Testing with an application that will require user training

Question

I have been assigned a project where I need to start doing Salesforce usability testing with a customer service department. My issue is that I've never tested an application that will require user training. I generally have an application where the user won't get training, so I give them a task and tell them to show me what they would do. Pretty easy.

So I've been looking for guidelines on how to accomplish testing with an application that will require user training but haven't had much luck. My thought process is divided between these three approaches:

• I could give them a task without any training and see where they go, just to see what they do. But my concern is that Salesforce is so large I'm not sure my findings would actually be helpful.

• I could require a quick overview a week (not sure on the best time here) before the testing and then pull them in to see how they do. But how much training is too much training?

• I could run a test or two and just have them run wild, then test with trained users, then switch back to new users, and so on.

I'll be testing once a week for the next three months so I don't have a ton of time before this project launches, but I do have a bit of wiggle room.

Does the UX Stack Exchange community have any advice on this or any documentation/guides/articles/books that might be able to help me plan my attack?

Answer 1

As with all forms of testing, you really need to establish a clear objective or outcome that you want to achieve from doing the testing. If you don't (and the client doesn't give you any directions on this) then the test results won't really mean anything.

From my understanding of the situation, an application that will require training means that there are lots of factors involved, whether it is due to a diverse range of users, complex transactions and features, compliance requirements, etc. That means you need to probably figure out which parts of the application the training is addressing, and which parts of the application should be intuitive and require very little or no training.

Your first approach of giving them no training and see what the users do is probably a good way to identify areas where training is required, but is this what you are really testing for?

Your second approach of a quick overview is good, but you will probably need to compare people who have received the overview versus no overview to see the difference that some instructions/training provides, but is that what you are really testing for?

Your third approach appears to be an iterative and agile way of developing an optimal method of doing the testing, but if you worked out what you need to test for with the client then you don't need to be spending time doing this.

I think going back to first principles is the key here. You are looking at a couple of variables, these being the different types of users (this could be based on their roles and experience with the application), the amount of training they receive (this could be time or type of training received) then seeing how this impacts on the successful completion of tasks.

If you don't have a lot of time, then you need to prioritize the testing activities based on the users groups that you select by number (i.e. volume of transactions or interactions with the application) or the importance of the task they do (hence the risk of not receiving proper training), and then cherry pick the tasks on the same principle and get the client to agree with the list. I think if you can justify your approach and decision with the testing in accordance with the outcome you want to achieve, that will be the best practice rather than following any set standards or guidelines.

Answer 2

I don't think training is a good idea, as you will miss a lot of feature testing and mental model validations. If the domain is new to the users some level of domain familiarization is required, not UI walkthrough. If they are new to the target device or new to computing itself, then you might need to provide a walkthrough of things work or design patterns used.

You could break the usability testing into different categories - Beginners, experienced and advanced users. Testing the users in phases.

Phase 1 - Beginner: You give them a goal or task and ask them to figure out how to achieve it using the applications. When they are stuck at any point guide guide them by progressively giving solutions (first you give a possibility, then you give the actual solution). The first phase should not be complex and aimed at revealing the functionality to the user.

Phase 2 (same users) - Provide more complex tasks to the user, this time around it should be unaided as much as possible. The more help you need to provide more the functionality is failing.

Phase 3 (same users) - Provide a complete end-to-end task and measure the features that are not obvious being completed and discovered by the users.

Answer 3

Training ensures employees beat the learning curve. The client knows it requires training and they know it won't go smooth. You're there to help identify those growing pains, and to tell them how to make the whole process go smoother.

What I would do:

1. Establish tasks around the main goals of the company

2. Interviews the users before, during and after a period parallel to their training (1 or 2 weeks).

3. As the training progresses they will quickly run into issues and lessons, forcing them to learn and adapt. Your test has to match that curve.

Employing static testing methods will cause polluted data because the user interactions will change during the period of training.

This will need iterative cooperation and input from the trainer and the employees being trained. Best of luck :)

Answer 4

Before you can do a usability test, you must think about what you want to get out the test. Not all usability testing requires the user to be trained.

For example, if you are interested in whether the user interface is obvious or not, then you can use absolutely anyone as your participant and no training is required. All you then need to do is design a test script which gets the participant to where you need them to be, then define how you are going to measure the obviousness of your solution (remember to use quantitative and qualitative measures). Don't make your instructions too specific because the goal of this test would be to see how obvious your solution feature are.

Answer 5

Short answer: Show them the basics.

Long answer: It depends.

I think the correct way is to show them what "new people" first using this system will actually know.

If there is a whole tutorial for all functions, super detailed, then, you should tell them the whole tutorial for all functions, super detailed.

If there is no tutorial at all, you should be telling them nothing at all.

If there are few things, you should tell few things.

This is how every people should do this. But only in my opinion.