9

I have been assigned a project where I need to start doing Salesforce usability testing with a customer service department. My issue is that I've never tested an application that will require user training. I generally have an application where the user won't get training, so I give them a task and tell them to show me what they would do. Pretty easy.

So I've been looking for guidelines on how to accomplish testing with an application that will require user training but haven't had much luck. My thought process is divided between these three approaches:

  • I could give them a task without any training and see where they go, just to see what they do. But my concern is that Salesforce is so large I'm not sure my findings would actually be helpful.

  • I could require a quick overview a week (not sure on the best time here) before the testing and then pull them in to see how they do. But how much training is too much training?

  • I could run a test or two and just have them run wild, then test with trained users, then switch back to new users, and so on.

I'll be testing once a week for the next three months so I don't have a ton of time before this project launches, but I do have a bit of wiggle room.

Does the UX Stack Exchange community have any advice on this or any documentation/guides/articles/books that might be able to help me plan my attack?

8
  • It sounds like you need to go back to the customer and ask them what they want you to do relative to the training requirement. Mar 28, 2016 at 19:24
  • 2
    You don't mention if there is already training material available for on boarding this particular user population. If materials exist, what about testing new users on the efficacy of the training materials relative to the critical tasks they must understand to effectively support their customers? Perhaps the usability testing results will speak to a need to adapt the training materials. Mar 28, 2016 at 19:40
  • 1
    I feel like it's inherently flawed to have training on a piece of software. If people need to memorize how actions are being done, then the software itself isn't good. Maybe build from there? Ask the users what the pain points are?
    – UXerUIer
    Mar 28, 2016 at 21:01
  • 1
    @Majo0od this would be naive to say. Would you say training is not necessary for systems like electronic medical records? or flights control tower? It doesn't mean users need to memorize the the actions, but they do need some explanations how the system works and how to operate it.
    – Dvir Adler
    Jun 23, 2016 at 4:51
  • What you are looking for are flaws in the software for which a solution can't be learnt. Eg missing functionality.
    – PhillipW
    Jun 23, 2016 at 7:08

5 Answers 5

6
+150

As with all forms of testing, you really need to establish a clear objective or outcome that you want to achieve from doing the testing. If you don't (and the client doesn't give you any directions on this) then the test results won't really mean anything.

From my understanding of the situation, an application that will require training means that there are lots of factors involved, whether it is due to a diverse range of users, complex transactions and features, compliance requirements, etc. That means you need to probably figure out which parts of the application the training is addressing, and which parts of the application should be intuitive and require very little or no training.

Your first approach of giving them no training and see what the users do is probably a good way to identify areas where training is required, but is this what you are really testing for?

Your second approach of a quick overview is good, but you will probably need to compare people who have received the overview versus no overview to see the difference that some instructions/training provides, but is that what you are really testing for?

Your third approach appears to be an iterative and agile way of developing an optimal method of doing the testing, but if you worked out what you need to test for with the client then you don't need to be spending time doing this.

I think going back to first principles is the key here. You are looking at a couple of variables, these being the different types of users (this could be based on their roles and experience with the application), the amount of training they receive (this could be time or type of training received) then seeing how this impacts on the successful completion of tasks.

If you don't have a lot of time, then you need to prioritize the testing activities based on the users groups that you select by number (i.e. volume of transactions or interactions with the application) or the importance of the task they do (hence the risk of not receiving proper training), and then cherry pick the tasks on the same principle and get the client to agree with the list. I think if you can justify your approach and decision with the testing in accordance with the outcome you want to achieve, that will be the best practice rather than following any set standards or guidelines.

3

I don't think training is a good idea, as you will miss a lot of feature testing and mental model validations. If the domain is new to the users some level of domain familiarization is required, not UI walkthrough. If they are new to the target device or new to computing itself, then you might need to provide a walkthrough of things work or design patterns used.

You could break the usability testing into different categories - Beginners, experienced and advanced users. Testing the users in phases.

Phase 1 - Beginner: You give them a goal or task and ask them to figure out how to achieve it using the applications. When they are stuck at any point guide guide them by progressively giving solutions (first you give a possibility, then you give the actual solution). The first phase should not be complex and aimed at revealing the functionality to the user.

Phase 2 (same users) - Provide more complex tasks to the user, this time around it should be unaided as much as possible. The more help you need to provide more the functionality is failing.

Phase 3 (same users) - Provide a complete end-to-end task and measure the features that are not obvious being completed and discovered by the users.

3

Training ensures employees beat the learning curve. The client knows it requires training and they know it won't go smooth. You're there to help identify those growing pains, and to tell them how to make the whole process go smoother.

What I would do:

  1. Establish tasks around the main goals of the company

  2. Interviews the users before, during and after a period parallel to their training (1 or 2 weeks).

  3. As the training progresses they will quickly run into issues and lessons, forcing them to learn and adapt. Your test has to match that curve.

Employing static testing methods will cause polluted data because the user interactions will change during the period of training.

This will need iterative cooperation and input from the trainer and the employees being trained. Best of luck :)

1

Before you can do a usability test, you must think about what you want to get out the test. Not all usability testing requires the user to be trained.

For example, if you are interested in whether the user interface is obvious or not, then you can use absolutely anyone as your participant and no training is required. All you then need to do is design a test script which gets the participant to where you need them to be, then define how you are going to measure the obviousness of your solution (remember to use quantitative and qualitative measures). Don't make your instructions too specific because the goal of this test would be to see how obvious your solution feature are.

0

Short answer: Show them the basics.

Long answer: It depends.

I think the correct way is to show them what "new people" first using this system will actually know.

If there is a whole tutorial for all functions, super detailed, then, you should tell them the whole tutorial for all functions, super detailed.

If there is no tutorial at all, you should be telling them nothing at all.

If there are few things, you should tell few things.

This is how every people should do this. But only in my opinion.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.