Formative Usability Testing - Script

Question

I'm running my first real usability test. I am doing this for an internal, enterprise application that will eventually be offered as a SasS.

I've recruited a small user group which literally account for 100% of our national users. (It will be an international product for a very niche industry.)

I've done a task analysis and have had multiple opportunities for contextual observation.

I've analyzed their current system, interviewed current users and reviewed analytics and business data to better understand user and business needs.

I've created a medium fidelity, semi-interactive prototype to run some initial, formative usability test.

My question is: how important is the script?

I've created a small script of tasks. I.e)

1)Login, search for Customer "X" by Name.

2)Identify customer's mailing address.

3)Find navigation instruction to customer's address.

Although these tasks are based off of a true representation of day-to-day tasks a user does, my understanding is that a Usability Test follows much more of a scripted version.

Is there anything wrong with doing a task based script if it is a true representation of a users day-to-day workflow?

Do I really needs to say things like "You are working for XXXX, a customer phones in. You need to locate this customer in the system." Or can I just give the user very small, specific tasks.

I'm asking because this, being my first time, makes me uncomfortable. I would rather give them simple tasks and analysis. But at the same time, if this will skew my results, I'll take the proper time to write a script.

What do you think? What has your experience been.

Answer 1

Below are some answers to different parts of your question w/ some tips:

1. A script is important because the way you ask can often be misleading or create bias. As a result, it's a good idea to run a pilot test, see what you get back, and refine your script as needed.

2. A script for this kind of test should start with contextualizing what you mentioned: "You are working for XXXX, a customer phones in..." You want to contextualize users as close to a real-life scenario as possible—and not assume anything.

3. A task based script is fine, but you should ask people to THINK OUT LOUD as they are performing the task. Follow up w/ questions relative to the task such as 'Do task XXXX. Was anything difficult or unclear? Why?'

4. Your user testing script will vary depending on whether you are running the test moderated or unmoderated (ie with you live with the user participant or not). For moderated, it's good to be loose with the script and ask lots of open ended follow up questions. For unmoderated, most the of same rules apply, but with more attention to wording of questions.

Here's a refined script based on the one you posted:

Imagine you are working for XXXX, a customer phones in. You need to locate this customer in the system. THINK OUT LOUD what you see, how you feel, and what catches your eye or interest as you go through each of the tasks below. 

1)Try logging in and searching for Customer "X" by Name. Anything unclear, unexpected or difficult? Why? 

2)Identify customer's mailing address. Think out loud as you try to do this.

3)Find navigation instruction to customer's address.

You can find more about this in an article I wrote here.

Answer 2

Adding a few more points to the answer by Junu :

The script is important because you are doing it for the first time and none of us are super humans to remember all steps in a test. Have a detailed script and print it out for your reference.

Apart from the tasks, you also have to start with few things like asking for their consent to record the session(If you are recording, which you should), make it clear to the user that the recordings will not be shared outside the design team and just for note taking purposes. Once you are done with the tasks, ask them few subjective questions and wrap it up with thanking them for their time and tell them you are looking forward to similar sessions in the future.