Usability testing for existing software

Question

I would like to test the usability of an iPad app released a few months ago.

This is an app that functions as a Point of Sale for retail businesses and cafes - there are at least two distinct user types.

There are features I know to be problematic and needing redesign, and I am also interested in the usability of simple main tasks.

I have been considering walkthroughs and talkalouds, but they seem to be better suited to pre-release stages.

I could work with both new and existing users, but budget restrictions are big and recording users on their busy cafe environments is not an option.

My questions are: What would be the optimal way of testing on this circumstances? How should I deal with different user types? How many tasks?

Answer 1

If direct interaction with the test participant is not possible without disrupting their actual work, I've done the following in a similar environment with front-line staff.

Pre-shift brief: Prior to their shift, make sure they know what's going on, are comfortable with everything and tell thing things to potentially make mental notes of to report or discuss afterwards.

Screen record: There are a few hacks that allow you to screen record on an iPad. Give some of them a try. I haven't personally used any on the iPad before.

Fly on the wall: Stand and look over the shoulder of the cashier. Usually best to have a cover story such as being an auditor. You can passively observe the software being used in its actual environment. Becareful not to get in the way or better yet be helpful. This is not always a possibility. I've had a need to usability test bank teller software and obviously it is not okay to have some guy standing next to the teller looking at their client's account info. If the environment is far too small or busy this can also be too disruptive

Selective or brief fly on the Wall: If there are certain aspects of the software that aren't often used (ie: refund), you can sit in a back room and ask them to come fetch you when this situation happens so that you can observe. You can also come out and observe at certain situation just to get snippets of observations rather than have to stand around all day. Make sure you pick representative times.

Post shift/task walkthrough: Interview immediately after their shift and have them dispense any mental notes they made of issues with the software, issues you observed, pros, etc. Best if they're able to walk through these on the software as you talk about them. Do NOT wait till the next day as you will lose a lot of detail.

You can use all or some of these depending on time and situation.

Answer 2

You can test for the what and how using slightly different tactics and the best part is you'd only need about 5 users per test. Rubin's Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests has been a great resource of mine over the years. Also, usability.gov has templates to get you going. Since it sounds like your app is intended for a broad non-specialized audience, then representative users should do fine as proxy.

With regard to your issues:

1. There are features I know to be problematic and needing redesign,
2. and I am also interested in the usability of simple main tasks.

For the first, you could do a Comparison Test which is used to compare two or more different feature designs. Collect performance and preference data - think alouds work well. I am big fan of the RITE or RITE+Krug method, which means test, then tweak, then test again to see if you've fixed the big issues. You might try to even test lo-fi prototypes to see how other designs meet users' expectations.

For both (but mainly for the second issue): you could test how efficient the problematic feature is with tracking how long it takes someone to complete the task and what is the failure rate. You could do this with a stop-watch or use usertesting.com and time it that way. This is known as a Validation or Verification Test meant to measure performance criteria (efficiency or effectiveness).

Above all else, some user testing IS better than none. I've found that even just talking to a few people informally can be worth heaps.

Good luck!

Answer 3

I would also opt to start with a short survey/interview with your users. This will generally give you feedback/information right away.

Don't focus on tasks, try to focus on time. I usually try to never go above 15-20 minutes in a test. I find that users get distracted pretty quickly.

Answer 4

Usability Testing ISN'T doing a survey of the users or an ethnographic study.

It's testing the software to see whether users can use it (and it doesn't matter that its released software).

Exactly the same principals apply with mobile devices as with PCs - except with a mobile device you get a few extra complications about capturing what is going on on screen (and really you should be testing in a variety of physical environments to reflect mobility).

Answer 5

1. Document current user interface:

There are features I know to be problematic and needing redesign, and I am also interested in the usability of simple main tasks.

This documentation and any assumptions you have will act as a point of reference when going through to the test stage. This being said, conducting usability testing in the workplace requires good understanding of how users work and collaborate to achieve tasks. So, gaining that knowledge is a prerequisite to capture how they use tools or software that assist them in doing their work. which takes us to (step 2)

2. Describing the scene :

Describe the physical aspects of the work environment, including the layout of workstations, desk space and clutter, collaboration and conversation areas.

Because testing will be conducted in a collaborative work environment, understanding the test environment, its limitations, constraints and flows make it easier to undertsand how users do their job. This is intended to give you High-level information that could prove useful down the line.

3. Observe and note user actions:

Reconstruct user/users action through observation and note taking:

What happened, and who did what? What is your impression of these incidents, and what are team members thoughts in regard to and interpretations of these events?

source: Ethnography in UX - UX Matters

This will help you gain better understanding of user actions in context (scene description would be useful here) while actively looking for patterns and making assumptions (hypothesis) check if these line-up with any assumptions you have made so far.

Tips:

• Choose best time to observe while being the least disruptive to them, don't stand in the way :)
• Record your thoughts, observations on your phone this is easier and makes you less noticeable!

4. Interview users while going through problematic tasks:

Contextual inquiry (CI) is a user-centered design (UCD) ethnographic research method, part of the Contextual Design methodology. A contextual inquiry interview is usually structured as an approximately two-hour, one-on-one interaction in which the researcher watches the user do their normal activities and discusses what they see with the user.

Source: Wikipedia

Once you have gathered enough information through the above steps you could interview users to gain insights about their pain-points, rule out assumptions and refine your test hypothesis further. at this stage recording user actions is ready to go forward:

5. Preparation for testing: facilitation and equipment:

There is a number of solutions that you could employ here, for example: a dedicated software to capture video on iPad which combined with observation and short structured task driven interviews (Contextual Inquiry) could yield good results to address the issues raised.

One of the solutions I have seen allows you to capture the user’s face via the camera, their audio through the mic and their on-screen activity (gestures,taps, swipes and scrolling)

Overall, how you test and what time you dedicate to the task will entirely depend on you and how you have planned for it. This might seem like a long list but I think its achievable.

Related: What's the term for modelling detailed customer behaviour in service/UX design?