8

I would like to test the usability of an iPad app released a few months ago.

This is an app that functions as a Point of Sale for retail businesses and cafes - there are at least two distinct user types.

There are features I know to be problematic and needing redesign, and I am also interested in the usability of simple main tasks.

I have been considering walkthroughs and talkalouds, but they seem to be better suited to pre-release stages.

I could work with both new and existing users, but budget restrictions are big and recording users on their busy cafe environments is not an option.

My questions are: What would be the optimal way of testing on this circumstances? How should I deal with different user types? How many tasks?

2
  • 3
    Why not consider observing those using your software instead of doing usability software? Are you in touch with people who use your software actively?
    – UXerUIer
    Mar 5, 2015 at 13:50
  • Cognitive Walkthroughs (aka Usability Testing) are perfectly fine for assessing released software. (An interface is an interface whether its finished or not). I think the question needs some refining to determine what the desired output from the research actually is.
    – PhillipW
    Mar 5, 2015 at 23:22

5 Answers 5

3

If direct interaction with the test participant is not possible without disrupting their actual work, I've done the following in a similar environment with front-line staff.

Pre-shift brief: Prior to their shift, make sure they know what's going on, are comfortable with everything and tell thing things to potentially make mental notes of to report or discuss afterwards.

Screen record: There are a few hacks that allow you to screen record on an iPad. Give some of them a try. I haven't personally used any on the iPad before.

Fly on the wall: Stand and look over the shoulder of the cashier. Usually best to have a cover story such as being an auditor. You can passively observe the software being used in its actual environment. Becareful not to get in the way or better yet be helpful. This is not always a possibility. I've had a need to usability test bank teller software and obviously it is not okay to have some guy standing next to the teller looking at their client's account info. If the environment is far too small or busy this can also be too disruptive

Selective or brief fly on the Wall: If there are certain aspects of the software that aren't often used (ie: refund), you can sit in a back room and ask them to come fetch you when this situation happens so that you can observe. You can also come out and observe at certain situation just to get snippets of observations rather than have to stand around all day. Make sure you pick representative times.

Post shift/task walkthrough: Interview immediately after their shift and have them dispense any mental notes they made of issues with the software, issues you observed, pros, etc. Best if they're able to walk through these on the software as you talk about them. Do NOT wait till the next day as you will lose a lot of detail.

You can use all or some of these depending on time and situation.

1
  • 1
    Agree. I would also add "Telemetry". Gather data about what a user does and when on the app. From this can (a) view workflow and see when it diverges from expected (b) build time-to-complete task picture and see where variance lies with a single user (are there outliers) between users (do some get stuck and others are ok) from target (e.g. order entry < 30 sec)
    – Jason A.
    Mar 6, 2015 at 10:40
0

You can test for the what and how using slightly different tactics and the best part is you'd only need about 5 users per test. Rubin's Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests has been a great resource of mine over the years. Also, usability.gov has templates to get you going. Since it sounds like your app is intended for a broad non-specialized audience, then representative users should do fine as proxy.

With regard to your issues:

  1. There are features I know to be problematic and needing redesign,
  2. and I am also interested in the usability of simple main tasks.

For the first, you could do a Comparison Test which is used to compare two or more different feature designs. Collect performance and preference data - think alouds work well. I am big fan of the RITE or RITE+Krug method, which means test, then tweak, then test again to see if you've fixed the big issues. You might try to even test lo-fi prototypes to see how other designs meet users' expectations.

For both (but mainly for the second issue): you could test how efficient the problematic feature is with tracking how long it takes someone to complete the task and what is the failure rate. You could do this with a stop-watch or use usertesting.com and time it that way. This is known as a Validation or Verification Test meant to measure performance criteria (efficiency or effectiveness).

Above all else, some user testing IS better than none. I've found that even just talking to a few people informally can be worth heaps.

Good luck!

-1

I would also opt to start with a short survey/interview with your users. This will generally give you feedback/information right away.

Don't focus on tasks, try to focus on time. I usually try to never go above 15-20 minutes in a test. I find that users get distracted pretty quickly.

-1

Usability Testing ISN'T doing a survey of the users or an ethnographic study.

It's testing the software to see whether users can use it (and it doesn't matter that its released software).

Exactly the same principals apply with mobile devices as with PCs - except with a mobile device you get a few extra complications about capturing what is going on on screen (and really you should be testing in a variety of physical environments to reflect mobility).

5
  • I am really sorry Phillip I had to down-vote because the OP needs to be aware of when to Use Which User-Experience Research Methods
    – Okavango
    Mar 5, 2015 at 14:46
  • 1
    it would also be good if you can elborate on why you don't agree with the use of ethnographic approach here and substantiate this with evidence and references.
    – Okavango
    Mar 5, 2015 at 14:55
  • User TESTING is a well defined approach which has been around for a long time and is given on the link in the answer. You test to tasks. Ethnographic research is a much more open approach where you concentrate on trying to 'be the user'. It's not constrained to doing specific tasks. If the question is about 'User Research' then that's a much more open question
    – PhillipW
    Mar 5, 2015 at 23:19
  • Your answer lacks any reference to the context of the question: 1st - The OP didn't ask what usability testing is but how to conduct testing under specific set of circumstances. 2nd - testing will be conducted in a collaborative work environment so understanding the test environment is key to understanding how the software is used, a number of answers here highlighted the need for observation (including mine) so ethnography is a valid approach. 3rd - Your understanding of what ethnography is flawed and incomplete, check my answer for references!
    – Okavango
    Mar 6, 2015 at 7:11
  • The heading on the question asks about "Usability testing for existing software". I've linked to a reputable source on Usability Testing. It does not describe ethnographic research. The major difference between what we are talking about is I'm using a lab and your approach suggests doing it in situ.
    – PhillipW
    Mar 6, 2015 at 13:55
-1

1. Document current user interface:

There are features I know to be problematic and needing redesign, and I am also interested in the usability of simple main tasks.

This documentation and any assumptions you have will act as a point of reference when going through to the test stage. This being said, conducting usability testing in the workplace requires good understanding of how users work and collaborate to achieve tasks. So, gaining that knowledge is a prerequisite to capture how they use tools or software that assist them in doing their work. which takes us to (step 2)

2. Describing the scene :

Describe the physical aspects of the work environment, including the layout of workstations, desk space and clutter, collaboration and conversation areas.

Because testing will be conducted in a collaborative work environment, understanding the test environment, its limitations, constraints and flows make it easier to undertsand how users do their job. This is intended to give you High-level information that could prove useful down the line.

3. Observe and note user actions:

Reconstruct user/users action through observation and note taking:

What happened, and who did what? What is your impression of these incidents, and what are team members thoughts in regard to and interpretations of these events?

source: Ethnography in UX - UX Matters

This will help you gain better understanding of user actions in context (scene description would be useful here) while actively looking for patterns and making assumptions (hypothesis) check if these line-up with any assumptions you have made so far.

Tips:

  • Choose best time to observe while being the least disruptive to them, don't stand in the way :)
  • Record your thoughts, observations on your phone this is easier and makes you less noticeable!

4. Interview users while going through problematic tasks:

Contextual inquiry (CI) is a user-centered design (UCD) ethnographic research method, part of the Contextual Design methodology. A contextual inquiry interview is usually structured as an approximately two-hour, one-on-one interaction in which the researcher watches the user do their normal activities and discusses what they see with the user.

Source: Wikipedia

Once you have gathered enough information through the above steps you could interview users to gain insights about their pain-points, rule out assumptions and refine your test hypothesis further. at this stage recording user actions is ready to go forward:

5. Preparation for testing: facilitation and equipment:

There is a number of solutions that you could employ here, for example: a dedicated software to capture video on iPad which combined with observation and short structured task driven interviews (Contextual Inquiry) could yield good results to address the issues raised.

One of the solutions I have seen allows you to capture the user’s face via the camera, their audio through the mic and their on-screen activity (gestures,taps, swipes and scrolling)

Overall, how you test and what time you dedicate to the task will entirely depend on you and how you have planned for it. This might seem like a long list but I think its achievable.

Related: What's the term for modelling detailed customer behaviour in service/UX design?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.