6

I work for a company that provides complex software for users to create augmented reality experiences. Think Unity in a browser for AR. We would like to perform usability studies and I'm a bit stuck about exactly what to test and how to test it.

Our tool is pretty complex and is also very freeform i.e. there isn't really an order of operations they need to complete once they have created their project and the end result varies widely between customers. This is similar to something like Figma where the tool can be used for a wide variety of tasks and also requires some knowledge of the product to be able to effectively use it.

I thought about testing individual features, but am concerned that won't be useful without using them in the context of the full project e.g. testing the animation tool without knowing how that animation will work in the context of the full end experience

My questions:

  1. How do you define a usability study and/or task list for something so complex?
  2. How do you test this on users who have been using the software for a while and are now used to the product?

Thanks!

1
  • That's a surprisingly well presented UX question from a new member. Typically first questions are ultra specific to a problem, and not presented as a general UX question, but yours is spot on.
    – JohnGB
    Commented Apr 13, 2023 at 22:29

2 Answers 2

6

Your problem, as I understand it, is that due to the wide variety of tasks that customers may be using your product for, you are having a challenge in specifying tasks for your usability tests.

I your case, I would suggest that your second question presents a solution to your first. Spend time with your existing customers learning about how they use the product, and what sorts of tasks they typically perform or are interested in performing. You should also use the time to try understand what may make their lives easier, or which tasks they would like to perform but currently can't. The results from several of those types of interviews / observation sessions will give you a much better idea of the task lists that you should be testing with new users. You may also find that 90% of your users are using the product for one or two types of projects, which may help with strategic decisions on possibly focusing the product (or a version of it) on specific tasks. The important part of the new user testing is that you test the set of task lists that a given new user would likely to be doing, and not just any random tasks.

From there it's an iterative process I would make changes to test with existing users first, and then use that feedback to make other changes which I would test with new users, and that feedback would feed into changes to test with existing users.

2
  • Thanks John! This is an interesting take. While we do want to improve the experience for current users and new users, I think the focus is actually on usability for current users. Your suggestion to observe/interview them to help discovery of tasks for new users is interesting though. My issue then though is creating a task list that ticks all those boxes. Using the Figma example, would a good test for that be to simply say "create a prototype for a shopping app" and then let them go at it? Commented Apr 13, 2023 at 22:56
  • @JustinPumpr If creating a prototype for a shopping app is a good example of what existing users are doing, then that would be a good example. Remember, that there's nothing stopping you from using the feedback from existing users to also create tasks lists for other existing users. However, existing users aren't a great indicator of usability as any usability issues may have effectively selected that group for you, and so you have a bit of survivorship bias to account for.
    – JohnGB
    Commented Apr 13, 2023 at 23:03
4

I think there's a couple of approaches you could try, both using Jobs to be Done as the framework.

1. How are users doing with the intended uses of the product? What jobs did your leaders have in mind when they developed the software? If your product were a drill, you'd be watching users drill on different surfaces, but you'd also watch them change drill bits, switch a bit's direction, deal with difficult surfaces like masonry, etc. They'd think out loud and give feedback while performing the tasks.

In your product, you wouldn't need to test every feature with every testing participant. You could prioritize based on early feedback. For example, if you're getting a lot of phone calls around how difficult it is to set up payment acceptance, you could bring in testers and watch them set up a brand-new payment section from scratch, and report back areas where they're getting frustrated or confused. Also, are your users taking a long time with certain tasks because there's no "order of operations"? That would be something to look at documenting and improving.

2. How are users using your product in unintended ways? Because your product is so "freeform", it could be interesting to recruit a set of target users, sit them down in front of your system, and just have them explore it while thinking out loud. You could ask, "What do you think you're able to do here?" If they say something the system can do, you could follow with, "How do you think you'd do that? Can you show me?" If they say something that the system can't do, you could ask, "Is that something that you'd expect the system to do? Is it important for you?" Then you have some product feedback that could help inform the roadmap and make things less abstract.

2
  • 1
    Thanks Izquierdo. I'd definitely like to try and study how they are doing with the intended use of the product and that's the part I'm struggling with because there's a LOT of ways the users could achieve their goal. Maybe observing existing users without giving them a task list is a better way to start to see what the pain points are first? Commented Apr 13, 2023 at 22:51
  • I think that's a good place to start. Then you can see what questions you have from the observation. Every step they struggle with is an opportunity to make the next step more discoverable.
    – Izquierdo
    Commented Apr 14, 2023 at 15:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.