How to perform usability studies on complex software

Question

I work for a company that provides complex software for users to create augmented reality experiences. Think Unity in a browser for AR. We would like to perform usability studies and I'm a bit stuck about exactly what to test and how to test it.

Our tool is pretty complex and is also very freeform i.e. there isn't really an order of operations they need to complete once they have created their project and the end result varies widely between customers. This is similar to something like Figma where the tool can be used for a wide variety of tasks and also requires some knowledge of the product to be able to effectively use it.

I thought about testing individual features, but am concerned that won't be useful without using them in the context of the full project e.g. testing the animation tool without knowing how that animation will work in the context of the full end experience

My questions:

1. How do you define a usability study and/or task list for something so complex?
2. How do you test this on users who have been using the software for a while and are now used to the product?

Thanks!

Answer 1

Your problem, as I understand it, is that due to the wide variety of tasks that customers may be using your product for, you are having a challenge in specifying tasks for your usability tests.

I your case, I would suggest that your second question presents a solution to your first. Spend time with your existing customers learning about how they use the product, and what sorts of tasks they typically perform or are interested in performing. You should also use the time to try understand what may make their lives easier, or which tasks they would like to perform but currently can't. The results from several of those types of interviews / observation sessions will give you a much better idea of the task lists that you should be testing with new users. You may also find that 90% of your users are using the product for one or two types of projects, which may help with strategic decisions on possibly focusing the product (or a version of it) on specific tasks. The important part of the new user testing is that you test the set of task lists that a given new user would likely to be doing, and not just any random tasks.

From there it's an iterative process I would make changes to test with existing users first, and then use that feedback to make other changes which I would test with new users, and that feedback would feed into changes to test with existing users.

Answer 2

I think there's a couple of approaches you could try, both using Jobs to be Done as the framework.

1. How are users doing with the intended uses of the product? What jobs did your leaders have in mind when they developed the software? If your product were a drill, you'd be watching users drill on different surfaces, but you'd also watch them change drill bits, switch a bit's direction, deal with difficult surfaces like masonry, etc. They'd think out loud and give feedback while performing the tasks.

In your product, you wouldn't need to test every feature with every testing participant. You could prioritize based on early feedback. For example, if you're getting a lot of phone calls around how difficult it is to set up payment acceptance, you could bring in testers and watch them set up a brand-new payment section from scratch, and report back areas where they're getting frustrated or confused. Also, are your users taking a long time with certain tasks because there's no "order of operations"? That would be something to look at documenting and improving.

2. How are users using your product in unintended ways? Because your product is so "freeform", it could be interesting to recruit a set of target users, sit them down in front of your system, and just have them explore it while thinking out loud. You could ask, "What do you think you're able to do here?" If they say something the system can do, you could follow with, "How do you think you'd do that? Can you show me?" If they say something that the system can't do, you could ask, "Is that something that you'd expect the system to do? Is it important for you?" Then you have some product feedback that could help inform the roadmap and make things less abstract.