Measuring UX of big enterpise application

Question

I'd like to measure usability and identify potential areas for UX improvements of rather big application. We've managed to identify 3 main user groups (based on roles) and around 3-4 main tasks performed by each of these groups. The app has around 200-300 users in total.

1st thing I'd like to do is to send a questionnaire to all users- but I don't like the idea of asking about the overall usability (e.g. with System Usability Scale (SUS) questionnaire).

I'd like to measure usability of each of the tasks for each of the roles so that when results come I know that for instance role 1 views task 2 as hard to perform or unusable, but I want to avoid asking same questions about each of the tasks (sending a SUS for each of the tasks). How should I approach this? Is it acceptable to send one for each of the tasks? How would you perform such project?

Answer 1

Reading your question, I think you need three factors:

• Which tasks do the roles perform and how often - for a later priorization.
• How usable they see the application.
• What is their top painpoint - for getting qualitative insights.

Let them mark your identified tasks and an "others-task" and let them priorize them. Which is the most valuable for them?

Ask the SUS questionnaire to get a benchmark where your app is. This is important for management communication. They like numbers. After your optimization take an other SUS and ideally it will be better. Thus you get your effort measureable and communicable. This is important for your standing.

Ask an open quesrion in your questionnaire like "If you could, what would you change first in the application?" Here you get ideas, what painpoints are worst.

I assume it is a B2B-app, so it will be easier to recruit participants because you know them. There are always feedback-rich answers. Ask this respondents if they are willing for an online usability test. I usually use Webex for screensharing and phone for speaking. Do a think-aloud formative test with them performing your identified tasks and collect insights. After the test ask for suggestions and ideas to optimize - there may be some quickwins for you.

After your test you will have a huge amount of data and recommendations. Now you will be lucky to have your collected priorization of your questionnaire, because you know which role see what task as top priority. With this backup you can give your development valueable insights, what feature should optimized first because they have most impact for your users.

Answer 2

Look for the point of least astonishment

I really like this tweet from Jared Spool...

How much user research do you need?

Ideally, you continue until you reach the Point of Least Astonishment.

Jared Spool describes this method of user research and design iteration by being able to guess what a user will do before they do it.

---

Here are the basic steps to this approach...

1. Allow customers from each of the groups you have identified to sign up for a chance to test drive new features and help decide the future of the product.

2. Decide on a list of primary tasks that are vital for each role to complete in a timely manner.

3. Have designers, project managers, developers independently come up with an expected path that users will take to complete each of the tasks.

4. Watch users complete each of the tasks and be sure not to guide them in any way. Assure them that you will be able to answer questions at the end of the session and that during the session it is more helpful if they explain what they are doing and what they think should be happening if they get stuck. Record these sessions.

5. Compare the expected results from step 3 with the actual results in step 4. If they reasonably line up then congratulations you have reached the point of least astonishment! (otherwise you need to adjust your design and iterate through these steps again)

Answer 3

There's a couple ways you could approach this in my opinion:

• Segment your roles and tasks so that each person surveyed gets 1 SUS, but you cover all of your tasks/roles equally for each.
• Send SUS for a role's most frequently done task to a group, and do user interviews from that group to analyze the other tasks.

Answer 4

I'm not a fan of questionnaires and trying to make usability measurable. Usability is subjective, you can't measure that.

Check out the book "Don't make me think" by Steve Krug. And do a usability test with a few (3) users. Just put them in front of their own computer and ask them to perform the task. Observate how different they do the task from how you would do it. It's an eye opener to see someone work with your application and if you can record and replay the test, it might be an eye opener for the people who asked to measure usability. I used to note how many mistakes the testee made and how long it took to fullfill the task. But I've noticed that the data is quite useless except for making pretty charts so somebody thinks it's quantifiable.