4

I am working in an organization where I am focusing on improving the usability of enterprise applications (HR applications and other applications people are forced to use in order to complete their work).

What I am trying to find is a consistent way to show improvement from the current state, both in terms of satisfaction and time, which can be easily converted to money.

Additionally, measuring usability improvements with standard metrics is difficult for many of these applications because they require specific domain knowledge to complete tasks. For example, consider an app that only financial analysts use. How would I go about getting good time, task, success rate data on something that already exists, where all of the people I would test with already have the domain knowledge and system expertise to get their job done (even if it's challenging)? I can't just pull anyone and have them test it, because the system itself is in no way part of their job, and it never would be.

What approach would I take to show quantitative improvements on domain-specific applications, where the user base all know how to use the current version (and have developed necessary workaround to become proficient), while factoring in learnability of a new deployment to accomplish the same tasks?

  • 1
    Welcome to the site, @Bil4l! As far as I can tell, your post was asking three separate questions: (1) what method to use to usability test software limited to specific domain, (2) at what point you can tell whether results are valid, and (3) whether there are any standard measures of satisfaction or joy you can use to gauge improvement. Because a Q&A-format site works best when each question thread has one question, I’ve removed two of the three. I’d suggest posting them as separate questions and linking to this one. – Graham Herrli Jan 24 '15 at 18:10
2

I'm afraid some of what you are asking for simply does not exist. Satisfaction is an emotion. You cannot measure it with any precision. It's like saying "Fred loves Wilma 4.18 times more than Barny loves Betty". I know that it would make your job easier if you could just have numbers, but psychometry doesn't work that way.

There are also no shortcuts in measuring satisfaction. If you want to know how a certain group of users will react to a system, you have to apply traditional metrics, using the real users your participants. If it turns out that you need to pay financial analysts' time at their going rates, then your usability experiment will be terribly expensive, but there is no way around it.

For measuring pure satisfaction, there is a number of scales and measures you can use. SUS is very popular in industry, and it does end with a simple score. Others, such as DeLone McLean, QUIS or PSSUQ will also work, you might have to just average the separate items if you need a number. But you should note that in most of these instruments, the result precision is higher than their accuracy - that is, you end up with a number normalized to some interval, maybe a whole number if it's normalized between 1 and 100, but it doesn't mean that you can say for sure that a system which measures at 79 is better than a system which measures at 76. Also, if you repeat measurements with different groups of users, or with a different task, or in a different context, the numbers become even less comparable. You can measure the numbers, and this is the best you can do, but if you are doing small incremental changes only, or if your overall usability is either already very high or still very low, you're unlikely to see a nice clean trend. And this doesn't mean that you are failing in your job at making the application better, it just means that human emotions cannot be expressed in numbers.

Measuring effectiveness and effectivity is easier, as the numbers are much more comparable. Again, you need the real users, and you need to give them realistic tasks. If you do it any other way, you'll get a bunch of rather meaningless numbers.

Measuring learnability is harder than measuring the satisfaction of people who already know the application. You need to find a user group naive to your application (but who already has solid mental models in your domain, if your expected users have them), test it, then train them and test again. Then you compare the measures between the two tests. You should do this in different sessions, at least in the beginning, with extensive training and use between both. It only makes sense to measure progress within a single session if you are confident that the application is so simple that the users don't need more than a typical usability test session to learn it, or if you are really interested on the level of performance after brief exposure, not after long use. But the second is rare with applications written for professional use.

Grossman at al did a very nice survey of learnability measurements. I suggest reading the whole paper, but for completeness, here is a list of the learnability measures they found in literature:

Task Metrics: Metrics based on task performance 
T1. Percentage of users who complete a task optimally. 
T2. Percentage of users who complete a task without any help. 
T3. Ability to complete task optimally after certain time frame. 
T4. Decrease in task errors made over certain time interval. 
T5. Time until user completes a certain task successfully. 
T6. Time until user completes a set of tasks within a time frame. 
T7. Quality of work performed during a task, as scored by judges. 

Command Metrics: Metrics based on command usage 
C1. Success rate of commands after being trained. 
C2. Increase in commands used over certain time interval. 
C3. Increase in complexity of commands over time interval. 
C4. Percent of commands known to user. 
C5. Percent of commands used by user. 

Mental Metrics: Metrics based on cognitive processes 
M1. Decrease in average think times over certain time interval. 
M2. Alpha vs. beta waves in EEG patterns during usage. 
M3. Change in chunk size over time. 
M4. Mental Model questionnaire pretest and post test results. 

Subjective Metrics: Metrics based on user feedback 
S1. Number of learnability related user comments. 
S2. Learnability questionnaire responses. 
S3. Twenty six Likert statements. 

Documentation Metrics: Metrics based on documentation usage 
D1. Decrease in help commands used over certain time interval. 
D2. Time taken to review documentation until starting a task. 
D3. Time to complete a task after reviewing documentation. 

Usability Metrics: Metrics based on change in usability 
U1. Comparing “quality of use” over time. 
U2. Comparing “usability” for novice and expert users. 

Rule Metrics: Metrics based on specific rules 
R1. Number of rules required to describe the system. 

Grossman, T., Fitzmaurice, G., & Attar, R. (2009). A Survey of Software Learnability: Metrics , Methodologies and Guidelines. In Conference on Human Factors in Computing Systems (pp. 649–658).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.