When user testing, how important are representative users?

Question

I work at a company where we design software that, in most cases, is targeted at leasing agents in the multi-family housing industry, a very specific group of people.

When I want to go about testing our UI with users, how important is it that I actually recruit leasing agents? The main issue here isn't necessarily finding them, but rather getting proper access to them (in other words, via their managers, which is slow and usually results in dead ends).

If I recruited more normal people "off the street", would the usability issues uncovered still hold? What might I be missing from a more representative group of users?

My goal is to make the recruiting effort easier if I can, because the amount of effort I'd need to put in currently is prohibitive to testing.

Answer 1

If I recruited more normal people "off the street", would the usability issues uncovered still hold? What might I be missing from a more representative group of users?

Some would. Some would not.

There are some issues that are pretty universal. If the text size is too small everybody will have problems reading it. If phone number field forces you to remove spaces and hyphens everybody will find that annoying.

What you'll miss is any domain related knowledge. For example things like:

• Yes this process is slightly opaque - but I need to do it N hundred times a day so faster is better than clearer
• Everybody knows what this acronym means
• Being able to take all this information in at a glance is more important than an aesthetic display with lots of whitespace
• This bit of information is really important to my problem, and this other bit isn't.
• ... and so on...

In short - testing with people "off the street" is better than not testing at all. But you will undoubtedly miss important issues that apply to people in your specific domain.

In one of the comments you talk about "friction" you encounter in the recruitment process. I think asking a question about ways to overcome that might be a more fruitful route.

Answer 2

In my honest opinion, it is best to test the app on a set of actual to-be users. This must be backed by administration. On the other hand, if the corporate mechanics are in they way, I'd personally find a to-be user and befriend him/her to do it on I-scratch-your-back-you-scratch-mine basis.

In any case, the tester must be familiar with the business logic and other specifics of the company/group-definition, thus you can't just put any random person to test the app. That would work only for user friendliness when the users are general public, but on a specific group of people you really need a real to-be user.

[EDIT] You'd be missing a number of factors if you don't use the representative group:

• Business logic specific important parts of the software won't be noticeable to normal users. Some parts will be used extensively by the real users and will need simplification and additional UI elements to ease the use, regular users won't notice the needed change because they don't know how much they would use the particular interface.
• For the same reason, will get better feedback from real users to change some specific parts, which in turn will let you know what you will need to focus on more.
• That feedback in return will also allow you to familiarize yourself more with business logic and day to day activity from the user, which will lead to a better software in the end.

I highly doubt that any usability issues uncovered by the "off the street" user will hold much value to the real users. At most it will change some few minor issues, but they won't uncover the real headaches that will come in production. "off the street" users will uncover some facts and you'd be happy fixing them, and you'd feel accomplishment which goes to waste when it comes to production.

But you know, I'm just basing this on my own experience.

[ENDEDIT]

Background: inner corporate application development.

Answer 3

If you can't get people from the real estate industry to even test the app, how are you going to get them to buy it? It sounds to me that the issues with your app might start up the chain a bit, particularly in what problems your app is trying to solve for these users. If you've approached them and explained how the app can help them and they've shown little interest, perhaps they don't have the problem you think they do?

Testing an app that you can't get in the hands of the actual consumers is what you'd call wastage in the lean startup world. Not to say that the app is wastage, because I don't know what it is or much about the real estate agent world, but you might have already learned a key thing and that is that your app isn't resonating with it's intended user base in it's current form.