Could I validate the user experience objectively from paper prototypes?

Question

I have paper prototypes for an application I am about to develop. I need to choose from alternatives. Is there any technique or methodology available to validate the different prototypes?

Answer 1

I just want to point out a couple of clarifications due to the objectivity you are concerned about. "Objectivity" is such a trigger word for most academics, you know. Because it is so difficult to really obtain objectivity.

Data colletion
First of all, think about the objectivity/subjectivity of the data you are collecting. Is the data you are collection objective or subjective. If you ask the participant "how did you feel when you saw that logo" or "how experienced are you with the Internet", then both of these statements are subjective. Both of these reflect the individual participant's subjective perceptions of the issue. If you want objective data, then you need formulate other questions or use other techniques than interview/questionnaire. You could measure hart rate, sweating, pupil-contraction, eye-tracing etc, or you can ask the user "objective" questions like "how often do you do tasks on the Internet" and "what kind of tasks do you do". This will give you objective data where the user don't need to evaluate the answer he/she gives.

Analysis
Then you have the analysis and interpretation of the data you collected. Judging individual or small datasets will be a subjective analysis of the information. Performing statistical analysis on the data will give you an objective analysis (that most often will be the basis for some subjective conclusion/decision).

The combination of collected data and analysis is interesting. If 30 users says that they feel good and 5 says they don't care, then the individually collected data is subjective, but the overall conclusion is objective. "30 of 35 users feels good".

Now, bear in mind how you word this conclusion. On the Internet experience example, it is a huge difference between the statements "80% of the users are experienced Internet users" and "80% of the users consider themselves to be experienced". The conclusion must reflect the actual underlying data.

Statistics
Statistics and analysis is a huge field, and it is important to know a little bit of this when you are doing statistical analysis. Statistic is all about confidence. How much error are you willing to accept. Given a 80% success rate, you'll have these confidence intervals for three different sample sizes.
- 4 out of 5 gives you a 95% confidence in the range 36%-98%.
- 16 out of 20 gives you a 95% confidence in the range 58%-95%.
- 80 out of 100 gives you a 95% confidence in the range 71%-86%.
(Eg. You can say with 95% confidence that somewhere between 36% and 98% of the larger population will be able to successfully complete the task. (Source, Tullis/Albert)

So, objectivity is very difficult.

---

Now, back to your question. Based on "I have paper prototype", "about to develop", "choose alternative" and "is there any...", I believe that objectivity is not what you really need. You don't need quantitative data, you need qualitative feedback so that you can make the right design decision. You don't event need to choose the best alternative, you can actually pick the best from both prototypes and generate a third alternative. Also known as parallel design.

You have chosen a very wise first step on your journey: Involve the user early. For further success: Evaluate often.

What you need to remember is this:

You need to know what you are looking for.
What is the purpose of the forthcoming research.

When you have clarified this, only your imagination will limit the opportunities you have.

With users, you can combine interview, questionnaire and empirical testing. Choose individuals, pairs of users or larger groups depending on the information you need and the group dynamic you want to achieve.

The material you use is not important. It could be sketches, prototypes, fully functional products, competing websites or even an "analog" moleskin.

Some methods you get you started:

• Perform a standard user test with 3-5 individual users. Ask them to perform some simple tasks and ask for pros and cons with each design. Remember to alternate which prototype you start with.
• Pair-testing. Put two particitamts together and let them dicuss the prototypes. You can give them both prototypes side by side immedeately, and take note of which propotype each pair start with.
• Workshop. Invide a group of 5 and perform an open evauation of the products.

All of these will give you a lot of feedback on your design.

Answer 2

I believe small group testing like that is great for exploration and terrible for validation. The fact hat 4 out of 6 total people liked version A vs. verson B should never mean that you go with version A. You have nearly zero statistical confidence. However, in exploration you might find places to improve or new ides that make it better or even a completely new approach.

Paper prototypes should be used because they are quick to produce and you can get feedback cheaply. Use them to learn, not to decide.

Answer 3

Yes, there is. Find about 5 people and have them pretend to use the prototypes. Take careful notes.

Answer 4

One thing to think about when you make paper prototypes is the fidelity. By keeping things rough, you'll get users to focus on broad, conceptual issues. This is why it can even be a distractor to use other design software (such as Photoshop) because your users will start focusing more on things like color and font, which aren't important in early stages.

On the flip side, you have to check to see if your paper prototype lacks any usability features that the real thing won't. i.e., if you're doing a website, do your links look like links on paper?

Bottom line is, before you run any usability tests, ask yourself what you're trying to get out of the test and what types of questions you're trying to answer. From there, you can decide if a paper prototype is right for you.

Once you've got your prototypes, there are a number of different methods you can use: cognitive walkthrough, verbal protocol, usability assessment, activity analysis (to name a few)

Answer 5

Not related to user experience, I just wanted to draw attention to the GOMS technique (Goals, Operators, Methods, Selection rules). It is a somewhat simplistic method to compare the efficiency of alternative solutions.

As I said: not related to user experience per se, but could be usable in order to compare different flows etc. Be sure to read to The Keystroke-Level Model for User Performance Time With Interactive Systems by Card, Moran, and Newell (1980, Communications of the ACM, July, 23(7), 396-410) before you start out.