What are the Pros / Cons of iterating on a prototype after each usability test session?

Question

I run a lot of guerilla usability tests on my prototypes at my company. Sometimes, after maybe the second or third participant runs into the same issue, I'll think about fixing that issue before testing with the next participant.

If I fix the issue, I'll immediately be able to test another design that could be better. On the other hand, I'm potentially skewing consistency in my results.

Will the usability of the final design be higher if I update the prototype after only a few participants? Or, should I refrain from updating any prototypes / mockups until all 4-6 participants have been tested?

Before posting this, I found "The RITE way to prototype" which loosely describes this process, but doesn't provide any quantitative evidence on the outcomes. I haven't found anything else, and wanted to see what others have done in similar circumstances.

Answer 1

Go for it.

Guerilla testing is all about rapid iterative prototyping. It's part of the creative design process, and not a quantitative scientific endeavour. Apart from the obvious advantage, of reducing risk of critical errors, Pros include

• Being able to defend your design decision with stakeholders (less chance of design by committee)
• Learning more about your users' environment, and generally getting more experience of user-focused design
• Designing in a 'safe' environment that allows risk taking (you may get a better design as a result)

Cons include... because you're not doing formal analysis, you can delude yourself that a feature is working / not working. This happened to me recently. I swore that users were rejecting a particular feature, because I was convinced that they would. It was only after going over the recordings and doing an analysis later that I realised I was deluding myself.

This isn't a reason not to do it. It's just something to watch out for.

Answer 2

There is always a risk of being misled by the spurious behavior of a single person who may perform certain actions by accident or in an unrepresentative manner.

Why You Only Need to Test with 5 Users

The accidental action taken by single user could be positive or negative, so your iteration could be for better or for worse. With two users (as with the RITE method you linked to) you could at least see if results are inconclusive on some parts of the UI. In which case you shouldn't probably iterate those elements of the design.

But if you can tell what the usability problem is and what causes it you could iterate after one user.

Changes to the user interface are made as soon as a problem is identified and a solution is clear. Sometimes this can occur after observing 1 participant. Once the data for a participant has been collected the usability engineer and team decide if they will be making any changes to the prototype prior to the next participant.

Using the RITE method to improve products; a definition and a case study

I couldn't find any studies on whether RITE or more traditional iteration methods produce better results in the end. That would be interesting to see. All we need is two separated test groups who start with same design, other using RITE and other something else.