Why people do guerrilla usability testing if the sampling is always non-representative?

Question

As far as I know guerrilla usability testing is very widely used by design teams. But why? In guerrilla usability testings the sampling method is called convenience method and it is known as a non-probability sampling (or non-representative). So you can't really say something like "our data shows us that 8% of the population will have problems with the navigation", for instance. And if you can't make such claims, is this kind of usability testing still useful?

Statically speaking the data collected by such surveys should be virtually worthless and could never be treated with the mathematical models usually employed because they don't hold true if the sample isn't perfectly random. And as far as I know, there are no mathematical models for such scenarios.

I get guerrilla usability testings are not expensive and fit agile teams. But if the data isn't good at all and if you can't really used it conclude a thing about the population, why people keep doing it at all? I mean, the data gathered could even be harmful and misleading.

Answer 1

If you need a clear statistical sampling of the population then guerrilla testing is useless. However often times one simply needs a "man-on-the-street" opinion. Small informal sampling is useful for generic things - "Is this design pleasing?"; "What feelings does it evoke?"; "Is the user-flow straight-forward?"; "Do you understand what to do next?"

Remember you're watching them and seeing where they are having problems, where they are confused and are uncertain in what to do next. What may seem clear cut to you may not be understandable to them. Those points of confusion are noteworthy to consider.

You'll also get feed back on the designs from people who couldn't care less about design and aren't bored by something that was done a year ago. This is very useful feedback if you're designing for the general public. And, you can find them - the general app-using public - at a local coffee shop.

For the price of a few cups of coffee and some muffins you can get useful feedback.

Answer 2

Usability testing (and the "guerrilla testing" version of it) is a qualitative research method. There's no need for statistical significance in order to discover problems with the system.

How many test users in a usability test? Nielsen/Norman says, "The answer is 5, except when it's not."

Answer 3

Guerrilla testing is similar to unit testing done by a developer. It's good practice for developers to run their code before releasing it to make sure it executes properly. Guerrilla testing is like that for design -- it's a way to check that the UI facilitates the tasks for which it was built.

High-value elements should undergo more rigorous testing, but guerrilla testing is adequate for fast-moving design and dev teams.