Is guerilla testing meant to be ad hoc and improvised?

Question

I think any type of research and testing activity should have some planning and structured involved, in the same way that adopting an Agile software development process doesn't mean throwing all the processes and documentation out the window.

Is it common practice for people to do guerilla testing as an ad hoc activity in research and testing, or should these still be activities that are scheduled when the time and resource constraints are known to be tight? And when people conduct guerilla testing, is it common for the process to be improvised or should it still be partially structured so as to allow easier synthesis of research/testing information?

Answer 1

Guerrilla testing is a common and low cost method of testing, which is by definition ad hoc. Its goal is to get quick feedback whenever you need it.

As you said, all testing should be thought about and planned in advance, even Guerrilla testing, so that you go in with a clear objective, and you still need to synthesise the results and write up the report afterwards.

The actual Guerrilla test itself needs to be kept very short, mainly because your are not really paying for the participant's time (although you might buy them a coffee, or vouchers etc.) and your objective has to be limited (which means what you might learn might also be limited), but from a process perspective time and resource constraints will always hinder us.Hopefully most businesses learn something.

Jakob Nielsen has his famous "Why you only need to test with 5 users" article, which is often the typical number of participants in a Guerrilla test.

https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/

As soon as you collect data from a single test user, your insights shoot up and you have already learned almost a third of all there is to know about the usability of the design. The difference between zero and even a little bit of data is astounding.

When you test the second user, you will discover that this person does some of the same things as the first user, so there is some overlap in what you learn. People are definitely different, so there will also be something new that the second user does that you did not observe with the first user. So the second user adds some amount of new insight, but not nearly as much as the first user did.

The third user will do many things that you already observed with the first user or with the second user and even some things that you have already seen twice. Plus, of course, the third user will generate a small amount of new data, even if not as much as the first and the second user did. As you add more and more users, you learn less and less because you will keep seeing the same things again and again. There is no real need to keep observing the same thing multiple times, and you will be very motivated to go back to the drawing board and redesign the site to eliminate the usability problems.

After the fifth user, you are wasting your time by observing the same findings repeatedly but not learning much new.

There are some other links in this article which might provide some more answers to your question:

http://www.uxbooth.com/articles/the-art-of-guerrilla-usability-testing/

So there are many similarities in the planning and post test activities in all testing methods - it is just the Guerrilla test itself which is different to more formal testing methods

Answer 2

The tl;dr version

Guerilla testing is a structured test set up and performed quickly to investigate one or very few immediate concerns.

It's a simple, actionable test.
Don't overcomplicate it.
Don't be sloppy.

Answer 3

While I agree with the definition of guerilla testing as 'ad hoc' and for 'quick feedback', I don't agree with the link made to qualitative testing - For me, qualitative testing (full user testing as outlined in the Nielsen quote posted by @SteveD) is much more in depth and requires much more careful design and planning which is why you only really need five test subjects.

Guerilla testing, on the other hand, is more about quantitative testing; short single-question tests that you ask to as many people as possible. These are great for getting the opinions and preferences of your users rather than studying user flows.

As an example I was working on a control panel and discovered what I thought might be a problem with one of the sliders being interpreted in the opposite direct to the way it was intended. It took about 90 minutes from the moment I thought of testing to getting usable (and, in this case, conclusive) results. (You can see the results here if you're interested).

In this case I used an online tool but I could just as easily have printed this out on little pieces of paper and gone out onto the street and asked people as they passed.

Answer 4

The guerrilla testing that I have done in the past has always been simple paper print outs. In house we have always called them hallway testing. We use it as a means to get past a difference of opinion within a design that does not need a full user test to determine.

This has always been very ad hoc in nature as it has either been walk down the hall to the break area or into a completely un-related department and ask if we can have 5 minutes. The sampling is all internal but it is also generally nice and random. By doing this we have gotten through many a quick win design choices that have later been proven out with structured testing.

The only structure that we have ever had for this type of testing is limiting the questions to no more 2 or 3 and if possible they are binary in nature.

Answer 5

NO! It should not be "ad hoc" - you need to PLAN it carefully. The process should be as planned as it can be, but the artifacts you test should be easily updated or disposed of, the tests should be very quick and lightweight in their level of effort, and you should be able to ideally set up easily and do it in a variety of locations.

Just because the tests are quick and you might scrap the artifacts regularly doesn't mean you want to shoot from the hip. Plan well, if you have time.

Answer 6

If Guerilla Testing was well-structured and planned, than it wouldn't be Guerilla Testing anymore. It would become a formal lab UX methodology, which already exists in many forms.

Every testing method has its ups and downs. Guerilla's Testing ups and downs benefit small companies, which is enough to assess it as a needed procedure.

After putting in balance the damage of the ad hoc measures/improvisations and the advantages you obtain with regard to rapid results and low costs, you might want to go with this option. Somebody else in your situation, with a different profile, might say otherwise.

I think it also depends on what you understand by the idea of structure. According to The Art of Guerrilla Usability Testing of course you can structure your Guerilla Testing procedures simply by asking yourself a few questions like:

• What shall we test?

• Where will we test?

• With whom will we test?

• How will we test?