Is it necessary to conduct both guerrilla testing and traditional testing for a single website?

Question

I am in the process of conducting some traditional usability test sessions for my sports website. I have some good feedback so far on the navigation, layout and design of the website.

But I've just read up on guerrilla testing and think it may be useful to get more feedback.

However the two types of usability test means there will be variations in the test proceedings. For example, the guerrilla test sessions would only last 15-20 minutes at most so that would mean I have to reduce the number of activities each participant does. That means I'll have one set of participants that did 10 tasks and one set of participants that did about 3 or 4 tasks.

Would the difference in testing environment, test type, tasks given to participants etc.. cause reliability or inconsistency issues when it comes to reporting results? Would it be suitable to draw conclusions with two sets of test results from 2 studies?

For example in my main usability test I'd ask to register tickets for an event and in the guerrilla test I'd ask the participant to do the same. If both report issues with the navigation, could I still draw conclusions taking consideration the environment and difference in test type?

Should I only stick to one type of evaluation method when gathering data from users?

I guess there isn't a generic answer for this question, but it's more to do with the necessity to conduct both test types, if it may improve or strengthen the quality of my feedback.

Answer 1

Don't be misled by the term guerrilla usability testing. It basically just means that you are very creative in the way you gather your findings and willing to cut corners on the methodology as compared to traditional usability testing.

There are two kinds of approaches to usability testing. You can test your design in one big bang, if you are pretty certain of the quality of your design or if there are very little options. Next to that you can test your design iterations, and change the design according to the findings. If you have the luxury of even considering doing both traditional usability testing and guerrilla usability testing on one project, why not use the guerrilla usability tests to decide what concept generally works best, and use the classical usability test to fine tune the winning concept.

As for combining findings from different studies, having multiple research setups to prove the same point is always a good idea. If you go about your research in a statistical way, you can't add findings from study A to study B, because they have different methods and you can't be sure you are measuring the same behavior of all users in the same way. Next to that, the stimuli are different. If you are doing qualitative studies, then go ahead and combine those findings, unless you are really going about it in a scientifical way.