Should I trust user tests outside a controlled environment

Question

Recently we started to do user tests for a new website we are launching. At first my boss was against user testing because 'reasons'*.

After I conducted the first few tests we already identified a couple of pain points. Solving these pain points made the product a lot better. My boss now had a renewed view on user tests and wanted to test it on more users** (mainly his neighbors / friends etc.), so i prepared a set of tasks for the users that I would use in our controlled environment and he could test tasks on users he knew.

My current problem

Tests done in our controlled environment reflect nearly the opposite results of his own user tests. I know he can be a bit projecting regarding, so this got me wondering: Should I trust user tests that are done outside a controlled environment?

*With reasons I mean the one that most people use is that 'I understand the target group better then the target group understands themselves etc'

**Yes these users fall within the target group, our target group is extremely wide. Basically anyone from 18 till 50 years old.

Answer 1

Yes / No

Both of the options above might be true depending on many factors.

See, conducting unmoderated tests isn't rare at all, as a matter of fact is pretty common (see this article to understand differences between unmoderated and moderated tests).

However, not all tests are the same, not all answers you're looking for can be obtained with the same tests, not always the "correct test" is correct. Then you have environment (controlled or uncontrolled, remote or local, user in her environment vs user at your office and so on), skills and knowledge of the interviewer, time, budget, assets.... well, a wide array of variables that will have influence in your testing.

All the above is related to research methodology. The problem is that many people thinks that research methods (or techniques) and research methodology are the same. To make it clear:

Research methods are the various procedures, schemes and algorithms used in research. All the methods used by a researcher during a research study are termed as research methods. They are essentially planned, scientific and value-neutral. Research methods help us collect samples, data and find a solution to a problem.

Research methodology is a systematic way to solve a problem. It is a science of studying how research is to be carried out. Essentially, the procedures by which researchers go about their work of describing, explaining and predicting phenomena are called research methodology. It is also defined as the study of methods by which knowledge is gained. Its aim is to give the work plan of research.

(definitions from RESEARCH METHODOLOGY paper)

As you can see... similar but not the same. And I learned this the hard way: I consider myself a very knowledgeable person when it comes to testing and research. After all, I studied the different tests available, how to conduct testing, how to screen participants... well, I knew enough to be fully qualified. Now, my wife/partner is a Psychology Doctor, professor of Methods and Techniques of Scientific Research and overall specialized in human behavior research. At that point, she didn't work with me. Years passed by and I noticed that my research skills were great, but sometimes I had results that proven wrong when tested against real world. I asked her for help and after careful review she told me: "OK, you know a lot about methods, much more than me. You simply don't know much about methodology, hence your results are not very reliable" (note: this was specially true with Qualitative Methods)

In short, going back to your answer

Everything is possible, but for what you describe, your manager's results are probably unreliable. The fact that he knows interviewees probably makes the test result's invalid (not necessarily, but very likely). Depending on the kind of test, I suggest you do it in your office or maybe some guerrilla testing

Answer 2

@Devin's answer is very valuable be sure to understand his point about the difference between methods and methodology, i would like to add :

---

Just to put this one away first

While it would have solved the riddle faster, i would prefer to avoid falling to my fast thinking heuristic that says **the typical boss that knows nothing and decides on everything ** so i would like to start this off by saying i am assuming here that your boss is qualified to perform and moderate test, if this does not apply i would definitely rely more on the non-moderated version, or in a statistical language if you prefer i would rely much more on a random uncontrolled sample over having a biased stratified sample, now the answer:

---

You might want to consider merging the findings

User tests produce different findings, while initially the main purpose is to assess the usability of our products they often bring different insights to the table varying from users needs insights, market insights and sometimes even how the value proposition is perceived on its own, it is very important to be able to distinguish between usability findings and other findings.

The issue with wide target groups is inconsistency in findings, so the difference between results could be due to either actually having different users or it could be due to the different environment of the test.

In Nielsen Norman Group's article https://www.nngroup.com/articles/how-many-test-users/

How Many Test Users in a Usability Study?

The answer is 5, except when it's not. Most arguments for using more test participants are wrong, but some tests should be bigger and some smaller.

To my understanding the answer above is more relevant to the usability aspect of the study, meaning from usability perspective the findings are representative using 5 users, but it does not mean the the other insights must representative as well, as an example if a user says "i wouldn't buy this product for this amount" does not necessarily mean the market will not buy it, although it might hint that the value proposition needs better verification.

So if we agree we have to differentiate between the types of conflicting findings you have arrived, and we are talking about findings that reflect specifically usability issues then it should be safe to merge them and treat the entire superset of findings rather than having to choose between Test A or Test B.

         Usability issue 1  Usability issue 2   Usability issue 3
Test A:  Exists             Does not Exist      Exists
Test B:  Does not exist     Exists              Exists

This way you can argue that your solution is more comprehensive.

---

One last comment this is a managerial way of handling things, some might disagree with the approach, yet it's up to you how to decide on it. I needed to bring spotlights at the different types of findings into your consideration while making your next move, best of luck.

Answer 3

Yes, you can trust results from usability tests done outside your controlled environment. (I assume you mean your in-house lab.) Check out Guerrilla Testing, which is done in the user's natural environment. (Just for kicks, also see Drunk User Testing.)

However, I think your real question is about whether you can trust usability tests done by your boss, or others who don't understand how easy it is to skew the results. And no, you can't trust those results. It's too easy to ask leading questions, give too much assistance, give the user hints, bias their attitudes, etc.