Good design or bad testing?

Question

If you are testing a simple design, should you be concerned if all results come back successful? The tested page has only three CTAs. To rule out the possibility that the participant only clicked on the available button, the test focused more on whether the user could explain why they clicked on this CTA and not the other two. The purpose was to see if the design and information was clear and what they expected to see next after they clicked on the CTA.

All participants showed a high level of understanding of the questions asked and were confident that they had all taken the correct action (which they all did correctly). Since this is such a simple design, should I expect a clear success rate, or because all participants did so well, is this a warning sign that something might be missing from the testing, or should I accept that the simple design met the requirements?

Answer 1

Very good question.

If it's as simple as you say, there's a good chance that users have performed the tasks correctly because the design is simple and clear enough for the task (not necessarily good, that's subjective).

However, if you have doubts, you can use this round as a control group and then change something and test with new users. Or even create a negative control group where you create a version that has to fail (warning: this is not so easy when testing designs).

Or you can reverse the test logic and ask users why they did not click on the wrong CTAs. If their logic is correct and supports your hypothesis, you can be sure that the design is correct.

Also keep in mind that qualitative research methods are more difficult than quantitative. So one way to 100% determine this is to run the same test with as many users as possible (at least 10, better 20). Then it's just a quantitative/statistical question: if the vast majority of users perform as expected, then your hypothesis is correct again.