2

We are developing a product and already tested with more than 50 users. A lot of times users would have ideas to suggest us move/duplicate an element (not common feedback, there are random ideas)

I wonder how to determine which suggestion is valuable that we should work on A/B testing to compare their idea and the existing design.

Improve this question

2 Answers 2

Reset to default
5

I'd look at plotting the ideas on an Impact/Effort Matrix, and use that to prioritize. The low-effort, high-impact ideas would be prioritized first; the high-effort, low-impact deliverables would be disregarded.

If you use Google Analytics, you can use Google Optimize to knock out the easy ones, like testing colors and button labels: https://www.google.com/analytics/optimize/capabilities/

Improve this answer
-1

The first and the most important thing you need to determine is whether you are actually able to run a valid A/B test - do you have a big enough sample size at your disposal?

Which is not the case when you are developing a new product.

A/B testing requires a lot of participants (sample size) - traffic, users and conversions (like a lot). A rough ballpark is 1k monthly transactions (conversions). It is possible with less but it becomes increasingly challenging and less effective with smaller sample sizes.

If you run an a/b test with an insufficient sample size, you'll be running an underpowered a/b test which means that there's a high probability that your results will be imaginary, for example:

  • You'll likely see an inconclusive result (no difference between the new and old design) which does not mean that there really isn't a difference (you'll need a bigger sample size to detect that difference)
  • You may even see a 'statistically significant' winner (or looser) but it does not mean that the variant is really a winner (or vice versa)

In simple words, if you ignore this, you are simply not going to be able to conclude if your new idea is better than the existing design. There's no way around it, the math is not going to work (statistics is tricky) - you need large amounts of data to detect 'real' differences between two (or more) variants:

Therefore, if you don't have the sample size yet, your best bet is user testing or other qualitative research (surveys, polls, interviews and etc.)

Second thing - A/B testing is widely misunderstood and highly complex. You are basically performing a science experiment (sort of) and there are many ways to screw it up :)

And the chances are high that it may even not give you the answers you are looking for so here are some great in-depth beginner resources to begin with (to learn when and how to use A/B testing properly):

But the answer to your actual question (how to determine if you should a/b test an idea or a suggestion by a user?) - many a/b testing pros use simple prioritization frameworks which are more or less based on the same effort/impact matrix. More details (and framework comparisons):

Improve this answer
5
  • 1
    "A/B testing requires a lot of participants (sample size) - traffic, users and conversions (like a lot). A rough ballpark is 1k monthly transactions (conversions)." This isn't true at all. In usability testing, there are often effect sizes large enough to justify very small samples while getting statistically significant results. You can see the math yourself in the following answer: ux.stackexchange.com/questions/100390/…
    – S. G. A.
    Jan 5, 2018 at 19:33
  • A/B testing and Usability testing are not the same things. Yes, in general, you are right - the higher the effect size the smaller sample size is needed but those high effect sizes (high lifts) don't happen that 'often' in real life. You can see the math by typing the minimum detectable effect (expected effect size) in this calculator - evanmiller.org/ab-testing/sample-size.html or this one - optimizely.com/sample-size-calculator (a bit easier to digest)
    – em-v
    Jan 5, 2018 at 19:58
  • And I also stated that it is possible to run an a/b tests with a smaller sample size but it becomes increasingly more difficult and ineffective.
    – em-v
    Jan 5, 2018 at 20:05
  • 1
    A/B testing is just a method which can be performed in different kinds of testing, including usability testing (which is what the OP is asking about, not marketing). It's not unique to getting conversions. While I agree that for marketing purposes you need huge samples (typically tiny effect sizes), we are discussing usability, where 1000s per month samples aren't often necessary, nor realistic.
    – S. G. A.
    Jan 8, 2018 at 14:04
  • How would you conduct an A/B test in usability testing? It certainly depends on what exactly are you testing, measuring and hoping to improve. If you test essential features or elements that are used often (therefore have high conversion rates) then yeah, you can do with smaller sample sizes (it's not just the effect sizes). However, in statistics, in general, small sample size is a problem that often leads to misleading and misinterpreted results.
    – em-v
    Jan 8, 2018 at 14:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.