0

I wanted to take my class design project up a notch by getting my designs in front of some users through an unmoderated usability test. Because of the platform I used for the test, I could only make the test 2 minutes long which meant I couldn't combine test a/b. I created two distinct tests and shared them with a specific Facebook group by creating a post on the platform.

On the bright side, I got responses. The downside is, that some users didn't complete the other test so now I have 14 responses to test A and 8 responses for test B.

How do I ensure that my data is equal? It would seem skewed to analyze the results with different numbers of responses. Would it be reasonable to recruit 6 additional users for test B in order to balance the response rates?

Improve this question
5
  • fwiw: At this sample size, it's unlikely you'll be able to draw quantitative conclusions either way. What you can do here is find some qualitative info, such as "oh dear, I completely overlooked this interaction and should fix that", but from your question it sounds like you want some sort of "design A is 37% better with p<0.05 confidence"
    – Leo Wattenberg
    Jul 29 at 21:51
  • @LeoWattenberg Yes, that was what I was trying to test. I wanted to see if version A might be better than version B. I do have 2 of the same follow-up questions after each test. That seems the closest I can get to qualitative. Wouldn't it still be skewed since I have a higher response on one versus the other?
    – yunglejack
    Jul 30 at 16:11
  • Your sample size is the much bigger issue: "I just did 14 coin flips of a 2€ coin which ended up at 10:4 heads:tails, 8 coin flips of a 1€ coin ended up as 2:6. I conclude both coins are heavily biased". While it's a true observation, it's not statistically significant, and any conclusions drawn from this are meaningless - and in my example obviously false. evanmiller.org/ab-testing/sample-size.html can help you estimate how many people you need to ask
    – Leo Wattenberg
    Jul 31 at 17:48
  • @LeoWattenberg I read the article that evan miller posted about a/b testing. And from what I read and further research, to draw a conclusion, I would have to expand the number of users who are participating and I can use sequential testing to let me know whether the feature should be implemented or not?
    – yunglejack
    Aug 2 at 19:59
  • The results will already be skewed by having everyone do test A before test B. Next time have half the people start with one and half with the other.
    – Ray Butterworth
    Aug 29 at 13:42

1 Answer 1

Reset to default
0

Well then there's your answer, the fact that people didn't finish one over the other means one was better and more usable than the other. The data is speaking to you, just not in the way you want.

Finding out where they stopped would be helpful as well, to know which section of your flow is struggling.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.