Skip to main content
22 votes
Accepted

How many test participants to 'prove' an issue?

You asked if there is any statistics that can be used to essentially determine if the 10 people you are testing with are especially dumb. Essentially what you're asking is: What is the likelihood ...
user6916458's user avatar
20 votes

A/B testing - client wanted the test run 70/30

Your statistical power is maximized if equal numbers of users are directed to A and B. That is, assuming one design really is better than the other, you’ll achieve statistical significance with the ...
Michael Zuschlag's user avatar
18 votes

How many test participants to 'prove' an issue?

In one study I discovered "5 out of 5 users couldn't complete the task" in the past, and my stakeholders simply didn't believe the data. So this is a something to consider. While Nielsen advocates 5 ...
SteveD's user avatar
  • 8,934
17 votes

Percentage of screen readers users in USA?

To answer the askers question, it looks like we have to use a little deduction. Nobody seems to collect these statistics outright. Someone wrote a very detailed article as to why. There are 326 ...
Nick Steele's user avatar
4 votes

Confidence intervals, Confidence levels and probability of simple tests

From your example, let's imagine you have run the fake door test, which resulted in 265 visitors clicking on it, where total visitors that day were 350.

 The real question (as you rightfully ...
Steven B. Peutz's user avatar
4 votes

A/B testing - how to deal with minority that chose B?

'C'. Option C should be a working model that introduces the compromised hybrid solution that attempts to find a way to appease both testing groups to a larger percentage. It'll always be changing, ...
Jeremy Schroeder's user avatar
4 votes

How many test participants to 'prove' an issue?

From a statistical perspective, the larger your test group, the more accurate your result. With an increasing number of participants, your result in the limit value always approaches the real value. ...
AnonRocketman's user avatar
4 votes

A/B testing - client wanted the test run 70/30

Run A/B test on 60% of the traffic then. Diverged exactly 50/50 into old and new variants. Diverge all other 40% of users/sessions/views to new experience (as your client requests), but ignore them in ...
Frax's user avatar
  • 141
4 votes

A/B testing - client wanted the test run 70/30

In theory, there isn't a problem with a client-ordered 70/30 test, as long as you're comparing conversion rates and not conversion amounts. If your hypothesis holds up, you'll still see the 5% dip ...
Izquierdo's user avatar
  • 12.7k
3 votes

How can i interpret the observed difference in a/b test?

You are correct that a significant null hypothesis result only tells you that A and B are different, but it doesn't quite tell you how different they could reasonably be, given the sample size and ...
Michael Zuschlag's user avatar
3 votes

What percentage of internet users save credentials in the browser?

Here's an article from Pew Research that points out which method is used by users to save their credentials from a survey in 2016. [W]hile 18% say that they save them using the built-in password ...
Utsav Shah's user avatar
2 votes
Accepted

Best way to approach statistics pages

Say for example, there are around 10 numerical stats that we want to show. How should hierarchy be determined when we are unsure of what data is most important? This must come from business. ...
Mayo's user avatar
  • 6,641
2 votes
Accepted

Graphs comparison placement

Having one before the other might be inevitable. You need to think what is more important to you. Here are a few questions that might make you choose one representation over another: What separation ...
mapto's user avatar
  • 481
2 votes

How many test participants to 'prove' an issue?

Most usability questions can be answered with small batch, qualitative testing, as mentioned above. Some questions require large samples and quantitative proof to answer. A/B testing uses large ...
RobC's user avatar
  • 1,549
2 votes

A/B testing - client wanted the test run 70/30

Your customer wants two things: Find out with evidence which method gives better sales, and don't lose sales during the test. A 50/50 split will give you the evidence quickest. A 70/30 split ...
gnasher729's user avatar
2 votes

How many test participants to 'prove' an issue?

Change the way you communicate with stakeholders If you need evidence to resolve a deadlock between stakeholders, the battle is already lost. Good design is not about quantitative science, it's about ...
Peter's user avatar
  • 4,813
1 vote

What percent of the web's images have alt text?

A couple top hits searching DuckDuckGo for: accessibility statistics "alternative text" ...recently report approx. 75% of "home pages" have alternative text for images. Sources ...
bloodyKnuckles's user avatar
1 vote
Accepted

How much does bounce rate vary according to page load speed?

According to Pingdom, the conventional wisdom has been more like two seconds (Google's recommendation), whereas the average website takes 3.21. The drop-off is much steeper than you suppose, based on ...
Luke Sawczak's user avatar
  • 1,121
1 vote

Any best practice on how to represent player progress stat in a game?

It seems like for a particular 'exercise' or 'task' you want to represent two bits of information. Progress of the exercise (i.e. how much of it they have completed) Competency of the exercise (i.e. ...
Michael Lai's user avatar
  • 27.3k
1 vote

Are there stats about height of browser windows worldwide / by industry?

I personally use this website call statcounter. I find it quite useful to decide with the team what viewports we should support, what breakpoint we should take into account, and what's the smallest ...
Aberouch's user avatar
1 vote

How to display data with variation to unsophisticated users?

Did you consider giving median with a label very uneven (red) : std > 4 uneven (orange) : std > 2 homogeneous (green) : std <= 2 Sometimes if you draw the statistical distribution, it shows ...
marcant0's user avatar
1 vote

How do I treat an 'Incomplete' in a Usability study?

I would mark them as "Incomplete" and write down the reason and details for not being able to complete the task. I also do this for tasks where the facilitator had to step in and help the participant ...
Ling's user avatar
  • 1,412
1 vote

Confidence intervals, Confidence levels and probability of simple tests

The test that you described is not a true experiment or test. The hypothesis is unclear, the results cannot be reproduced or generalized. The method is more of a user survey, so there's no point in ...
Ling's user avatar
  • 1,412
1 vote

What is the best way to statistically analyze a score system and number of steps for users?

Your metrics probably cannot be easily aggregated into one number. One is subjective (ranking) the other objective (number of steps). One way to summarize both could be to use a grading system (A+, B,...
Riche Design's user avatar
1 vote

What is the best way to statistically analyze a score system and number of steps for users?

John Brooke's SUS provides a kind of quantitative evaluation and final number/score that could be customized for doing task analysis. Template can be found here: https://www.usability.gov/how-to-and-...
gpgpgp's user avatar
  • 485
1 vote

What to do when sample size is too small during usability study?

Calculated with the Adjusted Wald Method, as N goes up the confidence intervals shrink. Are you evaluating specific tasks as pass/fail, or doing something more involved? Are you comparing something ...
Luke Smith's user avatar
  • 1,537
1 vote
Accepted

Should I invite existing users to my summative usability test?

Would inviting existing users hurt the results of my summative usability test too much? Yes Should I rather invite existing users for a separate test? Yes Is there a benefit of including ...
Devin's user avatar
  • 38.1k
1 vote

What's the best way to interpret Likert Scales with equal negative and positive Likert Items?

I would suggest that you might be able to present it on a single scale as "perceived data accuracy", and then just reverse the values of the negative ones and combine them with the positive ones. ...
Mattynabib's user avatar
  • 3,501
1 vote

Some stats about social networks used by iOS/Android users?

As of 2016 the number of US users of the Facebook app has reached 47 million a month on iOS and 55 million on Android. Facebook is the most popular app on iOS and it is only number 3 on Android. ...
Macrina Damian's user avatar
1 vote

Are there devices narrower than 320px (and data on their usage for web browsing)?

According to yesviz.com/devices.php, each of these devices has a viewport width of 162 px: Apple Watch SE, 40mm Apple Watch Series 6, 40mm Apple Watch Series 5, 40mm Apple Watch Series 4, 40mm ...
ma11hew28's user avatar
  • 121

Only top scored, non community-wiki answers of a minimum length are eligible