22
votes
Accepted
How many test participants to 'prove' an issue?
You asked if there is any statistics that can be used to essentially determine if the 10 people you are testing with are especially dumb. Essentially what you're asking is:
What is the likelihood ...
- 344
20
votes
A/B testing - client wanted the test run 70/30
Your statistical power is maximized if equal numbers of users are directed to A and B. That is, assuming one design really is better than the other, you’ll achieve statistical significance with the ...
- 37.7k
18
votes
How many test participants to 'prove' an issue?
In one study I discovered "5 out of 5 users couldn't complete the task" in the past, and my stakeholders simply didn't believe the data. So this is a something to consider.
While Nielsen advocates 5 ...
- 8,904
17
votes
Percentage of screen readers users in USA?
To answer the askers question, it looks like we have to use a little deduction. Nobody seems to collect these statistics outright. Someone wrote a very detailed article as to why.
There are 326 ...
- 271
5
votes
Is it worth spending time to design a tablet-friendly site in 2016?
I think your answer lies in your question, also it may only be 5% of traffic, but who are those users? I would find that out before ignoring them.
My employer's main traffic comes from desktops due ...
4
votes
A/B testing - how to deal with minority that chose B?
'C'.
Option C should be a working model that introduces the compromised hybrid solution that attempts to find a way to appease both testing groups to a larger percentage. It'll always be changing, ...
4
votes
Confidence intervals, Confidence levels and probability of simple tests
From your example, let's imagine you have run the fake door test, which resulted in 265 visitors clicking on it, where total visitors that day were 350.
The real question (as you rightfully ...
4
votes
Accepted
What would be a good way to visualize recent win/loss (player form) data?
If you want to visualize just the total wins and losses any simple chart or table will do. If you want to visualize the progress throughout the season though, what you have is similar to Edward Tufte'...
- 331
4
votes
How many test participants to 'prove' an issue?
From a statistical perspective, the larger your test group, the more accurate your result. With an increasing number of participants, your result in the limit value always approaches the real value. ...
4
votes
A/B testing - client wanted the test run 70/30
In theory, there isn't a problem with a client-ordered 70/30 test, as long as you're comparing conversion rates and not conversion amounts. If your hypothesis holds up, you'll still see the 5% dip ...
- 12.1k
4
votes
A/B testing - client wanted the test run 70/30
Run A/B test on 60% of the traffic then. Diverged exactly 50/50 into old and new variants.
Diverge all other 40% of users/sessions/views to new experience (as your client requests), but ignore them in ...
- 141
3
votes
What percentage of internet users save credentials in the browser?
Here's an article from Pew Research that points out which method is used by users to save their credentials from a survey in 2016.
[W]hile 18% say that they save them using the built-in password ...
- 31
3
votes
Statistics and Usability
It honestly depends on the type of work you are completing for your website.
You might be testing a new design out, just to see if your new design is usable. Or you could be completing A/B tests, ...
- 81
3
votes
Accepted
Statistics and Usability
Why Statistics?
Once you get beyond graphs, averages, and percents, the bulk of statistics concerns answering the question, “Is this sample size big enough to convince me that what I see in the ...
- 37.7k
3
votes
How to calculate user interface's efficiency time
The question for me is here: Can you define key metrics for success?
I do not think that there is a formula (at least I know none) that tells me "The interface works great!".
Usually, the question ...
- 1,681
3
votes
How can i interpret the observed difference in a/b test?
You are correct that a significant null hypothesis result only tells you that A and B are different, but it doesn't quite tell you how different they could reasonably be, given the sample size and ...
- 37.7k
2
votes
How to ask a user which factors are most important to them when calculating weights in a model?
According to User Centered Design, you should ask yourself what are the users' goals.
You gave a very good example of a person in a wheelchair. I would try to find out what are those goals, and let ...
- 4,988
2
votes
Accepted
Best way to approach statistics pages
Say for example, there are around 10 numerical stats that we want to
show. How should hierarchy be determined when we are unsure of what
data is most important?
This must come from business. ...
- 6,641
2
votes
How many test participants to 'prove' an issue?
Most usability questions can be answered with small batch, qualitative testing, as mentioned above.
Some questions require large samples and quantitative proof to answer. A/B testing uses large ...
- 1,529
2
votes
Accepted
Graphs comparison placement
Having one before the other might be inevitable. You need to think what is more important to you. Here are a few questions that might make you choose one representation over another:
What separation ...
- 471
2
votes
How many test participants to 'prove' an issue?
Change the way you communicate with stakeholders
If you need evidence to resolve a deadlock between stakeholders, the battle is already lost. Good design is not about quantitative science, it's about ...
- 4,811
2
votes
A/B testing - client wanted the test run 70/30
Your customer wants two things: Find out with evidence which method gives better sales, and don't lose sales during the test. A 50/50 split will give you the evidence quickest. A 70/30 split ...
- 476
1
vote
How do I treat an 'Incomplete' in a Usability study?
I would mark them as "Incomplete" and write down the reason and details for not being able to complete the task. I also do this for tasks where the facilitator had to step in and help the participant ...
- 1,402
1
vote
Confidence intervals, Confidence levels and probability of simple tests
The test that you described is not a true experiment or test.
The hypothesis is unclear, the results cannot be reproduced or generalized. The method is more of a user survey, so there's no point in ...
- 1,402
1
vote
What is the best way to statistically analyze a score system and number of steps for users?
Your metrics probably cannot be easily aggregated into one number. One is subjective (ranking) the other objective (number of steps).
One way to summarize both could be to use a grading system (A+, B,...
- 668
1
vote
What is the best way to statistically analyze a score system and number of steps for users?
John Brooke's SUS provides a kind of quantitative evaluation and final number/score that could be customized for doing task analysis. Template can be found here: https://www.usability.gov/how-to-and-...
- 485
1
vote
What to do when sample size is too small during usability study?
Calculated with the Adjusted Wald Method, as N goes up the confidence intervals shrink.
Are you evaluating specific tasks as pass/fail, or doing something more involved? Are you comparing something ...
- 1,537
1
vote
Accepted
Should I invite existing users to my summative usability test?
Would inviting existing users hurt the results of my summative
usability test too much?
Yes
Should I rather invite existing users for a separate test?
Yes
Is there a benefit of including ...
- 37.3k
1
vote
Accepted
Should I have different toggles for collecting usage data and crash reports?
Ask yourself what would you do?
If I was using the app:
Share Usage Data (Unchecked)
Report Crash Logs (Checked)
Share Usage Stats (Unchecked)
Someone else might do something else.
You need to ...
- 126
1
vote
Accepted
What tools are available to collect statistics on user behavioral patterns?
I haven't heard of a tool that gives you demographic information as you specified. And I'm not sure that's possible. The only think you can do is track specific users by using Mixpanel or Universal ...
- 11.5k
Only top scored, non community-wiki answers of a minimum length are eligible