20
votes
A/B testing - client wanted the test run 70/30
Your statistical power is maximized if equal numbers of users are directed to A and B. That is, assuming one design really is better than the other, you’ll achieve statistical significance with the ...
- 37.7k
10
votes
Accepted
USER TESTING alternative phrase
I think it's the word user that's more of an issue than the word testing.
After all, you're testing the product not the user. In fact, the chances are that your user testing script reminds you to ...
- 70.9k
6
votes
How to perform usability studies on complex software
Your problem, as I understand it, is that due to the wide variety of tasks that customers may be using your product for, you are having a challenge in specifying tasks for your usability tests.
I your ...
- 68.2k
5
votes
Accepted
Usability Testing a website redesign
You don't have to build the whole UI to test some features like navigation.
Early in the design process, you can test information architecture with a tool like tree testing or card sorting. These are ...
- 1,549
5
votes
Accepted
Should I trust user tests outside a controlled environment
Yes / No
Both of the options above might be true depending on many factors.
See, conducting unmoderated tests isn't rare at all, as a matter of fact is pretty common (see this article to understand ...
- 36.5k
4
votes
Confidence intervals, Confidence levels and probability of simple tests
From your example, let's imagine you have run the fake door test, which resulted in 265 visitors clicking on it, where total visitors that day were 350.
The real question (as you rightfully ...
4
votes
Running usability tests with the same people that have passed it previously
No.
People learn.
If the software is similar to the first time, your users will have learnt how it works.
- 11.2k
4
votes
How to perform usability studies on complex software
I think there's a couple of approaches you could try, both using Jobs to be Done as the framework.
1. How are users doing with the intended uses of the product? What jobs did your leaders have in mind ...
- 12k
4
votes
A/B testing - client wanted the test run 70/30
In theory, there isn't a problem with a client-ordered 70/30 test, as long as you're comparing conversion rates and not conversion amounts. If your hypothesis holds up, you'll still see the 5% dip ...
- 12k
4
votes
A/B testing - client wanted the test run 70/30
Run A/B test on 60% of the traffic then. Diverged exactly 50/50 into old and new variants.
Diverge all other 40% of users/sessions/views to new experience (as your client requests), but ignore them in ...
- 141
3
votes
Best way to get feedback from a user on a product?
Bringing the developer along will be beneficial for several reasons. Yes, they may have a chance to answer any specific technical questions but more importantly it will expose them to the overall ...
- 31
3
votes
Using both moderated and unmoderated Usability Testing in one research
If I were in this position, I might try treating the two in-person participants as a pilot. Do a study with them, and after you analyze the results, you'll be able to do a higher-quality remote study ...
- 1,455
3
votes
Accepted
Where to find 50+ year old people for guerrilla usability testing?
50-100 is a massive demographic.
I think you need to think about splitting it into subgroups, as these will behave very differently. Different age ranges have 'grown up' with different technology, ...
- 11.2k
3
votes
When testing a prototype, do you inform your users that everything is not functional?
This comes down to how you conduct your testing.
Personally, I always inform my test users that they will be working with a prototype so they cannot 'break' the system and will not destroy any data. ...
- 14.2k
3
votes
When testing a prototype, do you inform your users that everything is not functional?
If you don't want to immediately make the user aware that they're not using a finished product, then you should definitely make them aware when they go to click on a button (for example) and it isn't ...
- 309
3
votes
Audio for testing purposes
How about generating a short audio from a string that describes the feature? There should be plenty of ways to auto generate this kind of thing (as a c# developer I know it is a few lines of code to ...
- 12.5k
3
votes
Should I trust user tests outside a controlled environment
@Devin's answer is very valuable be sure to understand his point about the difference between methods and methodology, i would like to add :
Just to put this one away first
While it would have solved ...
- 2,445
3
votes
exploratory testing
If you are trying to validate a concept or determine a user's needs, you are in a generative research phase: you are trying to determine the right thing to do based on conditions your targeted users ...
- 3,386
3
votes
Accepted
Accessibility testing (with Microsoft Narrator as screenreader)? Somebody here with some practical guidelines?
I wouldn't bother with Narrator too much but by all means add it to your testing suite.
You will see from the webAim screenreader study that NVDA, JAWS and VoiceOver make up over 90% of screen readers....
- 2,588
3
votes
Running usability tests with the same people that have passed it previously
Yes, but their input should be treated differently than users who were not a part of the first round, and you certainly still require a group of users who have not previously tested it.
To expand as ...
- 131
3
votes
Running usability tests with the same people that have passed it previously
In learning we have received from PwC Experience Center on Qualitative Usability testing, it was noted participants should be new ones.
Advantages of this are:
if they are first time here, they are ...
- 2,656
2
votes
Best method for measuring the user's understanding of a pilot app?
You definitely want to user one-on-one interviews for this. Data can only tell you the what but talking to people will tell you the why.
I would structure these interviews as about 60 minutes each. ...
- 59
2
votes
Best method for measuring the user's understanding of a pilot app?
As a complimentary method to the @user5482 answer.
Conduct diary studies
A diary study is a research method used to collect qualitative data
about user behaviors, activities, and experiences ...
- 11.5k
2
votes
Accepted
How to test perception of specific elements on a page?
Start with why you are doing this test:
...just want to know if he notices it and reminds what he saw.
Sounds to me like the proposed design is intended to help users notice the elements and ...
- 1,537
2
votes
How to test perception of specific elements on a page?
Try and have page analytics enabled. You can see the conversion rate (% of viewers that clicked) for this page and the pages you go to when you click the highlighted areas. Do this before and after ...
- 7,905
2
votes
How can one test a design post development
This is borderline off-topic, but I'm answering because it's not purely about implementation. It's question about testing designs.
You can easily test pixel perfection using the Firefox add-in ...
- 11.6k
2
votes
Guerrilla usability testing
Make the participant feel comfortable
Ask the user to do whatever they feel comfortable with to clarify an idea. Some people are going to freeze when you ask them to draw anything. Others will grab ...
- 21.2k
2
votes
Best method for measuring the user's understanding of a pilot app?
The simplest method is to define the end goals that you expect users to accomplish.
Using stackexchange as an example, a goal might be: "did the user search for and find an answer to their question?"
...
- 994
2
votes
What is the best way to test three different versions of a piece of copy?
So does each of these descriptions has anything that distinguish it from the others? For example, does the 1st one talks more about "profitability" ("you can earn money/knowledge/friends if you ...
- 91
Only top scored, non community-wiki answers of a minimum length are eligible