Skip to main content
20 votes

A/B testing - client wanted the test run 70/30

Your statistical power is maximized if equal numbers of users are directed to A and B. That is, assuming one design really is better than the other, you’ll achieve statistical significance with the ...
Michael Zuschlag's user avatar
10 votes
Accepted

USER TESTING alternative phrase

I think it's the word user that's more of an issue than the word testing. After all, you're testing the product not the user. In fact, the chances are that your user testing script reminds you to ...
Roger Attrill's user avatar
6 votes

How to perform usability studies on complex software

Your problem, as I understand it, is that due to the wide variety of tasks that customers may be using your product for, you are having a challenge in specifying tasks for your usability tests. I your ...
JohnGB's user avatar
  • 68.4k
5 votes
Accepted

Usability Testing a website redesign

You don't have to build the whole UI to test some features like navigation. Early in the design process, you can test information architecture with a tool like tree testing or card sorting. These are ...
celinelenoble's user avatar
5 votes
Accepted

Should I trust user tests outside a controlled environment

Yes / No Both of the options above might be true depending on many factors. See, conducting unmoderated tests isn't rare at all, as a matter of fact is pretty common (see this article to understand ...
Devin's user avatar
  • 38.1k
4 votes

Confidence intervals, Confidence levels and probability of simple tests

From your example, let's imagine you have run the fake door test, which resulted in 265 visitors clicking on it, where total visitors that day were 350.

 The real question (as you rightfully ...
Steven B. Peutz's user avatar
4 votes

Running usability tests with the same people that have passed it previously

No. People learn. If the software is similar to the first time, your users will have learnt how it works.
PhillipW's user avatar
  • 11.3k
4 votes

How to perform usability studies on complex software

I think there's a couple of approaches you could try, both using Jobs to be Done as the framework. 1. How are users doing with the intended uses of the product? What jobs did your leaders have in mind ...
Izquierdo's user avatar
  • 12.7k
4 votes

A/B testing - client wanted the test run 70/30

In theory, there isn't a problem with a client-ordered 70/30 test, as long as you're comparing conversion rates and not conversion amounts. If your hypothesis holds up, you'll still see the 5% dip ...
Izquierdo's user avatar
  • 12.7k
4 votes

A/B testing - client wanted the test run 70/30

Run A/B test on 60% of the traffic then. Diverged exactly 50/50 into old and new variants. Diverge all other 40% of users/sessions/views to new experience (as your client requests), but ignore them in ...
Frax's user avatar
  • 141
3 votes

Best way to get feedback from a user on a product?

Bringing the developer along will be beneficial for several reasons. Yes, they may have a chance to answer any specific technical questions but more importantly it will expose them to the overall ...
Cameron's user avatar
  • 31
3 votes

Using both moderated and unmoderated Usability Testing in one research

If I were in this position, I might try treating the two in-person participants as a pilot. Do a study with them, and after you analyze the results, you'll be able to do a higher-quality remote study ...
Phillip Quintero's user avatar
3 votes
Accepted

Where to find 50+ year old people for guerrilla usability testing?

50-100 is a massive demographic. I think you need to think about splitting it into subgroups, as these will behave very differently. Different age ranges have 'grown up' with different technology, ...
PhillipW's user avatar
  • 11.3k
3 votes

When testing a prototype, do you inform your users that everything is not functional?

This comes down to how you conduct your testing. Personally, I always inform my test users that they will be working with a prototype so they cannot 'break' the system and will not destroy any data. ...
Roux Martin's user avatar
  • 14.2k
3 votes

When testing a prototype, do you inform your users that everything is not functional?

If you don't want to immediately make the user aware that they're not using a finished product, then you should definitely make them aware when they go to click on a button (for example) and it isn't ...
Jason Cemra's user avatar
3 votes

Audio for testing purposes

How about generating a short audio from a string that describes the feature? There should be plenty of ways to auto generate this kind of thing (as a c# developer I know it is a few lines of code to ...
Franchesca's user avatar
  • 12.5k
3 votes

Should I trust user tests outside a controlled environment

@Devin's answer is very valuable be sure to understand his point about the difference between methods and methodology, i would like to add : Just to put this one away first While it would have solved ...
UX Labs's user avatar
  • 2,455
3 votes

exploratory testing

If you are trying to validate a concept or determine a user's needs, you are in a generative research phase: you are trying to determine the right thing to do based on conditions your targeted users ...
Mattynabib's user avatar
  • 3,501
3 votes
Accepted

Accessibility testing (with Microsoft Narrator as screenreader)? Somebody here with some practical guidelines?

I wouldn't bother with Narrator too much but by all means add it to your testing suite. You will see from the webAim screenreader study that NVDA, JAWS and VoiceOver make up over 90% of screen readers....
GrahamTheDev's user avatar
  • 2,598
3 votes

Running usability tests with the same people that have passed it previously

Yes, but their input should be treated differently than users who were not a part of the first round, and you certainly still require a group of users who have not previously tested it. To expand as ...
TCooper's user avatar
  • 131
3 votes

Running usability tests with the same people that have passed it previously

In learning we have received from PwC Experience Center on Qualitative Usability testing, it was noted participants should be new ones. Advantages of this are: if they are first time here, they are ...
xul's user avatar
  • 2,676
2 votes

Best method for measuring the user's understanding of a pilot app?

You definitely want to user one-on-one interviews for this. Data can only tell you the what but talking to people will tell you the why. I would structure these interviews as about 60 minutes each. ...
sarahdoody's user avatar
2 votes

Best method for measuring the user's understanding of a pilot app?

As a complimentary method to the @user5482 answer. Conduct diary studies A diary study is a research method used to collect qualitative data about user behaviors, activities, and experiences ...
Kristiyan Lukanov's user avatar
2 votes
Accepted

How to test perception of specific elements on a page?

Start with why you are doing this test: ...just want to know if he notices it and reminds what he saw. Sounds to me like the proposed design is intended to help users notice the elements and ...
Luke Smith's user avatar
  • 1,537
2 votes

How to test perception of specific elements on a page?

Try and have page analytics enabled. You can see the conversion rate (% of viewers that clicked) for this page and the pages you go to when you click the highlighted areas. Do this before and after ...
Nick Groeneveld's user avatar
2 votes

How can one test a design post development

This is borderline off-topic, but I'm answering because it's not purely about implementation. It's question about testing designs. You can easily test pixel perfection using the Firefox add-in ...
dennislees's user avatar
  • 11.6k
2 votes

Guerrilla usability testing

Make the participant feel comfortable Ask the user to do whatever they feel comfortable with to clarify an idea. Some people are going to freeze when you ask them to draw anything. Others will grab ...
plainclothes's user avatar
  • 21.3k
2 votes

Best method for measuring the user's understanding of a pilot app?

The simplest method is to define the end goals that you expect users to accomplish. Using stackexchange as an example, a goal might be: "did the user search for and find an answer to their question?" ...
user5482's user avatar
  • 994
2 votes

What is the best way to test three different versions of a piece of copy?

So does each of these descriptions has anything that distinguish it from the others? For example, does the 1st one talks more about "profitability" ("you can earn money/knowledge/friends if you ...
Solomon's user avatar
  • 91

Only top scored, non community-wiki answers of a minimum length are eligible