Skip to main content

Questions tagged [ab-testing]

Use it for question related to A/B testing

Filter by
Sorted by
Tagged with
17 votes
2 answers
4k views

How much do “where did you hear about us?” type questions impact signup/conversion rates?

There's a truism I often hear - claiming that the less complicated the signup process for a web-service, the higher the conversion rate (as a rough measure of complexity, count mouse clicks to signup ...
blueberryfields's user avatar
16 votes
4 answers
661 views

A/B Testing Technique

My friend and I were having a discussion about different ways of A/B testing. His technique was to make a single small change (e.g. changing the colour of a button) and then perform A/B Testing to see ...
Mike's user avatar
  • 263
15 votes
12 answers
4k views

Technical Term for 'Annoyances' [closed]

Is there a technical UX term for an asset/object/function etc that may causes an annoyance to the user? I'm writing up a summary of a test and really don't want to use the word 'annoyances' as it ...
sclarke's user avatar
  • 1,895
14 votes
4 answers
4k views

A/B testing - client wanted the test run 70/30

We recently ran an A/B test for a new sidebar in a checkout flow. New variant 70% of traffic, old 30% of traffic. We tried to get client to run at 50/50 but they were sure our version was an ...
Ellouise's user avatar
  • 141
14 votes
2 answers
316 views

How to realistically test blog headlines before publishing?

I'm currently writing a post for my company's blog. I'm agonising over the headline, as I really, really want people to read this post (I'm trying to convince our users to opt in to our customer ...
Mal Ross's user avatar
  • 864
12 votes
5 answers
504 views

Tips for A/B testing?

There seems to an increasing number of A/B testing services (such as Google Website Optimiser) and people using them. Have you got any tips on using and analysing the results from them?
user avatar
12 votes
5 answers
4k views

How does e-commerce product quick view affect conversion rates?

Many large e-commerce retailers are using "quick views" for products on their collection pages where more details appear either on-hover of via some other trigger. Has anyone performed any A/B tests ...
Alex Czarto's user avatar
11 votes
4 answers
276 views

A/B Testing. How should it be done?

I just stumbled upon Google Website Optimizer. Is this the way to go? How do you normaly do A/B tests with your website?
user avatar
10 votes
3 answers
473 views

What do you do with an inconclusive A/B Test? Keep the change or lose it?

Short Version If you run an A/B Test, and the results are that there's no difference in your key metric, do you keep the experimental version, or revert the site back to the control? Story Version ...
fitzgeraldsteele's user avatar
9 votes
4 answers
1k views

How do Enterprise Softwares validate A/B testing?

I just started working for an enterprise software company, and I'm having a hard time integrating validation to see "what works" for the user. I'm used to working for more consumer-based software, ...
CleverNode's user avatar
8 votes
1 answer
373 views

A/B testing a site with low traffic

I've always been very interested in A/B testing, but have never had the opportunity to implement it on any websites. I work at a digital agency where the main reason why we haven't been able to ...
Rich's user avatar
  • 3,532
7 votes
5 answers
350 views

What are the best A/B tools? [closed]

I have a few pages to A/B test, and I am looking for the best tools to use for this? Is there a "standard" or favorite industry tool?
johnnietheblack's user avatar
7 votes
2 answers
354 views

A/B test shows unexplainable winner for a remote goal

We ran a test where we tested 3 different button colors for a Call to Action. The CTA led to our offers page (where we had special offers to sign up for a TV package). But of the 3 variations, none ...
Charles Shimooka's user avatar
7 votes
1 answer
151 views

Reliability of Remote UX Tests (Screen cast, A/B)

I'm curious on the community's opinion about reliability in UX testing. This is similar to the question: Purpose of Screen Recording but a little broader. I've discussed with multiple UX designers ...
Ash Hogarth's user avatar
7 votes
3 answers
435 views

Better purchase conversions for single page vs. multiple?

I'm mocking up a page for purchasing our product, and I'm wondering is it better to do everything on one page or split the ordering and payment into two different pages? Anyone have empirical data on ...
Nate Reed's user avatar
  • 173
7 votes
1 answer
1k views

Are simultaneous tests on different stages of a multi-step form statistically independent?

I want to do some test-and-learn on a multi-step form using infrastructure which appears to support conducting and reporting of A/B/n tests. If my site had two different forms I could simultaneously ...
Francis Norton's user avatar
6 votes
10 answers
729 views

Isn't all user testing essentially a way of gathering how people think things should work?

When user testing is performed, users are often presented with a serious of variations of a product and observed to see where they excelled and where they had trouble. People will often click in areas ...
Matt Rockwell's user avatar
6 votes
2 answers
429 views

Unexpected A/B testing results

We've just run an A/B test for one of our landing pages using Visual Website Optimizer. We are an online hotel booking reservation site. The landing page shows information about a particular hotel (...
Jose's user avatar
  • 61
6 votes
4 answers
2k views

How to A/B test a published IOS and Android app on actual users?

I've published several apps on IOS and Android adding flurry to measure the interaction that is happening in those apps, however, I'd love to enhance this interaction and make my apps more useful by ...
Abdullah Al-Khalidi's user avatar
6 votes
1 answer
4k views

A/B Testing vs Cohort Analysis

I had a few questions about A/B Testing vs Cohort Analysis (dividing users based on registration time). 1.) Is A/B testing mainly used for UX design changes (such as change in Sign Up button Color) ...
user1383202's user avatar
6 votes
2 answers
251 views

Are there any easy to use A/B Testing frameworks?

How to do A/B testing is a difficult problem. Are there any frameworks or tools out there that ease the pain?
tooshel's user avatar
  • 239
6 votes
2 answers
791 views

A/B testing for mobile apps? [duplicate]

Possible Duplicate: How to A/B test a published IOS and Android app on actual users? Is there any proper way and tool to do A/B testing in mobile apps to find the users' engagement and pain ...
Pratheep ch's user avatar
  • 1,515
6 votes
1 answer
161 views

Conducting different tests on a website at once

We have our own A/B test software which works this way: Every time a user visits a page with a test, he gets a cookie marker. For variant A in test one it may look like this: test1_a For test 2 and ...
user39061's user avatar
5 votes
5 answers
514 views

A/B Testing Homepage Variants

When trying out new homepage layouts / styles, is A/B testing the best way to do this? With typical A/B tests you have a specific conversion measurement (i.e. version X drives significantly more ...
JonW's user avatar
  • 37.4k
5 votes
2 answers
270 views

Which version of A/B do I show if you happen to get different versions in two instances?

Let's say I'm doing A/B testing, and this one particular user gets randomly assigned into group B for example. I have a choice right away to either store that on their account in my database, or on ...
Digital Ninja's user avatar
5 votes
2 answers
389 views

How do you mitigate internal/external validity issues resulting from user assignment difficulties associated with online A/B testing?

It's reasonably safe to say that it's not possible to track a website's users accurately. You can't track users accurately with cookies, because cookies are frequently cleared; you can't be confident ...
Dreadnaught's user avatar
5 votes
0 answers
58 views

UI/Process Design For A/B Testing [closed]

For fixed horizon A/B tests, the sample size needs to be determined based on alpha, beta, effect size and variance prior to a test. Therefore, in the experiment UI setup page, we need to ask users to ...
etang's user avatar
  • 151
4 votes
3 answers
6k views

What do Sample Size and Effect Size mean in terms of A/B testing?

What does "Sample size" and "effect size" means in terms of A/B testing? Can anyone explain in simple words with an example?
Jitendra Vyas's user avatar
4 votes
5 answers
859 views

Which spot has the best conversion in the grid layout?

We are going to show popup with 6 different offers, positioned by 3 in two rows. Of the these 6 offers, 2 are really beneficial for us (say, have high reward for click). Common sense says that we ...
RInatM's user avatar
  • 143
4 votes
3 answers
990 views

Can you carry out AB testing without the site being live?

I am new to A/B testing, i have looked at some sites for more understanding on what it is: https://www.optimizely.com/ab-testing/ https://vwo.com/ab-testing/ http://unbounce.com/landing-page-articles/...
kiempoturner's user avatar
4 votes
1 answer
85 views

Can we use A/B testing for analyze new features?

Hello I'm quite newbie in UX, especially in A/B Testing methods. In my opinion and after I read some of articles. A/B testing usually used for analyze some new call to action button, copywriting, ...
Nizariansyah Agung's user avatar
4 votes
1 answer
2k views

Understanding uplift in A/B testing

Let's say we have a(n) (positive) outcome rate which we try to optimize. Features affects this outcome. In a test, we find that users that used a specific feature X have a higher positive outcome ...
WJA's user avatar
  • 153
4 votes
1 answer
303 views

A/B Testing Home Page of an Ecommerce Site

I am very new to any sort of website testing (please be gentle!), and I have some questions (mostly just for my own understanding) about A/B testing an ecommerce site's home page. The company I work ...
NewTester's user avatar
3 votes
4 answers
1k views

Testing OK/Cancel buttons

Related: OK/Cancel on left/right? I have been looking in to whether OK/Cancel should be on left or right, and after looking at lots of views on each side it seems there are good reasons for both. I ...
TJH's user avatar
  • 2,689
3 votes
4 answers
2k views

Who should be responsible for A/B testing and conversion optimization?

Currently, I'm wearing many hats, but looking to expand our team and eventually dedicate someone to running A/B, MVT and conversion optimizations for us. With companies where even minor % ...
ljs.dev's user avatar
  • 279
3 votes
5 answers
341 views

A/B testing in user test

I designed 4 versions of a checkout page. I want to make some user test to know which version performs better. Do you think that I have to do the tests with the same person? the big change in the ...
Matteo Vacca's user avatar
3 votes
5 answers
455 views

Is A/B testing a worthwhile use of resources for the end user of a mobile app?

Web sites can enable A/B testing by serving a different page, for example. A mobile app would have to collect data and return it. As a matter of principle, is it okay to have software "phone home" ...
Don's user avatar
  • 866
3 votes
2 answers
1k views

Increasing add-to-cart rate, decreasing conversion?

I am currently optimising an e-commerce website. The request was specific to the product pages, after gaining insight from analytics and feedback two alternatives were proposed, and tested. The add-...
Davide's user avatar
  • 561
3 votes
1 answer
250 views

Which people do I choose for large scale A/B testing?

What are the best practices for deciding which users to admit to a large scale A/B test on a website? Below are two of my concerns on this choosing process. Veteran users' aversion to change Humans ...
JoJo's user avatar
  • 10.9k
3 votes
1 answer
220 views

A/B testing usability [determine test group size]

There are many A/B tests around but they have one thing in common, they're meant to improve the user conversion or increase the amount of sales. My goal is not to improve the conversions or increase ...
Bjorneh's user avatar
  • 33
3 votes
3 answers
188 views

How do I test optional form fields for removal?

I'm currently running a test on optional form input fields. I want to determine if these fields are used, or if some or all of them should be removed. What would be the optimal number, or percentage ...
Danny M.'s user avatar
3 votes
3 answers
1k views

Measuring activity on an invision prototype?

I'm wondering if any of you all could help me with this predicament. I am about to finish a certificate in human-computer interaction and the final gauntlet is A/B testing the prototypes we have ...
Mitch's user avatar
  • 31
3 votes
3 answers
397 views

What are the effects of A/B Testing if the user is aware of it?

I went on the same website on two different browsers, and I got two different layout + design. I was being A/B tested. I didn't like that, probably because: I don't like to be part of an experiment I ...
dayuloli's user avatar
  • 266
3 votes
2 answers
101 views

A/B testing usability with static user count

I'm new to A/B testing and I have few questions. The situation I would be testing an information system with no new users, so user count is more or less constant. In the system there is a big form ...
dace's user avatar
  • 33
3 votes
0 answers
71 views

How can I learn A/B testing independently? [closed]

I'm looking for an independent project to learn A/B testing on, and I'm trying to figure out which A/B testing tools are commonly used for: Mobile Games Web And I'm also aware that the common ...
user422318's user avatar
2 votes
2 answers
428 views

How to determine when the A/B testing is needed?

We are developing a product and already tested with more than 50 users. A lot of times users would have ideas to suggest us move/duplicate an element (not common feedback, there are random ideas) I ...
YogaPanda's user avatar
  • 214
2 votes
2 answers
971 views

Guidelines for stopping an A/B (or MVT) test early due to negative metrics

A test I've conducted recently reached significance within 24 hours showing a huge drop in conversion. As the site I work on is owned by a large company with high revenue streams, I needed to pause ...
sclarke's user avatar
  • 1,895
2 votes
1 answer
140 views

Multivariate test conversion: short vs long path

Let's assume we a workflow with 3 steps. Page 1 is our landing page. Page 2 is the form (signup). Page 3 is the conversion page (thank you page). To test variations on page 1, I am running a ...
Rdpi's user avatar
  • 697
2 votes
2 answers
3k views

Should you ever test multiple A/B tests at the same time?

Okay, so you have a website with lots of web pages on it. It has a header/nav and a footer that show up on every page, obviously. And you have a bunch of other unique individual pages. You want to ...
Jake Wilson's user avatar
2 votes
1 answer
183 views

Is there any testing methodology similar to A/B testing, but without a specific goal?

Please forgive me; this may be somewhat long-winded. I work in an academic library. I'd like to study how well assorted features of our site are serving our users' needs; for example, labeling of UI ...
Will Martin's user avatar