Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [ab-testing]

A way to test the UX of an application by displaying one version of it to 50% of the users of the application, and displaying another version to the rest of the users. You then compare how the two versions performed by using metrics such as click through rates, the average time on the site, etc.

1
vote
2answers
42 views

How do test/compare an existing function with prototypes

I want to compare an existing function in an existing solution (search filters not being seen and/or used on our corporate intranet) with one or two prototypes. What's the best way to this? How do I ...
1
vote
3answers
136 views

How do I know why people prefer our competitor's web site?

First let me explain the context: We have a web site that helps people to find jobs (e.g. https://www.careerbuilder.com/ https://www.glassdoor.com/index.htm ) In our country, there are some web ...
15
votes
12answers
4k views

Technical Term for 'Annoyances' [closed]

Is there a technical UX term for an asset/object/function etc that may causes an annoyance to the user? I'm writing up a summary of a test and really don't want to use the word 'annoyances' as it ...
2
votes
1answer
117 views

A/B testing usability [determine test group size]

There are many A/B tests around but they have one thing in common, they're meant to improve the user conversion or increase the amount of sales. My goal is not to improve the conversions or increase ...
0
votes
1answer
267 views

What is the minimum number of users i should do AB testing with to be effective?

Is testing with only 1000 user in AB testing, can give us a real data? or it will just be a waste of time?
0
votes
2answers
71 views

Is fair comparison a must in AB testing?

I wonder if the elements in AB test need to be exactly the same so as to allow a fair comparison? I found it personally depending on different scenarios. For example if it's a AB test in smaller ...
1
vote
2answers
130 views

How to determine when the A/B testing is needed?

We are developing a product and already tested with more than 50 users. A lot of times users would have ideas to suggest us move/duplicate an element (not common feedback, there are random ideas) I ...
1
vote
2answers
127 views

Guidelines for stopping an A/B (or MVT) test early due to negative metrics

A test I've conducted recently reached significance within 24 hours showing a huge drop in conversion. As the site I work on is owned by a large company with high revenue streams, I needed to pause ...
0
votes
1answer
35 views

How to get feedback and preference choice regarding several design screenshots? [closed]

I would like to show several (2-3) design screenshots of web pages to users and get feedback to their choice. Currently I am using UsabilityHub for that but the presentation of more than two screens ...
0
votes
3answers
347 views

Is A/B testing the color of your button ethically acceptable?

When doing A/B tests with the color of your button, you are not trying to improve qualitatively (goal of the button, explanation of user motivations). To me it sounds like you are trying to trigger ...
0
votes
1answer
90 views

A/B testing differences between US and Australia behaviour

I am running couple of a/b tests on Australia and US, tests are exactly same but results they are showing are totally reverse. Since we tend to move successful A/B tests to be hardcoded on the sites ...
5
votes
2answers
213 views

How do you mitigate internal/external validity issues resulting from user assignment difficulties associated with online A/B testing?

It's reasonably safe to say that it's not possible to track a website's users accurately. You can't track users accurately with cookies, because cookies are frequently cleared; you can't be confident ...
7
votes
1answer
112 views

Reliability of Remote UX Tests (Screen cast, A/B)

I'm curious on the community's opinion about reliability in UX testing. This is similar to the question: Purpose of Screen Recording but a little broader. I've discussed with multiple UX designers ...
5
votes
3answers
2k views

What do Sample Size and Effect Size mean in terms of A/B testing?

What does "Sample size" and "effect size" means in terms of A/B testing? Can anyone explain in simple words with an example?
3
votes
3answers
137 views

How do I test optional form fields for removal?

I'm currently running a test on optional form input fields. I want to determine if these fields are used, or if some or all of them should be removed. What would be the optimal number, or percentage ...
0
votes
1answer
122 views

Testing variables for video autoplay

I'm trying to test to see if having an auto play video helps more interaction with the video. Right now, I have a video on my website with a playlist of videos underneath it. Currently, the video ...
1
vote
1answer
80 views

How to determine if the conversion rate is high or low?

The test is to trace CTR. Only variation has button so the test results as below: Original - Visitors: 120170, Conversion rate: 0 Variation - Visitors: 121058, Conversion rate: 15256 Next ...
3
votes
1answer
313 views

Understanding uplift in A/B testing

Let's say we have a(n) (positive) outcome rate which we try to optimize. Features affects this outcome. In a test, we find that users that used a specific feature X have a higher positive outcome ...
1
vote
2answers
519 views

Should you ever test multiple A/B tests at the same time?

Okay, so you have a website with lots of web pages on it. It has a header/nav and a footer that show up on every page, obviously. And you have a bunch of other unique individual pages. You want to ...
0
votes
1answer
58 views

How to AB Test a particular component across the whole site?

I want to do a AB test for the navigation module across the whole site. I know AB/Multivariate tests are usually page specific. Is there any way to do AB Testing across a whole site? So, we can ...
4
votes
3answers
835 views

Measuring activity on an invision prototype?

I'm wondering if any of you all could help me with this predicament. I am about to finish a certificate in human-computer interaction and the final gauntlet is A/B testing the prototypes we have ...
1
vote
2answers
94 views

a/b testing hundreds of pages at once, with a different layout [closed]

I want to try a new layout for my products page. It would involve a new layout and new color scheme on every products page. There are hundreds of product pages. My question is, how do you a/b test ...
5
votes
3answers
402 views

Can you carry out AB testing without the site being live?

I am new to A/B testing, i have looked at some sites for more understanding on what it is: https://www.optimizely.com/ab-testing/ https://vwo.com/ab-testing/ http://unbounce.com/landing-page-articles/...
2
votes
1answer
173 views

How long should we test every option in an A/B test?

Currently we have an app with a number of features that fit well into a tab navigation bar below the action bar. Because of the vision of having more features in the future, business guys want to have ...
8
votes
4answers
574 views

How do Enterprise Softwares validate A/B testing?

I just started working for an enterprise software company, and I'm having a hard time integrating validation to see "what works" for the user. I'm used to working for more consumer-based software, ...
2
votes
2answers
86 views

What are the primary reasons that customers don't purchase AB Testing services?

I work in a UX consulting agency. I noticed while customers are happy to pay good money for our research and design services, AB testing and other post-launch services are seldom being asked for. ...
1
vote
1answer
54 views

Conversion based on clicks or orders

I'm running A/B split testing using Instapage. My dilemma is whether using clicks or orders versus visitors should be my conversion rate. This is what I had so far A: 556 visitors - 27 conversions (...
3
votes
3answers
239 views

What are the effects of A/B Testing if the user is aware of it?

I went on the same website on two different browsers, and I got two different layout + design. I was being A/B tested. I didn't like that, probably because: I don't like to be part of an experiment I ...
1
vote
2answers
53 views

Direction of fretboard

I know it might be a bit difficult to find musicians who have experience with UX, but I'll give my question a try here. I am designing an app for musicians and need an opinion on something: Which ...
4
votes
5answers
756 views

Which spot has the best conversion in the grid layout?

We are going to show popup with 6 different offers, positioned by 3 in two rows. Of the these 6 offers, 2 are really beneficial for us (say, have high reward for click). Common sense says that we ...
3
votes
5answers
289 views

A/B testing in user test

I designed 4 versions of a checkout page. I want to make some user test to know which version performs better. Do you think that I have to do the tests with the same person? the big change in the ...
12
votes
5answers
3k views

How does e-commerce product quick view affect conversion rates?

Many large e-commerce retailers are using "quick views" for products on their collection pages where more details appear either on-hover of via some other trigger. Has anyone performed any A/B tests ...
6
votes
1answer
130 views

Conducting different tests on a website at once

We have our own A/B test software which works this way: Every time a user visits a page with a test, he gets a cookie marker. For variant A in test one it may look like this: test1_a For test 2 and ...
0
votes
2answers
95 views

How to test for the best colour of an online store?

I already ran a test set up as following: I told the users what they were going to buy, showed them the same webpage in eight different colours (four webpages on two rows), and asked them on which ...
3
votes
0answers
70 views

How can I learn A/B testing independently? [closed]

I'm looking for an independent project to learn A/B testing on, and I'm trying to figure out which A/B testing tools are commonly used for: Mobile Games Web And I'm also aware that the common ...
1
vote
3answers
226 views

For landing page A/B testing, how do you recruit from a niche market?

I'm currently working with a client that has not launched the full website yet, but is A/B testing ads to drive member acquisition. The ads direct users to a landing page that collects user e-mails. ...
-2
votes
3answers
148 views

A/B or multi-variate testing as replacement for UX design?

It seems like in a business-driven organization (i.e. where the sales and marketing team has a bigger say) that offer software products or services, A/B testing is used as a way to improve and measure ...
4
votes
1answer
278 views

A/B Testing Home Page of an Ecommerce Site

I am very new to any sort of website testing (please be gentle!), and I have some questions (mostly just for my own understanding) about A/B testing an ecommerce site's home page. The company I work ...
6
votes
1answer
2k views

A/B Testing vs Cohort Analysis

I had a few questions about A/B Testing vs Cohort Analysis (dividing users based on registration time). 1.) Is A/B testing mainly used for UX design changes (such as change in Sign Up button Color) ...
16
votes
4answers
607 views

A/B Testing Technique

My friend and I were having a discussion about different ways of A/B testing. His technique was to make a single small change (e.g. changing the colour of a button) and then perform A/B Testing to see ...
14
votes
2answers
286 views

How to realistically test blog headlines before publishing?

I'm currently writing a post for my company's blog. I'm agonising over the headline, as I really, really want people to read this post (I'm trying to convince our users to opt in to our customer ...
8
votes
1answer
358 views

A/B testing a site with low traffic

I've always been very interested in A/B testing, but have never had the opportunity to implement it on any websites. I work at a digital agency where the main reason why we haven't been able to ...
2
votes
2answers
124 views

What metric should I use for A/B testing of a blog homepage?

We are launching a blog about web design. The goal of our blog is of an ordinary content site - to get visitors to read as many articles as possible (NOT to get ad impressions). Our designer has ...
7
votes
2answers
341 views

A/B test shows unexplainable winner for a remote goal

We ran a test where we tested 3 different button colors for a Call to Action. The CTA led to our offers page (where we had special offers to sign up for a TV package). But of the 3 variations, none ...
1
vote
1answer
152 views

I have a few significantly different layouts I want to test, is A/B or MVT testing the best approach?

I've read that A/B testing and MVT are best for testing smaller scale changes (i.e. button color, headline copy, etc), but are they also suitable for testing two (or more) very different designs? What'...
3
votes
4answers
704 views

Who should be responsible for A/B testing and conversion optimization?

Currently, I'm wearing many hats, but looking to expand our team and eventually dedicate someone to running A/B, MVT and conversion optimizations for us. With companies where even minor % ...
1
vote
1answer
202 views

A/B Testing and Duplicate Content [closed]

Now that Google can crawl/index content from JavaScript, how are page variants treated by Google by serving a slightly different versions when A/B testing? Is it advisable to add references to the ...
3
votes
4answers
531 views

Testing OK/Cancel buttons

Related: OK/Cancel on left/right? I have been looking in to whether OK/Cancel should be on left or right, and after looking at lots of views on each side it seems there are good reasons for both. I ...
6
votes
2answers
395 views

Unexpected A/B testing results

We've just run an A/B test for one of our landing pages using Visual Website Optimizer. We are an online hotel booking reservation site. The landing page shows information about a particular hotel (...
6
votes
4answers
2k views

How to A/B test a published IOS and Android app on actual users?

I've published several apps on IOS and Android adding flurry to measure the interaction that is happening in those apps, however, I'd love to enhance this interaction and make my apps more useful by ...