Skip to main content

Questions tagged [ab-testing]

Use it for question related to A/B testing

Filter by
Sorted by
Tagged with
1 vote
2 answers
126 views

Test Hypothesis On Multivariate Tests?

I was taught when A/B testing every test should be based on a test hypothesis, e.g. Control: Current Button (Blue) Test: White Button "We believe that changing the CTA colour (British English ...
1 vote
1 answer
133 views

How to determine Minimal Detectable Effect for AB Test with Click-through Rate as success metric?

I am conducting an AB test on my site. I want to decide what is considered a good MDE (Minimal Detectable Effect) when it comes to click-through rate. I can understand you can choose MDE when your ...
0 votes
1 answer
61 views

How to not skew results from usability research?

I wanted to take my class design project up a notch by getting my designs in front of some users through an unmoderated usability test. Because of the platform I used for the test, I could only make ...
2 votes
2 answers
3k views

Should you ever test multiple A/B tests at the same time?

Okay, so you have a website with lots of web pages on it. It has a header/nav and a footer that show up on every page, obviously. And you have a bunch of other unique individual pages. You want to ...
2 votes
3 answers
255 views

Selecting the right user testing method for mobile app

We have a mobile app for different platforms. And now we want to improve the UX and UI. Due to the security policy, we cannot install analytic tools. I'm thinking of options we have to measure our ...
1 vote
2 answers
132 views

Best practices for technically executing AB testing on ecommerce site?

I am working on a site redesign for my company's ecommerce site. We sell access to upcoming and recorded webinars. As a part of this redesign, I have recommended a new product strategy in which ...
14 votes
4 answers
4k views

A/B testing - client wanted the test run 70/30

We recently ran an A/B test for a new sidebar in a checkout flow. New variant 70% of traffic, old 30% of traffic. We tried to get client to run at 50/50 but they were sure our version was an ...
1 vote
2 answers
146 views

How can i interpret the observed difference in a/b test?

I conducted an a/b test but I was confused about how to interpret the result. I got 0.5% difference between control and treatment groups and statistical significance for the difference (p <0.05). ...
0 votes
2 answers
67 views

AB Test Dilemma [closed]

I am running an AB test on a site I launched. If I want to launch a new CTA button on my site and as a result, another section of my site performs worse, how do I account for this or how can I avoid ...
6 votes
2 answers
791 views

A/B testing for mobile apps? [duplicate]

Possible Duplicate: How to A/B test a published IOS and Android app on actual users? Is there any proper way and tool to do A/B testing in mobile apps to find the users' engagement and pain ...
4 votes
1 answer
85 views

Can we use A/B testing for analyze new features?

Hello I'm quite newbie in UX, especially in A/B Testing methods. In my opinion and after I read some of articles. A/B testing usually used for analyze some new call to action button, copywriting, ...
2 votes
3 answers
221 views

How do test/compare an existing function with prototypes

I want to compare an existing function in an existing solution (search filters not being seen and/or used on our corporate intranet) with one or two prototypes. What's the best way to this? How do I ...
3 votes
2 answers
101 views

A/B testing usability with static user count

I'm new to A/B testing and I have few questions. The situation I would be testing an information system with no new users, so user count is more or less constant. In the system there is a big form ...
1 vote
1 answer
154 views

A/B Usability Testing Technique Format?

I'm currently preparing a usability test for a module interface I'm designing and I want to be able to structure the test to ensure the best results. Currently, there are 3 versions of the interface, ...
1 vote
2 answers
82 views

Convention for users to opt out of A/B tests?

An associate was unable to follow a set of basic instructions I made for using a common website. It turned out my user was on some A/B test and was served a different UI. When I checked the same site ...
5 votes
0 answers
58 views

UI/Process Design For A/B Testing [closed]

For fixed horizon A/B tests, the sample size needs to be determined based on alpha, beta, effect size and variance prior to a test. Therefore, in the experiment UI setup page, we need to ask users to ...
0 votes
3 answers
110 views

Did not perform A/B Testing , how to define success of feature

The problem is regarding a educational matching platform. We match Learners together so they and the fellow learners can video call. Learners are entered into a ‘matching pool’ where they can see all ...
5 votes
2 answers
271 views

Which version of A/B do I show if you happen to get different versions in two instances?

Let's say I'm doing A/B testing, and this one particular user gets randomly assigned into group B for example. I have a choice right away to either store that on their account in my database, or on ...
1 vote
1 answer
92 views

Calculating minimal detectable effect with 0 historic conversions

I'm new to conversion rate optimisation, I'm currently prioritising my test ideas based on the effort and impact. Am I right in thinking that the MDE is the minimum uplift from baseline to be ...
1 vote
1 answer
191 views

How important is it to avoid A/B test collisions?

Let's say you have 3 A/B tests as such Control vs A Control vs B Control vs C Unless you specifically work to ensure that tests do not collide (run on top of each other), you will end up with ...
0 votes
3 answers
102 views

A/B/n Testing: All at Once or In Succession?

If I want to A/B test a button color, but for 5 colors, how do I do it? Do I equally split traffic to all 5 button colors for 5 weeks Or… do I run split traffic between red button vs blue button the ...
1 vote
3 answers
148 views

How do I know why people prefer our competitor's web site?

First let me explain the context: We have a web site that helps people to find jobs (e.g. https://www.careerbuilder.com/ https://www.glassdoor.com/index.htm ) In our country, there are some web ...
15 votes
12 answers
4k views

Technical Term for 'Annoyances' [closed]

Is there a technical UX term for an asset/object/function etc that may causes an annoyance to the user? I'm writing up a summary of a test and really don't want to use the word 'annoyances' as it ...
3 votes
1 answer
220 views

A/B testing usability [determine test group size]

There are many A/B tests around but they have one thing in common, they're meant to improve the user conversion or increase the amount of sales. My goal is not to improve the conversions or increase ...
2 votes
1 answer
2k views

What is the minimum number of users i should do AB testing with to be effective?

Is testing with only 1000 user in AB testing, can give us a real data? or it will just be a waste of time?
2 votes
2 answers
972 views

Guidelines for stopping an A/B (or MVT) test early due to negative metrics

A test I've conducted recently reached significance within 24 hours showing a huge drop in conversion. As the site I work on is owned by a large company with high revenue streams, I needed to pause ...
2 votes
2 answers
428 views

How to determine when the A/B testing is needed?

We are developing a product and already tested with more than 50 users. A lot of times users would have ideas to suggest us move/duplicate an element (not common feedback, there are random ideas) I ...
4 votes
3 answers
6k views

What do Sample Size and Effect Size mean in terms of A/B testing?

What does "Sample size" and "effect size" means in terms of A/B testing? Can anyone explain in simple words with an example?
0 votes
2 answers
105 views

Is fair comparison a must in AB testing?

I wonder if the elements in AB test need to be exactly the same so as to allow a fair comparison? I found it personally depending on different scenarios. For example if it's a AB test in smaller ...
0 votes
3 answers
536 views

Is A/B testing the color of your button ethically acceptable?

When doing A/B tests with the color of your button, you are not trying to improve qualitatively (goal of the button, explanation of user motivations). To me it sounds like you are trying to trigger ...
0 votes
1 answer
42 views

How to get feedback and preference choice regarding several design screenshots? [closed]

I would like to show several (2-3) design screenshots of web pages to users and get feedback to their choice. Currently I am using UsabilityHub for that but the presentation of more than two screens ...
0 votes
1 answer
160 views

Testing variables for video autoplay

I'm trying to test to see if having an auto play video helps more interaction with the video. Right now, I have a video on my website with a playlist of videos underneath it. Currently, the video ...
0 votes
1 answer
92 views

A/B testing differences between US and Australia behaviour

I am running couple of a/b tests on Australia and US, tests are exactly same but results they are showing are totally reverse. Since we tend to move successful A/B tests to be hardcoded on the sites ...
17 votes
2 answers
4k views

How much do “where did you hear about us?” type questions impact signup/conversion rates?

There's a truism I often hear - claiming that the less complicated the signup process for a web-service, the higher the conversion rate (as a rough measure of complexity, count mouse clicks to signup ...
5 votes
2 answers
389 views

How do you mitigate internal/external validity issues resulting from user assignment difficulties associated with online A/B testing?

It's reasonably safe to say that it's not possible to track a website's users accurately. You can't track users accurately with cookies, because cookies are frequently cleared; you can't be confident ...
12 votes
5 answers
4k views

How does e-commerce product quick view affect conversion rates?

Many large e-commerce retailers are using "quick views" for products on their collection pages where more details appear either on-hover of via some other trigger. Has anyone performed any A/B tests ...
7 votes
1 answer
151 views

Reliability of Remote UX Tests (Screen cast, A/B)

I'm curious on the community's opinion about reliability in UX testing. This is similar to the question: Purpose of Screen Recording but a little broader. I've discussed with multiple UX designers ...
3 votes
3 answers
188 views

How do I test optional form fields for removal?

I'm currently running a test on optional form input fields. I want to determine if these fields are used, or if some or all of them should be removed. What would be the optimal number, or percentage ...
1 vote
1 answer
110 views

How to determine if the conversion rate is high or low?

The test is to trace CTR. Only variation has button so the test results as below: Original - Visitors: 120170, Conversion rate: 0 Variation - Visitors: 121058, Conversion rate: 15256 Next step is to ...
4 votes
1 answer
2k views

Understanding uplift in A/B testing

Let's say we have a(n) (positive) outcome rate which we try to optimize. Features affects this outcome. In a test, we find that users that used a specific feature X have a higher positive outcome ...
3 votes
3 answers
1k views

Measuring activity on an invision prototype?

I'm wondering if any of you all could help me with this predicament. I am about to finish a certificate in human-computer interaction and the final gauntlet is A/B testing the prototypes we have ...
0 votes
1 answer
78 views

How to AB Test a particular component across the whole site?

I want to do a AB test for the navigation module across the whole site. I know AB/Multivariate tests are usually page specific. Is there any way to do AB Testing across a whole site? So, we can ...
1 vote
2 answers
107 views

a/b testing hundreds of pages at once, with a different layout [closed]

I want to try a new layout for my products page. It would involve a new layout and new color scheme on every products page. There are hundreds of product pages. My question is, how do you a/b test ...
4 votes
3 answers
990 views

Can you carry out AB testing without the site being live?

I am new to A/B testing, i have looked at some sites for more understanding on what it is: https://www.optimizely.com/ab-testing/ https://vwo.com/ab-testing/ http://unbounce.com/landing-page-articles/...
9 votes
4 answers
1k views

How do Enterprise Softwares validate A/B testing?

I just started working for an enterprise software company, and I'm having a hard time integrating validation to see "what works" for the user. I'm used to working for more consumer-based software, ...
2 votes
2 answers
94 views

What are the primary reasons that customers don't purchase AB Testing services?

I work in a UX consulting agency. I noticed while customers are happy to pay good money for our research and design services, AB testing and other post-launch services are seldom being asked for. ...
1 vote
1 answer
93 views

Conversion based on clicks or orders

I'm running A/B split testing using Instapage. My dilemma is whether using clicks or orders versus visitors should be my conversion rate. This is what I had so far A: 556 visitors - 27 conversions (...
3 votes
3 answers
397 views

What are the effects of A/B Testing if the user is aware of it?

I went on the same website on two different browsers, and I got two different layout + design. I was being A/B tested. I didn't like that, probably because: I don't like to be part of an experiment I ...
2 votes
1 answer
190 views

How long should we test every option in an A/B test?

Currently we have an app with a number of features that fit well into a tab navigation bar below the action bar. Because of the vision of having more features in the future, business guys want to have ...
10 votes
3 answers
473 views

What do you do with an inconclusive A/B Test? Keep the change or lose it?

Short Version If you run an A/B Test, and the results are that there's no difference in your key metric, do you keep the experimental version, or revert the site back to the control? Story Version ...