Questions tagged [ab-testing]

A way to test the UX of an application by displaying one version of it to 50% of the users of the application, and displaying another version to the rest of the users. You then compare how the two versions performed by using metrics such as click through rates, the average time on the site, etc.

Filter by
Sorted by
Tagged with
3
votes
2answers
59 views

A/B testing usability with static user count

I'm new to A/B testing and I have few questions. The situation I would be testing an information system with no new users, so user count is more or less constant. In the system there is a big form ...
1
vote
1answer
32 views

A/B Usability Testing Technique Format?

I'm currently preparing a usability test for a module interface I'm designing and I want to be able to structure the test to ensure the best results. Currently, there are 3 versions of the interface, ...
0
votes
0answers
40 views

A/B test in the moderated User Testing

Just wondering if it's common and practical to show our participants another version of our design to get their thoughts? My concern is validating version A without observing version B wouldn't be a ...
0
votes
0answers
26 views

Design test plans & evaluate results for an A/B test

My app aims at switching from monetizing with an ad only, to finding optimal revenue mechanisms between 03 models: ads, in-app purchases, and subscription. I want to increase its LTV at the same time ...
1
vote
2answers
22 views

Convention for users to opt out of A/B tests?

An associate was unable to follow a set of basic instructions I made for using a common website. It turned out my user was on some A/B test and was served a different UI. When I checked the same site ...
3
votes
0answers
41 views

UI/Process Design For A/B Testing

For fixed horizon A/B tests, the sample size needs to be determined based on alpha, beta, effect size and variance prior to a test. Therefore, in the experiment UI setup page, we need to ask users to ...
0
votes
3answers
104 views

Did not perform A/B Testing , how to define success of feature

The problem is regarding a educational matching platform. We match Learners together so they and the fellow learners can video call. Learners are entered into a ‘matching pool’ where they can see all ...
0
votes
2answers
55 views

Test Hypothesis On Multivariate Tests?

I was taught when A/B testing every test should be based on a test hypothesis, e.g. Control: Current Button (Blue) Test: White Button "We believe that changing the CTA colour (British English ...
2
votes
2answers
168 views

Selecting the right user testing method for mobile app

We have a mobile app for different platforms. And now we want to improve the UX and UI. Due to the security policy, we cannot install analytic tools. I'm thinking of options we have to measure our ...
1
vote
1answer
64 views

How important is it to avoid A/B test collisions?

Let's say you have 3 A/B tests as such Control vs A Control vs B Control vs C Unless you specifically work to ensure that tests do not collide (run on top of each other), you will end up with ...
1
vote
1answer
61 views

Calculating minimal detectable effect with 0 historic conversions

I'm new to conversion rate optimisation, I'm currently prioritising my test ideas based on the effort and impact. Am I right in thinking that the MDE is the minimum uplift from baseline to be ...
5
votes
2answers
111 views

Which version of A/B do I show if you happen to get different versions in two instances?

Let's say I'm doing A/B testing, and this one particular user gets randomly assigned into group B for example. I have a choice right away to either store that on their account in my database, or on ...
0
votes
3answers
83 views

A/B/n Testing: All at Once or In Succession?

If I want to A/B test a button color, but for 5 colors, how do I do it? Do I equally split traffic to all 5 button colors for 5 weeks Or… do I run split traffic between red button vs blue button the ...
2
votes
3answers
111 views

How do test/compare an existing function with prototypes

I want to compare an existing function in an existing solution (search filters not being seen and/or used on our corporate intranet) with one or two prototypes. What's the best way to this? How do I ...
1
vote
3answers
143 views

How do I know why people prefer our competitor's web site?

First let me explain the context: We have a web site that helps people to find jobs (e.g. https://www.careerbuilder.com/ https://www.glassdoor.com/index.htm ) In our country, there are some web ...
15
votes
12answers
4k views

Technical Term for 'Annoyances' [closed]

Is there a technical UX term for an asset/object/function etc that may causes an annoyance to the user? I'm writing up a summary of a test and really don't want to use the word 'annoyances' as it ...
3
votes
1answer
155 views

A/B testing usability [determine test group size]

There are many A/B tests around but they have one thing in common, they're meant to improve the user conversion or increase the amount of sales. My goal is not to improve the conversions or increase ...
1
vote
1answer
611 views

What is the minimum number of users i should do AB testing with to be effective?

Is testing with only 1000 user in AB testing, can give us a real data? or it will just be a waste of time?
0
votes
2answers
77 views

Is fair comparison a must in AB testing?

I wonder if the elements in AB test need to be exactly the same so as to allow a fair comparison? I found it personally depending on different scenarios. For example if it's a AB test in smaller ...
2
votes
2answers
302 views

How to determine when the A/B testing is needed?

We are developing a product and already tested with more than 50 users. A lot of times users would have ideas to suggest us move/duplicate an element (not common feedback, there are random ideas) I ...
2
votes
2answers
274 views

Guidelines for stopping an A/B (or MVT) test early due to negative metrics

A test I've conducted recently reached significance within 24 hours showing a huge drop in conversion. As the site I work on is owned by a large company with high revenue streams, I needed to pause ...
0
votes
1answer
37 views

How to get feedback and preference choice regarding several design screenshots? [closed]

I would like to show several (2-3) design screenshots of web pages to users and get feedback to their choice. Currently I am using UsabilityHub for that but the presentation of more than two screens ...
0
votes
3answers
391 views

Is A/B testing the color of your button ethically acceptable?

When doing A/B tests with the color of your button, you are not trying to improve qualitatively (goal of the button, explanation of user motivations). To me it sounds like you are trying to trigger ...
0
votes
1answer
91 views

A/B testing differences between US and Australia behaviour

I am running couple of a/b tests on Australia and US, tests are exactly same but results they are showing are totally reverse. Since we tend to move successful A/B tests to be hardcoded on the sites ...
5
votes
2answers
336 views

How do you mitigate internal/external validity issues resulting from user assignment difficulties associated with online A/B testing?

It's reasonably safe to say that it's not possible to track a website's users accurately. You can't track users accurately with cookies, because cookies are frequently cleared; you can't be confident ...
7
votes
1answer
135 views

Reliability of Remote UX Tests (Screen cast, A/B)

I'm curious on the community's opinion about reliability in UX testing. This is similar to the question: Purpose of Screen Recording but a little broader. I've discussed with multiple UX designers ...
4
votes
3answers
3k views

What do Sample Size and Effect Size mean in terms of A/B testing?

What does "Sample size" and "effect size" means in terms of A/B testing? Can anyone explain in simple words with an example?
3
votes
3answers
150 views

How do I test optional form fields for removal?

I'm currently running a test on optional form input fields. I want to determine if these fields are used, or if some or all of them should be removed. What would be the optimal number, or percentage ...
0
votes
1answer
126 views

Testing variables for video autoplay

I'm trying to test to see if having an auto play video helps more interaction with the video. Right now, I have a video on my website with a playlist of videos underneath it. Currently, the video ...
1
vote
1answer
91 views

How to determine if the conversion rate is high or low?

The test is to trace CTR. Only variation has button so the test results as below: Original - Visitors: 120170, Conversion rate: 0 Variation - Visitors: 121058, Conversion rate: 15256 Next step is to ...
4
votes
1answer
717 views

Understanding uplift in A/B testing

Let's say we have a(n) (positive) outcome rate which we try to optimize. Features affects this outcome. In a test, we find that users that used a specific feature X have a higher positive outcome ...
2
votes
2answers
1k views

Should you ever test multiple A/B tests at the same time?

Okay, so you have a website with lots of web pages on it. It has a header/nav and a footer that show up on every page, obviously. And you have a bunch of other unique individual pages. You want to ...
0
votes
1answer
63 views

How to AB Test a particular component across the whole site?

I want to do a AB test for the navigation module across the whole site. I know AB/Multivariate tests are usually page specific. Is there any way to do AB Testing across a whole site? So, we can ...
3
votes
3answers
1k views

Measuring activity on an invision prototype?

I'm wondering if any of you all could help me with this predicament. I am about to finish a certificate in human-computer interaction and the final gauntlet is A/B testing the prototypes we have ...
1
vote
2answers
98 views

a/b testing hundreds of pages at once, with a different layout [closed]

I want to try a new layout for my products page. It would involve a new layout and new color scheme on every products page. There are hundreds of product pages. My question is, how do you a/b test ...
4
votes
3answers
582 views

Can you carry out AB testing without the site being live?

I am new to A/B testing, i have looked at some sites for more understanding on what it is: https://www.optimizely.com/ab-testing/ https://vwo.com/ab-testing/ http://unbounce.com/landing-page-articles/...
2
votes
1answer
184 views

How long should we test every option in an A/B test?

Currently we have an app with a number of features that fit well into a tab navigation bar below the action bar. Because of the vision of having more features in the future, business guys want to have ...
9
votes
4answers
800 views

How do Enterprise Softwares validate A/B testing?

I just started working for an enterprise software company, and I'm having a hard time integrating validation to see "what works" for the user. I'm used to working for more consumer-based software, ...
2
votes
2answers
88 views

What are the primary reasons that customers don't purchase AB Testing services?

I work in a UX consulting agency. I noticed while customers are happy to pay good money for our research and design services, AB testing and other post-launch services are seldom being asked for. ...
1
vote
1answer
57 views

Conversion based on clicks or orders

I'm running A/B split testing using Instapage. My dilemma is whether using clicks or orders versus visitors should be my conversion rate. This is what I had so far A: 556 visitors - 27 conversions (...
3
votes
3answers
260 views

What are the effects of A/B Testing if the user is aware of it?

I went on the same website on two different browsers, and I got two different layout + design. I was being A/B tested. I didn't like that, probably because: I don't like to be part of an experiment I ...
1
vote
2answers
53 views

Direction of fretboard

I know it might be a bit difficult to find musicians who have experience with UX, but I'll give my question a try here. I am designing an app for musicians and need an opinion on something: Which ...
4
votes
5answers
786 views

Which spot has the best conversion in the grid layout?

We are going to show popup with 6 different offers, positioned by 3 in two rows. Of the these 6 offers, 2 are really beneficial for us (say, have high reward for click). Common sense says that we ...
3
votes
5answers
299 views

A/B testing in user test

I designed 4 versions of a checkout page. I want to make some user test to know which version performs better. Do you think that I have to do the tests with the same person? the big change in the ...
12
votes
5answers
4k views

How does e-commerce product quick view affect conversion rates?

Many large e-commerce retailers are using "quick views" for products on their collection pages where more details appear either on-hover of via some other trigger. Has anyone performed any A/B tests ...
6
votes
1answer
148 views

Conducting different tests on a website at once

We have our own A/B test software which works this way: Every time a user visits a page with a test, he gets a cookie marker. For variant A in test one it may look like this: test1_a For test 2 and ...
0
votes
2answers
98 views

How to test for the best colour of an online store?

I already ran a test set up as following: I told the users what they were going to buy, showed them the same webpage in eight different colours (four webpages on two rows), and asked them on which ...
3
votes
0answers
70 views

How can I learn A/B testing independently? [closed]

I'm looking for an independent project to learn A/B testing on, and I'm trying to figure out which A/B testing tools are commonly used for: Mobile Games Web And I'm also aware that the common ...
1
vote
3answers
236 views

For landing page A/B testing, how do you recruit from a niche market?

I'm currently working with a client that has not launched the full website yet, but is A/B testing ads to drive member acquisition. The ads direct users to a landing page that collects user e-mails. ...
-2
votes
3answers
149 views

A/B or multi-variate testing as replacement for UX design?

It seems like in a business-driven organization (i.e. where the sales and marketing team has a bigger say) that offer software products or services, A/B testing is used as a way to improve and measure ...