Questions tagged [ab-testing]
Use it for question related to A/B testing
85
questions
14
votes
4
answers
4k
views
A/B testing - client wanted the test run 70/30
We recently ran an A/B test for a new sidebar in a checkout flow. New variant 70% of traffic, old 30% of traffic. We tried to get client to run at 50/50 but they were sure our version was an ...
- 141
0
votes
1
answer
36
views
How to not skew results from usability research?
I wanted to take my class design project up a notch by getting my designs in front of some users through an unmoderated usability test. Because of the platform I used for the test, I could only make ...
1
vote
2
answers
94
views
How can i interpret the observed difference in a/b test?
I conducted an a/b test but I was confused about how to interpret the result.
I got 0.5% difference between control and treatment groups and statistical significance for the difference (p <0.05).
...
- 11
1
vote
1
answer
79
views
How to determine Minimal Detectable Effect for AB Test with Click-through Rate as success metric?
I am conducting an AB test on my site. I want to decide what is considered a good MDE (Minimal Detectable Effect) when it comes to click-through rate. I can understand you can choose MDE when your ...
- 113
0
votes
2
answers
67
views
AB Test Dilemma [closed]
I am running an AB test on a site I launched. If I want to launch a new CTA button on my site and as a result, another section of my site performs worse, how do I account for this or how can I avoid ...
- 113
4
votes
1
answer
78
views
Can we use A/B testing for analyze new features?
Hello I'm quite newbie in UX, especially in A/B Testing methods.
In my opinion and after I read some of articles. A/B testing usually used for analyze some new call to action button, copywriting, ...
1
vote
1
answer
104
views
Best practices for technically executing AB testing on ecommerce site?
I am working on a site redesign for my company's ecommerce site. We sell access to upcoming and recorded webinars. As a part of this redesign, I have recommended a new product strategy in which ...
- 41
3
votes
2
answers
99
views
A/B testing usability with static user count
I'm new to A/B testing and I have few questions.
The situation
I would be testing an information system with no new users, so user count is more or less constant. In the system there is a big form ...
- 33
1
vote
1
answer
106
views
A/B Usability Testing Technique Format?
I'm currently preparing a usability test for a module interface I'm designing and I want to be able to structure the test to ensure the best results.
Currently, there are 3 versions of the interface, ...
- 369
1
vote
2
answers
44
views
Convention for users to opt out of A/B tests?
An associate was unable to follow a set of basic instructions I made for using a common website. It turned out my user was on some A/B test and was served a different UI. When I checked the same site ...
- 289
5
votes
0
answers
58
views
UI/Process Design For A/B Testing [closed]
For fixed horizon A/B tests, the sample size needs to be determined based on alpha, beta, effect size and variance prior to a test. Therefore, in the experiment UI setup page, we need to ask users to ...
- 151
0
votes
3
answers
106
views
Did not perform A/B Testing , how to define success of feature
The problem is regarding a educational matching platform. We match Learners together so they and the fellow learners can video call. Learners are entered into a ‘matching pool’ where they can see all ...
1
vote
2
answers
118
views
Test Hypothesis On Multivariate Tests?
I was taught when A/B testing every test should be based on a test hypothesis, e.g.
Control: Current Button (Blue)
Test: White Button
"We believe that changing the CTA colour (British English ...
2
votes
2
answers
236
views
Selecting the right user testing method for mobile app
We have a mobile app for different platforms. And now we want to improve the UX and UI. Due to the security policy, we cannot install analytic tools. I'm thinking of options we have to measure our ...
1
vote
1
answer
150
views
How important is it to avoid A/B test collisions?
Let's say you have 3 A/B tests as such
Control vs A
Control vs B
Control vs C
Unless you specifically work to ensure that tests do not collide (run on top of each other), you will end up with ...
- 151
1
vote
1
answer
89
views
Calculating minimal detectable effect with 0 historic conversions
I'm new to conversion rate optimisation, I'm currently prioritising my test ideas based on the effort and impact.
Am I right in thinking that the MDE is the minimum uplift from baseline to be ...
- 61
5
votes
2
answers
155
views
Which version of A/B do I show if you happen to get different versions in two instances?
Let's say I'm doing A/B testing, and this one particular user gets randomly assigned into group B for example. I have a choice right away to either store that on their account in my database, or on ...
- 171
0
votes
3
answers
101
views
A/B/n Testing: All at Once or In Succession?
If I want to A/B test a button color, but for 5 colors, how do I do it?
Do I equally split traffic to all 5 button colors for 5 weeks
Or… do I run split traffic between red button vs blue button the ...
- 103
2
votes
3
answers
209
views
How do test/compare an existing function with prototypes
I want to compare an existing function in an existing solution (search filters not being seen and/or used on our corporate intranet) with one or two prototypes.
What's the best way to this? How do I ...
- 21
1
vote
3
answers
145
views
How do I know why people prefer our competitor's web site?
First let me explain the context:
We have a web site that helps people to find jobs (e.g. https://www.careerbuilder.com/ https://www.glassdoor.com/index.htm )
In our country, there are some web ...
- 63
15
votes
12
answers
4k
views
Technical Term for 'Annoyances' [closed]
Is there a technical UX term for an asset/object/function etc that may causes an annoyance to the user?
I'm writing up a summary of a test and really don't want to use the word 'annoyances' as it ...
- 1,855
3
votes
1
answer
210
views
A/B testing usability [determine test group size]
There are many A/B tests around but they have one thing in common, they're meant to improve the user conversion or increase the amount of sales. My goal is not to improve the conversions or increase ...
- 33
2
votes
1
answer
2k
views
What is the minimum number of users i should do AB testing with to be effective?
Is testing with only 1000 user in AB testing, can give us a real data? or it will just be a waste of time?
- 159
0
votes
2
answers
101
views
Is fair comparison a must in AB testing?
I wonder if the elements in AB test need to be exactly the same so as to allow a fair comparison?
I found it personally depending on different scenarios.
For example if it's a AB test in smaller ...
- 115
2
votes
2
answers
387
views
How to determine when the A/B testing is needed?
We are developing a product and already tested with more than 50 users.
A lot of times users would have ideas to suggest us move/duplicate an element (not common feedback, there are random ideas)
I ...
- 214
2
votes
2
answers
737
views
Guidelines for stopping an A/B (or MVT) test early due to negative metrics
A test I've conducted recently reached significance within 24 hours showing a huge drop in conversion. As the site I work on is owned by a large company with high revenue streams, I needed to pause ...
- 1,855
0
votes
1
answer
42
views
How to get feedback and preference choice regarding several design screenshots? [closed]
I would like to show several (2-3) design screenshots of web pages to users and get feedback to their choice.
Currently I am using UsabilityHub for that but the presentation of more than two screens ...
- 145
0
votes
3
answers
492
views
Is A/B testing the color of your button ethically acceptable?
When doing A/B tests with the color of your button, you are not trying to improve qualitatively (goal of the button, explanation of user motivations).
To me it sounds like you are trying to trigger ...
- 49
0
votes
1
answer
92
views
A/B testing differences between US and Australia behaviour
I am running couple of a/b tests on Australia and US, tests are exactly same but results they are showing are totally reverse. Since we tend to move successful A/B tests to be hardcoded on the sites ...
- 525
5
votes
2
answers
386
views
How do you mitigate internal/external validity issues resulting from user assignment difficulties associated with online A/B testing?
It's reasonably safe to say that it's not possible to track a website's users accurately. You can't track users accurately with cookies, because cookies are frequently cleared; you can't be confident ...
- 53
7
votes
1
answer
150
views
Reliability of Remote UX Tests (Screen cast, A/B)
I'm curious on the community's opinion about reliability in UX testing.
This is similar to the question:
Purpose of Screen Recording
but a little broader.
I've discussed with multiple UX designers ...
- 175
4
votes
3
answers
6k
views
What do Sample Size and Effect Size mean in terms of A/B testing?
What does "Sample size" and "effect size" means in terms of A/B testing? Can anyone explain in simple words with an example?
- 4,872
3
votes
3
answers
186
views
How do I test optional form fields for removal?
I'm currently running a test on optional form input fields. I want to determine if these fields are used, or if some or all of them should be removed.
What would be the optimal number, or percentage ...
- 33
0
votes
1
answer
156
views
Testing variables for video autoplay
I'm trying to test to see if having an auto play video helps more interaction with the video. Right now, I have a video on my website with a playlist of videos underneath it. Currently, the video ...
- 41
1
vote
1
answer
110
views
How to determine if the conversion rate is high or low?
The test is to trace CTR.
Only variation has button so the test results as below:
Original - Visitors: 120170, Conversion rate: 0
Variation - Visitors: 121058, Conversion rate: 15256
Next step is to ...
- 113
4
votes
1
answer
2k
views
Understanding uplift in A/B testing
Let's say we have a(n) (positive) outcome rate which we try to optimize.
Features affects this outcome.
In a test, we find that users that used a specific feature X have a higher positive outcome ...
- 153
2
votes
2
answers
2k
views
Should you ever test multiple A/B tests at the same time?
Okay, so you have a website with lots of web pages on it. It has a header/nav and a footer that show up on every page, obviously. And you have a bunch of other unique individual pages.
You want to ...
- 223
0
votes
1
answer
75
views
How to AB Test a particular component across the whole site?
I want to do a AB test for the navigation module across the whole site.
I know AB/Multivariate tests are usually page specific.
Is there any way to do AB Testing across a whole site? So, we can ...
- 103
3
votes
3
answers
1k
views
Measuring activity on an invision prototype?
I'm wondering if any of you all could help me with this predicament.
I am about to finish a certificate in human-computer interaction and the final gauntlet is A/B testing the prototypes we have ...
- 31
1
vote
2
answers
107
views
a/b testing hundreds of pages at once, with a different layout [closed]
I want to try a new layout for my products page. It would involve a new layout and new color scheme on every products page. There are hundreds of product pages.
My question is, how do you a/b test ...
- 2,343
4
votes
3
answers
823
views
Can you carry out AB testing without the site being live?
I am new to A/B testing, i have looked at some sites for more understanding on what it is:
https://www.optimizely.com/ab-testing/
https://vwo.com/ab-testing/
http://unbounce.com/landing-page-articles/...
- 105
2
votes
1
answer
187
views
How long should we test every option in an A/B test?
Currently we have an app with a number of features that fit well into a tab navigation bar below the action bar. Because of the vision of having more features in the future, business guys want to have ...
- 163
9
votes
4
answers
1k
views
How do Enterprise Softwares validate A/B testing?
I just started working for an enterprise software company, and I'm having a hard time integrating validation to see "what works" for the user.
I'm used to working for more consumer-based software, ...
- 315
2
votes
2
answers
92
views
What are the primary reasons that customers don't purchase AB Testing services?
I work in a UX consulting agency. I noticed while customers are happy to pay good money for our research and design services, AB testing and other post-launch services are seldom being asked for.
...
1
vote
1
answer
72
views
Conversion based on clicks or orders
I'm running A/B split testing using Instapage.
My dilemma is whether using clicks or orders versus visitors should be my conversion rate.
This is what I had so far
A: 556 visitors - 27 conversions (...
- 59
3
votes
3
answers
341
views
What are the effects of A/B Testing if the user is aware of it?
I went on the same website on two different browsers, and I got two different layout + design. I was being A/B tested. I didn't like that, probably because:
I don't like to be part of an experiment
I ...
- 266
1
vote
2
answers
56
views
Direction of fretboard
I know it might be a bit difficult to find musicians who have experience with UX, but I'll give my question a try here.
I am designing an app for musicians and need an opinion on something:
Which ...
- 177
4
votes
5
answers
857
views
Which spot has the best conversion in the grid layout?
We are going to show popup with 6 different offers, positioned by 3 in two rows. Of the these 6 offers, 2 are really beneficial for us (say, have high reward for click). Common sense says that we ...
- 143
3
votes
5
answers
339
views
A/B testing in user test
I designed 4 versions of a checkout page. I want to make some user test to know which version performs better. Do you think that I have to do the tests with the same person? the big change in the ...
- 963
12
votes
5
answers
4k
views
How does e-commerce product quick view affect conversion rates?
Many large e-commerce retailers are using "quick views" for products on their collection pages where more details appear either on-hover of via some other trigger.
Has anyone performed any A/B tests ...
- 171