0

I am running couple of a/b tests on Australia and US, tests are exactly same but results they are showing are totally reverse. Since we tend to move successful A/B tests to be hardcoded on the sites this causes some issues.

Test is performing really well on US but has massive negative impact on Australia. I know that different areas have different online behaviour and preferences but I wouldn't think that discrepancy would be so big. I am kinda tempted to think that there might be an error in our Aus analytics.

Tests are exact duplicates.

Has anyone encountered similar results? If so what actions you took?

Improve this question
3
  • 1
    Mind explaining briefly what the tests are for? I'm wondering if it is something that is very latency dependent, and with AU being further from your servers (guessing) it is giving a much different experience. (i.e. a large hero image may help conversion in US, but if it takes 10secs to load in AU and causes the page to be blank or jump around you'd likely see the opposite)
    – DasBeasto
    Jun 29, 2017 at 17:08
  • The two countries might have the same language - but they do have very different cultures !
    – PhillipW
    Jun 29, 2017 at 18:40
  • @DasBeasto test was to add accepted payment methods as an image on the cart page and checkout, no functional changes and image is about 140px by 30px at the bottom of the page so I don't think it latency should matter that much. PhillipW i know that, but I am wondering if someone else had such contradicting results.
    – Awfor
    Jun 30, 2017 at 8:34

1 Answer 1

Reset to default
1

Tests are exact duplicates.

Same local time of day?

Same average page load?

Same sample size?

Same target audience demographic?

Same experimental setup?

Same language and lexicon?

And is that sample big enough to really call Aus and US variants different?You said the results are different - how different? 75/25 split? Do your confidence intervals overlap? If they do - you can't be sure your test result is more than a fluke.

Try to prove your assumption wrong. Check the sample size. Check analytics setup. Check the test setup again and make sure there are no other differences. Until you rule out all independent variables besides target country, you can't conclude the result is what you think you're seeing.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.