Tag Info

Hot answers tagged

15

You are conflating 'subjective' and 'unreliable'. Usability tests aim to get reliable information about people's reactions. Self-reported opinions are also subjective, and are much less reliable indicators of how other people will react to the interface. If I test 100 people and their subjective opinion is that they hate an interface, I'm pretty sure that ...


13

Do you have to put 'OK' and 'Cancel' on the buttons? One of the problems with OK/Cancel in dialogs (and similarly, but worse, Yes/No) is that the user has to refer back to the original question to understand what the buttons will actually do. This is probably more of an issue than whether the OK or Cancel is on the left or the right. For example: Are ...


12

What you've got there is a null result - there's no real difference between the two. Let's backtrack from the percentages to actual numbers. Control (127): Submitted search 44, booked 6 Variation (123): Submitted search 34, booked 4 Just by eye balling the numbers it's not looking terribly compelling. If just one person less in the control and one more ...


12

No, neither is better. They deal with different aspects or strategies, and in general you need both. A small change lets you refine your design and have a better understanding of what affects conversion, but may let you end up with a local maxima. A more radical change with many elements will not help you understand what affects conversion, but may also ...


11

Here is what I would do to have A/B Testing done on Existing Customers / People who already have my app: Both the flows of the App you have in mind should be bundled as a part of the same app. You use the code injection to send information to Flurry. Use the same method to check how many users have downloaded the latest version of your App with both ...


9

Philip Re: Analysing the results. Take a look at the Easy statistics for AdWords A/B testing, and hamsters from Jason Cohen. It's one of the best articles i've ever read about understanding the statistical relevance of the results of an A/B test. Plus (and this is a big plus for me) it's fun to read. The main message: The way you determine whether an ...


9

Honestly speaking from personal experience,the only time I would bother to answer the question is I was applying for a job and I felt that selecting "employee referral" (provided I was referred by an employee :) ) might get me an interview or atleast get my resume in front of a recruiter. That said,every additional field that you require the user the fill ...


7

Google offers a free AB tester callled Website Optimizer


7

I used to work at Intuit and they tested everything. Here are some lessons I learned: You need traffic. You don't get statistical confidence unless you have enough traffic. It's a waste of effort to test something that will take 4 months to resolve. Multivariate testing takes even more traffic. What are you testing? At Intuit, they tested horrible ...


7

Assuming you're testing a change to an existing system, favor the control. It's what people are used to simply because it's already in place. If tests are inconclusive against users that aren't familiar with your site, at best the change won't help, at worst it briefly confuses current users. You failed to reject the null hypothesis, stick to what you have. ...


7

The only way that you are going to realistically test a headline is to A/B test it with your target audience. You could have focus groups (which is what book publishers usually do) but that isn't feasible for most web publishing due to the hight cost and turnaround time. Another solution would be to have an automated A/B test for different headlines. This ...


7

The point of doing A/B testing rather than cohort is that it eliminates the conflating variable of time. The data you gather is only valid if there aren't ulterior explanations for why two groups behave differently, and groups doing things at different times will often behave differently. For example: if you're an e-commerce site and you compare user ...


6

There's a really nice article by Cennydd Bowles that you'll probably find useful Statistical significance & other A/B test pitfalls


6

Well. Half the idea of an A/B-test is to be surprised... ;-) But I would definitely investigate this case more! I wouldn't say that the findings are compelling, either. The difference is too small to conclude that your visitors prefer A over B (and you should have more users). Do some user testing or after task interviews to get more "quality ...


6

"Massive" amounts of traffic is a subjective term. Split testing doesn't necessarily need the traffic volumes of Google or Amazon to be effective. The key question is statistical significance, which you can achieve with low amounts of traffic provided the difference in results is great enough. I built a quick and dirty split-testing calculator a while ago ...


5

This might be helpful. I'll remove it if it breaks the rules. I once converted a long and gruelling sign up form and chopped it up into wizard-like steps and compared them A/B style. In both cases there was a "Where did you hear about us?" field (I failed to get stakeholders to allow me to drop it). In both cases it was an optional step and I found that ...


5

Was this a One-tailed Test? First of all, I think your statistical test is giving you a 1-tailed p-value, rather than a 2-tailed p-value that you should use in what sounds like exploratory work. I think you’re saying your p-value is 0.02 (i.e., there is a 2% chance of getting the observed difference in conversions by random luck). However, if the number of ...


5

It sounds like you need multivariate testing. For example, select four elements on the page you'd like to test. You could radically redesign each of the elements so that in one version of the test, you're testing a mostly redesigned page. Then those four elements would be turned "on" or "off" alternately. You'd end up with 16 variations against your control ...


5

Some thoughts on the experiment are: Subjective. Your test seems to be more like Luscher color test, but bear in mind that color preferences are subjective and not permanent. Randomize. To get the proof of the golden triangle effect just mix randomly the webpages before each test session. Bias. You are biased experimentator, as you have some expectations ...


5

The answer to this can be a little complicated and i know lot of layout designers and grid specialists have spent a lifetime contemplating on this. Before entering into the discussion we need to ask whether you want to be open about the profits to your advertisers or whether its going to be an under the covers thing. If you are not going to be telling this ...


4

There is a google tool for web apps. www.google.com/websiteoptimizer


4

I would advise you to try this new tool: http://visualwebsiteoptimizer.com/ I think this should be the future of A/B testing! You can use it without any html/css knowledge. It is for business users, who know marketing and not programming. And it is reasonably priced!


4

You might find Smashing Magazine's The Ultimate Guide To A/B Testing a useful A/B testing resource: It includes a section on Tools For A/B Testing, as well as covering the following: What Is A/B Testing? What To Test? Create Your First A/B Test Do’s And Don’ts Classic A/B Testing Case Studies Resources For Deep-Diving Into A/B Testing


4

You're not measuring the quality, but which option is best. Keeping the short path will make the results more accurate if you don't have tons of users. If you set the long path - you're measuring the connection between the form and the landing page. Some users won't fill the form since they don't think that what you promised on page 1 is worth it. Some ...


4

If you're still looking for a way to do this, I've created an iOS Client Library and PHP Server for performing data-driven A/B Split Tests in your iOS Apps - it should give a kick-start to anybody's Mobile A/B testing effort. There are controls (extending UIButton) for A/B testing button text and images as well as the ability to generically split test String ...


4

Although all of the above could be part of a UX designers role if you are focused purely on analytics and conversions then potential roles titles would probably include the word analyst and be prepended by: Business/web/marketing/digital. http://www.clickz.com/clickz/column/1712911/becoming-experienced-invaluable-web-analyst The other angle to approach it ...


3

Without more information about the type of business you're in, it's hard to be specific, so I'll be broad. If your website is selling something in any way, you should be measuring sales by that recipe in aggregate, not individual actions taken from the homepage. If you sell items directly, then you would measure the average dollar amount sold per unique ...


3

I've been involved with Website Optimizer and Omniture, Marketo as well as some home-grown A/B testing tools. My key finding is this: Garbage In - Garbage Out. The key question is "What are you testing? WHo is deciding that?" Is it just "Let's try and beat the control" or is it "Lets learn about our audience." The latter is much more important. You ...


3

I'm a big fan of Optimizely. Visual editing. Easy goal setting. Real-time reporting. Retroactive goal data. Simple to use interface.


3

This is a really important question. Not all research is created equally. Honestly some published studies are fundamentally silly, but you have to evaluate each on its own merits, which is why the "methodology" section of any scientific paper is so important. It matters how results are obtained and analyzed and we would all do well to be more constructively ...



Only top voted, non community-wiki answers of a minimum length are eligible