How do I find out which one of two different ways of loading my website gives best perceived performance for the users?

Question

We have a rather heavy content website with about 400 000 unique daily users. The company has always focused on loading time, since we know this is important to our users, but I really want a shift in focus over to perceived performance rather than the technical time.

Our scenario is this:

• The developers have come up with a new way of loading, which in theory feels quicker to the user
• Our CTO thinks this is wrong and now believes the site to be slower than before

Is it a good way to split our users in half and give 50 % the old way of loading and the other 50 % the new way, and combine it all with a micro survey asking if they felt the site loaded e.g. a) instatly, b) very fast c) after a while or d) slowely

… or is there a better way?

Answer 1

A/B testing is a viable method to compare two solutions to a problem.

I would, however, like to propose a third. Perhaps it may be better, if possible, to tackle the underlying problem as to why the website takes so long to load. Focusing on your core proposal, offerings and gauging why people come to your website, then steering your content and design to service your business goals and visitors seems like a sensible decision to me, in doing so focus on making your website efficient and streamlined.

Answer 2

A/B Testing would be your best bet. We use Monetate (http://www.monetate.com) for A/B testing at our company, but there are others (Optimizely is another great tool: http://www.optimizely.com). The only drawback is that in order to measure true performance, you would have to create 2 separate pages (and then redirect 50% of users to the test page).

Another way you could test the 2 versions against each other would depend on if you can get sign-off from your CTO. You could run the current version for say, 1 month and then run the version you create for the following month & compare the 2 using both your site's analytics and moderated usability tests. But again, the only way this would work is if you can convince your CTO to allow you to test your hypothesis on your live site for a period of time... so maybe test a page with slightly less traffic if you're having trouble getting sign off on, say, the home page. If your hypothesis is correct (that perceived performance is important from a user perspective), you've also convinced your CTO; if it is not correct and users prefer the site as-is, at least then you can be sure.

There are a number of things you can do to improve both the perceived load time and the actual load time – why not do both? You already know that a faster site will lead to higher customer retention (and hopefully higher order size, conversion – whatever you're measuring); there are plenty of steps you can take:

• Reduce both the number of requests per page in addition to the size of each request
• Save images appropriately (JPEG for photos, PNG for transparency, GIF almost never)
• Compress your images (when exporting from Photoshop or the like, then compress again using https://imageoptim.com/ or a similar tool)
• Convert images to CSS3 or SVG where applicable
• Compress web fonts
• Load JavaScript asynchronously where it makes sense (don't block other requests)
• Minify your CSS and JS files
• Consider time to interactivity as opposed to full page load & optimize for that (make sure your user can perform the desired action as quickly as possible, even if the page hasn't fully finished loading)
• Consider optimizing your critical rendering path (so that you aren't relying on your very large stylesheet & JS files to fully load before the page does)

Perceived performance is beneficial when you can't improve actual performance further – you can't decrease total page weight, your users are still experiencing FOUC, or you are still observing very high bounce rates on a particular page – consider your options at this point.

If you're looking for a great resource on this topic, this book is excellent: (http://www.amazon.com/Designing-Performance-Weighing-Aesthetics-Speed/dp/1491902515)

Answer 3

In your situation, you can only test this A/B speed variation to only those site users who have used both version

But why ask users when you can measure and prove loading speed with a lot of online tools

Second thought is when site loading speed is concerned, loading speed is a quantifiable performance measure so you or your developer should use tools like

Pingdom Speed testing tool

to show the loading speed difference before and after your upgrade, if site loads in less milliseconds then that is the version to go with and nobody can disagree

For checking user perception of loading speed, do this simple on page top notification style one question survey via A/B Test

See the Image below for example wireframe

Answer 4

You are lucky to have 400,000 daily uniques to work with.

I think A/B testing will be effective for this decision.

You may want to keep in mind that interviewing or surveying users can create perception and framing biases which distorts results.

With such a large sample size at your disposal, an alternative approach is to do the A/B test and simply measure user behavior directly in terms of abandonment, stickiness, etc. instead of surveying users. Over such a large sample set, you can get very good confidence bounds.

Ultimately your goal here is not the perception of loading time, but something deep and more business related like ad clickthrough, content engagement, session stickiness or whatever.

A more modern approach would be simply to measure the direct impact of the two approaches on results, make sure that the statistics are empirically correct, then work backwards to test hypotheses around why it's working. This avoids the issues of survey bias and partial goal setting, and ensures you are measuring the correct, unbiased user behavior.

Answer 5

This is A/B testing.

You are right by separating user (randomly) and redirecting on one of the two way of loading.

You may add a link like How fast is this website? in a corner to display your survey but I would recommend to not believe what your user think but what they do ! (A link to not disturb users if they do not have time to spare)

---

Estimate what your business is and compare how users are acting in both ways.

If A give more money than B, choose A. Else choose B.

See Conversion marketing.

---

What about benchmarking to prove your CTO/developpers point?

---

EDIT : changing label according to @DarrylGodden comment