1

Question: Is there a method of defining potential profit value or loss on visual design tests?

Background: I was recently asked how to quantify a profit value or loss based on a set of tests for improving readability on a website. I had set up a few sets of 5 second and first-click tests. After each test I asked three open ended questions regarding why they made the choices they did and a finished with a 1 to 10 rating. I have been looking for a potential value equation or someone who had come up with a method to define value for visual tests, but have come up short.

3
  • 2
    How is your site making revenue? Are you selling ads, and do ad rates go up with longer engagement times?
    – Izquierdo
    Dec 27, 2021 at 18:30
  • The site I'm currently working on is an ecommerce website. We sell products related to healthcare and athletics as a distributor.
    – Huginn
    Dec 28, 2021 at 16:14
  • @Huginn to help answer the question better, perhaps you can provide some details about the type of test and also the metrics that you are currently measuring?
    – Michael Lai
    Dec 28, 2021 at 22:39

1 Answer 1

2
+100

Even though you have posted a generic question, because your background example is very specific I want to address it since it is more suited to UXSE. But the approach can be applied more generally to whatever it is that you want to test with your users.

Just to be sure that we are covering the right topics, it is probably good to understand some of the differences between legibility and readability, and also how they are related. This is because there are common factors that contribute to legibility and readability such as characters per line and content density that are very hard to separate out and test individually depending on the design of your content.

In general, legibility relates to how easy it is to perceive the content or information, so for example, am I able to read something if I am standing at a certain distance from the display device or holding it at a certain distance from my face. Another example might be the font style chosen and how easy it is to read the words due to the clarity and uniqueness of the font characteristics (e.g. kerning, leading, tracking). However, there may be specific reasons why one font is more legible compared to another due to user differences (e.g. dyslexia or other conditions).

In comparison, readability relates to the comprehensibility of the content, or how easy it is for the user to understand and retain the information. Both aspects are important because sometimes we want to make the user use the information at a point in time (so they don't need to remember it) while other times it is actually more important that they store this information for later use (e.g. public health safety practices during COVID). So being able to write in clear and simple language, staying away from jargons, and writing in a tone and style appropriate for the audience will contribute more to readability.

But this is where things get tricky because if something is not legible then it doesn't really matter if it is readable or not because the user is unable to perceive the information. Also, just how legible the content is will impact on the ability of the user to understand and retain the information because of the time and effort involved in processing the content. Therefore, I think it is important to first make sure that the content is legible and readable (or that you have considered both aspects), and then focus on your readability measures.

To be able to establish a profit or loss value based on readability alone, I would imagine that you need to first quantify what the metrics are for the existing/baseline design. This will then allow you to compare the variables that you manipulate and what the effects are (i.e. if they increase or decrease your baseline value).

Without knowing what the set of tests used to 'improve' readability are, my best guess is that they are variables that are either non-overlapping in the way that classic A/B tests are set up (e.g. compare difference in word usage or sentence length) or they are a set of co-related variables (e.g. writing style that incorporates particular sets of words and sentence length).

I would be very careful about drawing conclusions based on comparing the baseline metrics (e.g. conversion rate or click-through rate) for what appears to be non-overlapping variables (e.g. font style or font-size). This is because visual and interface design rely on a combination of factors that ultimately relate to the Gestalt Principles so they don't exist on their own. You often see articles written about A/B tests that assert the impact and value of design changes, like how doing an experiment shows the value of design changes, but because these kinds of tests are set up to show a particular outcome that the designer wants to achieve, it can often overlook a lot of other underlying facts. However, by carefully defining the control conditions and changing different variables you can observe the impact of the change and understand its effect more easily.

Given the relationship between legibility and readability, my preferred approach would be to analyze them together first. So the things you would be looking at should include things such as the combination of line width, line-height and font-size (see the Triangle Quiz), and how this determines what the optimum number of characters to display in different layouts. Another factor is the actual language that you are displaying (so not just thinking about English letters) because the average number of words (in a line or in a sentence) can also impact on the readability of a row of text. Even the type of readability scores and tests that are commonly accepted based on the type of content and audience you are writing for need to be applied carefully because it can have a profound effect on your test results.

There is a reason why legibility and readability are both very in-depth topics and involve specialist knowledge as they are related but separate disciplines in design (i.e. typography and copywriting), but there's no need to go into it in this question. Suffice to say there is a lot of detail in understanding the actual variables that contribute to the readability of your content so you need to be careful about teasing out the effects of each of them in the context of your audience and attribute them correctly to the metrics or value that you want to measure.

3
  • Great Explanation. What does UXSE stand for?
    – Shahriar
    Dec 28, 2021 at 10:43
  • 1
    UXSE I believe they're referring to this website.
    – Huginn
    Dec 28, 2021 at 16:17
  • 1
    @Shahriar User Experience StackExchange (UXSE)
    – Michael Lai
    Dec 28, 2021 at 22:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.