Is a usability study on just a newsletter effective?

Question

I'm doing a usability study in our lab next month to test a newsletter for a client. It will be my first test with only an email (instead of a website). I'll only have 3/4 tasks (open email, choose email, read email + reactions). Do you think I'll have good insights or should I include this test in a broader test that will include the whole website?

Answer 1

No your test is not broad enough. Signing up for and reading a newsletter has many steps, starting with locating the newsletter on the page. Users need to be able to complete all necessary steps in order for the function to be successful. Also you should be testing your user interface, not the user's email client's interface.

A successful newsletter function should have the following attributes:

• users are able to locate the newsletter on the page if they want to sign up
• users can register for the newsletter successfully
• users are able to read and access the newsletter when they receive it
• users can opt out of the newsletter if they so choose

This is a holistic approach that includes all aspects of accessing your organisation's newsletter. Failure at any of these steps (except maybe the last) will prevent your newsletter from being successful. Therefore you need to test for each of these.

Tasks to test for this could include:

• "Sign up for regular updates from this company." (user has to locate the sign up function and go through the registration steps)
• "Find out what information the regular updates contain." (user has to access the newsletter and understand the content)
• "Stop the company from sending you regular updates" (user has to find the unsubscribe function)

You should be asking users to think aloud as they complete the tasks, in order to understand their logic. You should also ask users how confident they are that they are successful at the end of this task. It is highly desirable to have users who are highly confident and who also completed the task successfully - any other combination is undesirable (eg. user failed at the task but is quite confident that they were successful suggests there is not enough feedback to the user regarding their progress or success/failure regarding the task)

You do not need to include tests for tasks such as 'delete email', 'choose email' etc because these do not test your user interface and are too specific. Broader tasks let you watch the logical steps that users take.

Answer 2

A newsletter implies that it's sent to users who have already signed up and used the site in question, so I'll only make sure the users you test it on qualify.
To get a balanced outcome you'll need to test different user types (use the site frequently, used it in the past and stopped, logged in just once, etc.), so you might have to "create" some of them, by letting them first use the site and then test the newsletter.

Other than that why not test it separately? It's like testing separately a landing page vs. an app page in your site - different mindset, goals, environment, etc.

Answer 3

I have learned that one can get insights from all kind of users. One's Mom, friends, colleagues etc. We want our applications/web pages/newsletters to be understood by everyone, unless it is by specialists, for that you will need some of those specialists to test it. In either case you will surely get some good insights.

Regarding testing the newsletter and the website together or not, I would suggest to test the newsletter itself if your focus is only on making sure that users understand the layout. It the newsletter should be the starting point of a process to get something on a web page, then I would include the web page as well in the test.

Answer 4

Though the purpose of the study might be just on how the users access the newsletter,I would also focus on aspects such as information retention and how specific things stood out to them and where their areas of focus were. If possible incorporate stuff like click points and heatmaps since that would also help you provide information about whether people are able understand the news letter and what are the main points of focus and where critical information should be placed to ensure users assimilate it at first glance

Answer 5

I guess it is. There is a complete, thick report on usability studies of (email) newsletters by Nielsen/Norman. It results in 199 recommendations. So, I would conclude there is plenty of room for study, as well as for making mistakes in this medium.

Answer 6

Most of the studies fail to first outline what the object of study should be optimized for. In other words, what is the goal of the newsletter, and what should one be able to do easily with it. Your client should articulate this clearly with your help. The ideal is to distinguish form from content. Form covers things like layout and mechanisms like unsubscribe, share, etc. Content is about what specifically fills the template. If the goal is for users to buy a product, or find more information about something, or retain an overview of what is happening, then you have your tasks. Then you need a couple other tasks for key operations with the template (e. g. unsubscribe).