Tree testing during a redesign - which answers are correct?

Question

I'm working with another UX person on reorganizing the information architecture for a large website. The IA work may eventually lead to a complete redesign of the website and will certainly impact its content.

I am the researcher running the studies on the IA. We have run a series of tree tests, including benchmarking, but the one we just ran has a mostly new set of tasks for a different audience. Some tasks have had multiple people nominate the same answers that we had originally thought were incorrect. Just like in the Atlassian wiki's guide, we've needed to add some new correct answers that we had missed initially.

We are still undecided about some other answers that we have frequently seen.

For example: There's a page called Contact Us. Let's say it has only a generic contact form for the entire organization in question. The task in the tree test involves finding someone in a specific department, which participants outside our organization would recognize (at least very generally). The correct answers initially dealt with that department, but about 20% of users nominated Contact Us. (This affects several other tasks too, but these are more complex, and changing the content on the pages would require more significant discussions with stakeholders.)

I've been saying that Contact Us is not the right answer for this task, based on its current purpose. Users won't find what the task is asking for on that page, and there is no cross link to pages where they can contact more specific people.

The other person (who designed the IA) is saying that Contact Us should be a right answer because the site is in a redesign and, if users are saying the content should be in that place, we should change the site to put it there. I'm concerned this may be a slippery slope, but I'm not sure.

In studying tree testing, I haven't seen anything on how to handle defining correct answers when the site's content itself is open to change. Are there any previous studies or best practices surrounding this?

Answer 1

I think the conflict comes from how the 'correct' answer is defined, and what the result of the analysis actually means.

It is common to define what the correct answer is in tree testing (e.g. using Treejack from Optimal Workshop) to help provide a reference or baseline for the output of the results.

If you (and the other UX person - I assume not a researcher?) were to define the purpose of the Contact Us page in one or more ways:

• for getting in touch with people in the organization
• for finding out contact details
• for users that can't find what they are looking

Then this has implications for how you interpret the result of the task that they are being asked to perform (e.g. if they fail to find something they might go to Contact Us as a last resort).

Adding to the complexity is framing the task in terms of what the Contact Us page on the site currently does versus what you want it to be able to do in the future.

So as you can see, the way the testing has been structured can create some ambiguity as to the best way to interpret the results, and this actually has to be set out before the testing is done (and then interpreted appropriately) rather than trying to move the goal posts during the analysis phase.

But the solution here is to just set a definition and reference point and interpret the data that has been collected. It just means framing your results and insights in a different way, but it shouldn't impact on the actual processing and analysis of the data if you have done it objectively.