Is there any testing methodology similar to A/B testing, but without a specific goal?

Question

Please forgive me; this may be somewhat long-winded.

I work in an academic library. I'd like to study how well assorted features of our site are serving our users' needs; for example, labeling of UI elements, layout, the copy of particular pages, and so on. But it's difficult to articulate a specific goal that I could measure via an A/B test, because whether a specific action is a good fit or not depends completely on what the user is looking for.

For example:

• Alice needs articles on treating childhood trauma for a class paper.
• Bob is looking for SWOT analyses of major shoe manufacturers.
• Charlie needs a copy of the original Latin text of the Legenda Aurea.
• Diane really just wants to print out the paper she just finished, but doesn't know if she can do that at the library, or when the library is open.

The library can answer any of these needs. Alice needs to search PsycInfo. Bob should look at Mergent Online. The catalog can tell Charlie whether we've got a copy of that, or he can request one through interlibrary loan. Diane can find out the library's hours and that she can print stuff from one or more pages on our site. All of those goals require different actions to achieve, and often there's more than one way to go about it. In short, since I don't know what a given visitor is looking for, I can't tell whether what they tried worked well or not.

I'm aware of surveys and direct observation usability testing using the think-aloud protocol, plus card sorts and other things. I've used some of those (direct observation and card sorts). But I'd like to test some specific design choices using data from live visitors rather than people in an artificial testing environment, and I'm struggling to figure out how to measure the results in a meaningful way.

I think what I need is some way to do an A/B test which does not presume a specific action as the metric of success. Suggestions?

Answer 1

(Edited: somehow I totally managed to miss the word 'site' when reading your original post! Sorry.)

I'm not sure whether you want a variant on A/B testing, since you don't really have one either/or you'd like to test. The only binary outcome you do have is "did the user find what they needed"? This might come in useful later when you look at which factors might have predicted that outcome.

The first thing that comes to mind is, if you have the resources, to basically shadow a smallish sample of users as they navigate the site, and interact with them in a sort of semi-structured interview, basically a usability testing session. The sorts of questions I have in mind are something like:

• What did you want to achieve today?
• Thinking about the resources available to you, what do you think the first step is?
• (after first step) Did that help you achieve your goal?
• Why do you think that was?
• (and then) Thinking about the resources available to you on the site, what is the next step towards achieving your goal?
• (repeat until goal is achieved or participant gives up)
• (at end of trail) Can you rate (e.g. out of 10) how well you achieved [your goal] today?

Of course, the fact that you are asking them to think about and describe their progress will probably alter their natural behaviour a little bit, but probably no worse than in any other kind of usability testing.

It might then be useful, when you've done this with a few people, to look at responses from participants who fail to achieve their goals and see whether they have anything in common. And then, if you have time, to make the changes you think will improve people's experience, then test again.

I'm probably missing out some critical questions here, but you will have a better idea of the kinds of library resources/navigational aids/etc that you want to test. (So for example if you wanted to test the site's navigability, you could frame questions with "Looking at the left/top of the screen, …" — that's quite a simplified idea, but I hope you know what I mean.)

Hope that's useful; really interested to find out how this works out for you. Good luck :)