Please forgive me; this may be somewhat long-winded.
I work in an academic library. I'd like to study how well assorted features of our site are serving our users' needs; for example, labeling of UI elements, layout, the copy of particular pages, and so on. But it's difficult to articulate a specific goal that I could measure via an A/B test, because whether a specific action is a good fit or not depends completely on what the user is looking for.
- Alice needs articles on treating childhood trauma for a class paper.
- Bob is looking for SWOT analyses of major shoe manufacturers.
- Charlie needs a copy of the original Latin text of the Legenda Aurea.
- Diane really just wants to print out the paper she just finished, but doesn't know if she can do that at the library, or when the library is open.
The library can answer any of these needs. Alice needs to search PsycInfo. Bob should look at Mergent Online. The catalog can tell Charlie whether we've got a copy of that, or he can request one through interlibrary loan. Diane can find out the library's hours and that she can print stuff from one or more pages on our site. All of those goals require different actions to achieve, and often there's more than one way to go about it. In short, since I don't know what a given visitor is looking for, I can't tell whether what they tried worked well or not.
I'm aware of surveys and direct observation usability testing using the think-aloud protocol, plus card sorts and other things. I've used some of those (direct observation and card sorts). But I'd like to test some specific design choices using data from live visitors rather than people in an artificial testing environment, and I'm struggling to figure out how to measure the results in a meaningful way.
I think what I need is some way to do an A/B test which does not presume a specific action as the metric of success. Suggestions?