What sort of tasks should be included in a lab test to determine baseline for a user?

Question

I'm starting some lab testing and I'm an on-the-job trained UXer and have watched a decent number of tests, but haven't constructed a lot on my own.

When doing a more formal test and not a guerrilla test, is it important to begin with simple baseline tasks to determine a participant's savvy?

If so, what type of tasks perform the best for determining that?

Right now, I ask participants to find the website based on how they would at home, hoping that will show me at least "Do they know about Google? Do they know how to use a computer well enough to navigate to a search engine and find a result inside the search engine?" as well as good data to show a client any frustrations in how a normal user may find their site.

Answer 1

This is an interesting question because I have never seen these types of results presented in usability studies. I think a lot of this depends on the particular context of the study. Sometimes the tests are already designed to eliminate these issues (e.g. by having the facilitator explain what needs to be done), and sometimes the sheer volume of participants will rule out the effect this will have on the results. But in general you would screen participates to make sure that you can get meaningful results from the usability testing. I do think, from a more scientific point of view, there should be some benchmark test that gives you an idea of how to compare between usability testing participants that gives you an indication of technology familiarity and proficiency.

I think the task(s) depend on the type of information that you want to capture from the users. If it is a qualitative measure that gives you some idea about how tech savvy the users are, and is designed so that you can compare results between different lab tests that you run (e.g. because you have small sample sizes and you want to be able to adjust the weightings from different tests) then you need to look at the different aspects that determine what makes different users more or less tech savvy. You'll also have to think about the variables that can influence the results of the tasks.

I don't think there's an easy answer to this question, but if you can provide some more background information it might help.