For a usability test, I'd like to have a simple measure for computer proficiency, to be able to see, whether it has an impact on successfully handling the tasks.
How can we best measure that?
A simple question with a likert scale would be nice, but is hard to create. As I think just asking "how proficient are you with computers?" probably won't wield valid results as there is nothing to compare your proficiency to.
I would use a set of questions on a scale: for example, "I only use a computer when I really have to" or " I use a computer for a number of tasks, but sometimes have to get help" right up to "I am a computer expert who works in computer problem solving"
The important thing is to define the limits of the scale, and make every entry positive, which is a difficult thing to do.
When I have needed to assess this sort of thing, I work on an amount of computer usage - "I do not own a computer, but occasionally have to use one" to "I use a computer all day at work, and a lot of my leisure time too". Defining the amount of time spent using computers roughly matches to how confident or knowledgeable the are about usage.
The other problem is what you mean by "proficiency" - I am a software professional, but do not know much about the hardware aspects. People can be very proficient at usage of applications, but know nothing about their workings or code aspects. So I think you need to define "proficiency" more clearly, which should help you to identify a scale you can use.
Self reporting doesn't work for something like this. Instead ask them a few proxy questions such as:
How often do you use a computer? How often do you use a computer in your free time?
I'm sure you can think of some others, but the general idea is to ask about something else that correlates to proficiency but is more objective.
The problem is that self reported skill levels are almost invariably misleading.
For one reason why, think about the classic Four stages of competence.
Someone who has unconscious incompetence isn't even aware that they lack skills. They'll typically overstate their skill level.
Someone who has conscious incompetence is aware of how much they dont' know. They'll typically understate their skill level.
Someone who has conscious competence knows they can do it, but it takes hard work and they often think they do it poorly. They'll typically understate their skill level.
Someone who has unconscious competence does it as second nature and has often forgotten how hard other people find it. They'll typically rate themselves highly and be accurate.
Add into this mix the idea that most people are perpetual intermediates (neither newbies nor experts, see Alan Cooper's book The inmates are running the asylum), and you find that self rating falls apart as the ratings have no strong correlation to actual skill.
Anything 'self reported' doesn't work, people's assessment of their own abilities is often highly inaccurate.
You'll get a general feel for how good (or not) they are with computers while doing the actual testing.
It is very difficult to get an objective scale for computer proficiency in one question. I have tried to do it for the purposes of screening out advanced computer users from usability tests, but not for your purposes of determining the relationship between proficiency and task completion. I don't think the response to a single answer will allow you to do this; I think the best approach for that is aomedia and ebeth's ideas for a set of task-related questions, from which you can try to define a proficiency score.
The way I would do this is come up with a list of tasks that would be the most relevant for your own test, and ask some variation of the following questions, give them 1-pt per question and add up the scores. Make sure the tasks either represent genuinely different levels of proficiency from each other, or represent different types of work, otherwise the correlation between tasks will catapault some users disproportionately higher relative to their actual proficiency.
When was the last time you did [task], if ever?
- Today or yesterday
- In the last week <--award 1 point for this answer or higher
- In the last month
- Never
How frequently do you [do task], if at all?
- Daily
- Several times per week <- 1 pt
- Weekly
- Monthly
For what it's worth, I have found variations on the following theme of "how do others see you" effective for screening out most highly proficient users. It's highly subjective and varies from person to person, but it was good enough for my purposes.
Do your friends and family ask for your advice or help in using computers?
Do your co-workers see you as an expert user of [software]?
I've used a list of potential tasks in which one would use a computer. For example:
I use a computer to:
I use something like this as a pre-test survey, then when they are in the testing room I will ask more probing questions like "How did you start researching the new car purchase?" or "How often do you edit photos." etc.
I use the names of actual software applications and sites as I can, usually but not always, determine their level of expertise. Someone who uses MS Office may be a very advanced Word or Excel user, but someone who uses Open Office may be more advanced in web search, application installation, and general knowledge of technology.
Knowing someone's proficiency in related tasks might be more helpful as most people tend to over or underestimate their GENERAL skills. So if you asked them if they are at ease with:
, you might end up with a more accurate profile.
I think it is important to understand the information or outcome that you want to achieve by asking the question, as it will determine the relevance of the response. It is difficult to do in a simple one question response as there might be a lot of contextual factors that you want to tease out.
For example, if you just want a general measure to sort the user groups into different levels of perceived computer literacy then there are probably some standard questions based on commonly used applications and tasks that you can choose from. However, if you already know something about your user group (e.g. they are an older population) then maybe you need something more specific to what they are familiar with so you can separate them into groups (otherwise you might just get everyone falling into one category).
Also, if you are testing or working on a specific type of application, you might want to ask a question that is targeted at the tasks that are typical of those applications (maybe based on usage statistics to determine how complex or common that feature/task is).
Finally, I assume that the reason you are interested in an one item question is because you might have seen things like the NPS score that can encapsulate a lot of variables into one simple figure to report on, but this is also assuming that you either don't need to be able to work out later on what those factors are by not collecting information on it.
I don't know of any 'magical' metrics in UX because the research that we do are so highly context dependent, but if you come across any then I'd be keen to test them out as well!
