How to measure self reported computer proficiency? Ideally with one item

Question

For a usability test, I'd like to have a simple measure for computer proficiency, to be able to see, whether it has an impact on successfully handling the tasks.

How can we best measure that?

A simple question with a likert scale would be nice, but is hard to create. As I think just asking "how proficient are you with computers?" probably won't wield valid results as there is nothing to compare your proficiency to.

Answer 1

I would use a set of questions on a scale: for example, "I only use a computer when I really have to" or " I use a computer for a number of tasks, but sometimes have to get help" right up to "I am a computer expert who works in computer problem solving"

The important thing is to define the limits of the scale, and make every entry positive, which is a difficult thing to do.

When I have needed to assess this sort of thing, I work on an amount of computer usage - "I do not own a computer, but occasionally have to use one" to "I use a computer all day at work, and a lot of my leisure time too". Defining the amount of time spent using computers roughly matches to how confident or knowledgeable the are about usage.

The other problem is what you mean by "proficiency" - I am a software professional, but do not know much about the hardware aspects. People can be very proficient at usage of applications, but know nothing about their workings or code aspects. So I think you need to define "proficiency" more clearly, which should help you to identify a scale you can use.

Answer 2

Self reporting doesn't work for something like this. Instead ask them a few proxy questions such as:

How often do you use a computer? How often do you use a computer in your free time?

I'm sure you can think of some others, but the general idea is to ask about something else that correlates to proficiency but is more objective.

Answer 3

Anything 'self reported' doesn't work, people's assessment of their own abilities is often highly inaccurate.

You'll get a general feel for how good (or not) they are with computers while doing the actual testing.

Answer 4

The problem is that self reported skill levels are almost invariably misleading.

For one reason why, think about the classic Four stages of competence.

• Someone who has unconscious incompetence isn't even aware that they lack skills. They'll typically overstate their skill level.

• Someone who has conscious incompetence is aware of how much they dont' know. They'll typically understate their skill level.

• Someone who has conscious competence knows they can do it, but it takes hard work and they often think they do it poorly. They'll typically understate their skill level.

• Someone who has unconscious competence does it as second nature and has often forgotten how hard other people find it. They'll typically rate themselves highly and be accurate.

Add into this mix the idea that most people are perpetual intermediates (neither newbies nor experts, see Alan Cooper's book The inmates are running the asylum), and you find that self rating falls apart as the ratings have no strong correlation to actual skill.

Answer 5

It is very difficult to get an objective scale for computer proficiency in one question. I have tried to do it for the purposes of screening out advanced computer users from usability tests, but not for your purposes of determining the relationship between proficiency and task completion. I don't think the response to a single answer will allow you to do this; I think the best approach for that is aomedia and ebeth's ideas for a set of task-related questions, from which you can try to define a proficiency score.

The way I would do this is come up with a list of tasks that would be the most relevant for your own test, and ask some variation of the following questions, give them 1-pt per question and add up the scores. Make sure the tasks either represent genuinely different levels of proficiency from each other, or represent different types of work, otherwise the correlation between tasks will catapault some users disproportionately higher relative to their actual proficiency.

When was the last time you did [task], if ever?

• Today or yesterday
• In the last week <--award 1 point for this answer or higher
• In the last month
• Never

How frequently do you [do task], if at all?

• Daily
• Several times per week <- 1 pt
• Weekly
• Monthly

For what it's worth, I have found variations on the following theme of "how do others see you" effective for screening out most highly proficient users. It's highly subjective and varies from person to person, but it was good enough for my purposes.

Do your friends and family ask for your advice or help in using computers?

Do your co-workers see you as an expert user of [software]?

Answer 6

I've used a list of potential tasks in which one would use a computer. For example:

I use a computer to:

• Check my email using Netscape Thunderbird
• Check my email using Microsoft Outlook
• Find out what my friends are doing on facebook
• Get information about using Twitter
• Research big purchases like a new car or TV using the web
• Create my own web page using Microsoft Publisher
• Edit photos using Adobe Photoshop

I use something like this as a pre-test survey, then when they are in the testing room I will ask more probing questions like "How did you start researching the new car purchase?" or "How often do you edit photos." etc.

I use the names of actual software applications and sites as I can, usually but not always, determine their level of expertise. Someone who uses MS Office may be a very advanced Word or Excel user, but someone who uses Open Office may be more advanced in web search, application installation, and general knowledge of technology.

Answer 7

Knowing someone's proficiency in related tasks might be more helpful as most people tend to over or underestimate their GENERAL skills. So if you asked them if they are at ease with:

• posting a picture to Facebook or any other site,
• editing this image on a website that offers that type of service,
• grabbing attachments from an email,
• editing images,
• ... all the way to complex tasks

, you might end up with a more accurate profile.