Take the 2-minute tour ×
User Experience Stack Exchange is a question and answer site for user experience researchers and experts. It's 100% free, no registration required.

For a usability test, I'd like to have a simple measure for computer proficiency, to be able to see, whether it has an impact on successfully handling the tasks.

How can we best measure that?

A simple question with a likert scale would be nice, but is hard to create. As I think just asking "how proficient are you with computers?" probably won't wield valid results as there is nothing to compare your proficiency to.

share|improve this question
3  
True story: I once asked a group of about 25 participants to rate themselves on a 1-7 scale on computer proficiency. EVERY SINGLE PERSON rated themselves a '6', except for one '7'--a person who didn't even know how to use a mouse. –  Alex Feinman Sep 9 '11 at 17:10
1  
    
@BenBrocka I created an account on Cog Sci just to upvote that question. –  msanford Sep 18 '12 at 20:37
1  
@msanford heh, be sure to browse the HCI tag and maybe ask some questions of your own :) –  Ben Brocka Sep 18 '12 at 20:44
add comment

7 Answers 7

It is very difficult to get an objective scale for computer proficiency in one question. I have tried to do it for the purposes of screening out advanced computer users from usability tests, but not for your purposes of determining the relationship between proficiency and task completion. I don't think the response to a single answer will allow you to do this; I think the best approach for that is aomedia and ebeth's ideas for a set of task-related questions, from which you can try to define a proficiency score.

The way I would do this is come up with a list of tasks that would be the most relevant for your own test, and ask some variation of the following questions, give them 1-pt per question and add up the scores. Make sure the tasks either represent genuinely different levels of proficiency from each other, or represent different types of work, otherwise the correlation between tasks will catapault some users disproportionately higher relative to their actual proficiency.

When was the last time you did [task], if ever?

  • Today or yesterday
  • In the last week <--award 1 point for this answer or higher
  • In the last month
  • Never

How frequently do you [do task], if at all?

  • Daily
  • Several times per week <- 1 pt
  • Weekly
  • Monthly

For what it's worth, I have found variations on the following theme of "how do others see you" effective for screening out most highly proficient users. It's highly subjective and varies from person to person, but it was good enough for my purposes.

Do your friends and family ask for your advice or help in using computers?

Do your co-workers see you as an expert user of [software]?

share|improve this answer
add comment

Knowing someone's proficiency in related tasks might be more helpful as most people tend to over or underestimate their GENERAL skills. So if you asked them if they are at ease with:

  • posting a picture to Facebook or any other site,
  • editing this image on a website that offers that type of service,
  • grabbing attachments from an email,
  • editing images,
  • ... all the way to complex tasks

, you might end up with a more accurate profile.

share|improve this answer
add comment

The problem is that self reported skill levels are almost invariably misleading.

For one reason why, think about the classic Four stages of competence.

  • Someone who has unconscious incompetence isn't even aware that they lack skills. They'll typically overstate their skill level.

  • Someone who has conscious incompetence is aware of how much they dont' know. They'll typically understate their skill level.

  • Someone who has conscious competence knows they can do it, but it takes hard work and they often think they do it poorly. They'll typically understate their skill level.

  • Someone who has unconscious competence does it as second nature and has often forgotten how hard other people find it. They'll typically rate themselves highly and be accurate.

Add into this mix the idea that most people are perpetual intermediates (neither newbies nor experts, see Alan Cooper's book The inmates are running the asylum), and you find that self rating falls apart as the ratings have no strong correlation to actual skill.

share|improve this answer
    
I must direct you to the most excellent work of Dunning and Kruger en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect –  colmcq Sep 9 '11 at 10:44
add comment

I've used a list of potential tasks in which one would use a computer. For example:

I use a computer to:

  • Check my email using Netscape Thunderbird
  • Check my email using Microsoft Outlook
  • Find out what my friends are doing on facebook
  • Get information about using Twitter
  • Research big purchases like a new car or TV using the web
  • Create my own web page using Microsoft Publisher
  • Edit photos using Adobe Photoshop

I use something like this as a pre-test survey, then when they are in the testing room I will ask more probing questions like "How did you start researching the new car purchase?" or "How often do you edit photos." etc.

I use the names of actual software applications and sites as I can, usually but not always, determine their level of expertise. Someone who uses MS Office may be a very advanced Word or Excel user, but someone who uses Open Office may be more advanced in web search, application installation, and general knowledge of technology.

share|improve this answer
add comment

I would use a set of questions on a scale: for example, "I only use a computer when I really have to" or " I use a computer for a number of tasks, but sometimes have to get help" right up to "I am a computer expert who works in computer problem solving"

The important thing is to define the limits of the scale, and make every entry positive, which is a difficult thing to do.

When I have needed to assess this sort of thing, I work on an amount of computer usage - "I do not own a computer, but occasionally have to use one" to "I use a computer all day at work, and a lot of my leisure time too". Defining the amount of time spent using computers roughly matches to how confident or knowledgeable the are about usage.

The other problem is what you mean by "proficiency" - I am a software professional, but do not know much about the hardware aspects. People can be very proficient at usage of applications, but know nothing about their workings or code aspects. So I think you need to define "proficiency" more clearly, which should help you to identify a scale you can use.

share|improve this answer
3  
Or (extreme example): I have experience reverse-engineering precompiled binaries for the MIPS R3000 architecture. But I couldn't tell you a single thing about setting up Active Directory on a local server. –  Jimmy Breck-McKye Sep 8 '11 at 11:05
    
@Jimmy Breck-McKye :) –  igor Sep 8 '11 at 12:18
2  
Also, focus the questions in the light of the actual usage. For example if the app involves sound editing, you could add an additional question like "How often do you use your computer for sound editing?" –  Vinko Vrsalovic Sep 8 '11 at 12:19
    
@Jimmy Breck-McKye Couldn't tell you a single thing about either of those. –  eBeth Sep 8 '11 at 13:47
add comment

Self reporting doesn't work for something like this. Instead ask them a few proxy questions such as:

How often do you use a computer? How often do you use a computer in your free time?

I'm sure you can think of some others, but the general idea is to ask about something else that correlates to proficiency but is more objective.

share|improve this answer
2  
Trouble is they can be spending an awful lot of time on say Facebook - without knowing much about computing generally. –  PhillipW Sep 8 '11 at 12:46
    
this is true, but there will be a general correlation between computer proficiency and number of hours spent on computer. If there's a better proxy I'd like to hear it... –  colmcq Sep 9 '11 at 10:43
    
I like the idea of proxy questions, but I have to agree with Phillip that I wouldn't use time spent or frequency of use. Now that so many office workers with a computer at home are simply surrounded by computers all day long, I wouldn't see much differentiation. I think what would be more relevant would be to ask about tasks like ebeth and aomedia, and the tasks should be connected to what Roland thinks would have the most impact on his task-completion test. –  Jonathan Sep 9 '11 at 18:57
    
There are things that you could ask to get more information, sure, but then you need to ask more questions. That will likely then get you a lower response rate. While time spent isn't ideal by any means, it has been shown to represent a general correlation. –  JohnGB Sep 10 '11 at 16:28
add comment

Anything 'self reported' doesn't work, people's assessment of their own abilities is often highly inaccurate.

You'll get a general feel for how good (or not) they are with computers while doing the actual testing.

share|improve this answer
    
see self selection bias en.wikipedia.org/wiki/Self-selection_bias –  colmcq Sep 9 '11 at 10:43
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.