10

For a usability test, I'd like to have a simple measure for computer proficiency, to be able to see, whether it has an impact on successfully handling the tasks.

How can we best measure that?

A simple question with a likert scale would be nice, but is hard to create. As I think just asking "how proficient are you with computers?" probably won't wield valid results as there is nothing to compare your proficiency to.

4

I would use a set of questions on a scale: for example, "I only use a computer when I really have to" or " I use a computer for a number of tasks, but sometimes have to get help" right up to "I am a computer expert who works in computer problem solving"

The important thing is to define the limits of the scale, and make every entry positive, which is a difficult thing to do.

When I have needed to assess this sort of thing, I work on an amount of computer usage - "I do not own a computer, but occasionally have to use one" to "I use a computer all day at work, and a lot of my leisure time too". Defining the amount of time spent using computers roughly matches to how confident or knowledgeable the are about usage.

The other problem is what you mean by "proficiency" - I am a software professional, but do not know much about the hardware aspects. People can be very proficient at usage of applications, but know nothing about their workings or code aspects. So I think you need to define "proficiency" more clearly, which should help you to identify a scale you can use.

  • 4
    Or (extreme example): I have experience reverse-engineering precompiled binaries for the MIPS R3000 architecture. But I couldn't tell you a single thing about setting up Active Directory on a local server. – Jimmy Breck-McKye Sep 8 '11 at 11:05
  • @Jimmy Breck-McKye :) – igor Sep 8 '11 at 12:18
  • 2
    Also, focus the questions in the light of the actual usage. For example if the app involves sound editing, you could add an additional question like "How often do you use your computer for sound editing?" – Vinko Vrsalovic Sep 8 '11 at 12:19
  • @Jimmy Breck-McKye Couldn't tell you a single thing about either of those. – eBeth Sep 8 '11 at 13:47
2

Self reporting doesn't work for something like this. Instead ask them a few proxy questions such as:

How often do you use a computer? How often do you use a computer in your free time?

I'm sure you can think of some others, but the general idea is to ask about something else that correlates to proficiency but is more objective.

  • 2
    Trouble is they can be spending an awful lot of time on say Facebook - without knowing much about computing generally. – PhillipW Sep 8 '11 at 12:46
  • this is true, but there will be a general correlation between computer proficiency and number of hours spent on computer. If there's a better proxy I'd like to hear it... – colmcq Sep 9 '11 at 10:43
  • I like the idea of proxy questions, but I have to agree with Phillip that I wouldn't use time spent or frequency of use. Now that so many office workers with a computer at home are simply surrounded by computers all day long, I wouldn't see much differentiation. I think what would be more relevant would be to ask about tasks like ebeth and aomedia, and the tasks should be connected to what Roland thinks would have the most impact on his task-completion test. – Jonathan Sep 9 '11 at 18:57
  • There are things that you could ask to get more information, sure, but then you need to ask more questions. That will likely then get you a lower response rate. While time spent isn't ideal by any means, it has been shown to represent a general correlation. – JohnGB Sep 10 '11 at 16:28
2

The problem is that self reported skill levels are almost invariably misleading.

For one reason why, think about the classic Four stages of competence.

  • Someone who has unconscious incompetence isn't even aware that they lack skills. They'll typically overstate their skill level.

  • Someone who has conscious incompetence is aware of how much they dont' know. They'll typically understate their skill level.

  • Someone who has conscious competence knows they can do it, but it takes hard work and they often think they do it poorly. They'll typically understate their skill level.

  • Someone who has unconscious competence does it as second nature and has often forgotten how hard other people find it. They'll typically rate themselves highly and be accurate.

Add into this mix the idea that most people are perpetual intermediates (neither newbies nor experts, see Alan Cooper's book The inmates are running the asylum), and you find that self rating falls apart as the ratings have no strong correlation to actual skill.

1

Anything 'self reported' doesn't work, people's assessment of their own abilities is often highly inaccurate.

You'll get a general feel for how good (or not) they are with computers while doing the actual testing.

1

It is very difficult to get an objective scale for computer proficiency in one question. I have tried to do it for the purposes of screening out advanced computer users from usability tests, but not for your purposes of determining the relationship between proficiency and task completion. I don't think the response to a single answer will allow you to do this; I think the best approach for that is aomedia and ebeth's ideas for a set of task-related questions, from which you can try to define a proficiency score.

The way I would do this is come up with a list of tasks that would be the most relevant for your own test, and ask some variation of the following questions, give them 1-pt per question and add up the scores. Make sure the tasks either represent genuinely different levels of proficiency from each other, or represent different types of work, otherwise the correlation between tasks will catapault some users disproportionately higher relative to their actual proficiency.

When was the last time you did [task], if ever?

  • Today or yesterday
  • In the last week <--award 1 point for this answer or higher
  • In the last month
  • Never

How frequently do you [do task], if at all?

  • Daily
  • Several times per week <- 1 pt
  • Weekly
  • Monthly

For what it's worth, I have found variations on the following theme of "how do others see you" effective for screening out most highly proficient users. It's highly subjective and varies from person to person, but it was good enough for my purposes.

Do your friends and family ask for your advice or help in using computers?

Do your co-workers see you as an expert user of [software]?

0

I've used a list of potential tasks in which one would use a computer. For example:

I use a computer to:

  • Check my email using Netscape Thunderbird
  • Check my email using Microsoft Outlook
  • Find out what my friends are doing on facebook
  • Get information about using Twitter
  • Research big purchases like a new car or TV using the web
  • Create my own web page using Microsoft Publisher
  • Edit photos using Adobe Photoshop

I use something like this as a pre-test survey, then when they are in the testing room I will ask more probing questions like "How did you start researching the new car purchase?" or "How often do you edit photos." etc.

I use the names of actual software applications and sites as I can, usually but not always, determine their level of expertise. Someone who uses MS Office may be a very advanced Word or Excel user, but someone who uses Open Office may be more advanced in web search, application installation, and general knowledge of technology.

0

Knowing someone's proficiency in related tasks might be more helpful as most people tend to over or underestimate their GENERAL skills. So if you asked them if they are at ease with:

  • posting a picture to Facebook or any other site,
  • editing this image on a website that offers that type of service,
  • grabbing attachments from an email,
  • editing images,
  • ... all the way to complex tasks

, you might end up with a more accurate profile.

0

I think it is important to understand the information or outcome that you want to achieve by asking the question, as it will determine the relevance of the response. It is difficult to do in a simple one question response as there might be a lot of contextual factors that you want to tease out.

For example, if you just want a general measure to sort the user groups into different levels of perceived computer literacy then there are probably some standard questions based on commonly used applications and tasks that you can choose from. However, if you already know something about your user group (e.g. they are an older population) then maybe you need something more specific to what they are familiar with so you can separate them into groups (otherwise you might just get everyone falling into one category).

Also, if you are testing or working on a specific type of application, you might want to ask a question that is targeted at the tasks that are typical of those applications (maybe based on usage statistics to determine how complex or common that feature/task is).

Finally, I assume that the reason you are interested in an one item question is because you might have seen things like the NPS score that can encapsulate a lot of variables into one simple figure to report on, but this is also assuming that you either don't need to be able to work out later on what those factors are by not collecting information on it.

I don't know of any 'magical' metrics in UX because the research that we do are so highly context dependent, but if you come across any then I'd be keen to test them out as well!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.