How to best ask for computer experience in a survey?

Question

Our business software has a heterogeneous user base ranging from tech-savvy engineers, over computer used to be salesman up to older, tech-averse craftsman.

In a coming survey I would like to ask for the computer expertise of them, in order to better sort their responses to our three main personas. I know its quite difficult and biased to ask for habits like how many times do you use or do you feel yourself a expert-user?

Thus I thought about asking for software they regularly use. And simply sum up how many programs they use. Having 9 options for mark, I would put an respondent to beginners (my craftsman) if he/she would mark only up to 4. Up to 7 used programs would be intermediate (my salesman) and more a power-user (my engineers). Using Shortcuts would give an extra point. Additionally I tried to find software, that matches computer expertise (even the understanding and naming of it). So first row for beginners, second intermediate, third pros.

How often do you use our software in a week? ________ times

What kind of software do you use often?
[ ] Outlook or other Mailclient
[ ] Internet Explorer or Browser
[ ] Word
[ ] Excel
[ ] Zip Archiver
[ ] CD/DVD Burner
[ ] CAD Software or 3D Software
[ ] Access or Database
[ ] Navision or Managment System

Do you regularly use Shortcuts? ( ) Yes ( ) No

What do you think about this approach? Any suggestions?

Or do you have any best practises regarding this theme?

Answer 1

It is a very difficult thing to do successfully. The problem is that some people may use shortcuts without knowing about it, and some of your craftsmen and salesmen may actually use a number of different programs, because they have to, without actually understanding them.

I would suggest, because you probably know your clients well enough, that you list a whole range of software, and ask them if they use it, with some details:

Outlook
Lotus Notes
Internet Explorer
Firefox
Chrome
Excel (for spreadsheets)
Excel (programming)

etc. Then you put weighting on each choice - firefox should weigh more than IE, and Chrome slightly more, because these are unlikely to be their defaults. You can put Excel programming with a high weighting, as they are probably fairly technically savvy.

If you also include some very techy stuff - whatever is appropriate for you most techy users - you should be able to get a score, and then divide into your 3 categories - or possibly more, because it is unlikely that all of you salesmen, for example, are at the same level of skill.

And this is why it is difficult to do - because there are no simple answers. Your best craftsman may be more technically savvy than your worst salesman.

Answer 2

You need to ask concisely in an easily measurable way. "Which software do you use" is probably not the best method. If you have three personas, try to come up with a single question that each would answer differently. If the difference in those personas in this regard can be summed up as "what is your level of comfort/experience with computers" then maybe you should just ask.

People complain about self reporting and say it's subjective, but the issue is any other measure is also subjective (it is a measure of an opinion of one's skill), and measures like "how many hours do you use a PC" or "do you use Excel" neither get to the heart of the issue nor are they easily measurable. You end up with a measure that you are subjectively evaluating without knowledge of the user.

Am I suddenly computer proficient because I can use CAD software? I've met engineers that aren't computer geeks. Am I a computer whiz because I spend hours on a computer every day? Many people use computers daily at work and at home and don't know basic shortcut keys or what a "browser" is.

Be direct. You'll save yourself and your survey takers some trouble by just asking the damn question.

Ask for their level of comfort. This tells you a lot about how they use computers; someone that tells you they're "good" with computers might be a novice, an expert or somewhere in between. Asking their level of comfort with complex or unfamiliar tasks is more likely to sift out those who are actual "experts" who solve their own problem, as opposed to people who never do anything complicated on a computer, thus consider themselves "good" at computers.

If you want a slightly more accurate measure, consider a small handful of Likert Scale questions like:

I am very comfortable with computers.
I know many advanced features in the software I use.
I am comfortable learning new software or using new computers.

This way you can average the score if you feel a single item is too imprecise, but the bottom line is there is no standard measure of computer proficiency and until there is, any random questions you ask vaguely relating to computer proficiency are probably going to be worse than just asking the direct question.

Consider some previously used measures of computer competance. See the Measures in the Appendix of the article Effects of User Characteristics on Computer Attitudes among Undergraduate Business Students.

If you want a more clear picture of any given user's level of expertise, try a single, open ended question asking them to elaborate on their level of expertise and comfort with computers and why they feel they are an expert/novice. If you really want to be accurate this will give you a lot more information without being as clumsy as the software usage checklist.

Answer 3

I would go even higher level than this, you will be surprised about how much people do not understand the software they use. Is this a survey or interview?

I would be asking them what software do they use to access the internet and send email. Do they know what an IM is? Asking about their habits outside of work may well illuminate more about their skill set, when they come into work. Also something about how and where they access the internet, mobile etc. May help frame and deliver some richness to your personas.

Answer 4

I think the answer needs to be tailored to the type of users and applications you are designing the survey for, but generally there would be three different things to keep in mind:

1. Frequency of usage, as this can determine whether there are set patterns or behaviours the users might be familiar with.
2. Complexity of usage, as this can vary depending on the context and type of tasks the user has to complete, and is also related to their experience and frequency of use.
3. Experience with other similar software applications, as this contributes to their overall knowledge and learning capabilities.

How you extract this information from the users needs to be carefully thought out, but it varies depending on the demographic and relative competency compared to other user groups.