17

Our business software has a heterogeneous user base ranging from tech-savvy engineers, over computer used to be salesman up to older, tech-averse craftsman.

In a coming survey I would like to ask for the computer expertise of them, in order to better sort their responses to our three main personas. I know its quite difficult and biased to ask for habits like how many times do you use or do you feel yourself a expert-user?

Thus I thought about asking for software they regularly use. And simply sum up how many programs they use. Having 9 options for mark, I would put an respondent to beginners (my craftsman) if he/she would mark only up to 4. Up to 7 used programs would be intermediate (my salesman) and more a power-user (my engineers). Using Shortcuts would give an extra point. Additionally I tried to find software, that matches computer expertise (even the understanding and naming of it). So first row for beginners, second intermediate, third pros.

How often do you use our software in a week? ________ times

What kind of software do you use often?
[ ] Outlook or other Mailclient
[ ] Internet Explorer or Browser
[ ] Word
[ ] Excel
[ ] Zip Archiver
[ ] CD/DVD Burner
[ ] CAD Software or 3D Software
[ ] Access or Database
[ ] Navision or Managment System

Do you regularly use Shortcuts? ( ) Yes ( ) No

What do you think about this approach? Any suggestions?

Or do you have any best practises regarding this theme?

5
  • 1
    If you can - do a trial run on a small sample first, before doing the full survey.
    – PhillipW
    Jan 11, 2012 at 10:00
  • That will be hard. People answer based on what they think they know; or even better, if they know the term or have used it at least once. The opposite of that is people who know something to an expert level, but don't feel confident with it.
    – b01
    Jan 12, 2012 at 15:20
  • I think there's a good chance that this scheme would classify Linus Torvalds and Don Knuth, amongst others, as novices: I could see them possibly scoring only 2/9.
    – Emmet
    Jul 24, 2014 at 17:49
  • @Emmet May be. If they have been working in our company two years ago. ;) It wasn't meant to be universal, but specific to my company and employees.
    – FrankL
    Jul 26, 2014 at 19:00

4 Answers 4

6

It is a very difficult thing to do successfully. The problem is that some people may use shortcuts without knowing about it, and some of your craftsmen and salesmen may actually use a number of different programs, because they have to, without actually understanding them.

I would suggest, because you probably know your clients well enough, that you list a whole range of software, and ask them if they use it, with some details:

Outlook
Lotus Notes
Internet Explorer
Firefox
Chrome
Excel (for spreadsheets)
Excel (programming)

etc. Then you put weighting on each choice - firefox should weigh more than IE, and Chrome slightly more, because these are unlikely to be their defaults. You can put Excel programming with a high weighting, as they are probably fairly technically savvy.

If you also include some very techy stuff - whatever is appropriate for you most techy users - you should be able to get a score, and then divide into your 3 categories - or possibly more, because it is unlikely that all of you salesmen, for example, are at the same level of skill.

And this is why it is difficult to do - because there are no simple answers. Your best craftsman may be more technically savvy than your worst salesman.

8
  • Agree with give them a list of prompts. People recognise items better than the recall them.
    – PhillipW
    Jan 11, 2012 at 9:52
  • And the point is "other MailClient" is not very clear. It is important not to make people feel stupid. So a list of apps, some of which they will use, is very neutral. Nobody will use all of them. Jan 11, 2012 at 9:54
  • "some people may use shortcuts without knowing about it" thats why they wouldn't get an extra point either. I assume a beginner wouldn't even know this term, but power-users for sure. The browser choice is a good one and divide Excel, too.
    – FrankL
    Jan 11, 2012 at 10:00
  • Not sure it would work. I don't know how I would answer this, but I am definately a power user. I think it needs more clarity, as part of the list. Jan 11, 2012 at 10:07
  • Weighting and scoring is worth considering, but makes an extra effort. Which I like to avoid. Actually we have some more personas who are kind of in-betweens in terms of computer experience. I posted the 3 main characters for a simpler explanation only. Thanks for your good ideas.
    – FrankL
    Jan 11, 2012 at 10:08
7

You need to ask concisely in an easily measurable way. "Which software do you use" is probably not the best method. If you have three personas, try to come up with a single question that each would answer differently. If the difference in those personas in this regard can be summed up as "what is your level of comfort/experience with computers" then maybe you should just ask.

People complain about self reporting and say it's subjective, but the issue is any other measure is also subjective (it is a measure of an opinion of one's skill), and measures like "how many hours do you use a PC" or "do you use Excel" neither get to the heart of the issue nor are they easily measurable. You end up with a measure that you are subjectively evaluating without knowledge of the user.

Am I suddenly computer proficient because I can use CAD software? I've met engineers that aren't computer geeks. Am I a computer whiz because I spend hours on a computer every day? Many people use computers daily at work and at home and don't know basic shortcut keys or what a "browser" is.

Be direct. You'll save yourself and your survey takers some trouble by just asking the damn question.

Ask for their level of comfort. This tells you a lot about how they use computers; someone that tells you they're "good" with computers might be a novice, an expert or somewhere in between. Asking their level of comfort with complex or unfamiliar tasks is more likely to sift out those who are actual "experts" who solve their own problem, as opposed to people who never do anything complicated on a computer, thus consider themselves "good" at computers.

If you want a slightly more accurate measure, consider a small handful of Likert Scale questions like:

I am very comfortable with computers.
I know many advanced features in the software I use.
I am comfortable learning new software or using new computers.

This way you can average the score if you feel a single item is too imprecise, but the bottom line is there is no standard measure of computer proficiency and until there is, any random questions you ask vaguely relating to computer proficiency are probably going to be worse than just asking the direct question.

Consider some previously used measures of computer competance. See the Measures in the Appendix of the article Effects of User Characteristics on Computer Attitudes among Undergraduate Business Students.

If you want a more clear picture of any given user's level of expertise, try a single, open ended question asking them to elaborate on their level of expertise and comfort with computers and why they feel they are an expert/novice. If you really want to be accurate this will give you a lot more information without being as clumsy as the software usage checklist.

1
  • 1
    I don't think it's good to ask for opinions in a usability survey. Its not about marketing, but behaviour driven. Probably I'll get a huge gender bias as all males feel comfortable with techs since birth and women not.
    – FrankL
    Sep 11, 2012 at 19:36
2

I would go even higher level than this, you will be surprised about how much people do not understand the software they use. Is this a survey or interview?

I would be asking them what software do they use to access the internet and send email. Do they know what an IM is? Asking about their habits outside of work may well illuminate more about their skill set, when they come into work. Also something about how and where they access the internet, mobile etc. May help frame and deliver some richness to your personas.

1
  • That's good points, but asking open questions aren't easy to analyse. I prefer just to sum and calculate in a spreadsheet instead of reading through it. This questions are intend for a short inspection of people in the beginning of my survey, so they shouldn't take much effort.
    – FrankL
    Sep 11, 2012 at 19:38
2

I think the answer needs to be tailored to the type of users and applications you are designing the survey for, but generally there would be three different things to keep in mind:

  1. Frequency of usage, as this can determine whether there are set patterns or behaviours the users might be familiar with.
  2. Complexity of usage, as this can vary depending on the context and type of tasks the user has to complete, and is also related to their experience and frequency of use.
  3. Experience with other similar software applications, as this contributes to their overall knowledge and learning capabilities.

How you extract this information from the users needs to be carefully thought out, but it varies depending on the demographic and relative competency compared to other user groups.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.