Classifying users into beginners, intermediate, experts

Question

In a survey, how can I find out about the experience of a user with a specific website? How can I classify the users into different types - novices/beginners, intermediate, experts? Are there standardised and reliable instruments that could be used to achieve this?

Answer 1

As Jonas said, the easiest way would be to just ask the users directly, especially since it is a survey. It may be in your interest to break your question down to specific areas of the website to be more granular. It may also help to phrase your questions/answers to fit in a Likert scale.

Examples would be

1. How comfortable are you with _____. (0 to 5)
2. How many features of the site would you say you use? (none to all)
3. How long/many times a day/much time per day/etc. do you used the site (range)
4. etc...

Answer 2

Like mentioned in the earlier answer, the simplest way is to ask the user.

But there are ways to identify users' expertise before a survey, and frame questions accordingly. The explanation of these methods will take you step by step to my suggestion at the end of this answer, that derives a probable way to evaluate users' expertise during a survey.

Brief on how Software giants do it

This is a brief on how giants like Adobe identify the experts:

1. Active involvement in forums
2. Identifying industry leaders (like famous ad gurus, photographers, etc)
3. Following portfolios

Communities and forums are some of the ways to identify novice, intermmediate, and expert users. But, this takes a very long time. For the users to use the forums actively, the app should be established and wide-spread.

Personal experience from a startup

In your case, I am unaware of how popular the website is or if it has a community. But, I can let you know my personal experience. I am an interaction designer at a web startup that provides programming lessons in native Indian languages. We have only around 80 active users and are not well known either.

We depend on analytics to do the trick. We all know the 80/20 principle. 80% of the users use only 20% of a product's features. So, we assign values to each of these features. These values are not assigned based on any research. But, this is a start, and this will eventually lead to a better study.

Example:

To keep it simple, let me take facebook as an example

• Using normal everyday feature (Scrolling through feeds and posting) - 10 points
• Using any other feature related to everyday feature (Tagging photos, adding desc, date and time)- 25 points
• Using hidden features (Privacy settings) - 50 points

You get the point. So, every user will have a score. Google analytics can also tell you what and how frequently a feature was used by a particular user. Crunching this data will let you know the classification of your users. But, it is better to understand the identified users in person, as the first few attempts are prone to technical and human errors. You can then correct the analysis repeatedly to get to an efficient scale.

Screenshots

In the above screenshot, every index corresponds to a particular feature or a page. We first started using Google analytics to understand site performance and identify usability issues. Now, we are figuring out a way to identify potential users for interviews and their needs, which is again just an extension of identifying usability issues on a more personal level.

IMPORTANT CONCLUSION

I understand what I have explained are all performed before a survey. May be you can tweak it to work during a survey. Categorizing and fixing value to features, and then evaluating a user's answer across these parameters will give you a fair idea on the user's expertise. May be you can use a variation of this method or dump it all together. Just thought of sharing a way that might help.

Answer 3

We did this classification for our educational website. As suggested by everyone, it's best to ask the user directly. However, it's important to let them know what you mean exactly by different levels of understanding. Just putting "Beginner, Intermediate and Advanced" would be vague. We gave a one line explanation of what we meant by different levels of knowledge. [Screenshot]

Also, the user should be given a choice to change his preference later.

Answer 4

This is a tough one. In any case you need to know certain properties of the web application before you can classify skill. Ask yourself if this is a user only survey, where we won't ask IT-Pros or developers of the application. The Second thing to ask is level of complexity. Is this a simple straight forward app where there is only one way to complete each task, or multiple ways where one set of actions is faster than the other? Third you need to know who your users are. Are they working in area where the use of this app is mandatory, or do they use it as a "nice to have" app, which really isn't necessary? A fourth notion would be the users self estimation. Can they really tell at what level they are, if they don't know the entire app? What kind of quality does that give us if we "Just ask them"? Is time a relevant factor? Can we tell that a user who worked with the app is more skilled than a user just using it for half a year? It depends on what types of tasks the user is exposed to, and can we tell which one knows more?

With all these parameters lined up, we can only do one thing to determine the level of skills of users, and that is to test them. Give them a task oriented test, determine the levels before what is beginner, intermediate and expert and let them try it out. This is the only way to be sure that an intermediate user is an intermediate user. She scored 56 point of a 100 at the test. Self estimation won't do.

Answer 5

I don't know of any tools to determine level of user "type". Sure, asking them to describe their expertise level is fine, but it shouldn't be trusted; just like customers know what they want, not how to get there, users know their expertise, but not what the expertise actually means.

As someone who came from QA, I have four classifications for users:

1. Beginner/non-tech savvy
2. Intermediate/somewhat tech savvy
3. Expert/tech savvy
4. Master/can replicate all previous types

The reason I add a fourth is because, as someone who's tested various products (hardware, software, appliances...you name it), each product type has it's own goals and user types but they also are all based around the same expertise levels. Since I have spent most of my career in tech, I also classify how tech savvy the user may be. And a mastery level does not mean "developer" or someone who worked on the product. It means very specifically someone who can follow all paths because a clear user understanding on all levels.

With that intro out of the way, determining user expertise is based on surveys and usage. The survey questions, however, need to be formatted in a way that the answer indirectly explains how savvy the user actually is.

For example, in the past I worked on a smart dayplanner app, that had a calendar, scheduling platform, contact list, task manager, and notes...all in one app. From our database we could see which users were using all of the app, or using any individual piece significantly, or using multiple pieces in tandem. Effectively, users who understood how to use the app and did so on a day-to-day basis and did so intelligently? Those were expert/advanced users. Those who used one or two features but did so regularly or semi-regularly? Those were intermediate users. And those who used it on rare occasion and/or only used one of the functions especially the simpler ones like the task manager and notes, they were the beginners.

With that data in mind, we created a survey and delivered it to our customers to ask how they liked the app and how they used it. We spent a good few hours on creating the survey to ask the questions correctly, to make sure we didn't guide the users to answer questions in a way that would affect their answers in any dishonest way.

That said, there were some numeric questions like "how many events do you schedule a week?" and "how often do you look at your calendar in the app?", which were intended to gauge usage. These questions are also more intended to get the user comfortable with answering questions; they lead to the more in-depth full-answer questions. These included: "What types of events do you schedule?", "How do you like the task manager compared to others?", and "What future features would you like to see?"

Each one of these questions is valuable for two reasons:

1. The answers literally answer the given question
2. The answers point to other specifics about the user, including intended use, desired use, understanding of the app, competitive analysis, and more. How? Because smart users who use the app will immediately run into problem areas that we are (or might not be) aware of. Expert users will communicate these because they want us to succeed because they're actively using the app (if they aren't, then they won't be answering the survey). Intermediate users, however, will have a much more limited answer set. Instead of offering suggestions, requesting features, complaining about bugs, etc. they'll likely say that they like the application but have trouble understanding it, or that they haven't tried those features (for no serious reason), or that they haven't gotten to trying something out yet.

Beginners, meanwhile, typically post low numbers and limited full-sentence answers. Like the person who just answers an app review with a star rating and "Nice app!"

There's no one-stop answer for determining user rating/expertise letter. Even with your survey and database queries, you'll still have to analyze the data and do so intelligently. That's why data science is turning into a major field. But for a small company or for light surveys, spending a little more time to make sure the questions provide more than just what's on the surface as an answer will go a long way.

Answer 6

A few quick comments about categorizing users by skill or performance level.

Perpetual-intermediate users. In his book About Face, Alan Cooper talks about the perpetual intermediate. He argues that there is no such thing as an expert—in the long run. Websites and software are always changing, and so experts necessarily slide down the learning curve as things change.

Rote users In his book, Designing and writing online documents, William Horton talks about users who have memorised an exact sequence of steps. They can do this task expertly, but if anything varies—the server is down or some other unexpected error—they have no idea how to proceed. They don't know how to trouble-shoot.

Transfer users. In the same book, Designing and writing online documents, William Horton talks about users who migrate. An occasional user may find the website has changed. Or a website visitor may be very capable of using a competitor's site. These users know the concepts and much of the terminology, but not where specifically to click or tap.

One person, multiple user types. A further complication is that some people are expert in some functions, but not in others. For example, you may be good at finding a book and completing a purchase on Amazon.com, but really inexperienced at sending a book back for a refund.

Asking users to rate their skill or performance level may not give you the data you need. Why don't you try a Five Whys exercise to see if you can uncover whether you really need user skill level, or whether you want something else. The first Why is probably this: Why do you want to know user skill level?

I hope that helps you think about the problem, so you can solve it well. :)

Answer 7

How about a remote usability test? You can combine user tasks with qualitative feedback.

First, find some test users. You could pay a recruiting service, put an intercept message on a web page, or send out a link to your contacts. Depending on who you are targeting, you will likely need to provide an incentive to encourage participation.

Then, use a software solution to set up a usability test. It might look like the following.

1. Question: Why do you typically use site/service X?

2. Task: Please go ahead and find the page you were looking for.

3. Question: Rate your experience.

4. Task: Imagine you were looking for page/feature Z. Go ahead and find it.

5. Question: How easy or hard was it to find Z?

6. Task: If you were to click on link Y, what would you expect to see?

etc., etc.

Some testing suites will return all kinds of data that, with sufficient user participation, you could turn into the kinds of insights you are looking for. I've used Loop11 before, for example, and it provides user clickthrough flows, time-on-task, completion/failure rates... all the kinds of things that might define the different user types you need. This adds a lot more information to a simple survey.

You could also hire a UX consultant to do the whole thing for you and provide a report.

Answer 8

Asking them is a very good and easy way to go. If you can't or are not sure on how much you can rely on their answers, measure their behaviour with Google Analytics, frequency and funnel/goal completions.

I have specific goals per user type to measure how well they know the site, and get them from our own Control Panel or from MixPanel ei. how many posts they have made in a particular period of time.

Answer 9

First think about the possible outcome, how you will use that data? We don't just categorize people for fun but for a reason, finding out your reason to do it will help to create the solution more than any best-practices.

Answer 10

• You are seeking a question (or questions), within the context of a survey, to determine the "experience" of users with a specific web site.
• You have expressed an aversion to self-reporting and frequency of use type measurements.
• You would like a "standardised" instrument.

You appear to want to "test" the users' experience, rather than allow them to self-report it. If that is the case, then you probably need to devise a series of skill-testing questions about the specific site. You could either weight each question by its difficulty, or select questions of equal weight. Then, you would need to categorize the users according to, perhaps, their weighted scores or percent correct. With some piloting of the survey, you might be able to pre-determine what scores will fall in what categories. You might also gather the data and see how you need to categorize the scores after the fact.

I believe your simultaneous aversion to self-reporting and desire for a standardised instrument are probably in conflict with each other. A standardized instrument will have been created, by definition, without knowledge of any specific web site and will, therefore, involve some degree of self-reporting.

Answer 11

As i already mentioned in my comment, the simple answer is just

It's worth noting that the user might have a different opinion about his own experience than his usage data reveals.

Maybe some people see them self as Facebook expert even tho they never visited the security settings while other see themself as Facebook intermediate even tho they submitted multiple security breaches inside the Facebook API.
The way the user behaves and the way the user feels about it might be completely different.

By asking you can obtain information about the feeling of the user about the website

As others already pointed out correctly - you can collect data about the user behavior and weight this information to determine how much experience the user has. Stackexchange does this too. Simply by being an active part of the community (upvoting good answers) you receive +1 point while you get more points for adding value to the network (good answers).

Stackexchange is a good example because someone who got 5 Badges in Java and 0 badges in Design probably is more of an programmer than a designer. This is measurable and gets more accurate with every action the user does on the website.

By determining both the feeling as well the statistics you can optimize your website in this direction. When you have lots of people saying that they're intermediate users while the statistics say that they're experts then you might reconsider the accessibility of your website. The user knows everything about the website while still thinking that he misses something.