4

In an web-app we have a bunch of command functions, each represented with an icon and a label.

a row of icons with labels

We've done usability testing of the UI as a whole, and taken note that a couple of icons are a bit vague. Our designers have come up with a bunch of alternative designs for each of the not-bleeding-obvious icons.

We'd like to test out these new designs, but it wouldn't be practical to do a full UI test for each of the variations.

What are some techniques and methods to test these icons?

Improve this question
5
  • 1
    General usability testing should work, no? Give users some tasks "Create a new enquiry" that sort of thing, and see if they use the icons to do that.
    – JonW ♦
    Apr 7, 2014 at 8:51
  • "and a label" = wouldn't that remedy any nit-picks about the icons?
    – DA01
    Apr 7, 2014 at 16:21
  • @DA01 read "and a label" as "and 10-20 text characters" and you'll see it's not much better
    – Erics
    Apr 8, 2014 at 21:29
  • I guess I'm not following. Are you saying each icon will also have a label accompanying it?
    – DA01
    Apr 8, 2014 at 21:41
  • I ended up doing something very similar using usabilla. Divide up the testers so each is shown one set of icons. Then take screen shots of the app with the icons. Label them as 1, 2, 3 etc and ask what do you expect to get when you click/tap on each of the numbers. Compare the results from one set of users to another to see if ones work better than others.
    – nightning
    Jun 11, 2014 at 20:06

11 Answers 11

Reset to default
4

You have a few options, depending on your time and resources.

  1. Use a survey. Show the icon, and give the survey respondent 4 options for what the icon could represent. If you do this, you'll need a relatively large number of respondents to do the statistical analysis necessary to get a good confidence interval.
  2. Ask users what they think icons represent. Let them give you multiple answers; tally up the response and see if you get your intended answer enough times. Again, you'll need enough respondents to be able to get a good confidence interval. You can do this in person, or you can use a survey.
  3. Conduct a small usability study, where you only consider time to locate the correct icon. This is probably a subset of the original usability study that you conducted.

Measuring Usability has some additional ideas for testing icons; these are the ones that I've found the most useful when I've specifically had questions about icons.

Improve this answer
2

I am a big fan of hallway testing. Go find someone who has never seen your application (e.g. Sarah in Accounting, or your mom) and ask them for a few minutes about your application. You can just use printouts to make it easier to go from person to person.

Improve this answer
6
  • Lightweight methods like hallway testing work if you're talking to people who are your users (or potential users). Unless Sarah or the poster's mother is a potential user of the system, asking their opinion about the icons is not going to lead to useful results.
    – nadyne
    Apr 10, 2014 at 1:51
  • 2
    @nadyne I disagree. Hallway data is useful. It's different, but still useful. Iconography shouldn't vary terribly from one type of user to another (in fact, good iconography strives to be relatively universal whenever it can). Of course, asking the end users is always preferred, but I wouldn't discount other sources of feedback.
    – DA01
    Apr 10, 2014 at 2:21
  • Icons are contextual. While I agree that some icons are "relatively universal whenever it can", a specialized application will have icons that make sense in the context of the application, its users, and their environment. Testing with Sarah or the poster's mother is only going to capture that context if they are users (or target users) of the application. If the icons are sufficiently important to an application and the application is not a "relatively universal", I could not recommend relying on hallway testing to give meaningful data.
    – nadyne
    Apr 10, 2014 at 16:59
  • 1
    @nadyne I agree, for specialized apps getting universal icons is surely difficult, and hallway testing should never be the end-all and be-all, but it's a quick and easy test that has very minimal cost (e.g. time, paying testers, paying graphics people, or talking to customers)
    – TruthOf42
    Apr 10, 2014 at 20:18
  • @nadyne I completely agree that they are contextual--hence the suggestion in my answer that it's important to provide this context. In this particular case, though, I'd argue that the fact that there are labels on these icons makes the concerns moot to begin with. In context, these icons will have labels that clearly state what they are. :)
    – DA01
    Apr 10, 2014 at 20:34
2

One question that comes to mind is are the functions/actions associated with those icons common knowledge or are they native to your product? If native to your product, one thing to think about is the primary action or purpose behind what they signify. You need to identify the expectations of the users for clicking those icons and create imagery aligning with that data.

For instance users usually expect a pencil icon for the ability to edit. Are you able to afford your user information, visually, to what those icons will be getting them?

https://medium.com/design-ux/bcc8e0257bc8
http://www.1stwebdesigner.com/design/choose-perfect-icons/
https://www.measuringusability.com/blog/icon-tests.php

Analysis of the icons and anything preexisting that is similar within the public space will help. Check out documents on common UI symbology to see if there is indeed precedence. Possibly do some quick surveying or your UI or even a Cafe-study with multiple iterations. That should allow you to quickly gauge variations quickly with minute, measured changes. Checkout how Google does this through their developer videos on Youtube for some how-tos.

Improve this answer
0
1

I guess you could try and ask your testers what the icons mean to them. You could try this by giving them more or less context (about where the icon will be, what tooltip will be available, etc.) to see if they can guess what they will be used for.

Improve this answer
4
  • That's not bad, but icons aren't always meant to be readily identifiable sans context. In other words, sometimes it's OK for an icon to be a learned element, rather than defacto intuitive. It all depends on the tool and the audience, of course, but I worry that you show 10 people an icon and ask them what it represents that you'll get 10 different answers.
    – DA01
    Apr 8, 2014 at 21:40
  • You should get more than 10 different answers from 10 users. You should get at least 30 users from those 10 users. With a large enough sample size, you can do a statistical analysis to determine whether the correct answer is likely to be one of the ones that people think of.
    – nadyne
    Apr 9, 2014 at 5:00
  • You could do a card sorting exercise with the icons and see how they react too.
    – UXerUIer
    Apr 9, 2014 at 5:12
  • @DA01 agree, icons do get used as obvious visual landmarks to help the user quickly find something they already know about.
    – Erics
    Apr 10, 2014 at 1:48
1

Such a subjective subject!

After some research, including reviewing this post, I created a variant test with icon candidates in context with obfuscated labels delivered in an on screen prototype in live usability test sessions. enter image description here

On each screen the main icon in question is swapped out and the user is asked to guess what the icon means and what clicking the button would do. We'll randomize order in several sets of studies to makes sure none of the icons has a context advantage over the others. enter image description here

Once we narrow down our options to a new icon candidate hopefully our team can AB test with current control...however without building in the ability to display site-wide variants and run a long test where at least returning/logged in users keep one variant - any new icon would have a disadvantage.

We've run it in four usability sessions so far and the feedback has been more enlightening than I expected. We have a clear, objective (well, given tiny sample group) front runner already.

Improve this answer
0

I think it's fair to say that Microsoft really managed to 'screw-the-pooch' on their Visual Studio 2012 icon re-design. Even after extensive usability test with positive results (approx. 50 users, but a poorly constructed test). It was the Beta program and community feedback that lowered the damage.

Taking a lesson from this you may try some community engagement. Sure along with the responses with some insight you will get a lot of "I like... / Don't like... " but that's no harm, because at worst only need to test the "most popular" icon set to ensure it does no damage.

Upside is you appear interested and engaged in your community.

Improve this answer
3
  • For those who aren't familiar with Visual Studio or the usability study that you criticize but don't give any information about it, it's difficult to determine whether MS really did "screw the pooch".
    – nadyne
    Apr 9, 2014 at 4:37
  • Hi @nadyne I think you misunderstood. The clear issue was with the icon set. There were multiple and significant icon design revisions to original icon set. And significant negative commentary recorded in the media. Yes, I did want to find and reference the MS blog entry which outlined the Usability test but unfortunately I could not re-locate this.
    – Jason A.
    Apr 9, 2014 at 18:51
  • 1
    BTW using tighter date ranges I did locate a reference (see the "Iconography" section) blogs.msdn.com/b/visualstudio/archive/2012/02/23/… . Quote "76 participants, 40 existing and 36 new VS users, showed no negative effect in icon recognition rates" Looks like good science, but it was wrong question. Should (also) have asked "Do users take longer to scan for an icon".
    – Jason A.
    Apr 9, 2014 at 19:10
0

Since your icons all have labels, it seems to me that the labels are the crucial part of the UI here. As long as the icons are unique, it shouldn't matter what they are--as long as the labels are meaningful to the users.

But if you still want to test the icons, I'd suggest a simple bucket-sorting type of exercise.

Show the icons:

1. -----   2. -----  3. -----  4. -----
   | X |      | O |     | * |     | ! |
   -----      -----     -----     -----

And then have them match them to concepts:

a. Which icon represents [concept a] to you? _______
b. Which icon represents [concept b] to you? _______
c. Which icon represents [concept c] to you? _______
Improve this answer
3
  • 2
    I think this test has merit, but one should also ask "What does this do?" without giving possible answers, simply because the end-user won't have a word-bank of actions to choose from.
    – TruthOf42
    Apr 10, 2014 at 12:56
  • @TruthOf42 it's worth a shot, but icons sans any context are often not going to produce consistent interpretations. Icons aren't necessarily designed to communicate fully the function of the button. Instead, they are meant to represent a known function. Show 20 random people the icons on the PhotoShop toolbar, for example, and it's unlikely you'll get people to get anywhere close to naming the actual functionality each represents.
    – DA01
    Apr 10, 2014 at 14:20
  • (and if the downvoter would care to explain, that's always helpful!)
    – DA01
    Apr 10, 2014 at 20:36
0

To expand on the "hallway testing" suggestions already mentioned. You can do the following things:

  • Create a paper mockup that uses the icon without a label and ask users what they think it represents. If they don't know, start feeding them bit by bit to see how far they are from working it out. Then tell them and ask whether they consider it a viable icon. (Only ask their opinion at the end)
  • Create a paper mockup which uses the icon with labels on the homescreen, but in isolation later on. Set them a task that focuses on some other part of the application, and ask whether they know what your icon stands for.
  • Create an interactive mockup and ask the user to perform a task that requires them to use the functions that the icon points to. Get them to think out loud. See if they associate the icon with the functionality, or if not, how they start testing the icons (eg. checking tooltips).

The point of icons is not only that people instantly know what they mean. Usually people don't, even if they're well designed. The point is that people can associate the icon to the concept quickly and use it as they become familiar with the application. That's what you're testing.

Improve this answer
0

Level of abstraction

firstly, you need to consider levels of abstraction associated with different types of signs (work done by Logician Charles Sanders Peirce is a great starting point to better understand the use of signs in general) without delving into too much details I would say that icons could interestingly be interpreted in many ways as they merely re-present the object they refer to, while labels and indeed all words are symbols based on a convention of use (languages) so the margin of multiple interpretation is far more reduced with words; The icon of an apple could refer to the physical object or to health and wellbeing.

Interpretation of icons

Secondly, when designing you need to take into account the perceived level of abstraction of each icon, is it prone to more or less possible interpretations? Ask yourself questions such as what the icon re-presents does, does culture play a role in how users will perceive it? is it commonly used? etc. than Link this to the context of use and associate a descriptive label to the icon as to mitigate and reduce the margin of interpretation. While this could be straightforward for many applications, difficulty will arise when dealing with complex and abstract concepts that don't have an immediate action associated to them: for example processes are generally very difficult to capture in iconic form.

Testing

When testing with users i would suggest testing icons and associated label together and incorporate variations both in terminology and icons to validate your design. This has the advantage of looking at both these items as a meaningful unit and reduce time used testing icons and labels separately. Hope this is of help

Improve this answer
-1

I haven't read stake exchange ux folks recommendations, so it can be a repeat suggested process.

I would strongly recommend to follow AB Testing process. When you are testing especially the icons or two objects this would help. Follow the below steps.

  1. Show them a single icon with plane background and ask what they relate to. After you complete the above
  2. Show the two or three icons that you have designed for a single activity / purpose and ask them rate according to the context relativity
  3. Show them in context with the page in which they are going to be used. But here one trick is show them entire page and provide them the label of the icon only and ask them to locate in the page.

Consolidate all the results and take the one which has high ratings.

Improve this answer
2
  • That is not what I would call an A/B test. A/B testing (in my experience) refers to testing two different designs in production (randomly assigning visitors design A or B) and measuring which has the highest conversion/efficiency/etc. It doesn't involve a moment when you can ask questions. I would call this formative testing.
    – Peter
    Apr 10, 2014 at 14:23
  • I agree with @Peter, this is not an A/B test. (Although it doesn't have to be in production, you can do an A/B test in a test environment; the important thing is random assignment and having a large enough group of participants that you can do statistical analysis.)
    – nadyne
    Apr 10, 2014 at 17:03
-1

I you have the project lunched already than you can use A/B testing with bounce rate as key metric.

If you have pages behind icons than that shouldn't be hard to implement with Google Analytics.

The main idea here is that low bounce rate shows that the page behind the icon didn't meet users expectations.

Improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.