Take the 2-minute tour ×
User Experience Stack Exchange is a question and answer site for user experience researchers and experts. It's 100% free, no registration required.

Unlike business and e-commerce sites, web applications don't necessarily have an end-goal or target for users to achieve. What are some metrics that can be used to measure how 'engaged' users are with the application? I was thinking if there are known ways for linking together user actions (e.g. button clicks, menu clicks, field input, etc.) or if there are some ways to infer this (e.g. time spent per screen). I believe this is better than putting social media links or provide forums for users to find out this type of information, so I would like to try adding this to the web application.

A quick research on the literature in this area includes some frameworks like the UES, HEARTS and UMUX. However, these describe general concepts and processes behind measuring user experience/engagement but does not mention exactly what is physically measured. It would be good to try and understand what is measured and how it might be relevant to user experience.

share|improve this question
    
I am not proposing that the metrics can provide the same type of insight that a detailed user group study might be able to provide, but rather as a way to monitor changes that might be happening with with behaviour as a whole. Qualitative measures are difficult to compare between different groups and time periods, and doesn't provide enough information to help point to where the problems might be. –  Michael Lai Apr 11 '13 at 6:05
    
It seems you are trying to operationalize "engagement" without reference to any underlying theory/model of what "engagement" is. –  uxzapper Apr 12 '13 at 0:04
    
From what I have read so far, the model of engagement seems to apply to traditional e-commerce applications where there are specific user goals, and you can base a discussion of engagement around user behaviour while they are trying to accomplish their goals. However, now we have lots of Saas products out there where the goal is defined by the users and not the product, so how do we measure engagement now that the underlying dynamics are different? –  Michael Lai Apr 12 '13 at 0:17
    
You have to start from a position of defining what engagement is. Clearly it is far more than behaviour because it has significant affective and cognitive dimensions to it. –  uxzapper Apr 12 '13 at 1:27
    
I see no qualitative difference between user goals on a web-site vs a web-app. Could you clarify why you have settled on engagement as a construct and why you want to measure it? Perhaps other constructs are what you really want to measure? –  uxzapper Apr 12 '13 at 1:41
show 2 more comments

6 Answers

up vote 11 down vote accepted

Below are the 6 factors used in the O'Brien/Toms User Engagement Scale.

1) Perceived Usability: users' perceived effort, their ability to accomplish tasks, the navigation and organization, and the emotions evoked in user.

2) Aesthetics: screen layout and graphics/images, overall aesthetic impressions of attractiveness, and sensory appeal.

3) Focused Attention: absorption (ability to become absorbed and lose oneself in the experience), awareness (about what was taking place outside of the interaction), and perceptions of time passing.

4) Endurability: the likelihood to remember things that we have enjoyed and a desire to do again an activity that has been fun.

5) Novelty: the curiosity evoked by the task and the participants' interest in the task.

6) Felt Involvement: feeling of being drawn into and involved in the task and the overall assessment of the experience as fun.

As you can see, these are things that require you to get feedback from the user. Measuring a complex (qualitative) construct like engagement purely through a pattern of user actions appears problematic.

References:

O’Brien, H.L. & Toms, E.G. (2010). The development and evaluation of a survey to measure user engagement in e-commerce environments. Journal of the American Society for Information Science & Technology, 61(1), 50-69. DOI: 10.1002/asi.v61:1.

O’Brien, H.L. & Toms, E.G. (2008). What is user engagement? A conceptual framework for defining user engagement with technology. Journal of the American Society for Information Science & Technology, 59(6), 938-955., DOI: 10.1002/asi.v59:6.

share|improve this answer
3  
+1 Nice summary of the User Engagement Scale, and good to see references. –  JohnGB Apr 11 '13 at 8:50
    
In the context of a website or web application where there is a clear goal (e.g. e-commerce website), the exact things you want to measure is probably easier. But if it is an application that supports other user activities, what would you measure to find out about these things by looking at the overall user behaviour. –  Michael Lai Apr 12 '13 at 1:53
    
Simple answer is that I don't think you can measure engagement effectively through user behavior alone. –  uxzapper Apr 12 '13 at 3:01
    
I understand that, and frameworks like HEART does look at both qualitative and quantitative measures. But of the quantitative user behaviour that you can measure, I would like to know what has typically been measured and how. –  Michael Lai Apr 14 '13 at 23:26
add comment

I disagree with you regarding the idea that web applications don't have an end goal or target for users to achieve or complete. Everything in your system should have a purpose. And every purpose can probably be measured.

To answer your question though, I wanna go ahead and acknowledge that engagement as a calculation is very broad, vague, and changes per application. Engagement on Pinterest is probably very different from engagement on Visual.ly. So to measure "engagement", you measure its parts. And to measure its parts, you must define them. (Also, I'm strictly talking about drawing conclusions from data, not user feedback.)

Let's take an example like Wolfram|Aplha.

W|A is a computation engine and doesn't really have typical pages. You enter a term/phrase and it "computes" results for you, and the types of data returned change depending on the context of the query.

Since users don't have a typical end or path (search > results > item), how do you measure whether users are engaging with the results, strictly using analytics? Well, I'd start with these questions:

How can the user interact with the results? Mostly, users can digest the information on screen, but they do have options (some premium), such as: Enlarge, Download, Customize & Save, Copy, Enable Interactivity, and Clip n' Share.

Also, they can change the query through suggested alternatives (relate or disambiguated); and expose more of less information, if applicable;

How can we measure those interactions? Just a few of things you could track: 1. How many clicks is each option getting? 2. How long between one option being clicked and another being clicked? 3. How long is a user staying on each query before leaving? 4. Does the user conduct another query? How long between queries? 5. Is this user a returning visitor?

Here's a glossary for Google analytics and should help to understand the language Google uses to define these types of metrics.

What could these measurements mean? Taken as a whole, the data could be telling you: if users come back (#5); how they interact with the results they receive (#1); if those interactions are appropriate (#1, #2); and if the results are relevant (#3, #4).

I'll note here that I wrote "what could these measurements mean" on purpose. Analytic data needs to be taken very much in context, and that context can be warped easily. So, relying on analytics without following up your assumptions with users can lead you down a wrong path. Louis Rosenfeld talks about this in more detail in this interview.

Interestingly, Kissmetrics notes that when used wrong, time on site can lead you in the wrong direction. This reinforces the idea that data is contextual, having different uses and meanings depending on the context.

The takeaway from this, for me at least, is that engagement is uniquely defined per product and is a constantly moving target. It requires combining lots of different metrics to abstract an idea of what's happening and those metrics will be different depending on what you're trying to measure. And having very specific user goals and tasks will help you define the metric you need.

share|improve this answer
    
Hi brad, welcome to UX.se! You have a nice perspective and answer, can you add links supporting your views to make it a stronger answer? –  rk. May 10 '13 at 21:33
2  
sure! links added! –  brad May 10 '13 at 22:34
    
The purpose of a web application may not be as specific or linear as an e-commerce site, because it may be an app that perhaps doesn't involve much activity at all (e.g. tracks your spending and budget), or that it has more than one major feature (e.g. cloud-based file sharing app). My point is that you can't really define a clear goal or path that you can measure a conversion of task completion process easily. This is where a mixture of different metrics may be required to define a specific type of activity or behaviour, and I am looking to see examples that are used to capture this data. –  Michael Lai Jul 16 '13 at 0:25
add comment

It is difficult to measure user engagement - you are right that sites like e-commerce sites don't measure changes in terms of usability or engagement but instead by a more easily measured metric (such as $$$).

You mention some good bits of data to collect (button clicks, time spent on a screen, etc.), but the key here is that you still need to define higher level engagement objectives, such as:

  • How many users are registering for your application (or how many users are registering after using a demo version?)
  • How many page hits are you getting on your application's call-to-action(s)?
  • How much are users using Feature X in your application as opposed to Feature Y?

Once you have defined engagement objectives, you can then go about collecting the appropriate data (button clicks, page hits, timers, etc.) to answer your research questions.

If defining objectives sounds hard, that's because it is. Note that some people object to the idea of measuring user engagement because of this ambiguity.

share|improve this answer
add comment

One of the most effective ways of finding out user engagement is using heat maps. From a google-fu :

EDIT Heat maps are, as the information in the link says, a tool to build behavior analytics of users. There are different types of heat maps like : a) Click Heatmaps Which show you exactly what visitors are clicking on b) Mouse Movement Heatmaps which track the mouse pointer to generate an eye tracking heatmap. This heatmaps shows users what parts of the page users are actually paying attention to. As per a research, a users eye movement is closely linked with the mouse pointer. So this helps in building metrics for measuring how engaged the user is with your website. to use heat maps, you can either purchase software, find if there are open source options available or pay someone to build a plugin for your website

share|improve this answer
    
Can you add some explanation here as to how this can be used? It's not really an answer currently, just a one-liner and a link off elsewhere. –  JonW Apr 11 '13 at 15:53
    
@JonW, done.I always forget this is not a technical forum. Nevertheless, if an answer suggesting usage of heat maps as a device to measure metrics of user engagement gets a downvote, I really wonder how the user community has really been measuring user involvement. –  happybuddha Apr 11 '13 at 18:55
    
I downvoted it, not because of the suggestion of heat maps, but because the answer itself was low quality - it gave no reasoning or explaination, it was just a brief one-line answer. It is better now so I have removed the downvote. –  JonW Apr 11 '13 at 19:32
    
Gotya. This is more reading on why/how This is an open sourceav‌​ailable. This is a paidsoftware site. I haven't used any of these. Just posting. –  happybuddha Apr 11 '13 at 20:34
add comment

I think about this question a lot.

Clicky (http://clicky.com/) is pretty good for one page analytics and can get you the kind of granularity you need to measure user engagement the Google analytics and others don't. They can actually trace mouse activity on specific buttons and get times between button clicks etc. Think of it as a poor mans heat map and timing diagram tracker for low volume high detail user behavior recording.

The easiest technique is to identify single spot measurement opportunities of particular interest such as an accordion widget. Measure the time between clicking to expand the accordion widget and when the click something next. If you graph all the times you'll see some peaks and humps in the histogram of time between expanding the accordion full of text and the next click. Hopefully one peak or hump in the timeing histogram is about the length of time it takes to read the entire content of the expanded accordion and is an engaged audience. The message in the accordion was what they wanted and they took the time to read it fully. For web apps there's lots steps that go in a set order so you can do this lots of places. There's lots of complicated caveats like are these new or returning visitors and returning visitors might not read the text if they already know what it says from the first time, but consider if they already know what it says and choose not to read it that should count as not very engaging and why would you think it's engaging to repeat your message annoyingly?

On a more generalized approach for multiple engagement measurements you benefit by segmenting your audience into engaged users and non-engaged users or assign some statistical measurement for how engaged they are. This can be done sort of like A/B testing only we keep the page designs constant and try to find patterns in the data of timing and order of clicks. For example lots of quick clicks to child pages followed by hitting the back button is a sign they are not finding the information they want and keep trying again and is a sign they are not engaged well by the main page they are jumping from. A good way to start this larger scale engagement measuring is to act like (or find) a very engaged user to generate an idealized engage persona's click stream. Usually good design relies on a concept of visual hierarchy, which is how a designer uses color and line and type etc. to try to get the audience to read and act in a specific order such as see the picture read the text click on the button for the next slide, etc. Then imagine and act like or find a disengaged user to record their clicksteam. Once you start profiling users or segmenting them based on their level of engagement you can separate the data into two sets or see trends that point to a more engaging design or less engaging design. Start with just a few people, watch closely and try to guess why they clicked what they did. It's often pretty easy to tell someone isn't engaged because they'll do things the web designer or web application developer don't expect and cause you to think thinks like "why did they do that? they're supposed to be interested in this picture and click on it, but they keep going up to the menu's when they land on this page." that would be a sign the page is not engaging because it is left in less time that it takes to be read. If they're not reading something or avoiding it we can confidently say that it doesn't engage them.

In general anything that gets clicked or read or if you have evidence it's looked at and used is engaging. That's kind of what engaging means in a web app. just counting clicks on buttons should show you some never get used and those can be called distractions or alternatively "not engaging".

share|improve this answer
add comment

Of course, you can measure how much time your users spend on each of your app's screen, actions performed by your users, user retention and churn and so on. But it won't tell you the full story, you'll understand the 'what' and not the 'why'. In order to understand the 'why', that is to say, understand how users use you app, I suggest to integrate visual mobile analytics solutions. Such tools actually enable you to put yourself on your users' shoes and SEE exactly how they engage with your app, by providing user recordings and touch heatmaps. I came across this article that presents this interesting approach of understanding user behavior via visual mobile analytics: http://online-behavior.com/analytics/in-app-analytics

And this articles referrers to Appsee Mobile App Analytics, which can be the tool you're looking for for measuring the user UX and engagement. I haven't tried it yet, but it offers a free account, so it's worth giving it a try.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.