3

I have a collaborative computing platform which I want to run a user study on, and compare it with an existing, paper-based system. Both systems are meant to be used in some specific types of meetings (e.g. brainstorming, ...). This is done for an academic publication.

My approach would be to run user studies and evaluate a set of metrics and then compare those metrics. More concretely, I think it makes sense to compare these two different approaches on a per metric basis, and finally come up with a table, with the mentioned metrics as rows and each system as columns.

The question is, what such metrics would be? Is there any widely accepted set of parameters for such studies? Any pointer would be highly appreciated.

Improve this question
2
  • Current question, very broad. If you can add some more specificity to it would be great.
    – Andy
    May 19, 2016 at 11:23
  • @Andy I edited the question accordingly.
    – Ali Alavi
    May 19, 2016 at 11:30

2 Answers 2

Reset to default
1

Valuable metrics are closely linked to goals

Good metrics are simply a way of measuring how well something meets goals.

You might be able to find a standard list of parameters to measure, but it might be argued that the more generic your metrics, the less value they actually provide.

Examples:

  • If a goal is to improve the perceived "performance" of website, 'page load speed' is a metric.
  • If it's a goal to improve an app that user's complain is tedious or inefficient, 'steps/clicks to task completion' is a goal.

In the case of your brainstorming example, seems like a goal might be 'ideas generated per session', or 'interactions of specific type in a certain time period', or 'perceived complexity'.

So first decide what you think are the crucial aspects of the system in the context of goals, and your metrics will practically define themselves.

Improve this answer
1
  • That's a good point, but this is basically the core of my question. In other words, how would researchers compare collaborative systems? What are the common criteria? For example, one might say, for a web usability study, 'performance/page load speed', 'ease of navigation', 'internationalization', and 'accessibility' are some generic critera that we can use to compare two websites. I'm looking for such criteria in CSCW domain.
    – Ali Alavi
    May 19, 2016 at 14:31
0

What do you hope to gain from the sessions? What is your hypothesis?

If you are expecting to see a specific change in interaction or user output - structure your session in a way to quantify it.

How many times did... How long did...

Also you are focusing on the output before completing your research! Depending on your metrics, tables may not be the best artefact to share your findings.

You may find that a 2x2 could provide greater insight. Or a journey map could identify how your users approach problems. You can cluster your findings and identify patterns.

Give them points. Compare. Quantify.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.