0

I need to elicit quantifiable responses to two designs. One is the live design, the other is a prototype, so I cannot evaluate directly via usability or ease of use. It has to be via Q and A. Anyone got any specific measures they would use?

Thanks all

Improve this question

2 Answers 2

Reset to default
2

You would be able to compare the two by making a prototype of the existing site. This prototype (of the existing site) should be presented with the same tool and have the same level of detail.

You needn't make a prototype of the entire existing site - only those points which you want to compare. Display to the users the same issues and show how it was solved in prototype A (the original) and prototype B (the new one).

I wouldn't tell the test takers that one example is from the live site. As far as they're concerned they are reviewing two prototypes.

Improve this answer
4
  • This will, I guess, work best with users who haven't already seen /used the live site.
    – TripeHound
    Commented Jul 18, 2018 at 15:47
  • @TripeHound - yes. If someone is quite familiar with the site it wouldn't work quite as well. The prototype of the existing site would be too "familiar".
    – Mayo
    Commented Jul 18, 2018 at 15:56
  • 1
    Agreed. You should be able to get some useful information even if the user recognises one of the prototypes as being the live system; but I strongly suspect you'll get better information if they've never seen the live system.
    – TripeHound
    Commented Jul 18, 2018 at 16:00
  • Thanks, both, there is an existing system that all people will know, and it is not possible to replicate all of the functionality in the test site (there are too many variables). I might resort to Product reaction cards - or maybe even a five-second design test! Thanks again
    – James Hoeksma
    Commented Jul 19, 2018 at 15:55
0

It's going to be tricky to get meaningful quantitative data comparing a live site and a prototype (or even between two sites). Especially if they're significantly different from each other. Even if the measured times and success rates are significant, how do you determine what factors caused the differences?

A Q&A survey has all the drawbacks that surveys have:

  • They typically rely on users recalling what they did and why, and that's unreliable.
  • Survey takers sometimes skew their answers toward their own points of view and what they think the moderator wants.

I'd suggest staying away from surveys and timed tasks.

Rather than timing users as they perform tasks, you can quantify the number of actions needed to complete common tasks. With Keystroke-Level Modeling you'll go through the tasks and note exactly what clicks and keystrokes they require to complete, and the time associated with each. You'll end up with quantities of actions and surprisingly accurate times for task completion. These you can compare. (This is done without users; you'll probably do it yourself.)

KLM is an old method, but I was always impressed at its accuracy at modeling expert behavior.

Improve this answer
2
  • Thanks, Ken, I have had a quick look at the method and might give it a try. Two things occur. Do you run the typing test first on yourself? Why is this time on task better by this method than time on task from a user? is this because of the weighted times in the methodology?
    – James Hoeksma
    Commented Jul 19, 2018 at 15:52
  • I wouldn't bother doing a typing test. You can just pick one of the K values (.28 sec is normal) and use it for both conditions. KLM is meant to be just an estimate - it won't predict actual performance times precisely - but it avoids introducing variables you can't control, allowing you make meaningful comparisons of the number of interactions and time to complete tasks.
    – Ken Mohnkern
    Commented Jul 19, 2018 at 16:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.