Take the 2-minute tour ×
User Experience Stack Exchange is a question and answer site for user experience researchers and experts. It's 100% free, no registration required.

We're currently doing usability analysis a set of kiosks and we're in the comparative analysis phase. The trouble is, we're not exactly sure how to compare the usability of a platform (as opposed to a particular implementation on that platform). This is akin to trying to do heuristic usability evaluation comparing SharePoint, Drupal, and Wordpress. Are there any guidelines for doing this kind of thing or are we on our own?

share|improve this question
    
Seems like SUS should still apply, right? It's made to compare (reasonably) like systems. But what's the task you're comparing, the process of programming (like making a Drupal site) the kiosk or what? –  Ben Brocka Feb 19 '12 at 19:05
    
@BenBrocka The SUS isn't something we'll be able to use because we don't have access to users at this stage and even if we did we don't have time to implement a survey. And no, we've been able to compare the usability of implementing systems on the kiosk with some defined metrics pretty well, but we've had a hard time comparing the usability of the output of the kiosks in any kind of consistent way since what you can build with each system is so variable. –  Zelbinian Feb 19 '12 at 19:15
    
Just to clarify: the end users are the users of the 'implementation' ? - and your developers are the users of the platform ?? –  PhillipW Feb 19 '12 at 22:14
    
You can run a SUS test with random people from the office that aren't familiar with the system too; it won't be as representative but it will help you notice any huge problems, the ones you'll want to find early as possible. –  Ben Brocka Feb 20 '12 at 1:19
add comment

3 Answers 3

It sounds like you want to evaluate the usability of the platform as a way to deliver usable products, so you have two kinds of stakeholder and two distinct sets of objectives: yours as developers, and those of your user base. If you were to use SUS (or any other usability evaluation framework), you'd have to apply it twice, and identify a way to weight the results (probably in favour of your end users, but that's your call).

Usability itself comprises 4 key measurable qualities:

  1. speedy; how quickly (time, number of interaction steps) can users achieve their objectives - fewer "clicks" are better
  2. intuitive; how obvious are the steps (explicit and familiar metaphors are better, but fewer visible choices are better - may compromise speed of navigation)
  3. accommodating; can a user recover incrementally from mistakes (undo, revert)
  4. helpful, not intrusive; can a user access tutorial and content-specific help or guidance; examples are often more usable than reference information

In terms of metrics, there are several pragmatic ways you could measure each platform, depending on how much effort you are able to invest:

  • the presence of specific usability-enhancing developer or user features
  • the time it (would) take to develop a small but sufficiently relevant and diverse set of specific apps on that platform
  • the time it takes a set of sample users to carry out a specific task (includes learning and operation), and then a variant of the task (mainly operation)

Features alone are not usability; ideally to measure anything that's inherently subjective, you are going to need experimentation and a set of objective measures.

Finally, don't overlook pleasure and satisfaction as aspects of usability; attractive, "high fidelity" and "physical" metaphors are more fun, and clear feedback (especially if it doesn't require any explicit interaction) enhances a user's sense of accomplishment.

As developers, aside from pure development, you have several "meta" objectives for a platform, e.g. how easily can you perform system tasks like porting, upgrading, instrumenting or internationalising apps on that platform; again, a combination of feature checking and empirical feedback should help you measure consistently.

share|improve this answer
add comment

I think you could try on making a rough draft or sketch for each of these platforms as a rapid prototype, knowing the technical limitiations and challenges of each of them, after studying it or brainstorming with your tech know-how guys. These will work out a way for such as to understand what could be good solutions in each of the platform.

Now group the pointers and devise a matrix or likert scale arrangment to map all essential items that needs to be compared. With your rapid prototype that is created earlier, will also be a visual imagery for comparison. Pull in a people to let you know what works well in the design. This is akin to think aloud protocol. Now you will have opinions also listed and noted down in the matrix.

Choosing a best platform or real comparison of the platforms can be made if you infer the survey results with each of the merits and demerits. I would leave you to pick your own measurable objectives.

share|improve this answer
add comment

Treat the platforms exactly as they are - applications o which to do some work to achieve a task. In this case, the task is the building of a system, but they are still user applications.

Your development team are the users. The task that is required, is to produce a good, usable application. Do an assessment on the platforms to see whether this set of users can achieve their required task in a smooth and clear way.

Some of this will be - can you make the finished product flexible enough to handle usability criteria? Is the performance of the platform sufficient/appropriate? Do the devlopment team need more or less training to use a particular platform?

It is not as easy as a front-end usability check, but it should be possible to apply the same principles. If the platform is not useable for building a usable system, then your end system will not be usable.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.