How do you go about comparing the usability of platforms?

Question

We're currently doing usability analysis a set of kiosks and we're in the comparative analysis phase. The trouble is, we're not exactly sure how to compare the usability of a platform (as opposed to a particular implementation on that platform). This is akin to trying to do heuristic usability evaluation comparing SharePoint, Drupal, and Wordpress. Are there any guidelines for doing this kind of thing or are we on our own?

Answer 1

It sounds like you want to evaluate the usability of the platform as a way to deliver usable products, so you have two kinds of stakeholder and two distinct sets of objectives: yours as developers, and those of your user base. If you were to use SUS (or any other usability evaluation framework), you'd have to apply it twice, and identify a way to weight the results (probably in favour of your end users, but that's your call).

Usability itself comprises 4 key measurable qualities:

1. speedy; how quickly (time, number of interaction steps) can users achieve their objectives - fewer "clicks" are better
2. intuitive; how obvious are the steps (explicit and familiar metaphors are better, but fewer visible choices are better - may compromise speed of navigation)
3. accommodating; can a user recover incrementally from mistakes (undo, revert)
4. helpful, not intrusive; can a user access tutorial and content-specific help or guidance; examples are often more usable than reference information

In terms of metrics, there are several pragmatic ways you could measure each platform, depending on how much effort you are able to invest:

• the presence of specific usability-enhancing developer or user features
• the time it (would) take to develop a small but sufficiently relevant and diverse set of specific apps on that platform
• the time it takes a set of sample users to carry out a specific task (includes learning and operation), and then a variant of the task (mainly operation)

Features alone are not usability; ideally to measure anything that's inherently subjective, you are going to need experimentation and a set of objective measures.

Finally, don't overlook pleasure and satisfaction as aspects of usability; attractive, "high fidelity" and "physical" metaphors are more fun, and clear feedback (especially if it doesn't require any explicit interaction) enhances a user's sense of accomplishment.

As developers, aside from pure development, you have several "meta" objectives for a platform, e.g. how easily can you perform system tasks like porting, upgrading, instrumenting or internationalising apps on that platform; again, a combination of feature checking and empirical feedback should help you measure consistently.

Answer 2

I think you could try on making a rough draft or sketch for each of these platforms as a rapid prototype, knowing the technical limitiations and challenges of each of them, after studying it or brainstorming with your tech know-how guys. These will work out a way for such as to understand what could be good solutions in each of the platform.

Now group the pointers and devise a matrix or likert scale arrangment to map all essential items that needs to be compared. With your rapid prototype that is created earlier, will also be a visual imagery for comparison. Pull in a people to let you know what works well in the design. This is akin to think aloud protocol. Now you will have opinions also listed and noted down in the matrix.

Choosing a best platform or real comparison of the platforms can be made if you infer the survey results with each of the merits and demerits. I would leave you to pick your own measurable objectives.

Answer 3

Treat the platforms exactly as they are - applications o which to do some work to achieve a task. In this case, the task is the building of a system, but they are still user applications.

Your development team are the users. The task that is required, is to produce a good, usable application. Do an assessment on the platforms to see whether this set of users can achieve their required task in a smooth and clear way.

Some of this will be - can you make the finished product flexible enough to handle usability criteria? Is the performance of the platform sufficient/appropriate? Do the devlopment team need more or less training to use a particular platform?

It is not as easy as a front-end usability check, but it should be possible to apply the same principles. If the platform is not useable for building a usable system, then your end system will not be usable.