How to evaluate an interface without users?

Question

I need to evaluate my application's novel interface. However, due to IRB issues I can not involve users for this stage of the evaluation. Previously, I asked How to quantitatively compare two interfaces?, which gave me some useful feedback but most of the metrics include humans.

At this point I have to do some form of evaluation. I plan on doing a formal study but that will be at a later date. My tool is an IDE so the two ideas that I can think of are:

• Count the number of actions to complete a task in a few IDEs and then compare the results. I would probably decide on 3-5 different common tasks that developer's perform and compare them for 2 or 3 IDEs.

• Write out a few use cases/user stories that describe the process of a user trying to accomplish a given task in a few different IDEs.

How else can I evaluate my tool without users?

Answer 1

A few possibilities come to mind:

• Heuristic analyses (requires 5-6 fellow HCI practitioners in order to be comprehensive)
• Validation against personas or behavioural goals (may not be reliable)
• GOMS / KLM analyses - basically your first bullet point (reliable, but doesn't uncover issues with IA or broken interface metaphors)
• Comparative assessment (researching the performance of similar interfaces - though this is risky, as it's hard to determine which differences between the UIs are significant and which are not)
• Accessibility validation (verifying that the UI works with assistive technologies - only relevant if you have to support disabled users)
• Cognitive walkthroughs (run through tasks and validate that users have all the information they need available when required and that users will not exhaust their working memory)

Answer 2

It's time to conduct a competitive heuristic evaluation. Your question has described most of the solution, so I'll talk about how you put those together.

First, if you haven't yet, write up the main use cases for your IDE. If your use cases are too broad ("an experienced developer will maintain ~100k lines of Python code"), then you add in additional details for those use cases and focus on an appropriate subset.

After you have a good set of use cases, perform a heuristic evaluation of your novel approach that focuses on those use cases. Then perform a heuristic evaluation of other IDEs following those same use cases. I generally use Nielsen's ten heuristics when I'm doing a heuristic evaluation, largely because they're well-known and have reasonable coverage of issues that you'll encounter. For example, someone else mentioned the idea of noting how keyboard-friendly the applications are; this would be covered under the heuristic of "flexibility and ease of use".

After you've conducted the individual heuristic evaluations, then you will need to analyze the data and determine the comparative strengths and weaknesses of each of the IDEs, and your plan going forward will probably include some design changes to your IDEs to learn from that evaluation and ensure that your IDE has all of the strengths that it needs.

I'm going to presume that you're aware of the Workshop on Evaluation and Usability of Programming Languages and Tools (PLATEAU), which is co-located with SPLASH (neé OOPSLA) this year, and that you're looking at that as a potential publication for your work (or at least a place to go and find others who are doing similar work).

Answer 3

Possibly how keyboard friendly it is. It is amazing how much time you spend removing your fingers from home row, moving the mouse, clicking then repositioning your fingers again.

Answer 4

Actually you are using humans for the evaluation of your novel UI, ain't you one? :-)
I'm sure there are a few more humans with you that might be helpful for this task.
The problem with you and the other humans around is that you know things about your UI that would slant the results of a naive user test.
To overcome the IRB restrictions and the slant thing you might want to hire a few non-humans: a set of personas.
You might not be able to hire humans and show them the UI, but you might yes be able to interview a number of would-be users without showing them nothing (if you didn't already) to learn their cravings and frustrations about the way they like to do things: how they do, what's the problem, and how would they like to. Especially, take note of what they label as unfair. Ask them to bring their notebooks so they can show the flagrant unfairness.
Analysis of the interviews will result in a list of items. After a while new interviews won't add any more items; this means that you are done.
Refrain from rushing to add features to your product, first you have to build the personas that will enact the humans that you can't hire.
If everything went normal you'll end up with a small number of different attitudes each resulting in a persona.
Now it's test time!
Choose one of your personas and wear his hat, say "Joe Eclipse".
Ask your fellows to watch you (or whoever) acting as Joe and to slap your hand if you ever derail from Joe's expected behavior.
Speak out every single action and do it at a slow pace so the humans around you can bear with you.
Take note of every little frustration or issue that reared.
If you have use cases, as Nadyne suggested, then they might be helpful. But remember that those might not be Joe's UCs.
Use only the name, the goal, of the UC. Blithely following all the steps is more like QC testing than usability testing. You should try different ways to do thing, ways you took note of during the interviews. Like "I copy all the code, do the heavyweight edition with vim, and then paste it back" (I do that).
When you are done with Joe go for Jane, and then for the other one. I guess you'll end up with three personas.
Don't forget to invite them to the launch party.