How to evaluate a console application?

Question

I have to perform an end-user evaluation on a console application.

• the application is a sort of a software library, i.e., a series of commands that a programmer can use to perform some specific tasks
• the commands are a sort of command-line interface: instead of pressing buttons, the users can assemble the commands to perform complicated things (as a composition of the results those commands generate)

I was thinking about a usability test, but I have some doubts. In the following, I try to exemplify it.

---

Let's make an example.

Suppose that my library is composed of two commands, i.e.,

1. computeRectangleArea(B, H) computes the area of a rectangle
2. computeTriangleArea(B, H) computes the area of a triangle

If I were to perform a usability test, I would:

1. define a use case, e.g., compute the area of a rectangle
2. define a goal for such use case, e.g.,compute the area of the rectangle R having height=8 and base=3
3. define the set of actions needed to perform the goal (which in this case is just calling computeRectangleArea(3,8))

In this case, if the user guesses the right sequence of actions to be done (which is just one), then the usability test succeeds, otherwise I have increased the failure rate.

---

My doubts

I see the possibility of making the user perform actually a usability test with this library, but I am worried that:

• use cases could become very complicated, requiring a lot of steps, a lot of commands to be written and launched
• such complicated use cases may be difficult to be validated, because I am worried that users may not guess which is the right sequence of actions and commands, and thus this may increase a lot the failure rate

This said, I do not know how to make my users evaluate such software library. The library was written according to a set of user requirements, and I see that I need them to test it and see if it covers the user requirements they specified at the beginning, if they are happy with it, if it needs refinements and so on. But I am worried a usability test would not be feasible.

---

Suggestions on how to perform this end-user evaluation?

Thanks.

Answer 1

The users of the application are the end-users, but for a library the users are developers. So test with them. To developers this is already known as unit test.

Answer 2

First off, it's great that you're attempting to do some usability evaluations on any application, let alone a console application.

I think you should avoid thinking of these as "use cases" so you don't end up in trapped in a developer's mindset. You're trying to create some test scenarios for a usability test.

Let me address your doubts:

• "use cases could become very complicated": Yes, in more complex applications the scenarios you're testing could be very complex, but software that does anything meaningful will have a certain level of complexity. Airplane cockpits undergo human factors research and those things are wicked complicated.
• "I am worried that users may not guess which is the right sequence of actions and commands": This is effectively the reason for doing a usability evaluation, right?

So, to me, your doubts only show that you're headed in the right direction, you just need to be cautious about how to proceed. I would start with figuring out some goals, measures, and metrics. These can help inform your test scenarios.

Determine if you want to focus on performance (how long it takes users to do something), perceived performance (how long they felt it took them to do it), satisfaction, learnability, or some other factor. There are also "tools" that already exist for getting a general "score" for the usability of a piece of software, such as SUS, but you may need something more specific. I would try to get multiple types of data--quantitative and qualitative--to have more confidence in your results.

A goal focused on learnability might be "users rarely have to check documentation after 10 minutes of use". The measure would be "number of documentation checks" and the metric for that might be as strict as "count of man <command> uses" if your application has a manual page. Then you can construct a test scenario that addresses that goal.

A goal focused on perceived performance might be "users can find areas of shapes in a reasonable time". The measure would be answers to question given after a task: "Did you feel this task took longer than expected or a reasonable amount of time?" The metric is the count of each answer. Caution: These questions can be kind of tricky to get right and users may be biased to say everything is OK when it's not. It's usually better to observe actions than to listen to what users says.

One last thing that seems minor but is important: in your task scenarios, don't reuse language that helps users complete the task. In the example use case you gave, "compute the area of a rectangle" is part of the command computeRectangleArea. The use case would give the user 3 of the words needed to complete the task. Try something like "given the height and width of this shape, find its area". It still has the word "area" in it and increases the cognitive overhead for users to figure out whether the shape is a rectangle or not, but that seems like a more real-world scenario and avoids giving away the entire command.