What methods do you use to test the usability of an API?

Question

Testing the usability of a programming interface poses a lot of challenges. It has a very different feel than testing a graphical interaction. The user can do almost anything at any time, as opposed to a graphical interface presenting a set of salient options.

What methods do you use to test APIs? What kinds of results have you had?

Answer 1

Alex's question was how to test an API, not what makes a good API. Post-deployment monitoring seems critical to me. It's very hard, if not impossible, to know in advance what uses developers are going to want to put your API to once it goes live. Logging good requests tells you which features are popular, but logging bad requests and user support questions might give you more of an insight into aspects of the API that need to be improved. If you have the resources (and it's appropriate for your domain), generating a dashboard view of your service/API from the early days would allow you to track and prioritize improvements.

Pre-deployment, I think the main things you can try to measure are developer satisfaction and the "elegance" of code that's written against the API. Both very subjective things to test. Having a good set of primary use cases might help I suppose.

Answer 2

In 2006, I was trying to choose a JavaScript framework. I tried Prototype, Dojo, Behavior, MooTools and jQuery. I choose jQuery for one reason: The API made sense to me and was easy. The jQuery team and John Resig in particular has obsessed about the ease of the API. They understood the universal truth about APIs: Easy is

1. compared to what? and
2. easy for whom?

Know thy audience. What are they used to? In jQuery the audience were people who knew CSS, so the syntax is very CSS-like. Here are some general tips:

• Shorter is better. Short words, short syntax. TinyUrls. Keep it simple and short.
• Use sensible language. Don't call something a "blah" if 50% of the time it's a "not-blah".
• Use consistent grammar. Don't say "Update" in place and "Updates" in another. Always use the same tense and the same grammatical conventions.
• A good API browser is essential. Check this one out from ExtJS. Super useful. jQuery API too.
• Have a sense of humor. It might sound irrelevant, but engineers are people too and spoonful of sugar always helps the medicine go down. I created an a javascript file called Munchkin.js and the api calls are called Munchkin Functions. Engineers smiled and many interactions were smoothed over.
• Think it through now, before someone starts using it. Once it's out there, you can't put the genie back in the bottle.

I hope these are a bit helpful.

Answer 3

An API is only as good as its documentation.

You could test an API by showing a developer the documentation and then asking them to explain how parts of it work.

Answer 4

Do usability testing. It probably is somewhat time consuming in this instance, but it has been done. Check out the paper Usability Implications of Requiring Parameters in Objects' Constructors by Stylos and Clarke for example.

As I haven't done usability testing on any API, I would go about it mimicking the article cited. Basically this would mean coming up with representative programming tasks for participants to do via the API.