I'm about to try to test the usability of our developer tools, how can this be done?

Articles I've read so far on UX testing seem to target website testing. I'm interested in reviewing the user experience of a developer using our language and developing an end application with out language.

Any specifics on how I might go about this?

  • If you have read only how to do website testing, then your knowledge seems to come from some patchy examples, maybe blog articles each of which tackles some small detail? As there is nothing special about testing developer tools when compared to other software tools, it seems that you need all the information needed for creating a standard usability study starting from zero. This means that you will probably have to read at least one textbook; this is way too much information to fit into a SE answer.
    – Rumi P.
    Jan 9, 2014 at 15:28
  • 2
    I would recommend Measure the user experience by Tullis and Albert for a solid treaty of how to make a quantitative usability study. It presumes that you already have user tasks defined from the requirements phase. If you don't, you will need another textbook on basic task analysis first.
    – Rumi P.
    Jan 9, 2014 at 15:29
  • 1
    You could try searching "heuristic evaluation". Agree with Rumi that this question requires too detailed an explanation to be answered here. Jan 9, 2014 at 15:41
  • Can you tell something more about your own specific language? What type is it? We have some experience developing our own specification language for the last 10 years, but the approach varies per type of language. Perl is not really suited for graphical design, PowerCenter is. Jan 9, 2014 at 17:41
  • a new upcoming keyword is "developer experience" which includes UI-free tools as well, I think general assumptions here come from workplace ergonomy in abstract sense; lean inspired waste reduction can be explored as well.
    – J. Doe
    Nov 15, 2017 at 16:56

1 Answer 1


The basic process would be:

  1. Decide what you want to know after the study. "Evaluate usability" is not enough, as the concept of usability is too complex. It has many aspects, and you can't evaluate them all with a single study. For example, achieving the research goal create a list of issues which have to be fixed in the next design iteration requires a very different study type from the goal I want to find out if my product is better suited for beginner developers than my competitor's.

  2. Evaluate the resources you have to conduct your study. The limiting resources will be usually the UX expert's worktime, the calendar time until a result has to be delivered, and the budget available for recruiting participants. You can save some UX expert time if you also allocate a budget for specialized tools, but it is not certain that there will be tools available for the type of study you want to conduct. You don't have to pay money for a statistical tool. Tullis and Albert suggest Excel, and it is enough if you are aware of its gotchas.

  3. Design the study. Decide how many participants you need to collect the metrics you want, or, if very limited by recruiting budget, see how many participants you can get and choose metrics which are reasonably good for answering your research questions but can be measured with the number of participants available.

  4. Organize the study. Prepare a workplace for the participants (if not doing unmoderated study from home - then prepare a way for them to have remote access to your system), prepare task descriptions, prepare data needed for completing the task, prepare questionnaires if needed (electronic ones are much more convenient, but a survey tool also costs money). Recruit participants, make appointments.

  5. Conduct the study itself.

  6. Evaluate the data. You will need statistics knowledge for it, else the results you get will look good but be actually totally meaningless (e.g. if you don't know statistics, you might decide to calculate an average value from the answers on a Likert scale). You don't have to be a wizard in statistics, but you have to know what types of data there are, what are basic descriptive statistics, what are confidence tests, and what is linear regression. Multivariate regression is more ambitious, but gives answers to questions which your managers will probably find most interesting. We are talking about maybe 4 to 8 credits worth of entry level statistics here. Of course, if you only have decided on a very specific study design, you can concentrate on the stuff you need for evaluating this one study only.

  7. Make a nice presentation out of your results. Whatever your statistics tools spews out as raw data, it is normally not suited for human consumption. You have to document how you conducted the study, what your results are, what conclusions can be made from them, and what conclusions cannot even if they seem logical.

There are books on how to do this all. "Measuring the user experience" by Tullis and Albert is one possibility which is aimed at practitioners, not academics, and it covers all of the steps above. (There are whole books written on each of the steps too, but this probably goes too deep for you).

For step 3, you will probably need task descriptions around which you will structure your study. They are supposed to be created in a project independently of any studies. If you don't have any, you will have to create them first, there are books about this too. For some types of research goal, it is also possible to do a study without tasks, but 1) this may not be applicable to your case, 2) you can't well evaluate the "test coverage" of this type of study, in the worst case you will think that everything is OK with your software only because your participants didn't think of trying out some tasks which your software supports badly and 3) you will need an alternative classification for structuring the results, and most non-task based ones (e.g. "they have trouble with items from the Edit menu and items from the Tools menu) do not let you recognize the connection between usability issues.

The above is not specific to developer tools, you can use it with any type of software. Discussing specific details connected to the fact that it is a developer tool is practically impossible at this level of abstraction.

Update in case the above seems overwhelming

If you have done any psychometry, econometry or similar in the past, the skills are very transferrable. If you find out that you need an intern to help you (it is lots of work), try to get one with experience in these fields, it is more likely to find students who have had courses in these than ones who have been involved in software usability.

Maybe you are now asking yourself if you need to go through all of this if what you want is a simple study. The problem is: there are simple studies, and they can indeed be done with little effort. But you need the background knowledge to know if what you did was right. Empirical studies of any kind have this unpleasant quality: results from bad studies look just as good as those from good studies, and you need somebody with enough knowledge to look at the procedure and see if what you found out is indeed good insights, or garbage.

It could be an easier way to emulate something other people already have done, this way you know that your design is right, which is the biggest problem for a beginner. Look around on Google scholar for papers about evaluating developer tools, read the methods section, and see if you want to find out about your tool whatever the authors found out about theirs. If yes, do it the same way as they did.

  • Thanks Rumi, slightly overwhelming for a developer who wants a clean and quick solution, I may have to spend more time at this than anticipated. I'll see can I get my hands some snippets of the book you've mentioned. Thank you All. Jan 10, 2014 at 15:24
  • question of budget - strategic lean tranformation projects would sposnor and assign enough time to procceed like that.
    – J. Doe
    Nov 15, 2017 at 17:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.