What's the best way to get reliable user feedback on a desktop application?

Question

Lately my company has been trying to figure out the best way to get reliable feedback from a our users. It seems that we hear things from our power users and mostly from those who have trouble, but never from those who have a somewhat easy time using our software. I would really like to find a solution that will get us a broad spectrum of feedback, without too much effort from the user. What we are really looking for is some confirmation that some things are working as intended, and what things could use some minor tweaking to create an over great experience. The current feedback we receive is far too polarized, and from a very small number of individuals.

What's the best way to get reliable user feedback on a desktop application, so that it is quick and easy for the user?

(This was inspired by Do online-feedback tools generate useful feedback?)

Answer 1

You have to deal with the fact that if everything works well for a user, she will not provide a feedback. Because she don't care. Because she has nothing to say.

There are several approaches, and no seem to work well.

For example, http://digitalwpc.com/ provides a link for a feedback. It is not targeted for people who are experiencing UX problems (because the form you have to fill is not intended to provide you help of further support), but I'm pretty sure that it's only used by people who actually have UX issues with the website. Personally, I used it to tell them that their keynotes cannot be watched full-screen on dual monitor configuration; if everything was fine, I would never use their feedback to actually tell them that their website rocks.

Another example is a more intrusive feedback. For example having a Logitech keyboard and mouse, I were asked several times for a feedback. Again, I used it to highlight some user experience flaws with their keyboard/mouse combo.

This being said, both times I also provided positive feedback. In the case of WPC, I selected the highest rank for content, because I really appreciated it. In the case of Logitech, I also selected highest marks for the points I appreciated the most, and even added a few points about what I enjoy about the keyboard.

It means that:

• You get a feedback mostly from the people who are experiencing some issues with your product,
• But this feedback doesn't have to be only negative if you invite the users to broaden their feedback.

If you provide just a text field asking people for their feedback, they will talk about their concern, and their concern only. If, on the other hand, you invite to talk about the general experience, to evaluate precise points, the same users may provide additional, positive info.

For example, if your desktop application is horribly slow, but with a nice design, easy to use and works on every device including smartphones, if you ask in the feedback to evaluate the performance, the design, the ease of use and the interoperability, you'll see some very low results for the first point and lots of negative comments about how slow your app is, but you'll also see more high ratings for the other three points.

Answer 2

Do some in-person testing!

Others have mentioned some good ways about gather feedback in aggregate and remotely, all of which is useful. However, I've always found there is no substitute for watching some real people really use the software for a significant period of time.

You won't necessarily be able to get feedback from a great many users, but getting good feedback from several representative users can be hugely productive. I've run private betas, usability testing on finished products, and even feedback sessions on prototypes, and every time I learn a ton about my users and what actually do with the product.

In my case, this is how I went about it:

1. Decide on what kinds of users I'd like to talk to, if you have personas written up that gives you a good starting place.
2. Go find the users that match your list- if you want a first time experience ask a friend, if you want a power user talk to sales or account management and see if they can put you in touch with someone, and if you have active users you already know- include them.
3. Figure out what you'd like to see them do- maybe outline some basic tasks and watch them. Don't interfere and just take some notes and ask them to talk out loud.
4. Once they're done, go through a few prepared questions you're particularly interested in.
5. Thank them for their time, and depending on who they are and your budget try to offer some kind of token like a coffee gift card or the like.

Don't Make Me Think by Steve Krug has a great section on guerrilla usability testing that covers a lot of this and is definitely worth a read.

Answer 3

I would go with Jacob Neilsen's You Only Need to Test with 5 Users approach.

Answer 4

Getting feedback on desktop apps is definitely a challenge compared to web apps. There is no single solution for collecting application feedback that I am aware of. Therefore, I recommend a combination of the following:

1. Add a feedback link or menu item somewhere in the app so that it is easy for users to provide their input when they are having a problem. This link or action can also be included in error messages.

2. Pop up a poll after the 100th session (or some similar number) for each user to collect general feedback and answers to any specific usage questions you may have. If you absolutely want to get people to answer, offer an amazon gift card award for a random selection of respondents.

3. Do intensive user feedback sessions (in person or remote live interviews) with a small group of 5-10 users. This will supplement the quantitative poll responses and feedback with in depth qualitative input.

3.

Answer 5

I have been working on UserMetrix, a feedback tool for desktop and mobile applications. At the moment it does a great job of automatically capturing crashes, and figuring out the context of use that led to these problems (i.e. People click open, then select text, press bold and finally right align before it crashes).

We are in the process of also adding the ability to capture freeform user feedback, in a similar fashion to existing online-feedback tools. The difference, is that we are clustering feedback around the context of use. This way, you can drill into the details about a particular feature, see what people are saying and how this feature is being used in a wider context. I.e. This is what people are saying about the "right align" feature, and people mostly "select text", and "press bold" before selecting "right align".