What is the most effective way to solicit and manage open source project UI feedback?

Question

I'm planning to release a project which will integrate with several popular content management systems as a plugin and operate as a stand-alone service - beyond providing users with a simple contact form which posts to a ticketing system, what are the most effective ways you have found to solicit useful feedback from within the interface and manage user feedback?

Edit:

@Nir & @Rahul - The scope of user feedback is obviously not going to be limited to UI suggestions (or bug reports, or ...) but I'd like to know whether there are any good ways to ferret out commonly-held problems with the application's UI, localization, et cetera. (Note that commonly-held problems might only be common for a limited subset of users - and, unfortunately, I don't foresee having the option to sit down and test every UI localization with a native speaker, for example)

I'm planning a "bug report/suggestion" modal which is accessible from any page and reports the page it was called from and I want to get as much information on the problem as possible. I'd expect that most users would need a nudge in the right direction to get usable feedback without any back-and-forth.

It's a two-part question, maybe it reads better as:

How do I encourage remote users of a web application to offer suggestions above and beyond "It doesn't work" directly through the application, rather than in discussion on a forum, plugin comments page, et cetera?

What is the best way to manage feedback aggregated from multiple sources? (when that inevitably occurs despite all best efforts to prevent it)

Edit w/Answer Accepted:

Thank you, Rahul, Nir, and indolering - I think I have a few of the missing pieces to the puzzle of comprehensive testing.

Given the recommendations I've received and my previous plans, my present strategy looks something like this:

1. Solicit feedback from within the UI with a simple contact/suggestion/bug report form which posts to a ticketing system
2. Provide a forum for users to post suggestions and get support from eachother (I'll probably start out spending a lot of time there myself)
3. Post a working demo version of the application, integrate click-tracking tools to watch how users interact with the interface
4. As the project matures, offer users incentives of some form to participate in beta testing and offer feedback
5. Consider third-party feedback collection and organization tools like UserVoice, GetSatisfaction, et cetera as appropriate to demand (I'm thinking that the choice of solution will depend a lot upon the type of userbase the project draws - given that I'm planning to see shared hosting users install my application, it might make sense to use a third-party service versus rolling my own... but I'll wait to solve that problem until I get to it)

Answer 1

Feedback

We use UserVoice for feedback. Users get 10 votes each and can create or vote on something (an idea, bug report, critique, etc) and assign up to 3 votes to each item. Once they're out of 10 votes they can't do anything until we close or complete an item they voted on, at which point their points for that item are refunded. It works great as a way to collect general ideas and feedback from people while allowing users to vote on each other's ideas. The UI could use some work but it's a nice service, especially since the free version is more than enough.

Customer support

GetSatisfaction works great as a customer support forum where you can get feedback but also allow users to help each other. It's great when you want to take a step beyond UserVoice and let the community evolve a little bit. People can offer feedback categorised by bugs, ideas and compliments. They can also vote up each other's items and indicate how they feel about it using a smiley face. It's a nice way to get an immediate impression per item of how people feel: if there are 22 sad faces, you're probably more motivated to take on this problem than a bunch of indifferent ones. The UI is pretty nice, although a little noisy and distracting when compared to something like UserVoice.

Ticketing email integration

Beyond that, consider allowing people to email you. Most ticketing systems will allow you to set up forwarding email addresses. The nice thing about email integration with ticketing is that you can keep the entire conversation with the user in email by replying back and forth, which brings the convenience and familiarity with UI to the user instead of expecting them to learn a potentially confusing ticketing system UI. Fogbugz does a good job of this and I've heard great things about Lighthouse.

Why not just use a forum or contact form?

Today's Web 2.0 era has a lot of solutions that are specific to certain situations. It makes sense to try and take advantage of a user experience designed exclusively for tackling the problem of "I built something and I want to collect feedback". You can always build your own solution, but you probably won't be able to spend enough time on it for it to be as good as products like the ones I mentioned above. Companies like 37signals, who consider customer support a major priority, have built their own superior solutions (for instance, Basecamp Answers).

Open source alternatives

My experience is with the above solutions, but I've seen mention of some open source alternatives for gathering feedback:

• IdeaTorrent - this looks like an interesting cross between a forum and something like UserVoice. Development of the new version has stopped, however
• OpenMind - a more enterprisey-feeling feedback app

Answer 2

For getting UI feedback do "hallway usability testing" - just grab a random victim, hm, I mean person that was unlucky enough to pass nearby (friend, family, coworker, delivery man whatever, just not another developer working on the same system) have him/her sit in front of your computer and complete some random task with the app while you are watching.

You can also use screen sharing software (Skype can do that, or you can use something like TeamViewer or GoToMeeting) to watch real users remotely when they are using the system or use a service like UserTesting.com

You need a surprisingly low number of such tests to find the problem that really matter (that are preventing users from completing the task or annoy users).

Also, A/B testing is a really good tool to use, always have tests running, if you don't know what to test just swap out random pieces of the UI and replace them with something completely different - if the original UI doesn't outperforms the replacement it's not optimal (or it just doesn't matter, not every piece of UI is critical after all).

From my experience selling commercial software (so my customers probably expect more support than yours) asking for feedback just doesn't work, at least for usability, once you fix the obvious problems that you already know about - especially if you use the system yourself - all UI-related feedback just stops.

For managing the problems you find any ticketing system will do - but it's important to prioritize the problems by severity and solve problems in order of priority, otherwise you are likely to implement the easy fun solutions to the small problems first.

Answer 3

Google analytics and clicktale. Everytime I open up clicktale and watch user videos I find new problems.

The heat and scrollmaps alone are invaluable.

Of course, nothing beats real usability testing!

Answer 4

This question also addresses a larger issue that FOSS projects, and all corporations face: how to weed out bad suggestions and ill-informed UI opinions from truly useful ones. Design documentation and previous Q&A you can point to (like idea torrent) is your first line of defense. Bill Fulton works on this a lot, I touched on it in this blog post (he doesn't publish much)