How often is too often to do focus groups while designing a web app?

Question

I'm trying to come up with a strategy for putting together focus groups while designing a web application, but I'm starting to wonder how often is too often to do focus groups? In my mind, they would be beneficial after just about every step of the design process.

Here is my process so far:

• Step 1.) Identify client's goals, abilities & project scope
• Step 2.) Focus groups with target market to discover useful features and ideal layout
• Step 3.) Compose wireframes
• Step 4.) Create static mockups (potentially with filler content and minor animations)
• Step 5.) Focus group for usability and visual style
• Step 6.) Seemingly endless development process...
• Step 7.) Focus groups with Alpha version
• Step 8.) Further development, bug fixes, probably more usability features
• Step 9.) Beta Release
• Step 10.) In-app user feedback if possible and/or more focus groups
• Step 11.) Step 8 all over again
• Step 11.) Launch

So what do you think of this process? Is any particular step unnecessary or missing? Too many focus groups?

Answer 1

Wow - this looks really nice. You've done your homework well, I can tell you that. However there is only one step I think you need to think through more. And it's this one:

Step 6.) Seemingly endless development process...

Depending on resources and scope, this step can be all from one week to 1½ years. And letting your focus group wait 1½ years doesn't do the trick. Try to break it down in 3 week sprints and let your focus group see where you are, and correct along the way.

If not you might end up with a project starting in 2008 for a Swedish bank, who didn't care about mobile banking, and five years later was scrubbed. Over 100 million USD out the window. Use Google Chrome to translate from Swedish.

Otherwise you've done a super job!

Answer 2

I'd say that it all depends on what you mean by focus group.

If you're going to sit down with your users and ask them what they think, and what they want, I would not give this too much of your energy. If users tell you it's wrong they are almost always right, but if users tell you what's wrong and what should change, they are almost always wrong (to paraphrase Neil Gaiman). You can use these to generate ideas at the start of a cycle, and as a sanity test at the end, but don't tie yourself down to the result in the project planning.

If you're going to observe your users and do formative evaluation, I'd plan as many sessions as you can get away with (but not with groups, with individuals). Every week during the wireframing stage is not a luxury. Basically, if dropping one focus group session with five people gets you five individual user tests, I'd drop all of them.

In your plan, step 5

Focus group for usability and visual style

sounds the most worrying to me. Usability and design should not be evaluated by focus group. You can get good feedback from users, but not usually by asking them what they would change, and not by putting them in a group. There's a real danger of devolving into design by committee here.

Answer 3

It depends what you mean by "focus group". If you mean just asking a group for general feedback on designs our concepts, then no, I don't think that's terribly useful to do repeatedly.

Focus groups are good for high level research at the start of a project when you're investigating what problems exist and how and why users are generally performing their tasks. They can also be used to validate personas if one on one interviews weren't possible during persona generation.

What they are not so suitable for is optimizing interfaces - the participants are not using the application, so feedback is at best hampered by lack of weight (we don't know which problems our users mentioned actually stop then completing tasks), at worst totally irrelevant (users start discussing things like graphic design).

Use moderated user testing (where users each individually - not as a group - attempt a particular task on a prototype) to optimize your interfaces instead. Visual design can be tested through "five second tests" with twenty our so individuals (possibly remotely), where participants are briefly exposed to a mockup and then asked questions about what they think the organization does and the alerts of values they think the organization possesses. Visual design at interface level is validated through the aforementioned user tests, not by asking users their opinions but actually observing whether visual metaphors, iconography choices and typography support our hinder the successful completion of tasks.

Finally, I would recommend budgeting for analytics / behavioural tracking features and reserving at least one post release iteration for improvements (our, even better, a continual improvement process if the sort Benny suggests above).