User research on 100 people

Question

Has anyone ever conducted user testing on a mobile app or service with a large amount of people? (around 100 people). We plan to give the users the application and have them use it for 1 week. It would be great if anyone could give me some pointers for preparing and conducting the study.

Answer 1

I'd suggest that the main aspects you want to address with such a large group would be;

• Ensure that you know what aspects of the app you are needing to test and that this is communicated to the testers. Since you do have access to such a large group of testers, it might be worth tasking them with testing different aspects of the app (broken down into groups of ~10) to obtain a more useful overview of different areas (Interaction, Process, UI etc.)
• Make it easy for the testers to get their feedback noted, UX field testing is really useful but a lot of information gets lost between the user interacting with the app and getting this information logged. There is a useful collection of methods for field studies here - http://www.allaboutux.org/field-methods

Hope this helps

Answer 2

I have already conducted an usability test with 45 people. We split the participants up into 5 peer groups of 9. Each peer group tested the same software, but in a different configuration (configurations varied in the gui design, in the interaction flow or in the automation level for certain tasks). Thus we were able to compare statistically, which configuration was the best to accomplish a certain task. For comparison, we calculated different metrics for each group:

• average time to accomplish a task in a certain configuration
• task success of a group - how many people managed to accomplish a task?
• average quality of the achieved result in a certain configuration - how good was the result (it depends on the task if a quality metric can be determined)

The more people you have in each peer group, the more reliable the statistics will be.

If there are any design decisions in your app, for which alternatives exist, you can find out the best alternative with the help of a big user test. Find out metrics to measure the convenience of each alternative and put together peer groups to test the different alternatives.