0

I intend to conduct a usability study on a website (say an elementary school) to find out about ease of use, learnability etc., the first group of prospective users I would approach are parents for taking a usability survey, as they represent the broader population (target users).

Do you believe it is practical to gather data from a different audience, say college students, who may not be using the website at all, but at least have an insight about features and content of a regular Web 2.0 website, and what is expected when a user would access the elementary school's site?

0

Yes, if the cost / difficulty of gathering data from college students is low and easy but needs to be kept in check if you are still going to test the actual target audience. It also depends on how many rounds of testing you plan on doing and time management. Project creep be lurking unless user testing is done in parallel?

Yes, why not so long as this is not so far from normal procedure with whomever you work [if any] that the potential fallout might later lead to consequences not worth taking such a risk. There is a lot to be gained from testing non-stakeholders. It is actually economically in your interest to test the group that provides the most feedback from the least resources. In fact the target audience could never end up being tested; which while extreme is not necessarily bad. Hallway testing between expensive programmers / QA will still almost always being cheaper / productive than user testing [unless you can deploy code and A/B test easily like Google to millions].

The question is really how similar are test audience XYZ and target audience ABC? And, thus how much valuable feedback will XYZ be able to provide at what cost?

Pros

  • Diversity in testing audience(s) will find many more unexpected usability problems in the rare 10% of use case scenarios.
  • Two user group tests of 3-5 people is fundamentally better than a user test made up of 20 users, and may even cost less but do not rely on that.
  • Multiple rounds of testing is magnitudes better than one round. It allows you to test your testing methods, is a group of related questions or a specific question falling flat and not creating constructive feedback? It also allows production to fix bugs which will render a smoothing user testing experience and get to the nitty gritty but also test whether the first groups feedback was actually constructive / implementable.

Cons: Dun dun. Dun....

  • Feedback from tech savvier college students may miss more common issues [90% of use case scenarios] that less experienced may be tripped up by. Hopefully you are good at putting at different hats on and empathize well with technophobes well enough that these are not the problems primarily that user testing is being intended to catch.
  • Stakeholder motivation, a group of parents on the other side of the world will probably have better aligned interests than a group of college students. Unless you do not randomly sample and actually test a subset, ie parents at college but not necessarily parents at said elementary school.
  • Thanks for the elaborated response. I found this number 3-5 for user testing on Nielsen's website as well. However, how can we prove if the result is statistically valid with 3-5 participants? Is it because we do not want to generalize our results to that type of website issues? Please clarify. – user39531 Jan 12 '16 at 2:42
  • 1
    It's rarely the goal of a usability study to be statistically valid. If you do want to be statistically valid, Nielsen covers this topic as well. It would be an immense waste of time and resources to collect statistically valid results for a group of non-users of your site. nngroup.com/articles/quantitative-studies-how-many-users – nadyne Jan 12 '16 at 5:38
  • I believe usability testing and usability study each has a different goal. If you are conducting a usability study, say for a journal publication, then it is essential to recruit enough participants >30 to test for statistical sig. This is reflected in the research papers from JUS. On the other hand, usability testing with 5 people is a workable solution. – user39531 Jan 21 '16 at 18:11
0

I'm fond of applying Steve Krug's maxim "Recruit loosely and grade on a curve".

If you can't find your ideal users feel free to recruit other folk, but apply more weight to the feedback of people who are closer to your target group.

0

Both the above answers are great. I will add that when possible I add extreme edge cases to my test panels. A trick I learned by accident.

I was testing a mobile banking app for business people. I usually order panels from an agency that recruits them for us. The first bunch was mistakenly switched with a panel of non-business people for another project. Anyway - when they arrived we decided to just go with the test anyway. We got so many great questions and insights from the "wrong group" that we improved the test for the right one. In this case it was especially important because our business people were very busy and hard to recruit - so it actually paid off optimizing the test itself this way.

It can be hard convincing stakeholders to fund or make time for non-core profiles but it is worth it.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.