1

If a website has 100 pages and you wish to learn from a survey which are the most important or useful to people, what percentage of the 100 pages should you ask people to choose?

Should you constrain them to five answers or allow them to choose (up to) twenty? Would the need to analyse four or five user segments affect the answer?

What are the statistical reasons?

9
  • 2
    Do you have analytics on the website?
    – Mervin
    Commented Mar 18, 2014 at 16:27
  • Yes, I can see which pages are most popular under the existing navigation scheme. I suspect that improving navigation will help surface content which tends to get buried and which users might not be aware of. Commented Mar 18, 2014 at 16:35
  • 3
    If they are not aware of those pages, then the survey won't provide much insight.
    – DA01
    Commented Mar 18, 2014 at 16:40
  • Possibly... I may need to reframe the question in terms of which information they do (or would) find most useful Commented Mar 18, 2014 at 16:48
  • What's the end goal of this survey? Are you re-designing? Getting ready for a new marketing push? Are you looking to trim back your content? Also - Don't forget about search terms in your analytics. That can provide great insight in to what people need/look for on your site.
    – Mark Bubel
    Commented Mar 18, 2014 at 17:07

4 Answers 4

1

From the phrasing of your question it sounds like you are looking for a quantitative solution ("what percentage of the 100 pages" and "constrain them to five answers"). It is really hard to design surveys properly. You need to ask the right questions to the right people. You need to be aware of self-selection of participants, and how this may impact your site's target demographic. There are lots of issues, and books have been written about this topic.

What I would suggest, as a start, is to not worry about statistics too much. Rather opt for a more qualitative approach, and ask people two simple questions:

  • What is the best part of the site?

  • Name one thing that you think this site is missing at the moment.

The first question will allow people to list things they like, in any order they like. If you aggregate all the results, it should (hopefully) align with your analytics data.

The second question will do three things:

  • Show you what people think they want from your site, but limit them to be very specific. If they could only list one thing, they will most likely list the most important thing to them. This is valuable information.

  • Highlight which portions of the site are actually useful, but there are people who do not know that it exist or where to find it (assuming they name something that already forms part of your site)

  • Provide you with a starting point from which to prioritize your redevelopment efforts.

Maybe people don't have a problem with your site. Maybe they actually want a better mobile version or app (as a hypothetical example). If you ask 100 people, and five say they want a mobile app, is it enough? That is a decision for your management team. But still, you would not have known about this apparent need in a structured "list your top 20 pages" style questionnaire format.

2
  • Thanks, I have some qualitative data from a previous (not website focused) survey and I will include an open question about information and communication needs or problems people have which aren't met by the site. As you say, survey design is not easy and I suspect I need the survey to be as simple as possible to ensure maximum participation. I don't want to overwhelm people with questions, so their top 20 pages from a list of 100 may be too big a request. If I aggregate similar pages and ask for top 5 from 50, will I get statistically significant information? Commented Mar 19, 2014 at 9:57
  • You will first have to decide what you want to measure or estimate, and then design your experiment around a question that can be answered with some degree of statistical significance. Going into more detail about the statistics would place this question beyond the scope of UX.SE.
    – CJF
    Commented Mar 19, 2014 at 19:07
1

Unless your website has a statically insignificant number of visitors, I would prefer the results of tracking your website via Google Analytics over any survey.

Surveys inherently posses a multitude of problems, foremost low completion rate and high dependence of result based on order of questions, question wording etc.

Use real-life tests of your pages with users for a general idea by observing their natural behavior and comments. You can then build upon that basis with using metrics from Google Analytics.

1
  • Sure, I'm using GA. What interests me in addition is finding out more about the needs of the six different user groups. It's possible this may lie in the data from different time periods throughout the year. Commented Sep 18, 2014 at 14:18
0

This article by Leisa Reichelt explains that confidence in decisions is more important than proof and suggests ways of improving understanding:

If you are doing qualitative design research, don’t worry about sample size. Sample size and statistical significance don’t matter.

The only thing that matters is how confident your team is about the next decision they need to make.

0

As the site exists with so many pages, you can also consider incorporating the survey into the site itself.

For example:

At the bottom of every Microsoft Support page, there is a question on the page that says "Was this page helpful?"

You are basically qualifying the value of each page.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.