How do you test the credibility of a community advice site?

Question

I am looking for standardised techniques or question sets that can be used to test the value, credibility or trustworthiness of a community advice site.

In particular, I'm interested in testing whether users interpret the site as:

1. having enough rich content to be worth searching for advice on their topic of interest
2. being specific enough to offer advice that can be acted on, or which will be effective in removing a worry
3. being populated by members who are sufficiently like themselves to offer advice that is relevant to the participant
4. being sufficiently community-led to be seen as independent, and not, for example, as pushing a particular product or approach

Does anyone know of any resources which might help to develop a test of these characteristics?

Note: I have an additional constraint, which is that I need to test designs, and not a live site, so there is a limit to the volume of actual content I can expose participants to. However, I am happy to adapt test questions designed for a live site as needed.

Answer 1

It's a very specific question, and will require a unique solution.

It doesn't appear that you've weighted your 4 criteria, and you've stated that you can't actually deploy the entire solution - so I'd focus on PAD testing with specific questions related to each of your 4 criteria.

Because of the limited amount of data provided, results for factors #2, #3 and #4 will not generalize to the final site. Content, not layout, will drive perceptions of Community (#2), Utility (#3) and Independence (#4).

The perception of "richness" though? That's fairly easy to establish through layout, presentation and marketing.

Answer 2

I am not sure about standardized techniques, but here are some suggestions for each of the questions you want to ask and some caveats. Since you are looking at four separate factors, the weighting that you give to each factor might also vary depending on the type of site, the features it provides or some other confounding factors.

1. Enough rich content - hard to define what enough is, but the metrics you can look at include: average number of answers per question, average rating of answers, average age of question and answers, average rating of users answering questions.

2. Being specific enough - average number of tags per question, average length of question and answer.

3. User similarity - I don't know if you can measure this without getting demographic information from users, but the general age, gender, interests, etc is a starting point.

4. Community-led - Most sites offering advice will have administrators and moderators, and these people will generally lead the community. I think if you look at the number of users that join, and the number that continue to actively participate (e.g. ask and answer questions) it will give you a starting point.