I want to know how will I set a cut-off scores in a Likert scale. I will be using 3 points system: agree, undecided, and disagree. I've been googling and read that I need to find the mean, variance and sd. But could not find what's next to do with them. Thanks!
closed as too broad by Jason A., Vitaly Mijiritsky, Bart Gijssens, JohnGB♦ Jul 6 '15 at 15:20
Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.
It sounds like you have two problems. One is getting a handle on what your questionnaire scores really mean (how good is 0.6?) and the other is how to assign cut-offs to scores to consistently decide on a course of action (at what score should I decide the design is ready for production?).
For the sake of an example, I’ll assume the questionnaire measures user satisfaction with a web site, and you want to determine if a web site is “acceptable” or not. Let’s say it has five items (each with your 3-point scale scored as 1=agree, 0=neutral,- 1= disagree) and we’ll average the items to get the overall scores.
Putting meaning to scores
Likert questionnaires have essentially arbitrary score values, so to really get a sense how good a number is, you need relate it to something. Here’re some options:
Study the semantic meaning of your Likert items and, for a given score, ask yourself what the equivalent proportion of “agrees” and “disagrees” mean for the web site. Say you have an average score of -0.2. That’s equivalent to your users agreeing with two items and disagreeing with three. If every item semantically represents a just sufficiently good opinion of the website (e.g., “I’d be willing to use this web site again if I had the need for this kind of information”), then disagreeing with most items on average suggest pretty poor performance. Good performance would be some kind of positive number. If, on the other hand, every item represents an outstanding opinion (e.g., “This is the most awesome site in the universe”), then -0.2 isn’t too bad –up to two “outstandings” out of five is actually pretty good.
Have a bunch of users use the questionnaire to rate a large random sample of web sites and use these data to compare where a given score lies relative to the others. For example, if you have ratings for 200 sites (not an unreasonable number), and you find that a -0.2 score is greater than the scores for 62 sites in your sample, then -0.2 corresponds to the 62/200 * 100 = 31th percentile. Pretty bad: over two-thirds of the sites out there are better. And have you seen what the average web site is like?
You mention mean and standard deviation. You don’t necessarily need the mean and standard deviation of the sample of sites to use norms. Just look where a score lies among the others. The mean and standard deviation can be used to calculate percentiles from standard scores. The standard score (z) is
z = (L – M)/ S
Where L is the average questionnaire score of one web site (from the same or different sample of users), M is the mean questionnaire score of the sample of sites, and S is the standard deviation of the sample of sites’ questionnaire scores (you don’t need the variance). With a standard score you can estimate the percentile using a normal distribution. The easiest way to do this is to plug z into Excel’s NORMSDIST() function.
Using standard scores is only useful when you don’t have a large sample of web sites so that simply looking where a score falls in the sample is too granular (e.g., if you have only a sample of 10, then it can only be precise to 10 percentile points). However, using standard scores assumes your scores are normally distributed. Maybe they’re not. And even if they are, you’re not going to get a very accurate estimate of the average web site out there from a small sample size, so don’t even bother.
Have users rate a random bunch of sites with the questionnaire and also take other measures of the sites’ usability (e.g., time to completion, number of user errors, conversion or not, number of correct answers to questions that require the user find the right information on the web site). Correlate the questionnaire scores with the other measures. For every measure with a high correlation (I’d say a Pearson correlation coefficient of at least 0.7, but you could go as low as 0.3), perform a linear regression of the questionnaire score on the other measure. The resulting equation will tell you the corresponding performance for any entered questionnaire score. For example, you could say that “users on average make 3.3 errors on a site that has a score of -0.2,” which may be more meaningful than the score by itself.
Generally this only works well for a bunch of sites with similar function and task (e.g., they’re all airline sites and the user is trying to book a flight). Otherwise, the other measures will be all over the place just because the tasks are so different, which lowers the correlations. Actually the correlations may be low anyway just because subjective satisfaction measured with the questionnaire is a different aspect of usability than things like efficiency or error prevention.
Obviously, this only makes sense if it’s easier for you to use the questionnaire for a site in question than for you to measure performance on the other measures directly.
Comparison to Other Site
Pick a site whose performance you want to match or exceed with your site in question, maybe a “standard” (e.g., Amazon), maybe a competitor, maybe the current site in production, maybe an earlier design iteration. Have a bunch of users rate both your site in question and the comparison site. Count how many users rate the new site better. Now you can say things like, “on average, about two out of three users rated the new site better.”
You can also subtract the score for the comparison site from your site for each user, calculate the mean (D) and standard deviation (s) of the difference scores for your sample of users, then calculate the standard score as:
z = D/s
Plug z in NORMSDIST() and you get the estimated percentage of users doing better on the site in question under the assumption that the difference scores have a normal distribution.
(Side note on inferential statistics)
For all the methods above, you’re getting questionnaire score on a site in question using a sample of users –probably a pretty small sample, so your score could easily be off by a certain range due to sampling error. To determine what that range reasonably could be, calculate the 95% or 90% confidence interval (however confident you want to be) to get an upper and lower score, and then perform your selected method above on each score to get a range of results (e.g., “between 16th and 45th percentile”). The confidence interval is:
Lower = L – t * s / sqrt(n)
Upper = L + t * s / sqrt(n)
Where L is the average questionnaire score in the sample, s is the standard deviation of the questionnaire scores in the sample, and n is the number of users in the sample. The t is the t-statistic, which you can get from Excel using the TINV() function.
TINV(0.05,n-1) = t for the 95% confidence interval
TINV(0.10,n-1) = t for the 90% confidence interval
A Likert scale instrument produces interval-scale numeric values. Assigning a cut-off reduces that range of evenly-space values to two categories. That process is inherently drawing a line through a gray zone, like deciding on a cutoff of when a backpack is “too heavy.” Is it 40, 50, or 60 pounds? And if you say “50 pounds,” is 51 really totally right out, while 49 is perfectly okay? There’s an element of arbitrariness to it, and no algorithm to arrive at a single right answer. It basically comes to making a reasonable judgment. How much better does a site have to be than the norms or the current design or the competition? How many errors or conversions do you want to shoot for? Is average good enough? Or do you want to shoot for truly superior –say the 75 percentile? Do you have the resources to get there?
And maybe you don’t need cut-off. Maybe it’s sufficient to use one the methods above and keep it in numeric terms of percentiles or such. Include those numbers with whatever other metrics you have to decide how good a site is.