I have an intern who helps me code data and metrics I'm measuring from Qualitative Usability Testing sessions. However, we have a deadline and she has been out of the office more than I'd like or expected her to be (another story). Would it invalidate the testing if I jump in and code some myself? Would we both have to code all of them, or can I code half and she codes half? I feel like that's a variable I shouldn't add to the results but project scope and deadline are of importance!
1
-
It shouldn't matter who/how many people create the results as long as they aren't bias.– sclarkeMar 16, 2018 at 15:41
Add a comment
|
2 Answers
I'm guessing this is moot since it's a month later but the approach that jumped to my mind would be to code some of the results she's already done too, and then see to what degree they agree with each other. That'd give you an indication of whether you're both on a similar track, and it's not specific to this context.
Specific to this context though, and more formally, what you're looking for is "Inter-Rater Reliability", of which there are many papers for calculating such:
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3402032/
IRR analysis aims to determine how much of the variance in the observed scores is due to variance in the true scores after the variance due to measurement error between coders has been removed (Novick, 1966)
...
For example, an IRR estimate of 0.80 would indicate that 80% of the observed variance is due to true score variance or similarity in ratings between coders, and 20% is due to error variance or differences in ratings between coders.
Without knowing more details I assume you and your intern share an approach and protocol for your analysis and review. Are you both working from the same definition of what does or doesn't qualify as a theme or pattern, etc?
Qualitative data analysis is inherently subjective. As long as you share an objective approach to analysis, it shouldn't matter too much if you have more than one person contributing.
