I created 2 surveys to analyze CES and SUS. My questions were about the software for people who work in call-centers suggesting real estate to potential clients. The software provides information about real eastate objects, its location, price, etc. It's not even a website, it's an old MVP created via loocker studio in google. About 40 people took part in those surveys and almost all of them said the software is perfect. I also wrote the intro saying that the real feedback is important for us because we are going to analyze the usability of the new version of that software comparing to what we have now. I also mentioned that it is an anon survey. What to do about it? Is there any other way to get a real feedback? Anything apart from the problem interview.
1 Answer
and welcome to the site!
Summary: Accept the good survey results, or observe users to gather your own data on the system's fit for the purpose.
not even a website, it's an old MVP So you are surprised that "an old" software is getting good survey results - is that your worry?
Let me first say that old software can actually be perfect. When my company switched from keyboard-and-menu-driven to mouse-driven, it turned out that experienced users suffered a slowdown, and they were asking that the keyboard-and-menu-interaction was to be kept. When we talk about the distinction between UI (user interface) and UX (user experience), we talk about the fact that software which is visually boring might actually be perfectly suited to the processes and goals of their users.
My second remark is that - given your description - I see the most important ingredients of a good survey (40 participants, anonymity announced, real feedback asked). I would only verify that "40 people" refers to "40 users"?
What do you mean by "we are going to analyze the usability of the new version of that software comparing to what we have now"? You are conducting a baseline study with the current system, which you want to compare to results from testing a new version? It sounds like the new version already exists? Why was is developed before feedback about the old version was collected? That misses the opportunity to improve on anything that might come up on the old version.
And that's leading to my only explanation (other than the perfection of the old software): There's something social, human going on - like fear of being pushed to a new system which is not an evolution (stepwise improvement), but a radical new implementation. Users will loose all their expertise, and have to start over learning a new system. Or like being presented with a new system they did not have anything to contribute to, where nobody asked for their input.
If you could imagine this to be the case, there's no other way than to observe the people using the old software. If you see them with post-its all over their screens, or hear them swearing, you'll know there are issues. I'm not sure why you wrote "Anything apart from the problem interview" and whether the same reason rules out observations, though. This statement also leads me to think there's something in the environment going on - are certain UX methods forbidden? Why?
All the best with your research!
-
Thanks a lot for the detailed answer! Yes, we are talking about 40 product users (operators) who took part in the survey. Yes, you are certainly right, and the good survey results can be really honest. The fact is that I have already conducted a problematic interview, operators (respondents) often perceived my questions as a test of their knowledge and gave irrelevant answers. Aug 31 at 8:49
-
My manager said to analyze these metrics through surveys to calculate them now. Then, when the new product is released, we will calculate the same metrics for it and compare the metrics to see if the efficiency of operators has improved in the new software. We conducted interviews with operators before designing a new version of the product and included requests from operators in the interface. But this is still a new product, which is fundamentally different from the old one: we cannot support the old one because the company is going to enter new markets, they need a modern, fast product Aug 31 at 8:49
-
Hi, thanks for the clarification! "perceived questions as a test" - then something has gone wrong in these interviews, either in the explanation of the interview goals or in the personal interaction. I don't know your involvement in this project (or the old product), but as we professionalize our user research, we have special non-project research people for such tasks - to eliminate bias in the moderator, and expectations in the interviewee. Aug 31 at 8:57
-
This might be an academic remark, but the ideal process would be to test a prototype (paper, interactive mock-up, or early implementation) of the new system, and incorporate improvements before releasing it. If you plan to have multiple releases, it'll be good as well, because you may deliver improvements in the next release(s). Aug 31 at 8:58