I created 2 surveys to analyze CES and SUS. My questions were about the software for people who work in call-centers suggesting real estate to potential clients. The software provides information about real eastate objects, its location, price, etc. It's not even a website, it's an old MVP created via loocker studio in google. About 40 people took part in those surveys and almost all of them said the software is perfect. I also wrote the intro saying that the real feedback is important for us because we are going to analyze the usability of the new version of that software comparing to what we have now. I also mentioned that it is an anon survey. What to do about it? Is there any other way to get a real feedback? Anything apart from the problem interview.
and welcome to the site!
Summary: Accept the good survey results, or observe users to gather your own data on the system's fit for the purpose.
not even a website, it's an old MVP So you are surprised that "an old" software is getting good survey results - is that your worry?
Let me first say that old software can actually be perfect. When my company switched from keyboard-and-menu-driven to mouse-driven, it turned out that experienced users suffered a slowdown, and they were asking that the keyboard-and-menu-interaction was to be kept. When we talk about the distinction between UI (user interface) and UX (user experience), we talk about the fact that software which is visually boring might actually be perfectly suited to the processes and goals of their users.
My second remark is that - given your description - I see the most important ingredients of a good survey (40 participants, anonymity announced, real feedback asked). I would only verify that "40 people" refers to "40 users"?
What do you mean by "we are going to analyze the usability of the new version of that software comparing to what we have now"? You are conducting a baseline study with the current system, which you want to compare to results from testing a new version? It sounds like the new version already exists? Why was is developed before feedback about the old version was collected? That misses the opportunity to improve on anything that might come up on the old version.
And that's leading to my only explanation (other than the perfection of the old software): There's something social, human going on - like fear of being pushed to a new system which is not an evolution (stepwise improvement), but a radical new implementation. Users will loose all their expertise, and have to start over learning a new system. Or like being presented with a new system they did not have anything to contribute to, where nobody asked for their input.
If you could imagine this to be the case, there's no other way than to observe the people using the old software. If you see them with post-its all over their screens, or hear them swearing, you'll know there are issues. I'm not sure why you wrote "Anything apart from the problem interview" and whether the same reason rules out observations, though. This statement also leads me to think there's something in the environment going on - are certain UX methods forbidden? Why?
All the best with your research!