I'm a big fan of About Face 3 by Alan Cooper and similar books about interaction design and how the software must be done correctly.
At the same time, working as a freelancer, I observe daily my customers, i.e. in general people with no technical background, asking to create products which violate every best practice and every rule of UX to finally obtain a mess I would never use myself. They are happy with it, and some will be angry and lost if they were forced to use a product designed according to the UX rules, scientific research¹, etc.
The difference between, on the one hand the rules and guidelines advocated by extremely competent and smart people, and on the other hand the users which often reject the products which follow those rules and guidelines, surprises me.
I imagine that this difference is too complicated to be explained in a short answer. Therefore, do you know any books or research papers on the subject, helping me to understand both why there is a such difference, and how to make products which will at the same time follow the best practices and not be rejected by people with no technical background?
¹ An example is a ribbon implemented in more and more Microsoft products. While created based on studies about the usage of software, productivity, etc., ribbon is perceived extremely badly by many users.