Should you bother with popups? Do users read them? How can you get them to read them? Should you try to design your application so it doesn't need or use pop up messages/notifications?
This applies to all pop up messages, not just error messages.
You can't rely on anyone reading them, and if you can do without them, you should.
Recommended read: Designing for People Who Have Better Things To Do With Their Lives by Joel Spolsky
Users don’t read messages boxes and it’s our own dang fault for over-using and misusing them. Too often message boxes are used to try to make up for design flaws, enforce security, or educate the users. It doesn't work. The result is message boxes are frequently irrelevant, uninformative, or incomprehensible. They “cry wolf” most of the time so users ignore them. For day-to-day use, when users get a message box:
Reading takes a long time. The odds are pretty slim that any careful reading of a message will result in a meaningful difference in user behavior. So from the users’ standpoint of just wanting to get on with their task without getting delayed or sidetracked, getting rid of the message boxes as efficaciously as possible can be seen as optimal behavior.
Don’t attempt to force users to read messages by making them complete some task (e.g., make them enter the nth word in the message into a blank) in order to dismiss the message. Such games are self-defeating. Users will focus mental resources on completing your game, rather than understanding the message.
If we are going to get users to read our messages, we’ve got to substantially improve the chance that any given message box is worth paying attention to, which means eliminating as many messages as possible so only the useful ones remain. My answer to Captchas to force user interaction? on SO includes both design alternatives that eliminate message boxes and ways to maximize the effectiveness of the few message boxes that should remain in your app. I’ve full details on the issue, including The Gallery of Pointless Messages at Of Dialogs and Detritus.
I can offer one instance where I - and the rest of the UX and development team to which I belonged - was proven wrong.
The site we worked on was for an email service provider. The boss wanted a popup for first time visitors - "Sign up for a short course on how to deliver email marketing successfully." It was one huge call to action, an email field, an OK button, and a No Thanks button.
Huge argument. HUGE. We assured the boss this was a terrible idea. No one would read it. It would annoy newcomers. It was cheap. Tawdry. Disaster loomed. His response was a command to do it.
Three months later 18,000 users had signed up. Eighteen thousand.
This is an associated number but is not the number I was intending to quote. A former colleague has assured me the number for signups - which is what I was referring to - was eight thousand in four weeks. Apparently A/B testing was taken - the alternative was a static on-page pane - and the popup was more effective by a mile. If we can find the stats I will post.
My takeaway was that context is king. People were coming to the site for our service, but also for the resources we could provide. To design the thing we had marketing experts work on the message itself, and the call to action was spot-on. It was simple.
I still hate popups (personally I don't read them), but they aren't the automatic devil we may think.
Given the number of people that ignore the pop ups that happen when you try to answer your own question or post a 2nd answer here on Stack Exchange you can safely assume that a very small percentage of people actually read them.
Even in cases where the pop up is a license agreement people (myself included) look for the "yes I've read this" box and click OK without actually reading it.
So, yes, you should design your application to avoid pop ups where ever possible unless you can incorporate the instructions/mechanism for the user to correct the problem in the pop up. This will keep the error message on the screen while they re-enter data etc.