What are the criteria that are used to measure the quality of an UI, or compare different UIs with each other?
There are several methods of measurement and I'm assuming your goal is to measure usability.
The first method would be doing usability tests. Set the major goals you want your users to achieve and let them accomplish them in all the UIs you have. See which UI facilitates reaching those goals better.
I suggest you read Jakob Nilsen's: Why You Only Need to Test with 5 Users on this topic.
This method is usually used in early stages where you present completely different UI schemes and have the luxury to change almost anything.
At later stages of the application or site, if you just want to make small changes, you can do A/B testing - display two versions of the element you want to test (usually in a random manner) and test the results (time spent, the desired link clicked, etc.)
A great article about the limitations of A/B testing can be found in Jeff Atwood's blog Coding Horror: Groundhog Day, or, the Problem with A/B Testing
Don't forget to run accessibility tests on the UI as well and make sure that people with disabilities can use the UI properly (the simplest problem to imagine is color blindness).
There are other ways of measurement, for example bringing in a UI expert to do a report, but it's just one person's opinion. It can be very helpful, but cannot replace actually putting the UI to the test by multiple users.
There's always the hallway usability test if you want a quick result :)
A hallway usability test is where you grab the next person that passes by in the hallway and force them to try to use the code you just wrote. If you do this to five people, you will learn 95% of what there is to learn about usability problems in your code.
Good user interface design is not as hard as you would think, and it's crucial if you want customers to love and buy your product. You can read my free online book on UI design, a short primer for programmers.
But the most important thing about user interfaces is that if you show your program to a handful of people, (in fact, five or six is enough) you will quickly discover the biggest problems people are having. Read Jakob Nielsen's article explaining why. Even if your UI design skills are lacking, as long as you force yourself to do hallway usability tests, which cost nothing, your UI will be much, much better.
The hallway usability test is useful for some quick guidance in the right direction before you (maybe) start doing more intensive UI testing. It can get you close to the mark, if not right on.
Alex Faaborg from Mozilla has a nice blog post about Usability in user interfaces:
He starts with some mistakes to avoid, then comes the part relative to this question: a list of rules that every user interface should follow. Put simply, every violation of these rules is a usability bug. (At the very end of his post is a link to this list, under the heading "Next Steps")
Personally I find this very helpful to evaluate interfaces, especially when comparing interfaces or while building your own.