What are the criteria that are used to measure the quality of an UI, or compare different UIs with each other?
There are several methods of measurement and I'm assuming your goal is to measure usability.
The first method would be doing usability tests. Set the major goals you want your users to achieve and let them accomplish them in all the UIs you have. See which UI facilitates reaching those goals better.
I suggest you read Jakob Nilsen's: Why You Only Need to Test with 5 Users on this topic.
This method is usually used in early stages where you present completely different UI schemes and have the luxury to change almost anything.
At later stages of the application or site, if you just want to make small changes, you can do A/B testing - display two versions of the element you want to test (usually in a random manner) and test the results (time spent, the desired link clicked, etc.)
A great article about the limitations of A/B testing can be found in Jeff Atwood's blog Coding Horror: Groundhog Day, or, the Problem with A/B Testing
Don't forget to run accessibility tests on the UI as well and make sure that people with disabilities can use the UI properly (the simplest problem to imagine is color blindness).
There are other ways of measurement, for example bringing in a UI expert to do a report, but it's just one person's opinion. It can be very helpful, but cannot replace actually putting the UI to the test by multiple users.
There's always the hallway usability test if you want a quick result :)
The hallway usability test is useful for some quick guidance in the right direction before you (maybe) start doing more intensive UI testing. It can get you close to the mark, if not right on.
Alex Faaborg from Mozilla has a nice blog post about Usability in user interfaces:
He starts with some mistakes to avoid, then comes the part relative to this question: a list of rules that every user interface should follow. Put simply, every violation of these rules is a usability bug. (At the very end of his post is a link to this list, under the heading "Next Steps")
Personally I find this very helpful to evaluate interfaces, especially when comparing interfaces or while building your own.