In some website projects, you are testing whether users can complete specific functionality. For example, with a shopping cart I can measure the percentage of users that complete checkout, the amount of time it took them to complete checkout, etc.
However with something like redesigning a user's dashboard, it seems like metrics might not be the best indicator of whether to go ahead with a redesign. You can test things like, "can people find the billing page", but that doesn't take into account the aesthetic.
What sort of questions can you ask during a user test to evaluate whether a new site design is better than the old site design? If you are just going on "how it looks", then the HIPPO (highest paid person's opinion) will win every time.