There are different ways to do this, different approaches and different techniques. It's also very important to define a multidimensional threshold for different quantitative and qualitative measures, since not all of the aspects you want to track are the same, not all of them are measured equally (and certainly not using the same tools), and like you say, time is very important.
Now, it's a known fact that users' first reaction is usually of resistance to anything new, which is logic, because they expect changes in the way they used to do things, and some kind of learning curve, so friction is added by default. If you did things right, this resistance will change to a state of happiness which will lead to a deeper engagement with your site.
However, if changes are badly planned, then it may imply the user is gone for good. This is why many companies do gradual changes, which is a conservative way to do it since it means less risks. Note that I don't totally support this approach since results vary once you start adding more elements, not to mention the user perceives a constant sense of "incompleteness", but I agree it's a good way if you are unsure, and nevertheless, a path to consider.
All this introduction is just a very short way to show you a few considerations on a subject that grows as I think in the many ways to approach your problem, so instead of going deeper, I'll tell you what we do, which is NOT the only way, maybe not even the best way, just the way we do it.
1. The Tools
We use a mix of analytics tools, including Google Analytics, CrazyEgg and a custom program we developed using heatmap.js (nothing fancy, just added some automation to send data to analytics sheets). Depending on site and project, sometimes we use surveys, whether they're "in house" or we also use Usabilla.
2. The Process
First, we take ALL DATA from the current data. This is a very thorough process, because we build from this data, improving weak spots, but think on this: we're very marketing and sales oriented, so this part could be different for you, for example if you are OK with results and just want to improve usability or add features, you may not need to do what we do. BUT the important part is to capture as much data as possible, because you'll compare from this data
After this we do our entire building process, for which I won't extend. Before launching, we do some pre-launch testing. Again, this differs with project: the more sensitive, the more testing. Granularity is the keyword, and getting the proper amount of granularity is really difficult.
After this round, we already have some insight and know what to change/improve if needed. For this purpose, we mainly use Zurb's Verify and some custom coded tools. Whenever possible, we reach real users of our client's site. We literally go to their homes or invite them to the client's offices and show them our testing versions (as you may imagine, this is for very specific cases where users are known and geolocation is relevant).
So there we launch (crossing fingers....)
3. The Analytics
Nothing fancy. If you bothered to read this boring article, quite possibly you're telling yourself "I know what are you doing here", but just in case, we take the same data, but with the new site. The only difference is that we define a time threshold, which means this data is analyzed throughout some time (at least through 1 month and at least every couple days, usually every day). This is because we know users need some time to get used to the changes. On some specific elements (usually buttons) we add special tracking, but we do it only if we feel there are issues with CTR or actions in general.
And so we start measuring traffic increase/decrease, bounce rates, time on page, paths followed by users... well, the usual stuff. As I said, nothing fancy
The above is our process since we like to use analytics and scientific approaches. A lot. Of course you can do as you probably have seen in many other sites, and it's OK as well: just have some notice telling users "hey, this is our new site, please let us know what do you think" and get analysis from there
Since I know our process may not fit anyone and everyone (and we spend quite some money per month, which I know is not to everyone's reach), take a look to these additional resources, some of them are not strictly about comparing results, but if you read hem, you'll realize you can easily use some of teh described techniques to compare before/after website changes:
Increasing Usability with User Feedback
Designing Great Feedback Loops
A Guide To Heuristic Website Reviews
30 Useful User Experience (UX) Tools
A guide to carrying out usability reviews