There are several things to consider here:
firstly is the way human beings create passworsds:
Firstly, users will just append a 1 or an ! to their favourite insecure dictionary password, or capitalise the first letter. This is a human trait called satificing (http://en.wikipedia.org/wiki/Satisficing#Decision_making). This human behaviour of aiding memory while just satisfying the rules immediately renders the 'x xxx xxx years of a desktop processor to crack' calculations nonsensical.
Secondly, you provide a mental hurdle for the user when they come back to get into your site again as the password they use for everything that isn't too serious is not working. All you've then done is given the user an extra conscious thought and possible mistake to overcome to get back into your site. This is the against the essential premise of 'Don't make me think' by Steve Krug.
You also have the added problem of obfuscation, as virtually all password fields are starred out all of the time, so visual memory is also taken out of the equation.
So your user is never going to use '642HfD77sk*7' they are going to use 'madonna1#' or '!sarah1957' or something else memorable and personal. You could probably brute force an account belonging to a close person like a parent by hand with a spreadsheet and a day or two, even if the passwords required numbers and special characters.
Here are some examples of popular passwords from recent database hacks ... http://netforbeginners.about.com/od/scamsandidentitytheft/tp/The-Most-Common-Passwords-2011.htm and again, from an attack on Adobe: http://www.bbc.co.uk/news/technology-24821528 ... the same group that selected 'adobe123' en masse is not suddenly going to select a random string of keyboard symbols if required to use numbers and special characters.
this article discusses some of these points also: https://medium.com/building-things-on-the-internet/a0c3eb525200
next is the number of possible types of attack you negate with the use of a strong password
There are many security issues with websites ... three immediate examples, that will gain passwords, are database breaches where millions of passwords are stolen in one go from the source; social engineering, where the user (or their representative) hands the password over without realising it; and phishing / spoofing, where the user is convinced they are using something legitimate when they are not. Not to mention that many people keep a list of password in their wallets / on their desks at home, waiting to be physically stolen. Then we get into the other possible security breaches, like unscrupulous employees, XSS and CSRF, so by insisting on a secure password you are not negating a great deal of potential risks.
The next consideration is how secure does your site have to be
Following on from @BennySkogberg's answer, where serious security is required the developers do take responsibility for the security of their users - and so must you.
Services like paypal and online banks, who protect serious information, use many other sophisticated ways of protecting your account. They monitor alsorts of other things, like the purchases I make, and will lock the account down in the case of suspicious transactions. In these cases the password is not a large part of the security - it basically stops opportunist access from others using the same devices as me. They still require special chars, but considering the first point it's obvious that they have other ways of stopping any brute force attack.
So, in conclusion:
When you put these three points together it is obvious that it isn't really necessary. If you are storing serious data about someone then you have many other threats and should have many other tactics up your sleeve (though you probably should still enforce in order to look secure and avoid bad PR), if not then you shouldn't increase the cognitive load on the repeat visiting human. In almost all cases making things nicer for the repeat visiting human should come out on top.
But, if you must do it, then you please also publish a reminder of your password rules on your login screen, i.e. 'we made you pick a password with at least one special character', allow the password to be created and entered in plain text and provide very good password reset.