As CAPTHCHAs have become something usual, the annoyance tend to be diluted, because they became expected (there is a good talk about that here). The user does not feel interrupted anymore (or not as interrupted as the first user that saw a CAPTCHA felt). That said, any extra step imposed to the user is a bad thing and may become reason of abandonment. And that is why we need to be careful when charging users with internal needs or problems.
In my own experience, I feel frustrated and will abandon the page if the CAPTCHA is really difficult to read, and I am not able to decipher it after a few tries. So, we may say that the UX for a CAPTCHA is a subject by itself: how to let the user show he is human in a simple, elegant, non-intrusive way?
And one of the answers for this problem nowadays are the identity management services (Facebook Connect, Google accounts, Twitter, Open ID, WordPress account, etc.), that let us login with our users that were already verified by the third party, all with just one click. In fact, it will be a good experiment to add those options to your form as a way to override the CAPTCHA and then see what the users tend to chose.