I can't find any research with comparison of different captchas that can clearly told which one causes the smalles loss in conversion rate of form.
If the form is on desktop, which captcha have smallest affect in convertion?
I can't use honeypot technique.
In my unbiased on nothing opinion is this one:
Because its the least intrusive and asks the user for only one click. It also the most appealing (As much a captcha can be)
Every website is different, and every customer segment is different. You aren't going to find any rock solid data that will give you the best captcha for your own site. The best way will be to split test a few different ideas and optimise for the best conversion rate for your product.
A honeypot field is the least intrusive to your user but it generally falls victim to autocomplete forms and screen readers. A honeypot captcha should slide past the form autocomplete scripts but may still cause problems with users using screen readers.
A simple checkbox (like the reCAPTCHA 'I am not a robot' above) is the next least amount of invested effort a user will have to do. You mentioned that it did not work on your site, but verify if that is because of the concept itself or just reCAPTCHA's implementation. You could roll your own that tests for the user physically clicking on the button that should work well.
A regular captcha is usually effective, but can come at the cost of user conversions. In order to make this less intrusive, you could hide the captcha for trusted users. These would include first time users and users finding the site via search or a reputable link.
With effective split testing, you can find out which captcha converts the most users, when and why you need a captcha, and even if the captcha is the reason your users aren't converting in the first place. As Steve Blank says, no facts exist inside the building; only opinions -- and the only way to know for sure is to test.
What is your goal? To reduce spam, or something else?
I don't think you will get a hard and fast rule on this.
Your conversion rate will depend on the demographics of your audience, what your site does, the content, the complexity of forms etc.
The best thing to do would be to MVT different captchas (and no captcha!) and track impact on conversion rate.
Not using one at all or using a different invisible technique to block spam. (Jake posted a nice link in the comments.)
I hate captchas personally. As a user and as a designer.
Idea: In the past when investigating my own alternative to a captcha I had a hidden field on the form that wasn't tabbable to or visible. (I hid it via overlapping it with another element so a bot wouldn't easily know that it was hidden to a human user). Then I left this field blank. A normal user wouldn't see it and therefore not fill it out. A bot however thought it was part of the form and entered content. If this field was not blank I would not send the form contents through to the server. But I would still redirect to a successful sent page so the bot thought it had achieved its goal.
This kind of technique is preferable in my opinion. It's non intrusive, and invisible to the user. User's don't care about your spam.
