I don't know of any studies, but I can give you some advice to help keep your results more meaningful.
If you really want to test the testing software it shouldn't be too hard to write server-side logic or "deployed" vs "not deployed" in some kind of A/A test.
- Software Flicker - if you use slow a/b testing platform, you're likely to exhibit page "flicker" which could invalidate tests (i have now seen the default and version B)
- Software Latency - if you introduce added latency you're probably harming your conversion rate (ironically). For this reason it's best to dish out the extra money and get a platform that is fast, and backed by SLAs.
- What you are measuring - if you test button clicks on the homepage when you're manipulating your purchase funnel, it's likely that you might not see a great deal of change. Your metrics should be as close to the changes as possible... not too upstream, not too downstream.
- Test Duration - Don't test for just a few days. The longer you test the closer to the "true average" your numbers will become and converge. You may also notice that as time goes on all versions converge to a similar conversion rate meaning that test didn't do much.
- Run A/A tests! - If you run an A/A test for 2 months and you notice uplift, then you should factor that in as an additional margin of error in your real test results. If You see 5% increase, and your next real tests results in a 5% increase, it is possible that it has no little to no effect.
Point #4 is important, personally. At the end of the day, if a visitor wants to purchase from your site they will, regardless of your testing habits. The only exception being things like Price and Benefits offered. But, those (price tests) will probably give you the greatest insights about what drives conversion...not just "do they like blue or green buttons"