If possible you should always try to use a combination of quantitative and qualitative methods. In general an A/B test is all about quantitative measures, but even though "numbers don't lie", they may still be deceiving.
For example, the assumption that shorter execution times leads to better UX might not necessarily hold true. In general it is probably mostly valid to believe that shorter execution times means better usability and usability can in most cases be seen as an enabler for providing a good user experience, but it really cannot stand alone and there are exceptions to the rule.
For example Flappy Birds was horrible from a usability perspective. It was so "unusable" that people could hardly play it. Yet, it succeded to communicate a disturbingly addictive experience (what is "good" UX?) that made people use the app over and over and over...
Same goes for number of launches. Maybe that specific build of the app kept crashing so users had to constantly relaunch it? Or maybe some other factor may have biased the results.
My point is that it is a good idea to set up some quantative KPIs to identify possible problematic areas, but without the ability to survey at least a small sample of the actual users or a test group, you will never know the truth about what people really think of your product.
Set up an A/B test, then try to get access to surveying some of the users about topics your A/B-test suggest might be problematic.
For the A/B test I would focus more on task completion than on execution times and launches. For example you could measure signups, users saving a document, users completing the first level in a game, users purchasing additional services, etc.
Additionally I would look at retention rates. The user just signed up - but does he log back in within XX days?