3

Web sites can enable A/B testing by serving a different page, for example. A mobile app would have to collect data and return it. As a matter of principle, is it okay to have software "phone home" to optimize parts of the display? Should it be disclosed to the user why I'm asking for this permission, even if they don't see the benefit in the current app?

EDIT: To clarify, even if disclosed to the user that I'd be doing so (which I'd absolutely do), is A/B testing important enough to the end user that it would be worth the trade off to them of having the software occasionally phone home? I've edited the question to reflect this, instead of my original question before I sidetracked myself.

Improve this question
1
  • Are you looking for technical answers or an answer as to if it's ethical to do so? We might need to know what platform you are looking for as well.
    – matto1990
    Aug 9, 2010 at 23:10

5 Answers 5

Reset to default
2

If you think it's important enough to do and the user is happy to let you do it I cant see the problem. You aren't going to be sending massive amounts of data back to you so I don't see why the user would mind. Most smartphone users have an unlimited internet contact anyway (at least in the UK they all come with one, bar some Blackberrys).

You could try something like MixPanel if you just want to see how users are using your app and how you might improve it. Never used it, but looks interesting.

Improve this answer
1
  • In the US, you have to pay a high premium for unlimited data, and many carriers have removed the option altogether. However, your point still stands in that smartphone users tend to be aware that background data use happens often and have the ability to restrict or disable it if they're getting close to their cap.
    – Yamikuronue
    Oct 24, 2012 at 17:09
2

We've implemented a type of mobile app A/B testing in our apps with some success. What's important to remember is that data charges differ across networks and countries. I feel that it's best to minimize data use as much as possible. As such, we implemented our A/B test as two different "themes" for the app. users could select which theme they wanted, and on the options page there was an option to vote on their favorite one and submit feedback.

In the end, collecting user feedback isn't easy on any mobile platform, even the more advanced ones. You're really stuck relying on the users willing to go the extra mile and reach out to you directly...

Improve this answer
1

I think the way you're wording the question is a bit confusing, but I'll try to answer anyway.

The best practice for installable apps is to let the user know up front what's going to happen and offer them the option of disabling it. A good example of this is Google Chrome, which notifies users that it will be transmitting data to Google and offers you the option then and there to disable it. It's also similar to the alerts iPhone will give you when an app wants to "send you push notifications". Just be up front, clear and honest and users will thank you for it.

As for the question "is A/B testing important enough for the end user" - I don't think the question is very relevant. Whether end users find A/B testing important or not shouldn't withhold you from wanting to improve your app, because if you do, they'll be happier eventually anyway. So do the A/B testing, but be sure to be honest about what data you're transmitting - these are two separate subjects.

Improve this answer
0

As matto1990 says, more info is required, but on principle I'd say the user should be notified in the event that any expense might be incurred.

Moreover, even if the call is free, it might still affect the user in problematic ways. Imagine the user is waiting for an important call, but it fails to come through since at that very moment the application initiated a call.

Improve this answer
2
  • 2
    I dont think they meant literally call home. It would likely be done over an internet connection which on all phones can work along side the texts and calls service.
    – matto1990
    Aug 10, 2010 at 19:48
  • You're right... it was before the question was edited and I missed that :)
    – Dan Barak
    Aug 12, 2010 at 17:01
0

Generally, you're going to want to focus on selling and supporting an application once it launches to production. Spending a not-insignificant amount of time testing interface tweaks when you should be marketing or bug-fixing may not be desirable.

If you want to run A/B tests (or multivariate tests if there's any complexity in the application at all), I'd get the software release-ready and release it in beta. As a beta, you can

  • forego the need to discourage users by suggesting their experience may be different from someone else's (it's a beta, things will be a little volatile),
  • release new test variants (i.e. "updates") as often as you want, and
  • collect better feedback (i.e. testing metrics) by proposing it as exchange for free use or early access to the software.
Improve this answer
2
  • I don't agree with your first paragraph. In my experience, "generally", you'd want to improve the app based on early feedback as much as possible, not ignore the option for real data/metric-based improvements for reasons like "marketing" (which is pretty vague to begin with and doesn't necessarily have to be done by the programmer/UI designer anyway).
    – Rahul
    Aug 18, 2010 at 15:47
  • 1
    Read the first paragraph again in the context of the second and third paragraphs -- we agree. I'm suggesting that the time to "improve the app based on early feedback" isn't once the app is production-live, but during a pre-production beta release. The user has a reduced expectation for optimum performance when in beta, will be more willing to provide the feedback you need without argument, and you have the potential to develop a stronger base by allowing some to "participate" in the development of the app.
    – Matt
    Aug 18, 2010 at 16:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.