2

Let's say I have a program with an automatic-updating feature. When I'm shipping a new update, I can do either of the following (or both):

  • Provide a beta channel for advanced users to test the latest updates before deploying them to everyone (the traditional approach), or
  • Roll out updates in a staged manner, gradually over random subsets of users until the update hits everyone (the Google approach).

The first approach gives technically-inclined users a chance to try out the new version ahead of everyone else, but expects the user to understand that there may be problems and that they have a responsibility to report them. Because beta versions are typically sampled by power users, this can lead to inadequate coverage of certain system configurations.

The second approach, pioneered by Google, ships the update to random subsets of users over time until everyone gets the update. This approach is much more likely to sample the target audience evenly and properly cover the myriad configurations users have. It can also be stopped at the first sign of trouble. However, it could leave some users waiting longer than expected for a new version to reach them, and a bad update could negatively impact a subset of users who aren't knowledgeable enough to report the problems as bugs or understand the staged rollout process.

How do I choose between these two approaches (or a combination of both)? What other approaches should I consider?

2

Personally I would favour both.

As you say the beta channel approach can lead to issues with power-users not doing things standard users might, in the simplest example a power user might know not to put text in a field asking for a number so you may miss the errors that would indicate you forgot to validate the entry until it goes live and lots of non-power users try to put text where numbers should go.

I would offer a beta channel, or even closed beta of some form, then once you have a production ready product roll it out over a day or two (maybe more depending on the usage of the software) to all users.

This way you still get rid of the majority of bugs before it hits average users but still have the opportunity to stop the roll out if a critical bug is discovered.

On a side note, Google adopts this two-fold process. Chrome for example still has a canary edition (read beta) for power-users but they still do a gradual roll-out when it comes to updating the release version.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.