Why Analytics Experiments Isn't Serving Your Pages 50/50

By Casey Davenport | Aug 27, 2012
More Articles by Casey


On August 1, we officially said goodbye to website optimizer and hello to Google Analytics Experiments. One of the most notable differences with Analytics Content Experiments is that A/B test pages are no longer always served 50/50.  Google determines the percent of daily visits allocated to each page based on that pages statistical likelihood of success, a method whose principles are adopted from the idea of the “Multi-Armed Bandit”.

What is a multi-armed bandit? (And should I be afraid)?

There is no need to be scared, the multi-armed bandit is our friend! Traditionally used in machine learning, the idea behind the multi-armed bandit is to maximize return while conducting a statistically valid experiment.  As the experiment progresses, the machine learns the actions (in our case, which page to serve) that will yield the highest return.  It then allocates a higher percentage of page views to the page with the highest probability of delivering a payoff.

Example time! Let’s say I run an online lamp store and currently have a charming chimpanzee delivering my call to action.  But I have this nagging hunch that a wise professor kitten would speak to my target audience better.

multi armed bandit experiment candidate #1
vs
multi armed bandit experiment candidate #2

My average conversion rate is currently a strong 20% and I don’t want to lose potential conversions, especially this close to the end of the quarter.  But the more conversions the better, right?

Who will be victorious?! Google content experiments in action

Using the multi-armed bandit, I can test my hunch without sacrificing a large number of conversions.  Google will look at performance metrics each day to decide the fraction of traffic that will be greeted by the chimpanzee and the fraction greeted by our wise kitty friend.

This means I will come out of the experiment with:

  • Lower opportunity cost (lost conversions) due to experimentation
  • Higher profit
  • An answer as to which page is the winner of the experiment

If you are interested in technical details about the statistical validity of multi-armed bandit experiments, Google has written a great explanation that I dare not butcher by attempting to summarize.

Have you had any experience with Analytics Experiments yet?  Who do you think would win the experiment, charming chimpanzee or professor kitten? Leave your comments below or find me on Twitter @CaseyDavenport.

Share this article

Share on LinkedIn Share on Twitter

Blog Search


Subscribe today!

Over 4,000 marketers are receiving actionable insights and tips for improving their digital marketing in each monthly issue - why not join them?