On August 1, we officially said goodbye to website optimizer and hello to Google Analytics Experiments. One of the most notable differences with Analytics Content Experiments is that A/B test pages are no longer always served 50/50. Google determines the percent of daily visits allocated to each page based on that pages statistical likelihood of success, a method whose principles are adopted from the idea of the “Multi-Armed Bandit”.
There is no need to be scared, the multi-armed bandit is our friend! Traditionally used in machine learning, the idea behind the multi-armed bandit is to maximize return while conducting a statistically valid experiment. As the experiment progresses, the machine learns the actions (in our case, which page to serve) that will yield the highest return. It then allocates a higher percentage of page views to the page with the highest probability of delivering a payoff.
Example time! Let’s say I run an online lamp store and currently have a charming chimpanzee delivering my call to action. But I have this nagging hunch that a wise professor kitten would speak to my target audience better.
My average conversion rate is currently a strong 20% and I don’t want to lose potential conversions, especially this close to the end of the quarter. But the more conversions the better, right?
Using the multi-armed bandit, I can test my hunch without sacrificing a large number of conversions. Google will look at performance metrics each day to decide the fraction of traffic that will be greeted by the chimpanzee and the fraction greeted by our wise kitty friend.
This means I will come out of the experiment with:
If you are interested in technical details about the statistical validity of multi-armed bandit experiments, Google has written a great explanation that I dare not butcher by attempting to summarize.
Have you had any experience with Analytics Experiments yet? Who do you think would win the experiment, charming chimpanzee or professor kitten? Leave your comments below or find me on Twitter @CaseyDavenport.