Live from ConvCon Chicago 2013: An Inconvenient Truth of MVT and A/B Testing

By Jenny DeGraff | Jun 12, 2013
More Articles by Jenny

As Day 2 of Conversion Conference continues, Zee Aganovic, CEO of HiConversion presents 10 inconvenient truths of multivariate and A/B testing:
An Inconvenient Truth of MVT and A/B Testing

1. Visitor reactions are unpredictable

Rules based targeting and personalization is very risky, nobody can predict visitor reactions. You must test every option so see how your visitors really act and react.

2. A/B testing is a poor relative of MVT

MVT can be a very arduous process, so most people will avoid this. Every element on the page impacts every other element. So testing elements in silos presents the possibility of making winning elements less effective. MVT allows you to test each element relative to the others.

3. A/B test results aren’t easier to interpret

A/B testing gives you just a yes or no response on the test. MVT testing empowers teams with actionable insights.

4. Testing is about end-to-end results

Page-by-page (iterative) testing is unlikely to yield sustainable results. You cannot assume that one successfully tested element works on every page of a site. Mult-step conversion actions become tricky for testing unless you are able to segment the control group and test group throughout the entire conversion process.

5. Technology Matters

Technology capabilities will impact your strategy and ROI. Better technology will enable best practices and vice versa.

6. Both A/B and MVT are risky

The losers often outweigh the winners. The prospect of losing money is always present. However, intelligent technology can throttle traffic based on predicted performance. Consider using a mature tool that offers an adaptive algorithm approach.

7. Both A/B and MVT are expensive

Consider total cost of ownership and ROI. Factors to consider include cost of tool, implementation, and most importantly (and mostly overlooked), the cost of lost revenue. You are not rich enough to afford cheap solutions. Having good technology pays.

8. Visitor behavior is always changing

There is a median value of visitor behavior, but that can vary as much as 10x. Lead generation is a little more stable, but this is especially true for ecommerce. There are many outside factors that influence a visitors’ decision that are outside of your control. The best way to counteract this is to use reactive technology that will automatically detect these changes and adapt in real time.

9. Lies, damned lies, and statistical significance

Statistical significance and confidence are misleading you. This is due to a lack of real time behavior views. Many times the delay in implementation will render a test void. There is no static solution for a dynamic problem.

10. Best practices are dead, long live best practices

Stop focusing on the “winner”. Look for the variable that consistently performs well and play that all the time as a part of your page.

Key take-aways: you must test. You also must recognize that visitor behavior is always changing and to compensate for that you need intelligent technology.

Share this article

Share on LinkedIn Share on Twitter

Receive Monthly Digital Marketing Tips

Subscribe to monthly updates from the Marketing Mojo Blog to get the latest digital marketing tips, best practices and insights - hot off the presses and straight to your inbox!


Blog Search