Live from ConvCon Chicago 2013: An Inconvenient Truth of MVT and A/B Testing

June 12, 2013 | 2 min read
By Jenny Knizner

As Day 2 of Conversion Conference continues, Zee Aganovic, CEO of HiConversion presents 10 inconvenient truths of multivariate and A/B testing:
An Inconvenient Truth of MVT and A/B Testing

1. Visitor reactions are unpredictable

Rules based targeting and personalization is very risky, nobody can predict visitor reactions. You must test every option so see how your visitors really act and react.

2. A/B testing is a poor relative of MVT

MVT can be a very arduous process, so most people will avoid this. Every element on the page impacts every other element. So testing elements in silos presents the possibility of making winning elements less effective. MVT allows you to test each element relative to the others.

3. A/B test results aren’t easier to interpret

A/B testing gives you just a yes or no response on the test. MVT testing empowers teams with actionable insights.

4. Testing is about end-to-end results

Page-by-page (iterative) testing is unlikely to yield sustainable results. You cannot assume that one successfully tested element works on every page of a site. Mult-step conversion actions become tricky for testing unless you are able to segment the control group and test group throughout the entire conversion process.

5. Technology Matters

Technology capabilities will impact your strategy and ROI. Better technology will enable best practices and vice versa.

6. Both A/B and MVT are risky

The losers often outweigh the winners. The prospect of losing money is always present. However, intelligent technology can throttle traffic based on predicted performance. Consider using a mature tool that offers an adaptive algorithm approach.

7. Both A/B and MVT are expensive

Consider total cost of ownership and ROI. Factors to consider include cost of tool, implementation, and most importantly (and mostly overlooked), the cost of lost revenue. You are not rich enough to afford cheap solutions. Having good technology pays.

8. Visitor behavior is always changing

There is a median value of visitor behavior, but that can vary as much as 10x. Lead generation is a little more stable, but this is especially true for ecommerce. There are many outside factors that influence a visitors’ decision that are outside of your control. The best way to counteract this is to use reactive technology that will automatically detect these changes and adapt in real time.

9. Lies, damned lies, and statistical significance

Statistical significance and confidence are misleading you. This is due to a lack of real time behavior views. Many times the delay in implementation will render a test void. There is no static solution for a dynamic problem.

10. Best practices are dead, long live best practices

Stop focusing on the “winner”. Look for the variable that consistently performs well and play that all the time as a part of your page.

Key take-aways: you must test. You also must recognize that visitor behavior is always changing and to compensate for that you need intelligent technology.


Live from SMX Advanced: You & A Keynote with Matt Cutts
Live From SMX Advanced Seattle: Pro-level Tips To Succeeding At Retargeting
Subscribe to the Digital Marketing Tip!
Get the latest insights and time-tested wisdom from veterans of the digital marketing industry.

Live from ConvCon Chicago 2013: An Inconvenient Truth of MVT and A/B Testing

June 12, 2013 | 2 min read
By Jenny Knizner

As Day 2 of Conversion Conference continues, Zee Aganovic, CEO of HiConversion presents 10 inconvenient truths of multivariate and A/B testing:
An Inconvenient Truth of MVT and A/B Testing

1. Visitor reactions are unpredictable

Rules based targeting and personalization is very risky, nobody can predict visitor reactions. You must test every option so see how your visitors really act and react.

2. A/B testing is a poor relative of MVT

MVT can be a very arduous process, so most people will avoid this. Every element on the page impacts every other element. So testing elements in silos presents the possibility of making winning elements less effective. MVT allows you to test each element relative to the others.

3. A/B test results aren’t easier to interpret

A/B testing gives you just a yes or no response on the test. MVT testing empowers teams with actionable insights.

4. Testing is about end-to-end results

Page-by-page (iterative) testing is unlikely to yield sustainable results. You cannot assume that one successfully tested element works on every page of a site. Mult-step conversion actions become tricky for testing unless you are able to segment the control group and test group throughout the entire conversion process.

5. Technology Matters

Technology capabilities will impact your strategy and ROI. Better technology will enable best practices and vice versa.

6. Both A/B and MVT are risky

The losers often outweigh the winners. The prospect of losing money is always present. However, intelligent technology can throttle traffic based on predicted performance. Consider using a mature tool that offers an adaptive algorithm approach.

7. Both A/B and MVT are expensive

Consider total cost of ownership and ROI. Factors to consider include cost of tool, implementation, and most importantly (and mostly overlooked), the cost of lost revenue. You are not rich enough to afford cheap solutions. Having good technology pays.

8. Visitor behavior is always changing

There is a median value of visitor behavior, but that can vary as much as 10x. Lead generation is a little more stable, but this is especially true for ecommerce. There are many outside factors that influence a visitors’ decision that are outside of your control. The best way to counteract this is to use reactive technology that will automatically detect these changes and adapt in real time.

9. Lies, damned lies, and statistical significance

Statistical significance and confidence are misleading you. This is due to a lack of real time behavior views. Many times the delay in implementation will render a test void. There is no static solution for a dynamic problem.

10. Best practices are dead, long live best practices

Stop focusing on the “winner”. Look for the variable that consistently performs well and play that all the time as a part of your page.

Key take-aways: you must test. You also must recognize that visitor behavior is always changing and to compensate for that you need intelligent technology.

Share This Post
PREVIOUS POST

Live from SMX Advanced: You & A Keynote with Matt Cutts
NEXT POST

Live From SMX Advanced Seattle: Pro-level Tips To Succeeding At Retargeting
Subscribe to the Digital Marketing Tip!
Get the latest insights and time-tested wisdom from veterans of the digital marketing industry.