The final session I attended on Tuesday was “Multivariate Testing and Conversion Tweaking”. Given my presentation earlier in the day, this is certainly a topic that interests me!
Vanessa Fox of Nine by Blue was up first. She recently wrote a book Marketing in the Age of Google, coming out in April 2010. Vanessa focused on thinking about your audience from a workflow perspective.
First thing to consider is searcher behavior. Searchers currently use mostly 1-4 words in a search query, but how can you know what the intent of that searcher is based on a few words? People are also using search engines more to find all sorts of queries too.
Vanessa reminded us that searching and browsing are two distinct behaviors. 20-25% of search queries have never been seen before! Think about the different searches people do and serve the right content for it:
If you do an offline ad, people will search. She gave an example of Hyundai’s Super Bowl ad where they ended the commercial with a URL: edityourown.com. But, many people searched “edit your own”, and they were not ranked there. Big fail.
Next up was Jason Cooper from Efficient Frontier. Jason started by discussing performance-based design, saying we can no longer assume that what we think as designers is best for users. How did we get to where we are? Technology on the web is maturing because of broadband penetration and the availability of free technologies, like Google Analytics.
What will this mean? Hopefully more testing and greater ROI through post-click optimization.
There’s absolutely a technology component to testing. Jason mentioned several tools, including:
Choose a tool that will allow you to rapidly roll out tests.
What should you test? Everything. Some common things to test:
37Signals tested a single headline on a page and improved conversion by 30%!
Jason gave a great quote: “Plans based on average assumptions are wrong on average.” – Sam Savage. Be careful how you look at data and segment it. One interesting tool that Jason shared was the Twitter Spectrum, where you put in two keywords and can visually see where they have keyword overlap.
Overall, Jason said it is important to build a “culture of testing” within your organization.
Finally was Glenn Alsup of Viewmark, Inc. who focused on qualitative vs. quantitative analysis.
For the qualitative side, they will look at goals, overview, description and scenario (GODS). In a case study, Glenn covered how Agilent Technologies tried to reduce support calls. Using the website, and testing usability, on the first pass, there was only a 27% success rate for users to find the product support info online. In the second pass, they added a feedback mechanism, they were able to increase the success rate to 53%!