Taking Testing to the Next Level

By Meaghan Lamarre:

Since I work mostly on the technical side of things here at Mandate Media, most folks assume I was either a computer science or political science major in college. Truth is, I majored in psychology. Specifically, social psychology. So while we were all at the Netroots Nation conference last week, I jumped at the chance to return to my academic roots in a workshop entitled "Beyond the A/B test: Using Social Science Theory to Get More from Your List" and I've returned to work with a new perspective on the hot-hot-hot topic of testing.

To provide a bit of context for those who aren't as steeped in this stuff as I am, it's become relatively routine to engage in various experiments with your website and/or email messages to make sure that you're getting the most bang for your messaging buck. Much has been written -- by Time, Businessweek and other major news outlets -- about how the Obama campaign took this testing to a whole new level. The basic idea is that we should test variations rather than assuming we know what works best. For example, test a green donate button and a yellow donate button, instead of just assuming that the red one will get the most clicks. With email testing, this is usually done with respect to a subject line: send subject line A to a small sample of your list and subject line B to a different small sample of your list; whichever version gets more opens/clicks gets sent to the rest of the list. This is all Testing 101.

In the "Beyond the A/B test" workshop, we graduated to Testing 201. The focus was less on the mechanics of testing and more on the theories behind the testing. And that's where things get interesting. The panelists, including moderator and testing guru Regina Schwartz, Deputy Director at The Analyst Institute and Jim Pugh, former Director of Analytics & Development for the Obama campaign, now at ShareProgress, invited us to think about the "why" behind the messages we're testing: what do we know about decision-making or social motivations, both well-researched topics in psychology, that might lead us to a more effective message? Once we have a theory, we can test it and see if we were right! Several panelists shared interesting examples about increasing engagement with their emails (as measured by clicks or opens) by crafting messages that appealed to supporters' sense of shared identity. Of course, we're not always right about our theories, as graduate student David Brockman reminded us with several examples, but that just reinforces the benefit of doing the testing.

At Mandate Media, we're already doing some of this testing, but I'm looking forward to doing more, not just in emails and ads, but on the websites we build, too. And I'm really excited at the prospect of moving beyond simply guessing -- I think the button should be blue! No I think it should be red! -- to actually using social science research to inform our experiments, which should lead us to even more effective campaign messaging via email and on the web. So if you can't find me at my computer, look for me in the library where I'll be doing some research!