You Should (A/B) Test That!

Capacity Interactive AUTHOR: Capacity Interactive
Sep 22, 2016
6 Min Read

As arts marketers, we’re always looking to optimize our digital campaigns and be more efficient with our budgets. We have to or we’ll see a direct impact on ticket sales. 

We optimize emails that we send and test display and paid social campaigns – the creative, the targeting methodology, the call-to- action, among other variables. We experiment with different search terms for our Google Grants and conduct search engine optimization (SEO) initiatives to improve how Google crawls and indexes our site.

Yet all of these initiatives drive people to the same location – the website – and we don’t really do much to optimize it. Generally speaking, we add content to templates that already exist, but we don’t really change the templates or the design until we have a major website redesign.

This is a really big problem.

Think about it. We spend time, energy, and money optimizing our digital advertising campaigns but we don’t optimize the experience that receives all of the traffic. Website redesigns shouldn’t be the only time we are optimizing our websites.

Luckily there’s an incredibly powerful process to help us iteratively optimize our sites: On-site A/B testing allows us to test a slightly different version of a page against the original and measure which version performs better.

The Benefits of A/B Testing
A/B testing allows us to prove causation, not just correlation.
If you make a change to a page and measure how it performed before vs. after the change, the design change could have influenced the variance in the data, but other variables might have, too (e.g. PR initiatives, site outages, snowstorms, a new marketing campaign, or some other macroeconomic item).

Pre/post analysis provides correlation – a change in the website’s performance is correlated with a change in its design – not that the change in design caused the change in performance. A/B tests let us prove causation.

A/B tests let us focus on one area at a time. We’re not redesigning the entire site – just one specific template at a time. We also mitigate risk – if our change doesn’t work, it’s good we tested it first instead of investing time and money building it, only to find out later on it wasn’t successful.

A/B testing allows us to measure, in real-time, if we’re winning or losing. You can create a test version without writing any code or designing assets in Photoshop. Testing platforms have visual editors that allow you to easily click and drag to make changes. 

A/B Testing in Action
We recently ran a test for Wolf Trap Center for the Performing Arts and focused on the mobile version of their Performance Detail page. The data showed how popular the page was and how it underperformed compared to the desktop version.

We saw where users dropped off in the purchase process and used that data to formulate a hypothesis – we can improve sales by giving mobile users an option to purchase directly on the ticketing site OR to call the box office.

Blog_AB_Testing-03.png Blog_AB_Testing-04.png
CTA Control CTA Test

We created a test version of the page, ran it for two weeks, and measured the test version against the control in real-time.

The results? Our test version increased sales by 19%. 

The test page only took a few hours to build. We used the testing platform to copy the original Performance Detail page and added a new button. That’s it! We didn’t build new functionality on the site or create brand new content. We were able to have a significant lift on one of the most important pages of the site without developers or designers. That’s a great ROI!

Another A/B testing success story comes from Pacific Northwest Ballet – also for a mobile Production Detail page. During their move to responsive design, the ‘buy tickets’ links were moved to the bottom of a long-scroll page.

We wanted to run a test on this – should the links stay at the bottom (letting users consume informational content and then click to begin purchasing) or should the links be moved higher up on the page, letting the user know right away that they can buy tickets on the site?
PNB Control PNB Test

The results? We used the testing platforms click and drag interface to move the buy ticket links higher up, let the test run, and saw a lift of 60% in clicks on those links.

A/B testing is the perfect vehicle to ensure your website continues to evolve over time. It is one of the most critical activities that you should be doing; yet we find that very few arts organizations are running A/B tests. Our goal at Capacity Interactive is to help change that for our industry – whether you want to partner with us on A/B testing initiatives or want to run them internally, you should be thinking about optimizing your website using A/B testing.

A/B testing gives us the ability to test hypotheses quickly. We’ll find out we’re wrong in many cases, which can be frustrating, but it’s powerful to be able to test something before committing dollars and time into something that won’t be impactful. When we’re correct and our tests win, we know for a fact that our design changes caused the improvement.

Remember: A/B testing is inherently a data-driven process. You must have good data if you want to run A/B tests. There’s no shortcut around this. If you want to run A/B tests, you must have good data.

Ready to run some A/B Tests? We’re ready for you. Get in touch with Yosaif to discuss the ABC’s of A/B Testing. 


Yosaif is the Senior Director of Analytics at Capacity Interactive. He will be speaking at this year’s Digital Marketing Boot Camp for the Arts on Iterative Optimization: Small Changes, Big Results.