March 7, 2018
As direct-response advertisers, testing is a huge part of our daily ad operations; we’re continually running creatives against each other, optimizing campaign bids and budgets, and analyzing down-funnel results between audiences. A lot of the time, well-structured campaigns can answer all the questions we have. Sometimes, though, we have to put our lab coats on, develop a solid hypothesis, isolate our variables and get our A/B testing on.
In this post, we’re sharing some of our experiences with Facebook’s newish testing feature, creative split testing, along with some advice on when and when NOT to leverage this feature.
First, a bit about the Facebook split testing feature.
Facebook Ads Split Testing 101:
Creative split testing (Facebook’s term for ad A/B testing) has evolved from an API-only feature (which we leveraged through our Facebook tech partner Smartly.io), into the self-serve tool it is today.
Launched in November 2017, Facebook advertisers can now A/B test any of the following variables:
- Delivery Optimization
Facebook A/B testing works by evenly splitting your ad set (up to five times), eliminating audience overlap and ensuring that users only see one variation for the duration of the test. It also guarantees that the exact same amount of budget is allocated to each variation, something that is impossible to achieve without this feature.
Of course, all the standard A/B testing best practices apply:
- Isolate and test only one variable at a time for best results.
- Give your test enough time to determine a statistically relevant winner. Facebook recommends four days, but we use this handy dandy statistical relevance calculator by Kissmetrics.
- Similarly, A/B testing typically works best with larger audiences. The bigger, the better, but aim for audiences of 1 million or more in each of your subsets.
- Set your KPI of interest up front, whether that’s click-through rate or cost-per-install.
For an in-depth look at how to set up Facebook ads for split testing, Adespresso has you covered.
When You SHOULD use Creative Split Testing
The value of creative split testing should be apparent; it helps advertisers determine the scientifically best performing ads, audiences, or bidding strategies. It does, however, require deliberate planning and dedicated budget, thus often lowering overall efficiency.
We’re constantly weighing the benefits of A/B testing with those costs, and have outlined some common scenarios in which you SHOULD utilize A/B tests, and others in which you SHOULDN’T.
Here are a few scenarios in which you should.
#1: Testing imagery to guide future creative direction
The golden rule of running split tests is quite simple–use them to test measurable hypotheses. As an advertising agency that strives to maintain campaign structure hygiene, we take this rule a step further. We run A/B tests against hypotheses that are meaningful to our overall strategy.
When set up correctly, A/B testing can provide black-and-white learnings to inform your creative strategy going forward.
Take this example…
We ran a split test that tested an existing high-performing ad against one with the same messaging, concept and copy but a different design element.
#2: Testing messaging to inform company-wide strategy
Similar to our previous example, split-testing can help inform messaging across channels, or even company-wide.
We ran a split test for an early-stage transportation company to determine which first-touch messaging resonated best with their core demographic.
Details: While we could have run typical ad sets, we knew that in helping our client determine future positioning, having a high level of confidence was important. Running a controlled A/B test also gave us insight into lifetime value, which we’ll discuss later in this post.
Results: In this case, our messaging variations didn’t have a major effect on the CTR. Although statistically relevant, our winning messaging had a less than 3% better CTR. It’s important to remember, however, that even those findings are valuable, prompting us to ask us other questions–do we need to focus on other areas like the imagery? Calls-to-action? Value-prop communication? Should we target different audiences with different messaging?
#3: Testing creative concept variations for specific personas
Split testing can also be useful when figuring out what creative resonates with new or hard-to-crack audiences. Employing the “ax first, sandpaper later” mentality, you can utilize a split test to determine which tactics resonate better, and make incremental optimizations afterward.
Take for example…
For one client, we wanted to run a split test to see if persona-specific creative would perform better than generic creative among a new demographic group.
Details: This particular test utilized existing successful ads against new ads to test our hypothesis that persona-specific creative would do better.
Results: These findings were significant in that this was an entirely untouched user segment for us, and gave conclusive guidelines on how to approach this audience in the future. The costs for the losing concept were 270% higher.
#4: Determining long-tail value of messaging or user groups
Although more time- and resource-intensive, you can also use split testing to create pseudo cohorts to monitor over time after your initial test has completed. Here are some ideas…
- Test what messaging produces the most bottom-of-the-funnel conversions (purchases, etc.).
- Test which Facebook placements have the lowest acquisition costs.
- Test which demographic (i.e., gender, age, interest, education) has the highest lifetime value.
We implemented this tactic for a subscription consumer goods startup to determine what upfront ad messaging produced the best return on ad spend after 28 days.
Results: After running our test long enough to get statistically relevant cost results, we continued to monitor our two groups. We determined that after 28 days, although conversion rates were virtually identical, one group did indeed have 27% higher ROAS.
When NOT to use Split-Testing:
Now it’s time for a reality check. We don’t always have time to set up highly scientific, highly controlled A/B tests. In many circumstances, our structured campaigns and ad sets work just fine, and there are other tailor-made solutions for other circumstances.
Here are three situations you should rethink running an A/B test…
#1. Testing several different creative variables
Although more of a black box, DCO can be more efficient at getting several learnings all at once. While it likely won’t inform your broader strategy, it’s a great use of resources if you want to test several ad copy variations, images, CTAs, etc.
#2: Testing minor creative tweaks that aren’t valuable to your business
At the end of the day, Facebook split testing is the only way to truly run a scientifically precise A/B test. That said, they do require planning and resources, and as a rule of thumb, you should trust that Facebook’s algorithm will deliver your best converting ads.
So, unless you NEED undisputable answers, standard ad sets should get the job done. Regardless, we urge you to weigh the pros and cons of setting up, running, and analyzing a Facebook creative split test up front and hope this post helped you reach a conclusion.
Want a leg up on A/B testing on Facebook? Feel free to check out our Facebook advertising expertise and drop us a line!
We design and execute highly scalable advertising programs for trailblazing brands.More About Us