Search the website

A/B/n Testing to the Max: How to Reduce Uncertainty in Turbulent Times

What's the point in testing, without declaring a winner? Exactly that

Brands that can create positive customer experiences may find that they maintain more of their market share and are even able to expand it. At the same time, irrelevant, aggressive marketing strategies could result in brand animosity that lasts past the coronavirus pandemic.

Mark that down as #14 in Marketing Truths That Are Even More Relevant in Times of a Global Pandemic.

One prominent way to find the right mix of messages and user experience is to test. Even before COVID-19, most marketers understood the value and importance of A/B/n testing their campaigns. And although not all will always choose to follow this strategy, there seems to be a consensus that it should be utilized whenever possible.

Testing campaigns means we submit to the fact that we don’t always know how the market will respond to each, and we know that blowing it can result in lower and even negative uplift.

Therefore, in this current reality, with customers’ priorities, preferences, and values shifting quickly, and budgets shrinking, marketers must test regularly.

A/B/n: Not a One-and-Done Thing

Generally, when A/B/n testing, once one version/campaign “won,” we remove the others. But it can’t be where it ends, right? We must ask how long should we test the campaign, how often should the campaign be reevaluated, and how do we maintain and regularly examine numerous campaigns?

Ensure Each Customer Gets the Best Option

At Optimove, we know that successful marketing comes from relevant contextual communications. That’s one of the reasons why Optimove created the self-optimizing-campaign. It allows the marketer to test campaigns and creatives against each other by continuously adjusting and deciding what is sent to each customer to maximize uplift.

A few weeks ago, we wanted to create a graph to visualize the uplift generated by some clients. So, we pulled the data from clients in some different industries, including eCommerce, retail, gaming, and streaming services.

For the most part, the results were consistent, and the revenue generated showed a constant uplift from that of classic testing:

Address Changes to Customer Preferences

The data from one of our customers was especially intriguing. And, it clearly showed the necessity of never declaring a winner from an A/B/n test.

In this case, the marketer set a self-optimizing-campaign with a control group, split as follows: 45% of the audience received treatment A, 45% of the audience received treatment B, and 10% were blocked from receiving any campaigns.

For the first three periods, treatment A was generating the most uplift, and so – more customers received that treatment. However, towards the third period, there was a change in customer behavior, which reduced the uplift generated by this campaign.

Optimove automatically identified this change and shifted more customers to receive treatment B, which in turn increased the uplift dramatically. See below.

Respond to Changes in Customer Preferences During COVID-19’s Outbreak

As seen above, customer preference changes all the time. Let alone in times like these.

Creating A/B/n campaigns and letting the winner take all, can reduce campaign uplift dramatically.

Once a winning campaign is selected and implemented, it becomes impossible to understand whether uplift reduction is a result of seasonality, customer preference, or other external factors.

And when we shift the same uncertainty to more turbulent times, such as COVID-19 – it’s virtually impossible to anticipate what campaign or creative will deliver the best results.

Using self-optimizing-campaigns, however, is a great way to reduce some of the guesswork around choosing the right campaign or creative and sending the optimal campaign to each member of your audience.

For more such ideas, best practices, analysis, and tips for #MarketingAmidCorona see:

Published on

Gabriella Laster

Gabriella Laster is Product Marketing Manager at Optimove, responsible for prospect marketing and product messaging. Gabriella holds a B.A. in International Relations and English Literature and an Executive MBA from Hebrew University.