Section Three: Creative Testing Recommendations 2.0

 

 

And here is what we think they mean:

 

Creative Testing Recommendations 2.0:

 

Because of the A/A test findings that we outlined in the past section, we decided to re-test our IPM winners while excluding the control video so we could determine if we have been killing potential winners.

Drawing from what we learned when we re-ran those tests, we updated our testing methodology. This is not the first time we have done this – our testing methodology is constantly evolving – but some significant changes had to be made once our understanding of Facebook’s testing algorithm evolved.

For Q1 2021, we recommend one of the three-phase testing plans outlined below, depending on whether your app monetizes with purchases or advertising.

But before we dive into that, there is one major issue we know you are curious about, and it applies to both IAP and IAA advertising: iOS 14.

 

Facebook’s Creative Testing Recommendations for Game Marketers in 2021

Facebook Gaming recently published a new report “Games Marketing Insights for 2021” in which they detail the impact of the COVID-19 pandemic over the past year on mobile gaming. By comparing cohorts of existing gamers to the millions of new mobile gamers that emerged in 2020, Facebook uncovered some interesting insights for mobile marketers.

Most notably, Facebook discovered that “Gamers want to see gameplay in ads.”

When it comes to gaming ads, both cohorts of existing and new gamers have similar preferences: everyone prefers to see the main gameplay and story. Specifically, gamers want to see ads that showcase the game’s art style, the characters, as well as high scores that players can achieve.

Mobile game marketers would do well to adopt a creative approach focused on storytelling, taking into consideration that people like to see gameplay in ads. By telling rich stories through immersive experiences and new formats, you can create familiarity, which we know drives discovery. Consider the creative elements in your ads with other creative means of engaging players such as streamer partnerships which have the potential to influence prospective gamers.

 

What happens when those iOS 14 privacy filters go up?

It is pretty simple: We’ll switch over to Android to do creative testing.

This is not even all that huge a change. We have generally tried to test on Android even before because it is less expensive. The only time we do not test on Android is when the client doesn’t have an Android app (which is very rare) or they’re just dead set against it.

The best news is that Android wins translate to iOS. So, in some ways, the iOS update is just cementing what we’ve been doing before. Also, CBO and business as usual winners translate over to AAA. So the iOS 14 change will have some effects, but it may not be as catastrophic as it first appeared.

 

Our 3-Step Creative Testing Process for IAP (In-App-Purchases)

 

Phase 1: IPM Test

• No control videos

• Create a new split test campaign using 3~6 new creatives (no control).
  • Setup campaign structure for basic App Install (No event optimization or value optimization)
  • Spend an equal amount on each creative. Ex: One ad per ad set.
  • Budget for at least 100 installs per creative
    • $200~$400 spend per ad is recommended (based on a CPI of $2-$4) if T1 English-speaking country
    • $20~$40 spend per ad/adset testing in India (based on $0.20-$0.40 CPI)
  • US Phase 1 testing.
    • 10-15% LAL with a seed audience similar to past 90-day installers, or past 90-day payers.
  • Non-US Phase 1 testing.
    • Use broad targeting & English speakers only
    • If not available in India, try other English-speaking countries with lower CPMs than U.S. and similar results. Ex: ZA, CA, IE, AU, PH, etc.
  • Use the OS (iOS or Android) you intend to scale in production
  • Use one body text
  • Headline is optional
  • FB Newsfeed or Facebook Audience Networking placement only (not both and not auto placements)
  • Be sure the winner has 100+ installs (50 installs acceptable in high CPI scenarios)
    • 100 installs: 70% confidence with 5% margin of error
    • 160 installs: 80% confidence with 5% margin of error
    • 270 installs: 90% confidence with 5% margin of error
    • IAP Titles: kill losers, top 1~3 winners go to phase 2
    • IAA Titles: kill losers, allow top 1~3 “possible winners” to exit the learning phase, and then put into “the Control’s” campaign

Which Creatives Move from Phase 1 > Phase 2?

• How To Pick A Phase 1 IPM Winner?
  • IPMs may range broadly or be clumped together
  • Goal: kill obvious losers and test remaining ads in phase 2
  • Ads (blue) have IPMs 6.77 & 6.34, move to phase 2

• If all ads are very close (e.g., within 5%), increase the budget

• IAA (in-app ads titles) you may need more LTV data before scaling

 

Phase 2: Initial ROAS

• No control videos

• Create a new campaign with AEO or VO optimization

• Place all creatives into a single ad set (Multi Ads Per Ad Set)

• Use IPM winner(s) from Phase 1 (you can combine winners from multiple Phase 1 tests into a single Phase 2 test)

• OS – Android or iOS. 5-10% LALs from top seeds (purchases, frequent users + purchase) + Auto Placements

• Testing can be done at a lower cost if you wish to run this campaign in other countries where ROAS is similar or higher but CPMs are much lower compared to the US – ie., South Africa, Ireland, Canada, etc.

• Lifetime budget $3,500-$4,900 or daily budgets of $500-$750 over 4-6 days (depending on your $/purchase).

• WARNING! Skipping this step is highly likely to result in one of the following scenarios:
  • Challenger immediately kills the champion/control but hasn’t achieved enough statistical relevance or exited the learning phase and therefore the sustained ROAS/KPI may not be sustained.
  • Champion/control video has a lot more statistical history and relevance and most likely has exited the learning phases and may immediately kill the challenger before it has a chance to get enough data to properly fight for ROAS.
• The latest change to our Phase 2 strategy: we have become more focused on retention than we were a few months ago. Before, we were just looking at pure ROAS. Now, we pay attention to retention information as it comes in on Day 1 to Day 3 or so. This is usually possible, because we’ll run Phase 2 ads for about 5 days, and that’s worth weighing in for which pieces of creative you take to Phase 3.

 

Phase 3: ROAS Scale

• No control videos
• Use strong CBO campaign
• Choose winner(s) from Phase 2 with good/decent ROAS
  • You have proven the ad has great IPM and “can monetize”
  • To win this phase, it must hit KPIs (D7 ROAS, etc.)
• Create a copy of an existing ad set
  • Delete old ads and replace them with your Phase 2 winner(s)
  • Allows new ads to spend in a competitive environment
• Then, create a new ad set, roll it out towards target audiences with solid ROAS / KPIs
• CBO controls budgets between ad sets with control creatives and ad sets with new creative winners.
  • Intervene with ad set min/max spend control only if new creatives do not receive spend from CBO.
• Require challenger to exit the learning phase before moving to challenge the control “Gladiator” video.
• Once the challenger has exited the learning phase, allow CBO to change budget distribution between challenger and champion.

 

Our 3-Step Creative Testing Process for IAA (In-App-Advertising)

 

Phase 1: IPM Test

• We use the same approach for IAA ads as we do for IAP ads in Phase 1.

 

Phase 2: Initial RPM

• Look at your MMP. Find whichever countries/regions/geos are paying out a good RPM, and just go with those.
• No control videos. Multiple ads per ad set. We call this “the gladiator battle.”
• Create a new campaign with AEO (not VO), but instead of the event being a purchase, have it be an event that could only occur after someone has played the game for a long time like they’ve achieved a certain level.
• For AEO, with a nonpurchase event, you can lower your budget down to even $250 per day.
• Auto placements are okay if that’s what you typically use for the account. Generally, just use the placements you already know will work. No need to reinvent the wheel here. Just run your default best setup.
• For audiences, just run your top two to three audiences. If you are concerned about budget, stack your audiences so you only have one ad set. This will get the ad set out of the learning phase faster and save you some money. (Keep in mind that you may not get out of the Learning Phase at all sometimes in Phase 2 testing.)

 

Phase 3: ROAS Scale

• For IAA, there really is not a Phase 3, but our recommendations for what to do at this stage depend on your budget:
  • If your budget is small, you are not going to know performance for several weeks and so you might as well just roll your best-performing ads out into production.
  • If your budget is large, do a scaled-down Phase 3 structure as we suggest for IAP advertising. This is especially important if you have control ads that are still doing well. Roll strong Phase 2 performers whenever you need a win.

 

Automation on Facebook AAA and Google UAC

 

We are running a lot of ads on Facebook and Google’s versions of automated ads. We do not tend to test in either of these campaign types (more on that below), but they’re an essential part of our work now, and they need to be run very differently than standard campaigns.

As with standard campaigns, IDFA is expected to have some major impacts, like:

• Value Optimization as we’ve been using it is over
• It’s possible Facebook iOS 14 ad accounts will have a campaign and ad limits
• Google hasn’t told us what they expect for UAC
• We should focus on upper-funnel events in Q1 so we can proxy future behavior
• Facebook’s audience network is going to be substantially smaller
• We will shift into testing on Android before we run on iOS

So that is what we expect once IDFA goes live. For now, here is how we are managing Facebook’s AAA and Google’s UAC.

 

Facebook’s AAA

 

Advantages

• You get creative performance reporting on images, videos, text, and you can get that tied to standard events. This reporting persists (Google’s doesn’t). We do all of our testings on Facebook because of the granular controls we have, both for bidding, creative reporting, and targeting.
• You have a choice between an auto bid and a bid cap.
• There are fewer restrictions on the text. You can use CAPS, exclamation points, emojis, etc.
• Flexibility with creative assets: In Facebook, you can basically do whatever you want with 50 assets, and the reporting will persist. So, if you want to run 49 videos and one image, great. (Note: We also suspect that Facebook will soon give advertisers even more flexibility with how many creative assets they can use. They know that creative testing is the best competitive advantage left for advertisers, and they want to give us the tools to aggressively optimize our ads.)
• You can assume there is going to be a CPM discount and a CPI discount. It’s just like CBO – whenever you use Facebook solutions that they are encouraging, the platform will give you a discount.
• Android and iOS performance can be comparable to standard campaigns.
• 4 x 5 and square aspect ratios tend to do best on Facebook, though in some placements, full portrait (9 x 16) is better. Be careful about Facebook allowing you to run creative assets that don’t entirely fit on given placements. Facebook will just shave off the top and bottom of your ads in some cases, which could mean they are chopping off the ad’s CTA or other critical information.

 

 

Google UAC

 

Advantages

• Flexibility in campaign bidding structure.
• You can have multiple campaigns running in any geo. You can have different CPAs, campaigns just running for installs, whatever.
• Better transparency for performance. You can have a 1:1 event reconciling with MMP data.
• See the performance by traffic source.
• Have multiple Ad Groups, so you can put different creative approaches into separate “buckets” and see how their performance compares. This also means you can run seasonal or ad hoc events without disturbing evergreen campaigns.
• Being able to create multiple Ad Groups also means you can test a lot more creative than you can on Facebook. Google does cap your assets at 20 videos and 10 images, but if you need more you can just create more Ad Groups.
• Portrait (9 x 16) and landscape (16 x 9) aspect ratios are the best bet right now for videos on UAC.

Challenges:

• In Google, you can have up to 20 videos and 10 images. Those are hard caps; you cannot swap out and have even 21 videos and 9 images. But if you swap those assets out to add new assets, you will not be able to see the stats for those creative assets. It is possible to get the stats back if you re-add the videos. So, the data isn’t lost. It’s just not shown.
• Reporting on creative performance is harder to get.
• You cannot control bid settings (like an auto bid and bid cap).
• You cannot control where your traffic is coming from, or how it is allocated throughout your campaigns, Ad Groups, and ads. This means you cannot really do a proper split-test. You can have one creative in one Ad Group, and then create as many Ad Groups as creative variations, but that is not a proper split-test.
• Google is more restrictive about the text. It will not allow CAPS, exclamation points, emojis, etc.
• There are no standard campaign options on Google. There’s only UAC.
• iOS does poorly on Google UAC. Android does fine. Compare this to Facebook, where both platforms tend to perform about the same.

 

Creative Sizes Best Practice

 

To maximize creative testing, run the best aspect ratio and media types across Facebook, Google, TikTok, Snap, and Apple Search Ads.  However, as the failure rate of new creative is 95% you will also want to minimize the number of sizes you produce until you have uncovered a fresh winning creative.  Below is our recommendation to maximize distribution and while minimizing creative production.

 

So, what does all this mean for User Acquisition Managers and Agencies in the Near Future?

As we have written about before, it is promising for anyone who focuses on creative optimization. That is absolutely the thing we will have the most control over going forward. So, pivot away from the hands-on campaign management that the algorithms can now actually do better than humans can. Focus on competitive analysis, creative optimization, and creative strategy.

In Section 4, we will talk about implementing these best practices and how to get help with the major changes that are coming up so fast.

 

section 2 Table of Contents section 4