Home Platforms Facebook Conversion Lift Measurement Issue Goes Undetected For 12 Months

Facebook Conversion Lift Measurement Issue Goes Undetected For 12 Months

SHARE:
For nearly a year, advertisers using Facebook’s conversion lift studies tool were unknowingly being fed faulty data.

For nearly a year, advertisers using Facebook’s conversion lift studies tool were unknowingly being fed faulty data.

A code error caused the reached conversions metric to be undercounted for conversion lift tests conducted on Facebook by several thousand advertisers between Aug. 15, 2019, and Aug. 31 of this year.

Advertisers use conversion lift studies, a free tool provided by Facebook, to measure the incrementality of their Facebook ads and make decisions about their Facebook spending. For affected advertisers, this information was erroneous.

Facebook fixed the bug on Sept. 1, and is “working with impacted advertisers,” a Facebook spokesperson told AdExchanger.

But Facebook didn’t alert buyers to the issue until last week.

This is far from Facebook’s first measurement-related misstep, which include a high-profile case of miscalculated video metrics in 2016 and an overestimation of its reach the following year.

What happened?

Specifically, the bug was caused by a data pipeline migration in August of last year that affected the way in which certain impressions were logged into Facebook’s conversion lift systems.

As a result, Facebook undercounted the number of conversions from people who were exposed to impressions on Facebook apps.

In other words, Facebook miscalculated the number of sales that came from people that saw an ad, which is a key ratio necessary to measure incrementality, because it’s used to calculate other metrics, including conversion lift percentage, which is the difference in conversions between the people who did and didn’t see ads during a test.

A token

Subscribe

AdExchanger Daily

Get our editors’ roundup delivered to your inbox every weekday.

Facebook is offering makegoods in the form of ad credits – or, in Facebook’s parlance, a “one-time coupon” – that must be used by March 31, 2021, but only for advertisers that were “meaningfully affected” by the data pipeline migration issue.

The coupons are being handed out on a case-by-case basis depending on an advertiser’s use of the conversion lift product and how much the bug impacted the results of their conversion testing.

Advertisers that were determined by Facebook not to have been meaningfully affected by the error aren’t getting anything.

According to a FAQ appended to an email AdExchanger obtained that was sent by Facebook measurement reps to affected advertisers last week, Facebook arrived at the specific amount for each coupon by “evaluating the potential incremental spend following study results,” taking into account how “incremental conversions may have shifted as a result of these issues” and how this may have affected marketing efforts on Facebook.

But without actually talking to the advertisers themselves to understand their marketing strategies, and it doesn’t appear that Facebook did, “I’m not really sure how their data science team would actually know that,” an agency representing an affected advertiser told AdExchanger.

This lack of transparency is par for the course for Facebook’s lift conversion tool, which has a reputation of being a bit of a black box, said the executive, who was actually more flummoxed by the “token” amount of the coupon received by one of the agency’s clients – tens of thousands of dollars back for an advertiser that had conducted multiple conversion lift tests and spent millions of dollars on Facebook media between August 2019 and August 2020.

Although Facebook’s conversion lift studies tool is free, the outcome of these studies often directly influences how much money an advertiser spends on Facebook.

“If you’re providing a tool for free as a media company and your data is being used to buy ads, then your liability should be the cost of the media – not what you decide the measurement errors are worth,” the executive said. “In this case, the tool is free and so I guess you got what you paid for.”

Facebook video metrics (comic)Sorry ‘bout that

According to the email Facebook sent affected account holders last week, the delay between discovering the conversion lift issue in late August and mid-November, which is when it gave advertisers the heads up, was because Facebook wanted to do its “due diligence to better understand the scope of the issue” and assess the impact on individual studies and metrics.

But that argument isn’t flying with some buyers, even though the bug didn’t directly relate to a billable metric.

“People continued to spend money after Facebook knew there was a bug, because they weren’t told,” said one account holder who received Facebook’s apologia email and was deeply frustrated by its “vague and corporate tone,” but asked to remain anonymous.

“The subject line of the email was very Facebook,” the account holder said. “It wasn’t, ‘Oh, my god, we effed up and we’re sorry!’ It was ‘Upcoming update to Facebook conversion lift studies.’”

During Facebook’s investigation into the original bug, it also came across and patched two additional, separate and, it says, “smaller” technical issues that affected conversion-based metrics for some lift tests.

The first was to do with the winning probability between Sept. 21 and Nov. 13 for multi-cell studies, which is when a target audience is split into multiple cells, each with its own test and control groups.

A second glitch caused the undercounting of the data that informs Facebook’s statistical models, which led Facebook to underestimate web conversions for some studies that started by June 15 and ended by July 26.

Facebook says that since uncovering the bugs, it’s implemented “additional measures” to better monitor and detect issues with its lift measurement systems going forward.

Must Read

Comic: Off-Platform Media

How RMNs Use MFA And Cheap Inventory To Game Attribution Rules

Retail media is built on its attribution quality, but real purchases can be gamed by programmatic metrics and create perverse incentives for RMNs to serve ads across low-quality inventory.

There’s A Lot Wrong With Google’s And Meta’s Non-Transparent ‘Refund’ Practices

Google and Meta are playing with fire. Their opaque refund practices have already exposed them to customer blowback – and could lead to class-action lawsuits by disgruntled advertisers.

Comic: The Great Online Privacy Battle

How US Intelligence Agencies Buy And Use Programmatic Data For Surveillance

Mike Yeagley, an independent contractor who has scouted and acquired commercial data and technology on behalf of intelligence agencies, is one of the earliest evangelists of using ad tech tracking information to identify and surveil government targets.

Privacy! Commerce! Connected TV! Read all about it. Subscribe to AdExchanger Newsletters

Comic: The MFA Cafe

Adalytics Report Torches Ad Tech For Touting MFA Prevention While Scarfing MFA Supply

Practically every ad tech vendor has put out a press release in recent months full of bluster about cutting out made for advertising sites – and yet supply sources remain oversaturated with garbage inventory.

Cloud-Based Collaboration Is Ad Tech’s Post-Cookie Lifeline – But Will It Last?

Cross-cloud data collaboration technology is the best bet for a post-cookie solution. But it’s vulnerable to similar privacy concerns.

Topsort Raises $20 Million To Seize The Post-Cookie Market Opportunity This Year

Topsort raised $20 million, with plans to seize the 2024 opportunity for post-cookie ad tech.