Measuring Incrementality with iROAS
Every advertiser wants to know: “When I spend a dollar on ads, what do I get back?”
For advertisers on DoorDash, this question can be phrased specifically as, “How many dollars of sales do I generate for each dollar of advertising I buy?” We answer this question by providing a “lift study” which can measure the amount of incrementality on your ad spend, or incremental return on ad spend (iROAS).
And now with Ghost Ads, DoorDash advertisers get more accurate lift studies, tighter confidence intervals, and greater trust in ROI. That means smarter investment decisions and more confidence in the value of every campaign.
Moving Past Intent-to-Treat Experiments
Prior to ghost ads, the industry standard for measuring lift was the intent-to-treat (ITT) randomized controlled trial (RCT). In these experiments, once an auction begins, eligible consumers are randomly assigned to either a treatment group, which may see the ad, or a control group, which does not. The control group is then removed from the auction, and their subsequent behavior is compared to that of the treatment group who viewed the ad.
The problem is that ITT compares consumers based on auction eligibility, not on whether they actually saw the ad. In practice, many “treated” consumers don’t even see the ads because of where the ad falls on the page amongst the organic content. This dilutes the measurement signal, making it harder to confidently assess incrementality.
For advertisers, that means ITT often underestimates the real ROI of your campaigns
Ghost Ads: A Better Approach
In 2017, Google and Amazon pioneered the ghost ads technique, which allows us to accurately identify equivalent populations of consumers in the control and treatment groups of a lift experiment:
Treatment consumers: those who actually saw the ad (based on impression data).
Control consumers: those who would have seen the ad if it had been shown (based on ghost ad impressions).
Instead of removing control consumers from auctions, both groups experience the same placements. For control consumers, the ad is replaced with organic content — a “ghost ad” — and impressions are measured on that content.
This makes lift studies fairer, more accurate, and closer to the real user experience. At DoorDash, this methodology now powers Sponsored Listings lift studies, allowing advertisers to measure incrementality with far greater precision than before.
(from Johnson et al)
Supporting Ghost Ads at DoorDash
Implementing ghost ads is challenging for ad platforms because the business logic must remain “fair” to both treatment and control groups, even though consumers in these groups experience the platform differently.
To create a fair implementation of ghost ads at DoorDash, our team focused on three principles:
Auctions must include ghost ads, but cannot penalize real ads.
Ghost ads must be blended into organic content just like real ads.
Automated controls must check for regressions that could introduce bias.
Finally, we evaluated fairness by checking whether treatment and control groups received impressions at the same rate — ensuring our implementation was both accurate and robust.
Auctions
To preserve fairness, our implementation ensures ghost ads participate in auctions but are skipped when establishing second prices, preventing real ads from ever bidding against ghosts. Treatment consumers are only selected once we know the ad would have won a seat at auction.
We automate bias checking at auction via our experimentation platform. The platform logs each time a consumer’s treatment group is evaluated, and standard statistical tests run automatically to detect experimentation bias at the auction stage.
Takeaway: Ghost ads participate in auctions but are skipped when establishing second prices, so real advertisers never bid against ghosts.
Blending
DoorDash navigated a lot of existing complexity in our blending algorithms to fairly support ghost ads for many concurrent experiments. Our blending algorithms for combining ads with organic content try to keep ads relevant and complementary for consumers. For example, ads must not duplicate the surrounding organic content and must be relevant to the organic surface (e.g. a “Date Night with Thai Food” carousel cannot contain a fried chicken sponsored listing).
Because blending is latency-sensitive, We created a real-time auditing system that asynchronously re-blends content for each experiment’s counterfactual to identify situations where we placed a ghost ad where a treatment ad would not have been or vice versa. After the blended response is returned to the consumer, we iteratively go through each blended campaign and look for these outliers one by one. Doing this allows us to be alerted in real time when bias exceeds acceptable thresholds.
Takeaway: Ghost ads are blended with organic content the same way real ads are, ensuring relevance and avoiding duplication.
Bias Detection
We measure the quality of our ghost ads implementation by comparing impression rates between treatment and control groups. By paying careful attention to how ads pass through auction and blending we consistently see less than half a percent difference between the target consumer exposure rates and the actuals.
For example, an experiment dividing consumers 80/20 treatment/control might see 80.3/19.7 when the population is reduced to only consumers with ghost or real ad impressions. Further statistical analysis (e.g. covariance and impact on conversion rates) shows this level of bias has no meaningful impact on experimental outcomes.
Takeaway: Automated controls log every evaluation and run statistical tests in real time to prevent experimentation bias.
In Practice: Sponsored Listings
On DoorDash, ghost ads are live for Sponsored Listings.
When running a lift study now, a treatment consumer might see a carousel where Panino e Basta appears in the first slot, while a control consumer sees the same carousel with Panino e Basta replaced by another store such as Liberty Baking & Roastery.
When we see an impression on Liberty Baking & Roastery, we know that the consumer would have registered an impression on the Panino e Basta ad in the same position with some minimal assumptions about consumer scrolling behavior. This ensures incrementality is measured only among consumers who were actually exposed or would have been exposed to the ad.
Redefining Ad Measurement on DoorDash
Ghost ads represent the new industry standard for incrementality measurement and DoorDash is proud to bring this innovation to our set of offerings. From our recent studies, we have found that ghost ads have reduced experimentation dilution by 92% and improved iROAS confidence intervals by 35%.
By ensuring fairness, reducing bias, and tightening confidence intervals, we’re giving advertisers greater clarity into what their ad spend delivers.
Citations
Johnson, Garrett and Lewis, Randall A. and Nubbemeyer, Elmar, Ghost Ads: Improving the Economics of Measuring Online Ad Effectiveness (January 12, 2017). Garrett A. Johnson, Randall A. Lewis, Elmar I. Nubbemeyer (2017) Ghost Ads: A Revolution in Measuring Ad Effectiveness Journal of Marketing Research, 54(6): 867-884. , Simon Business School Working Paper No. FR 15-21 , Available at SSRN: https://ssrn.com/abstract=2620078 or http://dx.doi.org/10.2139/ssrn.2620078
