What is incrementality?

The question attribution cannot answer

Attribution divides credit among the ads a converting user saw. It has nothing to say about the users who would have converted regardless — and in most accounts, a large share of attributed conversions fall into exactly that group.

Retargeting is the standard demonstration. It attributes superbly under last click, because it reaches people already close to purchasing and is therefore present at the end of most converting paths. Held out, a substantial part of that measured performance often turns out to be conversions that were going to happen anyway.

Incrementality is the correction. Withhold advertising from a randomly chosen group, compare, and the difference is what the spend caused.

How to run a test

1. Randomise. Split by user, or by geography where user-level suppression is not possible. Randomisation is what makes the comparison valid; a non-random control is just two different groups.
2. Withhold, genuinely. The control must receive no exposure from the channel under test, which is the part that requires discipline.
3. Size it before you run it. Small lift on a noisy base needs a large sample; running a test too small to detect the effect produces a confident null.
4. Run for a full purchase cycle, not for a convenient two weeks.
5. Compare the totals, not the attributed numbers. The whole point is to bypass attribution.

What it costs, and why it is still worth it

A holdout costs real revenue — the conversions the control group would have produced. That cost is the price of knowing, and it is usually small relative to the spend it informs. A 5% holdout on a large channel is cheap insurance against an eight-figure budget being allocated on a biased model.

The other cost is time. A test that respects the purchase cycle takes weeks, which is why it is a periodic instrument rather than a dashboard. Use attribution for the daily decisions and incrementality for the quarterly ones.

Why it is essential for AI media

On AI surfaces incrementality moves from best practice to the only defensible read. Recommendations produce action without clicks, conversions arrive through channels with no link to the exposure, and there is no independent counting layer. Every conventional instrument is guessing.

A holdout is unaffected by all of it. It does not need to observe the path, identify the user, or trust a platform's count — it needs only two comparable groups and an outcome you already record. That is why our guidance on measuring AI assistant ad performance honestly puts it first rather than last.