Two posts ago, in The Attribution Problem, we ran into a question that attribution models can never actually answer: what would have happened if this marketing didn’t exist?
That question has a name. It’s called incrementality, and it deserves more than a paragraph.
The question incrementality answers
Attribution asks: which touchpoint gets credit for this sale?
Incrementality asks something different: would this sale have happened anyway?
Those sound similar. They are not. A channel can win every attribution model you throw at it and still be creating zero new revenue. Retargeting is the classic example. It usually looks incredible in last-click reports, because by definition it’s showing ads to people who already visited your site, already added something to their cart, or already searched your brand name. Some meaningful share of those people were going to buy regardless of whether the ad ever loaded.
Incrementality testing is how you find out how many.
The basic setup: hold something back
The core mechanic is simple, even if the execution can get complicated. You take your audience, split it into two groups, and treat them differently:
- The exposed group sees the marketing (the ad, the email, the campaign) as normal.
- The holdout group doesn’t see it. Same audience, same time period, same everything else. They just don’t get the treatment.
Then you compare the outcomes. If the exposed group converts at 4% and the holdout group converts at 3.7%, your true lift is 0.3 percentage points, not the full 4%. That 0.3 is the number that’s actually yours. The rest would have happened either way.
This is the same logic as a clinical drug trial, and it’s not a coincidence. A drug that “works” for 90% of patients isn’t impressive if 88% of untreated patients also got better on their own. You only know the drug did something once you’ve measured against a group that didn’t take it.
Why this is different from an A/B test
People sometimes use “A/B test” and “incrementality test” interchangeably, but they’re usually answering different questions.
A typical A/B test compares two versions of something you’re already going to run: subject line A versus subject line B, landing page A versus landing page B. Both groups get treated, just differently. The question is which treatment performs better.
An incrementality test compares doing the thing versus not doing the thing at all. One group gets nothing. The question isn’t “which version works better,” it’s “does this channel work at all, beyond what would have happened anyway.”
You can run A/B tests forever without ever learning whether the channel itself is worth the budget. Incrementality is the test that tells you that.
A concrete example
Say you’re running paid search on your own branded terms, your business name plus variations. It converts beautifully. Cost per click is low, conversion rate is high, and every attribution model on earth gives it credit.
Here’s the test: turn branded search off in a handful of geographic markets for two or three weeks, while leaving it running everywhere else. Watch what happens to conversions in the markets where it’s off.
If conversions in those markets barely move, that tells you something important: most of those people were finding you through organic search anyway, since your brand name was already going to be the top organic result. The ad spend was buying clicks you already owned for free.
If conversions in those markets drop noticeably, that tells you the opposite: the ad is capturing demand that organic alone wasn’t catching, maybe because a competitor is bidding on your name, or your organic listing isn’t as dominant as it looks.
Either answer is useful. Neither one shows up in a standard attribution report, because branded search will look like a top performer in both scenarios.
What you need before you can run one
This is where incrementality testing runs into the same wall as everything we’ve written about measurement debt: it needs a baseline.
A holdout test only tells you something if you can trust the comparison. That means:
- Enough volume. If a market only gets 40 conversions a month, the difference between the exposed and holdout groups will be noise, not signal. Small businesses often need to test at the channel or campaign level, not the geography level, to get enough data.
- A clean starting point. If you already have five other campaigns running that also touch the same audience, isolating the effect of one channel gets harder. This is easier to set up before you’ve layered on a dozen overlapping campaigns than after.
- Patience. Two or three weeks minimum, longer for anything with a slow sales cycle. Cutting a test short because the first few days look flat is how you throw away a real result.
None of this requires a data science team or six-figure software. It requires deciding in advance what you’re comparing, holding your nerve for the length of the test, and being honest about the result even when it’s not the one you wanted.
When it’s worth it, and when it isn’t
You don’t need to run an incrementality test on every campaign. That would be its own kind of waste. A reasonable way to decide where to spend the effort:
Worth testing: Any channel that’s a large share of your budget, any channel where you suspect you’re paying for demand you’d get for free (branded search is the most common culprit), and any decision where you’re about to scale spend significantly based on what the attribution report is telling you.
Not worth testing yet: Small experimental channels where the spend itself is already low. New campaigns still finding their footing. Anything where you don’t yet have the volume to get a readable result.
The goal isn’t to test everything. It’s to test the handful of budget decisions where being wrong is expensive.
The takeaway
Attribution answers “who gets the credit.” Incrementality answers “did this actually work.” Most businesses only ever measure the first one, because it’s what the dashboards show by default and the second one takes deliberate effort to set up.
But the second question is the one your budget actually depends on. A channel that wins every attribution model and creates zero incremental revenue is still a channel you’re paying for nothing. The only way to tell the difference is to hold something back and see what happens when it’s gone.