You turn a channel off to see what happens. Three days in, revenue looks about the same, so you turn it back on and tell yourself the channel wasn’t doing much. That conclusion might be completely wrong, and the problem isn’t the test, it’s the clock.
Incrementality testing only works if the comparison it’s built on is actually trustworthy. Length is where most homemade tests quietly fall apart.
How long should an incrementality test run?
Most tests need a minimum of two to three weeks to produce a readable result, and considered purchases with longer sales cycles need longer still. The real floor is set by your sales cycle and your conversion volume, not by a fixed number of days.
Why doesn’t a few days tell you anything?
A few days of data is mostly noise. Day-to-day conversion swings from normal variation can easily be larger than the effect you’re trying to measure.
If your typical day sees anywhere from 30 to 70 conversions just from random variation, a one-day dip to 35 conversions after turning off a channel proves nothing, it’s well within the range that happens anyway. You need enough days in the sample for the noise to average out and the real signal, if there is one, to become visible above it.
How does your sales cycle set the floor?
The test needs to run at least as long as it typically takes a customer to go from first touch to purchase, or you’ll miss the customers who are still mid-decision.
A business selling a $20 impulse item might have a sales cycle measured in minutes. A B2B software company might have one measured in months. If your average customer takes three weeks to decide, a one-week test will only capture people who were already close to converting when the test started, it tells you almost nothing about customers earlier in the funnel. As a rule of thumb, your test should run at least as long as your typical sales cycle, not just long enough for traffic to accumulate.
How much volume do you need to trust the result?
Enough conversions in both the exposed and holdout groups that a real difference wouldn’t be explained by chance. As a practical floor, aim for at least a few hundred conversions per group before drawing conclusions.
This is the same sample-size problem that shows up everywhere in marketing measurement: a small subgroup produces numbers that look meaningful and are actually noise. A market with 40 conversions a month will never produce a trustworthy holdout result in two weeks, no matter how cleanly the test is designed. If your volume is too low at the geography or campaign level, test at a broader level, like the whole channel, instead.
What is a novelty effect, and why does it ruin short tests?
A novelty effect is the temporary bump or dip that shows up right after you change something, purely because it’s new, before behavior settles back to normal.
Turn off a channel your audience has seen every day for months, and the first few days of “nothing changed” might just be inertia; people still remember the brand and still convert out of habit. Conversely, a sudden change can also cause a short-lived overreaction in either direction. Running the test for a few extra days past where you expect it to flatten out helps you see the real, settled effect instead of the initial reaction to the change itself.
Should you ever stop a test early?
Only if something outside the test is clearly contaminating it, like a site outage or an unrelated promotion. Stopping early because the first few days “don’t look right” defeats the purpose of running a test at all.
The whole point of a holdout is that you don’t know the answer going in. Checking results daily and calling it the moment the data looks favorable is a way of fooling yourself with a process that was supposed to prevent that. Set the length before you start, based on sales cycle and volume, and let it run.
How do you know when you have enough data?
You know you have enough when the gap between your exposed and holdout groups has stayed roughly consistent for several days or weeks in a row, instead of still swinging day to day.
If the daily difference between groups is still bouncing between “clearly working” and “doing nothing” late into the test, you don’t have a stable result yet, you have more noise to wait out. A result that’s held steady for a meaningful stretch, measured in weeks for most businesses, is one you can actually act on.
The takeaway
The length of an incrementality test isn’t a formality, it’s the thing that determines whether the result means anything at all. Too short, and you’re reading noise, inertia, or a novelty effect and mistaking it for an answer.
Set the duration based on your sales cycle and your volume before you start, resist the urge to call it early, and let the test run long enough for the real pattern to separate from the day-to-day static. That patience is the whole difference between a test and a guess dressed up as one.