Experiments · 2 June 2026
When a holdout group is too small
A 2% withheld slice can get legal to nod and still tell you nothing. Holdout Grammar is about the smallest group that remains honest, not the smallest group that ships.
Product teams in Bangkok often arrive at the studio with a holdout that was negotiated down to almost nothing. CRM wanted 50/50. Legal wanted zero. They settled on 2% and called it science. The chart has error bars that could hide a bus.
Small is not the same as careful
We prefer rotating 10–15% holdouts on a single journey over a permanent 2% across everything. Rotation keeps any one user from living forever in the dark, and the slice is large enough to see a downstream action that the product already tracks. If your base is tiny, you may not get to run a messaging holdout at all. Say that. Do not invent precision.
Pawit’s fintech team, quoted on the alumni page, moved from begging for half the base to a 12% rotating holdout. Legal signed the one-paragraph risk note. That note is a template in Holdout Grammar. It is dull on purpose.
Contamination is how small holdouts die
Shared devices, household accounts, and retargeting pixels will leak treatment into the withheld group. If you cannot check contamination, a small holdout is decoration. Signal Path Studio spends a lab on this before anyone is allowed to brief a lift.
When to stop pretending
If stakeholders will only allow a slice that cannot detect the action you care about, stop calling it an experiment. Run the journey as a policy, annotate the calendar, and wait until access improves. Conservatory Year clinics spend a surprising amount of time on this pause. It is not cowardice. It is refusing to launder a guess through a percentage.