At a glance
A holdout test assigns eligible accounts at random to receive an additional campaign or continue without it, then compares outcomes. It can help assess whether the campaign added value, but small audiences and long sales cycles can leave the result uncertain. Use this guide to assess feasibility with your analyst and sales team before committing.
Your campaign reached accounts that later created opportunities. That’s encouraging, but your management team asks a harder question: how many would have progressed anyway? Linking campaign activity to pipeline doesn’t answer that by itself.
An account holdout can help you investigate the difference. Randomly selected accounts receive a defined additional campaign; the comparison group continues without that addition. The design only becomes useful if your team can maintain the intended difference and observe enough outcomes to learn from it.
Use this guide with your analyst and sales team to decide whether your audience and delivery process can support a useful test. The examples are hypothetical. The first decision is feasibility: can you learn enough to change an investment decision with the accounts and time available?
When we would recommend another measurement approach
Our recommendation is to commission a holdout only when three conditions are plausible: the additional intervention can remain meaningfully different between groups, enough independent accounts and outcomes are available, and the result could change an investment decision. An experiment that cannot meet those conditions may consume scarce account access without resolving the question.
For a narrow strategic programme, begin with a documented starting position and track the relevant account decisions. Our focused ABM case reports eight accounts entering tailored pilots and more than £1.7m in potential contract value at that stage. It also reports comparative meeting conversion and time to pilot. These are reported programme outcomes and comparisons, not results from a randomised holdout.
That kind of case evidence can explain delivery, progression and the decisions taken. It cannot answer the counterfactual with the same design as a valid randomised experiment. Use it to identify a feasible next test or decide what further evidence is worth collecting, while keeping the causal claim narrower.
If testing is feasible, have the analyst calculate the required sample for the actual baseline and commercially meaningful effect before committing. The illustrative interval below explains uncertainty; it is not a sample-size recommendation or a complete statistical protocol.
Be clear about the extra campaign you’re testing
State what the test group receives and what the control group continues to receive. For example, the intervention might be an additional account-specific workshop invitation and follow-up programme, while both groups retain normal customer service and existing sales activity.
This tests the addition, not the entire relationship with the account. Don’t describe the result as the effect of all marketing if several other activities remain active in both groups.
Keep the treatment reproducible: audience criteria, content, channels, timing and follow-up rules. Record deviations. If sales independently gives the control group the same intervention, the test no longer creates the intended contrast.
Google’s Conversion Lift documentation explains the treatment-control principle for advertising. An account-level B2B programme needs its own design and can’t inherit a platform experiment’s validity merely by using the word holdout.
Check whether the accounts really are independent
Decide whether accounts are genuinely independent. Two subsidiaries sharing a buying team may need to be grouped together. If one corporate parent controls the purchase, assigning subsidiaries to opposite groups can create contamination.
Resolve the account hierarchy before randomisation. Keep the group assignment stable and include all relevant people at that unit. Record why certain accounts are excluded from the experiment, such as contractual commitments or active negotiations that make the intervention inappropriate.
Balance important pre-existing differences through a planned design, such as randomisation within account tiers or regions. Don’t manually put the most promising accounts in the treatment group and call the remaining accounts a control.
Check whether a holdout can answer your question
| Requirement | Question to answer before launch | Consequence if unresolved |
|---|---|---|
| Clear intervention | What differs between the groups? | The result can’t be attributed to a defined change |
| Independent units | Can activity spill across the assigned accounts? | Effective sample and treatment contrast may be weaker |
| Measurable outcome | What event counts, and who records it? | Inconsistent classification can bias the result |
| Adequate sample | Can the design detect an effect worth acting on? | A realistic effect may remain inconclusive |
| Sufficient time | Can the chosen outcome mature within the window? | Early results may miss later progression |
| Operational adherence | Can sales and marketing preserve the assignment? | The intended comparison may disappear |
Work through these questions before commissioning the experiment. Discovering that your account list can’t support the answer you want is a useful planning result. It lets you choose a better measurement approach before spending the campaign budget.
Choose an outcome with commercial meaning
A primary outcome might be the share of eligible accounts reaching a defined qualified opportunity stage within a stated period. Define the stage criteria and whether each account can contribute only once.
Use earlier outcomes as secondary measures if they help explain the mechanism: relevant conversations, verified new relationships or accepted workshops. Don’t replace a failed primary outcome with whichever secondary metric looks strongest after the fact.
Include costs and guardrails. An intervention that creates more opportunities at an unsustainable cost may not be attractive. A treatment that generates complaints or disrupts existing relationships also needs consideration.
Estimate the sample you need
Required sample size depends on the baseline outcome rate, the smallest effect worth detecting, the chosen error thresholds and the design. NIST’s sample-size guidance explains the relationship between effect size, significance and power for proportion tests. Use a calculation appropriate to your actual two-group design rather than borrowing an unrelated minimum.
For rare opportunity outcomes, a list of a few dozen accounts may offer very little precision. Adding more contacts within the same account doesn’t create the same number of independent account observations.
Plan the analysis before launch, including handling of missing outcomes and the review date. Avoid repeatedly checking for a favourable result and stopping at the first apparent success unless the statistical design explicitly supports that approach.
Why an apparent 50% lift can remain inconclusive
Consider a fictional experiment with 200 accounts in each group. By the agreed cut-off, 24 treatment accounts and 16 control accounts reach the defined outcome. The observed rates are 12% and 8%, a four-percentage-point difference and a 50% relative lift.
The dramatic relative figure shouldn’t hide the counts. A simple large-sample approximation gives a 95% interval for the absolute difference of roughly minus 1.9 to plus 9.9 percentage points. That illustrative interval includes no effect. The analysis therefore doesn’t establish a dependable positive lift at that confidence level.
The appropriate calculation may differ for clustered accounts, rare events or a different pre-specified method. The example’s purpose is to show why an attractive point estimate can remain uncertain, not to provide a complete analysis template.
Compare the original groups, including non-responders
Report outcomes for the originally assigned groups, including accounts that didn’t engage with the treatment. Looking only at responders would select a more interested subset and undermine the randomised comparison.
Also report treatment delivery and contamination as diagnostic information. If many assigned accounts never received the intervention, the result estimates the effect of the programme as implemented, which may be weaker than the intended treatment.
Gordon and colleagues’ field-experiment research found that observational approaches often differed from randomised estimates in advertising. Its setting is distinct from ABM, but it explains why selecting exposed or engaged accounts after the fact isn’t an equivalent substitute.
Decide what the estimate applies to
Define the eligible population before assignment. A test among mid-tier accounts with no active opportunity doesn’t automatically estimate the effect for strategic accounts already in procurement. Keep the inclusion rules with the result so later planning doesn’t extend it beyond its scope.
Also define the treatment period and the outcome period separately. An intervention may run for four weeks while opportunities are observed for several months. Decide how to handle activity after the treatment ends, including whether normal follow-up continues equally in both groups.
If accounts leave the target list after assignment, preserve them in the analysis according to the pre-specified rules. Removing them because they became inconvenient can disturb the comparison. Record business changes and missing outcomes transparently rather than quietly replacing accounts.
For revenue outcomes, plan how very large deals will be handled. A single exceptional win can dominate a small experiment. Report the distribution and sensitivity of the result, and avoid changing exclusion rules only after seeing which group contains the large deal.
Prepare the operational agreement with your sales team
Explain the commercial question, the assigned groups and the behaviour that needs to remain different. Sales colleagues should know what normal activity continues and what intervention is being withheld. A holdout shouldn’t be confused with refusing appropriate responses to direct buyer requests.
Create a way to record legitimate exceptions. If a control account independently requests the specific experience being tested, respond according to the agreed customer-handling policy and record the crossover. The analysis should preserve the assignment and explain the resulting dilution of the contrast.
Agree who can alter the treatment and what constitutes a material protocol change. A mid-test change in the offer or qualification standard may create a different experiment from the one originally approved.
Choose a useful alternative when the test isn’t feasible
You can still improve operational reporting, inspect account progression and gather buyer explanations. A phased rollout or another evaluation design may be useful, but its assumptions need assessment; it shouldn’t be labelled causal proof automatically.
For a small strategic-account programme, a detailed case review may give you a more useful account of progress than an experiment with too little information to resolve the question. Record the starting position, actions, response and remaining uncertainty, then identify what the next phase needs to establish.
Our guide to measuring demand generation covers the wider reporting structure. Discuss your programme objectives with ABM Logic before deciding which measurement design can realistically support them.
