Skip to content
Back to search results

Blocked A/B Test — Block-Randomized Split Test

Introduced by
R. A. Fisher (blocking principle); adapted to online A/B testing by industry practitioners
Year
1926 (blocking principle); 2000s–2010s (online A/B testing application)
Type
Randomized controlled experiment with variance reduction
Family
Experimental design
Font size

What does it do?

A blocked A/B test is an experimental design that partitions units (users, subjects, or clusters) into homogeneous blocks before randomly assigning them to treatment A or treatment B within each block.

Blocking reduces within-experiment noise by ensuring that known sources of variation — such as device type, geography, or user tenure — are balanced across conditions, yielding more precise estimates of the treatment effect than a simple unblocked A/B test.

The intuition behind the method

Imagine testing a new checkout button color on an e-commerce site. Purchase rates differ markedly between mobile and desktop users, so a naive random split may accidentally give one variant more mobile users. Blocking solves this: first divide all users into mobile and desktop blocks, then flip a fair coin within each block to assign A or B. Now both variants see proportionally equal shares of mobile and desktop users. The treatment comparison is made within blocks, so device-driven variation is removed from the error term, making the experiment more sensitive to the real effect of the button color.

History

The principle of blocking in experimental design was formalized by R. A. Fisher in his 1926 paper on the arrangement of field experiments and developed further in his 1935 book The Design of Experiments. Fisher introduced blocking as a way to isolate and remove known sources of heterogeneity, thereby improving the precision of treatment comparisons. The approach was subsequently codified in the randomized complete block design (RCBD). In the 2000s and 2010s, technology companies adapted blocking to large-scale online A/B testing, using user attributes such as device type or account age as blocking variables to obtain more reliable causal estimates from web experiments.

When to use it

Use a blocked A/B test when you know in advance that a covariate (device type, user segment, geographic region, time period) is correlated with your primary metric and you want to prevent imbalance on that covariate from inflating your error estimate. It is especially valuable when sample sizes are moderate and you cannot afford the variance that simple randomization would leave in. Do not use blocking when you have no meaningful prognostic covariates — the overhead is unnecessary and block-size constraints may reduce power. Avoid it when the blocking variable is too fine-grained to yield blocks with at least a few units per condition, or when block membership cannot be determined before assignment.

Limitations and considerations

  • Requires pre-identification of blocking variables; variables not known or not measured before assignment cannot be used for blocking.
  • Very small block sizes (e.g., only 2–4 units per block) reduce the reliability of within-block randomization and can harm power rather than help it.
  • Analysis is more complex than a simple two-sample t-test; the block structure must be correctly incorporated in the statistical model.
  • Blocking on the wrong variable (one uncorrelated with the outcome) provides no variance reduction and adds analytical overhead.

Source conditions do not replace independent scientific review or validation by an analysis engine.

Strengths

  • Reduces experimental error variance attributable to known prognostic covariates, increasing statistical power without increasing sample size.
  • Guards against accidental covariate imbalance that could bias treatment effect estimates in finite samples.
  • Block estimates can themselves provide useful insights into heterogeneous treatment effects across segments.
  • Widely supported in standard statistical packages (ANOVA, regression with indicators, mixed models).
  • Directly analogous to the gold-standard block design used in clinical and agricultural research, giving the approach strong methodological credibility.

Applications

  • Online product experiments blocking by device type (mobile vs. desktop) or platform to compare UI variants.
  • Clinical pilot studies blocking by clinical site or baseline severity to compare two treatment protocols.
  • Educational technology experiments blocking by school or grade level to assess the effect of a new learning tool.
  • Marketing email tests blocking by customer lifecycle stage to measure the impact of subject-line variants on open rates.
  • Agricultural and environmental studies blocking by field location or soil type to compare crop varieties or interventions.

How it works

How it works

1

Specify the hypothesis (e.g., variant B increases conversion rate), choose the primary metric, and identify blocking variables — characteristics known or suspected to influence the outcome. Good blocking variables are measurable before treatment assignment, correlated with the outcome, and not caused by the treatment itself.

2

Partition all eligible units into mutually exclusive, exhaustive blocks based on the chosen blocking variables. Each block groups units that are as similar as possible on those variables. The number of blocks should be kept manageable; very fine-grained blocking can reduce statistical power if block sizes become too small.

3

Within each block, randomly assign units to condition A (control) or condition B (treatment) with equal (or pre-specified) probability. This within-block randomization is the core step that makes the design both fair and analytically tractable. Record block membership for each unit before exposure begins.

4

Deliver the assigned variant to each unit for the full experimental period. Maintain separation between conditions to prevent contamination. Collect outcome data along with block identifiers so that the block structure is preserved through to the analysis phase.

5

Analyze results using a model that accounts for blocks — typically a two-way ANOVA or a regression with block indicators as covariates, or the paired-comparison equivalent if blocks contain exactly one unit per condition. The block terms absorb variance attributable to the blocking variable, leaving a cleaner estimate of the treatment effect and a tighter confidence interval.

Common pitfalls

  • Ignoring the block structure at analysis time — running an ordinary t-test without block indicators discards the variance reduction that blocking was meant to deliver and can inflate the p-value.
  • Post-hoc blocking — defining blocks after seeing outcomes introduces bias and is not a valid blocked design.
  • Over-blocking with too many blocking variables produces tiny, impractical block sizes and can eliminate the statistical gains of the design.
  • Confusing blocking with stratified sampling — blocking is an assignment mechanism applied to a fixed population of experimental units, not a sampling strategy.

Frequently asked questions

What is the difference between a blocked A/B test and a stratified A/B test?

The terms are often used interchangeably in applied settings. Strictly, blocking is a design concept (assignment of units within groups) while stratification can refer to either design or analysis (post-hoc re-weighting). In a blocked A/B test, randomization happens separately within each block before exposure. In a stratified analysis, you may have randomized without blocking but reweight or adjust post-hoc by stratum. Blocking at design time is generally more powerful than stratification at analysis time.

How do I choose which variables to block on?

Block on variables that (1) you can measure before assigning treatment, (2) are expected to correlate with the primary outcome metric, and (3) are not caused by the treatment itself. Common choices in online experiments include device type, user tenure cohort, and geographic region. Avoid blocking on variables with many categories unless your sample is large enough to maintain adequate block sizes.

Does blocking change the sample size requirement?

Blocking can reduce the sample size needed to achieve a given power level, because it removes variance attributable to the blocking variable from the error term. The efficiency gain depends on how strongly the blocking variable correlates with the outcome. If the correlation is low, blocking yields little benefit and a simple A/B test may require fewer analytical resources.

Can I use a blocked design with more than two variants?

Yes. When there are more than two conditions the design is called a blocked multi-arm experiment or a randomized complete block design with multiple treatments. Units within each block are randomly assigned to one of k conditions. The analysis generalizes to ANOVA or regression with block and treatment indicators, followed by planned or post-hoc comparisons.

What happens if I forget to include block indicators in my analysis?

Omitting block indicators means the variance reduction from blocking is lost. Worse, if block sizes are unequal, the naive estimate may be biased. Always include block fixed effects (or a random-effects term) in the statistical model to honor the blocked design and recover the efficiency gains.

Related methods

Sources

  1. Fisher, R. A. (1926). The arrangement of field experiments. Journal of the Ministry of Agriculture of Great Britain, 33, 503–513.

    Open publication reference
    Copied.
  2. Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press.

    ISBN: 9781108724265
    Copied.