Skip to content
Back to search results

Adaptive A/B Test — Adaptive A/B Testing

Introduced by
Herbert Robbins (bandit framework); Thompson Sampling formalized by William R. Thompson
Year
1952 (Robbins); applied to A/B testing from ~2010s onward
Type
Adaptive experimental design
Family
Experimental design
Font size

What does it do?

An Adaptive A/B test is an experimental design that dynamically reallocates traffic or participants toward better-performing variants during the experiment itself, rather than holding allocations fixed until the end.

Drawing on multi-armed bandit algorithms such as Thompson Sampling or Upper Confidence Bound (UCB), it balances the exploration of uncertain variants with the exploitation of those already showing superior performance, typically yielding higher aggregate outcomes while still producing valid inferential conclusions.

The intuition behind the method

In a classic A/B test you split visitors 50/50 between two versions and wait until the end to declare a winner — meanwhile half the traffic keeps hitting the inferior variant. An adaptive A/B test works more like a learning system: it starts roughly even, but as evidence accumulates that variant B is outperforming variant A, it gradually shifts more traffic to B. The experiment continues to collect data for statistical inference, but the cost of running the worse option is actively minimized in real time. The analogy is a slot machine player who tracks which machines pay out more and shifts their play accordingly, while still sampling all machines enough to stay confident in the comparison.

History

The theoretical foundation of adaptive A/B testing lies in the multi-armed bandit problem, first formalized by Herbert Robbins in 1952. William R. Thompson had earlier proposed a Bayesian sampling strategy for clinical trials in 1933, later named Thompson Sampling, which became one of the most widely implemented algorithms. Academic interest in response-adaptive randomization in clinical trials grew through the 1970s–1990s. The application of bandit algorithms to online A/B testing at internet scale was popularized from roughly 2010 onward, driven by technology companies seeking to reduce opportunity costs during experimentation and by researchers linking bandit methods to causal inference frameworks.

When to use it

Adaptive A/B testing is most valuable when (1) exposure to an inferior variant carries a meaningful cost — in healthcare, digital products with high traffic, or time-limited campaigns; (2) the experiment runs long enough for the allocation algorithm to meaningfully shift (typically hundreds to thousands of observations); and (3) outcomes are observed quickly relative to the experiment's duration. It is also well-suited when testing multiple variants simultaneously (multi-arm setting) and when the goal is to maximize aggregate outcomes during the trial, not only after it. Do not use adaptive A/B testing when: the outcome is delayed relative to the decision epoch (delayed feedback makes updates unreliable); when interpretability of allocation ratios is required for regulatory review (e.g., some clinical trial contexts); when the sample size is very small (fewer than ~200 total observations), as the algorithm has insufficient data to meaningfully adapt; or when a simple balanced design will suffice and operational complexity is not warranted.

Limitations and considerations

  • Requires near-real-time or low-latency outcome observation; delayed feedback (e.g., outcomes measured weeks after treatment) breaks the update cycle and undermines adaptation.
  • Statistical inference is more complex than in fixed-allocation designs — standard t-tests are invalid; specialized sequential or Bayesian testing procedures are required.
  • Adaptive allocation can create imbalanced groups, which reduces power for detecting small effects and complicates subgroup analyses.
  • Implementation requires technical infrastructure for real-time data collection, posterior updating, and traffic routing — not readily available in standard survey or lab platforms.

Source conditions do not replace independent scientific review or validation by an analysis engine.

Strengths

  • Reduces cumulative exposure to inferior variants compared with fixed equal allocation, which matters when the outcome metric is morally or commercially important.
  • Well-suited to multi-arm settings where many variants are tested simultaneously and fixed allocation would spread the sample too thin.
  • Produces valid statistical inference when combined with pre-registered stopping rules, despite the non-fixed allocation.
  • Naturally integrates with Bayesian updating, enabling transparent quantification of uncertainty and evidence at any point in the trial.
  • Flexible to context — applicable in online experimentation, clinical trials (response-adaptive randomization), and behavioral field experiments.

Applications

  • Online product experimentation — routing web traffic toward higher-converting webpage variants during the test to minimize revenue loss from inferior designs.
  • Digital health interventions — adaptively allocating patients to more effective treatment arms in remote or app-based clinical trials.
  • Email and push-notification campaigns — dynamically shifting message delivery toward subject lines or content variants with higher open or click rates.
  • Educational technology — assigning students to more effective instructional variants in real time based on learning outcome signals.
  • Pricing and recommendation systems — testing multiple pricing levels or recommendation algorithms while steering users toward better-performing options during the test period.

How it works

How it works

1

The researcher specifies the variants to compare (typically two or more versions of a webpage, message, treatment, or product), the primary outcome metric (e.g., conversion rate, click-through, task completion), the minimum detectable effect size, and the stopping rule — either a fixed horizon (maximum sample size) or a sequential boundary (pre-registered alpha-spending rule).

2

Each variant is assigned a prior belief about its performance, expressed as a probability distribution over the expected outcome (e.g., a Beta distribution for binary outcomes). In frequentist UCB approaches, initial confidence intervals are set wide to reflect equal uncertainty. All variants start with equal or near-equal allocation to ensure sufficient early exploration.

3

At each decision epoch — which may be every new participant, batch of participants, or time unit — the adaptive algorithm determines how to allocate the next unit(s) of traffic. Under Thompson Sampling, a success probability is sampled from each variant's current posterior and the variant with the highest sampled value receives the next unit; under UCB, the variant with the highest upper confidence bound is chosen. This step is repeated continuously throughout the experiment.

4

After each observation, the outcome is recorded and the chosen variant's belief distribution is updated (Bayesian posterior update) or its confidence interval is recalculated (frequentist update). The updated beliefs immediately inform the next allocation decision, so the design learns incrementally from accumulating evidence.

5

The experiment stops when a pre-registered criterion is met: either the maximum sample size is reached, a sequential boundary is crossed, or the posterior probability that one variant is best exceeds a pre-specified threshold (e.g., 95%). Stopping rules must be declared before data collection to control Type I error inflation that arises from continuous monitoring.

6

The researcher reports the estimated effect size and uncertainty for each variant, the proportion of participants allocated to each variant, and the overall metric gain relative to a fixed-allocation benchmark. Inferential statements are made using the pre-registered stopping rule framework (frequentist or Bayesian) rather than conventional end-of-study t-tests, which do not account for adaptive allocation.

Common pitfalls

  • Peeking without a stopping rule — monitoring outcomes continuously and stopping when significance is first reached without a pre-registered alpha-spending rule inflates the Type I error rate substantially; always pre-register the stopping criterion.
  • Ignoring delayed outcomes — if the outcome metric lags the treatment (e.g., 30-day retention measured for a variant assigned today), the algorithm updates on incomplete data and allocates incorrectly; use designs that account for delayed feedback or switch to a fixed design.
  • Applying fixed-allocation analysis — reporting standard confidence intervals or p-values from end-of-study t-tests after adaptive allocation produces biased estimates; use likelihood-ratio sequential tests, posterior credible intervals, or off-policy correction estimators.
  • Too few observations for adaptation to matter — with a very small total sample the algorithm barely shifts allocation before the experiment ends; in small-sample settings a balanced randomized design is more efficient.
  • Conflating exploration goal with inference goal — adaptive allocation optimizes aggregate outcomes during the trial; if the sole goal is precise estimation of each variant's effect, a fixed balanced design provides narrower standard errors.

Frequently asked questions

Is an adaptive A/B test statistically valid?

Yes, provided that the stopping rule and analysis method are pre-registered and match the adaptive design. Standard t-tests and fixed-sample confidence intervals are not valid after adaptive allocation. Valid alternatives include alpha-spending sequential tests (e.g., O'Brien–Fleming boundaries), Thompson Sampling with Bayesian credible intervals, or off-policy correction estimators such as inverse propensity weighting. The key discipline is committing to the stopping rule before data collection begins.

How is this different from a standard multi-arm experiment?

A standard multi-arm experiment assigns participants to variants in fixed proportions (often equally) throughout the trial and analyzes all data at the end. An adaptive A/B test continuously updates the allocation ratios during the trial based on accumulating outcome data, so better-performing arms receive more participants over time. The inferential machinery must account for this non-fixed allocation, whereas standard multi-arm analysis assumes fixed randomization.

What is Thompson Sampling and why is it popular?

Thompson Sampling is a Bayesian algorithm that at each decision step samples a performance estimate from each variant's posterior distribution and assigns the next unit to the variant with the highest sampled value. It is popular because it is simple to implement, naturally trades off exploration and exploitation, and has been shown theoretically and empirically to achieve near-optimal cumulative outcomes. For binary outcomes it uses conjugate Beta distributions, making the posterior update a one-line calculation.

When should I stick with a classic fixed-allocation A/B test?

Use a fixed design when: outcomes are substantially delayed (feedback lag exceeds the inter-decision interval); the sample is small and the algorithm cannot meaningfully adapt; regulatory requirements mandate balanced allocation (e.g., certain Phase III clinical trials); or the sole objective is precise effect estimation rather than maximizing outcomes during the trial. In these situations a balanced randomized controlled design provides cleaner inference with simpler analysis.

Can adaptive A/B testing be used for more than two variants?

Yes. The multi-armed bandit framework naturally extends to any number of arms (variants). With many variants, adaptive allocation is especially valuable because fixed equal allocation in a multi-arm setting spreads the sample very thin across poorly-performing arms. Algorithms such as Thompson Sampling, UCB, and Successive Rejects scale directly to multi-arm settings.

Related methods

Sources

  1. Russo, D., Van Roy, B., Kazerouni, A., Osband, I., & Wen, Z. (2018). A Tutorial on Thompson Sampling. Foundations and Trends in Machine Learning, 11(1), 1–96.

    Open DOI reference
    Copied.
  2. Offer-Westort, M., Coppock, A., & Green, D. P. (2021). Adaptive Experimental Design: Prospects and Applications in Political Science. American Journal of Political Science, 65(4), 826–844.

    Open DOI reference
    Copied.