Bootstrap Simulation — Empirical Resampling for Statistical Inference
- Introduced by
- Bradley Efron
- Year
- 1979
- Type
- Simulation-based nonparametric inference
What does it do?
Bootstrap simulation, introduced by Bradley Efron in 1979, is a simulation-based inference method that derives the sampling distribution of virtually any statistic by repeatedly resampling with replacement from the observed data.
Because it requires no parametric distributional assumptions, it provides a robust, general-purpose alternative to analytical confidence intervals and parametric hypothesis tests across continuous, ordinal, binary, and count data.
The intuition behind the method
Normally, to understand how variable a statistic like the mean or median is, you would collect many independent samples from the population — but you only have one sample. Bootstrap resampling solves this by treating your one sample as if it were the population and drawing thousands of new samples from it with replacement. Each new draw is slightly different, and together they build an empirical picture of how the statistic would behave across repeated studies — no distributional formula required.
History
Bradley Efron introduced the bootstrap in a landmark 1979 paper and expanded the framework with Rob Tibshirani in their 1993 textbook, which remains the definitive reference. The method grew from earlier jackknife resampling ideas and was revolutionary because it offered a computationally tractable route to valid inference without requiring closed-form sampling distributions. Davison and Hinkley's 1997 Cambridge monograph extended bootstrap theory to regression models, time series, and censored data, consolidating the method as a cornerstone of modern applied statistics.
When to use it
Bootstrap simulation is appropriate whenever the sampling distribution of a statistic cannot be derived analytically or when parametric assumptions (normality, homoscedasticity) cannot be justified. It handles continuous, ordinal, binary, and count variables and works with virtually any statistical estimator — including complex ones such as regression coefficients, correlation differences, or model accuracy metrics. A minimum sample of around 20 observations is generally needed; with very small samples the bootstrap distribution may not adequately represent the population. For correlated or time-series data, use block bootstrap rather than the standard i.i.d. version.
Limitations and considerations
- Computationally intensive when B is large (10,000+ resamples) or the statistic is expensive to compute.
- Cannot correct for a biased or unrepresentative original sample — the bootstrap inherits any systematic error in the data.
- Small original samples (n < 20) may not provide a reliable empirical sampling distribution, even with many resamples.
- Standard i.i.d. bootstrap is invalid for dependent data; block bootstrap must be used instead, and block size selection adds complexity.
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Requires no parametric distributional assumption — valid under broad conditions where classical formulas break down.
- Applicable to virtually any statistic: means, medians, correlations, regression coefficients, model performance metrics, and custom estimators.
- Provides confidence intervals and hypothesis tests without closed-form sampling-distribution derivations.
- Straightforward to extend: block bootstrap handles dependent data; BCa intervals handle asymmetry.
Applications
- Constructing confidence intervals for complex estimators (medians, trimmed means, model accuracy) where analytical formulas are unavailable.
- Hypothesis testing when parametric assumptions cannot be verified — for example, comparing two groups on a skewed continuous outcome.
- Estimating prediction error and model stability in machine learning via bootstrap out-of-bag sampling.
- Biomedical research: deriving confidence intervals for survival curves, dose-response parameters, and diagnostic test characteristics.
- Social science and education research: inference on correlation differences, effect sizes, and composite scale scores.
How it works
How it works
Start with the observed dataset of n observations. Verify that the sample is representative of the target population — the bootstrap can only reproduce what is in the data, so a biased or unrepresentative sample will produce biased bootstrap estimates.
Draw B random samples of size n from the original data, each time sampling with replacement. Efron and Tibshirani (1993) recommend B ≥ 1,000 at minimum; for confidence intervals, B = 10,000 is the standard recommendation. For dependent data (time series, clustered observations) use block bootstrap to preserve the dependence structure.
Calculate the statistic of interest — mean, median, regression coefficient, correlation, or any other estimator — on each of the B bootstrap samples. This produces a bootstrap distribution of B statistic values.
Assemble the B bootstrap statistic values into an empirical sampling distribution. Summarise it with the bootstrap standard error and a bootstrap confidence interval. For skewed distributions the bias-corrected accelerated (BCa) interval is preferred; the percentile interval is the simplest alternative.
Use the aggregated bootstrap distribution to make inferences: report the confidence interval for the parameter of interest, or compute a p-value for a hypothesis test by comparing the observed statistic to the bootstrap null distribution. Because no parametric assumption was needed to reach this point, the inference is valid under broad conditions.
Common pitfalls
- Using too few resamples (B < 1,000) produces unstable confidence intervals — always use at least B = 1,000 and B = 10,000 for final reported CIs.
- Applying the standard i.i.d. bootstrap to time-series or clustered data violates the independence assumption; use block bootstrap for dependent observations.
- Treating bootstrap confidence intervals as equivalent regardless of method — the percentile interval can be biased when the statistic is skewed; prefer BCa in those cases.
- Confusing the bootstrap standard error with the population standard deviation — the bootstrap SE estimates variability of the estimator, not variability of the raw data.
Frequently asked questions
How many bootstrap resamples do I need?
A minimum of B = 1,000 resamples is required for bootstrap standard errors and rough confidence intervals. For final reported confidence intervals, B = 10,000 is the standard recommendation (Efron & Tibshirani, 1993). Fewer than 1,000 resamples produce unstable results and should not be reported.
Which confidence interval method should I use — percentile, BCa, or basic?
The BCa (bias-corrected accelerated) interval is preferred in most situations, particularly when the statistic's sampling distribution is asymmetric or the estimator is biased. The percentile interval is simpler but can be inaccurate under skewness. BCa adjusts for both bias and skewness, making it the most reliable general-purpose choice.
Can bootstrap be used with time-series or panel data?
Yes, but the standard i.i.d. bootstrap (which resamples individual observations) is invalid for dependent data because it breaks the dependence structure. Block bootstrap methods, which resample contiguous blocks of observations, preserve serial dependence and should be used whenever data are correlated over time or within clusters.
Does bootstrap work with small samples?
Bootstrap inference is generally reliable when n ≥ 20. With very small samples the observed data may not adequately represent the population, so the bootstrap distribution inherits this limitation — no amount of resampling compensates for a sample that is too small to capture the population's shape. In such cases, consider exact permutation tests or Bayesian methods.
Related methods
Sources
Efron, B. & Tibshirani, R.J. (1993). An Introduction to the Bootstrap. Chapman & Hall/CRC.
Open DOI referenceCopied.Davison, A.C. & Hinkley, D.V. (1997). Bootstrap Methods and their Application. Cambridge University Press.
Open DOI referenceCopied.