Skip to content

Compare methods

Compare source explanations, strengths, limitations and references for up to four methods.

10 results
Criterion Active Learning Uncertainty Quantification Conformal Prediction
Overview
  • Active learning is an iterative machine-learning paradigm in which a learning algorithm selectively queries an oracle — typically a human annotator — for labels on the most informative unlabeled examples. Formalized by Burr Settles in his seminal 2009 literature survey, active learning addresses the practical bottleneck of annotation cost by achieving high model accuracy with far fewer labeled examples than passive supervised learning requires.
  • Uncertainty Quantification (UQ) is a computational framework for systematically measuring how uncertainty in the inputs of a model propagates into uncertainty in its outputs. Building on Wiener's polynomial chaos theory (1938) and formalised for general stochastic problems by Xiu and Karniadakis (2002), UQ uses two primary strategies: Polynomial Chaos Expansion (PCE), which represents the model output as a series of orthogonal polynomials matched to the input distributions, and Kriging (Gaussian process) surrogates, which replace an expensive simulation with a fast statistical approximation fitted to a small set of carefully chosen runs.
  • Conformal Prediction is a distribution-free framework for constructing statistically valid prediction sets (for classification) or prediction intervals (for regression) around the output of any pre-trained machine learning model. Introduced by Vovk, Gammerman, and Shafer in their 2005 monograph, it provides a finite-sample marginal coverage guarantee — the true label falls inside the prediction set with at least 1-alpha probability — without requiring parametric assumptions about the data distribution.
Use and interpretation
  • Active learning is most beneficial when unlabeled data is abundant but expert annotation is expensive, slow, or scarce — common in medical imaging, legal document classification, scientific data curation, and natural language processing. It assumes oracle availability and that the query strategy can meaningfully identify informative examples. It is less suitable when annotation cost is negligible, the unlabeled pool is small, or the task distribution shifts substantially between queries. Alternatives include semi-supervised learning (when labels cannot be obtained interactively) and self-training (when oracle queries are infeasible).
  • UQ is appropriate whenever a computational model has uncertain inputs — parameters measured with error, boundary conditions that are only approximately known, or material properties with natural variability — and the analyst needs to characterise how that uncertainty affects model outputs. It is especially valuable when the simulator is computationally expensive (minutes to hours per run), making direct Monte Carlo infeasible. Input uncertainties must be expressible as probability distributions; if only worst-case bounds are available without probabilistic information, interval or fuzzy methods may be more suitable. PCE is well-suited to moderate numbers of inputs (roughly up to 10–15 dimensions before the curse of dimensionality requires sparse variants); Kriging surrogates can handle somewhat higher dimensions but require careful design-of-experiments to keep the number of training runs manageable. Both approaches assume that the model is sufficiently smooth for a polynomial or Gaussian process approximation to be accurate.
  • Use Conformal Prediction when you need rigorous, finite-sample coverage guarantees on a model's uncertainty output without distributional assumptions. It is appropriate for both classification and regression tasks where calibration data from the same distribution as test data is available. Limitations include the exchangeability assumption (violated under covariate shift), and that coverage is marginal rather than conditional, meaning some subgroups may be under- or over-covered. Alternatives include Bayesian credible intervals (require distributional assumptions) and quantile regression (no finite-sample guarantee without conformal wrapping).
Level of measurement
  • continuous
Distribution assumption
  • No
Output
  • Output distribution, mean, variance, Sobol sensitivity indices
Strengths
  • Drastically reduces labeling cost by focusing annotation effort on maximally informative examples.
  • Applicable to any supervised learner — classification, regression, structured prediction — without modifying the underlying model architecture.
  • Theoretically grounded: uncertainty sampling and query-by-committee have PAC-learning and Bayesian justifications.
  • Compatible with crowdsourcing workflows and modern annotation platforms.
  • Provides a full characterisation of the output distribution — not just a mean or worst case — enabling rigorous probabilistic statements about model predictions.
  • Sobol sensitivity indices identify which uncertain inputs drive output variability, guiding experimental design and data collection priorities.
  • Once the surrogate is built, propagation and sensitivity analysis are computationally negligible, making UQ practical even for expensive simulators.
  • PCE delivers analytical expressions for mean, variance, and sensitivity indices, avoiding Monte Carlo sampling noise in those quantities.
  • Provides finite-sample, distribution-free marginal coverage guarantees valid for any base model.
  • Model-agnostic: wraps any pre-trained classifier or regressor without retraining.
  • Computationally lightweight at inference time — only requires storing calibration scores and computing a quantile.
  • Flexible nonconformity scores allow adaptation to diverse tasks including object detection and natural language generation.
Limitations
  • Query strategies can introduce sampling bias, causing the labeled set to be non-representative of the true data distribution.
  • Performance gains depend heavily on the choice of query strategy, which is task- and model-specific.
  • Retraining the model after each query is computationally expensive for large neural networks.
  • Oracle noise (annotator disagreement or errors) can degrade performance if not explicitly modeled.
  • PCE suffers from the curse of dimensionality: the number of polynomial terms grows rapidly with the number of uncertain inputs and polynomial degree, requiring sparse PCE or dimensionality-reduction strategies for high-dimensional problems.
  • The accuracy of a Kriging surrogate depends critically on the design of experiments; too few or poorly placed training points yield unreliable predictions between them.
  • Both methods assume the simulator response is smooth enough to be approximated well by polynomials or a Gaussian process; discontinuities or sharp gradients in the response surface degrade accuracy.
  • The quality of the UQ result is only as good as the input distributions chosen; poorly specified input distributions propagate incorrect uncertainty into the output.
  • Coverage guarantee is marginal (averaged over test inputs), not conditional on each individual x.
  • Assumes exchangeability of calibration and test data; performance degrades under covariate or label shift.
  • Prediction sets can be uninformatively large when the base model is weak or the target miscoverage rate alpha is small.
  • Split-conformal methods require a dedicated calibration set, reducing the data available for training.
Common pitfalls
  • Cold-start problem: with very few initial labels, uncertainty estimates are unreliable, leading to poor early queries.
  • Batch-mode active learning may select redundant instances if inter-query correlations are ignored.
  • Applying active learning to deep neural networks with dropout-based uncertainty (e.g., MC Dropout) can underestimate uncertainty in out-of-distribution regions.
  • Comparing active learning runs across seeds or query budgets without confidence intervals leads to misleading conclusions about strategy superiority.
  • Choosing input distributions without physical or empirical justification — assigning a normal distribution simply because it is convenient can produce output distributions that misrepresent the true uncertainty.
  • Underestimating the polynomial degree or using a space-filling design that is too sparse relative to the response surface complexity leads to a surrogate that underfits and underestimates output variance.
  • Ignoring interactions among inputs by reporting only first-order Sobol indices can be misleading when input interactions are the dominant source of variance; always inspect total-order indices.
  • Applying PCE in high dimensions (above roughly 15 inputs) without sparse or adaptive truncation causes the number of terms to explode, making coefficient estimation unreliable.
  • Treating the surrogate as perfect and not validating it against held-out simulator evaluations before using it for decision-making is a common and costly error.
  • Using the training set as the calibration set inflates optimism and destroys the validity guarantee — always use a held-out calibration split.
  • Confusing marginal coverage with conditional coverage: the guarantee does not imply that every subpopulation is covered at level 1-alpha.
  • Ignoring distribution shift: applying a conformal predictor calibrated on historical data to a shifted deployment distribution breaks the theoretical guarantee.
  • Treating prediction set size as a direct confidence measure without recognizing that set size depends jointly on model quality and the chosen alpha.
Introduced by
  • Burr Settles
  • Norbert Wiener (polynomial chaos, 1938); extended to Wiener–Askey scheme by Xiu & Karniadakis (2002)
  • Vovk, Gammerman & Shafer
Year
  • 2009
  • Seminal modern form: 2002
  • 2005