Skip to content

Compare methods

Compare source explanations, strengths, limitations and references for up to four methods.

10 results
Criterion Conformal Prediction Uncertainty Quantification Model Calibration
Overview
  • Conformal Prediction is a distribution-free framework for constructing statistically valid prediction sets (for classification) or prediction intervals (for regression) around the output of any pre-trained machine learning model. Introduced by Vovk, Gammerman, and Shafer in their 2005 monograph, it provides a finite-sample marginal coverage guarantee — the true label falls inside the prediction set with at least 1-alpha probability — without requiring parametric assumptions about the data distribution.
  • Uncertainty Quantification (UQ) is a computational framework for systematically measuring how uncertainty in the inputs of a model propagates into uncertainty in its outputs. Building on Wiener's polynomial chaos theory (1938) and formalised for general stochastic problems by Xiu and Karniadakis (2002), UQ uses two primary strategies: Polynomial Chaos Expansion (PCE), which represents the model output as a series of orthogonal polynomials matched to the input distributions, and Kriging (Gaussian process) surrogates, which replace an expensive simulation with a fast statistical approximation fitted to a small set of carefully chosen runs.
  • Model calibration is a post-hoc technique that adjusts the probability outputs of a trained classifier so that predicted confidence scores match empirical outcome frequencies. A classifier is said to be perfectly calibrated if, among all predictions made with confidence p, exactly a fraction p of them are correct. Systematic miscalibration of modern deep neural networks was rigorously documented by Guo et al. (2017), who showed that networks trained with standard cross-entropy loss tend to be overconfident, and proposed temperature scaling as a simple, effective remedy.
Use and interpretation
  • Use Conformal Prediction when you need rigorous, finite-sample coverage guarantees on a model's uncertainty output without distributional assumptions. It is appropriate for both classification and regression tasks where calibration data from the same distribution as test data is available. Limitations include the exchangeability assumption (violated under covariate shift), and that coverage is marginal rather than conditional, meaning some subgroups may be under- or over-covered. Alternatives include Bayesian credible intervals (require distributional assumptions) and quantile regression (no finite-sample guarantee without conformal wrapping).
  • UQ is appropriate whenever a computational model has uncertain inputs — parameters measured with error, boundary conditions that are only approximately known, or material properties with natural variability — and the analyst needs to characterise how that uncertainty affects model outputs. It is especially valuable when the simulator is computationally expensive (minutes to hours per run), making direct Monte Carlo infeasible. Input uncertainties must be expressible as probability distributions; if only worst-case bounds are available without probabilistic information, interval or fuzzy methods may be more suitable. PCE is well-suited to moderate numbers of inputs (roughly up to 10–15 dimensions before the curse of dimensionality requires sparse variants); Kriging surrogates can handle somewhat higher dimensions but require careful design-of-experiments to keep the number of training runs manageable. Both approaches assume that the model is sufficiently smooth for a polynomial or Gaussian process approximation to be accurate.
  • Use model calibration whenever predicted probabilities will feed downstream decisions that depend on their face value, such as medical risk stratification, fraud scoring thresholds, or cost-sensitive classifiers. It is particularly important for deep neural networks, which Guo et al. (2017) demonstrated become increasingly overconfident as model depth and width grow. Calibration assumes a representative held-out set; it cannot correct for distribution shift between the calibration and deployment distributions. When prediction intervals rather than point probabilities are needed, consider conformal prediction instead.
Level of measurement
  • continuous
Distribution assumption
  • No
Output
  • Output distribution, mean, variance, Sobol sensitivity indices
  • Calibrated probability estimates aligned with empirical frequencies
Strengths
  • Provides finite-sample, distribution-free marginal coverage guarantees valid for any base model.
  • Model-agnostic: wraps any pre-trained classifier or regressor without retraining.
  • Computationally lightweight at inference time — only requires storing calibration scores and computing a quantile.
  • Flexible nonconformity scores allow adaptation to diverse tasks including object detection and natural language generation.
  • Provides a full characterisation of the output distribution — not just a mean or worst case — enabling rigorous probabilistic statements about model predictions.
  • Sobol sensitivity indices identify which uncertain inputs drive output variability, guiding experimental design and data collection priorities.
  • Once the surrogate is built, propagation and sensitivity analysis are computationally negligible, making UQ practical even for expensive simulators.
  • PCE delivers analytical expressions for mean, variance, and sensitivity indices, avoiding Monte Carlo sampling noise in those quantities.
  • Temperature scaling adds only one parameter and cannot reduce classification accuracy, making it extremely lightweight.
  • ECE provides an interpretable, actionable diagnostic that quantifies miscalibration before and after correction.
  • Post-hoc calibration is model-agnostic and can be applied to any pre-trained classifier without retraining.
  • Well-calibrated probabilities enable reliable threshold selection and expected-value calculations in downstream pipelines.
Limitations
  • Coverage guarantee is marginal (averaged over test inputs), not conditional on each individual x.
  • Assumes exchangeability of calibration and test data; performance degrades under covariate or label shift.
  • Prediction sets can be uninformatively large when the base model is weak or the target miscoverage rate alpha is small.
  • Split-conformal methods require a dedicated calibration set, reducing the data available for training.
  • PCE suffers from the curse of dimensionality: the number of polynomial terms grows rapidly with the number of uncertain inputs and polynomial degree, requiring sparse PCE or dimensionality-reduction strategies for high-dimensional problems.
  • The accuracy of a Kriging surrogate depends critically on the design of experiments; too few or poorly placed training points yield unreliable predictions between them.
  • Both methods assume the simulator response is smooth enough to be approximated well by polynomials or a Gaussian process; discontinuities or sharp gradients in the response surface degrade accuracy.
  • The quality of the UQ result is only as good as the input distributions chosen; poorly specified input distributions propagate incorrect uncertainty into the output.
  • Calibration requires a held-out validation set; small calibration sets produce unreliable temperature estimates.
  • Temperature scaling assumes a single global scalar suffices, which can fail when miscalibration is class- or region-specific.
  • ECE is sensitive to the number and width of bins, making cross-study comparisons noisy without standardization.
  • Calibration does not improve sharpness or discriminative ability; a poorly discriminating model remains so after calibration.
Common pitfalls
  • Using the training set as the calibration set inflates optimism and destroys the validity guarantee — always use a held-out calibration split.
  • Confusing marginal coverage with conditional coverage: the guarantee does not imply that every subpopulation is covered at level 1-alpha.
  • Ignoring distribution shift: applying a conformal predictor calibrated on historical data to a shifted deployment distribution breaks the theoretical guarantee.
  • Treating prediction set size as a direct confidence measure without recognizing that set size depends jointly on model quality and the chosen alpha.
  • Choosing input distributions without physical or empirical justification — assigning a normal distribution simply because it is convenient can produce output distributions that misrepresent the true uncertainty.
  • Underestimating the polynomial degree or using a space-filling design that is too sparse relative to the response surface complexity leads to a surrogate that underfits and underestimates output variance.
  • Ignoring interactions among inputs by reporting only first-order Sobol indices can be misleading when input interactions are the dominant source of variance; always inspect total-order indices.
  • Applying PCE in high dimensions (above roughly 15 inputs) without sparse or adaptive truncation causes the number of terms to explode, making coefficient estimation unreliable.
  • Treating the surrogate as perfect and not validating it against held-out simulator evaluations before using it for decision-making is a common and costly error.
  • Calibrating on the same data used for training leads to overfitting and spuriously low ECE on the training set.
  • Using calibrated probabilities as if they were uncertainty estimates conflates aleatoric and epistemic uncertainty.
  • Reporting only ECE can mask poor calibration in the tails; always visualize the reliability diagram alongside the scalar metric.
  • Applying a calibrated model to a shifted deployment distribution invalidates the calibration and may produce misleading confidence.
Introduced by
  • Vovk, Gammerman & Shafer
  • Norbert Wiener (polynomial chaos, 1938); extended to Wiener–Askey scheme by Xiu & Karniadakis (2002)
  • Platt; Guo et al.
Year
  • 2005
  • Seminal modern form: 2002
  • 2017