Skip to content
Back to search results

Stochastic Markov Model — Probabilistic State-Transition Simulation with Uncertainty Propagation

Introduced by
Markov, A. A. (probabilistic extension developed by Sonnenberg & Beck and others)
Year
1993
Type
Probabilistic state-transition model with Monte Carlo uncertainty propagation
Family
Simulation / optimization
Font size

What does it do?

A Stochastic Markov Model is a simulation technique that represents a system as a set of mutually exclusive health or decision states, moves a cohort (or individual agents) through those states using probabilistically sampled transition parameters, and aggregates outcomes across thousands of Monte Carlo iterations to produce full probability distributions over costs, outcomes, or rankings rather than single point estimates.

The intuition behind the method

Imagine tracking a population through disease stages — healthy, sick, recovered, dead — over many annual cycles. In a deterministic Markov model you fix the transition probabilities and get one trajectory. The stochastic version acknowledges that those probabilities are themselves uncertain: they come from studies with finite samples. By drawing new transition matrices from their posterior distributions in each Monte Carlo iteration, the model propagates that parameter uncertainty all the way through to final outputs, yielding credible intervals rather than false precision.

History

Markov chains were formalised by Andrei Markov in 1906. Their application to medical decision analysis was systematically introduced by Sonnenberg and Beck (1993), who provided practical guidance for health economists. Probabilistic extensions — drawing transition parameters from distributions across Monte Carlo iterations — became standard practice through the 1990s–2000s, codified in guidelines by bodies such as NICE and in textbooks by Briggs, Sculpher, and Claxton (2006). Today the stochastic Markov model is the dominant tool for probabilistic sensitivity analysis in health technology assessment.

When to use it

Use a Stochastic Markov Model when decision-making involves discrete states, recurring transition events over multiple time periods, and parameter uncertainty that must be formally propagated to outputs — most commonly in health technology assessment, pharmacoeconomics, and policy evaluation where uncertainty in transition probabilities and payoffs is substantial. It is the standard approach when regulatory or HTA bodies (NICE, CADTH, etc.) require probabilistic sensitivity analysis. Do NOT use it when the system has continuous state spaces better handled by differential equations, when the Markov assumption (memorylessness) is implausible and history clearly matters without being capturable via state expansion, when the analysis horizon is a single period, or when data are too sparse to justify specifying parameter distributions (in that case a deterministic one-way sensitivity analysis may be more appropriate).

Limitations and considerations

  • The Markov (memorylessness) assumption — that future transitions depend only on the current state, not history — is often violated in practice; workarounds via tunnel states add model complexity.
  • Specifying credible probability distributions for all parameters requires good-quality evidence; poorly specified priors propagate poorly calibrated uncertainty.
  • Computational cost grows with the number of states, cycles, and Monte Carlo iterations; very large models may require significant runtime or parallel computing resources.
  • Model structure itself (which states to include, cycle length) is typically fixed and not subject to probabilistic uncertainty, which can be a major source of unquantified structural uncertainty.
  • Results are sensitive to the choice of distributions and their correlations; ignoring correlation between parameters (e.g., costs and utilities) can underestimate or overestimate total variance.

Source conditions do not replace independent scientific review or validation by an analysis engine.

Strengths

  • Explicitly quantifies parameter uncertainty and delivers credible intervals rather than point estimates, supporting transparent decision-making under uncertainty.
  • The probabilistic output (e.g., cost-effectiveness acceptability curves) directly informs value-of-information analyses and budget allocation decisions.
  • State-transition structure is interpretable to non-technical stakeholders, and model logic can be audited cycle by cycle.
  • Naturally accommodates multiple simultaneously uncertain parameters without requiring manual scenario construction.
  • Widely accepted by health technology assessment bodies (NICE, CADTH, PBAC) as the standard for probabilistic sensitivity analysis in economic evaluations.
  • Extensible: individual-level microsimulation and time-inhomogeneous variants can incorporate patient heterogeneity and time-varying transitions within the same framework.

Applications

  • Cost-effectiveness analysis of new pharmaceuticals or medical devices requiring probabilistic sensitivity analysis for HTA submissions.
  • Disease progression modelling in oncology, cardiovascular disease, and infectious disease epidemiology.
  • Long-term outcome projection for chronic conditions such as diabetes or HIV, where patients cycle through multiple health states over years.
  • Health technology assessment submissions to regulatory bodies (NICE, CADTH, EMA) that mandate probabilistic uncertainty reporting.
  • Policy evaluation comparing screening or prevention programs, where cost-per-QALY credible intervals inform budget allocation.
  • Infrastructure or project risk modelling where assets transition through condition states (good, deteriorated, failed) with uncertain transition rates.

How it works

How it works

1

Define the state space: enumerate all mutually exclusive, collectively exhaustive states the modelled entity can occupy (e.g., disease stages, resource levels, project phases). Specify the time cycle length and the maximum number of cycles (time horizon). Each state carries associated payoffs — costs, utilities, or other outcome metrics — that may also be uncertain.

2

For every transition probability and every payoff parameter, specify a probability distribution reflecting uncertainty derived from the evidence base. Transition probabilities are typically drawn from a Dirichlet distribution (ensuring they sum to 1 per origin state); costs from gamma or log-normal distributions; utilities from beta distributions bounded in [0, 1]. This step encodes the full parameter uncertainty structure.

3

Draw one complete set of model parameters from all specified distributions — this constitutes one Monte Carlo iteration. A full parameter vector includes an entire transition matrix, all state-specific costs, and all utilities. Sampling is typically performed using Latin hypercube sampling or simple random sampling, with N ≥ 1000 iterations recommended for stable credible intervals.

4

Run the Markov cohort trace for the sampled parameter set: multiply the starting cohort distribution by the transition matrix repeatedly for each cycle until the time horizon is reached. Record the proportion of the cohort in each state at each cycle. Compute total expected costs and outcomes by multiplying state occupancies by sampled payoffs and summing across all cycles, applying discounting as appropriate.

5

Repeat steps 3–4 for all N iterations. This produces N replications of every aggregate outcome (e.g., total cost, quality-adjusted life years). The collection of N values for each output forms an empirical sampling distribution that directly reflects model uncertainty.

6

Summarise results across iterations: report the mean (or median) and 95% credible interval for each output. Construct cost-effectiveness acceptability curves (CEACs) or rank-acceptability indices if comparing alternatives. Identify which parameters drive the most variance using probabilistic sensitivity analysis (PSA) scatter plots or expected value of perfect information (EVPI) metrics.

Common pitfalls

  • Treating Monte Carlo iteration count as unimportant: fewer than 1000 iterations frequently yields unstable credible intervals, particularly in the tails; always report convergence diagnostics.
  • Ignoring parameter correlations when sampling: independently sampling correlated parameters (e.g., transition probabilities from separate Beta distributions rather than a joint Dirichlet) inflates apparent uncertainty and can violate the constraint that probabilities from a given state sum to one.
  • Conflating stochastic uncertainty (first-order, within-iteration variability across simulated individuals) with parameter uncertainty (second-order, across Monte Carlo iterations); these require different handling in cohort versus individual-level models.
  • Over-interpreting the width of credible intervals as a verdict on model quality rather than an honest reflection of data scarcity.
  • Failing to validate the deterministic base case before introducing stochastic layers — errors in the core model are amplified, not revealed, by probabilistic wrapping.

Frequently asked questions

What is the difference between a stochastic and a deterministic Markov model?

A deterministic Markov model uses fixed point estimates for all transition probabilities and payoffs, producing a single outcome value. A stochastic Markov model samples these parameters from probability distributions in each Monte Carlo iteration, producing a distribution of outcomes that reflects parameter uncertainty. Both use the same cohort-trace mechanics; only the parameter treatment differs.

How many Monte Carlo iterations are sufficient?

A minimum of 1000 iterations is commonly recommended; 10,000 is typical for stable credible intervals and cost-effectiveness acceptability curves. Convergence should be verified visually (running mean plots) or with formal diagnostics. Models with many correlated parameters or rare events may require more iterations.

When does the Markov assumption fail, and how can it be addressed?

The Markov assumption fails when the probability of transitioning out of a state depends on how long a patient has been in it (e.g., increasing mortality risk with disease duration). The standard fix is to introduce tunnel states that track time-in-state explicitly, at the cost of a larger state space. Alternatively, individual-level microsimulation can track each patient's history directly.

Can a stochastic Markov model handle individual heterogeneity?

Standard cohort-level stochastic Markov models capture only parameter uncertainty across iterations, not individual-level variability. To incorporate patient heterogeneity (different baseline risks, comorbidities), the model must be extended to an individual-level microsimulation where each simulated patient is tracked separately through the state space.

How is a stochastic Markov model different from a discrete-event simulation?

In a Markov model, time advances in fixed cycles and all events are governed by constant per-cycle transition probabilities. Discrete-event simulation models individual entities whose events occur at continuous times drawn from arbitrary distributions, allowing more flexible representation of queuing, resource constraints, and time-varying rates. Markov models are simpler to build and validate; DES is more flexible but harder to parameterise.

Related methods

Sources

  1. Sonnenberg, F. A., & Beck, J. R. (1993). Markov models in medical decision making: A practical guide. Medical Decision Making, 13(4), 322–338.

    Open DOI reference
    Copied.
  2. Briggs, A., Sculpher, M., & Claxton, K. (2006). Decision Modelling for Health Economic Evaluation. Oxford University Press.

    ISBN: 9780198526629
    Copied.