Markov Model — Probabilistic State-Transition Modeling
- Introduced by
- Andrei Markov
- Year
- 1906
- Type
- Probabilistic state-transition model
- Family
- Simulation / optimization
What does it do?
A Markov Model represents a system as a finite set of states and specifies the probability of moving from one state to another at each time step.
By capturing only the current state — not the full history — it enables tractable analysis of complex dynamic processes across health economics, engineering reliability, operations research, and social-science modeling.
The intuition behind the method
Imagine a patient who can be healthy, sick, or deceased. A Markov Model says that the chance of becoming sick tomorrow depends only on whether the patient is healthy today — not on the entire medical history. This memoryless property makes the mathematics tractable while still capturing realistic dynamics. At each step, a transition probability matrix governs how the population distributes across states, and analysts can ask: where does the system end up in the long run, and how quickly does it get there?
History
Andrei Markov introduced the mathematical framework for dependent random sequences in his 1906 study of letter transitions in Pushkin's verse, proving the law of large numbers for dependent variables. The discrete-time Markov chain formalism was extended by Kolmogorov and others in the 1930s. From the 1970s onward, Markov models became standard in operations research, queuing theory, and from the 1980s in health-economic modeling, cementing their role as a foundational tool in applied probability.
When to use it
Use a Markov Model when: (1) a process can be described as transitions among a small number of distinct states; (2) the memoryless assumption is defensible — future transitions depend only on the current state; (3) you need long-run or expected-value summaries (steady-state proportions, expected costs, expected quality-adjusted life years). It is particularly standard in health-technology assessment (HTA) and reliability engineering. Do NOT use when: history strongly affects future transitions (use higher-order chains or semi-Markov models instead); the number of states explodes with problem complexity (consider simulation instead); data for reliable transition-probability estimation are unavailable; or the process involves continuous state spaces without natural discretization.
Limitations and considerations
- The memoryless (Markov) assumption is often violated in practice — sojourn times that follow non-exponential distributions require semi-Markov or phase-type extensions.
- State-space explosion: adding detail to represent history or heterogeneity multiplies the number of states, rapidly increasing model complexity and data requirements.
- Transition probabilities must be estimated from data; sparse data lead to wide uncertainty intervals that can dominate model conclusions.
- Homogeneity assumption: standard Markov chains assume transition probabilities are constant over time, which may be unrealistic for aging cohorts or evolving systems.
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Transparent and auditable: the transition matrix fully specifies the model and can be inspected, validated, and communicated to non-technical stakeholders.
- Analytically tractable: closed-form solutions exist for steady-state distributions, mean passage times, and absorption probabilities.
- Widely accepted in regulatory contexts, especially health-technology assessment (NICE, CADTH guidelines recognize Markov cohort models).
- Flexible time horizon: supports both short-run transient analysis and long-run steady-state queries with the same model.
- Easily extended to Markov Decision Processes (MDPs) when control or policy optimization is needed.
Applications
- Health-technology assessment: modeling disease progression and computing quality-adjusted life years (QALYs) and incremental cost-effectiveness ratios (ICERs).
- Reliability engineering: computing failure rates, maintenance schedules, and mean time between failures for multi-component systems.
- Natural language processing: language models, part-of-speech tagging, and text generation based on n-gram transition probabilities.
- Finance and actuarial science: credit-rating migration models and insurance claim frequency analysis.
- Ecology: population viability analysis and habitat suitability transition modeling.
- Operations management: inventory control and customer lifetime value estimation using purchase-state transition matrices.
How it works
How it works
Enumerate all mutually exclusive, collectively exhaustive states the system can occupy. States must be defined precisely enough that membership is unambiguous at any point in time — for example, disease stages, inventory levels, or operational modes.
Construct the transition probability matrix P where each entry p_ij gives the one-step probability of moving from state i to state j. Every row must sum to 1. Probabilities are estimated from empirical data (cohort studies, historical records) or expert elicitation and should be reported with uncertainty ranges.
Formula
P = [p_{ij}], \quad p_{ij} = P(X_{t+1}=j \mid X_t=i), \quad \sum_j p_{ij} = 1Confirm that the next state depends only on the current state and not on earlier history. This Markov (memoryless) assumption is the model's key requirement. If history matters, consider higher-order Markov chains or semi-Markov extensions.
Formula
P(X_{t+1}=j \mid X_0,\ldots,X_t) = P(X_{t+1}=j \mid X_t)Starting from an initial state distribution vector pi^(0), multiply by P^t to obtain the distribution at time t. This shows how a cohort or system evolves step by step and allows computation of expected time spent in each state, known as sojourn times.
Formula
\pi^{(t)} = \pi^{(0)} P^tFor ergodic chains, solve the stationary distribution equation pi P = pi subject to the probabilities summing to 1. The steady-state vector pi describes the long-run proportion of time the system spends in each state and is a key output for cost-effectiveness and reliability analyses.
Formula
\pi P = \pi, \quad \sum_i \pi_i = 1
Common pitfalls
- Collapsing clinically distinct stages into a single state to simplify the model can mask important differences in costs or outcomes — always validate state definitions against clinical or domain expert judgment.
- Ignoring parameter uncertainty: reporting only point-estimate results without probabilistic sensitivity analysis (PSA) overstates confidence; always propagate uncertainty through the transition matrix.
- Choosing a cycle length that is too coarse relative to the speed of state transitions introduces half-cycle correction issues and biases cumulative estimates.
- Confusing absorbing states (e.g., death) with transient states can cause the model to produce infinite expected costs or times; verify that at least one absorbing state is reachable from every transient state.
Frequently asked questions
How many states should a Markov model have?
Use the fewest states that preserve clinically or operationally meaningful distinctions. More states improve realism but increase data requirements and verification burden. A practical starting point is 3-8 states; beyond 15-20, individual simulation approaches often become preferable.
What if my data suggest transition probabilities change over time?
A time-homogeneous Markov chain assumes constant transition probabilities. If probabilities vary with time or age, use a time-inhomogeneous chain (different P matrix per cycle), a semi-Markov model (sojourn-time-dependent transitions), or individual-level microsimulation.
How do I handle parameter uncertainty?
Fit Dirichlet distributions to each row of the transition matrix using observed counts, then sample from these distributions in a probabilistic sensitivity analysis (PSA). Report results as cost-effectiveness acceptability curves or credible intervals rather than point estimates.
What is the difference between a Markov model and a Markov Decision Process?
A Markov model describes a system passively — transitions are governed by fixed probabilities. A Markov Decision Process (MDP) extends this by adding actions and rewards, enabling optimization of decisions or policies over time.
When should I use a cohort Markov model versus individual-level microsimulation?
Cohort models track proportions across states and are computationally cheap and transparent; prefer them when the Markov assumption holds and heterogeneity is limited. Microsimulation tracks individual histories, making it suitable when memory effects, individual variability, or interaction between individuals matter, at the cost of greater computational and data demands.
Related methods
Sources
Norris, J. R. (1997). Markov Chains. Cambridge University Press, Cambridge.
ISBN: 9780521633963Copied.Markov chain. Wikipedia.
Open publication referenceCopied.