SHAP (SHapley Additive exPlanations)
- Introduced by
- Lundberg, S.M. & Lee, S.-I.
- Year
- 2017
- Type
- Model-explanation method (Shapley-value attribution)
What does it do?
SHAP is a model-explanation method, introduced by Scott Lundberg and Su-In Lee in 2017, that uses Shapley values from cooperative game theory to measure how much each feature contributes to an individual prediction, making the output of black-box machine-learning models interpretable.
It supports both global explanations (overall feature importance) and local explanations (why one specific prediction came out the way it did).
The intuition behind the method
A trained model such as a random forest or a deep network turns a row of features into a prediction, but it does not tell you why. SHAP borrows an idea from game theory: imagine the features are players cooperating to produce the prediction, and ask how the 'payout' — the gap between this prediction and the average prediction — should be fairly divided among them. Each feature's fair share is its SHAP value, so you can read off which features pushed a prediction up and which pulled it down, for a single case or across the whole dataset.
History
SHAP was introduced by Scott Lundberg and Su-In Lee in 2017 as a unified approach to interpreting model predictions, connecting several earlier explanation methods to the Shapley values of cooperative game theory. The accompanying TreeSHAP algorithm made exact, fast explanations practical for tree-based ensembles, and SHAP has since become a standard tool for opening up black-box models in fields from health and economics to the natural and behavioural sciences.
When to use it
Use SHAP when you already have a trained black-box model (such as XGBoost, random forest, or deep learning) on cross-sectional or panel data with continuous, categorical, or binary features, and you need to explain its predictions rather than just report accuracy. It works at both the global level (which features matter overall) and the local level (why one case was predicted as it was). At least about 30 observations are needed; below that, SHAP values become unstable and a directly interpretable model such as a decision tree is safer.
Limitations and considerations
- Requires an already-trained ML model; SHAP explains a model, it does not build one.
- On small samples (n below about 30) SHAP values are unstable and feature contributions become unreliable.
- KernelSHAP is model-agnostic but computationally slow, especially on large datasets or many features.
- Attributions describe the model's behaviour, not the true causal effect of a feature in the world.
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Turns black-box models into interpretable ones by attributing each prediction to its features.
- Provides both global feature importance and local single-prediction explanations from one framework.
- Grounded in Shapley values from game theory, giving the attributions a principled, additive fairness basis.
- TreeSHAP gives fast, exact explanations for tree-based models such as random forest and XGBoost.
- KernelSHAP is model-agnostic, so the same approach applies to almost any trained model.
Applications
- Explaining individual predictions of black-box models for clinical, financial, or risk decisions.
- Ranking overall feature importance to understand what a trained model relies on.
- Auditing models for fairness and unexpected dependence on sensitive features.
- Communicating model behaviour to reviewers and stakeholders who expect interpretable results.
How it works
How it works
SHAP rests on a trained black-box model — for example XGBoost, a random forest, or a deep network. The goal is not to refit anything but to explain what that model has already learned, attributing its predictions to the input features.
Each prediction is decomposed additively: it equals a baseline value (the average model output) plus the contributions of all the features. Every feature receives a SHAP value, its share of the difference between this prediction and the baseline, and those shares add up exactly to the prediction.
Formula
f(x) = \phi_0 + \sum_{i=1}^{M} \phi_iThe shares are computed as Shapley values, which average a feature's marginal contribution over all the ways features can be added to the prediction. TreeSHAP computes these exactly and quickly for tree-based models, while the model-agnostic KernelSHAP works for any model but is slower.
The resulting SHAP values can be read locally — explaining a single prediction by showing which features raised or lowered it — or aggregated across the dataset to give a global ranking of feature importance, turning a black-box model into something a reviewer can inspect.
Common pitfalls
- Computing SHAP values on a dataset too small (n below ~30) to give stable attributions.
- Reading SHAP values as causal effects rather than as explanations of the model's predictions.
- Using slow KernelSHAP on large data when TreeSHAP would give exact results far faster for tree models.
- Explaining a poorly fitted model and trusting the attributions of a model that does not predict well.
Frequently asked questions
Do I need to retrain my model to use SHAP?
No. SHAP explains a model you have already trained — such as XGBoost, a random forest, or a deep network — by attributing its existing predictions to the input features. It does not refit or change the model.
What is the difference between TreeSHAP and KernelSHAP?
TreeSHAP computes exact SHAP values quickly for tree-based models like random forest and XGBoost. KernelSHAP is model-agnostic and works for any model, but it is an approximation and is considerably slower, especially on large data.
Can SHAP explain a single prediction?
Yes. That is its local mode: for one case it shows how much each feature pushed the prediction above or below the baseline, with the contributions adding up exactly to the prediction. Aggregating these across the dataset gives global feature importance.
How much data do I need?
At least about 30 observations. Below that, SHAP values become unstable and feature contributions unreliable, so a directly interpretable model such as a decision tree is the safer choice.
Related methods
Sources
Lundberg, S.M. & Lee, S.-I. (2017). A Unified Approach to Interpreting Model Predictions. Advances in Neural Information Processing Systems, 30, 4766–4777.
Open publication referenceCopied.