Mixture of Experts
- Introduced by
- Shazeer, N. et al.
- Year
- 2017
- Type
- Sparse neural network architecture (conditional computation)
What does it do?
Mixture of Experts (MoE) is a sparse neural-network architecture, introduced by Shazeer and colleagues in 2017 with the sparsely-gated MoE layer, in which only a subset of expert sub-networks is activated for each input.
As seen in models such as Switch Transformer and Mixtral, it holds computation cost fixed even as the total parameter count grows.
The intuition behind the method
Instead of running every input through one huge network, MoE keeps a panel of specialised expert networks and a router that decides which few experts each input should consult. Because only a handful of experts fire per input, the model can carry an enormous number of parameters while doing roughly the same amount of work per example — much like consulting only the two specialists relevant to a question rather than the whole hospital.
History
The sparsely-gated Mixture-of-Experts layer was introduced by Shazeer and colleagues in 2017 in 'Outrageously Large Neural Networks', which showed that conditional computation could scale model capacity by orders of magnitude while keeping compute roughly constant. The idea was later carried into transformer language models — notably Switch Transformer and, in 2024, Mixtral — establishing MoE as a standard route to very large yet efficient models.
When to use it
Use MoE for large-scale prediction or classification on text and continuous-feature data when you have substantial data (about 1,000 observations or more) and large-scale training infrastructure with a GPU cluster. It assumes a router balancing loss is applied and that the training pipeline can support sparse gating. Below about 1,000 examples the router cannot learn balanced expert selection and training is unstable; below about 500 the model overfits and a single dense model is preferable.
Limitations and considerations
- Requires large-scale training infrastructure and a GPU cluster.
- Needs a router balancing loss; without it, load collapses onto a few experts.
- On small data (n below about 1,000) the router cannot balance expert selection and training is unstable.
- With very little data (n below about 500) the model overfits and a single dense model is enough.
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Decouples model capacity from per-input compute — total parameters can grow while computation per example stays fixed.
- Experts specialise, letting one architecture cover heterogeneous inputs.
- Proven at scale in systems such as Switch Transformer and Mixtral.
- Top-K sparse routing keeps inference cost far below that of a dense model of equal parameter count.
Applications
- Large language models that scale parameter count without scaling inference cost (e.g. Mixtral).
- Large-scale text prediction and classification.
- Prediction tasks over high-dimensional continuous features where capacity must grow efficiently.
- Conditional-computation architectures that route heterogeneous inputs to specialised sub-networks.
How it works
How it works
The layer holds many parallel expert sub-networks, each a small feed-forward network. The experts are trained to specialise, so different inputs end up routed to different experts. Only the selected experts process a given input, which is why total capacity can grow without growing the per-input compute.
A gating network (router) scores the experts for each input and keeps only the top-K. A softmax over those top-K scores produces the routing weights. During large-scale training a router balancing loss is applied so that load is spread across experts rather than collapsing onto a few favourites.
Formula
G(x) = \text{Softmax}(\text{TopK}(x \cdot W_g))The outputs of the selected experts are combined as a weighted sum, using the gating weights as coefficients. This sparse combination is what keeps computation constant while the parameter count — and thus model capacity — scales up.
Formula
y = \sum_{i \in \text{TopK}} G(x)_i \, E_i(x)Common pitfalls
- Omitting the router balancing loss, so a few experts dominate and the rest stay idle.
- Applying MoE to small datasets where a single dense model would train more reliably.
- Underestimating the GPU and infrastructure footprint required for sparse training.
- Treating expert count as free capacity without accounting for routing instability.
Frequently asked questions
Why does MoE add parameters without adding compute?
Only the top-K experts selected by the router run for each input, so even though the layer stores many experts, each example touches only a few. Total capacity scales with the number of experts while per-input computation stays roughly fixed.
What is the router balancing loss for?
Left alone, the router tends to send most inputs to a small number of experts, leaving the others untrained. A balancing loss penalises uneven load so that traffic is spread across experts and the full capacity is actually used.
How much data does MoE need?
It is intended for large-scale settings — roughly 1,000 examples or more. Below that the router cannot learn balanced expert selection and training becomes unstable; with under about 500 examples a single dense model overfits less and is the safer choice.
Do I need special hardware?
Yes. MoE assumes large-scale training infrastructure and a GPU cluster, since the many experts and sparse routing are designed for distributed, high-throughput training.
Related methods
Sources
Shazeer, N. et al. (2017). Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. ICLR. arXiv:1701.06538
Open publication referenceCopied.Jiang, A.Q. et al. (2024). Mixtral of Experts. arXiv.
Open publication referenceCopied.