LoRA and PEFT
- Introduced by
- Hu, E. J. et al.; Lester, B. et al.
- Year
- 2022
- Type
- Parameter-efficient fine-tuning of large pretrained models
What does it do?
LoRA (Low-Rank Adaptation), introduced by Hu et al. in 2022, and the broader family of parameter-efficient fine-tuning (PEFT) methods adapt large pretrained language models to new tasks by training only a small number of extra parameters instead of every weight in the model.
This makes fine-tuning possible with far less GPU memory and compute while leaving the original model largely untouched.
The intuition behind the method
Fully fine-tuning a large language model means updating billions of weights, which is expensive in memory and compute and produces a full copy of the model for every task. The key insight behind LoRA is that the change a task requires is usually low-rank: it can be captured by a small pair of matrices rather than a dense update to every weight. PEFT methods exploit this by freezing the original model and training only a tiny set of new parameters, so a single base model can be cheaply specialised to many tasks.
History
LoRA was introduced by Hu et al. in 2022, proposing that the weight update needed to adapt a large language model can be represented as a low-rank decomposition trained in place of full fine-tuning. It built on earlier parameter-efficient ideas, including the prompt tuning of Lester et al. (2021), which showed that adapting only a small set of soft-prompt parameters becomes increasingly competitive as model scale grows. Together these methods established parameter-efficient fine-tuning as the standard way to specialise large pretrained models cheaply.
When to use it
Use LoRA and PEFT when you want to adapt a large pretrained language model to a specific text classification or prediction task but cannot afford the memory and compute of full fine-tuning. A large pretrained model must already be available, task-specific data should be prepared, and a GPU is recommended. These methods work best with a substantial task dataset; with very small datasets (below a few hundred examples) the adaptation tends to overfit and a simpler model such as random forest or gradient boosting is safer.
Limitations and considerations
- Requires a large pretrained model to start from; it does not train a model from scratch.
- A GPU is recommended, and task-specific data must be prepared in advance.
- On small task datasets (below about 500 examples) the fine-tuning may not provide enough adaptation and risks overfitting.
- With very little task data (below about 50 examples) even parameter-efficient fine-tuning becomes meaningless.
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Drastically lower GPU memory and compute than full fine-tuning, since only a small set of parameters is trained.
- The original pretrained model is frozen and reused across tasks, so a single base model can be specialised many times.
- The learned low-rank update can be merged back into the weights, adding no inference-time overhead.
- Covers a family of options — LoRA, prefix tuning, prompt tuning — so the adaptation can be placed where it works best.
Applications
- Adapting a large pretrained language model to a domain-specific text classification task.
- Specialising one base model to many tasks without storing a full copy per task.
- Fine-tuning under limited GPU memory where full fine-tuning is infeasible.
- Text prediction tasks that build on an existing pretrained language model.
How it works
How it works
A large model that has already been pretrained is taken as a fixed starting point. Instead of fine-tuning all of its weights, the original parameters are frozen and only a small set of additional parameters is introduced and trained for the target task. LoRA, prefix tuning, and prompt tuning are the principal methods in this family, each adding parameters in a different place.
In LoRA, the update to a frozen weight matrix is constrained to be low-rank: it is written as the product of two small matrices, so only those small matrices are trained while the original weights stay fixed. Because the rank is much smaller than the weight dimensions, the number of trainable parameters drops dramatically, which is what cuts the memory and compute cost of fine-tuning.
Formula
W = W_0 + \Delta W = W_0 + BA,\quad B \in \mathbb{R}^{d\times r},\ A \in \mathbb{R}^{r\times k},\ r \ll \min(d,k)Only the small added parameters are optimised on the task-specific data, using far less GPU memory than full fine-tuning. The base model is reused unchanged across tasks, and the learned low-rank update can be merged back into the weights for inference. A GPU is recommended, and a reasonably sized task dataset is needed for the adaptation to be meaningful.
Common pitfalls
- Applying LoRA/PEFT to a tiny dataset and expecting reliable adaptation instead of falling back to a classical model.
- Treating parameter-efficient fine-tuning as a substitute for having a suitable pretrained base model.
- Ignoring overfitting risk on small task data rather than monitoring validation performance.
- Assuming the method removes the need for a GPU when one is recommended.
Frequently asked questions
What is the difference between LoRA and PEFT?
PEFT (parameter-efficient fine-tuning) is the broad family of methods that adapt a large model by training only a small number of extra parameters. LoRA is one specific PEFT method that trains a low-rank update to the weights; prefix tuning and prompt tuning are other members of the same family.
How much data do I need?
These methods need a reasonably sized task dataset. Below about 500 examples the adaptation may be insufficient and prone to overfitting, and below about 50 examples parameter-efficient fine-tuning becomes meaningless — a classical model is the better choice there.
Do I need a GPU?
A GPU is recommended. Although PEFT methods use far less memory than full fine-tuning, training the added parameters on a large pretrained model still benefits substantially from GPU acceleration.
Why is the update low-rank?
The change required to adapt a pretrained model to a new task can usually be captured by a small low-rank correction rather than a dense update to every weight. Representing it as the product of two small matrices is what makes the number of trainable parameters — and the cost — drop dramatically.
Related methods
Sources
Hu, E. J. et al. (2022). LoRA: Low-Rank Adaptation of Large Language Models. ICLR.
Open publication referenceCopied.Lester, B. et al. (2021). The Power of Scale for Parameter-Efficient Prompt Tuning. EMNLP.
Open DOI referenceCopied.