Update every weight, or a carefully chosen few.
Full fine-tuning updates every parameter in the model. Parameter-efficient fine-tuning (PEFT) freezes the base and trains a small set of added parameters instead. The gap between them is narrower than most teams expect, and the operational difference is large.
Reach for PEFT first. It is cheaper, faster to iterate on, and produces adapters you can swap per customer or per task. Escalate to full fine-tuning when PEFT has plateaued below your target and you have ruled out data quality as the cause — which, in practice, is where most plateaus actually come from.
| Dimension | Full Fine-Tuning | PEFT (LoRA family) |
|---|---|---|
| Parameters updated | All of them | A small added subset |
| GPU memory | High — weights, gradients and optimiser state | Much lower |
| Artefact size | A full model checkpoint | A small adapter file |
| Serving many variants | A full copy per variant | One base, many hot-swappable adapters |
| Catastrophic forgetting | Higher risk | Lower — base weights are frozen |
| Ceiling on adaptation | Highest | High, but bounded by adapter capacity |
| Iteration speed | Slow and expensive per run | Fast enough to run many experiments |
On most adaptation tasks, very little — often within noise of full fine-tuning. The gap widens when the target domain is far from what the base model saw in pre-training.
Yes, and it is one of the strongest arguments for PEFT. One set of base weights in memory can serve many adapters, which makes per-customer models economically viable.
Check the data first. A plateau far more often reflects label noise, insufficient examples or a mismatched task framing than adapter capacity. Full fine-tuning on bad data just fits the noise faster.