Same adapter maths. Very different memory bills.
LoRA and QLoRA both freeze the base model and train a small set of low-rank adapter matrices. The difference is what precision the frozen base is held at during training — and that single choice decides which GPU you need.
Start with QLoRA if the model does not comfortably fit on your GPU in 16-bit. Use LoRA when it does — you get slightly faster steps and one less moving part. Adapter quality between the two is close enough that memory, not accuracy, is usually the deciding factor.
| Dimension | LoRA | QLoRA |
|---|---|---|
| Base model precision | 16-bit (bf16/fp16), frozen | 4-bit NF4 quantised, frozen |
| Trained parameters | Low-rank adapters only | Low-rank adapters only |
| Relative GPU memory | Baseline | Substantially lower — this is the whole point |
| Training step speed | Faster per step | Slower per step from de-quantisation overhead |
| Typical use | Model already fits in VRAM | Large model on a single smaller GPU |
| Merging adapters | Merge straight back into 16-bit weights | Merge into a de-quantised copy of the base |
| Main risk | Runs out of memory on big models | Quantisation error on sensitive tasks |
Usually the gap is small, but it is task-dependent. The honest answer is to run both on your own eval set — the difference that matters is the one on your data, not a benchmark average.
Yes, but you merge into a de-quantised copy of the base, not into the 4-bit weights. Budget the memory for that step.
Rank is a separate axis from LoRA vs QLoRA. Start low, raise it only if training loss plateaus above where you need it — higher rank costs memory and rarely fixes a data problem.