What does LoRA actually do differently from full fine-tuning?

LoRA freezes every original weight in the pretrained model and never updates them. Instead, for chosen weight matrices, it adds two small trainable matrices (commonly called A and B) alongside the frozen original, and only those small matrices get trained. The paper that introduced this, Hu et al.'s "LoRA: Low-Rank Adaptation of Large Language Models" (arXiv:2106.09685), reports that compared to full fine-tuning of GPT-3 175B with Adam, this reduces the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times.

What is QLoRA and how is it different from LoRA?

QLoRA (Dettmers et al., "QLoRA: Efficient Finetuning of Quantized LLMs," arXiv:2305.14314) combines LoRA with a quantized, frozen base model: the original weights are compressed to 4-bit precision (using a technique the paper calls 4-bit NormalFloat) before LoRA adapters are trained on top of them. The paper reports this reduces memory usage enough to fine-tune a 65-billion-parameter model on a single 48GB GPU, and their resulting Guanaco model reached 99.3% of ChatGPT's performance on the Vicuna benchmark after only 24 hours of fine-tuning on one GPU.

AI & ML — INTERMEDIATE

LoRA and Parameter-Efficient Fine-tuning: Why You Don’t Need to Retrain Everything

The Fine-tuning Basics Intermediate level covers what fine-tuning is and when it beats prompting alone. This page assumes that context and goes one layer deeper: full fine-tuning updates every single weight in a model, which for a large model means enormous GPU memory just to hold gradients and optimizer state for billions of parameters. LoRA (Low-Rank Adaptation) is the technique that made fine-tuning large models practical on far more modest hardware — by changing what actually gets trained.

The Quick Answer

LoRA freezes every original weight in the pretrained model and trains a small pair of low-rank matrices alongside each chosen weight instead. Because those adapter matrices are tiny compared to the full weight matrix, LoRA trains a dramatically smaller number of parameters — Hu et al. report a 10,000x reduction in trainable parameters and a 3x reduction in GPU memory versus full fine-tuning of GPT-3 175B with Adam. QLoRA pushes this further by also quantizing the frozen base model to 4-bit precision, which is what let Dettmers et al. fine-tune a 65B-parameter model on a single 48GB GPU.

What LoRA Actually Freezes and Trains

A pretrained model’s weights live in large matrices — for example, the matrix that projects a token’s representation into a query or value vector inside a self-attention layer. Full fine-tuning updates every number in that matrix directly. LoRA does something different: it leaves the original matrix completely untouched (frozen) and adds a separate, much smaller update computed from two new matrices trained from scratch. Those two matrices are far smaller than the original because they share one small dimension called the rank— this is the “low-rank” part of Low-Rank Adaptation, covered in full mechanical detail in the Advanced level.

The paper that introduced this, Edward Hu and colleagues’ “LoRA: Low-Rank Adaptation of Large Language Models”, reports a concrete real-world figure for exactly how much smaller this makes the training problem: compared to full fine-tuning of GPT-3 175B with the Adam optimizer, LoRA reduces the number of trainable parameters by 10,000 times and the GPU memory requirement during training by 3 times.

Source: Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models,” arXiv:2106.09685

Why Training Fewer Parameters Is Dramatically Cheaper

The saving isn’t just about the trained matrices being smaller files — it changes what has to sit in GPU memory during training. Training a parameter with a gradient-based optimizer like Adam requires storing, per trainable parameter: the parameter itself, its gradient, and (for Adam specifically) two additional running-average states used to adapt the learning rate. Full fine-tuning pays that four-times-over memory cost for every single weight in the model. LoRA only pays it for the small adapter matrices — the frozen original weights still have to be loaded into memory to run the forward pass, but since they never receive gradients or optimizer state, they cost far less to keep around during training.

That’s the real mechanism behind the 3x memory reduction Hu et al. report: it’s smaller than the 10,000x parameter reduction because the frozen base weights still occupy memory — LoRA doesn’t shrink the model that gets loaded, only the part of it that gets trained.

See how this plays out at different model sizes, using the real reduction ratios Hu et al. report:

QLoRA: Quantizing the Frozen Base Model Too

LoRA already shrinks the training problem by freezing the base model. QLoRA (Tim Dettmers and colleagues, “QLoRA: Efficient Finetuning of Quantized LLMs”) asks a follow-up question: since those frozen weights never get updated, do they even need to sit in memory at full precision? QLoRA’s answer is no — it quantizes the frozen base model down to 4-bit precision (using a technique the paper calls 4-bit NormalFloat, designed specifically for how pretrained weights are distributed), then trains LoRA adapters on top of that quantized, frozen base. The mechanics of quantization itself — turning full-precision weights into lower-bit integers, and what’s actually lost doing it — are covered in full in Quantization Advanced; this page only covers what changes when a quantized model becomes the frozen base for LoRA training.

The reported result is concrete: QLoRA reduces memory usage enough to fine-tune a 65-billion-parameter model on a single 48GB GPU, while the paper reports preserving full 16-bit fine-tuning task performance. Using QLoRA, the authors trained a model family they called Guanaco, which reached 99.3% of ChatGPT’s performance level on the Vicuna benchmark after just 24 hours of fine-tuning on a single GPU.

Source: Dettmers et al., “QLoRA: Efficient Finetuning of Quantized LLMs,” arXiv:2305.14314

Fun Fact

The name “QLoRA” and the way it’s built reads almost like a checklist of this site’s own topics stacked on top of each other: quantization (compressing the frozen base model) plus LoRA (training small adapters instead of the full model), combined into one fine-tuning recipe. Neither technique replaces the other — QLoRA just applies both at once, to two different parts of the same problem.

Test Yourself

What happens to the original pretrained model weights when fine-tuning with LoRA?

According to Hu et al.'s LoRA paper, compared to full fine-tuning of GPT-3 175B with Adam, what does LoRA reduce?

What does QLoRA add on top of LoRA, according to the Dettmers et al. paper?

Ready for the real matrix math behind the LoRA update, how rank actually controls the adapter’s size and expressiveness, merging adapters versus keeping them separate, and when full fine-tuning is still worth its extra cost? Continue to the Advanced level →