LoRA in Production: Rank Selection, Merging, and Serving Multiple Adapters

The Actual LoRA Update Rule

Hu et al. define the modified forward pass, for a frozen pretrained weight matrix W₀ of shape d × k, as:

h = W0 x + delta_W x = W0 x + (B A) x

where B has shape d × r, A has shape r × k, and r (the rank) is chosen to be much smaller than either d or k. The product BA is scaled by α/r (a constant α divided by the rank) before being added — the paper notes tuning α behaves roughly like tuning a learning rate, and that this scaling reduces how much hyperparameters need retuning when rank is changed. Only B and A ever receive gradients — W₀ stays frozen throughout training, exactly as covered in the Intermediate level.

Worked as illustrative Python, with small, checkable numbers rather than a real transformer’s actual weights:

import numpy as np

d, k, r = 4, 4, 2          # tiny illustrative shapes; real layers use much larger d, k
alpha = 8                  # constant chosen alongside r, per Hu et al.
scaling = alpha / r        # scaling = alpha / r, applied to B @ A

W0 = np.array([            # frozen original weight matrix (never updated)
    [0.10, 0.20, 0.00, 0.30],
    [0.05, 0.15, 0.10, 0.00],
    [0.00, 0.10, 0.20, 0.05],
    [0.20, 0.00, 0.05, 0.10],
])

B = np.zeros((d, r))       # LoRA convention: B starts at all zeros
A = np.random.randn(r, k) * 0.01   # A starts small-random (Hu et al.'s init scheme)

# At initialization, B @ A is all zeros, so W_new == W0 exactly —
# training starts from the frozen model's original behavior, unchanged.
delta_W = (B @ A) * scaling
W_new = W0 + delta_W
assert np.allclose(W_new, W0)      # true only before any training step

# After some training updates B and A (illustrative, not a real gradient step):
B_trained = np.array([[0.4, -0.1], [0.0, 0.3], [-0.2, 0.1], [0.1, 0.2]])
A_trained = np.array([[0.5, -0.3, 0.2, 0.1], [0.1, 0.4, -0.2, 0.3]])

delta_W_trained = (B_trained @ A_trained) * scaling
W_new_trained = W0 + delta_W_trained
# W_new_trained is now W0 plus a rank-2 correction — never a full d x k
# matrix's worth of independently trained numbers.

Two details in that code matter beyond the shapes. First, B is initialized to all zeros (with A started small and random), so B @ A is exactly zero before training starts — the adapted model behaves identically to the original frozen model on step zero, and only diverges as training updates B and A. Second, the product of a d × r matrix and an r × k matrix is mathematically constrained to be at most rank r — this is the literal reason the technique is called Low-Rank Adaptation.

Source: Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models,” arXiv:2106.09685

What Rank Actually Controls

Rank (r) is the shared inner dimension of B (d × r) and A (r × k). Every other dimension in those matrices is fixed by the layer’s existing shape — rank is the one number an engineer actually chooses, and it directly sets both matrices’ total parameter count and how rich a correction they can represent:

Hu et al.’s own ablation in the LoRA paper found that increasing rank well beyond small values often produced little additional benefit for the tasks they tested — a finding that has made small ranks (roughly 4 to 64) a common practical starting range, rather than something requiring a search across hundreds of possible values. Try different ranks below and see exactly how the adapter matrices’ dimensions and parameter count change:

Merging Into the Base Model vs. Keeping Adapters Separate

Once B and A are trained, there are two ways to deploy them. Merging computes W_new = W_original + (B @ A) * scaling once, ahead of time, and saves the result as a single ordinary weight matrix — at inference time there is no adapter left to apply, just one dense matrix identical in shape to the original. Keeping the adapter separate means storing B and A apart from W₀, and computing the adapted output at every forward pass.

The real trade-off is concrete, not abstract: a customer support deployment that only ever needs one fine-tuned behavior can merge and get zero-overhead inference. A platform serving many customers, each with their own task-specific adapter trained against the same base model, needs to keep adapters separate — merging would mean maintaining a full separate copy of the entire base model’s weights per customer, instead of one shared frozen base plus many small adapters swapped in per request.

When Full Fine-tuning Is Still Worth It

LoRA’s efficiency comes from a real constraint: the update it can apply to any given weight matrix is mathematically limited to rank r, no matter how it’s trained. For tasks where the needed change in behavior is a narrow adjustment to what the pretrained model can already mostly do, that constraint costs little — which is why LoRA performs on par with full fine-tuning on many of the tasks Hu et al. tested. But for tasks that require a genuinely large representational shift from the pretrained model’s existing behavior — learning something the base model’s pretraining gave it little foundation for — a low-rank update may simply not have enough capacity to represent the needed change, however it’s trained. In that situation, full fine-tuning’s ability to move every weight independently is a real advantage worth its much higher training cost, not just a more expensive way to reach the same result LoRA already gets.

Fun Fact

Because B is initialized to all zeros, a freshly created (untrained) LoRA adapter changes nothing about the model it’s attached to — B @ A is exactly zero before a single training step runs. Every LoRA adapter starts life as a mathematically perfect no-op.

Test Yourself