Quantization Math & Why It Makes Inference Faster

The Actual Math: Scale and Zero-Point

Quantizing a set of weights to b bits means mapping them onto integers from 0 to 2^b − 1. Two numbers make that mapping reversible: a scale (the real-valued gap each integer step represents) and a zero-point (which integer represents real value zero).

import numpy as np

def quantize(weights, bits=8):
    qmin, qmax = 0, (2 ** bits) - 1
    w_min, w_max = weights.min(), weights.max()
    scale = (w_max - w_min) / (qmax - qmin)
    zero_point = round(qmin - w_min / scale)
    quantized = np.round(weights / scale + zero_point).clip(qmin, qmax).astype(np.int32)
    return quantized, scale, zero_point

def dequantize(quantized, scale, zero_point):
    return (quantized.astype(np.float32) - zero_point) * scale

weights = np.array([-0.42, -0.15, 0.03, 0.18, 0.35, 0.61, -0.08, 0.02], dtype=np.float32)
quantized, scale, zero_point = quantize(weights, bits=8)
restored = dequantize(quantized, scale, zero_point)

print(quantized)
# [  0  67 111 149 191 255  84 109]
print(round(float(scale), 6), zero_point)
# 0.004039 104
print(np.round(restored, 4))
# [-0.4201 -0.1495  0.0283  0.1818  0.3514  0.6099 -0.0808  0.0202]
print(round(float(np.max(np.abs(weights - restored))), 5))
# 0.00176  -- max error at 8-bit

Every quantized weight in this example is off from its original value by less than 0.002 — small enough that it barely affects the model’s output. Try the same math at lower bit-widths below, and watch the error grow as fewer integers are available to represent the same range.

Mixed Precision: Not Every Weight Is Equal

The Intermediate level introduced k-quants as a method that assigns different precision to different parts of a model rather than quantizing everything uniformly. The reasoning is straightforward: some tensors contribute disproportionately to output quality, so spending a few extra bits on those specific parts — while quantizing the rest more aggressively — recovers most of the quality of a fully higher-precision model at a fraction of the size. This is exactly the trade-off encoded in a name like Q4_K_M: it isn’t uniformly 4-bit throughout, even though the name leads with “Q4.”

The Real Speed Numbers

llama.cpp’s own published benchmark for Llama 3.1 8B shows something that isn’t obvious from the size numbers alone — text generation speed actually improves with quantization:

The reason is memory bandwidth, not arithmetic. Generating one token at a time requires reading the entire model’s weights from memory for each step — the same GPU offloading bottleneck covered in Ollama Advanced. A quantized model has fewer bytes to move per weight, so more of it can be streamed through memory each second, directly speeding up generation — independent of whatever raw arithmetic speedup lower-precision math might also provide.

Source: llama.cpp’s quantization documentation.

Fun Fact

In that same benchmark, prompt processing speed — reading and understanding the input you send, before generating anything — actually got very slightly slower at Q4_K_M than at F16 (821.81 tok/s vs. 923.49 tok/s). Prompt processing is compute-bound rather than memory-bound, so quantization’s memory-bandwidth advantage doesn’t apply there the way it does to text generation.

Test Yourself