trendingAI & ML — INTERMEDIATE
Quantization Levels Explained: What Q4_K_M Actually Means
The Introduction level showed that quantization shrinks a model. It didn’t explain how “how much quality is lost” actually gets measured, or what all those quantization level names — Q8_0, Q4_K_M, Q2_K — are actually telling you.
The Quick Answer
Quality loss from quantization is typically measured with perplexity — the same metric covered in How LLMs Work Advanced— comparing how well the model predicts text before and after quantization. There are two ways to get there: post-training quantization (quantize a model after it’s already trained — fast and simple) or quantization-aware training (simulate quantization during training itself — slower, but usually preserves more quality). And a name like Q4_K_Misn’t arbitrary — every part of it describes a real design decision about how that specific quantized file was built.
Measuring Quality Loss: Perplexity
Perplexity measures how “surprised” a model is by real text — lower is better, meaning the model predicted the actual next words more confidently. Quantizing a model almost always raises its perplexity at least slightly, since some precision is genuinely lost. The goal of a good quantization method isn’t to avoid that increase entirely — it’s to keep it small enough that the difference is barely noticeable in practice, while still shrinking the file dramatically.
Post-Training Quantization vs. Quantization-Aware Training
- 1
Post-training quantization (PTQ). Take a model that’s already fully trained, and convert its weights to lower precision afterward. This is what almost every quantized model you download — including everything in Ollama’s library — actually is. It’s fast, requires no retraining, and works on any existing model.
- 2
Quantization-aware training (QAT). Simulate the effect of quantization while the model is still being trained, so the model’s weights adjust to compensate for the precision loss before it ever happens for real. This usually preserves more quality than PTQ at the same bit-width, but it’s far more expensive — it requires access to training infrastructure and data, not just the finished model.
The practical difference: PTQ is why you can quantize any model you download in minutes. QAT is why some providers release a dedicated “quantized” variant of a model trained specifically to hold up well at low precision, rather than just quantizing the standard release after the fact.
Decoding the Name: Q4_K_M
Every quantized GGUF file (the format Ollama uses) has a name that encodes real information:
Q4— the weights are stored at roughly 4 bits each (other common values: Q8, Q6, Q3, Q2).K— “k-quants,” a llama.cpp quantization method that doesn’t treat every weight identically — some parts of the model get assigned higher precision than others.M— small, medium, or large, describing how aggressively that mixed-precision strategy is applied.Mis the commonly recommended default, balancing size against quality.
Source: llama.cpp, GitHub Discussion #2094 and llama.cpp’s quantization documentation.
Real benchmark figures for a single model, across five common quantization levels — pick one to compare:
Fun Fact
“K-quants” aren’t named after a person or a company — the K refers to the method itself. According to llama.cpp’s own maintainers, k-quantizations are designed to be strictly better than the older, uniform quantization methods at the same file size, which is why nearly every quantized model published today uses a K-quant name rather than the older naming scheme.
Test Yourself
What is commonly used to measure how much quality a model loses from quantization?
What is the key difference between post-training quantization (PTQ) and quantization-aware training (QAT)?
In the name Q4_K_M, what does the "K" refer to?
Ready for the actual quantization math, mixed-precision layer decisions, and why quantized models run faster on the same hardware? Continue to the Advanced level →