What is quantization in AI models?

Quantization is the process of storing a model’s weights using fewer bits of precision — for example, converting 16-bit numbers into 4-bit numbers. This shrinks the model’s file size and memory requirements dramatically, at the cost of a small amount of accuracy, which is why a model that would otherwise need a data-center GPU can run on ordinary consumer hardware once quantized.

AI & ML — INTRODUCTION

What Is Quantization? How Big Models Run on Small Hardware

A 70-billion-parameter model sounds like it should need a data center to run. Quantization is the main reason it doesn’t have to — and it’s the same trick that lets Ollama run serious models on an ordinary laptop.

The Quick Answer

Every number inside a model — its weights — normally gets stored with high precision (16 bits each, or more). Quantization rewrites those same weights using far fewer bits (8, or even 4), the same way a compressed JPEG stores a photo with less data than the original. The file gets dramatically smaller and needs much less memory to run, in exchange for a small, usually barely noticeable, drop in accuracy.

What the Experts Say

llama.cpp — the engine Ollama itself runs on — describes quantization plainly in its own documentation: converting weights from a high-precision format down to fewer bits shrinks the model’s size, at the cost of some accuracy loss that’s typically measured using a metric called perplexity. Their own worked example shows a Llama 3.1 8B model shrinking from 32.1 GB down to 4.9 GB after quantization. See llama.cpp’s quantization documentation.

Real Models, Before and After

These are actual download sizes from Ollama’s own model library — not estimates — comparing each model at full 16-bit precision (FP16) against Ollama’s default 4-bit quantized version (Q4_K_M):

Model	Full precision (FP16)	Quantized (Q4_K_M)	Size reduction
Llama 3.2 3B	6.4 GB	2.0 GB	~3.2x smaller
Llama 3.1 8B	16 GB	4.9 GB	~3.3x smaller
Llama 3.1 70B	141 GB	43 GB	~3.3x smaller

The Llama 3.1 70B row is the clearest example: a model whose full-precision version alone is bigger than most consumer GPUs’ entire storage drive shrinks to something that fits on a single high-end graphics card, once quantized.

Try It Yourself

Pick a model and a precision level, and see how much the download size actually changes.

Fun Fact

Ollama’s Q4_K_M — the default quantization for most models in its library — isn’t the smallest option available. More aggressive levels exist, but Q4_K_M is chosen as the default because it keeps most of the model’s quality while still shrinking it dramatically — the sweet spot rather than the extreme.

Test Yourself

What does quantization actually do to a model?

Based on the real Llama 3.1 70B numbers, roughly how much smaller is the Q4_K_M version than the full-precision (FP16) version?

Ready to see how quality loss actually gets measured, and what the different quantization level names like Q4_K_M mean? Continue to the Intermediate level →