trendingAI & ML — ADVANCED
Ollama in Production: Concurrency, GPU Offloading & Memory Math
A single “ollama run” session hides two production questions: how many requests can hit that model at once, and what happens to speed the moment your model doesn’t fully fit in GPU memory.
The Quick Answer
Two environment variables govern concurrency: OLLAMA_NUM_PARALLEL (how many requests one loaded model handles simultaneously) and OLLAMA_MAX_LOADED_MODELS (how many different models stay resident in memory at once). Required memory scales with parallel slots × context length, so concurrency has a real, calculable memory cost. Separately, if a model doesn’t fully fit in GPU VRAM, Ollama splits it between GPU and system RAM — and that split can cost far more throughput than the layer count alone would suggest.
Concurrency: Two Separate Knobs
Ollama exposes two independent settings for handling multiple requests at once:
OLLAMA_NUM_PARALLEL— how many requests a single already-loaded model processes simultaneously. Defaults to 1 or 4 depending on available memory.OLLAMA_MAX_LOADED_MODELS— how many distinct models can stay loaded in memory at the same time. Defaults to 3 per GPU, or 3 for CPU-only inference.
Every parallel slot needs its own KV-cache, so required memory scales directly with OLLAMA_NUM_PARALLEL × context_length. A worked example, using the same KV-cache formula from the Context Window Advanced page (32 layers, 8 KV heads, head dimension 128, FP16, a modest 4096-token context):
num_layers = 32
num_kv_heads = 8
head_dim = 128
context_length = 4096
bytes_per_element = 2 # FP16
def kv_cache_gb(parallel_slots):
total_bytes = (
2 * num_layers * num_kv_heads * head_dim
* context_length * bytes_per_element * parallel_slots
)
return total_bytes / (1000 ** 3)
for slots in [1, 4, 8]:
print(slots, "parallel slots ->", round(kv_cache_gb(slots), 2), "GB")
# 1 parallel slots -> 0.54 GB
# 4 parallel slots -> 2.15 GB
# 8 parallel slots -> 4.29 GBThe relationship is exactly linear — doubling parallel slots doubles KV-cache memory, with everything else held fixed. If a production deployment has insufficient memory to load a new request while others are loaded, Ollama queues incoming requests rather than failing outright; if the queue itself fills past OLLAMA_MAX_QUEUE (default 512), further requests get rejected with a 503.
GPU Offloading: Not All Layers Are Equal
A model’s transformer layers can run on GPU (fast) or spill over to system RAM, processed by the CPU (much slower) — controlled by num_gpu, the number of layers Ollama is told to place on the GPU. When VRAM can’t hold every layer, the remainder falls back to CPU — and every one of those CPU layers requires shuttling data back and forth over the PCIe bus, which is dramatically slower than VRAM’s own internal bandwidth.
A documented benchmark on Qwen 3 8B (36 transformer layers) shows just how nonlinear this drop-off is: fully offloaded to VRAM, the model ran at roughly 40 tokens/second. With only 25 of those 36 layers on the GPU — still the large majority — throughput fell to around 8 tokens/second, a 5× slowdown from removing barely a third of the layers from VRAM.
Source: “llama.cpp --n-gpu-layers guide: -1, 0, and partial offload”
Drag the slider below to see the shape of that same trade-off — throughput barely changes near full GPU offload, then falls off a cliff once a meaningful chunk of layers spills to system RAM.
Fun Fact
The default 5-minute keep_alive window — the same setting covered in the Prompt Engineering Advanced Ollama case study — can be set to a negative value to mean “never unload,” or exactly 0 to unload the model the instant a response finishes generating — the same parameter covers both extremes.
Test Yourself
What does OLLAMA_NUM_PARALLEL control?
What happens to required memory as OLLAMA_NUM_PARALLEL increases?
In the documented Qwen 3 8B benchmark, what happened to throughput when only 25 of 36 layers fit in VRAM?
Why does partial GPU offloading slow inference down so much?
What is the default value of keep_alive if not otherwise configured?