Ollama in Production: Concurrency, GPU Offloading & Memory Math

Concurrency: Two Separate Knobs

Ollama exposes two independent settings for handling multiple requests at once:

• OLLAMA_NUM_PARALLEL — how many requests a single already-loaded model processes simultaneously. Defaults to 1 or 4 depending on available memory.
• OLLAMA_MAX_LOADED_MODELS — how many distinct models can stay loaded in memory at the same time. Defaults to 3 per GPU, or 3 for CPU-only inference.

Every parallel slot needs its own KV-cache, so required memory scales directly with OLLAMA_NUM_PARALLEL × context_length. A worked example, using the same KV-cache formula from the Context Window Advanced page (32 layers, 8 KV heads, head dimension 128, FP16, a modest 4096-token context):

num_layers = 32
num_kv_heads = 8
head_dim = 128
context_length = 4096
bytes_per_element = 2  # FP16

def kv_cache_gb(parallel_slots):
    total_bytes = (
        2 * num_layers * num_kv_heads * head_dim
        * context_length * bytes_per_element * parallel_slots
    )
    return total_bytes / (1000 ** 3)

for slots in [1, 4, 8]:
    print(slots, "parallel slots ->", round(kv_cache_gb(slots), 2), "GB")

# 1 parallel slots -> 0.54 GB
# 4 parallel slots -> 2.15 GB
# 8 parallel slots -> 4.29 GB

The relationship is exactly linear — doubling parallel slots doubles KV-cache memory, with everything else held fixed. If a production deployment has insufficient memory to load a new request while others are loaded, Ollama queues incoming requests rather than failing outright; if the queue itself fills past OLLAMA_MAX_QUEUE (default 512), further requests get rejected with a 503.

Source: Ollama, “Frequently Asked Questions”

GPU Offloading: Not All Layers Are Equal

A model’s transformer layers can run on GPU (fast) or spill over to system RAM, processed by the CPU (much slower) — controlled by num_gpu, the number of layers Ollama is told to place on the GPU. When VRAM can’t hold every layer, the remainder falls back to CPU — and every one of those CPU layers requires shuttling data back and forth over the PCIe bus, which is dramatically slower than VRAM’s own internal bandwidth.

A documented benchmark on Qwen 3 8B (36 transformer layers) shows just how nonlinear this drop-off is: fully offloaded to VRAM, the model ran at roughly 40 tokens/second. With only 25 of those 36 layers on the GPU — still the large majority — throughput fell to around 8 tokens/second, a 5× slowdown from removing barely a third of the layers from VRAM.

Source: “llama.cpp --n-gpu-layers guide: -1, 0, and partial offload”

Drag the slider below to see the shape of that same trade-off — throughput barely changes near full GPU offload, then falls off a cliff once a meaningful chunk of layers spills to system RAM.

Ollama Beyond Local: Turbo and the Cloud Models

Everything above assumes the model runs on your own GPU or CPU. Since mid-2025, Ollama also offers a way to run models on hardware you don’t own, using the exact same ollama run workflow.

Ollama Turbois a paid cloud inference service, in preview at roughly $20/month, that runs models on Ollama’s own datacenter-grade hardware instead of your local machine. It uses the same CLI, API, and desktop app you already use for local models — the difference is where the computation happens. This makes it possible to run models that simply won’t fit on consumer GPUs, such as gpt-oss:120b or deepseek-v3.1:671b, without buying the hardware to hold them.

Source: Ollama, “Turbo Preview”

A related but distinct feature is the :cloud model suffix. Models tagged this way — for example qwen3-coder:480b-cloud — are ordinary entries in the Ollama model library that route to Ollama’s cloud infrastructure rather than running locally. You address them through the identical ollama run qwen3-coder:480b-cloud command used for any local model — no separate client, no different API shape, just a different suffix on the model name.

Source: Ollama Blog, “Cloud models”

On the local side, Apple Silicon Macs got their own dedicated speed path: an MLX backend — Apple’s own machine learning framework, built around the unified memory that Apple Silicon shares between CPU and GPU — shipped in preview as an alternative to the llama.cpp-based engine Ollama uses on Linux and Windows. On the hardware it targets, Ollama’s own benchmarks show prefill throughput rising from roughly 1,154 to 1,810 tokens/second, and decode throughput from roughly 58 to 112 tokens/second.

Source: Ollama Blog, “Ollama is now powered by MLX on Apple Silicon in preview”

Fun Fact

The default 5-minute keep_alive window — the same setting covered in the Prompt Engineering Advanced Ollama case study — can be set to a negative value to mean “never unload,” or exactly 0 to unload the model the instant a response finishes generating — the same parameter covers both extremes.

Test Yourself