Context Windows in Production: KV-Cache Cost & Lost-in-the-Middle

The Real Cost: KV-Cache Memory

The formula practitioners actually use to size KV-cache memory:

KV_cache_bytes = 2 × num_layers × num_kv_heads × head_dim × seq_len × bytes_per_element

The leading 2 is for the K and V vectors — one each, per token, per layer, per head. Plug in real numbers for a 70B-parameter model (80 layers, 8 KV heads under grouped-query attention, head dimension 128) at a 128K-token context length, storing in FP16 (2 bytes per number):

num_layers = 80
num_kv_heads = 8
head_dim = 128
seq_len = 131072       # 128K tokens
bytes_per_element = 2  # FP16

kv_cache_bytes = 2 * num_layers * num_kv_heads * head_dim * seq_len * bytes_per_element
kv_cache_gb = kv_cache_bytes / (1000 ** 3)

print(round(kv_cache_gb, 1))
# 42.9

Forty-three gigabytes — for the KV cache of a single request, before the model’s own weights (which take up their own tens of gigabytes) are even counted. This is why providers quantize the KV cache itself: dropping to FP8 storage roughly halves that number, and lower-precision formats push it lower still, in each case trading some retrieval quality for the memory headroom to serve more concurrent long-context requests on the same hardware.

The Lost-in-the-Middle Problem

A 2023 study tested this directly: place a single relevant fact at different positions inside a long document, then ask a question that requires retrieving it. Across six different model families — including GPT-3.5, GPT-4, and Claude — the result was consistently U-shaped: accuracy was highest when the fact sat at the very start or very end of the input, and dropped by more than 30% when the same fact sat near the middle.

The likely architectural cause ties back to how positional encoding interacts with self-attention: many positional schemes systematically weaken the similarity score between tokens that are far apart in the sequence, and softmax normalization then concentrates attention further on whichever tokens already score highest — reinforcing an advantage for tokens near the edges of the context at the expense of the middle.

Source: Liu et al., “Lost in the Middle: How Language Models Use Long Contexts,” arXiv:2307.03172

Needle in a Haystack, Yourself

This test design — burying one distinctive sentence inside a much larger document at varying depths, then checking whether the model can retrieve it — was popularized by Greg Kamradt and has since become one of the standard benchmarks for long-context retrieval. The demo below models that same shape: pick a depth and see how retrieval confidence changes.

Managing Context in Production

Given both costs — memory that grows with every token kept, and reliability that degrades for facts buried in the middle — the practical response most production systems converge on isn’t “maximize the window,” it’s “minimize what actually needs to be in it”:

• Retrieval instead of stuffing.Store documents externally and pull in only the chunks relevant to the current query, rather than keeping an entire knowledge base loaded in context on every request — the approach covered in this series’ upcoming RAG topics.
• Query-aware placement. When you do control what goes into the prompt, put the most important facts near the start or end, not buried in a long middle section, given the retrieval-position effect above.
• Prompt caching. For content that repeats across requests (a system prompt, a shared document), caching the already-computed KV state avoids reprocessing it — covered in depth on the Prompt Engineering Advanced page.
• Summarization and compaction. When a conversation genuinely needs to span beyond what fits, periodically summarizing older turns into a shorter recap trades some detail for staying inside the budget, instead of simply truncating.

Fun Fact

The famous “needle” sentence used in the original Needle in a Haystack test is almost comically mundane: “The best thing to do in San Francisco is eat a sandwich and sit in Dolores Park on a sunny day.” It was chosen precisely because it has no logical connection to the surrounding essay text, making it easy to tell whether the model actually retrieved it or just pattern-matched something related.

Test Yourself