Context Windows for Engineers: Tokens, Truncation & Cost

Why It’s Expensive, Not Just Limited

Self-attention’s core operation — softmax(QK^T / √d_k) — computes an affinity score between every pair of tokens in the input. Double the number of tokens and you don’t double the work — you roughly quadruple it, since the number of pairs grows as n². A model built for a 4K-token window and one built for a 1M-token window aren’t running the same architecture at a different setting; the longer one needs real engineering (sparse attention variants, better KV-cache management, and more) to make that cost tractable at all.

Estimating Tokens Before You Hit the Limit

You rarely have the exact tokenizer handy while drafting a prompt, so a rough estimate is usually enough to sanity-check whether you’re anywhere near the limit. OpenAI’s own published rule of thumb: roughly 1 token per 4 characters of English text. Try it below — type or paste something and watch the live estimate.

What Happens When You Exceed It

Once a conversation’s token count would exceed the window, something has to be dropped before the next request can even be sent. The simplest and most common approach is a sliding window: keep the most recent messages that fit inside the budget, and drop the oldest ones first — exactly the FIFO behavior you saw in the Introduction-level demo, just now driven by an actual token budget instead of a message count. Other strategies exist (summarizing dropped turns into a shorter recap, or retrieving only the relevant older messages back in), but sliding-window truncation is the default nearly every chat client falls back to.

Below is a fixed support conversation with a small 40-token budget. Add messages one at a time and watch which ones get pushed out once the budget is full.

Context Window ≠ Memory

It’s tempting to treat the context window as the model’s “memory,” but it’s better understood as scratch space for the current request. It’s scoped to one conversation and empties out completely the moment that conversation ends. Systems that need to remember facts across sessions — a user’s preferences, a document library, prior conversations — store that information externally (commonly in a vector database, retrieved back into the context window only when relevant) rather than trying to keep it permanently loaded in the window itself. That retrieval pattern is its own topic, coming later in this series.

Bigger Isn’t Automatically Better

A larger context window doesn’t guarantee a model actually uses everything inside it equally well. Research on long-context retrieval has repeatedly found a U-shaped accuracy curve: models retrieve facts most reliably when they sit near the very start or very end of the context, and least reliably when they’re buried in the middle — a finding that held up across six different model families in the original study. The practical takeaway: don’t assume a 1M-token window means the model will notice a fact you buried at token 500,000 as reliably as one you put in the first paragraph. The mechanism behind this — and how to actually test for it — is covered in the Advanced level.

Fun Fact

The original “Lost in the Middle” study found accuracy on a middle-positioned fact dropping by more than 30% compared to the same fact placed at the start or end of the context — even though the model could, in principle, attend to every token equally.

Source: Liu et al., “Lost in the Middle: How Language Models Use Long Contexts,” arXiv:2307.03172

Test Yourself

Ready for production-scale numbers — real KV-cache memory math, and a hands-on Needle-in-a-Haystack demo? Continue to the Advanced level →