Advanced Prompt Engineering: Caching, Constrained Decoding & Self-Consistency

Prompt Caching

This is KV caching’s counterpart at the API level (see the “How LLMs Work” Advanced page). Long system prompts, few-shot examples, or reference documents that stay identical across many requests don’t need to be reprocessed from scratch every call — the provider caches the already-computed attention state for that prefix and reuses it. Anthropic’s own documentation puts a real number on it: cache reads cost roughly 0.1× the price of a normal input token — about a 90% reduction — while a cache write costs 1.25× (5-minute TTL) or 2× (1-hour TTL) the base price, paid once to set the cache up.

Source: Anthropic, Prompt Caching Documentation

Case Study: Implementing This on Ollama

Anthropic’s API charges differently for cache reads vs. writes, but the same idea applies to a fully local system built on Ollama — you just get it for free instead of a billing line item, with two conditions to actually earn it.

First, the model has to stay resident in memory between requests. Ollama unloads a model 5 minutes after its last request by default; set keep_alive to a longer duration (or -1to never unload) in every request, otherwise there’s nothing left in memory to reuse between calls:

import requests

response = requests.post("http://localhost:11434/api/generate", json={
    "model": "llama3",
    "prompt": full_prompt,      # static system prompt + dynamic user question
    "keep_alive": "30m",        # keep the model loaded between requests —
                                # otherwise there's nothing to reuse a cache from
})

Second — and this is the part people miss — Ollama’s inference engine (built on llama.cpp) can only reuse the already-computed Key/Value state for tokens that form an exact, unchanged leading prefixof the new prompt, on the same server session. The moment the very first tokens differ from the last request — even a changed timestamp or a reordered instruction near the top — the whole prompt gets reprocessed from scratch. The practical fix is to put everything static (system prompt, instructions, few-shot examples) first, byte-for-byte identical every time, and put the one thing that actually changes — the user’s question — at the very end:

SYSTEM_PROMPT = """You are a support assistant for Acme Router hardware.
Answer using only the troubleshooting steps below.
... (long, fixed instructions + few-shot examples) ...
"""

def build_prompt(user_question):
    # Static content first (byte-for-byte identical every call), dynamic
    # content last — this is what lets llama.cpp's cache-reuse actually
    # match a previous request's leading prefix instead of reprocessing
    # the whole thing from token 0 every time.
    return f"{SYSTEM_PROMPT}\n\nUser question: {user_question}"

Source: llama.cpp, “Mastering Host-Memory Prompt Caching in llama-server”

Constrained Decoding: Guaranteeing Valid Structure

The Intermediate page’s “ask for JSON” approach still lets the model sample any token — including ones that break your schema. Constrained decoding restricts which tokens are even eligible at each sampling step, so only tokens that keep the output on a valid path through the schema can be chosen. The output isn’t just usually correct — it’s structurally incapable of being invalid:

Self-Consistency: Sampling Multiple Reasoning Paths

Chain-of-thought helps, but any single reasoning path can still take a wrong turn. Self-consistency samples the same chain-of-thought prompt several times — at a temperature above 0, so the paths actually differ — and takes a majority vote on the final answers. Individual paths are allowed to be wrong; the technique only needs most of them to be right:

Source: Wang et al., “Self-Consistency Improves Chain of Thought Reasoning in Language Models,” 2022

Prompt Injection: When the Prompt Isn’t Fully Yours

Everything so far assumed you fully control the prompt. Real systems often paste in untrusted content — a scraped webpage, a user’s message, a document someone else wrote — and LLMs process instructions and that pasted data in the exact same channel, with no built-in way to tell them apart. If the pasted content contains something that reads like an instruction, the model can end up following it instead of the developer’s actual system prompt. OWASP has ranked this the #1 risk for LLM applications for two editions in a row.

Source: OWASP Gen AI Security Project, “LLM01:2025 Prompt Injection”

Automatic Prompt Optimization

Every technique above still assumes a human is hand-writing the prompt. A newer line of work treats the prompt itself as something to search over: generate candidate prompts, score them against real examples, and keep iterating — using the model to improve its own instructions rather than a person hand-tuning wording by trial and error. It’s the same instinct as hyperparameter search, aimed at English instructions instead of learning rates.

Worth being precise about: this isn’t a background feature running inside your API calls that needs turning off. Tools that do this — DSPy’s optimizers, APE, and similar frameworks — only run when you explicitly invoke their optimization step over a training set. If you never call that step, nothing rewrites your prompt; the exact wording you wrote is exactly what gets sent, every time. “Disabling” it is just not opting in.

Fun Fact

When OpenAI introduced Structured Outputs in August 2024, they reported that gpt-4o-2024-08-06 achieved 100% schema adherence on their complex-JSON-schema evals with constrained decoding turned on — compared to plain JSON mode, which produces valid JSON but was never designed to guarantee it matches any particular schema at all.

Source: OpenAI, “Introducing Structured Outputs in the API” (2024)

Test Yourself