Why not just log everything for an LLM-powered system?

Logging the full prompt and response is exactly what makes a bad answer debuggable later, but a prompt or response can also contain sensitive user data — names, account numbers, health details — that shouldn’t be stored any more broadly than necessary. The real practice is masking or redacting sensitive fields before a log is written, not logging less overall, so debugging power and data-minimization can both be satisfied at once.

AI & ML — INTERMEDIATE

Production Monitoring for LLM Systems: What to Actually Log

Q: What should you actually log for an LLM system running in production?

Four things earn their keep in almost every LLM system: the full prompt and response (for debugging and audit), token counts and latency per call (real structural facts, not benchmark guesses), user feedback signals (explicit thumbs up/down or implicit signals like an immediate retry), and error rates (tool-call failures and is_error responses). What changes system to system is how much of the prompt and response can be logged in full versus how much needs to be masked first.

An LLM-powered system in production is still software — but the failures that show up in it are different from a typical web service’s. A wrong answer doesn’t throw an exception. A slow response doesn’t always mean a bug. Getting visibility into what a system is actually doing starts with deciding, deliberately, what to log — and what not to.

The Quick Answer

Four categories are worth logging in almost any LLM-powered system: the full prompt and response (so a bad answer can actually be reproduced and inspected), token counts and latency per call (real structural facts, not invented benchmark numbers), user feedback signals (explicit thumbs up/down, or implicit signals like an immediate retry), and error rates (tool-call failures and is_error responses). The genuine tension is between logging enough to debug problems and not logging sensitive user data any more broadly than necessary — solved by masking, not by logging less.

The Full Prompt and Response

When a user reports a bad answer weeks later, the only way to actually understand what happened is to see exactly what the model was given and exactly what it produced — not a paraphrase, not just the final rendered text, but the real prompt (including any retrieved context, system instructions, and conversation history) and the real response. Without this, debugging a production LLM issue turns into guessing.

This is also what makes an audit possible after the fact: if a regulator, a customer, or an internal reviewer asks “what did the system actually say, and why,” a full prompt/response log is the only record that answers that question with certainty, rather than reconstructing it from memory or partial UI logs.

Token Counts and Latency Per Call

Every LLM call has two real, structural facts attached to it, independent of what the content of the call was: how many tokens it used, and how long it took. These aren’t benchmark scores or estimates — they’re measured, per-request facts that come back with every API response, and logging them per call is what turns “this feels slow/expensive” into an actual, graphable trend over time.

This ties directly into the same idea covered in Agentic Workflows vs. Chatbots Advanced: LLM call count is a real cost signal, not a guess. Logging token counts and latency per call is what makes that signal visible in a running system, rather than only knowable in theory from the architecture.

User Feedback Signals: Explicit and Implicit

The most direct feedback signal is explicit: a thumbs up/down button next to a response, logged alongside the exchange it’s rating. It’s simple to collect and simple to aggregate into a satisfaction rate over time.

Just as valuable, and easy to miss, are implicitsignals — behavior that suggests a response failed even though the user never said so directly. The clearest example: a user immediately re-asking essentially the same question, often rephrased. That pattern is a strong hint the first answer didn’t actually help, and it’s available in the logs whether or not any explicit feedback mechanism exists at all.

Error Rates: Tool-Call Failures and is_error

For any system where the model calls tools, a failed tool call is a distinct, loggable event — reported back to the model as a tool_result with is_error: true. The exact mechanics of how that error is constructed and reported are covered in Tool Calling Advanced — the monitoring angle here is simpler: every is_error result is worth logging and counting, because a rising rate of tool failures is one of the clearest, earliest signs something in production has actually broken — a downstream API, a changed schema, a rate limit newly being hit.

What to Log, By Scenario

The four categories above apply everywhere, but how much of the raw prompt and response can be logged in full — versus needing to be masked first — depends heavily on what the system actually handles. Try a few different production scenarios below:

Logging Enough to Debug, Without Logging Sensitive Data Unnecessarily

The real tension isn’t debugging versusprivacy — it’s that a naive “log everything” policy solves debugging but ignores privacy, while a naive “log nothing sensitive” policy solves privacy but makes debugging much harder. The actual fix is neither extreme: mask or redact sensitive fields (names, account numbers, health details) before a log is written, so the debugging-relevant shape of the prompt and response is preserved without storing the raw sensitive values any more broadly than necessary.

The concrete masking technique for this — detecting sensitive substrings and replacing them with reversible placeholder tokens — is covered in depth in Cloud AI vs Local AI Advanced. The same reversible-masking pattern used there to protect data sent to a cloud model applies directly to what gets written into a production log.

Fun Fact

A rising rate of “immediate retry with a rephrased question” is often a more honest quality signal than a thumbs-down rate — most users who get a bad answer never bother clicking a feedback button, but almost everyone who gets a bad answer and still needs the information will try asking again.

Test Yourself

Why is logging the full prompt and response worth the storage cost for an LLM system?

What is an implicit user feedback signal, as opposed to an explicit one like thumbs up/down?

What is the real tension between debugging and privacy when deciding what to log?

Ready for what happens when the input distribution shifts, a provider changes a model underneath a fixed API name, and a structured way to respond when a production incident actually hits? Continue to the Advanced level →