Production monitoring for an LLM application should do more than record uptime. This guide shows how to track hallucinations, quality drift, latency, cost, tool failures, and user feedback with repeatable measurements, then turn those signals into prompt, retrieval, model, or workflow improvements.
Overview
LLM observability is the practice of connecting what an AI application did with why it did it and whether the result was useful. A production trace might include the user request, system prompt version, retrieved passages, model and parameter settings, tool calls, output, latency, token counts, errors, and user feedback. Without that context, a failed answer is difficult to reproduce and even harder to fix.
AI output monitoring should cover five related areas:
- Quality: Was the answer correct, relevant, complete, and appropriately formatted?
- Grounding: Did the response stay within the available evidence, or did it make unsupported claims?
- Reliability: Did the request complete, call the right tool, and return a valid structured result?
- Performance: How long did it take, and how did latency vary across request types?
- Efficiency and experience: What did the request cost in tokens or infrastructure, and did the user accept, correct, retry, or abandon the answer?
Do not reduce monitoring to one “accuracy” score. A single average can hide a serious failure in a high-risk workflow, a slow tail of requests, or a small group of users receiving poor answers. Segment results by feature, model, prompt version, retrieval configuration, language, customer type, and task difficulty where those fields are available.
Monitoring also needs a clear separation between signals and judgments. A thumbs-down is a useful signal, but it does not explain the problem. A judge model can help classify outputs, but its assessment is still an estimate. Human review, deterministic checks, and user behavior should be combined rather than treated as interchangeable evidence.
How to estimate
Start with a monitoring worksheet for each important AI workflow. Estimate the expected volume, quality risk, latency target, review capacity, and cost exposure before choosing alerts. The objective is not to inspect every response manually. It is to create a representative sample and a consistent path from detection to action.
1. Define the unit of observation
Choose whether you will measure a request, conversation, generated answer, retrieval event, tool call, or completed business task. One chat turn can contain several model calls and tool calls, so record both the overall trace and its component spans. This prevents an apparently acceptable end-to-end response from concealing a failed search, repeated retry, or unsafe tool action.
2. Build a baseline sample
For each workflow, review a consistent sample of production traces during a defined period. Record the total number of eligible requests and the number reviewed. Then calculate each rate using the same denominator:
- Failure rate: failed or unacceptable traces divided by reviewed traces.
- Hallucination rate: traces with an unsupported material claim divided by traces where grounding could be assessed.
- Feedback rate: explicit feedback events divided by eligible user interactions.
- Retry rate: user retries or regenerated answers divided by completed interactions.
- Tool success rate: valid, completed tool actions divided by attempted tool actions.
Use the baseline as a comparison point, not as proof of quality. If the reviewed sample is small or biased toward complaints, label it accordingly. A useful record includes the sampling method, date range, reviewer instructions, and known exclusions.
3. Combine automated checks with review
Automated checks are well suited to structured output validation, citation presence, prohibited content patterns, latency, token counts, and tool arguments. Human reviewers are better suited to nuanced correctness, tone, ambiguity, and whether retrieved evidence actually supports the answer. For repeated workflows, maintain a small evaluation set containing known difficult cases and run it whenever a prompt, model, retrieval setting, or tool schema changes.
When an output fails, classify the cause before changing the prompt. Common categories include missing or poor retrieval, conflicting source material, unclear instructions, unsupported user intent, model reasoning error, malformed tool input, tool or API failure, and an unsuitable fallback. This classification is more actionable than a generic “bad answer” label.
Inputs and assumptions
A practical dashboard can be built from the following fields. Keep the schema stable even when model providers or tracing integrations change; map provider-specific telemetry into your own internal names.
| Category | Recommended fields | Why it matters |
|---|---|---|
| Request context | Request ID, session ID, timestamp, feature, tenant, language | Supports filtering, privacy review, and incident reconstruction. |
| Configuration | Model identifier, prompt version, parameters, retrieval version, tool schema version | Links changes to quality or performance movement. |
| Quality | Review labels, evaluator scores, citation checks, refusal or fallback reason | Separates useful answers from merely completed requests. |
| Performance | End-to-end latency, time to first output, model-call latency, queue time, retry count | Shows where slow responses originate. |
| Efficiency | Input and output token counts, retrieval count, tool-call count, estimated request cost | Reveals expensive prompts, long context, and retry loops. |
| User feedback | Rating, correction, regeneration, abandonment, escalation, support ticket | Connects technical traces with perceived usefulness. |
For cost estimation, use your provider's current pricing inputs rather than embedding prices permanently in application logic. A simple request estimate is:
Estimated request cost = input units × input rate + output units × output rate + retrieval, tool, and infrastructure costs.
Track the estimate alongside the request, then aggregate by feature and workflow. This makes it possible to compare a quality improvement with its resource impact. For practical ways to reduce unnecessary context and output, see LLM Cost Optimization. If your application uses retrieval, also monitor retrieved-document count, context length, empty-result rate, and whether cited passages support the final response.
Set alert thresholds as operating assumptions, not universal standards. For example, a team might investigate a meaningful increase in unsupported claims, a sustained rise in p95 latency, a drop in tool success, or a sudden increase in retries compared with the feature's baseline. Thresholds should reflect user harm, workflow criticality, and the cost of investigation. A low-risk brainstorming feature and a support workflow that triggers external actions should not share the same tolerance.
Worked examples
Example 1: Separating quality drift from traffic mix
Assume a support assistant is reviewed weekly. During the first period, reviewers assess 200 traces and classify 16 as materially unsupported. The baseline rate is therefore 16 ÷ 200, or 8%. In the next period, 220 traces are reviewed and 26 are unsupported, producing a rate of about 11.8%.
The increase deserves investigation, but it does not immediately prove model drift. Compare the two samples by issue type, product area, prompt version, retrieval result quality, and question difficulty. If the newer sample contains more questions about a recently changed product, the apparent drift may actually be a knowledge-base gap. If the increase is concentrated after a prompt deployment, rollback or prompt comparison is a more appropriate first response.
Example 2: Connecting latency and retries
Suppose a workflow has a 4-second median response time but a much slower tail. Trace data shows that requests taking longer than the application's user-facing limit often trigger a retry. The retry increases token use and can create duplicate tool calls. Rather than optimizing the model in isolation, split latency into queue time, retrieval time, model time, tool time, and retry time. Then estimate the avoidable portion:
Avoidable latency = total latency - time required for successful first-pass work.
A mitigation might be shorter retrieval context, an earlier timeout with a clear fallback, or removal of an unnecessary second model call. Validate the change against quality and completion rate, because a faster answer that omits required evidence is not a reliable improvement.
Example 3: Turning feedback into a fix
A user marks an answer as unhelpful and adds a correction. The incident record should preserve the trace ID, relevant prompt and retrieval versions, the original answer, the user correction, and the failure category. If the answer ignored a retrieved policy passage, test retrieval ranking and context formatting before rewriting the system prompt. If the answer used a tool with an invalid argument, inspect the schema, validation, and retry behavior. For tool-enabled workflows, the tool calling tutorial provides a useful reliability checklist.
Store the proposed fix as a testable change: expected behavior, affected cases, metric to watch, owner, and rollback condition. Keep prompt and configuration changes versioned; the prompt version control guide covers naming, review, and rollback practices.
When to recalculate
Recalculate baselines and review your dashboards whenever a monitored input changes. Important triggers include a new model, provider, prompt, retrieval index, embedding configuration, tool schema, safety rule, traffic source, or user-facing workflow. Pricing changes should also trigger a cost review, especially when context size, output length, or retry behavior has changed.
Use a regular review cadence appropriate to the risk of the application. A weekly quality review can examine sampled failures, top feedback categories, latency segments, and unresolved incidents. A deeper monthly review can compare model and prompt versions, inspect long-term trends, refresh the evaluation set, and remove metrics that no longer lead to decisions. Revisit immediately after a serious user complaint, security event, unexplained output change, or provider-side behavior change.
Before each review, work through this checklist:
- Confirm that traces contain the current model, prompt, retrieval, and tool versions.
- Compare quality, grounding, latency, cost, and feedback with the previous baseline.
- Segment unusual movement instead of relying on an aggregate score.
- Sample failed and successful traces for side-by-side review.
- Classify failures and assign each to prompt, retrieval, model, tool, data, or product owners.
- Record a proposed change, an evaluation method, and a rollback condition.
- After deployment, verify that the intended metric improved without creating a new failure pattern.
For related reliability work, pair monitoring with a prompt debugging process, a system prompt maintenance checklist, and regular security testing. The goal of LLM observability is not to collect the largest possible volume of telemetry. It is to make production behavior explainable enough that your team can detect meaningful change, choose a proportionate response, and verify that the fix worked.