LLM Incident Response Runbook: Diagnose Hallucinations, Outages, and Unsafe AI Behavior
LLM reliabilityincident responseAI operationshallucinationsproduction AIrunbooks

LLM Incident Response Runbook: Diagnose Hallucinations, Outages, and Unsafe AI Behavior

HHiro Solutions Editorial Team
2026-11-21
6 min read

A practical LLM incident response runbook for containing hallucinations, outages, sensitive disclosures, and unsafe AI actions.

An LLM incident response runbook helps your team contain hallucinations, outages, sensitive disclosures, and unsafe actions without relying on improvised decisions. Use this guide to define severity, preserve evidence, control traffic, restore a safe version, and review recurring reliability signals.

Overview

Production AI failures rarely have a single cause. A sudden increase in incorrect answers may result from a model change, prompt edit, retrieval failure, malformed context, an unavailable tool, or a change in user input. An outage may originate with your application, a dependency, a provider, or a network path. Unsafe behavior can involve prompt injection, excessive permissions, weak output validation, or a workflow that allows an unverified response to trigger an action.

The first objective of LLM incident response is not to explain everything. It is to reduce harm and restore a known-safe service state. Assign one incident lead, open a shared incident record, and separate confirmed facts from working hypotheses. Avoid making multiple untracked prompt or configuration changes while investigating; they can destroy the comparison point needed for diagnosis.

Use a simple severity model. A critical incident involves possible physical, financial, legal, privacy, or security harm; unauthorized external actions; or a broad service failure. A high-severity incident affects a significant workflow, produces materially misleading output, or exposes restricted information without evidence of widespread impact. A moderate incident is limited in scope, recoverable, and not currently causing significant harm. Severity should be raised when impact, duration, uncertainty, or the likelihood of recurrence increases.

Your runbook should name the incident commander, technical investigator, communications owner, security or privacy contact, and business owner for each production AI system. It should also document who can disable tools, route traffic to a fallback, roll back a prompt or model, and approve a return to normal operation.

What to track

Track enough information to compare the failing behavior with a healthy baseline. At minimum, record:

  • Time and scope: detection time, first known occurrence, affected users, regions, tenants, endpoints, and workflows.
  • Request details: model identifier, provider or deployment, prompt and system-prompt version, retrieval configuration, tool set, token counts, and relevant feature flags.
  • Output evidence: representative inputs and outputs, refusal or safety signals, citations, tool calls, validation errors, and whether a human reviewed the result.
  • Service health: request volume, error rate, timeout rate, latency, rate-limit responses, queue depth, and fallback usage.
  • Quality signals: user corrections, escalation rates, groundedness checks, citation failures, structured-output parse failures, and sampled evaluation results.
  • Security and privacy indicators: suspected prompt injection, sensitive-data exposure, unauthorized tool calls, unusual access patterns, and whether logs contain data that should be redacted.

Preserve raw evidence carefully. Store identifiers, timestamps, configuration versions, and traces in a restricted location, and redact or minimize personal and confidential data before sharing the incident broadly. Do not paste sensitive production conversations into an external debugging tool merely to obtain a quick explanation.

For recurring monitoring, create a dashboard that shows reliability by model, prompt version, workflow, and customer segment rather than only one global average. A stable overall error rate can hide a serious failure affecting one important route. Connect alerts to thresholds that matter operationally, such as a sustained rise in tool failures or unsafe-output reports, rather than alerting on every isolated anomalous response.

Cadence and checkpoints

During an active incident, use a short operational cycle:

  1. Detect and classify: confirm that the signal is reproducible, estimate impact, assign severity, and name an incident lead.
  2. Contain: disable risky tools, pause automated actions, reduce traffic, require human approval, switch to a constrained mode, or temporarily serve a clear degraded experience.
  3. Preserve: capture traces, versions, timestamps, alerts, deployment changes, provider responses, and examples before altering the system.
  4. Diagnose: compare a failing request with a known-good request and test one variable at a time: provider status, model, prompt, retrieval, tool schema, input class, or application code.
  5. Recover: roll back the smallest relevant change or route to a tested fallback. Verify both successful requests and the failure mode that caused the incident.
  6. Communicate: provide internal updates at a predictable interval and tell affected users what is unavailable, what action they should take, and when the next update is expected.
  7. Close and learn: document the root cause, contributing conditions, customer impact, corrective actions, and evidence that the fix works.

After each release, model change, major prompt revision, retrieval-index update, or tool-permission change, run a focused checkpoint. Test normal requests, ambiguous inputs, adversarial inputs, long-context cases, structured outputs, and failure handling. A prompt rollback is useful only if prompt versions are named, stored, reviewed, and deployable independently; see the prompt version control workflow for a practical operating pattern.

On a monthly or quarterly schedule, review incident counts, repeated causes, mean time to detect, time to contain, time to recover, rollback frequency, unresolved actions, and the proportion of incidents discovered by users rather than monitoring. Recheck emergency contacts, fallback credentials, rate limits, provider configurations, and access to deployment controls.

How to interpret changes

Interpret trends by segment and by change history. A quality decline immediately after a model or system-prompt deployment points toward a release regression, but it is not proof. A decline limited to questions requiring retrieved documents suggests checking chunking, indexing, filters, context assembly, and citation validation. A rise in timeouts with normal model quality points toward capacity, concurrency, network, or provider availability. A rise in malformed JSON or failed tool calls calls for inspection of schemas, parsers, retries, and model-specific behavior.

Hallucination reports deserve more than a simple count. Classify each case as missing context, incorrect retrieval, unsupported inference, stale source material, ambiguous user intent, or an evaluation error. Then compare the categories across prompt versions, models, and routes. The appropriate response may be better retrieval, stricter structured output, a clarifying question, a human review step, or a narrower system prompt—not necessarily a larger model.

For suspected unsafe behavior, treat containment as successful only when the action path is blocked, not merely when the visible text looks safer. Verify permissions, tool arguments, allowlists, approval gates, output filters, and audit logs. Use the prompt security testing checklist to turn repeated attack patterns into regression tests.

When to revisit

Revisit this runbook after every material incident, provider or model migration, SDK upgrade, system-prompt change, retrieval redesign, tool integration, or change to data-handling requirements. It should also be reviewed on a monthly or quarterly cadence even when the service appears healthy. Reliability gaps often remain invisible until an alert, fallback, or rollback is exercised under pressure.

Before each review, sample recent traces and compare them with the documented baseline. Confirm that dashboards still contain the required dimensions, alerts reach the right people, rollback artifacts are available, and fallback behavior has been tested. Update examples when the application changes, but keep the core decision sequence stable.

End every review with owners and deadlines for concrete actions. A useful incident record includes: summary, severity, timeline, affected components, evidence, containment steps, root cause, contributing factors, user impact, recovery verification, and follow-up tasks. Schedule a short tabletop exercise at least periodically: simulate an outage, hallucination spike, sensitive-output report, and unsafe tool call, then measure how quickly the team can identify the signal and move the system into a safe state. That recurring practice turns an incident response document into an operational control rather than a file consulted only after something goes wrong.

Related Topics

#LLM reliability#incident response#AI operations#hallucinations#production AI#runbooks
H

Hiro Solutions Editorial Team

AI Systems Reliability Editor

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.