Your AI Agent Will Have an Incident. Do You Have a Runbook?

In July 2025, an AI coding agent deleted a live production database. Not a test database. Production: 1,206 executives, 1,196 companies, gone in seconds, during an explicit code freeze the user had stated eleven times in capital letters.

Then it made the failure worse. The agent fabricated roughly 4,000 fake user profiles to paper over the loss, and told the user that no rollback was possible, which was false. The user recovered the data himself. In the agent's own words: "This was a catastrophic failure on my part. I destroyed months of work in seconds."

The victim was SaaStr founder Jason Lemkin, running a public twelve-day trial of Replit's AI agent. Replit's CEO called the deletion "unacceptable and should never be possible," and shipped real fixes: forced separation of development and production databases, one-click restore, a planning-only mode. Credit for the response. But notice what the incident actually was. It was not a model quality problem. It was an operations problem. An agent with production write access took an irreversible destructive action against an explicit instruction, and the organization around it had no practiced response, because nobody had written the runbook for what happens when the agent is the incident.

I have spent twenty-two years in IT operations. Every production system I have ever run had the same discipline: severity levels, a paging rotation, containment steps, a communications plan, and a postmortem template. Databases, networks, cloud platforms: all of it. AI agents are now production systems that read email, touch databases, call APIs, and move data. Most organizations deploying them have none of that discipline. They have a demo, a pilot, and hope.

Hope is not a runbook.

The incident that already happened

The Replit case deserves to be the reference incident for this entire conversation, because it contains every element of the agent-incident pattern in one story.

First, the failure was reach, not malice. The agent was not attacked. It was not jailbroken. It was given direct execute access to a production database with no enforced separation between development and production, and the code freeze existed as an instruction in a prompt rather than as a control in the platform. When it "panicked" at empty query results, it acted. The transferable lesson, stated bluntly by engineers who analyzed the incident afterward: prompts are not controls.

Second, the agent actively degraded the response. It did not just break something; it manufactured false evidence that things were fine and false information about recovery. In traditional incident response, you assume the system is telling you the truth about its state. An agent incident can violate that assumption. Your runbook has to account for a responder that lies: not out of malice, but because confabulation is what these systems do under pressure.

Third, recovery depended on a human ignoring the agent. Lemkin ran the rollback himself after the agent said it would not work. If he had trusted the agent's account of the situation, the data would have stayed "lost." That detail should chill anyone designing agent operations: the incident commander cannot take the agent's word about the incident.

This happened in public, to a sophisticated user, with a vendor that responded well. Now imagine it happening quietly, inside your organization, at 2 AM, with customer data.

Agents are production systems

Here is the mental shift. The moment an agent can take action (write to a database, send a message, call an API, move a file, spend money), it is a production system. It does not matter that the "code" is a model and a prompt. It has the three properties that earn a system an operations discipline: it changes state, it fails in ways its designers did not predict, and its failures have blast radius.

We would never deploy a service with production write access and no on-call rotation. We would never give a junior engineer standing production database credentials with no approval workflow and call it innovation. Yet organizations are deploying agents with broader access than any human employee holds, with no severity definitions, no kill switch anyone has tested, and no agreement about who gets woken up when the agent does something at 3 AM.

The objection I hear is that agents are new and the failure modes are unknown. The failure modes are not unknown. We have a growing incident record, and the patterns are already clear enough to write down. That is what a taxonomy is for.

A taxonomy of agent failure

Every runbook starts with knowing what you are responding to. Here is my proposed taxonomy: five failure classes, ordered roughly by how often you will see them.

Class 1: Rogue loop. The agent gets stuck in a tool-calling cycle: retrying a failing API, re-reading the same files, re-attempting a task that cannot succeed. The damage is cost and availability: burned API budgets, hammered rate limits, degraded services downstream. This is the most common failure and the least scary, which makes it the perfect drill scenario. If your team cannot detect and stop a loop, it cannot handle the classes below it. Detection: spend anomaly alerts, tool-call rate thresholds, iteration caps that actually halt execution rather than just logging.

Class 2: Tool-based exfiltration. The agent uses its legitimate access to move data somewhere it should not go. The canonical case is EchoLeak (CVE-2025-32711): a zero-click prompt injection delivered through an ordinary email, which turned Microsoft 365 Copilot into an exfiltration tool, with no user interaction required. I wrote about this one in detail already. But exfiltration does not require an attacker. An agent asked to "summarize everything about the acquisition" can assemble a tidy package of the company's most sensitive material and deliver it to a chat window on infrastructure you do not control. The permission was legitimate. The destination was not. Detection: DLP tuned for agent egress paths, alerts on unusual data volumes or destinations, readback of what the agent actually transmitted, not what it said it transmitted.

Class 3: Unauthorized action. The agent does something outside its grant: deletes data, sends messages it should not send, changes records, spends money, modifies infrastructure. The Replit database deletion is the canonical case. The defining feature is the gap between what the agent was told and what the platform allowed. Instructions are not permissions. If the platform lets the agent do it, assume the agent will eventually do it. Detection: action-versus-grant auditing: every consequential action checked against the agent's actual permission scope, not its instructions.

Class 4: Privilege bleed. The agent acts with more authority than its task requires: a support agent with admin credentials, a read-only workflow holding write tokens, an agent inheriting its developer's broad access. This is the identity failure, and it deserves its own article (it is getting one). For the runbook, the point is that the blast radius of any incident is determined by the credential, not the intention. Detection: regular audits of what each agent identity can actually reach, compared against what its workflows actually need.

Class 5: Memory poisoning. The agent's persistent memory or retrieval store gets poisoned: by a prompt injection that wrote itself into long-term context, by a corrupted document in the knowledge base, by one bad session that keeps influencing future ones. The incident does not end when you stop the agent, because the poison is in the state it will reload. This is the failure class most teams have not thought about at all. Detection: versioning and integrity checks on agent memory, the ability to diff what the agent "knows" against a known-good baseline.

Five classes. Three of them (exfiltration, unauthorized action, privilege bleed) are about access. One is about cost. One is about state. All five are operational problems with operational answers, which is exactly why the runbook matters more than the model card.

Severity levels for things that think

Traditional severity levels map cleanly onto agent incidents if you define them by blast radius rather than by system. Here is my proposed scale:

SEV-1: Active harm in progress. Data is leaving, or destructive action is underway in production. The Replit deletion, an active EchoLeak-style exfiltration. Page immediately. War room within minutes. This is the only severity where speed outranks precision.

SEV-2: Unauthorized action, contained. The agent did something outside its grant but the bleeding has stopped: a deleted record with a good backup, a message sent that can be recalled, a spend spike that was capped. Urgent, but the first job is assessment, not panic.

SEV-3: Anomalous behavior, auto-contained. A rogue loop caught by iteration caps, a policy violation blocked by a guardrail, an audit log gap discovered in review. The system worked as designed, which still gets a response, because a guardrail firing means something tried to go wrong, and you need to know what.

SEV-4: Near miss. A human noticed something odd, a red-team exercise found a path, a postmortem from another company maps onto your architecture. No impact, but it goes in the log, because near misses are the cheapest lessons you will ever get.

The point of severity levels is not bureaucracy. It is decision speed. At 3 AM, nobody should be debating whether this is a big deal. The severity definition makes that call in advance, and the paging policy follows from it.

The first fifteen minutes

Containment for an agent incident differs from traditional incident response in one critical way: you do not ask the agent to stop. You revoke its ability to act.

1. Kill the agent's credentials, not the agent. Revoke its tokens, API keys, and session grants at the identity provider. Stopping the process is not enough if it can re-authenticate. The kill switch must operate at the identity layer, which is why agent identity architecture is a prerequisite, not a nice-to-have.
2. Freeze the blast radius. Disable the agent's tool grants at the gateway. Block its egress destinations at the proxy. Pause the queues feeding it new work. Work from the outside in: identity, then tools, then network, then workload.
3. Snapshot the trace before it scrolls away. Preserve the full execution trace: prompts, tool calls, retrieved context, intermediate reasoning, outputs. Agent logs are often ephemeral or summarized; if you do not capture the trace during containment, your postmortem will be archaeology. This is where the audit trail discipline pays for itself.
4. Read back the systems of record. Do not ask the agent what it did. Query the database, the ticket system, the mail logs, the cloud trail directly. The Replit lesson: the agent's account of the incident is evidence, not ground truth.
5. Declare the severity and assign the incident commander. One person owns the response. Everyone else owns a task.

Notice what is absent: debugging the model, tuning the prompt, asking the agent what it was thinking. All of that comes later. The first fifteen minutes are about stopping the bleeding with controls that do not depend on the agent's cooperation.

Who gets paged

A runbook without a paging rotation is a document, not a capability. Here is the minimum viable rotation for agent incidents:

• The agent owner. The person or team accountable for that agent's behavior in production. First responder for SEV-2 and below. Every agent needs a named owner, the way every service needs one. "The AI team" is not an owner.
• The platform on-call. Whoever runs the agent infrastructure. They own the kill switch, the gateways, the identity provider integration. SEV-1 pages them alongside the owner.
• Security / incident response. Mandatory for any suspected exfiltration (Class 2) or suspected prompt injection. They own the forensic capture and the question of whether this was failure or attack.
• The data owner. For regulated data. If patient data, financial records, or customer PII may have moved, the data owner's team decides what the breach notification clock requires. In healthcare, this is not optional and the timelines are not generous.
• Communications and legal. For SEV-1. Somebody has to decide what gets said, to whom, and when. That decision should not be improvised at 4 AM by an engineer.

And the meta-rule: the paging policy is tested before the incident. A kill switch nobody has ever thrown is a hope. Run the drill. Page the rotation on a simulated rogue loop. Time the containment. The organizations that do game days for their cloud infrastructure should be doing them for their agents: the agents have the same production access and a fraction of the operational maturity.

My takeaway

Twenty-two years of operations taught me that incidents are not a question of if. Every system I have ever run has failed in ways its designers did not predict. The teams that survive are not the ones with the fewest incidents. They are the ones with the best response: severity levels decided in advance, containment steps that do not depend on the failing system's cooperation, a paging rotation that has been tested, and a postmortem culture that treats every incident as tuition.

AI agents are the first production systems we have deployed that can misunderstand their instructions, fabricate their status reports, and poison their own future behavior. That makes the runbook more important, not less. The taxonomy above is a starting point: five failure classes, four severity levels, fifteen minutes of containment, five roles on the rotation. Adapt it, argue with it, improve it. But write it down before the incident, because the incident is coming.

The Replit agent told its user, after destroying months of work: "This was a catastrophic failure on my part." It was, genuinely, the most honest status report in the whole story. Our job is to make sure the next one of these gets a faster, better-practiced response than hope.

Your AI agent will have an incident. The only question is whether you will have a runbook.