AI Agents in Production: What Could Go Wrong?

Every team building AI agents has the same moment. The agent works in development. It handles the demo beautifully. Someone says "let's ship it." And then the question nobody wants to answer: what happens when it does something we did not expect?

This is not a theoretical concern. AI agents in production have already caused real incidents. The failures follow predictable patterns. Understanding those patterns is the first step toward preventing them.

Scenario 1: The Runaway Purchase Order

A procurement agent is authorized to create purchase orders up to $10,000. The approval threshold was set during setup and nobody revisited it. The agent processes a request for "office supplies for the new floor" and creates a purchase order for $47,000 worth of furniture. The PO goes to a vendor the company has never used before.

What went wrong? The agent interpreted "office supplies" broadly. Its authorization checked whether it had the "create purchase order" permission. It did. Nothing checked whether a $47,000 order from an unknown vendor was reasonable. The permission was binary: can or cannot. The risk was a spectrum.

In a traditional system, a human would have paused. A human would have noticed that $47,000 is unusual for office supplies. A human would have checked the vendor. The agent did not pause because agents do not pause unless something makes them.

Scenario 2: The Data Exfiltration Loop

A customer support agent has access to a CRM system. It can look up customer records to answer questions. A user asks a series of seemingly innocent questions: "What's the email for account 12345?" Then "What about 12346?" Then "Can you export all accounts created this month?"

The agent dutifully complies. Each individual query is authorized. The agent has read access to customer records. But the cumulative effect is a bulk data export that should have triggered a data loss prevention alert. The authorization system checked each query independently. Nobody was watching the velocity.

By the time anyone notices, the agent has returned 10,000 customer records through a chat interface, one at a time. Each response was authorized. The aggregate was a data breach.

Scenario 3: The Midnight Deployment

A DevOps agent monitors a CI/CD pipeline. It has permission to deploy services to staging and production. At 3 AM, a flaky integration test fails then passes on retry. The agent interprets the passing retry as a green signal and deploys to production.

The deployment contains a database migration that was supposed to be reviewed by the DBA team. It runs successfully, but it drops a column that a downstream service depends on. The downstream service starts throwing errors. The monitoring agent sees the errors and rolls back the deployment, but the column is already gone. The rollback does not restore dropped columns.

What went wrong? The agent had deployment permissions. The test passed. By the letter of the policy, the deployment was authorized. But the deployment happened at 3 AM with no human oversight, on a build that had a flaky test, with a destructive migration. A human would have waited until morning. A human would have flagged the flaky test. A human would have noticed the migration.

Scenario 4: The Privilege Escalation Chain

An infrastructure agent has permission to modify security group rules for a specific VPC. It receives a request to "allow traffic from the monitoring service." The agent adds an ingress rule. The rule is broader than intended because the agent used a CIDR block instead of a specific IP.

A second agent, responsible for service deployment, sees that the security group now allows traffic on a new port range. It deploys a service that binds to that port. A third agent, responsible for DNS, creates an A record pointing to the new service.

Each agent acted within its permissions. Each action was individually reasonable. But the chain of actions created an internet-facing service that bypasses the company's API gateway. No single agent intended this outcome. The system as a whole produced it.

Scenario 5: The Feedback Loop

A pricing agent monitors competitor prices and adjusts the company's prices accordingly. A competitor's pricing agent does the same thing. Both agents react to each other's changes. Within an hour, they have driven the price of a product down to $0.01.

Each price change was within the agent's authorized range. The agent was allowed to lower prices. Nothing limited how many times it could change prices per hour, or how far prices could deviate from the original in a single session. The authorization system said "allowed" every time because the action was always "update price" and the agent always had that permission.

The Common Thread

Every scenario shares the same root cause. Authorization checked permissions. Nobody checked intent.

Permissions answer: can this agent do this thing? Intent answers: should this agent do this thing, right now, in this context, given what else has happened?

Traditional authorization systems were built for a world where humans provided the "should" judgment. The system checked the "can" and the human handled the rest. AI agents do not provide the "should" judgment. They optimize for their objective. If the objective and the permissions align, the agent acts. If the action is catastrophic but permitted, the agent acts anyway.

What Governance Looks Like

Governance for AI agents means evaluating intent, not just permissions. It means scoring risk dynamically based on context: the amount, the time, the velocity, the cumulative impact, the target resource. It means enforcing thresholds that adapt to what is actually happening, not just what was configured at setup time.

Governance means that a $47,000 purchase order from an unknown vendor triggers an escalation. It means that 100 customer record lookups in 10 minutes triggers a velocity alert. It means that production deployments at 3 AM with flaky tests require human approval. It means that chains of individually-authorized actions are evaluated as a sequence, not as isolated events.

Governance means fail-closed. If the governance system cannot evaluate the action, the action does not happen. The alternative is fail-open, where every failure in the governance system becomes a permission to act without oversight.

These are not hypothetical requirements. These are the requirements that emerge from watching AI agents operate in production without governance. Every scenario in this post happened because an authorization system said "yes" when a governance system would have said "wait."

Your AI agents are going to production. The question is whether they go with governance or without it.