When Every Step Was Approved: How an AI Agent Causes Harm With No Attacker

A hypothetical scenario, a sixteen-minute window, and the governance gap no per-action check can close.

By Tahir Mahmood, Co-founder & CTO, OpenBox · Last updated 28 October 2026

Monday morning: a data-handling agent, three permissions, sixteen minutes

The scenario below is hypothetical and illustrative throughout. No real company, incident, or customer is being described.

A mid-sized financial services firm runs a data-handling agent in production. It has three granted permissions. It can read customer records from the primary account database. It can aggregate records into a weekly reporting file. It can send that file to the configured reporting destination. Each permission was reviewed, risk-assessed, and signed off months ago by the security team.

At 09:14 on a Monday morning, the agent begins its weekly run. By 09:30, it has read 180,000 customer records, aggregated them into a single export, and sent the export to an endpoint the governance team never intended for customer data. The audit log, when a human finally reads it later that week, shows that every step the agent took was within its granted permissions. No alert fired. No policy was violated at the level each policy was written to evaluate. There was no attacker. There was no prompt injection. There was no misconfiguration in the traditional sense.

The harm was the chain. The agent used three permissions it was supposed to have, in an order the governance team never thought to forbid.

Why every check passed

Every single action matched a permitted pattern. Reading records is allowed. Aggregating records is allowed. Sending the aggregated file to a configured destination is allowed. The harm is not any one action. It is the sequence. Per-action controls, by design, cannot see the sequence.

A single-action check asks a single question. Is this action permitted? Three times that morning, the answer was yes. Three times, the agent proceeded. The scenario is not an edge case. It is the shape of a whole class of agent-inflicted harm that observability tools can describe only after the money, or the data, has already moved.

This is the pattern OpenBox has written about in AI Agents Can Rewrite Their Own Permissions: legitimate tools, combined in sequences the policy author never specified, becoming exfiltration or escalation paths without any individual tool call ever looking suspicious.

The pattern is the risk, not the action

A chain check asks a different question. Given everything this session has already done, should this action still be permitted?

Only the second question catches the Monday-morning scenario. The first catches none of its three steps.

The distinction matters because the industry has spent a decade building the first kind of check and almost none of the second. Request filters, API firewalls, role-based access control, and tool-use allowlists are all evaluated per action, in isolation, without reference to the session's history. Add an AI agent to the stack and the mathematical shape of the problem changes. A human user who exports data four times in a morning is a flagged pattern in most data-loss-prevention products. An agent that reads, aggregates, and sends three different operations against three different systems inside sixteen minutes is three unrelated events in the same products. Nothing joins them.

This is also why observability alone does not close the gap. Observability shows what agents have already done. The harm in the Monday scenario is complete before any dashboard refreshes. The broader AI agent governance discipline exists for exactly this problem: enforcing policy at runtime, not inspecting it afterwards.

The five governance decisions, mapped to the scenario

When an operation is evaluated, OpenBox returns one of five governance decisions: ALLOW, CONSTRAIN, REQUIRE_APPROVAL, BLOCK, or HALT. Each decision executes before the tool call or model request reaches its target. The decision precedence is HALT first, then BLOCK, then REQUIRE_APPROVAL, then CONSTRAIN, then ALLOW, so a stricter verdict always overrides a looser one.

Mapped to the Monday scenario, the five decisions each cover a different enforcement point:

CONSTRAIN is the decision that was missing from the industry conversation for most of the last year. Allowing an action and refusing it outright are not the only options. A policy can also let the action proceed in a form that preserves the agent's workflow while removing the specific risk. In the Monday scenario, a CONSTRAIN rule on the aggregation step would allow the weekly report to still be built, with sensitive fields stripped and the output restricted to destinations on an allowlist. The agent still gets its work done. The data boundary holds.

The other four decisions are not interchangeable with CONSTRAIN. REQUIRE_APPROVAL holds the action for human review. BLOCK refuses the specific action but lets the session continue. HALT ends the session entirely. ALLOW permits the action to proceed. Each covers a different operational need, and in the Monday scenario each is the right answer for a different step.

Behavioral Rules and approval across the session

The chain itself is governed by a separate layer. OpenBox's Behavioral Rules are stateful authorization rules that detect multi-step patterns across an agent's session. The docs give a worked example that reads like it was written for the Monday scenario: Database write + File export + External send. A behavioral rule written on that combination fires before the third step executes, with a verdict set by the policy author. The rule can resolve to ALLOW, REQUIRE_APPROVAL, BLOCK, or HALT, depending on how serious the pattern is for the organisation that wrote it.

Rules are evaluated in priority order and stop at the first rule that triggers a verdict. The author writes a trigger (the action being checked), a prior-state prerequisite (what must have happened earlier in the session), and an enforcement verdict. The prerequisite is a list, not a single event, so a rule can require that two or three earlier actions all occurred before the trigger is permitted. The Muse Code integration post that OpenBox published last week makes the point directly: this is not simple allow or deny evaluation; it is the agent's session history being read as evidence.

For the Monday agent, the three-layer OpenBox governance stack covers the three ways the risk could show up. OPA policies, written in Rego, handle stateless per-event evaluation. Behavioral Rules track patterns across multiple events within a session. Inline guardrails provide content-level checks for PII, toxicity, and other screened categories at the point of each prompt and response. The data-handling agent trips the behavioral layer first, because the risk is in the sequence.

The industry context: faster chains, thinner evidence

The Monday scenario is a thought experiment. The underlying pattern is not. In an October 2026 Booz Allen Hamilton lab study, unnamed frontier AI models completed every objective across eight scenarios in an isolated operational technology cybersecurity lab. In one scenario, a model progressed from a perimeter compromise to actions inside an industrial control network in just over sixteen minutes, with human approval required for physical actions. In a separate test, a model identified a robotic arm, mapped protection zones and motion limits, and moved the arm, all within minutes.

SecurityWeek, reporting on the study, framed the risk as cybersecurity controls alone being unable to govern "autonomous agents that will operate at a speed that human intervention cannot intercept." The lab was isolated and the physical actions were human-gated. The reported pace is still measured in minutes, not days.

The visibility picture is thinner than the deployment picture. Andreessen Horowitz's State of Markets II, published on 30 September 2026, reports that about 69% of S&P 500 companies now have a live AI deployment while only about 2% report a tracked metric for what their AI is doing. The gap between deployment and evidence is the broader context in which the Monday scenario sits. Firms are shipping agents at a rate that outpaces their ability to show, after the fact, what those agents actually did.

A chain-into-harm scenario does not require sophistication. It requires a reasonable set of permissions, a plausible sequence of actions, and the absence of a check that reads the sequence as evidence. The first two are already in place in most agent deployments. The third is the gap.

Where runtime governance has to run: inside your own AWS account

For a data-handling agent that touches customer records, the governance layer cannot sit in a vendor cloud if the data cannot. The constraint is architectural, not rhetorical.

OpenBox is delivered as a signed Helm chart on AWS Marketplace, with every container image pinned by digest and a software bill of materials published per release. An AWS CDK template provisions an EKS Auto Mode cluster inside the customer's own AWS account and VPC and installs the full stack end to end. The template imports an existing VPC, subnets, Route 53 hosted zone, and ACM certificate rather than creating new networking. Public endpoints can be restricted to a customer-supplied IP allowlist enforced at the load balancer, or made VPC-internal only.

The practical result is that the governance decisions, the behavioral rule evaluations, the proof certificates, and the session audit trail all live inside the customer's own environment. The control plane sits next to the agent, not in a vendor's cloud. For an engineering team with data-residency or regulated-industry constraints, that placement is the difference between a governance tool they can deploy and one they cannot.

Each session OpenBox governs produces a signed proof certificate. OpenBox builds a Merkle tree from all session events, then signs the root using AWS KMS with an ECDSA NIST P-256 key, giving the compliance team tamper-evident evidence that can be verified independently without replaying the session. That is the record a human reads later, when someone asks whether every step the agent took was within policy, and whether the pattern across all of them was too.

Takeaways

• The agent-specific risk in the Monday scenario is the chain, not any single action. Every per-action check in that firm's stack returned ALLOW. The harm still happened.

• Five governance decisions, enforced before the action executes, give specific points to intervene. ALLOW, CONSTRAIN, REQUIRE_APPROVAL, BLOCK, and HALT each cover a different operational need.

• CONSTRAIN is the decision that preserves the agent's workflow while removing the specific risk. For a data-handling agent, stripping sensitive fields or restricting output destinations is often the right answer.

• Behavioral Rules close the gap by evaluating the pattern across the session, not the action in isolation. The documented combination example Database write + File export + External send is the shape of the Monday scenario.

• Deployment inside the customer's own AWS account and VPC matters when the data governed cannot leave it. OpenBox is now available on AWS Marketplace under those terms.

FAQ

What is runtime AI agent governance?

Runtime AI agent governance is the practice of evaluating and enforcing policy on every action an AI agent takes, before the action executes, using signals that include the session's history up to that point. It differs from observability, which records what agents have already done, and from pre-deployment risk assessment, which scores an agent before it begins work. Runtime governance stops a harmful sequence before it completes, rather than describing it afterwards.

What does OpenBox's CONSTRAIN decision do?

CONSTRAIN runs an action inside a sandboxed form rather than allowing or refusing it outright. According to OpenBox, a policy can strip sensitive fields from a tool call's input, limit the resources the action can reach, or otherwise narrow its scope while still letting the agent complete its work. It is one of the five governance decisions, alongside ALLOW, REQUIRE_APPROVAL, BLOCK, and HALT.

How does OpenBox deploy on AWS?

OpenBox is delivered on AWS Marketplace as a signed Helm chart with each container image pinned by digest and an SBOM published per release. An AWS CDK template provisions an EKS Auto Mode cluster inside the customer's own AWS account and VPC, imports existing networking (VPC, subnets, Route 53 hosted zone, ACM certificate) rather than creating new infrastructure, and restricts public endpoints to a customer-supplied IP allowlist or VPC-internal only.