AI Agent Security: How to Protect Agents at Runtime

A practical guide to the risks of tool-using AI agents and the identity, policy, runtime and monitoring controls that keep them safe in production.

A traditional application does what its code says. An AI agent decides what to do next, then does it. It reads a prompt, plans a sequence of steps, calls tools, and acts on the results with limited human supervision. When the plan is wrong, or an attacker bends it, the agent does not just return a bad answer. It takes a bad action, with real credentials, against real systems.

AI agent security is the practice of controlling what an agent is permitted to do at each step, proving what it actually did, and containing it when something goes wrong. This guide sets out the agent-specific threat model and the controls security and engineering teams can implement today, from pre-deployment hardening through runtime enforcement to monitoring and incident response.

What Is AI Agent Security?

AI agent security is the discipline of protecting autonomous, tool-using AI systems from misuse, manipulation and failure across their full lifecycle. It extends beyond model safety to cover the agent’s identity, permissions, tools, memory and actions. The goal is not a safer model output. The goal is a governed action.

Ordinary LLM safety asks whether a model produces harmful text. Agent security asks a harder question: what happens when the model is allowed to act. A jailbreak that produces an offensive paragraph is a content problem. The same jailbreak inside an agent with database access and a payment tool is an operational one. The threat surface moves from the response to the action.

Familiar cybersecurity practice still applies, but it needs adaptation for this new surface. NIST’s National Cybersecurity Center of Excellence frames the shift plainly: as enterprises move AI from generating outputs to taking actions, the scale and range of those actions can grow sharply, which brings new risks alongside the gains (NCCoE, 2026). Identity, least privilege, logging and monitoring remain the right instincts. What changes is that they now have to govern a system whose next step is decided at runtime.

The AI Agent Attack Surface

The AI agent attack surface spans every component an agent touches: the model, its memory, its identity, the tools and MCP servers it calls, the data and APIs behind those tools, other agents it coordinates with, and the human approval points that gate high-impact actions. Each is a place where an attacker can influence a decision or an action.

Each component in the table below is both a capability and an exposure. A visual of this attack surface would sit here in the published article.

OWASP’s Agentic Security Initiative, part of the OWASP GenAI Security Project, has published agentic threat guidance since early 2025 and, in December 2025, released a formal Top 10 framework for these risks as the OWASP Top 10 for Agentic Applications (ASI01 to ASI10), grounded in real incidents and attacks rather than hypotheticals alone. The sections below map each major risk to that framework.

The Main AI Agent Security Risks

The main AI agent security risks are goal hijacking, tool misuse, identity and authorisation failures, memory poisoning, insecure tool and MCP integrations, supply-chain compromise, cascading multi-agent failures, and rogue behaviour. OWASP’s Top 10 for Agentic Applications catalogues them as ASI01 to ASI10.

The table maps each risk to how it happens and to its OWASP identifier, so the taxonomy stays traceable to a primary source.

Two further OWASP categories cut across the table. Unexpected code execution (ASI05) is where a natural-language instruction reaches a code path and runs. Human-agent trust exploitation (ASI09) is where a confident, well-worded agent persuades a human reviewer to approve a harmful action. OWASP also publishes a dedicated guide for secure MCP server development, because the Model Context Protocol is now a common channel for tool and supply-chain risk.

These are not projections. OWASP’s wider body of work, including its State of Agentic AI Security and Governance report, describes a year in which documented incidents accumulated across many of the ten categories, and treats the taxonomy as a shared language for securing autonomous systems rather than a list of future problems (OWASP GenAI Security Project, 2025). That shift, from plausible risk to documented incident, is the reason the controls that follow belong in production rather than on a roadmap.

Pre-Deployment Controls

Pre-deployment controls shrink the blast radius before an agent ever runs. The core moves are threat modelling the agent’s specific tools and goals, testing it adversarially, granting least privilege, pinning and signing dependencies, sandboxing execution, and defining clear human-approval boundaries for high-impact actions.

Threat model the agent, not just the app. Map the tools it can call, the data it can read, the identities it uses, and the downstream systems its actions reach. The question is not what the agent is meant to do, but what it becomes capable of once it holds those tools.

Test adversarially before production. Red-team the agent for prompt injection, tool misuse and privilege escalation. Model-layer defences are probabilistic, so testing has to be continuous rather than a one-time gate. Reducing the attack volume is a real gain even though it does not remove the risk.

Grant least privilege. Scope each tool and API to the task, not to the operator who launched the agent. NIST is working the same territory: its NCCoE is exploring how existing identity and authorisation standards should apply to agents, which is the foundation least privilege depends on (NCCoE, 2026).

Harden the supply chain and sandbox execution. Pin and sign the models, plugins, dependencies and MCP servers the agent relies on, and run the agent inside a sandbox with resource and time limits. Both steps contain what a compromised component can do.

Set approval boundaries. Decide, before launch, which actions an agent may take on its own and which require a human to sign off. A clear boundary is what makes runtime escalation meaningful later.

Runtime Controls and AI Runtime Security

Runtime controls govern the agent while it is acting, because pre-deployment hardening cannot anticipate every prompt or plan. AI runtime security means evaluating each action before it executes, enforcing identity and policy, watching behaviour across steps, capping rate and transaction limits, escalating to humans, and keeping a way to stop the agent immediately.

Per-action authorisation. The central runtime control is a check on each action before it runs, rather than a check on the prompt. Stateless per-operation policies decide whether a single action is permitted; stateful checks look at patterns across a whole session. Policy-as-code engines such as OPA/Rego are a common way to express these permission rules.

Agent identity. Give each agent its own cryptographic identity, distinct from a shared service account, so an action can be attributed to the agent that took it. Decentralised identifiers (DIDs) with Ed25519 signing keys are an emerging pattern for this, separating the credential that reaches the platform from the key that proves which agent produced a request.

The separation matters when a credential leaks. In OpenBox, an agent’s API key authenticates the HTTP call while its Ed25519 key signs the payload, so a leaked API key alone does not let an attacker act as the agent, and a per-agent setting can reject any unsigned request (docs.openbox.ai). NIST’s NCCoE is exploring how existing identity standards should extend to agents in exactly this way, and NIST’s AI Agent Standards Initiative names agent authentication and identity as a core research area (NIST, 2026).

Behavioural monitoring, limits and escalation. Watch for multi-step patterns and goal drift that no single action would reveal. Cap rate, spend and transaction limits so a runaway loop cannot do unbounded damage. Route high-impact actions to a human for approval, and keep the agent paused until that decision returns.

Reversibility and kill switches. Prefer actions that can be undone, and keep a kill switch that halts the whole session at once. Blocking a single action stops one mistake; halting the session stops a compromised agent.

Multi-agent coordination. When agents call other agents, two OWASP risks compound: insecure inter-agent communication (ASI07) and cascading failures (ASI08). A spoofed message can misdirect a cluster, and one wrong output can propagate through the chain with escalating impact. The runtime defences are the same primitives applied to agent-to-agent traffic: authenticate the sending agent, authorise the requested action, and cap how far a single failure can spread before a circuit breaker or a session halt stops it.

Runtime governance for AI agents has become a defined engineering category over the past year. Microsoft released the open-source, MIT-licensed Agent Governance Toolkit in April 2026, describing it as the first toolkit to address all ten OWASP agentic risks with deterministic, sub-millisecond policy enforcement that intercepts each action before it executes (Microsoft Open Source Blog, 2026). It is framework-agnostic and ships seven packages, including a stateless policy engine and DID-based identity. Its arrival, alongside managed platforms, is the clearest sign that runtime interception, identity and tool-call control are now treated as an engineering discipline rather than an afterthought.

Layered enforcement is how this looks in one managed platform. OpenBox, an AI agent governance platform, runs each operation through guardrails that validate or transform input and output, then stateless Policies written in OPA/Rego, then stateful Behavioral Rules that detect multi-step patterns across a session (docs.openbox.ai). The pipeline returns one of four governance decisions: ALLOW, REQUIRE_APPROVAL, BLOCK or HALT, with precedence HALT over BLOCK over REQUIRE_APPROVAL over ALLOW.

Monitoring and Incident Response

Monitoring and incident response turn runtime governance into something you can prove and act on. At minimum, log every action, tool call, decision and approval; detect anomalies such as goal drift and unusual tool sequences; and be able to contain, roll back and investigate an incident after it happens.

What to log and prove. Capture each governance event, the decision taken at each step, and the human approval trail. For evidence an auditor or investigator can verify independently, the record itself needs to be tamper-evident, so they can confirm the timeline was not altered after it was recorded.

OpenBox is one worked example of that evidence model. It hashes each session event with SHA-256, combines the hashes into a Merkle tree, and signs the root with ECDSA NIST P-256 through AWS KMS by default, producing a proof certificate for each session (docs.openbox.ai). The certificate carries the Merkle root, the signature and the event count, which is what makes the audit trail tamper-evident rather than merely stored.

Detect. Score goal alignment across sessions and flag drift events when an agent’s actions diverge from its stated goal. Track tool and MCP health so a degraded dependency shows up before it causes a failure. Anomalous tool sequences and sudden shifts in behaviour are the signals worth alerting on.

Contain, roll back and investigate. Containment is the kill switch: halt the session and stop pending actions. Investigation is replay. OpenBox’s Session Replay steps through a session event by event, showing each tool call, the governance decision at that step, and the full input and output payload, so an investigator can see which action broke and what it was given (docs.openbox.ai). Because the underlying record is signed, the events you replay can be verified against that signature and shown not to have been altered after they were recorded. That establishes the integrity of the governance record, not that every downstream action succeeded in the outside world.

No control set removes agent risk entirely. The realistic aim is to reduce it, and to make what remains visible, attributable and reversible.

AI Agent Security Checklist

This AI agent security checklist gathers the controls a team should have in place before and during production. It is written to be scanned by both developers and security teams and to be adopted incrementally.

Before deployment

• Threat model the agent’s tools, data, identities and downstream actions

• Red-team for prompt injection, tool misuse and privilege escalation before launch

• Grant least privilege: scope each tool and API to the task, not the operator

• Pin and sign models, plugins, dependencies and MCP servers

• Sandbox the agent’s execution with resource and time limits

• Define which actions require human approval

At runtime

• Evaluate every action against policy before it executes

• Give each agent a distinct cryptographic identity, not a shared service account

• Detect multi-step patterns and goal drift across a session

• Enforce rate, spend and transaction limits

• Route high-impact actions to a human for sign-off

• Keep a kill switch that halts the session immediately

Monitoring and evidence

• Log every action, tool call, decision and approval

• Produce a tamper-evident record of each session

• Alert on anomalies, degraded tools and misalignment

• Be able to replay, roll back and investigate an incident

Where OpenBox Fits

OpenBox is an AI agent governance platform that applies these runtime controls as a managed service. It wraps existing agents, including those built on Temporal, LangGraph, Mastra and CrewAI, gives each a cryptographic identity, and governs each action through the four decisions above (docs.openbox.ai).

Microsoft’s Agent Governance Toolkit shows how quickly this category has formed, and the two products sit at different ends of it. The toolkit is an open-source, MIT-licensed library that teams install and run themselves, with a stateless policy engine, DID-based identity and mappings to the OWASP Top 10 (Microsoft Open Source Blog, 2026). OpenBox sits at the managed end: a hosted Trust Lifecycle, a dashboard, per-session cryptographic proof certificates and session replay, rather than a toolkit to self-host. Which one fits depends on whether a team wants to own the governance stack or consume it.

For security and compliance teams, the practical payoff is a single record. Every agent action carries a governance decision, and every session carries verifiable evidence that the decision was recorded as stated (docs.openbox.ai). That is what turns AI agent security from a policy on paper into something a team can enforce at runtime and prove afterwards.

Frequently Asked Questions

What is AI agent security?

AI agent security is the practice of protecting autonomous, tool-using AI systems from misuse, manipulation and failure across their lifecycle. It governs an agent’s identity, permissions, tools, memory and actions, not just its text output, so that each action the agent takes is controlled, attributable and reversible.

How is agentic AI security different from LLM security?

LLM security focuses on the model’s output: whether it produces harmful or leaked text. Agentic AI security focuses on the model’s actions: what happens when it can call tools and change real systems. The same jailbreak is a content issue in a chatbot and an operational one in an agent with tool access.

What are the biggest AI agent security risks?

The biggest risks are goal hijacking, tool misuse, identity and authorisation failures, memory poisoning, insecure MCP and third-party tools, supply-chain compromise, cascading multi-agent failures and rogue behaviour. OWASP catalogues them as ASI01 to ASI10 in its Top 10 for Agentic Applications, published in December 2025.

What are AI agent guardrails?

AI agent guardrails are controls that validate or transform an agent’s inputs and outputs, such as masking sensitive data or blocking unsafe content. They are one layer of runtime enforcement, sitting alongside per-operation policy checks and multi-step behavioural rules that together decide whether an action may proceed.

What is runtime governance for AI agents?

Runtime governance for AI agents is the practice of evaluating and controlling each action while the agent runs, rather than only hardening it beforehand. It covers per-action authorisation, agent identity, behavioural monitoring, rate and transaction limits, human escalation and a kill switch, plus a record of every decision taken.

Can AI agent security risks be eliminated?

No. Model-layer defences are probabilistic and no single control removes agent risk entirely, a point both OWASP and Microsoft make. The realistic goal is defence in depth: reduce the attack surface before deployment, enforce controls at runtime, and make what remains visible, attributable and reversible.

Sources