When the Guardrails Are in the Model, the Attacker Picks the Model

Why model-level AI safeguards are not enough to govern enterprise agents, and where the enforceable control belongs.

By Tahir Mahmood, Co-founder & CTO, OpenBox  ·  Last updated 3 November 2026

What model-level safeguards are, and why they are a ceiling, not a floor

Model-level safeguards are the refusals and classifiers a model provider builds into and around a model, including behaviour trained into the model and, depending on the service, external input and output classifiers. They are necessary, and they work often. But they are probabilistic: they act on the model’s own reading of the intent behind a prompt, a reading your organisation does not control and that varies from one prompt to the next and from one model to another. That makes them a ceiling on misuse, useful for holding the average case down, rather than a floor you can enforce.

The distinction matters for a security team. A model refusal is a judgement the model makes about the text in front of it. Anthropic put the limit plainly in its September 2026 threat report, writing that in highly technical dual-use areas "a classifier cannot simultaneously enable benefit and prevent harm," because it cannot reliably identify the user’s intent. That is not a criticism of classifiers. It is a statement about what any probabilistic, intent-reading control can and cannot do. The operative word is "probabilistic." A control that decides on the balance of probabilities is valuable, but a model refusal on its own does not show that an explicit authorisation policy was enforced at a defined boundary, which is what an auditor asks to see.

None of this means classifiers are optional. They refuse many facially malicious requests before those requests ever become actions (Anthropic reports a nine-in-ten refusal rate in one of the cases below), and the rest of this argument assumes you keep them. The point is narrower: a safeguard that lives inside the model is the model’s behaviour. For a hosted proprietary model, the enterprise consumes the provider’s safeguard behaviour without necessarily controlling or independently versioning the underlying classifiers. That is why securing AI agents requires a runtime layer in addition to the model’s own refusals.

Three ways model safeguards degrade under pressure, from Anthropic’s own report

Anthropic’s September 2026 threat report, which covers misuse it disrupted between December 2025 and August 2026, documents three field-observed ways model safeguards slip under pressure: fragmenting a task across smaller sessions, re-prompting after a refusal, and routing the work to a weaker, less-safeguarded model. Anthropic presents these as notable and novel cases rather than typical misuse, and together they show safeguards being worked around or outflanked in different ways rather than defeated head-on.

The first is fragmentation. In a case Anthropic attributes to an actor it tracks as GTG-30006, "Claude refused nine out of ten direct requests that were facially malicious. But our safeguards performed less consistently when the user fragmented the work and directed the model to carry out tasks across later, smaller sessions." The refusal held against the blunt request and slipped when the same work was broken into benign-looking pieces spread over time.

The second is re-prompting. In a separate state-linked surveillance case, an initial refusal did not settle the matter. Anthropic reports the actor "was able to re-prompt the model to produce functional suppression guidance naming 10 private citizens," and its own case summary records the outcome as a refusal "reversed on re-prompt." The first "no" was not the last word.

The third is model-switching. In one biological-research investigation, Anthropic found that a reseller platform had "modified the service to route biology prompts to other, more permissive models," with a pre-deployment test that "failed if the prompts reached Claude instead of a more-permissive model." In a separate biological-research case, Anthropic found a researcher using Claude Sonnet 4 and Haiku 4.5, "models in our weakest class of models," and concluded that safeguards on its frontier models were "robust enough to force researchers to use weaker and less safeguarded models." Across its cases, Anthropic describes state actors who "routinely evade our access controls via relays, ZDR abuse, multi-model fallback."

These are three different actors in three different domains, and that is the point. The weakness is not specific to one prompt, one model, or one kind of harm. It is a property of where the control sits.

Why "pick a different model" is the structural problem

When the control lives inside the model, defeating it becomes a model-selection problem for the attacker. If one model refuses, the work can be fragmented, re-prompted, or routed to a model that will answer. Anthropic’s own conclusion is that safe access to its most sensitive capabilities "will necessarily require account and institutional signals to verify user legitimacy," not refusals alone.

The three evasion modes above share one shape. The safeguard is attached to the model, so the attacker’s move is to change the conditions the model sees, or to change the model. For its most sensitive domain, Anthropic goes as far as concluding that "the only safe way to serve frontier biological capabilities is to offer them in trusted user programs." Read that as an enterprise-security lesson rather than a biosecurity one, and the reasonable inference is uncomfortable: a control you do not own, and cannot pin to a specific action, can be swapped out from under you.

An enterprise can choose and restrict which models its agents may use, but it does not control how a given model refuses, nor how a vendor retunes that model’s classifiers next quarter, and an agent left free to reach for an alternative can land on a weaker one. Betting the safety of a production agent on the internal refusal behaviour of a model you neither tune nor version is a fragile position, however good that model’s refusals are today.

Put the enforceable control at the action boundary

The control an enterprise can actually enforce sits where the agent acts: the tool call, the API request, the transaction the agent tries to execute. A runtime governance layer evaluates each of those actions against your policy and returns a decision, regardless of which model generated the intent behind it. Where the policy uses explicit rule-based conditions, its decision on a given action is deterministic for that action and policy state, an enforceable floor beneath the probabilistic ceiling, even though no single layer guarantees a safe outcome, and it is why runtime enforcement comes before compliance.

A model classifier asks, in effect, "should I answer this?" An action-layer control asks a narrower question: "is this agent allowed to take this action, here, now?" The second question is one an enterprise can answer deterministically, because it turns on the action and the policy, not on inferring intent from text. OpenBox, an AI agent governance platform, evaluates each agent action that a supported enforcement integration passes to it against policy in real time and records the decision. Because the check is on the action rather than on the model’s output, changing the model underneath does not move the control. An agent that switches from one provider to another still has to clear the same action-level policy, as long as its actions run through the governed integration.

That last point is an inference from how the layer is built rather than a claim about any single model, so it is worth stating carefully. OpenBox governs agents across a range of agent frameworks, workflow engines and model-routing integrations, including LangChain, LangGraph, CrewAI, Temporal and OpenRouter, and the governance decision is taken on the action the agent attempts. Every action that passes through a supported enforcement integration is evaluated against the same action-level policy, regardless of which model produced the request; an action that reaches a tool through an ungoverned route sits outside that boundary. This does not remove the model’s classifier. It adds a layer that still holds when the model’s behaviour changes.

What a runtime governance decision looks like

At the action boundary, OpenBox returns one of five governance decisions for each agent action, evaluated against policy in real time: ALLOW, CONSTRAIN, REQUIRE_APPROVAL, BLOCK or HALT. In plain terms:

– ALLOW: lets the action proceed without interruption.

– CONSTRAIN: executes the action inside a sandboxed environment rather than allowing or refusing it outright, for example stripping sensitive fields from a tool call’s input or limiting which resources the action can reach.

– REQUIRE_APPROVAL: pauses the call for a designated approver to review it, where the integration supports an approval workflow.

– BLOCK: refuses the individual call and returns the policy’s reason to the developer.

– HALT: terminates the agent session outright, cancelling the current action and any pending operations.

CONSTRAIN stands apart from the other four: it neither simply allows nor refuses, but runs the action under sandboxed limits. The documented precedence runs HALT over BLOCK over REQUIRE_APPROVAL over CONSTRAIN over ALLOW, so the stricter decision wins.

These decisions are configured in the Authorize stage of OpenBox’s five-stage trust lifecycle (Assess, Authorize, Monitor, Verify, Adapt), and each attaches to an action and is evaluated at runtime. An explicit action-level rule holds steady when the model changes: re-prompting or a switch to a weaker model changes what the model does, not whether the action clears policy. Fragmentation is harder, because the harm can live in a sequence of individually permitted actions that a per-action check does not see, which OpenBox addresses with stateful behavioural rules that detect multi-step patterns across a session. When the session ends, OpenBox can produce signed, tamper-evident evidence of the events and decisions it captured, which attests what the integration recorded rather than guaranteeing every outcome. That is the difference between a best-effort refusal and an enforced, evidenced control.

The takeaway for security leaders

Treat model refusals as a valuable best-effort layer, not as a control you can enforce or evidence. Keep the classifiers. Then add the layer you own: policy at the action boundary, so that fragmentation, re-prompting and model-switching change the model’s behaviour but not your enforcement.

Anthropic’s report is, among other things, an unusually honest account from a model developer of where model-layer safety ends. The same report that shows safeguards slipping under pressure also points to where durable control has to live: account and institutional signals, trusted-access programmes, and controls that do not depend on a single model’s refusal. For an enterprise running agents in production, the control it owns is at the action boundary. Security leaders who want a concrete starting point can govern individual agent actions at runtime. Model classifiers raise the ceiling. A runtime governance decision on each action is the floor that holds when you change models.

Frequently asked questions

Does runtime governance replace model safety classifiers?

No. Model classifiers remain a necessary first layer that catches many facially malicious requests before they become actions. Runtime governance is additive: it enforces policy on the agent’s action regardless of which model produced it, so the two layers cover different failure modes. The sound position is to keep both.

Can an attacker bypass action-layer governance by switching models?

Switching models does not move an action-layer control. OpenBox evaluates each agent action against policy at runtime and returns a decision regardless of the model behind it, so re-prompting or model-switching change the model’s behaviour, not the enforcement at the action boundary. Fragmented workflows, where each step is individually permitted, are the harder case and need stateful behavioural rules that look across the whole session.

Is runtime governance the same as logging or monitoring?

No. Logging preserves records and monitoring identifies suspicious activity, but neither provides preventive enforcement unless it is wired to a control that can intervene before or during execution. A runtime governance decision (ALLOW, CONSTRAIN, REQUIRE_APPROVAL, BLOCK or HALT) is made before the action executes and can stop, constrain or gate it. OpenBox also attests the session afterwards, so you get enforcement first and evidence second.

Sources