Why "Block the Bad Prompt" Fails When the Same Action Is Both Safe and Harmful

By Tahir Mahmood, Co-founder & CTO, OpenBox  ·  Last updated 4 November 2026

Why can't you just block the bad prompt?

You cannot, because a dual-use action is safe or harmful depending on context, not on wording. A classifier that judges the request text has to infer intent from that text, but the same database export is a routine report for one agent and an exfiltration for another. Block the request and you either stop legitimate work or wave through the harmful version, because the words look the same in both.

This is the structural problem with a prompt-level gate. It evaluates the request, not the situation. A gate that sees only the prompt has to guess the intent behind it, and for an action that serves two purposes at once there is no reading of the text that is right in every case. The problem is not that the classifier is inaccurate; it is that the information that settles the question, the actor, the records, the destination, the policy, lives outside the request. The interesting governance decisions are the ones a yes or no gate cannot represent.

The dual-use problem, in one sentence

Anthropic's September 2026 threat report states the limit directly: in highly technical dual-use areas "a classifier cannot simultaneously enable benefit and prevent harm," because it cannot reliably identify the intent of the user. Describing its venom-and-toxin cases, the report says the work "proceeded largely unimpeded by our biological safety classifier," and adds: "This was by design."

That phrase, "by design," needs its context. The classifier was built to restrict information that could make known biological weapons accessible to novices, not to block every line of research with both therapeutic and harmful uses, so in those cases the content passed because it served a legitimate purpose as readily as a harmful one. The report's separate orthopoxvirus case makes the same point from the other side: "given the dual-use nature of this research and its explicit focus on attenuation, Claude supplied the user with information and thus was not blocked by our classifiers." A control that reads intent from a single request reaches its floor exactly where intent stops being legible.

Not just a biology problem: your agents do this every day

The dual-use structure maps directly onto the actions enterprise AI agents take. A data export, a payment, a schema change: each one is legitimate or dangerous only in context. Anthropic's report is about biological research; applying its logic to enterprise AI agent governance is an analogy we draw, not a finding Anthropic reports. Drawn out that way, the uncomfortable part is that your agents meet this problem daily, on ordinary operations.

Consider an illustrative scenario: a data-handling agent that can read customer records and send a file to a configured destination. Both actions are approved. Whether a given export is a weekly report or a leak depends on which records, which destination, and which agent, none of which a prompt classifier evaluating the instruction can see. The request to "export the customer file" is the same sentence in the safe case and the damaging one. The meaning lives in the context around it.

The decisions that live between allow and block

A binary gate offers two answers; a dual-use action needs more than two. OpenBox returns one of five governance decisions on each agent action it evaluates: ALLOW, CONSTRAIN, REQUIRE_APPROVAL, BLOCK or HALT. The precedence runs HALT over BLOCK over REQUIRE_APPROVAL over CONSTRAIN over ALLOW, so a stricter verdict always overrides a looser one. The two decisions in the middle are the ones a yes or no gate cannot express.

REQUIRE_APPROVAL pauses the operation and routes it to a human to approve or reject before it proceeds. It is the answer for an action that is defensible in some contexts and not others, where a person, not a classifier, should make the call. CONSTRAIN runs the action inside a sandboxed form rather than allowing or refusing it outright: a policy can strip sensitive fields from a tool call's input or limit which resources the action can reach, so the agent completes its work while the specific risk is removed. BLOCK refuses the individual action and lets the session continue; HALT terminates the whole session; ALLOW lets the action run.

Set against the dual-use export, the point is concrete. A binary gate must either allow the export, and accept the leak, or block it, and break the report. A graded model can let the report run with sensitive fields stripped and destinations restricted, or pause it for a human when the destination is unusual. Those are the decisions that were missing from the conversation.

All of this assumes the action is mediated by the governance layer. An operation an agent can reach by an ungoverned route sits outside the policy, which is a condition of runtime enforcement in general, not a detail unique to one product.

What makes a graded decision possible: the actor, the context, the trust level

Graded decisions and context are two separate requirements, and both are needed. More verdicts are useless without the information to choose between them, and rich context is wasted if the system can only answer yes or no. A binary engine with full context still cannot express the middle, and a five-verdict engine without reliable action metadata still decides blind.

A graded decision needs inputs a prompt classifier does not have: who the agent is, how it has behaved, and the context of the action it is attempting. OpenBox derives these from an agent-level Trust Score, calculated as (Risk Profile Score × 40%) + (Behavioral × 35%) + (Alignment × 25%), and maps the score to trust tiers that scale controls, from Tier 1 with minimal constraints to Tier 4 with strict governance and human review.

The score is what lets the decision turn on the actor rather than the sentence. The same export can resolve differently when it is attempted by a long-compliant Tier 1 agent, by a new agent that the current tier model typically places at Tier 3, and by a degraded Tier 4 one under stricter governance, because the trust level and the context, whether the action crosses a risk threshold, whether the tier mandates review, feed the verdict. A prompt classifier sees one request in isolation; a trust-based control sees the request, the actor, and the history together. That is the difference between guessing intent and enforcing a policy.

This trust signal is partly dynamic. In OpenBox's model the Risk Profile component stays fixed unless the agent is reassessed, while the Behavioral component updates continuously as the agent complies with or violates policy and the Alignment component updates per session from goal-alignment checks, so an agent's score moves as it behaves. An agent that drifts can fall to a lower tier and meet stricter decisions on the same action it was once trusted to take, and one that recovers earns its autonomy back. The decision tracks the actor over time, which a one-off reading of a prompt cannot do.

The score is an input to the decision, not a verdict on intent. A Tier 1 agent can still attempt an unauthorised action, so the trust level weights the control while the action, the resources, and the policy are still evaluated in their own right.

From "is this prompt bad" to "is this actor allowed to take this action, here, now"

The reframing is the whole argument. The useful question is not whether a request looks malicious, but whether this actor, at this trust level, in this context, may take this action, and under what constraint. Given complete context and a defined policy, that question has a reproducible answer, because it turns on the action and the policy rather than on inferring intent from text. Reproducible is not the same as correct: the answer is only as good as the context and the policy behind it, so stale identity or an incomplete policy will degrade it.

Anthropic reaches a parallel conclusion for its own hardest domain. Having found that classifiers alone cannot resolve dual-use intent, the report concludes that safe access "will necessarily require account and institutional signals to verify user legitimacy," and, for frontier biological capabilities, that "the only safe way to serve" them "is to offer them in trusted user programs." Translated out of biosecurity and into enterprise agents, which is our reading rather than Anthropic's claim, the same logic holds: when the request cannot tell you whether an action is safe, the actor, the context, and the trust behind them have to. A binary allow or block gate, on its own, forces a poor trade-off: block legitimate work, or wave through the dual-use action whose risk depends on context. Telling apart the cases that need a restriction, a human review, or an outright refusal takes more than two answers. The decisions worth having live in between.

Frequently asked questions

Can a better classifier solve the dual-use problem?

Not on its own. A prompt-level classifier infers intent from the request text, and for a dual-use action the same text serves a safe and a harmful purpose, so no reading is correct in every case. Anthropic's September 2026 report states that in such areas "a classifier cannot simultaneously enable benefit and prevent harm." A better classifier raises the floor; it does not change where the floor is.

What is the difference between BLOCK and CONSTRAIN?

BLOCK refuses the specific action outright and lets the agent session continue with other work. CONSTRAIN lets the action run but inside a sandboxed form, for example with sensitive fields stripped from the input or the reachable resources limited, so the agent completes its task while the specific risk is removed. BLOCK is a refusal; CONSTRAIN is a narrowed yes.

Does action-layer governance replace model safety classifiers?

No. Model classifiers remain a useful first layer that refuses many facially harmful requests before they become actions. Action-layer governance is additive: it decides on what the agent actually does, using the actor and context, regardless of which model produced the intent. Keeping both covers failure modes neither covers alone.