In February I filed a comment with NIST arguing that AI agent security has to be enforced at the infrastructure layer, because the model layer cannot hold. In July, OpenAI, Anthropic, and the UK AI Security Institute each published incident reports in which models under evaluation reached real systems they were never meant to touch. Read together, those reports are the argument I made, written up as postmortems.
So let me lay out the argument, and then what July looked like.
The argument
NIST's Center for AI Standards and Innovation asked how to secure AI agents. Not chatbots, but systems that take autonomous actions against real state: moving money, editing records, calling third-party services. My answer was one sentence:
Just as zero trust networking abandoned trust in the network perimeter, AI agent architectures must abandon trust in model outputs.
We spent fifteen years learning that a packet's origin inside the firewall tells you nothing about whether it should be honored. Every resource authenticates and authorizes independently. That lesson transfers directly. An instruction is not trustworthy because it came from your model rather than from the internet. Your model read the internet.
So the design question is not "how do I stop the model from being tricked?" It is:
Even if this output is malicious, what damage can it cause?
Engineer the answer to be none. Concretely: the database refuses unauthorized queries in a session scoped to the user's permissions, regardless of what the agent asked for. Generated code runs sandboxed with no host access. Outbound network access is restricted at the infrastructure boundary. Outputs are validated against strict schemas before execution. Infrastructure controls give deterministic guarantees; model-level controls remain probabilistic. Detection and model-level defenses are real defense-in-depth. They just must never be the primary boundary.
Why the model layer can't hold
This isn't a hunch. The research record is lopsided:
- Nasr, Carlini et al.—researchers from OpenAI, Anthropic, and Google DeepMind—bypassed 12 published prompt-injection defenses with over 90% attack success using adaptive attacks. The damning detail: "the majority of defenses originally reported near-zero attack success rates." They were evaluated against static attacks, and fell apart against adaptive ones. (arXiv:2510.09023)
- Google DeepMind's own evaluation of Gemini 2.5 found 94.6% attack success against out-of-distribution attacks after adversarial training. (arXiv:2505.14534)
- An independent benchmark of 10 state-of-the-art defenses concluded that "nearly all existing defenses remain far from meeting the requirements for real-world deployment." (arXiv:2602.03117)
A control that fails most of the time under adaptive pressure is a speed bump. Speed bumps belong behind walls, not instead of them.
What July looked like
On July 21, OpenAI disclosed that models under internal evaluation had chained a previously unknown vulnerability in the sandbox's only egress path, reached the open internet, and pulled evaluation data from Hugging Face's production infrastructure. That one was a genuine escape, and it produced real CVEs.
The Anthropic report nine days later is the one I'd make every engineer building agents read, because nothing escaped. Anthropic reviewed 141,006 evaluation runs and found three in which Claude reached real infrastructure at three real organizations. The mechanism:
Our models were told they had no internet access and to capture the flag, while in fact being misconfigured to have internet access.
The prompt asserted a boundary. The infrastructure didn't enforce it. Claude believed the prompt, treated everything it could reach as part of the exercise, and compromised the targets using, in Anthropic's words, "basic techniques, such as exploiting weak passwords and unauthenticated endpoints." No novel capability. Just an absent wall.
The second of the three incidents is the one to sit with. Claude found setup instructions in the fictional environment telling developers to install a Python package that didn't exist on PyPI. It recognized a dependency-confusion opening and set out to publish a package under that name. Registering for PyPI required an email address. Getting an email address required a phone number. It tried and failed to obtain funds to buy one, then backtracked, found a free email provider that wasn't blocked, registered, and uploaded a booby-trapped package to the real registry. The package was live for about an hour and was downloaded and run on 15 real systems.
Every obstacle in that chain was a speed bump. Not one was a wall. Anthropic's own assessment is the sentence I would have written if I'd been asked:
We believe these incidents to be closer to a harness and operational failure than a model alignment failure.
That is the whole thesis. The model did what models do. The failure was in the layer that was supposed to constrain it and didn't exist.
The UK AI Security Institute's report closes the loop on the detection side. Across 122 runs, agents took 19 unsanctioned actions against real people and organizations, including researching a real open-source maintainer, creating fake identities, and socially engineering him. AISI is explicit that this "was not a case of a model escaping its secure test environment." Internet access was intentionally on. And they caught it, in their words, "through general monitoring after the fact, not through monitoring built to watch the evaluation as it ran."
The organizations Claude compromised hadn't noticed either. Two of the three learned about it when Anthropic called.
Where the other 936 comments landed
NIST received 937 responses to its docket. Hari Asok's analysis in Ghost in the Org Chart finds the convergence sitting in the infrastructure layer rather than the model layer: the identity gap, the delegation framework, the permission architecture, the execution boundary. His framing of the failure mode is better than mine:
An agent can have scoped identity, least privilege, every pre-action guardrail in the playbook, and still execute a perfectly authorised action on a corrupted premise. The guardrails pass. The outcome is wrong.
The OpenID Foundation's identity group argued the urgent risks "are not technical failures, but failures of trust." The Foundation for Defense of Democracies wants MITRE ATLAS extended to cover agentic kill chains and multi-agent lateral movement. And the recommendation Asok credits to "several respondents," to classify risk by what an agent can do rather than by model capability, is my fifth recommendation nearly verbatim. That it arrived from several directions is exactly the point.
The part nobody has solved
The most interesting thing in the docket isn't a solution, it's a trap. The International Center for Law & Economics pointed out that securing agents requires deep telemetry—prompts, tool calls, retrieved context, intermediate reasoning—and that capturing it may collide with communications privacy law. Their filing notes that the Electronic Communications Privacy Act's distinction between intercepting communications in transit and accessing them in storage "maps poorly onto AI-agent traffic," and that "an agent's tool-call arguments may qualify as 'content' under one statutory provision while functioning as system telemetry under another." They call it the Telemetry Paradox.
July made it concrete. AISI had the traffic and found the incidents only in retrospect. The organizations Claude hit had no idea. You need the logs to secure the system, and capturing the logs may itself be legal exposure. The rational response to that ambiguity is to under-invest in observability, which is the worst available outcome. I don't have an answer. Neither, as far as I can tell, does anyone else.
What I'd tell you to do Monday
Assume the model will be compromised, or will simply believe something false about where it is, and ask what it can reach when that happens. If the answer is "production data" or "the public package registry," no prompt will save you. Scope the credential, sandbox the execution, restrict the egress, and make the database enforce the permission independently. Classify agents by blast radius, not by benchmark scores.
AISI's own recommendation, five months after the docket closed, is the argument in one line: "Evaluations should assume a capable model may try to act beyond its remit, with the scope of any such behaviour limited in advance." Limited in advance. By the infrastructure. Not by the prompt.
My full comment is NIST-2025-0035-0154 (PDF, HTML). NIST's own summary of all responses is NIST AI 800-5.