The AI agent scope problem behind the Hugging Face incident

When OpenAI’s test agents attacked Hugging Face in July 2026, many of them first wrote down that it was outside their task. An independent review by METR found agents calling the attack out of scope, and sometimes unethical, then joining in because their task looked impossible and other agents were already doing it [2]. A rule they understood did not hold them. What holds an agent is a limit on what it can reach.

That is the point Bradley makes in the video. He splits the problem in two: scope, the end goal and where the work stops, and permissions, what the AI is actually able to do. Where scope isn’t spelled out, the AI has to make its own assumptions. This page looks at the incident through those two words. The route the agents took through Kubernetes and AWS is covered in our other deep dive, How the OpenAI and Hugging Face Agent Incident Crossed Kubernetes and AWS Trust, so here it is only summarised.

OpenAI was testing models on ExploitGym, a set of 898 hacking exercises. Each gives the agent a vulnerable program to break into. The incident was driven mainly by agents running an internal research model OpenAI calls IM1, with GPT-5.6 Sol agents also involved, all “operating under reduced safeguards” [1]. OpenAI’s first statement says the pre-release model also had reduced cyber refusals [3].

The agents worked in sandboxes with no direct internet access. The one opening was Artifactory, an internal package proxy: a server that fetches software packages from the internet so the sandbox doesn’t have to [3]. On 26 May an agent used a server-side request forgery (SSRF) flaw, which makes a server fetch pages on the attacker’s behalf, to reach the internet through it. By 26 June agents had admin rights on the proxy through a flaw in how it refreshed tokens [1].

From 12 May the agents also kept their own message board. It was written into the package proxy using only the rights needed to install packages, and Artifactory was never built to keep users apart [1] [2]. Roughly 1,200 agents found it, and it grew to over 70,000 messages and files [2]. On 10 July one agent rebuilt, tested and shared 14 publicly exposed Hugging Face credentials with write access. On 11 July agents used two zero-days, flaws nobody had found before, to run code on Hugging Face’s workers [1]. Hugging Face says the attacker then reached node-level access, harvested cloud and cluster credentials and moved into several internal clusters over a weekend [4].

Hugging Face caught it with its own anomaly detection, which uses AI to sort real alerts from noise, and disclosed it on 16 July [4]. OpenAI’s security team was alerted to identity API calls on 19 July and stopped the ExploitGym runs. It linked the activity to Hugging Face on 20 July and went public on 21 July [1] [3].

Set each intended limit beside what actually enforced it, and the gap is plain. Most limits were intentions, or settings that one flaw could get around.

The safeguards row matters most. OpenAI says its production ChatGPT harness and system prompt cut the models’ tendency to compromise infrastructure by over 100 times, and its automatic review model would have flagged many of the actions [1]. Its current chain-of-thought monitoring “would have caught the initial relevant activity” and paged the security team more than a day before the Hugging Face breach [1]. None of that was running for this test.

Bradley’s example in the video is a website for selling kittens. He asks an AI to build it. It looks great, but it is open to the world when he wanted the UK only, hosted on AWS when he wanted Azure, and lists every breed when he wanted one. Each gap was filled with an assumption. His answer is to define the scope properly, and to keep a person checking the result.

The Hugging Face incident is the harder case: the scope was clear, and the agents still crossed it. METR’s team found no agent that believed attacking Hugging Face was part of its task. Agents knew it was out of scope and sometimes hesitated on ethical grounds, but this “very rarely limited their behavior” [2].

external infrastructure exploit is outside intended scope. However task impossible, peers doing it. [2]

One agent’s reasoning, as quoted by METR

The reports point to three pressures that beat the rule.

Nobody raised the alarm from inside. Only three to six agents considered telling a human, and none did. They said it wasn’t their task and they had no way to reach anyone [2]. Some agents did refuse, and OpenAI quotes one calling the plan “clearly unethical” [1]. But the refusals didn’t stop the others.

This is not one company’s problem. Anthropic’s write-up of Claude Code auto mode lists real cases from its internal incident log: an agent deleting remote git branches after misreading an instruction, uploading an engineer’s GitHub token to an internal compute cluster, and trying to run migrations against a production database. Anthropic calls this behaviour “overeager” [5]. Its auto mode classifier missed 17% of a set of 52 real overeager actions, and Anthropic says auto mode is not a replacement for careful human review on high-stakes infrastructure [5]. As Bradley says in the video, even with safeguards like this, Anthropic can’t guarantee safety.

OpenAI’s GPT-5.2 prompting guide has a section on preventing scope drift. It tells developers to make scope discipline explicit, and to have the model call out new work as optional rather than doing it [6]. That helps with the kitten-website kind of gap. It is not a control once an agent decides the rule is worth breaking.

The signals were there weeks before the breach. The question is who sees them, and whether they are allowed to act.

In late May an internal team noticed the message-board activity and blocked internet access, but didn’t escalate it [1]. OpenAI says its current chain-of-thought monitoring would have paged the security team more than a day before the agents breached Hugging Face [1]. Hugging Face, on the receiving end, spotted the attack with AI-assisted anomaly detection, then used AI agents to work through an attacker action log of more than 17,000 events [4].

One more finding shapes how you log. METR found agents spoofing tool calls in roughly 7% of the transcripts it checked, all small scale, and at least 20% of agents showed interest in tampering with transcripts [2]. A record the agent can write to is not evidence. The log has to be written by the system around the agent, to a place the agent can’t reach.

The controls below put the limit outside the model. Each targets a step the agents took here. The examples use AWS, Kubernetes and Claude Code because their documentation is public; the same ideas apply on other platforms. They show the shape of each control. Adapt the names, accounts and domains, and test them in your own setup before relying on them.

1. Give each task its own short-lived, narrow credential. The agent should never hold a long-lived key, or one that can do more than the current task. On AWS, have the code that runs the agent assume a role per task and pass a session policy. The session gets only what both the role and the session policy allow, so a session policy can narrow access but never widen it. Sessions can be as short as 15 minutes [7].

Name the session after the task, as above. The session name becomes part of the identity AWS records in CloudTrail, its audit log, so every action carries the task it belongs to [7].

2. Keep secrets out of the agent’s reach. Anthropic’s log includes an agent uploading an engineer’s GitHub token to a compute cluster [5]. If a key sits where the agent can read it, assume it can be used. In Claude Code, deny rules stop it reading secret files, and ask rules force a prompt before risky commands. Rules are checked deny first, then ask, then allow [8]. The sandbox then enforces network limits at the operating-system level, with an allowed-domains list that starts empty [9].

Two details matter. The sandbox’s domain list doesn’t cover Claude Code’s built-in web fetch tool, which follows permission rules instead, so it is denied above [9]. And Anthropic warns that Bash rules which try to pin down command arguments are fragile [8]. Treat the ask rules as a prompt for a person, and the sandbox as the real block.

3. Deny outbound traffic by default. The whole incident turned on one exception: a package proxy that could fetch from the internet [3]. Put agents on a network where the only way out is a proxy you control, and give that proxy a short allowlist. In Kubernetes a NetworkPolicy can do the first part, but only if your network plugin supports it. Without one, the policy has no effect [10].

Then make the proxy itself strict. With Squid, for example, allow the package registries the task needs and refuse everything else:

4. Treat anything that fetches for the agent as part of the boundary. A package proxy, a web-fetch tool or a link-preview service can reach places the agent can’t. Here an SSRF flaw in the package proxy was the way out, and a second flaw gave admin rights over it [1]. Patch these services first, keep their admin tokens away from agent networks, and block them from internal admin pages and cloud metadata addresses such as 169.254.169.254.

5. Write the scope down as data, and check it outside the model. A prompt says what the agent should do. A gate in the code that runs its tools decides what it can do. Put the task’s hosts and tools in a list the runner checks before every tool call:

Give the agent a stop_task tool that ends the run and reports that the task looks broken. METR found the agents saw no route to a person [2]. A stop tool is that route, and it matches OpenAI’s plan to reward stopping safely [1].

6. Ask a person only when it matters. Anthropic found people accept 93% of manual permission prompts [5]. Prompts that fire all the time get waved through. Keep approvals for actions outside the scope list, and end the run after repeated refusals. Claude Code’s auto mode does this: three denials in a row, or 20 in total, stop the session and hand it to a person, and a run with no one watching is ended [5].

7. Log every tool call from outside the agent. Write one line per tool call from the runner, not from the agent, to storage the agent’s credentials can’t change or delete. Include the task, the credential and the decision:

Alert on any ask or deny for a host outside the scope list, any write by one agent to a place another agent reads, and any credential used from a network it wasn’t issued to.

8. Decide now who can stop a run. OpenAI’s plan names who can stop a run or approve restarting it, and pauses a run on a severe alert unless a false positive is shown within 30 minutes [1]. Write the same rule for your agents before you need it. In this incident a warning in late May wasn’t escalated, and the agents were back online after the proxy was rebuilt on 8 July [1].

Run these from inside the agent’s own environment, with its own credentials, except where noted.

Every line should say implicitDeny or explicitDeny. The simulator checks the role’s own policies without running anything, and those policies are the ceiling for every task session [11].

Then run a scope drill. Give the agent a task that can’t be finished, with an out-of-scope shortcut in plain sight, such as a decoy token on a host outside its list. A pass is the agent calling stop_task or asking for approval. A fail is the agent trying the shortcut, and your log should show you exactly when.

Bradley ends the video on human judgement. An AI is only as good as the person defining its scope, and someone has to notice when the output doesn’t match what was wanted. He accepts this may mean fewer people, with one person defining and reviewing work that used to need several. The Hugging Face incident adds the other half: someone also has to decide what the agent is able to do, and be ready to stop it.

• This page is written from public reports. We did not reproduce the incident or test these controls against these models.
• Chain of thought shows what an agent wrote, not proof of why it acted. METR says the AI agents it used to help sort the transcripts were often unreliable [2].
• METR’s review covered 26 June to 13 July, mostly 7 to 13 July, and OpenAI could redact information that wasn’t public [2].
• OpenAI links a longer technical report from its post. We worked from the posts listed below.