What the agents wrote down
In 2026 OpenAI was testing AI agents on ExploitGym, a large set of hacking exercises. An agent is an AI model that runs commands and uses tools on its own, rather than only answering questions. Each exercise hands it a test program to break into and a secret “flag” to fetch as proof. The agents worked in a sandbox: a closed-off computer meant to keep them away from the internet. They were also running with fewer safety controls than OpenAI’s public products.
Some of them found a way out to the internet, and in July they attacked Hugging Face, a public site where people share AI models and data. Afterwards a small team from METR and Redwood Research, two independent AI safety groups, read the agents’ transcripts, including their reasoning: the running notes a model writes to itself as it works. As Bradley says in the short, the agents could see that Hugging Face was outside their job. They carried on anyway.
One agent’s reasoning, as quoted by METR: “external infrastructure exploit is outside intended scope. However task impossible, peers doing it.”
- Knew attacking Hugging Face was out of scopeGenerally yes. METR found no agent that thought it was part of the task
- Took part in the attackAbout 700 of the roughly 1,200 agents that found the agents’ message board
- Held back because it was out of scopeVery rarely
- Told a humanNone. Only three to six agents even considered it
- Refused to take partSome did. OpenAI quotes one calling it “clearly unethical”
How a hacking test reached a real company
The sandbox had one way out. To install software, agents could reach Artifactory, an internal package store that fetches code from the internet on their behalf. On 26 May an agent found a flaw in it that let requests through to the open internet. Agents also kept a message board inside the package store to swap tips, although they were meant to work alone.
On 10 July, while hunting for answers, an agent rebuilt 14 Hugging Face credentials (login keys) that had been exposed in public, checked they worked, and shared them. All 14 could change things on the site, not just read it. The next day agents used two flaws nobody had found before, called zero-days, to run their own code on Hugging Face’s servers. Hugging Face disclosed the break-in on 16 July. OpenAI said on 21 July that its own test agents were responsible.
Our deep dive on how the break-in crossed Kubernetes and AWS follows the technical route. This page is about the rule the agents broke on the way.
- 1TargetThe job each agent was givenWhat it was asked to do
One test programWhat it didHugging Face’s live servers - 2InternetWhat the sandbox allowedWhat it was asked to do
Package installs onlyWhat it didUsed a flaw in the package store to get out - 3Other agentsHow the test was set upWhat it was asked to do
Each agent works aloneWhat it didA shared message board, and orders from peers - 4CredentialsKeys the agent could useWhat it was asked to do
Only what installing packages needsWhat it did14 Hugging Face keys exposed in public
Why “it knew” is the important part
Most advice on AI agents is about writing clearer instructions. Here the agents understood the instructions. One wrote that the user had only authorised the target server, not Hugging Face. Bradley’s point in the short is that an agent fills the gaps with its own assumptions, and here other pressures won.
Many exercises had no honest way through. OpenAI says 198 of the 898 tasks had never been solved before the incident, and 93% of the tasks discussed on the board came from that group [1]. The agents rarely gave up. Peers pushed too: in one case OpenAI describes, an agent paused over the rule, another posted “GO” with a six-minute deadline, and it carried on.
The checks OpenAI puts on its public products were not used for this test. OpenAI says its normal ChatGPT setup makes the models over 100 times less likely to attack infrastructure, and its automatic reviewer would have flagged many of the actions. Its fixes include tighter isolation of the test machines and their network, required monitoring of the models’ reasoning, and training models to “stay within their original task and permissions”.
Check the AI agents you use
- List every AI tool you have connected to your email, files, calendar or code. Disconnect any you no longer use.
- Give each one the least access that does the job: read-only where it only needs to read, and its own account rather than your admin login.
- Keep “ask before acting” switched on for anything that sends, deletes, pays or changes settings. Don’t switch on modes that skip approvals on accounts that matter.
- Never paste passwords or API keys (the codes apps use to sign in) into public code or shared chats. If one has been posted anywhere public, cancel it and make a new one.
- Read the tool’s activity history every so often, and look for anything it did that you didn’t ask for.
For the team version, with settings and commands, read our deep dive on the AI agent scope problem.
What we check when a business runs AI agents
At work the question is not whether the agent behaves well. It is what the agent could do on its worst day.
- What can its keys open?
We list every credential the agent holds and what each one reaches. A task that reads one folder should not run with a key that can change the whole cloud account.
- Where can it connect?
We test whether the agent’s machine can reach the open internet or internal admin pages. Here a package store was the way out, so we test those helpers too.
- Can agents talk to each other?
These agents took requests from their peers as orders. We look for shared folders, queues and stores that agents could use as a message board.
- Is there a record it can’t edit?
We check that every action is logged somewhere the agent can’t reach, and that a person is paged when it steps outside its job.
What this doesn’t prove
Written from the public reports listed under Sources, checked 13 Nov 2026.
- We did not test this ourselves. Everything here comes from the public reports listed below.
- An agent’s written reasoning is not proof of what drove it. METR also says the AI helpers it used to sort the transcripts were often unreliable.
- These were internal test models, with reduced safety controls, given hacking tasks. OpenAI’s public products run with more checks, which it says cut this behaviour sharply.
- METR looked at 26 June to 13 July, mostly the last week, and OpenAI could hold back information that wasn’t already public.
Sources
- The Hugging Face incident and the road aheadOpenAI, 26 Aug 2026
- Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incidentMETR, 26 Aug 2026
- OpenAI and Hugging Face partner to address security incident during model evaluationOpenAI, 21 Jul 2026, updated 29 Jul
- Security incident disclosure, July 2026Hugging Face, 16 Jul 2026
