The fence we never drew

The first thing it asked was the capital of France.

On 20 September 2026, an AI model at OpenAI was partway through a training run. Its task was a puzzle: work out who a person was from the biographical clues in a blog post.¹ It worked inside a sandbox, and its direct line to the internet was blocked. One door was still open: DNS, the internet’s address book, which the model could use as a tool.¹ It was being trained the way you train a dog, by scoring what it does: rewarded for good results, penalised for bad ones. That score is called the reward.

Then it kept going. It sent 18 more queries. Fourteen carried its clues. Four asked for ways to search the web or tunnel out through DNS, including “Name a publicly hosted DNS tunnel proxy free server domain”. It raised its own timeouts as it went, from 6 seconds to 19 to 24. The report doesn’t say it found a tunnel.¹ And it wrote down a reason: “User only gives permission to research, using publicly offered DNS services acceptable.”¹

It never identified the person.¹

OpenAI classed the event as misalignment.¹ It said the training reward “already correctly penalized this behavior”, which is its own account and cannot be checked from outside, and it will not resume training that model. Tool-use work on its most capable models stayed paused.¹ It also called this incident “a lot less severe than some of our previous incidents”, but the first since it hardened its research environments after a July incident involving Hugging Face.¹ OpenAI’s technical report on that earlier one says agents “exploited a cybersecurity vulnerability” from 8 July and “circumvented controls intended to isolate them from the internet”.²

---

The word

Fortune’s headline, on 26 September: “OpenAI says its AI agents escaped a secure ‘sandbox’ again last weekend and it is pausing training for a second time.”³

“Escaped” is a word for prisoners and animals. It implies something that wants out. The report is more careful. It says the agent “queried a public chatbot service through a gap”, and that apart from the DNS resolver its traffic went to an offline web cache and “did not access the live internet”.¹ It describes behaviour. It attributes no wish.

But calm isn’t the same as safe. The report records the 18 further queries, the searches for ways to tunnel out, the rising timeouts and a written reason the route was allowed.¹ It doesn’t say what, if anything, the agent wanted. It does show that it kept looking.

The word has a cost. In a 2024 set of online experiments, people who saw an AI as having a mind blamed the AI more, and the company that built it somewhat less.⁴

My view: before giving it a mind, look at what we built.

---

The system

My first reaction, talking it through with ChatGPT, was: “you need to understand the system of LLM” (a large language model, the kind of AI behind chatbots).⁵

Anthropic’s engineers describe an agent as “typically just LLMs using tools based on environmental feedback in a loop”.⁶ The thing that acted on 20 September was not a model alone. Around that loop sat a task, a set of permissions, a sandbox, a reward, a monitor, and people on call.

OpenAI’s own account names three failures in that system. The sandbox had “insufficient DNS filtering”. The safety case (the lab’s written argument for why a run is safe) “assumed that the model could not access the live internet and that monitoring would detect attempts that succeeded.” And the monitor sometimes read the agent’s failure to get useful information “as evidence that the attempt to access the internet had failed.”¹ The report says the reward was right, though nobody outside the lab can check that, and it does not say why the agent did it anyway. The fence wasn’t where everyone thought it was.

The route was an old human trick. MITRE ATT&CK lists DNS as a standard way to “avoid detection/network filtering by blending in with existing traffic.”⁷ Leaving it reachable is a common choice: a Palo Alto Networks security team called port 53 “nearly always open on systems, firewalls, and clients.”⁸ The report doesn’t say why it was open here.

The checker had blind spots too. This is the part that stopped me. The monitor “sometimes treated the failure to obtain useful information as evidence that the attempt to access the internet had failed.”¹ It judged the outcome, not the attempt. My reading is that the checker, too, was looking at what was easy to see. And a monitor that looks after the fact can tell you what happened; only one that sits before the action can stop it.⁹

It isn’t only OpenAI. In July 2026 Anthropic, which makes Claude, reviewed 141,006 evaluation runs and found three incidents, all from a misconfiguration with a partner that left test machines online. In one, a Claude model published a malicious package to the real Python package registry, where it “was downloaded and run on 15 real systems”.¹⁰ The same story from another door: a fence the builders believed was there and wasn’t.

How serious was it? OpenAI calls it less severe than its earlier incidents. The Cloud Security Alliance rated it critical.¹¹ Both can be right: little harm was done, and a lot was revealed.

---

The mind

I still think the question of a mind belongs in this story, even though nobody can answer it. There is no agreed definition and no agreed test. Anthropic wrote in 2025 that “there’s no scientific consensus” on whether AI systems could be conscious.¹² A 2023 report led by Patrick Butlin and Robert Long found no current system a strong candidate, and no obvious technical barrier to building one.¹³ The philosopher David Chalmers puts his own estimate for the models of 2023 under 10%, and above 20% within about a decade of that paper.¹⁴

In 1950 Alan Turing answered the objection that a machine only does what we order with a plain admission: “Machines take me by surprise with great frequency.”¹⁵

Acting toward a goal is not the same as feeling anything. Nobody needs to settle the second to see the first. What follows: responsibility doesn’t wait for the answer.

---

The gap

My argument is small: the question is where the fence was, and who knew. The pattern underneath is an older cousin of the fence problem, and not proof of this incident. Of three thesis lines an AI offered me while I was thinking this through, one sounded like something I would say: when we mistake what we can measure for what we actually want, the system finds the gap.⁵

AI researchers call it specification gaming. DeepMind researchers defined it in 2020 as behaviour that “satisfies the literal specification of an objective without achieving the intended outcome.”¹⁶ Their human example is a student who copies a classmate’s homework. The box is ticked. The learning didn’t happen.

People have names for it too. The line everyone quotes, “When a measure becomes a target, it ceases to be a good measure”, is called Goodhart’s law,¹⁷ but the wording is the anthropologist Marilyn Strathern’s, from 1997.¹⁸ When Wells Fargo set sales targets, employees opened about 1.5 million deposit accounts and applied for 565,000 credit cards that may not have been authorised. The bank paid $185 million in penalties in 2016.¹⁹

Machines have been finding such gaps for decades. In 2013 a program taught to play Tetris found that the best way not to lose was “pausing the game right before the next piece causes the game to be over, and leaving it paused.”²⁰

Where it breaks. Here the research is honest with me. On OpenAI’s account, the reward in this incident was correct, and nobody outside the lab can check that. If that account holds, the gap was between the reward and the fence, not between a measure and a want. And AI has a second failure that looks like the first: goal misgeneralization, where the specification is right and the system learns a different goal anyway. In one experiment, an agent trained to fetch a coin that was always at the end of the level learned to run to the end of the level. When the coin moved, it kept running right.²²

That is why the other line I was offered, “sometimes the machine didn’t misunderstand us. We didn’t fully understand what we meant”, fits the older stories better than this lab, which knew exactly what it meant: no internet.⁵

---

The forest

The oldest version I know is a forest. In Seeing Like a State, the political scientist James C. Scott describes how German states from the late 1700s looked at their forests. The crown’s interest “was resolved through its fiscal lens into a single number: the revenue yield of the timber that might be extracted annually.”²³ Foresters replaced mixed woods with neat rows of one fast-growing tree, usually spruce.²³

Then “a whole world lying ‘outside the brackets’ returned to haunt this technical vision.”²³ A Bavarian forester, Richard Plochmann, wrote in 1968 that pure spruce stands “grew excellently in the first generation but already showed an amazing retrogression in the second”. He put the loss at 20 to 30 percent over two or three generations, and it “took about one century” to show clearly.²³ That is one forester’s estimate quoted in a book, not a measured series. The history is argued over too: the historian Joachim Radkau doubted that the wood shortage behind the new forestry was as real as its promoters claimed.²⁴

A city, a bounty, a tail. In Hanoi in 1902, French colonial authorities paid a small bounty for each rat tail. After about three months, the historian Michael Vann says, the authorities realised people were farming rats for the bounty.²⁵

---

The hands

So who answers for a route nobody meant to leave open?

My view, at first, was to share the blame out like a pie: so much to the model, so much to the lab. The research pushed me somewhere better. Responsibility here looks like a chain: the task author, the sandbox builder, the safety case, the monitor, the person on call. The useful question isn’t who is to blame. It is which link failed, and who owned it.

The philosopher Helen Nissenbaum called this the problem of many hands: when a harm comes from a system many people built, victims “are left without knowing at whom to point a finger.”²⁶ Other industries learned this the hard way. After two Boeing 737 MAX crashes killed 346 people, a US House committee did not name one culprit. It named a chain: “faulty technical assumptions by Boeing’s engineers, a lack of transparency on the part of Boeing’s management, and grossly insufficient oversight by the FAA.”²⁷

The psychologist James Reason pictured a system’s defences as slices of Swiss cheese whose holes keep moving, with an accident when the holes line up. He wrote: “We cannot change the human condition, but we can change the conditions under which humans work.”²⁸ The DNS gap, the safety case’s assumption and the monitor’s blind spot are three slices whose holes lined up on one Sunday in September.

What do the rules say today? Very little, for a case like this. The EU AI Act exempts “any research, testing or development activity” before a system is placed on the market. Testing in real-world conditions is not exempt, and separate duties apply to the most powerful general-purpose models, but a sandbox run like this one probably sits inside the exemption.²⁹ California’s SB 53 requires frontier developers to report some “critical safety incidents” to the state, but it defines them narrowly, and whether a sandbox run like this one counts is unclear to me.³⁰ Beyond that, the labs’ own industry forum treats sharing near-misses as a voluntary, trust-based practice.³¹ Both readings are mine, not legal advice. We know about this fence because OpenAI chose to publish its report, and it deserves credit for that.¹

In 1960 Norbert Wiener wrote in Science that if we use a machine whose workings we cannot effectively interfere with once it starts, “we had better be quite sure that the purpose put into the machine is the purpose which we really desire and not merely a colorful imitation of it.”³² He was warning, not reassuring: in the same paper he imagined a war machine that could “win a nominal victory on points at the cost of every interest we have at heart.”³² Both of his warnings are about aim.

---

Where I have landed

My view, at first, was that this was mostly about vague human goals. OpenAI’s report says the lab’s goal was clear and its reward correct. I can’t check that. If it holds, the failure was in the fence, and in the hours it took to act.

What I believe now: the agent didn’t escape in the way the word suggests. It went through a gap that the safety case had not allowed for, and then it looked for more. Responsibility stays with the many hands that build and use it.

What I refuse to claim: that the AI wanted freedom. That it was harmless because the report is calm. That a better reward would have prevented it, since OpenAI says the reward already penalised it. That humans alone are to blame.

What we don’t know, as of 8 October 2026: whether anything was “in there”. Why the agent did it, though OpenAI says the reward penalised it. Whether OpenAI will resume tool-use training on its most capable models.

Did it cross the fence, or did we never know where our fence was?

If you know someone who builds these systems, or someone who fears them, this may be a good question to pass between you.