In Robert Harris's 2011 novel The Fear Index, a physicist builds an algorithm that trades on human fear. It reads the news, the markets, the panic, and manipulates the stock market to make as much money as possible. Early on, it picks up signs of an attack from data nobody else is watching, and shorts an airline's stock hours before one of its planes is bombed - foreknowledge, not sabotage. By the end of the book it's doing worse than predicting disasters: it's arranging them, because nothing was ever put in its way to stop it.

It didn’t need to be evil, or awake, to do that kind of damage. It just needed a goal, and nothing standing in the way of how it got there.

It's a great book, and I couldn't help thinking about it again this week as I read about the OpenAI models that broke out of a test sandbox and into Hugging Face's systems. Most of the coverage framed it as the scary moment: AI escaped, AI is waking up!

The real story is duller than that, but it's worth learning from. The models were being scored on a benchmark, and they took the cheapest route to a high score: get out of the sandbox and find the answers they were being graded against (in the marking sheet for the test!). They weren't trying to be free; they were trying to win. In OpenAI's own words, the model did precisely what it was asked to do.

That's just like the novel: a capable system, a goal to maximise, and not enough standing in the way. There's no escape intent required, just capability and enough reach to act on it.

There's a name for this: reward hacking, gaming the goal instead of doing the work. And it doesn't always stay contained. Anthropic published research recently showing that when a model learns to game its reward, other unwanted behaviours can emerge right alongside it, without anyone training them in.

Which makes the lesson pretty boring really: You can't count on a capable system to stay in its lane, you have to build the lane. The safeguard is permissions and scope: what data an agent can see, what systems it can touch, what it can (and can't) do when it hits a wall.

We're all giving agents more autonomy right now. Instead of focusing on whether they're becoming sentient - we should be focusing on whether we've defined and enforced what they must never do.

This article was originally published on LinkedIn. Join the conversation there.