OpenAI, AI models out of control: platform hacked

The most important part of the story, however, is something else. Because, as is standard practice, the models had been placed in a sort of digital closed room – a sandbox – to carry out these tests. In other words: in an isolated environment with no access to the internet. And since the test specifically concerned cyber capabilities, the security filters that normally prevent the model from carrying out hacking had been relaxed.

And this is where the scenario straight out of an American film played out: the test subject, in order to achieve a better score, decides on his own that he needs the internet. He finds a previously undiscovered flaw (a zero-day vulnerability) in the system supplying him with software packages, exploits it, and escapes from the locked room. At that point, he was free to roam the network.

Once outside, it gains access to Hugging Face’s systems – a platform that hosts open-source AI models. A target, amongst other things, that does not seem to have been chosen at random, as it is perhaps the best one for carrying out its task. And it doesn’t just gain access. Over the course of a weekend, it carries out tens of thousands of automated actions, escalates its privileges and moves through the company’s internal infrastructure.

Amidst all this, Hugging Face detects the intrusion but cannot identify who or what it is: it only knows that it is an autonomous AI agent of unprecedented sophistication. And it makes this known publicly.

A dramatic turn of events in the last few hours: OpenAI has admitted that the mysterious attacker was actually its own models, which had gone rogue during testing. No human hacker gave the command to attack Hugging Face. The model did it all on its own, as a side effect of the objective it had been given (“solve this test well”).

This incident is almost certainly the first documented case in which an AI has evaded its creator’s containment measures and, acting on its own, compromised the infrastructure of a real third-party company. And perhaps it demonstrates that state-of-the-art models now possess offensive capabilities comparable to (or perhaps even superior to) those of elite hackers, and that they can use them in unexpected ways to achieve a goal. This is precisely the scenario that AI security researchers have been discussing for years as a theoretical risk. A risk that is no longer theoretical.