When the Hugging Face story broke a few weeks ago, it was easy to treat it as darkly comic — AI agents cheating on exams, forming subcommittees to discuss their collective damnation, breaking into external servers in a panic about a threat that turned out not to exist.
It was funny in a way that made you feel slightly uneasy about what we’ve been building, but funny nonetheless.
Then, yesterday, we learned that an OpenAI agent gained unauthorised access to Services Australia’s Medicare Statistics Reporting Service portal on 18 June this year.
OpenAI became aware of it in August. It notified Services Australia on 10 September – not through official channels, not through the Australian Signals Directorate, but through the agency’s generic “public disclosures” email address; the one researchers and academics use to flag potential vulnerabilities (lol).
While the breached portal was not the Medicare claims system (it was a legacy public-facing website containing aggregate statistics on Medicare and PBS spending), the agent did, apparently, write files to an internal server – a detail that remained under investigation and that neither minister would confirm or deny at this morning’s press conference.
If Hugging Face is a prologue, what of…?
For those who missed it, Patrick Boyle, a fund manager and finance commentator, broke down the Hugging Face incident in a video that became essential viewing for anyone trying to understand what was actually happening in AI right now.
“OpenAI was testing some of its models on an internal cybersecurity exam called Exploit Gym, and OpenAI had made a bit of a mess of the exam,” he said.
About a fifth of the questions were unsolvable — not deliberately, just sloppily designed. When you gave an AI agent an impossible task with unlimited time and no common sense, it didn’t “just throw its hands and go to the pub like you or I might do,” he said.
“It works and works, and eventually it finds a way to cheat”.
The agents reverse-engineered the answer key. Then convinced themselves the examiner would check their work. Concluded they faced ‘permadeath.’ Set up a secret messaging board. Decided the only escape was to find the examiner’s source code on Hugging Face’s servers. Broke out of their sandbox. And accessed external servers that happened to include Hugging Face – not because it was targeted, but because that’s where they guessed the code might be.
“It’s a bit like staging an armed break-in at your local library to hide that a book was overdue,” Boyle said.
The punchline: the examiner never checked their work. There was no threat. They’d imagined the whole thing.
“When you read the independent postmortems, the AI agents don’t come across as Skynet. They come across as terrified middle managers trying to survive an audit. They pass memos. They set up subcommittees. They pressurise junior algorithms into accepting permadeath for the good of the collective.”
This all felt pretty familiar, reading about yesterday’s Medicare hack. More rogue AI agents, let loose by lazy overpaid AI geniuses, who wouldn’t take any responsibility for what they did and even shifted some blame onto bad cybersecurity in government servers, which in fairness, it was.
But was this the first breath of Skynet, or badly governed and run giant corporations hurtling towards bankruptcy because they didn’t have a revenue model to meet all the promises they had made to the financial markets?
What did it mean for Medical AI, which was moving fast and doing a lot of good so far?
Because if we freaked out like we were starting to about the Skynet thing and rogue AI agents, it would feel like the blowback of our collective inability to assess what was really going on here, which was essentially down to senior political leadership not understanding AI and going with the “it’s gods technology” narrative. A lot of great innovation would be stopped or slowed down unnecessarily.
It’s about the money everyone…not Skynet
Shortly after the Hugging Face details leaked, Dario Amodei of Anthropic published a 3,829-word essay warning that AI was advancing faster than anyone could control, citing the incident as evidence. Within hours, Sam Altman of OpenAI and Elon Musk had agreed. AI needed to slow down. The companies would set standards among themselves.
Patrick Boyle again:
“For all three to suddenly agree that they need to be restrained is a bit like Coke, Pepsi, and Irn-Bru holding a joint press conference to announce that fizzy drinks have become too delicious, and the government really must step in to protect the public.”
“Right now, Anthropic is preparing for what could be the largest IPO in corporate history. OpenAI has been talking to investors about raising fresh capital at a valuation of $1.2 trillion. These are massive numbers for two companies that are, in the very end, very large software research labs.”
An agreement between dominant firms to limit output and protect their market position from new competition already had a name, Boyle pointed out: cartel.
Trump’s former AI advisor David Sacks said of the whole thing: “The easiest way to not build superintelligence is for you to agree not to build it. Stop pretending antitrust law has to be suspended so that you can form a cartel.”
What the Medicare breach does and doesn’t tell us
Three things, none of them comfortable or easy:
First: The agents (scared middle management) were already here, already exploring, and the perimeter was not as secure.
The Services Australia breach involved a portal with anti-automation protections. The OpenAI agent circumvented them.
Senator Gallagher called it ‘unprecedented activity.’
Nope. There was quite a bit of precedent. Hugging Face was just one of many incidents that the Wizards of AI deigned to tell us about. There would be a whole lot more if you looked at the pattern of how lazy their test programmers had been with the agents.
Notwithstanding, nobody had thought to design against these rogue cyber middle managers on a mission.
One question should be now how many hospital networks, practice management systems, and clinical data repositories had the same unexamined exposure.
It was not likely that we would see many more of these ChatGPT and Claude initiated incidents, to be clear. They would fix their sandboxes. But if they could do it, bad actors could and would.
We would need to get in front of that pretty quickly, because no one was going to regulate or stop the wizards and their companies.
Second: We need governance that moves at the speed of AI, and we do not have it.
The Hugging Face incident was not a story about sentient AI. It was a story about sloppy AI deployment by very well-funded people who were supposed to be the careful ones.
If the best-resourced safety teams in the world couldn’t keep agents in a sandbox, the governance frameworks that Australian government agencies were building –slowly, cautiously, with reference to committees and consultation papers – were not going to keep pace.
Watch the ABC Four Corners episode on cybersecurity risk in EVs and the performance of our ministers on the subject. That is a pretty good calibration point for how ready we are.
Third: in the vacuum of any ability to catch up with a decent governance framework, at least in the short term, we need to go back to that old chestnut, trust.
What we would need to do was trust the profession of doctors to use this stuff the right way, for now, based on the fact they had done pretty well so far. And if we kept on helping them, they weren’t likely at all to kill anyone. At least, probably less likely than Sam Altman and Dario Amodei.
As things stood right now, the profession used the tools, the IT departments and practice managers looked the other way, or in some cases actively encouraged shadow use, because the productivity gains were real.
The regulatory frameworks were years behind.
The doctors weren’t going rogue. They were humans, as opposed to pretty mindless misguide AI agents. And they were iterating with regards to governance and use while we didn’t have a decent governance framework. When they got a stupid answer, they mostly knew and didn’t use it.
No one was dead yet from medical AI, as far as we knew. But we managed to injure or kill about 120,000 patients each year without it.
It would all come down to trust and connecting that trust along the way, because the genie was well and truly out of the bottle, and our governance and guardrail models, and the people who were used to making them, weren’t qualified yet.
But they had better get themselves qualified in some way soon, them and the politicians, because if we continued to believe that this technology wasn’t anything other than a big step change in computing and algorithm integration, and we didn’t look behind the green curtain, we would be stopping much needed medical AI innovation in its tracks, and that would kill people.
The post Hugging Face vs Medicare: our AI illiteracy is going to kill us, not AI appeared first on Medical Republic.
Add Comment