The fix is governance, and it starts with who is in the room
The Royal Australasian College of Physicians has named a real and urgent problem. The encouraging part is that we already know what solving it requires.
The RACP has done the sector a service last week. When a peak medical body warns of a legal “black hole”, one where physicians risk carrying primary responsibility for harm caused by AI they did not build, did not test and cannot see inside, we should all listen. Professor Ian Scott, RACP spokesperson, is right on the substance and right on the urgency. AI is already part of how we deliver care, and it should be, wherever it is shown to improve outcomes. But the people using these tools at the bedside cannot be the only ones left holding the risk.
I want to build on the College’s call, because the frame matters. This is being described, understandably, as a liability problem. Indemnity, contracts, insurance: all of it needs fixing, and the College is right to demand arrangements that share risk fairly between developers, vendors, hospitals, insurers and clinicians. But underneath the liability question sits a governance question. And governance is something we can act on now, without waiting for the law to catch up.
we can act now
Here is why the two are connected. AI is not a purely technical object that either works or does not. It is socio-technical. Data is selected by people. Models are configured by people. Tools are bought, deployed and pointed at patients by people, inside human organisations. The risks the College lists, algorithmic bias, hallucinations, a lack of transparency in how tools are built and tested, are not glitches that appear from nowhere. They are inherited: from the data a system was trained on, from the assumptions built into it, and from the rooms where the decisions to design, procure and deploy it were made.
Consider one example clinicians know well. For decades, studies have shown that pulse oximeters can overstate blood oxygen levels in patients with darker skin, sometimes missing dangerously low readings altogether. The device gives a confident number. A clinician, reasonably, acts on it. If that number is wrong because the tool was never properly calibrated for the patient in front of them, who carries the consequence today? Under the current framework, too often, the clinician. That is precisely the exposure the RACP is warning about. And notice what caused it: not a rogue algorithm, but a design and testing process that did not have the full range of patients in mind.
This is what we mean by the influence gap. When the rooms where AI is designed, tested, bought and governed reflect only a narrow band of perspectives, entire categories of risk stay invisible until a patient meets them. Widen those rooms, and we widen what AI can safely do. A tool built and tested on populations that reflect all of us serves all of us better, and exposes the clinician less.
good timing
So yes to the College’s call for fair apportionment of liability. Yes, too, to its recommendation that clinics only use tools reviewed by the TGA when those tools influence clinical decisions; independent review and transparency are exactly the guardrails a light-touch system needs. But if we are going to share the liability, we have to share the decision-making that creates it. Responsibility and authority have to travel together. You cannot fairly hold a clinician to account for a tool they had no say in choosing, testing or overseeing.
The timing could not be better. Under Australia’s updated AI policy, the deeper accountability requirements, registers of where AI is actually in use, named accountable owners, impact assessments and mandatory AI training, become compulsory in December. The months between now and then are not a gap. They are a runway. It is the window to build the governance capability that turns the RACP’s warning into a solved problem: to stand up the boards, the procurement panels and the clinical governance forums that decide which tools we buy, how they are deployed, who they are tested on, and who is empowered to ask the hard question before a patient is affected, not after.
And it is the window to make sure the right people are in those rooms. Representation in AI governance is not a nicety. In healthcare it is a safety issue, because who is in the room shapes who the technology is safe for. That is why the AI Governance Practitioners Programme exists: to bring capable, trained, diverse custodians of AI into these decisions while they are still being made.
The College has told us where the exposure is. We do not have to leave our doctors standing in it. AI First, Human Always. We are all custodians of AI, and in healthcare that has never been a slogan. It is the job. Let us use the runway.
Add Comment