Dr Monica Trujillo.
The recent OpenAI Medicare breach highlights why strong governance is critical as AI becomes embedded in healthcare, writes Dr Monica Trujillo.
AI is reshaping the way healthcare organisations operate and deliver care. As these capabilities move closer to core clinical and business processes, governance becomes a strategic necessity to ensure innovation is matched by accountability, trust, and organisational resilience.
The recent breach of Australia’s Medicare statistics reporting portal has reinforced a broader reality: trust is now a strategic consideration in every technology decision. While not all AI applications carry the same level of clinical risk, they share a common requirement for clear transparency and oversight.
For health leaders, the question is no longer whether AI will influence healthcare delivery, but how organisations will ensure its use remains safe, trusted, and aligned with clinical intent.
Imagine this scenario: at a rural hospital, an AI algorithm flags sepsis five hours before clinical presentation. The algorithm fires, the nurse acts, the patient improves. Success. But then the algorithm fires again, incorrectly, and the same nurse follows its recommendation. The patient deteriorates. In that moment, who is accountable? The vendor who built it? The nurse who acted on it? The organisation that deployed it? If you cannot answer these questions clearly, your governance needs a refresh.
AI is an Executive Leadership Challenge
There is a misconception that AI is primarily a technology challenge. In my experience, it is fundamentally an executive governance and leadership challenge.
Through my engagement with healthcare leaders across Australia, I have seen a common theme emerge: accountability becomes harder to define as AI moves into clinical practice. The questions leaders need to answer are no longer just about capability or implementation. They are about governance, leadership and the accountability needed to use AI safely, ethically and responsibly.
At a recent Royal Australasian College of Medical Administrators event on AI and Digital Leadership, I spoke about three critical distances where accountability can disappear.
Distance 1 – Data to Decision: Who owns the output? Between an algorithm’s recommendation and the care that a person receives sits a series of human judgements and professional responsibility.
If that pathway isn’t deliberately designed and governed, responsibility becomes unclear. This is also where data quality becomes a clinical governance issue, not an IT issue. When data is incomplete or unrepresentative, algorithms reproduce existing biases at scale. Research showed a widely used algorithm systematically underestimated health needs for African American patients because training data reflected existing inequity in the healthcare system. The algorithm faithfully reproduced that bias. That is why data integrity is clinical safety.
Distance 2 – Vendor to Governance: Who owns the system? Many organisations assume the AI tool will operate as the vendor designed it. But clinical use evolves. Scope expands. Workflows change. The vendor designed for one use case, but your organisation is now using it for three. If that drift isn’t monitored, an AI scribe recommending tests has quietly become a medical device requiring regulatory approval.
Distance 3 – Tool to Judgement: Who owns the gap? The more a clinician trusts a tool, the less likely they are to interrogate its output critically. That is not a clinician failure; it is a governance failure. Building the conditions for good judgement must become a leadership priority. Organisations should capture clinician overrides of AI recommendations as valuable data, invest in clinicians’ understanding of failure modes, and foster a culture where questioning AI is expected, not discouraged.
The principles of clinical governance have not changed. We still need good data, capable people, clear accountability, defined escalation pathways and ongoing oversight.
What has changed is the speed, scale and complexity with which decisions can now be influenced. AI amplifies the consequences when governance fundamentals are not strong enough.
Building Trust Through Governance
Trust in AI is not created by algorithms alone. It is earned through strong governance, transparency and accountability.
Australia’s 3rd National Policy Roadmap for AI – ‘Don’t Walk, Run’ reinforces the need for stronger, adaptable governance as AI adoption accelerates. For health organisations, that means oversight needs to extend across the lifecycle of AI, from adoption to ongoing use and monitoring.
Healthcare has spent decades developing pharmacovigilance to monitor medicines after they enter clinical practice. We will need the same discipline for AI. More than 10,000 AI-related incidents have been recorded globally since 2014. Governance cannot stop at implementation. It must be continuous.
We are also seeing regulators begin to grapple with what this needs to look like in practice. In August, the U.S. Food and Drug Administration released a discussion paper seeking feedback on the regulation of generative AI-enabled medical devices, including approaches to risk assessment, pre-market evaluation and ongoing monitoring.
Importantly, it explores how benchmarking could be combined with clinical confirmation before deployment, alongside post-market monitoring proportionate to risk. It also considers the challenges presented by foundation models and agentic AI systems.
While this is a consultation rather than a final regulatory requirement, the direction is important. For Australian health organisations, it provides a useful emerging assurance benchmark: demonstrate clinical performance before deployment, govern how models change over time, and continue to monitor safety once they are in use.
The regulatory settings will continue to evolve, but health leaders should not wait for regulation to define good governance. Building those disciplines now is fundamental to maintaining trust as AI becomes more deeply embedded in care.
The Opportunity is Significant. So is the Responsibility
AI has the potential to improve outcomes, reduce inequity, support overstretched clinicians and help health and care systems meet growing demand. But technology alone will never deliver these benefits. Success depends on the governance, culture and accountability that surround it and build trust.
While we welcome the progress AI is making in health and care settings, we should also recognise the leadership work happening behind the scenes – by clinicians, governance professionals, digital health leaders and health organisations – to ensure these technologies are safe, trusted, and implemented responsibly.
Ultimately, the future of AI in healthcare will be shaped by the quality of the leadership, governance and clinical judgement that surround it, not by the technology itself.
The technology is ready. Our responsibility is to ensure our governance is, too.
If you lead a health organisation, ask yourself today: Do we have a clinical governance committee specifically for AI? Do we know which tools are deployed and for what purpose? Have we assessed TGA classification? If the answer to any is “no,” that is not a technology gap – it is a leadership gap. That is where your work begins.
Viewpoint articles are the author’s opinion, produced without payment or sponsorship.
Add Comment