Introduction by Croakey: The Therapeutic Goods Administration expects to soon publish its review of the use of artificial intelligence-powered clinical scribes, along with updated guidance for developers, according to Health Services Daily (paywall).
In a report on the Australian Institute of Digital Health’s HIC2026 conference in Sydney on Monday, the publication said the TGA plans to begin compliance action against suppliers of some AI-powered clinical scribes after the review found evidence of regulatory non-compliance.
The TGA’s product quality division head Tracey Duffy said the regulator was “now moving into compliance for those organisations who have deployed a scribe that’s operated as a medical device and have not sought medical device regulatory approval”.
She said the regulator’s broader review of AI regulation had concluded that Australia’s existing legislative framework remained fit for purpose, with refinements rather than new laws required.
The TGA has previously flagged regulatory concerns such as ‘scope creep’ – where an AI scribe, for example, does not initially meet the criteria of being a medical device but this changes when a new feature is added.
The Australian Commission on Safety and Quality in Health Care has issued advice to clinicians on the use of AI clinical scribes, stressing the importance of informed consent, as has Ahpra and the National Boards. The Royal Australian College of General Practitioners has also published guidance.
Meanwhile, Digital Rights Watch recently launched a discussion paper on AI scribes in healthcare in Australia, calling for them to be regulated as clinical devices and tested for bias and accuracy, and saying that patients should have a ‘right to refuse’ their use.
“Primary healthcare providers in Australia are rapidly adopting AI scribes – software tools that record and transcribe clinical consultations,” said Digital Rights Watch.
“While these tools may offer some administrative benefits for medical practices, this report finds that their rate of adoption is outpacing both regulation and due diligence, creating serious and largely unaddressed risks for patients.”
Dr Andrew Cullen, Senior Research Fellow at the University of Melbourne, says the use of AI scribes raises many concerns, including around patients’ data and privacy, and he calls for tougher regulatory action. His article first appeared in The Conversation.
Andrew Cullen writes:
Visits to a new doctor, physio or specialist often begin with: “Do you mind if I use AI?”
According to Off the Record, a new report by Digital Rights Watch, more than 40 percent of Australian doctors are now using AI scribes. These tools listen to consultations, feeding what they hear into Large Language Models – similar to ChatGPT or Claude – to generate medical notes.
For clinicians, AI scribes are an easy sell. They eliminate hours of paperwork, freeing energy to focus on the patient. This can profoundly change how chronically burned-out doctors feel about their work. And with Medicare only compensating doctors for face-to-face time, and not note-taking, the financial and emotional incentives for practitioners are undeniable.
Yet the benefit for patients is less clear, especially when we have little transparency about who has access to deeply personal medical data, or how this data is handled by these lightly regulated AI scribes.
What can go wrong
Many AI scribes don’t just transcribe text. They summarise and interpret conversations based on what the AI assumes a medical professional might say.
However, AIs often hallucinate, confidently stating things that never happened. A doctor counselling a patient to quit smoking might find the AI confidently recorded that the patient was advised to avoid house fires.
AI systems also frequently show biases relating to race, ethnicity, class and gender. For patients from culturally and linguistically diverse backgrounds, or those for whom English is a second language, AI interpretations of speech can lead to medical records that wildly diverge from reality.
Early studies suggested up to 90 percent of AI-generated notes required correction. More recent studies show that approximately 20 percent of AI clinical notes contain errors significant enough to impact patient diagnoses.
And catching these errors is hard for clinicians, as humans really struggle to identify others’ mistakes under time pressure. This is especially true when automation bias leads humans to accept an AI’s output without scrutiny. But failing to correct these errors harms both patients and any subsequent clinician relying on the notes.
We’ve seen this in the legal space, where judges and lawyers are now forced to meticulously fact-check every submission to ensure no AI hallucinations slip through, slowing down proceedings.
If clinicians must meticulously verify every note for potential AI-related mistakes, the promised productivity gains likely evaporate. If they don’t, the health system will be flooded with unreliable data.
The issues with trust run much deeper. Medical professionals aren’t well-equipped to provide guidance on where the data they give to AI scribes goes.
At the moment, patients are asked just to trust that their clinician understands what is happening. But healthcare professionals aren’t AI experts and aren’t well equipped to discuss these privacy issues. Their professional bodies provide only high-level information.
Complicating this is the fact AI scribes typically fail to disclose how data is handled, who has access to it, and what AI models are used. This means that beyond vague assurances that data remains in Australia, healthcare professionals simply cannot give patients concrete answers about who is accessing their data or what they are doing with it.
Regulation needed
Regulators are struggling to keep pace: they’re working with laws that don’t clearly define who is responsible for this rapidly evolving space.
As Digital Rights Watch argues, Australia’s Therapeutic Goods Administration (TGA) should regulate AI scribes as it would any other medical device. It should be subject to the same safety testing requirements as any other piece of medical equipment.
The TGA currently claims scribes are only medical devices if they provide diagnostic advice. This allows vendors to evade scrutiny.
The TGA’s position ignores how generative AI is used in clinical practice, and that even transcriptions can contain hallucinations or biases that could affect diagnoses. This leaves Australia’s guidance out of step with nations such as the United Kingdom.
Fundamentally, the Government must ensure Australians’ private data is properly safeguarded, through both legislation and bodies such as the AI Safety Institute. This should include forcing AI service providers to clearly disclose what data they have access to, how they use it, and who can access it, with rigorous protections to ensure privacy and real penalties for breaches.
More broadly, there is a need for rigorous independent testing to ensure the AI systems with which we interact are fair, reasonable and unbiased for all Australians. This should include testing whether these systems produce genuine productivity gains that are beneficial for patients.
Without these safeguards, Australian patients may lose trust in the medical system. And once that trust is broken, it will hurt us all.
Author details:
Dr Andrew Cullen is a Senior Research Fellow, School of Computing and Information Systems, The University of Melbourne. He is a board member of Music Technology Australia, a member of the World Intellectual Property Organisation’s AI Technical Exchange, and manages the consultancy EthiCan.
See Croakey’s archive of articles on AI and health 
Add Comment