Introduction by Croakey: Following debates about the need for patient consent for AI scribes, MJA Insight+ has recently published a new guide for GPs to AI adoption in healthcare.
With a section on consent, this guide recognises the importance of educating medical professionals in consent policy and practice at a time when nearly one-third of Australian practitioners surveyed recently indicate they are using AI daily, particularly AI scribe tools.
As CEO of Allied Health Professions Australia, Bronwyn Morris-Donovan, told Health Services Daily recently, it is critical that practitioners know they are responsible for reviewing the outcomes of AI-supported work, as they are still responsible for making sure records are “accurate, complete, and appropriate”.
“These technologies have moved quickly, and Australia’s regulatory and professional standards are continuing to evolve alongside them,” she told HSD.
“While AI tools may be used differently across health professions, the fundamental expectations around clinical quality, safety, and professional accountability remain the same whether the user is a doctor, nurse, or allied health professional.”
Even so, the complexity of AI consent ethics raises questions such as whether it is even possible for patients to withdraw consent once their data has been used by AI tools, particularly where it has influenced the training of a machine model.
In our final Croakey Conference News Service report from the recent #Intelligence26 conference, Marie McInerney explores the critical problems of interpretation and practice facing patients and healthcare providers as they negotiate AI ethics and patient consent.
Marie McInerney writes:
Current patient consent requirements and procedures are “not fit for purpose” for the artificial intelligence juggernaut now arriving in healthcare, a national conference on AI issues in radiation and radiation oncology has heard.
Dr Yves Saint James Aquino told the recent Royal Australian and New Zealand College of Radiologists #Intelligence26 conference that current guidance on consent processes for AI-enabled care is limited, as are documented patient perspectives on AI use.
Aquino, a research fellow at the University of Wollongong’s (UoW) Australian Centre for Health Engagement, Evidence and Values (ACHEEV), acknowledged that clinicians are seeking better guidance on how to manage consent, given the very particular and untested challenges that AI raises.
He highlighted five particular issues that make their decision-making complicated.
The first is that AI is really an umbrella term for a set of technologies so diverse that it can be impossible to define what is and what is not included. This makes it difficult for health and medical workers to distinguish when separate consent might be required.
A second challenge is AI’s operational complexity. Increasingly much of it can be explained, he said, but the way some algorithms work may not be understandable even for clinicians. This means “it’s not easy for them to explain that to patients” for informed consent purposes.
“We know the input and the output but we don’t know how the AI system arrived [at its conclusion],” he said.
Admin or medicine?
A third challenge is the blurred distinction between AI’s use as an administrative tool versus a medical device that would need approval from the Therapeutic Goods Administration.
Currently a lot of AI systems are being marketed as administrative, and therefore not needing approval, he said. “But in practice, they are a medical device.”
Aquino acknowledges that the definition of a medical device is “obviously complex”, but said it should apply when a tool provides diagnosis or treatment.
So there’s a question of how to deal, for example, with a scribing device that summarises a medical consultation, an administrative task, but in practice may also give diagnosis and treatment.
For example, as the MJA Insight+ article recently warned, ambient scribes that simply capture and translate talk into text do not generally require an ARTG listing (Australian Register of Therapeutic Goods).
“However, if a scribe examines or interprets clinical discussions (for example, by offering differential diagnoses or treatment recommendations you did not suggest), it falls into the AI medical device category and must be registered with the TGA,” it said.
More challenges
A fourth challenge, Aquino told the conference, is that AI systems have multiple components located in different places or jurisdictions, unlike other medical hardware, such as an x-ray machine.
An AI system may involve hardware, software, and a cloud storage system that might exist in another country “so the privacy obligations … might differ [depending] where the cloud storage is located and where the hardware is located,” he said.
Equally the risk assessment of such components will vary depending on the jurisdiction.
Finally, he said, some AI systems change over time as they interact with more information. That means they can get better, but they can also get worse, and either way that might mean a deviation from the evidence base on which they were initially tested or approved.
Patient preferences
Meanwhile, he said, there are still very few empirical studies on patient views about AI use in their care. What is currently available shows people generally prefer to be notified about AI use, but that varies across gender, age, income and the level of risk, seriousness or precedent in their care.
Aquino was part of the UoW-led team that, in 2023, conducted the first national Australian citizens’ jury on using AI in healthcare. It asked, in essence, under what circumstances, if any, should AI be used in the Australian health system to detect or diagnose disease?
Thirty Australian adults were recruited for the jury to reflect population proportions by gender, age, ancestry, highest level of education, and residential location (state/territory; urban, regional, rural).
They discussed the question in depth over 18 days between March and April 2023, with 15 of those sessions online and three days face‐to‐face in Sydney.
They came up with 15 recommendations across ten categories, including the call for a charter for AI in the Australian health system and services, which they said must include Aboriginal and Torres Strait Islander peoples and other under-represented people, rural and remote populations, sustainability and the environment, Australian security and sovereignty, and ethics and human rights.
Croakey has previously reported on the jury’s findings from a webinar conducted in 2024 by the Consumers Health Forum of Australia (CHF), which also asked whether AI tools for clinicians and patients are fit for purpose in healthcare.
Acknowledging that healthcare consumers, healthcare services, and healthcare financing institutions could all potentially benefit from wider adoption of generative AI, CHF warned that the risks are primarily borne by consumers.
“We know that artificial intelligence or AI can improve things, but we shouldn’t assume that that’ll be the case,” CHF CEO Dr Elizabeth Deveny said.
In terms of clinical governance and training, the jury recommended that professional bodies must have clear directions regarding the use and intended outcomes of AI in the domains for which they are responsible and that healthcare workers must be trained in AI systems that are to be implemented to their practice environment before clinical use.
On patients’ rights and choice, there were three recommendations — see slide below.
The recommendations suggest that Australians are very concerned about being excluded from the benefits of AI-supported care, but also that they are given the opportunity to learn about its potential and make informed choices.
“They were afraid that AI is being developed outside or independent of the interests of consumers and patients,” Aquino said. There was also fear that AI will increase inequities in care.
RANZCR and three health research translation centres, Monash Partners Academic Health Science Centre, Maridulu Budyari Gumal, the Sydney Partnership for Health, Education, Research and Enterprise (SPHERE), and Western Australian Health Translation Network supported the jury process.
You can read RANZCR’s responses to the jury recommendations here.
In response to the jury’s work, ACHEEV also worked with a 12-member consumer panel to develop an introductory guide for consumer or patient representatives for research and healthcare organisations (see example questions below).
UoW is now leading a national $2.25 million initiative to address the rapidly evolving ethical challenges posed by AI-related health research.
Emerging questions
Many questions were raised by Aquino and others at #Intelligence26 on consent and governance, including how consent can be negotiated, for example, with CTs and MRI that already employ AI. If a consumer said no to AI use, would that then mean they couldn’t access such widely used equipment? What might that mean for treatment?
Other questions included what happens when AI is standard operating procedure? When and how often will consent have to be secured — just in the first instance or every time AI is being used? Who should secure consent: could it be the receptionist or always the clinician?
And is consent on AI the responsibility of the individual clinician, or should it be a larger education community education strategy, or an organisational requirement?
Aquinas also noted that guidelines about consent, such as those from the Australian Health Practitioner Regulation Agency (Ahpra), also talk about “meaningful” consent, not just “informed”.
Acknowledging that current codes or regulations have no single approach to the consent process, he indicated “this is what’s giving a lot of people headaches because the question remains: How do we do these in practice?”
Designing consent procedures
The most obvious model is opt-in consent, where patients are asked to agree for AI to be used in their care, either signing a form or agreeing verbally. An alternative is the opt-out approach, where patients have to verbally, or via a written document, reject AI applications in their care plan.
Or there is the indirect disclosure approach, where a clinician or service doesn’t ask for consent, but posts information on the premises which say that AI is being used in their care across the board.
That’s also why more research studies are needed to establish a co-designed consent form or consent process because regulators and clinicians “can’t assume that every patient will have the same level of preference to the level of information that they will receive in order to meaningfully provide their consent”, Aquino said.
The stakes may be high, he said, noting that Ahpra guidelines currently say informed consent is particularly important in AI models that refer private conversations or consultations, as there may be criminal implications if consent is not obtained before recording.
Clinical responsibilities
A recent article in the Australian Journal of General Practice outlines some of the responsibilities that clinicians will have, which could be significant, he said.
However, the questions outlined in the article (see below), which help GPs test their knowledge of AI systems, will take some study and time for most people to answer effectively.
“We can’t just expect clinicians to do everything overnight. It’s a huge burden,” Aquino said.
Associate Professor Rosie Dobson, a psychologist and Principal Advisor of AI Research and Evaluation at Health New Zealand | Te Whatu Ora, told the conference that consumers in Aotearoa New Zealand have consistently informed authorities that “they want to be able to trust the healthcare system”.
They don’t want to have to understand technical issues or concerns around cyber security, “they want to trust us to do that”, she said.
That’s putting a lot of responsibility on the system, she acknowledged, which makes governance essential, as well as acknowledgement that the benefits and risks for patients, clinicians and health systems can be different and “not always aligned”.
Who gets access and why?
Dobson said issues of access and equity are also vital.
AI tools can be expensive, so health services need to consider whether an algorithm or device is adding value or shifting funding from better, existing applications.
The value also may not just be monetary, she said. An important consideration will be whether it can perform everywhere equally.
“So it’s all very well in radiology, saying we’ve got this really cool AI tool, It’s going to be great for patients, and clinicians really want it, it’s awesome. But it can only be used on the latest machines, and it just so happens that those latest machines are in urban areas, they’re not in our rural hospitals.
“Is that something we’re willing to accept?” she asked.
Importantly, she said, in Aotearoa New Zealand, obligations under Te Tiriti, the Treaty of Waitangi mean AI governance must ensure that anything done with AI is for the benefit of Māori people.
Dobson has been involved in work to ensure a voice for Māori to address health inequities that are exacerbated and perpetuated through breaches of Te Tiriti, that recognises the challenge goes beyond simply modifying existing risk frameworks or governance guidance.
She and colleagues looked in 2023 at governance for AI tools within the Te Whatu Ora Waitematā public health service, which is responsible for the health of approximately 650,000 people in Auckland.
To address gaps that were detected, the National Artificial Intelligence and Algorithm Expert Advisory Group of which she’s a member, developed a framework that includes specific considerations for Māori, consumers, equity, and ethics, all as separate issues.
There are nine lenses, each with separate questions for practitioners to consider and which informed consumers could pose to their treating team.
Consumer representative Murray McLachlan, who is deputy chair of Health Consumers NSW and sits on the RANZCR AI Committee, also offered the College some advice.
“I suspect that the first time a patient who’s not familiar with how AI is being used will be bamboozled,” he said. “It’s your job as clinicians to try and make that as straightforward as possible.
“I suspect at the second consultation they’ll have a lot more questions, and maybe that’s something you need to start [thinking about] in terms of how that process goes ahead.”
Disclaimer from Marie McInerney: AI assisted with transcribing but all quotes were checked by the journalist.
Bookmark this link to follow Croakey Conference News Service’s coverage of #Intelligence26 and see this playlist of video interviews.
Add Comment