Australian health services are deploying AI systems that are largely unable to remove the influence of patient data once it has been used to train them, according to researchers from the Australian Institute for Machine Learning (AIML).
AIML researchers Dr Anthony Porter and Associate Professor Emily Kirkpatrick say “machine unlearning” – the ability to modify an AI model so it behaves as though specified data had never been used in its training – is emerging as a significant gap in clinical AI governance.
The emerging issue was presented at the Health Informatics Conference (HIC) in Sydney earlier this month, with machine unlearning described as a “missing capability in clinical AI governance”.
A/Prof Kirkpatrick told Pulse+IT the problem was “near-universal, though largely invisible.”
“Most machine learning models cannot selectively ‘forget’ once patient data has shaped a model’s parameters, its influence persists even if the source records are later deleted,” she said.
“Health services routinely delete data to meet privacy obligations, but deleting the data does not remove its influence from any model trained on it.
“Until recently, this gap was rarely discussed in procurement or governance conversations, which is precisely why we’ve focused on it across this research.”
Deleting data not enough
The researchers said the problem stemmed from how machine learning models are trained.
Rather than retaining a patient’s record in one identifiable location, training data influences potentially millions of internal model settings, meaning its influence becomes distributed throughout the model.
The HIC presentation suggested that “there is no row to delete – forgetting has to be engineered.”
Machine unlearning aims to modify an existing model so it behaves as closely as possible to a model that has never encountered the specified data.
It differs from deleting the source record or suppressing particular outputs. Completely retraining a model without the affected data provides the strongest assurance, but can be too expensive and time-consuming to perform routinely – particularly with large models.
A/Prof Kirkpatrick said removing data from large foundation models remained particularly difficult.
“Full retraining without the data is the gold standard but is prohibitively expensive for large models, and approximate unlearning techniques are still maturing,” she said.
“AIML has been working on methodologies for verifying how that influence has truly been removed.
“That’s why the procurement stage matters so much, as it’s far easier to negotiate whether your data enters a foundation model than to extract its influence afterwards.”
Questions for vendors
The researchers said health services needed to consider the ability to remove data influence when procuring AI, rather than waiting until a patient or organisation requests its removal.
The HIC presentation outlined several possible approaches, ranging from complete retraining through to approximate unlearning, partitioning models so affected components can be retrained, and certified removal.
However, some of those capabilities needed to be designed into an AI system from the outset.
A/Prof Kirkpatrick said health services should ask vendors four key questions.
“First, is our data used to train or fine-tune your models, and if so, which ones? Second, if we withdraw, or a patient withdraws consent, can you remove that data’s influence and how would you demonstrate that to us?
“Third, do you maintain the data lineage and training records that make unlearning technically feasible? Fourth, what happens contractually at the end of engagement. How does our data’s influence persist in models you continue to commercialise?”
A/Prof Kirkpatrick said any vendor who cannot answer these questions “clearly should be seen as a flag to the contract manager.”
The researchers said health services would also need model registries and data lineage capable of identifying which datasets influenced particular model versions and downstream services.
Once those foundations were in place, an unlearning request could be managed similarly to other clinical technology change controls, including mapping affected systems, selecting an appropriate removal method, validating performance and maintaining an audit trail.
However, A/Prof Kirkpatrick said she was not aware of any health service currently capable of conducting a governed machine unlearning process from end to end.
“The technical capability exists in research settings, but an operational version requires things most services don’t yet have which includes the data lineage tracking, contractual rights, defined governance triggers and verification methods,” she said.
“This is one of the reasons we have linked machine unlearning with AUScribe to better understand what the reality of unlearning is for health service AI.”
Indigenous data sovereignty
The researchers also viewed machine unlearning as having potentially important implications for Indigenous data sovereignty.
The HIC presentation highlighted Australian Aboriginal and Torres Strait Islander data sovereignty principles, the international CARE principles and Te Mana Raraunga Māori data sovereignty framework in New Zealand.
The researchers noted that while communities could authorise, refuse, condition and withdraw uses of their data, those rights risked becoming largely performative if an AI system cannot technically act on that decision.
“We see this as one of the most important applications,” A/Prof Kirkpatrick said.
“Indigenous data sovereignty principles, including CARE and Māori data sovereignty frameworks, assert that communities hold ongoing authority over their data, including the right to withdraw it.
“At present that right is largely unenforceable once data has trained an AI model, where the records can be returned or deleted, but the model retains what it learned.”
The researchers proposed an “assurance ladder” ranging from deletion of source records through to containment of affected models, targeted retraining or unlearning and ultimately certified or independently verified removal.
They warned that simply preventing a model or dataset from being used for a particular purpose should not be described as unlearning unless the removal of its influence has actually been validated.
Procurement a faster lever
A/Prof Kirkpatrick said machine unlearning ultimately needed to be considered across procurement, governance and regulation, but noted that procurement offered health services the most immediate opportunity to act.
“Procurement is the fastest lever and health services can start asking the question of unlearning today, with contract clauses on data influence being standard,” she said.
“Governance frameworks should define the triggers for an unlearning event to include consent withdrawal, a data breach, identified bias, or a community exercising data sovereignty.
“Regulation will likely follow where we see privacy law, which already provides deletion rights, and the logical next step is recognising that deletion of data without removal of its influence is incomplete.”
The researchers said AI systems should ultimately be designed for “reversibility, not permanence”, noting that the ability to remove data influence also exposes where control over healthcare AI ultimately resides.
Their presentation noted that “AI that cannot forget is incompatible with modern healthcare governance.”
“The ability, or inability, to remove data influence reveals whether health services, communities, and clinicians retain authority, or whether that power has silently shifted to vendors and legacy architectures.”
Add Comment