A European study has highlighted potential challenges for using automated tools in healthcare. The Spanish researchers who conducted the study concluded it was important to “investigate the errors that humans (including doctors) make when working with algorithms, in order to learn how to minimise the problems that arise from them.
The University of the Basque Country researchers tested over 220 physicians onine, presenting them with a hypothetical situation. The physicians were asked to imagine they had the option to treat patients for a rare disease using a not-yet-proven treatment still under development.
They were told that an AI system had identified which patients were more or less likely to benefit from the treatment. The physicians then chose which patients to treat, and after being presented with data on patient recovery, rated their perceptions of how reliable the AI was.
Crucially, the actual effectiveness of the hypothetical treatment did not align with the AI recommendations. In one experiment, the treatment was equally moderately effective for all patients, and in a second experiment, it was equally ineffective for all.
However, in both experiments, the physicians tended to rate the AI system as reliable and apparently did not use the patient recovery data to conclude that the AI recommendations were incorrect. In the second experiment, the physicians did not realise that the treatment was entirely ineffective.
Aranzazu Vinas of the University of the Basque Country, Spain, and colleagues presented their findings in the open-access journal PLOS Digital Health.
Lead author Aranzazu Vinas noted that in both experiments, physicians mostly trusted the AI’s classifications and had trouble learning from the feedback. “Furthermore, in the second experiment, professionals did not notice that the treatment was completely ineffective.”
Co-author Helena Matute said, “People tend to say that there is always a human controlling the algorithm, but our experiments show that doctors (as well as anyone else) have problems in learning from the available evidence when it contradicts the suggestions of an algorithm.”
Add Comment