The benefits of medical AI assistance vary based on user expertise
Study finds non-experts deferred to LLM-based diagnostic assistance, even when it was wrong, while clinicians caught AI errors. New research found non-experts deferred to AI-based assistance in diagnosing skin cancer, even when it was wrong, while clinicians were more likely to catch AI errors. Adam Zewe | MIT News Publication Date : August 4, 2026 Press Inquiries Press Contact : Abby Abazorius Email: abbya@mit.
Key Takeaways
- edu MIT News Office : A new MIT study found that non-experts tended to trust AI-generated diagnostic advice - even when it was incorrect - while clinicians were more likely to recognize the AI's mistakes.
Credits : Credit: iStock Previous image Next image A one-size-fits-all approach likely isn't the best strategy when designing artificial intelligence systems that assist users in disease diagnosis.
- Explainable AI methods help users know when to trust a model's predictions by describing or validating the model's decision-making.
For instance, a model might use a heat map to highlight image regions that were most important in its diagnosis or a large language model (LLM) to explain the prediction in plain language.
- Non-experts trusted LLM-based explanations whether they were right or wrong, and found explanations more convincing when they were vague or generic.
By contrast, clinicians were not tripped up by incorrect AI assistance and performed best when given only a model's prediction, with no accompanying explanation.
- Our findings show that those with the least medical knowledge are most likely to be led astray when explainable AI models give an erroneous output," says Roxana Daneshjou, a co-author and assistant professor of biomedical data science and dermatology at Stanford University.
These results underscore the importance of building AI systems with users in mind and of developing explainability methods that encourage critical thinking rather than overreliance on the model, the researchers say.
- Often the people who could benefit most from AI are the ones most likely to be led astray by it, so how we present a recommendation matters as much as whether it's correct," says lead author Orson Xu, an assistant professor in the Department of Biomedical Informatics at Columbia University.
New research found non-experts deferred to AI-based assistance in diagnosing skin cancer, even when it was wrong, while clinicians were more likely to catch AI errors. Study finds non-experts deferred to LLM-based diagnostic assistance, even when it was wrong, while clinicians caught AI errors. Adam Zewe | MIT News Publication Date : August 4, 2026 Press Inquiries Press Contact : Abby Abazorius Email: abbya@mit.
edu MIT News Office : A new MIT study found that non-experts tended to trust AI-generated diagnostic advice - even when it was incorrect - while clinicians were more likely to recognize the AI's mistakes. Credits : Credit: iStock Previous image Next image A one-size-fits-all approach likely isn't the best strategy when designing artificial intelligence systems that assist users in disease diagnosis. A new study by researchers at MIT and elsewhere found that, while AI assistance generally improved the accuracy of non-experts and clinicians in diagnosing skin diseases, AI explainability methods had different impacts depending on the users' knowledge level.
Explainable AI methods help users know when to trust a model's predictions by describing or validating the model's decision-making. For instance, a model might use a heat map to highlight image regions that were most important in its diagnosis or a large language model (LLM) to explain the prediction in plain language. In this study, researchers tested non-experts and primary care providers in skin disease diagnosis, with and without the help of different explainable AI systems.
For more details please read the original article at MIT News AI.
Continue Learning
Comments
Sign in to join the conversation