A one-size-fits-all approach is not the best strategy when designing artificial intelligence systems that assist users in disease diagnosis. A new study found that while AI assistance generally improved the accuracy of non-experts and clinicians in diagnosing skin diseases, AI explainability methods had different impacts depending on the users’ knowledge level. Explainable AI methods help users know when to trust a model’s predictions by describing or validating the model’s decision-making. For instance, a model might use a heat map to highlight image regions that were most important in its diagnosis or a large language model (LLM) to explain the prediction in plain language. The researchers tested non-experts and primary care providers in skin disease diagnosis, with and without the help of different explainable AI systems. They found that non-experts’ diagnostic accuracy improved, but it was largely due to deference to the AI system. Non-experts trusted LLM-based explanations whether they were right or wrong, and found explanations more convincing when they were vague or generic. By contrast, clinicians were not tripped up by incorrect AI assistance and performed best when given only a model’s prediction, with no accompanying explanation. "Good AI systems can improve performance in some health settings, but this has to be balanced carefully with algorithmic deference that can lead to more error. We know that both AI and explainability methods can engage automation bias in humans, and this anchoring effect is something that must be accounted for when we design AI systems," says Marzyeh Ghassemi. Blogger's Review: This study highlights the importance of considering the user's expertise and usage habits when designing medical AI assistance systems to avoid over-reliance on models and resulting errors. It also emphasizes the need to develop explainability methods that encourage critical thinking rather than over-reliance on the model.