Right Diagnoses, Decorative Reasoning:A Perturbation Audit of Medical Chain-of-Thought

Imagine you're a clinician relying on AI to assist in diagnosing patients. You want to trust that the AI's reasoning process is sound, but how do you know if the explanations it provides are genuinely reflective of its decision-making? Current evaluations often treat the reasoning chain as a black box, leading to uncertainty about whether the AI's outputs are reliable or just coincidental. This is problematic, especially in high-stakes medical settings where the cost of errors can be significant. The authors address this issue by developing a medical perturbation audit that rigorously tests how changes to the input affect the AI's reasoning and answers. They use a set of clinically motivated edits to both the questions and the reasoning chains, allowing them to classify the models based on their failure modes. Their findings reveal that a significant portion of the models do not register meaningful changes in their answers when the reasoning chain is altered, indicating a lack of faithfulness in the CoT process. This new framework not only provides a reusable yardstick for evaluating medical LLMs but also highlights the importance of understanding the relationship between reasoning and outcomes in AI applications. For builders, this means there's now a structured way to assess and improve the reliability of AI systems in critical domains like healthcare.

The approach introduces a novel framework for evaluating CoT in medical LLMs.

The study employs multiple independent tests and clinician re-annotations for validation.