Structured reasoning failures compromise LLM interpretation of clinical oncology notes

Abstract

Large language models (LLMs) show strong performance on clinical benchmarks, yet their reasoning reliability in real-world oncology care remains unclear. We evaluated LLM reasoning on authentic oncology notes using a novel hierarchical error taxonomy across two retrospective cohorts spanning breast, pancreatic, and prostate cancer. GPT-4 produced reasoning errors in 23.1% of note interpretations, the majority reflecting cognitive bias patterns. Errors were more frequent in recommendation tasks and were strongly associated with guideline-discordant recommendations and lower clinician-rated clinical impact scores. Confirmation bias, anchoring bias, and omission errors were most strongly linked to potentially harmful outputs. Compared to GPT-4, GPT-5.1 demonstrated reduced error rates and improved clinical performance but retained structured reasoning failure patterns. Automated LLM-based evaluators detected error presence but failed to reliably classify subtypes, and a preliminary self-mitigation strategy yielded only modest improvement. Endpoint accuracy alone may mask clinically meaningful reasoning failures. Therefore, evaluating and monitoring reasoning fidelity should be a prerequisite for safe deployment of LLMs in oncology decision support.

Citation

@article{Kenaston26,
  author={Kenaston, Matthew W and Ayub, Umair and Parmar, Mihir and Kumar, Priya and Anjum, Muhammad Umair and Naqvi, Syed Arsalan Ahmed and Chaudhuri, Aadel A and Zakharia, Yousef and Heath, Elisabeth I and Bekaii-Saab, Tanios S and Tao, Cui and Van Allen, Eliezer M and Zhou, Ben and Choi, YooJung and Baral, Chitta and Riaz, Irbaz Bin},
  title={Structured reasoning failures compromise LLM interpretation of clinical oncology notes},  
  journal={npj Digital Medicine},
  year={2026},
  publisher={Nature Publishing Group UK London},
}