{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,2]],"date-time":"2026-11-02T21:53:02Z","timestamp":1793656382868,"version":"4.5.1"},"reference-count":32,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Med Internet Res"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec sec-type=\"background\">\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Large language model (LLM) agents capable of generating and executing statistical code from natural language may broaden access to clinical data analysis, yet which pipeline stages they perform reliably and which require expert oversight remain poorly defined.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"objective\">\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>This study aimed to evaluate the performance and systematic failure modes of an LLM agent across 5 stages of a clinical data analysis workflow.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"methods\">\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>The publicly available dataset and R script (R Foundation for Statistical Computing) were drawn from a previously published study of 12-year outcomes in 7802 patients with eyes with neovascular age-related macular degeneration at Moorfields Eye Hospital. Participants were evaluated using an LLM agent (Claude; Anthropic) across 3 interaction modes (Chat, Code, and Cowork). It was asked to perform 3 levels of data analysis practice: prompt A, to generate research questions from raw data only; prompt B, to develop a statistical analysis plan (SAP) from a high-level clinical objective, then execute it; and prompt C, to execute an analysis given an investigator-drafted SAP. Each was replicated 3 times (27 total runs). Qualitative evaluation of research question thematic coverage (prompt A), SAP completeness against a reference checklist (prompt B), and evaluation of execution outputs against validated reference values and of result text and narrative summaries against execution logs (prompts B and C) was conducted.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"results\">\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>The agent generated 18 clinically grounded questions spanning 7 domains; Cowork mode uniquely reached 3 thematic areas requiring data-driven methods. All 9 SAPs correctly identified the statistical framework. Kaplan-Meier estimates were near-identical across 17 completed runs. Systematic execution errors emerged: SAP quality did not predict code correctness, and within-mode errors propagated identically across independent repetitions. Result text accurately reflected execution logs in nearly all runs, though unit propagation and an undisclosed postcrash rerun were identified. Of 17 narrative summaries, 8 were fully satisfactory; 2 runs produced clinically meaningful errors.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec sec-type=\"conclusions\">\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>LLM agents perform reliably for question generation and SAP drafting but require expert verification of formula composition, cohort boundary logic, and concordance computation before results are reported. Using an ophthalmology dataset as a controlled testbed, this study develops and applies an evaluation framework whose lessons are likely applicable across clinical specialties.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.2196\/99597","type":"journal-article","created":{"date-parts":[[2026,9,8]],"date-time":"2026-09-08T18:45:11Z","timestamp":1788893111000},"page":"e99597-e99597","source":"Crossref","is-referenced-by-count":2,"title":["Performance, Failures, and Oversight of a Large Language Model Agent for Clinical Data Analysis: Evaluation Study"],"prefix":"10.2196","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0493-9958","authenticated-orcid":false,"given":"Yilan","family":"Wu","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2852-6912","authenticated-orcid":false,"given":"Dun Jack","family":"Fu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0840-6422","authenticated-orcid":false,"given":"Yukun","family":"Zhou","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4915-4353","authenticated-orcid":false,"given":"Siegfried K","family":"Wagner","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9239-745X","authenticated-orcid":false,"given":"Pearse A","family":"Keane","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,9,8]]},"reference":[{"issue":"7972","key":"R1","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","journal-title":"Nature"},{"key":"R2","unstructured":"Nori H King N McKinney SM Carignan D Horvitz E . Capabilities of GPT-4 on medical challenge problems. arXiv. Preprint posted online on Mar 20, 2023. [doi: 10.48550\/arXiv.2303.13375]"},{"issue":"8","key":"R3","doi-asserted-by":"publisher","first-page":"1930","DOI":"10.1038\/s41591-023-02448-8","article-title":"Large language models in medicine","volume":"29","author":"Thirunavukarasu","journal-title":"Nat Med"},{"issue":"15","key":"R4","doi-asserted-by":"publisher","first-page":"1320","DOI":"10.1001\/jama.2023.27861","article-title":"Comparative analysis of multimodal large language model performance on clinical vignette questions","volume":"331","author":"Han","journal-title":"JAMA"},{"issue":"11","key":"R5","doi-asserted-by":"publisher","DOI":"10.2196\/19612","article-title":"Amplifying domain expertise in clinical data pipelines","volume":"8","author":"Rahman","journal-title":"JMIR Med Inform"},{"issue":"8","key":"R6","doi-asserted-by":"publisher","first-page":"577","DOI":"10.7326\/M18-2516","article-title":"Inappropriate statistical analysis and reporting in medical research: perverse incentives and institutional solutions","volume":"169","author":"Localio","journal-title":"Ann Intern Med"},{"issue":"1","key":"R7","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s12909-019-1842-1","article-title":"Medical graduate views on statistical learning needs for clinical practice: a comprehensive survey","volume":"20","author":"MacDougall","journal-title":"BMC Med Educ"},{"key":"R8","doi-asserted-by":"publisher","DOI":"10.2196\/68135","article-title":"Leveraging large language models and agent-based systems for scientific data analysis: validation study","volume":"12","author":"Peasley","journal-title":"JMIR Ment Health"},{"issue":"2","key":"R9","doi-asserted-by":"publisher","first-page":"245","DOI":"10.1038\/s41551-025-01463-z","article-title":"CRISPR-GPT for agentic automation of gene-editing experiments","volume":"10","author":"Qu","journal-title":"Nat Biomed Eng"},{"issue":"1","key":"R10","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-023-00896-7","article-title":"Evaluating large language models on medical evidence summarization","volume":"6","author":"Tang","journal-title":"NPJ Digit Med"},{"key":"R11","unstructured":"Jimenez CE Yang J Wettig A et al. SWE-bench: can language models resolve real-world github issues. Presented at: International Conference on Learning Representations 2024 (ICLR 2024); May 7-11, 2024; Vienna, Austria. URL: https:\/\/proceedings.iclr.cc\/paper_files\/paper\/2024\/hash\/edac78c3e300629acfe6cbe9ca88fb84-Abstract-Conference.html [Accessed 20-08-2026]"},{"issue":"1","key":"R12","doi-asserted-by":"publisher","first-page":"57","DOI":"10.1001\/jamaophthalmol.2020.5044","article-title":"Insights from survival analyses during 12 years of anti-vascular endothelial growth factor therapy for neovascular age-related macular degeneration","volume":"139","author":"Fu","journal-title":"JAMA Ophthalmol"},{"key":"R13","unstructured":"UK research ethics committee (REC) policy document. Health Research Authority; 2026. URL: https:\/\/s3.eu-west-2.amazonaws.com\/www.hra.nhs.uk\/media\/documents\/UK_REC_Policy_document_Final_v_1.0_28th_April_2026.pdf#page=3.11 [Accessed 26-08-2026]"},{"issue":"2","key":"R14","doi-asserted-by":"publisher","first-page":"186","DOI":"10.1038\/s41591-021-01229-5","article-title":"DECIDE-AI: new reporting guidelines to bridge the development-to-implementation gap in clinical artificial intelligence","volume":"27","author":"The DECIDE-AI Steering Group","journal-title":"Nat Med"},{"key":"R15","unstructured":"Fu DJ Keane P . Insights from survival analyses during 12 years of anti-VEGF therapy for neovascular age-related macular degeneration. Dryad. 2020. URL: https:\/\/datadryad.org\/dataset\/doi:10.5061\/dryad.nvx0k6dqg [Accessed 20-08-2026]"},{"key":"R16","unstructured":"Si C Yang D Hashimoto T . Can LLMs generate novel research ideas? a large-scale human study with 100+ NLP researchers. Presented at: International Conference on Learning Representations 2025 (ICLR 2025); Apr 24-28, 2025; Singapore. URL: https:\/\/openreview.net\/forum?id=M23dTGWCZy [Accessed 20-08-2026]"},{"key":"R17","unstructured":"Ren S Xie C Jian P Ren Z Leng C Zhang J . Towards scientific intelligence: a survey of LLM-based scientific agents. arXiv. Preprint posted online on Mar 31, 2026. [doi: 10.48550\/arXiv.2503.24047]"},{"key":"R18","unstructured":"Yeti\u015ftiren B \u00d6zsoy I Ayerdem M T\u00fcz\u00fcn E . Evaluating the code quality of AI-assisted code generation tools: an empirical study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPT. arXiv. Preprint posted online on Apr 21, 2023. [doi: 10.48550\/arXiv.2304.10778]"},{"key":"R19","doi-asserted-by":"crossref","unstructured":"Merler M Pan R Krishna R Ho TK Pavuluri R Vukovic M . Usage, effects and requirements for ai coding assistants in the enterprise: an empirical study. Presented at: LLM4Code \u201926: The 3rd International Workshop on Large Language Models For Code; Apr 12-18, 2026; Rio de Janeiro Brazil. [doi: 10.1145\/3786181.378872]","DOI":"10.1145\/3786181.3788727"},{"issue":"1","key":"R20","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-024-55628-6","article-title":"Large language models lack essential metacognition for reliable medical reasoning","volume":"16","author":"Griot","journal-title":"Nat Commun"},{"issue":"1","key":"R21","doi-asserted-by":"publisher","first-page":"39426","DOI":"10.1038\/s41598-025-22940-0","article-title":"Limitations of large language models in clinical problem-solving arising from inflexible reasoning","volume":"15","author":"Kim","journal-title":"Sci Rep"},{"key":"R22","unstructured":"Golchin S Surdeanu M . Time travel in LLMs: tracing data contamination in large language models. Presented at: The 12th International Conference on Learning Representations (ICLR 2024); May 7-11, 2024; Vienna, Austria. URL: https:\/\/openreview.net\/forum?id=2Rwq6c3tvr [Accessed 07-08-2026]"},{"key":"R23","doi-asserted-by":"crossref","unstructured":"Ye W Hu J Li L Wang H Chen G Zhao J . Data contamination calibration for black-box llms. In: Ku LW Martins A Srikumar V , editors. Findings of the Association for Computational Linguistics: ACL 2024. Association for Computational Linguistics; 2024:10845-10861. [doi: 10.18653\/v1\/2024.findings-acl.644]","DOI":"10.18653\/v1\/2024.findings-acl.644"},{"key":"R24","doi-asserted-by":"publisher","DOI":"10.1162\/TACL.a.20","article-title":"Data contamination quiz: a tool to detect and estimate contamination in large language models","volume":"13","author":"Golchin","journal-title":"Trans Assoc Comput Linguist"},{"issue":"1","key":"R25","doi-asserted-by":"publisher","DOI":"10.1038\/s41597-026-06731-4","article-title":"StatLLM: a dataset for evaluating the performance of large language models in statistical analysis","volume":"13","author":"Song","journal-title":"Sci Data"},{"issue":"8","key":"R26","doi-asserted-by":"publisher","first-page":"1732","DOI":"10.1038\/s41551-025-01587-2","article-title":"Making large language models reliable data science programming copilots for biomedical research","volume":"10","author":"Wang","journal-title":"Nat Biomed Eng"},{"issue":"1","key":"R27","doi-asserted-by":"publisher","DOI":"10.1056\/AIoa2400555","article-title":"Autonomous LLM-driven research \u2014 from data to human-verifiable research papers","volume":"2","author":"Ifargan","journal-title":"NEJM AI"},{"issue":"8107","key":"R28","doi-asserted-by":"publisher","first-page":"914","DOI":"10.1038\/s41586-026-10265-5","article-title":"Towards end-to-end automation of AI research","volume":"651","author":"Lu","journal-title":"Nature"},{"key":"R29","doi-asserted-by":"crossref","unstructured":"Gao R Xiao SC . Nonstandard errors in AI agents. SSRN. Preprint posted online on Mar 16, 2026. [doi: 10.2139\/ssrn.6427518]","DOI":"10.2139\/ssrn.6427518"},{"key":"R30","doi-asserted-by":"crossref","unstructured":"Zhu Y Du S Li B Luo Y Tang N . Are large language models good statisticians? Presented at: Advances in Neural Information Processing Systems 37; Dec 10-15, 2024; Vancouver, BC, Canada. [doi: 10.52202\/079017-2005]","DOI":"10.52202\/079017-2005"},{"issue":"4","key":"R31","doi-asserted-by":"publisher","first-page":"231","DOI":"10.54912\/jci.2025.0020","article-title":"ChatGPT and medical statistics: a narrative review on opportunities, pitfalls, and the principle of \u201ctrust, but verify\u201d","volume":"4","author":"Jeon","journal-title":"J Cardiovasc Interv"},{"issue":"6","key":"R32","doi-asserted-by":"publisher","first-page":"467","DOI":"10.5468\/ogs.25232","article-title":"Statistical analysis using ChatGPT in medical research","volume":"68","author":"Kim","journal-title":"Obstet Gynecol Sci"}],"container-title":["Journal of Medical Internet Research"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,9,8]],"date-time":"2026-09-08T18:45:15Z","timestamp":1788893115000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jmir.org\/2026\/1\/e99597"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,8]]},"references-count":32,"URL":"https:\/\/doi.org\/10.2196\/99597","relation":{},"ISSN":["1438-8871"],"issn-type":[{"value":"1438-8871","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,8]]},"article-number":"v28i7972e99597"}}