{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,2]],"date-time":"2026-10-02T11:21:27Z","timestamp":1790940087193,"version":"4.1.0"},"reference-count":35,"publisher":"Public Library of Science (PLoS)","issue":"5","license":[{"start":{"date-parts":[[2026,5,26]],"date-time":"2026-05-26T00:00:00Z","timestamp":1779753600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["U54 TW012043-01"],"award-info":[{"award-number":["U54 TW012043-01"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000009","name":"Foundation for the National Institutes of Health","doi-asserted-by":"publisher","award":["OT2OD032701"],"award-info":[{"award-number":["OT2OD032701"]}],"id":[{"id":"10.13039\/100000009","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["2148451"],"award-info":[{"award-number":["2148451"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100020549","name":"Food Industry Research and Development Institute","doi-asserted-by":"publisher","award":["RS-2024-00403047"],"award-info":[{"award-number":["RS-2024-00403047"]}],"id":[{"id":"10.13039\/100020549","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000156","name":"Fonds de Recherche du Qu\u00e9bec - Sant\u00e9","doi-asserted-by":"publisher","award":["Frqs - 331771"],"award-info":[{"award-number":["Frqs - 331771"]}],"id":[{"id":"10.13039\/501100000156","id-type":"DOI","asserted-by":"publisher"}]},{"name":"2024 Detweiler Travelling Fellowship"},{"DOI":"10.13039\/100006492","name":"Division of Intramural Research, National Institute of Allergy and Infectious Diseases","doi-asserted-by":"publisher","award":["1K08AI181606-01A1"],"award-info":[{"award-number":["1K08AI181606-01A1"]}],"id":[{"id":"10.13039\/100006492","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.plosdigitalhealth.org"],"crossmark-restriction":false},"short-container-title":["PLOS Digit Health"],"abstract":"<jats:p>Rapid integration of Large Language Models (LLMs) into healthcare has exposed a critical disconnect between technical performance and clinical value. While state-of-the-art models achieve impressive scores on standardized medical examinations, their real-world impact remains limited, with few models progressing to successful clinical integration. This disconnect persists, in part, due to a proliferation of evaluation practices that prioritize static, decontextualized benchmarks. To help address this gap, we propose five foundational principles to guide contextually appropriate evaluations of healthcare AI: Local (grounded in specific deployment contexts), Task-specific (aligned with intended clinical use), Agile (continuously adaptive), Reflective (acknowledging limitations and inherent value-sensitivity), and Community-partnered (centering affected voices). We argue that emphasis on these principles can help shift evaluation practice towards assessment of artificial intelligence. This reorientation is essential for developing healthcare AI that not only performs well technically, but also can meaningfully improve patient care, serve communities for defined purposes, and mitigate (rather than exacerbate) health disparities.<\/jats:p>","DOI":"10.1371\/journal.pdig.0001115","type":"journal-article","created":{"date-parts":[[2026,5,26]],"date-time":"2026-05-26T17:27:50Z","timestamp":1779816470000},"page":"e0001115","update-policy":"https:\/\/doi.org\/10.1371\/journal.pdig.corrections_policy","source":"Crossref","is-referenced-by-count":2,"title":["Moving beyond the benchmarks: Five foundational principles for meaningful AI evaluation in healthcare"],"prefix":"10.1371","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1871-3909","authenticated-orcid":true,"given":"Catherine G.","family":"Bielick","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Aya","family":"Awwad","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jacob","family":"Ellen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Laleh","family":"Jalilian","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Liam G.","family":"McCoy","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vishala","family":"Mishra","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Esli","family":"Osmanlliu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Stephen R.","family":"Pfohl","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Leo A.","family":"Celi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2026,5,26]]},"reference":[{"key":"pdig.0001115.ref001","article-title":"How large language models perform on the United States medical licensing examination: a systematic review","author":"D Brin","year":"2023","journal-title":"MedRxiv"},{"issue":"3","key":"pdig.0001115.ref002","doi-asserted-by":"crossref","first-page":"943","DOI":"10.1038\/s41591-024-03423-7","article-title":"Toward expert-level medical question answering with large language models","volume":"31","author":"K Singhal","year":"2025","journal-title":"Nat Med"},{"issue":"1","key":"pdig.0001115.ref003","doi-asserted-by":"crossref","first-page":"58","DOI":"10.1038\/s41746-024-01064-1","article-title":"To warrant clinical adoption AI models require a multi-faceted implementation evaluation","volume":"7","author":"D van de Sande","year":"2024","journal-title":"NPJ Digit Med"},{"key":"pdig.0001115.ref004","author":"X Wang","year":"2024"},{"key":"pdig.0001115.ref005","unstructured":"HLAB-AI U. Governing AI for humanity. 2024."},{"key":"pdig.0001115.ref006","unstructured":"World Health Organization. Ethics and governance of artificial intelligence for health: Large multi-modal models. World Health Organization; 2024."},{"issue":"4","key":"pdig.0001115.ref007","doi-asserted-by":"crossref","first-page":"319","DOI":"10.1001\/jama.2024.21700","article-title":"Testing and evaluation of health care applications of large language models: a systematic review","volume":"333","author":"S Bedi","year":"2025","journal-title":"JAMA"},{"issue":"10","key":"pdig.0001115.ref008","doi-asserted-by":"crossref","first-page":"2718","DOI":"10.1038\/s41591-024-03203-3","article-title":"Not all AI health tools with regulatory authorization are clinically validated","volume":"30","author":"S Chouffani El Fassi","year":"2024","journal-title":"Nat Med"},{"key":"pdig.0001115.ref009","article-title":"Medical large language model benchmarks should prioritize construct validity (version 1)","author":"A Alaa","year":"2025","journal-title":"arXiv"},{"key":"pdig.0001115.ref010","article-title":"Measurement to meaning: a validity-centered framework for AI evaluation (version 4)","author":"O Salaudeen","year":"2025","journal-title":"arXiv"},{"issue":"3","key":"pdig.0001115.ref011","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1038\/s42256-021-00307-0","article-title":"Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans","volume":"3","author":"M Roberts","year":"2021","journal-title":"Nat Mach Intell"},{"key":"pdig.0001115.ref012","article-title":"Stakeholder perspectives of clinical artificial intelligence implementation: systematic review of qualitative evidence","volume":"25","author":"HDJ Hogg","year":"2023","journal-title":"J Med Internet Res"},{"key":"pdig.0001115.ref013","article-title":"The illusion of readiness in health AI","author":"Y Gu","year":"2025","journal-title":"arXiv preprint"},{"issue":"1","key":"pdig.0001115.ref014","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1186\/s41512-022-00136-8","article-title":"Targeted validation: validating clinical prediction models in their intended population and setting","volume":"6","author":"M Sperrin","year":"2022","journal-title":"Diagn Progn Res"},{"issue":"1","key":"pdig.0001115.ref015","doi-asserted-by":"crossref","first-page":"90","DOI":"10.1377\/hlthaff.2024.00842","article-title":"Current use and evaluation of artificial intelligence and predictive models in US hospitals","volume":"44","author":"P Nong","year":"2025","journal-title":"Health Aff"},{"issue":"3","key":"pdig.0001115.ref016","doi-asserted-by":"crossref","first-page":"283","DOI":"10.1056\/NEJMc2104626","article-title":"The clinician and dataset shift in artificial intelligence","volume":"385","author":"SG Finlayson","year":"2021","journal-title":"N Engl J Med"},{"issue":"8","key":"pdig.0001115.ref017","doi-asserted-by":"crossref","first-page":"1065","DOI":"10.1001\/jamainternmed.2021.2626","article-title":"External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients","volume":"181","author":"A Wong","year":"2021","journal-title":"JAMA Intern Med"},{"issue":"9","key":"pdig.0001115.ref018","doi-asserted-by":"crossref","first-page":"2613","DOI":"10.1038\/s41591-024-03097-1","article-title":"Evaluation and mitigation of the limitations of large language models in clinical decision-making","volume":"30","author":"P Hager","year":"2024","journal-title":"Nat Med"},{"key":"pdig.0001115.ref019","article-title":"Attention-driven reasoning: unlocking the potential of large language models","author":"B Liao","year":"2024","journal-title":"CoRR"},{"issue":"6464","key":"pdig.0001115.ref020","doi-asserted-by":"crossref","first-page":"447","DOI":"10.1126\/science.aax2342","article-title":"Dissecting racial bias in an algorithm used to manage the health of populations","volume":"366","author":"Z Obermeyer","year":"2019","journal-title":"Science"},{"key":"pdig.0001115.ref021","article-title":"Keeping medical AI healthy: a review of detection and correction methods for system degradation","author":"H Guan","year":"2025","journal-title":"arXiv Preprint"},{"issue":"10","key":"pdig.0001115.ref022","doi-asserted-by":"crossref","first-page":"1358","DOI":"10.7326\/M23-0949","article-title":"Implications of the use of artificial intelligence predictive models in health care settings\u00a0: a simulation study","volume":"176","author":"A Vaid","year":"2023","journal-title":"Ann Intern Med"},{"key":"pdig.0001115.ref023","doi-asserted-by":"crossref","DOI":"10.2196\/37685","article-title":"The need to prioritize model-updating processes in clinical Artificial Intelligence (AI) models: protocol for a scoping review","volume":"12","author":"AU Otokiti","year":"2023","journal-title":"JMIR Res Protoc"},{"key":"pdig.0001115.ref024","author":"MH Rahman","year":"2024"},{"key":"pdig.0001115.ref025","doi-asserted-by":"crossref","DOI":"10.2196\/46407","article-title":"Evaluating artificial intelligence in clinical settings-let us not reinvent the wheel","volume":"26","author":"K Cresswell","year":"2024","journal-title":"J Med Internet Res"},{"issue":"3","key":"pdig.0001115.ref026","doi-asserted-by":"crossref","first-page":"491","DOI":"10.1093\/jamia\/ocz192","article-title":"A governance model for the application of AI in health care","volume":"27","author":"S Reddy","year":"2020","journal-title":"J Am Med Inform Assoc"},{"key":"pdig.0001115.ref027","doi-asserted-by":"crossref","DOI":"10.1515\/9780804795722","volume-title":"Evaluation foundations revisited: cultivating a life of the mind for practice","author":"T Schwandt","year":"2020"},{"issue":"11","key":"pdig.0001115.ref028","doi-asserted-by":"crossref","first-page":"1989","DOI":"10.1093\/jamia\/ocac136","article-title":"Engaging the disability community in informatics research: rationales and practical steps","volume":"29","author":"RS Valdez","year":"2022","journal-title":"J Am Med Inform Assoc"},{"issue":"8","key":"pdig.0001115.ref029","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pdig.0000561","article-title":"Community-engaged artificial intelligence research: a scoping review","volume":"3","author":"TJ Loftus","year":"2024","journal-title":"PLOS Digit Health"},{"key":"pdig.0001115.ref030","article-title":"Understanding challenges to the interpretation of disaggregated evaluations of algorithmic fairness","author":"SR Pfohl","year":"2025","journal-title":"arXiv"},{"key":"pdig.0001115.ref031","doi-asserted-by":"crossref","first-page":"108331","DOI":"10.1016\/j.drugalcdep.2020.108331","article-title":"Community dashboards to support data-informed decision-making in the HEALing communities study","volume":"217","author":"E Wu","year":"2020","journal-title":"Drug Alcohol Depend"},{"issue":"10","key":"pdig.0001115.ref032","article-title":"Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension","volume":"2","author":"SPIRIT-AI and CONSORT-AI Working Group","year":"2020","journal-title":"Lancet Digit Health"},{"key":"pdig.0001115.ref033","doi-asserted-by":"crossref","DOI":"10.1136\/bmj-2023-078378","article-title":"TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods","volume":"385","author":"GS Collins","year":"2024","journal-title":"BMJ"},{"key":"pdig.0001115.ref034","doi-asserted-by":"crossref","unstructured":"NIST AI. Artificial intelligence risk management framework (AI RMF 1.0). 2023. pp. 100\u20131. Available from: https:\/\/nvlpubs.nist.gov\/nistpubs\/ai\/nist.ai","DOI":"10.6028\/NIST.AI.100-1.jpn"},{"key":"pdig.0001115.ref035","unstructured":"U.S. Food and Drug Administration. Guiding principles for good machine learning practice for medical device development. Available from: https:\/\/www.fda.gov\/medical-devices\/software-medical-device-samd\/good-machine-learning-practice-medical-device-development-guiding-principles. 2021. Accessed 2026 February 7."}],"container-title":["PLOS Digital Health"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pdig.0001115","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,26]],"date-time":"2026-05-26T17:27:55Z","timestamp":1779816475000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pdig.0001115"}},"subtitle":[],"editor":[{"given":"Yue","family":"Li","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2026,5,26]]},"references-count":35,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2026,5,26]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pdig.0001115","relation":{},"ISSN":["2767-3170"],"issn-type":[{"value":"2767-3170","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,5,26]]}}}