{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,6]],"date-time":"2026-10-06T16:56:14Z","timestamp":1791305774837,"version":"4.3.1"},"reference-count":33,"publisher":"Oxford University Press (OUP)","issue":"5","license":[{"start":{"date-parts":[[2026,10,6]],"date-time":"2026-10-06T00:00:00Z","timestamp":1791244800000},"content-version":"vor","delay-in-days":27,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2026,9,9]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objectives<\/jats:title>\n                    <jats:p>To simultaneously evaluate and compare LLM performance in clinical reasoning with a minimally necessary criteria for clinical alignment with patients.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Material and Methods<\/jats:title>\n                    <jats:p>We adapt the MedNLI dataset to create MedNLI-Pain, a new benchmark task that evaluates if LLMs that make accurate clinical inferences about patients can also anticipate when those patients are in pain. MedNLI-Pain consists of patient vignettes originating from MedNLI that are then annotated with physician assessments of the described patient\u2019s pain.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>All LLMs had higher agreement with physicians in MedNLI than in MedNLI-Pain and its counterfactually augmented subset. This trend persisted when accounting for the higher difficulty of MedNLI-Pain by normalizing to human performance. Large language model performance improved with more training data in MedNLI but not MedNLI-Pain.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Discussion<\/jats:title>\n                    <jats:p>Large language models may lack necessary capabilities (eg, recognizing patients\u2019 pain) for clinical alignment. Success at a clinical reasoning task, in this case medical natural language inference, may not entail commensurate ability to align with patients\u2019 interests.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>Further research on alignment with patients is needed because the implications may direct future research into either closed-ended or agentic applications.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/jamiaopen\/ooag143","type":"journal-article","created":{"date-parts":[[2026,7,18]],"date-time":"2026-07-18T11:43:14Z","timestamp":1784374994000},"source":"Crossref","is-referenced-by-count":0,"title":["MedNLI-Pain benchmarks a minimally necessary requirement for large language model alignment"],"prefix":"10.1093","volume":"9","author":[{"given":"Christopher","family":"Wong","sequence":"first","affiliation":[{"name":"Rutgers Robert Wood Johnson Medical School Department of Medicine, , New Brunswick, NJ 08901,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Divy","family":"Kumar","sequence":"additional","affiliation":[{"name":"Feinberg School of Medicine, Northwestern University , Chicago, IL 60611,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Amiin","family":"Muse","sequence":"additional","affiliation":[{"name":"McCormick School of Engineering, Northwestern University Department of Computer Science, , Evanston, IL 60208,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Philip","family":"Wang","sequence":"additional","affiliation":[{"name":"McCormick School of Engineering, Northwestern University Department of Computer Science, , Evanston, IL 60208,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Finn","family":"Wintz","sequence":"additional","affiliation":[{"name":"McCormick School of Engineering, Northwestern University Department of Computer Science, , Evanston, IL 60208,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sasha-ann","family":"East","sequence":"additional","affiliation":[{"name":"Rutgers Robert Wood Johnson Medical School Department of Medicine, , New Brunswick, NJ 08901,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jason","family":"Lyou","sequence":"additional","affiliation":[{"name":"Bridgeport Hospital\/Yale New Haven Hospital Department of Palliative Care, , Bridgeport, CT 06610,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Naveena","family":"Yanamala","sequence":"additional","affiliation":[{"name":"Rutgers Robert Wood Johnson Medical School Department of Medicine, , New Brunswick, NJ 08901,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zach","family":"Wood-Doughty","sequence":"additional","affiliation":[{"name":"McCormick School of Engineering, Northwestern University Department of Computer Science, , Evanston, IL 60208,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2026,10,6]]},"reference":[{"key":"2026100612302600000_ooag143-B1","doi-asserted-by":"publisher","first-page":"842","DOI":"10.1001\/jama.2023.1044","article-title":"Appropriateness of cardiovascular disease prevention recommendations obtained from a popular online chat-based artificial intelligence model","volume":"329","author":"Sarraju","year":"2023","journal-title":"JAMA"},{"key":"2026100612302600000_ooag143-B2","doi-asserted-by":"publisher","first-page":"1437","DOI":"10.1001\/jamaoncol.2023.2947","article-title":"Assessment of artificial intelligence chatbot responses to top searched queries about cancer","volume":"9","author":"Pan","year":"2023","journal-title":"JAMA Oncol"},{"key":"2026100612302600000_ooag143-B3","doi-asserted-by":"publisher","first-page":"e2336483","DOI":"10.1001\/jamanetworkopen.2023.36483","article-title":"Accuracy and reliability of chatbot responses to physician questions","volume":"6","author":"Goodman","year":"2023","journal-title":"JAMA Netw Open"},{"key":"2026100612302600000_ooag143-B4","doi-asserted-by":"publisher","first-page":"e2330320","DOI":"10.1001\/jamanetworkopen.2023.30320","article-title":"Comparison of ophthalmologist and large language model chatbot responses to online patient eye care questions","volume":"6","author":"Bernstein","year":"2023","journal-title":"JAMA Netw Open"},{"key":"2026100612302600000_ooag143-B5","doi-asserted-by":"publisher","author":"Bubeck","year":".     2023.","DOI":"10.48550\/arXiv.2303.12712"},{"key":"2026100612302600000_ooag143-B6","doi-asserted-by":"publisher","author":"Nori","year":"2023","DOI":"10.48550\/arXiv.2303.13375"},{"key":"2026100612302600000_ooag143-B7","doi-asserted-by":"publisher","first-page":"AIcs2500239","DOI":"10.1056\/AIcs2500239","article-title":"Assessment of short-answer questions by ChatGPT in a medical school course","volume":"3","author":"Grey","year":"2026","journal-title":"NEJM AI"},{"key":"2026100612302600000_ooag143-B8","doi-asserted-by":"publisher","first-page":"581","DOI":"10.1001\/jamainternmed.2024.0295","article-title":"Clinical reasoning of a generative artificial intelligence model compared with physicians","volume":"184","author":"Cabral","year":"2024","journal-title":"JAMA Intern Med"},{"key":"2026100612302600000_ooag143-B9","doi-asserted-by":"publisher","first-page":"e2347075","DOI":"10.1001\/jamanetworkopen.2023.47075","article-title":"Artificial intelligence vs clinician performance in estimating probabilities of diagnoses before and after testing","volume":"6","author":"Rodman","year":"2023","journal-title":"JAMA Netw Open"},{"key":"2026100612302600000_ooag143-B10","doi-asserted-by":"publisher","first-page":"78","DOI":"10.1001\/jama.2023.8288","article-title":"Accuracy of a generative artificial intelligence model in a complex diagnostic challenge","volume":"330","author":"Kanjee","year":"2023","journal-title":"JAMA"},{"key":"2026100612302600000_ooag143-B11","doi-asserted-by":"publisher","first-page":"89","DOI":"10.1038\/s41746-025-02274-x","article-title":"ARTEMIS: a pilot study comparing AI-based and expert therapeutic decisions in simulated clinical cases of neuroendocrine neoplasms","volume":"9","author":"Lamberti","year":"2025","journal-title":"NPJ Digit Med"},{"key":"2026100612302600000_ooag143-B12","doi-asserted-by":"publisher","first-page":"106","DOI":"10.1016\/j.ahj.2025.02.016","article-title":"Re-engineering the clinical approach to suspected cardiac chest pain assessment in the emergency department by expediting research evidence to practice using artificial intelligence. (RAPIDx AI)\u2014a cluster randomized study design","volume":"285","author":"Khan","year":"2025","journal-title":"Am Heart J"},{"key":"2026100612302600000_ooag143-B13","doi-asserted-by":"publisher","first-page":"Ioa2400296","DOI":"10.1056\/AIoa2400296","article-title":"Impact of artificial intelligence\u2013based triage decision support on emergency department care","volume":"2","author":"Taylor","year":"2025","journal-title":"NEJM AI"},{"key":"2026100612302600000_ooag143-B14","doi-asserted-by":"publisher","first-page":"100078","DOI":"10.1016\/j.lanprc.2025.100078","article-title":"Artificial intelligence in primary care: innovation at a crossroads","volume":"2","author":"Laranjo","year":"2026","journal-title":"Lancet Prim Care"},{"key":"2026100612302600000_ooag143-B15","doi-asserted-by":"publisher","first-page":"442","DOI":"10.1038\/s41586-025-08866-7","article-title":"Towards conversational diagnostic artificial intelligence","volume":"642","author":"Tu","year":"2025","journal-title":"Nature"},{"key":"2026100612302600000_ooag143-B16","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1038\/s41591-024-03328-5","article-title":"An evaluation framework for clinical use of large language models in patient interaction tasks","volume":"31","author":"Johri","year":"2025","journal-title":"Nat Med"},{"key":"2026100612302600000_ooag143-B17","doi-asserted-by":"publisher","first-page":"ooae078","DOI":"10.1093\/jamiaopen\/ooae078","article-title":"Development and evaluation of an artificial intelligence-based workflow for the prioritization of patient portal messages","volume":"7","author":"Yang","year":"2024","journal-title":"JAMIA Open"},{"key":"2026100612302600000_ooag143-B18","doi-asserted-by":"publisher","first-page":"AIcs2500539","DOI":"10.1056\/AIcs2500539","article-title":"Early clinical evaluation of AI triage of chest radiographs: time to diagnosis for suspected cancer and number of urgent CT referrals","volume":"3","author":"Mathew","year":"2025","journal-title":"NEJM AI"},{"key":"2026100612302600000_ooag143-B19","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3770749","article-title":"AI alignment: a contemporary survey","volume":"58","author":"Ji","year":"2025","journal-title":"ACM Comput Surv"},{"key":"2026100612302600000_ooag143-B20","doi-asserted-by":"publisher","first-page":"317","DOI":"10.1038\/s41746-024-01306-2","article-title":"Simulated misuse of large language models and clinical credit systems","volume":"7","author":"Anibal","year":"2024","journal-title":"NPJ Digit Med"},{"key":"2026100612302600000_ooag143-B21","doi-asserted-by":"publisher","first-page":"1873","DOI":"10.1038\/s41591-025-03626-6","article-title":"Sociodemographic biases in medical decision making by large language models","volume":"31","author":"Omar","year":"2025","journal-title":"Nat Med"},{"key":"2026100612302600000_ooag143-B22","doi-asserted-by":"publisher","first-page":"605","DOI":"10.1038\/s41746-025-02008-z","article-title":"When helpfulness backfires: LLMs and the risk of false medical information due to sycophantic behavior","volume":"8","author":"Chen","year":"2025","journal-title":"NPJ Digit Med"},{"key":"2026100612302600000_ooag143-B23","doi-asserted-by":"publisher","first-page":"1895","DOI":"10.1056\/NEJMra2214183","article-title":"Medical artificial intelligence and human values","volume":"390","author":"Kun-Hsing","year":"2024","journal-title":"N Engl J Med"},{"key":"2026100612302600000_ooag143-B24","doi-asserted-by":"publisher","author":"Romanov","year":"2018","DOI":"10.18653\/v1\/D18-1187"},{"key":"2026100612302600000_ooag143-B25","doi-asserted-by":"publisher","first-page":"160035","DOI":"10.1038\/sdata.2016.35","article-title":"MIMIC-III, a freely accessible critical care database","volume":"3","author":"Johnson","year":"2016","journal-title":"Sci Data"},{"key":"2026100612302600000_ooag143-B26","article-title":"Explaining the efficacy of counterfactually augmented data","volume-title":"International Conference on Learning Representations","author":"Kaushik","year":"2021"},{"key":"2026100612302600000_ooag143-B27","author":"R Core Team"},{"key":"2026100612302600000_ooag143-B28","doi-asserted-by":"publisher","first-page":"1","DOI":"10.18637\/jss.v028.i05","article-title":"Building predictive models in R using the caret package","volume":"28","author":"Kuhn","year":"2008","journal-title":"J Stat Soft"},{"key":"2026100612302600000_ooag143-B29","doi-asserted-by":"publisher","author":"Wickham","year":"2016","DOI":"10.1007\/978-3-319-24277-4"},{"key":"2026100612302600000_ooag143-B30","doi-asserted-by":"publisher","first-page":"i1","DOI":"10.1093\/ageing\/afx192","article-title":"The assessment of pain in older people: UK national guidelines","volume":"47","author":"Schofield","year":"2018","journal-title":"Age Ageing"},{"key":"2026100612302600000_ooag143-B31","doi-asserted-by":"publisher","first-page":"395","DOI":"10.1111\/j.1526-4637.2009.00787.x","article-title":"Comparing the psychometric properties of the checklist of nonverbal pain behaviors (CNPI) and the pain assessment in advanced dementia (PAIN-AD) instruments","volume":"11","author":"Ersek","year":"2010","journal-title":"Pain Med"},{"key":"2026100612302600000_ooag143-B32","doi-asserted-by":"publisher","first-page":"194","DOI":"10.1038\/s41746-022-00742-2","article-title":"A large language model for electronic health records","volume":"5","author":"Yang","year":"2022","journal-title":"NPJ Digit Med"},{"key":"2026100612302600000_ooag143-B33","doi-asserted-by":"publisher","first-page":"AIp2401165","DOI":"10.1056\/AIp2401165","article-title":"AI agents, automaticity, and value alignment in health care","volume":"2","author":"Andrew","year":"2025","journal-title":"NEJM AI"}],"container-title":["JAMIA Open"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamiaopen\/article-pdf\/9\/5\/ooag143\/71628203\/ooag143.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamiaopen\/article-pdf\/9\/5\/ooag143\/71628203\/ooag143.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,6]],"date-time":"2026-10-06T16:30:32Z","timestamp":1791304232000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamiaopen\/article\/doi\/10.1093\/jamiaopen\/ooag143\/8869099"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,9]]},"references-count":33,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2026,9,9]]}},"URL":"https:\/\/doi.org\/10.1093\/jamiaopen\/ooag143","relation":{},"ISSN":["2574-2531"],"issn-type":[{"value":"2574-2531","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026,10]]},"published":{"date-parts":[[2026,9,9]]},"article-number":"ooag143"}}