{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,7]],"date-time":"2026-10-07T02:57:56Z","timestamp":1791341876809,"version":"4.3.1"},"reference-count":30,"publisher":"American Medical Association (AMA)","issue":"6","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JAMA Netw Open"],"abstract":"\n Importance<\/jats:title>\n Emergency department (ED) quality review often uses administrative electronic triggers (eTriggers), but yields on detecting missed opportunities for diagnosis (MODs) are low. A commercial large language model (LLM) may help screen for MODs, yet evaluation data in real-world cohorts remain limited.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n To evaluate LLMs for identifying MODs in ED eTrigger cohorts.<\/jats:p>\n <\/jats:sec>\n \n Design, Setting, and Participants<\/jats:title>\n This retrospective diagnostic study of 2 eTrigger cohorts, ED discharge with return hospital admission within 72 hours and ED admission to the floor with intensive care unit (ICU) escalation within 24 hours, was conducted from April 2015 through March 2025 across 9 EDs (2 academic and 7 community) in 1 US health system. Samples included 200 encounters from the 72-hour return cohort and 100 encounters from the floor-to-ICU cohort; each case was adjudicated by 2 emergency physicians using a review process based on the Safer Dx framework.<\/jats:p>\n <\/jats:sec>\n \n Exposures<\/jats:title>\n Cases were evaluated by Claude Sonnet 4, Claude Sonnet 4.6, Claude Opus 4.6, Gemini 3 Pro, GPT-5, and GPT-5 mini.<\/jats:p>\n <\/jats:sec>\n \n Main Outcomes and Measures<\/jats:title>\n Main outcomes were sensitivity, specificity, positive predictive value, negative predictive value, area under the receiver operating characteristic curve (AUC), and reviewer-reviewer and reviewer-model concordance.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n Among 300 sampled encounters, 12 were excluded, leaving 288 analyzed encounters (median [IQR] age, 69 [54-79] years; 135 female [46.9%]) with 39 MODs (13.5%), including 21 of 191 (11.0%) in the 72-hour return cohort and 18 of 97 (18.6%) in the floor-to-ICU cohort. Interrater agreement was 81.9% (95% CI, 77.4%-86.1%), with Gwet AC1 of 0.77 (95% CI, 0.70-0.83). In the 72-hour return cohort, model sensitivity ranged from 42.9% (95% CI, 24.5%-63.5%) for GPT-5 mini to 85.7% (95% CI, 65.4%-95.0%) for Claude Sonnet 4, specificity from 55.9% (95% CI, 48.4%-63.1%) for Claude Sonnet 4 to 82.9% (95% CI, 76.6%-87.9%) for GPT-5 mini, and AUC from 0.65 (95% CI, 0.53-0.77) for GPT-5 mini to 0.73 (95% CI, 0.61-0.85) for Claude Sonnet 4. In the floor-to-ICU cohort, sensitivity ranged from 5.6% (95% CI, 1.0-25.8%) for GPT-5 mini to 55.6% (95% CI, 33.7%-75.4%) for Claude Sonnet 4, specificity from 64.6% (53.6%-74.2%) for Claude Sonnet 4 to 97.5% (95% CI, 91.2%-99.3%) for GPT-5 mini, and AUC from 0.57 (95% CI, 0.46-0.67) for GPT-5 mini to 0.82 (95% CI, 0.73-0.91) for GPT-5. Across cohorts, LLMs showed similar discrimination but different sensitivity-specificity tradeoffs; Claude Sonnet 4 generally favored higher sensitivity, whereas GPT-5 mini favored higher specificity.<\/jats:p>\n <\/jats:sec>\n \n Conclusions and Relevance<\/jats:title>\n In this diagnostic study of 2 ED eTrigger cohorts, model performance varied by cohort, with LLMs showing similar discrimination but different binary thresholds. These findings suggest that evaluation within the review workflow is needed before implementation and that reviewer-like concordance captures a distinct dimension of model behavior from discrimination.<\/jats:p>\n <\/jats:sec>","DOI":"10.1001\/jamanetworkopen.2026.20939","type":"journal-article","created":{"date-parts":[[2026,6,29]],"date-time":"2026-06-29T15:32:21Z","timestamp":1782747141000},"page":"e2620939","source":"Crossref","is-referenced-by-count":1,"title":["Screening for Missed Opportunities for Diagnosis in the ED Using eTriggers and Large Language Models"],"prefix":"10.1001","volume":"9","author":[{"given":"Clifford M.","family":"Marks","sequence":"first","affiliation":[{"name":"Department of Emergency Medicine, Georgetown University, Washington, DC"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sean","family":"Gibney","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bryan","family":"Stenson","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Deesha","family":"Sarma","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cynthia","family":"Gaudet","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, Boston Medical Center, Boston University Chobanian & Avedisian School of Medicine, Boston, Massachusetts"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Haadi","family":"Mombini","sequence":"additional","affiliation":[{"name":"Beth Israel Lahey Health Technology and Innovation, Beth Israel Lahey Health, Boston, Massachusetts"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thomas A.","family":"Buckley","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Harvard Medical School, Boston, Massachusetts"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mario","family":"Keko","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Larry A.","family":"Nathanson","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Laura G.","family":"Burke","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, Warren Alpert Medical School of Brown University, Providence, Rhode Island"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nathan I.","family":"Shapiro","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jonathan L.","family":"Burstein","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shamai A.","family":"Grossman","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anika","family":"Parab","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, Lahey Hospital & Medical Center, UMass Chan-Lahey School of Medicine, Burlington, Massachusetts"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alexander T.","family":"Janke","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, University of Michigan Medical School, Ann Arbor"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Arjun K.","family":"Manrai","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Harvard Medical School, Boston, Massachusetts"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Richard A.","family":"Taylor","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, University of Virginia School of Medicine, Charlottesville"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Carlo L.","family":"Rosen","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Adam","family":"Rodman","sequence":"additional","affiliation":[{"name":"Department of Medicine, Division of General Medicine, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Adrian D.","family":"Haimovich","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"10","published-online":{"date-parts":[[2026,6,29]]},"reference":[{"issue":"Suppl 2","key":"zoi260579r1","doi-asserted-by":"publisher","first-page":"ii21","DOI":"10.1136\/bmjqs-2012-001615","article-title":"The incidence of diagnostic error in medicine.","volume":"22","author":"Graber","year":"2013","journal-title":"BMJ Qual Saf"},{"issue":"1","key":"zoi260579r3","doi-asserted-by":"publisher","first-page":"e11","DOI":"10.1097\/PTS.0000000000000243","article-title":"Development of an emergency department trigger tool using a systematic search and modified Delphi process.","volume":"16","author":"Griffey","year":"2020","journal-title":"J Patient Saf"},{"issue":"2","key":"zoi260579r4","doi-asserted-by":"publisher","first-page":"230","DOI":"10.1016\/j.annemergmed.2019.07.032","article-title":"The emergency department trigger tool: a novel approach to screening for quality and safety events.","volume":"76","author":"Griffey","year":"2020","journal-title":"Ann Emerg Med"},{"issue":"6","key":"zoi260579r5","doi-asserted-by":"publisher","first-page":"625","DOI":"10.1016\/j.annemergmed.2025.06.613","article-title":"Multicenter trial of the emergency department trigger tool for adverse event detection.","volume":"86","author":"Griffey","year":"2025","journal-title":"Ann Emerg Med"},{"issue":"2","key":"zoi260579r6","doi-asserted-by":"publisher","first-page":"103","DOI":"10.1136\/bmjqs-2014-003675","article-title":"Advancing the science of measurement of diagnostic errors in healthcare: the Safer Dx framework.","volume":"24","author":"Singh","year":"2015","journal-title":"BMJ Qual Saf"},{"issue":"9","key":"zoi260579r7","doi-asserted-by":"publisher","first-page":"1605","DOI":"10.1377\/hlthaff.2009.0513","article-title":"Why diagnostic errors don\u2019t get any respect\u2013and what can be done about them.","volume":"29","author":"Wachter","year":"2010","journal-title":"Health Aff (Millwood)"},{"issue":"2","key":"zoi260579r8","doi-asserted-by":"publisher","first-page":"143","DOI":"10.1001\/jamainternmed.2024.6214","article-title":"Implementation of electronic triggers to identify diagnostic errors in emergency departments.","volume":"185","author":"Vaghani","year":"2025","journal-title":"JAMA Intern Med"},{"issue":"2","key":"zoi260579r9","doi-asserted-by":"publisher","first-page":"164","DOI":"10.1001\/jamainternmed.2023.7347","article-title":"Diagnostic errors in hospitalized adults who died or were transferred to intensive care.","volume":"184","author":"Auerbach","year":"2024","journal-title":"JAMA Intern Med"},{"issue":"1","key":"zoi260579r10","doi-asserted-by":"publisher","first-page":"103","DOI":"10.1007\/s11606-017-4164-1","article-title":"Defining and measuring diagnostic uncertainty in medicine: a systematic review.","volume":"33","author":"Bhise","year":"2018","journal-title":"J Gen Intern Med"},{"issue":"3","key":"zoi260579r11","doi-asserted-by":"publisher","first-page":"359","DOI":"10.1016\/j.ajem.2017.08.019","article-title":"72h returns: a trigger tool for diagnostic error.","volume":"36","author":"Aaronson","year":"2018","journal-title":"Am J Emerg Med"},{"issue":"1","key":"zoi260579r12","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s44401-024-00011-2","article-title":"Current and future state of evaluation of large language models for medical summarization tasks.","volume":"2","author":"Croxford","year":"2025","journal-title":"Npj Health Syst"},{"issue":"2","key":"zoi260579r13","doi-asserted-by":"publisher","DOI":"10.1136\/bmjoq-2023-002654","article-title":"Performance evaluation of ChatGPT in detecting diagnostic errors and their contributing factors: an analysis of 545 case reports of diagnostic errors.","volume":"13","author":"Harada","year":"2024","journal-title":"BMJ Open Qual"},{"issue":"1","key":"zoi260579r14","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.232714","article-title":"Potential of GPT-4 for detecting errors in radiology reports: implications for reporting accuracy.","volume":"311","author":"Gertz","year":"2024","journal-title":"Radiology"},{"issue":"2","key":"zoi260579r15","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.242575","article-title":"Generative large language models trained for detecting errors in radiology reports.","volume":"315","author":"Sun","year":"2025","journal-title":"Radiology"},{"issue":"1","key":"zoi260579r17","doi-asserted-by":"publisher","first-page":"274","DOI":"10.1038\/s41746-025-01670-7","article-title":"A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation.","volume":"8","author":"Asgari","year":"2025","journal-title":"NPJ Digit Med"},{"issue":"3","key":"zoi260579r18","doi-asserted-by":"publisher","first-page":"327","DOI":"10.1111\/acem.15066","article-title":"Leveraging artificial intelligence to reduce diagnostic errors in emergency medicine: challenges, opportunities, and future directions.","volume":"32","author":"Taylor","year":"2025","journal-title":"Acad Emerg Med"},{"issue":"1","key":"zoi260579r19","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1038\/s41591-024-03425-5","article-title":"The TRIPOD-LLM reporting guideline for studies using large language models.","volume":"31","author":"Gallifant","year":"2025","journal-title":"Nat Med"},{"issue":"4","key":"zoi260579r20","doi-asserted-by":"publisher","first-page":"390","DOI":"10.1111\/j.1553-2712.2011.01042.x","article-title":"Seventy-two-hour returns may not be a good indicator of safety in the emergency department: a national study.","volume":"18","author":"Pham","year":"2011","journal-title":"Acad Emerg Med"},{"issue":"11","key":"zoi260579r21","doi-asserted-by":"publisher","first-page":"1389","DOI":"10.1109\/LSP.2014.2337313","article-title":"Fast implementation of DeLong\u2019s algorithm for comparing the areas under correlated receiver operating characteristic curves.","volume":"21","author":"Sun","year":"2014","journal-title":"IEEE Signal Process Lett"},{"issue":"1","key":"zoi260579r22","doi-asserted-by":"publisher","first-page":"61","DOI":"10.1186\/1471-2288-13-61","article-title":"A comparison of Cohen\u2019s kappa and Gwet\u2019s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples.","volume":"13","author":"Wongpakaran","year":"2013","journal-title":"BMC Med Res Methodol"},{"issue":"1","key":"zoi260579r23","doi-asserted-by":"publisher","first-page":"8236","DOI":"10.1038\/s41467-024-52415-1","article-title":"Evaluating the use of large language models to provide clinical recommendations in the emergency department.","volume":"15","author":"Williams","year":"2024","journal-title":"Nat Commun"},{"issue":"4","key":"zoi260579r24","doi-asserted-by":"publisher","first-page":"319","DOI":"10.1001\/jama.2024.21700","article-title":"Testing and evaluation of health care applications of large language models: a systematic review.","volume":"333","author":"Bedi","year":"2025","journal-title":"JAMA"},{"key":"zoi260579r27","doi-asserted-by":"publisher","DOI":"10.2196\/63733","article-title":"Designing clinical decision support systems (CDSS)\u2014a user-centered lens of the design characteristics, challenges, and implications: systematic review.","volume":"27","author":"Bayor","year":"2025","journal-title":"J Med Internet Res"},{"key":"zoi260579r28","doi-asserted-by":"publisher","DOI":"10.2196\/69678","article-title":"Trust in artificial intelligence\u2013based clinical decision support systems among health care workers: systematic review.","volume":"27","author":"Tun","year":"2025","journal-title":"J Med Internet Res"},{"key":"zoi260579r29","doi-asserted-by":"publisher","DOI":"10.3389\/frhs.2023.1211150","article-title":"Implementing AI in healthcare-the relevance of trust: a scoping review.","volume":"3","author":"Steerling","year":"2023","journal-title":"Front Health Serv"},{"key":"zoi260579r30","doi-asserted-by":"publisher","DOI":"10.2196\/63649","article-title":"Facilitators and barriers to implementing AI in routine medical imaging: systematic review and qualitative analysis.","volume":"27","author":"Wenderott","year":"2025","journal-title":"J Med Internet Res"},{"key":"zoi260579r2","volume-title":"Improving Diagnosis In Health Care","author":"Ball","year":"2015"},{"key":"zoi260579r16","unstructured":"Brodeur? PG, Buckley? TA, Kanjee? Z, . Superhuman performance of a large language model on the reasoning tasks of a physician.? arXiv. Preprint posted online December 14, 2024. doi:10.48550\/arXiv.2412.10849"},{"key":"zoi260579r25","unstructured":"Li? J, Klinger? R. IPrOp: Interactive prompt optimization for large language models with a human in the loop. Paper presented at: 63rd Annual Meeting of the Association for Computational Linguistics; July 27 to August 1, 2025; Vienna, Austria. Accessed June 9, 2026. https:\/\/aclanthology.org\/2025.acl-srw.18\/"},{"key":"zoi260579r26","unstructured":"Trivedi? P, Chakraborty? S, Reddy? A, Aggarwal? V, Bedi? AS, Atia? GK. Align-pro: a principled approach to prompt optimization for LLM alignment. Paper presented at: 63rd Annual Meeting of the Association for Computational Linguistics; July 27 to August 1, 2025; Vienna, Austria. Accessed June 9, 2026. https:\/\/ojs.aaai.org\/index.php\/AAAI\/article\/view\/34979\/37134"}],"container-title":["JAMA Network Open"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/jamanetwork.com\/journals\/jamanetworkopen\/articlepdf\/2850946\/marks_2026_oi_260579_1782852605.009.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T21:28:09Z","timestamp":1782854889000},"score":1,"resource":{"primary":{"URL":"https:\/\/jamanetwork.com\/journals\/jamanetworkopen\/fullarticle\/2850946"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,29]]},"references-count":30,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2026,6,1]]}},"URL":"https:\/\/doi.org\/10.1001\/jamanetworkopen.2026.20939","relation":{},"ISSN":["2574-3805"],"issn-type":[{"value":"2574-3805","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,29]]}}}