{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,15]],"date-time":"2026-08-15T00:51:33Z","timestamp":1786755093027,"version":"build-2736575974"},"reference-count":51,"publisher":"American Medical Association (AMA)","issue":"11","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JAMA Netw Open"],"abstract":"\n Importance<\/jats:title>\n Validation of prognostic tools is an essential component of their integration into clinical practice.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n To validate 4 delirium risk stratification tools in an independent emergency department (ED) cohort.<\/jats:p>\n <\/jats:sec>\n \n Design, Setting, and Participants<\/jats:title>\n This prognostic study included a retrospective cohort of patients 65 years or older who presented to an academic ED between January 1, 2021, and December 31, 2024. Data were analyzed from February 10 to August 21, 2025.<\/jats:p>\n <\/jats:sec>\n \n Exposure<\/jats:title>\n Delirium screening included the Delirium Triage Screen followed by the brief Confusional Assessment Method as part of the standard of care. Four delirium risk scores were obtained, including the Kennedy rule, the Zucchelli rule, the Mayo Delirium Prediction (MDP) tools, and the Recognizing Delirium in Emergency Medicine (REDEEM) score.<\/jats:p>\n <\/jats:sec>\n \n Main Outcomes and Measures<\/jats:title>\n Main outcomes included risk score performances and diagnostic test accuracy measured using the area under the receiver operating characteristics curve (AUROC), sensitivity, specificity, negative (NPV) and positive (PPV) predictive values, and positive and negative likelihood ratios (LRs) with 95% CIs. Secondary outcomes were discriminative and calibration measures.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n A total of 44\u202f578 patients were included, of whom 1701 (3.8%) were diagnosed with delirium. The median age was 80.0 (IQR, 75.0-85.0) years, and 22\u202f786 patients (51.1%) were female. The Kennedy rule had an AUROC of 0.777 (95% CI, 0.766-0.789). At a cutoff of 5, sensitivity was 0.55 (95% CI, 0.52-0.57); specificity, 0.85 (95% CI, 0.84-0.85); positive LR, 3.54 (95% CI, 3.37-3.72); negative LR, 0.54 (95% CI, 0.51-0.57); PPV, 0.12 (95% CI, 0.12-0.13); and NPV, 0.98 (95% CI, 0.98-0.98). The Zucchelli tool had an AUROC of 0.701 (95% CI, 0.686-0.713). At a cutoff of 5, sensitivity was 0.68 (95% CI, 0.66-0.70); specificity, 0.66 (95% CI, 0.65-0.66); positive LR, 1.99 (95% CI, 1.92-2.06); negative LR, 0.49 (95% CI, 0.45-0.52); PPV, 0.07 (95% CI, 0.07-0.08); and NPV, 0.98 (95% CI, 0.98-0.98). The MDP tool had an AUROC of 0.898 (95% CI, 0.891-0.905). At a 30% cutoff, sensitivity was 0.51 (95% CI, 0.48-0.53); specificity, 0.95 (95% CI, 0.95-0.95); positive LR, 9.69 (95% CI, 9.11-10.31); negative LR, 0.52 (95% CI, 0.50-0.55); PPV, 0.28 (95% CI, 0.26-0.29); and NPV, 0.98 (95% CI, 0.98-0.98). The REDEEM score demonstrated the highest AUROC at 0.921 (95% CI, 0.914-0.929). Using a cutoff of 11 or greater, sensitivity was 0.83 (95% CI, 0.81-0.85); specificity, 0.92 (95% CI, 0.91-0.92), negative LR, 0.18 (95% CI, 0.17-0.20); positive LR, 9.91 (95% CI, 9.54-10.29); PPV, 0.28 (95% CI, 0.27-0.29); and NPV, 0.99 (95% CI, 0.99-0.99).<\/jats:p>\n <\/jats:sec>\n \n Conclusions and Relevance<\/jats:title>\n In this prognostic study comparing 4 delirium tools to improve delirium detection and identify high-risk patients, the REDEEM score and the MDP tool had better performance for delirium detection in the ED. These findings lay the groundwork for integrating validated risk stratification into ED workflows to improve early delirium detection and inform prevention strategies for high-risk older adults.<\/jats:p>\n <\/jats:sec>","DOI":"10.1001\/jamanetworkopen.2025.40920","type":"journal-article","created":{"date-parts":[[2025,11,3]],"date-time":"2025-11-03T16:00:57Z","timestamp":1762185657000},"page":"e2540920","source":"Crossref","is-referenced-by-count":3,"title":["Validation of 4 Risk Stratification Tools for Delirium in the Emergency Department"],"prefix":"10.1001","volume":"8","author":[{"given":"Manuela","family":"Bartolacci","sequence":"first","affiliation":[{"name":"Department of Emergency Medicine, Mayo Clinic, Rochester, Minnesota"},{"name":"Department of Internal Medicine and Medical Therapy, University of Pavia, Pavia, Italy"},{"name":"Department of Industrial and Information Engineering, University of Pavia, Pavia, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kayla P.","family":"Carpenter","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, Mayo Clinic, Rochester, Minnesota"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Molly M.","family":"Jeffery","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, Mayo Clinic, Rochester, Minnesota"},{"name":"Robert D. and Patricia E. Kern Center for the Science of Health Care Delivery, Mayo Clinic, Rochester, Minnesota"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Aidan F.","family":"Mullan","sequence":"additional","affiliation":[{"name":"Department of Quantitative Health Sciences, Mayo Clinic, Rochester, Minnesota"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christopher R.","family":"Carpenter","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, Mayo Clinic, Rochester, Minnesota"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fernanda","family":"Bellolio","sequence":"additional","affiliation":[{"name":"Department of Emergency Medicine, Mayo Clinic, Rochester, Minnesota"},{"name":"Robert and Arlene Kogod Center on Aging, Mayo Clinic, Rochester, Minnesota"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"10","published-online":{"date-parts":[[2025,11,3]]},"reference":[{"key":"zoi251122r1","doi-asserted-by":"publisher","first-page":"105","DOI":"10.1016\/j.ajem.2023.01.040","article-title":"Association of delirium with increased short-term mortality among older emergency department patients: a cohort study.","volume":"66","author":"Arneson","year":"2023","journal-title":"Am J Emerg Med"},{"key":"zoi251122r2","doi-asserted-by":"publisher","first-page":"141","DOI":"10.1186\/s12916-014-0141-2","article-title":"The DSM-5 criteria, level of arousal and delirium diagnosis: inclusiveness is safer.","volume":"12","author":"European Delirium Association","year":"2014","journal-title":"BMC Med"},{"issue":"10","key":"zoi251122r3","doi-asserted-by":"publisher","first-page":"1014","DOI":"10.1111\/acem.14935","article-title":"Delirium detection in the emergency department: a diagnostic accuracy meta-analysis of history, physical examination, laboratory tests, and screening instruments.","volume":"31","author":"Carpenter","year":"2024","journal-title":"Acad Emerg Med"},{"issue":"1","key":"zoi251122r4","doi-asserted-by":"publisher","first-page":"19","DOI":"10.1111\/acem.14166","article-title":"Delirium prevention, detection, and treatment in emergency medicine settings: a Geriatric Emergency Care Applied Research (GEAR) network scoping review and consensus statement.","volume":"28","author":"Carpenter","year":"2021","journal-title":"Acad Emerg Med"},{"key":"zoi251122r5","doi-asserted-by":"publisher","first-page":"121","DOI":"10.1016\/j.ajem.2022.05.058","article-title":"Delirium prevalence in geriatric emergency department patients: a systematic review and meta-analysis.","volume":"59","author":"Chen","year":"2022","journal-title":"Am J Emerg Med"},{"issue":"12","key":"zoi251122r6","doi-asserted-by":"publisher","first-page":"5","DOI":"10.17294\/2694-4715.1010","article-title":"A survey of delirium self-reported knowledge and practices among emergency physicians in the United States.","volume":"2","author":"Chary","year":"2021","journal-title":"J Geriatr Emerg Med"},{"issue":"2","key":"zoi251122r7","doi-asserted-by":"publisher","first-page":"341","DOI":"10.1111\/jgs.17581","article-title":"Evaluating older adults with cognitive dysfunction: a qualitative study with emergency clinicians.","volume":"70","author":"Chary","year":"2022","journal-title":"J Am Geriatr Soc"},{"issue":"12","key":"zoi251122r8","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.20616","article-title":"Emergency clinicians\u2019 perceptions of communication tools to establish the mental baseline of older adults: a qualitative study.","volume":"13","author":"Chary","year":"2021","journal-title":"Cureus"},{"issue":"4","key":"zoi251122r9","doi-asserted-by":"publisher","first-page":"1328","DOI":"10.1111\/jgs.18147","article-title":"Language discordance in emergency department delirium screening: results from a qualitative interview-based study.","volume":"71","author":"Chary","year":"2023","journal-title":"J Am Geriatr Soc"},{"issue":"5","key":"zoi251122r10","doi-asserted-by":"publisher","first-page":"1368","DOI":"10.1111\/jgs.17745","article-title":"Balancing vision with pragmatism: the geriatric emergency department guidelines\u2014realistic expectations from emergency medicine and geriatric medicine.","volume":"70","author":"Shih","year":"2022","journal-title":"J Am Geriatr Soc"},{"issue":"4","key":"zoi251122r11","doi-asserted-by":"publisher","first-page":"549","DOI":"10.1016\/j.annemergmed.2021.03.005","article-title":"Risk factors for delirium in older adults in the emergency department: a systematic review and meta-analysis.","volume":"78","author":"Oliveira J E Silva","year":"2021","journal-title":"Ann Emerg Med"},{"key":"zoi251122r12","doi-asserted-by":"publisher","DOI":"10.1016\/j.annemergmed.2025.02.003","article-title":"Outcomes of older adults with delirium discharged from the emergency department.","author":"Howick","year":"2025","journal-title":"Ann Emerg Med"},{"issue":"5","key":"zoi251122r13","doi-asserted-by":"publisher","first-page":"551","DOI":"10.1016\/j.annemergmed.2013.11.010","article-title":"Screening for delirium in the emergency department: a systematic review.","volume":"63","author":"LaMantia","year":"2014","journal-title":"Ann Emerg Med"},{"issue":"2","key":"zoi251122r14","doi-asserted-by":"publisher","DOI":"10.1002\/emp2.12421","article-title":"ED-DEL: development of a change package and toolkit for delirium in the emergency department.","volume":"2","author":"Kennedy","year":"2021","journal-title":"J Am Coll Emerg Physicians Open"},{"issue":"6","key":"zoi251122r15","doi-asserted-by":"publisher","first-page":"681","DOI":"10.1016\/j.annemergmed.2023.05.013","article-title":"An outcome comparison between geriatric and nongeriatric emergency departments.","volume":"82","author":"Gettel","year":"2023","journal-title":"Ann Emerg Med"},{"issue":"4","key":"zoi251122r16","article-title":"Examination of geriatric care processes implemented in level 1 and level 2 geriatric emergency departments.","volume":"3","author":"Santangelo","year":"2022","journal-title":"J Geriatr Emerg Med"},{"issue":"5","key":"zoi251122r17","doi-asserted-by":"publisher","first-page":"956","DOI":"10.1111\/jgs.16437","article-title":"Delirium in the emergency department: moving from tool-based research to system-wide change.","volume":"68","author":"Kennedy","year":"2020","journal-title":"J Am Geriatr Soc"},{"key":"zoi251122r18","doi-asserted-by":"publisher","DOI":"10.56392\/001c.55690","article-title":"A qualitative study of emergency department delirium prevention initiatives.","author":"Chary","year":"2022","journal-title":"Delirium Commun"},{"issue":"4","key":"zoi251122r19","doi-asserted-by":"publisher","first-page":"367","DOI":"10.1016\/j.annemergmed.2021.06.013","article-title":"Reach and adoption of a geriatric emergency department accreditation program in the United States.","volume":"79","author":"Kennedy","year":"2022","journal-title":"Ann Emerg Med"},{"issue":"4","key":"zoi251122r20","doi-asserted-by":"publisher","first-page":"1137","DOI":"10.1007\/s11739-023-03239-2","article-title":"Emergency department crowding: an overview of reviews describing measures causes, and harms.","volume":"18","author":"Pearce","year":"2023","journal-title":"Intern Emerg Med"},{"issue":"4","key":"zoi251122r21","doi-asserted-by":"publisher","first-page":"476","DOI":"10.1111\/acem.14423","article-title":"Recognizing Delirium in Geriatric Emergency Medicine: the REDEEM risk stratification score.","volume":"29","author":"Oliveira J E Silva","year":"2022","journal-title":"Acad Emerg Med"},{"issue":"10","key":"zoi251122r22","doi-asserted-by":"publisher","first-page":"969","DOI":"10.1111\/acem.14939","article-title":"Risk factors and risk stratification approaches for delirium screening: a Geriatric Emergency Department Guidelines 2.0 systematic review.","volume":"31","author":"Seidenfeld","year":"2024","journal-title":"Acad Emerg Med"},{"key":"zoi251122r23","doi-asserted-by":"publisher","first-page":"202","DOI":"10.1016\/j.jclinepi.2021.11.024","article-title":"GRADE concept paper 2: concepts for judging certainty on the calibration of prognostic models in a body of validation studies.","volume":"143","author":"Foroutan","year":"2022","journal-title":"J Clin Epidemiol"},{"key":"zoi251122r24","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2024.111344","article-title":"GRADE concept paper 8: judging the certainty of discrimination performance estimates of prognostic models in a body of validation studies.","volume":"170","author":"Foroutan","year":"2024","journal-title":"J Clin Epidemiol"},{"key":"zoi251122r25","doi-asserted-by":"publisher","first-page":"40","DOI":"10.1186\/1471-2288-14-40","article-title":"External validation of multivariable prediction models: a systematic review of methodological conduct and reporting.","volume":"14","author":"Collins","year":"2014","journal-title":"BMC Med Res Methodol"},{"issue":"1","key":"zoi251122r26","doi-asserted-by":"publisher","first-page":"12","DOI":"10.1186\/s12874-021-01469-6","article-title":"Completeness of reporting of clinical prediction models developed using supervised machine learning: a systematic review.","volume":"22","author":"Andaur Navarro","year":"2022","journal-title":"BMC Med Res Methodol"},{"issue":"3","key":"zoi251122r27","doi-asserted-by":"publisher","first-page":"478","DOI":"10.1055\/a-2073-3736","article-title":"Leveraging the electronic health record to implement emergency department delirium screening.","volume":"14","author":"Chary","year":"2023","journal-title":"Appl Clin Inform"},{"issue":"6","key":"zoi251122r28","doi-asserted-by":"publisher","first-page":"521","DOI":"10.1016\/j.jaclp.2022.05.006","article-title":"Validation and recalibration of modified Mayo Delirium Prediction Tool in a hospitalized cohort.","volume":"63","author":"Pagali","year":"2022","journal-title":"J Acad Consult Liaison Psychiatry"},{"key":"zoi251122r29","doi-asserted-by":"publisher","first-page":"S574","DOI":"10.21037\/jtd.2019.01.25","article-title":"Developing prediction models for clinical use using logistic regression: an overview.","volume":"11","author":"Shipe","year":"2019","journal-title":"J Thorac Dis"},{"issue":"6","key":"zoi251122r30","doi-asserted-by":"publisher","first-page":"1658","DOI":"10.1111\/jgs.18704","article-title":"Around the EQUATOR with Clin-STAR: prediction modeling opportunities and challenges in aging research.","volume":"72","author":"Deardorff","year":"2024","journal-title":"J Am Geriatr Soc"},{"issue":"11","key":"zoi251122r31","doi-asserted-by":"publisher","first-page":"1251","DOI":"10.1111\/acem.13449","article-title":"Validation of the brief confusion assessment method for screening delirium in elderly medical patients in a German emergency department.","volume":"25","author":"Baten","year":"2018","journal-title":"Acad Emerg Med"},{"issue":"5","key":"zoi251122r32","doi-asserted-by":"publisher","first-page":"457","DOI":"10.1016\/j.annemergmed.2013.05.003","article-title":"Diagnosing delirium in older emergency department patients: validity and reliability of the delirium triage screen and the brief confusion assessment method.","volume":"62","author":"Han","year":"2013","journal-title":"Ann Emerg Med"},{"issue":"3","key":"zoi251122r33","doi-asserted-by":"publisher","first-page":"462","DOI":"10.1111\/jgs.12692","article-title":"Delirium risk prediction, healthcare use and mortality of elderly adults in the emergency department.","volume":"62","author":"Kennedy","year":"2014","journal-title":"J Am Geriatr Soc"},{"issue":"10","key":"zoi251122r34","doi-asserted-by":"publisher","first-page":"2753","DOI":"10.1007\/s40520-021-01792-4","article-title":"Development and validation of a delirium risk assessment tool in older patients admitted to the emergency department observation unit.","volume":"33","author":"Zucchelli","year":"2021","journal-title":"Aging Clin Exp Res"},{"issue":"5","key":"zoi251122r35","doi-asserted-by":"publisher","first-page":"1229","DOI":"10.1016\/j.mayocp.2020.08.049","article-title":"Predicting delirium risk using an automated Mayo Delirium Prediction Tool: development and validation of a risk-stratification model.","volume":"96","author":"Pagali","year":"2021","journal-title":"Mayo Clin Proc"},{"issue":"2","key":"zoi251122r37","doi-asserted-by":"publisher","first-page":"292","DOI":"10.1067\/mem.2003.274","article-title":"Evidence-based emergency medicine\/skills for evidence-based emergency care\u2014interval likelihood ratios: another advantage for the evidence-based diagnostician.","volume":"42","author":"Brown","year":"2003","journal-title":"Ann Emerg Med"},{"issue":"1","key":"zoi251122r38","doi-asserted-by":"publisher","first-page":"142","DOI":"10.1177\/0962280214541852","article-title":"Does McNemar\u2019s test compare the sensitivities and specificities of two diagnostic tests?","volume":"26","author":"Kim","year":"2017","journal-title":"Stat Methods Med Res"},{"issue":"4","key":"zoi251122r39","doi-asserted-by":"publisher","first-page":"621","DOI":"10.1093\/jamia\/ocz228","article-title":"A tutorial on calibration measurements and calibration models for clinical prediction models.","volume":"27","author":"Huang","year":"2020","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"zoi251122r40","doi-asserted-by":"publisher","first-page":"128","DOI":"10.1097\/EDE.0b013e3181c30fb2","article-title":"Assessing the performance of prediction models: a framework for traditional and novel measures.","volume":"21","author":"Steyerberg","year":"2010","journal-title":"Epidemiology"},{"key":"zoi251122r41","doi-asserted-by":"publisher","first-page":"S99","DOI":"10.1111\/epi.17406","article-title":"Learning to generalize seizure forecasts.","volume":"64","author":"Leguia","year":"2023","journal-title":"Epilepsia"},{"issue":"14","key":"zoi251122r42","doi-asserted-by":"publisher","first-page":"1377","DOI":"10.1001\/jama.2017.12126","article-title":"Discrimination and calibration of clinical prediction models: users\u2019 guides to the medical literature.","volume":"318","author":"Alba","year":"2017","journal-title":"JAMA"},{"key":"zoi251122r43","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpsychores.2019.109850","article-title":"Evaluation of emergency department derived delirium prediction models using a hospital-wide cohort.","volume":"127","author":"Lee","year":"2019","journal-title":"J Psychosom Res"},{"issue":"5","key":"zoi251122r44","doi-asserted-by":"publisher","first-page":"575","DOI":"10.1016\/S0196-0644(99)70346-X","article-title":"Likelihood ratio: a powerful tool for incorporating the results of a diagnostic test into clinical decisionmaking.","volume":"33","author":"Hayden","year":"1999","journal-title":"Ann Emerg Med"},{"key":"zoi251122r45","doi-asserted-by":"publisher","first-page":"62","DOI":"10.1016\/j.jclinepi.2019.12.023","article-title":"GRADE Guidelines 28: use of GRADE for the assessment of evidence about prognostic factors: rating certainty in identification of groups of patients with different absolute risks.","volume":"121","author":"Foroutan","year":"2020","journal-title":"J Clin Epidemiol"},{"issue":"11","key":"zoi251122r46","doi-asserted-by":"publisher","first-page":"1194","DOI":"10.1111\/acem.12255","article-title":"Understanding the direction of bias in studies of diagnostic test accuracy.","volume":"20","author":"Kohn","year":"2013","journal-title":"Acad Emerg Med"},{"issue":"4","key":"zoi251122r47","doi-asserted-by":"publisher","first-page":"826","DOI":"10.5811\/westjem.2020.4.43387","article-title":"Fall prevention knowledge, attitudes, and behaviors: a survey of emergency providers.","volume":"21","author":"Davenport","year":"2020","journal-title":"West J Emerg Med"},{"issue":"6","key":"zoi251122r48","doi-asserted-by":"publisher","first-page":"1838","DOI":"10.1111\/jgs.17740","article-title":"Can we improve delirium prevention and treatment in the emergency department? a systematic review.","volume":"70","author":"Lee","year":"2022","journal-title":"J Am Geriatr Soc"},{"issue":"11","key":"zoi251122r49","doi-asserted-by":"publisher","first-page":"3551","DOI":"10.1111\/jgs.19052","article-title":"A scoping review of geriatric emergency medicine research transparency in diversity, equity, and inclusion reporting.","volume":"72","author":"Chary","year":"2024","journal-title":"J Am Geriatr Soc"},{"issue":"12","key":"zoi251122r50","doi-asserted-by":"publisher","first-page":"941","DOI":"10.7326\/0003-4819-113-12-941","article-title":"Clarifying confusion: the confusion assessment method\u2014a new method for detection of delirium.","volume":"113","author":"Inouye","year":"1990","journal-title":"Ann Intern Med"},{"issue":"8","key":"zoi251122r51","doi-asserted-by":"publisher","first-page":"S30","DOI":"10.1097\/MLR.0b013e31829b1dbd","article-title":"Caveats for the use of operational electronic health record data in comparative effectiveness research.","volume":"51","author":"Hersh","year":"2013","journal-title":"Med Care"},{"key":"zoi251122r36","unstructured":"R: A Language and Environment for Statistical Computing. Version 4.5.0. R Foundation for Statistical Computing; 2025. Accessed February 10, 2025. https:\/\/www.R-project.org\/"}],"container-title":["JAMA Network Open"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/jamanetwork.com\/journals\/jamanetworkopen\/articlepdf\/2840816\/bartolacci_2025_oi_251122_1761576099.84409.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,11,3]],"date-time":"2025-11-03T16:00:59Z","timestamp":1762185659000},"score":1,"resource":{"primary":{"URL":"https:\/\/jamanetwork.com\/journals\/jamanetworkopen\/fullarticle\/2840816"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,11,3]]},"references-count":51,"journal-issue":{"issue":"11","published-print":{"date-parts":[[2025,11,3]]}},"URL":"https:\/\/doi.org\/10.1001\/jamanetworkopen.2025.40920","relation":{},"ISSN":["2574-3805"],"issn-type":[{"value":"2574-3805","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,11,3]]}}}