{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,25]],"date-time":"2026-06-25T17:29:18Z","timestamp":1782408558688,"version":"3.54.5"},"reference-count":49,"publisher":"Oxford University Press (OUP)","issue":"1","license":[{"start":{"date-parts":[[2022,10,28]],"date-time":"2022-10-28T00:00:00Z","timestamp":1666915200000},"content-version":"vor","delay-in-days":1,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["R01AG062499-01"],"award-info":[{"award-number":["R01AG062499-01"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["K99LM014097-01"],"award-info":[{"award-number":["K99LM014097-01"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,12,13]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objective<\/jats:title>\n                  <jats:p>To develop and test an accurate deep learning model for predicting new onset delirium in hospitalized adult patients.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Methods<\/jats:title>\n                  <jats:p>Using electronic health record (EHR) data extracted from a large academic medical center, we developed a model combining long short-term memory (LSTM) and machine learning to predict new onset delirium and compared its performance with machine-learning-only models (logistic regression, random forest, support vector machine, neural network, and LightGBM). The labels of models were confusion assessment method (CAM) assessments. We evaluated models on a hold-out dataset. We calculated Shapley additive explanations (SHAP) measures to gauge the feature impact on the model.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>A total of 331\u00a0489 CAM assessments with 896 features from 34\u00a0035 patients were included. The LightGBM model achieved the best performance (AUC 0.927 [0.924, 0.929] and F1 0.626 [0.618, 0.634]) among the machine learning models. When combined with the LSTM model, the final model\u2019s performance improved significantly (P\u2009=\u2009.001) with AUC 0.952 [0.950, 0.955] and F1 0.759 [0.755, 0.765]. The precision value of the combined model improved from 0.497 to 0.751 with a fixed recall of 0.8. Using the mean absolute SHAP values, we identified the top 20 features, including age, heart rate, Richmond Agitation-Sedation Scale score, Morse fall risk score, pulse, respiratory rate, and level of care.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>Leveraging LSTM to capture temporal trends and combining it with the LightGBM model can significantly improve the prediction of new onset delirium, providing an algorithmic basis for the subsequent development of clinical decision support tools for proactive delirium interventions.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocac210","type":"journal-article","created":{"date-parts":[[2022,10,28]],"date-time":"2022-10-28T09:06:56Z","timestamp":1666948016000},"page":"120-131","source":"Crossref","is-referenced-by-count":34,"title":["New onset delirium prediction using machine learning and long short-term memory (LSTM) in electronic health record"],"prefix":"10.1093","volume":"30","author":[{"given":"Siru","family":"Liu","sequence":"first","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, Tennessee, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Joseph J","family":"Schlesinger","sequence":"additional","affiliation":[{"name":"Division of Critical Care Medicine, Department of Anesthesiology, Vanderbilt University Medical Center , Nashville, Tennessee, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2292-9147","authenticated-orcid":false,"given":"Allison B","family":"McCoy","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, Tennessee, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1081-1670","authenticated-orcid":false,"given":"Thomas J","family":"Reese","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, Tennessee, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bryan","family":"Steitz","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, Tennessee, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Elise","family":"Russo","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, Tennessee, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Brian","family":"Koh","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, Tennessee, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6844-145X","authenticated-orcid":false,"given":"Adam","family":"Wright","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, Tennessee, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2022,10,27]]},"reference":[{"issue":"9920","key":"2022121408281564500_ocac210-B1","doi-asserted-by":"crossref","first-page":"911","DOI":"10.1016\/S0140-6736(13)60688-1","article-title":"Delirium in elderly people","volume":"383","author":"Inouye","year":"2014","journal-title":"Lancet"},{"issue":"8","key":"2022121408281564500_ocac210-B2","doi-asserted-by":"crossref","first-page":"759","DOI":"10.1111\/j.1532-5415.1992.tb01846.x","article-title":"Post-operative delirium: predictors and prognosis in elderly orthopedic patients","volume":"40","author":"Williams-Russo","year":"1992","journal-title":"J Am Geriatr Soc"},{"issue":"5","key":"2022121408281564500_ocac210-B3","doi-asserted-by":"crossref","first-page":"678","DOI":"10.1046\/j.1532-5415.2001.49133.x","article-title":"Delirium after hip fracture: to be or not to be?","volume":"49","author":"Inouye","year":"2001","journal-title":"J Am Geriatr Soc"},{"issue":"3","key":"2022121408281564500_ocac210-B4","doi-asserted-by":"crossref","first-page":"278","DOI":"10.1016\/0002-9343(94)90011-6","article-title":"The dilemma of delirium: clinical and research controversies regarding diagnosis and evaluation of delirium in hospitalized elderly medical patients","volume":"97","author":"Inouye","year":"1994","journal-title":"Am J Med"},{"issue":"4","key":"2022121408281564500_ocac210-B5","doi-asserted-by":"crossref","first-page":"234","DOI":"10.1046\/j.1525-1497.1998.00073.x","article-title":"Does delirium contribute to poor hospital outcomes?","volume":"13","author":"Inouye","year":"1998","journal-title":"J Gen Intern Med"},{"issue":"2","key":"2022121408281564500_ocac210-B6","doi-asserted-by":"crossref","first-page":"174","DOI":"10.1111\/j.1532-5415.1997.tb04503.x","article-title":"The prognostic significance of delirium in older hospital patients","volume":"45","author":"O'Keeffe","year":"1997","journal-title":"J Am Geriatr Soc"},{"issue":"6","key":"2022121408281564500_ocac210-B7","doi-asserted-by":"crossref","first-page":"601","DOI":"10.1111\/j.1532-5415.1992.tb02111.x","article-title":"Prognosis after hospital discharge of older medical patients with delirium","volume":"40","author":"Francis","year":"1992","journal-title":"J Am Geriatr Soc"},{"issue":"3","key":"2022121408281564500_ocac210-B8","doi-asserted-by":"crossref","first-page":"234","DOI":"10.1176\/appi.psy.50.3.234","article-title":"Impact of delirium on short-term mortality in elderly inpatients: a prospective cohort study","volume":"50","author":"Gonz\u00e1lez","year":"2009","journal-title":"Psychosomatics"},{"issue":"12","key":"2022121408281564500_ocac210-B9","doi-asserted-by":"crossref","first-page":"848","DOI":"10.7326\/0003-4819-156-12-201206190-00005","article-title":"Adverse outcomes after hospitalization and delirium in persons with Alzheimer disease","volume":"156","author":"Fong","year":"2012","journal-title":"Ann Intern Med"},{"issue":"1","key":"2022121408281564500_ocac210-B10","doi-asserted-by":"crossref","first-page":"R28","DOI":"10.1186\/cc5708","article-title":"Risk factors for post-traumatic stress disorder symptoms following critical illness requiring mechanical ventilation: a prospective cohort study","volume":"11","author":"Girard","year":"2007","journal-title":"Crit Care"},{"issue":"8","key":"2022121408281564500_ocac210-B11","doi-asserted-by":"crossref","first-page":"823","DOI":"10.1016\/S1474-4422(15)00101-5","article-title":"The interface between delirium and dementia in elderly adults","volume":"14","author":"Fong","year":"2015","journal-title":"Lancet Neurol"},{"issue":"1","key":"2022121408281564500_ocac210-B12","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1016\/j.psc.2014.11.002","article-title":"Cognitive functioning, mental health, and quality of life in ICU survivors: an overview","volume":"38","author":"Jackson","year":"2015","journal-title":"Psychiatr Clin North Am"},{"issue":"1","key":"2022121408281564500_ocac210-B13","doi-asserted-by":"crossref","first-page":"112","DOI":"10.1097\/CCM.0b013e31822e9fc9","article-title":"Delirium in critically ill patients: impact on long-term health-related quality of life and cognitive functioning","volume":"40","author":"van den Boogaard","year":"2012","journal-title":"Crit Care Med"},{"key":"2022121408281564500_ocac210-B14","first-page":"1","article-title":"Delirium","volume":"6","author":"Wilson","year":"2020","journal-title":"Nat Rev Dis Prim"},{"issue":"1","key":"2022121408281564500_ocac210-B15","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1097\/NUR.0000000000000098","article-title":"Implementing clinical practice guidelines for screening and detection of delirium in a 21-hospital system in Northern California","volume":"29","author":"Adams","year":"2015","journal-title":"Clin Nurse Spec"},{"issue":"5","key":"2022121408281564500_ocac210-B16","doi-asserted-by":"crossref","first-page":"516","DOI":"10.1046\/j.1532-5415.2001.49108.x","article-title":"Reducing delirium after hip fracture: a randomized trial","volume":"49","author":"Marcantonio","year":"2001","journal-title":"J Am Geriatr Soc"},{"issue":"10","key":"2022121408281564500_ocac210-B17","doi-asserted-by":"crossref","first-page":"1089","DOI":"10.1007\/s00134-021-06503-1","article-title":"Delirium in critical illness: clinical manifestations, outcomes, and management","volume":"47","author":"Stollings","year":"2021","journal-title":"Intensive Care Med"},{"issue":"2","key":"2022121408281564500_ocac210-B18","doi-asserted-by":"crossref","first-page":"160","DOI":"10.5603\/AIT.a2018.0011","article-title":"ICU delirium\u2014a diagnostic and therapeutic challenge in the intensive care unit","volume":"50","author":"Kotfis","year":"2018","journal-title":"Anaesthesiol Intensive Ther"},{"issue":"1","key":"2022121408281564500_ocac210-B19","doi-asserted-by":"crossref","first-page":"38","DOI":"10.4037\/ccn2018633","article-title":"Delirium assessment tools for use in critically ill adults: a psychometric analysis and systematic review","volume":"38","author":"G\u00e9linas","year":"2018","journal-title":"Crit Care Nurse"},{"issue":"6","key":"2022121408281564500_ocac210-B20","doi-asserted-by":"crossref","first-page":"555","DOI":"10.4037\/ajcc2008.17.6.555","article-title":"Assessment of delirium in the intensive care unit: nursing practices and perceptions","volume":"17","author":"Devlin","year":"2008","journal-title":"Am J Crit Care"},{"issue":"1","key":"2022121408281564500_ocac210-B21","doi-asserted-by":"crossref","first-page":"e12\u201320","DOI":"10.4037\/ajcc2012735","article-title":"Implications of objective vs subjective delirium assessment in surgical intensive care patients","volume":"21","author":"Guenther","year":"2012","journal-title":"Am J Crit Care"},{"issue":"10","key":"2022121408281564500_ocac210-B22","doi-asserted-by":"crossref","first-page":"1658","DOI":"10.1111\/j.1532-5415.2005.53503.x","article-title":"Haloperidol prophylaxis for elderly hip-surgery patients at risk for delirium: a randomized placebo-controlled study","volume":"53","author":"Kalisvaart","year":"2005","journal-title":"J Am Geriatr Soc"},{"issue":"3","key":"2022121408281564500_ocac210-B23","doi-asserted-by":"crossref","first-page":"391","DOI":"10.1093\/bja\/aew476","article-title":"Risk prediction models for delirium in the intensive care unit after cardiac surgery: a systematic review and independent external validation","volume":"118","author":"Lee","year":"2017","journal-title":"Br J Anaesth"},{"key":"2022121408281564500_ocac210-B24","first-page":"1","article-title":"Predicting brain function status changes in critically ill patients via machine learning","volume":"2021","author":"Yan","year":"2021","journal-title":"J Am Med Informatics Assoc"},{"issue":"9","key":"2022121408281564500_ocac210-B25","doi-asserted-by":"crossref","first-page":"1383","DOI":"10.1093\/jamia\/ocaa113","article-title":"Risk prediction of delirium in hospitalized patients using machine learning: an implementation and prospective evaluation study","volume":"27","author":"Jauk","year":"2020","journal-title":"J Am Med Inform Assoc"},{"issue":"8","key":"2022121408281564500_ocac210-B26","doi-asserted-by":"crossref","first-page":"945","DOI":"10.1002\/pds.4226","article-title":"Evaluation of algorithms to identify delirium in administrative claims and drug utilization database","volume":"26","author":"Kim","year":"2017","journal-title":"Pharmacoepidemiol Drug Saf"},{"issue":"12","key":"2022121408281564500_ocac210-B27","doi-asserted-by":"crossref","first-page":"941","DOI":"10.7326\/0003-4819-113-12-941","article-title":"Clarifying confusion: the confusion assessment method. A new method for detection of delirium","volume":"113","author":"Inouye","year":"1990","journal-title":"Ann Intern Med"},{"key":"2022121408281564500_ocac210-B28","author":"Healthcare Cost and Utilization Project. Clinical Classifications Software Refined (CCSR)","year":"2021"},{"key":"2022121408281564500_ocac210-B29","first-page":"3147","article-title":"LightGBM: a highly efficient gradient boosting decision tree","author":"Ke","year":"2017","journal-title":"Advances in Neural Information Processing Systems"},{"issue":"2","key":"2022121408281564500_ocac210-B30","doi-asserted-by":"crossref","first-page":"e0246306","DOI":"10.1371\/journal.pone.0246306","article-title":"Predicting mortality of patients with acute kidney injury in the ICU using XGBoost model","volume":"16","author":"Liu","year":"2021","journal-title":"PLoS One"},{"issue":"6","key":"2022121408281564500_ocac210-B31","doi-asserted-by":"crossref","first-page":"e36958","DOI":"10.2196\/36958","article-title":"Predicting risk of hypoglycemia in patients with type 2 diabetes by electronic health record-based machine learning: development and validation","volume":"10","author":"Yang","year":"2022","journal-title":"JMIR Med Inform"},{"issue":"8","key":"2022121408281564500_ocac210-B32","doi-asserted-by":"crossref","first-page":"e38082","DOI":"10.2196\/38082","article-title":"Predicting mortality in intensive care unit patients with heart failure using an interpretable machine learning model: retrospective cohort study","volume":"24","author":"Li","year":"2022","journal-title":"J Med Internet Res"},{"issue":"1","key":"2022121408281564500_ocac210-B33","doi-asserted-by":"crossref","first-page":"86","DOI":"10.1214\/aoms\/1177731944","article-title":"A comparison of alternative tests of significance for the problem of m rankings","volume":"11","author":"Friedman","year":"1940","journal-title":"Ann Math Statist"},{"key":"2022121408281564500_ocac210-B34","author":"Nemenyi","year":"1963"},{"key":"2022121408281564500_ocac210-B35","first-page":"1","article-title":"Statistical comparisons of classifiers over multiple data sets","volume":"7","author":"Demsar","year":"2006","journal-title":"J Mach Learn Res"},{"key":"2022121408281564500_ocac210-B36","first-page":"4766","author":"Lundberg","year":"2017"},{"issue":"8","key":"2022121408281564500_ocac210-B37","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput"},{"key":"2022121408281564500_ocac210-B38","first-page":"1","article-title":"Hyperband: a novel bandit-based approach to hyperparameter optimization","volume":"18","author":"Li","year":"2016","journal-title":"J Mach Learn Res"},{"issue":"2\u20133","key":"2022121408281564500_ocac210-B39","doi-asserted-by":"crossref","first-page":"177","DOI":"10.1016\/S0197-0186(02)00040-2","article-title":"Interorgan ammonia metabolism in liver failure","volume":"41","author":"Olde Damink","year":"2002","journal-title":"Neurochem Int"},{"issue":"4","key":"2022121408281564500_ocac210-B40","doi-asserted-by":"crossref","first-page":"341","DOI":"10.1017\/S1478951512000600","article-title":"Assessment and management of delirium: a focus on hepatic encephalopathy","volume":"11","author":"Coggins","year":"2013","journal-title":"Palliat Support Care"},{"issue":"1","key":"2022121408281564500_ocac210-B41","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1186\/s12871-019-0694-x","article-title":"Postoperative delirium in critically ill surgical patients: incidence, risk factors, and predictive scores","volume":"19","author":"Chaiwat","year":"2019","journal-title":"BMC Anesthesiol"},{"issue":"2","key":"2022121408281564500_ocac210-B42","doi-asserted-by":"crossref","first-page":"129","DOI":"10.1017\/S1478951510000672","article-title":"Phenomenology of the subtypes of delirium: phenomenological differences between hyperactive and hypoactive delirium","volume":"9","author":"Boettger","year":"2011","journal-title":"Palliat Support Care"},{"issue":"3","key":"2022121408281564500_ocac210-B43","doi-asserted-by":"crossref","first-page":"236","DOI":"10.1016\/j.jpsychores.2011.11.013","article-title":"A longitudinal study of motor subtypes in delirium: frequency and stability during episodes","volume":"72","author":"Meagher","year":"2012","journal-title":"J Psychosom Res"},{"issue":"5","key":"2022121408281564500_ocac210-B44","doi-asserted-by":"crossref","first-page":"970","DOI":"10.1111\/jgs.13334","article-title":"Stability of postoperative delirium psychomotor subtypes in individuals with hip fracture","volume":"63","author":"Albrecht","year":"2015","journal-title":"J Am Geriatr Soc"},{"issue":"6","key":"2022121408281564500_ocac210-B45","doi-asserted-by":"crossref","first-page":"e480","DOI":"10.1097\/CCM.0000000000004313","article-title":"Association of hypoactive and hyperactive delirium with cognitive function after critical illness","volume":"48","author":"Hayhurst","year":"2020","journal-title":"Crit Care Med"},{"issue":"20","key":"2022121408281564500_ocac210-B46","doi-asserted-by":"crossref","first-page":"2467","DOI":"10.1001\/archinte.161.20.2467","article-title":"Nurses\u2019 recognition of delirium and its symptoms","volume":"161","author":"Inouye","year":"2001","journal-title":"Arch Intern Med"},{"issue":"6","key":"2022121408281564500_ocac210-B47","doi-asserted-by":"crossref","first-page":"299","DOI":"10.1097\/NUR.0b013e318234897b","article-title":"Nurses\u2019 recognition of delirium in the hospitalized older adult","volume":"25","author":"Rice","year":"2011","journal-title":"Clin Nurse Spec"},{"key":"2022121408281564500_ocac210-B48","doi-asserted-by":"publisher","author":"Djerioui","year":"2020","DOI":"10.1109\/ICEE49691.2020.9249935"},{"issue":"2","key":"2022121408281564500_ocac210-B49","doi-asserted-by":"crossref","first-page":"ooac048","DOI":"10.1093\/jamiaopen\/ooac048","article-title":"Delirium prediction in the ICU: designing a screening tool for preventive interventions","volume":"5","author":"Bhattacharyya","year":"2022","journal-title":"JAMIA Open"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/30\/1\/120\/47830100\/ocac210.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/30\/1\/120\/47830100\/ocac210.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,14]],"date-time":"2022-12-14T11:51:35Z","timestamp":1671018695000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/30\/1\/120\/6775898"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,10,27]]},"references-count":49,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2022,10,27]]},"published-print":{"date-parts":[[2022,12,13]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocac210","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023,1,1]]},"published":{"date-parts":[[2022,10,27]]}}}