{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,23]],"date-time":"2026-10-23T05:59:01Z","timestamp":1792735141086,"version":"4.5.0"},"reference-count":49,"publisher":"Oxford University Press (OUP)","issue":"2","license":[{"start":{"date-parts":[[2022,6,10]],"date-time":"2022-06-10T00:00:00Z","timestamp":1654819200000},"content-version":"vor","delay-in-days":65,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,4,6]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Introduction<\/jats:title>\n                  <jats:p>Delirium occurrence is common and preventive strategies are resource intensive. Screening tools can prioritize patients at risk. Using machine learning, we can capture time and treatment effects that pose a challenge to delirium prediction. We aim to develop a delirium prediction model that can be used as a screening tool.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Methods<\/jats:title>\n                  <jats:p>From the eICU Collaborative Research Database (eICU-CRD) and the Medical Information Mart for Intensive Care version III (MIMIC-III) database, patients with one or more Confusion Assessment Method-Intensive Care Unit (CAM-ICU) values and intensive care unit (ICU) length of stay greater than 24\u2009h were included in our study. We validated our model using 21 quantitative clinical parameters and assessed performance across a range of observation and prediction windows, using different thresholds and applied interpretation techniques. We evaluate our models based on stratified repeated cross-validation using 3 algorithms, namely Logistic Regression, Random Forest, and Bidirectional Long Short-Term Memory (BiLSTM). BiLSTM represents an evolution from recurrent neural network-based Long Short-Term Memory, and with a backward input, preserves information from both past and future. Model performance is measured using Area Under Receiver Operating Characteristic, Area Under Precision Recall Curve, Recall, Precision (Positive Predictive Value), and Negative Predictive Value metrics.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>We evaluated our results on 16\u200a546 patients (47% female) and 6294 patients (44% female) from eICU-CRD and MIMIC-III databases, respectively. Performance was best in BiLSTM models where, precision and recall changed from 37.52% (95% confidence interval [CI], 36.00%\u201339.05%) to 17.45 (95% CI, 15.83%\u201319.08%) and 86.1% (95% CI, 82.49%\u201389.71%) to 75.58% (95% CI, 68.33%\u201382.83%), respectively as prediction window increased from 12 to 96\u2009h. After optimizing for higher recall, precision and recall changed from 26.96% (95% CI, 24.99%\u201328.94%) to 11.34% (95% CI, 10.71%\u201311.98%) and 93.73% (95% CI, 93.1%\u201394.37%) to 92.57% (95% CI, 88.19%\u201396.95%), respectively. Comparable results were obtained in the MIMIC-III cohort.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusions<\/jats:title>\n                  <jats:p>Our model performed comparably to contemporary models using fewer variables. Using techniques like sliding windows, modification of threshold to augment recall and feature ranking for interpretability, we addressed shortcomings of current models.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamiaopen\/ooac048","type":"journal-article","created":{"date-parts":[[2022,6,10]],"date-time":"2022-06-10T09:36:33Z","timestamp":1654853793000},"source":"Crossref","is-referenced-by-count":32,"title":["Delirium prediction in the ICU: designing a screening tool for preventive interventions"],"prefix":"10.1093","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1646-709X","authenticated-orcid":false,"given":"Anirban","family":"Bhattacharyya","sequence":"first","affiliation":[{"name":"Critical Care Services, Mayo Clinic , Jacksonville, Florida, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Seyedmostafa","family":"Sheikhalishahi","sequence":"additional","affiliation":[{"name":"Fondazione Bruno Kessler Research Institute , Trento, Italy"},{"name":"University of Trento , Trento, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Heather","family":"Torbic","sequence":"additional","affiliation":[{"name":"Department of Pharmacy, Cleveland Clinic , Cleveland, Ohio, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wesley","family":"Yeung","sequence":"additional","affiliation":[{"name":"National University of Singapore , Singapore, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tiffany","family":"Wang","sequence":"additional","affiliation":[{"name":"Division of Pulmonary, Critical Care and Sleep Medicine, Beth Israel Deaconess Medical Center , Boston, Massachusetts, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jennifer","family":"Birst","sequence":"additional","affiliation":[{"name":"Physical and Occupational Therapy, Mayo Clinic , Jacksonville, Florida, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Abhijit","family":"Duggal","sequence":"additional","affiliation":[{"name":"Respiratory Institute, Cleveland Clinic , Cleveland, Ohio, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Leo Anthony","family":"Celi","sequence":"additional","affiliation":[{"name":"Division of Pulmonary, Critical Care and Sleep Medicine, Beth Israel Deaconess Medical Center , Boston, Massachusetts, USA"},{"name":"Institute of Medical Engineering and Science, Massachusetts Institute of Technology , Boston, Massachusetts, USA"},{"name":"Department of Biostatistics, Harvard T.H. Chan School of Public Health , Boston, Massachusetts, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7306-2972","authenticated-orcid":false,"given":"Venet","family":"Osmani","sequence":"additional","affiliation":[{"name":"Fondazione Bruno Kessler Research Institute , Trento, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2022,6,10]]},"reference":[{"issue":"12","key":"2022061010061040800_ooac048-B1","doi-asserted-by":"crossref","first-page":"2029","DOI":"10.1097\/CCM.0000000000003402","article-title":"Incidence and prevalence of delirium subtypes in an adult ICU","volume":"46","author":"Krewulak","year":"2018","journal-title":"Crit Care Med"},{"issue":"5","key":"2022061010061040800_ooac048-B2","doi-asserted-by":"crossref","first-page":"572","DOI":"10.1016\/j.hrtlng.2020.03.006","article-title":"Time trends of delirium rates in the intensive care unit","volume":"49","author":"Khan","year":"2020","journal-title":"Heart Lung"},{"issue":"5","key":"2022061010061040800_ooac048-B3","doi-asserted-by":"crossref","first-page":"793","DOI":"10.1017\/S1041610214002658","article-title":"Poor functional recovery after delirium is associated with other geriatric syndromes and additional illnesses","volume":"27","author":"Dasgupta","year":"2015","journal-title":"Int Psychogeriatr"},{"issue":"1","key":"2022061010061040800_ooac048-B4","doi-asserted-by":"crossref","first-page":"261","DOI":"10.1186\/s13054-016-1433-z","article-title":"The association of sleep quality, delirium, and sedation status with daily participation in physical therapy in the ICU","volume":"20","author":"Kamdar","year":"2016","journal-title":"Crit Care"},{"issue":"1","key":"2022061010061040800_ooac048-B5","doi-asserted-by":"crossref","first-page":"68","DOI":"10.1016\/j.mayocp.2017.11.005","article-title":"Potentially modifiable risk factors for long-term cognitive impairment after critical illness: a systematic review","volume":"93","author":"Sakusic","year":"2018","journal-title":"Mayo Clin Proc"},{"issue":"1","key":"2022061010061040800_ooac048-B6","doi-asserted-by":"crossref","first-page":"550","DOI":"10.1186\/s12913-018-3345-x","article-title":"A hospital-wide evaluation of delirium prevalence and outcomes in acute care patients\u2014a cohort study","volume":"18","author":"Schubert","year":"2018","journal-title":"BMC Health Serv Res"},{"issue":"14","key":"2022061010061040800_ooac048-B7","doi-asserted-by":"crossref","first-page":"1753","DOI":"10.1001\/jama.291.14.1753","article-title":"Delirium as a predictor of mortality in mechanically ventilated patients in the intensive care unit","volume":"291","author":"Ely","year":"2004","journal-title":"JAMA"},{"issue":"1","key":"2022061010061040800_ooac048-B8","first-page":"34","article-title":"Prevalence and risk factors for development of delirium in surgical and trauma intensive care unit patients","volume":"65","author":"Pandharipande","year":"2008","journal-title":"J Trauma"},{"issue":"2","key":"2022061010061040800_ooac048-B9","doi-asserted-by":"crossref","first-page":"405","DOI":"10.1097\/CCM.0b013e31826ab49b","article-title":"Delirium in the cardiovascular ICU","volume":"41","author":"McPherson","year":"2013","journal-title":"Crit Care Med"},{"issue":"14","key":"2022061010061040800_ooac048-B10","doi-asserted-by":"crossref","first-page":"1306","DOI":"10.1056\/NEJMoa1301372","article-title":"Long-term cognitive impairment after critical illness","volume":"369","author":"Pandharipande","year":"2013","journal-title":"N Engl J Med"},{"issue":"4","key":"2022061010061040800_ooac048-B11","doi-asserted-by":"crossref","first-page":"210","DOI":"10.1038\/nrneurol.2009.24","article-title":"Delirium in elderly adults: diagnosis, prevention and treatment","volume":"5","author":"Fong","year":"2009","journal-title":"Nat Rev Neurol"},{"issue":"2","key":"2022061010061040800_ooac048-B12","doi-asserted-by":"crossref","first-page":"225","DOI":"10.1016\/j.ccc.2016.12.005","article-title":"The ABCDEF bundle in critical care","volume":"33","author":"Marra","year":"2017","journal-title":"Crit Care Clin"},{"issue":"9","key":"2022061010061040800_ooac048-B13","doi-asserted-by":"crossref","first-page":"e825\u201373","DOI":"10.1097\/CCM.0000000000003259","article-title":"Clinical practice guidelines for the prevention and management of pain, agitation\/sedation, delirium, immobility, and sleep disruption in adult patients in the ICU","volume":"46","author":"Devlin","year":"2018","journal-title":"Crit Care Med"},{"key":"2022061010061040800_ooac048-B14","doi-asserted-by":"crossref","first-page":"885","DOI":"10.1097\/CCM.0000000000003765","article-title":"Staged implementation of awakening and breathing, coordination, delirium monitoring and management, and early mobilization bundle improves patient outcomes and reduces hospital costs","volume":"47","author":"Hsieh","year":"2019","journal-title":"Crit Care Med"},{"key":"2022061010061040800_ooac048-B15","doi-asserted-by":"crossref","first-page":"1370","DOI":"10.1097\/00003246-200107000-00012","article-title":"Evaluation of delirium in critically ill patients: validation of the confusion assessment method for the intensive care unit (CAM-ICU)","volume":"29","author":"Ely","year":"2001","journal-title":"Crit Care Med"},{"key":"2022061010061040800_ooac048-B16","doi-asserted-by":"crossref","first-page":"S3","DOI":"10.1186\/cc6149","article-title":"Delirium in the intensive care unit","volume":"12 (Suppl 3)","author":"Girard","year":"2008","journal-title":"Crit Care"},{"issue":"9","key":"2022061010061040800_ooac048-B17","doi-asserted-by":"crossref","first-page":"2196","DOI":"10.1097\/CCM.0b013e31829a6f1e","article-title":"Implementing delirium screening in the ICU: secrets to success","volume":"41","author":"Brummel","year":"2013","journal-title":"Crit Care Med"},{"issue":"3","key":"2022061010061040800_ooac048-B18","doi-asserted-by":"crossref","first-page":"353","DOI":"10.1097\/CCM.0000000000004139","article-title":"Biomarkers of delirium duration and delirium severity in the ICU","volume":"48","author":"Khan","year":"2020","journal-title":"Crit Care Med"},{"issue":"12","key":"2022061010061040800_ooac048-B19","doi-asserted-by":"crossref","first-page":"e0296","DOI":"10.1097\/CCE.0000000000000296","article-title":"ICU delirium-prediction models: a systematic review","volume":"2","author":"Ruppert","year":"2020","journal-title":"Crit Care Explor"},{"issue":"4","key":"2022061010061040800_ooac048-B20","doi-asserted-by":"crossref","first-page":"e019223","DOI":"10.1136\/bmjopen-2017-019223","article-title":"Systematic review of prediction models for delirium in the older adult inpatient","volume":"8","author":"Lindroth","year":"2018","journal-title":"Bmj Open"},{"issue":"4","key":"2022061010061040800_ooac048-B21","doi-asserted-by":"crossref","first-page":"e181018","DOI":"10.1001\/jamanetworkopen.2018.1018","article-title":"Development and validation of an electronic health record\u2013based machine learning model to estimate delirium risk in newly hospitalized patients without known cognitive impairment","volume":"1","author":"Wong","year":"2018","journal-title":"JAMA Netw Open"},{"key":"2022061010061040800_ooac048-B22","doi-asserted-by":"crossref","first-page":"180178","DOI":"10.1038\/sdata.2018.178","article-title":"The eICU Collaborative Research Database, a freely available multi-center database for critical care research","volume":"5","author":"Pollard","year":"2018","journal-title":"Sci Data"},{"key":"2022061010061040800_ooac048-B23","doi-asserted-by":"crossref","first-page":"160035","DOI":"10.1038\/sdata.2016.35","article-title":"MIMIC-III, a freely accessible critical care database","volume":"3","author":"Johnson","year":"2016","journal-title":"Sci Data"},{"issue":"6","key":"2022061010061040800_ooac048-B24","doi-asserted-by":"crossref","first-page":"474","DOI":"10.7326\/0003-4819-119-6-199309150-00005","article-title":"A predictive model for delirium in hospitalized elderly medical patients based on admission characteristics","volume":"119","author":"Inouye","year":"1993","journal-title":"Ann Intern Med"},{"issue":"8","key":"2022061010061040800_ooac048-B25","doi-asserted-by":"crossref","first-page":"809","DOI":"10.1111\/j.1532-5415.1994.tb06551.x","article-title":"Delirium in hospitalized older persons: outcomes and predictors","volume":"42","author":"Pompei","year":"1994","journal-title":"J Am Geriatr Soc"},{"issue":"12","key":"2022061010061040800_ooac048-B26","doi-asserted-by":"crossref","first-page":"e3072","DOI":"10.1097\/MD.0000000000003072","article-title":"DELirium prediction based on hospital information (Delphi) in general surgery patients","volume":"95","author":"Kim","year":"2016","journal-title":"Medicine (Baltimore)"},{"key":"2022061010061040800_ooac048-B27","first-page":"226","article-title":"Delirium risk stratification in consecutive unselected admissions to acute medicine: validation of a susceptibility score based on factors identified externally in pooled data for use at entry to the acute care pathway","volume":"46","author":"Pendlebury","year":"2016","journal-title":"Age Ageing"},{"issue":"2","key":"2022061010061040800_ooac048-B28","doi-asserted-by":"crossref","first-page":"229","DOI":"10.1161\/CIRCULATIONAHA.108.795260","article-title":"Derivation and validation of a preoperative prediction rule for delirium after cardiac surgery","volume":"119","author":"Rudolph","year":"2009","journal-title":"Circulation"},{"issue":"3","key":"2022061010061040800_ooac048-B29","doi-asserted-by":"crossref","first-page":"346","DOI":"10.1093\/ageing\/aft141","article-title":"Development and validation of a delirium predictive score in older people","volume":"43","author":"Carrasco","year":"2014","journal-title":"Age Ageing"},{"issue":"10","key":"2022061010061040800_ooac048-B30","doi-asserted-by":"crossref","first-page":"946","DOI":"10.1016\/j.jagp.2013.01.069","article-title":"Does preoperative risk for delirium moderate the effects of postoperative pain and opiate use on postoperative delirium?","volume":"21","author":"Leung","year":"2013","journal-title":"Am J Geriatr Psychiatry"},{"issue":"4","key":"2022061010061040800_ooac048-B31","doi-asserted-by":"crossref","first-page":"317","DOI":"10.1093\/ageing\/25.4.317","article-title":"Predicting delirium in elderly patients: development and validation of a risk-stratification model","volume":"25","author":"O'Keeffe","year":"1996","journal-title":"Age Ageing"},{"issue":"5","key":"2022061010061040800_ooac048-B32","doi-asserted-by":"crossref","first-page":"817","DOI":"10.1111\/j.1532-5415.2006.00704.x","article-title":"Risk factors and prediction of postoperative delirium in elderly hip-surgery patients: implementation and validation of a medical risk factor model","volume":"54","author":"Kalisvaart","year":"2006","journal-title":"J Am Geriatr Soc"},{"issue":"6","key":"2022061010061040800_ooac048-B33","doi-asserted-by":"crossref","first-page":"1082","DOI":"10.5535\/arm.2016.40.6.1082","article-title":"Risk factors for delirium during acute and subacute stages of various disorders in patients admitted to rehabilitation units","volume":"40","author":"Jang","year":"2016","journal-title":"Ann Rehabil Med"},{"issue":"3","key":"2022061010061040800_ooac048-B34","doi-asserted-by":"crossref","first-page":"244","DOI":"10.1016\/j.jamda.2015.10.020","article-title":"Validation of a delirium risk assessment using electronic medical record information","volume":"17","author":"Rudolph","year":"2016","journal-title":"J Am Med Dir Assoc"},{"key":"2022061010061040800_ooac048-B35","doi-asserted-by":"crossref","first-page":"S289","DOI":"10.1111\/j.1532-5415.2011.03677.x","article-title":"Validation of a medical record-based delirium risk assessment","volume":"59 (Suppl 2)","author":"Rudolph","year":"2011","journal-title":"J Am Geriatr Soc"},{"key":"2022061010061040800_ooac048-B36","first-page":"2079","article-title":"On over-fitting in model selection and subsequent selection bias in performance evaluation","volume":"11","author":"Cawley","year":"2010","journal-title":"J Mach Learn Res"},{"issue":"8","key":"2022061010061040800_ooac048-B37","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput"},{"issue":"11","key":"2022061010061040800_ooac048-B38","doi-asserted-by":"crossref","first-page":"2673","DOI":"10.1109\/78.650093","article-title":"Bidirectional recurrent neural networks","volume":"45","author":"Schuster","year":"1997","journal-title":"IEEE Trans Signal Process"},{"issue":"6","key":"2022061010061040800_ooac048-B39","doi-asserted-by":"crossref","first-page":"1236","DOI":"10.1093\/bib\/bbx044","article-title":"Deep learning for healthcare: review, opportunities and challenges","volume":"19","author":"Miotto","year":"2018","journal-title":"Brief Bioinform"},{"issue":"5","key":"2022061010061040800_ooac048-B40","doi-asserted-by":"crossref","first-page":"1726","DOI":"10.1016\/j.cor.2008.04.004","article-title":"Polynomial calculation of the Shapley value based on sampling","volume":"36","author":"Castro","year":"2009","journal-title":"Comput Oper Res"},{"key":"2022061010061040800_ooac048-B41","first-page":"1","article-title":"An efficient explanation of individual classifications using game theory","volume":"11","author":"Strumbelj","year":"2010","journal-title":"J Mach Learn Res"},{"key":"2022061010061040800_ooac048-B42","first-page":"6441","author":"Ismail","year":"2020"},{"key":"2022061010061040800_ooac048-B43","first-page":"6441","author":"Sundararajan","year":"2017"},{"key":"2022061010061040800_ooac048-B44","author":"Springenberg","year":"2014"},{"issue":"4","key":"2022061010061040800_ooac048-B45","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3343440","article-title":"A systematic review on imbalanced data challenges in machine learning","volume":"52","author":"Kaur","year":"2020","journal-title":"ACM Comput Surv"},{"issue":"9","key":"2022061010061040800_ooac048-B46","doi-asserted-by":"crossref","first-page":"e489","DOI":"10.1016\/S2589-7500(20)30186-2","article-title":"The myth of generalisability in clinical research and machine learning in health care","volume":"2","author":"Futoma","year":"2020","journal-title":"Lancet Digital Heal"},{"key":"2022061010061040800_ooac048-B47","doi-asserted-by":"crossref","first-page":"e420","DOI":"10.1136\/bmj.e420","article-title":"Development and validation of PRE-DELIRIC (PREdiction of DELIRium in ICu patients) delirium prediction model for intensive care patients: observational multicentre study","volume":"344","author":"van den Boogaard","year":"2012","journal-title":"BMJ"},{"issue":"6","key":"2022061010061040800_ooac048-B48","doi-asserted-by":"crossref","first-page":"1048","DOI":"10.1007\/s00134-015-3777-2","article-title":"Multinational development and validation of an early prediction model for delirium in ICU patients","volume":"41","author":"Wassenaar","year":"2015","journal-title":"Intensive Care Med"},{"issue":"4","key":"2022061010061040800_ooac048-B49","doi-asserted-by":"crossref","first-page":"e181404","DOI":"10.1001\/jamanetworkopen.2018.1404","article-title":"Machine learning for prediction in electronic health data","volume":"1","author":"Rose","year":"2018","journal-title":"JAMA Netw Open"}],"container-title":["JAMIA Open"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamiaopen\/article-pdf\/5\/2\/ooac048\/44002754\/ooac048.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamiaopen\/article-pdf\/5\/2\/ooac048\/44002754\/ooac048.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,6,10]],"date-time":"2022-06-10T10:07:09Z","timestamp":1654855629000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamiaopen\/article\/doi\/10.1093\/jamiaopen\/ooac048\/6605167"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,4,6]]},"references-count":49,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2022,4,6]]}},"URL":"https:\/\/doi.org\/10.1093\/jamiaopen\/ooac048","relation":{},"ISSN":["2574-2531"],"issn-type":[{"value":"2574-2531","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022,7,1]]},"published":{"date-parts":[[2022,4,6]]},"article-number":"ooac048"}}