{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,18]],"date-time":"2026-01-18T10:52:37Z","timestamp":1768733557257,"version":"3.49.0"},"reference-count":53,"publisher":"Oxford University Press (OUP)","issue":"9","license":[{"start":{"date-parts":[[2021,7,16]],"date-time":"2021-07-16T00:00:00Z","timestamp":1626393600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"funder":[{"name":"National Institute for Nursing Research (NINR) funded CONCERN Study","award":["1R01NR016941"],"award-info":[{"award-number":["1R01NR016941"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2021,8,13]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Objective<\/jats:title><jats:p>To propose an algorithm that utilizes only timestamps of longitudinal electronic health record data to classify clinical deterioration events.<\/jats:p><\/jats:sec><jats:sec><jats:title>Materials and methods<\/jats:title><jats:p>This retrospective study explores the efficacy of machine learning algorithms in classifying clinical deterioration events among patients in intensive care units using sequences of timestamps of vital sign measurements, flowsheets comments, order entries, and nursing notes. We design a data pipeline to partition events into discrete, regular time bins that we refer to as timesteps. Logistic regressions, random forest classifiers, and recurrent neural networks are trained on datasets of different length of timesteps, respectively, against a composite outcome of death, cardiac arrest, and Rapid Response Team calls. Then these models are validated on a holdout dataset.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>A total of 6720 intensive care unit encounters meet the criteria and the final dataset includes 830 578 timestamps. The gated recurrent unit model utilizes timestamps of vital signs, order entries, flowsheet comments, and nursing notes to achieve the best performance on the time-to-outcome dataset, with an area under the precision-recall curve of 0.101 (0.06, 0.137), a sensitivity of 0.443, and a positive predictive value of 0. 092 at the threshold of 0.6.<\/jats:p><\/jats:sec><jats:sec><jats:title>Discussion and Conclusion<\/jats:title><jats:p>This study demonstrates that our recurrent neural network models using only timestamps of longitudinal electronic health record data that reflect healthcare processes achieve well-performing discriminative power.<\/jats:p><\/jats:sec>","DOI":"10.1093\/jamia\/ocab111","type":"journal-article","created":{"date-parts":[[2021,5,19]],"date-time":"2021-05-19T19:28:36Z","timestamp":1621452516000},"page":"1955-1963","source":"Crossref","is-referenced-by-count":12,"title":["Utilizing timestamps of longitudinal electronic health record data to classify clinical deterioration events"],"prefix":"10.1093","volume":"28","author":[{"given":"Li-Heng","family":"Fu","sequence":"first","affiliation":[{"name":"Department of Biomedical Informatics, Columbia University, New York, New York, USA"}]},{"given":"Chris","family":"Knaplund","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Columbia University, New York, New York, USA"}]},{"given":"Kenrick","family":"Cato","sequence":"additional","affiliation":[{"name":"School of Nursing, Columbia University, New York, New York, USA"}]},{"given":"Adler","family":"Perotte","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Columbia University, New York, New York, USA"}]},{"given":"Min-Jeoung","family":"Kang","sequence":"additional","affiliation":[{"name":"The Catholic University of Korea, College of Nursing, Seoul, Republic of Korea"}]},{"given":"Patricia C","family":"Dykes","sequence":"additional","affiliation":[{"name":"Division of General Internal Medicine and Primary Care, Brigham and Women\u2019s Hospital, Boston, Massachusetts, USA"},{"name":"Harvard Medical School, Boston, Massachusetts, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5369-526X","authenticated-orcid":false,"given":"David","family":"Albers","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Columbia University, New York, New York, USA"},{"name":"Department of Pediatrics, Section of Informatics and Data Science, University of Colorado, Aurora, Colorado, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2632-8867","authenticated-orcid":false,"given":"Sarah","family":"Collins Rossetti","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Columbia University, New York, New York, USA"},{"name":"School of Nursing, Columbia University, New York, New York, USA"}]}],"member":"286","published-online":{"date-parts":[[2021,7,16]]},"reference":[{"issue":"7148","key":"2021081407012218600_ocab111-B1","doi-asserted-by":"crossref","first-page":"1853","DOI":"10.1136\/bmj.316.7148.1853","article-title":"Confidential inquiry into quality of care before admission to intensive care. BMJ","volume":"316","author":"McQuillan","year":"1998","journal-title":"Clin Res Ed"},{"issue":"1","key":"2021081407012218600_ocab111-B2","doi-asserted-by":"crossref","first-page":"43","DOI":"10.7748\/ns.23.1.43.s55","article-title":"Deterioration in hospital patients: early signs and appropriate actions","volume":"23","author":"Beaumont","year":"2008","journal-title":"Nurs Stand"},{"issue":"11","key":"2021081407012218600_ocab111-B3","doi-asserted-by":"crossref","first-page":"1629","DOI":"10.1007\/s00134-002-1496-y","article-title":"Duration of life-threatening antecedents prior to intensive care admission","volume":"28","author":"Hillman","year":"2002","journal-title":"Intensive Care Med"},{"issue":"2","key":"2021081407012218600_ocab111-B4","doi-asserted-by":"crossref","first-page":"68","DOI":"10.1002\/jhm.812","article-title":"Unplanned transfers to a medical intensive care unit: causes and relationship to preventable errors in care","volume":"6","author":"Bapoje","year":"2011","journal-title":"J Hosp Med"},{"issue":"6","key":"2021081407012218600_ocab111-B5","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1046\/j.1365-2044.1999.00837.x","article-title":"Physiological values and procedures in the 24 h before ICU admission from the ward","volume":"54","author":"Goldhill","year":"1999","journal-title":"Anaesthesia"},{"issue":"6","key":"2021081407012218600_ocab111-B6","doi-asserted-by":"crossref","first-page":"1388","DOI":"10.1378\/chest.98.6.1388","article-title":"Clinical antecedents to in-hospital cardiopulmonary arrest","volume":"98","author":"Schein","year":"1990","journal-title":"Chest"},{"key":"2021081407012218600_ocab111-B7","first-page":"100","article-title":"An Early Warning Scoring System for detecting developing critical illness","volume":"8","author":"Morgan","year":"1997","journal-title":"Clin Intens Care"},{"issue":"8","key":"2021081407012218600_ocab111-B8","doi-asserted-by":"crossref","first-page":"932","DOI":"10.1016\/j.resuscitation.2010.04.014","article-title":"ViEWS\u2013Towards a national early warning score for detecting adult inpatient deterioration","volume":"81","author":"Prytherch","year":"2010","journal-title":"Resuscitation"},{"issue":"10","key":"2021081407012218600_ocab111-B9","doi-asserted-by":"crossref","first-page":"521","DOI":"10.1093\/qjmed\/94.10.521","article-title":"Validation of a modified Early Warning Score in medical admissions","volume":"94","author":"Subbe","year":"2001","journal-title":"QJM"},{"issue":"4","key":"2021081407012218600_ocab111-B10","doi-asserted-by":"crossref","first-page":"841","DOI":"10.1097\/CCM.0000000000000038","article-title":"Using electronic health record data to develop and validate a prediction model for adverse outcomes in the wards","volume":"42","author":"Churpek","year":"2014","journal-title":"Crit Care Med"},{"issue":"5","key":"2021081407012218600_ocab111-B11","doi-asserted-by":"crossref","first-page":"388","DOI":"10.1002\/jhm.1929","article-title":"Early detection of impending physiologic deterioration among patients who are not in intensive care: development of predictive models using data from an automated electronic medical record","volume":"7","author":"Escobar","year":"2012","journal-title":"J Hosp Med"},{"issue":"7","key":"2021081407012218600_ocab111-B12","doi-asserted-by":"crossref","first-page":"2102","DOI":"10.1097\/CCM.0b013e318250aa5a","article-title":"Derivation of a cardiac arrest prediction model using ward vital signs","volume":"40","author":"Churpek","year":"2012","journal-title":"Crit Care Med"},{"key":"2021081407012218600_ocab111-B13","volume-title":"National Early Warning Score (NEWS): Standardising the Assessment of Acute Illness Severity in the NHS","year":"2012"},{"issue":"9","key":"2021081407012218600_ocab111-B14","doi-asserted-by":"crossref","first-page":"2463","DOI":"10.1097\/01.CCM.0000235743.38172.6E","article-title":"Findings of the first consensus conference on medical emergency teams","volume":"34","author":"Devita","year":"2006","journal-title":"Crit Care Med"},{"issue":"2","key":"2021081407012218600_ocab111-B15","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1056\/NEJMra0910926","article-title":"Rapid-response teams","volume":"365","author":"Jones","year":"2011","journal-title":"N Engl J Med"},{"issue":"2","key":"2021081407012218600_ocab111-B16","doi-asserted-by":"crossref","first-page":"150","DOI":"10.1016\/j.resuscitation.2010.09.480","article-title":"An eight-year audit before and after the introduction of modified early warning score (MEWS) charts, of patients admitted to a tertiary referral intensive care unit after CPR","volume":"82","author":"Moon","year":"2011","journal-title":"Resuscitation"},{"issue":"6","key":"2021081407012218600_ocab111-B17","doi-asserted-by":"crossref","first-page":"438","DOI":"10.1002\/jhm.2554","article-title":"Effectiveness of rapid response teams on rates of in-hospital cardiopulmonary arrest and mortality: a systematic review and meta-analysis","volume":"11","author":"Solomon","year":"2016","journal-title":"J Hosp Med"},{"issue":"1","key":"2021081407012218600_ocab111-B18","doi-asserted-by":"crossref","first-page":"254","DOI":"10.1186\/s13054-015-0973-y","article-title":"Rapid response systems: a systematic review and meta-analysis","volume":"19","author":"Maharaj","year":"2015","journal-title":"Crit Care"},{"key":"2021081407012218600_ocab111-B19","doi-asserted-by":"crossref","first-page":"102848","DOI":"10.1016\/j.iccn.2020.102848","article-title":"The impact of rapid response systems on mortality and cardiac arrests: a literature review","volume":"59","author":"Teuma Custo","year":"2020","journal-title":"Intensive Crit Care Nurs"},{"issue":"9","key":"2021081407012218600_ocab111-B20","doi-asserted-by":"crossref","first-page":"1454","DOI":"10.1513\/AnnalsATS.201403-102OC","article-title":"Early warning system scores for clinical deterioration in hospitalized patients: a systematic review","volume":"11","author":"Smith","year":"2014","journal-title":"Ann Am Thorac Soc"},{"key":"2021081407012218600_ocab111-B21","doi-asserted-by":"crossref","first-page":"103410","DOI":"10.1016\/j.jbi.2020.103410","article-title":"Development and validation of early warning score system: a systematic literature review","volume":"105","author":"Fu","year":"2020","journal-title":"J Biomed Inform"},{"issue":"5","key":"2021081407012218600_ocab111-B22","doi-asserted-by":"crossref","first-page":"837","DOI":"10.1016\/j.jbi.2013.06.011","article-title":"Development and validation of a continuous measure of patient condition using the electronic medical record","volume":"46","author":"Rothman","year":"2013","journal-title":"J Biomed Inform"},{"issue":"5","key":"2021081407012218600_ocab111-B23","doi-asserted-by":"crossref","first-page":"563","DOI":"10.1016\/j.resuscitation.2011.10.023","article-title":"Rapid Response Team composition, resourcing and calling criteria in Australia","volume":"83","year":"2012","journal-title":"Resuscitation"},{"issue":"12","key":"2021081407012218600_ocab111-B24","doi-asserted-by":"crossref","first-page":"2877","DOI":"10.1111\/jan.13398","article-title":"Early warning systems and rapid response to the deteriorating patient in hospital: A systematic realist review","volume":"73","author":"McGaughey","year":"2017","journal-title":"J Adv Nurs"},{"issue":"4","key":"2021081407012218600_ocab111-B25","doi-asserted-by":"crossref","first-page":"251","DOI":"10.1016\/j.apnr.2011.06.002","article-title":"Reading between the lines\u201d of flow sheet data: nurses\u2019 optional documentation associated with cardiac arrest outcomes","volume":"25","author":"Collins","year":"2012","journal-title":"Appl Nurs Res"},{"issue":"10","key":"2021081407012218600_ocab111-B26","doi-asserted-by":"crossref","first-page":"1992","DOI":"10.1111\/j.1365-2648.2009.05109.x","article-title":"Nurses\u2019 role in detecting deterioration in ward patients: systematic literature review","volume":"65","author":"Odell","year":"2009","journal-title":"J Adv Nurs"},{"issue":"4","key":"2021081407012218600_ocab111-B27","doi-asserted-by":"crossref","first-page":"306","DOI":"10.4037\/ajcc2013426","article-title":"Relationship between nursing documentation and patients\u2019 mortality","volume":"22","author":"Collins","year":"2013","journal-title":"Am J Crit Care"},{"key":"2021081407012218600_ocab111-B28","article-title":"Workarounds used by nurses to overcome design constraints of electronic health records","volume-title":"In: proceedings of the 11th International Congress on Nursing Informatics","author":"Collins"},{"issue":"1","key":"2021081407012218600_ocab111-B29","doi-asserted-by":"crossref","first-page":"230","DOI":"10.1186\/s13054-015-0950-5","article-title":"Nurses\u2019 worry or concern and early recognition of deteriorating patients on general wards in acute care hospitals: a systematic review","volume":"19","author":"Douw","year":"2015","journal-title":"Crit Care"},{"issue":"1","key":"2021081407012218600_ocab111-B30","doi-asserted-by":"crossref","first-page":"117","DOI":"10.1136\/amiajnl-2012-001145","article-title":"Next-generation phenotyping of electronic health records","volume":"20","author":"Hripcsak","year":"2013","journal-title":"J Am Med Inform Assoc"},{"issue":"Suppl 1","key":"2021081407012218600_ocab111-B31","doi-asserted-by":"crossref","first-page":"i109","DOI":"10.1136\/amiajnl-2011-000463","article-title":"Exploiting time in electronic health record correlations","volume":"18","author":"Hripcsak","year":"2011","journal-title":"J Am Med Inform Assoc"},{"key":"2021081407012218600_ocab111-B32","first-page":"323","article-title":"Leveraging clinical expertise as a feature - not an outcome - of predictive models: evaluation of an early warning system use case","volume-title":"AMIA Annu Symp Proc","author":"Rossetti","year":"2019"},{"key":"2021081407012218600_ocab111-B33","doi-asserted-by":"crossref","first-page":"1242","DOI":"10.1093\/jamia\/ocab006","article-title":"Healthcare process modeling to phenotype clinician behaviors for exploiting the signal gain of clinical expertise (HPM-ExpertSignals): development and evaluation of a conceptual framework","author":"Rossetti","year":"2021","journal-title":"J Am Med Inform Assoc"},{"key":"2021081407012218600_ocab111-B34","author":"Chung","year":"2014"},{"issue":"8","key":"2021081407012218600_ocab111-B35","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput"},{"issue":"1","key":"2021081407012218600_ocab111-B36","doi-asserted-by":"crossref","first-page":"267","DOI":"10.1111\/j.2517-6161.1996.tb02080.x","article-title":"Regression shrinkage and selection via the Lasso","volume":"58","author":"Tibshirani","year":"1996","journal-title":"J Roy Stat Soc Ser B (Methodol)"},{"issue":"1","key":"2021081407012218600_ocab111-B37","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1080\/00401706.1970.10488634","article-title":"Ridge regression: biased estimation for nonorthogonal problems","volume":"12","author":"Hoerl","year":"1970","journal-title":"Technometrics"},{"issue":"1","key":"2021081407012218600_ocab111-B38","first-page":"6765","article-title":"Hyperband: a novel bandit-based approach to hyperparameter optimization","volume":"18","author":"Li","year":"2017","journal-title":"J Mach Learn Res"},{"issue":"1","key":"2021081407012218600_ocab111-B39","doi-asserted-by":"crossref","first-page":"286","DOI":"10.1186\/s13054-018-2194-7","article-title":"Multicenter derivation and validation of an early warning score for acute respiratory failure or death in the hospital","volume":"22","author":"Dziadzko","year":"2018","journal-title":"Crit Care"},{"key":"2021081407012218600_ocab111-B40","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1016\/j.jbi.2016.09.013","article-title":"Development and validation of an electronic medical record-based alert score for detection of inpatient deterioration outside the ICU","volume":"64","author":"Kipnis","year":"2016","journal-title":"J Biomed Inform"},{"key":"2021081407012218600_ocab111-B41","doi-asserted-by":"crossref","first-page":"28","DOI":"10.1186\/1472-6947-13-28","article-title":"Predicting out of intensive care unit cardiopulmonary arrest or death using electronic medical record data","volume":"13","author":"Alvarez","year":"2013","journal-title":"BMC Med Inform Decis Mak"},{"key":"2021081407012218600_ocab111-B42","first-page":"93","article-title":"Workarounds used by nurses to overcome design constraints of electronic health records","volume":"2012","author":"Collins","year":"2012","journal-title":"Ni 2012 (2012)"},{"issue":"1","key":"2021081407012218600_ocab111-B43","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s11749-009-0138-x","article-title":"Missing data methods in longitudinal studies: a review","volume":"18","author":"Ibrahim","year":"2009","journal-title":"Test (Madr)"},{"issue":"2","key":"2021081407012218600_ocab111-B44","doi-asserted-by":"crossref","first-page":"368","DOI":"10.1097\/CCM.0000000000001571","article-title":"Multicenter comparison of machine learning methods and conventional regression for predicting clinical deterioration on the wards","volume":"44","author":"Churpek","year":"2016","journal-title":"Crit Care Med"},{"key":"2021081407012218600_ocab111-B45","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.resuscitation.2016.02.005","article-title":"The value of vital sign trends for detecting clinical deterioration on the wards","volume":"102","author":"Churpek","year":"2016","journal-title":"Resuscitation"},{"issue":"6","key":"2021081407012218600_ocab111-B46","doi-asserted-by":"crossref","first-page":"649","DOI":"10.1164\/rccm.201406-1022OC","article-title":"Multicenter development and validation of a risk stratification tool for ward patients","volume":"190","author":"Churpek","year":"2014","journal-title":"Am J Respir Crit Care Med"},{"issue":"1","key":"2021081407012218600_ocab111-B47","doi-asserted-by":"crossref","first-page":"96","DOI":"10.1038\/s41597-019-0103-9","article-title":"Multitask learning and benchmarking with clinical time series data","volume":"6","author":"Harutyunyan","year":"2019","journal-title":"Sci Data"},{"key":"2021081407012218600_ocab111-B48","first-page":"460","article-title":"An interpretable ICU mortality prediction model based on logistic regression and recurrent neural networks with LSTM units","volume":"2018","author":"Ge","year":"2018","journal-title":"AMIA Annu Symp Proc"},{"issue":"1","key":"2021081407012218600_ocab111-B49","doi-asserted-by":"crossref","first-page":"6085","DOI":"10.1038\/s41598-018-24271-9","article-title":"Recurrent neural networks for multivariate time series with missing values","volume":"8","author":"Che","year":"2018","journal-title":"Sci Rep"},{"key":"2021081407012218600_ocab111-B50","author":"Lipton","year":"2016"},{"issue":"3","key":"2021081407012218600_ocab111-B51","doi-asserted-by":"crossref","first-page":"289","DOI":"10.1093\/jamia\/ocx110","article-title":"High-fidelity phenotyping: richness and freedom from bias","volume":"25","author":"Hripcsak","year":"2018","journal-title":"J Am Med Inform Assoc"},{"key":"2021081407012218600_ocab111-B52","doi-asserted-by":"crossref","first-page":"k1479","DOI":"10.1136\/bmj.k1479","article-title":"Biases in electronic health record data due to processes within the healthcare system: retrospective observational study","volume":"361","author":"Agniel","year":"2018","journal-title":"BMJ (Clin Res Ed)"},{"key":"2021081407012218600_ocab111-B53","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1016\/j.jbi.2014.03.016","article-title":"Identifying and mitigating biases in EHR laboratory tests","volume":"51","author":"Pivovarov","year":"2014","journal-title":"J Biomed Inform"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/28\/9\/1955\/39731746\/ocab111.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/academic.oup.com\/jamia\/article-pdf\/28\/9\/1955\/39731746\/ocab111.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,8,31]],"date-time":"2024-08-31T06:16:19Z","timestamp":1725084979000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/28\/9\/1955\/6322899"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,7,16]]},"references-count":53,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2021,7,16]]},"published-print":{"date-parts":[[2021,8,13]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocab111","relation":{},"ISSN":["1527-974X"],"issn-type":[{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021,9,1]]},"published":{"date-parts":[[2021,7,16]]}}}