{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,15]],"date-time":"2026-06-15T11:03:00Z","timestamp":1781521380542,"version":"3.54.1"},"reference-count":42,"publisher":"Oxford University Press (OUP)","issue":"6","license":[{"start":{"date-parts":[[2024,4,29]],"date-time":"2024-04-29T00:00:00Z","timestamp":1714348800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000057","name":"National Institute of General Medical Sciences","doi-asserted-by":"publisher","award":["R35-13362546"],"award-info":[{"award-number":["R35-13362546"]}],"id":[{"id":"10.13039\/100000057","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000050","name":"National Heart, Lung, and Blood Institute","doi-asserted-by":"publisher","award":["R01HL157262"],"award-info":[{"award-number":["R01HL157262"]}],"id":[{"id":"10.13039\/100000050","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000062","name":"National Institute of Diabetes, Digestive and Kidney Diseases","doi-asserted-by":"crossref","award":["R01-DK126933A"],"award-info":[{"award-number":["R01-DK126933A"]}],"id":[{"id":"10.13039\/100000062","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,5,20]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objectives<\/jats:title>\n                  <jats:p>To compare and externally validate popular deep learning model architectures and data transformation methods for variable-length time series data in 3 clinical tasks (clinical deterioration, severe acute kidney injury [AKI], and suspected infection).<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Materials and Methods<\/jats:title>\n                  <jats:p>This multicenter retrospective study included admissions at 2 medical centers that spanned 2007-2022. Distinct datasets were created for each clinical task, with 1 site used for training and the other for testing. Three feature engineering methods (normalization, standardization, and piece-wise linear encoding with decision trees [PLE-DTs]) and 3 architectures (long short-term memory\/gated recurrent unit [LSTM\/GRU], temporal convolutional network, and time-distributed wrapper with convolutional neural network [TDW-CNN]) were compared in each clinical task. Model discrimination was evaluated using the area under the precision-recall curve (AUPRC) and the area under the receiver operating characteristic curve (AUROC).<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>The study comprised 373\u00a0825 admissions for training and 256\u00a0128 admissions for testing. LSTM\/GRU models tied with TDW-CNN models with both obtaining the highest mean AUPRC in 2 tasks, and LSTM\/GRU had the highest mean AUROC across all tasks (deterioration: 0.81, AKI: 0.92, infection: 0.87). PLE-DT with LSTM\/GRU achieved the highest AUPRC in all tasks.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Discussion<\/jats:title>\n                  <jats:p>When externally validated in 3 clinical tasks, the LSTM\/GRU model architecture with PLE-DT transformed data demonstrated the highest AUPRC in all tasks. Multiple models achieved similar performance when evaluated using AUROC.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusion<\/jats:title>\n                  <jats:p>The LSTM architecture performs as well or better than some newer architectures, and PLE-DT may enhance the AUPRC in variable-length time series data for predicting clinical outcomes during external validation.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jamia\/ocae088","type":"journal-article","created":{"date-parts":[[2024,4,29]],"date-time":"2024-04-29T05:30:00Z","timestamp":1714368600000},"page":"1322-1330","source":"Crossref","is-referenced-by-count":12,"title":["Development and external validation of deep learning clinical prediction models using variable-length time series data"],"prefix":"10.1093","volume":"31","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0153-4958","authenticated-orcid":false,"given":"Fereshteh S","family":"Bashiri","sequence":"first","affiliation":[{"name":"Department of Medicine, University of Wisconsin-Madison , Madison, WI 53792, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kyle A","family":"Carey","sequence":"additional","affiliation":[{"name":"Department of Medicine, University of Chicago , Chicago, IL 60637, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jennie","family":"Martin","sequence":"additional","affiliation":[{"name":"Department of Medicine, University of Wisconsin-Madison , Madison, WI 53792, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jay L","family":"Koyner","sequence":"additional","affiliation":[{"name":"Department of Medicine, University of Chicago , Chicago, IL 60637, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dana P","family":"Edelson","sequence":"additional","affiliation":[{"name":"Department of Medicine, University of Chicago , Chicago, IL 60637, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Emily R","family":"Gilbert","sequence":"additional","affiliation":[{"name":"Department of Medicine, Loyola University , Chicago, IL 60153, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anoop","family":"Mayampurath","sequence":"additional","affiliation":[{"name":"Department of Medicine, University of Wisconsin-Madison , Madison, WI 53792, United States"},{"name":"Department of Biostatistics and Medical Informatics, University of Wisconsin-Madison , Madison, WI 53726, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Majid","family":"Afshar","sequence":"additional","affiliation":[{"name":"Department of Medicine, University of Wisconsin-Madison , Madison, WI 53792, United States"},{"name":"Department of Biostatistics and Medical Informatics, University of Wisconsin-Madison , Madison, WI 53726, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Matthew M","family":"Churpek","sequence":"additional","affiliation":[{"name":"Department of Medicine, University of Wisconsin-Madison , Madison, WI 53792, United States"},{"name":"Department of Biostatistics and Medical Informatics, University of Wisconsin-Madison , Madison, WI 53726, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2024,4,29]]},"reference":[{"issue":"6","key":"2024052019461201200_ocae088-B1","doi-asserted-by":"crossref","first-page":"395","DOI":"10.1038\/nrg3208","article-title":"Mining electronic health records: towards better research applications and clinical care","volume":"13","author":"Jensen","year":"2012","journal-title":"Nat Rev Genet"},{"key":"2024052019461201200_ocae088-B2","doi-asserted-by":"crossref","first-page":"86","DOI":"10.1016\/j.resuscitation.2017.10.028","article-title":"Comparison of the between the flags calling criteria to the MEWS, NEWS and the electronic cardiac arrest risk triage (eCART) score for the identification of deteriorating ward patients","volume":"123","author":"Green","year":"2018","journal-title":"Resuscitation"},{"issue":"8","key":"2024052019461201200_ocae088-B3","doi-asserted-by":"crossref","first-page":"e2012892","DOI":"10.1001\/jamanetworkopen.2020.12892","article-title":"Internal and external validation of a machine learning risk score for acute kidney injury","volume":"3","author":"Churpek","year":"2020","journal-title":"JAMA Netw Open"},{"issue":"1","key":"2024052019461201200_ocae088-B4","doi-asserted-by":"crossref","first-page":"e0245157","DOI":"10.1371\/journal.pone.0245157","article-title":"A comparison of machine learning models versus clinical evaluation for mortality prediction in patients with sepsis","volume":"16","author":"van Doorn","year":"2021","journal-title":"PLoS One"},{"issue":"7-8","key":"2024052019461201200_ocae088-B5","doi-asserted-by":"crossref","first-page":"1360","DOI":"10.1111\/jocn.14238","article-title":"Clinical deterioration: a concept analysis","volume":"27","author":"Padilla","year":"2018","journal-title":"J Clin Nurs"},{"issue":"4","key":"2024052019461201200_ocae088-B6","doi-asserted-by":"crossref","first-page":"307","DOI":"10.1097\/01.NCQ.0000290410.19769.41","article-title":"Implementation and outcomes of a rapid response team","volume":"22","author":"McFarlan","year":"2007","journal-title":"J Nurs Care Qual"},{"issue":"2","key":"2024052019461201200_ocae088-B7","doi-asserted-by":"crossref","first-page":"e187571","DOI":"10.1001\/jamanetworkopen.2018.7571","article-title":"Prevalence, underlying causes, and preventability of sepsis-associated mortality in US acute care hospitals","volume":"2","author":"Rhee","year":"2019","journal-title":"JAMA Netw Open"},{"issue":"10212","key":"2024052019461201200_ocae088-B8","doi-asserted-by":"crossref","first-page":"1949","DOI":"10.1016\/S0140-6736(19)32563-2","article-title":"Acute kidney injury","volume":"394","author":"Ronco","year":"2019","journal-title":"Lancet"},{"issue":"1","key":"2024052019461201200_ocae088-B9","doi-asserted-by":"crossref","first-page":"295","DOI":"10.1186\/s12884-022-04631-0","article-title":"Comparison of early warning scores for predicting clinical deterioration and infection in obstetric patients","volume":"22","author":"Arnolds","year":"2022","journal-title":"BMC Pregnancy Childb"},{"issue":"6","key":"2024052019461201200_ocae088-B10","doi-asserted-by":"crossref","first-page":"1059","DOI":"10.1097\/SLA.0000000000002665","article-title":"Validating the electronic cardiac arrest risk triage (eCART) score for risk stratification of surgical inpatients in the postoperative setting: retrospective cohort study","volume":"269","author":"Bartkowiak","year":"2019","journal-title":"Ann Surg"},{"issue":"5","key":"2024052019461201200_ocae088-B11","doi-asserted-by":"crossref","first-page":"1589","DOI":"10.1109\/JBHI.2017.2767063","article-title":"Deep EHR: a survey of recent advances in deep learning techniques for electronic health record (EHR) analysis","volume":"22","author":"Shickel","year":"2017","journal-title":"IEEE J Biomed Health Inform"},{"issue":"10","key":"2024052019461201200_ocae088-B12","doi-asserted-by":"crossref","first-page":"1696","DOI":"10.1093\/jamia\/ocac109","article-title":"Identifying infected patients using semi-supervised and transfer learning","volume":"29","author":"Bashiri","year":"2022","journal-title":"J Am Med Inform Assoc"},{"key":"2024052019461201200_ocae088-B13","doi-asserted-by":"crossref","first-page":"112","DOI":"10.1016\/j.jbi.2018.04.007","article-title":"Benchmarking deep learning models on large healthcare datasets","volume":"83","author":"Purushotham","year":"2018","journal-title":"J Biomed Inform"},{"issue":"1","key":"2024052019461201200_ocae088-B14","doi-asserted-by":"crossref","first-page":"462","DOI":"10.1186\/s12967-020-02620-5","article-title":"Predicting 30-days mortality for MIMIC-III patients with sepsis-3: a machine learning approach using XGboost","volume":"18","author":"Hou","year":"2020","journal-title":"J Transl Med"},{"key":"2024052019461201200_ocae088-B15","doi-asserted-by":"crossref","first-page":"103269","DOI":"10.1016\/j.jbi.2019.103269","article-title":"ISeeU: visually interpretable deep learning for mortality prediction inside the ICU","volume":"98","author":"Caicedo-Torres","year":"2019","journal-title":"J Biomed Inform"},{"issue":"6","key":"2024052019461201200_ocae088-B16","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3516367","article-title":"Self-supervised transformer for sparse and irregularly sampled multivariate clinical time-series","volume":"16","author":"Tipirneni","year":"2022","journal-title":"ACM Trans Knowl Disc Data (TKDD)"},{"issue":"1","key":"2024052019461201200_ocae088-B17","doi-asserted-by":"crossref","first-page":"160035","DOI":"10.1038\/sdata.2016.35","article-title":"MIMIC-III, a freely accessible critical care database","volume":"3","author":"Johnson","year":"2016","journal-title":"Sci Data"},{"issue":"7","key":"2024052019461201200_ocae088-B18","doi-asserted-by":"crossref","first-page":"1070","DOI":"10.1097\/CCM.0000000000003123","article-title":"The development of a machine learning inpatient acute kidney injury prediction model","volume":"46","author":"Koyner","year":"2018","journal-title":"Crit Care Med"},{"issue":"1","key":"2024052019461201200_ocae088-B19","first-page":"1","article-title":"Kidney disease: improving global outcomes (KDIGO) acute kidney injury work group. KDIGO clinical practice guideline for acute kidney injury","volume":"2","author":"Kellum","year":"2012","journal-title":"Kidney Int Suppl (2011)"},{"issue":"8","key":"2024052019461201200_ocae088-B20","doi-asserted-by":"crossref","first-page":"762","DOI":"10.1001\/jama.2016.0288","article-title":"Assessment of clinical criteria for sepsis: for the third international consensus definitions for sepsis and septic shock (Sepsis-3)","volume":"315","author":"Seymour","year":"2016","journal-title":"JAMA"},{"issue":"7","key":"2024052019461201200_ocae088-B21","doi-asserted-by":"crossref","first-page":"e673","DOI":"10.1097\/CCM.0000000000004968","article-title":"Determining the electronic signature of infection in electronic health record data","volume":"49","author":"Churpek","year":"2021","journal-title":"Crit Care Med"},{"key":"2024052019461201200_ocae088-B22","first-page":"24991","volume-title":"Advances in Neural Information Processing Systems","author":"Gorishniy","year":"2022"},{"key":"2024052019461201200_ocae088-B23","author":"Bai","year":"2018"},{"key":"2024052019461201200_ocae088-B24","author":"Remy","year":"2020"},{"key":"2024052019461201200_ocae088-B25","author":"Dallanoce","year":"2021"},{"key":"2024052019461201200_ocae088-B26","doi-asserted-by":"crossref","first-page":"60039","DOI":"10.1109\/ACCESS.2022.3179577","article-title":"Timedistributed-CNN-LSTM: a hybrid approach combining CNN and LSTM to classify brain tumor on 3d MRI scans performing ablation study","volume":"10","author":"Montaha","year":"2022","journal-title":"IEEE Access"},{"key":"2024052019461201200_ocae088-B27","first-page":"1","author":"Siddique","year":"2022"},{"key":"2024052019461201200_ocae088-B28","first-page":"473","author":"Hochreiter","year":"1996"},{"issue":"5-6","key":"2024052019461201200_ocae088-B29","doi-asserted-by":"crossref","first-page":"602","DOI":"10.1016\/j.neunet.2005.06.042","article-title":"Framewise phoneme classification with bidirectional LSTM and other neural network architectures","volume":"18","author":"Graves","year":"2005","journal-title":"Neural Netw"},{"key":"2024052019461201200_ocae088-B30","author":"Cho","year":"2014"},{"key":"2024052019461201200_ocae088-B31","author":"Ba","year":"2016"},{"key":"2024052019461201200_ocae088-B32","first-page":"448","author":"Ioffe","year":"2015"},{"issue":"3","key":"2024052019461201200_ocae088-B33","doi-asserted-by":"crossref","first-page":"837","DOI":"10.2307\/2531595","article-title":"Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach","volume":"44","author":"DeLong","year":"1988","journal-title":"Biometrics"},{"issue":"11","key":"2024052019461201200_ocae088-B34","doi-asserted-by":"crossref","first-page":"1389","DOI":"10.1109\/LSP.2014.2337313","article-title":"Fast implementation of DeLong\u2019s algorithm for comparing the areas under correlated receiver operating characteristic curves","volume":"21","author":"Sun","year":"2014","journal-title":"IEEE Signal Process Lett"},{"issue":"8","key":"2024052019461201200_ocae088-B35","doi-asserted-by":"crossref","first-page":"e37658","DOI":"10.2196\/37658","article-title":"Predicting abnormalities in laboratory values of patients in the intensive care unit using different deep learning models: comparative study","volume":"10","author":"Ayad","year":"2022","journal-title":"JMIR Med Inform"},{"key":"2024052019461201200_ocae088-B36","first-page":"2415","author":"Gopali","year":"2021"},{"key":"2024052019461201200_ocae088-B37","author":"Almqvist","year":"2019"},{"issue":"7767","key":"2024052019461201200_ocae088-B38","doi-asserted-by":"crossref","first-page":"116","DOI":"10.1038\/s41586-019-1390-1","article-title":"A clinically applicable approach to continuous prediction of future acute kidney injury","volume":"572","author":"Toma\u0161ev","year":"2019","journal-title":"Nature"},{"key":"2024052019461201200_ocae088-B39","doi-asserted-by":"crossref","first-page":"100407","DOI":"10.1016\/j.bdr.2023.100407","article-title":"A large comparison of normalization methods on time series","volume":"34","author":"Lima","year":"2023","journal-title":"Big Data Res"},{"key":"2024052019461201200_ocae088-B40","author":"Gorishniy","year":"2023"},{"issue":"17","key":"2024052019461201200_ocae088-B41","doi-asserted-by":"crossref","first-page":"5658","DOI":"10.3390\/jcm12175658","article-title":"Machine learning-based early prediction of sepsis using electronic health records: a systematic review","volume":"12","author":"Islam","year":"2023","journal-title":"J Clin Med"},{"issue":"12","key":"2024052019461201200_ocae088-B42","doi-asserted-by":"crossref","first-page":"2266","DOI":"10.1093\/ckj\/sfac181","article-title":"Machine learning models for predicting acute kidney injury: a systematic review and critical appraisal","volume":"15","author":"Vagliano","year":"2022","journal-title":"Clin Kidney J"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/6\/1322\/57768913\/ocae088.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/6\/1322\/57768913\/ocae088.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,5,20]],"date-time":"2024-05-20T19:46:36Z","timestamp":1716234396000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/31\/6\/1322\/7659472"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,29]]},"references-count":42,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2024,4,29]]},"published-print":{"date-parts":[[2024,5,20]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocae088","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,6,1]]},"published":{"date-parts":[[2024,4,29]]}}}