{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,18]],"date-time":"2026-09-18T20:41:23Z","timestamp":1789764083232,"version":"4.0.1"},"reference-count":41,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2025,1,8]],"date-time":"2025-01-08T00:00:00Z","timestamp":1736294400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JCM"],"abstract":"<jats:p>Background Predictive machine learning models have made use of a variety of scoring systems to identify clinical deterioration in ICU patients. However, most of these scores include variables that are dependent on medical staff examining the patient. We present the development of a real-time prediction model using clinical variables that are digital and automatically generated for the early detection of patients at risk of deterioration. Methods Routine monitoring data were used in this analysis. ICU patients with at least 24 h of vital sign recordings were included. Deterioration was defined as qSOFA \u2265 2. Model development and validation were performed internally by splitting the cohort into training and test datasets and validating the results on the test dataset. Five different models were trained, tested, and compared against each other. The models were an artificial neural network (ANN), a random forest (RF), a support vector machine (SVM), a linear discriminant analysis (LDA), and a logistic regression (LR). Results In total, 7156 ICU patients were screened for inclusion in the study, which resulted in models trained from a total of 28,348 longitudinal measurements. The artificial neural network showed a superior predictive performance for deterioration, with an area under the curve of 0.81 over 0.78 (RF), 0.78 (SVM), 0.77 (LDA), and 0.76 (LR), by using only four vital parameters. The sensitivity was higher than the specificity for the artificial neural network. Conclusions The artificial neural network, only using four automatically recorded vital signs, was best able to predict deterioration, 10 h before documentation in clinical records. This real-time prediction model has the potential to flag at-risk patients to the healthcare providers treating them, for closer monitoring and further investigation.<\/jats:p>","DOI":"10.3390\/jcm14020350","type":"journal-article","created":{"date-parts":[[2025,1,8]],"date-time":"2025-01-08T09:30:45Z","timestamp":1736328645000},"page":"350","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":11,"title":["Machine Learning Models for the Early Real-Time Prediction of Deterioration in Intensive Care Units\u2014A Novel Approach to the Early Identification of High-Risk Patients"],"prefix":"10.3390","volume":"14","author":[{"given":"Dominik","family":"Thiele","sequence":"first","affiliation":[{"name":"Department of Neurology and Experimental Neurology, Charit\u00e9\u2014Universit\u00e4tsmedizin Berlin, 10117 Berlin, Germany"},{"name":"TCC Analytics, Telehealth Competence Center (TCC) GmbH, 22083 Hamburg, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3779-7805","authenticated-orcid":false,"given":"Reitze","family":"Rodseth","sequence":"additional","affiliation":[{"name":"Netcare Limited, Johannesburg 2196, South Africa"},{"name":"Department of Anaesthesiology and Critical Care, University of KwaZulu-Natal, Durban 4001, South Africa"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Richard","family":"Friedland","sequence":"additional","affiliation":[{"name":"Netcare Limited, Johannesburg 2196, South Africa"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fabian","family":"Berger","sequence":"additional","affiliation":[{"name":"Netcare Limited, Johannesburg 2196, South Africa"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-9164-5225","authenticated-orcid":false,"given":"Chris","family":"Mathew","sequence":"additional","affiliation":[{"name":"Netcare Limited, Johannesburg 2196, South Africa"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0145-3672","authenticated-orcid":false,"given":"Caroline","family":"Maslo","sequence":"additional","affiliation":[{"name":"Netcare Limited, Johannesburg 2196, South Africa"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vanessa","family":"Moll","sequence":"additional","affiliation":[{"name":"Department of Anesthesiology, Division of Critical Care Medicine, University of Minnesota School of Medicine, Minneapolis, MN 55455, USA"},{"name":"Department of Anesthesiology, Division of Critical Care Medicine, Emory University School of Medicine, Atlanta, GA 30322, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christoph","family":"Leithner","sequence":"additional","affiliation":[{"name":"Department of Neurology and Experimental Neurology, Charit\u00e9\u2014Universit\u00e4tsmedizin Berlin, 10117 Berlin, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christian","family":"Storm","sequence":"additional","affiliation":[{"name":"TCC Analytics, Telehealth Competence Center (TCC) GmbH, 22083 Hamburg, Germany"},{"name":"Department of Nephrology and Medical Intensive Care, Charit\u00e9\u2014Universit\u00e4tsmedizin Berlin, 22083 Berlin, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alexander","family":"Krannich","sequence":"additional","affiliation":[{"name":"TCC Analytics, Telehealth Competence Center (TCC) GmbH, 22083 Hamburg, Germany"},{"name":"Experimental and Clinical Research Center (ECRC), Charit\u00e9\u2014Universit\u00e4tsmedizin Berlin, 22083 Berlin, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9888-4358","authenticated-orcid":false,"given":"Jens","family":"Nee","sequence":"additional","affiliation":[{"name":"Department of Nephrology and Medical Intensive Care, Charit\u00e9\u2014Universit\u00e4tsmedizin Berlin, 22083 Berlin, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2025,1,8]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"133","DOI":"10.1093\/bja\/aer194","article-title":"Rodseth Utility of clinical risk predictors for preoperative cardiovascular risk prediction","volume":"107","author":"Biccard","year":"2011","journal-title":"Br. J. Anaesth."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Doyle, D.J. (2018). Clinical Early Warning Scores: New Clinical Tools in Evolution. Open Anesth. J., 12.","DOI":"10.2174\/2589645801812010026"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"e0116","DOI":"10.1097\/CCE.0000000000000116","article-title":"Early Detection of In-Patient Deterioration: One Prediction Model Does Not Fit All","volume":"2","author":"Blackwell","year":"2020","journal-title":"Crit. Care Explor."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"220","DOI":"10.4103\/0972-5229.130573","article-title":"Scoring systems in the intensive care unit: A compendium","volume":"18","author":"Rapsang","year":"2014","journal-title":"Indian J. Crit. Care Med."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"779","DOI":"10.1016\/j.colegn.2023.05.006","article-title":"Queensland Adult Deterioration Detection System observation chart diagnostic accuracy in detecting patient deterioration: A retrospective case\u2013control study","volume":"30","author":"Flenady","year":"2023","journal-title":"Collegian"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"170","DOI":"10.1016\/j.jacc.2013.08.1630","article-title":"The Prognostic Value of Pre-Operative and Post-Operative B-Type Natriuretic Peptides in Patients Undergoing Noncardiac Surgery","volume":"63","author":"Rodseth","year":"2014","journal-title":"J. Am. Coll. Cardiol."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"P\u00e9rez-Fern\u00e1ndez, J., Raimondi, N.A., and Murillo Cabezas, F. (2020). Digital Transformation: The Smart ICU. Critical Care Administration, Springer.","DOI":"10.1007\/978-3-030-33808-4_9"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.cmpb.2018.12.027","article-title":"Prediction of sepsis patients using machine learning approach: A meta-analysis","volume":"170","author":"Islam","year":"2019","journal-title":"Comput. Methods Programs Biomed."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"e35293","DOI":"10.2196\/35293","article-title":"Comparison of Severity of Illness Scores and Artificial Intelligence Models That Are Predictive of Intensive Care Unit Mortality: Meta-analysis and Review of the Literature","volume":"10","author":"Barboi","year":"2022","journal-title":"JMIR Med. Inform."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Choi, M.H., Kim, D., Choi, E.J., Jung, Y.J., Choi, Y.J., Cho, J.H., and Jeong, S.H. (2022). Mortality prediction of patients in intensive care units using machine learning algorithms based on electronic health records. Sci. Rep., 12.","DOI":"10.1038\/s41598-022-11226-4"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"g7594","DOI":"10.1136\/bmj.g7594","article-title":"Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): The TRIPOD statement","volume":"350","author":"Collins","year":"2015","journal-title":"BMJ"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Ghalati, P.F., Samal, S.S., Bhat, J.S., Deisz, R., Marx, G., and Schuppert, A. (2019). Critical Transitions in Intensive Care Units: A Sepsis Case Study. Sci. Rep., 9.","DOI":"10.1038\/s41598-019-49006-2"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Moor, M., Rieck, B., Horn, M., Jutzeler, C.R., and Borgwardt, K. (2021). Early Prediction of Sepsis in the ICU Using Machine Learning: A Systematic Review. Front. Med., 8.","DOI":"10.3389\/fmed.2021.607952"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"383","DOI":"10.1007\/s00134-019-05872-y","article-title":"Machine learning for the prediction of sepsis: A systematic review and meta-analysis of diagnostic test accuracy","volume":"46","author":"Fleuren","year":"2020","journal-title":"Intensiv. Care Med."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Harrel, F.E. (2015). Regression Modeling Strategies\u2014With Applications to Linear Models, Logistic and Ordinal Regression, and Survival Analysis, Springer. R Software.","DOI":"10.1007\/978-3-319-19425-7"},{"key":"ref_16","unstructured":"R Core Team (2020). A Language and Environment for Statistical Computing, R Foundation for Statistical Computing. Available online: https:\/\/www.R-project.org\/."},{"key":"ref_17","unstructured":"Fritsch, S., Guenther, F., and Wright, M. (2024, December 18). Neuralnet: Training of Neural Networks. Available online: https:\/\/CRANR-project.org\/package=neuralnet."},{"key":"ref_18","unstructured":"Meyer, D., Dimitriadou, E., Hornik, K., Weingessel, A., Leisch, F., Chang, C.C., and Lin, C.C. (2022). e1071: Misc Functions of the Department of Statistics, Probability Theory Group (Formerly: E1071), TU Wien. R package version 1.7-12."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1","DOI":"10.18637\/jss.v077.i01","article-title":"Ranger: A fast implementation of random forests for high dimensional data in C++ and R","volume":"77","author":"Wright","year":"2017","journal-title":"J. Stat. Softw."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Venables, W.N., and Ripley, B.D. (2002). Modern Applied Statistics with S, Springer. [4th ed.].","DOI":"10.1007\/978-0-387-21706-2"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"547","DOI":"10.1097\/CCM.0000000000002936","article-title":"An Interpretable Machine Learning Model for Accurate Prediction of Sepsis in the ICU","volume":"46","author":"Nemati","year":"2018","journal-title":"Crit. Care Med."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Kaji, D.A., Zech, J.R., Kim, J.S., Cho, S.K., Dangayach, N.S., Costa, A.B., and Oermann, E.K. (2019). An attention based deep learning model of clinical events in the intensive care unit. PLoS ONE, 14.","DOI":"10.1371\/journal.pone.0211057"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1505","DOI":"10.1016\/j.ajem.2016.05.009","article-title":"A critical analysis of unplanned ICU transfer within 48 hours from ED admission as a quality measure","volume":"34","author":"Dahn","year":"2016","journal-title":"Am. J. Emerg. Med."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1177\/0310057X1504300211","article-title":"Intensive care unit admission in patients following rapid response team activation: Call factors, patient characteristics and hospital outcomes","volume":"43","author":"Tobin","year":"2015","journal-title":"Anaesth Intensive Care"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Moss, T.J., Clark, M.T., Calland, J.F., Enfield, K.B., Voss, J.D., Lake, D.E., and Moorman, J.R. (2017). Cardiorespiratory dynamics measured from continuous ECG monitoring improves detection of deterioration in acute care patients: A retrospective cohort study. PLoS ONE, 12.","DOI":"10.1371\/journal.pone.0181448"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1639","DOI":"10.1097\/CCM.0000000000001738","article-title":"Signatures of Subacute Potentially Catastrophic Illness in the ICU: Model Development and Validation*","volume":"44","author":"Moss","year":"2016","journal-title":"Crit. Care Med."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1629","DOI":"10.1007\/s00134-002-1496-y","article-title":"Duration of life-threatening antecedents prior to intensive care admission","volume":"28","author":"Hillman","year":"2002","journal-title":"Intensive Care Med."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"363","DOI":"10.1016\/j.jcrc.2014.11.001","article-title":"Medical intensive care unit consults occurring within 48 hours of admission: A prospective study","volume":"30","author":"Cohen","year":"2015","journal-title":"J. Crit. Care"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"68","DOI":"10.1002\/jhm.812","article-title":"Unplanned transfers to a medical intensive care unit: Causes and relationship to preventable errors in care","volume":"6","author":"Bapoje","year":"2011","journal-title":"J. Hosp. Med."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"273","DOI":"10.1016\/j.cnc.2018.02.009","article-title":"Advancing Continuous Predictive Analytics Monitoring: Moving from Implementation to Clinical Action in a Learning Health System","volume":"30","author":"Kitzmiller","year":"2018","journal-title":"Crit. Care Nurs. Clin. N. Am."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"1749","DOI":"10.1097\/CCM.0000000000000330","article-title":"Empiric antibiotic treatment reduces mortality in severe sepsis and septic shock from the first hour: Results from a guideline-based performance improvement program","volume":"42","author":"Ferrer","year":"2014","journal-title":"Crit. Care Med."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"1589","DOI":"10.1097\/01.CCM.0000217961.75225.E9","article-title":"Duration of hypotension before initiation of effective antimicrobial therapy is the critical determinant of survival in human septic shock","volume":"34","author":"Kumar","year":"2006","journal-title":"Crit. Care Med."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1016\/j.jbi.2016.09.013","article-title":"Development and validation of an electronic medical record-based alert score for detection of inpatient deterioration outside the ICU","volume":"64","author":"Kipnis","year":"2016","journal-title":"J. Biomed. Inform."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"S18","DOI":"10.1002\/jhm.2652","article-title":"Piloting electronic medical record\u2013based early detection of inpatient deterioration in community hospitals","volume":"11","author":"Escobar","year":"2016","journal-title":"J. Hosp. Med."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"e000234","DOI":"10.1136\/bmjresp-2017-000234","article-title":"Effect of a machine learning-based severe sepsis prediction algorithm on patient survival and hospital length of stay: A randomised clinical trial","volume":"4","author":"Shimabukuro","year":"2017","journal-title":"BMJ Open Respir. Res."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"e017833","DOI":"10.1136\/bmjopen-2017-017833","article-title":"Multicentre validation of a sepsis prediction algorithm using only vital sign data in the emergency department, general ward and ICU","volume":"8","author":"Mao","year":"2018","journal-title":"BMJ Open"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Horng, S., Sontag, D.A., Halpern, Y., Jernite, Y., Shapiro, N.I., and Nathanson, L.A. (2017). Creating an automated trigger for sepsis clinical decision support at emergency department triage using machine learning. PLoS ONE, 12.","DOI":"10.1371\/journal.pone.0174708"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"1485","DOI":"10.1097\/CCM.0000000000003891","article-title":"A Machine Learning Algorithm to Predict Severe Sepsis and Septic Shock","volume":"47","author":"Giannini","year":"2019","journal-title":"Crit. Care Med."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"1477","DOI":"10.1097\/CCM.0000000000003803","article-title":"Clinician Perception of a Machine Learning\u2013Based Early Warning System Designed to Predict Severe Sepsis and Septic Shock","volume":"47","author":"Ginestra","year":"2019","journal-title":"Crit. Care Med."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"e884","DOI":"10.1097\/CCM.0000000000004494","article-title":"A Time-Phased Machine Learning Model for Real-Time Prediction of Sepsis in Critical Care","volume":"48","author":"Li","year":"2020","journal-title":"Crit. Care Med."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"1181","DOI":"10.1007\/s00134-021-06506-y","article-title":"Surviving sepsis campaign: International guidelines for management of sepsis and septic shock 2021","volume":"47","author":"Evans","year":"2021","journal-title":"Intensive Care Med."}],"container-title":["Journal of Clinical Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2077-0383\/14\/2\/350\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,8]],"date-time":"2025-10-08T10:25:09Z","timestamp":1759919109000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2077-0383\/14\/2\/350"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,1,8]]},"references-count":41,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2025,1]]}},"alternative-id":["jcm14020350"],"URL":"https:\/\/doi.org\/10.3390\/jcm14020350","relation":{},"ISSN":["2077-0383"],"issn-type":[{"value":"2077-0383","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,1,8]]}}}