{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,11]],"date-time":"2026-07-11T03:27:40Z","timestamp":1783740460077,"version":"3.55.0"},"reference-count":53,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2022,2,12]],"date-time":"2022-02-12T00:00:00Z","timestamp":1644624000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Institute of Health","award":["RO1-GM117622"],"award-info":[{"award-number":["RO1-GM117622"]}]},{"name":"National Institute of Health","award":["R01-NR013912"],"award-info":[{"award-number":["R01-NR013912"]}]},{"name":"Department of Defence","award":["W81XWH-19-C-0101"],"award-info":[{"award-number":["W81XWH-19-C-0101"]}]},{"DOI":"10.13039\/100000185","name":"Defense Advanced Research Projects Agency","doi-asserted-by":"publisher","award":["FA8750-17-2-0130"],"award-info":[{"award-number":["FA8750-17-2-0130"]}],"id":[{"id":"10.13039\/100000185","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Early recognition of pathologic cardiorespiratory stress and forecasting cardiorespiratory decompensation in the critically ill is difficult even in highly monitored patients in the Intensive Care Unit (ICU). Instability can be intuitively defined as the overt manifestation of the failure of the host to adequately respond to cardiorespiratory stress. The enormous volume of patient data available in ICU environments, both of high-frequency numeric and waveform data accessible from bedside monitors, plus Electronic Health Record (EHR) data, presents a platform ripe for Artificial Intelligence (AI) approaches for the detection and forecasting of instability, and data-driven intelligent clinical decision support (CDS). Building unbiased, reliable, and usable AI-based systems across health care sites is rapidly becoming a high priority, specifically as these systems relate to diagnostics, forecasting, and bedside clinical decision support. The ICU environment is particularly well-positioned to demonstrate the value of AI in saving lives. The goal is to create AI models embedded in a real-time CDS for forecasting and mitigation of critical instability in ICU patients of sufficient readiness to be deployed at the bedside. Such a system must leverage multi-source patient data, machine learning, systems engineering, and human action expertise, the latter being key to successful CDS implementation in the clinical workflow and evaluation of bias. We present one approach to create an operationally relevant AI-based forecasting CDS system.<\/jats:p>","DOI":"10.3390\/s22041408","type":"journal-article","created":{"date-parts":[[2022,2,13]],"date-time":"2022-02-13T20:34:45Z","timestamp":1644784485000},"page":"1408","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":31,"title":["Intelligent Clinical Decision Support"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6166-700X","authenticated-orcid":false,"given":"Michael","family":"Pinsky","sequence":"first","affiliation":[{"name":"Department of Critical Care Medicine, University of Pittsburgh School of Medicine, Pittsburgh, PA 15261, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2372-0831","authenticated-orcid":false,"given":"Artur","family":"Dubrawski","sequence":"additional","affiliation":[{"name":"Auton Laboratory, School of Computer Science, Carnegie Mellon University, Pittsburgh, PA 15213, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gilles","family":"Clermont","sequence":"additional","affiliation":[{"name":"Department of Critical Care Medicine, University of Pittsburgh School of Medicine, Pittsburgh, PA 15261, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,2,12]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"180","DOI":"10.1111\/j.1365-2648.2006.03901.x","article-title":"Medical emergency teams: A strategy for improving patient care and nursing work environments","volume":"55","author":"Galhotra","year":"2006","journal-title":"J. Adv. Nurs."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"e140","DOI":"10.1097\/PTS.0000000000000720","article-title":"Failure to Rescue Deteriorating Patients: A Systematic Review of Root Causes and Improvement Strategies","volume":"18","author":"Burke","year":"2022","journal-title":"J. Patient Saf."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"S3","DOI":"10.1097\/PTS.0000000000000748","article-title":"The Use of Rapid Response Teams to Reduce Failure to Rescue Events: A Systematic Review","volume":"16","author":"Hall","year":"2020","journal-title":"J. Patient Saf."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1477","DOI":"10.1097\/CCM.0000000000003803","article-title":"Clinician Perception of a Machine Learning\u2013Based Early Warning System Designed to Predict Severe Sepsis and Septic Shock","volume":"47","author":"Ginestra","year":"2019","journal-title":"Crit. Care Med."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1065","DOI":"10.1001\/jamainternmed.2021.2626","article-title":"External Validation of a Widely Imple-mented Proprietary Sepsis Prediction Model in Hospitalized Patients","volume":"181","author":"Wong","year":"2021","journal-title":"JAMA Intern. Med."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"566","DOI":"10.1186\/cc3927","article-title":"Functional hemodynamic monitoring","volume":"9","author":"Pinsky","year":"2005","journal-title":"Crit. Care."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"606","DOI":"10.1164\/rccm.201404-0716CP","article-title":"Gleaning Knowledge from Data in the Intensive Care Unit","volume":"190","author":"Pinsky","year":"2014","journal-title":"Am. J. Respir. Crit. Care Med."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"e0058","DOI":"10.1097\/CCE.0000000000000058","article-title":"Increasing Cardiovascular Data Sampling Frequency and Referencing It to Baseline Improve Hemorrhage Detection","volume":"1","author":"Wertz","year":"2019","journal-title":"Crit. Care Explor."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1097\/CCM.0b013e3181fb7b1c","article-title":"Cardiorespiratory instability before and after im-plementing an integrated monitoring system","volume":"39","author":"Hravnak","year":"2011","journal-title":"Crit. Care Med."},{"key":"ref_10","first-page":"813","article-title":"Heart-Rate Complexity for Prediction of Prehospital Lifesaving Interventions in Trauma Patients","volume":"65","author":"Cancio","year":"2008","journal-title":"J. Trauma Inj. Infect. Crit. Care"},{"key":"ref_11","first-page":"512","article-title":"Prehospital Loss of R-to-R Interval Complexity is Associated with Mortality in Trauma Patients","volume":"63","author":"Batchinsky","year":"2007","journal-title":"J. Trauma Inj. Infect. Crit. Care"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"30","DOI":"10.1038\/s41746-019-0107-z","article-title":"An algorithm strategy for precise patient monitoring in a connected healthcare enterprise","volume":"2","author":"Hu","year":"2019","journal-title":"NPJ Digit. Med."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"133","DOI":"10.1016\/j.ccc.2014.08.007","article-title":"Using what you get: Dynamic physiologic signatures of critical illness","volume":"31","author":"Holder","year":"2015","journal-title":"Crit. Care Clin."},{"key":"ref_14","first-page":"47","article-title":"Learning temporal rules to forecast instability in continuously monitored patients","volume":"24","author":"Dubrawski","year":"2016","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1013","DOI":"10.1016\/j.resuscitation.2011.03.006","article-title":"Centile-based early warning scores derived from statistical distributions of vital signs","volume":"82","author":"Tarassenko","year":"2011","journal-title":"Resuscitation"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"661","DOI":"10.1186\/s13054-020-03379-3","article-title":"Prediction of hypotension events with physi-ologic vital sign signatures in the intensive care unit","volume":"24","author":"Yoon","year":"2020","journal-title":"Crit. Care."},{"key":"ref_17","first-page":"1841","article-title":"Modelling Risk of Cardio-Respiratory Instability as a Heterogeneous Process","volume":"2015","author":"Chen","year":"2015","journal-title":"Annu. Symp. Am. Med. Inform. Assoc."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"384","DOI":"10.1513\/AnnalsATS.201611-905OC","article-title":"Dynamic and Personalized Risk Forecast in Step-Down Units. Implications for Monitoring Paradigms","volume":"14","author":"Chen","year":"2017","journal-title":"Ann. Am. Thorac. Soc."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"703","DOI":"10.1007\/s10877-018-0194-4","article-title":"Impact of predictive analytics based on continuous cardiorespiratory monitoring in a surgical and trauma intensive care unit","volume":"33","author":"Ruminski","year":"2018","journal-title":"Int. J. Clin. Monit. Comput."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"570","DOI":"10.1038\/pr.2013.136","article-title":"Septicemia mortality reduction in neonates in a heart rate characteristics monitoring trial","volume":"74","author":"Fairchild","year":"2013","journal-title":"Pediatr. Res."},{"key":"ref_21","unstructured":"Gisolfi, N. (2021). Model-Centric Verification of Artificial Intelligence. [Ph.D. Thesis, The Robotics Institute, Carnegie Mellon University]."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1136\/amiajnl-2011-000089","article-title":"Automation bias: A systematic review of frequency, effect mediators, and mitigators","volume":"19","author":"Goddard","year":"2012","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"652776","DOI":"10.3389\/frobt.2021.652776","article-title":"Adaptive Cognitive Mechanisms to Maintain Calibrated Trust and Reliance in Automation","volume":"8","author":"Lebiere","year":"2021","journal-title":"Front. Robot. AI"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"680","DOI":"10.1055\/s-0040-1709707","article-title":"Graphical Presentations of Clinical Data in a Learning Electronic Medical Record","volume":"11","author":"Calzoni","year":"2020","journal-title":"Appl. Clin. Inform."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"1544","DOI":"10.1001\/jamainternmed.2018.3763","article-title":"Potential Biases in Machine Learning Algorithms Using Electronic Health Record Data","volume":"178","author":"Gianfrancesco","year":"2018","journal-title":"JAMA Intern. Med."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1479","DOI":"10.1136\/bmj.k1479","article-title":"Biases in electronic health record data due to processes within the healthcare system: Ret-rospective observational study","volume":"361","author":"Agniel","year":"2018","journal-title":"BMJ"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"4:1","DOI":"10.1147\/JRD.2019.2942287","article-title":"AI Fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias","volume":"63","author":"Bellamy","year":"2019","journal-title":"IBM J. Res. Dev."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"51","DOI":"10.7326\/M18-1376","article-title":"PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies","volume":"170","author":"Wolff","year":"2019","journal-title":"Ann. Intern. Med."},{"key":"ref_29","unstructured":"Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., and Ghani, R. (2018). Aequitas: A Bias and Fairness Audit Toolkit. arXiv."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"2377","DOI":"10.1001\/jama.2019.18058","article-title":"Addressing Bias in Artificial Intelligence in Health Care","volume":"322","author":"Parikh","year":"2019","journal-title":"JAMA J. Am. Med. Assoc."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"22","DOI":"10.1513\/AnnalsATS.201808-564PS","article-title":"Control of Confounding and Reporting of Results in Causal Inference Studies. Guidance for Authors from Editors of Respiratory, Sleep, and Critical Care Journals","volume":"16","author":"Lederer","year":"2019","journal-title":"Ann. Am. Thorac. Soc."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1097\/CCM.0000000000003531","article-title":"Causal Inference from Observational Data: New Guidance from Pulmonary, Critical Care, and Sleep Journals","volume":"47","author":"Maslove","year":"2019","journal-title":"Crit. Care Med."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"100185","DOI":"10.1016\/j.resplu.2021.100185","article-title":"Sex differences in post cardiac arrest discharge locations","volume":"8","author":"Jeanselme","year":"2021","journal-title":"Resusc. Plus"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"150","DOI":"10.1007\/s00134-020-06295-w","article-title":"Artificial intelligence in telemetry: What clinicians should know","volume":"47","author":"Maslove","year":"2021","journal-title":"Intensiv. Care Med."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"1951","DOI":"10.1056\/NEJMsa2001090","article-title":"Automated Identification of Adults at Risk for In-Hospital Clinical Deterioration","volume":"383","author":"Escobar","year":"2020","journal-title":"N. Engl. J. Med."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"649","DOI":"10.1164\/rccm.201406-1022OC","article-title":"Multicenter Development and Validation of a Risk Stratification Tool for Ward Patients","volume":"190","author":"Churpek","year":"2014","journal-title":"Am. J. Respir. Crit. Care Med."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"1059","DOI":"10.1097\/SLA.0000000000002665","article-title":"Validating the Electronic Cardiac Arrest Risk Triage (eCART) Score for Risk Stratification of Surgical Inpatients in the Postoperative Setting: Retrospective Cohort Study","volume":"269","author":"Bartkowiak","year":"2019","journal-title":"Ann. Surg."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"1052","DOI":"10.1001\/jama.2020.0592","article-title":"Effect of a Machine Learning-Derived Early Warning System for Intraoperative Hypotension vs. Standard Care on Depth and Duration of Intraoperative Hypotension during Elective Noncardiac Surgery: The HYPE Randomized Clinical Trial","volume":"323","author":"Wijnberge","year":"2020","journal-title":"JAMA"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"055010","DOI":"10.1088\/1361-6579\/ab8cb2","article-title":"Development of hemorrhage identification model using non-invasive vital signs","volume":"41","author":"Chen","year":"2020","journal-title":"Physiol. Meas."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Chen, Y., Hong, C., Pinsky, M.R., Ma, T., and Clermont, G. (2020). Estimating Surgical Blood Loss Volume Using Continuously Monitored Vital Signs. Sensors, 20.","DOI":"10.3390\/s20226558"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"1176","DOI":"10.1213\/ANE.0000000000004564","article-title":"Parsimony of Hemodynamic Monitoring Data Sufficient for the Detection of Hemorrhage","volume":"130","author":"Pinsky","year":"2020","journal-title":"Anesth. Analg."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"160035","DOI":"10.1038\/sdata.2016.35","article-title":"MIMIC-III, a freely accessible critical care database","volume":"3","author":"Johnson","year":"2016","journal-title":"Sci. Data"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"973","DOI":"10.1007\/s10877-019-00277-0","article-title":"Predicting tachycardia as a surrogate for instability in the intensive care unit","volume":"33","author":"Yoon","year":"2019","journal-title":"Int. J. Clin. Monit. Comput."},{"key":"ref_44","first-page":"S389","article-title":"Multi-Tier ground truth Elicitation Framework with Application to Artifact Classification for Predicting Patient Instability","volume":"40","author":"Wang","year":"2014","journal-title":"Intensive Care Med."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"A769","DOI":"10.1186\/2197-425X-3-S1-A769","article-title":"Semi automated adjudication of vital sign alerts in step-down units","volume":"3","author":"Fiterau","year":"2015","journal-title":"Intensive Care Med. Exp."},{"key":"ref_46","first-page":"S289","article-title":"Artifact adjudication for vital sign step-down unit data can be improved using Active Learning with low-dimensional models","volume":"40","author":"Fiterau","year":"2014","journal-title":"Intensive Care Med."},{"key":"ref_47","unstructured":"Xu, Y., Zhang, H., Miller, K., Singh, A., and Dubrawski, A. (2017, January 4\u20139). Noise-Tolerant Interactive Learning Using Pairwise Comparisons. Proceedings of the31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"606","DOI":"10.1097\/01.ccm.0000552003.61806.05","article-title":"Using Comparisons to Reduce Cost of Data Annotation Required to Train Models for Bedside Monitoring","volume":"47","author":"Sheng","year":"2019","journal-title":"Crit. Care Med."},{"key":"ref_49","unstructured":"Goswami, M., Boecking, B., and Dubrawski, A. (2021). Weak Supervision for Affordable Modeling of Electrocardiogram Data. arXiv."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"104643","DOI":"10.1016\/j.ijmedinf.2021.104643","article-title":"Engaging Clinicians Early During the Development of a Graphical User Display of An Intelligent Alerting System at the Bedside","volume":"159","author":"Helman","year":"2022","journal-title":"Internat. J. Med. Inform."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s12916-019-1426-2","article-title":"Key challenges for delivering clinical impact with artificial intelligence","volume":"17","author":"Kelly","year":"2019","journal-title":"BMC Med."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"1351","DOI":"10.1001\/jama.2019.10306","article-title":"Making Machine Learning Models Clinically Useful","volume":"322","author":"Shah","year":"2019","journal-title":"JAMA J. Am. Med. Assoc."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"065002","DOI":"10.1088\/1361-6579\/abfc9b","article-title":"The critical care data exchange format: A proposed flexible data standard for combining clinical and high-frequency physiologic data in critical care","volume":"42","author":"Laird","year":"2021","journal-title":"Physiol. Meas."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/4\/1408\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:18:06Z","timestamp":1760134686000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/4\/1408"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,2,12]]},"references-count":53,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2022,2]]}},"alternative-id":["s22041408"],"URL":"https:\/\/doi.org\/10.3390\/s22041408","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,2,12]]}}}