{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,17]],"date-time":"2025-10-17T12:41:20Z","timestamp":1760704880746,"version":"build-2065373602"},"reference-count":38,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2025,10,13]],"date-time":"2025-10-13T00:00:00Z","timestamp":1760313600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100002347","name":"German Federal Ministry of Education and Research","doi-asserted-by":"publisher","award":["16SV9030"],"award-info":[{"award-number":["16SV9030"]}],"id":[{"id":"10.13039\/501100002347","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>Background: Accurate prediction of postoperative care requirements is critical for patient safety and resource allocation. Although numerous approaches involving artificial intelligence (AI) and machine learning (ML) have been proposed to support such predictions, their implementation in practice has so far been insufficiently successful. One reason for this is that the performance of the algorithms is difficult to assess in practical use, as the accuracy of clinical decisions has not yet been systematically quantified. As a result, models are often assessed purely from a technical perspective, neglecting the socio-technical context. Methods: We conducted a retrospective, single-center observational study at the University Hospital Augsburg, including 35,488 elective surgical cases documented between August 2023 and January 2025. For each case, preoperative care-level predictions by surgical and anesthesiology teams were compared with the actual postoperative care provided. Predictive performance was evaluated using accuracy and sensitivity. Since this is a highly imbalanced dataset, in addition to sensitivity and specificity, the balanced accuracy and the F\u03b2-score were also calculated. The results were contrasted with published Machine-Learning (ML)-based approaches. Results: Overall prediction accuracy was high (surgery: 91.2%; anesthesiology: 87.1%). However, sensitivity for identifying patients requiring postoperative intensive care was markedly lower than reported for ML models in the literature, with the largest discrepancies observed in patients ultimately admitted to the ICU (surgery: 38.05%; anesthesiology: 56.84%; ML: 70%). Nevertheless, clinical judgment demonstrated a superior F1-score, indicating a more balanced performance between sensitivity and precision (surgery: 0.527; anesthesiology: 0.551; ML: 0.28). Conclusions: This study provides the first real-world benchmark of clinical expertise in postoperative care prediction and shows a way in which modern ML approaches must be evaluated in a specific sociotechnical context. By quantifying the predictive performance of surgeons and anesthesiologists, it enables an evaluation of existing ML approaches. Thus the strength of our work is the provision of a real-world benchmark against which all ML methods for preoperative prediction of ICU demand can be systematically evaluated. This enables, for the first time, a comparison of different approaches on a common, practice-oriented basis and thus significantly facilitates translation into clinical practice, thereby closing the translational gap. Furthermore it offers a data-driven framework to support the integration of ML into preoperative decision-making.<\/jats:p>","DOI":"10.3390\/info16100888","type":"journal-article","created":{"date-parts":[[2025,10,17]],"date-time":"2025-10-17T11:39:34Z","timestamp":1760701174000},"page":"888","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Prediction of Postoperative ICU Requirements: Closing the Translational Gap with a Real-World Clinical Benchmark for Artificial Intelligence Approaches"],"prefix":"10.3390","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2323-2739","authenticated-orcid":false,"given":"Alexander","family":"Althammer","sequence":"first","affiliation":[{"name":"Anaesthesiology and Operative Intensive Care, Faculty of Medicine, University of Augsburg, 86156 Augsburg, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2826-1363","authenticated-orcid":false,"given":"Felix","family":"Berger","sequence":"additional","affiliation":[{"name":"Anaesthesiology and Operative Intensive Care, Faculty of Medicine, University of Augsburg, 86156 Augsburg, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-9086-9217","authenticated-orcid":false,"given":"Oliver","family":"Spring","sequence":"additional","affiliation":[{"name":"Anaesthesiology and Operative Intensive Care, Faculty of Medicine, University of Augsburg, 86156 Augsburg, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2696-3254","authenticated-orcid":false,"given":"Philipp","family":"Simon","sequence":"additional","affiliation":[{"name":"Anaesthesiology and Operative Intensive Care, Faculty of Medicine, University of Augsburg, 86156 Augsburg, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8371-6463","authenticated-orcid":false,"given":"Felix","family":"Girrbach","sequence":"additional","affiliation":[{"name":"Anaesthesiology and Operative Intensive Care, Faculty of Medicine, University of Augsburg, 86156 Augsburg, Germany"}]},{"given":"Maximilian","family":"Dieing","sequence":"additional","affiliation":[{"name":"Faculty of Business and Economics, University of Augsburg, Universit\u00e4tsstra\u00dfe 2, 86159 Augsburg, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2700-4795","authenticated-orcid":false,"given":"Jens O.","family":"Brunner","sequence":"additional","affiliation":[{"name":"Faculty of Business and Economics, University of Augsburg, Universit\u00e4tsstra\u00dfe 2, 86159 Augsburg, Germany"},{"name":"Department of Technology, Management, and Economics, Technical University of Denmark, Anker Engelunds Vej 1, Bygning 101A, 2800 Kongens Lyngby, Denmark"},{"name":"Center for Excellence in Healthcare Operations Planning, Next Generation Technology, Region Zealand, \u00c6rtekildevej 1, 4100 Ringsted, Denmark"}]},{"given":"Sergey","family":"Shmygalev","sequence":"additional","affiliation":[{"name":"Anaesthesiology and Operative Intensive Care, Faculty of Medicine, University of Augsburg, 86156 Augsburg, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8300-1249","authenticated-orcid":false,"given":"Christina C.","family":"Bartenschlager","sequence":"additional","affiliation":[{"name":"Anaesthesiology and Operative Intensive Care, Faculty of Medicine, University of Augsburg, 86156 Augsburg, Germany"},{"name":"Applied Data Science in Health Care, N\u00fcrnberg School of Health, Ohm University of Applied Sciences Nuremberg, 90489 Nuremberg, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6144-3874","authenticated-orcid":false,"given":"Axel R.","family":"Heller","sequence":"additional","affiliation":[{"name":"Anaesthesiology and Operative Intensive Care, Faculty of Medicine, University of Augsburg, 86156 Augsburg, Germany"}]}],"member":"1968","published-online":{"date-parts":[[2025,10,13]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"644","DOI":"10.1097\/00000539-200203000-00030","article-title":"The effect of outpatient preoperative evaluation of hospital inpatients on cancellation of surgery and length of hospital stay","volume":"94","author":"Moons","year":"2002","journal-title":"Anesth. Analg."},{"key":"ref_2","first-page":"30","article-title":"Financial cost of elective day of surgery cancellations","volume":"7","author":"Turunen","year":"2018","journal-title":"J. Hosp. Adm."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"753","DOI":"10.1213\/ANE.0000000000003946","article-title":"The Impact of Postoperative Intensive Care Unit Admission on Postoperative Hospital Length of Stay and Costs: A Prespecified Propensity-Matched Cohort Study","volume":"129","author":"Thevathasan","year":"2019","journal-title":"Anesth. Analg."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1133","DOI":"10.1097\/SLA.0000000000003297","article-title":"Utilizing Machine Learning Methods for Preoperative Prediction of Postsurgical Mortality and Intensive Care Unit Admission","volume":"272","author":"Chiew","year":"2020","journal-title":"Ann. Surg."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"455","DOI":"10.1016\/j.ejso.2019.11.513","article-title":"Impact of delay to surgery on survival in stage I-III colon cancer","volume":"46","author":"Grass","year":"2020","journal-title":"Eur. J. Surg. Oncol."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"651","DOI":"10.1177\/0194599818803330","article-title":"Impact of Delay in Treatment Initiation on Overall Survival in Laryngeal Cancers","volume":"160","author":"Kompelli","year":"2019","journal-title":"Otolaryngol. Head Neck Surg."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Nothofer, S., Geipel, J., Aehling, K., Sommer, B., Heller, A.R., Shiban, E., and Simon, P. (2025). Postoperative Surveillance in the Postoperative vs. Intensive Care Unit for Patients Undergoing Elective Supratentorial Brain Tumor Removal: A Retrospective Observational Study. J. Clin. Med., 14.","DOI":"10.3390\/jcm14082632"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1186\/s13613-016-0129-5","article-title":"SAPS 3 score as a predictive factor for postoperative referral to intensive care unit","volume":"6","author":"Silva","year":"2016","journal-title":"Ann. Intensive Care"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"959","DOI":"10.1097\/ALN.0b013e3182a4e94d","article-title":"Risk stratification tools for predicting morbidity and mortality in adult patients undergoing major surgery: Qualitative systematic review","volume":"119","author":"Moonesinghe","year":"2013","journal-title":"Anesthesiology"},{"key":"ref_10","first-page":"e193","article-title":"An international prospective cohort study evaluating major vascular complications among patients undergoing noncardiac surgery: The VISION Pilot Study","volume":"5","author":"Devereaux","year":"2011","journal-title":"Open Med."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"167","DOI":"10.1016\/j.jocn.2022.10.029","article-title":"Use of random forest machine learning algorithm to predict short term outcomes following posterior cervical decompression with instrumented fusion","volume":"107","author":"Cabrera","year":"2023","journal-title":"J. Clin. Neurosci."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"790","DOI":"10.1016\/j.surg.2021.05.005","article-title":"Predicting need for hospital-specific interventional care after surgery using electronic health record data","volume":"170","author":"Verhoef","year":"2021","journal-title":"Surgery"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1052","DOI":"10.1001\/jama.2020.0592","article-title":"Effect of a Machine Learning-Derived Early Warning System for Intraoperative Hypotension vs Standard Care on Depth and Duration of Intraoperative Hypotension During Elective Noncardiac Surgery: The HYPE Randomized Clinical Trial","volume":"323","author":"Wijnberge","year":"2020","journal-title":"JAMA"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1186\/s13741-025-00544-6","article-title":"A warning model for predicting patient admissions to the intensive care unit (ICU) following surgery","volume":"14","author":"Li","year":"2025","journal-title":"Perioper. Med."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"207","DOI":"10.1016\/j.hrtlng.2023.08.001","article-title":"Development and validation of a prediction model for postoperative intensive care unit admission in patients with non-cardiac surgery","volume":"62","author":"Xu","year":"2023","journal-title":"Heart Lung"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"259","DOI":"10.23736\/S0375-9393.24.18587-2","article-title":"Effectiveness of ChatGPT-4 in predicting the human decision to send patients to the postoperative intensive care unit: A prospective multicentric study","volume":"91","author":"Turan","year":"2025","journal-title":"Minerva Anestesiol."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"111810","DOI":"10.1016\/j.jclinane.2025.111810","article-title":"Predicting admission to and length of stay in intensive care units after general anesthesia: Time-dependent role of pre- and intraoperative data for clinical decision-making","volume":"103","author":"Stieger","year":"2025","journal-title":"J. Clin. Anesth."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"293","DOI":"10.1186\/s40001-025-02553-z","article-title":"Prediction of postoperative intensive care unit admission with artificial intelligence models in non-small cell lung carcinoma","volume":"30","author":"Kaynak","year":"2025","journal-title":"Eur. J. Med. Res."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"226","DOI":"10.1016\/j.cursur.2006.02.005","article-title":"Risk scoring in perioperative and surgical intensive care patients: A review","volume":"63","author":"Hariharan","year":"2006","journal-title":"Curr. Surg."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1276","DOI":"10.1097\/ALN.0b013e318293065c","article-title":"Development and validation of a score for prediction of postoperative respiratory complications","volume":"118","author":"Brueckmann","year":"2013","journal-title":"Anesthesiology"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"505","DOI":"10.1097\/ALN.0b013e3181979440","article-title":"Development and validation of an acute kidney injury risk index for patients undergoing general surgery: Results from a national data set","volume":"110","author":"Kheterpal","year":"2009","journal-title":"Anesthesiology"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"344","DOI":"10.1001\/jamasurg.2021.7580","article-title":"Development and Validation of a Multivariable Prediction Model for Postoperative Intensive Care Unit Stay in a Broad Surgical Population","volume":"157","author":"Rozeboom","year":"2022","journal-title":"JAMA Surg."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1799","DOI":"10.1007\/s00134-019-05775-y","article-title":"Post-operative intensive care: Is it really necessary?","volume":"45","author":"Cashmore","year":"2019","journal-title":"Intensive Care Med."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"S2055","DOI":"10.21037\/jtd.2018.05.156","article-title":"Who benefits from postoperative ICU admissions?-more research is needed","volume":"10","author":"Park","year":"2018","journal-title":"J. Thorac. Dis."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Waydhas, E., Herting, S., Kluge, A., Markewitz, G., Marx, E., Muhl, T., Nicolai, K., and Notz, V. (2017). Intermediate Care Station Empfehlungen zur Ausstattung und Struktur, Deutsche Interdisziplin\u00e4re Vereinigung f\u00fcr Intensiv-und Notfallmedizin eV (DIVI).","DOI":"10.1007\/s00063-017-0369-7"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"115","DOI":"10.1038\/nature21056","article-title":"Dermatologist-level classification of skin cancer with deep neural networks","volume":"542","author":"Esteva","year":"2017","journal-title":"Nature"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1016\/j.ejca.2019.05.023","article-title":"Deep neural networks are superior to dermatologists in melanoma image classification","volume":"119","author":"Brinker","year":"2019","journal-title":"Eur. J. Cancer"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Rajpurkar, P., Irvin, J., Ball, R.L., Zhu, K., Yang, B., Mehta, H., Duan, T., Ding, D., Bagul, A., and Langlotz, C.P. (2018). Deep learning for chest radiograph diagnosis: A retrospective comparison of the CheXNeXt algorithm to practicing radiologists. PLoS Med., 15.","DOI":"10.1371\/journal.pmed.1002686"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"1636","DOI":"10.1097\/PAS.0000000000001151","article-title":"Impact of Deep Learning Assistance on the Histopathologic Review of Lymph Nodes for Metastatic Breast Cancer","volume":"42","author":"Steiner","year":"2018","journal-title":"Am. J. Surg. Pathol."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"148","DOI":"10.1001\/jamasurg.2019.4917","article-title":"Artificial Intelligence and Surgical Decision-making","volume":"155","author":"Loftus","year":"2020","journal-title":"JAMA Surg."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"345","DOI":"10.1038\/s41746-025-01725-9","article-title":"Artificial intelligence should genuinely support clinical reasoning and decision making to bridge the translational gap","volume":"8","author":"Sokol","year":"2025","journal-title":"npj Digit. Med."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"85","DOI":"10.1097\/ALN.0000000000004764","article-title":"Prediction of Complications and Prognostication in Perioperative Medicine: A Systematic Review and PROBAST Assessment of Machine Learning Tools","volume":"140","author":"Arina","year":"2024","journal-title":"Anesthesiology"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"549","DOI":"10.1038\/s41746-025-01952-0","article-title":"AI assisted prediction of unplanned intensive care admissions using natural language processing in elective neurosurgery","volume":"8","author":"Ive","year":"2025","journal-title":"npj Digit. Med."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Cao, Y., Wang, Y., Liu, H., and Wu, L. (2025). Artificial intelligence revolutionizing anesthesia management: Advances and prospects in intelligent anesthesia technology. Front. Med., 12.","DOI":"10.3389\/fmed.2025.1571725"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"1066","DOI":"10.23736\/S0375-9393.22.16739-8","article-title":"Artificial intelligence in intensive care: Moving towards clinical decision support systems","volume":"88","author":"Montomoli","year":"2022","journal-title":"Minerva Anestesiol."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Sokolova, M., Japkowicz, N., and Szpakowicz, S. (2006, January 4\u20138). Beyond Accuracy, F-Score and ROC: A Family of Discriminant Measures for Performance Evaluation. Proceedings of the AI 2006: Advances in Artificial Intelligence: 19th Australian Joint Conference on Artificial Intelligence, Hobart, Australia.","DOI":"10.1007\/11941439_114"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Hicks, S.A., Str\u00fcmke, I., Thambawita, V., Hammou, M., Riegler, M.A., Halvorsen, P., and Parasa, S. (2022). On evaluation metrics for medical applications of artificial intelligence. Sci. Rep., 12.","DOI":"10.1038\/s41598-022-09954-8"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"652","DOI":"10.1097\/SLA.0000000000002706","article-title":"MySurgeryRisk: Development and Validation of a Machine-learning Risk Algorithm for Major Complications and Death After Surgery","volume":"269","author":"Bihorac","year":"2019","journal-title":"Ann. Surg."}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/16\/10\/888\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,17]],"date-time":"2025-10-17T12:03:16Z","timestamp":1760702596000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/16\/10\/888"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,13]]},"references-count":38,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2025,10]]}},"alternative-id":["info16100888"],"URL":"https:\/\/doi.org\/10.3390\/info16100888","relation":{},"ISSN":["2078-2489"],"issn-type":[{"value":"2078-2489","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,10,13]]}}}