{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,7]],"date-time":"2026-05-07T23:48:46Z","timestamp":1778197726850,"version":"3.51.4"},"reference-count":45,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2021,2,19]],"date-time":"2021-02-19T00:00:00Z","timestamp":1613692800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,2,19]],"date-time":"2021-02-19T00:00:00Z","timestamp":1613692800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The aim of this work was to develop and evaluate the reinforcement learning algorithm VentAI, which is able to suggest a dynamically optimized mechanical ventilation regime for critically-ill patients. We built, validated and tested its performance on 11,943 events of volume-controlled mechanical ventilation derived from 61,532 distinct ICU admissions and tested it on an independent, secondary dataset (200,859 ICU stays; 25,086 mechanical ventilation events). A patient \u201cdata fingerprint\u201d of 44 features was extracted as multidimensional time series in 4-hour time steps. We used a Markov decision process, including a reward system and a Q-learning approach, to find the optimized settings for positive end-expiratory pressure (PEEP), fraction of inspired oxygen (FiO<jats:sub>2<\/jats:sub>) and ideal body weight-adjusted tidal volume (Vt). The observed outcome was in-hospital or 90-day mortality. VentAI reached a significantly increased estimated performance return of 83.3 (primary dataset) and 84.1 (secondary dataset) compared to physicians\u2019 standard clinical care (51.1). The number of recommended action changes per mechanically ventilated patient constantly exceeded those of the clinicians. VentAI chose 202.9% more frequently ventilation regimes with lower Vt (5\u20137.5\u2009mL\/kg), but 50.8% less for regimes with higher Vt (7.5\u201310\u2009mL\/kg). VentAI recommended 29.3% more frequently PEEP levels of 5\u20137\u2009cm H<jats:sub>2<\/jats:sub>O and 53.6% more frequently PEEP levels of 7\u20139 cmH<jats:sub>2<\/jats:sub>O. VentAI avoided high (&gt;55%) FiO<jats:sub>2<\/jats:sub> values (59.8% decrease), while preferring the range of 50\u201355% (140.3% increase). In conclusion, VentAI provides reproducible high performance by dynamically choosing an optimized, individualized ventilation strategy and thus might be of benefit for critically ill patients.<\/jats:p>","DOI":"10.1038\/s41746-021-00388-6","type":"journal-article","created":{"date-parts":[[2021,2,20]],"date-time":"2021-02-20T16:02:53Z","timestamp":1613836973000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":118,"title":["Development and validation of a reinforcement learning algorithm to dynamically optimize mechanical ventilation in critical care"],"prefix":"10.1038","volume":"4","author":[{"given":"Arne","family":"Peine","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3932-4873","authenticated-orcid":false,"given":"Ahmed","family":"Hallawa","sequence":"additional","affiliation":[]},{"given":"Johannes","family":"Bickenbach","sequence":"additional","affiliation":[]},{"given":"Guido","family":"Dartmann","sequence":"additional","affiliation":[]},{"given":"Lejla Begic","family":"Fazlic","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9929-2925","authenticated-orcid":false,"given":"Anke","family":"Schmeink","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4068-3558","authenticated-orcid":false,"given":"Gerd","family":"Ascheid","sequence":"additional","affiliation":[]},{"given":"Christoph","family":"Thiemermann","sequence":"additional","affiliation":[]},{"given":"Andreas","family":"Schuppert","sequence":"additional","affiliation":[]},{"given":"Ryan","family":"Kindle","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6712-6626","authenticated-orcid":false,"given":"Leo","family":"Celi","sequence":"additional","affiliation":[]},{"given":"Gernot","family":"Marx","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8650-5090","authenticated-orcid":false,"given":"Lukas","family":"Martin","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2021,2,19]]},"reference":[{"key":"388_CR1","doi-asserted-by":"publisher","first-page":"41","DOI":"10.1097\/SHK.0000000000000702","volume":"47","author":"FG Zampieri","year":"2017","unstructured":"Zampieri, F. G. & Mazza, B. Mechanical ventilation in sepsis: a reappraisal. Shock 47, 41\u201346 (2017).","journal-title":"Shock"},{"key":"388_CR2","doi-asserted-by":"publisher","first-page":"1872","DOI":"10.1001\/jama.2018.14280","volume":"320","author":"Writing Group for the PReVENT Investigators et al.","year":"2018","unstructured":"Writing Group for the PReVENT Investigators et al. Effect of a low vs intermediate tidal volume strategy on ventilator-free days in intensive care unit patients without ARDS: a randomized clinical trial. JAMA 320, 1872\u20131880 (2018).","journal-title":"JAMA"},{"key":"388_CR3","doi-asserted-by":"publisher","first-page":"2126","DOI":"10.1056\/NEJMra1208707","volume":"369","author":"AS Slutsky","year":"2013","unstructured":"Slutsky, A. S. & Ranieri, V. M. Ventilator-induced lung injury. N. Engl. J. Med. 369, 2126\u20132136 (2013).","journal-title":"N. Engl. J. Med."},{"key":"388_CR4","doi-asserted-by":"publisher","first-page":"66","DOI":"10.1097\/ALN.0000000000000706","volume":"123","author":"A Serpa Neto","year":"2015","unstructured":"Serpa Neto, A. et al. Protective versus conventional ventilation for surgery: a systematic review and individual patient data meta-analysis. Anesthesiology 123, 66\u201378 (2015).","journal-title":"Anesthesiology"},{"key":"388_CR5","doi-asserted-by":"publisher","first-page":"183","DOI":"10.1186\/s13054-017-1750-x","volume":"21","author":"L Gattinoni","year":"2017","unstructured":"Gattinoni, L. et al. The future of mechanical ventilation: lessons from the present and the past. Crit. Care Lond. Engl. 21, 183 (2017).","journal-title":"Crit. Care Lond. Engl."},{"key":"388_CR6","doi-asserted-by":"publisher","first-page":"1519","DOI":"10.1164\/rccm.201708-1629CI","volume":"196","author":"SK Sahetya","year":"2017","unstructured":"Sahetya, S. K., Mancebo, J. & Brower, R. G. Fifty years of research in ARDS. Vt selection in acute respiratory distress syndrome. Am. J. Respir. Crit. Care Med. 196, 1519\u20131525 (2017).","journal-title":"Am. J. Respir. Crit. Care Med."},{"key":"388_CR7","doi-asserted-by":"publisher","first-page":"847","DOI":"10.1007\/s00134-012-2787-6","volume":"39","author":"T Bein","year":"2013","unstructured":"Bein, T. et al. Lower tidal volume strategy (\u22483 ml\/kg) combined with extracorporeal CO2 removal versus \u2018conventional\u2019 protective ventilation (6 ml\/kg) in severe ARDS: the prospective randomized Xtravent-study. Intensive Care Med. 39, 847\u2013856 (2013).","journal-title":"Intensive Care Med."},{"key":"388_CR8","doi-asserted-by":"publisher","unstructured":"Combes, A., Fanelli, V., Pham, T., Ranieri, V. M. & European Society of Intensive Care Medicine Trials Group and the \u201cStrategy of Ultra-Protective lung ventilation with Extracorporeal CO2 Removal for New-Onset moderate to severe ARDS\u201d (SUPERNOVA) investigators. Feasibility and safety of extracorporeal CO2 removal to enhance protective ventilation in acute respiratory distress syndrome: the SUPERNOVA study. Intensive Care Med. (2019) https:\/\/doi.org\/10.1007\/s00134-019-05567-4.","DOI":"10.1007\/s00134-019-05567-4"},{"key":"388_CR9","doi-asserted-by":"publisher","first-page":"44","DOI":"10.1038\/s41591-018-0300-7","volume":"25","author":"EJ Topol","year":"2019","unstructured":"Topol, E. J. High-performance medicine: the convergence of human and artificial intelligence. Nat. Med. 25, 44 (2019).","journal-title":"Nat. Med."},{"key":"388_CR10","doi-asserted-by":"publisher","first-page":"1716","DOI":"10.1038\/s41591-018-0213-5","volume":"24","author":"M Komorowski","year":"2018","unstructured":"Komorowski, M., Celi, L. A., Badawi, O., Gordon, A. C. & Faisal, A. A. The Artificial Intelligence Clinician learns optimal treatment strategies for sepsis in intensive care. Nat. Med. 24, 1716 (2018).","journal-title":"Nat. Med."},{"key":"388_CR11","doi-asserted-by":"crossref","unstructured":"Sutton, R. S. & Barto, A. G. Reinforcement Learning: An Introduction. (A Bradford Book, 1998).","DOI":"10.1109\/TNN.1998.712192"},{"key":"388_CR12","doi-asserted-by":"publisher","first-page":"16","DOI":"10.1038\/s41591-018-0310-5","volume":"25","author":"O Gottesman","year":"2019","unstructured":"Gottesman, O. et al. Guidelines for reinforcement learning in healthcare. Nat. Med. 25, 16 (2019).","journal-title":"Nat. Med."},{"key":"388_CR13","doi-asserted-by":"publisher","DOI":"10.1038\/sdata.2016.35","volume":"3","author":"AEW Johnson","year":"2016","unstructured":"Johnson, A. E. W. et al. MIMIC-III, a freely accessible critical care database. Sci. Data 3, 160035 (2016).","journal-title":"Sci. Data"},{"key":"388_CR14","doi-asserted-by":"publisher","DOI":"10.1038\/sdata.2018.178","volume":"5","author":"TJ Pollard","year":"2018","unstructured":"Pollard, T. J. et al. The eICU Collaborative Research Database, a freely available multi-center database for critical care research. Sci. Data 5, 180178 (2018).","journal-title":"Sci. Data"},{"key":"388_CR15","unstructured":"Precup, D., Sutton, R. S. & Dasgupta, S. Off-policy temporal difference learning with function approximation. Proceedings of the Eighteenth International Conference on Machine Learning. Morgan Kaufmann Publishers Inc. pp. 417\u2013424 (San Francisco, CA, USA, 2001)."},{"key":"388_CR16","doi-asserted-by":"publisher","first-page":"205","DOI":"10.4236\/ojs.2011.13024","volume":"01","author":"MW Mitchell","year":"2011","unstructured":"Mitchell, M. W. Bias of the random forest out-of-bag (OOB) error for certain input parameters. Open J. Stat. 01, 205 (2011).","journal-title":"Open J. Stat."},{"key":"388_CR17","doi-asserted-by":"publisher","first-page":"1311","DOI":"10.1097\/01.CCM.0000215598.84885.01","volume":"34","author":"J Villar","year":"2006","unstructured":"Villar, J., Kacmarek, R. M., P\u00e9rez-M\u00e9ndez, L. & Aguirre-Jaime, A. A high positive end-expiratory pressure, low tidal volume ventilatory strategy improves outcome in persistent acute respiratory distress syndrome: a randomized, controlled trial. Crit. Care Med. 34, 1311\u20131318 (2006).","journal-title":"Crit. Care Med."},{"key":"388_CR18","doi-asserted-by":"publisher","first-page":"651","DOI":"10.1016\/S2213-2600(18)30287-X","volume":"6","author":"PR Lawler","year":"2018","unstructured":"Lawler, P. R. & Fan, E. Heterogeneity and phenotypic stratification in acute respiratory distress syndrome. Lancet Respir. Med. 6, 651\u2013653 (2018).","journal-title":"Lancet Respir. Med."},{"key":"388_CR19","doi-asserted-by":"publisher","first-page":"26","DOI":"10.21037\/atm.2017.12.06","volume":"6","author":"B Lobo","year":"2018","unstructured":"Lobo, B., Hermosa, C., Abella, A. & Gordo, F. Electrical impedance tomography. Ann. Transl. Med. 6, 26 (2018).","journal-title":"Ann. Transl. Med."},{"key":"388_CR20","doi-asserted-by":"publisher","first-page":"788","DOI":"10.1001\/jama.2016.0291","volume":"315","author":"G Bellani","year":"2016","unstructured":"Bellani, G. et al. Epidemiology, patterns of care, and mortality for patients with acute respiratory distress syndrome in intensive care units in 50 countries. JAMA 315, 788\u2013800 (2016).","journal-title":"JAMA"},{"key":"388_CR21","doi-asserted-by":"publisher","first-page":"347","DOI":"10.1056\/NEJM199802053380602","volume":"338","author":"MB Amato","year":"1998","unstructured":"Amato, M. B. et al. Effect of a protective-ventilation strategy on mortality in the acute respiratory distress syndrome. N. Engl. J. Med. 338, 347\u2013354 (1998).","journal-title":"N. Engl. J. Med."},{"key":"388_CR22","doi-asserted-by":"crossref","unstructured":"National Heart, Lung, and Blood Institute ARDS Clinical Trials Network. Higher versus lower positive end-expiratory pressures in patients with the acute respiratory distress syndrome. N. Engl. J. Med. 351, 327\u2013336 (2004).","DOI":"10.1056\/NEJMoa032193"},{"key":"388_CR23","unstructured":"Batista, G. & Monard, M. C. A study of K-nearest neighbour as an imputation method. HIS. 87, 251\u2013260 (2003)."},{"key":"388_CR24","doi-asserted-by":"publisher","first-page":"637","DOI":"10.1001\/jama.299.6.637","volume":"299","author":"MO Meade","year":"2008","unstructured":"Meade, M. O. et al. Ventilation strategy using low tidal volumes, recruitment maneuvers, and high positive end-expiratory pressure for acute lung injury and acute respiratory distress syndrome: a randomized controlled trial. JAMA 299, 637\u2013645 (2008).","journal-title":"JAMA"},{"key":"388_CR25","doi-asserted-by":"publisher","first-page":"646","DOI":"10.1001\/jama.299.6.646","volume":"299","author":"A Mercat","year":"2008","unstructured":"Mercat, A. et al. Positive end-expiratory pressure setting in adults with acute lung injury and acute respiratory distress syndrome: a randomized controlled trial. JAMA 299, 646\u2013655 (2008).","journal-title":"JAMA"},{"key":"388_CR26","doi-asserted-by":"publisher","first-page":"1174","DOI":"10.1016\/j.rmed.2009.02.008","volume":"103","author":"Y Oba","year":"2009","unstructured":"Oba, Y., Thameem, D. M. & Zaza, T. High levels of PEEP may improve survival in acute respiratory distress syndrome: A meta-analysis. Respir. Med. 103, 1174\u20131181 (2009).","journal-title":"Respir. Med."},{"key":"388_CR27","doi-asserted-by":"publisher","first-page":"865","DOI":"10.1001\/jama.2010.218","volume":"303","author":"M Briel","year":"2010","unstructured":"Briel, M. et al. Higher vs lower positive end-expiratory pressure in patients with acute lung injury and acute respiratory distress syndrome: systematic review and meta-analysis. JAMA 303, 865\u2013873 (2010).","journal-title":"JAMA"},{"key":"388_CR28","first-page":"840","volume":"115","author":"F Fichtner","year":"2018","unstructured":"Fichtner, F. et al. Mechanical ventilation and extracorporeal membrane oxygena tion in acute respiratory insufficiency. Dtsch. Arzteblatt Int. 115, 840\u2013847 (2018).","journal-title":"Dtsch. Arzteblatt Int."},{"key":"388_CR29","doi-asserted-by":"publisher","unstructured":"Santa Cruz, R., Rojas, J. I., Nervi, R., Heredia, R. & Ciapponi, A. High versus low positive end-expiratory pressure (PEEP) levels for mechanically ventilated adult patients with acute lung injury and acute respiratory distress syndrome. Cochrane Database Syst. Rev. CD009098 (2013) https:\/\/doi.org\/10.1002\/14651858.CD009098.pub2.","DOI":"10.1002\/14651858.CD009098.pub2"},{"key":"388_CR30","unstructured":"Le, H. M., Voloshin, C. & Yue, Y. Batch policy learning under constraints. Proceedings of the 36th International Conference on Machine Learning, in Proceedings of Machine Learning Research 97, 3703\u20133712 Available from http:\/\/proceedings.mlr.press\/v97\/le19a.html (2019)."},{"key":"388_CR31","unstructured":"Raghu, A. et al. Behaviour policy estimation in off-policy policy evaluation: calibration matters. Preprint at https:\/\/arxiv.org\/abs\/1807.01066 (2018)."},{"key":"388_CR32","unstructured":"Liu, Y. et al. Representation balancing MDPs for off-policy policy evaluation. NeurIPS. Preprint at https:\/\/arxiv.org\/abs\/1805.09044 (2018)."},{"key":"388_CR33","unstructured":"Li, L., Komorowski, M. & Faisal, A. A. The actor search tree critic (ASTC) for off-policy POMDP learning in medical decision making. Preprint at https:\/\/arxiv.org\/abs\/1805.11548 (2018)."},{"key":"388_CR34","first-page":"239","volume":"2017","author":"S Parbhoo","year":"2017","unstructured":"Parbhoo, S., Bogojeska, J., Zazzi, M., Roth, V. & Doshi-Velez, F. Combining kernel and model based learning for HIV therapy selection. AMIA Summits Transl. Sci. Proc. 2017, 239\u2013248 (2017).","journal-title":"AMIA Summits Transl. Sci. Proc."},{"key":"388_CR35","unstructured":"Guez, A., Vincent, R. D., Avoli, M. & Pineau, J. Adaptive Treatment of Epilepsy via Batch-mode Reinforcement Learning. in Proceedings of the 20th National Conference on Innovative Applications of Artificial Intelligence - Volume 3 1671\u20131678 (AAAI Press, 2008)."},{"key":"388_CR36","unstructured":"Prasad, N., Cheng, L.-F., Chivers, C., Draugelis, M. & Engelhardt, B. E. A reinforcement learning approach to weaning of mechanical ventilation in intensive care units. Preprint at https:\/\/arxiv.org\/abs\/1704.06300 (2017)."},{"key":"388_CR37","doi-asserted-by":"publisher","first-page":"401","DOI":"10.1165\/ajrcmb.22.4.f184","volume":"22","author":"E Abraham","year":"2000","unstructured":"Abraham, E. Coagulation abnormalities in acute lung injury and sepsis. Am. J. Respir. Cell Mol. Biol. 22, 401\u2013404 (2000).","journal-title":"Am. J. Respir. Cell Mol. Biol."},{"key":"388_CR38","unstructured":"Johansson, F. D., Shalit, U. & Sontag, D. Learning Representations for Counterfactual Inference. in Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48 3020\u20133029 (JMLR.org, 2016)."},{"key":"388_CR39","unstructured":"Shalit, U., Johansson, F. D. & Sontag, D. Estimating individual treatment effect: generalization bounds and algorithms. ICML. Preprint at https:\/\/arxiv.org\/abs\/1606.03976 (2016)."},{"key":"388_CR40","unstructured":"Mitra, S. K. Digital Signal Processing: A Computer Based Approach. (McGraw-Hill Education - Europe, 2010)."},{"key":"388_CR41","doi-asserted-by":"publisher","unstructured":"Salgado, C. M., Azevedo, C., Proen\u00e7a, H. & Vieira, S. M. Missing Data. in Secondary Analysis of Electronic Health Records (ed. MIT Critical Data) 143\u2013162 (Springer International Publishing, 2016). https:\/\/doi.org\/10.1007\/978-3-319-43742-2_13.","DOI":"10.1007\/978-3-319-43742-2_13"},{"key":"388_CR42","doi-asserted-by":"publisher","first-page":"474","DOI":"10.1177\/0272989X09353194","volume":"30","author":"O Alagoz","year":"2010","unstructured":"Alagoz, O., Hsu, H., Schaefer, A. J. & Roberts, M. S. Markov decision processes: a tool for sequential decision making under uncertainty. Med. Decis. Mak. 30, 474\u2013483 (2010).","journal-title":"Med. Decis. Mak."},{"key":"388_CR43","unstructured":"Neumann, G. & Peters, J. R. Fitted Q-iteration by Advantage Weighted Regression. in Advances in Neural Information Processing Systems 21 (eds. Koller, D., Schuurmans, D., Bengio, Y. & Bottou, L.) 1177\u20131184 (Curran Associates, Inc., 2009)."},{"key":"388_CR44","first-page":"279","volume":"8","author":"CJCH Watkins","year":"1992","unstructured":"Watkins, C. J. C. H. & Dayan, P. Q-learning. Mach. Learn. 8, 279\u2013292 (1992).","journal-title":"Mach. Learn."},{"key":"388_CR45","doi-asserted-by":"crossref","unstructured":"Thomas, P., Theocharous, G. & Ghavamzadeh, M. High-confidence off-policy evaluation. In Proceedings of the AAAI Conference on Artificial Intelligence. 29, (2015).","DOI":"10.1609\/aaai.v29i1.9541"}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-021-00388-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-021-00388-6","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-021-00388-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,3]],"date-time":"2022-12-03T18:44:46Z","timestamp":1670093086000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-021-00388-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,2,19]]},"references-count":45,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2021,12]]}},"alternative-id":["388"],"URL":"https:\/\/doi.org\/10.1038\/s41746-021-00388-6","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,2,19]]},"assertion":[{"value":"6 January 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"11 January 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"19 February 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"A.P., G.D., A.S., C.T., G.M., and L.M. are co-founders of Clinomic GmbH. A.P. and L.M. are chief executive officers of Clinomic GmbH. C.T. is chief executive officer of William Harvey Research Limited outside of the submitted work. G.M. received restricted research grants and consultancy fees from BBraun Melsungen, Biotest, Adrenomed, and Sphingotec GmbH outside of the submitted work. L.M. and A.P. received consultancy fees from Sphingotec GmbH. All remaining authors declare that they have no conflict of interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"32"}}