{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,8]],"date-time":"2026-09-08T03:41:54Z","timestamp":1788838914532,"version":"build-2803163510"},"reference-count":29,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2022,4,14]],"date-time":"2022-04-14T00:00:00Z","timestamp":1649894400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100014764","name":"China International Medical Foundation","doi-asserted-by":"publisher","award":["Z-2018-35-1902"],"award-info":[{"award-number":["Z-2018-35-1902"]}],"id":[{"id":"10.13039\/501100014764","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100005090","name":"Beijing Nova Program","doi-asserted-by":"publisher","award":["Z201100006820126"],"award-info":[{"award-number":["Z201100006820126"]}],"id":[{"id":"10.13039\/501100005090","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Med."],"abstract":"<jats:sec><jats:title>Objective<\/jats:title><jats:p>Fluid therapy for sepsis patients has always been a problem that puzzles clinicians, that is, knowing when patients need fluid infusion and when they need negative fluid balance. Different clinicians may have different judgment criteria and make different decisions. Recently, studies have suggested that different fluid treatment strategies can cause different clinical outcomes. This study is intended to establish and verify a model for judging the direction of fluid therapy based on machine learning.<\/jats:p><\/jats:sec><jats:sec><jats:title>Method<\/jats:title><jats:p>This study included 2705 sepsis patients from the Peking Union Medical College Hospital Intensive Care Medical Information System and Database (PICMISD) from January 2016 to April 2020. The training set and test set (January 2016 to June 2019) were randomly divided. Twenty-seven features were extracted for modeling, including 25 state features (bloc, vital sign, laboratory examination, blood gas assay and demographics), 1 action feature (fluid balance) and 1 outcome feature (ICU survival or death). SARSA was used to learn the data rules of the training set. Deep Q-learning (DQN) was used to learn the relationship between states and actions of the training set and predict the next balance. A double-robust estimator was used to evaluate the average expected reward of the test set in the deep Q-learning model. Lastly, we verified the difference between the predicted fluid therapy model and the actual treatment for the patient's prognoses, with sepsis patient data from July 2019 to April 2020 as the validation set.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>The training set and test set were extracted from the same database, and the distribution of liquid balance was similar. Actions were divided into five intervals corresponding to 0\u201320, 20\u201340, 40\u201360, 60\u201380, and 80\u2013100% percentiles of fluid balance. The higher the reward of <jats:italic>Q<\/jats:italic>(<jats:italic>s, a<\/jats:italic>) calculated by SARSA from the training set, the lower the mortality rate. Deep Q-learning indicates that both fluid balance differences that are too high and too low show an increase in mortality. The more consistent the fluid balance prediction with the real result, the lower the mortality rate. The smaller the difference between the prediction and the reality, the lower the mortality rate. The double-robust estimator shows that the model has satisfactory stability. The validation set indicates that the mortality rate of patients in the \u201cpredicted negative fluid balance and actual negative fluid balance\u201d subgroup was the lowest, which was statistically significant, indicating that the model can be used for clinical verification.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusion<\/jats:title><jats:p>We used reinforcement learning to propose a possible prediction model for guiding the direction of fluid therapy for sepsis patients in the ICU. This model may accurately predict the best direction for fluid therapy, thereby improving patient prognosis.<\/jats:p><\/jats:sec>","DOI":"10.3389\/fmed.2022.766447","type":"journal-article","created":{"date-parts":[[2022,4,14]],"date-time":"2022-04-14T12:13:14Z","timestamp":1649938394000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":13,"title":["Establishment and Implementation of Potential Fluid Therapy Balance Strategies for ICU Sepsis Patients Based on Reinforcement Learning"],"prefix":"10.3389","volume":"9","author":[{"given":"Longxiang","family":"Su","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yansheng","family":"Li","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shengjun","family":"Liu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Siqi","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xiang","family":"Zhou","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Li","family":"Weng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mingliang","family":"Su","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bin","family":"Du","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Weiguo","family":"Zhu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yun","family":"Long","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2022,4,14]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"251","DOI":"10.1186\/s13054-015-0970-1","article-title":"A positive fluid balance is an independent prognostic factor in patients with sepsis","volume":"19","author":"Acheampong","year":"2015","journal-title":"Crit Care."},{"key":"B2","doi-asserted-by":"publisher","first-page":"259","DOI":"10.1097\/CCM.0b013e3181feeb15","article-title":"Fluid resuscitation in septic shock: a positive fluid balance and elevated central venous pressure are associated with increased mortality","volume":"39","author":"Boyd","year":"2011","journal-title":"Crit Care Med."},{"key":"B3","doi-asserted-by":"publisher","first-page":"R76","DOI":"10.1186\/cc11333","article-title":"Higher vs. lower fluid volume for septic shock: clinical characteristics and outcome in unselected patients in a prospective, multicenter cohort","volume":"16","author":"Smith","year":"2012","journal-title":"Crit Care."},{"key":"B4","doi-asserted-by":"publisher","first-page":"68","DOI":"10.1097\/SHK.0000000000000268","article-title":"Fluid overload in patients with severe sepsis and septic shock treated with early goal-directed therapy is associated with increased acute need for fluid-related medical interventions and hospital death","volume":"43","author":"Kelm","year":"2015","journal-title":"Shock."},{"key":"B5","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1016\/j.kint.2016.11.029","article-title":"Capillary leak syndrome: etiologies, pathophysiology, and management","volume":"92","author":"Siddall","year":"2017","journal-title":"Kidney Int."},{"key":"B6","doi-asserted-by":"publisher","first-page":"16","DOI":"10.1016\/j.jcrc.2019.09.025","article-title":"Association of negative fluid balance during the de-escalation phase of sepsis management with mortality: a cohort study","volume":"55","author":"Dhondup","year":"2020","journal-title":"J Crit Care."},{"key":"B7","doi-asserted-by":"publisher","first-page":"243","DOI":"10.1186\/s13054-021-03682-7","article-title":"Individualized resuscitation strategy for septic shock formalized by finite mixture modeling and dynamic treatment regimen","volume":"25","author":"Ma","year":"2021","journal-title":"Crit Care."},{"key":"B8","first-page":"773","article-title":"Is Deep reinforcement learning ready for practical applications in healthcare? A sensitivity analysis of duel-DDQN for hemodynamic management in sepsis patients AMIA","volume":"2020","author":"Lu","year":"2020","journal-title":"Annu Symp Proc."},{"key":"B9","doi-asserted-by":"publisher","first-page":"801","DOI":"10.1001\/jama.2016.0287","article-title":"The third international consensus definitions for sepsis and septic shock (Sepsis-3)","volume":"315","author":"Singer","year":"2016","journal-title":"JAMA."},{"key":"B10","volume-title":"GARM: On-line Q-learning using connectionist systems","year":"1994"},{"key":"B11","first-page":"2613","article-title":"Double Q-learning","volume":"23","author":"HV","year":"2010","journal-title":"Adv Neural Inf Process Syst."},{"key":"B12","first-page":"06461","article-title":"David Silver: Deep reinforcement learning with double Q-learning","volume":"1509","author":"Hado van Hasselt","year":"2015","journal-title":"arXiv [Preprint]"},{"key":"B13","doi-asserted-by":"publisher","first-page":"1716","DOI":"10.1038\/s41591-018-0213-5","article-title":"The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care","volume":"24","author":"Komorowski","year":"2018","journal-title":"Nat Med."},{"key":"B14","first-page":"4601","article-title":"Doubly robust policy evaluation and learning","volume":"1103","author":"Miroslav Dudik","year":"2011","journal-title":"arXiv [Preprint]"},{"key":"B15","first-page":"03722","article-title":"Doubly robust off-policy value evaluation for reinforcement learning","volume":"1511","author":"Nan Jiang","year":"2016","journal-title":"arXiv [Preprint]"},{"key":"B16","doi-asserted-by":"publisher","first-page":"399","DOI":"10.1080\/00273171.2011.568786","article-title":"An introduction to propensity score methods for reducing the effects of confounding in observational studies","volume":"46","author":"Austin","year":"2011","journal-title":"Multivariate Behav Res."},{"key":"B17","doi-asserted-by":"publisher","first-page":"01347","DOI":"10.48550\/arXiv.1806.01347","article-title":"Importance sampling policy evaluation with an estimated behavior policy","volume":"1806","year":"2019","journal-title":"arXiv preprint"},{"key":"B18","doi-asserted-by":"publisher","first-page":"1749","DOI":"10.1378\/chest.117.6.1749","article-title":"Negative fluid balance predicts survival in patients with septic shock: a retrospective pilot study","volume":"117","author":"Alsous","year":"2000","journal-title":"Chest."},{"key":"B19","doi-asserted-by":"publisher","first-page":"2564","DOI":"10.1056\/NEJMoa062200","article-title":"Comparison of two fluid-management strategies in acute lung injury","volume":"354","author":"Wiedemann","year":"2006","journal-title":"N Engl J Med."},{"key":"B20","doi-asserted-by":"publisher","first-page":"257","DOI":"10.1001\/jamapediatrics.2017.4540","article-title":"Association between fluid balance and outcomes in critically ill children: a systematic review and meta-analysis","volume":"172","author":"Alobaidi","year":"2018","journal-title":"JAMA Pediatr."},{"key":"B21","doi-asserted-by":"publisher","first-page":"625","DOI":"10.1007\/s00134-016-4675-y","article-title":"Fluid administration in severe sepsis and septic shock, patterns, and outcomes: an analysis of a large national database","volume":"43","author":"Marik","year":"2017","journal-title":"Intensive Care Med."},{"key":"B22","doi-asserted-by":"publisher","first-page":"1726","DOI":"10.1056\/NEJMra1208943","article-title":"Circulatory shock","volume":"369","author":"Vincent","year":"2013","journal-title":"N Engl J Med."},{"key":"B23","doi-asserted-by":"publisher","first-page":"740","DOI":"10.1093\/bja\/aeu300","article-title":"Four phases of intravenous fluid therapy: a conceptual model","volume":"113","author":"Hoste","year":"2014","journal-title":"Br J Anaesth."},{"key":"B24","doi-asserted-by":"publisher","first-page":"66","DOI":"10.1186\/s13613-018-0402-x","article-title":"Principles of fluid management and stewardship in septic shock: it is time to consider the four D's and the four phases of fluid therapy","volume":"8","author":"Malbrain","year":"2018","journal-title":"Ann Intensive Care."},{"key":"B25","doi-asserted-by":"publisher","first-page":"104","DOI":"10.1186\/s13054-017-1692-3","article-title":"Association between fluid intake and mortality in critically ill patients with negative fluid balance: a retrospective cohort study","volume":"21","author":"Shen","year":"2017","journal-title":"Crit Care."},{"key":"B26","article-title":"Continuous state-space models for optimal sepsis treatment - a deep reinforcement learning approach","author":"Raghu","year":"2017","journal-title":"arXiv [Preprint]. arXiv"},{"key":"B27","article-title":"Identifying distinct, effective treatments for acute hypotension with Soda-RL: safely optimized diverse accurate reinforcement learning","author":"Joseph Futoma","year":"2020","journal-title":"arXiv 200103224."},{"key":"B28","article-title":"Improving sepsis treatment strategies by combining deep and kernel-based reinforcement learning","author":"Xuefeng Peng","year":"2019","journal-title":"arXiv [Preprint]. arXiv"},{"key":"B29","doi-asserted-by":"publisher","first-page":"17874","DOI":"10.1038\/s41598-020-74906-z","article-title":"Individualized fluid administration for critically ill patients with sepsis with an interpretable dynamic treatment regimen model","volume":"10","author":"Zhang","year":"2020","journal-title":"Sci Rep."}],"container-title":["Frontiers in Medicine"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fmed.2022.766447\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,4,14]],"date-time":"2022-04-14T12:13:22Z","timestamp":1649938402000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fmed.2022.766447\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,4,14]]},"references-count":29,"alternative-id":["10.3389\/fmed.2022.766447"],"URL":"https:\/\/doi.org\/10.3389\/fmed.2022.766447","relation":{},"ISSN":["2296-858X"],"issn-type":[{"value":"2296-858X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,4,14]]},"article-number":"766447"}}