{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:56:31Z","timestamp":1782845791749,"version":"3.54.5"},"reference-count":34,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Med Inform"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Offline reinforcement learning (RL) has been increasingly applied to clinical decision-making problems. However, due to the lack of a standardized pipeline, prior work often relied on strategies that may lead to overfitted policies and inaccurate evaluations.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n In this work, we present a practical pipeline\u2014Pipeline for Learning Robust Policies in Reinforcement Learning (PROP-RL)\u2014designed to improve robustness and minimize disruption to clinical workflow. We demonstrate its efficacy in the context of learning treatment policies for administering loop diuretics in hospitalized patients.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n Our cohort included adult inpatients admitted to the emergency department at Michigan Medicine between 2015 and 2019 who required supplemental oxygen. We modeled the management of loop diuretics as an offline RL problem using a discrete state space based on features extracted from electronic health records, a binary action space corresponding to the daily use of loop diuretics, and a reward function based on in-hospital mortality. The policy was trained on data from 2015 to 2018 and evaluated on a held-out set of hospitalizations from 2019, in terms of estimated reduction in mortality compared to clinician behavior.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n The final study cohort included 36,570 hospitalizations. The learned treatment policy was based on 60 states: the policy deferred to clinicians in 36 states, recommended the majority action in 22 states, and diverged significantly from clinician behavior in 2 of the states. Among the cases where the policy meaningfully diverged from the behavior policy, the learned policy was estimated to significantly reduce the mortality rate from 3.8% to 2.2% by 1.6% (95% CI 0.4\u20102.7;\n P<\/jats:italic>\n =.006).\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n We applied our pipeline to the clinical problem of loop diuretic treatment, highlighting the importance of robust state representation and thoughtful policy selection and evaluation. Our work reveals areas of potential improvement in current clinical care for loop diuretics and serves as a blueprint for using offline RL for sequential treatment selection in clinical settings.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/69145","type":"journal-article","created":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T20:13:44Z","timestamp":1760127224000},"page":"e69145-e69145","source":"Crossref","is-referenced-by-count":1,"title":["Optimizing Loop Diuretic Treatment for Mortality Reduction in Patients With Acute Dyspnea Using a Practical Offline Reinforcement Learning Pipeline for Health Care: Retrospective Single-Center Simulation Study"],"prefix":"10.2196","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5341-8529","authenticated-orcid":false,"given":"Jung Min","family":"Lee","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4213-2015","authenticated-orcid":false,"given":"Shengpu","family":"Tang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0535-9659","authenticated-orcid":false,"given":"Michael","family":"Sjoding","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1057-7722","authenticated-orcid":false,"given":"Jenna","family":"Wiens","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2025,10,10]]},"reference":[{"key":"R1","unstructured":"Sutton RS Barto AG . Reinforcement Learning, Second Edition: An Introduction. MIT Press; 2018. ISBN: 978-0-262-03924-6"},{"issue":"11","key":"R2","doi-asserted-by":"publisher","first-page":"1716","DOI":"10.1038\/s41591-018-0213-5","article-title":"The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care","volume":"24","author":"Komorowski","journal-title":"Nat Med"},{"issue":"10","key":"R3","doi-asserted-by":"publisher","first-page":"5087","DOI":"10.1109\/JBHI.2023.3303367","article-title":"Offline deep reinforcement learning and off-policy evaluation for personalized basal insulin control in type 1 diabetes","volume":"27","author":"Zhu","journal-title":"IEEE J Biomed Health Inform"},{"issue":"1","key":"R4","doi-asserted-by":"publisher","first-page":"173","DOI":"10.1038\/s41746-022-00708-4","article-title":"An interpretable RL framework for pre-deployment modeling in ICU hypotension management","volume":"5","author":"Zhang","journal-title":"NPJ Digit Med"},{"key":"R5","unstructured":"Levine S Kumar A Tucker G Fu J . Offline reinforcement learning: tutorial, review, and perspectives on open problems. arXiv. Preprint posted online on May 4, 2020. [doi: 10.48550\/arXiv.2005.01643]"},{"key":"R6","unstructured":"Paine TL Paduraru C Michi A et al. Hyperparameter selection for offline reinforcement learning. arXiv. Preprint posted online on Jul 17, 2020. [doi: 10.48550\/arXiv.2007.09055]"},{"key":"R7","unstructured":"Nie A Flet-Berliac Y Jordan D Steenbergen W Brunskill E . Data-efficient pipeline for offline reinforcement learning with limited data. Presented at: Proceedings of the 36th Conference on Neural Information Processing Systems; Nov 28 to Dec 9, 2022; New Orleans, LA, USA. URL: https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2022\/file\/5ee7ed60a7e8169012224dec5fe0d27f-Paper-Conference.pdf [Accessed 29-09-2025]"},{"key":"R8","unstructured":"Kumar A Zhou A Tucker G Levine S . Conservative q-learning for offline reinforcement learning. Presented at: Proceedings of the 34th Conference on Neural Information Processing Systems; Dec 6-12, 2020. URL: https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2020\/file\/0d2b2061826a5df3221116a5085a6052-Paper.pdf [Accessed 21-09-2025]"},{"issue":"1","key":"R9","doi-asserted-by":"publisher","first-page":"16","DOI":"10.1038\/s41591-018-0310-5","article-title":"Guidelines for reinforcement learning in healthcare","volume":"25","author":"Gottesman","journal-title":"Nat Med"},{"issue":"8","key":"R10","doi-asserted-by":"publisher","first-page":"803","DOI":"10.1093\/eurjhf\/hfs105","article-title":"ESC guidelines for the diagnosis and treatment of acute and chronic heart failure 2012: The Task Force for the Diagnosis and Treatment of Acute and Chronic Heart Failure 2012 of the European Society of Cardiology. Developed in collaboration with the Heart Failure Association (HFA) of the ESC","volume":"14","author":"McMurray","journal-title":"Eur J Heart Fail"},{"key":"R11","doi-asserted-by":"crossref","unstructured":"Killian TW Zhang H Subramanian J Fatemi M Ghassemi M . An empirical study of representation learning for reinforcement learning in healthcare. arXiv. Preprint posted online on Nov 23, 2020. [doi: 10.48550\/arXiv.2011.11235]","DOI":"10.1007\/978-981-15-4095-0_11"},{"key":"R12","unstructured":"Tang S Wiens J . Model selection for offline reinforcement learning: practical considerations for healthcare settings. Presented at: Proceedings of the 6th Machine Learning for Healthcare Conference, PMLR; Aug 6, 2021. URL: https:\/\/proceedings.mlr.press\/v149\/tang21a.html [Accessed 28-05-2024]"},{"issue":"1","key":"R13","doi-asserted-by":"publisher","first-page":"17","DOI":"10.5049\/EBP.2015.13.1.17","article-title":"Loop diuretics in clinical practice","volume":"13","author":"Oh","journal-title":"Electrolyte Blood Press"},{"issue":"3","key":"R14","doi-asserted-by":"publisher","first-page":"156","DOI":"10.1038\/nrcardio.2012.191","article-title":"Causes and treatment of oedema in patients with heart failure","volume":"10","author":"Clark","journal-title":"Nat Rev Cardiol"},{"issue":"49","key":"R15","doi-asserted-by":"publisher","first-page":"834","DOI":"10.3238\/arztebl.2016.0834","article-title":"The differential diagnosis of dyspnea","volume":"113","author":"Berliner","journal-title":"Dtsch Arztebl Int"},{"issue":"3","key":"R16","doi-asserted-by":"publisher","first-page":"210","DOI":"10.1016\/j.cardfail.2003.10.006","article-title":"Searching for evidence: refractory questions in advanced heart failure","volume":"10","author":"Shah","journal-title":"J Card Fail"},{"issue":"1","key":"R17","doi-asserted-by":"publisher","DOI":"10.1186\/s13054-015-1017-3","article-title":"Loop diuretics in acute heart failure: beyond the decongestive relief for the kidney","volume":"19","author":"Palazzuoli","journal-title":"Crit Care"},{"issue":"25","key":"R18","doi-asserted-by":"publisher","first-page":"3042","DOI":"10.1016\/j.jacc.2017.04.042","article-title":"Time-to-furosemide treatment and mortality in patients hospitalized with acute heart failure","volume":"69","author":"Matsue","journal-title":"J Am Coll Cardiol"},{"key":"R19","unstructured":"PROP-RL: pipeline for learning robust policies in RL. GitHub. URL: https:\/\/github.com\/MLD3\/PROP-RL [Accessed 23-04-2024]"},{"issue":"12","key":"R20","doi-asserted-by":"publisher","first-page":"1921","DOI":"10.1093\/jamia\/ocaa139","article-title":"Democratizing EHR analyses with FIDDLE: a flexible data-driven preprocessing pipeline for structured clinical data","volume":"27","author":"Tang","journal-title":"J Am Med Inform Assoc"},{"key":"R21","unstructured":"Shi C Blei D Veitch V . Adapting neural networks for the estimation of treatment effects. Presented at: Proceedings of the 33rd Conference on Neural Information Processing Systems; Dec 8-14, 2019; Vancouver, Canada. URL: https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2019\/file\/8fb5f8be2aa9d6c64a04e3ab9f63feee-Paper.pdf [Accessed 28-05-2024]"},{"key":"R22","doi-asserted-by":"publisher","DOI":"10.1162\/153244303321897735","article-title":"Cluster ensembles --- a knowledge reuse framework for combining multiple partitions","volume":"3","author":"Strehl","journal-title":"J Mach Learn Res"},{"key":"R23","unstructured":"Fujimoto S Meger D Precup D . Off-policy deep reinforcement learning without exploration. Presented at: Proceedings of the 36th International Conference on Machine Learning, PMLR; Jun 9-15, 2019; Long Beach, California, USA. URL: https:\/\/proceedings.mlr.press\/v97\/fujimoto19a.html [Accessed 21-09-2025]"},{"key":"R24","unstructured":"Kidambi R Rajeswaran A Netrapalli P Joachims T . MOReL: model-based offline reinforcement learning. Presented at: Proceedings of the 34th Conference on Neural Information Processing Systems; Dec 6-12, 2020. URL: https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2020\/file\/f7efa4f864ae9b88d43527f4b14f750f-Paper.pdf [Accessed 28-05-2024]"},{"key":"R25","unstructured":"Precup D Sutton RS Singh S . Eligibility traces for off-policy policy evaluation. Presented at: ICML \u201900: Proceedings of the Seventeenth International Conference on Machine Learning; Jun 29 to Jul 2, 2000; Stanford University, Stanford, CA, USA."},{"key":"R26","doi-asserted-by":"publisher","DOI":"10.1016\/j.sigpro.2016.08.025","article-title":"Effective sample size for importance sampling based on discrepancy measures","volume":"131","author":"Martino","journal-title":"Signal Processing"},{"key":"R27","unstructured":"Moore D Notz W Fligner M . The Basic Practice of Statistics. WH Freeman & Co Ltd; 2021. ISBN: 978-1-319-34463-4"},{"key":"R28","unstructured":"Shen SP Ma Y Gottesman O Doshi-Velez F . State relevance for off-policy evaluation. Presented at: Proceedings of the 38th International Conference on Machine Learning, PMLR; Jul 18-24, 2021. URL: https:\/\/proceedings.mlr.press\/v139\/shen21d.html [Accessed 21-09-2025]"},{"key":"R29","doi-asserted-by":"publisher","DOI":"10.1136\/bmj-2023-078378","article-title":"TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods","volume":"385","author":"Collins","journal-title":"BMJ"},{"issue":"13","key":"R30","doi-asserted-by":"publisher","first-page":"15696","DOI":"10.1609\/aaai.v37i13.26862","article-title":"Towards safe mechanical ventilation treatment using deep offline reinforcement learning","volume":"37","author":"Kondrup","journal-title":"AAAI"},{"key":"R31","unstructured":"Liu R Greenstein JL Fackler JC Bergmann J Bembea MM Winslow RL . Offline reinforcement learning with uncertainty for treatment strategies in sepsis. arXiv. Preprint posted online on Jul 9, 2021. [doi: 10.48550\/arXiv.2107.04491]"},{"issue":"1","key":"R32","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1093\/jamia\/ocy139","article-title":"Cranky comments: detecting clinical decision support malfunctions through free-text override reasons","volume":"26","author":"Aaron","journal-title":"J Am Med Inform Assoc"},{"issue":"5","key":"R33","doi-asserted-by":"publisher","first-page":"496","DOI":"10.1093\/jamia\/ocx106","article-title":"Clinical decision support alert malfunctions: analysis and empirically derived taxonomy","volume":"25","author":"Wright","journal-title":"J Am Med Inform Assoc"},{"key":"R34","doi-asserted-by":"publisher","DOI":"10.1136\/bmj-2022-070904","article-title":"Reporting guideline for the early stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI","volume":"377","author":"Vasey","journal-title":"BMJ"}],"container-title":["JMIR Medical Informatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/medinform.jmir.org\/2025\/1\/e69145","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:56:36Z","timestamp":1782842196000},"score":1,"resource":{"primary":{"URL":"https:\/\/medinform.jmir.org\/2025\/1\/e69145"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,10]]},"references-count":34,"URL":"https:\/\/doi.org\/10.2196\/69145","relation":{},"ISSN":["2291-9694"],"issn-type":[{"value":"2291-9694","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,10,10]]},"article-number":"v13i11e69145"}}