{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,24]],"date-time":"2026-07-24T11:00:01Z","timestamp":1784890801030,"version":"3.55.0"},"reference-count":23,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2022,3,11]],"date-time":"2022-03-11T00:00:00Z","timestamp":1646956800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,3,11]],"date-time":"2022-03-11T00:00:00Z","timestamp":1646956800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["U01 CA230694"],"award-info":[{"award-number":["U01 CA230694"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000738","name":"U.S. Department of Veterans Affairs","doi-asserted-by":"publisher","award":["IIR 16-024"],"award-info":[{"award-number":["IIR 16-024"]}],"id":[{"id":"10.13039\/100000738","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Med Inform Decis Mak"],"published-print":{"date-parts":[[2022,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec>\n                <jats:title>Background<\/jats:title>\n                <jats:p>Evaluation of new treatment policies is often costly and challenging in complex conditions, such as hepatitis C virus (HCV) treatment, or in limited-resource settings. We sought to identify hypothetical policies for HCV treatment that could best balance the prevention of cirrhosis while preserving resources (financial or otherwise).<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Methods<\/jats:title>\n                <jats:p>The cohort consisted of 3792 HCV-infected patients without a history of cirrhosis or hepatocellular carcinoma at baseline from the national Veterans Health Administration from 2015 to 2019. To estimate the efficacy of hypothetical treatment policies, we utilized historical data and reinforcement learning to allow for greater flexibility when constructing new HCV treatment strategies. We tested and compared four new treatment policies: a simple stepwise policy based on Aspartate Aminotransferase to Platelet Ratio Index (APRI), a logistic regression based on APRI, a logistic regression on multiple longitudinal and demographic indicators that were prespecified for clinical significance, and a treatment policy based on a risk model developed for HCV infection.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Results<\/jats:title>\n                <jats:p>The risk-based hypothetical treatment policy achieved the lowest overall risk with a score of 0.016\u00a0(90% CI 0.016, 0.019) while treating the most high-risk (346.4\u2009\u00b1\u20091.4) and the fewest low-risk (361.0\u2009\u00b1\u200920.1) patients. Compared to hypothetical treatment policies that treated approximately the same number of patients (1843.7 vs. 1914.4 patients), the risk-based policy had more untreated time per patient (7968.4 vs. 7742.9 patient visits), signaling cost reduction for the healthcare system.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Conclusions<\/jats:title>\n                <jats:p>Off-policy evaluation strategies are useful to evaluate hypothetical treatment policies without implementation. If a quality risk model is available, risk-based treatment strategies can reduce overall risk and prioritize patients while reducing healthcare system costs.<\/jats:p>\n              <\/jats:sec>","DOI":"10.1186\/s12911-022-01789-7","type":"journal-article","created":{"date-parts":[[2022,3,11]],"date-time":"2022-03-11T04:28:39Z","timestamp":1646972919000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["Reinforcement learning evaluation of treatment policies for patients with hepatitis C virus"],"prefix":"10.1186","volume":"22","author":[{"given":"Brandon","family":"Oselio","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Amit G.","family":"Singal","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xuefei","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tony","family":"Van","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Boang","family":"Liu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ji","family":"Zhu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Akbar K.","family":"Waljee","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2022,3,11]]},"reference":[{"key":"1789_CR1","doi-asserted-by":"publisher","first-page":"106263","DOI":"10.1016\/j.ypmed.2020.106263","volume":"141","author":"J Cohen","year":"2020","unstructured":"Cohen J, Rodgers YVM. Contributing factors to personal protective equipment shortages during the COVID-19 pandemic. Prev Med. 2020;141:106263.","journal-title":"Prev Med"},{"issue":"4","key":"1789_CR2","doi-asserted-by":"publisher","first-page":"211","DOI":"10.12788\/jhm.3539","volume":"16","author":"AT Janke","year":"2021","unstructured":"Janke AT, et al. Analysis of hospital resource availability and COVID-19 mortality across the United States. J Hosp Med. 2021;16(4):211\u20134.","journal-title":"J Hosp Med"},{"key":"1789_CR3","doi-asserted-by":"crossref","unstructured":"Gristina GR, Piccinni M. Covid-19 pandemic in ICU. Limited resources for many patients: approaches and criteria for triaging. Minerva Anestesiol, 2021.","DOI":"10.23736\/S0375-9393.21.15736-0"},{"issue":"6","key":"1789_CR4","doi-asserted-by":"publisher","first-page":"1218","DOI":"10.1093\/geronb\/gbaa123","volume":"76","author":"R Tipirneni","year":"2021","unstructured":"Tipirneni R, et al. Changes in health care access and utilization for low-SES adults aged 51\u201364 years after medicaid expansion. J Gerontol B Psychol Sci Soc Sci. 2021;76(6):1218\u201330.","journal-title":"J Gerontol B Psychol Sci Soc Sci"},{"issue":"10","key":"1789_CR5","doi-asserted-by":"publisher","first-page":"1501","DOI":"10.1001\/jamainternmed.2016.4419","volume":"176","author":"BD Sommers","year":"2016","unstructured":"Sommers BD, et al. Changes in utilization and health among low-income adults after medicaid expansion or expanded private insurance. JAMA Intern Med. 2016;176(10):1501\u20139.","journal-title":"JAMA Intern Med"},{"issue":"5","key":"1789_CR6","doi-asserted-by":"publisher","first-page":"469","DOI":"10.23736\/S0375-9393.20.14619-4","volume":"86","author":"M Vergano","year":"2020","unstructured":"Vergano M, et al. SIAARTI recommendations for the allocation of intensive care treatments in exceptional, resource-limited circumstances. Minerva Anestesiol. 2020;86(5):469\u201372.","journal-title":"Minerva Anestesiol"},{"issue":"2","key":"1789_CR7","doi-asserted-by":"publisher","first-page":"87","DOI":"10.1136\/bmjstel-2018-000370","volume":"6","author":"G Lam\u00e9","year":"2020","unstructured":"Lam\u00e9 G, Dixon-Woods M. Using clinical simulation to study how to improve quality and safety in healthcare. BMJ Simul Technol Enhanc Learn. 2020;6(2):87\u201394.","journal-title":"BMJ Simul Technol Enhanc Learn"},{"key":"1789_CR8","doi-asserted-by":"crossref","unstructured":"Yu C, et al., Reinforcement learning in healthcare: a survey. ACM Comput Surv 2021;55(1):Article 5.","DOI":"10.1145\/3453160"},{"key":"1789_CR9","doi-asserted-by":"publisher","first-page":"102193","DOI":"10.1016\/j.artmed.2021.102193","volume":"121","author":"S Ebrahimi","year":"2021","unstructured":"Ebrahimi S, Lim GJ. A reinforcement learning approach for finding optimal policy of adaptive radiation therapy considering uncertain tumor biological response. Artif Intell Med. 2021;121:102193.","journal-title":"Artif Intell Med"},{"issue":"11","key":"1789_CR10","doi-asserted-by":"publisher","first-page":"1716","DOI":"10.1038\/s41591-018-0213-5","volume":"24","author":"M Komorowski","year":"2018","unstructured":"Komorowski M, et al. The Artificial Intelligence Clinician learns optimal treatment strategies for sepsis in intensive care. Nat Med. 2018;24(11):1716\u201320.","journal-title":"Nat Med"},{"issue":"7","key":"1789_CR11","doi-asserted-by":"publisher","first-page":"e18477","DOI":"10.2196\/18477","volume":"22","author":"S Liu","year":"2020","unstructured":"Liu S, et al. Reinforcement learning for clinical decision support in critical care: comprehensive review. J Med Internet Res. 2020;22(7):e18477.","journal-title":"J Med Internet Res"},{"issue":"1","key":"1789_CR12","doi-asserted-by":"publisher","first-page":"350","DOI":"10.1186\/s12911-021-01712-6","volume":"21","author":"H Zheng","year":"2021","unstructured":"Zheng H, et al. Reinforcement learning assisted oxygen therapy for COVID-19 patients under intensive care. BMC Med Inform Decis Mak. 2021;21(1):350.","journal-title":"BMC Med Inform Decis Mak"},{"issue":"Suppl 3","key":"1789_CR13","doi-asserted-by":"publisher","first-page":"124","DOI":"10.1186\/s12911-020-1120-5","volume":"20","author":"C Yu","year":"2020","unstructured":"Yu C, Ren G, Dong Y. Supervised-actor-critic reinforcement learning for intelligent mechanical ventilation and sedative dosing in intensive care units. BMC Med Inform Decis Mak. 2020;20(Suppl 3):124.","journal-title":"BMC Med Inform Decis Mak"},{"issue":"Suppl 2","key":"1789_CR14","doi-asserted-by":"publisher","first-page":"57","DOI":"10.1186\/s12911-019-0763-6","volume":"19","author":"C Yu","year":"2019","unstructured":"Yu C, Liu J, Zhao H. Inverse reinforcement learning for intelligent mechanical ventilation and sedative dosing in intensive care units. BMC Med Inform Decis Mak. 2019;19(Suppl 2):57.","journal-title":"BMC Med Inform Decis Mak"},{"issue":"1","key":"1789_CR15","doi-asserted-by":"publisher","first-page":"347","DOI":"10.1186\/s12911-021-01711-7","volume":"21","author":"LA Beste","year":"2021","unstructured":"Beste LA, et al. Adapted time-varying covariates Cox model for predicting future cirrhosis development performs well in a large hepatitis C cohort. BMC Med Inform Decis Mak. 2021;21(1):347.","journal-title":"BMC Med Inform Decis Mak"},{"issue":"1","key":"1789_CR16","doi-asserted-by":"publisher","first-page":"e0208141","DOI":"10.1371\/journal.pone.0208141","volume":"14","author":"MA Konerman","year":"2019","unstructured":"Konerman MA, et al. Machine learning models to predict disease progression among veterans with hepatitis C virus. PLoS ONE. 2019;14(1):e0208141.","journal-title":"PLoS ONE"},{"issue":"6","key":"1789_CR17","doi-asserted-by":"publisher","first-page":"1832","DOI":"10.1002\/hep.27750","volume":"61","author":"MA Konerman","year":"2015","unstructured":"Konerman MA, et al. Improvement of predictive models of risk of disease progression in chronic hepatitis C by incorporating longitudinal data. Hepatology. 2015;61(6):1832\u201341.","journal-title":"Hepatology"},{"key":"1789_CR18","unstructured":"Sutton RS, Barto AG. Reinforcement learning: An introduction, 2nd ed. Reinforcement learning: An introduction, 2nd ed. 2018, Cambridge, MA, US: The MIT Press. xxii, 526-xxii, 526."},{"key":"1789_CR19","doi-asserted-by":"crossref","unstructured":"Doroudi S, Thomas P, Brunskill E. Importance of sampling for fair policy selection. In: 2018: Proceedings of the 27th International Joint Conference on Artificial Intelligence, pp. 5239\u20135243.","DOI":"10.24963\/ijcai.2018\/729"},{"key":"1789_CR20","doi-asserted-by":"publisher","first-page":"26","DOI":"10.1109\/MSP.2017.2743240","volume":"34","author":"K Arulkumaran","year":"2017","unstructured":"Arulkumaran K, Deisenroth MP, Brundage M, Bharath AA. A brief survey of deep reinforcement learning. IEEE Signal Process Mag. 2017;34:26\u201338.","journal-title":"IEEE Signal Process Mag"},{"key":"1789_CR21","unstructured":"Raghu A, et al. Behaviour policy estimation in off-policy policy evaluation: Calibration matters. arXiv preprint arXiv:1807.01066 (2018)."},{"key":"1789_CR22","unstructured":"Thomas P, Brunskill E. Data-efficient off-policy policy evaluation for reinforcement learning. In: International Conference on Machine Learning. 2016. PMLR."},{"key":"1789_CR23","doi-asserted-by":"publisher","first-page":"117","DOI":"10.1093\/jnci\/djq495","volume":"103","author":"A Mariotto","year":"2011","unstructured":"Mariotto A, Yabroff KR, Shao Y, Feuer EJ, Brown ML. Projections of the cost of cancer care in the United States: 2010\u20132020. J Natl Cancer Inst. 2011;103:117\u201328.","journal-title":"J Natl Cancer Inst"}],"container-title":["BMC Medical Informatics and Decision Making"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-022-01789-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12911-022-01789-7\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-022-01789-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,3,11]],"date-time":"2022-03-11T04:29:39Z","timestamp":1646972979000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcmedinformdecismak.biomedcentral.com\/articles\/10.1186\/s12911-022-01789-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3,11]]},"references-count":23,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2022,12]]}},"alternative-id":["1789"],"URL":"https:\/\/doi.org\/10.1186\/s12911-022-01789-7","relation":{},"ISSN":["1472-6947"],"issn-type":[{"value":"1472-6947","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,3,11]]},"assertion":[{"value":"24 September 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"22 February 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"11 March 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"All methods of the present study were performed in accordance with the relevant guidelines and regulations of the ethical committee of the VA Ann Arbor Healthcare System.Approval to conduct the study and a waiver of informed consent was obtained (IRB-2013\u2013219).from the Institutional Review Board at the VA Ann Arbor Healthcare System.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"A.G.S has received research support from Gilead. All other authors report no conflicts of interest.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"63"}}