{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,4]],"date-time":"2026-06-04T23:12:48Z","timestamp":1780614768721,"version":"3.54.1"},"reference-count":25,"publisher":"Springer Science and Business Media LLC","issue":"S3","license":[{"start":{"date-parts":[[2020,7,1]],"date-time":"2020-07-01T00:00:00Z","timestamp":1593561600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,7,9]],"date-time":"2020-07-09T00:00:00Z","timestamp":1594252800000},"content-version":"vor","delay-in-days":8,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Med Inform Decis Mak"],"published-print":{"date-parts":[[2020,7]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec>\n<jats:title>Background<\/jats:title>\n<jats:p>Reinforcement learning (RL) provides a promising technique to solve complex sequential decision making problems in healthcare domains. Recent years have seen a great progress of applying RL in addressing decision-making problems in Intensive Care Units (ICUs). However, since the goal of traditional RL algorithms is to maximize a long-term reward function, exploration in the learning process may have a fatal impact on the patient. As such, a short-term goal should also be considered to keep the patient stable during the treating process.<\/jats:p>\n<\/jats:sec><jats:sec>\n<jats:title>Methods<\/jats:title>\n<jats:p>We use a Supervised-Actor-Critic (SAC) RL algorithm to address this problem by combining the long-term goal-oriented characteristics of RL with the short-term goal of supervised learning. We evaluate the differences between SAC and traditional Actor-Critic (AC) algorithms in addressing the decision making problems of ventilation and sedative dosing in ICUs.<\/jats:p>\n<\/jats:sec><jats:sec>\n<jats:title>Results<\/jats:title>\n<jats:p>Results show that SAC is much more efficient than the traditional AC algorithm in terms of convergence rate and data utilization.<\/jats:p>\n<\/jats:sec><jats:sec>\n<jats:title>Conclusions<\/jats:title>\n<jats:p>The SAC algorithm not only aims to cure patients in the long term, but also reduces the degree of deviation from the strategy applied by clinical doctors and thus improves the therapeutic effect.<\/jats:p>\n<\/jats:sec>","DOI":"10.1186\/s12911-020-1120-5","type":"journal-article","created":{"date-parts":[[2020,7,9]],"date-time":"2020-07-09T08:08:50Z","timestamp":1594282130000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":29,"title":["Supervised-actor-critic reinforcement learning for intelligent mechanical ventilation and sedative dosing in intensive care units"],"prefix":"10.1186","volume":"20","author":[{"given":"Chao","family":"Yu","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Guoqi","family":"Ren","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yinzhao","family":"Dong","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2020,7,9]]},"reference":[{"key":"1120_CR1","volume-title":"Reinforcement Learning: An Introduction","author":"RS Sutton","year":"1998","unstructured":"Sutton RS, Barto AG. Reinforcement Learning: An Introduction. Cambridge, Massachusetts: The MIT press; 1998."},{"key":"1120_CR2","unstructured":"Yu C, Liu J, Nemati S. Reinforcement learning in healthcare: A survey. 2019. arXiv preprint arXiv:1908.08796."},{"issue":"5","key":"1120_CR3","doi-asserted-by":"publisher","first-page":"661","DOI":"10.1586\/17434440.2013.827515","volume":"10","author":"MK Bothe","year":"2013","unstructured":"Bothe MK, Dickens L, Reichel K, Tellmann A, Ellger B, Westphal M, Faisal AA. The use of reinforcement learning algorithms to meet the challenges of an artificial pancreas. Expert Rev Med Devices. 2013; 10(5):661\u201373.","journal-title":"Expert Rev Med Devices"},{"issue":"12","key":"1120_CR4","doi-asserted-by":"publisher","first-page":"6690","DOI":"10.1002\/mp.12625","volume":"44","author":"HH Tseng","year":"2017","unstructured":"Tseng HH, Luo Y, Cui S, Chien JT, Ten Haken RK, El Naqa I. Deep reinforcement learning for automated radiation adaptation in lung cancer. Med Phys. 2017; 44(12):6690\u2013705.","journal-title":"Med Phys"},{"key":"1120_CR5","volume-title":"2019 IEEE International Conference on Healthcare Informatics (ICHI)","author":"C Yu","year":"2019","unstructured":"Yu C, Ren G, Liu J. Deep Inverse Reinforcement Learning for Sepsis Treatment. In: 2019 IEEE International Conference on Healthcare Informatics (ICHI). New York: IEEE: 2019. p. 1\u20133."},{"issue":"1-2","key":"1120_CR6","doi-asserted-by":"publisher","first-page":"109","DOI":"10.1007\/s10994-010-5229-0","volume":"84","author":"SM Shortreed","year":"2011","unstructured":"Shortreed SM, Laber E, Lizotte DJ, Stroup TS, Pineau J, Murphy SA. Informing sequential clinical decision-making through reinforcement learning: an empirical study. Mach Learn. 2011; 84(1-2):109\u201336.","journal-title":"Mach Learn"},{"issue":"07","key":"1120_CR7","doi-asserted-by":"publisher","first-page":"1750012","DOI":"10.1142\/S0129065717500125","volume":"27","author":"V Nagaraj","year":"2017","unstructured":"Nagaraj V, Lamperski A, Netoff TI. Seizure control in a computational model using a reinforcement learning stimulation paradigm. Int J Neural Syst. 2017; 27(07):1750012.","journal-title":"Int J Neural Syst"},{"issue":"2","key":"1120_CR8","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1186\/s12911-019-0755-6","volume":"19","author":"C Yu","year":"2019","unstructured":"Yu C, Dong Y, Liu J, Ren G. Incorporating causal factors into reinforcement learning for dynamic treatment regimes in HIV. BMC Med Inform Decis Making. 2019; 19(2):60.","journal-title":"BMC Med Inform Decis Making"},{"key":"1120_CR9","volume-title":"Advances in neural information processing systems","author":"VR Konda","year":"2000","unstructured":"Konda VR, Tsitsiklis JN. Actor-critic algorithms. In: Advances in neural information processing systems. Cambridge: MIT Press: 2000. p. 1008\u201314."},{"issue":"2","key":"1120_CR10","doi-asserted-by":"publisher","first-page":"444","DOI":"10.1109\/JPROC.2015.2501978","volume":"104","author":"AE Johnson","year":"2016","unstructured":"Johnson AE, Ghassemi MM, Nemati S, Niehaus KE, Clifton DA, Clifford GD. Machine learning and decision support in critical care. Proc IEEE Inst Electr Electron Eng. 2016; 104(2):444\u201366.","journal-title":"Proc IEEE Inst Electr Electron Eng"},{"issue":"11","key":"1120_CR11","doi-asserted-by":"publisher","first-page":"1716","DOI":"10.1038\/s41591-018-0213-5","volume":"24","author":"M Komorowski","year":"2018","unstructured":"Komorowski M, Celi LA, Badawi O, Gordon AC, Faisal AA. The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care. Nat Med. 2018; 24(11):1716\u201320.","journal-title":"Nat Med"},{"key":"1120_CR12","unstructured":"Raghu A, Komorowski M, Ahmed I, Celi L, Szolovits P, Ghassemi M. Deep reinforcement learning for sepsis treatment. 2017. arXiv preprint arXiv:1711.09602."},{"key":"1120_CR13","volume-title":"Machine Learning for Healthcare Conference","author":"A Raghu","year":"2017","unstructured":"Raghu A, Komorowski M, Celi LA, Szolovits P, Ghassemi M. Continuous State-Space Models for Optimal Sepsis Treatment: A Deep Reinforcement Learning Approach. In: Machine Learning for Healthcare Conference. Cambridge: MIT Press: 2017. p. 147\u201363."},{"key":"1120_CR14","doi-asserted-by":"publisher","first-page":"54","DOI":"10.1016\/j.bspc.2015.05.013","volume":"22","author":"R Padmanabhan","year":"2015","unstructured":"Padmanabhan R, Meskin N, Haddad WM. Closed-loop control of anesthesia and mean arterial pressure using reinforcement learning. Biomed Signal Process Control. 2015; 22:54\u201364.","journal-title":"Biomed Signal Process Control"},{"key":"1120_CR15","doi-asserted-by":"publisher","first-page":"131","DOI":"10.1016\/j.mbs.2019.01.012","volume":"309","author":"R Padmanabhan","year":"2019","unstructured":"Padmanabhan R, Meskin N, Haddad WM. Optimal adaptive control of drug dosing using integral reinforcement learning. Math Biosci. 2019; 309:131\u201342.","journal-title":"Math Biosci"},{"key":"1120_CR16","unstructured":"Prasad N, Cheng LF, Chivers C, Draugelis M, Engelhardt BE. A reinforcement learning approach to weaning of mechanical ventilation in intensive care units. 2017. arXiv preprint arXiv:1704.06300."},{"key":"1120_CR17","volume-title":"39th International Conference on Information Systems","author":"CP Utomo","year":"2018","unstructured":"Utomo CP, Li X, Chen W. Treatment Recommendation in Critical Care: A Scalable and Interpretable Approach in Partially Observable Health States. In: 39th International Conference on Information Systems. New York: Curran Associates: 2018. p. 1\u20139."},{"key":"1120_CR18","volume-title":"2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)","author":"S Nemati","year":"2016","unstructured":"Nemati S, Ghassemi MM, Clifford GD. Optimal medication dosing from suboptimal clinical examples: A deep reinforcement learning approach. In: 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). New York: IEEE: 2016. p. 2978\u201381."},{"issue":"2","key":"1120_CR19","doi-asserted-by":"publisher","first-page":"57","DOI":"10.1186\/s12911-019-0763-6","volume":"19","author":"C Yu","year":"2019","unstructured":"Yu C, Liu J, Zhao H. Inverse reinforcement learning for intelligent mechanical ventilation and sedative dosing in intensive care units. BMC Med Inform Decis Making. 2019; 19(2):57.","journal-title":"BMC Med Inform Decis Making"},{"key":"1120_CR20","unstructured":"Chang CH, Mai M, Goldenberg A. Dynamic Measurement Scheduling for Event Forecasting using Deep RL. 2019. arXiv preprint arXiv:1901.09699."},{"key":"1120_CR21","doi-asserted-by":"publisher","first-page":"160035","DOI":"10.1038\/sdata.2016.35","volume":"3","author":"AE Johnson","year":"2016","unstructured":"Johnson AE, Pollard TJ, Shen L, Li-wei HL, Feng M, Ghassemi M, Moody B, Szolovits P, Celi LA, Mark RG. MIMIC-III, a freely accessible critical care database. Sci Data. 2016; 3:160035.","journal-title":"Sci Data"},{"key":"1120_CR22","volume-title":"Support vector machines. An Introduction to Support Vector Machines and Other Kernel-based Learning Methods","author":"J Shawe-Taylor","year":"2000","unstructured":"Shawe-Taylor J, Cristianini N. Support vector machines. An Introduction to Support Vector Machines and Other Kernel-based Learning Methods. United Kingdom: Cambridge university press; 2000, pp. 93\u2013112."},{"key":"1120_CR23","doi-asserted-by":"publisher","DOI":"10.1109\/9780470544785","volume-title":"Handbook of learning and approximate dynamic programming","author":"J Si","year":"2004","unstructured":"Si J, Barto AG, Powell WB, Wunsch D. Supervised actor-critic reinforcement learning. In: Handbook of learning and approximate dynamic programming. London: IEEE Press: 2004. p. 359\u201380."},{"key":"1120_CR24","volume-title":"Advances in neural information processing system","author":"M Zinkevich","year":"2010","unstructured":"Zinkevich M, Weimer M, Li L, Smola AJ. Parallelized stochastic gradient descent. In: Advances in neural information processing system. Cambridge: MIT Press: 2010. p. 2595\u2013603."},{"key":"1120_CR25","volume-title":"Proc. 3rd Annual PLD Design Conference & Exhibit, vol. 1","author":"S Golson","year":"1993","unstructured":"Golson S. One-hot state machine design for FPGAs. In: Proc. 3rd Annual PLD Design Conference & Exhibit, vol. 1. New York: IEEE: 1993."}],"container-title":["BMC Medical Informatics and Decision Making"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-020-1120-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12911-020-1120-5\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-020-1120-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,7,8]],"date-time":"2021-07-08T23:20:28Z","timestamp":1625786428000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcmedinformdecismak.biomedcentral.com\/articles\/10.1186\/s12911-020-1120-5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,7]]},"references-count":25,"journal-issue":{"issue":"S3","published-print":{"date-parts":[[2020,7]]}},"alternative-id":["1120"],"URL":"https:\/\/doi.org\/10.1186\/s12911-020-1120-5","relation":{},"ISSN":["1472-6947"],"issn-type":[{"value":"1472-6947","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,7]]},"assertion":[{"value":"9 July 2020","order":1,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Not applicable.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare that they have no competing interests.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"124"}}