{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,16]],"date-time":"2026-03-16T13:55:36Z","timestamp":1773669336923,"version":"3.50.1"},"reference-count":33,"publisher":"Wiley","issue":"7","license":[{"start":{"date-parts":[[2025,5,31]],"date-time":"2025-05-31T00:00:00Z","timestamp":1748649600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000054","name":"National Cancer Institute","doi-asserted-by":"publisher","award":["R37CA214639"],"award-info":[{"award-number":["R37CA214639"]}],"id":[{"id":"10.13039\/100000054","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000054","name":"National Cancer Institute","doi-asserted-by":"publisher","award":["R01CA254377"],"award-info":[{"award-number":["R01CA254377"]}],"id":[{"id":"10.13039\/100000054","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000054","name":"National Cancer Institute","doi-asserted-by":"publisher","award":["R01CA237269"],"award-info":[{"award-number":["R01CA237269"]}],"id":[{"id":"10.13039\/100000054","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["aapm.onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Medical Physics"],"published-print":{"date-parts":[[2025,7]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Achieving highly efficient treatment planning in intensity\u2010modulated radiotherapy (IMRT) is challenging due to the complex interactions between radiation beams and the human body. The introduction of artificial intelligence (AI) has automated treatment planning, significantly improving efficiency. However, existing automatic treatment planning agents often rely on supervised or unsupervised AI models that require large datasets of high\u2010quality patient data for training. Additionally, these networks are generally not universally applicable across patient cases from different institutions and can be vulnerable to adversarial attacks. Deep reinforcement learning (DRL), which mimics the trial\u2010and\u2010error process used by human planners, offers a promising new approach to address these challenges.\u00a0<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Purpose<\/jats:title>\n                    <jats:p>This work aims to develop a stochastic policy\u2010based DRL agent for automatic treatment planning that facilitates effective training with limited datasets, universal applicability across diverse patient datasets, and robust performance under adversarial attacks.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We employ an actor\u2013critic with experience replay (ACER) architecture to develop the automatic treatment planning agent. This agent operates the treatment planning system (TPS) for inverse treatment planning by automatically tuning treatment planning parameters (TPPs). We use prostate cancer IMRT patient cases as our testbed, which includes one target and two organs at risk (OARs), along with 18 discrete TPP tuning actions. The network takes dose\u2013volume histograms (DVHs) as input and outputs a policy for effective TPP tuning, accompanied by an evaluation function for that policy. Training utilizes DVHs from treatment plans generated by an in\u2010house TPS under randomized TPPs for a single patient case, with validation conducted on two other independent cases. Both online asynchronous learning and offline, sample\u2010efficient experience replay methods are employed to update the network parameters. After training, six groups, comprising more than 300 initial treatment plans drawn from three datasets, were used for testing. These groups have beam and anatomical configurations distinct from those of the training case. The ProKnow scoring system for prostate cancer IMRT, with a maximum score of 9, is used to evaluate plan quality. The robustness of the network is further assessed through adversarial attacks using the fast gradient sign method (FGSM).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Despite being trained on treatment plans from a single patient case, the network converges efficiently when validated on two independent cases. For testing performance, the mean  standard deviation of the plan scores across all test cases before ACER\u2010based treatment planning is . After implementing ACER\u2010based treatment planning,  of the cases achieve a perfect score of 9, with only  scoring between 8 and 9, and no cases being below 7. The corresponding mean  standard deviation is . This performance highlights the ACER agent's high generality across patient data from various sources. Further analysis indicates that the ACER agent effectively prioritizes leading reasonable TPP tuning actions over obviously unsuitable ones by several orders of magnitude, showing its efficacy. Additionally, results from FGSM attacks demonstrate that the ACER\u2010based agent remains comparatively robust against various levels of\u00a0perturbation.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>We successfully trained a DRL agent using the ACER technique for high\u2010quality treatment planning in prostate cancer IMRT. It achieves high generality across diverse patient datasets and exhibits high robustness against adversarial\u00a0attacks.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1002\/mp.17915","type":"journal-article","created":{"date-parts":[[2025,5,31]],"date-time":"2025-05-31T23:44:48Z","timestamp":1748735088000},"update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Actor critic with experience replay\u2010based automatic treatment planning for prostate cancer intensity modulated radiotherapy"],"prefix":"10.1002","volume":"52","author":[{"given":"Md Mainul","family":"Abrar","sequence":"first","affiliation":[{"name":"Department of Physics The University of Texas at Arlington Arlington Texas USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Parvat","family":"Sapkota","sequence":"additional","affiliation":[{"name":"Department of Physics The University of Texas at Arlington Arlington Texas USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Damon","family":"Sprouts","sequence":"additional","affiliation":[{"name":"Department of Physics The University of Texas at Arlington Arlington Texas USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xun","family":"Jia","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology and Molecular Radiation Sciences Johns Hopkins University Baltimore Maryland USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yujie","family":"Chi","sequence":"additional","affiliation":[{"name":"Department of Physics The University of Texas at Arlington Arlington Texas USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"311","published-online":{"date-parts":[[2025,5,31]]},"reference":[{"key":"e_1_2_8_2_1","doi-asserted-by":"publisher","DOI":"10.1259\/bjr\/65676879"},{"key":"e_1_2_8_3_1","doi-asserted-by":"publisher","DOI":"10.1088\/1361-6560\/aba5eb"},{"key":"e_1_2_8_4_1","doi-asserted-by":"crossref","unstructured":"HeK ZhangX RenS SunJ.Deep Residual Learning for Image Recognition. In:2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2016:770\u2010778.","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_2_8_5_1","doi-asserted-by":"crossref","unstructured":"HuangG LiuZ Van DerMaatenL WeinbergerKQ.Densely Connected Convolutional Networks. In:2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2017:2261\u20102269.","DOI":"10.1109\/CVPR.2017.243"},{"key":"e_1_2_8_6_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejmp.2022.05.008"},{"key":"e_1_2_8_7_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.phro.2021.07.008"},{"key":"e_1_2_8_8_1","doi-asserted-by":"crossref","unstructured":"RonnebergerO FischerP BroxT.U\u2010net: Convolutional networks for biomedical image segmentation. In:Medical Image Computing and Computer\u2010Assisted Intervention\u2010MICCAI 2015: 18th International Conference Munich Germany October 5\u20109 2015 proceedings part III 18 Springer;2015:234\u2010241.","DOI":"10.1007\/978-3-319-24574-4_28"},{"key":"e_1_2_8_9_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.radonc.2020.10.027"},{"key":"e_1_2_8_10_1","doi-asserted-by":"publisher","DOI":"10.1126\/science.aaw4399"},{"key":"e_1_2_8_11_1","volume-title":"Reinforcement Learning: An Introduction","author":"Sutton RS","year":"2018"},{"key":"e_1_2_8_12_1","doi-asserted-by":"publisher","DOI":"10.1038\/nature14236"},{"key":"e_1_2_8_13_1","doi-asserted-by":"publisher","DOI":"10.1088\/1361-6560\/ab18bf"},{"key":"e_1_2_8_14_1","doi-asserted-by":"publisher","DOI":"10.1002\/mp.14114"},{"key":"e_1_2_8_15_1","doi-asserted-by":"publisher","DOI":"10.1002\/mp.14712"},{"key":"e_1_2_8_16_1","doi-asserted-by":"publisher","DOI":"10.1088\/1361-6560\/ac09a2"},{"key":"e_1_2_8_17_1","doi-asserted-by":"publisher","DOI":"10.1088\/2057-1976\/ac6d82"},{"key":"e_1_2_8_18_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.radonc.2023.109685"},{"key":"e_1_2_8_19_1","doi-asserted-by":"crossref","unstructured":"ZhuJ WuF ZhaoJ.An overview of the action space for deep reinforcement learning. In:Proceedings of the 2021 4th International Conference on Algorithms Computing and Artificial Intelligence 2021:1\u201310.","DOI":"10.1145\/3508546.3508598"},{"key":"e_1_2_8_20_1","unstructured":"HuangS PapernotN GoodfellowI DuanY AbbeelP.Adversarial attacks on neural network policies.arXiv preprint arXiv:1702.02284.2017."},{"key":"e_1_2_8_21_1","article-title":"Sample efficient actor\u2010critic with experience replay","author":"Wang Z","year":"2016","journal-title":"arXiv preprint arXiv:1611.01224"},{"key":"e_1_2_8_22_1","doi-asserted-by":"crossref","unstructured":"AbrarMM JiaX ChiY.Deep Actor Critic Experience Replay (ACER) Reinforcement Learning\u2010Based Automatic Treatment Planning for Intensity Modulated Radiotherapy. In:AAPM 66th Annual Meeting & Exhibition AAPM;2024.","DOI":"10.1002\/mp.17915"},{"key":"e_1_2_8_23_1","unstructured":"MnihV BadiaAP MirzaM et\u00a0al.Asynchronous methods for deep reinforcement learning. In:International Conference on Machine LearningPMLR;2016:1928\u20101937."},{"key":"e_1_2_8_24_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1022628806385"},{"key":"e_1_2_8_25_1","unstructured":"Varian. Eclipse Photon and Electron Algorithms Reference Guide.2014."},{"key":"e_1_2_8_26_1","first-page":"1057","article-title":"Policy gradient methods for reinforcement learning with function approximation","volume":"12","author":"Sutton RS","year":"1999","journal-title":"Adv Neural Inf Process Syst"},{"key":"e_1_2_8_27_1","first-page":"1054","article-title":"Safe and efficient off\u2010policy reinforcement learning","volume":"29","author":"Munos R","year":"2016","journal-title":"Adv Neural Inf Process Syst"},{"key":"e_1_2_8_28_1","unstructured":"SchulmanJ LevineS AbbeelP JordanM MoritzP.Trust region policy optimization. In:International Conference on Machine Learning.PMLR;2015:1889\u20131897."},{"issue":"2","key":"e_1_2_8_29_1","first-page":"26","article-title":"Lecture 6.5\u2010rmsprop: Divide the gradient by a running average of its recent magnitude","volume":"4","author":"Tieleman T","year":"2012","journal-title":"COURSERA: Neural Net Mach Learn"},{"key":"e_1_2_8_30_1","doi-asserted-by":"publisher","DOI":"10.1186\/2047-217X-3-37"},{"key":"e_1_2_8_31_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.dib.2017.03.037"},{"key":"e_1_2_8_32_1","article-title":"Intriguing properties of neural networks","author":"Szegedy C","year":"2013","journal-title":"arXiv preprint arXiv:1312.6199"},{"key":"e_1_2_8_33_1","doi-asserted-by":"crossref","unstructured":"BiggioB RoliF.Wild patterns: Ten years after the rise of adversarial Mach Learn. In:Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security 2018:2154\u20132156.","DOI":"10.1145\/3243734.3264418"},{"key":"e_1_2_8_34_1","article-title":"Explaining and harnessing adversarial examples","author":"Goodfellow IJ","year":"2014","journal-title":"arXiv preprint arXiv:1412.6572"}],"container-title":["Medical Physics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/aapm.onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/mp.17915","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,12,8]],"date-time":"2025-12-08T10:33:20Z","timestamp":1765190000000},"score":1,"resource":{"primary":{"URL":"https:\/\/aapm.onlinelibrary.wiley.com\/doi\/10.1002\/mp.17915"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,5,31]]},"references-count":33,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2025,7]]}},"alternative-id":["10.1002\/mp.17915"],"URL":"https:\/\/doi.org\/10.1002\/mp.17915","archive":["Portico"],"relation":{},"ISSN":["0094-2405","2473-4209"],"issn-type":[{"value":"0094-2405","type":"print"},{"value":"2473-4209","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,5,31]]},"assertion":[{"value":"2024-11-06","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-05-08","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-05-31","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"e17915"}}