{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,10]],"date-time":"2026-07-10T18:34:30Z","timestamp":1783708470141,"version":"3.55.0"},"reference-count":34,"publisher":"Wiley","issue":"6","license":[{"start":{"date-parts":[[2024,4,26]],"date-time":"2024-04-26T00:00:00Z","timestamp":1714089600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000905","name":"Commonwealth Fund","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000905","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["aapm.onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Medical Physics"],"published-print":{"date-parts":[[2024,6]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Background<\/jats:title><jats:p>Volumetric modulated arc therapy (VMAT) machine parameter optimization (MPO) remains computationally expensive and sensitive to input dose objectives creating challenges for manual and automatic planning. Reinforcement learning (RL) involves machine learning through extensive trial\u2010and\u2010error, demonstrating performance exceeding humans, and existing algorithms in several domains.<\/jats:p><\/jats:sec><jats:sec><jats:title>Purpose<\/jats:title><jats:p>To develop and evaluate an RL approach for VMAT MPO for localized prostate cancer to rapidly and automatically generate deliverable VMAT plans for a clinical linear accelerator (linac) and compare resultant dosimetry to clinical plans.<\/jats:p><\/jats:sec><jats:sec><jats:title>Methods<\/jats:title><jats:p>We extended our previous RL approach to enable VMAT MPO of a 3D beam model for a clinical linac through a policy network. It accepts an input state describing the current control point and predicts continuous machine parameters for the next control point, which are used to update the input state, repeating until plan termination. RL training was conducted to minimize a dose\u2010based cost function for prescription of 60\u00a0Gy in 20 fractions using CT scans and contours from 136 retrospective localized prostate cancer patients, 20 of which had existing plans used to initialize training. Data augmentation was employed to mitigate over\u2010fitting, and parameter exploration was achieved using Gaussian perturbations. Following training, RL VMAT was applied to an independent cohort of 15 patients, and the resultant dosimetry was compared to clinical plans. We also combined the RL approach with our clinical treatment planning system (TPS) to automate final plan refinement, and creating the potential for manual review and edits as required for clinical use.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>RL training was conducted for 5000 iterations, producing 40 000 plans during exploration. Mean\u00a0\u00b1\u00a0SD execution time to produce deliverable VMAT plans in the test cohort was 3.3\u00a0\u00b1\u00a00.5 s which were automatically refined in the TPS taking an additional 77.4\u00a0\u00b1\u00a05.8 s. When normalized to provide equivalent target coverage, the RL+TPS plans provided a similar mean\u00a0\u00b1\u00a0SD overall maximum dose of 63.2\u00a0\u00b1\u00a00.6\u00a0Gy and a lower mean rectum dose of 17.4\u00a0\u00b1\u00a07.4 compared to 63.9\u00a0\u00b1\u00a01.5\u00a0Gy (<jats:italic>p<\/jats:italic>\u00a0=\u00a00.061) and 21.0\u00a0\u00b1\u00a06.0 (<jats:italic>p<\/jats:italic>\u00a0=\u00a00.024) for the clinical plans.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusions<\/jats:title><jats:p>An approach for VMAT MPO using RL for a clinical linac model was developed and applied to automatically generate deliverable plans for localized prostate cancer patients, and when combined with the clinical TPS shows potential to rapidly generate high\u2010quality plans. The RL VMAT approach shows promise to discover advanced linac control policies through trial\u2010and\u2010error, and algorithm limitations and future directions are identified and discussed.<\/jats:p><\/jats:sec>","DOI":"10.1002\/mp.17100","type":"journal-article","created":{"date-parts":[[2024,4,26]],"date-time":"2024-04-26T14:24:07Z","timestamp":1714141447000},"page":"3972-3984","update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":14,"title":["Clinical VMAT machine parameter optimization for localized prostate cancer using deep reinforcement learning"],"prefix":"10.1002","volume":"51","author":[{"given":"William T.","family":"Hrinivich","sequence":"first","affiliation":[{"name":"Department of Radiation Oncology and Molecular Radiation Sciences Johns Hopkins University  Baltimore Maryland USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mahasweta","family":"Bhattacharya","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology and Molecular Radiation Sciences Johns Hopkins University  Baltimore Maryland USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lina","family":"Mekki","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology and Molecular Radiation Sciences Johns Hopkins University  Baltimore Maryland USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Todd","family":"McNutt","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology and Molecular Radiation Sciences Johns Hopkins University  Baltimore Maryland USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xun","family":"Jia","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology and Molecular Radiation Sciences Johns Hopkins University  Baltimore Maryland USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Heng","family":"Li","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology and Molecular Radiation Sciences Johns Hopkins University  Baltimore Maryland USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Daniel Y.","family":"Song","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology and Molecular Radiation Sciences Johns Hopkins University  Baltimore Maryland USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Junghoon","family":"Lee","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology and Molecular Radiation Sciences Johns Hopkins University  Baltimore Maryland USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"311","published-online":{"date-parts":[[2024,4,26]]},"reference":[{"key":"e_1_2_8_2_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijrobp.2008.02.047"},{"key":"e_1_2_8_3_1","doi-asserted-by":"publisher","DOI":"10.1118\/1.4908224"},{"key":"e_1_2_8_4_1","doi-asserted-by":"publisher","DOI":"10.1002\/mp.14074"},{"key":"e_1_2_8_5_1","doi-asserted-by":"publisher","DOI":"10.1118\/1.3491675"},{"key":"e_1_2_8_6_1","doi-asserted-by":"publisher","DOI":"10.1088\/0031\u20109155\/57\/14\/4569"},{"key":"e_1_2_8_7_1","doi-asserted-by":"publisher","DOI":"10.1177\/1533033819873922"},{"key":"e_1_2_8_8_1","doi-asserted-by":"publisher","DOI":"10.1002\/mp.13526"},{"key":"e_1_2_8_9_1","doi-asserted-by":"publisher","DOI":"10.1002\/acm2.13337"},{"key":"e_1_2_8_10_1","doi-asserted-by":"publisher","DOI":"10.1002\/mp.13896"},{"key":"e_1_2_8_11_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0232697"},{"key":"e_1_2_8_12_1","doi-asserted-by":"publisher","DOI":"10.1088\/1361\u20106560\/abdd8a"},{"key":"e_1_2_8_13_1","doi-asserted-by":"publisher","DOI":"10.1002\/mp.13955"},{"key":"e_1_2_8_14_1","doi-asserted-by":"publisher","DOI":"10.1088\/1361\u20106560\/aba5eb"},{"key":"e_1_2_8_15_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejmp.2022.05.008"},{"key":"e_1_2_8_16_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.adro.2021.100672"},{"key":"e_1_2_8_17_1","volume-title":"Reinforcement Learning: An Introduction","author":"Sutton RS","year":"2018"},{"key":"e_1_2_8_18_1","doi-asserted-by":"publisher","DOI":"10.1038\/nature24270"},{"key":"e_1_2_8_19_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41586\u2010019\u20101724\u2010z"},{"key":"e_1_2_8_20_1","doi-asserted-by":"publisher","DOI":"10.1038\/nature14236"},{"key":"e_1_2_8_21_1","unstructured":"OpenAI.GPT\u20104 Technical Report. Published online March 15 2023.http:\/\/arxiv.org\/abs\/2303.08774"},{"key":"e_1_2_8_22_1","unstructured":"ChristianoP LeikeJ BrownTB MarticM LeggS AmodeiD.Deep reinforcement learning from human preferences. Published online June 12 2017.http:\/\/arxiv.org\/abs\/1706.03741"},{"key":"e_1_2_8_23_1","doi-asserted-by":"publisher","DOI":"10.1088\/1361\u20106560\/ab18bf"},{"key":"e_1_2_8_24_1","doi-asserted-by":"publisher","DOI":"10.1088\/1361\u20106560\/ac09a2"},{"key":"e_1_2_8_25_1","doi-asserted-by":"publisher","DOI":"10.1002\/mp.14114"},{"key":"e_1_2_8_26_1","doi-asserted-by":"publisher","DOI":"10.1002\/mp.14544"},{"key":"e_1_2_8_27_1","unstructured":"SilverD LeverG HeessN DegrisT WierstraD RiedmillerM.Deterministic policy gradient algorithms. InInternational conference on machinelearning pp.387\u2010395. Pmlr 2014."},{"key":"e_1_2_8_28_1","doi-asserted-by":"publisher","DOI":"10.1120\/jacmp.v14i2.4136"},{"key":"e_1_2_8_29_1","doi-asserted-by":"publisher","DOI":"10.1118\/1.3483785"},{"key":"e_1_2_8_30_1","doi-asserted-by":"publisher","DOI":"10.1118\/1.596360"},{"key":"e_1_2_8_31_1","doi-asserted-by":"publisher","DOI":"10.1016\/S1470\u20102045(16)30102\u20104"},{"key":"e_1_2_8_32_1","unstructured":"KingmaDP BaJL.Adam: a method for stochastic optimization.3rd International Conference on Learning Representations ICLR 2015 \u2010 Conference Track Proceedings;2015.http:\/\/arxiv.org\/abs\/1412.6980"},{"key":"e_1_2_8_33_1","unstructured":"LillicrapTP HuntJJ PritzelA et\u00a0al.Continuous control with deep reinforcement learning. Published online September 9 2015.http:\/\/arxiv.org\/abs\/1509.02971"},{"key":"e_1_2_8_34_1","doi-asserted-by":"publisher","DOI":"10.3171\/jns.2000.93.supplement"},{"key":"e_1_2_8_35_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41591\u2010021\u201001359\u2010w"}],"container-title":["Medical Physics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/aapm.onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/mp.17100","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,23]],"date-time":"2025-08-23T00:22:21Z","timestamp":1755908541000},"score":1,"resource":{"primary":{"URL":"https:\/\/aapm.onlinelibrary.wiley.com\/doi\/10.1002\/mp.17100"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,26]]},"references-count":34,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2024,6]]}},"alternative-id":["10.1002\/mp.17100"],"URL":"https:\/\/doi.org\/10.1002\/mp.17100","archive":["Portico"],"relation":{},"ISSN":["0094-2405","2473-4209"],"issn-type":[{"value":"0094-2405","type":"print"},{"value":"2473-4209","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,4,26]]},"assertion":[{"value":"2023-09-07","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-04-14","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-04-26","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}