{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,24]],"date-time":"2026-09-24T12:01:52Z","timestamp":1790251312972,"version":"4.1.0"},"reference-count":34,"publisher":"IOP Publishing","issue":"1","license":[{"start":{"date-parts":[[2024,12,24]],"date-time":"2024-12-24T00:00:00Z","timestamp":1734998400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/iopscience.iop.org\/page\/copyright"},{"start":{"date-parts":[[2024,12,24]],"date-time":"2024-12-24T00:00:00Z","timestamp":1734998400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/iopscience.iop.org\/info\/page\/text-and-data-mining"}],"funder":[{"DOI":"10.13039\/100007325","name":"American Association of Physicists in Medicine","doi-asserted-by":"crossref","award":["AAPM\/RSNA Doctoral Graduate Fellowship"],"award-info":[{"award-number":["AAPM\/RSNA Doctoral Graduate Fellowship"]}],"id":[{"id":"10.13039\/100007325","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100007210","name":"Varian","doi-asserted-by":"crossref","award":["Master Research Agreement Grant"],"award-info":[{"award-number":["Master Research Agreement Grant"]}],"id":[{"id":"10.13039\/100007210","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["iopscience.iop.org"],"crossmark-restriction":false},"short-container-title":["Phys. Med. Biol."],"published-print":{"date-parts":[[2025,1,5]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    <jats:italic>Purpose.<\/jats:italic>\n                    To develop a deep reinforcement learning (DRL) agent to self-interact with the treatment planning system to automatically generate intensity modulated radiation therapy (IMRT) treatment plans for head-and-neck (HN) cancer with consistent organ-at-risk (OAR) sparing performance.\n                    <jats:italic>Methods.<\/jats:italic>\n                    With IRB approval, one hundred and twenty HN patients receiving IMRT were included. The DRL agent was trained with 20 patients. During each inverse optimization process, the intermediate dosimetric endpoints\u2019 values, dose volume constraints\u2019 values and structure objective function losses were collected as the DRL\n                    <jats:italic>states<\/jats:italic>\n                    . By adjusting the objective constraints as\n                    <jats:italic>actions<\/jats:italic>\n                    , the agent learned to seek optimal rewards by balancing OAR sparing and planning target volume (PTV) coverage. Reward computed from current dosimetric endpoints and clinical objectives were sent back to the agent to update action policy during model training. The trained agent was evaluated with the rest 100 patients.\n                    <jats:italic>Results.<\/jats:italic>\n                    The DRL agent was able to generate a clinically acceptable IMRT plan within\n                    <jats:inline-formula>\n                      <jats:tex-math>\n                        \n                      <\/jats:tex-math>\n                      <mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\" overflow=\"scroll\">\n                        <mml:mrow>\n                          <mml:mn>12.4<\/mml:mn>\n                          <mml:mo>\u00b1<\/mml:mo>\n                          <mml:mn>3.1<\/mml:mn>\n                        <\/mml:mrow>\n                      <\/mml:math>\n                    <\/jats:inline-formula>\n                    min without human intervention. DRL plans showed lower PTV maximum dose (109.2%) compared to clinical plans (112.4%) (\n                    <jats:italic>p<\/jats:italic>\n                    &lt; .05). Average median dose of left parotid, right parotid, oral cavity, larynx, pharynx of DRL plans were 15.6 Gy, 12.2 Gy, 25.7 Gy, 27.3 Gy and 32.1 Gy respectively, comparable to 17.1 Gy, 15.7 Gy, 24.4 Gy, 23.7 Gy and 35.5 Gy of corresponding clinical plans. The maximum dose of cord + 5 mm, brainstem and mandible were also comparable between the two groups. In addition, DRL plans demonstrated reduced variability, as evidenced by smaller 95% confidence intervals. The total MU of the DRL plans was 1611 vs 1870 (\n                    <jats:italic>p<\/jats:italic>\n                    &lt; .05) of clinical plans. The results signaled the DRL\u2019s consistent planning strategy compared to the planners\u2019 occasional back-and-forth decision-making during planning.\n                    <jats:italic>Conclusion.<\/jats:italic>\n                    The proposed DRL agent is capable of efficiently generating HN IMRT plans with consistent quality.\n                  <\/jats:p>","DOI":"10.1088\/1361-6560\/ad965d","type":"journal-article","created":{"date-parts":[[2024,11,22]],"date-time":"2024-11-22T17:59:17Z","timestamp":1732298357000},"page":"015010","update-policy":"https:\/\/doi.org\/10.1088\/crossmark-policy","source":"Crossref","is-referenced-by-count":9,"title":["Automated treatment planning with deep reinforcement learning for head-and-neck (HN) cancer intensity modulated radiation therapy (IMRT)"],"prefix":"10.1088","volume":"70","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4559-5701","authenticated-orcid":true,"given":"Dongrong","family":"Yang","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1108-7626","authenticated-orcid":true,"given":"Xin","family":"Wu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9995-8101","authenticated-orcid":true,"given":"Xinyi","family":"Li","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-8209-4982","authenticated-orcid":true,"given":"Ryan","family":"Mansfield","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yibo","family":"Xie","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0748-7280","authenticated-orcid":true,"given":"Qiuwen","family":"Wu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Q","family":"Jackie Wu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yang","family":"Sheng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"266","published-online":{"date-parts":[[2024,12,24]]},"reference":[{"key":"pmbad965dbib1","doi-asserted-by":"publisher","first-page":"233","DOI":"10.1053\/j.seminoncol.2019.07.002","article-title":"Modern radiotherapy for head and neck cancer","volume":"46","author":"Alterio","year":"2019","journal-title":"Semin. Oncol."},{"key":"pmbad965dbib2","doi-asserted-by":"publisher","first-page":"4912","DOI":"10.3390\/cancers13194912","article-title":"An updated review on head and neck cancer treatment with radiation therapy","volume":"13","author":"Anderson","year":"2021","journal-title":"Cancers"},{"key":"pmbad965dbib3","doi-asserted-by":"publisher","first-page":"1695","DOI":"10.1016\/S0140-6736(08)60728-X","article-title":"Head and neck cancer","volume":"371","author":"Argiris","year":"2008","journal-title":"Lancet"},{"key":"pmbad965dbib4","doi-asserted-by":"publisher","first-page":"e99","DOI":"10.1016\/j.prro.2012.06.009","article-title":"How important is dosimetrist experience for intensity modulated radiation therapy? A comparative analysis of a head and neck case","volume":"3","author":"Batumalai","year":"2013","journal-title":"Pract. Radiat. Oncol."},{"key":"pmbad965dbib5","article-title":"Soft actor-critic for discrete action settings 2019 October 01","author":"Christodoulou","year":"2019"},{"key":"pmbad965dbib6","doi-asserted-by":"publisher","first-page":"7","DOI":"10.1016\/j.ejmp.2020.12.015","article-title":"Personalized automation of treatment planning in head-neck cancer: a step forward for quality in radiation therapy?","volume":"82","author":"Cilla","year":"2021","journal-title":"Phys. Med."},{"key":"pmbad965dbib7","article-title":"Addressing function approximation error in actor-critic methods","author":"Fujimoto","year":"2018","edition":"(eds)"},{"key":"pmbad965dbib8","doi-asserted-by":"publisher","DOI":"10.1088\/1361-6560\/ad4b90","article-title":"Human-like intelligent automatic treatment planning of head and neck cancer radiation therapy","volume":"69","author":"Gao","year":"2024","journal-title":"Phys. Med. Biol."},{"key":"pmbad965dbib9","doi-asserted-by":"publisher","first-page":"343","DOI":"10.1016\/j.radonc.2012.07.001","article-title":"Three-dimensional conformal radiotherapy (3D-CRT) versus intensity modulated radiation therapy (IMRT) in squamous cell carcinoma of the head and neck: a randomized controlled trial","volume":"104","author":"Gupta","year":"2012","journal-title":"Radiother. Oncol."},{"key":"pmbad965dbib10","first-page":"1861","article-title":"Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor","author":"Haarnoja","year":"2018a"},{"key":"pmbad965dbib11","article-title":"Soft actor-critic algorithms and applications","author":"Haarnoja","year":"2018b"},{"key":"pmbad965dbib12","doi-asserted-by":"publisher","first-page":"2","DOI":"10.1016\/j.ctro.2016.08.001","article-title":"Automatic treatment planning improves the clinical quality of head and neck cancer treatment plans","volume":"1","author":"Hansen","year":"2016","journal-title":"Clin. Transl. Radiat. Oncol."},{"key":"pmbad965dbib13","doi-asserted-by":"publisher","first-page":"6140","DOI":"10.1002\/mp.14544","article-title":"Artificial intelligence-based radiotherapy machine parameter optimization using reinforcement learning","volume":"47","author":"Hrinivich","year":"2020","journal-title":"Med. Phys."},{"key":"pmbad965dbib14","doi-asserted-by":"publisher","first-page":"4873","DOI":"10.1200\/JCO.2007.11.5501","article-title":"Prospective randomized study of intensity-modulated radiotherapy on salivary gland function in early-stage nasopharyngeal carcinoma patients","volume":"25","author":"Kam","year":"2007","journal-title":"J. Clin. Oncol."},{"key":"pmbad965dbib15","doi-asserted-by":"publisher","first-page":"4909","DOI":"10.1109\/TITS.2021.3054625","article-title":"Deep reinforcement learning for autonomous driving: a survey","volume":"23","author":"Kiran","year":"2021","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"pmbad965dbib16","doi-asserted-by":"publisher","first-page":"1238","DOI":"10.1177\/0278364913495721","article-title":"Reinforcement learning in robotics: a survey","volume":"32","author":"Kober","year":"2013","journal-title":"Int. J. Robot. Res."},{"key":"pmbad965dbib17","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s13014-017-0947-0","article-title":"Planning comparison of five automated treatment planning solutions for locally advanced head and neck cancer","volume":"13","author":"Krayenbuehl","year":"2018","journal-title":"Radiat. Oncol."},{"key":"pmbad965dbib18","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v31i1.10827","article-title":"Playing FPS games with deep reinforcement learning","author":"Lample","year":"2017","edition":"(eds)"},{"key":"pmbad965dbib19","doi-asserted-by":"publisher","first-page":"387","DOI":"10.1002\/hed.20332","article-title":"Intensity-modulated radiation therapy in head and neck cancers: an update","volume":"29","author":"Lee","year":"2007","journal-title":"Head Neck"},{"key":"pmbad965dbib20","doi-asserted-by":"publisher","first-page":"9","DOI":"10.1016\/j.radonc.2013.11.010","article-title":"Intensity-modulated radiation therapy for head and neck cancer: systematic review and meta-analysis","volume":"110","author":"Marta","year":"2014","journal-title":"Radiother. Oncol."},{"key":"pmbad965dbib21","doi-asserted-by":"publisher","first-page":"c4684","DOI":"10.1136\/bmj.c4684","article-title":"Head and neck cancer\u2014Part 1: epidemiology, presentation, and prevention","volume":"341","author":"Mehanna","year":"2010","journal-title":"Bmj"},{"key":"pmbad965dbib22","doi-asserted-by":"publisher","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"pmbad965dbib23","doi-asserted-by":"publisher","first-page":"14","DOI":"10.1038\/s41591-018-0320-3","article-title":"A call for deep-learning healthcare","volume":"25","author":"Norgeot","year":"2019","journal-title":"Nat. Med."},{"key":"pmbad965dbib24","doi-asserted-by":"publisher","first-page":"127","DOI":"10.1016\/S1470-2045(10)70290-4","article-title":"PARSPORT trial management group. Parotid-sparing intensity modulated versus conventional radiotherapy in head and neck cancer (PARSPORT): a phase 3 multicentre randomised controlled trial","volume":"12","author":"Nutting","year":"2011","journal-title":"Lancet Oncol."},{"key":"pmbad965dbib25","author":"Rummery","year":"1994"},{"key":"pmbad965dbib26","doi-asserted-by":"publisher","DOI":"10.1088\/1361-6560\/ac09a2","article-title":"A hierarchical deep reinforcement learning framework for intelligent automatic treatment planning of prostate cancer intensity modulated radiation therapy","volume":"66","author":"Shen","year":"2021","journal-title":"Phys. Med. Biol."},{"key":"pmbad965dbib27","doi-asserted-by":"publisher","DOI":"10.1088\/1361-6560\/ab18bf","article-title":"Intelligent inverse treatment planning via deep reinforcement learning, a proof-of-principle study in high dose-rate brachytherapy for cervical cancer","volume":"64","author":"Shen","year":"2019","journal-title":"Phys. Med. Biol."},{"key":"pmbad965dbib28","doi-asserted-by":"publisher","first-page":"2329","DOI":"10.1002\/mp.14114","article-title":"Operating a treatment planning system using a deep-reinforcement learning-based virtual treatment planner for prostate cancer intensity-modulated radiation therapy treatment planning","volume":"47","author":"Shen","year":"2020","journal-title":"Med. Phys."},{"key":"pmbad965dbib29","doi-asserted-by":"publisher","DOI":"10.3389\/fphy.2024.1331849","article-title":"A reinforcement learning agent for head and neck intensity-modulated radiation therapy","volume":"12","author":"Stephens","year":"2024","journal-title":"Front. Phys."},{"key":"pmbad965dbib30","doi-asserted-by":"publisher","first-page":"511","DOI":"10.1016\/j.ctrv.2011.01.004","article-title":"Intensity-modulated radiotherapy versus conventional and 3D conformal radiotherapy in patients with head and neck cancer: is there a worthwhile quality of life gain?","volume":"37","author":"Tribius","year":"2011","journal-title":"Cancer Treat Rev."},{"key":"pmbad965dbib31","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.ijrobp.2008.07.059","article-title":"Intensity-modulated radiotherapy reduces radiation-induced morbidity and improves health-related quality of life: results of a nonrandomized prospective study using a standardized follow-up program","volume":"74","author":"Vergeer","year":"2009","journal-title":"Int. J. Radiat. Oncol. Biol. Phys."},{"key":"pmbad965dbib32","doi-asserted-by":"publisher","first-page":"866","DOI":"10.1016\/j.ijrobp.2012.04.015","article-title":"Toward fully automated multicriterial plan generation: a prospective clinical study","volume":"85","author":"Voet","year":"2013","journal-title":"Int. J. Radiat. Oncol. Biol. Phys."},{"key":"pmbad965dbib33","doi-asserted-by":"publisher","first-page":"1076","DOI":"10.1016\/j.ijrobp.2020.10.019","article-title":"An interpretable planning bot for pancreas stereotactic body radiation therapy","volume":"109","author":"Zhang","year":"2021","journal-title":"Int. J. Radiat. Oncol. Biol. Phys."},{"key":"pmbad965dbib34","article-title":"Artificial intelligence-driven pancreas stereotactic body radiation therapy (SBRT) treatment planning","author":"Zhang","year":"2019","edition":"(eds)"}],"container-title":["Physics in Medicine &amp; Biology"],"original-title":[],"link":[{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ad965d","content-type":"text\/html","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ad965d\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ad965d","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ad965d\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ad965d\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ad965d\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ad965d\/pdf","content-type":"application\/pdf","content-version":"am","intended-application":"similarity-checking"},{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ad965d\/pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,12,24]],"date-time":"2024-12-24T01:32:05Z","timestamp":1735003925000},"score":1,"resource":{"primary":{"URL":"https:\/\/iopscience.iop.org\/article\/10.1088\/1361-6560\/ad965d"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,24]]},"references-count":34,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,12,24]]},"published-print":{"date-parts":[[2025,1,5]]}},"URL":"https:\/\/doi.org\/10.1088\/1361-6560\/ad965d","relation":{},"ISSN":["0031-9155","1361-6560"],"issn-type":[{"value":"0031-9155","type":"print"},{"value":"1361-6560","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,12,24]]},"assertion":[{"value":"Automated treatment planning with deep reinforcement learning for head-and-neck (HN) cancer intensity modulated radiation therapy (IMRT)","name":"article_title","label":"Article Title"},{"value":"Physics in Medicine & Biology","name":"journal_title","label":"Journal Title"},{"value":"paper","name":"article_type","label":"Article Type"},{"value":"\u00a9 2024 Institute of Physics and Engineering in Medicine. All rights, including for text and data mining, AI training, and similar technologies, are reserved.","name":"copyright_information","label":"Copyright Information"},{"value":"2024-08-26","name":"date_received","label":"Date Received","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2024-11-22","name":"date_accepted","label":"Date Accepted","group":{"name":"publication_dates","label":"Publication dates"}},{"value":"2024-12-24","name":"date_epub","label":"Online publication date","group":{"name":"publication_dates","label":"Publication dates"}}]}}