{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,26]],"date-time":"2026-10-26T17:00:26Z","timestamp":1793034026039,"version":"4.5.0"},"reference-count":33,"publisher":"Oxford University Press (OUP)","issue":"1174","license":[{"start":{"date-parts":[[2025,7,8]],"date-time":"2025-07-08T00:00:00Z","timestamp":1751932800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"name":"Radiation Research Unit at the Cancer Research UK City of London Centre","award":["C7893\/A28990"],"award-info":[{"award-number":["C7893\/A28990"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025,10,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objectives<\/jats:title>\n                  <jats:p>To comprehensively evaluate the ability of ART-Plan (autocontouring software using artificial intelligence [AI]) to contour head and neck cancer (HNC) radiotherapy structures (organs-at-risk and elective nodal volumes) in a real-world setting, as required by NICE guidelines.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Methods<\/jats:title>\n                  <jats:p>Retrospective evaluation (n\u2009=\u200960) compared clinically used contours to AI-contours, using volumetric dice similarity coefficient (VDSC) and blinded radiation oncologist (RO) contour preference assessment. AI-contours were then generated prospectively for HNC radiotherapy patients (n\u2009=\u200961), before review by ROs. ROs recorded qualitative scoring and review\/editing time. Geometric and dosimetric comparison of AI-contours and final contours was undertaken. Correlation coefficients between all metrics were calculated.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>Retrospective median VDSC varied widely for different structures (0.23-0.88). 31.4% blinded contour assessments preferred clinician-generated contours, 32.9% preferred AI-generated contours, 35.7% saw no significant difference. Prospective evaluation showed AI-contour yielded significant time-saving in reviewing\/editing for all structures compared to manual contouring. Qualitative scores demonstrated most structures had median scoring \u22654 (indicating no editing required). Geometric metrics showed high similarity for all structures except larynx. Dosimetric evaluation demonstrated clinically significant dose differences for larynx and elective nodal volumes. Strong correlation was seen between qualitative scoring and all geometric metrics.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusions<\/jats:title>\n                  <jats:p>AI-contours showed excellent qualitative performance and facilitated time-saving. Protocol differences exist between commercial AI-solutions and implementing centres. Ongoing final clinician review of AI-contours remains essential.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Advances in knowledge<\/jats:title>\n                  <jats:p>This is the first study demonstrating ART-Plan\u2019s capability for excellent qualitative ratings and significant time-savings for HNC radiotherapy contouring in a real-world workflow setting. 5-point Likert-scale qualitative scoring correlates strongly with geometric metrics.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/bjr\/tqaf098","type":"journal-article","created":{"date-parts":[[2025,7,8]],"date-time":"2025-07-08T11:34:11Z","timestamp":1751974451000},"page":"1632-1641","source":"Crossref","is-referenced-by-count":5,"title":["Real-world performance evaluation of commercial autocontouring software for head and neck cancer radiotherapy"],"prefix":"10.1093","volume":"98","author":[{"ORCID":"https:\/\/orcid.org\/0009-0009-4085-4362","authenticated-orcid":false,"given":"Tom","family":"Young","sequence":"first","affiliation":[{"name":"Radiotherapy Department, Guy\u2019s and St Thomas\u2019 NHS Foundation Trust (GSTT) , London, SE1 9RT,","place":["United Kingdom"]},{"name":"King\u2019s College London , London, SE1 7EU,","place":["United Kingdom"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Victoria","family":"Butterworth","sequence":"additional","affiliation":[{"name":"Radiotherapy Department, Guy\u2019s and St Thomas\u2019 NHS Foundation Trust (GSTT) , London, SE1 9RT,","place":["United Kingdom"]},{"name":"King\u2019s College London , London, SE1 7EU,","place":["United Kingdom"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sarah","family":"Misson","sequence":"additional","affiliation":[{"name":"Radiotherapy Department, Guy\u2019s and St Thomas\u2019 NHS Foundation Trust (GSTT) , London, SE1 9RT,","place":["United Kingdom"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Delali","family":"Adjogatse","sequence":"additional","affiliation":[{"name":"Radiotherapy Department, Guy\u2019s and St Thomas\u2019 NHS Foundation Trust (GSTT) , London, SE1 9RT,","place":["United Kingdom"]},{"name":"King\u2019s College London , London, SE1 7EU,","place":["United Kingdom"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anthony","family":"Kong","sequence":"additional","affiliation":[{"name":"Radiotherapy Department, Guy\u2019s and St Thomas\u2019 NHS Foundation Trust (GSTT) , London, SE1 9RT,","place":["United Kingdom"]},{"name":"King\u2019s College London , London, SE1 7EU,","place":["United Kingdom"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Imran","family":"Petkar","sequence":"additional","affiliation":[{"name":"Radiotherapy Department, Guy\u2019s and St Thomas\u2019 NHS Foundation Trust (GSTT) , London, SE1 9RT,","place":["United Kingdom"]},{"name":"King\u2019s College London , London, SE1 7EU,","place":["United Kingdom"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Miguel","family":"Reis Ferreira","sequence":"additional","affiliation":[{"name":"Radiotherapy Department, Guy\u2019s and St Thomas\u2019 NHS Foundation Trust (GSTT) , London, SE1 9RT,","place":["United Kingdom"]},{"name":"King\u2019s College London , London, SE1 7EU,","place":["United Kingdom"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mary","family":"Lei","sequence":"additional","affiliation":[{"name":"Radiotherapy Department, Guy\u2019s and St Thomas\u2019 NHS Foundation Trust (GSTT) , London, SE1 9RT,","place":["United Kingdom"]},{"name":"King\u2019s College London , London, SE1 7EU,","place":["United Kingdom"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andrew","family":"King","sequence":"additional","affiliation":[{"name":"King\u2019s College London , London, SE1 7EU,","place":["United Kingdom"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3090-541X","authenticated-orcid":false,"given":"Teresa","family":"Guerrero Urbano","sequence":"additional","affiliation":[{"name":"Radiotherapy Department, Guy\u2019s and St Thomas\u2019 NHS Foundation Trust (GSTT) , London, SE1 9RT,","place":["United Kingdom"]},{"name":"King\u2019s College London , London, SE1 7EU,","place":["United Kingdom"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2025,7,8]]},"reference":[{"key":"2025101113172100600_tqaf098-B1","doi-asserted-by":"publisher","first-page":"335","DOI":"10.1016\/j.clon.2024.03.005","article-title":"How long does contouring really take? Results of the Royal College of Radiologists contouring surveys","volume":"36","author":"Montague","year":"2024","journal-title":"Clin Oncol (R Coll Radiol)"},{"key":"2025101113172100600_tqaf098-B2","author":"National Institute of Clinical Excellence","year":"2023"},{"key":"2025101113172100600_tqaf098-B3","doi-asserted-by":"publisher","first-page":"254","DOI":"10.1515\/raon-2016-0023","article-title":"Uncertainties in target volume delineation in radiotherapy\u2014are they relevant and what can we do about them?","volume":"50","author":"Segedin","year":"2016","journal-title":"Radiol Oncol"},{"key":"2025101113172100600_tqaf098-B4","doi-asserted-by":"publisher","first-page":"130","DOI":"10.1016\/j.radonc.2019.03.004","article-title":"Rapid advances in auto-segmentation of organs at risk and target volumes in head and neck cancer","volume":"135","author":"Kosmin","year":"2019","journal-title":"Radiother Oncol"},{"key":"2025101113172100600_tqaf098-B5","doi-asserted-by":"publisher","first-page":"541","DOI":"10.1007\/s10278-021-00460-3","article-title":"Novel autosegmentation spatial similarity metrics capture the time required to correct segmentations better than traditional metrics in a thoracic cavity segmentation workflow","volume":"34","author":"Kiser","year":"2021","journal-title":"J Digit Imaging"},{"key":"2025101113172100600_tqaf098-B6","author":"Paragios","year":"2024"},{"key":"2025101113172100600_tqaf098-B7","doi-asserted-by":"publisher","first-page":"68","DOI":"10.1016\/j.radonc.2019.05.010","article-title":"Benefits of deep learning for delineation of organs at risk in head and neck cancer","volume":"138","author":"van der Veen","year":"2019","journal-title":"Radiother Oncol"},{"key":"2025101113172100600_tqaf098-B8","doi-asserted-by":"publisher","first-page":"801","DOI":"10.1016\/j.ijrobp.2020.10.005","article-title":"Generating high-quality lymph node clinical target volumes for head and neck cancer radiation therapy using a fully automated deep learning-based approach","volume":"109","author":"Cardenas","year":"2021","journal-title":"Int J Radiat Oncol Biol Phys"},{"key":"2025101113172100600_tqaf098-B9","doi-asserted-by":"publisher","first-page":"1115258","DOI":"10.3389\/fonc.2023.1115258","article-title":"Deep learning for automatic head and neck lymph node level delineation provides expert-level accuracy","volume":"13","author":"Weissmann","year":"2023","journal-title":"Front Oncol"},{"key":"2025101113172100600_tqaf098-B10","doi-asserted-by":"publisher","first-page":"160","DOI":"10.1186\/1748-717X-7-160","article-title":"Systematic evaluation of three different commercial software solutions for automatic segmentation for adaptive therapy in head-and-neck, prostate and pleural cancer","volume":"7","author":"La Macchia","year":"2012","journal-title":"Radiat Oncol"},{"key":"2025101113172100600_tqaf098-B11","doi-asserted-by":"publisher","first-page":"354","DOI":"10.1016\/j.clon.2023.01.016","article-title":"A review of the metrics used to assess auto-contouring systems in radiotherapy","volume":"35","author":"Mackay","year":"2023","journal-title":"Clin Oncol (R Coll Radiol)"},{"key":"2025101113172100600_tqaf098-B12","author":"Banerjee","year":"2024"},{"key":"2025101113172100600_tqaf098-B13","doi-asserted-by":"publisher","first-page":"e150","DOI":"10.1016\/j.prro.2023.10.011","article-title":"Framework for Radiation Oncology Department-wide evaluation and implementation of commercial artificial intelligence autocontouring","volume":"14","author":"Maes","year":"2024","journal-title":"Pract Radiat Oncol"},{"key":"2025101113172100600_tqaf098-B14","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.phro.2019.12.001","article-title":"Evaluation of measures for assessing time-saving of automatic organ-at-risk segmentation in radiotherapy","volume":"13","author":"Vaassen","year":"2020","journal-title":"Phys Imaging Radiat Oncol"},{"key":"2025101113172100600_tqaf098-B15","doi-asserted-by":"publisher","first-page":"185","DOI":"10.1016\/j.radonc.2021.05.003","article-title":"Metrics to evaluate the performance of auto-segmentation for radiation treatment planning: a critical review","volume":"160","author":"Sherer","year":"2021","journal-title":"Radiother Oncol"},{"key":"2025101113172100600_tqaf098-B16","author":"TheraPanacea","year":"2024"},{"key":"2025101113172100600_tqaf098-B17","doi-asserted-by":"publisher","first-page":"1213068","DOI":"10.3389\/fonc.2023.1213068","article-title":"A clinical evaluation of the performance of five commercial artificial intelligence contouring systems for radiotherapy","volume":"13","author":"Doolan","year":"2023","journal-title":"Front Oncol"},{"key":"2025101113172100600_tqaf098-B18","author":"Kiser","year":"2024"},{"key":"2025101113172100600_tqaf098-B19","article-title":"Surface distance","author":"DeepMind","year":"2024"},{"key":"2025101113172100600_tqaf098-B20","doi-asserted-by":"publisher","first-page":"e13647","DOI":"10.1002\/acm2.13647","article-title":"Automatic contouring QA method using a deep learning-based autocontouring system","volume":"23","author":"Rhee","year":"2022","journal-title":"J Appl Clin Med Phys"},{"key":"2025101113172100600_tqaf098-B21","doi-asserted-by":"publisher","first-page":"197","DOI":"10.1016\/j.zemedi.2023.05.001","article-title":"Automatic AI-based contouring of prostate MRI for online adaptive radiotherapy","volume":"34","author":"Nachbar","year":"2024","journal-title":"Z Med Phys"},{"key":"2025101113172100600_tqaf098-B22"},{"key":"2025101113172100600_tqaf098-B23","doi-asserted-by":"publisher","first-page":"741","DOI":"10.1016\/j.ijrobp.2007.01.010","article-title":"Dosimetric predictors of laryngeal edema","volume":"68","author":"Sanguineti","year":"2007","journal-title":"Int J Radiat Oncol Biol Phys"},{"key":"2025101113172100600_tqaf098-B24","volume-title":"Prescribing, Recording, and Reporting Intensity-Modulated Photon-Beam Therapy (IMRT)","author":"International Commission on Radiation Units and Measurements","year":"2010"},{"key":"2025101113172100600_tqaf098-B25","doi-asserted-by":"publisher","first-page":"109574","DOI":"10.1016\/j.radonc.2023.109574","article-title":"Benefits of automated gross tumor volume segmentation in head and neck cancer using multi-modality information","volume":"182","author":"Bollen","year":"2023","journal-title":"Radiother Oncol"},{"key":"2025101113172100600_tqaf098-B26","doi-asserted-by":"publisher","first-page":"e26151","DOI":"10.2196\/26151","article-title":"Clinically applicable segmentation of head and neck anatomy for radiotherapy: deep learning algorithm development and validation study","volume":"23","author":"Nikolov","year":"2021","journal-title":"J Med Internet Res"},{"key":"2025101113172100600_tqaf098-B27","doi-asserted-by":"publisher","first-page":"248","DOI":"10.1016\/j.clon.2020.10.017","article-title":"Structure and processes of existing practice in radiotherapy peer review: a systematic review of the literature","volume":"33","author":"Lewis","year":"2021","journal-title":"Clin Oncol (R Coll Radiol)"},{"key":"2025101113172100600_tqaf098-B28","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.radonc.2021.02.040","article-title":"Clinical implementation of deep learning contour autosegmentation for prostate radiotherapy","volume":"159","author":"Cha","year":"2021","journal-title":"Radiother Oncol"},{"key":"2025101113172100600_tqaf098-B29","doi-asserted-by":"publisher","first-page":"1632","DOI":"10.3390\/diagnostics14151632","article-title":"Analyzing the relationship between dose and geometric agreement metrics for auto-contouring in head and neck normal tissues","volume":"14","author":"Marquez","year":"2024","journal-title":"Diagnostics"},{"key":"2025101113172100600_tqaf098-B30","author":"Royal College of Radiologists"},{"key":"2025101113172100600_tqaf098-B31","doi-asserted-by":"publisher","first-page":"101394","DOI":"10.1016\/j.adro.2023.101394","article-title":"Artificial intelligence-based autosegmentation: advantages in delineation, absorbed Dose-distribution, and logistics","volume":"9","author":"Sarria","year":"2024","journal-title":"Adv Radiat Oncol"},{"key":"2025101113172100600_tqaf098-B32","doi-asserted-by":"publisher","first-page":"69","DOI":"10.1186\/s13014-024-02451-4","article-title":"Evaluation of multiple-vendor AI autocontouring solutions","volume":"19","author":"Goddard","year":"2024","journal-title":"Radiat Oncol"},{"key":"2025101113172100600_tqaf098-B33","author":"Royal College of Radiologists"}],"container-title":["British Journal of Radiology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bjr\/advance-article-pdf\/doi\/10.1093\/bjr\/tqaf098\/63704035\/tqaf098.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bjr\/article-pdf\/98\/1174\/1632\/63704035\/tqaf098.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bjr\/article-pdf\/98\/1174\/1632\/63704035\/tqaf098.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T17:17:29Z","timestamp":1760203049000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bjr\/article\/98\/1174\/1632\/8193704"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,8]]},"references-count":33,"journal-issue":{"issue":"1174","published-online":{"date-parts":[[2025,7,8]]},"published-print":{"date-parts":[[2025,10,1]]}},"URL":"https:\/\/doi.org\/10.1093\/bjr\/tqaf098","relation":{},"ISSN":["0007-1285","1748-880X"],"issn-type":[{"value":"0007-1285","type":"print"},{"value":"1748-880X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2025,10]]},"published":{"date-parts":[[2025,7,8]]}}}