{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,29]],"date-time":"2026-08-29T15:15:37Z","timestamp":1788016537693,"version":"build-2784847793"},"reference-count":43,"publisher":"Wiley","issue":"6","license":[{"start":{"date-parts":[[2024,1,23]],"date-time":"2024-01-23T00:00:00Z","timestamp":1705968000000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["aapm.onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["J Applied Clin Med Phys"],"published-print":{"date-parts":[[2024,6]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Purpose<\/jats:title><jats:p>Artificial intelligence (AI) based commercial software can be used to automatically delineate organs at risk (OAR), with potential for efficiency savings in the radiotherapy treatment planning pathway, and reduction of inter\u2010 and intra\u2010observer variability. There has been little research investigating gross failure rates and failure modes of such systems.<\/jats:p><\/jats:sec><jats:sec><jats:title>Method<\/jats:title><jats:p>50 head and neck (H&amp;N) patient data sets with \u201cgold standard\u201d contours were compared to AI\u2010generated contours to produce expected mean and standard deviation values for the Dice Similarity Coefficient (DSC), for four common H&amp;N OARs (brainstem, mandible, left and right parotid). An AI\u2010based commercial system was applied to 500 H&amp;N patients. AI\u2010generated contours were compared to manual contours, outlined by an expert human, and a gross failure was set at three standard deviations below the expected mean DSC. Failures were inspected to assess reason for failure of the AI\u2010based system with failures relating to suboptimal manual contouring censored. True failures were classified into 4 sub\u2010types (setup position, anatomy, image artefacts and unknown).<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>There were 24 true failures of the AI\u2010based commercial software, a gross failure rate of 1.2%. Fifteen failures were due to patient anatomy, four were due to dental image artefacts, three were due to patient position and two were unknown. True failure rates by OAR were 0.4% (brainstem), 2.2% (mandible), 1.4% (left parotid) and 0.8% (right parotid).<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusion<\/jats:title><jats:p>True failures of the AI\u2010based system were predominantly associated with a non\u2010standard element within the CT scan. It is likely that these non\u2010standard elements were the reason for the gross failure, and suggests that patient datasets used to train the AI model did not contain sufficient heterogeneity of data. Regardless of the reasons for failure, the true failure rate for the AI\u2010based system in the H&amp;N region for the OARs investigated was low (\u223c1%).<\/jats:p><\/jats:sec>","DOI":"10.1002\/acm2.14273","type":"journal-article","created":{"date-parts":[[2024,1,24]],"date-time":"2024-01-24T07:49:31Z","timestamp":1706082571000},"update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":17,"title":["Gross failure rates and failure modes for a commercial AI\u2010based auto\u2010segmentation algorithm in head and neck cancer patients"],"prefix":"10.1002","volume":"25","author":[{"given":"Simon W. P.","family":"Temple","sequence":"first","affiliation":[{"name":"Medical Physics Department The Clatterbridge Cancer Centre NHS Foundation Trust  Liverpool UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Carl G.","family":"Rowbottom","sequence":"additional","affiliation":[{"name":"Medical Physics Department The Clatterbridge Cancer Centre NHS Foundation Trust  Liverpool UK"},{"name":"Department of Physics University of Liverpool  Liverpool UK"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"311","published-online":{"date-parts":[[2024,1,23]]},"reference":[{"key":"e_1_2_9_2_1","doi-asserted-by":"publisher","DOI":"10.1002\/mp.13147"},{"key":"e_1_2_9_3_1","doi-asserted-by":"publisher","DOI":"10.1016\/J.SEMRADONC.2019.02.001"},{"key":"e_1_2_9_4_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.prro.2011.11.012"},{"key":"e_1_2_9_5_1","doi-asserted-by":"publisher","DOI":"10.1186\/s13014\u2010021\u201001809\u20102"},{"key":"e_1_2_9_6_1","doi-asserted-by":"publisher","DOI":"10.1016\/J.RADONC.2019.05.010"},{"key":"e_1_2_9_7_1","doi-asserted-by":"publisher","DOI":"10.1186\/s13014\u2010021\u201001896\u20101"},{"key":"e_1_2_9_8_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.phro.2022.11.003"},{"key":"e_1_2_9_9_1","doi-asserted-by":"publisher","DOI":"10.1002\/acm2.13949"},{"key":"e_1_2_9_10_1","doi-asserted-by":"publisher","DOI":"10.3389\/fonc.2023.1137803"},{"key":"e_1_2_9_11_1","doi-asserted-by":"publisher","DOI":"10.1186\/1748\u2010717X\u20108\u2010154"},{"key":"e_1_2_9_12_1","doi-asserted-by":"publisher","DOI":"10.1002\/mp.14320"},{"key":"e_1_2_9_13_1","doi-asserted-by":"publisher","DOI":"10.1111\/1754\u20109485.13286"},{"key":"e_1_2_9_14_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.clon.2021.12.003"},{"key":"e_1_2_9_15_1","doi-asserted-by":"publisher","DOI":"10.1016\/J.RADONC.2019.10.019"},{"key":"e_1_2_9_16_1","doi-asserted-by":"publisher","DOI":"10.1162\/99608f92.5a8a3a3d"},{"key":"e_1_2_9_17_1","doi-asserted-by":"publisher","DOI":"10.1111\/jgh.15384"},{"key":"e_1_2_9_18_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.radonc.2020.09.008"},{"key":"e_1_2_9_19_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.phro.2020.11.002"},{"key":"e_1_2_9_20_1","doi-asserted-by":"publisher","DOI":"10.2307\/1932409"},{"key":"e_1_2_9_21_1","doi-asserted-by":"publisher","DOI":"10.1002\/mp.13141"},{"key":"e_1_2_9_22_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.radonc.2019.09.022"},{"key":"e_1_2_9_23_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.radonc.2015.07.041"},{"key":"e_1_2_9_24_1","doi-asserted-by":"publisher","DOI":"10.1186\/s12885\u2010015\u20101598\u2010x"},{"key":"e_1_2_9_25_1","doi-asserted-by":"publisher","DOI":"10.1111\/1754\u20109485.12462"},{"key":"e_1_2_9_26_1","doi-asserted-by":"publisher","DOI":"10.1080\/00031305.1994.10476030"},{"key":"e_1_2_9_27_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.radonc.2021.05.003"},{"key":"e_1_2_9_28_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejmp.2013.09.001"},{"key":"e_1_2_9_29_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.radonc.2017.09.029"},{"key":"e_1_2_9_30_1","doi-asserted-by":"publisher","DOI":"10.3389\/fdata.2020.577974"},{"key":"e_1_2_9_31_1","doi-asserted-by":"publisher","DOI":"10.1186\/s12909\u2010023\u201004698\u2010z"},{"key":"e_1_2_9_32_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41388\u2010023\u201002826\u2010z"},{"key":"e_1_2_9_33_1","doi-asserted-by":"publisher","DOI":"10.3174\/ajnr.a7963"},{"key":"e_1_2_9_34_1","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.43262"},{"key":"e_1_2_9_35_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cca.2023.117519"},{"key":"e_1_2_9_36_1","doi-asserted-by":"publisher","DOI":"10.1002\/mp.13271"},{"key":"e_1_2_9_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3458723"},{"key":"e_1_2_9_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/iemc.2001.960553"},{"key":"e_1_2_9_39_1","doi-asserted-by":"publisher","DOI":"10.1016\/0167\u2010188X(88)90025\u20100"},{"key":"e_1_2_9_40_1","doi-asserted-by":"publisher","DOI":"10.1002\/MP.13814"},{"key":"e_1_2_9_41_1","doi-asserted-by":"publisher","DOI":"10.1016\/J.CANRAD.2021.06.023"},{"key":"e_1_2_9_42_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.phro.2020.10.001"},{"key":"e_1_2_9_43_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.oraloncology.2018.04.025"},{"key":"e_1_2_9_44_1","doi-asserted-by":"publisher","DOI":"10.1186\/s13014\u2010020\u201001677\u20102"}],"container-title":["Journal of Applied Clinical Medical Physics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/aapm.onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/acm2.14273","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,6,12]],"date-time":"2024-06-12T18:13:37Z","timestamp":1718216017000},"score":1,"resource":{"primary":{"URL":"https:\/\/aapm.onlinelibrary.wiley.com\/doi\/10.1002\/acm2.14273"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,1,23]]},"references-count":43,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2024,6]]}},"alternative-id":["10.1002\/acm2.14273"],"URL":"https:\/\/doi.org\/10.1002\/acm2.14273","archive":["Portico"],"relation":{},"ISSN":["1526-9914","1526-9914"],"issn-type":[{"value":"1526-9914","type":"print"},{"value":"1526-9914","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,1,23]]},"assertion":[{"value":"2023-10-04","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-12-20","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-01-23","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"e14273"}}