{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,25]],"date-time":"2026-08-25T18:29:07Z","timestamp":1787682547185,"version":"build-2784847793"},"reference-count":22,"publisher":"Wiley","issue":"4","license":[{"start":{"date-parts":[[2025,2,13]],"date-time":"2025-02-13T00:00:00Z","timestamp":1739404800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R01CA272991"],"award-info":[{"award-number":["R01CA272991"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R01DE033512"],"award-info":[{"award-number":["R01DE033512"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R56EB033332"],"award-info":[{"award-number":["R56EB033332"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["P30CA008748"],"award-info":[{"award-number":["P30CA008748"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["aapm.onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["J Applied Clin Med Phys"],"published-print":{"date-parts":[[2025,4]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec><jats:title>Purpose<\/jats:title><jats:p>Deep learning\u2010based segmentation of organs\u2010at\u2010risk (OAR) is emerging to become mainstream in clinical practice because of the superior performance over atlas and model\u2010based autocontouring methods. While several commercial deep learning\u2010based autosegmentation solutions are now available, the implementation of these tools is still at such a primitive stage that acceptance criteria are underdeveloped due to a lack of knowledge about the systems\u2019 segmentation tendencies and failure modes. As the starting point of the iterative process of clinical implementation, this study focuses on the outlier analysis of four commercial autocontouring tools for the abdominal OARs.<\/jats:p><\/jats:sec><jats:sec><jats:title>Materials and methods<\/jats:title><jats:p>The autosegmentation software, developed by Limbus AI, MIM Contour Prot\u00e9g\u00e9AI, Radformation AutoContour, and Siemens syngo.via, were used to segment 111 patient cases. Geometric segmentation accuracy was quantitatively compared with clinical contours using the dice similarity coefficient (DSC) and 95% Hausdorff distance (HD95). The outliers from quantitative evaluations of each software were analyzed for the liver, stomach, and kidneys with the possible causes of outliers summarized into six categories: (1) difference in contouring style or guideline, (2) image acquisition and quality, (3) abnormal anatomy of the OAR, (4) abnormal anatomy of abutting organs\/tissues, (5) external\/internal devices, and (6) other causes.<\/jats:p><\/jats:sec><jats:sec><jats:title>Results<\/jats:title><jats:p>For the liver segmentation, the most prominent cause of discrepancies for Limbus, which occurred in four of its six outliers, was the existence of biliary stent or internal\/external biliary drain as well as the resulting pneumobilia. Siemens included the abutting organs that shared CT numbers similar to those of the liver in 5\/8 outliers. 12 of 13 Radformation's liver segmentation outliers included the heart and\/or stomach while MIM not only included the stomach in the presence of barium in 5\/11 outliers, but also produced fragmented contours in 5\/11 other cases. Only Limbus and Radformation provided stomach segmentation, and imaging with barium contrast directly caused incomplete stomach delineation in 10\/12 Limbus outliers and 21\/25 Radformation outliers. As for the kidneys, Radformation and Siemens consistently followed the RTOG contouring guidelines, whereas the institutional contours excluded the renal pelvis in some cases, resulting in 19\/25 Radformation outliers and 18\/23 Siemens outliers. By contrast, Limbus contours appeared to follow different contouring guidelines that exclude the renal pelvis. Fragmented kidney contours were found in 10\/15 Limbus outliers and 25\/26 MIM outliers. The ones in MIM were directly linked to the use of IV contrast in imaging, but there was not enough evidence to identify the origin of Limbus's fragmented contours.<\/jats:p><\/jats:sec><jats:sec><jats:title>Conclusion<\/jats:title><jats:p>The causes of the segmentation outliers of the four commercial deep learning\u2010based autocontouring solutions were summarized for each OAR. This work can help the vendors improve their autosegmentation software and also inform the users of potential modes of failure when using the tools.<\/jats:p><\/jats:sec>","DOI":"10.1002\/acm2.70010","type":"journal-article","created":{"date-parts":[[2025,2,13]],"date-time":"2025-02-13T17:38:56Z","timestamp":1739468336000},"update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":11,"title":["Evaluation and failure analysis of four commercial deep learning\u2010based autosegmentation software for abdominal organs at risk"],"prefix":"10.1002","volume":"26","author":[{"given":"Mingdong","family":"Fan","sequence":"first","affiliation":[{"name":"Department of Radiation Oncology and Winship Cancer Institute Emory University  Atlanta Georgia USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tonghe","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Medical Physics Memorial Sloan Kettering Cancer Center  New York New York USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3572-0345","authenticated-orcid":false,"given":"Yang","family":"Lei","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology Icahn School of Medicine at Mount Sinai  New York New York USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pretesh R.","family":"Patel","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology and Winship Cancer Institute Emory University  Atlanta Georgia USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8191-0852","authenticated-orcid":false,"given":"Sean","family":"Dresser","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology and Winship Cancer Institute Emory University  Atlanta Georgia USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Beth Bradshaw","family":"Ghavidel","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology and Winship Cancer Institute Emory University  Atlanta Georgia USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7877-1900","authenticated-orcid":false,"given":"Richard L. J.","family":"Qiu","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology and Winship Cancer Institute Emory University  Atlanta Georgia USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6078-9424","authenticated-orcid":false,"given":"Jun","family":"Zhou","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology and Winship Cancer Institute Emory University  Atlanta Georgia USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3781-6666","authenticated-orcid":false,"given":"Kirk","family":"Luca","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology and Winship Cancer Institute Emory University  Atlanta Georgia USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Oluwatosin","family":"Kayode","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology and Winship Cancer Institute Emory University  Atlanta Georgia USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jeffrey D.","family":"Bradley","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology University of Pennsylvania  Philadelphia Pennsylvania USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9023-5855","authenticated-orcid":false,"given":"Xiaofeng","family":"Yang","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology and Winship Cancer Institute Emory University  Atlanta Georgia USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Justin","family":"Roper","sequence":"additional","affiliation":[{"name":"Department of Radiation Oncology and Winship Cancer Institute Emory University  Atlanta Georgia USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"311","published-online":{"date-parts":[[2025,2,13]]},"reference":[{"key":"e_1_2_10_2_1","first-page":"185","volume-title":"Seminars in Radiation Oncology","author":"Cardenas CE","year":"2019"},{"key":"e_1_2_10_3_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ctro.2020.09.004"},{"key":"e_1_2_10_4_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijrobp.2020.04.011"},{"key":"e_1_2_10_5_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.radonc.2020.05.038"},{"key":"e_1_2_10_6_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.prro.2018.12.003"},{"key":"e_1_2_10_7_1","doi-asserted-by":"publisher","DOI":"10.1186\/1748-717X-7-32"},{"key":"e_1_2_10_8_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.radonc.2019.04.006"},{"key":"e_1_2_10_9_1","first-page":"1","article-title":"Interobserver variability in organ at risk delineation in head and neck cancer","volume":"16","author":"Veen J","year":"2021","journal-title":"Radiat Oncol"},{"key":"e_1_2_10_10_1","first-page":"421","volume-title":"Seminars in Radiation Oncology","author":"Claessens M","year":"2022"},{"key":"e_1_2_10_11_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.radonc.2020.09.008"},{"key":"e_1_2_10_12_1","first-page":"S0360","article-title":"NRG Oncology assessment on AI deep\u2010learning based auto\u2010segmentation for radiotherapy: current development, clinical consideration, and future direction","author":"Rong Y","year":"2023","journal-title":"Int J Radiat Oncol Biol Phys"},{"key":"e_1_2_10_13_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.meddos.2022.11.001"},{"key":"e_1_2_10_14_1","article-title":"Multicenter study of pelvic nodal autosegmentation algorithm of siemens healthineers: comparison of male versus female pelvis","volume":"28","author":"Rayn K","year":"2023","journal-title":"Adv Radiat Oncol"},{"key":"e_1_2_10_15_1","doi-asserted-by":"publisher","DOI":"10.3390\/life12122088"},{"key":"e_1_2_10_16_1","doi-asserted-by":"publisher","DOI":"10.1186\/s13014-021-01831-4"},{"key":"e_1_2_10_17_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.phro.2023.100515"},{"key":"e_1_2_10_18_1","doi-asserted-by":"publisher","DOI":"10.1002\/mp.16537"},{"key":"e_1_2_10_19_1","doi-asserted-by":"publisher","DOI":"10.3389\/fonc.2023.1213068"},{"key":"e_1_2_10_20_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.prro.2023.10.011"},{"key":"e_1_2_10_21_1","doi-asserted-by":"publisher","DOI":"10.3389\/fonc.2023.1089807"},{"key":"e_1_2_10_22_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.prro.2013.06.004"},{"key":"e_1_2_10_23_1","doi-asserted-by":"publisher","DOI":"10.1259\/bjr.20180131"}],"container-title":["Journal of Applied Clinical Medical Physics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/aapm.onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/acm2.70010","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,4,5]],"date-time":"2025-04-05T12:34:04Z","timestamp":1743856444000},"score":1,"resource":{"primary":{"URL":"https:\/\/aapm.onlinelibrary.wiley.com\/doi\/10.1002\/acm2.70010"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,2,13]]},"references-count":22,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2025,4]]}},"alternative-id":["10.1002\/acm2.70010"],"URL":"https:\/\/doi.org\/10.1002\/acm2.70010","archive":["Portico"],"relation":{},"ISSN":["1526-9914","1526-9914"],"issn-type":[{"value":"1526-9914","type":"print"},{"value":"1526-9914","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,2,13]]},"assertion":[{"value":"2024-08-30","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-12-06","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-02-13","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"e70010"}}