{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,22]],"date-time":"2026-10-22T04:37:08Z","timestamp":1792643828917,"version":"4.5.0"},"reference-count":31,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2024,9,1]],"date-time":"2024-09-01T00:00:00Z","timestamp":1725148800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,9,2]],"date-time":"2024-09-02T00:00:00Z","timestamp":1725235200000},"content-version":"vor","delay-in-days":1,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001774","name":"University of Sydney","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100001774","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Phys Eng Sci Med"],"published-print":{"date-parts":[[2024,9]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Manual contouring of organs at risk (OAR) is time-consuming and subject to inter-observer variability. AI-based auto-contouring is proposed as a solution to these problems if it can produce clinically acceptable results. This study investigated the performance of multiple AI-based auto-contouring systems in different OAR segmentations. The auto-contouring was performed using seven different AI-based segmentation systems (Radiotherapy AI, Limbus AI version 1.5 and 1.6, Therapanacea, MIM, Siemens AI-Rad Companion and RadFormation) on a total of 42 clinical cases with varying anatomical sites. Volumetric and surface dice similarity coefficients and maximum Hausdorff distance (HD) between the expert\u2019s contours and automated contours were calculated to evaluate their performance. Radiotherapy AI has shown better performance than other software in most tested structures considered in the head and neck, and brain cases. No specific software had shown overall superior performance over other software in lung, breast, pelvis and abdomen cases. Each tested AI system was able to produce comparable contours to the experts\u2019 contours of organs at risk which can potentially be used for clinical use. A reduced performance of AI systems in the case of small and complex anatomical structures was found and reported, showing that it is still essential to review each contour produced by AI systems for clinical uses. This study has also demonstrated a method of comparing contouring software options which could be replicated in clinics or used for ongoing quality assurance of purchased systems.<\/jats:p>","DOI":"10.1007\/s13246-024-01434-9","type":"journal-article","created":{"date-parts":[[2024,9,2]],"date-time":"2024-09-02T09:02:49Z","timestamp":1725267769000},"page":"1123-1140","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":24,"title":["Investigation on performance of multiple AI-based auto-contouring systems in organs at risks (OARs) delineation"],"prefix":"10.1007","volume":"47","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5818-950X","authenticated-orcid":false,"given":"Young Woo","family":"Kim","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Simon","family":"Biggs","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Elizabeth","family":"Claridge Mackonis","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,9,2]]},"reference":[{"issue":"1","key":"1434_CR1","doi-asserted-by":"publisher","first-page":"213","DOI":"10.1186\/s13014-019-1392-z","volume":"14","author":"SH Ahn","year":"2019","unstructured":"Ahn SH, Yeo AU, Kim KH et al (2019) Comparative clinical evaluation of atlas and deep-learning-based auto-segmentation of organ structures in liver cancer. Radiat Oncol 14(1):213. https:\/\/doi.org\/10.1186\/s13014-019-1392-z","journal-title":"Radiat Oncol"},{"issue":"3","key":"1434_CR2","doi-asserted-by":"publisher","first-page":"1150","DOI":"10.1002\/mp.12752","volume":"45","author":"EA AlBadawy","year":"2018","unstructured":"AlBadawy EA, Saha A, Mazurowski MA (2018) Deep learning for segmentation of brain tumors: impact of cross-institutional training and testing. Med Phys 45(3):1150\u20131158. https:\/\/doi.org\/10.1002\/mp.12752","journal-title":"Med Phys"},{"key":"1434_CR3","doi-asserted-by":"publisher","DOI":"10.3390\/diagnostics13040667","author":"H Baroudi","year":"2023","unstructured":"Baroudi H, Brock KK, Cao W et al (2023) Automated contouring and planning in radiation therapy: What is \u2018clinically acceptable\u2019? Diagnostics. https:\/\/doi.org\/10.3390\/diagnostics13040667","journal-title":"Diagnostics"},{"issue":"3","key":"1434_CR4","doi-asserted-by":"publisher","first-page":"185","DOI":"10.1016\/j.semradonc.2019.02.001","volume":"29","author":"CE Cardenas","year":"2019","unstructured":"Cardenas CE, Yang J, Anderson BM et al (2019) Advances in auto-segmentation. Semin Radiat Oncol 29(3):185\u2013197. https:\/\/doi.org\/10.1016\/j.semradonc.2019.02.001","journal-title":"Semin Radiat Oncol"},{"issue":"1","key":"1434_CR5","doi-asserted-by":"publisher","first-page":"176","DOI":"10.1186\/s13014-020-01617-0","volume":"15","author":"W Chen","year":"2020","unstructured":"Chen W, Li Y, Dyer BA et al (2020) Deep learning vs. atlas-based models for fast auto-segmentation of the masticatory muscles on head and neck ct images. Radiat Oncol 15(1):176. https:\/\/doi.org\/10.1186\/s13014-020-01617-0","journal-title":"Radiat Oncol"},{"issue":"1","key":"1434_CR6","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1016\/j.radonc.2007.11.016","volume":"87","author":"Z Chen","year":"2008","unstructured":"Chen Z, King W, Pearcey R et al (2008) The relationship between waiting time for radiotherapy and clinical outcomes: a systematic review of the literature. Radiother Oncol 87(1):3\u201316. https:\/\/doi.org\/10.1016\/j.radonc.2007.11.016","journal-title":"Radiother Oncol"},{"key":"1434_CR7","unstructured":"Chlap P, Finnegan R (2021) Platipy (processing library and analysis toolkit for medical imaging in python). http:\/\/https:\/\/pypi.org\/project\/platipy\/"},{"key":"1434_CR8","doi-asserted-by":"crossref","unstructured":"Choi MS, Choi BS, Chung SY, et\u00a0al (2020) Clinical evaluation of atlas- and deep learning-based automatic segmentation of multiple organs and clinical target volumes for breast cancer. Radiotherapy and Oncology 153:139\u2013145. https:\/\/www.sciencedirect.com\/science\/article\/pii\/S0167814020308203","DOI":"10.1016\/j.radonc.2020.09.045"},{"key":"1434_CR9","doi-asserted-by":"publisher","first-page":"154","DOI":"10.1186\/1748-717X-8-154","volume":"8","author":"JF Daisne","year":"2013","unstructured":"Daisne JF, Blumhofer A (2013) Atlas-based automatic segmentation of head and neck organs at risk and nodal target volumes: a clinical validation. Radiat Oncol 8:154. https:\/\/doi.org\/10.1186\/1748-717X-8-154","journal-title":"Radiat Oncol"},{"key":"1434_CR10","doi-asserted-by":"publisher","DOI":"10.3389\/fonc.2023.1213068","author":"PJ Doolan","year":"2023","unstructured":"Doolan PJ, Charalambous S, Roussakis Y et al (2023) A clinical evaluation of the performance of five commercial artificial intelligence contouring systems for radiotherapy. Front Oncol. https:\/\/doi.org\/10.3389\/fonc.2023.1213068","journal-title":"Front Oncol"},{"key":"1434_CR11","unstructured":"Dosovitskiy A, Beyer L, Kolesnikov A, et\u00a0al (2020) An image is worth 16x16 words: transformers for image recognition at scale. arXiv:2010.11929"},{"issue":"3","key":"1434_CR12","doi-asserted-by":"publisher","first-page":"542","DOI":"10.1016\/j.radonc.2015.10.007","volume":"117","author":"G Eminowicz","year":"2015","unstructured":"Eminowicz G, McCormack M (2015) Variability of clinical target volume delineation for definitive radiotherapy in cervix cancer. Radiother Oncol 117(3):542\u2013547. https:\/\/doi.org\/10.1016\/j.radonc.2015.10.007","journal-title":"Radiother Oncol"},{"key":"1434_CR13","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1214\/09-SS051","volume":"4","author":"MP Fay","year":"2010","unstructured":"Fay MP, Proschan MA (2010) Wilcoxon-Mann-Whitney or t-test? on assumptions for hypothesis tests and multiple interpretations of decision rules. Stat Surv 4:1\u201339. https:\/\/doi.org\/10.1214\/09-SS051","journal-title":"Stat Surv"},{"key":"1434_CR14","doi-asserted-by":"publisher","unstructured":"He K, Zhang X, Ren S, et\u00a0al (2015) Deep residual learning for image recognition. https:\/\/doi.org\/10.48550\/arXiv.1512.03385","DOI":"10.48550\/arXiv.1512.03385"},{"key":"1434_CR15","unstructured":"Healthineers S (2021) Customer experience ai-rad companion organs rt the power of automated contouring at ccgm montpellier [white paper]. Siemens Healthcare GmbH, Erlangen https:\/\/pep.siemens-info.com\/en-us\/whitepaper-ai-rad-companion-organs-rt-the-power-of-automated-contouring-at-ccgm-montpellier\/view"},{"issue":"7553","key":"1434_CR16","doi-asserted-by":"publisher","first-page":"436","DOI":"10.1038\/nature14539","volume":"521","author":"Y LeCun","year":"2015","unstructured":"LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436\u2013444. https:\/\/doi.org\/10.1038\/nature14539","journal-title":"Nature"},{"issue":"3","key":"1434_CR17","doi-asserted-by":"publisher","first-page":"944","DOI":"10.1016\/j.ijrobp.2008.10.034","volume":"73","author":"XA Li","year":"2009","unstructured":"Li XA, Tai A, Arthur DW et al (2009) Variability of target and normal structure delineation for breast cancer radiotherapy: an rtog multi-institutional and multiobserver study. Int J Radiat Oncol Biol Phys 73(3):944\u2013951. https:\/\/doi.org\/10.1016\/j.ijrobp.2008.10.034","journal-title":"Int J Radiat Oncol Biol Phys"},{"key":"1434_CR18","unstructured":"Liu C, Tierney K, Blackwell T (2024) Autocontour: State-of-the-art deep-learning auto-segmentation [white paper]. Radformation, New York, NY, https:\/\/21696034.fs1.hubspotusercontent-na1.net\/hubfs\/21696034\/AutoContour%20White%20Paper.pdf?utm_campaign=2023%20Product%3A%20AutoContour &utm_medium=email &_hsmi=246459087 &_hsenc=p2ANqtz-9xITM8hj5VxtNRYKaG7Wkyc2dIo03hDqh73fPipBCz_i_CwUl_kCZQA7aGzEBOje1In7qzOUgRZv_DEdS5_5wFmuRAZg &utm_content=246459087 &utm_source=hs_automation\u201d"},{"issue":"1","key":"1434_CR19","doi-asserted-by":"publisher","first-page":"104","DOI":"10.1186\/s12938-023-01159-y","volume":"22","author":"P Liu","year":"2023","unstructured":"Liu P, Sun Y, Zhao X et al (2023) Deep learning algorithm performance in contouring head and neck organs at risk: a systematic review and single-arm meta-analysis. BioMed Eng OnLine 22(1):104. https:\/\/doi.org\/10.1186\/s12938-023-01159-y","journal-title":"BioMed Eng OnLine"},{"key":"1434_CR20","doi-asserted-by":"publisher","first-page":"40","DOI":"10.1016\/j.ctro.2018.07.006","volume":"12","author":"SP Ng","year":"2018","unstructured":"Ng SP, Dyer BA, Kalpathy-Cramer J et al (2018) A prospective in silico analysis of interdisciplinary and interobserver spatial variability in post-operative target delineation of high-risk oral cavity cancers: Does physician specialty matter? Clin Transl Radiat Oncol 12:40\u201346. https:\/\/doi.org\/10.1016\/j.ctro.2018.07.006","journal-title":"Clin Transl Radiat Oncol"},{"key":"1434_CR21","doi-asserted-by":"publisher","unstructured":"Nikolov S, Blackwell S, Zverovitch A et al (2021) Clinically applicable segmentation of head and neck anatomy for radiotherapy: deep learning algorithm development and validation study. J Med Internet Res 23(7):e26151. https:\/\/doi.org\/10.2196\/26151, https:\/\/www.ncbi.nlm.nih.gov\/pubmed\/34255661","DOI":"10.2196\/26151"},{"key":"1434_CR22","doi-asserted-by":"publisher","first-page":"2088","DOI":"10.3390\/life12122088","volume":"12","author":"L Radici","year":"2022","unstructured":"Radici L, Ferrario S, Borca V et al (2022) Implementation of a commercial deep learning-based auto segmentation software in radiotherapy: evaluation of effectiveness and impact on workflow. Life 12:2088. https:\/\/doi.org\/10.3390\/life12122088","journal-title":"Life"},{"key":"1434_CR23","doi-asserted-by":"publisher","unstructured":"Rhee DJ, Akinfenwa CPA, Rigaud B et al (2022) Automatic contouring qa method using a deep learning-based autocontouring system. J Appl Clin Med Phys 23(8):e13647. https:\/\/doi.org\/10.1002\/acm2.13647, https:\/\/www.ncbi.nlm.nih.gov\/pubmed\/35580067, pMC9359039","DOI":"10.1002\/acm2.13647"},{"key":"1434_CR24","doi-asserted-by":"publisher","unstructured":"Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. https:\/\/doi.org\/10.48550\/arXiv.1505.04597, arXiv:1505.04597","DOI":"10.48550\/arXiv.1505.04597"},{"issue":"1","key":"1434_CR25","doi-asserted-by":"publisher","first-page":"29","DOI":"10.1186\/s12880-015-0068-x","volume":"15","author":"AA Taha","year":"2015","unstructured":"Taha AA, Hanbury A (2015) Metrics for evaluating 3d medical image segmentation: analysis, selection, and tool. BMC Med Imaging 15(1):29. https:\/\/doi.org\/10.1186\/s12880-015-0068-x","journal-title":"BMC Med Imaging"},{"issue":"1","key":"1434_CR26","doi-asserted-by":"publisher","first-page":"175","DOI":"10.1186\/s13014-021-01896-1","volume":"16","author":"Y Urago","year":"2021","unstructured":"Urago Y, Okamoto H, Kaneda T et al (2021) Evaluation of auto-segmentation accuracy of cloud-based artificial intelligence and atlas-based models. Radiat Oncol 16(1):175. https:\/\/doi.org\/10.1186\/s13014-021-01896-1","journal-title":"Radiat Oncol"},{"key":"1434_CR27","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.phro.2019.12.001","volume":"13","author":"F Vaassen","year":"2020","unstructured":"Vaassen F, Hazelaar C, Vaniqui A et al (2020) Evaluation of measures for assessing time-saving of automatic organ-at-risk segmentation in radiotherapy. Phys Imaging Radiat Oncol 13:1\u20136. https:\/\/doi.org\/10.1016\/j.phro.2019.12.001","journal-title":"Phys Imaging Radiat Oncol"},{"key":"1434_CR28","unstructured":"Vaswani A, Shazeer N, Parmar N, et\u00a0al (2023) Attention is all you need. arXiv:1706.03762"},{"key":"1434_CR29","unstructured":"Wan H (2024) Automated contouring using neural networks [white paper]. MIM software inc, Cleveland https:\/\/5300642.fs1.hubspotusercontent-na1.net\/hubfs\/5300642\/TD650-Contour_ProtegeAI_White_Paper-20211208.pdf"},{"issue":"6","key":"1434_CR30","doi-asserted-by":"publisher","first-page":"509","DOI":"10.1186\/cc1820","volume":"6","author":"E Whitley","year":"2002","unstructured":"Whitley E, Ball J (2002) Statistics review 6: nonparametric methods. Crit Care 6(6):509. https:\/\/doi.org\/10.1186\/cc1820","journal-title":"Crit Care"},{"issue":"1","key":"1434_CR31","doi-asserted-by":"publisher","first-page":"e80","DOI":"10.1016\/j.prro.2020.05.013","volume":"11","author":"WJ Zabel","year":"2021","unstructured":"Zabel WJ, Conway JL, Gladwish A et al (2021) Clinical evaluation of deep learning and atlas-based auto-contouring of bladder and rectum for prostate radiation therapy. Pract Radiat Oncol 11(1):e80\u2013e89. https:\/\/doi.org\/10.1016\/j.prro.2020.05.013","journal-title":"Pract Radiat Oncol"}],"container-title":["Physical and Engineering Sciences in Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s13246-024-01434-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s13246-024-01434-9\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s13246-024-01434-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,18]],"date-time":"2024-09-18T13:03:20Z","timestamp":1726664600000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s13246-024-01434-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,9]]},"references-count":31,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2024,9]]}},"alternative-id":["1434"],"URL":"https:\/\/doi.org\/10.1007\/s13246-024-01434-9","relation":{},"ISSN":["2662-4729","2662-4737"],"issn-type":[{"value":"2662-4729","type":"print"},{"value":"2662-4737","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,9]]},"assertion":[{"value":"29 August 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"30 April 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 September 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Simon Biggs is the founder of Radiotherapy AI. The authors have no other Conflict of interest\/Conflict of interest to disclose.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"This study is a retrospective study. The study was approved by the Radiation Oncology Research Committee at Chris O\u2019Brien Lifehouse to run under an existing ethics committee approval.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics Approval"}}]}}