{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,6]],"date-time":"2026-11-06T06:49:16Z","timestamp":1793947756510,"version":"4.8.0"},"reference-count":32,"publisher":"Springer Science and Business Media LLC","issue":"4","license":[{"start":{"date-parts":[[2025,4,1]],"date-time":"2025-04-01T00:00:00Z","timestamp":1743465600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,4,4]],"date-time":"2025-04-04T00:00:00Z","timestamp":1743724800000},"content-version":"vor","delay-in-days":3,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001826","name":"ZonMw","doi-asserted-by":"publisher","award":["10070012010006"],"award-info":[{"award-number":["10070012010006"]}],"id":[{"id":"10.13039\/501100001826","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Neuroradiology"],"published-print":{"date-parts":[[2025,4]]},"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:sec>\n            <jats:title>Purpose<\/jats:title>\n            <jats:p>To evaluate the clinimetric reliability of automated vestibular schwannoma (VS) segmentations by a comparison with human inter-observer variability on T1-weighted contrast-enhanced MRI scans.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>Methods<\/jats:title>\n            <jats:p>This retrospective study employed MR images, including follow-up, from 1,015 patients (median age: 59, 511 men), resulting in 1,856 unique scans. Two nnU-Net models were trained using fivefold cross-validation to create a single-center segmentation model, along with a multi-center model using additional publicly available data. Geometric-based segmentation metrics (e.g. the Dice score) were used to evaluate model performance. To quantitatively assess the clinimetric reliability of the models, automated tumor volumes from a separate test set were compared to human inter-observer variability using the limits of agreement with the mean (LOAM) procedure. Additionally, new agreement limits that include automated annotations are calculated.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>Results<\/jats:title>\n            <jats:p>Both models performed comparable to current state-of-the-art VS segmentation models, with median Dice scores of 91.6% and 91.9% for the single and multi-center models, respectively. There is a stark difference in clinimetric performance between both models: automated tumor volumes of the multi-center model fell within human agreement limits in 73% of the cases, compared to 44% for the single-center model. Newly calculated agreement limits including the single-center model, resulted in very high and wide limits. For the multi-center model, the new agreement limits were comparable to human inter-observer variability.<\/jats:p>\n          <\/jats:sec>\n          <jats:sec>\n            <jats:title>Conclusion<\/jats:title>\n            <jats:p>Models with excellent geometric-based metrics do not necessarily imply high clinimetric reliability, demonstrating the need to clinimetrically evaluate models as part of the clinical implementation process. The multi-center model displayed high reliability, warranting its possible future use in clinical practice. However, caution should be exercised when employing the model for small tumors, as the reliability was found to be volume-dependent.<\/jats:p>\n          <\/jats:sec>","DOI":"10.1007\/s00234-025-03611-3","type":"journal-article","created":{"date-parts":[[2025,4,5]],"date-time":"2025-04-05T15:57:39Z","timestamp":1743868659000},"page":"1049-1059","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Towards clinical implementation of automated segmentation of vestibular schwannomas: a reliability study comparing AI and human performance"],"prefix":"10.1007","volume":"67","author":[{"given":"Stefan","family":"Cornelissen","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sammy M.","family":"Schouten","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Patrick P. J. H.","family":"Langenhuizen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Henricus P. M.","family":"Kunst","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jeroen B.","family":"Verheul","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Peter H. N.","family":"De With","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2025,4,4]]},"reference":[{"key":"3611_CR1","doi-asserted-by":"publisher","first-page":"1335","DOI":"10.1056\/NEJMra2020394","volume":"384","author":"ML Carlson","year":"2021","unstructured":"Carlson ML, Link MJ (2021) Vestibular schwannomas. N Engl J Med 384:1335\u20131348. https:\/\/doi.org\/10.1056\/NEJMra2020394","journal-title":"N Engl J Med"},{"key":"3611_CR2","doi-asserted-by":"publisher","first-page":"440","DOI":"10.1177\/0194599815590105","volume":"153","author":"ML Carlson","year":"2015","unstructured":"Carlson ML, Habermann EB, Wagie AE et al (2015) The changing landscape of vestibular Schwannoma management in the united States - A shift toward conservatism. Otolaryngol - Head Neck Surg (United States) 153:440\u2013446. https:\/\/doi.org\/10.1177\/0194599815590105","journal-title":"Otolaryngol - Head Neck Surg (United States)"},{"key":"3611_CR3","doi-asserted-by":"publisher","first-page":"271","DOI":"10.1097\/MAO.0b013e3181c34bda","volume":"31","author":"SE Stangerup","year":"2010","unstructured":"Stangerup SE, Thomsen J, Tos M, Cay\u00e9-Thomasen P (2010) Long-term hearing preservation in vestibular Schwannoma. Otol Neurotol 31:271\u2013275. https:\/\/doi.org\/10.1097\/MAO.0b013e3181c34bda","journal-title":"Otol Neurotol"},{"key":"3611_CR4","doi-asserted-by":"publisher","first-page":"2034","DOI":"10.3174\/ajnr.A5213","volume":"38","author":"EP Lin","year":"2017","unstructured":"Lin EP, Crane BT (2017) The management and imaging of vestibular schwannomas. Am J Neuroradiol 38:2034\u20132043. https:\/\/doi.org\/10.3174\/ajnr.A5213","journal-title":"Am J Neuroradiol"},{"key":"3611_CR5","doi-asserted-by":"publisher","first-page":"97","DOI":"10.1111\/j.1749-4486.2010.02099.x","volume":"35","author":"JK Varughese","year":"2010","unstructured":"Varughese JK, Wentzel-Larsen T, Vassbotn F et al (2010) Analysis of vestibular Schwannoma size in multiple dimensions: A comparative cohort study of different measurement techniques. Clin Otolaryngol 35:97\u2013103. https:\/\/doi.org\/10.1111\/j.1749-4486.2010.02099.x","journal-title":"Clin Otolaryngol"},{"key":"3611_CR6","doi-asserted-by":"publisher","first-page":"867","DOI":"10.1007\/s00405-018-4865-z","volume":"275","author":"S MacKeith","year":"2018","unstructured":"MacKeith S, Das T, Graves M et al (2018) A comparison of semi-automated volumetric vs linear measurement of small vestibular schwannomas. Eur Arch Otorhinolaryngol 275:867\u2013874. https:\/\/doi.org\/10.1007\/s00405-018-4865-z","journal-title":"Eur Arch Otorhinolaryngol"},{"key":"3611_CR7","doi-asserted-by":"publisher","first-page":"535","DOI":"10.1177\/0194599818770413","volume":"159","author":"KA Lees","year":"2018","unstructured":"Lees KA, Tombers NM, Link MJ et al (2018) Natural history of sporadic vestibular Schwannoma: A volumetric study of tumor growth. Otolaryngol - Head Neck Surg (United States) 159:535\u2013542. https:\/\/doi.org\/10.1177\/0194599818770413","journal-title":"Otolaryngol - Head Neck Surg (United States)"},{"key":"3611_CR8","doi-asserted-by":"publisher","first-page":"123","DOI":"10.1111\/j.1749-4486.2006.01161.x","volume":"31","author":"JJ Cross","year":"2006","unstructured":"Cross JJ, Baguley DM, Antoun NM et al (2006) Reproducibility of volume measurements of vestibular schwannomas - A preliminary study. Clin Otolaryngol 31:123\u2013129. https:\/\/doi.org\/10.1111\/j.1749-4486.2006.01161.x","journal-title":"Clin Otolaryngol"},{"key":"3611_CR9","doi-asserted-by":"publisher","first-page":"517","DOI":"10.1007\/s00234-009-0529-4","volume":"51","author":"R Van De Langenberg","year":"2009","unstructured":"Van De Langenberg R, De Bondt BJ, Nelemans PJ et al (2009) Follow-up assessment of vestibular schwannomas: volume quantification versus two-dimensional measurements. Neuroradiology 51:517\u2013524. https:\/\/doi.org\/10.1007\/s00234-009-0529-4","journal-title":"Neuroradiology"},{"key":"3611_CR10","doi-asserted-by":"publisher","first-page":"171","DOI":"10.3171\/2019.9.JNS191949","volume":"134","author":"J Shapey","year":"2021","unstructured":"Shapey J, Wang G, Dorent R et al (2021) An artificial intelligence framework for automatic segmentation and volumetry of vestibular schwannomas from contrast-enhanced T1-weighted and high-resolution T2-weighted MRI. J Neurosurg 134:171\u2013179. https:\/\/doi.org\/10.3171\/2019.9.JNS191949","journal-title":"J Neurosurg"},{"key":"3611_CR11","doi-asserted-by":"crossref","unstructured":"Wang G, Shapey J, Li W et al (2019) Automatic segmentation of vestibular Schwannoma from T2-Weighted MRI by deep Spatial attention with Hardness-Weighted loss. Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in Bioinformatics). Springer Science and Business Media Deutschland GmbH, pp 264\u2013272","DOI":"10.1007\/978-3-030-32245-8_30"},{"key":"3611_CR12","doi-asserted-by":"publisher","first-page":"3106","DOI":"10.1038\/s41598-021-82665-8","volume":"11","author":"C Lee","year":"2021","unstructured":"Lee C, Chia, Lee WK, Wu CC et al (2021) Applying artificial intelligence to longitudinal imaging analysis of vestibular Schwannoma following radiosurgery. Sci Rep 11:3106. https:\/\/doi.org\/10.1038\/s41598-021-82665-8","journal-title":"Sci Rep"},{"key":"3611_CR13","doi-asserted-by":"publisher","unstructured":"Neve OM, Chen Y, Tao Q et al (2022) Fully automated 3D vestibular Schwannoma segmentation with and without gadolinium contrast: A multicenter, multivendor study. https:\/\/doi.org\/10.1148\/ryai.210300. Radiol Artif Intell","DOI":"10.1148\/ryai.210300"},{"key":"3611_CR14","doi-asserted-by":"publisher","unstructured":"Kujawa A, Dorent R, Connor S et al (2024) Deep learning for automatic segmentation of vestibular Schwannoma: a retrospective study from multi-center routine MRI. Front Comput Neurosci 18. https:\/\/doi.org\/10.3389\/fncom.2024.1365727","DOI":"10.3389\/fncom.2024.1365727"},{"key":"3611_CR15","doi-asserted-by":"publisher","first-page":"35","DOI":"10.1016\/j.wneu.2024.04.145","volume":"188","author":"K Nernekli","year":"2024","unstructured":"Nernekli K, Persad AR, Hori YS et al (2024) Automatic segmentation of vestibular schwannomas: A systematic review. World Neurosurg 188:35\u201344. https:\/\/doi.org\/10.1016\/j.wneu.2024.04.145","journal-title":"World Neurosurg"},{"key":"3611_CR16","doi-asserted-by":"publisher","DOI":"10.1007\/s00234-024-03449-1","author":"P \u0141ajczak","year":"2024","unstructured":"\u0141ajczak P, Matyja J, J\u00f3\u017awik K, Nawrat Z (2024) Accuracy of vestibular Schwannoma segmentation using deep learning models - a systematic review & meta-analysis. Neuroradiology. https:\/\/doi.org\/10.1007\/s00234-024-03449-1","journal-title":"Neuroradiology"},{"key":"3611_CR17","doi-asserted-by":"publisher","first-page":"5105","DOI":"10.1002\/mp.13200","volume":"45","author":"MJ Gooding","year":"2018","unstructured":"Gooding MJ, Smith AJ, Tariq M et al (2018) Comparative evaluation of autocontouring in clinical practice: A practical method using the turing test. Med Phys 45:5105\u20135115. https:\/\/doi.org\/10.1002\/mp.13200","journal-title":"Med Phys"},{"key":"3611_CR18","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41467-018-07619-7","volume":"9","author":"L Maier-Hein","year":"2018","unstructured":"Maier-Hein L, Eisenmann M, Reinke A et al (2018) Why rankings of biomedical image analysis competitions should be interpreted with care. Nat Commun 9:1\u201313. https:\/\/doi.org\/10.1038\/s41467-018-07619-7","journal-title":"Nat Commun"},{"key":"3611_CR19","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.radonc.2021.02.040","volume":"159","author":"E Cha","year":"2021","unstructured":"Cha E, Elguindi S, Onochie I et al (2021) Clinical implementation of deep learning contour autosegmentation for prostate radiotherapy. Radiother Oncol 159:1\u20137. https:\/\/doi.org\/10.1016\/j.radonc.2021.02.040","journal-title":"Radiother Oncol"},{"key":"3611_CR20","doi-asserted-by":"publisher","first-page":"1311","DOI":"10.1007\/s11548-023-02936-9","volume":"18","author":"TN Tran","year":"2023","unstructured":"Tran TN, Adler TJ, Yamlahi A et al (2023) Sources of performance variability in deep learning-based polyp detection. Int J Comput Assist Radiol Surg 18:1311\u20131322. https:\/\/doi.org\/10.1007\/s11548-023-02936-9","journal-title":"Int J Comput Assist Radiol Surg"},{"key":"3611_CR21","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.phro.2019.12.001","volume":"13","author":"F Vaassen","year":"2020","unstructured":"Vaassen F, Hazelaar C, Vaniqui A et al (2020) Evaluation of measures for assessing time-saving of automatic organ-at-risk segmentation in radiotherapy. Phys Imaging Radiat Oncol 13:1\u20136. https:\/\/doi.org\/10.1016\/j.phro.2019.12.001","journal-title":"Phys Imaging Radiat Oncol"},{"key":"3611_CR22","doi-asserted-by":"publisher","first-page":"185","DOI":"10.1016\/j.radonc.2021.05.003","volume":"160","author":"MV Sherer","year":"2021","unstructured":"Sherer MV, Lin D, Elguindi S et al (2021) Metrics to evaluate the performance of auto-segmentation for radiation treatment planning: A critical review. Radiother Oncol 160:185\u2013191. https:\/\/doi.org\/10.1016\/j.radonc.2021.05.003","journal-title":"Radiother Oncol"},{"key":"3611_CR23","doi-asserted-by":"publisher","first-page":"739","DOI":"10.1007\/s10278-022-00735-3","volume":"36","author":"M Yeung","year":"2023","unstructured":"Yeung M, Rundo L, Nan Y et al (2023) Calibrating the dice loss to handle neural network overconfidence for biomedical image segmentation. J Digit Imaging 36:739\u2013752. https:\/\/doi.org\/10.1007\/s10278-022-00735-3","journal-title":"J Digit Imaging"},{"key":"3611_CR24","doi-asserted-by":"publisher","first-page":"843","DOI":"10.7326\/0003-4819-99-6-843","volume":"99","author":"AR Feinstein","year":"1983","unstructured":"Feinstein AR (1983) An additional basic science for clinical medicine: IV. The development of clinimetrics. Ann Intern Med 99:843\u2013848. https:\/\/doi.org\/10.7326\/0003-4819-99-6-843","journal-title":"Ann Intern Med"},{"key":"3611_CR25","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s12874-020-01182-w","volume":"20","author":"HS Christensen","year":"2020","unstructured":"Christensen HS, Borgbjerg J, B\u00f8rty L, B\u00f8gsted M (2020) On Jones et al.\u2019s method for extending Bland-Altman plots to limits of agreement with the mean for multiple observers. BMC Med Res Methodol 20:1\u20138. https:\/\/doi.org\/10.1186\/s12874-020-01182-w","journal-title":"BMC Med Res Methodol"},{"key":"3611_CR26","doi-asserted-by":"publisher","unstructured":"Cornelissen S, Schouten SM, Langenhuizen PPJH et al (2024) Defining tumor growth in vestibular schwannomas: a volumetric inter-observer variability study in contrast-enhanced T1-weighted MRI. https:\/\/doi.org\/10.1007\/s00234-024-03416-w. Neuroradiology","DOI":"10.1007\/s00234-024-03416-w"},{"key":"3611_CR27","doi-asserted-by":"publisher","first-page":"1619","DOI":"10.3171\/2015.4.JNS142415","volume":"124","author":"S Klijn","year":"2016","unstructured":"Klijn S, Verheul JB, Beute GN et al (2016) Gamma knife radiosurgery for vestibular schwannomas: evaluation of tumor control and its predictors in a large patient cohort in the Netherlands. J Neurosurg 124:1619\u20131626. https:\/\/doi.org\/10.3171\/2015.4.JNS142415","journal-title":"J Neurosurg"},{"key":"3611_CR28","doi-asserted-by":"publisher","first-page":"1405","DOI":"10.3171\/2018.6.JNS18516","volume":"131","author":"PPJH Langenhuizen","year":"2019","unstructured":"Langenhuizen PPJH, Zinger S, Hanssens PEJ et al (2019) Influence of pretreatment growth rate on gamma knife treatment response for vestibular Schwannoma: A volumetric analysis. J Neurosurg 131:1405\u20131412. https:\/\/doi.org\/10.3171\/2018.6.JNS18516","journal-title":"J Neurosurg"},{"key":"3611_CR29","doi-asserted-by":"publisher","unstructured":"Shapey J, Kujawa A, Dorent R et al (2021) Segmentation of vestibular Schwannoma from MRI \u2014 An open annotated dataset and baseline algorithm. https:\/\/doi.org\/10.1101\/2021.08.04.21261588. medRxiv 2021.08.04.21261588","DOI":"10.1101\/2021.08.04.21261588"},{"key":"3611_CR30","doi-asserted-by":"publisher","first-page":"203","DOI":"10.1038\/s41592-020-01008-z","volume":"18","author":"F Isensee","year":"2021","unstructured":"Isensee F, Jaeger PF, Kohl SAA et al (2021) nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nat Methods 18:203\u2013211. https:\/\/doi.org\/10.1038\/s41592-020-01008-z","journal-title":"Nat Methods"},{"key":"3611_CR31","doi-asserted-by":"publisher","first-page":"2754","DOI":"10.1002\/lary.30516","volume":"133","author":"K Wang","year":"2023","unstructured":"Wang K, George-Jones NA, Chen L et al (2023) Joint vestibular Schwannoma enlargement prediction and segmentation using a deep Multi-task model. Laryngoscope 133:2754\u20132760. https:\/\/doi.org\/10.1002\/lary.30516","journal-title":"Laryngoscope"},{"key":"3611_CR32","doi-asserted-by":"publisher","first-page":"906","DOI":"10.1097\/MAO.0000000000003055","volume":"42","author":"AM Selleck","year":"2021","unstructured":"Selleck AM, Rodriguez JD, Brown KD (2021) Vestibular Schwannoma Measurements-Is volumetric analysis clinically necessary?? Otol Neurotol 42:906\u2013911. https:\/\/doi.org\/10.1097\/MAO.0000000000003055","journal-title":"Otol Neurotol"}],"container-title":["Neuroradiology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00234-025-03611-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00234-025-03611-3\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00234-025-03611-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,4,29]],"date-time":"2025-04-29T08:02:00Z","timestamp":1745913720000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s00234-025-03611-3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,4]]},"references-count":32,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2025,4]]}},"alternative-id":["3611"],"URL":"https:\/\/doi.org\/10.1007\/s00234-025-03611-3","relation":{},"ISSN":["0028-3940","1432-1920"],"issn-type":[{"value":"0028-3940","type":"print"},{"value":"1432-1920","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,4]]},"assertion":[{"value":"10 January 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"28 March 2025","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"4 April 2025","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"This study was performed in line with the principles of the Declaration of Helsinki. The institutional review board for Radboud University Medical Center (CMO region Arnhem - Nijmegen) waived the need for ethical approval of this study, since our study is not subject to the Dutch Medical Research Involving Human Subjects Act (WMO).","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval"}},{"value":"Informed consent requirement was waived by the institutional review board, since our study is not subject to the Dutch Medical Research Involving Human Subjects Act (WMO).","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Informed consent"}},{"value":"The authors declare no competing interests.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}]}}