{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,2]],"date-time":"2026-02-02T07:45:40Z","timestamp":1770018340779,"version":"3.49.0"},"reference-count":31,"publisher":"Springer Science and Business Media LLC","issue":"4","license":[{"start":{"date-parts":[[2023,3,17]],"date-time":"2023-03-17T00:00:00Z","timestamp":1679011200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,3,17]],"date-time":"2023-03-17T00:00:00Z","timestamp":1679011200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Mayo Clinic Ventures, nference, and the Mayo Clinic Office of Translation to Practice"},{"DOI":"10.13039\/100000062","name":"National Institute of Diabetes and Digestive and Kidney Diseases","doi-asserted-by":"publisher","award":["K01DK110136"],"award-info":[{"award-number":["K01DK110136"]}],"id":[{"id":"10.13039\/100000062","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000062","name":"National Institute of Diabetes and Digestive and Kidney Diseases","doi-asserted-by":"publisher","award":["R03DK125632"],"award-info":[{"award-number":["R03DK125632"]}],"id":[{"id":"10.13039\/100000062","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Digit Imaging"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The aim of this study is to investigate the use of an exponential-plateau model to determine the required training dataset size that yields the maximum medical image segmentation performance. CT and MR images of patients with renal tumors acquired between 1997 and 2017 were retrospectively collected from our nephrectomy registry. Modality-based datasets of 50, 100, 150, 200, 250, and 300 images were assembled to train models with an 80\u201320 training-validation split evaluated against 50 randomly held out test set images. A third experiment using the KiTS21 dataset was also used to explore the effects of different model architectures. Exponential-plateau models were used to establish the relationship of dataset size to model generalizability performance. For segmenting non-neoplastic kidney regions on CT and MR imaging, our model yielded test Dice score plateaus of <jats:inline-formula><jats:alternatives><jats:tex-math>$$0.93\\pm 0.02$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                  <mml:mrow>\n                    <mml:mn>0.93<\/mml:mn>\n                    <mml:mo>\u00b1<\/mml:mo>\n                    <mml:mn>0.02<\/mml:mn>\n                  <\/mml:mrow>\n                <\/mml:math><\/jats:alternatives><\/jats:inline-formula> and <jats:inline-formula><jats:alternatives><jats:tex-math>$$0.92\\pm 0.04$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                  <mml:mrow>\n                    <mml:mn>0.92<\/mml:mn>\n                    <mml:mo>\u00b1<\/mml:mo>\n                    <mml:mn>0.04<\/mml:mn>\n                  <\/mml:mrow>\n                <\/mml:math><\/jats:alternatives><\/jats:inline-formula> with the number of training-validation images needed to reach the plateaus of 54 and 122, respectively. For segmenting CT and MR tumor regions, we modeled a test Dice score plateau of <jats:inline-formula><jats:alternatives><jats:tex-math>$$0.85\\pm 0.20$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                  <mml:mrow>\n                    <mml:mn>0.85<\/mml:mn>\n                    <mml:mo>\u00b1<\/mml:mo>\n                    <mml:mn>0.20<\/mml:mn>\n                  <\/mml:mrow>\n                <\/mml:math><\/jats:alternatives><\/jats:inline-formula> and <jats:inline-formula><jats:alternatives><jats:tex-math>$$0.76\\pm 0.27$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                  <mml:mrow>\n                    <mml:mn>0.76<\/mml:mn>\n                    <mml:mo>\u00b1<\/mml:mo>\n                    <mml:mn>0.27<\/mml:mn>\n                  <\/mml:mrow>\n                <\/mml:math><\/jats:alternatives><\/jats:inline-formula>, with 125 and 389 training-validation images needed to reach the plateaus. For the KiTS21 dataset, the best Dice score plateaus for nn-UNet 2D and 3D architectures were <jats:inline-formula><jats:alternatives><jats:tex-math>$$0.67\\pm 0.29$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                  <mml:mrow>\n                    <mml:mn>0.67<\/mml:mn>\n                    <mml:mo>\u00b1<\/mml:mo>\n                    <mml:mn>0.29<\/mml:mn>\n                  <\/mml:mrow>\n                <\/mml:math><\/jats:alternatives><\/jats:inline-formula> and <jats:inline-formula><jats:alternatives><jats:tex-math>$$0.84\\pm 0.18$$<\/jats:tex-math><mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\">\n                  <mml:mrow>\n                    <mml:mn>0.84<\/mml:mn>\n                    <mml:mo>\u00b1<\/mml:mo>\n                    <mml:mn>0.18<\/mml:mn>\n                  <\/mml:mrow>\n                <\/mml:math><\/jats:alternatives><\/jats:inline-formula> with number to reach performance plateau of 177 and 440. Our research validates that differing imaging modalities, target structures, and model architectures all affect the amount of training images required to reach a performance plateau. The modeling approach we developed will help future researchers determine for their experiments when additional training-validation images will likely not further improve model performance.\n<\/jats:p>","DOI":"10.1007\/s10278-023-00804-1","type":"journal-article","created":{"date-parts":[[2023,3,17]],"date-time":"2023-03-17T19:02:51Z","timestamp":1679079771000},"page":"1770-1781","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":12,"title":["Effect of Dataset Size and Medical Image Modality on Convolutional Neural Network Model Performance for Automated Segmentation: A CT and MR Renal Tumor Imaging Study"],"prefix":"10.1007","volume":"36","author":[{"given":"Harrison C.","family":"Gottlich","sequence":"first","affiliation":[]},{"given":"Adriana V.","family":"Gregory","sequence":"additional","affiliation":[]},{"given":"Vidit","family":"Sharma","sequence":"additional","affiliation":[]},{"given":"Abhinav","family":"Khanna","sequence":"additional","affiliation":[]},{"given":"Amr U.","family":"Moustafa","sequence":"additional","affiliation":[]},{"given":"Christine M.","family":"Lohse","sequence":"additional","affiliation":[]},{"given":"Theodora A.","family":"Potretzke","sequence":"additional","affiliation":[]},{"given":"Panagiotis","family":"Korfiatis","sequence":"additional","affiliation":[]},{"given":"Aaron M.","family":"Potretzke","sequence":"additional","affiliation":[]},{"given":"Aleksandar","family":"Denic","sequence":"additional","affiliation":[]},{"given":"Andrew D.","family":"Rule","sequence":"additional","affiliation":[]},{"given":"Naoki","family":"Takahashi","sequence":"additional","affiliation":[]},{"given":"Bradley J.","family":"Erickson","sequence":"additional","affiliation":[]},{"given":"Bradley C.","family":"Leibovich","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7917-9853","authenticated-orcid":false,"given":"Timothy L.","family":"Kline","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2023,3,17]]},"reference":[{"key":"804_CR1","doi-asserted-by":"crossref","unstructured":"Tsui KH, Shvarts O, Smith RB, Figlin R, de Kernion JB, Belldegrun A: Renal cell carcinoma: prognostic significance of incidentally detected tumors. The Journal of Urology 163:426\u2013430, 20000","DOI":"10.1016\/S0022-5347(05)67892-5"},{"key":"804_CR2","doi-asserted-by":"publisher","first-page":"1331","DOI":"10.1093\/jnci\/djj362","volume":"98","author":"JM Hollingsworth","year":"2006","unstructured":"Hollingsworth JM, Miller DC, Daignault S, Hollenbeck BK: Rising incidence of small renal masses: a need to reassess treatment effect. Journal of the National Cancer Institute 98:1331-1334, 2006","journal-title":"Journal of the National Cancer Institute"},{"key":"804_CR3","doi-asserted-by":"publisher","first-page":"1119","DOI":"10.1016\/S0140-6736(09)60229-4","volume":"373","author":"BI Rini","year":"2009","unstructured":"Rini BI, Campbell SC, Escudier B: Renal cell carcinoma. Lancet (London, England) 373:1119-1132, 2009","journal-title":"Lancet (London, England)"},{"key":"804_CR4","doi-asserted-by":"publisher","first-page":"692","DOI":"10.1016\/j.euf.2021.07.017","volume":"7","author":"J Rickman","year":"2021","unstructured":"Rickman J, Struyk G, Simpson B, Byun BC, Papanikolopoulos N: The growing role for semantic segmentation in urology. European Urology Focus 7:692-695, 2021","journal-title":"European Urology Focus"},{"key":"804_CR5","doi-asserted-by":"crossref","unstructured":"Cuingnet R, Prevost R, Lesage D, Cohen LD, Mory B, Ardon R: Automatic detection and segmentation of kidneys in 3D CT images using random forests. Proc. International conference on medical image computing and computer-assisted intervention City, 2012 Year","DOI":"10.1007\/978-3-642-33454-2_9"},{"key":"804_CR6","doi-asserted-by":"crossref","unstructured":"Torres H, et al.: Kidney segmentation in 3D CT images using B-spline explicit active surfaces. Proc. 2016 IEEE International Conference on Serious Games and Applications for Health (SeGAH): City, May 1, 2016 Year","DOI":"10.1109\/SeGAH.2016.7586276"},{"key":"804_CR7","doi-asserted-by":"crossref","unstructured":"Roth H, et al.: A new 2.5D representation for lymph node detection using random sets of deep convolutional neural network observations. Proc. Med Image Comput Comput Assist Interv: City, June 6, 2014 Year","DOI":"10.1007\/978-3-319-10404-1_65"},{"key":"804_CR8","doi-asserted-by":"publisher","first-page":"61","DOI":"10.1016\/j.media.2016.10.004","volume":"36","author":"K Kamnitsas","year":"2017","unstructured":"Kamnitsas K, et al.: Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation. Medical Image Analysis 36:61-78, 2017","journal-title":"Medical Image Analysis"},{"key":"804_CR9","doi-asserted-by":"crossref","unstructured":"Janssens R, Zeng G, Zheng G: Fully automatic segmentation of lumbar vertebrae from CT images using cascaded 3D fully convolutional networks. arXiv:171201509 [cs], 2017","DOI":"10.1109\/ISBI.2018.8363715"},{"key":"804_CR10","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1016\/j.media.2017.07.005","volume":"42","author":"G Litjens","year":"2017","unstructured":"Litjens G, et al.: A survey on deep learning in medical image analysis. Medical Image Analysis 42:60-88, 2017","journal-title":"Medical Image Analysis"},{"key":"804_CR11","doi-asserted-by":"crossref","unstructured":"Ronneberger O, Fischer P, Brox T: U-Net: convolutional networks for biomedical image segmentation. arXiv:150504597 [cs], 2015","DOI":"10.1007\/978-3-319-24574-4_28"},{"key":"804_CR12","doi-asserted-by":"publisher","DOI":"10.1117\/1.JMI.6.3.034001","volume":"6","author":"S M\u00fcller","year":"2019","unstructured":"M\u00fcller S, et al.: Benchmarking Wilms\u2019 tumor in multisequence MRI data: why does current clinical practice fail? Which popular segmentation algorithms perform well? Journal of Medical Imaging 6:034001, 2019","journal-title":"Journal of Medical Imaging"},{"key":"804_CR13","unstructured":"Heller N, et al.: The KiTS19 challenge data: 300 kidney tumor cases with clinical context, ct semantic segmentations, and surgical outcomes. arXiv:190400445 [cs, q-bio, stat], 2020"},{"key":"804_CR14","unstructured":"Isensee F, J\u00e4ger PF, Kohl SA, Petersen J, Maier-Hein KH: Automated design of deep learning methods for biomedical image segmentation. arXiv preprint arXiv:190408128, 2019"},{"key":"804_CR15","doi-asserted-by":"publisher","first-page":"203","DOI":"10.1038\/s41592-020-01008-z","volume":"18","author":"F Isensee","year":"2021","unstructured":"Isensee F, J\u00e4ger PF, Kohl SAA, Petersen J, Maier-Hein KH: Automated design of deep learning methods for biomedical image segmentation. Nature Methods 18:203-211, 2021","journal-title":"Nature Methods"},{"key":"804_CR16","unstructured":"Chen ZZ: A coarse-to-fine framework for the 2021 kidney and kidney tumor segmentation challenge, 2021"},{"key":"804_CR17","doi-asserted-by":"crossref","unstructured":"Golts A, Khapun D, Shats D, Shoshan Y, Gilboa-Solomon F: An ensemble of 3D U-Net based models for segmentation of kidney and masses in CT scans. International Challenge on Kidney and Kidney Tumor Segmentation, 2021","DOI":"10.1007\/978-3-030-98385-7_14"},{"key":"804_CR18","unstructured":"The 2021 kidney and kidney tumor segmentation challenge. Available at https:\/\/kits21.kits-challenge.org\/. Accessed 01\/21\/2023 2023."},{"key":"804_CR19","doi-asserted-by":"publisher","first-page":"2642","DOI":"10.1681\/ASN.2020040449","volume":"31","author":"A Denic","year":"2020","unstructured":"Denic A, et al.: Larger nephron size and nephrosclerosis predict progressive CKD and mortality after radical nephrectomy for tumor and independent of kidney function. Journal of the American Society of Nephrology 31:2642-2652, 2020","journal-title":"Journal of the American Society of Nephrology"},{"key":"804_CR20","doi-asserted-by":"publisher","first-page":"1116","DOI":"10.1016\/j.neuroimage.2006.01.015","volume":"31","author":"PA Yushkevich","year":"2006","unstructured":"Yushkevich PA, et al.: User-guided 3D active contour segmentation of anatomical structures: significantly improved efficiency and reliability. Neuroimage 31:1116-1128, 2006","journal-title":"Neuroimage"},{"key":"804_CR21","doi-asserted-by":"publisher","first-page":"1514","DOI":"10.1681\/ASN.2018090902","volume":"30","author":"MD van Gastel","year":"2019","unstructured":"van Gastel MD, Edwards ME, Torres VE, Erickson BJ, Gansevoort RT, Kline TL: Automatic measurement of kidney and liver volumes from MR images of patients affected by autosomal dominant polycystic kidney disease. Journal of the American Society of Nephrology 30:1514-1522, 2019","journal-title":"Journal of the American Society of Nephrology"},{"key":"804_CR22","unstructured":"nnUNet. Available at https:\/\/github.com\/MIC-DKFZ\/nnUNet. Accessed 1\/15 2022."},{"key":"804_CR23","doi-asserted-by":"crossref","unstructured":"Taha AA, Hanbury AJBmi: Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool15:1\u201328, 2015","DOI":"10.1186\/s12880-015-0068-x"},{"key":"804_CR24","unstructured":"Bressert E: SciPy and NumPy: an overview for developers, 2012"},{"key":"804_CR25","doi-asserted-by":"crossref","unstructured":"Ma J, et al.: Abdomenct-1k: Is abdominal organ segmentation a solved problem, 2021","DOI":"10.1109\/TPAMI.2021.3100536"},{"key":"804_CR26","doi-asserted-by":"publisher","first-page":"1391","DOI":"10.1007\/s00330-018-5695-5","volume":"29","author":"L Joskowicz","year":"2019","unstructured":"Joskowicz L, Cohen D, Caplan N, Sosna J: Inter-observer variability of manual contour delineation of structures in CT. European Radiology 29:1391-1399, 2019","journal-title":"European Radiology"},{"key":"804_CR27","doi-asserted-by":"publisher","first-page":"1125","DOI":"10.1002\/mrm.28768","volume":"86","author":"AJ Daniel","year":"2021","unstructured":"Daniel AJ, et al.: Automated renal segmentation in healthy and chronic kidney disease subjects using a convolutional neural network. Magnetic Resonance in Medicine 86:1125-1136, 2021","journal-title":"Magnetic Resonance in Medicine"},{"key":"804_CR28","doi-asserted-by":"crossref","unstructured":"Haarburger C, M\u00fcller-Franzes G, Weninger L, Kuhl C, Truhn D, Merhof DJSr: Radiomics feature reproducibility under inter-rater variability in segmentations of CT images10:1\u201310, 2020","DOI":"10.1038\/s41598-020-69534-6"},{"key":"804_CR29","doi-asserted-by":"publisher","first-page":"1676","DOI":"10.1007\/s00330-020-07233-6","volume":"31","author":"K Nguyen","year":"2021","unstructured":"Nguyen K, Schieda N, James N, McInnes MD, Wu M, Thornhill RE: Effect of phase of enhancement on texture analysis in renal masses evaluated with non-contrast-enhanced, corticomedullary, and nephrographic phase\u2013enhanced CT images. European Radiology 31:1676-1686, 2021","journal-title":"European Radiology"},{"key":"804_CR30","doi-asserted-by":"crossref","unstructured":"Lappas G, et al.: Inter-observer variability of organ contouring for preclinical studies with cone beam Computed Tomography imaging21:11\u201317, 2022","DOI":"10.1016\/j.phro.2022.01.002"},{"key":"804_CR31","unstructured":"Bilic P, et al.: The liver tumor segmentation benchmark (lits), 2019"}],"container-title":["Journal of Digital Imaging"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10278-023-00804-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10278-023-00804-1\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10278-023-00804-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,8,7]],"date-time":"2023-08-07T17:07:33Z","timestamp":1691428053000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10278-023-00804-1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,3,17]]},"references-count":31,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2023,8]]}},"alternative-id":["804"],"URL":"https:\/\/doi.org\/10.1007\/s10278-023-00804-1","relation":{},"ISSN":["1618-727X"],"issn-type":[{"value":"1618-727X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,3,17]]},"assertion":[{"value":"11 September 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"28 February 2023","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"1 March 2023","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"17 March 2023","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The following statement was supplied by the Mayo Clinic IRB. \u201cThe Committee discussed the investigator\u2019s justification for waiver of consent and waiver of HIPAA authorization as provided in the IRBe application (sections \u201cWaiver of Informed Consent\u201d and \u201cHIPAA Authorization Waiver\u201d) and approved waiver of consent in accordance with 45CFR46.116 and waiver of HIPAA authorization in accordance with applicable HIPAA regulations.\u201d","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics Approval"}},{"value":"The authors declare no competing interests.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing Interests"}},{"value":"The content is solely the responsibility of the authors and does not necessarily represent the official views of Mayo Clinic or nference or the National Institutes of Health.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Disclaimer"}}]}}