{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,4]],"date-time":"2026-11-04T19:33:59Z","timestamp":1793820839956,"version":"4.6.1"},"reference-count":24,"publisher":"Public Library of Science (PLoS)","issue":"2","license":[{"start":{"date-parts":[[2024,2,7]],"date-time":"2024-02-07T00:00:00Z","timestamp":1707264000000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Cancer Institute","award":["U01-CA214183"],"award-info":[{"award-number":["U01-CA214183"]}]},{"name":"National Cancer Institute","award":["R01-CA185138"],"award-info":[{"award-number":["R01-CA185138"]}]},{"name":"DOD Breast Cancer Research Program","award":["W81XWH-14-1-0473"],"award-info":[{"award-number":["W81XWH-14-1-0473"]}]},{"DOI":"10.13039\/100001006","name":"Breast Cancer Research Foundation","doi-asserted-by":"crossref","award":["BCRF-16-183"],"award-info":[{"award-number":["BCRF-16-183"]}],"id":[{"id":"10.13039\/100001006","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100001006","name":"Breast Cancer Research Foundation","doi-asserted-by":"crossref","award":["BCRF-17-073"],"award-info":[{"award-number":["BCRF-17-073"]}],"id":[{"id":"10.13039\/100001006","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Cancer Research UK and Dutch Cancer Society","award":["C38317\/A24043"],"award-info":[{"award-number":["C38317\/A24043"]}]}],"content-domain":{"domain":["www.plosone.org"],"crossmark-restriction":false},"short-container-title":["PLoS ONE"],"abstract":"<jats:sec id=\"sec001\">\n                    <jats:title>Objectives<\/jats:title>\n                    <jats:p>To assess the performance bias caused by sampling data into training and test sets in a mammography radiomics study.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"sec002\">\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>Mammograms from 700 women were used to study upstaging of ductal carcinoma in situ. The dataset was repeatedly shuffled and split into training (n = 400) and test cases (n = 300) forty times. For each split, cross-validation was used for training, followed by an assessment of the test set. Logistic regression with regularization and support vector machine were used as the machine learning classifiers. For each split and classifier type, multiple models were created based on radiomics and\/or clinical features.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"sec003\">\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Area under the curve (AUC) performances varied considerably across the different data splits (e.g., radiomics regression model: train 0.58\u20130.70, test 0.59\u20130.73). Performances for regression models showed a tradeoff where better training led to worse testing and vice versa. Cross-validation over all cases reduced this variability, but required samples of 500+ cases to yield representative estimates of performance.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"sec004\">\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>In medical imaging, clinical datasets are often limited to relatively small size. Models built from different training sets may not be representative of the whole dataset. Depending on the selected data split and model, performance bias could lead to inappropriate conclusions that might influence the clinical significance of the findings.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"sec005\">\n                    <jats:title>Advances in knowledge<\/jats:title>\n                    <jats:p>Performance bias can result from model testing when using limited datasets. Optimal strategies for test set selection should be developed to ensure study conclusions are appropriate.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1371\/journal.pone.0282402","type":"journal-article","created":{"date-parts":[[2024,2,7]],"date-time":"2024-02-07T13:27:27Z","timestamp":1707312447000},"page":"e0282402","update-policy":"https:\/\/doi.org\/10.1371\/journal.pone.corrections_policy","source":"Crossref","is-referenced-by-count":14,"title":["Classification performance bias between training and test sets in a limited mammography dataset"],"prefix":"10.1371","volume":"19","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0348-8772","authenticated-orcid":true,"given":"Rui","family":"Hou","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9540-5072","authenticated-orcid":true,"given":"Joseph Y.","family":"Lo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jeffrey R.","family":"Marks","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"E. Shelley","family":"Hwang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lars J.","family":"Grimm","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2024,2,7]]},"reference":[{"key":"pone.0282402.ref001","unstructured":"American Cancer Society. Breast Cancer Facts & Figures 2019\u20132020, Atlanta: American Cancer Society, Inc., 2019."},{"issue":"9","key":"pone.0282402.ref002","doi-asserted-by":"crossref","first-page":"952","DOI":"10.1093\/jnci\/djy220","article-title":"Cancer Outcomes in DCIS Patients Without Locoregional Treatment","volume":"111","author":"M. D. Ryser","year":"2019","journal-title":"JNCI: Journal of the National Cancer Institute"},{"issue":"12","key":"pone.0282402.ref003","doi-asserted-by":"crossref","first-page":"2481","DOI":"10.1002\/cncr.21069","article-title":"The natural history of low-grade ductal carcinoma in situ of the breast in women treated by biopsy only revealed over 30 years of long-term follow-up","volume":"103","author":"M. E. Sanders","year":"2005","journal-title":"Cancer"},{"issue":"27_suppl","key":"pone.0282402.ref004","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1200\/jco.2011.29.27_suppl.39","article-title":"Long-term outcome of DCIS patients: p53 as a biomarker of ipsilateral recurrence","volume":"29","author":"T. J. Hieken","year":"2011","journal-title":"Journal of Clinical Oncology"},{"issue":"1","key":"pone.0282402.ref005","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1148\/radiol.11102368","article-title":"Ductal carcinoma in situ at core-needle biopsy: meta-analysis of underestimation and predictors of invasive breast cancer","volume":"260","author":"M. E. Brennan","year":"2011","journal-title":"Radiology"},{"issue":"12","key":"pone.0282402.ref006","doi-asserted-by":"crossref","first-page":"3534","DOI":"10.1245\/s10434-017-6018-9","article-title":"Surgical Upstaging Rates for Vacuum Assisted Biopsy Proven DCIS: Implications for Active Surveillance Trials","volume":"24","author":"L. J. Grimm","year":"2017","journal-title":"Ann Surg Oncol"},{"issue":"1","key":"pone.0282402.ref007","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1148\/radiol.210407","article-title":"Prediction of Upstaging in Ductal Carcinoma in Situ Based on Mammographic Radiomic Features","volume":"303","author":"R. Hou","year":"2022","journal-title":"Radiology"},{"key":"pone.0282402.ref008","doi-asserted-by":"crossref","unstructured":"X. Wang, Y. Peng, L. Lu, Z. Lu, M. Bagheri, and R. M. Summers, \"ChestX-Ray8: Hospital-Scale Chest X-Ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases,\" presented at the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.","DOI":"10.1109\/CVPR.2017.369"},{"key":"pone.0282402.ref009","doi-asserted-by":"crossref","unstructured":"J. Irvin et al., \"Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison,\" in Proceedings of the AAAI Conference on Artificial Intelligence, 2019, vol. 33, pp. 590\u2013597.","DOI":"10.1609\/aaai.v33i01.3301590"},{"issue":"3","key":"pone.0282402.ref010","first-page":"11","article-title":"DeepLesion: automated mining of large-scale lesion annotations and universal lesion detection with deep learning","volume":"5","author":"K. Yan","year":"2018","journal-title":"J. Medical Imaging"},{"key":"pone.0282402.ref011","doi-asserted-by":"crossref","first-page":"303","DOI":"10.1016\/j.media.2016.07.007","article-title":"Large scale deep learning for computer aided detection of mammographic lesions","volume":"35","author":"T. Kooi","year":"2017","journal-title":"Medical image analysis"},{"issue":"3","key":"pone.0282402.ref012","doi-asserted-by":"crossref","first-page":"e200265","DOI":"10.1001\/jamanetworkopen.2020.0265","article-title":"Evaluation of Combined Artificial Intelligence and Radiologist Assessment to Interpret Screening Mammograms","volume":"3","author":"T. Schaffter","year":"2020","journal-title":"JAMA Netw Open"},{"issue":"3","key":"pone.0282402.ref013","doi-asserted-by":"crossref","first-page":"e138","DOI":"10.1016\/S2589-7500(20)30003-0","article-title":"Changes in cancer detection and false-positive recall in mammography using artificial intelligence: a retrospective, multireader study","volume":"2","author":"H.-E. Kim","year":"2020","journal-title":"The Lancet Digital Health"},{"issue":"7788","key":"pone.0282402.ref014","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1038\/s41586-019-1799-6","article-title":"International evaluation of an AI system for breast cancer screening","volume":"577","author":"S. M. McKinney","year":"2020","journal-title":"Nature"},{"issue":"7639","key":"pone.0282402.ref015","doi-asserted-by":"crossref","first-page":"115","DOI":"10.1038\/nature21056","article-title":"Dermatologist-level classification of skin cancer with deep neural networks","volume":"542","author":"A. Esteva","year":"2017","journal-title":"Nature"},{"issue":"1","key":"pone.0282402.ref016","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41598-019-48995-4","article-title":"Deep learning to improve breast cancer detection on screening mammography","volume":"9","author":"L. Shen","year":"2019","journal-title":"Scientific reports"},{"issue":"xxxx","key":"pone.0282402.ref017","doi-asserted-by":"crossref","first-page":"20170576","DOI":"10.1259\/bjr.20170576","article-title":"Classification of breast cancer in ultrasound imaging using a generic deep learning analysis software: a pilot study","volume":"91","author":"A. S. Becker","year":"2018","journal-title":"Br J Radiol"},{"key":"pone.0282402.ref018","doi-asserted-by":"crossref","first-page":"68","DOI":"10.1016\/j.neuroimage.2017.06.061","article-title":"Cross-validation failure: Small sample sizes lead to large error bars","volume":"180","author":"G. Varoquaux","year":"2018","journal-title":"Neuroimage"},{"issue":"3","key":"pone.0282402.ref019","doi-asserted-by":"crossref","first-page":"249","DOI":"10.1007\/s41664-018-0068-2","article-title":"On splitting training and validation set: a comparative study of cross-validation, bootstrap and systematic sampling for estimating the generalization performance of supervised learning","volume":"2","author":"Y. Xu","year":"2018","journal-title":"Journal of analysis and testing"},{"issue":"8","key":"pone.0282402.ref020","doi-asserted-by":"crossref","first-page":"e0256152","DOI":"10.1371\/journal.pone.0256152","article-title":"Radiomics machine learning study with a small sample size: Single random training-test set split may lead to unreliable results","volume":"16","author":"C. An","year":"2021","journal-title":"PLoS One"},{"key":"pone.0282402.ref021","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s13244-021-01115-1","article-title":"Measuring the bias of incorrect application of feature selection when using cross-validation in radiomics","volume":"12","author":"A. Demircio\u011flu","year":"2021","journal-title":"Insights into Imaging"},{"issue":"3","key":"pone.0282402.ref022","doi-asserted-by":"crossref","first-page":"527","DOI":"10.1016\/j.jacr.2017.11.036","article-title":"Prediction of occult invasive disease in ductal carcinoma in situ using deep learning features","volume":"15","author":"B. Shi","year":"2018","journal-title":"Journal of the American College of Radiology"},{"key":"pone.0282402.ref023","article-title":"Can Occult Invasive Disease in Ductal Carcinoma In Situ Be Predicted Using Computer-extracted Mammographic Features?","author":"B. Shi","year":"2017","journal-title":"Acad Radiol"},{"issue":"6","key":"pone.0282402.ref024","doi-asserted-by":"crossref","first-page":"1565","DOI":"10.1109\/TBME.2019.2940195","article-title":"Prediction of Upstaged Ductal Carcinoma In Situ Using Forced Labeling and Domain Adaptation","volume":"67","author":"R. Hou","year":"2020","journal-title":"IEEE Trans Biomed Eng"}],"container-title":["PLOS ONE"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0282402","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,10]],"date-time":"2024-11-10T14:43:16Z","timestamp":1731249796000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0282402"}},"subtitle":[],"editor":[{"given":"Michal","family":"Rosen-Zvi","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2024,2,7]]},"references-count":24,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2024,2,7]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pone.0282402","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2023.02.15.23285985","asserted-by":"object"}]},"ISSN":["1932-6203"],"issn-type":[{"value":"1932-6203","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,2,7]]}}}