{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"institution":[{"name":"medRxiv"}],"indexed":{"date-parts":[[2026,1,16]],"date-time":"2026-01-16T12:16:07Z","timestamp":1768565767396,"version":"3.49.0"},"posted":{"date-parts":[[2023,2,23]]},"group-title":"Radiology and Imaging","reference-count":20,"publisher":"openRxiv","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"accepted":{"date-parts":[[2023,2,23]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                <jats:sec>\n                  <jats:title>Objectives<\/jats:title>\n                  <jats:p>To assess the performance bias caused by sampling data into training and test sets in a mammography radiomics study.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Methods<\/jats:title>\n                  <jats:p>Mammograms from 700 women were used to study upstaging of ductal carcinoma in situ. The dataset was repeatedly shuffled and split into training (n=400) and test cases (n=300) forty times. For each split, cross-validation was used for training, followed by an assessment of the test set. Logistic regression with regularization and support vector machine were used as the machine learning classifiers. For each split and classifier type, multiple models were created based on radiomics and\/or clinical features.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>Area under the curve (AUC) performances varied considerably across the different data splits (e.g., radiomics regression model: train 0.58-0.70, test 0.59\u20130.73). Performances for regression models showed a tradeoff where better training led to worse testing and vice versa. Cross-validation over all cases reduced this variability, but required samples of 500+ cases to yield representative estimates of performance.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Conclusions<\/jats:title>\n                  <jats:p>In medical imaging, clinical datasets are often limited to relatively small size. Models built from different training sets may not be representative of the whole dataset. Depending on the selected data split and model, performance bias could lead to inappropriate conclusions that might influence the clinical significance of the findings. Optimal strategies for test set selection should be developed to ensure study conclusions are appropriate.<\/jats:p>\n                <\/jats:sec>","DOI":"10.1101\/2023.02.15.23285985","type":"posted-content","created":{"date-parts":[[2023,2,23]],"date-time":"2023-02-23T18:25:18Z","timestamp":1677176718000},"source":"Crossref","is-referenced-by-count":0,"title":["Classification performance bias between training and test sets in a limited mammography dataset"],"prefix":"10.64898","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0348-8772","authenticated-orcid":false,"given":"Rui","family":"Hou","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Joseph Y.","family":"Lo","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jeffrey R.","family":"Marks","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"E. Shelley","family":"Hwang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lars J.","family":"Grimm","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"54368","reference":[{"key":"2023022508150338000_2023.02.15.23285985v1.1","unstructured":"American Cancer Society. Breast Cancer Facts & Figures 2019-2020. Atlanta: American Cancer Society, Inc. 2019."},{"issue":"9","key":"2023022508150338000_2023.02.15.23285985v1.2","first-page":"952","volume":"111","year":"2019","journal-title":"Cancer Outcomes in DCIS Patients Without Locoregional Treatment. JNCI: Journal of the National Cancer Institute"},{"key":"2023022508150338000_2023.02.15.23285985v1.3","doi-asserted-by":"publisher","DOI":"10.1002\/cncr.21069"},{"issue":"27_suppl","key":"2023022508150338000_2023.02.15.23285985v1.4","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1200\/jco.2011.29.27_suppl.39","article-title":"Long-term outcome of DCIS patients: p53 as a biomarker of ipsilateral recurrence","volume":"29","year":"2011","journal-title":"Journal of Clinical Oncology"},{"issue":"1","key":"2023022508150338000_2023.02.15.23285985v1.5","first-page":"119","volume":"260","year":"2011","journal-title":"Ductal carcinoma in situ at core-needle biopsy: meta-analysis of underestimation and predictors of invasive breast cancer. Radiology"},{"issue":"12","key":"2023022508150338000_2023.02.15.23285985v1.6","first-page":"3534","volume":"24","year":"2017","journal-title":"Surgical Upstaging Rates for Vacuum Assisted Biopsy Proven DCIS: Implications for Active Surveillance Trials. Annals of Surgical Oncology"},{"issue":"1","key":"2023022508150338000_2023.02.15.23285985v1.7","first-page":"54","volume":"303","year":"2022","journal-title":"Prediction of Upstaging in Ductal Carcinoma in Situ Based on Mammographic Radiomic Features. Radiology"},{"key":"2023022508150338000_2023.02.15.23285985v1.8","doi-asserted-by":"crossref","unstructured":"Wang X , Peng Y , Lu L , Lu Z , Bagheri M , Summers RM. ChestX-Ray8: Hospital-Scale Chest X-Ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2017. p. 3462\u201371.","DOI":"10.1109\/CVPR.2017.369"},{"key":"2023022508150338000_2023.02.15.23285985v1.9","doi-asserted-by":"crossref","unstructured":"Irvin J , Rajpurkar P , Ko M , Yu Y , Ciurea-Ilcus S , Chute C , et al., editors. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. Proceedings of the AAAI Conference on Artificial Intelligence; 2019.","DOI":"10.1609\/aaai.v33i01.3301590"},{"key":"2023022508150338000_2023.02.15.23285985v1.10","doi-asserted-by":"crossref","unstructured":"Yan K , Wang X , Lu L , Summers RM , editors. DeepLesion: automated mining of large-scale lesion annotations and universal lesion detection with deep learning. J Medical Imaging; 2018: SPIE.","DOI":"10.1117\/1.JMI.5.3.036501"},{"key":"2023022508150338000_2023.02.15.23285985v1.11","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2016.07.007"},{"issue":"3","key":"2023022508150338000_2023.02.15.23285985v1.12","first-page":"e200265","volume":"3","year":"2020","journal-title":"Evaluation of Combined Artificial Intelligence and Radiologist Assessment to Interpret Screening Mammograms. JAMA Netw Open"},{"issue":"3","key":"2023022508150338000_2023.02.15.23285985v1.13","first-page":"e138","volume":"2","year":"2020","journal-title":"Changes in cancer detection and false-positive recall in mammography using artificial intelligence: a retrospective, multireader study. The Lancet Digital Health"},{"issue":"7788","key":"2023022508150338000_2023.02.15.23285985v1.14","first-page":"89","volume":"577","year":"2020","journal-title":"International evaluation of an AI system for breast cancer screening. Nature"},{"issue":"7639","key":"2023022508150338000_2023.02.15.23285985v1.15","first-page":"115","volume":"542","year":"2017","journal-title":"Dermatologist-level classification of skin cancer with deep neural networks. Nature"},{"issue":"1","key":"2023022508150338000_2023.02.15.23285985v1.16","first-page":"1","article-title":"Deep learning to improve breast cancer detection on screening mammography","volume":"9","year":"2019","journal-title":"Scientific reports"},{"issue":"xxxx","key":"2023022508150338000_2023.02.15.23285985v1.17","first-page":"20170576","article-title":"Classification of breast cancer in ultrasound imaging using a generic deep learning analysis software: a pilot study","volume":"91","year":"2018","journal-title":"Br J Radiol"},{"issue":"3","key":"2023022508150338000_2023.02.15.23285985v1.18","first-page":"527","volume":"15","year":"2018","journal-title":"Prediction of occult invasive disease in ductal carcinoma in situ using deep learning features. Journal of the American College of Radiology"},{"key":"2023022508150338000_2023.02.15.23285985v1.19","doi-asserted-by":"crossref","unstructured":"Shi B , Grimm LJ , Mazurowski MA , Baker JA , Marks JR , King LM , et al. Can Occult Invasive Disease in Ductal Carcinoma In Situ Be Predicted Using Computer-extracted Mammographic Features? Acad Radiol. 2017.","DOI":"10.1117\/12.2255731"},{"issue":"6","key":"2023022508150338000_2023.02.15.23285985v1.20","first-page":"1565","volume":"67","year":"2020","journal-title":"Prediction of Upstaged Ductal Carcinoma In Situ Using Forced Labeling and Domain Adaptation. IEEE Trans Biomed Eng"}],"container-title":[],"original-title":[],"link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.1101\/2023.02.15.23285985","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,15]],"date-time":"2026-01-15T19:27:17Z","timestamp":1768505237000},"score":1,"resource":{"primary":{"URL":"http:\/\/medrxiv.org\/lookup\/doi\/10.1101\/2023.02.15.23285985"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,2,23]]},"references-count":20,"URL":"https:\/\/doi.org\/10.1101\/2023.02.15.23285985","relation":{"is-preprint-of":[{"id-type":"doi","id":"10.1371\/journal.pone.0282402","asserted-by":"subject"}]},"subject":[],"published":{"date-parts":[[2023,2,23]]},"subtype":"preprint"}}