{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,11]],"date-time":"2026-06-11T12:37:04Z","timestamp":1781181424107,"version":"3.54.1"},"reference-count":33,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2024,1,10]],"date-time":"2024-01-10T00:00:00Z","timestamp":1704844800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,1,10]],"date-time":"2024-01-10T00:00:00Z","timestamp":1704844800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100000054","name":"National Cancer Institute","doi-asserted-by":"publisher","award":["R01CA247910"],"award-info":[{"award-number":["R01CA247910"]}],"id":[{"id":"10.13039\/100000054","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000054","name":"National Cancer Institute","doi-asserted-by":"publisher","award":["U54CA132378"],"award-info":[{"award-number":["U54CA132378"]}],"id":[{"id":"10.13039\/100000054","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Digit Imaging. Inform. med."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Deep neural networks have demonstrated promising performance in screening mammography with recent studies reporting performance at or above the level of trained radiologists on internal datasets. However, it remains unclear whether the performance of these trained models is robust and replicates across external datasets. In this study, we evaluate four state-of-the-art publicly available models using four publicly available mammography datasets (CBIS-DDSM, INbreast, CMMD, OMI-DB). Where test data was available, published results were replicated. The best-performing model, which achieved an area under the ROC curve (AUC) of 0.88 on internal data from NYU, achieved here an AUC of 0.9 on the external CMMD dataset (<jats:italic>N<\/jats:italic>\u2009=\u2009826 exams). On the larger OMI-DB dataset (<jats:italic>N<\/jats:italic>\u2009=\u200911,440 exams), it achieved an AUC of 0.84 but did not match the performance of individual radiologists (at a specificity of 0.92, the sensitivity was 0.97 for the radiologist and 0.53 for the network for a 1-year follow-up). The network showed higher performance for in situ cancers, as opposed to invasive cancers. Among invasive cancers, it was relatively weaker at identifying asymmetries and was relatively stronger at identifying masses. The three other trained models that we evaluated all performed poorly on external datasets. Independent validation of trained models is an essential step to ensure safe and reliable use. Future progress in AI for mammography may depend on a concerted effort to make larger datasets publicly available that span multiple clinical sites.<\/jats:p>","DOI":"10.1007\/s10278-023-00943-5","type":"journal-article","created":{"date-parts":[[2024,1,10]],"date-time":"2024-01-10T18:01:32Z","timestamp":1704909692000},"page":"536-546","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":11,"title":["Robustness of Deep Networks for Mammography: Replication Across Public Datasets"],"prefix":"10.1007","volume":"37","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7291-4018","authenticated-orcid":false,"given":"Osvaldo M.","family":"Velarde","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Clarissa","family":"Lin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sarah","family":"Eskreis-Winkler","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lucas C.","family":"Parra","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,1,10]]},"reference":[{"key":"943_CR1","unstructured":"U.S. Breast Cancer Statistics. Breastcancer.org https:\/\/www.breastcancer.org\/symptoms\/understand_bc\/statistics . (2021)."},{"key":"943_CR2","doi-asserted-by":"publisher","first-page":"1784","DOI":"10.1056\/NEJMoa050518","volume":"353","author":"DA Berry","year":"2005","unstructured":"Berry, D. A. et al. Effect of Screening and Adjuvant Therapy on Mortality from Breast Cancer. N. Engl. J. Med. 353, 1784\u20131792 (2005).","journal-title":"N. Engl. J. Med."},{"key":"943_CR3","unstructured":"Screening Mammography | Health First Breast Center. https:\/\/hf.org\/breasthealth\/digitalmammo.cfm."},{"key":"943_CR4","doi-asserted-by":"publisher","first-page":"20190580","DOI":"10.1259\/bjr.20190580","volume":"93","author":"H-P Chan","year":"2020","unstructured":"Chan, H.-P., Samala, R. K. & Hadjiiski, L. M. CAD and AI for breast cancer\u2014recent development and challenges. Br. J. Radiol. 93, 20190580 (2020).","journal-title":"Br. J. Radiol."},{"key":"943_CR5","doi-asserted-by":"publisher","first-page":"150360","DOI":"10.1109\/ACCESS.2020.3016715","volume":"8","author":"N Fatima","year":"2020","unstructured":"Fatima, N., Liu, L., Hong, S. & Ahmed, H. Prediction of Breast Cancer, Comparative Review of Machine Learning Techniques, and Their Analysis. IEEE Access 8, 150360\u2013150376 (2020).","journal-title":"IEEE Access"},{"key":"943_CR6","doi-asserted-by":"publisher","first-page":"4165","DOI":"10.1038\/s41598-018-22437-z","volume":"8","author":"D Ribli","year":"2018","unstructured":"Ribli, D., Horv\u00e1th, A., Unger, Z., Pollner, P. & Csabai, I. Detecting and classifying lesions in mammograms with Deep Learning. Sci. Rep. 8, 4165 (2018).","journal-title":"Sci. Rep."},{"key":"943_CR7","doi-asserted-by":"publisher","first-page":"12495","DOI":"10.1038\/s41598-019-48995-4","volume":"9","author":"L Shen","year":"2019","unstructured":"Shen, L. et al. Deep Learning to Improve Breast Cancer Detection on Screening Mammography. Sci. Rep. 9, 12495 (2019).","journal-title":"Sci. Rep."},{"key":"943_CR8","doi-asserted-by":"publisher","first-page":"89","DOI":"10.1038\/s41586-019-1799-6","volume":"577","author":"SM McKinney","year":"2020","unstructured":"McKinney, S. M. et al. International evaluation of an AI system for breast cancer screening. Nature 577, 89\u201394 (2020).","journal-title":"Nature"},{"key":"943_CR9","doi-asserted-by":"publisher","first-page":"244","DOI":"10.1038\/s41591-020-01174-9","volume":"27","author":"W Lotter","year":"2021","unstructured":"Lotter, W. et al. Robust breast cancer detection in mammography and digital breast tomosynthesis using an annotation-efficient deep learning approach. Nat. Med. 27, 244\u2013249 (2021).","journal-title":"Nat. Med."},{"key":"943_CR10","doi-asserted-by":"publisher","first-page":"236","DOI":"10.1016\/j.acra.2011.09.014","volume":"19","author":"IC Moreira","year":"2012","unstructured":"Moreira, I. C. et al. INbreast: toward a full-field digital mammographic database. Acad. Radiol. 19, 236\u2013248 (2012).","journal-title":"Acad. Radiol."},{"key":"943_CR11","doi-asserted-by":"publisher","DOI":"10.1038\/sdata.2017.177","volume":"4","author":"RS Lee","year":"2017","unstructured":"Lee, R. S. et al. A curated mammography data set for use in computer-aided detection and diagnosis research. Sci. Data 4, 170177 (2017).","journal-title":"Sci. Data"},{"key":"943_CR12","doi-asserted-by":"publisher","unstructured":"Cui, C. et al. The Chinese Mammography Database (CMMD): An online mammography database with biopsy confirmed types for machine diagnosis of breast. (2021).\u00a0https:\/\/doi.org\/10.7937\/TCIA.EQDE-4B16.","DOI":"10.7937\/TCIA.EQDE-4B16"},{"key":"943_CR13","doi-asserted-by":"publisher","DOI":"10.1016\/j.media.2020.101908","volume":"68","author":"Y Shen","year":"2021","unstructured":"Shen, Y. et al. An interpretable classifier for high-resolution breast cancer screening images utilizing weakly supervised localization. Med. Image Anal. 68, 101908 (2021).","journal-title":"Med. Image Anal."},{"key":"943_CR14","doi-asserted-by":"crossref","unstructured":"Yala, A. et al. Toward robust mammography-based models for breast cancer risk. Sci. Transl. Med. 13, eaba4373 (2021).","DOI":"10.1126\/scitranslmed.aba4373"},{"key":"943_CR15","doi-asserted-by":"publisher","first-page":"1184","DOI":"10.1109\/TMI.2019.2945514","volume":"39","author":"N Wu","year":"2020","unstructured":"Wu, N. et al. Deep Neural Networks Improve Radiologists\u2019 Performance in Breast Cancer Screening. IEEE Trans. Med. Imaging 39, 1184\u20131194 (2020).","journal-title":"IEEE Trans. Med. Imaging"},{"key":"943_CR16","unstructured":"Liu, K. et al. Weakly-supervised High-resolution Segmentation of Mammography Images for Breast Cancer Diagnosis. ArXiv210607049 Cs (2021)."},{"key":"943_CR17","doi-asserted-by":"publisher","DOI":"10.1148\/ryai.2020200103","volume":"3","author":"MD Halling-Brown","year":"2021","unstructured":"Halling-Brown, M. D. et al. OPTIMAM Mammography Image Database: A Large-Scale Resource of Mammography Images and Clinical Data. Radiol. Artif. Intell. 3, e200103 (2021).","journal-title":"Radiol. Artif. Intell."},{"key":"943_CR18","unstructured":"Wu, N. et al. The NYU Breast Cancer Screening Dataset v1.0. 9."},{"key":"943_CR19","unstructured":"Stadnick, B. et al. Meta-repository of screening mammography classifiers. Preprint at http:\/\/arxiv.org\/abs\/2108.04800 (2022)."},{"key":"943_CR20","doi-asserted-by":"publisher","first-page":"e138","DOI":"10.1016\/S2589-7500(20)30003-0","volume":"2","author":"H-E Kim","year":"2020","unstructured":"Kim, H.-E. et al. Changes in cancer detection and false-positive recall in mammography using artificial intelligence: a retrospective, multireader study. Lancet Digit. Health 2, e138\u2013e148 (2020).","journal-title":"Lancet Digit. Health"},{"key":"943_CR21","unstructured":"Pedemonte, S. et al. A deep learning algorithm for reducing false positives in screening mammography. Preprint at http:\/\/arxiv.org\/abs\/2204.06671 (2022)."},{"key":"943_CR22","unstructured":"Bassett, L. W., Conner, K. & Ms, I. The Abnormal Mammogram. in Holland-Frei Cancer Medicine. 6th edition (BC Decker, 2003)."},{"key":"943_CR23","doi-asserted-by":"publisher","first-page":"347","DOI":"10.1148\/radiol.2512081235","volume":"251","author":"S Vinnicombe","year":"2009","unstructured":"Vinnicombe, S. et al. Full-Field Digital versus Screen-Film Mammography: Comparison within the UK Breast Screening Program and Systematic Review of Published Data. Radiology 251, 347\u2013358 (2009).","journal-title":"Radiology"},{"key":"943_CR24","doi-asserted-by":"publisher","first-page":"214","DOI":"10.1016\/j.semcancer.2020.06.002","volume":"72","author":"I Sechopoulos","year":"2021","unstructured":"Sechopoulos, I., Teuwen, J. & Mann, R. Artificial intelligence for breast cancer detection in mammography and digital breast tomosynthesis: State of the art. Semin. Cancer Biol. 72, 214\u2013225 (2021).","journal-title":"Semin. Cancer Biol."},{"key":"943_CR25","doi-asserted-by":"publisher","first-page":"3654","DOI":"10.1002\/mp.15598","volume":"49","author":"J Bai","year":"2022","unstructured":"Bai, J., Jin, A., Wang, T., Yang, C. & Nabavi, S. Feature fusion Siamese network for breast cancer detection comparing current and prior mammograms. Med. Phys. 49, 3654\u20133669 (2022).","journal-title":"Med. Phys."},{"key":"943_CR26","doi-asserted-by":"publisher","DOI":"10.1117\/1.JMI.4.4.044501","volume":"4","author":"T Kooi","year":"2017","unstructured":"Kooi, T. & Karssemeijer, N. Classifying symmetrical differences and temporal change for the detection of malignant masses in mammography using deep neural networks. J. Med. Imaging 4, 044501 (2017).","journal-title":"J. Med. Imaging"},{"key":"943_CR27","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1148\/radiol.2019182716","volume":"292","author":"A Yala","year":"2019","unstructured":"Yala, A., Lehman, C., Schuster, T., Portnoi, T. & Barzilay, R. A Deep Learning Mammography-based Model for Improved Breast Cancer Risk Prediction. Radiology 292, 60\u201366 (2019).","journal-title":"Radiology"},{"key":"943_CR28","doi-asserted-by":"publisher","first-page":"303","DOI":"10.1016\/j.media.2016.07.007","volume":"35","author":"T Kooi","year":"2017","unstructured":"Kooi, T. et al. Large scale deep learning for computer aided detection of mammographic lesions. Med. Image Anal. 35, 303\u2013312 (2017).","journal-title":"Med. Image Anal."},{"key":"943_CR29","doi-asserted-by":"publisher","unstructured":"Huang, G., Liu, Z., van der Maaten, L. & Weinberger, K. Q. Densely Connected Convolutional Networks. Preprint at https:\/\/doi.org\/10.48550\/arXiv.1608.06993 (2018).","DOI":"10.48550\/arXiv.1608.06993"},{"key":"943_CR30","unstructured":"CBIS-DDSM - The Cancer Imaging Archive (TCIA) Public Access - Cancer Imaging Archive Wiki. https:\/\/wiki.cancerimagingarchive.net\/display\/Public\/CBIS-DDSM."},{"key":"943_CR31","unstructured":"INbreast Dataset | Kaggle. https:\/\/www.kaggle.com\/ramanathansp20\/inbreast-dataset."},{"key":"943_CR32","doi-asserted-by":"publisher","first-page":"1045","DOI":"10.1007\/s10278-013-9622-7","volume":"26","author":"K Clark","year":"2013","unstructured":"Clark, K. et al. The Cancer Imaging Archive (TCIA): Maintaining and Operating a Public Information Repository. J. Digit. Imaging 26, 1045\u20131057 (2013).","journal-title":"J. Digit. Imaging"},{"key":"943_CR33","unstructured":"Huff, T., Mahabadi, N. & Tadi, P. Neuroanatomy, Visual Cortex. in StatPearls (StatPearls Publishing, 2022)."}],"container-title":["Journal of Imaging Informatics in Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10278-023-00943-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10278-023-00943-5\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10278-023-00943-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,4,19]],"date-time":"2024-04-19T15:04:59Z","timestamp":1713539099000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10278-023-00943-5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,1,10]]},"references-count":33,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2024,4]]}},"alternative-id":["943"],"URL":"https:\/\/doi.org\/10.1007\/s10278-023-00943-5","relation":{},"ISSN":["2948-2933"],"issn-type":[{"value":"2948-2933","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,1,10]]},"assertion":[{"value":"27 July 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"4 October 2023","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"5 October 2023","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 January 2024","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing Interests"}}]}}