{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,18]],"date-time":"2026-11-18T18:02:20Z","timestamp":1795024940144,"version":"4.10.2"},"reference-count":29,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:00:00Z","timestamp":1750291200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"MARBLE"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Diagnostics"],"abstract":"<jats:p>Background: The integration of artificial intelligence (AI) in digital mammography (DM) screening holds promise for early breast cancer detection, potentially enhancing accuracy and efficiency. However, AI performance is not identical to that of human observers. We aimed to identify common morphological image characteristics of true cancers that are missed by either AI or human screening when their interpretations are discrepant. Methods: Twenty-six breast cancer-positive cases, identified from a large retrospective multi-institutional digital mammography dataset based on discrepant AI and human interpretations, were included in a reader study. Ground truth was confirmed by histopathology or \u22651-year follow-up. Fourteen radiologists assessed lesion visibility, morphological features, and likelihood of malignancy. AI performance was evaluated using receiver operating characteristic (ROC) analysis and area under the curve (AUC). The reader study results were analyzed using interobserver agreement measures and descriptive statistics. Results: AI demonstrated high discriminative capability in the full dataset, with AUCs ranging from 0.903 (95% CI: 0.862\u20130.944) to 0.946 (95% CI: 0.896\u20130.996). Cancers missed by AI had a significantly smaller median size (9.0 mm, IQR 6.5\u201312.0) compared to those missed by human readers (21.0 mm, IQR 10.5\u201341.0) (p = 0.0014). Cancers in discrepant cases were often described as having \u2018low visibility\u2019, \u2018indistinct margins\u2019, or \u2018irregular shape\u2019. Calcifications were observed in 27% of human-missed cancers (42\/154) versus 18% of AI-missed cancers (38\/210). A very high likelihood of malignancy was assigned in 32.5% (50\/154) of human-missed cancers compared to 19.5% (41\/210) of AI-missed cancers. Overall inter-rater agreement was poor to fair (&lt;0.40), indicating interpretation challenges of the selected images. Among the human-missed cancers, calcifications were more frequent (42\/154; 27%) than among the AI-missed cancers (38\/210; 18%) (p = 0.396). Furthermore, 50\/154 (32.5%) human-missed cancers were deemed to have a very high likelihood of malignancy, compared to 41\/210 (19.5%) AI-missed cancers (p = 0.8). Overall inter-rater agreement on the items assessed during the reader study was poor to fair (&lt;0.40), suggesting that interpretation of the selected images was challenging. Conclusions: Lesions missed by AI were smaller and less often calcified than cancers missed by human readers. Cancers missed by AI tended to show lower levels of suspicion than those missed by human readers. While definitive conclusions are premature, the findings highlight the complementary roles of AI and human readers in mammographic interpretation.<\/jats:p>","DOI":"10.3390\/diagnostics15121566","type":"journal-article","created":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T08:43:58Z","timestamp":1750322638000},"page":"1566","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["An Exploration of Discrepant Recalls Between AI and Human Readers of Malignant Lesions in Digital Mammography Screening"],"prefix":"10.3390","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7273-4386","authenticated-orcid":false,"given":"Suzanne L.","family":"van Winkel","sequence":"first","affiliation":[{"name":"Radboudumc\u2014Radboud University Nijmegen Medical Centre, Medical Imaging Department, Geert Grooteplein Zuid 10, 6525 GA Nijmegen, The Netherlands"},{"name":"Amsterdam UMC, Location VUmc (Vrije Universiteit Medical Center), Faculty of Medicine, Department of Ethics, Law & Humanities, De Boelelaan 1089a, 1081 HV Amsterdam, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9615-8205","authenticated-orcid":false,"given":"Ioannis","family":"Sechopoulos","sequence":"additional","affiliation":[{"name":"Radboudumc\u2014Radboud University Nijmegen Medical Centre, Medical Imaging Department, Geert Grooteplein Zuid 10, 6525 GA Nijmegen, The Netherlands"},{"name":"LRCB\u2014Dutch Expert Centre for Screening, Wijchenseweg 101, 6538 SW Nijmegen, The Netherlands"},{"name":"Technical Medicine Centre, University of Twente, Hallenweg 5, 7522 NH Enschede, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alejandro","family":"Rodr\u00edguez-Ruiz","sequence":"additional","affiliation":[{"name":"ScreenPoint Medical BV, Toernooiveld 300, 6525 EC Nijmegen, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wouter J. H.","family":"Veldkamp","sequence":"additional","affiliation":[{"name":"Leiden University Medical Center, Department of Radiology, Albinusdreef 2, 2333 ZA Leiden, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2444-1778","authenticated-orcid":false,"given":"Gisella","family":"Gennaro","sequence":"additional","affiliation":[{"name":"Veneto Institute of Oncology (IOV), Department of Imaging and Radiotherapy, Unit of Breast Radiology, IRCCS, via Gattamelata 64, 35128 Padua, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3848-9783","authenticated-orcid":false,"given":"Margarita","family":"Chevalier","sequence":"additional","affiliation":[{"name":"Department of Radiology and Medical Physics, Universidad Complutense de Madrid, 28040 Madrid, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3169-778X","authenticated-orcid":false,"given":"Thomas H.","family":"Helbich","sequence":"additional","affiliation":[{"name":"Department of Biomedical Imaging and Image-Guided Therapy, Division of General and Pediatric Radiology, Medical University of Vienna and General Hospital, Waehringer Guertel 18-20, A-1090 Vienna, Austria"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tianyu","family":"Zhang","sequence":"additional","affiliation":[{"name":"Radboudumc\u2014Radboud University Nijmegen Medical Centre, Medical Imaging Department, Geert Grooteplein Zuid 10, 6525 GA Nijmegen, The Netherlands"},{"name":"Netherlands Cancer Institute, Antoni van Leeuwenhoek (NKI), Department of Radiology, Plesmanlaan 121, 1066 CX Amsterdam, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Matthew G.","family":"Wallis","sequence":"additional","affiliation":[{"name":"Cambridge Breast Unit and NIHR Cambridge Biomedical Research Centre, Addenbrooke\u2019s Hospital, Cambridge CB2 2QQ, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ritse M.","family":"Mann","sequence":"additional","affiliation":[{"name":"Radboudumc\u2014Radboud University Nijmegen Medical Centre, Medical Imaging Department, Geert Grooteplein Zuid 10, 6525 GA Nijmegen, The Netherlands"},{"name":"Netherlands Cancer Institute, Antoni van Leeuwenhoek (NKI), Department of Radiology, Plesmanlaan 121, 1066 CX Amsterdam, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2025,6,19]]},"reference":[{"key":"ref_1","first-page":"320","article-title":"Breast cancer diagnosis and death in The Netherlands: A changing burden","volume":"25","author":"Verbeek","year":"2014","journal-title":"Eur. J. Public Health"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1387","DOI":"10.7150\/ijbs.21635","article-title":"Risk factors and preventions of breast cancer","volume":"13","author":"Sun","year":"2017","journal-title":"Int. J. Biol. Sci."},{"key":"ref_3","first-page":"404","article-title":"Cancer screening: Evidence and practice in Europe 2008","volume":"44","author":"Hakema","year":"2008","journal-title":"Eur. J. Cancer"},{"key":"ref_4","unstructured":"Weerakkody, Y., Manning, T., and Lemos, P. (2025, June 11). Breast Imaging-Reporting and Data System (BI-RADS). Radiopaedia. Available online: https:\/\/radiopaedia.org\/articles\/breast-imaging-reporting-and-data-system-bi-rads?lang=us."},{"key":"ref_5","unstructured":"American College of Radiology (2020). ACR BI-RADS\u00ae Atlas: Breast Imaging Reporting and Data System, American College of Radiology. [5th ed.]."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"E19","DOI":"10.1038\/s41586-020-2679-9","article-title":"International evaluation of an AI system for breast cancer screening","volume":"577","author":"McKinney","year":"2020","journal-title":"Nature"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"916","DOI":"10.1093\/jnci\/djy222","article-title":"Stand-Alone Artificial Intelligence for Breast Cancer Detection in Mammography: Comparison with 101 Radiologists","volume":"111","author":"Broeders","year":"2019","journal-title":"J. Natl. Cancer Inst."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1184","DOI":"10.1109\/TMI.2019.2945514","article-title":"Deep neural networks improve radiologists\u2019 performance in breast cancer screening","volume":"39","author":"Wu","year":"2019","journal-title":"IEEE Trans. Med. Imaging."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"254","DOI":"10.1016\/j.breast.2019.12.014","article-title":"Stand-alone artificial intelligence\u2014The future of breast cancer screening?","volume":"49","author":"Sechopoulos","year":"2020","journal-title":"Breast"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"P936","DOI":"10.1016\/S1470-2045(23)00298-X","article-title":"Artificial intelligence-supported screen reading versus standard double reading in the Mammography Screening with Artificial Intelligence trial (MASAI)","volume":"24","author":"Joseffson","year":"2023","journal-title":"Lancet Oncol."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"4825","DOI":"10.1007\/s00330-019-06186-9","article-title":"Can we reduce the workload of mammographic screening by automatic identification of normal exams with artificial intelligence? A feasibility study","volume":"29","author":"Teuwen","year":"2019","journal-title":"Eur. Radiol."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"38","DOI":"10.1148\/radiol.2019182908","article-title":"A deep learning model to triage screening mammograms: A simulation study","volume":"1","author":"Yala","year":"2019","journal-title":"Radiology"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"e468","DOI":"10.1016\/S2589-7500(20)30185-0","article-title":"Effect of artificial intelligence-based triaging of breast cancer screening mammograms on cancer detection and radiologist workload: A retrospective simulation study","volume":"2","author":"Dembrower","year":"2020","journal-title":"Lancet Digit. Health"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"305","DOI":"10.1148\/radiol.2018181371","article-title":"Detection of Breast Cancer with Mammography: Effect of an Artificial Intelligence Support System","volume":"290","author":"Krupinski","year":"2019","journal-title":"Radiology"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"8682","DOI":"10.1007\/s00330-021-07992-w","article-title":"Impact of artificial intelligence support on accuracy and reading time in breast tomosynthesis image interpretation: A multi-reader multi-case study","volume":"31","author":"Appelman","year":"2021","journal-title":"Eur. Radiol."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"214","DOI":"10.1016\/j.semcancer.2020.06.002","article-title":"Artificial intelligence for breast cancer detection in mammography and digital breast tomosynthesis: State of the art","volume":"72","author":"Sechopoulos","year":"2021","journal-title":"Semin. Cancer Biol."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"93","DOI":"10.1007\/s00330-012-2562-7","article-title":"Stand-alone computer-aided detection compared to radiologists\u2019 performance for the detection of mamographic masses","volume":"23","author":"Hupse","year":"2013","journal-title":"Eur. Radiol."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"664","DOI":"10.1007\/s00330-012-2649-1","article-title":"Performance comparison of single view digital breast thomosynthesis plus single view digital mammography with tweo-view digital mammography","volume":"23","author":"Gennaro","year":"2013","journal-title":"Eur. Radiol."},{"key":"ref_19","first-page":"788","article-title":"Two-view and single-view tomosynthesis versus full-filed digital mammography: High- resolotion x-ray imaging observer study","volume":"262","author":"Wallis","year":"2012","journal-title":"Eur. Radiol."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"908","DOI":"10.1007\/s00330-011-2320-2","article-title":"Increase in perceived case suspiciousness due to local contrast optimisation in digital screening mammography","volume":"22","author":"Visser","year":"2012","journal-title":"Eur. Radiol."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"565","DOI":"10.1007\/s00330-017-4991-9","article-title":"Diagnostic value of the stand-alone synthetic image in digital breast tomosynthesis examinations","volume":"28","author":"Garayoa","year":"2018","journal-title":"Eur. Radiol."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1938","DOI":"10.1007\/s00330-017-5167-3","article-title":"One-view digital breast tomosynthesis as a atand-alone modality for breast cancer detection: Do we need more?","volume":"28","author":"Lardenoije","year":"2018","journal-title":"Eur. Radiol."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"83","DOI":"10.1097\/RLI.0000000000000513","article-title":"Synthetic 2d mammography can replace digital mammography as an adjuct to wide-angle digital breast tomosynthesis","volume":"54","author":"Clauser","year":"2019","journal-title":"Investig. Radiol."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"614","DOI":"10.1093\/annonc\/mdm481","article-title":"European Guidelines for quality assurance in breast cancer screening and diagnosis","volume":"19","author":"Perry","year":"2008","journal-title":"Ann. Oncol."},{"key":"ref_25","unstructured":"Nikitin, V., Filatov, A., Bagotskaya, N., Kil, I., Lossev, I., and Losseva, N. (2014, January 6\u201310). Improvement in ROC curves of readers with next generation of mammography CAD. Proceedings of the European Congress of Radiology (ECR) 2014, Vienna, Austria."},{"key":"ref_26","first-page":"20220018","article-title":"Strengths and challenges of the artificial intelligence in the assessment of dense breasts","volume":"4","author":"Mansour","year":"2022","journal-title":"BJR Open"},{"key":"ref_27","unstructured":"Meakin, J., Gerke, P.K., Kerkstra, S., Groeneveld, M., van Leeuwen, K., Mickan, A., Overkamp, M., van Run, C., van Zeeland, H., and van Ginneken, B. (Grand-Challenge.org [Software], 2023). Grand-Challenge.org [Software], Version v2023.05."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"159","DOI":"10.2307\/2529310","article-title":"The measurement of observer agreement for categorical data","volume":"33","author":"Landis","year":"1977","journal-title":"Biometrics"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"941","DOI":"10.1148\/rg.2020190090","article-title":"Missed breast cancer: Effects of subconscious bias and lesion characteristics","volume":"40","author":"Lamb","year":"2020","journal-title":"Radiographics"}],"container-title":["Diagnostics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2075-4418\/15\/12\/1566\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T17:55:03Z","timestamp":1760032503000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2075-4418\/15\/12\/1566"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,19]]},"references-count":29,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2025,6]]}},"alternative-id":["diagnostics15121566"],"URL":"https:\/\/doi.org\/10.3390\/diagnostics15121566","relation":{},"ISSN":["2075-4418"],"issn-type":[{"value":"2075-4418","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,19]]}}}