{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,16]],"date-time":"2026-10-16T15:36:01Z","timestamp":1792164961958,"version":"4.4.0"},"reference-count":25,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2024,6,9]],"date-time":"2024-06-09T00:00:00Z","timestamp":1717891200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,6,9]],"date-time":"2024-06-09T00:00:00Z","timestamp":1717891200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"The Daffodil Centre","award":["Postgraduate Research Scholarship"],"award-info":[{"award-number":["Postgraduate Research Scholarship"]}]},{"DOI":"10.13039\/501100001026","name":"National Breast Cancer Foundation","doi-asserted-by":"publisher","award":["EC-21-001"],"award-info":[{"award-number":["EC-21-001"]}],"id":[{"id":"10.13039\/501100001026","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001026","name":"National Breast Cancer Foundation","doi-asserted-by":"publisher","award":["2023\/IIRS0028"],"award-info":[{"award-number":["2023\/IIRS0028"]}],"id":[{"id":"10.13039\/501100001026","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000925","name":"National Health and Medical Research Council","doi-asserted-by":"publisher","award":["1194410"],"award-info":[{"award-number":["1194410"]}],"id":[{"id":"10.13039\/501100000925","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000925","name":"National Health and Medical Research Council","doi-asserted-by":"publisher","award":["1194108"],"award-info":[{"award-number":["1194108"]}],"id":[{"id":"10.13039\/501100000925","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001774","name":"University of Sydney","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100001774","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Breast Cancer Res Treat"],"published-print":{"date-parts":[[2024,8]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec>\n                <jats:title>Purpose<\/jats:title>\n                <jats:p>Artificial intelligence (AI) for reading breast screening mammograms could potentially replace (some) human-reading and improve screening effectiveness. This systematic review aims to identify and quantify the types of AI errors to better understand the consequences of implementing this technology.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Methods<\/jats:title>\n                <jats:p>Electronic databases were searched for external validation studies of the accuracy of AI algorithms in real-world screening mammograms. Descriptive synthesis was performed on error types and frequency. False negative proportions (FNP) and false positive proportions (FPP) were pooled within AI positivity thresholds using random-effects meta-analysis.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Results<\/jats:title>\n                <jats:p>Seven retrospective studies (447,676 examinations; published 2019\u20132022) met inclusion criteria. Five studies reported AI error as false negatives or false positives. Pooled FPP decreased incrementally with increasing positivity threshold (71.83% [95% CI 69.67, 73.90] at Transpara 3 to 10.77% [95% CI 8.34, 13.79] at Transpara 9). Pooled FNP increased incrementally from 0.02% [95% CI 0.01, 0.03] (Transpara 3) to 0.12% [95% CI 0.06, 0.26] (Transpara 9), consistent with a trade-off with FPP. Heterogeneity within thresholds reflected algorithm version and completeness of the reference standard. Other forms of AI error were reported rarely (location error and technical error in one study each).<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Conclusion<\/jats:title>\n                <jats:p>AI errors are largely interpreted in the framework of test accuracy. FP and FN errors show expected variability not only by positivity threshold, but also by algorithm version and study quality. Reporting of other forms of AI errors is sparse, despite their potential implications for adoption of the technology. Considering broader types of AI error would add nuance to reporting that can inform inferences about AI\u2019s utility.<\/jats:p>\n              <\/jats:sec>","DOI":"10.1007\/s10549-024-07353-3","type":"journal-article","created":{"date-parts":[[2024,6,9]],"date-time":"2024-06-09T16:01:23Z","timestamp":1717948883000},"page":"1-13","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":27,"title":["Frequency and characteristics of errors by artificial intelligence (AI) in reading screening mammography: a systematic review"],"prefix":"10.1007","volume":"207","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7528-8630","authenticated-orcid":false,"given":"Aileen","family":"Zeng","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3641-952X","authenticated-orcid":false,"given":"Nehmat","family":"Houssami","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5662-1496","authenticated-orcid":false,"given":"Naomi","family":"Noguchi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8100-4278","authenticated-orcid":false,"given":"Brooke","family":"Nickel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3801-8180","authenticated-orcid":false,"given":"M. Luke","family":"Marinovich","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,6,9]]},"reference":[{"issue":"1","key":"7353_CR1","doi-asserted-by":"publisher","first-page":"e054005","DOI":"10.1136\/bmjopen-2021-054005","volume":"12","author":"ML Marinovich","year":"2022","unstructured":"Marinovich ML, Wylie E, Lotter W, Pearce A, Carter SM, Lund H et al (2022) Artificial intelligence (AI) to enhance breast cancer screening: protocol for population-based cohort study of cancer detection. BMJ Open 12(1):e054005. https:\/\/doi.org\/10.1136\/bmjopen-2021-054005","journal-title":"BMJ Open"},{"key":"7353_CR2","doi-asserted-by":"publisher","first-page":"104498","DOI":"10.1016\/j.ebiom.2023.104498","volume":"90","author":"ML Marinovich","year":"2023","unstructured":"Marinovich ML, Wylie E, Lotter W, Lund H, Waddell A, Madeley C et al (2023) Artificial intelligence (AI) for breast cancer screening: BreastScreen population-based cohort study of cancer detection. EBioMedicine 90:104498. https:\/\/doi.org\/10.1016\/j.ebiom.2023.104498","journal-title":"EBioMedicine"},{"issue":"2020","key":"7353_CR3","doi-asserted-by":"publisher","first-page":"151","DOI":"10.1145\/3368555.3384468","volume":"2020","author":"L Oakden-Rayner","year":"2020","unstructured":"Oakden-Rayner L, Dunnmon J, Carneiro G, R\u00e9 C (2020) Hidden stratification causes clinically meaningful failures in machine learning for medical imaging. Proc ACM Conf Health Inference Learn 2020(2020):151\u2013159. https:\/\/doi.org\/10.1145\/3368555.3384468","journal-title":"Proc ACM Conf Health Inference Learn"},{"issue":"6433","key":"7353_CR4","doi-asserted-by":"publisher","first-page":"1287","DOI":"10.1126\/science.aaw4399","volume":"363","author":"SG Finlayson","year":"2019","unstructured":"Finlayson SG, Bowers JD, Ito J, Zittrain JL, Beam AL, Kohane IS (2019) Adversarial attacks on medical machine learning. Science 363(6433):1287\u20131289","journal-title":"Science"},{"issue":"4","key":"7353_CR5","doi-asserted-by":"publisher","first-page":"e222176","DOI":"10.1148\/radiol.222176","volume":"307","author":"T Dratsch","year":"2023","unstructured":"Dratsch T, Chen X, Rezazade Mehrizi M, Kloeckner R, M\u00e4hringer-Kunz A, P\u00fcsken M et al (2023) Automation bias in mammography: the impact of artificial intelligence BI-RADS suggestions on reader performance. Radiology 307(4):e222176. https:\/\/doi.org\/10.1148\/radiol.222176","journal-title":"Radiology"},{"issue":"21","key":"7353_CR6","doi-asserted-by":"publisher","first-page":"259","DOI":"10.1016\/j.jacr.2021.11.008","volume":"19","author":"AW Anderson","year":"2022","unstructured":"Anderson AW, Marinovich ML, Houssami N, Lowry KP, Elmore JG, Buist DSM et al (2022) Independent external validation of artificial intelligence algorithms for automated interpretation of screening mammography: a systematic review. J Am College Radiol 19(21):259\u201373. https:\/\/doi.org\/10.1016\/j.jacr.2021.11.008","journal-title":"J Am College Radiol"},{"issue":"5","key":"7353_CR7","doi-asserted-by":"publisher","first-page":"351","DOI":"10.1080\/17434440.2019.1610387","volume":"16","author":"N Houssami","year":"2019","unstructured":"Houssami N, Kirkpatrick-Jones G, Noguchi N, Lee CI (2019) Artificial Intelligence (AI) for the early detection of breast cancer: a scoping review to assess AI\u2019s potential in breast screening practice. Expert Rev Med Devices 16(5):351\u2013362. https:\/\/doi.org\/10.1080\/17434440.2019.1610387","journal-title":"Expert Rev Med Devices"},{"key":"7353_CR8","doi-asserted-by":"publisher","first-page":"n1872","DOI":"10.1136\/bmj.n1872","volume":"374","author":"K Freeman","year":"2021","unstructured":"Freeman K, Geppert J, Stinton C, Todkill D, Johnson S, Clarke A et al (2021) Use of artificial intelligence for image analysis in breast cancer screening programmes: systematic review of test accuracy. BMJ 374:n1872. https:\/\/doi.org\/10.1136\/bmj.n1872","journal-title":"BMJ"},{"issue":"4","key":"7353_CR9","doi-asserted-by":"publisher","first-page":"388","DOI":"10.1001\/jama.2017.19163","volume":"319","author":"MD McInnes","year":"2018","unstructured":"McInnes MD, Moher D, Thombs BD, McGrath TA, Bossuyt PM, Clifford T et al (2018) Preferred reporting items for a systematic review and meta-analysis of diagnostic test accuracy studies: the PRISMA-DTA statement. JAMA 319(4):388\u2013396. https:\/\/doi.org\/10.1001\/jama.2017.19163","journal-title":"JAMA"},{"key":"7353_CR10","doi-asserted-by":"publisher","first-page":"111","DOI":"10.1016\/j.compchemeng.2017.10.008","volume":"114","author":"JH Lee","year":"2018","unstructured":"Lee JH, Shin J, Realff MJ (2018) Machine learning: overview of the recent progresses and implications for the process systems engineering field. Comput Chem Eng 114:111\u2013121. https:\/\/doi.org\/10.1016\/j.compchemeng.2017.10.008","journal-title":"Comput Chem Eng"},{"issue":"10","key":"7353_CR11","doi-asserted-by":"publisher","first-page":"1663","DOI":"10.1038\/s41591-021-01517-0","volume":"27","author":"V Sounderajah","year":"2021","unstructured":"Sounderajah V, Ashrafian H, Rose S, Shah NH, Ghassemi M, Golub R et al (2021) A quality assessment tool for artificial intelligence-centered diagnostic test accuracy studies: QUADAS-AI. Nat Med 27(10):1663\u20131665. https:\/\/doi.org\/10.1038\/s41591-021-01517-0","journal-title":"Nat Med"},{"issue":"3","key":"7353_CR12","doi-asserted-by":"publisher","first-page":"261","DOI":"10.3102\/10769986030003261","volume":"30","author":"W Viechtbauer","year":"2005","unstructured":"Viechtbauer W (2005) Bias and efficiency of meta-analytic variance estimators in the random-effects model. J Educ Behav Stat 30(3):261\u2013293. https:\/\/doi.org\/10.3102\/10769986030003261","journal-title":"J Educ Behav Stat"},{"key":"7353_CR13","first-page":"295","volume-title":"Analyzing effect sizes: random-effects models. The handbook of research synthesis and meta-analysis","author":"SW Raudenbush","year":"2009","unstructured":"Raudenbush SW (2009) Analyzing effect sizes: random-effects models. The handbook of research synthesis and meta-analysis, 2nd edn. Russell Sage Foundation, New York, pp 295\u2013315","edition":"2"},{"issue":"3","key":"7353_CR14","doi-asserted-by":"publisher","first-page":"1","DOI":"10.18637\/jss.v036.i03","volume":"36","author":"W Viechtbauer","year":"2010","unstructured":"Viechtbauer W (2010) Conducting meta-analyses in R with the metafor package. J Stat Softw 36(3):1\u201348. https:\/\/doi.org\/10.18637\/jss.v036.i03","journal-title":"J Stat Softw"},{"key":"7353_CR15","unstructured":"Wickham H, editor An implementation of the grammar of graphics in R: ggplot. Book of Abstracts; 2006."},{"key":"7353_CR16","doi-asserted-by":"publisher","unstructured":"Balta C, Rodriguez-Ruiz A, Mieskes C, Karssemeijer N, Heywang-K\u00f6brunner SH. Going from double to single reading for screening exams labeled as likely normal by AI: what is the impact? Proceedings of SPIE; 2020. https:\/\/doi.org\/10.1117\/12.2564179","DOI":"10.1117\/12.2564179"},{"issue":"3","key":"7353_CR17","doi-asserted-by":"publisher","first-page":"1687","DOI":"10.1007\/s00330-020-07165-1","volume":"31","author":"K Lang","year":"2021","unstructured":"Lang K, Dustler M, Dahlblom V, Akesson A, Andersson I, Zackrisson S (2021) Identifying normal mammograms in a large screening population using artificial intelligence. Eur Radiol 31(3):1687\u20131692. https:\/\/doi.org\/10.1007\/s00330-020-07165-1","journal-title":"Eur Radiol"},{"key":"7353_CR18","doi-asserted-by":"publisher","first-page":"212381","DOI":"10.1148\/radiol.212381","volume":"303","author":"M Larsen","year":"2022","unstructured":"Larsen M, Aglen CF, Lee CI, Hoff SR, Lund-Hanssen H, Lang K et al (2022) Artificial intelligence evaluation of 122969 mammography examinations from a population-based screening program. Radiology 303:212381. https:\/\/doi.org\/10.1148\/radiol.212381","journal-title":"Radiology"},{"key":"7353_CR19","doi-asserted-by":"publisher","first-page":"210948","DOI":"10.1148\/radiol.210948","volume":"304","author":"AD Lauritzen","year":"2022","unstructured":"Lauritzen AD, Rodriguez-Ruiz A, von Euler-Chelpin MC, Lynge E, Vejborg I, Nielsen M et al (2022) An artificial intelligence-based mammography screening protocol for breast cancer: outcome and radiologist workload. Radiology 304:210948. https:\/\/doi.org\/10.1148\/radiol.210948","journal-title":"Radiology"},{"issue":"4","key":"7353_CR20","doi-asserted-by":"publisher","first-page":"618","DOI":"10.1007\/s10278-018-0168-6","volume":"32","author":"RC Mayo","year":"2019","unstructured":"Mayo RC, Kent D, Sen LC, Kapoor M, Leung JWT, Watanabe AT (2019) Reduction of false-positive markings on mammograms: a retrospective comparison study using an artificial intelligence-based CAD. J Digit Imaging 32(4):618\u2013624. https:\/\/doi.org\/10.1007\/s10278-018-0168-6","journal-title":"J Digit Imaging"},{"issue":"3","key":"7353_CR21","doi-asserted-by":"publisher","first-page":"e200265","DOI":"10.1001\/jamanetworkopen.2020.0265","volume":"3","author":"T Schaffter","year":"2020","unstructured":"Schaffter T, Buist DSM, Lee CI, Nikulin Y, Ribli D, Guan Y et al (2020) Evaluation of combined artificial intelligence and radiologist assessment to interpret screening mammograms. JAMA Netw 3(3):e200265. https:\/\/doi.org\/10.1001\/jamanetworkopen.2020.0265","journal-title":"JAMA Netw"},{"issue":"1","key":"7353_CR22","doi-asserted-by":"publisher","first-page":"57","DOI":"10.1148\/radiol.2021203555","volume":"300","author":"JL Raya-Povedano","year":"2021","unstructured":"Raya-Povedano JL, Romero-Martin S, Elias-Cabot E, Gubern-Merida A, Rodriguez-Ruiz A, Alvarez-Benito M (2021) AI-based strategies to reduce workload in breast cancer screening with mammography and tomosynthesis: a retrospective evaluation. Radiology 300(1):57\u201365. https:\/\/doi.org\/10.1148\/radiol.2021203555","journal-title":"Radiology"},{"issue":"12","key":"7353_CR23","doi-asserted-by":"publisher","first-page":"8238","DOI":"10.1007\/s00330-022-08909-x","volume":"32","author":"M Larsen","year":"2022","unstructured":"Larsen M, Aglen CF, Hoff SR, Lund-Hanssen H, Hofvind S (2022) Possible strategies for use of artificial intelligence in screen-reading of mammograms, based on retrospective data from 122,969 screening examinations. Eur Radiol 32(12):8238\u20138246. https:\/\/doi.org\/10.1007\/s00330-022-08909-x","journal-title":"Eur Radiol"},{"key":"7353_CR24","doi-asserted-by":"publisher","first-page":"146","DOI":"10.1016\/j.breast.2019.09.005","volume":"52","author":"CI Lee","year":"2020","unstructured":"Lee CI, Houssami N, Elmore JG, Buist DSM (2020) Pathways to breast cancer screening artificial intelligence algorithm validation. Breast 52:146\u2013149","journal-title":"Breast"},{"issue":"1","key":"7353_CR25","doi-asserted-by":"publisher","first-page":"e100293","DOI":"10.1136\/bmjhci-2020-100293","volume":"28","author":"N Lennox-Chhugani","year":"2021","unstructured":"Lennox-Chhugani N, Chen Y, Pearson V, Trzcinski B, James J (2021) Women\u2019s attitudes to the use of AI image readers: a case study from a national breast screening programme. BMJ Health Care Inform 28(1):e100293. https:\/\/doi.org\/10.1136\/bmjhci-2020-100293","journal-title":"BMJ Health Care Inform"}],"container-title":["Breast Cancer Research and Treatment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10549-024-07353-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10549-024-07353-3\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10549-024-07353-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,7,8]],"date-time":"2024-07-08T06:08:16Z","timestamp":1720418896000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10549-024-07353-3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6,9]]},"references-count":25,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,8]]}},"alternative-id":["7353"],"URL":"https:\/\/doi.org\/10.1007\/s10549-024-07353-3","relation":{},"ISSN":["0167-6806","1573-7217"],"issn-type":[{"value":"0167-6806","type":"print"},{"value":"1573-7217","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,6,9]]},"assertion":[{"value":"15 December 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"24 April 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 June 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors have no relevant financial or non-financial interests to disclose.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}},{"value":"This a systematic review and does not require ethics approval.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval"}},{"value":"Not applicable.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent to participate"}},{"value":"Not applicable.","order":5,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent to publish"}}]}}