{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,17]],"date-time":"2026-08-17T15:19:42Z","timestamp":1786979982615,"version":"build-2736575974"},"reference-count":24,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2021,8,12]],"date-time":"2021-08-12T00:00:00Z","timestamp":1628726400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,8,12]],"date-time":"2021-08-12T00:00:00Z","timestamp":1628726400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100013276","name":"Interreg","doi-asserted-by":"publisher","award":["122207"],"award-info":[{"award-number":["122207"]}],"id":[{"id":"10.13039\/100013276","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Eur Radiol"],"published-print":{"date-parts":[[2022,2]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objectives<\/jats:title>\n                    <jats:p>To evaluate if artificial intelligence (AI) can discriminate recalled benign from recalled malignant mammographic screening abnormalities to improve screening performance.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>A total of 2257 full-field digital mammography screening examinations, obtained 2011\u20132013, of women aged 50\u201369 years which were recalled for further assessment of 295 malignant out of 305 truly malignant lesions and 2289 benign lesions after independent double-reading with arbitration, were included in this retrospective study. A deep learning AI system was used to obtain a score (0\u201395) for each recalled lesion, representing the likelihood of breast cancer. The sensitivity on the lesion level and the proportion of women without false-positive ratings (non-FPR) resulting under AI were estimated as a function of the classification cutoff and compared to that of human readers.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>\n                      Using a cutoff of 1, AI decreased the proportion of women with false-positives from 89.9 to 62.0%, non-FPR 11.1% vs. 38.0% (difference 26.9%, 95% confidence interval 25.1\u201328.8%;\n                      <jats:italic>p<\/jats:italic>\n                      &lt; .001), preventing 30.1% of reader-induced false-positive recalls, while reducing sensitivity from 96.7 to 91.1% (5.6%, 3.1\u20138.0%) as compared to human reading. The positive predictive value of recall (PPV-1) increased from 12.8 to 16.5% (3.7%, 3.5\u20134.0%). In women with mass-related lesions (\n                      <jats:italic>n<\/jats:italic>\n                      = 900), the non-FPR was 14.2% for humans vs. 36.7% for AI (22.4%, 19.8\u201325.3%) at a sensitivity of 98.5% vs. 97.1% (1.5%, 0\u20133.5%).\n                    <\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>The application of AI during consensus conference might especially help readers to reduce false-positive recalls of masses at the expense of a small sensitivity reduction. Prospective studies are needed to further evaluate the screening benefit of AI in practice.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Key Points<\/jats:title>\n                    <jats:p>\n                      <jats:italic>\u2022 Integrating the use of artificial intelligence in the arbitration process reduces benign recalls and increases the positive predictive value of recall at the expense of some sensitivity loss.<\/jats:italic>\n                    <\/jats:p>\n                    <jats:p>\n                      <jats:italic>\u2022 Application of the artificial intelligence system to aid the decision to recall a woman seems particularly beneficial for masses, where the system reaches comparable sensitivity to that of the readers, but with considerably reduced false-positives.<\/jats:italic>\n                    <\/jats:p>\n                    <jats:p>\n                      <jats:italic>\u2022 About one-fourth of all recalled malignant lesions are not automatically marked by the system such that their evaluation (AI score) must be retrieved manually by the reader. A thorough reading of screening mammograms by readers to identify suspicious lesions therefore remains mandatory.<\/jats:italic>\n                    <\/jats:p>\n                  <\/jats:sec>","DOI":"10.1007\/s00330-021-08217-w","type":"journal-article","created":{"date-parts":[[2021,8,12]],"date-time":"2021-08-12T10:03:04Z","timestamp":1628762584000},"page":"842-852","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":15,"title":["Using deep learning to assist readers during the arbitration process: a lesion-based retrospective evaluation of breast cancer screening performance"],"prefix":"10.1007","volume":"32","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1195-7399","authenticated-orcid":false,"given":"Laura","family":"Kerschke","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Stefanie","family":"Weigel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alejandro","family":"Rodriguez-Ruiz","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nico","family":"Karssemeijer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Walter","family":"Heindel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2021,8,12]]},"reference":[{"key":"8217_CR1","doi-asserted-by":"publisher","first-page":"146","DOI":"10.1016\/j.breast.2019.09.005","volume":"52","author":"CI Lee","year":"2019","unstructured":"Lee CI, Houssami N, Elmore JG, Buist DSM (2019) Pathways to breast cancer screening artificial intelligence algorithm validation. Breast 52:146\u2013149","journal-title":"Breast"},{"key":"8217_CR2","doi-asserted-by":"publisher","first-page":"38","DOI":"10.1148\/radiol.2019182908","volume":"293","author":"A Yala","year":"2019","unstructured":"Yala A, Schuster T, Miles R, Barzilay R, Lehman C (2019) A deep learning model to triage screening mammograms: a simulation study. Radiology 293:38\u201346","journal-title":"Radiology"},{"key":"8217_CR3","doi-asserted-by":"publisher","first-page":"305","DOI":"10.1148\/radiol.2018181371","volume":"290","author":"A Rodr\u00edguez-Ruiz","year":"2019","unstructured":"Rodr\u00edguez-Ruiz A, Krupinski E, Mordang JJ et al (2019) Detection of breast cancer with mammography: effect of an artificial intelligence support system. Radiology 290:305\u2013314","journal-title":"Radiology"},{"key":"8217_CR4","doi-asserted-by":"publisher","first-page":"4825","DOI":"10.1007\/s00330-019-06186-9","volume":"29","author":"A Rodriguez-Ruiz","year":"2019","unstructured":"Rodriguez-Ruiz A, L\u00e5ng K, Gubern-Merida A et al (2019) Can we reduce the workload of mammographic screening by automatic identification of normal exams with artificial intelligence? A feasibility study. Eur Radiol 29:4825\u20134832","journal-title":"Eur Radiol"},{"key":"8217_CR5","doi-asserted-by":"publisher","first-page":"e200265","DOI":"10.1001\/jamanetworkopen.2020.0265","volume":"3","author":"T Schaffter","year":"2020","unstructured":"Schaffter T, Buist DSM, Lee CI et al (2020) Evaluation of combined artificial intelligence and radiologist assessment to interpret screening mammograms. JAMA Netw Open 3:e200265","journal-title":"JAMA Netw Open"},{"key":"8217_CR6","doi-asserted-by":"publisher","first-page":"89","DOI":"10.1038\/s41586-019-1799-6","volume":"577","author":"SM McKinney","year":"2020","unstructured":"McKinney SM, Sieniek M, Godbole V et al (2020) International evaluation of an AI system for breast cancer screening. Nature 577:89\u201394","journal-title":"Nature"},{"key":"8217_CR7","doi-asserted-by":"publisher","first-page":"246","DOI":"10.1148\/radiol.2019182627","volume":"293","author":"KJ Geras","year":"2019","unstructured":"Geras KJ, Mann RM, Moy L (2019) Artificial intelligence for mammography and digital breast tomosynthesis: current concepts and future perspectives. Radiology 293:246\u2013259","journal-title":"Radiology"},{"key":"8217_CR8","doi-asserted-by":"publisher","first-page":"351","DOI":"10.1080\/17434440.2019.1610387","volume":"16","author":"N Houssami","year":"2019","unstructured":"Houssami N, Kirkpatrick-Jones G, Noguchi N, Lee CI (2019) Artificial Intelligence (AI) for the early detection of breast cancer: a scoping review to assess AI\u2019s potential in breast screening practice. Expert Rev Med Device 16:351\u2013362","journal-title":"Expert Rev Med Device"},{"key":"8217_CR9","doi-asserted-by":"publisher","first-page":"357","DOI":"10.1016\/j.crad.2019.02.006","volume":"74","author":"EPV Le","year":"2019","unstructured":"Le EPV, Wang Y, Huang Y, Hickman S, Gilbert FJ (2019) Artificial intelligence in breast imaging. Clin Radiol 74:357\u2013366","journal-title":"Clin Radiol"},{"key":"8217_CR10","doi-asserted-by":"publisher","first-page":"e138","DOI":"10.1016\/S2589-7500(20)30003-0","volume":"2","author":"HE Kim","year":"2020","unstructured":"Kim HE, Kim HH, Han BK et al (2020) Changes in cancer detection and false-positive recall in mammography using artificial intelligence: a retrospective, multireader study. Lancet Digit Health 2:e138\u2013e148","journal-title":"Lancet Digit Health"},{"key":"8217_CR11","doi-asserted-by":"publisher","first-page":"2205","DOI":"10.1038\/bjc.2013.177","volume":"108","author":"MG Marmot","year":"2013","unstructured":"Marmot MG, Altman DG, Cameron DA, Dewar JA, Thompson SG, Wilcox M (2013) The benefits and harms of breast cancer screening: an independent review. Br J Cancer 108:2205\u20132240","journal-title":"Br J Cancer"},{"key":"8217_CR12","doi-asserted-by":"publisher","first-page":"954","DOI":"10.1001\/jamainternmed.2014.981","volume":"174","author":"AN Tosteson","year":"2014","unstructured":"Tosteson AN, Fryback DG, Hammond CS et al (2014) Consequences of false-positive screening mammograms. JAMA Intern Med 174:954\u2013961","journal-title":"JAMA Intern Med"},{"key":"8217_CR13","volume-title":"European guidelines for quality assurance in breast cancer screening and diagnosis","author":"N Perry","year":"2006","unstructured":"Perry N, Broeders M, de Wolf C, T\u00f6rnberg S, Holland R, van Karsa L (2006) European guidelines for quality assurance in breast cancer screening and diagnosis, 4th edn. Office for Official Publications of the European Communities, Luxembourg","edition":"4"},{"key":"8217_CR14","unstructured":"Kooperationsgemeinschaft Mammographie (2020) Jahresbericht Evaluation 2018 Deutsches Mammographie-Screening-Programm. Available via https:\/\/fachservice.mammo-programm.de\/download\/evaluationsberichte\/Jahresbericht-Evaluation_2018.pdf. Accessed 19 Jan 2021."},{"key":"8217_CR15","unstructured":"National Evaluation Team for Breast Cancer Screening (2014) National evaluation of breast cancer screening in the Netherlands 1990 - 2011\/2012 NETB XIII. Available via https:\/\/www.lrcb.nl\/resources\/uploads\/2017\/02\/LETB-XIII-English.pdf. Accessed 19 Jan 2021."},{"key":"8217_CR16","doi-asserted-by":"publisher","first-page":"38","DOI":"10.1148\/radiol.13131498","volume":"271","author":"S Weigel","year":"2014","unstructured":"Weigel S, Heindel W, Heidinger O, Berkemeyer S, Hense HW (2014) Digital mammography screening: association between detection rate and nuclear grade of ductal carcinoma in situ. Radiology 271:38\u201344","journal-title":"Radiology"},{"key":"8217_CR17","doi-asserted-by":"publisher","first-page":"424","DOI":"10.1148\/radiol.2017170673","volume":"286","author":"S Weigel","year":"2018","unstructured":"Weigel S, Khil L, Hense HW et al (2018) Detection rates of ductal carcinoma in situ with biennial digital mammography screening: radiologic findings support pathologic model of tumor progression. Radiology 286:424\u2013432","journal-title":"Radiology"},{"key":"8217_CR18","doi-asserted-by":"publisher","first-page":"916","DOI":"10.1093\/jnci\/djy222","volume":"111","author":"A Rodriguez-Ruiz","year":"2019","unstructured":"Rodriguez-Ruiz A, L\u00e5ng K, Gubern-Merida A (2019) Stand-alone artificial intelligence for breast cancer detection in mammography: comparison with 101 radiologists. J Natl Cancer Inst 111:916\u2013922","journal-title":"J Natl Cancer Inst"},{"key":"8217_CR19","doi-asserted-by":"publisher","first-page":"915","DOI":"10.1016\/j.acra.2013.03.001","volume":"20","author":"DP Chakraborty","year":"2013","unstructured":"Chakraborty DP (2013) A brief history of free-response receiver operating characteristic paradigm data analysis. Acad Radiol 20:915\u2013919","journal-title":"Acad Radiol"},{"key":"8217_CR20","doi-asserted-by":"publisher","first-page":"891","DOI":"10.1002\/(SICI)1097-0258(19980430)17:8<891::AID-SIM780>3.0.CO;2-B","volume":"17","author":"T Tango","year":"1998","unstructured":"Tango T (1998) Equivalence test and confidence interval for the difference in proportions for the paired-sample design. Stat Med 17:891\u2013908","journal-title":"Stat Med"},{"key":"8217_CR21","doi-asserted-by":"publisher","first-page":"964","DOI":"10.1002\/sim.5587","volume":"32","author":"AS Kosinski","year":"2013","unstructured":"Kosinski AS (2013) A weighted generalized score statistic for comparison of predictive values of diagnostic tests. Stat Med 32:964\u2013977","journal-title":"Stat Med"},{"key":"8217_CR22","doi-asserted-by":"publisher","first-page":"2520","DOI":"10.1007\/s00330-015-4074-8","volume":"26","author":"L Domingo","year":"2016","unstructured":"Domingo L, Hofvind S, Hubbard RA et al (2016) Cross-national comparison of screening mammography accuracy measures in US, Norway, and Spain. Eur Radiol 26:2520\u20132528","journal-title":"Eur Radiol"},{"key":"8217_CR23","doi-asserted-by":"publisher","first-page":"5902","DOI":"10.1158\/1078-0432.CCR-18-1115","volume":"24","author":"SS Aboutalib","year":"2018","unstructured":"Aboutalib SS, Mohamed AA, Berg WA, Zuley ML, Sumkin JH, Wu S (2018) Deep learning to distinguish recalled but benign mammography images in breast cancer screening. Clin Cancer Res 24:5902\u20135909","journal-title":"Clin Cancer Res"},{"key":"8217_CR24","doi-asserted-by":"publisher","first-page":"27","DOI":"10.1631\/FITEE.1700808","volume":"19","author":"QS Zhang","year":"2018","unstructured":"Zhang QS, Zhu SC (2018) Visual interpretability for deep learning: a survey. Front Inf Technol Electronic Eng 19:27\u201339","journal-title":"Front Inf Technol Electronic Eng"}],"container-title":["European Radiology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00330-021-08217-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00330-021-08217-w\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00330-021-08217-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,1,27]],"date-time":"2022-01-27T07:08:37Z","timestamp":1643267317000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s00330-021-08217-w"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,8,12]]},"references-count":24,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2022,2]]}},"alternative-id":["8217"],"URL":"https:\/\/doi.org\/10.1007\/s00330-021-08217-w","relation":{},"ISSN":["0938-7994","1432-1084"],"issn-type":[{"value":"0938-7994","type":"print"},{"value":"1432-1084","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,8,12]]},"assertion":[{"value":"12 March 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"4 June 2021","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"13 July 2021","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 August 2021","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The scientific guarantor of this publication is Walter Heindel.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Guarantor"}},{"value":"The authors of this manuscript declare relationships as project partners of the EU-funded INTERREG V A-Projekt with the following companies: ScreenPoint Medical BV, Nijmegen, The Netherlands (AR and NK are employees).","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"One of the authors (LK) has significant statistical expertise.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Statistics and biometry"}},{"value":"Written informed consent was not required for this study due to the evaluation of internal anonymized data.","order":5,"name":"Ethics","group":{"name":"EthicsHeading","label":"Informed consent"}},{"value":"Institutional Review Board approval was obtained.","order":6,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical approval"}},{"value":"\u2022 retrospective\u2022 diagnostic study\/experimental\u2022 performed at one institution","order":7,"name":"Ethics","group":{"name":"EthicsHeading","label":"Methodology"}}]}}