{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,3]],"date-time":"2026-11-03T20:10:05Z","timestamp":1793736605435,"version":"4.6.1"},"reference-count":49,"publisher":"Oxford University Press (OUP)","issue":"9","license":[{"start":{"date-parts":[[2019,3,5]],"date-time":"2019-03-05T00:00:00Z","timestamp":1551744000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2019,9,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Background<\/jats:title>\n                  <jats:p>Artificial intelligence (AI) systems performing at radiologist-like levels in the evaluation of digital mammography (DM) would improve breast cancer screening accuracy and efficiency. We aimed to compare the stand-alone performance of an AI system to that of radiologists in detecting breast cancer in DM.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Methods<\/jats:title>\n                  <jats:p>Nine multi-reader, multi-case study datasets previously used for different research purposes in seven countries were collected. Each dataset consisted of DM exams acquired with systems from four different vendors, multiple radiologists\u2019 assessments per exam, and ground truth verified by histopathological analysis or follow-up, yielding a total of 2652 exams (653 malignant) and interpretations by 101 radiologists (28\u2009296 independent interpretations). An AI system analyzed these exams yielding a level of suspicion of cancer present between 1 and 10. The detection performance between the radiologists and the AI system was compared using a noninferiority null hypothesis at a margin of 0.05.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>The performance of the AI system was statistically noninferior to that of the average of the 101 radiologists. The AI system had a 0.840 (95% confidence interval [CI] = 0.820 to 0.860) area under the ROC curve and the average of the radiologists was 0.814 (95% CI = 0.787 to 0.841) (difference 95% CI = \u22120.003 to 0.055). The AI system had an AUC higher than 61.4% of the radiologists.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusions<\/jats:title>\n                  <jats:p>The evaluated AI system achieved a cancer detection accuracy comparable to an average breast radiologist in this retrospective setting. Although promising, the performance and impact of such a system in a screening setting needs further investigation.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/jnci\/djy222","type":"journal-article","created":{"date-parts":[[2018,11,30]],"date-time":"2018-11-30T15:11:45Z","timestamp":1543590705000},"page":"916-922","source":"Crossref","is-referenced-by-count":626,"title":["Stand-Alone Artificial Intelligence for Breast Cancer Detection in Mammography: Comparison With 101 Radiologists"],"prefix":"10.1093","volume":"111","author":[{"given":"Alejandro","family":"Rodriguez-Ruiz","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kristina","family":"L\u00e5ng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Albert","family":"Gubern-Merida","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mireille","family":"Broeders","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gisella","family":"Gennaro","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Paola","family":"Clauser","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thomas H","family":"Helbich","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Margarita","family":"Chevalier","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tao","family":"Tan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thomas","family":"Mertelmeier","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Matthew G","family":"Wallis","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ingvar","family":"Andersson","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sophia","family":"Zackrisson","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ritse M","family":"Mann","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ioannis","family":"Sechopoulos","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2019,3,5]]},"reference":[{"issue":"5","key":"2019091715471745000_djy222-B1","doi-asserted-by":"crossref","DOI":"10.1002\/ijc.29210","article-title":"Cancer incidence and mortality worldwide: sources, methods and major patterns in GLOBOCAN 2012","volume":"136","author":"Ferlay","year":"2015","journal-title":"Int J Cancer"},{"key":"2019091715471745000_djy222-B2","doi-asserted-by":"crossref","first-page":"14","DOI":"10.1258\/jms.2012.012078","article-title":"The impact of mammographic screening on breast cancer mortality in Europe: a review of observational studies","volume":"19(suppl 1)","author":"Broeders","year":"2012","journal-title":"J Med Screen"},{"issue":"24","key":"2019091715471745000_djy222-B3","doi-asserted-by":"crossref","first-page":"2353","DOI":"10.1056\/NEJMsr1504363","article-title":"Breast cancer screening\u2013viewpoint of the IARC Working Group","volume":"372","author":"Lauby-Secretan","year":"2015","journal-title":"N Engl J Med"},{"issue":"11","key":"2019091715471745000_djy222-B4","doi-asserted-by":"crossref","first-page":"2205","DOI":"10.1038\/bjc.2013.177","article-title":"The benefits and harms of breast cancer screening: an independent review","volume":"108","author":"Marmot","year":"2013","journal-title":"Br J Cancer"},{"issue":"2","key":"2019091715471745000_djy222-B5","doi-asserted-by":"crossref","first-page":"100","DOI":"10.3322\/caac.21392","article-title":"Cancer screening in the United States, 2017: a review of current American Cancer Society guidelines and current issues in cancer screening","volume":"67","author":"Smith","year":"2017","journal-title":"CA Cancer J Clin"},{"issue":"3","key":"2019091715471745000_djy222-B6","doi-asserted-by":"crossref","first-page":"613","DOI":"10.1148\/radiology.184.3.1509041","article-title":"Analysis of cancers missed at screening mammography","volume":"184","author":"Bird","year":"1992","journal-title":"Radiology"},{"issue":"4","key":"2019091715471745000_djy222-B7","doi-asserted-by":"crossref","first-page":"881","DOI":"10.1148\/rg.234025083","article-title":"Missed breast carcinoma: pitfalls and pearls","volume":"23","author":"Majid","year":"2003","journal-title":"Radiographics"},{"issue":"3","key":"2019091715471745000_djy222-B8","doi-asserted-by":"crossref","first-page":"471","DOI":"10.1007\/s10549-016-3882-0","article-title":"Characteristics and prognosis of interval cancers after biennial screen-film or full-field digital screening mammography","volume":"158","author":"Weber","year":"2016","journal-title":"Breast Cancer Res Treat"},{"issue":"12","key":"2019091715471745000_djy222-B9","doi-asserted-by":"crossref","first-page":"1770","DOI":"10.1016\/S0959-8049(03)00311-3","article-title":"Use of previous screening mammograms to identify features indicating cases that would have a possible gain in prognosis following earlier detection","volume":"39","author":"Broeders","year":"2003","journal-title":"Eur J Cancer"},{"key":"2019091715471745000_djy222-B10","article-title":"Radiologist shortage leaves patient care at risk, warns royal college","volume":"359","author":"Rimmer","year":"2017","journal-title":"BMJ"},{"key":"2019091715471745000_djy222-B11","year":"2017"},{"issue":"2","key":"2019091715471745000_djy222-B12","doi-asserted-by":"crossref","first-page":"370","DOI":"10.2214\/AJR.08.1665","article-title":"Workforce shortages in breast imaging: impact on mammography utilization","volume":"192","author":"Wing","year":"2009","journal-title":"AJR Am J Roentgenol"},{"issue":"14","key":"2019091715471745000_djy222-B13","doi-asserted-by":"crossref","first-page":"1399","DOI":"10.1056\/NEJMoa066099","article-title":"Influence of computer-aided detection on performance of screening mammography","volume":"356","author":"Fenton","year":"2007","journal-title":"N Engl J Med"},{"issue":"11","key":"2019091715471745000_djy222-B14","doi-asserted-by":"crossref","first-page":"1828","DOI":"10.1001\/jamainternmed.2015.5231","article-title":"Diagnostic accuracy of digital screening mammography with and without computer-aided detection","volume":"175","author":"Lehman","year":"2015","journal-title":"JAMA Intern Med"},{"key":"2019091715471745000_djy222-B15","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1016\/j.media.2017.07.005","article-title":"A survey on deep learning in medical image analysis","volume":"42","author":"Litjens","year":"2017","journal-title":"Med Image Anal"},{"issue":"22","key":"2019091715471745000_djy222-B16","doi-asserted-by":"crossref","first-page":"2199","DOI":"10.1001\/jama.2017.14585","article-title":"Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer","volume":"318","author":"Bejnordi","year":"2017","journal-title":"JAMA"},{"key":"2019091715471745000_djy222-B17","doi-asserted-by":"crossref","DOI":"10.1001\/jamaoncol.2017.0473","article-title":"Will machine learning tip the balance in breast cancer screening?","author":"Trister","year":"2017","journal-title":"JAMA Oncol"},{"issue":"7","key":"2019091715471745000_djy222-B18","doi-asserted-by":"crossref","first-page":"434","DOI":"10.1097\/RLI.0000000000000358","article-title":"Deep learning in mammography: diagnostic accuracy of a multipurpose image analysis software in the detection of breast cancer","volume":"52","author":"Becker","year":"2017","journal-title":"Invest Radiol"},{"key":"2019091715471745000_djy222-B19","doi-asserted-by":"crossref","first-page":"303","DOI":"10.1016\/j.media.2016.07.007","article-title":"Large scale deep learning for computer aided detection of mammographic lesions","volume":"35","author":"Kooi","year":"2017","journal-title":"Med Image Anal"},{"key":"2019091715471745000_djy222-B20","author":"Mordang"},{"issue":"2","key":"2019091715471745000_djy222-B21","doi-asserted-by":"crossref","first-page":"241","DOI":"10.1016\/j.media.2013.10.014","article-title":"Learning from unbalanced data: a cascade-based approach for detecting clustered microcalcifications","volume":"18","author":"Bria","year":"2014","journal-title":"Med Image Anal"},{"issue":"12","key":"2019091715471745000_djy222-B22","doi-asserted-by":"crossref","first-page":"2033","DOI":"10.1109\/TMI.2009.2028611","article-title":"Use of normal tissue context in computer-aided detection of masses in mammograms","volume":"28","author":"Hupse","year":"2009","journal-title":"IEEE Trans Med Imaging"},{"issue":"2","key":"2019091715471745000_djy222-B23","doi-asserted-by":"crossref","first-page":"365","DOI":"10.1088\/0031-9155\/43\/2\/011","article-title":"Automated classification of parenchymal patterns in mammograms","volume":"43","author":"Karssemeijer","year":"1998","journal-title":"Phys Med Biol"},{"issue":"5","key":"2019091715471745000_djy222-B24","doi-asserted-by":"crossref","first-page":"611","DOI":"10.1109\/42.538938","article-title":"Detection of stellate distortions in mammograms","volume":"15","author":"Karssemeijer","year":"1996","journal-title":"IEEE Trans Med Imaging"},{"issue":"3","key":"2019091715471745000_djy222-B25","doi-asserted-by":"crossref","first-page":"788","DOI":"10.1148\/radiol.11103514","article-title":"Two-view and single-view tomosynthesis versus full-field digital mammography: high-resolution X-ray imaging observer study","volume":"262","author":"Wallis","year":"2012","journal-title":"Radiology"},{"issue":"4","key":"2019091715471745000_djy222-B26","doi-asserted-by":"crossref","first-page":"908","DOI":"10.1007\/s00330-011-2320-2","article-title":"Increase in perceived case suspiciousness due to local contrast optimisation in digital screening mammography","volume":"22","author":"Visser","year":"2012","journal-title":"Eur Radiol"},{"issue":"1","key":"2019091715471745000_djy222-B27","doi-asserted-by":"crossref","first-page":"123","DOI":"10.1148\/radiol.12120218","article-title":"Computer-aided detection of masses at mammography: interactive decision support versus prompts","volume":"266","author":"Hupse","year":"2013","journal-title":"Radiology"},{"issue":"3","key":"2019091715471745000_djy222-B28","doi-asserted-by":"crossref","first-page":"664","DOI":"10.1007\/s00330-012-2649-1","article-title":"Performance comparison of single-view digital breast tomosynthesis plus single-view digital mammography with two-view digital mammography","volume":"23","author":"Gennaro","year":"2013","journal-title":"Eur Radiol"},{"key":"2019091715471745000_djy222-B29","author":"Siemens Medical Solutions USA Inc","year":"2015"},{"issue":"2","key":"2019091715471745000_djy222-B30","doi-asserted-by":"crossref","first-page":"565","DOI":"10.1007\/s00330-017-4991-9","article-title":"Diagnostic value of the stand-alone synthetic image in digital breast tomosynthesis examinations","volume":"28","author":"Garayoa","year":"2018","journal-title":"Eur Radiol"},{"key":"2019091715471745000_djy222-B31","doi-asserted-by":"crossref","DOI":"10.1007\/s00330-017-5167-3","article-title":"One-view digital breast tomosynthesis as a stand-alone modality for breast cancer detection: do we need more?","author":"Rodriguez-Ruiz","year":"2018","journal-title":"Eur Radiol"},{"key":"2019091715471745000_djy222-B32","doi-asserted-by":"crossref","DOI":"10.1097\/RLI.0000000000000513","article-title":"Synthetic 2D mammography can replace digital mammography as an adjunct to wide-angle digital breast tomosynthesis","author":"Clauser","year":"2019","journal-title":"Invest Radiol"},{"issue":"4","key":"2019091715471745000_djy222-B33","doi-asserted-by":"crossref","first-page":"614","DOI":"10.1093\/annonc\/mdm481","article-title":"European guidelines for quality assurance in breast cancer screening and diagnosis. Fourth edition\u2014summary document","volume":"19","author":"Perry","year":"2007","journal-title":"Ann Oncol"},{"issue":"4","key":"2019091715471745000_djy222-B34","doi-asserted-by":"crossref","first-page":"345","DOI":"10.1016\/0197-2456(82)90024-1","article-title":"\u201cProving the null hypothesis\u201d in clinical trials","volume":"3","author":"Blackwelder","year":"1982","journal-title":"Control Clin Trials"},{"issue":"9","key":"2019091715471745000_djy222-B35","doi-asserted-by":"crossref","first-page":"1158","DOI":"10.1016\/j.acra.2012.04.011","article-title":"Hypothesis testing in noninferiority and equivalence MRMC ROC studies","volume":"19","author":"Chen","year":"2012","journal-title":"Acad Radiol"},{"issue":"15","key":"2019091715471745000_djy222-B36","doi-asserted-by":"crossref","first-page":"2586","DOI":"10.1080\/03610920802610084","article-title":"A framework for random-effects ROC analysis: biases with the bootstrap and other variance estimators","volume":"38","author":"Gallas","year":"2009","journal-title":"Commun Stat Theory Methods"},{"key":"2019091715471745000_djy222-B37","author":"Gallas","year":"2017"},{"key":"2019091715471745000_djy222-B38"},{"key":"2019091715471745000_djy222-B39","author":"Chen","year":"2018"},{"issue":"2\u20133","key":"2019091715471745000_djy222-B40","doi-asserted-by":"crossref","first-page":"387","DOI":"10.1016\/j.neunet.2007.12.013","article-title":"Reader studies for validation of CAD systems","volume":"21","author":"Gallas","year":"2008","journal-title":"Neural Netw"},{"issue":"1","key":"2019091715471745000_djy222-B41","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1148\/radiol.10091394","article-title":"BI-RADS data should not be used to estimate ROC curves","volume":"256","author":"Jiang","year":"2010","journal-title":"Radiology"},{"issue":"1040","key":"2019091715471745000_djy222-B42","doi-asserted-by":"crossref","first-page":"20140016.","DOI":"10.1259\/bjr.20140016","article-title":"The average receiver operating characteristic curve in multireader multicase imaging studies","volume":"87","author":"Chen","year":"2014","journal-title":"Br J Radiol"},{"issue":"3","key":"2019091715471745000_djy222-B43","doi-asserted-by":"crossref","first-page":"031006.","DOI":"10.1117\/1.JMI.1.3.031006","article-title":"Generalized Roe and Metz receiver operating characteristic model: analytic link between simulated decision scores and empirical AUC variances and covariances","volume":"1","author":"Gallas","year":"2014","journal-title":"J Med Imaging"},{"issue":"1","key":"2019091715471745000_djy222-B44","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1007\/s12282-016-0699-y","article-title":"Breast cancer screening with digital breast tomosynthesis","volume":"24","author":"Skaane","year":"2017","journal-title":"Breast Cancer"},{"issue":"24","key":"2019091715471745000_djy222-B45","doi-asserted-by":"crossref","first-page":"1840","DOI":"10.1093\/jnci\/djh333","article-title":"Accuracy of screening mammography interpretation by characteristics of radiologists","volume":"96","author":"Barlow","year":"2004","journal-title":"J Natl Cancer Inst"},{"issue":"1","key":"2019091715471745000_djy222-B46","doi-asserted-by":"crossref","first-page":"47","DOI":"10.1148\/radiol.2491072025","article-title":"The \u201claboratory\u201d effect: comparing radiologists' performance and variability during prospective clinical and laboratory mammography interpretations","volume":"249","author":"Gur","year":"2008","journal-title":"Radiology"},{"issue":"5","key":"2019091715471745000_djy222-B47","doi-asserted-by":"crossref","first-page":"e64366.","DOI":"10.1371\/journal.pone.0064366","article-title":"If you don\u2019t find it often, you often don\u2019t find it: why some cancers are missed in breast cancer screening","volume":"8","author":"Evans","year":"2013","journal-title":"PLoS One"},{"issue":"16","key":"2019091715471745000_djy222-B48","doi-asserted-by":"crossref","first-page":"1675","DOI":"10.1056\/NEJMoa0803545","article-title":"Single reading with computer-aided detection for screening mammography","volume":"359","author":"Gilbert","year":"2008","journal-title":"N Engl J Med"},{"issue":"5","key":"2019091715471745000_djy222-B49","doi-asserted-by":"crossref","first-page":"1137","DOI":"10.1148\/radiographics.18.5.9747612","article-title":"The false-negative mammogram","volume":"18","author":"Huynh","year":"1998","journal-title":"Radiographics"}],"container-title":["JNCI: Journal of the National Cancer Institute"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/jnci\/article-pdf\/111\/9\/916\/30011693\/djy222.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2019,9,17]],"date-time":"2019-09-17T15:52:22Z","timestamp":1568735542000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jnci\/article\/111\/9\/916\/5307077"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,3,5]]},"references-count":49,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2019,3,5]]},"published-print":{"date-parts":[[2019,9,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jnci\/djy222","relation":{},"ISSN":["0027-8874","1460-2105"],"issn-type":[{"value":"0027-8874","type":"print"},{"value":"1460-2105","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2019,9]]},"published":{"date-parts":[[2019,3,5]]}}}