{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,26]],"date-time":"2026-10-26T01:35:32Z","timestamp":1792978532207,"version":"4.5.0"},"reference-count":18,"publisher":"American Medical Association (AMA)","issue":"11","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JAMA Netw Open"],"abstract":"<jats:sec id=\"ab-zoi221192-4\"><jats:title>Importance<\/jats:title><jats:p>With a shortfall in fellowship-trained breast radiologists, mammography screening programs are looking toward artificial intelligence (AI) to increase efficiency and diagnostic accuracy. External validation studies provide an initial assessment of how promising AI algorithms perform in different practice settings.<\/jats:p><\/jats:sec><jats:sec id=\"ab-zoi221192-5\"><jats:title>Objective<\/jats:title><jats:p>To externally validate an ensemble deep-learning model using data from a high-volume, distributed screening program of an academic health system with a diverse patient population.<\/jats:p><\/jats:sec><jats:sec id=\"ab-zoi221192-6\"><jats:title>Design, Setting, and Participants<\/jats:title><jats:p>In this diagnostic study, an ensemble learning method, which reweights outputs of the 11 highest-performing individual AI models from the Digital Mammography Dialogue on Reverse Engineering Assessment and Methods (DREAM) Mammography Challenge, was used to predict the cancer status of an individual using a standard set of screening mammography images. This study was conducted using retrospective patient data collected between 2010 and 2020 from women aged 40 years and older who underwent a routine breast screening examination and participated in the Athena Breast Health Network at the University of California, Los Angeles (UCLA).<\/jats:p><\/jats:sec><jats:sec id=\"ab-zoi221192-7\"><jats:title>Main Outcomes and Measures<\/jats:title><jats:p>Performance of the challenge ensemble method (CEM) and the CEM combined with radiologist assessment (CEM+R) were compared with diagnosed ductal carcinoma in situ and invasive cancers within a year of the screening examination using performance metrics, such as sensitivity, specificity, and area under the receiver operating characteristic curve (AUROC).<\/jats:p><\/jats:sec><jats:sec id=\"ab-zoi221192-8\"><jats:title>Results<\/jats:title><jats:p>Evaluated on 37\u202f317 examinations from 26\u202f817 women (mean [SD] age, 58.4 [11.5] years), individual model AUROC estimates ranged from 0.77 (95% CI, 0.75-0.79) to 0.83 (95% CI, 0.81-0.85). The CEM model achieved an AUROC of 0.85 (95% CI, 0.84-0.87) in the UCLA cohort, lower than the performance achieved in the Kaiser Permanente Washington (AUROC, 0.90) and Karolinska Institute (AUROC, 0.92) cohorts. The CEM+R model achieved a sensitivity (0.813 [95% CI, 0.781-0.843] vs 0.826 [95% CI, 0.795-0.856]; <jats:italic>P<\/jats:italic>\u2009=\u2009.20) and specificity (0.925 [95% CI, 0.916-0.934] vs 0.930 [95% CI, 0.929-0.932]; <jats:italic>P<\/jats:italic>\u2009=\u2009.18) similar to the radiologist performance. The CEM+R model had significantly lower sensitivity (0.596 [95% CI, 0.466-0.717] vs 0.850 [95% CI, 0.766-0.923]; <jats:italic>P<\/jats:italic>\u2009&amp;amp;lt;\u2009.001) and specificity (0.803 [95% CI, 0.734-0.861] vs 0.945 [95% CI, 0.936-0.954]; <jats:italic>P<\/jats:italic>\u2009&amp;amp;lt;\u2009.001) than the radiologist in women with a prior history of breast cancer and Hispanic women (0.894 [95% CI, 0.873-0.910] vs 0.926 [95% CI, 0.919-0.933]; <jats:italic>P<\/jats:italic>\u2009=\u2009.004).<\/jats:p><\/jats:sec><jats:sec id=\"ab-zoi221192-9\"><jats:title>Conclusions and Relevance<\/jats:title><jats:p>This study found that the high performance of an ensemble deep-learning model for automated screening mammography interpretation did not generalize to a more diverse screening cohort, suggesting that the model experienced underspecification. This study suggests the need for model transparency and fine-tuning of AI models for specific target populations prior to their clinical adoption.<\/jats:p><\/jats:sec>","DOI":"10.1001\/jamanetworkopen.2022.42343","type":"journal-article","created":{"date-parts":[[2022,11,21]],"date-time":"2022-11-21T16:33:02Z","timestamp":1669048382000},"page":"e2242343","source":"Crossref","is-referenced-by-count":60,"title":["External Validation of an Ensemble Model for Automated Mammography Interpretation by Artificial Intelligence"],"prefix":"10.1001","volume":"5","author":[{"given":"William","family":"Hsu","sequence":"first","affiliation":[{"name":"Medical and Imaging Informatics, Department of Radiological Sciences, David Geffen School of Medicine at University California, Los Angeles"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Daniel S.","family":"Hippe","sequence":"additional","affiliation":[{"name":"Clinical Research Division, Fred Hutchinson Cancer Center, Seattle, Washington"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Noor","family":"Nakhaei","sequence":"additional","affiliation":[{"name":"Medical and Imaging Informatics, Department of Radiological Sciences, David Geffen School of Medicine at University California, Los Angeles"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pin-Chieh","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Medicine, David Geffen School of Medicine at University California, Los Angeles"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bing","family":"Zhu","sequence":"additional","affiliation":[{"name":"Medical and Imaging Informatics, Department of Radiological Sciences, David Geffen School of Medicine at University California, Los Angeles"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nathan","family":"Siu","sequence":"additional","affiliation":[{"name":"Medical Informatics Home Area, Graduate Programs in Biosciences, David Geffen School of Medicine at University California, Los Angeles, Los Angeles, California"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mehmet Eren","family":"Ahsen","sequence":"additional","affiliation":[{"name":"Gies College of Business, University of Illinois at Urbana-Champaign"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"William","family":"Lotter","sequence":"additional","affiliation":[{"name":"DeepHealth, RadNet AI Solutions, Cambridge, Massachusetts"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"A. Gregory","family":"Sorensen","sequence":"additional","affiliation":[{"name":"DeepHealth, RadNet AI Solutions, Cambridge, Massachusetts"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Arash","family":"Naeim","sequence":"additional","affiliation":[{"name":"Center for Systematic, Measurable, Actionable, Resilient, and Technology-driven Health, Clinical and Translational Science Institute, David Geffen School of Medicine at University California, Los Angeles"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Diana S. M.","family":"Buist","sequence":"additional","affiliation":[{"name":"Kaiser Permanente Washington Health Research Institute, Seattle, Washington"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thomas","family":"Schaffter","sequence":"additional","affiliation":[{"name":"Computational Oncology, Sage Bionetworks, Seattle, Washington"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Justin","family":"Guinney","sequence":"additional","affiliation":[{"name":"Tempus Labs, Chicago, Illinois"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Joann G.","family":"Elmore","sequence":"additional","affiliation":[{"name":"Department of Medicine, David Geffen School of Medicine at University California, Los Angeles"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christoph I.","family":"Lee","sequence":"additional","affiliation":[{"name":"Department of Radiology, University of Washington School of Medicine, Seattle"},{"name":"Department of Health Services, University of Washington School of Public Health, Seattle"},{"name":"Hutchinson Institute for Cancer Outcomes Research, Fred Hutchinson Cancer Center, Seattle, Washington"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"10","published-online":{"date-parts":[[2022,11,21]]},"reference":[{"issue":"7","key":"zoi221192r1","doi-asserted-by":"publisher","first-page":"4872","DOI":"10.1007\/s00330-020-07640-9","article-title":"Deep learning in breast radiology: current progress and future directions.","volume":"31","author":"Ou","year":"2021","journal-title":"Eur Radiol"},{"issue":"2","key":"zoi221192r2","doi-asserted-by":"publisher","first-page":"293","DOI":"10.2214\/AJR.18.20532","article-title":"Artificial intelligence in breast imaging: potentials and limitations.","volume":"212","author":"Mendelson","year":"2019","journal-title":"AJR Am J Roentgenol"},{"issue":"5","key":"zoi221192r3","doi-asserted-by":"publisher","first-page":"351","DOI":"10.1080\/17434440.2019.1610387","article-title":"Artificial intelligence (AI) for the early detection of breast cancer: a scoping review to assess AI\u2019s potential in breast screening practice.","volume":"16","author":"Houssami","year":"2019","journal-title":"Expert Rev Med Devices"},{"issue":"4","key":"zoi221192r4","doi-asserted-by":"publisher","first-page":"304","DOI":"10.1093\/jbi\/wbaa033","article-title":"Artificial intelligence: a primer for breast imaging radiologists.","volume":"2","author":"Bahl","year":"2020","journal-title":"J Breast Imaging"},{"issue":"6","key":"zoi221192r7","doi-asserted-by":"publisher","DOI":"10.1148\/ryai.2021210097","article-title":"Toward generalizability in the deployment of artificial intelligence in radiology: role of computation stress testing to overcome underspecification.","volume":"3","author":"Eche","year":"2021","journal-title":"Radiol Artif Intell"},{"issue":"4","key":"zoi221192r8","doi-asserted-by":"publisher","first-page":"305","DOI":"10.1001\/jama.2019.20866","article-title":"Challenges to the reproducibility of machine learning models in health care.","volume":"323","author":"Beam","year":"2020","journal-title":"JAMA"},{"issue":"3","key":"zoi221192r9","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2020.0265","article-title":"Evaluation of combined artificial intelligence and radiologist assessment to interpret screening mammograms.","volume":"3","author":"Schaffter","year":"2020","journal-title":"JAMA Netw Open"},{"issue":"11","key":"zoi221192r10","doi-asserted-by":"publisher","first-page":"1463","DOI":"10.1001\/jamaoncol.2017.0473","article-title":"Will machine learning tip the balance in breast cancer screening?","volume":"3","author":"Trister","year":"2017","journal-title":"JAMA Oncol"},{"issue":"2","key":"zoi221192r12","doi-asserted-by":"publisher","first-page":"417","DOI":"10.1007\/s10549-013-2612-0","article-title":"The Athena Breast Health Network: developing a rapid learning system in breast cancer prevention, screening, treatment, and care.","volume":"140","author":"Elson","year":"2013","journal-title":"Breast Cancer Res Treat"},{"key":"zoi221192r13","doi-asserted-by":"publisher","first-page":"93","DOI":"10.1016\/j.jclinepi.2019.09.016","article-title":"Validation of clinical prediction models: what does the \u201ccalibration slope\u201d really measure?","volume":"118","author":"Stevens","year":"2020","journal-title":"J Clin Epidemiol"},{"issue":"15","key":"zoi221192r14","doi-asserted-by":"publisher","first-page":"2595","DOI":"10.1093\/bioinformatics\/btv153","article-title":"PRROC: computing and visualizing precision-recall and receiver operating characteristic curves in R.","volume":"31","author":"Grau","year":"2015","journal-title":"Bioinformatics"},{"key":"zoi221192r15","doi-asserted-by":"publisher","first-page":"259","DOI":"10.1016\/j.jacr.2021.11.008","article-title":"Independent external validation of artificial intelligence algorithms for automated interpretation of screening mammography: a systematic review.","volume":"19","author":"Anderson","year":"2022","journal-title":"J Am Coll Radiol"},{"issue":"10","key":"zoi221192r16","doi-asserted-by":"publisher","first-page":"1581","DOI":"10.1001\/jamaoncol.2020.3321","article-title":"External evaluation of 3 commercial artificial intelligence algorithms for independent assessment of screening mammograms.","volume":"6","author":"Salim","year":"2020","journal-title":"JAMA Oncol"},{"issue":"1","key":"zoi221192r17","doi-asserted-by":"publisher","first-page":"51","DOI":"10.7326\/M18-1376","article-title":"PROBAST: a tool to assess the risk of bias and applicability of prediction model studies.","volume":"170","author":"Wolff","year":"2019","journal-title":"Ann Intern Med"},{"key":"zoi221192r18","doi-asserted-by":"publisher","first-page":"261","DOI":"10.1038\/s42256-019-0055-y","article-title":"Establishing the rules for building trustworthy AI.","volume":"1","author":"Floridi","year":"2019","journal-title":"Nat Mach Intell"},{"key":"zoi221192r11","volume-title":"Protection of human subjects: general requirements for informed consent","author":"US Department of Health and Human Services"},{"key":"zoi221192r5","unstructured":"US Food and Drug Administration. Artificial intelligence and machine learning (AI\/ML)-enabled medical devices. Accessed March 20, 2022. https:\/\/www.fda.gov\/medical-devices\/software-medical-device-samd\/artificial-intelligence-and-machine-learning-aiml-enabled-medical-devices"},{"key":"zoi221192r6","unstructured":"American College of Radiology Data Science Institute.? AI central. Accessed March 20, 2022. https:\/\/aicentral.acrdsi.org\/"}],"container-title":["JAMA Network Open"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/jamanetwork.com\/journals\/jamanetworkopen\/articlepdf\/2798829\/hsu_2022_oi_221192_1668176872.82758.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,11,21]],"date-time":"2022-11-21T16:33:05Z","timestamp":1669048385000},"score":1,"resource":{"primary":{"URL":"https:\/\/jamanetwork.com\/journals\/jamanetworkopen\/fullarticle\/2798829"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,11,21]]},"references-count":18,"journal-issue":{"issue":"11","published-print":{"date-parts":[[2022,11,1]]}},"URL":"https:\/\/doi.org\/10.1001\/jamanetworkopen.2022.42343","relation":{},"ISSN":["2574-3805"],"issn-type":[{"value":"2574-3805","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,11,21]]}}}