{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,15]],"date-time":"2025-11-15T10:34:21Z","timestamp":1763202861418,"version":"3.41.2"},"reference-count":20,"publisher":"Oxford University Press (OUP)","issue":"1","license":[{"start":{"date-parts":[[2023,12,12]],"date-time":"2023-12-12T00:00:00Z","timestamp":1702339200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"IBM Watson Health Imaging"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2023,12,12]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Objectives<\/jats:title>\n                  <jats:p>This diagnostic study assessed the accuracy of radiologists retrospectively, using the deep learning and natural language processing chest algorithms implemented in Clinical Review version 3.2 for: pneumothorax, rib fractures in digital chest X-ray radiographs (CXR); aortic aneurysm, pulmonary nodules, emphysema, and pulmonary embolism in CT images.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Methods<\/jats:title>\n                  <jats:p>The study design was double-blind (artificial intelligence [AI] algorithms and humans), retrospective, non-interventional, and at a single NHS Trust. Adult patients (\u226518\u2009years old) scheduled for CXR and CT were invited to enroll as participants through an opt-out process. Reports and images were de-identified, processed retrospectively, and AI-flagged discrepant findings were assigned to two lead radiologists, each blinded to patient identifiers and original radiologist. The radiologist\u2019s findings for each clinical condition were tallied as a verified discrepancy (true positive) or not (false positive).<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>The missed findings were: 0.02% rib fractures, 0.51% aortic aneurysm, 0.32% pulmonary nodules, 0.92% emphysema, and 0.28% pulmonary embolism. The positive predictive values (PPVs) were: pneumothorax (0%), rib fractures (5.6%), aortic dilatation (43.2%), pulmonary emphysema (46.0%), pulmonary embolus (11.5%), and pulmonary nodules (9.2%). The PPV for pneumothorax was nil owing to lack of available studies that were analysed for outpatient activity.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Conclusions<\/jats:title>\n                  <jats:p>The number of missed findings was far less than generally predicted. The chest algorithms deployed retrospectively were a useful quality tool and AI augmented the radiologists\u2019 workflow.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Advances in knowledge<\/jats:title>\n                  <jats:p>The diagnostic accuracy of our radiologists generated missed findings of 0.02% for rib fractures CXR, 0.51% for aortic dilatation, 0.32% for pulmonary nodule, 0.92% for pulmonary emphysema, and 0.28% for pulmonary embolism for CT studies, all retrospectively evaluated with AI used as a quality tool to flag potential missed findings. It is important to account for prevalence of these chest conditions in clinical context and use appropriate clinical thresholds for decision-making, not relying solely on AI.<\/jats:p>\n               <\/jats:sec>","DOI":"10.1093\/bjro\/tzad009","type":"journal-article","created":{"date-parts":[[2024,1,10]],"date-time":"2024-01-10T00:11:44Z","timestamp":1704845504000},"source":"Crossref","is-referenced-by-count":1,"title":["A real-world evaluation of the diagnostic accuracy of radiologists using positive predictive values verified from deep learning and natural language processing chest algorithms deployed retrospectively"],"prefix":"10.1093","volume":"6","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4522-2665","authenticated-orcid":false,"given":"Bahadar S","family":"Bhatia","sequence":"first","affiliation":[{"name":"Directorate of Diagnostic Radiology, Sandwell & West Birmingham NHS Trust , Lyndon , West Bromwich B71 4HJ, United Kingdom"},{"name":"Space Research Centre, Physics & Astronomy, University of Leicester , 92 Corporation Road , Leicester LE4 5SP, United Kingdom"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"John F","family":"Morlese","sequence":"additional","affiliation":[{"name":"Directorate of Diagnostic Radiology, Sandwell & West Birmingham NHS Trust , Lyndon , West Bromwich B71 4HJ, United Kingdom"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sarah","family":"Yusuf","sequence":"additional","affiliation":[{"name":"Directorate of Diagnostic Radiology, Sandwell & West Birmingham NHS Trust , Lyndon , West Bromwich B71 4HJ, United Kingdom"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yiting","family":"Xie","sequence":"additional","affiliation":[{"name":"Merge, Merative (Formerly, IBM Watson Health Imaging) , Ann Arbor, Michigan, MI 48108, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Bob","family":"Schallhorn","sequence":"additional","affiliation":[{"name":"Merge, Merative (Formerly, IBM Watson Health Imaging) , Ann Arbor, Michigan, MI 48108, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"David","family":"Gruen","sequence":"additional","affiliation":[{"name":"Jefferson Radiology and Radiology Partners , 111 Founders Plaza , East Hartford, Connecticut CT 06108, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2023,12,12]]},"reference":[{"issue":"1","key":"2024011000105738900_tzad009-B1","doi-asserted-by":"crossref","first-page":"171","DOI":"10.1007\/s13244-016-0534-1","article-title":"Error and discrepancy in radiology: inevitable or avoidable?","volume":"8","author":"Brady","year":"2017","journal-title":"Insights Imaging"},{"issue":"6","key":"2024011000105738900_tzad009-B2","doi-asserted-by":"crossref","first-page":"350","DOI":"10.1097\/RTI.0000000000000361","article-title":"Radiologist performance in the detection of pulmonary embolism: features that favor correct interpretation and risk factors for errors","volume":"33","author":"Kligerman","year":"2018","journal-title":"J Thorac Imaging"},{"issue":"1150","key":"2024011000105738900_tzad009-B3","doi-asserted-by":"crossref","first-page":"20221031","DOI":"10.1259\/bjr.20221031","article-title":"Clinical applications of artificial intelligence in radiology","volume":"96","author":"Mello-Thoms","year":"2023","journal-title":"Br J Radiol"},{"issue":"1","key":"2024011000105738900_tzad009-B4","doi-asserted-by":"crossref","first-page":"38","DOI":"10.1016\/j.crad.2019.08.005","article-title":"Identifying pulmonary nodules or masses on chest radiography using deep learning: external validation and strategies to improve clinical practice","volume":"75","author":"Liang","year":"2020","journal-title":"Clin Radiol"},{"issue":"8","key":"2024011000105738900_tzad009-B5","doi-asserted-by":"crossref","first-page":"e496","DOI":"10.1016\/S2589-7500(21)00106-0","article-title":"Effect of a comprehensive deep-learning model on the accuracy of chest x-ray interpretation by radiologists: a retrospective, multireader multicase study","volume":"3","author":"Seah","year":"2021","journal-title":"Lancet Digit Health"},{"issue":"1","key":"2024011000105738900_tzad009-B6","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.crad.2022.08.129","article-title":"Validation study of machine-learning chest radiograph software in primary and emergency medicine","volume":"78","author":"van Beek","year":"2023","journal-title":"Clin Radiol"},{"issue":"7","key":"2024011000105738900_tzad009-B7","doi-asserted-by":"crossref","first-page":"4468","DOI":"10.1007\/s00330-022-08593-x","article-title":"Artificial intelligence system for identification of false-negative interpretations in chest radiographs","volume":"32","author":"Hwang","year":"2022","journal-title":"Eur Radiol"},{"issue":"3","key":"2024011000105738900_tzad009-B8","doi-asserted-by":"crossref","first-page":"e180084","DOI":"10.1148\/ryai.2019180084","article-title":"Evaluating a fully automated pulmonary nodule detection approach and its impact on radiologist performance","volume":"1","author":"Liu","year":"2019","journal-title":"Radiol Artif Intell"},{"issue":"2","key":"2024011000105738900_tzad009-B9","doi-asserted-by":"crossref","first-page":"e220163","DOI":"10.1148\/ryct.220163","article-title":"Artificial intelligence tool for detection and worklist prioritization reduces time to diagnosis of incidental pulmonary embolism at CT","volume":"5","author":"Topff","year":"2023","journal-title":"Radiol Cardiothorac Imaging"},{"year":"2023","author":"Radiological Society of North America Clinical Trials Processor, CTP USA","key":"2024011000105738900_tzad009-B10"},{"year":"2023","author":"National Institute of Health-National Library of Medicine Scrubber, USA","key":"2024011000105738900_tzad009-B11"},{"issue":"7522","key":"2024011000105738900_tzad009-B12","doi-asserted-by":"crossref","first-page":"940","DOI":"10.1136\/bmj.38583.625613.AE","article-title":"Recruiting patients to medical research: double blind randomized trial of \u201copt-in\u201d versus \u201copt-out\u201d strategies","volume":"331","author":"Junghans","year":"2005","journal-title":"BMJ"},{"issue":"4","key":"2024011000105738900_tzad009-B13","doi-asserted-by":"crossref","first-page":"932","DOI":"10.1007\/s00330-014-3487-0","article-title":"Disadvantages of using the area under the receiver operating characteristic curve to assess imaging tests: a discussion and proposal for an alternative approach","volume":"25","author":"Halligan","year":"2015","journal-title":"Eur Radiol"},{"issue":"2","key":"2024011000105738900_tzad009-B14","doi-asserted-by":"crossref","first-page":"373","DOI":"10.1093\/oxfordjournals.aje.a115507","article-title":"A simple method to calculate the confidence interval of a standardized mortality ratio (SMR)","volume":"131","author":"Ulm","year":"1990","journal-title":"Am J Epidemiol"},{"issue":"24","key":"2024011000105738900_tzad009-B15","doi-asserted-by":"crossref","first-page":"e334","DOI":"10.1161\/CIR.0000000000001106","article-title":"2022 ACC\/AHA guideline for the diagnosis and management of aortic disease: a report of the American Heart Association\/American College of Cardiology Joint Committee on Clinical Practice Guidelines","volume":"146","author":"Isselbacher","year":"2022","journal-title":"Circulation"},{"year":"2022","author":"Xie","key":"2024011000105738900_tzad009-B16"},{"year":"2021","author":"Xie","key":"2024011000105738900_tzad009-B17"},{"issue":"9","key":"2024011000105738900_tzad009-B18","doi-asserted-by":"crossref","first-page":"5831","DOI":"10.1007\/s00330-022-08645-2","article-title":"How artificial intelligence improves radiological interpretation in suspected embolism","volume":"32","author":"Cheikh","year":"2022","journal-title":"Eur Radiol"},{"issue":"2","key":"2024011000105738900_tzad009-B19","doi-asserted-by":"crossref","first-page":"1185","DOI":"10.1007\/s00330-022-09071-0","article-title":"Incidental pulmonary embolism in patients with cancer: prevalence, underdiagnosis and evaluation of an AI algorithm for automatic detection of pulmonary embolism","volume":"33","author":"Wiklund","year":"2023","journal-title":"Eur Radiol"},{"issue":"3","key":"2024011000105738900_tzad009-B20","doi-asserted-by":"crossref","first-page":"e222268","DOI":"10.1148\/radiol.222268","article-title":"Autonomous chest radiograph reporting using AI: estimation of clinical impact","volume":"307","author":"Plesner","year":"2023","journal-title":"Radiology"}],"container-title":["BJR|Open"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bjro\/article-pdf\/6\/1\/tzad009\/54878887\/tzad009.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bjro\/article-pdf\/6\/1\/tzad009\/54878887\/tzad009.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,1,12]],"date-time":"2024-01-12T14:27:01Z","timestamp":1705069621000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bjro\/article\/7470369"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,12,12]]},"references-count":20,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2023,12,12]]}},"URL":"https:\/\/doi.org\/10.1093\/bjro\/tzad009","relation":{},"ISSN":["2513-9878"],"issn-type":[{"type":"electronic","value":"2513-9878"}],"subject":[],"published-other":{"date-parts":[[2024,1,1]]},"published":{"date-parts":[[2023,12,12]]},"article-number":"tzad009"}}