{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,5]],"date-time":"2026-11-05T00:04:23Z","timestamp":1793837063385,"version":"4.6.1"},"reference-count":34,"publisher":"BMJ","issue":"11","license":[{"start":{"date-parts":[[2023,11,8]],"date-time":"2023-11-08T00:00:00Z","timestamp":1699401600000},"content-version":"unspecified","delay-in-days":7,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100010269","name":"Wellcome Trust","doi-asserted-by":"publisher","award":["205188\/Z\/16\/Z"],"award-info":[{"award-number":["205188\/Z\/16\/Z"]}],"id":[{"id":"10.13039\/100010269","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100010269","name":"Wellcome Trust","doi-asserted-by":"publisher","award":["222930\/Z\/21\/Z"],"award-info":[{"award-number":["222930\/Z\/21\/Z"]}],"id":[{"id":"10.13039\/100010269","id-type":"DOI","asserted-by":"publisher"}]},{"name":"4ward North","award":["203914\/Z\/16\/"],"award-info":[{"award-number":["203914\/Z\/16\/"]}]}],"content-domain":{"domain":["bmj.com"],"crossmark-restriction":true},"short-container-title":["BMJ Open"],"accepted":{"date-parts":[[2023,10,16]]},"published-print":{"date-parts":[[2023,11]]},"abstract":"<jats:sec>\n                    <jats:title>Objectives<\/jats:title>\n                    <jats:p>Early identification of lung cancer on chest radiographs improves patient outcomes. Artificial intelligence (AI) tools may increase diagnostic accuracy and streamline this pathway. This study evaluated the performance of commercially available AI-based software trained to identify cancerous lung nodules on chest radiographs.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Design<\/jats:title>\n                    <jats:p>This retrospective study included primary care chest radiographs acquired in a UK centre. The software evaluated each radiograph independently and outputs were compared with two reference standards: (1) the radiologist report and (2) the diagnosis of cancer by multidisciplinary team decision. Failure analysis was performed by interrogating the software marker locations on radiographs.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Participants<\/jats:title>\n                    <jats:p>5722 consecutive chest radiographs were included from 5592 patients (median age 59 years, 53.8% women, 1.6% prevalence of cancer).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Compared with radiologist reports for nodule detection, the software demonstrated sensitivity 54.5% (95% CI 44.2% to 64.4%), specificity 83.2% (82.2% to 84.1%), positive predictive value (PPV) 5.5% (4.6% to 6.6%) and negative predictive value (NPV) 99.0% (98.8% to 99.2%). Compared with cancer diagnosis, the software demonstrated sensitivity 60.9% (50.1% to 70.9%), specificity 83.3% (82.3% to 84.2%), PPV 5.6% (4.8% to 6.6%) and NPV 99.2% (99.0% to 99.4%). Normal or variant anatomy was misidentified as an abnormality in 69.9% of the 943 false positive cases.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>The software demonstrated considerable underperformance in this real-world patient cohort. Failure analysis suggested a lack of generalisability in the training and testing datasets as a potential factor. The low PPV carries the risk of over-investigation and limits the translation of the software to clinical practice. Our findings highlight the importance of training and testing software in representative datasets, with broader implications for the implementation of AI tools in imaging.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1136\/bmjopen-2023-077348","type":"journal-article","created":{"date-parts":[[2023,11,8]],"date-time":"2023-11-08T11:51:12Z","timestamp":1699444272000},"page":"e077348","update-policy":"https:\/\/doi.org\/10.1136\/crossmarkpolicy","source":"Crossref","is-referenced-by-count":20,"title":["Evaluating the performance of artificial intelligence software for lung nodule detection on chest radiographs in a retrospective real-world UK population"],"prefix":"10.1136","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4999-2608","authenticated-orcid":false,"given":"Ahmed","family":"Maiter","sequence":"first","affiliation":[{"name":"School of Medicine and Population Health, The University of Sheffield, Sheffield, UK"},{"name":"Radiology, Sheffield Teaching Hospitals NHS Foundation Trust, Sheffield, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Katherine","family":"Hocking","sequence":"additional","affiliation":[{"name":"Radiology, Sheffield Teaching Hospitals NHS Foundation Trust, Sheffield, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Suzanne","family":"Matthews","sequence":"additional","affiliation":[{"name":"Radiology, Sheffield Teaching Hospitals NHS Foundation Trust, Sheffield, UK"},{"name":"Medical Imaging and Medical Physics, Sheffield Teaching Hospitals NHS Foundation Trust, Sheffield, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jonathan","family":"Taylor","sequence":"additional","affiliation":[{"name":"Medical Imaging and Medical Physics, Sheffield Teaching Hospitals NHS Foundation Trust, Sheffield, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael","family":"Sharkey","sequence":"additional","affiliation":[{"name":"Medical Imaging and Medical Physics, Sheffield Teaching Hospitals NHS Foundation Trust, Sheffield, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Peter","family":"Metherall","sequence":"additional","affiliation":[{"name":"Medical Imaging and Medical Physics, Sheffield Teaching Hospitals NHS Foundation Trust, Sheffield, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Samer","family":"Alabed","sequence":"additional","affiliation":[{"name":"School of Medicine and Population Health, The University of Sheffield, Sheffield, UK"},{"name":"Radiology, Sheffield Teaching Hospitals NHS Foundation Trust, Sheffield, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Krit","family":"Dwivedi","sequence":"additional","affiliation":[{"name":"School of Medicine and Population Health, The University of Sheffield, Sheffield, UK"},{"name":"Radiology, Sheffield Teaching Hospitals NHS Foundation Trust, Sheffield, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yousef","family":"Shahin","sequence":"additional","affiliation":[{"name":"School of Medicine and Population Health, The University of Sheffield, Sheffield, UK"},{"name":"Radiology, Sheffield Teaching Hospitals NHS Foundation Trust, Sheffield, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Elizabeth","family":"Anderson","sequence":"additional","affiliation":[{"name":"Radiology, Sheffield Teaching Hospitals NHS Foundation Trust, Sheffield, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sarah","family":"Holt","sequence":"additional","affiliation":[{"name":"Radiology, Sheffield Teaching Hospitals NHS Foundation Trust, Sheffield, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Charlotte","family":"Rowbotham","sequence":"additional","affiliation":[{"name":"Radiology, Sheffield Teaching Hospitals NHS Foundation Trust, Sheffield, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mohamed A","family":"Kamil","sequence":"additional","affiliation":[{"name":"Radiology, Sheffield Teaching Hospitals NHS Foundation Trust, Sheffield, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nigel","family":"Hoggard","sequence":"additional","affiliation":[{"name":"School of Medicine and Population Health, The University of Sheffield, Sheffield, UK"},{"name":"Radiology, Sheffield Teaching Hospitals NHS Foundation Trust, Sheffield, UK"},{"name":"NIHR Sheffield Biomedical Research Centre, Sheffield, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Saba P","family":"Balasubramanian","sequence":"additional","affiliation":[{"name":"Medical Imaging and Medical Physics, Sheffield Teaching Hospitals NHS Foundation Trust, Sheffield, UK"},{"name":"Surgical directorate, Sheffield Teaching Hospitals Foundation NHS Trust, Sheffield, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andrew","family":"Swift","sequence":"additional","affiliation":[{"name":"School of Medicine and Population Health, The University of Sheffield, Sheffield, UK"},{"name":"Radiology, Sheffield Teaching Hospitals NHS Foundation Trust, Sheffield, UK"},{"name":"NIHR Sheffield Biomedical Research Centre, Sheffield, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3724-0430","authenticated-orcid":false,"given":"Christopher S","family":"Johns","sequence":"additional","affiliation":[{"name":"Radiology, Sheffield Teaching Hospitals NHS Foundation Trust, Sheffield, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"239","published-online":{"date-parts":[[2023,11,8]]},"reference":[{"key":"2025120600242413000_13.11.e077348.1","doi-asserted-by":"publisher","DOI":"10.1016\/j.chest.2017.07.032"},{"key":"2025120600242413000_13.11.e077348.2","doi-asserted-by":"crossref","first-page":"282","DOI":"10.1016\/S2213-2600(15)00029-6","article-title":"Lung cancer diagnosis and staging with endobronchial ultrasound-guided transbronchial needle aspiration compared with conventional approaches: an open-label, pragmatic, randomised controlled trial","volume":"3","author":"Navani","year":"2015","journal-title":"Lancet Respir Med"},{"key":"2025120600242413000_13.11.e077348.3","unstructured":"Cancer Research UK . Survival | lung cancer. Available: https:\/\/www.cancerresearchuk.org\/about-cancer\/lung-cancer\/survival [Accessed 10 Dec 2022]."},{"key":"2025120600242413000_13.11.e077348.4","article-title":"Clinical characteristics and prognosis of incidentally detected lung cancers","volume":"2015","author":"Quadrelli","year":"2015","journal-title":"Int J Surg Oncol"},{"key":"2025120600242413000_13.11.e077348.5","unstructured":"National Institute for Health and Care Excellence (NICE) . Lung cancer: diagnosis and management. Available: https:\/\/www.nice.org.uk\/guidance\/ng122 [Accessed 10 Dec 2022]."},{"key":"2025120600242413000_13.11.e077348.6","unstructured":"NHS England . Rapid cancer diagnostic and assessment pathways. Available: https:\/\/www.england.nhs.uk\/publication\/rapid-cancer-diagnostic-and-assessment-pathways\/ [Accessed 10 Dec 2022]."},{"key":"2025120600242413000_13.11.e077348.7","unstructured":"The Royal College of Radiologists . Clinical radiology census report 2021. 2022. Available: https:\/\/www.rcr.ac.uk\/clinical-radiology\/rcr-clinical-radiology-census-report-2021 [Accessed 5 Jan 2023]."},{"key":"2025120600242413000_13.11.e077348.8","doi-asserted-by":"publisher","DOI":"10.1038\/s41568-018-0016-5"},{"key":"2025120600242413000_13.11.e077348.9","doi-asserted-by":"crossref","DOI":"10.1038\/s41571-020-0329-7","article-title":"AI Outperforms Radiologists in Mammographic screening","volume":"17","author":"Killock","year":"2020","journal-title":"Nat Rev Clin Oncol"},{"key":"2025120600242413000_13.11.e077348.10","doi-asserted-by":"crossref","first-page":"44","DOI":"10.1186\/s13244-019-0738-2","article-title":"What the radiologist should know about artificial intelligence \u2013 an ESR white paper","volume":"10","author":"Neri","year":"2019","journal-title":"Insights Imaging"},{"key":"2025120600242413000_13.11.e077348.11","doi-asserted-by":"crossref","DOI":"10.1001\/jamanetworkopen.2019.1095","article-title":"Development and validation of a deep learning\u2013based automated detection algorithm for major thoracic diseases on chest radiographs","volume":"2","author":"Hwang","year":"2019","journal-title":"JAMA Netw Open"},{"key":"2025120600242413000_13.11.e077348.12","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pone.0204155","article-title":"Deep learning in chest radiography: detection of findings and presence of change","volume":"13","author":"Singh","year":"2018","journal-title":"PLoS One"},{"key":"2025120600242413000_13.11.e077348.13","doi-asserted-by":"crossref","DOI":"10.1097\/MD.0000000000025663","article-title":"Evaluation of a deep learning-based computer-aided detection algorithm on chest radiographs: case-control study","volume":"100","author":"Choi","year":"2021","journal-title":"Medicine (Baltimore)"},{"key":"2025120600242413000_13.11.e077348.14","doi-asserted-by":"crossref","DOI":"10.1183\/13993003.03061-2020","article-title":"Development and validation of a deep learning algorithm detecting 10 common abnormalities on chest radiographs","volume":"57","author":"Nam","year":"2021","journal-title":"Eur Respir J"},{"key":"2025120600242413000_13.11.e077348.15","doi-asserted-by":"crossref","DOI":"10.1038\/s41598-021-03265-0","article-title":"Independent evaluation of 12 artificial intelligence solutions for the detection of tuberculosis","volume":"11","author":"Codlin","year":"2021","journal-title":"Sci Rep"},{"key":"2025120600242413000_13.11.e077348.16","doi-asserted-by":"crossref","first-page":"1069","DOI":"10.1007\/s00330-020-07219-4","article-title":"Deep learning-based automated detection algorithm for active pulmonary tuberculosis on chest radiographs: diagnostic performance in systematic screening of asymptomatic individuals","volume":"31","author":"Lee","year":"2021","journal-title":"Eur Radiol"},{"key":"2025120600242413000_13.11.e077348.17","doi-asserted-by":"crossref","DOI":"10.1038\/s41598-019-56589-3","article-title":"Deep learning, computer-aided radiography reading for tuberculosis: a diagnostic accuracy study from a tertiary hospital in India","volume":"10","author":"Nash","year":"2020","journal-title":"Sci Rep"},{"key":"2025120600242413000_13.11.e077348.18","doi-asserted-by":"crossref","DOI":"10.1001\/jamanetworkopen.2020.17135","article-title":"Validation of a deep learning algorithm for the detection of malignant pulmonary nodules in chest radiographs","volume":"3","author":"Yoo","year":"2020","journal-title":"JAMA Netw Open"},{"key":"2025120600242413000_13.11.e077348.19","doi-asserted-by":"crossref","first-page":"15","DOI":"10.1111\/1754-9485.13105","article-title":"Extravalidation and reproducibility results of a commercial deep learning-based automatic detection algorithm for pulmonary nodules on chest radiographs at tertiary hospital","volume":"65","author":"Koo","year":"2021","journal-title":"J Med Imaging Radiat Oncol"},{"key":"2025120600242413000_13.11.e077348.20","doi-asserted-by":"publisher","DOI":"10.1148\/radiol.2018180237"},{"key":"2025120600242413000_13.11.e077348.21","doi-asserted-by":"publisher","DOI":"10.1016\/j.crad.2021.02.005"},{"key":"2025120600242413000_13.11.e077348.22","doi-asserted-by":"crossref","DOI":"10.1148\/radiol.2019194005","article-title":"Automated Triaging of adult chest Radiographs with deep artificial neural networks","volume":"291","author":"Annarumma","year":"2019","journal-title":"Radiology"},{"key":"2025120600242413000_13.11.e077348.23","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1148\/radiol.2019182465","article-title":"Deep Convolutional neural network\u2013based software improves Radiologist detection of malignant lung nodules on chest Radiographs","volume":"294","author":"Sim","year":"2020","journal-title":"Radiology"},{"key":"2025120600242413000_13.11.e077348.24","doi-asserted-by":"crossref","first-page":"210","DOI":"10.1148\/radiol.2019192252","article-title":"Medical image perception research in the emerging age of artificial intelligence","volume":"294","author":"Jacobson","year":"2020","journal-title":"Radiology"},{"key":"2025120600242413000_13.11.e077348.25","doi-asserted-by":"crossref","DOI":"10.1148\/ryai.2020200029","article-title":"Checklist for artificial intelligence in medical imaging (CLAIM): A guide for authors and reviewers","volume":"2","author":"Mongan","year":"2020","journal-title":"Radiol Artif Intell"},{"key":"2025120600242413000_13.11.e077348.26","doi-asserted-by":"publisher","DOI":"10.1056\/NEJMoa1208962"},{"key":"2025120600242413000_13.11.e077348.27","doi-asserted-by":"crossref","first-page":"e406","DOI":"10.1016\/S2589-7500(22)00063-2","article-title":"AI recognition of patient race in medical imaging: a modelling study","volume":"4","author":"Gichoya","year":"2022","journal-title":"Lancet Digit Health"},{"key":"2025120600242413000_13.11.e077348.28","doi-asserted-by":"crossref","first-page":"e399","DOI":"10.1016\/S2589-7500(22)00092-9","article-title":"AI models in health care are not colour blind and we should not be either","volume":"4","author":"Wiens","year":"2022","journal-title":"Lancet Digit Health"},{"key":"2025120600242413000_13.11.e077348.29","unstructured":"NHS England . Diagnostic imaging dataset 2021-22 data. Available: https:\/\/www.england.nhs.uk\/statistics\/statistical-work-areas\/diagnostic-imaging-dataset\/diagnostic-imaging-dataset-2021-22-data\/ [Accessed 5 Jan 2023]."},{"key":"2025120600242413000_13.11.e077348.30","doi-asserted-by":"crossref","first-page":"2176","DOI":"10.1038\/s41591-021-01595-0","article-title":"Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations","volume":"27","author":"Seyyed-Kalantari","year":"2021","journal-title":"Nat Med"},{"key":"2025120600242413000_13.11.e077348.31","doi-asserted-by":"publisher","DOI":"10.1056\/NEJMp1714229"},{"key":"2025120600242413000_13.11.e077348.32","doi-asserted-by":"crossref","DOI":"10.3389\/fcvm.2022.956811","article-title":"Quality of reporting in AI cardiac MRI segmentation studies - a systematic review and recommendations for future studies","volume":"9","author":"Alabed","year":"2022","journal-title":"Front Cardiovasc Med"},{"key":"2025120600242413000_13.11.e077348.33","doi-asserted-by":"crossref","first-page":"607","DOI":"10.1016\/j.crad.2021.03.021","article-title":"Augmenting lung cancer diagnosis on chest radiographs: positioning artificial intelligence to improve radiologist performance","volume":"76","author":"Tam","year":"2021","journal-title":"Clin Radiol"},{"key":"2025120600242413000_13.11.e077348.34","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-019-1799-6"}],"container-title":["BMJ Open"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.1136\/bmjopen-2023-077348","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,12,6]],"date-time":"2025-12-06T08:24:37Z","timestamp":1765009477000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmjopen.bmj.com\/lookup\/doi\/10.1136\/bmjopen-2023-077348"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,11]]},"references-count":34,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2023,11,8]]},"published-print":{"date-parts":[[2023,11]]}},"alternative-id":["10.1136\/bmjopen-2023-077348"],"URL":"https:\/\/doi.org\/10.1136\/bmjopen-2023-077348","relation":{},"ISSN":["2044-6055","2044-6055"],"issn-type":[{"value":"2044-6055","type":"print"},{"value":"2044-6055","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,11]]}}}