{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,30]],"date-time":"2026-09-30T19:58:40Z","timestamp":1790798320983,"version":"4.1.0"},"reference-count":28,"publisher":"Springer Science and Business Media LLC","issue":"11","license":[{"start":{"date-parts":[[2024,5,17]],"date-time":"2024-05-17T00:00:00Z","timestamp":1715904000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,5,17]],"date-time":"2024-05-17T00:00:00Z","timestamp":1715904000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Eur Radiol"],"abstract":"<jats:title>Abstract<\/jats:title><jats:sec>\n                <jats:title>Introduction<\/jats:title>\n                <jats:p>This study investigates the performance of a commercially available artificial intelligence (AI) system to identify normal chest radiographs and its potential to reduce radiologist workload.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Methods<\/jats:title>\n                <jats:p>Retrospective analysis included consecutive chest radiographs from two medical centers between Oct 1, 2016 and Oct 14, 2016. Exclusions comprised follow-up exams within the inclusion period, bedside radiographs, incomplete images, imported radiographs, and pediatric radiographs. Three chest radiologists categorized findings into normal, clinically irrelevant, clinically relevant, urgent, and critical. A commercial AI system processed all radiographs, scoring 10 chest abnormalities on a 0\u2013100 confidence scale. AI system performance was evaluated using the area under the ROC curve (AUC), assessing the detection of normal radiographs. Sensitivity was calculated for the default and a conservative operating point. the detection of negative predictive value (NPV) for urgent and critical findings, as well as the potential workload reduction, was calculated.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Results<\/jats:title>\n                <jats:p>A total of 2603 radiographs were acquired in 2141 unique patients. Post-exclusion, 1670 radiographs were analyzed. Categories included 479 normal, 332 clinically irrelevant, 339 clinically relevant, 501 urgent, and 19 critical findings. The AI system achieved an AUC of 0.92. Sensitivity for normal radiographs was 92% at default and 53% at the conservative operating point. At the conservative operating point, NPV was 98% for urgent and critical findings, and could result in a 15% workload reduction.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Conclusion<\/jats:title>\n                <jats:p>A commercially available AI system effectively identifies normal chest radiographs and holds the potential to lessen radiologists\u2019 workload by omitting half of the normal exams from reporting.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Clinical relevance statement<\/jats:title>\n                <jats:p>The AI system is able to detect half of all normal chest radiographs at a clinically acceptable operating point, thereby potentially reducing the workload for the radiologists by 15%.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Key Points<\/jats:title>\n                <jats:p><jats:list list-type=\"bullet\">\n                  <jats:list-item>\n                    <jats:p><jats:italic>The AI system reached an AUC of 0.92 for the detection of normal chest radiographs<\/jats:italic>.<\/jats:p>\n                  <\/jats:list-item>\n                  <jats:list-item>\n                    <jats:p><jats:italic>Fifty-three percent of normal chest radiographs were identified with a NPV of 98% for urgent findings. AI can reduce the workload of chest radiography reporting by 15%<\/jats:italic>.<\/jats:p>\n                  <\/jats:list-item>\n                <\/jats:list><\/jats:p>\n              <\/jats:sec>","DOI":"10.1007\/s00330-024-10794-5","type":"journal-article","created":{"date-parts":[[2024,5,17]],"date-time":"2024-05-17T07:02:01Z","timestamp":1715929321000},"page":"7255-7263","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":30,"title":["Performance of AI to exclude normal chest radiographs to reduce radiologists\u2019 workload"],"prefix":"10.1007","volume":"34","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3984-3596","authenticated-orcid":false,"given":"Steven","family":"Schalekamp","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kicky","family":"van Leeuwen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Erdi","family":"Calli","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Keelin","family":"Murphy","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Matthieu","family":"Rutten","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bram","family":"Geurts","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Liesbeth","family":"Peters-Bax","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bram","family":"van Ginneken","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mathias","family":"Prokop","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,5,17]]},"reference":[{"key":"10794_CR1","doi-asserted-by":"publisher","first-page":"545","DOI":"10.1378\/chest.10-1302","volume":"141","author":"S Raoof","year":"2012","unstructured":"Raoof S, Feigin D, Sung A, Raoof S, Irugulpati L, Rosenow 3rd EC (2012) Interpretation of plain chest roentgenogram. Chest 141:545\u2013558","journal-title":"Chest"},{"key":"10794_CR2","doi-asserted-by":"publisher","unstructured":"van Leeuwen KG, Schalekamp S, Rutten MJCM, van Ginneken B, de Rooij M (2021) Artificial intelligence in radiology: 100 commercially available products and their scientific evidence. Eur Radiol. https:\/\/doi.org\/10.1007\/s00330-021-07892-z","DOI":"10.1007\/s00330-021-07892-z"},{"key":"10794_CR3","doi-asserted-by":"publisher","first-page":"102125","DOI":"10.1016\/j.media.2021.102125","volume":"72","author":"E Calli","year":"2021","unstructured":"Calli E, Sogancioglu E, van Ginneken B, van Leeuwen KG, Murphy K (2021) Deep learning for chest X-ray analysis: a survey. Med Image Anal 72:102125","journal-title":"Med Image Anal"},{"key":"10794_CR4","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejrad.2019.108774","volume":"123","author":"G Chassagnon","year":"2020","unstructured":"Chassagnon G, Vakalopoulou M, Paragios N, Revel MP (2020) Artificial intelligence applications for thoracic imaging. Eur J Radiol 123:108774","journal-title":"Eur J Radiol"},{"key":"10794_CR5","doi-asserted-by":"crossref","unstructured":"Fanni SC, Marcucci A, Volpi F, Valentino S, Neri E, Romei C (2023) Artificial intelligence-based software with CE mark for chest X-ray interpretation: opportunities and challenges. Diagnostics (Basel) 13:2020","DOI":"10.3390\/diagnostics13122020"},{"key":"10794_CR6","doi-asserted-by":"publisher","first-page":"e2229289","DOI":"10.1001\/jamanetworkopen.2022.29289","volume":"5","author":"JS Ahn","year":"2022","unstructured":"Ahn JS, Ebrahimian S, McDermott S et al (2022) Association of artificial intelligence-aided chest radiograph interpretation with reader performance and efficiency. JAMA Netw Open 5:e2229289","journal-title":"JAMA Netw Open"},{"key":"10794_CR7","doi-asserted-by":"publisher","first-page":"e2141096","DOI":"10.1001\/jamanetworkopen.2021.41096","volume":"4","author":"F Homayounieh","year":"2021","unstructured":"Homayounieh F, Digumarthy S, Ebrahimian S et al (2021) An artificial intelligence-based chest X-ray model on human nodule detection accuracy from a multicenter study. JAMA Netw Open 4:e2141096","journal-title":"JAMA Netw Open"},{"key":"10794_CR8","doi-asserted-by":"publisher","first-page":"573","DOI":"10.1148\/radiol.2019191225","volume":"293","author":"EJ Hwang","year":"2019","unstructured":"Hwang EJ, Nam JG, Lim WH et al (2019) Deep learning for chest radiograph diagnosis in the emergency department. Radiology 293:573\u2013580","journal-title":"Radiology"},{"key":"10794_CR9","doi-asserted-by":"publisher","first-page":"e191095","DOI":"10.1001\/jamanetworkopen.2019.1095","volume":"2","author":"EJ Hwang","year":"2019","unstructured":"Hwang EJ, Park S, Jin K-N et al (2019) Development and validation of a deep learning-based automated detection algorithm for major thoracic diseases on chest radiographs. JAMA Netw Open 2:e191095","journal-title":"JAMA Netw Open"},{"key":"10794_CR10","doi-asserted-by":"publisher","first-page":"252","DOI":"10.1148\/radiol.14131315","volume":"272","author":"S Schalekamp","year":"2014","unstructured":"Schalekamp S, van Ginneken B, Koedam E et al (2014) Computer-aided detection improves detection of pulmonary nodules in chest radiographs beyond the support by bone-suppressed images. Radiology 272:252\u2013261","journal-title":"Radiology"},{"key":"10794_CR11","doi-asserted-by":"publisher","first-page":"e0282123","DOI":"10.1371\/journal.pone.0282123","volume":"18","author":"HJ Shin","year":"2023","unstructured":"Shin HJ, Lee S, Kim S, Son NH, Kim EK (2023) Hospital-wide survey of clinical experience with artificial intelligence applied to daily chest radiographs. PLoS One 18:e0282123","journal-title":"PLoS One"},{"key":"10794_CR12","doi-asserted-by":"publisher","first-page":"272","DOI":"10.1148\/radiol.2019194005","volume":"291","author":"M Annarumma","year":"2019","unstructured":"Annarumma M, Withey SJ, Bakewell RJ, Pesce E, Goh V, Montana G (2019) Automated triaging of adult chest radiographs with deep artificial neural networks. Radiology 291:272","journal-title":"Radiology"},{"key":"10794_CR13","doi-asserted-by":"publisher","first-page":"537","DOI":"10.1148\/radiol.2018181422","volume":"290","author":"JA Dunnmon","year":"2019","unstructured":"Dunnmon JA, Yi D, Langlotz CP, Re C, Rubin DL, Lungren MP (2019) Assessment of convolutional neural networks for automated classification of chest radiographs. Radiology 290:537\u2013544","journal-title":"Radiology"},{"key":"10794_CR14","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-021-93967-2","volume":"11","author":"Z Nabulsi","year":"2021","unstructured":"Nabulsi Z, Sellergren A, Jamshy S et al (2021) Deep learning for distinguishing normal versus abnormal chest radiographs and generalization to two unseen diseases tuberculosis and COVID-19. Sci Rep 11:15523","journal-title":"Sci Rep"},{"key":"10794_CR15","doi-asserted-by":"publisher","first-page":"1006","DOI":"10.1016\/j.jacr.2022.06.009","volume":"19","author":"A Khurana","year":"2022","unstructured":"Khurana A, Patel B, Sharpe Jr R (2022) Geographic variations in growth of radiologists and medicare enrollees from 2012 to 2019. J Am Coll Radiol 19:1006\u20131014","journal-title":"J Am Coll Radiol"},{"key":"10794_CR16","unstructured":"The Royal College of Radiologists (2018) Clinical radiology UK workforce census report 2018. https:\/\/www.rcr.ac.uk\/publication\/clinical-radiology-uk-workforce-census-report-2018"},{"key":"10794_CR17","doi-asserted-by":"publisher","DOI":"10.1186\/s12960-020-0448-3","volume":"18","author":"X Zhang","year":"2020","unstructured":"Zhang X, Lin D, Pforsich H, Lin VW (2020) Physician workforce in the United States of America: forecasting nationwide shortages. Hum Resour Health 18:8","journal-title":"Hum Resour Health"},{"key":"10794_CR18","doi-asserted-by":"crossref","unstructured":"Bruls RJM, Kwee RM (2020) Workload for radiologists during on-call hours: dramatic increase in the past 15 years. Insights Imaging 11:121","DOI":"10.1186\/s13244-020-00925-z"},{"key":"10794_CR19","doi-asserted-by":"crossref","unstructured":"Lantsman CD, Barash Y, Klang E, Guranda L, Konen E, Tau N (2022) Trend in radiologist workload compared to number of admissions in the emergency department. Eur J Radiol 149:110195","DOI":"10.1016\/j.ejrad.2022.110195"},{"key":"10794_CR20","doi-asserted-by":"crossref","unstructured":"Kwee TC, Kwee RM (2021) Workload of diagnostic radiologists in the foreseeable future based on recent scientific advances: growth expectations and role of artificial intelligence. Insights Imaging 12:88","DOI":"10.1186\/s13244-021-01031-4"},{"key":"10794_CR21","doi-asserted-by":"publisher","first-page":"473.e479","DOI":"10.1016\/j.crad.2021.01.015","volume":"76","author":"T Dyer","year":"2021","unstructured":"Dyer T, Dillard L, Harrison M et al (2021) Diagnosis of normal chest radiographs using an autonomous deep-learning algorithm. Clin Radiol 76:473.e479\u2013473.e415","journal-title":"Clin Radiol"},{"key":"10794_CR22","doi-asserted-by":"publisher","unstructured":"Keski-Filppula T, Nikki M, Haapea M, Ramanauskas N, Tervonen O (2022) Using artificial intelligence to detect chest X-rays with no significant findings in a primary health care setting in Oulu, Finland. https:\/\/doi.org\/10.48550\/arXiv.2205.08123","DOI":"10.48550\/arXiv.2205.08123"},{"key":"10794_CR23","doi-asserted-by":"publisher","first-page":"e222268","DOI":"10.1148\/radiol.222268","volume":"307","author":"LL Plesner","year":"2023","unstructured":"Plesner LL, Muller FC, Nybing JD et al (2023) Autonomous chest radiograph reporting using AI: estimation of clinical impact. Radiology 307:e222268","journal-title":"Radiology"},{"key":"10794_CR24","doi-asserted-by":"publisher","first-page":"826","DOI":"10.1148\/radiol.2015151516","volume":"277","author":"Bossuyt","year":"2015","unstructured":"Bossuyt, Reitsma PM, Bruns DE JB et al (2015) STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. Radiology 277:826\u2013832","journal-title":"Radiology"},{"key":"10794_CR25","doi-asserted-by":"crossref","unstructured":"Kaviani P, Kalra MK, Digumarthy SR et al (2022) Frequency of missed findings on chest radiographs (CXRs) in an International, Multicenter Study: application of AI to reduce missed findings. Diagnostics (Basel) 12:2382","DOI":"10.3390\/diagnostics12102382"},{"key":"10794_CR26","doi-asserted-by":"publisher","first-page":"e827","DOI":"10.3399\/bjgp19X706853","volume":"69","author":"SH Bradley","year":"2019","unstructured":"Bradley SH, Abraham S, Callister ME et al (2019) Sensitivity of chest X-ray for detecting lung cancer in people presenting with symptoms: a systematic review. Br J Gen Pract 69:e827\u2013e835","journal-title":"Br J Gen Pract"},{"key":"10794_CR27","doi-asserted-by":"publisher","first-page":"310","DOI":"10.1016\/j.ajem.2021.06.034","volume":"49","author":"DG Dillon","year":"2021","unstructured":"Dillon DG, Rodriguez RM (2021) Screening performance of the chest X-ray in adult blunt trauma evaluation: Is it effective and what does it miss? Am J Emerg Med 49:310\u2013314","journal-title":"Am J Emerg Med"},{"key":"10794_CR28","doi-asserted-by":"publisher","first-page":"720","DOI":"10.1378\/chest.115.3.720","volume":"115","author":"LG Quekel","year":"1999","unstructured":"Quekel LG, Kessels AG, Goei R, van Engelshoven JM (1999) Miss rate of lung cancer on the chest radiograph in clinical practice. Chest 115:720\u2013724","journal-title":"Chest"}],"container-title":["European Radiology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00330-024-10794-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00330-024-10794-5\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00330-024-10794-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,28]],"date-time":"2024-10-28T15:07:09Z","timestamp":1730128029000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s00330-024-10794-5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,5,17]]},"references-count":28,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2024,11]]}},"alternative-id":["10794"],"URL":"https:\/\/doi.org\/10.1007\/s00330-024-10794-5","relation":{},"ISSN":["1432-1084"],"issn-type":[{"value":"1432-1084","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,5,17]]},"assertion":[{"value":"23 February 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 April 2024","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"22 April 2024","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"17 May 2024","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Compliance with ethical standards"}},{"value":"The scientific guarantor of this publication is Steven Schalekamp","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Guarantor"}},{"value":"The authors of this manuscript declare no relationships with any companies, whose products or services may be related to the subject matter of the article.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"No complex statistical methods were necessary for this paper.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Statistics and biometry"}},{"value":"Written informed consent was waived by the Institutional Review Board.","order":5,"name":"Ethics","group":{"name":"EthicsHeading","label":"Informed consent"}},{"value":"Institutional Review Board approval was obtained. Institutional Review Board approval was not required because of the use of de-identified retrospective data (local institution review board waiver number 2023-16333).","order":6,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical approval"}},{"value":"Not applicable.","order":7,"name":"Ethics","group":{"name":"EthicsHeading","label":"Study subjects or cohorts overlap"}},{"order":8,"name":"Ethics","group":{"name":"EthicsHeading","label":"Methodology"}}]}}