{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,23]],"date-time":"2026-09-23T13:19:55Z","timestamp":1790169595257,"version":"4.1.0"},"reference-count":28,"publisher":"Springer Science and Business Media LLC","issue":"5","license":[{"start":{"date-parts":[[2022,1,1]],"date-time":"2022-01-01T00:00:00Z","timestamp":1640995200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,1,1]],"date-time":"2022-01-01T00:00:00Z","timestamp":1640995200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100003710","name":"Korea Health Industry Development Institute","doi-asserted-by":"publisher","award":["HI19C0847"],"award-info":[{"award-number":["HI19C0847"]}],"id":[{"id":"10.13039\/501100003710","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Eur Radiol"],"published-print":{"date-parts":[[2022,5]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec>\n                <jats:title>Objectives<\/jats:title>\n                <jats:p>We aim\ned to evaluate a commercial artificial intelligence (AI) solution on a multicenter cohort of chest radiographs and to compare physicians' ability to detect and localize referable thoracic abnormalities with and without AI assistance.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Methods<\/jats:title>\n                <jats:p>In this retrospective diagnostic cohort study, we investigated 6,006 consecutive patients who underwent both chest radiography and CT. We evaluated a commercially available AI solution intended to facilitate the detection of three chest abnormalities (nodule\/masses, consolidation, and pneumothorax) against a reference standard to measure its diagnostic performance. Moreover, twelve physicians, including thoracic radiologists, board-certified radiologists, radiology residents, and pulmonologists, assessed a dataset of 230 randomly sampled chest radiographic images. The images were reviewed twice per physician, with and without AI, with a 4-week washout period. We measured the impact of AI assistance on observer's AUC, sensitivity, specificity, and the area under the alternative free-response ROC (AUAFROC).<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Results<\/jats:title>\n                <jats:p>In the entire set (<jats:italic>n<\/jats:italic>\u2009=\u20096,006), the AI solution showed average sensitivity, specificity, and AUC of 0.885, 0.723, and 0.867, respectively. In the test dataset (<jats:italic>n<\/jats:italic>\u2009=\u2009230), the average AUC and AUAFROC across observers significantly increased with AI assistance (from 0.861 to 0.886; <jats:italic>p<\/jats:italic>\u2009=\u20090.003 and from 0.797 to 0.822; <jats:italic>p<\/jats:italic>\u2009=\u20090.003, respectively).<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Conclusions<\/jats:title>\n                <jats:p>The diagnostic performance of the AI solution was found to be acceptable for the images from respiratory outpatient clinics. The diagnostic performance of physicians marginally improved with the use of AI solutions. Further evaluation of AI assistance for chest radiographs using a prospective design is required to prove the efficacy of AI assistance.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Key Points<\/jats:title>\n                <jats:p>\u2022 <jats:italic>AI assistance for chest radiographs marginally improved physicians\u2019 performance in detecting and localizing referable thoracic abnormalities on chest radiographs.<\/jats:italic><\/jats:p>\n                <jats:p>\u2022 <jats:italic>The detection or localization of referable thoracic abnormalities by pulmonologists and radiology residents improved with the use of AI assistance.<\/jats:italic><\/jats:p>\n              <\/jats:sec>","DOI":"10.1007\/s00330-021-08397-5","type":"journal-article","created":{"date-parts":[[2022,1,1]],"date-time":"2022-01-01T07:16:29Z","timestamp":1641021389000},"page":"3469-3479","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":35,"title":["Diagnostic effect of artificial intelligence solution for referable thoracic abnormalities on chest radiography: a multicenter respiratory outpatient diagnostic cohort study"],"prefix":"10.1007","volume":"32","author":[{"given":"Kwang Nam","family":"Jin","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Eun Young","family":"Kim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Young Jae","family":"Kim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gi Pyo","family":"Lee","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hyungjin","family":"Kim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sohee","family":"Oh","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yong Suk","family":"Kim","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ju Hyuck","family":"Han","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0632-1011","authenticated-orcid":false,"given":"Young Jun","family":"Cho","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2022,1,1]]},"reference":[{"key":"8397_CR1","doi-asserted-by":"publisher","first-page":"47","DOI":"10.1097\/00004032-200307000-00011","volume":"85","author":"E Ron","year":"2003","unstructured":"Ron E (2003) Cancer risks from medical radiation. Health Phys 85:47\u201359","journal-title":"Health Phys"},{"key":"8397_CR2","unstructured":"National Council on Radiation Protection and Measurements (2009) Ionizing Radiation Exposure of the Population of the United States (NCRP Report No. 160). National Council on Radiation Protection and Measurements, Bethesda"},{"key":"8397_CR3","doi-asserted-by":"publisher","first-page":"688","DOI":"10.1378\/chest.130.3.688","volume":"130","author":"PF Pinsky","year":"2006","unstructured":"Pinsky PF, Freedman M, Kvale P et al (2006) Abnormalities on chest radiograph reported in subjects in a cancer screening trial. Chest 130:688\u2013693","journal-title":"Chest"},{"key":"8397_CR4","doi-asserted-by":"publisher","first-page":"54","DOI":"10.2214\/AJR.13.10888","volume":"202","author":"BP Little","year":"2014","unstructured":"Little BP, Gilman MD, Humphrey KL et al (2014) Outcome of recommendations for radiographic follow-up of pneumonia on outpatient chest radiography. AJR Am J Roentgenol 202:54\u201359","journal-title":"AJR Am J Roentgenol"},{"key":"8397_CR5","doi-asserted-by":"publisher","first-page":"938","DOI":"10.1053\/crad.2001.0858","volume":"56","author":"R Fitzgerald","year":"2001","unstructured":"Fitzgerald R (2001) Error in radiology. Clin Radiol 56:938\u2013946","journal-title":"Clin Radiol"},{"key":"8397_CR6","doi-asserted-by":"publisher","first-page":"173","DOI":"10.1111\/j.1754-9485.2012.02348.x","volume":"56","author":"JJ Donald","year":"2012","unstructured":"Donald JJ, Barnard SA (2012) Common patterns in 558 diagnostic radiology errors. J Med Imaging Radiat Oncol 56:173\u2013178","journal-title":"J Med Imaging Radiat Oncol"},{"key":"8397_CR7","doi-asserted-by":"publisher","first-page":"453","DOI":"10.1148\/radiol.2532090200","volume":"253","author":"CL Sistrom","year":"2009","unstructured":"Sistrom CL, Dreyer KJ, Dang PP et al (2009) Recommendations for additional imaging in radiology reports: multifactorial analysis of 5.9 million examinations. Radiology 253:453\u2013461","journal-title":"Radiology"},{"key":"8397_CR8","doi-asserted-by":"publisher","first-page":"262","DOI":"10.1148\/radiol.14140583","volume":"275","author":"HB Harvey","year":"2015","unstructured":"Harvey HB, Gilman MD, Wu CC et al (2015) Diagnostic yield of recommendations for chest CT examination prompted by outpatient chest radiographic findings. Radiology 275:262\u2013271","journal-title":"Radiology"},{"key":"8397_CR9","doi-asserted-by":"publisher","first-page":"218","DOI":"10.1148\/radiol.2018180237","volume":"290","author":"JG Nam","year":"2019","unstructured":"Nam JG, Park S, Hwang EJ et al (2019) Development and validation of deep learning-based automatic detection algorithm for malignant pulmonary nodules on chest radiographs. Radiology 290:218\u2013228","journal-title":"Radiology"},{"key":"8397_CR10","doi-asserted-by":"publisher","first-page":"739","DOI":"10.1093\/cid\/ciy967","volume":"69","author":"EJ Hwang","year":"2019","unstructured":"Hwang EJ, Park S, Jin KN et al (2019) Development and validation of a deep learning-based automatic detection algorithm for active pulmonary tuberculosis on chest radiographs. Clin Infect Dis 69:739\u2013747","journal-title":"Clin Infect Dis"},{"key":"8397_CR11","doi-asserted-by":"publisher","first-page":"e191095","DOI":"10.1001\/jamanetworkopen.2019.1095","volume":"2","author":"EJ Hwang","year":"2019","unstructured":"Hwang EJ, Park S, Jin KN et al (2019) Development and validation of a deep learning-based automated detection algorithm for major thoracic diseases on chest radiographs. JAMA Netw Open 2:e191095","journal-title":"JAMA Netw Open"},{"key":"8397_CR12","doi-asserted-by":"publisher","first-page":"800","DOI":"10.1148\/radiol.2017171920","volume":"286","author":"SH Park","year":"2018","unstructured":"Park SH, Han K (2018) Methodologic guide for evaluating clinical performance and effect of artificial intelligence technology for medical diagnosis and prediction. Radiology 286:800\u2013809","journal-title":"Radiology"},{"key":"8397_CR13","doi-asserted-by":"publisher","first-page":"272","DOI":"10.1148\/radiol.2018182294","volume":"290","author":"SH Park","year":"2019","unstructured":"Park SH (2019) Diagnostic case-control versus diagnostic cohort studies for clinical validation of artificial intelligence algorithm performance. Radiology 290:272\u20132731","journal-title":"Radiology"},{"key":"8397_CR14","doi-asserted-by":"publisher","first-page":"573","DOI":"10.1148\/radiol.2019191225","volume":"293","author":"EJ Hwang","year":"2019","unstructured":"Hwang EJ, Nam JG, Lim WH et al (2019) Deep learning for chest radiograph diagnosis in the emergency department. Radiology 293:573\u2013580","journal-title":"Radiology"},{"key":"8397_CR15","doi-asserted-by":"publisher","first-page":"687","DOI":"10.1148\/radiol.2020201240","volume":"297","author":"JH Lee","year":"2020","unstructured":"Lee JH, Sun HY, Park SG et al (2020) Performance of a deep learning algorithm compared with radiologic interpretation for lung cancer detection on chest radiographs in a health screening population. Radiology 297:687\u2013696","journal-title":"Radiology"},{"key":"8397_CR16","unstructured":"Rajpurkar P, Irvin J, Zhu K et al (2017) CheXNet: radiologist-level pneumonia detection on chest X-rays with deep learning. arXiv [cs.CV]"},{"key":"8397_CR17","doi-asserted-by":"publisher","first-page":"697","DOI":"10.1148\/radiol.2462070712","volume":"246","author":"DM Hansell","year":"2008","unstructured":"Hansell DM, Bankier AA, MacMahon H et al (2008) Fleischner Society: glossary of terms for thoracic imaging. Radiology 246:697\u2013722","journal-title":"Radiology"},{"key":"8397_CR18","doi-asserted-by":"crossref","unstructured":"Johnson AEW, Pollard TJ, Greenbaum NR et al (2019) MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs. arXiv [cs.CV]","DOI":"10.1038\/s41597-019-0322-0"},{"key":"8397_CR19","unstructured":"World Health Organization, Others (2016) ICD-10 Version: 2016.apps.who.int\/classifications\/icd10\/browse\/2016\/en. F00-F09. Accessed 25 February 2016"},{"key":"8397_CR20","doi-asserted-by":"publisher","first-page":"178","DOI":"10.1016\/S1076-6332(03)00671-8","volume":"11","author":"KH Zou","year":"2004","unstructured":"Zou KH, Warfield SK, Bharatha A et al (2004) Statistical validation of image segmentation quality based on a spatial overlap index. Acad Radiol 11:178\u2013189","journal-title":"Acad Radiol"},{"key":"8397_CR21","doi-asserted-by":"publisher","first-page":"647","DOI":"10.1016\/j.acra.2007.12.015","volume":"15","author":"SL Hillis","year":"2008","unstructured":"Hillis SL, Berbaum KS, Metz CE (2008) Recent developments in the DorfmanBerbaum-Metz procedure for multireader ROC study analysis. Acad Radiol 15:647\u2013661","journal-title":"Acad Radiol"},{"key":"8397_CR22","doi-asserted-by":"publisher","DOI":"10.1183\/13993003.03061-2020","author":"JG Nam","year":"2020","unstructured":"Nam JG, Kim M, Park J et al (2020) Development and validation of a deep learning algorithm detecting 10 common abnormalities on chest radiographs. Eur Respir J. https:\/\/doi.org\/10.1183\/13993003.03061-2020","journal-title":"Eur Respir J"},{"key":"8397_CR23","doi-asserted-by":"publisher","DOI":"10.1007\/s00330-021-08074-7","author":"H Yoo","year":"2021","unstructured":"Yoo H, Lee SH, Arru CD et al (2021) AI-based improvement in lung cancer detection on chest radiographs: results of a multi-reader study in NLST dataset. Eur Radiol. https:\/\/doi.org\/10.1007\/s00330-021-08074-7","journal-title":"Eur Radiol"},{"key":"8397_CR24","doi-asserted-by":"publisher","first-page":"511","DOI":"10.3348\/kjr.2019.0821","volume":"21","author":"EJ Hwang","year":"2020","unstructured":"Hwang EJ, Park CM (2020) Clinical implementation of deep learning in thoracic radiology: potential applications and challenges. Korean J Radiol 21:511\u2013525","journal-title":"Korean J Radiol"},{"key":"8397_CR25","doi-asserted-by":"publisher","first-page":"537","DOI":"10.1148\/radiol.2018181422","volume":"290","author":"JA Dunnmon","year":"2019","unstructured":"Dunnmon JA, Yi D, Langlotz CP et al (2019) Assessment of convolutional neural networks for automated classification of chest radiographs. Radiology 290:537\u2013544","journal-title":"Radiology"},{"key":"8397_CR26","doi-asserted-by":"publisher","first-page":"1359","DOI":"10.1007\/s00330-019-06532-x","volume":"30","author":"S Park","year":"2020","unstructured":"Park S, Lee SM, Lee KH et al (2020) Deep learning-based detection system for multiclass lesions on chest radiographs: comparison with observer readings. Eur Radiol 30:1359\u20131368","journal-title":"Eur Radiol"},{"key":"8397_CR27","doi-asserted-by":"publisher","first-page":"421","DOI":"10.1148\/radiol.2019191293","volume":"294","author":"A Majkowska","year":"2020","unstructured":"Majkowska A, Mittal S, Steiner DF et al (2020) Chest radiograph interpretation with deep learning models: assessment with radiologist-adjudicated reference standards and population-adjusted evaluation. Radiology 294:421\u2013431","journal-title":"Radiology"},{"key":"8397_CR28","doi-asserted-by":"publisher","first-page":"199","DOI":"10.1148\/radiol.2019182465","volume":"294","author":"Y Sim","year":"2020","unstructured":"Sim Y, Chung MJ, Kotter E et al (2020) Deep Convolutional neural network\u2013based software improves radiologist detection of malignant lung nodules on chest radiographs. Radiology 294:199\u2013209","journal-title":"Radiology"}],"container-title":["European Radiology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00330-021-08397-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00330-021-08397-5\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00330-021-08397-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,4,25]],"date-time":"2022-04-25T15:12:19Z","timestamp":1650899539000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s00330-021-08397-5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,1,1]]},"references-count":28,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2022,5]]}},"alternative-id":["8397"],"URL":"https:\/\/doi.org\/10.1007\/s00330-021-08397-5","relation":{},"ISSN":["1432-1084"],"issn-type":[{"value":"1432-1084","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,1,1]]},"assertion":[{"value":"8 June 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"6 September 2021","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 October 2021","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"1 January 2022","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The scientific guarantor of this publication is Young Jun Cho, MD.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Guarantor"}},{"value":"Kwang Nam Jin and Hyungjin Kim received a research grant from Lunit for activities not related to the present article. Other authors of this manuscript declare no relationships with any companies, whose products or services may be related to the subject matter of the article.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"Sohee Oh kindly provided statistical advice for this manuscript.No complex statistical methods were necessary for this paper.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Statistics and biometry"}},{"value":"The study design was approved by the appropriate ethics review board and the requirement of informed consent was waived.","order":5,"name":"Ethics","group":{"name":"EthicsHeading","label":"Informed consent"}},{"value":"Institutional Review Board approval was obtained. IRB No. was 10\u20132019-48 (Boramae Medical Center), 2019\u201305-022 (Konyang University Hospital), and 2020\u2013322 (Gil Medical Center), respectively.","order":6,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical approval"}},{"value":"\u2022 retrospective\u2022 diagnostic study\u2022 multicenter study \/ performed at three institutions","order":7,"name":"Ethics","group":{"name":"EthicsHeading","label":"Methodology"}}]}}