{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,12]],"date-time":"2026-09-12T19:44:00Z","timestamp":1789242240288,"version":"build-2803163510"},"reference-count":24,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2023,3,13]],"date-time":"2023-03-13T00:00:00Z","timestamp":1678665600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Korea Health Industry Development Institute","award":["H19C1129"],"award-info":[{"award-number":["H19C1129"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Diagnostics"],"abstract":"<jats:p>It is unclear whether the visualization methods for artificial-intelligence-based computer-aided detection (AI-CAD) of chest radiographs influence the accuracy of readers\u2019 interpretation. We aimed to evaluate the accuracy of radiologists\u2019 interpretations of chest radiographs using different visualization methods for the same AI-CAD. Initial chest radiographs of patients with acute respiratory symptoms were retrospectively collected. A commercialized AI-CAD using three different methods of visualizing was applied: (a) closed-line method, (b) heat map method, and (c) combined method. A reader test was conducted with five trainee radiologists over three interpretation sessions. In each session, the chest radiographs were interpreted using AI-CAD with one of the three visualization methods in random order. Examination-level sensitivity and accuracy, and lesion-level detection rates for clinically significant abnormalities were evaluated for the three visualization methods. The sensitivity (p = 0.007) and accuracy (p = 0.037) of the combined method are significantly higher than that of the closed-line method. Detection rates using the heat map method (p = 0.043) and the combined method (p = 0.004) are significantly higher than those using the closed-line method. The methods for visualizing AI-CAD results for chest radiographs influenced the performance of radiologists\u2019 interpretations. Combining the closed-line and heat map methods for visualizing AI-CAD results led to the highest sensitivity and accuracy of radiologists.<\/jats:p>","DOI":"10.3390\/diagnostics13061089","type":"journal-article","created":{"date-parts":[[2023,3,14]],"date-time":"2023-03-14T02:33:36Z","timestamp":1678761216000},"page":"1089","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["Methods of Visualizing the Results of an Artificial-Intelligence-Based Computer-Aided Detection System for Chest Radiographs: Effect on the Diagnostic Performance of Radiologists"],"prefix":"10.3390","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1810-7650","authenticated-orcid":false,"given":"Sungho","family":"Hong","sequence":"first","affiliation":[{"name":"Department of Radiology, Seoul National University Hospital, Seoul 03082, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3697-5542","authenticated-orcid":false,"given":"Eui Jin","family":"Hwang","sequence":"additional","affiliation":[{"name":"Department of Radiology, Seoul National University Hospital, Seoul 03082, Republic of Korea"},{"name":"Department of Radiology, Seoul National University College of Medicine, Seoul 03082, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Soojin","family":"Kim","sequence":"additional","affiliation":[{"name":"Department of Radiology, Seoul National University Hospital, Seoul 03082, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6171-876X","authenticated-orcid":false,"given":"Jiyoung","family":"Song","sequence":"additional","affiliation":[{"name":"Department of Radiology, Seoul National University Hospital, Seoul 03082, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0592-8268","authenticated-orcid":false,"given":"Taehee","family":"Lee","sequence":"additional","affiliation":[{"name":"Department of Radiology, Seoul National University Hospital, Seoul 03082, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gyeong Deok","family":"Jo","sequence":"additional","affiliation":[{"name":"Department of Radiology, Seoul National University Hospital, Seoul 03082, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9679-919X","authenticated-orcid":false,"given":"Yelim","family":"Choi","sequence":"additional","affiliation":[{"name":"Department of Radiology, Seoul National University Hospital, Seoul 03082, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chang Min","family":"Park","sequence":"additional","affiliation":[{"name":"Department of Radiology, Seoul National University Hospital, Seoul 03082, Republic of Korea"},{"name":"Department of Radiology, Seoul National University College of Medicine, Seoul 03082, Republic of Korea"},{"name":"Institute of Radiation Medicine, Seoul National University Medical Research Center, Seoul 03082, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1791-7942","authenticated-orcid":false,"given":"Jin Mo","family":"Goo","sequence":"additional","affiliation":[{"name":"Department of Radiology, Seoul National University Hospital, Seoul 03082, Republic of Korea"},{"name":"Department of Radiology, Seoul National University College of Medicine, Seoul 03082, Republic of Korea"},{"name":"Institute of Radiation Medicine, Seoul National University Medical Research Center, Seoul 03082, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2023,3,13]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"European Society of Radiology (2022). Current practical experience with artificial intelligence in clinical radiology: A survey of the European Society of Radiology. Insights Imaging, 13, 107.","DOI":"10.1186\/s13244-022-01247-y"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1743","DOI":"10.3348\/kjr.2021.0544","article-title":"Use of Artificial Intelligence-Based Software as Medical Devices for Chest Radiography: A Position Paper from the Korean Society of Thoracic Radiology","volume":"22","author":"Hwang","year":"2021","journal-title":"Korean J. Radiol."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"847","DOI":"10.3348\/kjr.2022.0193","article-title":"Successful Implementation of an Artificial Intelligence-Based Computer-Aided Detection System for Chest Radiography in Daily Clinical Practice","volume":"23","author":"Lee","year":"2022","journal-title":"Korean J. Radiol."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"511","DOI":"10.3348\/kjr.2019.0821","article-title":"Clinical Implementation of Deep Learning in Thoracic Radiology: Potential Applications and Challenges","volume":"21","author":"Hwang","year":"2020","journal-title":"Korean J. Radiol."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1363","DOI":"10.1016\/j.jacr.2020.08.016","article-title":"Workflow Applications of Artificial Intelligence in Radiology and an Overview of Available Tools","volume":"17","author":"Kapoor","year":"2020","journal-title":"J. Am. Coll. Radiol."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"120","DOI":"10.1016\/j.carj.2018.02.002","article-title":"Canadian Association of Radiologists White Paper on Artificial Intelligence in Radiology","volume":"69","author":"Tang","year":"2018","journal-title":"Can. Assoc. Radiol. J."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"e2229289","DOI":"10.1001\/jamanetworkopen.2022.29289","article-title":"Association of Artificial Intelligence-Aided Chest Radiograph Interpretation with Reader Performance and Efficiency","volume":"5","author":"Ahn","year":"2022","journal-title":"JAMA Netw. Open"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"102125","DOI":"10.1016\/j.media.2021.102125","article-title":"Deep learning for chest X-ray analysis: A survey","volume":"72","author":"Calli","year":"2021","journal-title":"Med. Image Anal."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"109319","DOI":"10.1016\/j.asoc.2022.109319","article-title":"Chest X-ray analysis empowered with deep learning: A systematic review","volume":"126","author":"Meedeniya","year":"2022","journal-title":"Appl. Soft Comput."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"433","DOI":"10.1148\/radiol.211706","article-title":"Deep Learning for Detecting Pneumothorax on Chest Radiographs after Needle Biopsy: Clinical Implementation","volume":"303","author":"Hong","year":"2022","journal-title":"Radiology"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"455","DOI":"10.1148\/radiol.2021210578","article-title":"Deep Learning for Detection of Pulmonary Metastasis on Chest Radiographs","volume":"301","author":"Hwang","year":"2021","journal-title":"Radiology"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"573","DOI":"10.1148\/radiol.2019191225","article-title":"Deep Learning for Chest Radiograph Diagnosis in the Emergency Department","volume":"293","author":"Hwang","year":"2019","journal-title":"Radiology"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"6902","DOI":"10.1007\/s00330-020-07062-7","article-title":"Automated identification of chest radiographs with referable abnormality with deep learning: Need for recalibration","volume":"30","author":"Hwang","year":"2020","journal-title":"Eur. Radiol."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"e191095","DOI":"10.1001\/jamanetworkopen.2019.1095","article-title":"Development and Validation of a Deep Learning-Based Automated Detection Algorithm for Major Thoracic Diseases on Chest Radiographs","volume":"2","author":"Hwang","year":"2019","journal-title":"JAMA Netw. Open"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1263","DOI":"10.1002\/(SICI)1097-0258(19970615)16:11<1263::AID-SIM550>3.0.CO;2-M","article-title":"A marginal regression modelling framework for evaluating medical diagnostic tests","volume":"16","author":"Leisenring","year":"1997","journal-title":"Stat. Med."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"e26611","DOI":"10.2196\/26611","article-title":"Population Preferences for Performance and Explainability of Artificial Intelligence in Health Care: Choice-Based Conjoint Survey","volume":"23","author":"Ploug","year":"2021","journal-title":"J. Med. Internet Res."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"e745","DOI":"10.1016\/S2589-7500(21)00208-9","article-title":"The false hope of current approaches to explainable artificial intelligence in health care","volume":"3","author":"Ghassemi","year":"2021","journal-title":"Lancet Digit. Health"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"e214","DOI":"10.1016\/S2589-7500(22)00029-2","article-title":"Explainability and artificial intelligence in medicine","volume":"4","author":"Reddy","year":"2022","journal-title":"Lancet Digit. Health"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. (2017, January 22\u201329). Grad-cam: Visual explanations from deep networks via gradient-based localization. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.","DOI":"10.1109\/ICCV.2017.74"},{"key":"ref_20","unstructured":"Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (July, January 26). You only look once: Unified, real-time object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1137","DOI":"10.1109\/TPAMI.2016.2577031","article-title":"Faster r-cnn: Towards real-time object detection with region proposal networks","volume":"39","author":"Ren","year":"2017","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"3797","DOI":"10.1007\/s00330-021-07892-z","article-title":"Artificial intelligence in radiology: 100 commercially available products and their scientific evidence","volume":"31","author":"Schalekamp","year":"2021","journal-title":"Eur. Radiol."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41746-021-00385-9","article-title":"Do as AI say: Susceptibility in deployment of clinical decision-aids","volume":"4","author":"Gaube","year":"2021","journal-title":"NPJ Digit. Med."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"475","DOI":"10.1007\/s00330-020-07151-7","article-title":"Implementation of the cloud-based computerized interpretation system in a nationwide lung cancer screening with low-dose CT: Comparison with the conventional reading system","volume":"31","author":"Hwang","year":"2021","journal-title":"Eur. Radiol."}],"container-title":["Diagnostics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2075-4418\/13\/6\/1089\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T18:54:15Z","timestamp":1760122455000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2075-4418\/13\/6\/1089"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,3,13]]},"references-count":24,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2023,3]]}},"alternative-id":["diagnostics13061089"],"URL":"https:\/\/doi.org\/10.3390\/diagnostics13061089","relation":{},"ISSN":["2075-4418"],"issn-type":[{"value":"2075-4418","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,3,13]]}}}