{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,5]],"date-time":"2026-03-05T01:57:51Z","timestamp":1772675871029,"version":"3.50.1"},"reference-count":27,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2024,2,8]],"date-time":"2024-02-08T00:00:00Z","timestamp":1707350400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,2,8]],"date-time":"2024-02-08T00:00:00Z","timestamp":1707350400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"the National Key R&D Program of China","award":["2020AAA0109505"],"award-info":[{"award-number":["2020AAA0109505"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Digit Imaging. Inform. med."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>This study aimed to assess the performance of a deep learning algorithm in helping radiologist achieve improved efficiency and accuracy in chest radiograph diagnosis. We adopted a deep learning algorithm to concurrently detect the presence of normal findings and 13 different abnormalities in chest radiographs and evaluated its performance in assisting radiologists. Each competing radiologist had to determine the presence or absence of these signs based on the label provided by the AI. The 100 radiographs were randomly divided into two sets for evaluation: one without AI assistance (control group) and one with AI assistance (test group). The accuracy, false-positive rate, false-negative rate, and analysis time of 111 radiologists (29 senior, 32 intermediate, and 50 junior) were evaluated. A radiologist was given an initial score of 14 points for each image read, with 1 point deducted for an incorrect answer and 0 points given for a correct answer. The final score for each doctor was automatically calculated by the backend calculator. We calculated the mean scores of each radiologist in the two groups (the control group and the test group) and calculated the mean scores to evaluate the performance of the radiologists with and without AI assistance. The average score of the 111 radiologists was 597 (587\u2013605) in the control group and 619 (612\u2013626) in the test group (<jats:italic>P<\/jats:italic>\u2009&lt;\u20090.001). The time spent by the 111 radiologists on the control and test groups was 3279 (2972\u20133941) and 1926 (1710\u20132432) s, respectively (<jats:italic>P<\/jats:italic>\u2009&lt;\u20090.001). The performance of the 111 radiologists in the two groups was evaluated by the area under the receiver operating characteristic curve (AUC). The radiologists showed better performance on the test group of radiographs in terms of normal findings, pulmonary fibrosis, heart shadow enlargement, mass, pleural effusion, and pulmonary consolidation recognition, with AUCs of 1.0, 0.950, 0.991, 1.0, 0.993, and 0.982, respectively. The radiologists alone showed better performance in aortic calcification (0.993), calcification (0.933), cavity (0.963), nodule (0.923), pleural thickening (0.957), and rib fracture (0.987) recognition. This competition verified the positive effects of deep learning methods in assisting radiologists in interpreting chest X-rays. AI assistance can help to improve both the efficacy and efficiency of radiologists.<\/jats:p>","DOI":"10.1007\/s10278-024-00990-6","type":"journal-article","created":{"date-parts":[[2024,2,8]],"date-time":"2024-02-08T22:02:01Z","timestamp":1707429721000},"page":"922-934","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":26,"title":["Deep Learning for Chest X-ray Diagnosis: Competition Between Radiologists with or Without Artificial Intelligence Assistance"],"prefix":"10.1007","volume":"37","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4034-6968","authenticated-orcid":false,"given":"Lili","family":"Guo","sequence":"first","affiliation":[]},{"given":"Changsheng","family":"Zhou","sequence":"additional","affiliation":[]},{"given":"Jingxu","family":"Xu","sequence":"additional","affiliation":[]},{"given":"Chencui","family":"Huang","sequence":"additional","affiliation":[]},{"given":"Yizhou","family":"Yu","sequence":"additional","affiliation":[]},{"given":"Guangming","family":"Lu","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2024,2,8]]},"reference":[{"key":"990_CR1","doi-asserted-by":"publisher","first-page":"227","DOI":"10.1016\/j.jbi.2014.02.018","volume":"49","author":"C Kurtz","year":"2014","unstructured":"Kurtz C, Beaulieu CF, Napel S, et al. A hierarchical knowledge-based approach for retrieving similar medical images described with semantic annotations. J Biomed Inform. 2014; 49: 227-44. https:\/\/doi.org\/10.1016\/j.jbi.2014.02.018.","journal-title":"J Biomed Inform."},{"issue":"11","key":"990_CR2","doi-asserted-by":"publisher","first-page":"e1002686","DOI":"10.1371\/journal.pmed.1002686","volume":"15","author":"P Rajpurkar","year":"2018","unstructured":"Rajpurkar P, Irvin J, Ball RL, et al. Deep learning for chest radiograph diagnosis: A retrospective comparison of the CheXNeXt algorithm to practicing radiologists. PLoS Med. 2018; 15(11): e1002686. https:\/\/doi.org\/10.1371\/journal.pmed.1002686.","journal-title":"PLoS Med."},{"key":"990_CR3","doi-asserted-by":"publisher","first-page":"115","DOI":"10.1038\/s41746-020-00322-2","volume":"3","author":"P Rajpurkar","year":"2020","unstructured":"Rajpurkar P, O'Connell C, Schechter A, et al. CheXaid: deep learning assistance for physician diagnosis of tuberculosis using chest x-rays in patients with HIV. NPJ Digit Med. 2020; 3: 115. https:\/\/doi.org\/10.1038\/s41746-020-00322-2.","journal-title":"NPJ Digit Med."},{"issue":"1","key":"990_CR4","doi-asserted-by":"publisher","first-page":"272","DOI":"10.1148\/radiol.2019194005","volume":"291","author":"M Annarumma","year":"2019","unstructured":"Annarumma M, Withey SJ, Bakewell RJ, et al. Automated Triaging of Adult Chest Radiographs with Deep Artificial Neural Networks. Radiology. 2019; 291(1): 272. https:\/\/doi.org\/10.1148\/radiol. 2019194005.","journal-title":"Radiology."},{"key":"990_CR5","doi-asserted-by":"publisher","first-page":"26","DOI":"10.1016\/j.media.2018.12.007","volume":"53","author":"E Pesce","year":"2019","unstructured":"Pesce E, Ypsilantis P-P, Withey S, et al. Learning to detect chest radiographs containing lung nodules using visual attention networks. Med Image Anal. 2019; 53: 26-38. https:\/\/doi.org\/10.1016\/j.media.2018.12.007.","journal-title":"Med Image Anal."},{"issue":"8","key":"990_CR6","doi-asserted-by":"publisher","first-page":"2042","DOI":"10.1109\/TMI.2021.3070847","volume":"40","author":"J Lian","year":"2021","unstructured":"Lian J, Liu J, Zhang S, et al. A Structure-Aware Relation Network for Thoracic Diseases Detection and Segmentation. IEEE Trans Med Imaging, 2021; 40(8): 2042-2052. https:\/\/doi.org\/10.1109\/TMI.2021.3070847.","journal-title":"IEEE Trans Med Imaging"},{"key":"990_CR7","doi-asserted-by":"publisher","unstructured":"Ruhan Sa, Owens W, Wiegand R, et al. Intervertebral disc detection in X-ray images using faster R-CNN[J]. Annu Int Conf Proc IEEE Eng Med Biol Soc, 2017:564\u2013567. https:\/\/doi.org\/10.1109\/EMBC.2017.8036887.","DOI":"10.1109\/EMBC.2017.8036887"},{"issue":"9","key":"990_CR8","doi-asserted-by":"publisher","first-page":"2428","DOI":"10.1109\/TMI.2021.3077913","volume":"40","author":"G Zhao","year":"2021","unstructured":"Zhao G, Fang C, Li G, et al. Contralaterally Enhanced Networks for Thoracic Disease Detection[J]. IEEE Trans Med Imaging, 2021;40(9):2428-2438. https:\/\/doi.org\/10.1109\/TMI.2021.3077913.","journal-title":"IEEE Trans Med Imaging"},{"key":"990_CR9","doi-asserted-by":"publisher","DOI":"10.1016\/j.chaos.2021.110749","volume":"145","author":"S Rajpal","year":"2021","unstructured":"Rajpal S, Lakhyani N, Singh AK, et al. Using Handpicked Features in Conjunction with ResNet-50 for Improved Detection of COVID-19 from Chest X-Ray Images[J]. Chaos Solitons Fractals, 2021;145:110749. https:\/\/doi.org\/10.1016\/j.chaos.2021.110749.","journal-title":"Chaos Solitons Fractals"},{"issue":"6","key":"990_CR10","doi-asserted-by":"publisher","first-page":"1452","DOI":"10.1109\/TPAMI.2017.2723009","volume":"40","author":"B Zhou","year":"2018","unstructured":"Zhou B, Lapedriza A, Khosla A, et al. Places: A 10 Million Image Database for Scene Recognition [J]. IEEE Trans Pattern Anal Mach Intell.\u00a02018; 40(6): 1452-1464. https:\/\/doi.org\/10.1109\/TPAMI.2017.2723009.","journal-title":"IEEE Trans Pattern Anal Mach Intell."},{"issue":"1","key":"990_CR11","doi-asserted-by":"publisher","first-page":"271","DOI":"10.3390\/ijerph18010271","volume":"18","author":"D Lee","year":"2021","unstructured":"Lee D, Yoon SN. Application of Artificial Intelligence-Based Technologies in the Healthcare Industry: Opportunities and Challenges. Int J Environ Res Public Health. 2021;18(1):271. https:\/\/doi.org\/10.3390\/ijerph18010271.","journal-title":"Int J Environ Res Public Health."},{"issue":"7","key":"990_CR12","doi-asserted-by":"publisher","first-page":"869","DOI":"10.3348\/kjr.2019.0651","volume":"21","author":"QQ Zhou","year":"2020","unstructured":"Zhou QQ, Tang W, Wang J, et al. Automatic Detection and Classification of Rib Fractures on Thoracic CT Using Convolutional Neural Network: Accuracy and Feasibility. Korean J Radiol. 2020; 21(7): 869-879. https:\/\/doi.org\/10.3348\/kjr.2019.0651.","journal-title":"Korean J Radiol."},{"issue":"3","key":"990_CR13","doi-asserted-by":"publisher","first-page":"251","DOI":"10.1097\/MPA.0000000000001762","volume":"50","author":"B Kenner","year":"2021","unstructured":"Kenner B, Chari ST, Kelsen D, et al. Artificial Intelligence and Early Detection of Pancreatic Cancer. Pancreas. 2021; 50(3): 251-279. https:\/\/doi.org\/10.1097\/MPA.0000000000001762.","journal-title":"Pancreas."},{"key":"990_CR14","doi-asserted-by":"publisher","first-page":"111","DOI":"10.1038\/s41746-019-0189-7","volume":"2","author":"BN Patel","year":"2019","unstructured":"Patel BN, Rosenberg L, Willcox G, et al. Human\u2013machine partnership with artificial intelligence for chest radiograph diagnosis. NPJ Digit Med. 2019; 2: 111. https:\/\/doi.org\/10.1038\/s41746-019-0189-7.","journal-title":"NPJ Digit Med."},{"key":"990_CR15","doi-asserted-by":"publisher","unstructured":"Hyeyoung Hah, Deana Shevit Goldin. How\u00a0Clinicians\u00a0Perceive\u00a0Artificial\u00a0Intelligence\u2013Assisted\u00a0Technologies\u00a0in\u00a0Diagnostic\u00a0Decision Making:\u00a0Mixed\u00a0Methods\u00a0Approach. J Med Internet Res.\u00a02021;\u00a023(12): e33540.\u00a0Published online 2021 Dec 16. https:\/\/doi.org\/10.2196\/33540.","DOI":"10.2196\/33540"},{"key":"990_CR16","doi-asserted-by":"publisher","unstructured":"Thomas Weikert, Marco Francone, Suhny Abbara, et al. Machine learning in cardiovascular\u00a0radiology: ESCR position statement on design requirements, quality assessment, current applications, opportunities, and challenges. Eur Radiol.\u00a02021;\u00a031(6): 3909\u20133922.\u00a0Published online 2020 Nov 19. https:\/\/doi.org\/10.1007\/s00330-020-07417-0.","DOI":"10.1007\/s00330-020-07417-0"},{"key":"990_CR17","doi-asserted-by":"publisher","first-page":"32","DOI":"10.1016\/j.compmedimag.2016.04.002","volume":"51","author":"Sema Candemir","year":"2016","unstructured":"Sema Candemir, Stefan Jaeger, Sameer Antani,\u00a0et al. Atlas-based rib-bone detection in chest X-rays. Comput Med Imaging Graph. 2016; 51: 32-39. https:\/\/doi.org\/10.1016\/j.compmedimag.2016.04.002.","journal-title":"Comput Med Imaging Graph."},{"key":"990_CR18","doi-asserted-by":"publisher","first-page":"61","DOI":"10.1038\/s41746-020-0266-y","volume":"3","author":"SC Huang","year":"2020","unstructured":"Huang SC, Kothari T, Banerjee I, et al. PENet\u2014a scalable deep-learning model for automated diagnosis of pulmonary embolism using volumetric CT imaging. NPJ Digit Med. 2020; 3: 61. https:\/\/doi.org\/10.1038\/s41746-020-0266-y.","journal-title":"NPJ Digit Med."},{"key":"990_CR19","doi-asserted-by":"publisher","unstructured":"Cao XF, Li Y, Xin HN, et al. Application of artificial intelligence in digital chest radiography reading for pulmonary tuberculosis screening. Chronic Dis Transl Med.\u00a02021; 7(1): 35-40. https:\/\/doi.org\/10.1016\/j.cdtm.2021.02.001.","DOI":"10.1016\/j.cdtm.2021.02.001"},{"issue":"11","key":"990_CR20","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0242013","volume":"15","author":"HY Wang","year":"2020","unstructured":"Wang HY, Gu H, Qin P, et al. CheXLocNet: Automatic localization of pneumothorax in chest radiographs using deep convolutional neural networks. PLoS One.\u00a02020; 15(11): e0242013.\u00a0https:\/\/doi.org\/10.1371\/journal.pone.0242013.","journal-title":"PLoS One."},{"issue":"5","key":"990_CR21","doi-asserted-by":"publisher","first-page":"320","DOI":"10.5114\/pg.2013.38736","volume":"8","author":"D Kozie\u0142","year":"2013","unstructured":"Kozie\u0142 D, Koz\u0142owska M, Deneka J, et al. Retrospective analysis of clinical problems concerning acute pancreatitis in one treatment center. Prz Gastroenterol. 2013; 8(5): 320-6. https:\/\/doi.org\/10.5114\/pg.2013.38736.","journal-title":"Prz Gastroenterol."},{"issue":"1","key":"990_CR22","doi-asserted-by":"publisher","first-page":"93","DOI":"10.1186\/s12955-019-1164-3","volume":"17","author":"X Duan","year":"2019","unstructured":"Duan X, Ni X, Shi L, et al. The impact of workplace violence on job satisfaction, job burnout, and turnover intention: the mediating role of social support. Health Qual Life Outcomes. 2019; 17(1): 93.\u00a0https:\/\/doi.org\/10.1186\/s12955-019-1164-3.","journal-title":"Health Qual Life Outcomes."},{"issue":"1","key":"990_CR23","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.6833","volume":"12","author":"S Albano","year":"2020","unstructured":"Albano S, Ramnot A, Siddiqi J,\u00a0et al. Medical and Legal Implications of MRI Scout Imaging in a Surgical Patient with Case Presentation. Cureus. 2020; 12(1): e6833.\u00a0https:\/\/doi.org\/10.7759\/cureus.6833.","journal-title":"Cureus."},{"issue":"2","key":"990_CR24","doi-asserted-by":"publisher","first-page":"507","DOI":"10.3390\/cancers12020507","volume":"12","author":"WY Chuang","year":"2020","unstructured":"Chuang WY, Chang SH, Yu WH, et al. Successful Identification of Nasopharyngeal Carcinoma in Nasopharyngeal Biopsies Using Deep Learning. Cancers (Basel). 2020; 12(2): 507. https:\/\/doi.org\/10.3390\/cancers12020507.","journal-title":"Cancers (Basel)."},{"key":"990_CR25","doi-asserted-by":"publisher","unstructured":"Pranav Rajpurkar, Jeremy Irvin, Robyn L, et al. Deep learning\u00a0for\u00a0chest\u00a0radiograph\u00a0diagnosis: A\u00a0retrospective\u00a0comparison\u00a0of the\u00a0CheXNeXt\u00a0algorithm\u00a0to\u00a0practicing\u00a0radiologists. PLoS Med.\u00a02018;\u00a015(11): e1002686. https:\/\/doi.org\/10.1371\/journal.pmed.1002686.","DOI":"10.1371\/journal.pmed.1002686"},{"issue":"6","key":"990_CR26","doi-asserted-by":"publisher","first-page":"743","DOI":"10.1038\/s41551-023-01045-x","volume":"7","author":"HY Zhou","year":"2023","unstructured":"Zhou HY, Yu YZ, Wang CD. et al. A transformer-based representation-learning model with unified processing of multimodal input for clinical diagnostics. Nat Biomed Eng 2023; 7(6):743-755.","journal-title":"Nat Biomed Eng"},{"key":"990_CR27","doi-asserted-by":"publisher","unstructured":"X-ray dark-field chest imaging for detection and quantification of emphysema in patients with chronic obstructive pulmonary disease: a diagnostic accuracy study The Lancet Digital Health 3(11) e733-e744. https:\/\/doi.org\/10.1016\/S2589-7500(21)00146-1","DOI":"10.1016\/S2589-7500(21)00146-1"}],"container-title":["Journal of Imaging Informatics in Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10278-024-00990-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10278-024-00990-6\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10278-024-00990-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,6,12]],"date-time":"2024-06-12T12:15:46Z","timestamp":1718194546000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10278-024-00990-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,2,8]]},"references-count":27,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2024,6]]}},"alternative-id":["990"],"URL":"https:\/\/doi.org\/10.1007\/s10278-024-00990-6","relation":{},"ISSN":["2948-2933"],"issn-type":[{"value":"2948-2933","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,2,8]]},"assertion":[{"value":"9 September 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"13 December 2023","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"13 December 2023","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"8 February 2024","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Institutional Review Board approval was obtained by Ethics Committee of our hospital (approval number YX-P-2020\u2013085-01).","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics Approval"}},{"value":"Written informed consent was waived by the Institutional Review Board for this retrospective analysis.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Informed Consent"}},{"value":"The authors affirm that consent to publish has been received from all human research participants.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for Publication"}},{"value":"The authors declare no competing interests.","order":5,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing Interests"}}]}}