{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,13]],"date-time":"2026-11-13T22:44:06Z","timestamp":1794609846163,"version":"4.9.1"},"reference-count":48,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2020,10,9]],"date-time":"2020-10-09T00:00:00Z","timestamp":1602201600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,10,9]],"date-time":"2020-10-09T00:00:00Z","timestamp":1602201600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Nat Commun"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    Early detection of COVID-19 based on chest CT enables timely treatment of patients and helps control the spread of the disease. We proposed an artificial intelligence (AI) system for rapid COVID-19 detection and performed extensive statistical analysis of CTs of COVID-19 based on the AI system. We developed and evaluated our system on a large dataset with more than 10 thousand CT volumes from COVID-19, influenza-A\/B, non-viral community acquired pneumonia (CAP) and non-pneumonia subjects. In such a difficult multi-class diagnosis task, our deep convolutional neural network-based system is able to achieve an area under the receiver operating characteristic curve (AUC) of 97.81% for multi-way classification on test cohort of 3,199 scans, AUC of 92.99% and 93.25% on two publicly available datasets, CC-CCII and MosMedData respectively. In a reader study involving five radiologists, the AI system outperforms all of radiologists in more challenging tasks at a speed of two orders of magnitude above them. Diagnosis performance of chest x-ray (CXR) is compared to that of CT. Detailed interpretation of deep network is also performed to relate system outputs with CT presentations. The code is available at\n                    <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"uri\" xlink:href=\"https:\/\/github.com\/ChenWWWeixiang\/diagnosis_covid19\">https:\/\/github.com\/ChenWWWeixiang\/diagnosis_covid19<\/jats:ext-link>\n                    .\n                  <\/jats:p>","DOI":"10.1038\/s41467-020-18685-1","type":"journal-article","created":{"date-parts":[[2020,10,9]],"date-time":"2020-10-09T06:03:32Z","timestamp":1602223412000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":353,"title":["Development and evaluation of an artificial intelligence system for COVID-19 diagnosis"],"prefix":"10.1038","volume":"11","author":[{"given":"Cheng","family":"Jin","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Weixiang","family":"Chen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8391-841X","authenticated-orcid":false,"given":"Yukun","family":"Cao","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhanwei","family":"Xu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zimeng","family":"Tan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xin","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lei","family":"Deng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2435-1417","authenticated-orcid":false,"given":"Chuansheng","family":"Zheng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jie","family":"Zhou","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0877-3054","authenticated-orcid":false,"given":"Heshui","family":"Shi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5940-0063","authenticated-orcid":false,"given":"Jianjiang","family":"Feng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2020,10,9]]},"reference":[{"key":"18685_CR1","unstructured":"World Health Organization. Laboratory Testing For Coronavirus Disease 2019 (COVID-19) in Suspected Human Cases: Interim Guidance, 2 March 2020. https:\/\/apps.who.int\/iris\/handle\/10665\/331329 (2020)."},{"key":"18685_CR2","doi-asserted-by":"publisher","first-page":"E46","DOI":"10.1148\/radiol.2020200823","volume":"296","author":"HX Bai","year":"2020","unstructured":"Bai, H. X. et al. Performance of radiologists in differentiating COVID-19 from viral pneumonia on chest CT. Radiology 296, E46\u2013E54 (2020).","journal-title":"Radiology"},{"key":"18685_CR3","doi-asserted-by":"publisher","first-page":"E32","DOI":"10.1148\/radiol.2020200642","volume":"296","author":"T Ai","year":"2020","unstructured":"Ai, T. et al. Correlation of chest CT and RT-PCR testing in coronavirus disease 2019 (COVID-19) in China: a report of 1014 cases. Radiology 296, E32\u2013E40 (2020).","journal-title":"Radiology"},{"key":"18685_CR4","doi-asserted-by":"publisher","first-page":"106","DOI":"10.1016\/j.chest.2020.04.003","volume":"158","author":"GD Rubin","year":"2020","unstructured":"Rubin, G. D. et al. The role of chest imaging in patient management during the COVID-19 pandemic: a multinational consensus statement from the Fleischner Society. Chest 158, 106\u2013116 (2020).","journal-title":"Chest"},{"key":"18685_CR5","doi-asserted-by":"publisher","first-page":"E72","DOI":"10.1148\/radiol.2020201160","volume":"296","author":"HYF Wong","year":"2020","unstructured":"Wong, H. Y. F. et al. Frequency and distribution of chest radiographic findings in COVID-19 positive patients. Radiology 296, E72\u2013E78 (2020).","journal-title":"Radiology"},{"key":"18685_CR6","doi-asserted-by":"publisher","first-page":"425","DOI":"10.1016\/S1473-3099(20)30086-4","volume":"20","author":"H Shi","year":"2020","unstructured":"Shi, H. et al. Radiological findings from 81 patients with COVID-19 pneumonia in Wuhan, China: a descriptive study. Lancet Infect. Dis. 20, 425\u2013434 (2020).","journal-title":"Lancet Infect. Dis."},{"key":"18685_CR7","doi-asserted-by":"crossref","unstructured":"Deng, J. et al. Imagenet: a large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, 248\u2013255 (IEEE, Miami, FL, 2009).","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"18685_CR8","doi-asserted-by":"publisher","first-page":"436","DOI":"10.1038\/nature14539","volume":"521","author":"Y LeCun","year":"2015","unstructured":"LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436\u2013444 (2015).","journal-title":"Nature"},{"key":"18685_CR9","unstructured":"Krizhevsky, A., Sutskever, I. & Hinton, G. E. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, 1097\u20131105 (Curran Associates, Inc., 2012)."},{"key":"18685_CR10","unstructured":"Ren, S., He, K., Girshick, R. & Sun, J. Faster R-CNN: towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems, 91\u201399 (IEEE, 2015 Published)."},{"key":"18685_CR11","doi-asserted-by":"publisher","first-page":"115","DOI":"10.1038\/nature21056","volume":"542","author":"A Esteva","year":"2017","unstructured":"Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115\u2013118 (2017).","journal-title":"Nature"},{"key":"18685_CR12","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1016\/j.media.2017.07.005","volume":"42","author":"G Litjens","year":"2017","unstructured":"Litjens, G. et al. A survey on deep learning in medical image analysis. Med. Image Anal. 42, 60\u201388 (2017).","journal-title":"Med. Image Anal."},{"key":"18685_CR13","doi-asserted-by":"publisher","first-page":"24","DOI":"10.1038\/s41591-018-0316-z","volume":"25","author":"A Esteva","year":"2019","unstructured":"Esteva, A. et al. A guide to deep learning in healthcare. Nat. Med. 25, 24\u201329 (2019).","journal-title":"Nat. Med."},{"key":"18685_CR14","doi-asserted-by":"publisher","first-page":"44","DOI":"10.1038\/s41591-018-0300-7","volume":"25","author":"EJ Topol","year":"2019","unstructured":"Topol, E. J. High-performance medicine: the convergence of human and artificial intelligence. Nat. Med. 25, 44\u201356 (2019).","journal-title":"Nat. Med."},{"key":"18685_CR15","doi-asserted-by":"publisher","first-page":"954","DOI":"10.1038\/s41591-019-0447-x","volume":"25","author":"D Ardila","year":"2019","unstructured":"Ardila, D. et al. End-to-end lung cancer screening with three-dimensional deep learning on low-dose chest computed tomography. Nat. Med. 25, 954\u2013961 (2019).","journal-title":"Nat. Med."},{"key":"18685_CR16","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41598-018-37186-2","volume":"9","author":"ZZ Qin","year":"2019","unstructured":"Qin, Z. Z. et al. Using artificial intelligence to read chest radiographs for tuberculosis detection: a multi-site evaluation of the diagnostic accuracy of three deep learning systems. Sci. Rep. 9, 1\u201310 (2019).","journal-title":"Sci. Rep."},{"key":"18685_CR17","doi-asserted-by":"publisher","first-page":"1224","DOI":"10.1038\/s41591-020-0931-3","volume":"26","author":"X Mei","year":"2020","unstructured":"Mei, X. et al. Artificial intelligence\u2013enabled rapid diagnosis of patients with COVID-19. Nat. Med. 26, 1224\u20131228 (2020).","journal-title":"Nat. Med."},{"key":"18685_CR18","doi-asserted-by":"publisher","first-page":"109","DOI":"10.1016\/j.patcog.2018.07.031","volume":"85","author":"H Xie","year":"2019","unstructured":"Xie, H., Yang, D., Sun, N., Chen, Z. & Zhang, Y. Automated pulmonary nodule detection in CT images using deep convolutional neural networks. Pattern Recognit 85, 109\u2013119 (2019).","journal-title":"Pattern Recognit"},{"key":"18685_CR19","doi-asserted-by":"publisher","first-page":"3484","DOI":"10.1109\/TNNLS.2019.2892409","volume":"30","author":"F Liao","year":"2019","unstructured":"Liao, F., Liang, M., Li, Z., Hu, X. & Song, S. Evaluate the malignancy of pulmonary nodules using the 3-d deep leaky noisy-or network. IEEE Trans. neural Netw. Learn. Syst. 30, 3484\u20133495 (2019).","journal-title":"IEEE Trans. neural Netw. Learn. Syst."},{"key":"18685_CR20","doi-asserted-by":"crossref","unstructured":"Zhu, W., Liu, C., Fan, W. & Xie, X. Deeplung: deep 3d dual path nets for automated pulmonary nodule detection and classification. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), 673\u2013681 (IEEE, 2018).","DOI":"10.1109\/WACV.2018.00079"},{"key":"18685_CR21","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41598-019-56589-3","volume":"10","author":"M Nash","year":"2020","unstructured":"Nash, M. et al. Deep learning, computer-aided radiography reading for tuberculosis: a diagnostic accuracy study from a tertiary hospital in India. Sci. Rep. 10, 1\u201310 (2020).","journal-title":"Sci. Rep."},{"key":"18685_CR22","doi-asserted-by":"crossref","unstructured":"Shi, F. et al. Review of artificial intelligence techniques in imaging data acquisition, segmentation and diagnosis for COVID-19. IEEE Rev. Biomed. Eng. (2020).","DOI":"10.1109\/RBME.2020.2987975"},{"key":"18685_CR23","doi-asserted-by":"crossref","unstructured":"Dong, D. et al. The role of imaging in the detection and management of COVID-19: a review. IEEE Rev. Biomed. Eng. (2020).","DOI":"10.1109\/RBME.2020.2990959"},{"key":"18685_CR24","doi-asserted-by":"publisher","first-page":"1423","DOI":"10.1016\/j.cell.2020.04.045","volume":"181","author":"K Zhang","year":"2020","unstructured":"Zhang, K. et al. Clinically applicable AI system for accurate diagnosis, quantitative measurements and prognosis of COVID-19 pneumonia using computed tomography. Cell 181, 1423\u20131433 (2020).","journal-title":"Cell"},{"key":"18685_CR25","doi-asserted-by":"publisher","first-page":"E65","DOI":"10.1148\/radiol.2020200905","volume":"296","author":"L Li","year":"2020","unstructured":"Li, L. et al. Using Artificial Intelligence to Detect COVID-19 and Community-acquired Pneumonia Based on Pulmonary CT: Evaluation of the Diagnostic Accuracy. Radiology 296, E65\u2013E71 (2020).","journal-title":"Radiology"},{"key":"18685_CR26","doi-asserted-by":"publisher","first-page":"E156","DOI":"10.1148\/radiol.2020201491","volume":"296","author":"HX Bai","year":"2020","unstructured":"Bai, H. X. et al. AI augmentation of radiologist performance in distinguishing COVID-19 from pneumonia of other etiology on chest CT. Radiology 296, E156\u2013E165 (2020).","journal-title":"Radiology"},{"key":"18685_CR27","doi-asserted-by":"publisher","first-page":"2615","DOI":"10.1109\/TMI.2020.2995965","volume":"39","author":"X Wang","year":"2020","unstructured":"Wang, X. et al. A weakly-supervised framework for COVID-19 Classification and lesion localization from chest CT. IEEE Trans. Med. Imaging 39, 2615\u20132625 (2020).","journal-title":"IEEE Trans. Med. Imaging"},{"key":"18685_CR28","doi-asserted-by":"publisher","first-page":"2584","DOI":"10.1109\/TMI.2020.2996256","volume":"39","author":"Z Han","year":"2020","unstructured":"Han, Z. et al. Accurate screening of COVID-19 using attention based deep 3D multiple instance learning. IIEEE Trans. Med. Imaging 39, 2584\u20132594 (2020).","journal-title":"IIEEE Trans. Med. Imaging"},{"key":"18685_CR29","doi-asserted-by":"publisher","first-page":"2595","DOI":"10.1109\/TMI.2020.2995508","volume":"39","author":"X Ouyang","year":"2020","unstructured":"Ouyang, X. et al. Dual-sampling attention network for diagnosis of COVID-19 from community acquired pneumonia. IEEE Trans. Med. Imaging 39, 2595\u20132605 (2020).","journal-title":"IEEE Trans. Med. Imaging"},{"key":"18685_CR30","doi-asserted-by":"publisher","first-page":"E166","DOI":"10.1148\/radiol.2020201874","volume":"296","author":"K Murphy","year":"2020","unstructured":"Murphy, K. et al. COVID-19 on the chest radiograph: a multi-reader evaluation of an AI system. Radiology 296, E166\u2013E172 (2020).","journal-title":"Radiology"},{"key":"18685_CR31","doi-asserted-by":"publisher","first-page":"915","DOI":"10.1118\/1.3528204","volume":"38","author":"SG Armato III","year":"2011","unstructured":"Armato, S. G. III et al. The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI): a completed reference database of lung nodules on CT scans. Med. Phys. 38, 915\u2013931 (2011).","journal-title":"Med. Phys."},{"key":"18685_CR32","unstructured":"Tianchi Competition https:\/\/tianchi.aliyun.com\/competition\/entrance\/231601\/information (2017)."},{"key":"18685_CR33","doi-asserted-by":"crossref","unstructured":"Morozov, S. et al. MosMedData: chest CT scans with COVID-19 related findings. Preprint at https:\/\/arxiv.org\/abs\/2005.06465 (2020).","DOI":"10.1101\/2020.05.20.20100362"},{"key":"18685_CR34","doi-asserted-by":"crossref","unstructured":"Selvaraju, R. R. et al. Grad-cam: visual explanations from deep networks via gradient-based localization. in Proceedings of the IEEE international conference on computer vision, 618\u2013626 (IEEE, 2017).","DOI":"10.1109\/ICCV.2017.74"},{"key":"18685_CR35","first-page":"2579","volume":"9","author":"Lvd Maaten","year":"2008","unstructured":"Maaten, Lvd & Hinton, G. Visualizing data using t-SNE. J. Mach. Learn. Res. 9, 2579\u20132605 (2008).","journal-title":"J. Mach. Learn. Res."},{"key":"18685_CR36","doi-asserted-by":"publisher","first-page":"e104","DOI":"10.1158\/0008-5472.CAN-17-0339","volume":"77","author":"JJ Van Griethuysen","year":"2017","unstructured":"Van Griethuysen, J. J. et al. Computational radiomics system to decode the radiographic phenotype. Cancer Res. 77, e104\u2013e107 (2017).","journal-title":"Cancer Res."},{"key":"18685_CR37","doi-asserted-by":"publisher","first-page":"200463","DOI":"10.1148\/radiol.2020200463","volume":"295","author":"A Bernheim","year":"2020","unstructured":"Bernheim, A. et al. Chest CT findings in coronavirus disease-19 (COVID-19): relationship to duration of infection. Radiology 295, 200463 (2020).","journal-title":"Radiology"},{"key":"18685_CR38","unstructured":"Xiong, Q. et al. Women may play a more important role in the transmission of the corona virus disease (COVID-19) than men. Lancet (2020)."},{"key":"18685_CR39","doi-asserted-by":"crossref","unstructured":"Kanne, J. P. Chest CT Findings nn 2019 Novel Coronavirus (2019-nCoV) Infections from Wuhan, China: Key Points for the Radiologist. Report No. 0033-8419, 16\u201317 (Radiological Society of North America, 2020).","DOI":"10.1148\/radiol.2020200241"},{"key":"18685_CR40","doi-asserted-by":"crossref","unstructured":"Ronneberger, O., Fischer, P. & Brox, T. U-net: convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-assisted Intervention, 234\u2013241 (Springer, 2015).","DOI":"10.1007\/978-3-319-24574-4_28"},{"key":"18685_CR41","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 770\u2013778 (IEEE, 2016).","DOI":"10.1109\/CVPR.2016.90"},{"key":"18685_CR42","unstructured":"Paszke, A. et al. PyTorch: an imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, 8024\u20138035 (NeurIPS, 2019)."},{"key":"18685_CR43","unstructured":"Cascella, M., Rajnik, M., Cuomo, A., Dulebohn, S. C. & Di Napoli, R. Features, Evaluation and Treatment Coronavirus (COVID-19) (StatPearls Publishing, 2020)."},{"key":"18685_CR44","unstructured":"Dail, D. H. & Hammar, S. P. Dail and Hammar\u2019s Pulmonary Pathology (Springer Science & Business Media, 2013)."},{"key":"18685_CR45","unstructured":"Cotes, J. E., Chinn, D. J. & Miller, M. R. Lung Function: Physiology, Measurement and Application in Medicine (Wiley, 2009)."},{"key":"18685_CR46","doi-asserted-by":"publisher","first-page":"160053","DOI":"10.1183\/16000617.0053-2016","volume":"26","author":"M Panagiotou","year":"2017","unstructured":"Panagiotou, M., Church, A. C., Johnson, M. K. & Peacock, A. J. Pulmonary vascular and cardiac impairment in interstitial lung disease. Eur. Respir. Rev. 26, 160053 (2017).","journal-title":"Eur. Respir. Rev."},{"key":"18685_CR47","first-page":"207","volume":"9","author":"JW Breslin","year":"2011","unstructured":"Breslin, J. W. et al. Lymphatic vessel network structure and physiology. Compr. Physiol. 9, 207\u2013299 (2011).","journal-title":"Compr. Physiol."},{"key":"18685_CR48","first-page":"1","volume":"2","author":"B Moldoveanu","year":"2009","unstructured":"Moldoveanu, B. et al. Inflammatory mechanisms in the lung. J. Inflamm. Res. 2, 1 (2009).","journal-title":"J. Inflamm. Res."}],"container-title":["Nature Communications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41467-020-18685-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41467-020-18685-1","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41467-020-18685-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,5]],"date-time":"2022-12-05T20:14:28Z","timestamp":1670271268000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41467-020-18685-1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,10,9]]},"references-count":48,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2020,12]]}},"alternative-id":["18685"],"URL":"https:\/\/doi.org\/10.1038\/s41467-020-18685-1","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2020.03.20.20039834","asserted-by":"object"}]},"ISSN":["2041-1723"],"issn-type":[{"value":"2041-1723","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,10,9]]},"assertion":[{"value":"11 June 2020","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"4 September 2020","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 October 2020","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"5088"}}