{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,10]],"date-time":"2026-02-10T18:49:33Z","timestamp":1770749373632,"version":"3.50.0"},"reference-count":36,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2022,1,14]],"date-time":"2022-01-14T00:00:00Z","timestamp":1642118400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,1,14]],"date-time":"2022-01-14T00:00:00Z","timestamp":1642118400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100006418","name":"Brown University","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100006418","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100008536","name":"Amazon Web Services","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100008536","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000093","name":"U.S. Department of Health & Human Services | NIH | Center for Information Technology","doi-asserted-by":"publisher","award":["F30CA239407,"],"award-info":[{"award-number":["F30CA239407,"]}],"id":[{"id":"10.13039\/100000093","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>While COVID-19 diagnosis and prognosis artificial intelligence models exist, very few can be implemented for practical use given their high risk of bias. We aimed to develop a diagnosis model that addresses notable shortcomings of prior studies, integrating it into a fully automated triage pipeline that examines chest radiographs for the presence, severity, and progression of COVID-19 pneumonia. Scans were collected using the DICOM Image Analysis and Archive, a system that communicates with a hospital\u2019s image repository. The authors collected over 6,500 non-public chest X-rays comprising diverse COVID-19 severities, along with radiology reports and RT-PCR data. The authors provisioned one internally held-out and two external test sets to assess model generalizability and compare performance to traditional radiologist interpretation. The pipeline was evaluated on a prospective cohort of 80 radiographs, reporting a 95% diagnostic accuracy. The study mitigates bias in AI model development and demonstrates the value of an end-to-end COVID-19 triage platform.<\/jats:p>","DOI":"10.1038\/s41746-021-00546-w","type":"journal-article","created":{"date-parts":[[2022,1,14]],"date-time":"2022-01-14T11:02:40Z","timestamp":1642158160000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":31,"title":["An automated COVID-19 triage pipeline using artificial intelligence based on chest radiographs and clinical data"],"prefix":"10.1038","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6146-3581","authenticated-orcid":false,"given":"Chris K.","family":"Kim","sequence":"first","affiliation":[]},{"given":"Ji Whae","family":"Choi","sequence":"additional","affiliation":[]},{"given":"Zhicheng","family":"Jiao","sequence":"additional","affiliation":[]},{"given":"Dongcui","family":"Wang","sequence":"additional","affiliation":[]},{"given":"Jing","family":"Wu","sequence":"additional","affiliation":[]},{"given":"Thomas Y.","family":"Yi","sequence":"additional","affiliation":[]},{"given":"Kasey C.","family":"Halsey","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1417-7096","authenticated-orcid":false,"given":"Feyisope","family":"Eweje","sequence":"additional","affiliation":[]},{"given":"Thi My Linh","family":"Tran","sequence":"additional","affiliation":[]},{"given":"Chang","family":"Liu","sequence":"additional","affiliation":[]},{"given":"Robin","family":"Wang","sequence":"additional","affiliation":[]},{"given":"John","family":"Sollee","sequence":"additional","affiliation":[]},{"given":"Celina","family":"Hsieh","sequence":"additional","affiliation":[]},{"given":"Ken","family":"Chang","sequence":"additional","affiliation":[]},{"given":"Fang-Xue","family":"Yang","sequence":"additional","affiliation":[]},{"given":"Ritambhara","family":"Singh","sequence":"additional","affiliation":[]},{"given":"Jie-Lin","family":"Ou","sequence":"additional","affiliation":[]},{"given":"Raymond Y.","family":"Huang","sequence":"additional","affiliation":[]},{"given":"Cai","family":"Feng","sequence":"additional","affiliation":[]},{"given":"Michael D.","family":"Feldman","sequence":"additional","affiliation":[]},{"given":"Tao","family":"Liu","sequence":"additional","affiliation":[]},{"given":"Ji Sheng","family":"Gong","sequence":"additional","affiliation":[]},{"given":"Shaolei","family":"Lu","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9895-4061","authenticated-orcid":false,"given":"Carsten","family":"Eickhoff","sequence":"additional","affiliation":[]},{"given":"Xue","family":"Feng","sequence":"additional","affiliation":[]},{"given":"Ihab","family":"Kamel","sequence":"additional","affiliation":[]},{"given":"Ronnie","family":"Sebro","sequence":"additional","affiliation":[]},{"given":"Michael K.","family":"Atalay","sequence":"additional","affiliation":[]},{"given":"Terrance","family":"Healey","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9869-4685","authenticated-orcid":false,"given":"Yong","family":"Fan","sequence":"additional","affiliation":[]},{"given":"Wei-Hua","family":"Liao","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1516-0480","authenticated-orcid":false,"given":"Jianxin","family":"Wang","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7460-8866","authenticated-orcid":false,"given":"Harrison X.","family":"Bai","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2022,1,14]]},"reference":[{"key":"546_CR1","doi-asserted-by":"publisher","first-page":"2012","DOI":"10.1056\/NEJMoa2004500","volume":"382","author":"PK Bhatraju","year":"2020","unstructured":"Bhatraju, P. K. et al. Covid-19 in Critically ill patients In the Seattle region\u2014case series. N. Engl. J. Med. 382, 2012\u20132022 (2020).","journal-title":"N. Engl. J. Med."},{"key":"546_CR2","doi-asserted-by":"publisher","first-page":"1708","DOI":"10.1056\/NEJMoa2002032","volume":"382","author":"W Guan","year":"2020","unstructured":"Guan, W. et al. Clinical characteristics of coronavirus disease 2019 in China. N. Engl. J. Med. 382, 1708\u20131720 (2020).","journal-title":"N. Engl. J. Med."},{"key":"546_CR3","unstructured":"COVID-19 Map. Johns Hopkins Coronavirus Resource Center. https:\/\/coronavirus.jhu.edu\/map.html."},{"key":"546_CR4","doi-asserted-by":"publisher","first-page":"1151","DOI":"10.1016\/S1473-3099(20)30457-6","volume":"20","author":"AJ Kucharski","year":"2020","unstructured":"Kucharski, A. J. et al. Effectiveness of isolation, testing, contact tracing, and physical distancing on reducing transmission of SARS-CoV-2 in different settings: a mathematical modelling study. Lancet Infect. Dis. 20, 1151\u20131160 (2020).","journal-title":"Lancet Infect. Dis."},{"key":"546_CR5","doi-asserted-by":"publisher","first-page":"23","DOI":"10.2807\/1560-7917.ES.2020.25.3.2000045","volume":"25","author":"VM Corman","year":"2020","unstructured":"Corman, V. M. et al. Detection of 2019 novel coronavirus (2019-nCoV) by real-time RT-PCR. Eurosurveillance 25, 23\u201330 (2020).","journal-title":"Eurosurveillance"},{"key":"546_CR6","doi-asserted-by":"publisher","first-page":"623","DOI":"10.3348\/kjr.2020.0195","volume":"21","author":"Z Chen","year":"2020","unstructured":"Chen, Z. et al. A patient with COVID-19 presenting a false-negative reverse transcriptase polymerase chain reaction result. Korean J. Radiol. 21, 623\u2013624 (2020).","journal-title":"Korean J. Radiol."},{"key":"546_CR7","doi-asserted-by":"publisher","first-page":"e00297","DOI":"10.1128\/JCM.00297-20","volume":"58","author":"P Winichakoon","year":"2020","unstructured":"Winichakoon, P. et al. Negative nasopharyngeal and oropharyngeal swabs do not rule out COVID-19. J. Clin. Microbiol. 58, e00297\u201320 (2020).","journal-title":"J. Clin. Microbiol."},{"key":"546_CR8","doi-asserted-by":"publisher","first-page":"2249","DOI":"10.1001\/jama.2020.8259","volume":"323","author":"N Sethuraman","year":"2020","unstructured":"Sethuraman, N., Jeremiah, S. S. & Ryo, A. Interpreting diagnostic tests for SARS-CoV-2. JAMA 323, 2249\u20132251 (2020).","journal-title":"JAMA"},{"key":"546_CR9","unstructured":"ASM Advocacy. ASM Expresses Concern About Coronavirus Test Reagent Shortages https:\/\/asm.org\/Articles\/Policy\/2020\/March\/ASM-Expresses-Concern-about-Test-Reagent-Shortages (2020)."},{"key":"546_CR10","doi-asserted-by":"publisher","first-page":"103792","DOI":"10.1016\/j.compbiomed.2020.103792","volume":"121","author":"T Ozturk","year":"2020","unstructured":"Ozturk, T. et al. Automated detection of COVID-19 cases using deep neural networks with X-ray images. Comput. Biol. Med. 121, 103792 (2020).","journal-title":"Comput. Biol. Med."},{"key":"546_CR11","doi-asserted-by":"publisher","first-page":"E197","DOI":"10.1148\/radiol.2020201754","volume":"297","author":"D Toussie","year":"2020","unstructured":"Toussie, D. et al. Clinical and chest radiography features determine patient outcomes in young and middle-aged adults with COVID-19. Radiology 297, E197\u2013E206 (2020).","journal-title":"Radiology"},{"key":"546_CR12","doi-asserted-by":"publisher","first-page":"105532","DOI":"10.1016\/j.cmpb.2020.105532","volume":"194","author":"RM Pereira","year":"2020","unstructured":"Pereira, R. M., Bertolini, D., Teixeira, L. O., Silla, C. N. & Costa, Y. M. G. COVID-19 identification in chest X-ray images on flat and hierarchical classification scenarios. Comput. Methods Prog. Biomed. 194, 105532 (2020).","journal-title":"Comput. Methods Prog. Biomed."},{"key":"546_CR13","unstructured":"American College of Radiology. ACR Recommendations for the use of Chest Radiography and Computed Tomography (CT) for Suspected COVID-19 Infection https:\/\/www.acr.org\/Advocacy-and-Economics\/ACR-Position-Statements\/Recommendations-for-Chest-Radiography-and-CT-for-Suspected-COVID19-Infection."},{"key":"546_CR14","doi-asserted-by":"publisher","first-page":"35","DOI":"10.1016\/j.clinimag.2020.04.001","volume":"64","author":"A Jacobi","year":"2020","unstructured":"Jacobi, A., Chung, M., Bernheim, A. & Eber, C. Portable chest X-ray in coronavirus disease-19 (COVID-19): a pictorial review. Clin. Imaging 64, 35 (2020).","journal-title":"Clin. Imaging"},{"key":"546_CR15","doi-asserted-by":"publisher","first-page":"E72","DOI":"10.1148\/radiol.2020201160","volume":"296","author":"HYF Wong","year":"2020","unstructured":"Wong, H. Y. F. et al. Frequency and distribution of chest radiographic findings in patients positive for COVID-19. Radiology 296, E72\u2013E78 https:\/\/doi.org\/10.1148\/radiol.2020201160 (2020).","journal-title":"Radiology"},{"key":"546_CR16","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s10140-020-01808-y","volume":"27","author":"HW Kim","year":"2020","unstructured":"Kim, H. W. et al. The role of initial chest X-ray in triaging patients with suspected COVID-19 during the pandemic. Emerg. Radiol. 27, 1 (2020).","journal-title":"Emerg. Radiol."},{"key":"546_CR17","doi-asserted-by":"publisher","first-page":"9887","DOI":"10.1038\/s41598-021-88807-2","volume":"11","author":"A Zargari Khuzani","year":"2021","unstructured":"Zargari Khuzani, A., Heidari, M. & Shariati, S. A. COVID-Classifier: an automated machine learning model to assist in the diagnosis of COVID-19 infection in chest X-ray images. Sci. Rep. 11, 9887, https:\/\/doi.org\/10.1038\/s41598-021-88807-2 (2021).","journal-title":"Sci. Rep."},{"key":"546_CR18","first-page":"e200079","volume":"2","author":"MD Li","year":"2020","unstructured":"Li, M. D. et al. Automated assessment and tracking of COVID-19 pulmonary disease severity on chest radiographs using convolutional Siamese. Neural Netw. 2, e200079 (2020).","journal-title":"Neural Netw."},{"key":"546_CR19","doi-asserted-by":"publisher","first-page":"e286","DOI":"10.1016\/S2589-7500(21)00039-X","volume":"3","author":"Z Jiao","year":"2021","unstructured":"Jiao, Z. et al. Prognostication of patients with COVID-19 using artificial intelligence based on chest x-rays and clinical data: a retrospective study. Lancet Digit. Health 3, e286\u2013e294 (2021).","journal-title":"Lancet Digit. Health"},{"key":"546_CR20","doi-asserted-by":"publisher","first-page":"199","DOI":"10.1038\/s42256-021-00307-0","volume":"3","author":"M Roberts","year":"2021","unstructured":"Roberts, M. et al. Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans. Nat. Mach. Intell. 3, 199\u2013217 (2021).","journal-title":"Nat. Mach. Intell."},{"key":"546_CR21","doi-asserted-by":"publisher","first-page":"336","DOI":"10.1007\/s11263-019-01228-7","volume":"128","author":"RR Selvaraju","year":"2016","unstructured":"Selvaraju, R. R. et al. Grad-CAM: visual explanations from deep networks via gradient-based localization. Int. J. Comput. Vis. 128, 336\u2013359 (2016).","journal-title":"Int. J. Comput. Vis."},{"key":"546_CR22","first-page":"10691","volume":"2019","author":"M Tan","year":"2019","unstructured":"Tan, M. & Le, Q. V. EfficientNet: rethinking model scaling for convolutional neural networks. 36th Int. Conf. Mach. Learn. ICML 2019 2019, 10691\u201310700 (2019).","journal-title":"36th Int. Conf. Mach. Learn. ICML 2019"},{"key":"546_CR23","unstructured":"Cohen, J. P. et al. COVID-19 Image data collection: prospective predictions are the future. undefined 2020, 2\u20133 (2020)."},{"key":"546_CR24","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.inffus.2021.04.008","volume":"76","author":"G Maguolo","year":"2020","unstructured":"Maguolo, G. & Nanni, L. A critic evaluation of methods for COVID-19 automatic detection from X-ray images. Inf. Fusion 76, 1\u20137 (2020).","journal-title":"Inf. Fusion"},{"key":"546_CR25","doi-asserted-by":"publisher","first-page":"1405","DOI":"10.1007\/s10278-021-00488-5","volume":"34","author":"T Yi","year":"2021","unstructured":"Yi, T. et al. DICOM image analysis and archive (DIANA): an open-source system for clinical AI applications. J. Digit. Imaging. 34, 1405\u20131413, https:\/\/doi.org\/10.1007\/s10278-021-00488-5 (2021).","journal-title":"J. Digit. Imaging."},{"key":"546_CR26","doi-asserted-by":"publisher","first-page":"65","DOI":"10.1007\/s10140-020-01824-y","volume":"28","author":"EL Tung","year":"2021","unstructured":"Tung, E. L., Dubble, E. H., Jindal, G., Movson, J. S. & Swenson, D. W. Survey of radiologists and emergency department providers after implementation of a global radiology report categorization system. Emerg. Radiol. 28, 65\u201375 (2021).","journal-title":"Emerg. Radiol."},{"key":"546_CR27","doi-asserted-by":"crossref","unstructured":"Irvin, J. et al. CheXpert: a large chest radiograph dataset with uncertainty labels and expert comparison. In 33rd AAAI Conference on Artificial Intelligence AAAI 2019, 31st Innovations in Applied Artificial Intelligence Conference IAAI 2019 9th AAAI Symp. Educ. Adv. Artificial Intelligence EAAI 2019 590\u2013597 (AAAI, 2019).","DOI":"10.1609\/aaai.v33i01.3301590"},{"key":"546_CR28","unstructured":"Vay\u00e1, M. et al. BIMCV COVID-19+: A Large Annotated Dataset of RX and CT Images from COVID-19 Patients (2020). arXiv:2006.01174."},{"key":"546_CR29","doi-asserted-by":"crossref","unstructured":"Wang, X. et al. Chest X-ray 8: hospital-scale chest X-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In Proc. of the 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017 January. 3462\u20133471 (IEEE, 2017).","DOI":"10.1109\/CVPR.2017.369"},{"key":"546_CR30","first-page":"475","volume":"4","author":"S Jaeger","year":"2014","unstructured":"Jaeger, S. et al. Two public chest X-ray datasets for computer-aided screening of pulmonary diseases. Quant. Imaging Med. Surg. 4, 475 (2014).","journal-title":"Quant. Imaging Med. Surg."},{"key":"546_CR31","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1007\/s11263-015-0816-y","volume":"115","author":"O Russakovsky","year":"2014","unstructured":"Russakovsky, O. et al. ImageNet large scale visual recognition challenge. Int. J. Comput. Vis. 115, 211\u2013252 (2014).","journal-title":"Int. J. Comput. Vis."},{"key":"546_CR32","unstructured":"Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. In Proc. 3rd International Conference on Learning Representations, ICLR 2015\u2014Conference Track Proc. (ICLR, 2015)."},{"key":"546_CR33","first-page":"234","volume":"9351","author":"O Ronneberger","year":"2015","unstructured":"Ronneberger, O., Fischer, P. & Brox, T. U-Net: convolutional networks for biomedical image segmentation. Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinforma.) 9351, 234\u2013241 (2015).","journal-title":"Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinforma.)"},{"key":"546_CR34","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1214\/aos\/1176344552","volume":"7","author":"B Efron","year":"1979","unstructured":"Efron, B. Bootstrap methods: another look at the Jackknife. Ann. Stat. 7, 1\u201326 (1979).","journal-title":"Ann. Stat."},{"key":"546_CR35","first-page":"119","volume":"52","author":"A Agresti","year":"1998","unstructured":"Agresti, A. & Coull, B. A. Approximate Is better than \u2018exact\u2019 for interval estimation of binomial proportions. Am. Stat. 52, 119 (1998).","journal-title":"Am. Stat."},{"key":"546_CR36","doi-asserted-by":"publisher","first-page":"361","DOI":"10.1002\/(SICI)1097-0258(19960229)15:4<361::AID-SIM168>3.0.CO;2-4","volume":"28","author":"FE Harrel Jr","year":"1996","unstructured":"Harrel, F. E. Jr, Lee, K. L. & Mark, D. B. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat. Med. 28, 361\u2013387 (1996).","journal-title":"Stat. Med."}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-021-00546-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-021-00546-w","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-021-00546-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,11,25]],"date-time":"2022-11-25T08:19:39Z","timestamp":1669364379000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-021-00546-w"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,1,14]]},"references-count":36,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2022,12]]}},"alternative-id":["546"],"URL":"https:\/\/doi.org\/10.1038\/s41746-021-00546-w","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,1,14]]},"assertion":[{"value":"25 May 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"28 November 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"14 January 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"One of the co-authors (X.F.) is employed by Carina Medical. The remaining authors declare no competing interests or other disclosures to make.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"5"}}