{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,24]],"date-time":"2026-06-24T19:44:10Z","timestamp":1782330250303,"version":"3.54.5"},"reference-count":38,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2022,1,28]],"date-time":"2022-01-28T00:00:00Z","timestamp":1643328000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,1,28]],"date-time":"2022-01-28T00:00:00Z","timestamp":1643328000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["npj Digit. Med."],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>To identify Coronavirus disease (COVID-19) cases efficiently, affordably, and at scale, recent work has shown how audio (including cough, breathing and voice) based approaches can be used for testing. However, there is a lack of exploration of how biases and methodological decisions impact these tools\u2019 performance in practice. In this paper, we explore the realistic performance of audio-based digital testing of COVID-19. To investigate this, we collected a large crowdsourced respiratory audio dataset through a mobile app, alongside symptoms and COVID-19 test results. Within the collected dataset, we selected 5240 samples from 2478 English-speaking participants and split them into participant-independent sets for model development and validation. In addition to controlling the language, we also balanced demographics for model training to avoid potential acoustic bias. We used these audio samples to construct an audio-based COVID-19 prediction model. The unbiased model took features extracted from breathing, coughs and voice signals as predictors and yielded an AUC-ROC of 0.71 (95% CI: 0.65\u20130.77). We further explored several scenarios with different types of unbalanced data distributions to demonstrate how biases and participant splits affect the performance. With these different, but less appropriate, evaluation strategies, the performance could be overestimated, reaching an AUC up to 0.90 (95% CI: 0.85\u20130.95) in some circumstances. We found that an unrealistic experimental setting can result in misleading, sometimes over-optimistic, performance. Instead, we reported complete and reliable results on crowd-sourced data, which would allow medical professionals and policy makers to accurately assess the value of this technology and facilitate its deployment.<\/jats:p>","DOI":"10.1038\/s41746-021-00553-x","type":"journal-article","created":{"date-parts":[[2022,1,28]],"date-time":"2022-01-28T11:02:30Z","timestamp":1643367750000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":84,"title":["Sounds of COVID-19: exploring realistic performance of audio-based digital testing"],"prefix":"10.1038","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5776-6849","authenticated-orcid":false,"given":"Jing","family":"Han","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6994-6318","authenticated-orcid":false,"given":"Tong","family":"Xia","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9761-951X","authenticated-orcid":false,"given":"Dimitris","family":"Spathis","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Erika","family":"Bondareva","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chlo\u00eb","family":"Brown","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jagmohan","family":"Chauhan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ting","family":"Dang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andreas","family":"Grammenos","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Apinan","family":"Hasthanasombat","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andres","family":"Floto","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9193-8496","authenticated-orcid":false,"given":"Pietro","family":"Cicuta","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cecilia","family":"Mascolo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2022,1,28]]},"reference":[{"key":"553_CR1","first-page":"1","volume":"371","author":"M Cevik","year":"2020","unstructured":"Cevik, M., Kuppalli, K., Kindrachuk, J. & Peiris, M. Virology, transmission, and pathogenesis of SARS-CoV-2. Br. Med. J. 371, 1\u20136 (2020).","journal-title":"Br. Med. J."},{"key":"553_CR2","doi-asserted-by":"publisher","first-page":"1299","DOI":"10.1038\/s41564-020-0761-6","volume":"5","author":"CB Vogels","year":"2020","unstructured":"Vogels, C. B. et al. Analytical sensitivity and efficiency comparisons of SARS-CoV-2 RT\u2013qPCR primer\u2013probe sets. Nat. Microbiol. 5, 1299\u20131305 (2020).","journal-title":"Nat. Microbiol."},{"key":"553_CR3","doi-asserted-by":"publisher","unstructured":"Garg, A. et al. Evaluation of seven commercial RT-PCR kits for COVID-19 testing in pooled clinical specimens. J. Med. Virol. https:\/\/doi.org\/10.1002\/jmv.26691 (2020).","DOI":"10.1002\/jmv.26691"},{"key":"553_CR4","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1016\/j.cca.2020.03.009","volume":"505","author":"R Liu","year":"2020","unstructured":"Liu, R. et al. Positive rate of RT-PCR detection of SARS-CoV-2 infection in 4880 cases from one hospital in Wuhan, China, from Jan to Feb 2020. Clin. Chim. Acta 505, 172\u2013175 (2020).","journal-title":"Clin. Chim. Acta"},{"key":"553_CR5","doi-asserted-by":"publisher","unstructured":"van Kasteren, P. B. et al. Comparison of seven commercial RT-PCR diagnostic kits for COVID-19. J. Clin. Virol. https:\/\/doi.org\/10.1016\/j.jcv.2020.104412 (2020).","DOI":"10.1016\/j.jcv.2020.104412"},{"key":"553_CR6","doi-asserted-by":"publisher","first-page":"5214","DOI":"10.1007\/s00330-020-06898-3","volume":"30","author":"L Fan","year":"2020","unstructured":"Fan, L. & Liu, S. CT and COVID-19: Chinese experience and recommendations concerning detection, staging and follow-up. Eur. Radiol. 30, 5214\u20135216 (2020).","journal-title":"Eur. Radiol."},{"key":"553_CR7","doi-asserted-by":"publisher","first-page":"1189","DOI":"10.1016\/S0140-6736(20)30728-5","volume":"395","author":"MD Hope","year":"2020","unstructured":"Hope, M. D., Raptis, C. A., Shah, A., Hammer, M. M. & Henry, T. S. A role for CT in COVID-19? What data really tell us so far. Lancet 395, 1189\u20131190 (2020).","journal-title":"Lancet"},{"key":"553_CR8","doi-asserted-by":"publisher","first-page":"1037","DOI":"10.1038\/s41591-020-0916-2","volume":"26","author":"C Menni","year":"2020","unstructured":"Menni, C. et al. Real-time tracking of self-reported symptoms to predict potential COVID-19. Nat. Med. 26, 1037\u20131040 (2020).","journal-title":"Nat. Med."},{"key":"553_CR9","doi-asserted-by":"publisher","first-page":"1224","DOI":"10.1038\/s41591-020-0931-3","volume":"26","author":"X Mei","year":"2020","unstructured":"Mei, X. et al. Artificial intelligence\u2013enabled rapid diagnosis of patients with COVID-19. Nat. Med. 26, 1224\u20131228 (2020).","journal-title":"Nat. Med."},{"key":"553_CR10","doi-asserted-by":"publisher","unstructured":"Harmon, S. A. et al. Artificial intelligence for the detection of COVID-19 pneumonia on chest CT using multinational datasets. Nat. Commun. https:\/\/doi.org\/10.1038\/s41467-020-17971-2 (2020).","DOI":"10.1038\/s41467-020-17971-2"},{"key":"553_CR11","doi-asserted-by":"publisher","unstructured":"Javaheri, T. et al. CovidCTNet: an open-source deep learning approach to identify covid-19 using CT image. NPJ Digit. Med. https:\/\/doi.org\/10.1038\/s41746-021-00399-3 (2021).","DOI":"10.1038\/s41746-021-00399-3"},{"key":"553_CR12","doi-asserted-by":"publisher","unstructured":"Xu, Q. et al. Ai-based analysis of ct images for rapid triage of covid-19 patients. NPJ Digit. Med. https:\/\/doi.org\/10.1038\/s41746-021-00446-z (2021).","DOI":"10.1038\/s41746-021-00446-z"},{"key":"553_CR13","doi-asserted-by":"publisher","unstructured":"Lee, E. H. et al. Deep covid detect: an international experience on covid-19 lung detection and prognosis using chest ct. NPJ Digit. Med. https:\/\/doi.org\/10.1038\/s41746-020-00369-1 (2021).","DOI":"10.1038\/s41746-020-00369-1"},{"key":"553_CR14","doi-asserted-by":"publisher","unstructured":"Imran, A. et al. AI4COVID-19: AI enabled preliminary diagnosis for COVID-19 from cough samples via an app. Informatics in Medicine Unlocked 20 https:\/\/doi.org\/10.1016\/j.imu.2020.100378 (2020).","DOI":"10.1016\/j.imu.2020.100378"},{"key":"553_CR15","doi-asserted-by":"crossref","unstructured":"Brown, C. et al. Exploring automatic diagnosis of COVID-19 from crowdsourced respiratory sound data. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD) (San Diego, 2020).","DOI":"10.1145\/3394486.3412865"},{"key":"553_CR16","doi-asserted-by":"publisher","first-page":"275","DOI":"10.1109\/OJEMB.2020.3026928","volume":"1","author":"J Laguarta","year":"2020","unstructured":"Laguarta, J., Hueto, F. & Subirana, B. COVID-19 artificial intelligence diagnosis using only cough recordings. IEEE Open J. Eng. Med. Biol. 1, 275\u2013281 (2020).","journal-title":"IEEE Open J. Eng. Med. Biol."},{"key":"553_CR17","doi-asserted-by":"publisher","first-page":"268","DOI":"10.1109\/OJEMB.2020.3026468","volume":"1","author":"G Pinkas","year":"2020","unstructured":"Pinkas, G. et al. SARS-CoV-2 detection from voice. IEEE Open J. Eng. Med. Biol. 1, 268\u2013274 (2020).","journal-title":"IEEE Open J. Eng. Med. Biol."},{"key":"553_CR18","doi-asserted-by":"crossref","unstructured":"Han, J. et al. Exploring automatic covid-19 diagnosis via voice and symptoms from crowdsourced data. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 8328\u20138332 (Toronto, 2021).","DOI":"10.1109\/ICASSP39728.2021.9414576"},{"key":"553_CR19","doi-asserted-by":"publisher","unstructured":"Andreu-Perez, J. et al. A generic deep learning based cough analysis system from clinically validated samples for point-of-need COVID-19 test and severity levels. IEEE Trans. Serv. Comput. https:\/\/doi.org\/10.1109\/TSC.2021.3061402 (2021).","DOI":"10.1109\/TSC.2021.3061402"},{"key":"553_CR20","doi-asserted-by":"publisher","first-page":"356","DOI":"10.1136\/bmjinnov-2021-000668","volume":"7","author":"H Coppock","year":"2021","unstructured":"Coppock, H. et al. End-to-end convolutional neural network enables COVID-19 detection from breath and cough audio: a pilot study. BMJ Innov. 7, 356\u2013362 (2021).","journal-title":"BMJ Innov."},{"key":"553_CR21","doi-asserted-by":"crossref","unstructured":"Al Ismail, M., Deshmukh, S. & Singh, R. Detection of covid-19 through the analysis of vocal fold oscillations. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1035\u20131039 (IEEE, 2021).","DOI":"10.1109\/ICASSP39728.2021.9414201"},{"key":"553_CR22","doi-asserted-by":"publisher","unstructured":"Wynants, L. et al. Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal. Br. Med. J. https:\/\/doi.org\/10.1136\/bmj.m1328 (2020).","DOI":"10.1136\/bmj.m1328"},{"key":"553_CR23","doi-asserted-by":"crossref","unstructured":"Deshmukh, S., Al Ismail, M. & Singh, R. Interpreting glottal flow dynamics for detecting covid-19 from voice. In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 1055\u20131059 (IEEE, 2021).","DOI":"10.1109\/ICASSP39728.2021.9414530"},{"key":"553_CR24","doi-asserted-by":"crossref","unstructured":"Han, J. et al. An early study on intelligent analysis of speech under COVID-19: severity, sleep quality, fatigue, and anxiety. In Annual Conference of the International Speech Communication Association (INTERSPEECH), 4946\u20134950 (2020).","DOI":"10.21437\/Interspeech.2020-2223"},{"key":"553_CR25","doi-asserted-by":"crossref","unstructured":"Xue, H. & Salim, F. D. Exploring self-supervised representation ensembles for COVID-19 cough classification. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD) (Singapore, 2021).","DOI":"10.1145\/3447548.3467263"},{"key":"553_CR26","unstructured":"Subirana, B. et al. Hi sigma, do i have the coronavirus?: call for a new artificial intelligence approach to support health care professionals dealing with the covid-19 pandemic Preprint at arXiv https:\/\/arxiv.org\/abs\/2004.06510 (2020)."},{"key":"553_CR27","doi-asserted-by":"publisher","first-page":"1874","DOI":"10.1016\/S0140-6736(20)32589-7","volume":"396","author":"EJ Topol","year":"2020","unstructured":"Topol, E. J. Is my cough COVID-19? Lancet 396, 1874 (2020).","journal-title":"Lancet"},{"key":"553_CR28","doi-asserted-by":"publisher","first-page":"199","DOI":"10.1038\/s42256-021-00307-0","volume":"3","author":"M Roberts","year":"2021","unstructured":"Roberts, M. et al. Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans. Nat. Mach. Intell. 3, 199\u2013217 (2021).","journal-title":"Nat. Mach. Intell."},{"key":"553_CR29","doi-asserted-by":"publisher","unstructured":"Moons, K. G. et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): explanation and elaboration. Ann. Int. Med. https:\/\/doi.org\/10.7326\/M14-0698 (2015).","DOI":"10.7326\/M14-0698"},{"key":"553_CR30","doi-asserted-by":"publisher","first-page":"336","DOI":"10.1016\/j.ijid.2021.08.067","volume":"111","author":"S Louca","year":"2021","unstructured":"Louca, S. Covid-19 prevalence in 161 countries and over time. Int. J. Infect. Dis. 111, 336\u2013346 (2021).","journal-title":"Int. J. Infect. Dis."},{"key":"553_CR31","unstructured":"McKight, P. E. & Najab, J. The Corsini Encyclopedia of Psychology (John Wiley & Sons, Inc, 2010)."},{"key":"553_CR32","doi-asserted-by":"publisher","unstructured":"Pahar, M. et al. COVID-19 detection in cough, breath and speech using deep transfer learning and bottleneck features. Comput. Biol. Med. 141, 105153 https:\/\/doi.org\/10.1016\/j.compbiomed.2021.105153 (2021).","DOI":"10.1016\/j.compbiomed.2021.105153"},{"key":"553_CR33","doi-asserted-by":"publisher","first-page":"73","DOI":"10.1038\/s41591-020-1123-x","volume":"27","author":"G Quer","year":"2021","unstructured":"Quer, G. et al. Wearable sensor data and self-reported symptoms for covid-19 detection. Nat. Med. 27, 73\u201377 (2021).","journal-title":"Nat. Med."},{"key":"553_CR34","unstructured":"Guo, C., Pleiss, G., Sun, Y. & Weinberger, K. Q. On calibration of modern neural networks. In International Conference on Machine Learning (ICML), 1321\u20131330 (2017)."},{"key":"553_CR35","doi-asserted-by":"crossref","unstructured":"Xia, T., Han, J., Qendro, L., Dang, T. & Mascolo, C. Uncertainty-aware COVID-19 detection from imbalanced sound data. In Annual Conference of the International Speech Communication Association (INTERSPEECH), 2951\u20132955 (2021).","DOI":"10.21437\/Interspeech.2021-1320"},{"key":"553_CR36","doi-asserted-by":"crossref","unstructured":"Hershey, S. et al. CNN architectures for large-scale audio classification. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2017).","DOI":"10.1109\/ICASSP.2017.7952132"},{"key":"553_CR37","unstructured":"Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. In International Conference on Learning Representations (ICLR) (San Diego, CA, 2015)."},{"key":"553_CR38","doi-asserted-by":"publisher","first-page":"1141","DOI":"10.1002\/(SICI)1097-0258(20000515)19:9<1141::AID-SIM479>3.0.CO;2-F","volume":"19","author":"J Carpenter","year":"2000","unstructured":"Carpenter, J. & Bithell, J. Bootstrap confidence intervals: when, which, what? A practical guide for medical statisticians. Stat. Med. 19, 1141\u20131164 (2000).","journal-title":"Stat. Med."}],"container-title":["npj Digital Medicine"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41746-021-00553-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-021-00553-x","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41746-021-00553-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,11,25]],"date-time":"2022-11-25T08:22:53Z","timestamp":1669364573000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41746-021-00553-x"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,1,28]]},"references-count":38,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2022,12]]}},"alternative-id":["553"],"URL":"https:\/\/doi.org\/10.1038\/s41746-021-00553-x","relation":{},"ISSN":["2398-6352"],"issn-type":[{"value":"2398-6352","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,1,28]]},"assertion":[{"value":"21 June 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"13 December 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"28 January 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"16"}}