{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,28]],"date-time":"2026-08-28T17:15:45Z","timestamp":1787937345286,"version":"build-2784847793"},"reference-count":45,"publisher":"MDPI AG","issue":"19","license":[{"start":{"date-parts":[[2024,9,24]],"date-time":"2024-09-24T00:00:00Z","timestamp":1727136000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Audio-based classification techniques for body sounds have long been studied to aid in the diagnosis of respiratory diseases. While most research is centered on the use of coughs as the main acoustic biomarker, other body sounds also have the potential to detect respiratory diseases. Recent studies on the coronavirus disease 2019 (COVID-19) have suggested that breath and speech sounds, in addition to cough, correlate with the disease. Our study proposes fused audio instance and representation (FAIR) as a method for respiratory disease detection. FAIR relies on constructing a joint feature vector from various body sounds represented in waveform and spectrogram form. We conduct experiments on the use case of COVID-19 detection by combining waveform and spectrogram representation of body sounds. Our findings show that the use of self-attention to combine extracted features from cough, breath, and speech sounds leads to the best performance with an area under the receiver operating characteristic curve (AUC) score of 0.8658, a sensitivity of 0.8057, and a specificity of 0.7958. Compared to models trained solely on spectrograms or waveforms, the use of both representations results in an improved AUC score, demonstrating that combining spectrogram and waveform representation helps to enrich the extracted features and outperforms the models that use only one representation. While this study focuses on COVID-19, FAIR\u2019s flexibility allows it to combine various multi-modal and multi-instance features in many other diagnostic applications, potentially leading to more accurate diagnoses across a wider range of diseases.<\/jats:p>","DOI":"10.3390\/s24196176","type":"journal-article","created":{"date-parts":[[2024,9,24]],"date-time":"2024-09-24T08:56:06Z","timestamp":1727168166000},"page":"6176","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Fused Audio Instance and Representation for Respiratory Disease Detection"],"prefix":"10.3390","volume":"24","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0284-4269","authenticated-orcid":false,"given":"Tuan","family":"Truong","sequence":"first","affiliation":[{"name":"Bayer AG, 13353 Berlin, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3771-2012","authenticated-orcid":false,"given":"Matthias","family":"Lenga","sequence":"additional","affiliation":[{"name":"Bayer AG, 13353 Berlin, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8268-6832","authenticated-orcid":false,"given":"Antoine","family":"Serrurier","sequence":"additional","affiliation":[{"name":"Clinic for Phoniatrics, Pedaudiology and Communication Disorders, University Hospital of RWTH Aachen, 52074 Aachen, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sadegh","family":"Mohammadi","sequence":"additional","affiliation":[{"name":"Bayer AG, 13353 Berlin, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2024,9,24]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"88","DOI":"10.1016\/j.smrv.2019.03.001","article-title":"Systematic review of the different aspects of primary snoring","volume":"45","author":"Jacquet","year":"2019","journal-title":"Sleep Med. Rev."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"158","DOI":"10.4103\/1817-1737.160831","article-title":"Auscultation of the respiratory system","volume":"10","author":"Sarkar","year":"2015","journal-title":"Ann. Thorac. Med."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Song, I. (2015, January 12\u201317). Diagnosis of pneumonia from sounds collected using low cost cell phones. Proceedings of the 2015 International Joint Conference on Neural Networks (IJCNN), Killarney, Ireland.","DOI":"10.1109\/IJCNN.2015.7280317"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"275","DOI":"10.1109\/OJEMB.2020.3026928","article-title":"COVID-19 Artificial Intelligence Diagnosis Using Only Cough Recordings","volume":"1","author":"Laguarta","year":"2020","journal-title":"IEEE Open J. Eng. Med. Biol."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"045005","DOI":"10.1088\/1361-6579\/aab6d0","article-title":"Detection of tuberculosis by automatic cough sound analysis","volume":"39","author":"Botha","year":"2018","journal-title":"Physiol. Meas."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1344","DOI":"10.1109\/JBHI.2019.2931395","article-title":"Deep Learning on Computerized Analysis of Chronic Obstructive Pulmonary Disease","volume":"24","author":"Altan","year":"2020","journal-title":"IEEE J. Biomed. Health Inform."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Zhang, H., Song, C., Wang, A., Xu, C., Li, D., and Xu, W. (2019, January 21\u201325). PDVocal: Towards Privacy-preserving Parkinson\u2019s Disease Detection using Non-speech Body Sounds. Proceedings of the 25th Annual International Conference on Mobile Computing and Networking, Los Cabos, Mexico.","DOI":"10.1145\/3300061.3300125"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"671","DOI":"10.1007\/s11517-017-1706-y","article-title":"Apnea and heart rate detection from tracheal body sounds for the diagnosis of sleep-related breathing disorders","volume":"56","author":"Kalkbrenner","year":"2018","journal-title":"Med Biol. Eng. Comput."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"407","DOI":"10.1016\/j.dsx.2020.04.020","article-title":"Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2): An overview of viral structure and host response","volume":"14","author":"Astuti","year":"2020","journal-title":"Diabetes Metab. Syndr. Clin. Res. Rev."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"2100441","DOI":"10.2807\/1560-7917.ES.2021.26.44.2100441","article-title":"Comparative sensitivity evaluation for 122 CE-marked rapid diagnostic tests for SARS-CoV-2 antigen, Germany, September 2020 to April 2021","volume":"26","author":"Scheiblauer","year":"2021","journal-title":"Eurosurveillance"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Huang, Y., Meng, S., Zhang, Y., Wu, S., Zhang, Y., Zhang, Y., Ye, Y., Wei, Q., Zhao, N., and Jiang, J. (2020). The respiratory sound features of COVID-19 patients fill gaps between clinical data and screening methods. medRxiv.","DOI":"10.1101\/2020.04.07.20051060"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Al Ismail, M., Deshmukh, S., and Singh, R. (2021, January 6\u201311). Detection of Covid-19 Through the Analysis of Vocal Fold Oscillations. Proceedings of the ICASSP 2021\u20142021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toronto, ON, Canada.","DOI":"10.1109\/ICASSP39728.2021.9414201"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1120","DOI":"10.1121\/10.0003434","article-title":"Artificial intelligence enabled preliminary diagnosis for COVID-19 from voice cues and questionnaires","volume":"149","author":"Shimon","year":"2021","journal-title":"J. Acoust. Soc. Am."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Suppakitjanusant, P., Sungkanuparph, S., Wongsinin, T., Virapongsiri, S., Kasemkosin, N., Chailurkit, L., and Ongphiphadhanakul, B. (2021). Identifying individuals with recent COVID-19 through voice classification using deep learning. Sci. Rep., 11.","DOI":"10.1038\/s41598-021-98742-x"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"105014","DOI":"10.1088\/1361-6579\/ac2fb8","article-title":"Automatic cough classification for tuberculosis screening in a real-world environment","volume":"42","author":"Pahar","year":"2021","journal-title":"Physiol. Meas."},{"key":"ref_16","first-page":"1","article-title":"Listen2Cough: Leveraging End-to-End Deep Learning Cough Detection Model to Enhance Lung Health Assessment Using Passively Sensed Audio","volume":"Volume 5","author":"Xu","year":"2021","journal-title":"Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Khanaghavalle, G., Rahul, G., Senajith, S., Vishnuvasan, T., and Keerthana, S. (2024, January 12\u201314). Chronic Obstructive Pulmonary Disease Severity Classification using lung Sound. Proceedings of the 2024 10th International Conference on Communication and Signal Processing (ICCSP), Melmaruvathur, India.","DOI":"10.1109\/ICCSP60870.2024.10543344"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Luo, K., Yang, G., Li, Y., Lan, S., Wang, Y., He, L., and Hu, B. (2024). Croup and pertussis cough sound classification algorithm based on channel attention and multiscale Mel-spectrogram. Biomed. Signal Process. Control., 91.","DOI":"10.1016\/j.bspc.2024.106073"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Kim, B.J., Kim, B.S., Mun, J.H., Lim, C., and Kim, K. (2022). An accurate deep learning model for wheezing in children using real world data. Sci. Rep., 12.","DOI":"10.1038\/s41598-022-25953-1"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Petmezas, G., Cheimariotis, G.A., Stefanopoulos, L., Rocha, B., Paiva, R.P., Katsaggelos, A.K., and Maglaveras, N. (2022). Automated Lung Sound Classification Using a Hybrid CNN-LSTM Network and Focal Loss Function. Sensors, 22.","DOI":"10.3390\/s22031232"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Choi, Y., and Lee, H. (2023). Interpretation of lung disease classification with light attention connected module. Biomed. Signal Process. Control., 84.","DOI":"10.1016\/j.bspc.2023.104695"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Serrurier, A., Neuschaefer-Rube, C., and R\u00f6hrig, R. (2022). Past and Trends in Cough Sound Acquisition, Automatic Detection and Automatic Classification: A Comparative Review. Sensors, 22.","DOI":"10.3390\/s22082896"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"2053","DOI":"10.1177\/15353702221115428","article-title":"Exploring machine learning for audio-based respiratory condition screening: A concise review of databases, methods, and open issues","volume":"247","author":"Xia","year":"2022","journal-title":"Exp. Biol. Med."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"156","DOI":"10.1038\/s41597-021-00937-4","article-title":"The COUGHVID crowdsourcing dataset, a corpus for the study of large-scale cough analysis algorithms","volume":"8","author":"Orlandic","year":"2021","journal-title":"Sci. Data"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Sharma, N., Krishnan, P., Kumar, R., Ramoji, S., Chetupalli, S.R., R., N., Ghosh, P.K., and Ganapathy, S. (2020, January 25\u201329). Coswara\u2014A Database of Breathing, Cough, and Voice Sounds for COVID-19 Diagnosis. Proceedings of the Interspeech 2020, Virtual Event, Shanghai, China.","DOI":"10.21437\/Interspeech.2020-2768"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Brown, C., Chauhan, J., Grammenos, A., Han, J., Hasthanasombat, A., Spathis, D., Xia, T., Cicuta, P., and Mascolo, C. (2020, January 6\u201310). Exploring Automatic Diagnosis of COVID-19 from Crowdsourced Respiratory Sound Data. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, Virtual Event.","DOI":"10.1145\/3394486.3412865"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Fakhry, A., Jiang, X., Xiao, J., Chaudhari, G., Han, A., and Khanzada, A. (2021). Virufy: A Multi-Branch Deep Learning Network for Automated Detection of COVID-19. arXiv.","DOI":"10.21437\/Interspeech.2021-378"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Meister, J.A., Nguyen, K.A., and Luo, Z. (2021). Audio feature ranking for sound-based COVID-19 patient detection. arXiv.","DOI":"10.1007\/978-3-031-16474-3_13"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Pahar, M., Klopper, M., Warren, R., and Niesler, T. (2021). COVID-19 cough classification using machine learning and global smartphone recordings. Comput. Biol. Med., 135.","DOI":"10.1016\/j.compbiomed.2021.104572"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Topuz, E.K., and Kaya, Y. (2024). SUPER-COUGH: A Super Learner-based ensemble machine learning method for detecting disease on cough acoustic signals. Biomed. Signal Process. Control., 93.","DOI":"10.1016\/j.bspc.2024.106165"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Rao, S., Narayanaswamy, V., Esposito, M., Thiagarajan, J., and Spanias, A. (2021, January 12\u201314). Deep Learning with hyper-parameter tuning for COVID-19 Cough Detection. Proceedings of the 2021 12th International Conference on Information, Intelligence, Systems & Applications (IISA), Chania Crete, Greece.","DOI":"10.1109\/IISA52424.2021.9555564"},{"key":"ref_32","unstructured":"Simonyan, K., and Zisserman, A. (2015). Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv."},{"key":"ref_33","unstructured":"Xia, T., Spathis, D., Brown, C., Chauhan, J., Grammenos, A., Han, J., Hasthanasombat, A., Bondareva, E., Dang, T., and Floto, A. (2021, January 6\u201314). COVID-19 Sounds: A Large-Scale Audio Dataset for Digital Respiratory Screening. Proceedings of the 35th Conference on Neural Information Processing Systems Datasets and Benchmarks Track, Virtual."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Wall, C., Zhang, L., Yu, Y., Kumar, A., and Gao, R. (2022). A Deep Ensemble Neural Network with Attention Mechanisms for Lung Abnormality Classification Using Audio Inputs. Sensors, 22.","DOI":"10.3390\/s22155566"},{"key":"ref_35","first-page":"1","article-title":"Attention is All you Need","volume":"30","author":"Vaswani","year":"2017","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_36","unstructured":"Truong, T., Mohammadi, S., and Lenga, M. (2021). How Transferable are Self-supervised Features in Medical Image Classification Tasks. Machine Learning for Health, PMLR."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"21262","DOI":"10.1109\/ACCESS.2024.3361943","article-title":"Lung sound classification with multi-feature integration utilizing lightweight CNN model","volume":"12","author":"Wanasinghe","year":"2024","journal-title":"IEEE Access"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"236","DOI":"10.1109\/TASSP.1984.1164317","article-title":"Signal estimation from modified short-time Fourier transform","volume":"32","author":"Griffin","year":"1984","journal-title":"IEEE Trans. Acoust. Speech Signal Process."},{"key":"ref_39","unstructured":"Baevski, A., Zhou, H., Mohamed, A., and Auli, M. (2020). wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations. arXiv."},{"key":"ref_40","unstructured":"Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2021). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv."},{"key":"ref_41","unstructured":"Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and J\u00e9gou, H. (2021, January 18\u201324). Training data-efficient image transformers & distillation through attention. Proceedings of the International Conference on Machine Learning, Virtual."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"397","DOI":"10.1038\/s41597-023-02266-0","article-title":"Coswara: A respiratory sounds and symptoms dataset for remote screening of SARS-CoV-2 infection","volume":"10","author":"Bhattacharya","year":"2023","journal-title":"Sci. Data"},{"key":"ref_43","unstructured":"Loshchilov, I., and Hutter, F. (2019). Decoupled Weight Decay Regularization. arXiv."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (July, January 26). Deep Residual Learning for Image Recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Hu, J., Shen, L., and Sun, G. (2018, January 18\u201323). Squeeze-and-Excitation Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00745"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/19\/6176\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T16:01:42Z","timestamp":1760112102000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/19\/6176"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,9,24]]},"references-count":45,"journal-issue":{"issue":"19","published-online":{"date-parts":[[2024,10]]}},"alternative-id":["s24196176"],"URL":"https:\/\/doi.org\/10.3390\/s24196176","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,9,24]]}}}