{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,25]],"date-time":"2026-09-25T03:40:50Z","timestamp":1790307650801,"version":"4.1.0"},"reference-count":59,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2025,5,22]],"date-time":"2025-05-22T00:00:00Z","timestamp":1747872000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"KAU Endowment (WAQF) at King Abdulaziz University, Jeddah, Saudi Arabia"},{"name":"Deanship of Scientific Research (DSR)"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Bioengineering"],"abstract":"<jats:p>In recent years, deep learning has shown promise in automating heart-sound classification. Although this approach is fast, non-invasive, and cost-effective, its diagnostic accuracy still mainly depends on the clinician\u2019s expertise, making it particularly challenging to detect rare or complex conditions. This study is motivated by two key concerns in the field of heart-sound classification. First, we observed that automatic heart-sound segmentation algorithms\u2014commonly used for data augmentation\u2014produce varying outcomes, raising concerns about the accuracy of both the segmentation process and the resulting classification performance. Second, we noticed inconsistent accuracy scores across different pretrained models, prompting the need for interpretable explanations to validate these results. We argue that without interpretability to support reported metrics, accuracy scores can be misleading because of ambiguity in how training data interact with pretrained models. Specifically, it remains unclear whether these models classify spectrogram images\u2014generated from heart-sound signals\u2014in a way that aligns with clinical reasoning, where experts focus on specific components of the heart cycle, such as S1, systole, S2, and diastole. To address this, we applied explainable AI (XAI) techniques with two primary objectives: (1) to assess whether the model truly focuses on clinically relevant features, thereby allowing classification results to be verified and trusted, and (2) to investigate whether incorporating attention mechanisms can improve both the performance and the model\u2019s focus on meaningful segments of the signal. To the best of our knowledge, this is the first study conducted on a manually segmented dataset, which objectively evaluates the model\u2019s behavior using XAI and explores performance enhancement by combining attention mechanisms with pretrained models. We employ the Grad-CAM method to visualize the model\u2019s attention and gain insights into the decision-making process. The experimental results show that integrating multi-head attention significantly improves both the classification accuracy and interpretability. Notably, ResNet50 with multi-head attention achieved an accuracy of 97.3%, outperforming those of both the baseline and SE-enhanced models. Moreover, the mean intersection over union (mIoU) for interpretability increased from 75.7% to 82.0%, indicating the model\u2019s improved focus on diagnostically relevant regions.<\/jats:p>","DOI":"10.3390\/bioengineering12060558","type":"journal-article","created":{"date-parts":[[2025,5,22]],"date-time":"2025-05-22T08:49:57Z","timestamp":1747903797000},"page":"558","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Are Artificial Intelligence Models Listening Like Cardiologists? Bridging the Gap Between Artificial Intelligence and Clinical Reasoning in Heart-Sound Classification Using Explainable Artificial Intelligence"],"prefix":"10.3390","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4674-6081","authenticated-orcid":false,"given":"Sami","family":"Alrabie","sequence":"first","affiliation":[{"name":"Faculty of Computing and Information Technology (FCIT), King Abdulaziz University, Jeddah 21589, Saudi Arabia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0516-8331","authenticated-orcid":false,"given":"Ahmed","family":"Barnawi","sequence":"additional","affiliation":[{"name":"Faculty of Computing and Information Technology (FCIT), King Abdulaziz University, Jeddah 21589, Saudi Arabia"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2025,5,22]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1201\/b23266-2","article-title":"Cardiovascular diseases worldwide","volume":"1","author":"Gaziano","year":"2022","journal-title":"Public Health Approach Cardiovasc. Dis. Prev. Manag."},{"key":"ref_2","unstructured":"Barnawi, S., and Alrabie, A. (2025, April 02). Evaluation of Pre-Trained CNN Models for Cardiovascular Disease Classification: A Benchmark Study. Available online: https:\/\/www.researchgate.net\/profile\/Sami-Alrabie\/publication\/374875705_Evaluation_of_Pre-Trained_CNN_Models_for_Cardiovascular_Disease_Evaluation_of_Pre-Trained_CNN_Models_for_Cardiovascular_Disease_Classification_A_Benchmark_Study_Classification_A_Benchmark_Study_Evalua\/links\/6533bd351d6e8a707041e413\/Evaluation-of-Pre-Trained-CNN-Models-for-Cardiovascular-Disease-Evaluation-of-Pre-Trained-CNN-Models-for-Cardiovascular-Disease-Classification-A-Benchmark-Study-Classification-A-Benchmark-Study-Evalua.pdf."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1186\/s13634-018-0545-9","article-title":"Localization and Classification of Heart Beats in Phonocardiography Signals\u2014A Comprehensive Review","volume":"2018","author":"Ismail","year":"2018","journal-title":"EURASIP J. Adv. Signal Process."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1056","DOI":"10.1016\/j.eswa.2006.12.015","article-title":"Comparison of Envelope Extraction Algorithms for Cardiac Sound Signal Segmentation","volume":"34","author":"Choi","year":"2008","journal-title":"Expert Syst. Appl."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Schmidt, S.E., Toft, E., Holst-Hansen, C., Graff, C., and Struijk, J.J. (2008, January 14\u201317). Segmentation of Heart Sound Recordings from an Electronic Stethoscope by a Duration Dependent Hidden-Markov Model. Proceedings of the 2008 Computers in Cardiology, Bologna, Italy.","DOI":"10.1109\/CIC.2008.4749049"},{"key":"ref_6","first-page":"822","article-title":"Logistic Regression-HSMM-Based Heart Sound Segmentation","volume":"63","author":"Springer","year":"2015","journal-title":"IEEE Trans. Biomed. Eng."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Boulares, M., Alotaibi, R., AlMansour, A., and Barnawi, A. (2021). Cardiovascular Disease Recognition Based on Heartbeat Segmentation and Selection Process. Int. J. Environ. Res. Public Health, 18.","DOI":"10.3390\/ijerph182010952"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"118722","DOI":"10.1109\/ACCESS.2023.3325749","article-title":"HeartWave: A Multiclass Dataset of Heart Sounds for Cardiovascular Diseases Detection","volume":"11","author":"Alrabie","year":"2023","journal-title":"IEEE Access"},{"key":"ref_9","first-page":"44","article-title":"DARPA\u2019s Explainable Artificial Intelligence (XAI) Program","volume":"40","author":"Gunning","year":"2019","journal-title":"AI Mag."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"129701","DOI":"10.1016\/j.neucom.2025.129701","article-title":"TIxAI: A Trustworthiness Index for eXplainable AI in skin lesions classification","volume":"630","author":"Ieracitano","year":"2025","journal-title":"Neurocomputing"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"108517","DOI":"10.1016\/j.engappai.2024.108517","article-title":"An explainable embedded neural system for on-board ship detection from optical satellite imagery","volume":"133","author":"Ieracitano","year":"2024","journal-title":"Eng. Appl. Artif. Intell."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"S174","DOI":"10.1016\/j.jtho.2023.09.260","article-title":"MA19.11 Predicting Future Lung Cancer Risk with Low-Dose Screening CT Using an Artificial Intelligence Model","volume":"18","author":"Yuan","year":"2023","journal-title":"J. Thorac. Oncol."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Devnath, L., Fan, Z., Luo, S., Summons, P., and Wang, D. (2022). Detection and Visualisation of Pneumoconiosis Using an Ensemble of Multi-Dimensional Deep Features Learned from Chest X-rays. Int. J. Environ. Res. Public Health, 19.","DOI":"10.3390\/ijerph191811193"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Zhang, Y., Weng, Y., and Lund, J. (2022). Applications of Explainable Artificial Intelligence in Diagnosis and Surgery. Diagnostics, 12.","DOI":"10.3390\/diagnostics12020237"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"101286","DOI":"10.1016\/j.imu.2023.101286","article-title":"Application of Explainable Artificial Intelligence in Medical Health: A Systematic Review of Interpretability Methods","volume":"40","author":"Band","year":"2023","journal-title":"Informatics Med. Unlocked"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"109475","DOI":"10.1109\/ACCESS.2020.3002151","article-title":"Transfer Learning Benchmark for Cardiovascular Disease Recognition","volume":"8","author":"Boulares","year":"2020","journal-title":"IEEE Access"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Zhang, C., Yang, T., and Yang, J. (2022). Image Recognition of Wind Turbine Blade Defects Using Attention-Based MobileNetv1-YOLOv4 and Transfer Learning. Sensors, 22.","DOI":"10.3390\/s22166009"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s12911-021-01736-y","article-title":"ECG Signal Classification Based on Deep CNN and BiLSTM","volume":"21","author":"Cheng","year":"2021","journal-title":"BMC Med. Inform. Decis. Mak."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"e13791","DOI":"10.1111\/exsy.13791","article-title":"An Attention-Driven Hybrid Deep Neural Network for Enhanced Heart Disease Classification","volume":"42","author":"Lilhore","year":"2025","journal-title":"Expert Syst."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Li, M., He, Z., and Wang, H. (2025). Heart Sound Classification Based on Multi-Scale Feature Fusion and Channel Attention Module. Bioengineering, 12.","DOI":"10.3390\/bioengineering12030290"},{"key":"ref_21","unstructured":"Han, J., and Shaout, A. (2025). ENACT-Heart\u2013ENsemble-Based Assessment Using CNN and Transformer on Heart Sounds. arXiv."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Wu, T., Huang, Z., Li, S., Zhao, Q., and Pan, F. (2024). Heart Murmur Quality Detection Using Deep Neural Networks with Attention Mechanism. Appl. Sci., 14.","DOI":"10.3390\/app14156825"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Ranipa, K., Zhu, W.-P., and Swamy, M.N.S. (2024). A Novel Feature-Level Fusion Scheme with Multimodal Attention CNN for Heart Sound Classification. Comput. Methods Programs Biomed., 248.","DOI":"10.1016\/j.cmpb.2024.108122"},{"key":"ref_24","unstructured":"Oliveira, B.F.F.A. (2024). Deep Learning for Heart Sounds and Electrocardiogram Signal Analysis: On the Impact of Multimodal Data for Explainable AI. [Master\u2019s Thesis, University of Porto]."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Mohanty, S., and Patra, S. (2024). Integration of Multi-Feature Analysis with Lightweight CNN Model for Heart Sound Classification. Proceedings of the International Conference on Advanced Network Technologies and Intelligent Computing, Springer.","DOI":"10.1007\/978-3-031-83790-6_17"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1007\/s13755-024-00302-w","article-title":"Rapid Detection and Interpretation of Heart Murmurs Using Phonocardiograms, Transfer Learning and Explainable Artificial Intelligence","volume":"12","year":"2024","journal-title":"Health Inf. Sci. Syst."},{"key":"ref_27","first-page":"1","article-title":"Automated Detection of Heart Valve Diseases Using Stationary Wavelet Transform and Attention-Based Hierarchical LSTM Network","volume":"72","author":"Das","year":"2023","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Yadav, H., Shah, P., Gandhi, N., Vyas, T., Nair, A., Desai, S., Gohil, L., Tanwar, S., Sharma, R., and Verdes, M. (2023). CNN and Bidirectional GRU-Based Heartbeat Sound Classification Architecture for Elderly People. Mathematics, 11.","DOI":"10.3390\/math11061365"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"3811","DOI":"10.1007\/s00371-023-03031-5","article-title":"PCTMF-Net: Heart Sound Classification with Parallel CNNs\u2013Transformer and Second-Order Spectral Analysis","volume":"39","author":"Wang","year":"2023","journal-title":"Vis. Comput."},{"key":"ref_30","unstructured":"Netto, A.N., Abraham, L., and Philip, S. (2023, January 26\u201328). Detection of Congenital Heart Disease from Heart Sounds Using 2D CNN-BiLSTM with Attention Mechanism. Proceedings of the International Conference on Machine Learning and Big Data Analytics, Shenzhen, China."},{"key":"ref_31","first-page":"438","article-title":"Abnormal heart sound classification and model interpretability: A transfer learning approach with deep learning","volume":"2","author":"Marocchi","year":"2023","journal-title":"J. Vasc. Dis."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Bao, X., Xu, Y., Lam, H.-K., Trabelsi, M., Chihi, I., Sidhom, L., and Kamavuako, E.N. (2023). Time-Frequency Distributions of Heart Sound Signals: A Comparative Study Using Convolutional Neural Networks. Biomed. Eng. Adv., 5.","DOI":"10.1016\/j.bea.2023.100093"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Cheng, J., and Sun, K. (2023). Heart Sound Classification Network Based on Convolution and Transformer. Sensors, 23.","DOI":"10.3390\/s23198168"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Barnawi, A., Boulares, M., and Somai, R. (2023). Simple and Powerful PCG Classification Method Based on Selection and Transfer Learning for Precision Medicine Application. Bioengineering, 10.","DOI":"10.3390\/bioengineering10030294"},{"key":"ref_35","first-page":"100322","article-title":"Deep Attention-Based Neural Networks for Explainable Heart Sound Classification","volume":"9","author":"Ren","year":"2022","journal-title":"Mach. Learn. Appl."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"1378","DOI":"10.1007\/s12559-022-10009-3","article-title":"Imbalanced Heart Sound Signal Classification Based on Two-Stage Trained DsaNet","volume":"14","author":"Tian","year":"2022","journal-title":"Cogn. Comput."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"108900","DOI":"10.1016\/j.apacoust.2022.108900","article-title":"Diagnosis of Exercise-Induced Cardiac Fatigue Based on Deep Learning and Heart Sounds","volume":"197","author":"Yin","year":"2022","journal-title":"Appl. Acoust."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Ren, Z., Qian, K., Dong, F., Dai, Z., Yamamoto, Y., and Schuller, B.W. (2021). Deep Attention-Based Representation Learning for Heart Sound Classification. arXiv.","DOI":"10.1016\/j.mlwa.2022.100322"},{"key":"ref_39","first-page":"1","article-title":"Detection of Phonocardiogram Event Patterns in Mitral Valve Prolapse: An Automated Clinically Relevant Explainable Diagnostic Framework","volume":"72","author":"Rajeshwari","year":"2023","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"ref_40","first-page":"679","article-title":"The auscultatory areas and the interpretation of heart sounds and murmurs","volume":"12","author":"Littmann","year":"1963","journal-title":"Am. J. Cardiol."},{"key":"ref_41","unstructured":"(2025, May 14). Stethophone. Available online: https:\/\/stethophone.com\/us\/en\/about\/."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"102470","DOI":"10.1016\/j.media.2022.102470","article-title":"Explainable Artificial Intelligence (XAI) in Deep Learning-Based Medical Image Analysis","volume":"79","author":"Kuijf","year":"2022","journal-title":"Med. Image Anal."},{"key":"ref_43","unstructured":"Husby, U.E. (2024). Exploring Breast Cancer Diagnosis: A Study of SHAP and LIME in XAI-Driven Medical Imaging. [Master\u2019s Thesis, Norwegian University of Life Sciences]."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"110786","DOI":"10.1016\/j.ejrad.2023.110786","article-title":"Explainable AI in Medical Imaging: An Overview for Clinical Practitioners\u2014Beyond Saliency-Based XAI Approaches","volume":"162","author":"Borys","year":"2023","journal-title":"Eur. J. Radiol."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Singh, A., Sengupta, S., and Lakshminarayanan, V. (2020). Explainable Deep Learning Models in Medical Image Analysis. J. Imaging, 6.","DOI":"10.3390\/jimaging6060052"},{"key":"ref_46","first-page":"5998","article-title":"Attention Is All You Need","volume":"30","author":"Vaswani","year":"2017","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Hu, J., Shen, L., and Sun, G. (2018, January 18\u201323). Squeeze-and-Excitation Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00745"},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Sallam, A., Gaid, A.S.A., Saif, W.Q.A., Kaid, H.A.S., Abdulkareem, R.A., Ahmed, K.J.A., Saeed, A.Y.A., and Radman, A. (2021, January 22\u201324). Early Detection of Glaucoma Using Transfer Learning from Pre-Trained CNN Models. Proceedings of the 2021 International Conference of Technology, Science and Administration (ICTSA), Taiz, Yemen.","DOI":"10.1109\/ICTSA52017.2021.9406522"},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27\u201330). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_50","unstructured":"Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017). MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv."},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.-C. (2018, January 18\u201322). MobileNetV2: Inverted Residuals and Linear Bottlenecks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00474"},{"key":"ref_52","unstructured":"Simonyan, K., and Zisserman, A. (2014). Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv."},{"key":"ref_53","unstructured":"Tan, M., and Le, Q. (2021, January 18\u201324). EfficientNetV2: Smaller Models and Faster Training. Proceedings of the 38th International Conference on Machine Learning, PMLR, Virtual."},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Kim, H.E., Cosa-Linan, A., Santhanam, N., Jannesari, M., Maros, M.E., and Ganslandt, T. (2022). Transfer Learning for Medical Image Classification: A Literature Review. BMC Med. Imaging, 22.","DOI":"10.1186\/s12880-022-00793-7"},{"key":"ref_55","unstructured":"Team, K. (2024, February 10). Keras Applications Documentation. Keras.io, Available online: https:\/\/keras.io\/api\/applications\/."},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. (2017, January 22\u201329). Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy.","DOI":"10.1109\/ICCV.2017.74"},{"key":"ref_57","unstructured":"Holzinger, A., Biemann, C., Pattichis, C.S., and Kell, D.B. (2017). What do we need to build explainable AI systems for the medical domain?. arXiv."},{"key":"ref_58","doi-asserted-by":"crossref","first-page":"44","DOI":"10.1038\/s41591-018-0300-7","article-title":"High-performance medicine: The convergence of human and artificial intelligence","volume":"25","author":"Topol","year":"2019","journal-title":"Nat. Med."},{"key":"ref_59","unstructured":"European Commission (2025, April 20). Proposal for a Regulation Laying Down Harmonised Rules on Artificial Intelligence (Artificial Intelligence Act) and Amending Certain Union Legislative Acts. COM\/2021\/206 Final, Brussels. Available online: https:\/\/eur-lex.europa.eu\/legal-content\/EN\/TXT\/?uri=celex:52021PC0206."}],"container-title":["Bioengineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2306-5354\/12\/6\/558\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T17:38:30Z","timestamp":1760031510000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2306-5354\/12\/6\/558"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,5,22]]},"references-count":59,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2025,6]]}},"alternative-id":["bioengineering12060558"],"URL":"https:\/\/doi.org\/10.3390\/bioengineering12060558","relation":{},"ISSN":["2306-5354"],"issn-type":[{"value":"2306-5354","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,5,22]]}}}