{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,19]],"date-time":"2026-08-19T17:43:29Z","timestamp":1787161409681,"version":"build-2736575974"},"reference-count":39,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Med Inform"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Echocardiography is a fundamental imaging modality for the diagnosis of heart disease (HD), but its interpretation remains operator-dependent and lacks standardized, data-driven decision support. Although artificial intelligence has improved image-based diagnosis, the added value and interpretability of integrating routinely collected electronic medical records (EMRs) with echocardiogram (ECHO) for large-scale screening remain underexplored.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aims to develop an explainable artificial intelligence framework that integrates multimodal data, including ECHO images and EMRs, with derived temporal and clinical features, to enhance the screening and interpretability of heart disease diagnosis.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n In this retrospective single-center study, we analyzed 26,936 patients with linked ECHO and EMR records (1470 with clinically confirmed HD and 25,466 without). We constructed cross-sectional EMR features (demographics, comorbidities, and registration history) and extracted view-specific image features with a simplified Inception-v3 backbone. Five modality-specific feature vectors (4 imaging views + EMR) were concatenated and input to an extreme gradient boosting classifier within a patient-level stratified 5-fold cross-validation framework. To mitigate class imbalance during training, non-HD cases in the training folds were randomly downsampled to a 1:3 HD:non-HD ratio; held-out test folds preserved the original prevalence. Post hoc explainability was provided via Grad-CAM heatmaps for images and Shapley additive explanations analysis for EMR features.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n For the primary binary task (presence vs absence of HD), the multimodal model achieved an AUC of 0.8147 (SD 0.009) on held-out test folds, compared with 0.7785 (SD 0.014) for the image-only baseline, and 0.7343 (SD 0.009) for the EMR-only baseline. At the clinically selected decision threshold, sensitivity, specificity, positive predictive value, and negative predictive value were 84.6% (SD 1.6%), 72.4% (SD 2%), 15% (SD 2.3%), 98.8% (SD 0.3%) for the multimodal model, versus 82.3% (SD 1.9%), 73.5% (SD 2%), 15.2% (SD 2.1%), 98.6% (SD 0.3%) (image-only), and 77.5% (SD 2.1%), 67% (SD 2.3%), 11.9% (SD 1.7%), 98.1% (SD 0.4%) (EMR-only). Grad-CAM visualizations qualitatively highlighted anatomically and physiologically meaningful regions (eg, valvular structures and abnormal flow jets), while Shapley additive explanations analysis identified age, registration years, sex, and hypertension among the top EMR contributors\u2014findings that align with established cardiovascular risk factors.<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n Integrating echocardiographic images and EMR data in an explainable multimodal framework yields improved and clinically plausible screening performance for heart disease. Future work should focus on external multicenter validation, quantitative assessment of visual explanations against expert annotations, and prospective studies assessing clinical use and workflow integration.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/78949","type":"journal-article","created":{"date-parts":[[2026,5,14]],"date-time":"2026-05-14T19:42:46Z","timestamp":1778787766000},"page":"e78949-e78949","source":"Crossref","is-referenced-by-count":2,"title":["Multimodal Fusion of Echocardiogram Images and Electronic Medical Records for Heart Disease Screening: Retrospective Algorithm Development and Validation Study"],"prefix":"10.2196","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6667-0910","authenticated-orcid":false,"given":"Bokai","family":"Yang","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-6122-5840","authenticated-orcid":false,"given":"Yirong","family":"Qin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5351-8546","authenticated-orcid":false,"given":"Ye","family":"Li","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4180-8212","authenticated-orcid":false,"given":"Ruitao","family":"Xie","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6321-7571","authenticated-orcid":false,"given":"Limai","family":"Jiang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-6942-6309","authenticated-orcid":false,"given":"Juan","family":"He","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4071-0245","authenticated-orcid":false,"given":"Jikui","family":"Liu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8797-4243","authenticated-orcid":false,"given":"Yunpeng","family":"Cai","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,5,14]]},"reference":[{"issue":"1","key":"R1","doi-asserted-by":"publisher","DOI":"10.1186\/s12889-023-16466-1","article-title":"The global estimate of premature cardiovascular mortality: a systematic review and meta-analysis of age-standardized mortality rate","volume":"23","author":"Hasani","journal-title":"BMC Public Health"},{"issue":"7","key":"R2","doi-asserted-by":"publisher","first-page":"465","DOI":"10.1038\/s41569-020-00503-2","article-title":"Artificial intelligence-enhanced electrocardiography in cardiovascular disease management","volume":"18","author":"Siontis","journal-title":"Nat Rev Cardiol"},{"key":"R3","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2025.104800","article-title":"Enhanced heart failure mortality prediction through model-independent hybrid feature selection and explainable machine learning","volume":"163","author":"Petmezas","journal-title":"J Biomed Inform"},{"issue":"1","key":"R4","doi-asserted-by":"publisher","first-page":"19","DOI":"10.1007\/s12551-022-01040-7","article-title":"Recent developments in modeling, imaging, and monitoring of cardiovascular diseases using machine learning","volume":"15","author":"Moradi","journal-title":"Biophys Rev"},{"issue":"8","key":"R5","doi-asserted-by":"publisher","first-page":"1153","DOI":"10.1016\/j.ultrasmedbio.2025.03.015","article-title":"Deep learning in echocardiography for enhanced detection of left ventricular function and wall motion abnormalities","volume":"51","author":"Alhussein","journal-title":"Ultrasound Med Biol"},{"issue":"11","key":"R6","doi-asserted-by":"publisher","first-page":"987","DOI":"10.1016\/s0195-668x(03)00114-3","article-title":"Estimation of ten-year risk of fatal cardiovascular disease in Europe: the SCORE project","volume":"24","author":"Conroy","journal-title":"Eur Heart J"},{"issue":"7611","key":"R7","doi-asserted-by":"publisher","first-page":"136","DOI":"10.1136\/bmj.39261.471806.55","article-title":"Derivation and validation of QRISK, a new cardiovascular disease risk score for the United Kingdom: prospective open cohort study","volume":"335","author":"Hippisley-Cox","journal-title":"BMJ"},{"key":"R8","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.c6624","article-title":"Derivation, validation, and evaluation of a new QRISK model to estimate lifetime risk of cardiovascular disease: cohort study using QResearch database","volume":"341","author":"Hippisley-Cox","journal-title":"BMJ"},{"issue":"6","key":"R9","doi-asserted-by":"publisher","first-page":"690","DOI":"10.1001\/jamacardio.2021.0948","article-title":"Performance of the American Heart Association\/American College of Cardiology pooled cohort equations to estimate atherosclerotic cardiovascular disease risk by self-reported physical activity levels","volume":"6","author":"Pandey","journal-title":"JAMA Cardiol"},{"issue":"1","key":"R10","doi-asserted-by":"publisher","first-page":"33","DOI":"10.1016\/j.jacasi.2021.10.007","article-title":"Evaluation of atherosclerotic cardiovascular risk prediction models in China: results from the CHERRY study","volume":"2","author":"Liu","journal-title":"JACC Asia"},{"issue":"9","key":"R11","doi-asserted-by":"publisher","first-page":"6187","DOI":"10.1007\/s11042-024-19127-6","article-title":"Early detection and prediction of heart disease using wearable devices and deep learning algorithms","volume":"84","author":"Sivasubramaniam","journal-title":"Multimed Tools Appl"},{"key":"R12","doi-asserted-by":"publisher","DOI":"10.1016\/j.hjc.2021.07.006","article-title":"Updated knowledge and practical implementations of stress echocardiography in ischemic and non-ischemic cardiac diseases: an expert consensus of the working group of echocardiography of the Hellenic Society of Cardiology","volume":"64","author":"Kadoglou","journal-title":"Hellenic J Cardiol"},{"key":"R13","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijcard.2023.01.024","article-title":"Handheld echocardiographic screening for rheumatic heart disease by non-experts in rural South Kordofan, Sudan: supporting task shifting for control of a serious disease","volume":"377","author":"Elazrag","journal-title":"Int J Cardiol"},{"issue":"6","key":"R14","doi-asserted-by":"publisher","first-page":"e983","DOI":"10.1016\/S2214-109X(24)00127-X","article-title":"Handheld echocardiography for the screening and diagnosis of rheumatic heart disease: a systematic review to inform WHO guidelines","volume":"12","author":"Provid\u00eancia","journal-title":"Lancet Glob Health"},{"issue":"7","key":"R15","doi-asserted-by":"publisher","first-page":"733","DOI":"10.1016\/j.echo.2023.02.007","article-title":"Abbreviated echocardiographic screening for rheumatic heart disease by nonexperts with and without offsite expert review: a diagnostic accuracy study","volume":"36","author":"Francis","journal-title":"J Am Soc Echocardiogr"},{"issue":"1","key":"R16","doi-asserted-by":"publisher","first-page":"10","DOI":"10.1016\/j.rec.2022.04.002","article-title":"Cardiac tomography-echocardiography imaging fusion: a new approach to congenital heart disease","volume":"76","author":"Fournier","journal-title":"Revista Espa\u00f1ola de Cardiolog\u00eda (English Edition)"},{"issue":"8","key":"R17","doi-asserted-by":"publisher","first-page":"1051","DOI":"10.1093\/ehjci\/jeae127","article-title":"Prevalence of asymptomatic valvular heart disease in the elderly population: a community-based echocardiographic study","volume":"25","author":"Tsampasian","journal-title":"Eur Heart J Cardiovasc Imaging"},{"issue":"1","key":"R18","doi-asserted-by":"publisher","DOI":"10.1186\/s44348-024-00019-0","article-title":"2023 Korean Society of Echocardiography position paper for diagnosis and management of valvular heart disease, part I: aortic valve disease","volume":"32","author":"Lee","journal-title":"J Cardiovasc Imaging"},{"issue":"1","key":"R19","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1016\/j.echo.2022.10.009","article-title":"Recommendations for the use of echocardiography in the evaluation of rheumatic heart disease: a report from the American Society of Echocardiography","volume":"36","author":"Pandian","journal-title":"J Am Soc Echocardiogr"},{"issue":"4","key":"R20","doi-asserted-by":"publisher","first-page":"250","DOI":"10.1038\/s41569-023-00940-9","article-title":"2023 World Heart Federation guidelines for the echocardiographic diagnosis of rheumatic heart disease","volume":"21","author":"Rwebembera","journal-title":"Nat Rev Cardiol"},{"issue":"20","key":"R21","doi-asserted-by":"publisher","first-page":"10633","DOI":"10.1007\/s00500-022-06780-y","article-title":"Heart disease diagnosis using deep learning and cardiac color Doppler ultrasound","volume":"26","author":"Wang","journal-title":"Soft Comput"},{"key":"R22","doi-asserted-by":"crossref","unstructured":"Bhagawati M Paul S . Generative adversarial network-based deep learning framework for cardiovascular disease risk prediction. Presented at: 2024 5th International Conference on Innovative Trends in Information Technology (ICITIIT); Mar 15-16, 2024:1-4; Kottayam, India. [doi: 10.1109\/ICITIIT61487.2024.10580562]","DOI":"10.1109\/ICITIIT61487.2024.10580562"},{"key":"R23","volume":"219","author":"Huang","journal-title":"Proc Mach Learn Res"},{"issue":"4","key":"R24","doi-asserted-by":"publisher","first-page":"551","DOI":"10.1016\/j.jcmg.2021.08.015","article-title":"Automated analysis of Doppler echocardiographic videos as a screening tool for valvular heart diseases","volume":"15","author":"Yang","journal-title":"JACC Cardiovasc Imaging"},{"issue":"20","key":"R25","doi-asserted-by":"publisher","first-page":"1592","DOI":"10.1136\/heartjnl-2021-319725","article-title":"Artificial intelligence for the echocardiographic assessment of valvular heart disease","volume":"108","author":"Nedadur","journal-title":"Heart"},{"issue":"3","key":"R26","doi-asserted-by":"publisher","DOI":"10.1136\/wjps-2023-000580","article-title":"A deep learning-based method for pediatric congenital heart disease detection with seven standard views in echocardiography","volume":"6","author":"Jiang","journal-title":"World J Pediatr Surg"},{"key":"R27","doi-asserted-by":"publisher","DOI":"10.34133\/research.0319","article-title":"Development and validation of a deep-learning network for detecting congenital heart disease from multi-view multi-modal transthoracic echocardiograms","volume":"7","author":"Cheng","journal-title":"Research (Wash D C)"},{"key":"R28","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2023.3316719","article-title":"Deep learning technique for congenital heart disease detection using stacking-based CNN-LSTM models from fetal echocardiogram: a pilot study","volume":"11","author":"Rahman","journal-title":"IEEE Access"},{"issue":"6","key":"R29","doi-asserted-by":"publisher","first-page":"733","DOI":"10.1002\/pd.5892","article-title":"Artificial intelligence, fetal echocardiography, and congenital heart disease","volume":"41","author":"Day","journal-title":"Prenat Diagn"},{"issue":"5","key":"R30","doi-asserted-by":"publisher","first-page":"1007","DOI":"10.1007\/s10554-022-02566-3","article-title":"Application of machine learning in screening for congenital heart diseases using fetal echocardiography","volume":"38","author":"Truong","journal-title":"Int J Cardiovasc Imaging"},{"issue":"2","key":"R31","doi-asserted-by":"publisher","DOI":"10.1161\/JAHA.123.031257","article-title":"Using artificial intelligence for rheumatic heart disease detection by echocardiography: focus on mitral regurgitation","volume":"13","author":"Brown","journal-title":"J Am Heart Assoc"},{"issue":"9","key":"R32","doi-asserted-by":"publisher","first-page":"1834","DOI":"10.1093\/jamia\/ocab061","article-title":"Towards automatic diagnosis of rheumatic heart disease on echocardiographic exams through video-based deep learning","volume":"28","author":"Martins","journal-title":"J Am Med Inform Assoc"},{"issue":"3","key":"R33","doi-asserted-by":"publisher","first-page":"469","DOI":"10.3390\/sym17030469","article-title":"AI-driven technology in heart failure detection and diagnosis: a review of the advancement in personalized healthcare","volume":"17","author":"Udoy","journal-title":"Symmetry (Basel)"},{"issue":"1","key":"R34","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1093\/ehjdh\/ztad072","article-title":"External validation of a deep learning algorithm for automated echocardiographic strain measurements","volume":"5","author":"Myhre","journal-title":"Eur Heart J Digit Health"},{"issue":"9","key":"R35","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0310107","article-title":"Enhancing reginal wall abnormality detection accuracy: Integrating machine learning, optical flow algorithms, and temporal convolutional networks in multi-view echocardiography","volume":"19","author":"Kasim","journal-title":"PLoS ONE"},{"key":"R36","doi-asserted-by":"crossref","unstructured":"Jaenputra JJ Sigit R Harsono T Firdaus AAA . Heart disease detection from PSAX echocardiography view using ultrasound portable based on machine learning method. Presented at: 2024 International Electronics Symposium (IES); Aug 6-8, 2024:637-643; Denpasar, Indonesia. [doi: 10.1109\/IES63037.2024.10665785]","DOI":"10.1109\/IES63037.2024.10665785"},{"key":"R37","doi-asserted-by":"publisher","DOI":"10.56979\/702\/2024","article-title":"Enhancing heart disease detection in echocardiogram images using optimized EfficientNetB3 architecture","volume":"7","author":"Mahtab","journal-title":"J Comput Biomed Inform"},{"key":"R38","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2024.125849","article-title":"Automatic cardiac pathology recognition in echocardiography images using higher order dynamic mode decomposition and a vision transformer for small datasets","volume":"264","author":"Bell-Navas","journal-title":"Expert Syst Appl"},{"issue":"5","key":"R39","doi-asserted-by":"publisher","first-page":"1505","DOI":"10.1002\/ima.22885","article-title":"CarpNet: transformer for mitral valve disease classification in echocardiographic videos","volume":"33","author":"Vafaeezadeh","journal-title":"Int J Imaging Syst Tech"}],"container-title":["JMIR Medical Informatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/medinform.jmir.org\/2026\/1\/e78949","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:11:46Z","timestamp":1782843106000},"score":1,"resource":{"primary":{"URL":"https:\/\/medinform.jmir.org\/2026\/1\/e78949"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,5,14]]},"references-count":39,"URL":"https:\/\/doi.org\/10.2196\/78949","relation":{},"ISSN":["2291-9694"],"issn-type":[{"value":"2291-9694","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,5,14]]},"article-number":"v14i1e78949"}}