{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,29]],"date-time":"2026-09-29T07:18:04Z","timestamp":1790666284443,"version":"4.1.0"},"reference-count":20,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["J Med Internet Res"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Vision-enabled large language models (VE-LLMs) have the potential to provide flexible and explainable medical image interpretation. However, their real-world performance on clinical data, such as 12-lead electrocardiograms (ECGs), has not been systematically assessed.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aimed to evaluate the diagnostic accuracy and reliability of state-of-the-art generalist VE-LLMs in interpreting real-world ECG images.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n We tested 6 generalist VE-LLMs (ChatGPT-5, ChatGPT-4, Gemini 2.5, Copilot, Claude Sonnet-4, and Claude Opus-4.1) using 70 deidentified ECG images. A standardized prompt requested 9 determinations: rhythm, first-degree atrioventricular (AV) block, intraventricular conduction block and pattern, corrected QT (QTc) prolongation, premature atrial and ventricular contractions, ischemic ST-segment deviation, and axis deviation. An expert consensus served as the reference standard. Moreover, 2 image-based ECG-specialized LLMs (PULSE-7B and ECG-Instruct-Llama-3.2-11B-Vision) were tested for exploratory comparison. Model outputs were evaluated using overall and per-category diagnostic metrics.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n Overall balanced accuracy across generalist models ranged from 50.1% to 61.8% (Cochran Q,\n P<\/jats:italic>\n <.001). ChatGPT-5 achieved the highest balanced accuracy (61.8%) but had the slowest response time (median 276, IQR 110-407 s), whereas Copilot responded within a median of 3 (IQR 2-4) seconds. Balanced accuracy for rhythm classification ranged from 38.6% to 55.8%, but sensitivity for atrial fibrillation among generalist models was \u226411.1%, detecting either none or only 1 of the 9 cases. Detection of first-degree AV block (sensitivity 0%\u201022%; 0\/9 to 2\/9) and QTc prolongation (sensitivity 0%\u201045.5%; 0\/22 to 10\/22) was poor. Intraventricular block was identified with up to 67.8% balanced accuracy, but correct subtype assignment was \u226444% (\u226411\/25). ST-segment deviation sensitivity was <25% for all generalist models (highest 3\/14). Agreement with expert interpretation was low, with Cohen \u03ba indicating poor-to-fair concordance (\u03ba\u22640.39). Specialized models achieved overall balanced accuracy of 56.5% (ECG-Instruct-Llama-3.2-11B-Vision) and 64.4% (PULSE-7B), with PULSE-7B showing higher task-specific balanced accuracy in rhythm classification and ectopic beats detection (up to 86.3% and 89.2%, respectively).\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n VE-LLMs showed moderate overall performance but mostly low sensitivity and limited agreement with expert ECG interpretation. Current performance remains inconsistent across models and diagnostic categories and is insufficient to support clinical deployment.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/86692","type":"journal-article","created":{"date-parts":[[2026,4,24]],"date-time":"2026-04-24T17:50:07Z","timestamp":1777053007000},"page":"e86692-e86692","source":"Crossref","is-referenced-by-count":3,"title":["Performance of Vision-Enabled Large Language Models in Image-Based Electrocardiogram Interpretation: Exploratory Evaluation"],"prefix":"10.2196","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0272-0525","authenticated-orcid":false,"given":"Nibras","family":"Soubh","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-6418-5883","authenticated-orcid":false,"given":"Eva","family":"Rasenack","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-0566-5827","authenticated-orcid":false,"given":"Helge","family":"Haarmann","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8168-0025","authenticated-orcid":false,"given":"Felix","family":"Wiedmann","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0895-4602","authenticated-orcid":false,"given":"Markus","family":"Zabel","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8702-4977","authenticated-orcid":false,"given":"Constanze","family":"Schmidt","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-2327-7702","authenticated-orcid":false,"given":"Rayan","family":"Suliman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7025-9315","authenticated-orcid":false,"given":"Leonard","family":"Bergau","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,6,3]]},"reference":[{"issue":"3","key":"R1","doi-asserted-by":"publisher","first-page":"943","DOI":"10.1038\/s41591-024-03423-7","article-title":"Toward expert-level medical question answering with large language models","volume":"31","author":"Singhal","journal-title":"Nat Med"},{"issue":"1","key":"R2","doi-asserted-by":"publisher","DOI":"10.1186\/s12909-024-05944-8","article-title":"Performance of ChatGPT-3.5 and GPT-4 in national licensing examinations for medicine, pharmacy, dentistry, and nursing: a systematic review and meta-analysis","volume":"24","author":"Jin","journal-title":"BMC Med Educ"},{"key":"R3","doi-asserted-by":"publisher","DOI":"10.2196\/60807","article-title":"Performance of ChatGPT across different versions in medical licensing examinations worldwide: systematic review and meta-analysis","volume":"26","author":"Liu","journal-title":"J Med Internet Res"},{"issue":"2","key":"R4","doi-asserted-by":"publisher","DOI":"10.1001\/jamanetworkopen.2024.60637","article-title":"Clinician experiences with ambient scribe technology to assist with documentation burden and efficiency","volume":"8","author":"Duggan","journal-title":"JAMA Netw Open"},{"issue":"12","key":"R5","doi-asserted-by":"publisher","first-page":"1898","DOI":"10.1016\/j.jacr.2024.06.018","article-title":"The impact of large language model-generated radiology report summaries on patient comprehension: a randomized controlled trial","volume":"21","author":"Berigan","journal-title":"J Am Coll Radiol"},{"issue":"12","key":"R6","doi-asserted-by":"publisher","first-page":"1131","DOI":"10.1001\/jamacardio.2023.3701","article-title":"Deep learning of electrocardiograms in sinus rhythm from US veterans to predict atrial fibrillation","volume":"8","author":"Yuan","journal-title":"JAMA Cardiol"},{"issue":"46","key":"R7","doi-asserted-by":"publisher","first-page":"4920","DOI":"10.1093\/eurheartj\/ehae595","article-title":"Prediction of incident atrial fibrillation using deep learning, clinical models, and polygenic scores","volume":"45","author":"Jabbour","journal-title":"Eur Heart J"},{"issue":"8075","key":"R8","doi-asserted-by":"publisher","first-page":"221","DOI":"10.1038\/s41586-025-09227-0","article-title":"Detecting structural heart disease from electrocardiograms using AI","volume":"644","author":"Poterucha","journal-title":"Nature"},{"key":"R9","doi-asserted-by":"publisher","DOI":"10.1016\/j.ajem.2024.07.043","article-title":"The accuracy of Gemini, GPT-4, and GPT-4o in ECG analysis: a comparison with cardiologists and emergency medicine specialists","volume":"84","author":"G\u00fcnay","journal-title":"Am J Emerg Med"},{"key":"R10","doi-asserted-by":"publisher","DOI":"10.2196\/54607","article-title":"Multimodal ChatGPT-4V for electrocardiogram interpretation: promise and limitations","volume":"26","author":"Zhu","journal-title":"J Med Internet Res"},{"issue":"1","key":"R11","doi-asserted-by":"publisher","DOI":"10.1136\/bmjdhai-2025-000031","article-title":"High-accuracy ECG image interpretation using parameter-efficient Low-Rank Adaptation (LoRA) fine-tuning with multimodal LLaMA V.3.2","volume":"1","author":"Mukkunnoth","journal-title":"BMJ Digit Health"},{"key":"R12","unstructured":"Liu R Bai Y Yue X Zhang P . Teach multimodal LLMs to comprehend electrocardiographic images. arXiv. Preprint posted online on Oct 21, 2024. [doi: 10.48550\/arXiv.2410.19008]"},{"key":"R13","doi-asserted-by":"publisher","DOI":"10.2196\/74426","article-title":"Effectiveness of the GPT-4o model in interpreting electrocardiogram images for cardiac diagnostics: diagnostic accuracy study","volume":"4","author":"Engelstein","journal-title":"JMIR AI"},{"issue":"7","key":"R14","doi-asserted-by":"publisher","first-page":"1406","DOI":"10.1016\/j.cjca.2025.01.036","article-title":"Beyond text: the impact of clinical context on GPT-4\u2019s 12-lead electrocardiogram interpretation accuracy","volume":"41","author":"Zeljkovic","journal-title":"Can J Cardiol"},{"key":"R15","doi-asserted-by":"publisher","DOI":"10.2196\/75910","article-title":"Comparative diagnostic performance of a multimodal large language model versus a dedicated electrocardiogram AI in detecting myocardial infarction from electrocardiogram images: comparative study","volume":"4","author":"Lee","journal-title":"JMIR AI"},{"issue":"1","key":"R16","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-020-15432-4","article-title":"Automatic diagnosis of the 12-lead ECG using a deep neural network","volume":"11","author":"Ribeiro","journal-title":"Nat Commun"},{"issue":"1","key":"R17","doi-asserted-by":"publisher","DOI":"10.1007\/s12672-025-02005-4","article-title":"A preliminary investigation into the potential, pitfalls, and limitations of large language models for mammography interpretation","volume":"16","author":"Pesapane","journal-title":"Discov Oncol"},{"key":"R18","doi-asserted-by":"publisher","DOI":"10.3389\/fcvm.2025.1458289","article-title":"Assessing the performance of zero-shot visual question answering in multimodal large language models for 12-lead ECG image interpretation","volume":"12","author":"Seki","journal-title":"Front Cardiovasc Med"},{"issue":"221","key":"R19","doi-asserted-by":"publisher","first-page":"0221","DOI":"10.34133\/hds.0221","article-title":"ECG-LM: understanding electrocardiogram with a large language model","volume":"5","author":"Yang","journal-title":"Health Data Sci"},{"key":"R20","doi-asserted-by":"crossref","unstructured":"Xia Y Li J Sun Y Bu J Wang K . ECG-aBcDe: overcoming model dependence, encoding ECG into a universal language for any large language model. Comput Biol Med. Feb 1, 2026;202:111439. [doi: 10.1016\/j.compbiomed.2025.111439] [Medline: 41483694]","DOI":"10.1016\/j.compbiomed.2025.111439"}],"container-title":["Journal of Medical Internet Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.jmir.org\/2026\/1\/e86692","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T22:25:25Z","timestamp":1785536725000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.jmir.org\/2026\/1\/e86692"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,3]]},"references-count":20,"URL":"https:\/\/doi.org\/10.2196\/86692","relation":{},"ISSN":["1438-8871"],"issn-type":[{"value":"1438-8871","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,3]]},"article-number":"v28i3e86692"}}