{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,13]],"date-time":"2026-08-13T11:37:14Z","timestamp":1786621034514,"version":"build-2736575974"},"reference-count":25,"publisher":"BMJ","issue":"8","license":[{"start":{"date-parts":[[2026,8,13]],"date-time":"2026-08-13T00:00:00Z","timestamp":1786579200000},"content-version":"unspecified","delay-in-days":12,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"content-domain":{"domain":["bmj.com"],"crossmark-restriction":true},"short-container-title":["BMJ Open"],"accepted":{"date-parts":[[2026,7,31]]},"published-print":{"date-parts":[[2026,8]]},"abstract":"<jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>To compare the diagnostic accuracy of four available automated electronic medical record (EMR) retrieval methods, including a large language model (LLM)-assisted workflow, against manual chart adjudication for identifying cardiovascular events.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Design<\/jats:title>\n                    <jats:p>Retrospective diagnostic accuracy study.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Setting<\/jats:title>\n                    <jats:p>Three sites within a single US tertiary health system.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Participants<\/jats:title>\n                    <jats:p>Two adult cohorts with previously adjudicated cardiovascular outcomes were included. Cohort 1 included 2258 patients treated with immune checkpoint inhibitors, and Cohort 2 included 1426 patients who underwent transcatheter aortic valve replacement.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Primary and secondary outcome measures<\/jats:title>\n                    <jats:p>The reference standard was clinician manual chart adjudication. Outcomes included ischaemic stroke or transient ischaemic attack, myocardial infarction (MI), heart failure (HF) exacerbation or hospitalisation and a composite major adverse cardiovascular events (MACE) outcome. Automated retrieval methods included International Classification of Diseases (ICD) codes, primary diagnosis, problem list and a zero-shot LLM workflow. Area under the (receiver operating characteristic) curve (AUC), sensitivity, specificity and net reclassification improvement were assessed.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>In Cohort 1, the LLM achieved the highest AUC for stroke (0.920; 95% CI 0.881 to 0.958), MI (0.938; 95% CI 0.905 to 0.971) and composite MACE (0.880; 95% CI 0.854 to 0.907), whereas ICD-based retrieval had a higher AUC for HF (0.882; 95% CI 0.845 to 0.918\u2009vs 0.873; 95% CI 0.831 to 0.914). In Cohort 2, the LLM achieved the highest AUC for all evaluated outcomes: stroke (0.915; 95% CI 0.862 to 0.968), MI (0.928; 95% CI 0.839 to 1.000), HF (0.844; 95% CI 0.803 to 0.884) and composite MACE (0.862; 95% CI 0.829 to 0.895). In Cohort 1, differences in AUC between the LLM and ICD methods were not statistically significant across outcomes, whereas in Cohort 2 the LLM showed significantly higher AUC for stroke and composite MACE.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>In this multisite retrospective validation study, the LLM-assisted workflow showed strong but context-dependent performance for identifying cardiovascular events from the EMR. Performance varied by outcome and cohort, and ICD-based retrieval remained competitive for some use cases. These findings support a complementary role for LLM-assisted extraction in retrospective cardiovascular outcomes research.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1136\/bmjopen-2025-116133","type":"journal-article","created":{"date-parts":[[2026,8,13]],"date-time":"2026-08-13T11:10:20Z","timestamp":1786619420000},"page":"e116133","update-policy":"https:\/\/doi.org\/10.1136\/crossmarkpolicy","source":"Crossref","is-referenced-by-count":0,"title":["Diagnostic accuracy of electronic medical record retrieval methods and a large language model for identifying cardiovascular events: a multisite retrospective validation study in a medical system in the United States"],"prefix":"10.1136","volume":"16","author":[{"given":"Omar","family":"Ibrahim","sequence":"first","affiliation":[{"name":"Department of Cardiovascular Medicine, Mayo Clinic Arizona, Phoenix, Arizona, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Juan","family":"Farina","sequence":"additional","affiliation":[{"name":"Department of Cardiovascular Medicine, Mayo Clinic Arizona, Phoenix, Arizona, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-2530-7850","authenticated-orcid":false,"given":"Milagros","family":"Pereyra Pietri","sequence":"additional","affiliation":[{"name":"Department of Cardiovascular Medicine, Mayo Clinic Arizona, Phoenix, Arizona, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kamal","family":"Awad","sequence":"additional","affiliation":[{"name":"Department of Cardiovascular Medicine, Mayo Clinic Arizona, Phoenix, Arizona, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mohammed Tiseer","family":"Abbas","sequence":"additional","affiliation":[{"name":"Department of Cardiovascular Medicine, Mayo Clinic Arizona, Phoenix, Arizona, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Isabel G","family":"Scalia","sequence":"additional","affiliation":[{"name":"Department of Cardiovascular Medicine, Mayo Clinic Arizona, Phoenix, Arizona, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hesham","family":"Sheashaa","sequence":"additional","affiliation":[{"name":"Department of Cardiovascular Medicine, Mayo Clinic Arizona, Phoenix, Arizona, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fatmaelzahraa E","family":"Abdelfattah","sequence":"additional","affiliation":[{"name":"Department of Cardiovascular Medicine, Mayo Clinic Arizona, Phoenix, Arizona, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mahshad","family":"Razaghi","sequence":"additional","affiliation":[{"name":"Department of Cardiovascular Medicine, Mayo Clinic Arizona, Phoenix, Arizona, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0278-2192","authenticated-orcid":false,"given":"Cecilia C","family":"Villa Etchegoyen","sequence":"additional","affiliation":[{"name":"Department of Cardiovascular Medicine, Mayo Clinic Arizona, Phoenix, Arizona, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vinod C","family":"Kaggal","sequence":"additional","affiliation":[{"name":"Information Technology, Mayo Clinic Minnesota, Rochester, Minnesota, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9922-0083","authenticated-orcid":false,"given":"Santiago","family":"Romero-Brufau","sequence":"additional","affiliation":[{"name":"AI and Systems Engineering, Department of Otolaryngology, Mayo Clinic, Rochester, Minnesota, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Reza","family":"Arsanjani","sequence":"additional","affiliation":[{"name":"Department of Cardiovascular Medicine, Mayo Clinic Arizona, Phoenix, Arizona, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chadi","family":"Ayoub","sequence":"additional","affiliation":[{"name":"Department of Cardiovascular Medicine, Mayo Clinic Arizona, Phoenix, Arizona, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"239","published-online":{"date-parts":[[2026,8,13]]},"reference":[{"key":"2026081304101264000_16.8.e116133.1","doi-asserted-by":"crossref","DOI":"10.2196\/59024","article-title":"Twenty-Five Years of Evolution and Hurdles in Electronic Health Records and Interoperability in Medical Research: Comprehensive Review","volume":"27","author":"Shen","year":"2025","journal-title":"J Med Internet Res"},{"key":"2026081304101264000_16.8.e116133.2","doi-asserted-by":"crossref","first-page":"157","DOI":"10.1044\/2016_AJSLP-16-0025","article-title":"Retrospective Studies: A Fresh Look","volume":"25","author":"Abbott","year":"2016","journal-title":"Am J Speech Lang Pathol"},{"key":"2026081304101264000_16.8.e116133.3","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pone.0138649","article-title":"Factors Affecting Accuracy of Data Abstracted from Medical Records","volume":"10","author":"Zozus","year":"2015","journal-title":"PLoS One"},{"key":"2026081304101264000_16.8.e116133.4","first-page":"71","article-title":"Evaluation of the Performance of a Large Language Model to Extract Signs and Symptoms from Clinical Notes","volume":"323","author":"Reategui-Rivera","year":"2025","journal-title":"Stud Health Technol Inform"},{"key":"2026081304101264000_16.8.e116133.5","doi-asserted-by":"crossref","DOI":"10.1186\/s12874-025-02470-z","article-title":"Scalable information extraction from free text electronic health records using large language models","volume":"25","author":"Gu","year":"2025","journal-title":"BMC Med Res Methodol"},{"key":"2026081304101264000_16.8.e116133.6","doi-asserted-by":"crossref","first-page":"4134","DOI":"10.1200\/JCO.24.00326","article-title":"Enhancing Precision in Detecting Severe Immune-Related Adverse Events: Comparative Analysis of Large Language Models and International Classification of Disease Codes in Patient Records","volume":"42","author":"Sun","year":"2024","journal-title":"J Clin Oncol"},{"key":"2026081304101264000_16.8.e116133.7","doi-asserted-by":"crossref","DOI":"10.1186\/s12911-025-02954-4","article-title":"A systematic review of large language model (LLM) evaluations in clinical medicine","volume":"25","author":"Shool","year":"2025","journal-title":"BMC Med Inform Decis Mak"},{"key":"2026081304101264000_16.8.e116133.8","doi-asserted-by":"crossref","DOI":"10.1136\/bmjopen-2016-012799","article-title":"STARD 2015 guidelines for reporting diagnostic accuracy studies: explanation and elaboration","volume":"6","author":"Cohen","year":"2016","journal-title":"BMJ Open"},{"key":"2026081304101264000_16.8.e116133.9","doi-asserted-by":"crossref","first-page":"2140","DOI":"10.1016\/j.mayocp.2024.12.017","article-title":"Characterization of Cardiovascular Events and Prognosis in Immune Checkpoint Inhibitor-Related Myocarditis","volume":"100","author":"Pereyra Pietri","year":"2025","journal-title":"Mayo Clin Proc"},{"key":"2026081304101264000_16.8.e116133.10","doi-asserted-by":"crossref","first-page":"295","DOI":"10.1093\/ehjdh\/ztae022","article-title":"The prognostic value of artificial intelligence to predict cardiac amyloidosis in patients with severe aortic stenosis undergoing transcatheter aortic valve replacement","volume":"5","author":"Pereyra Pietri","year":"2024","journal-title":"Eur Heart J Digit Health"},{"key":"2026081304101264000_16.8.e116133.11","doi-asserted-by":"publisher","DOI":"10.1161\/STR.0000000000000418"},{"key":"2026081304101264000_16.8.e116133.12","doi-asserted-by":"publisher","DOI":"10.1093\/eurheartj\/ehad191"},{"key":"2026081304101264000_16.8.e116133.13","doi-asserted-by":"publisher","DOI":"10.1093\/eurheartj\/ehad195"},{"key":"2026081304101264000_16.8.e116133.14","doi-asserted-by":"publisher","DOI":"10.1161\/STR.0000000000000211"},{"key":"2026081304101264000_16.8.e116133.15","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pone.0218919","article-title":"A diagnostic accuracy study validating cardiovascular ICD-9-CM codes in healthcare administrative databases. The Umbria Data-Value Project","volume":"14","author":"Cozzolino","year":"2019","journal-title":"PLoS One"},{"key":"2026081304101264000_16.8.e116133.16","doi-asserted-by":"publisher","DOI":"10.1097\/01.mlr.0000160417.39497.a9"},{"key":"2026081304101264000_16.8.e116133.17","doi-asserted-by":"crossref","first-page":"1095","DOI":"10.2147\/CLEP.S265619","article-title":"Validity of Acute Cardiovascular Outcome Diagnoses Recorded in European Electronic Health Records: A Systematic Review","volume":"12","author":"Davidson","year":"2020","journal-title":"Clin Epidemiol"},{"key":"2026081304101264000_16.8.e116133.18","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pone.0092286","article-title":"Validity of myocardial infarction diagnoses in administrative databases: a systematic review","volume":"9","author":"McCormick","year":"2014","journal-title":"PLoS One"},{"key":"2026081304101264000_16.8.e116133.19","doi-asserted-by":"publisher","DOI":"10.1002\/pds.3921"},{"key":"2026081304101264000_16.8.e116133.20","doi-asserted-by":"crossref","DOI":"10.1161\/CIRCOUTCOMES.121.008190","article-title":"Validation of Cardiovascular End Points Ascertainment Leveraging Multisource Electronic Health Records Harmonized Into a Common Data Model in the ADAPTABLE Randomized Clinical Trial","volume":"14","author":"Marquis-Gravel","year":"2021","journal-title":"Circ Cardiovasc Qual Outcomes"},{"key":"2026081304101264000_16.8.e116133.21","doi-asserted-by":"crossref","DOI":"10.1136\/bmjopen-2022-068938","article-title":"Validation of diagnosis of acute myocardial infarction and stroke in electronic medical records: a primary care cross-sectional study in Madrid, Spain (the e-MADVEVA Study)","volume":"13","author":"de Burgos-Lunar","year":"2023","journal-title":"BMJ Open"},{"key":"2026081304101264000_16.8.e116133.22","doi-asserted-by":"publisher","DOI":"10.1002\/pds.5150"},{"key":"2026081304101264000_16.8.e116133.23","doi-asserted-by":"publisher","DOI":"10.1161\/CIRCULATIONAHA.115.018610"},{"key":"2026081304101264000_16.8.e116133.24","doi-asserted-by":"crossref","DOI":"10.1186\/s13104-017-2600-2","article-title":"A rule-based electronic phenotyping algorithm for detecting clinically relevant cardiovascular disease cases","volume":"10","author":"Esteban","year":"2017","journal-title":"BMC Res Notes"},{"key":"2026081304101264000_16.8.e116133.25","doi-asserted-by":"crossref","DOI":"10.5888\/pcd10.120097","article-title":"An algorithm that identifies coronary and heart failure events in the electronic health record","volume":"10","author":"Kottke","year":"2013","journal-title":"Prev Chronic Dis"}],"container-title":["BMJ Open"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.1136\/bmjopen-2025-116133","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,13]],"date-time":"2026-08-13T11:10:23Z","timestamp":1786619423000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmjopen.bmj.com\/lookup\/doi\/10.1136\/bmjopen-2025-116133"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8]]},"references-count":25,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2026,8,13]]},"published-print":{"date-parts":[[2026,8]]}},"alternative-id":["10.1136\/bmjopen-2025-116133"],"URL":"https:\/\/doi.org\/10.1136\/bmjopen-2025-116133","relation":{},"ISSN":["2044-6055","2044-6055"],"issn-type":[{"value":"2044-6055","type":"print"},{"value":"2044-6055","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8]]}}}