{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,20]],"date-time":"2026-08-20T16:24:57Z","timestamp":1787243097512,"version":"build-2736575974"},"reference-count":27,"publisher":"Oxford University Press (OUP)","issue":"7","license":[{"start":{"date-parts":[[2026,7,28]],"date-time":"2026-07-28T00:00:00Z","timestamp":1785196800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100010447","name":"German Centre for Cardiovascular Research","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100010447","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100002347","name":"Federal Ministry of Education and Research","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100002347","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100002839","name":"Charit\u00e9 \u2013 Universit\u00e4tsmedizin Berlin","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100002839","id-type":"DOI","asserted-by":"publisher"}]},{"name":"BIH Charit\u00e9 Digital Clinician Scientist Program"},{"DOI":"10.13039\/501100017268","name":"Berlin Institute of Health","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100017268","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2026,7,29]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Aims<\/jats:title>\n                    <jats:p>Risk scores are essential to evidence-based cardiovascular care, but manual calculation is labour intensive and error prone. Large language models (LLMs) could automate this process, yet LLMs are limited by their propensity for calculation errors and factual hallucinations. Pipelines separating LLM-based data extraction from deterministic score computation may improve reliability and transparency.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods and results<\/jats:title>\n                    <jats:p>We conducted a retrospective diagnostic study at a quaternary heart centre in Germany (January 2020 to July 2023). Patients with atrial fibrillation (n = 179) from an ablation registry and patients with severe aortic stenosis (n = 76) evaluated by a heart team were included. Six LLMs (GPT-5.2, Gemini 3.1 Pro, DeepSeek-R1, Qwen3, GPT-OSS 120B, and Kimi K2.5) were tested in standalone, retrieval-augmented generation (RAG), and pipeline configurations to compute HAS-BLED, CHA2DS2-VASc, and EuroSCORE II scores from routine clinical reports. Accuracy was assessed against expert-adjudicated ground truth using root mean squared error (RMSE) and Krippendorff\u2019s \u03b1 to evaluate numerical deviation and categorical agreement, respectively. Pipeline-generated scores showed substantially higher agreement with expert adjudication than standalone LLMs, LLMs with RAG, and treating physicians (mean Krippendorff\u2019s \u03b1: 0.78 vs. 0.32 vs. 0.39 vs. 0.31) and lower deviation from ground truth (mean RMSE: 0.89 vs. 5.81 vs. 1.85 vs. 1.34).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>Pipelines combining expert-curated knowledge injection, LLM-based clinical data extraction, and deterministic score calculation enable accurate and scalable cardiovascular risk score computation from unstructured real-world clinical data, outperforming physician-documented scores. Such pipelines could form the basis for clinical decision-support systems that automate routine risk assessment, reduce clinician workload, and promote more consistent evidence-based care.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/ehjdh\/ztag124","type":"journal-article","created":{"date-parts":[[2026,7,25]],"date-time":"2026-07-25T11:38:35Z","timestamp":1784979515000},"source":"Crossref","is-referenced-by-count":0,"title":["Development of an LLM pipeline exceeding physician-documented cardiovascular risk scores under routine clinical conditions"],"prefix":"10.1093","volume":"7","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8376-3366","authenticated-orcid":false,"given":"Tobias","family":"Roeschl","sequence":"first","affiliation":[{"name":"Deutsches Herzzentrum der Charit\u00e9, Department of Cardiology, Angiology and Intensive Care Medicine , Charit\u00e9platz 1, Berlin 10117 ,","place":["Germany"]},{"name":"Charit\u00e9 \u2013 Universit\u00e4tsmedizin Berlin, corporate Member of Freie Universit\u00e4t Berlin and Humboldt-Universit\u00e4t zu Berlin , Charit\u00e9platz 1, Berlin 10117 ,","place":["Germany"]},{"name":"Charit\u00e9 \u2013 Universit\u00e4tsmedizin Berlin, Institute for Artificial Intelligence in Medicine , Invalidenstra\u00dfe 120, Berlin 10115 ,","place":["Germany"]},{"name":"Berlin Institute of Health at Charit\u00e9 \u2013 Universit\u00e4tsmedizin Berlin , Charit\u00e9platz 1, Berlin 10117 ,","place":["Germany"]},{"name":"DZHK (German Centre for Cardiovascular Research), partner Site Berlin , Potsdamer Str. 58, 10785 Berlin ,","place":["Germany"]},{"name":"Deutsches Herzzentrum der Charit\u00e9, Department of Cardiothoracic and Vascular Surgery , Augustenburger Platz 1, Berlin 13353 ,","place":["Germany"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marie","family":"Hoffmann","sequence":"additional","affiliation":[{"name":"Charit\u00e9 \u2013 Universit\u00e4tsmedizin Berlin, corporate Member of Freie Universit\u00e4t Berlin and Humboldt-Universit\u00e4t zu Berlin , Charit\u00e9platz 1, Berlin 10117 ,","place":["Germany"]},{"name":"Charit\u00e9 \u2013 Universit\u00e4tsmedizin Berlin, Institute for Artificial Intelligence in Medicine , Invalidenstra\u00dfe 120, Berlin 10115 ,","place":["Germany"]},{"name":"DZHK (German Centre for Cardiovascular Research), partner Site Berlin , Potsdamer Str. 58, 10785 Berlin ,","place":["Germany"]},{"name":"Deutsches Herzzentrum der Charit\u00e9, Department of Cardiothoracic and Vascular Surgery , Augustenburger Platz 1, Berlin 13353 ,","place":["Germany"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Axel","family":"Unbehaun","sequence":"additional","affiliation":[{"name":"Charit\u00e9 \u2013 Universit\u00e4tsmedizin Berlin, corporate Member of Freie Universit\u00e4t Berlin and Humboldt-Universit\u00e4t zu Berlin , Charit\u00e9platz 1, Berlin 10117 ,","place":["Germany"]},{"name":"Deutsches Herzzentrum der Charit\u00e9, Department of Cardiothoracic and Vascular Surgery , Augustenburger Platz 1, Berlin 13353 ,","place":["Germany"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Henryk","family":"Dreger","sequence":"additional","affiliation":[{"name":"Charit\u00e9 \u2013 Universit\u00e4tsmedizin Berlin, corporate Member of Freie Universit\u00e4t Berlin and Humboldt-Universit\u00e4t zu Berlin , Charit\u00e9platz 1, Berlin 10117 ,","place":["Germany"]},{"name":"Deutsches Herzzentrum der Charit\u00e9, Department of Cardiology, Angiology and Intensive Care Medicine , Augustenburger Platz 1, Berlin 13353 ,","place":["Germany"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0132-0773","authenticated-orcid":false,"given":"Gerhard","family":"Hindricks","sequence":"additional","affiliation":[{"name":"Deutsches Herzzentrum der Charit\u00e9, Department of Cardiology, Angiology and Intensive Care Medicine , Charit\u00e9platz 1, Berlin 10117 ,","place":["Germany"]},{"name":"Charit\u00e9 \u2013 Universit\u00e4tsmedizin Berlin, corporate Member of Freie Universit\u00e4t Berlin and Humboldt-Universit\u00e4t zu Berlin , Charit\u00e9platz 1, Berlin 10117 ,","place":["Germany"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7911-8620","authenticated-orcid":false,"given":"Volkmar","family":"Falk","sequence":"additional","affiliation":[{"name":"Charit\u00e9 \u2013 Universit\u00e4tsmedizin Berlin, corporate Member of Freie Universit\u00e4t Berlin and Humboldt-Universit\u00e4t zu Berlin , Charit\u00e9platz 1, Berlin 10117 ,","place":["Germany"]},{"name":"DZHK (German Centre for Cardiovascular Research), partner Site Berlin , Potsdamer Str. 58, 10785 Berlin ,","place":["Germany"]},{"name":"Deutsches Herzzentrum der Charit\u00e9, Department of Cardiothoracic and Vascular Surgery , Augustenburger Platz 1, Berlin 13353 ,","place":["Germany"]},{"name":"Translational Cardiovascular Technologies, Institute of Translational Medicine, Department of Health Sciences and Technology, Swiss Federal Institute of Technology , Z\u00fcrich 8092 ,","place":["Switzerland"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7783-6362","authenticated-orcid":false,"given":"Ran","family":"Balicer","sequence":"additional","affiliation":[{"name":"Charit\u00e9 \u2013 Universit\u00e4tsmedizin Berlin, corporate Member of Freie Universit\u00e4t Berlin and Humboldt-Universit\u00e4t zu Berlin , Charit\u00e9platz 1, Berlin 10117 ,","place":["Germany"]},{"name":"Innovation Division, Clalit Health Services, Clalit Research Institute , Tel Aviv ,","place":["Israel"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2193-9813","authenticated-orcid":false,"given":"Radu","family":"Tanacli","sequence":"additional","affiliation":[{"name":"Charit\u00e9 \u2013 Universit\u00e4tsmedizin Berlin, corporate Member of Freie Universit\u00e4t Berlin and Humboldt-Universit\u00e4t zu Berlin , Charit\u00e9platz 1, Berlin 10117 ,","place":["Germany"]},{"name":"Deutsches Herzzentrum der Charit\u00e9, Department of Cardiology, Angiology and Intensive Care Medicine , Augustenburger Platz 1, Berlin 13353 ,","place":["Germany"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Felix","family":"Hohendanner","sequence":"additional","affiliation":[{"name":"Deutsches Herzzentrum der Charit\u00e9, Department of Cardiology, Angiology and Intensive Care Medicine , Charit\u00e9platz 1, Berlin 10117 ,","place":["Germany"]},{"name":"Charit\u00e9 \u2013 Universit\u00e4tsmedizin Berlin, corporate Member of Freie Universit\u00e4t Berlin and Humboldt-Universit\u00e4t zu Berlin , Charit\u00e9platz 1, Berlin 10117 ,","place":["Germany"]},{"name":"Deutsches Herzzentrum der Charit\u00e9, Department of Cardiology, Angiology and Intensive Care Medicine , Augustenburger Platz 1, Berlin 13353 ,","place":["Germany"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6944-2478","authenticated-orcid":false,"given":"Alexander","family":"Meyer","sequence":"additional","affiliation":[{"name":"Charit\u00e9 \u2013 Universit\u00e4tsmedizin Berlin, corporate Member of Freie Universit\u00e4t Berlin and Humboldt-Universit\u00e4t zu Berlin , Charit\u00e9platz 1, Berlin 10117 ,","place":["Germany"]},{"name":"Charit\u00e9 \u2013 Universit\u00e4tsmedizin Berlin, Institute for Artificial Intelligence in Medicine , Invalidenstra\u00dfe 120, Berlin 10115 ,","place":["Germany"]},{"name":"Berlin Institute of Health at Charit\u00e9 \u2013 Universit\u00e4tsmedizin Berlin , Charit\u00e9platz 1, Berlin 10117 ,","place":["Germany"]},{"name":"DZHK (German Centre for Cardiovascular Research), partner Site Berlin , Potsdamer Str. 58, 10785 Berlin ,","place":["Germany"]},{"name":"Berlin Institute for the Foundations of Learning and Data \u2013 TU Berlin , Berlin ,","place":["Germany"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2026,7,28]]},"reference":[{"key":"2026082011421908400_ztag124-B1","doi-asserted-by":"crossref","first-page":"1751","DOI":"10.1111\/risa.13682","article-title":"Risk analysis in the age of big data: the promises and pitfalls","volume":"41","author":"Nateghi","year":"2021","journal-title":"Risk Anal"},{"key":"2026082011421908400_ztag124-B2","doi-asserted-by":"publisher","author":"Chuang","DOI":"10.18653\/v1\/2024.emnlp-main.84"},{"key":"2026082011421908400_ztag124-B3","doi-asserted-by":"crossref","first-page":"84730","DOI":"10.52202\/079017-2690","article-title":"Medcalc-bench: evaluating large language models for medical calculations","volume":"37","author":"Khandekar","year":"2024","journal-title":"Adv Neural Inf Process Syst"},{"key":"2026082011421908400_ztag124-B4","doi-asserted-by":"publisher","author":"Boye","DOI":"10.48550\/arXiv.2502.11574"},{"key":"2026082011421908400_ztag124-B5","author":"Mirzadeh"},{"key":"2026082011421908400_ztag124-B6","doi-asserted-by":"crossref","first-page":"163","DOI":"10.1038\/s41746-025-01475-8","article-title":"Large language model agents can use tools to perform clinical calculations","volume":"8","author":"Goodell","year":"2025","journal-title":"NPJ Digit Med"},{"key":"2026082011421908400_ztag124-B7","author":"Wan"},{"key":"2026082011421908400_ztag124-B8","doi-asserted-by":"publisher","author":"Zhu","DOI":"10.18653\/v1\/2025.naacl-long.263"},{"key":"2026082011421908400_ztag124-B9","first-page":"9459","article-title":"Retrieval-augmented generation for knowledge-intensive NLP tasks","volume":"33","author":"Lewis","year":"2020","journal-title":"Adv Neural Inf Process Syst"},{"key":"2026082011421908400_ztag124-B10","doi-asserted-by":"crossref","first-page":"263","DOI":"10.1378\/chest.09-1584","article-title":"Refining clinical risk stratification for predicting stroke and thromboembolism in atrial fibrillation using a novel risk factor-based approach: the euro heart survey on atrial fibrillation","volume":"137","author":"Lip","year":"2010","journal-title":"Chest"},{"key":"2026082011421908400_ztag124-B11","doi-asserted-by":"crossref","first-page":"1093","DOI":"10.1378\/chest.10-0134","article-title":"A novel user-friendly score (HAS-BLED) to assess 1-year risk of Major bleeding in patients with atrial fibrillation: the euro heart survey","volume":"138","author":"Pisters","year":"2010","journal-title":"Chest"},{"key":"2026082011421908400_ztag124-B12","doi-asserted-by":"crossref","first-page":"734","DOI":"10.1093\/ejcts\/ezs043","article-title":"EuroSCORE II","volume":"41","author":"Nashef","year":"2012","journal-title":"Eur J Cardiothorac Surg"},{"key":"2026082011421908400_ztag124-B13","doi-asserted-by":"crossref","first-page":"373","DOI":"10.1093\/eurheartj\/ehaa612","article-title":"2020 ESC guidelines for the diagnosis and management of atrial fibrillation developed in collaboration with the European association for cardio-thoracic surgery (EACTS): the task force for the diagnosis and management of atrial fibrillation of the European Society of Cardiology (ESC) developed with the special contribution of the European heart rhythm association (EHRA) of the ESC","volume":"42","author":"Hindricks","year":"2020","journal-title":"Eur Heart J"},{"key":"2026082011421908400_ztag124-B14","doi-asserted-by":"crossref","first-page":"561","DOI":"10.1093\/eurheartj\/ehab395","article-title":"2021 ESC\/EACTS guidelines for the management of valvular heart disease: developed by the task force for the management of valvular heart disease of the European Society of Cardiology (ESC) and the European association for cardio-thoracic surgery (EACTS)","volume":"43","author":"Vahanian","year":"2021","journal-title":"Eur Heart J"},{"key":"2026082011421908400_ztag124-B15","doi-asserted-by":"publisher","author":"Longpre","DOI":"10.18653\/v1\/2021.emnlp-main.565"},{"key":"2026082011421908400_ztag124-B16","doi-asserted-by":"publisher","author":"Liu","DOI":"10.1162\/tacl_a_00638"},{"key":"2026082011421908400_ztag124-B17","doi-asserted-by":"crossref","first-page":"856","DOI":"10.3390\/math13050856","article-title":"Hallucination mitigation for retrieval-augmented large language models: a review","volume":"13","author":"Zhang","year":"2025","journal-title":"Mathematics"},{"key":"2026082011421908400_ztag124-B18","doi-asserted-by":"crossref","first-page":"600","DOI":"10.1038\/s41746-025-01963-x","article-title":"The evaluation illusion of large language models in medicine","volume":"8","author":"Agrawal","year":"2025","journal-title":"NPJ Digit Med"},{"key":"2026082011421908400_ztag124-B19","doi-asserted-by":"crossref","first-page":"319","DOI":"10.1001\/jama.2024.21700","article-title":"Testing and evaluation of health care applications of large language models: a systematic review","volume":"333","author":"Bedi","year":"2025","journal-title":"JAMA"},{"key":"2026082011421908400_ztag124-B20","doi-asserted-by":"crossref","first-page":"1152","DOI":"10.1038\/s41591-026-04229-5","article-title":"LLM-assisted systematic review of large language models in clinical medicine","volume":"32","author":"Chen","year":"2026","journal-title":"Nat Med"},{"key":"2026082011421908400_ztag124-B21","doi-asserted-by":"crossref","first-page":"2613","DOI":"10.1038\/s41591-024-03097-1","article-title":"Evaluation and mitigation of the limitations of large language models in clinical decision-making","volume":"30","author":"Hager","year":"2024","journal-title":"Nat Med"},{"key":"2026082011421908400_ztag124-B22","doi-asserted-by":"crossref","first-page":"9377","DOI":"10.1038\/s41467-025-64430-x","article-title":"AgentMD: empowering language agents for risk prediction with large-scale clinical tool learning","volume":"16","author":"Jin","year":"2025","journal-title":"Nat Commun"},{"key":"2026082011421908400_ztag124-B23","doi-asserted-by":"publisher","author":"Liu","DOI":"10.48550\/arXiv.2503.03802"},{"key":"2026082011421908400_ztag124-B24","doi-asserted-by":"crossref","first-page":"e74899","DOI":"10.2196\/74899","article-title":"Assessing the limitations of large language models in clinical practice guideline\u2013concordant treatment decision-making on real-world data: retrospective study","volume":"6","author":"Roeschl","year":"2025","journal-title":"JMIRx Med"},{"key":"2026082011421908400_ztag124-B25","doi-asserted-by":"crossref","first-page":"314","DOI":"10.1093\/europace\/euaa358","article-title":"The interpretation of CHA2DS2-VASc score components in clinical practice: a joint survey by the European heart rhythm association (EHRA) scientific initiatives committee, the EHRA young electrophysiologists, the association of cardiovascular nursing and allied professionals, and the European Society of Cardiology council on stroke","volume":"23","author":"Zhang","year":"2020","journal-title":"EP Europace"},{"key":"2026082011421908400_ztag124-B26","doi-asserted-by":"crossref","first-page":"64","DOI":"10.1080\/00325481.2021.1996815","article-title":"Perceptions and knowledge gaps on CHA2DS2-VASc score components: a joint survey of Chinese clinicians and clinical pharmacists","volume":"134","author":"Zhang","year":"2022","journal-title":"Postgrad Med"},{"key":"2026082011421908400_ztag124-B27","doi-asserted-by":"crossref","first-page":"359","DOI":"10.1097\/EDE.0000000000001173","article-title":"Aim for clinical utility, not just predictive accuracy","volume":"31","author":"Sachs","year":"2020","journal-title":"Epidemiology"}],"container-title":["European Heart Journal - Digital Health"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/ehjdh\/advance-article-pdf\/doi\/10.1093\/ehjdh\/ztag124\/70299008\/ztag124.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/ehjdh\/article-pdf\/7\/7\/ztag124\/70299008\/ztag124.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/ehjdh\/article-pdf\/7\/7\/ztag124\/70299008\/ztag124.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,20]],"date-time":"2026-08-20T15:42:31Z","timestamp":1787240551000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/ehjdh\/article\/doi\/10.1093\/ehjdh\/ztag124\/8746371"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,28]]},"references-count":27,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2026,7,29]]}},"URL":"https:\/\/doi.org\/10.1093\/ehjdh\/ztag124","relation":{},"ISSN":["2634-3916"],"issn-type":[{"value":"2634-3916","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026,8]]},"published":{"date-parts":[[2026,7,28]]},"article-number":"ztag124"}}