{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,23]],"date-time":"2026-10-23T12:31:13Z","timestamp":1792758673892,"version":"4.5.0"},"reference-count":46,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Form Res"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Pediatric heart disease (PHD), including congenital heart defects, is often incompletely captured in electronic health records, particularly when clinical significance must be inferred from unstructured echocardiogram reports. Automated methods capable of extracting clinically meaningful PHD from narrative reports could improve clinical decision support and research applications.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n The aim of the study is to evaluate the feasibility of using supervised fine-tuning of large language models (LLMs), with and without chain-of-thought (CoT) reasoning, to characterize patients with clinically significant or historical PHD from unstructured echocardiogram reports.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n We developed a PHD detection algorithm using fine-tuned open-source LLMs, including LLaMA (Meta) and Qwen (Alibaba), to analyze 9749 echocardiogram reports. A subset of 712 reports was adjudicated by 2 pediatric cardiac anesthesiologists, classifying 506 (71.1%) as clinically significant PHD and 206 (28.9%) as not significant. While DeepSeek R1 has shown improved performance with CoT reasoning, its application in medical contexts is underexplored. We incorporated R1-generated CoT into model prompts and fine-tuned backbone LLMs.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n The fine-tuned Qwen-7B-10k-overthink-CoT achieved the highest accuracy (92.4%), outperforming Qwen-7B-without-CoT (90%), LLaMA-3B-without-CoT (87.9%), Qwen-3B-without-CoT (85.6%), Qwen-3B-10k-overthink-CoT (68.5%), and LLaMA-3B-10k-overthink-CoT (46.2%). In a second dataset, an external validation was performed (n=113; 64 positive, 49 negative), Qwen-7B-10k-overthink-CoT sustained a strong, balanced performance (82.7%), followed by Qwen-7B-without-CoT (88.4%), LLaMA-3B-without-CoT (86.8%), Qwen-3B-without-CoT (84.5%), Qwen-3B-10k-overthink-CoT (58.9%), and LLaMA-3B-10k-overthink-CoT (46.2%). The fine-tuned Qwen-7B model with overthinking CoT (10,000 tokens) achieved the highest internal accuracy (92.4%), with balanced sensitivity and specificity. Across repeated runs, CoT-enhanced models demonstrated improved classification consistency compared to non-CoT models (Qwen-7B-without-CoT: 90%, LLaMA-3B-without-CoT: 87.9%, Qwen-3B-without-CoT: 85.6%). In external validation (n=113), non-CoT variants achieved higher accuracy (up to 88.4%), whereas the Qwen-7B CoT model demonstrated more balanced class performance (accuracy=82.7%).<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n Supervised fine-tuning of LLMs with CoT offers an effective approach for automated PHD detection within unstructured data in the electronic medical record. While CoT-enhanced models demonstrated improved internal performance and more balanced classification, they did not consistently achieve higher accuracy in external validation, highlighting trade-offs between accuracy and class balance. These findings highlight the promise of LLM-based approaches for clinical text phenotyping while underscoring the need for larger, multicenter validation and careful calibration for real-world deployment. Continued validation and integration into the electronic medical record are essential for real-world, artificial intelligence\u2013driven clinical decision support.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/90968","type":"journal-article","created":{"date-parts":[[2026,6,8]],"date-time":"2026-06-08T17:52:26Z","timestamp":1780941146000},"page":"e90968-e90968","source":"Crossref","is-referenced-by-count":1,"title":["Supervised Fine-Tuning of Large Language Models With Chain-of-Thought Reasoning for Pediatric Heart Disease Detection in Unstructured Echocardiogram Reports: Algorithm Development and Validation"],"prefix":"10.2196","volume":"10","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0228-6116","authenticated-orcid":false,"given":"Haoming","family":"Shi","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7660-6063","authenticated-orcid":false,"given":"Justin B","family":"Long","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-0944-4161","authenticated-orcid":false,"given":"Michael C","family":"Fiedorek","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-9198-0787","authenticated-orcid":false,"given":"Hannah D","family":"Kilday","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-5973-2579","authenticated-orcid":false,"given":"Henry P","family":"Foote","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7056-8759","authenticated-orcid":false,"given":"Christoph P","family":"Hornik","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6389-2179","authenticated-orcid":false,"given":"Aditya","family":"Nagori","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-2931-6100","authenticated-orcid":false,"given":"Yifan","family":"Xiang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8366-4811","authenticated-orcid":false,"given":"Rishikesan","family":"Kamaleswaran","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,6,8]]},"reference":[{"key":"R1","unstructured":"About congenital heart defects. Centers for Disease Control and Prevention: Congenital Heart Defects (CHDs). 2024. URL: https:\/\/www.cdc.gov\/heart-defects\/about\/index.html [Accessed 23-09-2024]"},{"issue":"1","key":"R2","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1186\/s40949-020-00034-y","article-title":"Clinical Spectrum of congenital heart defects (CHD) detected at the child health Clinic in a Tertiary Health Facility in Ghana: a retrospective analysis","volume":"4","author":"Thomford","journal-title":"J Congenit Heart Dis"},{"issue":"1","key":"R3","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-023-27935-3","article-title":"Prevalence, predictors, and outcomes of major congenital anomalies: a population-based register study","volume":"13","author":"Al-Dewik","journal-title":"Sci Rep"},{"issue":"1","key":"R4","doi-asserted-by":"publisher","first-page":"34","DOI":"10.1136\/hrt.82.1.34","article-title":"Spectrum of congenital heart defects and extracardiac malformations associated with chromosomal abnormalities: results of a seven year necropsy study","volume":"82","author":"Tennstedt","journal-title":"Heart"},{"issue":"9","key":"R5","doi-asserted-by":"publisher","first-page":"2366","DOI":"10.21037\/tp-21-297","article-title":"Genetics of congenital heart disease: a narrative review of recent advances and clinical implications","volume":"10","author":"Yasuhara","journal-title":"Transl Pediatr"},{"issue":"2","key":"R6","doi-asserted-by":"publisher","first-page":"126","DOI":"10.1038\/s41569-022-00749-y","article-title":"Changing epidemiology of congenital heart disease: effect on outcomes and quality of care in adults","volume":"20","author":"Liu","journal-title":"Nat Rev Cardiol"},{"issue":"8","key":"R7","doi-asserted-by":"publisher","DOI":"10.1161\/JAHA.118.009450","article-title":"Public health approach to improve outcomes for congenital heart disease across the life span","volume":"8","author":"Jenkins","journal-title":"J Am Heart Assoc"},{"issue":"16","key":"R8","doi-asserted-by":"publisher","DOI":"10.1161\/JAHA.123.030821","article-title":"Positive predictive value of , and , codes for identification of congenital heart defects","volume":"12","author":"Ivey","journal-title":"J Am Heart Assoc"},{"issue":"1","key":"R9","doi-asserted-by":"publisher","DOI":"10.23889\/ijpds.v5i1.1123","article-title":"Unlocking the potential of electronic health records for health research","volume":"5","author":"Lee","journal-title":"Int J Popul Data Sci"},{"issue":"18","key":"R10","doi-asserted-by":"publisher","first-page":"1693","DOI":"10.1002\/bdr2.2245","article-title":"A machine learning model for predicting congenital heart defects from administrative data","volume":"115","author":"Shi","journal-title":"Birth Defects Res"},{"issue":"5","key":"R11","doi-asserted-by":"publisher","first-page":"276","DOI":"10.1038\/nrcardio.2014.26","article-title":"Epidemiological studies of CHD and the evolution of preventive cardiology","volume":"11","author":"Wong","journal-title":"Nat Rev Cardiol"},{"issue":"10","key":"R12","doi-asserted-by":"publisher","DOI":"10.1161\/JAHA.119.012030","article-title":"Congenital heart disease and risk of cardiovascular disease: a meta-analysis of cohort studies","volume":"8","author":"Wang","journal-title":"J Am Heart Assoc"},{"issue":"2","key":"R13","doi-asserted-by":"publisher","DOI":"10.1002\/bdr2.2440","article-title":"A generalized machine learning model for identifying congenital heart defects (CHDs) using ICD Codes","volume":"117","author":"Shi","journal-title":"Birth Defects Res"},{"key":"R14","doi-asserted-by":"crossref","unstructured":"Ivey LC Rodriguez FH IIIShi H et al. International Classification of Diseases (ICD) codes for congenital heart defects (CHD) have variable and limited accuracy for detecting CHD cases. medRxiv. Preprint posted online on Apr 20, 2023. [doi: 10.1101\/2023.04.20.23288898]","DOI":"10.1101\/2023.04.20.23288898"},{"issue":"3","key":"R15","doi-asserted-by":"publisher","DOI":"10.1002\/bdr2.2451","article-title":"Machine learning and natural language processing to improve classification of atrial septal defects in electronic health records","volume":"117","author":"Guo","journal-title":"Birth Defects Res"},{"issue":"6","key":"R16","doi-asserted-by":"publisher","first-page":"923","DOI":"10.1161\/CIRCRESAHA.116.309140","article-title":"Genetics and genomics of congenital heart disease","volume":"120","author":"Zaidi","journal-title":"Circ Res"},{"issue":"10","key":"R17","doi-asserted-by":"publisher","first-page":"1757","DOI":"10.1111\/apa.14921","article-title":"Adult congenital heart disease: past, present and future","volume":"108","author":"Brida","journal-title":"Acta Paediatr"},{"issue":"4","key":"R18","doi-asserted-by":"publisher","first-page":"171","DOI":"10.4103\/jcecho.jcecho_52_23","article-title":"Echocardiography in simple congenital heart diseases: guiding adult patient management","volume":"33","author":"Galzerano","journal-title":"J Cardiovasc Echogr"},{"issue":"1","key":"R19","doi-asserted-by":"publisher","DOI":"10.1186\/s44156-022-00006-5","article-title":"A practical guideline for performing a comprehensive transthoracic echocardiogram in the congenital heart disease patient: consensus recommendations from the British Society of Echocardiography","volume":"9","author":"Corbett","journal-title":"Echo Res Pract"},{"issue":"1","key":"R20","doi-asserted-by":"publisher","DOI":"10.1186\/s12872-017-0580-8","article-title":"Unlocking echocardiogram measurements for heart disease research through natural language processing","volume":"17","author":"Patterson","journal-title":"BMC Cardiovasc Disord"},{"issue":"3","key":"R21","doi-asserted-by":"publisher","DOI":"10.1136\/wjps-2023-000580","article-title":"A deep learning-based method for pediatric congenital heart disease detection with seven standard views in echocardiography","volume":"6","author":"Jiang","journal-title":"World J Pediatr Surg"},{"issue":"10","key":"R22","doi-asserted-by":"publisher","DOI":"10.3390\/healthcare10101940","article-title":"Recent advancements in emerging technologies for healthcare management systems: a survey","volume":"10","author":"Junaid","journal-title":"Healthcare (Basel)"},{"key":"R23","doi-asserted-by":"publisher","DOI":"10.1016\/j.tacc.2021.02.007","article-title":"Natural language processing in medicine: a review","volume":"38","author":"Locke","journal-title":"Trends Anaesth Crit Care"},{"key":"R24","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2017.07.012","article-title":"Natural language processing systems for capturing and standardizing unstructured clinical information: a systematic review","volume":"73","author":"Kreimeyer","journal-title":"J Biomed Inform"},{"key":"R25","doi-asserted-by":"publisher","DOI":"10.1016\/j.csbj.2024.09.031","article-title":"Large language models and their applications in bioinformatics","volume":"23","author":"Sarumi","journal-title":"Comput Struct Biotechnol J"},{"issue":"1","key":"R26","doi-asserted-by":"publisher","DOI":"10.1016\/j.mcpdig.2024.11.005","article-title":"Fine-tuning large language models for specialized use cases","volume":"3","author":"Anisuzzaman","journal-title":"Mayo Clin Proc Digit Health"},{"issue":"1","key":"R27","doi-asserted-by":"publisher","first-page":"95","DOI":"10.1038\/s41746-025-01489-2","article-title":"The foundational capabilities of large language models in predicting postoperative risks using clinical notes","volume":"8","author":"Alba","journal-title":"NPJ Digit Med"},{"issue":"1","key":"R28","doi-asserted-by":"publisher","first-page":"148","DOI":"10.3390\/medicina60010148","article-title":"Chain of thought utilization in large language models and application in nephrology","volume":"60","author":"Miao","journal-title":"Med Bogota Colomb"},{"issue":"9","key":"R29","doi-asserted-by":"publisher","first-page":"1929","DOI":"10.1093\/jamia\/ocae095","article-title":"RT: a Retrieving and Chain-of-Thought framework for few-shot medical named entity recognition","volume":"31","author":"Li","journal-title":"J Am Med Inform Assoc"},{"issue":"1","key":"R30","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-024-44930-y","article-title":"Congenital heart disease detection by pediatric electrocardiogram based deep learning integrated with human concepts","volume":"15","author":"Chen","journal-title":"Nat Commun"},{"issue":"5","key":"R31","doi-asserted-by":"publisher","first-page":"698","DOI":"10.1016\/j.neuron.2024.01.016","article-title":"Data science opportunities of large language models for neuroscience and biomedicine","volume":"112","author":"Bzdok","journal-title":"Neuron"},{"issue":"1","key":"R32","doi-asserted-by":"publisher","first-page":"97","DOI":"10.1016\/j.jacc.2024.05.003","article-title":"Transforming cardiovascular care with artificial intelligence: from discovery to practice","volume":"84","author":"Khera","journal-title":"J Am Coll Cardiol"},{"key":"R33","doi-asserted-by":"publisher","DOI":"10.1016\/j.amjcard.2023.06.104","article-title":"Cardiovascular care innovation through data-driven discoveries in the electronic health record","volume":"203","author":"Dhingra","journal-title":"Am J Cardiol"},{"issue":"3","key":"R34","doi-asserted-by":"publisher","first-page":"223","DOI":"10.1016\/j.jchf.2019.09.008","article-title":"Heart failure management innovation enabled by electronic health records","volume":"8","author":"Kao","journal-title":"JACC Heart Fail"},{"issue":"1","key":"R35","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-022-22398-4","article-title":"Data-driven identification of heart failure disease states and progression pathways using electronic health records","volume":"12","author":"Nagamine","journal-title":"Sci Rep"},{"issue":"1","key":"R36","doi-asserted-by":"publisher","first-page":"86","DOI":"10.1111\/cts.12884","article-title":"Precision medicine, AI, and the future of personalized health care","volume":"14","author":"Johnson","journal-title":"Clin Transl Sci"},{"key":"R37","volume":"2021","author":"Eyre","journal-title":"AMIA Annu Symp Proc"},{"key":"R38","unstructured":"Best prompting practices for using Meta Llama 3 with Amazon SageMaker JumpStart. AWS Machine Learning Blog. 2024. URL: https:\/\/aws.amazon.com\/blogs\/machine-learning\/best-prompting-practices-for-using-meta-llama-3-with-amazon-sagemaker-jumpstart\/ [Accessed 12-11-2024]"},{"key":"R39","doi-asserted-by":"crossref","unstructured":"Renze M Guven E . The effect of sampling temperature on problem solving in large language models. arXiv. Preprint posted online on Oct 2, 2024. [doi: 10.48550\/arXiv.2402.05201]","DOI":"10.18653\/v1\/2024.findings-emnlp.432"},{"key":"R40","unstructured":"Llama 3.2: revolutionizing edge AI and vision with open, customizable models. Meta AI. URL: https:\/\/ai.meta.com\/blog\/llama-3-2-connect-2024-vision-edge-mobile-devices\/ [Accessed 12-11-2024]"},{"key":"R41","unstructured":"Grattafiori A Dubey A Jauhri A et al. The Llama 3 herd of models. arXiv. Preprint posted online on Nov 23, 2024. [doi: 10.48550\/arXiv.2407.21783]"},{"key":"R42","unstructured":"Yang A Yang B Zhang B et al. Qwen2.5 technical report. arXiv. Preprint posted online on Jan 3, 2025. [doi: 10.48550\/arXiv.2412.15115]"},{"key":"R43","unstructured":"DeepSeek-AIGuo D Yang D et al. DeepSeek-R1: incentivizing reasoning capability in LLMs via reinforcement learning. arXiv. Preprint posted online on Jan 4, 2026. [doi: 10.48550\/arXiv.2501.12948]"},{"key":"R44","unstructured":"Qunash\/r1-overthinker: force DeepSeek r1 models to think for as long as you wish. GitHub. URL: https:\/\/github.com\/qunash\/r1-overthinker\/tree\/master [Accessed 27-02-2025]"},{"key":"R45","unstructured":"Hu EJ Shen Y Wallis P et al. LoRA: low-rank adaptation of large language models. arXiv. Preprint posted online on Oct 16, 2021. [doi: 10.48550\/arXiv.2106.09685]"},{"key":"R46","doi-asserted-by":"crossref","unstructured":"Li Y Yue X Xu Z et al. Small models struggle to learn from strong reasoners. Presented at: Findings of the Association for Computational Linguistics: ACL 2025; Jul 27 to Aug 1, 2025; Vienna, Austria. [doi: 10.18653\/v1\/2025.findings-acl.1301]","DOI":"10.18653\/v1\/2025.findings-acl.1301"}],"container-title":["JMIR Formative Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/formative.jmir.org\/2026\/1\/e90968","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T22:24:16Z","timestamp":1785536656000},"score":1,"resource":{"primary":{"URL":"https:\/\/formative.jmir.org\/2026\/1\/e90968"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,8]]},"references-count":46,"URL":"https:\/\/doi.org\/10.2196\/90968","relation":{},"ISSN":["2561-326X"],"issn-type":[{"value":"2561-326X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,8]]},"article-number":"v10i1e90968"}}