{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,1]],"date-time":"2026-08-01T00:43:04Z","timestamp":1785544984435,"version":"3.56.0"},"reference-count":33,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Form Res"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Machine learning models for surgical applications require large, diverse datasets; however, data scarcity remains a critical limitation due to privacy regulations, institutional variability, and the rarity of many surgical procedures. Large language models (LLMs) offer a potential solution through synthetic data generation, but their performance and reliability in specialized surgical domains remain underexplored.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aimed to evaluate the linguistic fidelity of LLM-generated operative notes for cleft lip and palate procedures and to assess their impact on natural language processing classifier performance under varying data availability conditions.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n A total of 630 authentic operative notes were obtained from cleft procedures (86 primary cleft lip repairs, 101 primary cleft palate repairs, and 62 primary alveolar bone grafting [ABG] procedures) performed between 2013 and 2024. GPT-4o generated matched synthetic notes using multishot prompting with anonymized examples. Linguistic fidelity was evaluated using BERTScore for semantic similarity, Jensen-Shannon divergence of part-of-speech trigrams for syntactic structure, and Bilingual Evaluation Understudy (BLEU) scores for lexical overlap. Binary classifiers using ClinicalBERT embeddings and logistic regression were trained under both full data and data-scarce (retaining 5% or 10% of positive training cases while retaining the full negative training set) conditions, with and without synthetic augmentation at approximate ratios of synthetic to real notes (1:1, 2:1, 5:1, and 10:1).<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n Synthetic notes demonstrated high semantic fidelity across all procedures (BERTScore\n F<\/jats:italic>\n 1<\/jats:sub>\n -score: 0.86\u20100.88) and low syntactic divergence (Jensen-Shannon divergence: 0.06\u20100.08). BLEU scores indicated moderate lexical variation (0.14\u20100.19), reflecting distinct but contextually consistent phrasing. With full datasets, synthetic augmentation did not meaningfully affect classifier performance. Under data-scarce conditions retaining 5% of positive training cases while preserving the full negative training set, the area under the curve improved from 0.915 (SD 0.056) to 0.929 (SD 0.033) for cleft lip and from 0.935 (SD 0.036) to 0.949 (SD 0.033) for cleft palate, with smaller gains for ABG (mean 0.983, SD 0.014 to mean 0.987, SD 0.015). When retaining 10% of positive training cases, performance changes were minor across procedures (cleft lip: mean 0.952, SD 0.036 to mean 0.948, SD 0.028; cleft palate: mean 0.945, SD 0.044 to mean 0.956, SD 0.036; and ABG: mean 0.989, SD 0.008 to mean 0.985, SD 0.012).\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n LLM-generated operative notes exhibit strong semantic and syntactic fidelity to authentic documentation and can enhance model performance in a task-dependent manner when authentic data are limited. These findings suggest that synthetic data generation may address data scarcity challenges in specialized surgical domains, particularly for rare or underrepresented procedures, enabling robust machine learning model development.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/87276","type":"journal-article","created":{"date-parts":[[2026,7,3]],"date-time":"2026-07-03T18:23:06Z","timestamp":1783102986000},"page":"e87276-e87276","source":"Crossref","is-referenced-by-count":0,"title":["Linguistic Fidelity and Classification Performance of Large Language Models for Generating Synthetic Operative Notes: Evaluation Study"],"prefix":"10.2196","volume":"10","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9642-653X","authenticated-orcid":false,"given":"Meredith","family":"Cox","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9078-1186","authenticated-orcid":false,"given":"Elaine","family":"Lin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5415-5161","authenticated-orcid":false,"given":"Nicholas","family":"Oleck","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-2336-8129","authenticated-orcid":false,"given":"Carlee","family":"Jones","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9149-4859","authenticated-orcid":false,"given":"Neill Y","family":"Li","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3762-0831","authenticated-orcid":false,"given":"Suhail K","family":"Mithani","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3334-9877","authenticated-orcid":false,"given":"Alexander C","family":"Allori","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,7,3]]},"reference":[{"key":"R1","doi-asserted-by":"publisher","DOI":"10.3389\/fsurg.2024.1403540","article-title":"Applications of natural language processing tools in the surgical journey","volume":"11","author":"Le","journal-title":"Front Surg"},{"issue":"5","key":"R2","doi-asserted-by":"publisher","first-page":"900","DOI":"10.1097\/SLA.0000000000004419","article-title":"Natural language processing in surgery: a systematic review and meta-analysis","volume":"273","author":"Mellia","journal-title":"Ann Surg"},{"issue":"6","key":"R3","doi-asserted-by":"publisher","first-page":"1496","DOI":"10.1016\/j.surg.2024.02.019","article-title":"Generative artificial intelligence in surgery","volume":"175","author":"Rodler","journal-title":"Surgery"},{"issue":"11","key":"R4","doi-asserted-by":"publisher","DOI":"10.3390\/jcm13113041","article-title":"Clinical and surgical applications of large language models: a systematic review","volume":"13","author":"Pressman","journal-title":"J Clin Med"},{"issue":"11","key":"R5","doi-asserted-by":"publisher","first-page":"2929","DOI":"10.1038\/s41591-023-02608-w","article-title":"The value of standards for health datasets in artificial intelligence-based applications","volume":"29","author":"Arora","journal-title":"Nat Med"},{"issue":"2","key":"R6","doi-asserted-by":"publisher","first-page":"790","DOI":"10.1109\/JBHI.2022.3185673","article-title":"Handling privacy-sensitive medical data with federated learning: challenges and future directions","volume":"27","author":"Aouedi","journal-title":"IEEE J Biomed Health Inform"},{"issue":"4","key":"R7","doi-asserted-by":"publisher","first-page":"329","DOI":"10.1111\/cts.12638","article-title":"Clinical data: sources and types, regulatory constraints, applications","volume":"12","author":"Ahalt","journal-title":"Clin Transl Sci"},{"issue":"11","key":"R8","doi-asserted-by":"publisher","first-page":"2355","DOI":"10.1007\/s11606-019-05025-3","article-title":"Variation in physicians\u2019 electronic health record documentation and potential patient harm from that variation","volume":"34","author":"Cohen","journal-title":"J Gen Intern Med"},{"key":"R9","doi-asserted-by":"publisher","DOI":"10.1109\/EMBC.2012.6347140","article-title":"Improving surgical models through one\/two class learning","volume":"2012","author":"Chia","journal-title":"Annu Int Conf IEEE Eng Med Biol Soc"},{"issue":"5","key":"R10","doi-asserted-by":"publisher","first-page":"264","DOI":"10.3390\/info15050264","article-title":"Addressing data scarcity in the medical domain: a GPT-based approach for synthetic data generation and feature extraction","volume":"15","author":"Sufi","journal-title":"Information"},{"key":"R11","doi-asserted-by":"publisher","DOI":"10.3389\/frai.2025.1533508","article-title":"Large language models generating synthetic clinical datasets: a feasibility and comparative analysis with real-world perioperative data","volume":"8","author":"Barr","journal-title":"Front Artif Intell"},{"key":"R12","unstructured":"Gregg F Eder D . Dedupe. GitHub. URL: https:\/\/github.com\/dedupeio\/dedupe [Accessed 15-03-2026]"},{"issue":"1","key":"R13","doi-asserted-by":"publisher","first-page":"191","DOI":"10.1162\/coli_a_00540","article-title":"Evaluating synthetic data generation from user generated text","volume":"51","author":"Chim","journal-title":"Comput Linguist"},{"key":"R14","doi-asserted-by":"crossref","unstructured":"Zhang T Kishore V Wu F Weinberger KQ Artzi Y . BERTScore: evaluating text generation with BERT. arXiv. Preprint posted online on Apr 21, 2019. [doi: 10.48550\/arXiv.1904.09675]","DOI":"10.5260\/chara.21.2.8"},{"issue":"2","key":"R15","doi-asserted-by":"publisher","first-page":"307","DOI":"10.1016\/S0016-0032(96)00063-4","article-title":"The Jensen-Shannon divergence","volume":"334","author":"Men\u00e9ndez","journal-title":"J Frankl Inst"},{"key":"R16","doi-asserted-by":"crossref","unstructured":"Papineni K Roukos S Ward T Zhu WJ . BLEU: a method for automatic evaluation of machine translation. In: ACL \u201902: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. Association for Computational Linguistics; 2002. [doi: 10.3115\/1073083.1073135]","DOI":"10.3115\/1073083.1073135"},{"issue":"2","key":"R17","doi-asserted-by":"publisher","DOI":"10.2196\/16492","article-title":"Analyzing medical research results based on synthetic data and their relation to real data results: systematic comparison from five observational studies","volume":"8","author":"Reiner Benaim","journal-title":"JMIR Med Inform"},{"key":"R18","doi-asserted-by":"crossref","unstructured":"Laderas T Vasilevsky N Pederson B Haendel M McWeeney S Dorr DA . Teaching data science fundamentals through realistic synthetic clinical cardiovascular data. bioRxiv. Preprint posted online on Apr 21, 2018. [doi: 10.1101\/232611]","DOI":"10.1101\/232611"},{"issue":"8","key":"R19","doi-asserted-by":"publisher","first-page":"901","DOI":"10.1007\/s10552-019-01189-9","article-title":"The sensitivity of reported effects of EMF on childhood leukemia to uncontrolled confounding by residential mobility: a hybrid simulation study and an empirical analysis using CAPS data","volume":"30","author":"Amoon","journal-title":"Cancer Causes Control"},{"issue":"2-3","key":"R20","doi-asserted-by":"publisher","first-page":"267","DOI":"10.1016\/j.healthpol.2010.05.014","article-title":"Using micro-simulation to create a synthesised data set and test policy options: the case of health service effects under demographic ageing","volume":"97","author":"Davis","journal-title":"Health Policy"},{"key":"R21","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-020-0267-x","article-title":"Generation and evaluation of artificial mental health records for natural language processing","volume":"3","author":"Ive","journal-title":"NPJ Digit Med"},{"issue":"12","key":"R22","doi-asserted-by":"publisher","DOI":"10.1007\/s10916-022-01880-6","article-title":"DataSifterText: partially synthetic text generation for sensitive clinical notes","volume":"46","author":"Zhou","journal-title":"J Med Syst"},{"key":"R23","doi-asserted-by":"publisher","DOI":"10.1186\/1471-2458-14-1144","article-title":"A systematic review of barriers to data sharing in public health","volume":"14","author":"van Panhuis","journal-title":"BMC Public Health"},{"issue":"1","key":"R24","doi-asserted-by":"publisher","first-page":"18","DOI":"10.1177\/1833358320917207","article-title":"Developing a strategy to improve data sharing in health research: a mixed-methods study to identify barriers and facilitators","volume":"52","author":"Krahe","journal-title":"Health Inf Manag"},{"key":"R25","doi-asserted-by":"crossref","unstructured":"Feng SY Gangal V Wei J et al. A survey of data augmentation approaches for NLP. arXiv. Preprint posted online on May 7, 2021. [doi: 10.48550\/arXiv.2105.03075]","DOI":"10.18653\/v1\/2021.findings-acl.84"},{"key":"R26","doi-asserted-by":"crossref","unstructured":"Ding B Qin C Zhao R et al. Data augmentation using large language models: data perspectives, learning paradigms and challenges. arXiv. Preprint posted online on Mar 5, 2024. [doi: 10.48550\/arXiv.2403.02990]","DOI":"10.18653\/v1\/2024.findings-acl.97"},{"key":"R27","doi-asserted-by":"crossref","unstructured":"Vardhan M Nathani D Vardhan S Aggarwal A Simini F . Large language models as synthetic electronic health record data generators. In: 2024 IEEE Conference on Artificial Intelligence (CAI). IEEE; 2024. [doi: 10.1109\/CAI59869.2024.00152]","DOI":"10.1109\/CAI59869.2024.00152"},{"issue":"1","key":"R28","doi-asserted-by":"publisher","DOI":"10.3171\/2025.4.FOCUS25225","article-title":"Synthetic neurosurgical data generation with generative adversarial networks and large language models: an investigation on fidelity, utility, and privacy","volume":"59","author":"Barr","journal-title":"Neurosurg Focus"},{"key":"R29","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2024.3358206","article-title":"Socially aware synthetic data generation for suicidal ideation detection using large language models","volume":"12","author":"Ghanadian","journal-title":"IEEE Access"},{"key":"R30","doi-asserted-by":"crossref","unstructured":"Bucur AM . Leveraging LLM-generated data for detecting depression symptoms on social media. In: Goeuriot L Mulhem P Qu\u00e9not G et al, editors. Experimental IR Meets Multilinguality, Multimodality, and Interaction. Springer-Verlag; 2024:193-204. [doi: 10.1007\/978-3-031-71736-9_14]","DOI":"10.1007\/978-3-031-71736-9_14"},{"issue":"3","key":"R31","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pdig.0001290","article-title":"SynthCraft: an AI partner for synthetic data generation to support data access and augmentation in healthcare","volume":"5","author":"Callender","journal-title":"PLOS Digit Health"},{"issue":"2","key":"R32","doi-asserted-by":"publisher","DOI":"10.1162\/99608f92.5317da47","article-title":"How is ChatGPT\u2019s behavior changing over time?","volume":"6","author":"Chen","journal-title":"Harv Data Sci Rev"},{"key":"R33","unstructured":"OpenAI. URL: https:\/\/openai.com\/ [Accessed 18-06-2026]"}],"container-title":["JMIR Formative Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/formative.jmir.org\/2026\/1\/e87276","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T22:25:17Z","timestamp":1785536717000},"score":1,"resource":{"primary":{"URL":"https:\/\/formative.jmir.org\/2026\/1\/e87276"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,3]]},"references-count":33,"URL":"https:\/\/doi.org\/10.2196\/87276","relation":{},"ISSN":["2561-326X"],"issn-type":[{"value":"2561-326X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,3]]},"article-number":"v10i5e87276"}}