{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:47:47Z","timestamp":1782841667095,"version":"3.54.5"},"reference-count":41,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Med Inform"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n The accurate classification of operative notes is essential for surgical outcomes research; however, CPT code classification is notoriously nonspecific for many procedures. In such situations, the operative note (or \u201cdictation\u201d) must be reviewed manually, a process that is labor-intensive and unsustainable. Natural language processing demonstrates tremendous potential for improving the efficiency and accuracy of procedure classification from unstructured operative notes. To date, it remains unexplored whether natural language processing can reliably differentiate between complex, multicomponent procedures, such as those involved in the care of cleft lip or palate and craniofacial anomalies.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aims to develop and evaluate a machine learning framework for the automated classification of operative notes for cleft and craniofacial procedures.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n \n This single-institution, retrospective observational study used operative notes from patients undergoing cleft and craniofacial procedures at a single academic medical center from 2016 to 2024. Each note in the database had been manually classified previously. Notes were preprocessed and vectorized using term frequency-inverse document frequency. A One-vs-Rest classification framework with random forest as the base classifier was developed to categorize procedures at 3 levels: primary procedure type (cleft lip repair, alveolar bone grafting, cleft palate repair, velopharyngeal insufficiency correction, rhinoplasty, and other), procedural subtype (primary vs revision), and specific surgical technique used (eg, Fisher, Mulliken, or rotation-advancement technique for cleft lip repair). Each hierarchical level was developed and evaluated using cross-validation. To improve procedural subtype classification for classes with few samples, synthetic notes were added to the dataset. Area under the receiver operating characteristic curve (AUC), an area under the precision-recall curve, micro- and macro-averaged\n F<\/jats:italic>\n 1<\/jats:sub>\n -scores, and Hamming loss were used to assess model performance.\n <\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n The dataset comprised 630 operative notes from 311 pediatric patients undergoing cleft and craniofacial procedures between 2016 and 2024, with a mean age of 3.75 (range 0\u201019) years. The primary classification model achieved strong performance in distinguishing procedure types with an AUC of 0.93 (SD 0.04), area under the precision-recall curve of 0.84 (SD 0.05), micro-averaged\n F<\/jats:italic>\n 1<\/jats:sub>\n -score of 0.88 (SD 0.02), a macro-averaged\n F<\/jats:italic>\n 1<\/jats:sub>\n -score of 0.84 (SD 0.03), and a Hamming loss of 0.04 (SD 0.01). Secondary classifiers achieved AUC scores of 1.0 (SD 0.00) for cleft lip revision classification but failed to discriminate between alveolar bone grafting primary and revision procedures (AUC 0.49, SD 0.02). Tertiary classifiers for surgical technique identification showed AUC scores of 0.88 (SD 0.03), 0.89 (SD 0.03), and 0.89 (SD 0.09) for cleft lip, cleft palate, and velopharyngeal insufficiency repair techniques, respectively.\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n This pilot study demonstrates that machine learning approaches can automate the classification of pediatric craniofacial operative notes across multiple levels of procedural detail. The implementation of such systems could significantly reduce the administrative burden related to surgical research, operations, and quality improvement.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/87133","type":"journal-article","created":{"date-parts":[[2026,5,12]],"date-time":"2026-05-12T04:28:32Z","timestamp":1778560112000},"page":"e87133-e87133","source":"Crossref","is-referenced-by-count":0,"title":["Natural Language Processing for Automated Classification of Cleft and Craniofacial Procedures From Operative Notes: Model Development and Feasibility Study"],"prefix":"10.2196","volume":"14","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9642-653X","authenticated-orcid":false,"given":"Meredith","family":"Cox","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9078-1186","authenticated-orcid":false,"given":"Elaine","family":"Lin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5415-5161","authenticated-orcid":false,"given":"Nicholas","family":"Oleck","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-2336-8129","authenticated-orcid":false,"given":"Carlee","family":"Jones","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9149-4859","authenticated-orcid":false,"given":"Neill Y","family":"Li","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3762-0831","authenticated-orcid":false,"given":"Suhail K","family":"Mithani","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3334-9877","authenticated-orcid":false,"given":"Alexander C","family":"Allori","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,5,11]]},"reference":[{"issue":"5","key":"R1","doi-asserted-by":"publisher","first-page":"504","DOI":"10.1097\/SHK.0000000000000898","article-title":"Machine learning for predicting outcomes in trauma","volume":"48","author":"Liu","journal-title":"Shock"},{"issue":"11","key":"R2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0293684","article-title":"Predicting amputation using machine learning: a systematic review","volume":"18","author":"Yao","journal-title":"PLOS ONE"},{"issue":"6","key":"R3","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0234722","article-title":"A systematic review of machine learning models for predicting outcomes of stroke with structured data","volume":"15","author":"Wang","journal-title":"PLOS ONE"},{"issue":"6","key":"R4","doi-asserted-by":"publisher","DOI":"10.1002\/wics.1549","article-title":"Challenges and opportunities beyond structured data in analysis of electronic health records","volume":"13","author":"Tayefi","journal-title":"WIREs Computational Stats"},{"issue":"5","key":"R5","doi-asserted-by":"publisher","first-page":"1224","DOI":"10.1097\/SCS.0000000000003667","article-title":"Variations in current procedural terminology coding for craniofacial surgery: a need for review and change","volume":"28","author":"Ewings","journal-title":"J Craniofac Surg"},{"issue":"6","key":"R6","doi-asserted-by":"publisher","first-page":"577","DOI":"10.1111\/ceo.14257","article-title":"Optimising vitrectomy operation note coding with machine learning","volume":"51","author":"Lee","journal-title":"Clinical Exper Ophthalmol"},{"issue":"4","key":"R7","doi-asserted-by":"publisher","first-page":"738","DOI":"10.1097\/ALN.0000000000003150","article-title":"Classification of current procedural terminology codes from electronic health record data using machine learning","volume":"132","author":"Burns","journal-title":"Anesthesiology"},{"issue":"1","key":"R8","doi-asserted-by":"publisher","DOI":"10.1007\/s10916-025-02149-4","article-title":"Evaluating large language models for automated CPT code prediction in endovascular neurosurgery","volume":"49","author":"Roy","journal-title":"J Med Syst"},{"issue":"6","key":"R9","doi-asserted-by":"publisher","DOI":"10.3171\/2023.3.FOCUS2371","article-title":"Surgical classification using natural language processing of informed consent forms in spine surgery","volume":"54","author":"Shost","journal-title":"Neurosurg Focus"},{"issue":"3","key":"R10","doi-asserted-by":"publisher","first-page":"439","DOI":"10.1097\/SLA.0000000000006793","article-title":"An open-architecture AI model for CPT coding in breast surgery: development, validation, and prospective testing","volume":"282","author":"El Moheb","journal-title":"Ann Surg"},{"issue":"7","key":"R11","doi-asserted-by":"publisher","DOI":"10.3390\/diagnostics15070829","article-title":"Artificial intelligence applications in pediatric craniofacial surgery","volume":"15","author":"Harrison","journal-title":"Diagnostics (Basel)"},{"issue":"1","key":"R12","doi-asserted-by":"publisher","DOI":"10.1007\/s10006-025-01371-1","article-title":"Revolutionizing cleft lip and palate management through artificial intelligence: a scoping review","volume":"29","author":"Zambrano","journal-title":"Oral Maxillofac Surg"},{"issue":"5","key":"R13","doi-asserted-by":"publisher","first-page":"540","DOI":"10.1597\/15-292","article-title":"A standard set of outcome measures for the comprehensive appraisal of cleft care","volume":"54","author":"Allori","journal-title":"Cleft Palate Craniofac J"},{"key":"R14","unstructured":"ACCQUIREnet. Duke University School of Medicine (Duke Surgery). URL: https:\/\/surgery.duke.edu\/divisions\/plastic-maxillofacial-and-oral-surgery\/research\/clinical-research\/datalab-clinical-care-and-population-health\/accquirenet [Accessed 13-05-2025]"},{"key":"R15","unstructured":"Hurst A Lerer A Goucher AP et al. GPT-4o system card. arXiv. Preprint posted online on Oct 25, 2024. [doi: 10.48550\/arXiv.2410.21276]"},{"key":"R16","unstructured":"Dedupe 3.0.2 documentation. Dedupe.io. URL: https:\/\/docs.dedupe.io\/en\/latest\/ [Accessed 15-03-2026]"},{"issue":"10","key":"R17","doi-asserted-by":"publisher","DOI":"10.1007\/s10916-021-01761-4","article-title":"Deep learning-based natural language processing in radiology: the impact of report complexity, disease prevalence, dataset size, and algorithm type on model performance","volume":"45","author":"Olthof","journal-title":"J Med Syst"},{"key":"R18","doi-asserted-by":"crossref","unstructured":"Chaturvedi J Shamsutdinova D Zimmer F et al. Sample size in natural language processing within healthcare research. arXiv. Preprint posted online on Sep 5, 2023. [doi: 10.48550\/arXiv.2309.02237]","DOI":"10.2139\/ssrn.4553964"},{"issue":"1","key":"R19","doi-asserted-by":"publisher","DOI":"10.2196\/73374","article-title":"Improving explainability and integrability of medical AI to promote health care professional acceptance and use: mixed systematic review","volume":"27","author":"Liu","journal-title":"J Med Internet Res"},{"key":"R20","doi-asserted-by":"publisher","DOI":"10.21037\/jmai-21-25","article-title":"Factors influencing trust in medical artificial intelligence for healthcare professionals: a narrative review","volume":"5","author":"Tucci","journal-title":"J Med Artif Intell"},{"key":"R21","doi-asserted-by":"publisher","DOI":"10.1177\/10556656251338640","article-title":"Navigating Coding challenges in craniofacial surgery: a national survey analysis on CPT variability","author":"Elver","journal-title":"Cleft Palate Craniofac J"},{"issue":"4","key":"R22","doi-asserted-by":"publisher","first-page":"996","DOI":"10.1097\/SCS.0000000000006362","article-title":"Variability in Current Procedural Terminology codes for craniomaxillofacial trauma reconstruction: a national survey","volume":"31","author":"Jazayeri","journal-title":"J Craniofac Surg"},{"issue":"1","key":"R23","doi-asserted-by":"publisher","DOI":"10.2196\/71176","article-title":"Language models for multilabel document classification of surgical concepts in exploratory laparotomy operative notes: algorithm development study","volume":"13","author":"Balch","journal-title":"JMIR Med Inform"},{"issue":"7","key":"R24","doi-asserted-by":"publisher","first-page":"2022","DOI":"10.1177\/21925682231164935","article-title":"Can a novel natural language processing model and artificial intelligence automatically generate billing codes from spine surgical operative notes?","volume":"14","author":"Zaidat","journal-title":"Global Spine J"},{"issue":"7","key":"R25","doi-asserted-by":"publisher","first-page":"1946","DOI":"10.1177\/21925682211062831","article-title":"Can natural language processing and artificial intelligence automate the generation of billing codes from operative note dictations?","volume":"13","author":"Kim","journal-title":"Global Spine J"},{"key":"R26","doi-asserted-by":"publisher","DOI":"10.4103\/jpi.jpi_52_21","article-title":"Comparison of machine-learning algorithms for the prediction of current procedural terminology (CPT) codes from pathology reports","volume":"13","author":"Levy","journal-title":"J Pathol Inform"},{"issue":"3","key":"R27","doi-asserted-by":"publisher","first-page":"831","DOI":"10.1097\/SCS.0000000000010575","article-title":"Evaluating the efficacy of large language models in CPT coding for craniofacial surgery: a comparative analysis","volume":"36","author":"Isch","journal-title":"J Craniofac Surg"},{"issue":"5","key":"R28","doi-asserted-by":"publisher","DOI":"10.1056\/AIdbp2300040","article-title":"Large language models are poor medical coders\u2014benchmarking of medical code querying","volume":"1","author":"Soroush","journal-title":"NEJM AI"},{"key":"R29","doi-asserted-by":"publisher","DOI":"10.1016\/j.avsg.2025.10.045","article-title":"Evaluating artificial intelligence-assisted current procedural terminology coding in vascular surgery: a comparison of ChatGPT Plus and Perplexity Pro against finance department","volume":"124","author":"Madris","journal-title":"Ann Vasc Surg"},{"key":"R30","doi-asserted-by":"publisher","DOI":"10.1097\/SCS.0000000000011390","article-title":"Bridging the coding gap: assessing large language models for accurate modifier assignment in craniofacial operative notes","volume":"36","author":"Isch","journal-title":"J Craniofac Surg"},{"issue":"6","key":"R31","doi-asserted-by":"publisher","DOI":"10.1097\/GOX.0000000000003633","article-title":"ChartSweep: a HIPAA-compliant tool to automate chart review for plastic surgery research","volume":"9","author":"Chartier","journal-title":"Plast Reconstr Surg Glob Open"},{"issue":"6","key":"R32","doi-asserted-by":"publisher","first-page":"1404","DOI":"10.1093\/jamia\/ocae081","article-title":"Constructing synthetic datasets with generative artificial intelligence to train large language models to classify acute renal failure from clinical notes","volume":"31","author":"Litake","journal-title":"J Am Med Inform Assoc"},{"issue":"4","key":"R33","doi-asserted-by":"publisher","first-page":"686","DOI":"10.1007\/s41666-025-00203-x","article-title":"Clinical assessment of fine-tuned open-source LLMs in cardiology: from progress notes to discharge summary","volume":"9","author":"Jung","journal-title":"J Healthc Inform Res"},{"key":"R34","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2024.3384496","article-title":"Evaluation and analysis of large language models for clinical text augmentation and generation","volume":"12","author":"Latif","journal-title":"IEEE Access"},{"issue":"5","key":"R35","doi-asserted-by":"publisher","first-page":"264","DOI":"10.3390\/info15050264","article-title":"Addressing data scarcity in the medical domain: a GPT-based approach for synthetic data generation and feature extraction","volume":"15","author":"Sufi","journal-title":"Information"},{"key":"R36","doi-asserted-by":"crossref","unstructured":"Benoit JRA . ChatGPT for clinical vignette generation, revision, and evaluation. medRxiv. Preprint posted online on Feb 8, 2023. [doi: 10.1101\/2023.02.04.23285478]","DOI":"10.1101\/2023.02.04.23285478"},{"key":"R37","unstructured":"Kang A Chen JY Lee-Youngzie Z Fu S . Synthetic data generation with LLM for improved depression prediction. arXiv. Preprint posted online on Nov 26, 2024. [doi: 10.48550\/arXiv.2411.17672]"},{"key":"R38","doi-asserted-by":"publisher","DOI":"10.38124\/ijisrt\/IJISRT24MAY2085","article-title":"Enhancing clinical documentation with synthetic data: leveraging generative models for improved accuracy","volume":"9","author":"Biswas","journal-title":"Int J Innov Sci Res Technol"},{"key":"R39","unstructured":"Alshaikhdeeb B Hemedan AA Ghosh S Balaur I Satagopam V . Generation of synthetic clinical text: a systematic review. arXiv. Preprint posted online on Jul 24, 2025. [doi: 10.48550\/arXiv.2507.18451]"},{"key":"R40","doi-asserted-by":"publisher","DOI":"10.3389\/frai.2025.1533508","article-title":"Large language models generating synthetic clinical datasets: a feasibility and comparative analysis with real-world perioperative data","volume":"8","author":"Barr","journal-title":"Front Artif Intell"},{"key":"R41","unstructured":"Meredithcox\/CleftNoteClassifier. GitHub. URL: https:\/\/github.com\/meredithcox\/CleftNoteClassifier [Accessed 28-04-2026]"}],"container-title":["JMIR Medical Informatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/medinform.jmir.org\/2026\/1\/e87133","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:14:40Z","timestamp":1782839680000},"score":1,"resource":{"primary":{"URL":"https:\/\/medinform.jmir.org\/2026\/1\/e87133"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,5,11]]},"references-count":41,"URL":"https:\/\/doi.org\/10.2196\/87133","relation":{},"ISSN":["2291-9694"],"issn-type":[{"value":"2291-9694","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,5,11]]},"article-number":"v14i5e87133"}}