{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,11]],"date-time":"2026-09-11T04:28:34Z","timestamp":1789100914761,"version":"build-2803163510"},"reference-count":54,"publisher":"Oxford University Press (OUP)","issue":"11","license":[{"start":{"date-parts":[[2025,9,12]],"date-time":"2025-09-12T00:00:00Z","timestamp":1757635200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"name":"National Institute of Neurological Disorders and Stroke of the National Institutes of Health","award":["R01NS131112"],"award-info":[{"award-number":["R01NS131112"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025,11,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Introduction<\/jats:title>\n                    <jats:p>Neurofibromatosis type 1 (NF1) is a rare genetic disorder affecting multiple organ systems with significant clinical heterogeneity. Managing individuals with NF1 is challenging due to variability in disease progression and outcomes and limited early risk assessment tools.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>This study aims to develop an effective, generalizable, user-friendly clinical entity extraction pipeline for identifying NF1-related phenotypes from unstructured clinical notes to enhance research and risk-modeling efforts. We compare the benefits of rule-based natural language processing (NLP) vs large language models (LLMs) for this purpose.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Materials and Methods<\/jats:title>\n                    <jats:p>Four phenotype extraction pipelines (3 LLM-based vs 1 rule-based) were developed to automatically extract selected NF1-relevant phenotypes. Subject matter experts manually reviewed clinical notes, generating a gold-standard annotation dataset for evaluation. In Phase 1, notes authored by a single NF1 physician were used to guide pipeline development and refinement. In Phase 2, notes from a second NF1 physician were used to assess pipeline generalizability, followed by further refinement to accommodate differences in physician terminology.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>With refinement, the rule-based model had higher distributions of F1 scores than the LLMs in both Phase 1 and Phase 2. However, the LLMs demonstrated better generalizability between physicians without refinement, showing lesser performance decreases (4.4%-5.1%) when transitioning from Phase 1 to Phase 2 without refinement, compared to an 8.8% decrease for the rule-based model.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>We highlight trade-offs between the effectiveness of rule-based NLP vs generalizability and ease of implementation of LLMs for clinical entity extraction, with implications for pipeline portability across providers and institutions.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/jamia\/ocaf155","type":"journal-article","created":{"date-parts":[[2025,9,2]],"date-time":"2025-09-02T11:52:31Z","timestamp":1756813951000},"page":"1663-1673","source":"Crossref","is-referenced-by-count":11,"title":["Comparison of rule- and large language model-based phenotype extraction from clinical notes for neurofibromatosis type 1"],"prefix":"10.1093","volume":"32","author":[{"ORCID":"https:\/\/orcid.org\/0009-0002-3773-801X","authenticated-orcid":false,"given":"Levi","family":"Kaster","sequence":"first","affiliation":[{"name":"Institute for Informatics, Data Science and Biostatistics, Washington University School of Medicine in St Louis , St Louis, MO 63110,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ethan","family":"Hillis","sequence":"additional","affiliation":[{"name":"Institute for Informatics, Data Science and Biostatistics, Washington University School of Medicine in St Louis , St Louis, MO 63110,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0780-8710","authenticated-orcid":false,"given":"Inez Y","family":"Oh","sequence":"additional","affiliation":[{"name":"Institute for Informatics, Data Science and Biostatistics, Washington University School of Medicine in St Louis , St Louis, MO 63110,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Elizabeth C","family":"Cordell","sequence":"additional","affiliation":[{"name":"Department of Neurology, Washington University School of Medicine in St Louis , St Louis, MO 63110,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Randi E","family":"Foraker","sequence":"additional","affiliation":[{"name":"Institute for Informatics, Data Science and Biostatistics, Washington University School of Medicine in St Louis , St Louis, MO 63110,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9241-2656","authenticated-orcid":false,"given":"Albert M","family":"Lai","sequence":"additional","affiliation":[{"name":"Institute for Informatics, Data Science and Biostatistics, Washington University School of Medicine in St Louis , St Louis, MO 63110,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Stephanie M","family":"Morris","sequence":"additional","affiliation":[{"name":"Center for Autism Services, Science, and Innovation (CASSI), Kennedy Krieger Institute , Baltimore, MD 21211,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"David H","family":"Gutmann","sequence":"additional","affiliation":[{"name":"Department of Neurology, Washington University School of Medicine in St Louis , St Louis, MO 63110,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9532-2998","authenticated-orcid":false,"given":"Philip R O","family":"Payne","sequence":"additional","affiliation":[{"name":"Institute for Informatics, Data Science and Biostatistics, Washington University School of Medicine in St Louis , St Louis, MO 63110,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Aditi","family":"Gupta","sequence":"additional","affiliation":[{"name":"Institute for Informatics, Data Science and Biostatistics, Washington University School of Medicine in St Louis , St Louis, MO 63110,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2025,9,12]]},"reference":[{"key":"2025111813565238900_ocaf155-B1","doi-asserted-by":"publisher","first-page":"704","DOI":"10.1136\/jmg.26.11.704","article-title":"A genetic study of von Recklinghausen neurofibromatosis in south east Wales. I. Prevalence, fitness, mutation rate, and effect of parental transmission on severity","volume":"26","author":"Huson","year":"1989","journal-title":"J Med Genet"},{"key":"2025111813565238900_ocaf155-B2","volume-title":"Neurofibromatosis 1","author":"Friedman","year":"1993"},{"key":"2025111813565238900_ocaf155-B3","doi-asserted-by":"publisher","first-page":"17004","DOI":"10.1038\/nrdp.2017.4","article-title":"Neurofibromatosis type 1","volume":"3","author":"Gutmann","year":"2017","journal-title":"Nat Rev Dis Primers"},{"key":"2025111813565238900_ocaf155-B4","doi-asserted-by":"publisher","first-page":"51","DOI":"10.1001\/jama.1997.03550010065042","article-title":"The diagnostic evaluation and multidisciplinary management of neurofibromatosis 1 and neurofibromatosis 2","volume":"278","author":"Gutmann","year":"1997","journal-title":"JAMA"},{"key":"2025111813565238900_ocaf155-B5","doi-asserted-by":"publisher","first-page":"309","DOI":"10.1002\/ana.24093","article-title":"Sex Is a major determinant of neuronal dysfunction in neurofibromatosis type 1","volume":"75","author":"Diggs-Andrews","year":"2014","journal-title":"Ann Neurol"},{"key":"2025111813565238900_ocaf155-B6","doi-asserted-by":"publisher","first-page":"1276","DOI":"10.1001\/jamapsychiatry.2016.2600","article-title":"Disease burden and symptom structure of autism in neurofibromatosis type 1: a study of the international NF1-ASD consortium team (INFACT)","volume":"73","author":"Morris","year":"2016","journal-title":"JAMA Psychiatry"},{"key":"2025111813565238900_ocaf155-B7","doi-asserted-by":"publisher","first-page":"2835","DOI":"10.1007\/s10803-022-05571-6","article-title":"Sex- and age-related differences in autistic behaviours in children with neurofibromatosis type 1","volume":"53","author":"Chisholm","year":"2023","journal-title":"J Autism Dev Disord"},{"key":"2025111813565238900_ocaf155-B8","doi-asserted-by":"publisher","first-page":"299","DOI":"10.1002\/humu.23929","article-title":"Clinical spectrum of individuals with pathogenic NF1 missense variants affecting p.Met1149, p.Arg1276, and p.Lys1423: genotype\u2013phenotype study in neurofibromatosis type 1","volume":"41","author":"Koczkowska","year":"2020","journal-title":"Hum Mutat"},{"key":"2025111813565238900_ocaf155-B9","doi-asserted-by":"publisher","first-page":"E1506","DOI":"10.1002\/humu.21271","article-title":"NF1 microdeletions in neurofibromatosis type 1: from genotype to phenotype","volume":"31","author":"Pasmant","year":"2010","journal-title":"Hum Mutat"},{"key":"2025111813565238900_ocaf155-B10","doi-asserted-by":"publisher","first-page":"79","DOI":"10.1038\/s10038-019-0695-0","article-title":"Phenotype categorization of neurofibromatosis type I and correlation to NF1 mutation types","volume":"65","author":"Kang","year":"2020","journal-title":"J Hum Genet"},{"key":"2025111813565238900_ocaf155-B11","doi-asserted-by":"publisher","first-page":"1052","DOI":"10.1002\/humu.22832","article-title":"High incidence of Noonan syndrome features including short stature and pulmonic stenosis in patients carrying NF1 Missense mutations affecting p.Arg1809: genotype\u2013phenotype correlation","volume":"36","author":"Rojnueangnit","year":"2015","journal-title":"Hum Mutat"},{"key":"2025111813565238900_ocaf155-B12","doi-asserted-by":"publisher","first-page":"e497","DOI":"10.1212\/CPJ.0000000000001089","article-title":"Predictive modeling for clinical features associated with neurofibromatosis type 1","volume":"11","author":"Morris","year":"2021","journal-title":"Neurol Clin Pract"},{"key":"2025111813565238900_ocaf155-B13","volume-title":"Registries for Evaluating Patient Outcomes: A User\u2019s Guide [Internet]","author":"Gliklich","year":"2014","edition":"3rd ed"},{"key":"2025111813565238900_ocaf155-B14","doi-asserted-by":"publisher","first-page":"106","DOI":"10.1186\/s13023-023-02719-0","article-title":"A systematic overview of rare disease patient registries: challenges in design, quality management, and maintenance","volume":"18","author":"Hageman","year":"2023","journal-title":"Orphanet J Rare Dis"},{"key":"2025111813565238900_ocaf155-B15","doi-asserted-by":"publisher","first-page":"e20","DOI":"10.1093\/jamia\/ocv130","article-title":"Combining billing codes, clinical notes, and medications from electronic health records provides superior phenotyping performance","volume":"23","author":"Wei","year":"2016","journal-title":"J Am Med Inform Assoc"},{"key":"2025111813565238900_ocaf155-B16","doi-asserted-by":"publisher","first-page":"104039","DOI":"10.1016\/j.jbi.2022.104039","article-title":"Clinical notes: an untapped opportunity for improving risk prediction for hospitalization and emergency department visit during home health care","volume":"128","author":"Song","year":"2022","journal-title":"J Biomed Inform"},{"key":"2025111813565238900_ocaf155-B17","doi-asserted-by":"publisher","first-page":"295","DOI":"10.1186\/s12911-020-01318-4","article-title":"Predicting mortality in critically ill patients with diabetes using machine learning and clinical notes","volume":"20","author":"Ye","year":"2020","journal-title":"BMC Med Inform Decis Mak"},{"key":"2025111813565238900_ocaf155-B18","doi-asserted-by":"publisher","first-page":"1753","DOI":"10.1016\/j.jvs.2016.11.031","article-title":"Mining peripheral arterial disease cases from narrative clinical notes using natural language processing","volume":"65","author":"Afzal","year":"2017","journal-title":"J Vasc Surg"},{"key":"2025111813565238900_ocaf155-B19","doi-asserted-by":"publisher","first-page":"ooad014","DOI":"10.1093\/jamiaopen\/ooad014","article-title":"Extraction of clinical phenotypes for Alzheimer\u2019s disease dementia from clinical notes using natural language processing","volume":"6","author":"Oh","year":"2023","journal-title":"JAMIA Open"},{"key":"2025111813565238900_ocaf155-B20","doi-asserted-by":"publisher","first-page":"272","DOI":"10.1016\/j.jbi.2014.06.006","article-title":"Complex epilepsy phenotype extraction from narrative clinical discharge summaries","volume":"51","author":"Cui","year":"2014","journal-title":"J Biomed Inform"},{"key":"2025111813565238900_ocaf155-B21","doi-asserted-by":"publisher","first-page":"e52289","DOI":"10.2196\/52289","article-title":"Mining clinical notes for physical rehabilitation exercise information: natural language processing algorithm development and validation study","volume":"12","author":"Sivarajkumar","year":"2024","journal-title":"JMIR Med Inform."},{"key":"2025111813565238900_ocaf155-B22","doi-asserted-by":"publisher","DOI":"10.1093\/jamiaopen\/ooae060","article-title":"Leveraging GPT-4 for identifying cancer phenotypes in electronic health records: a performance comparison between GPT-4, GPT-3.5-turbo, Flan-T5, Llama-3-8B, and spaCy\u2019s rule-based and machine learning-based methods","volume":"7","author":"Bhattarai","year":"2024","journal-title":"JAMIA Open"},{"key":"2025111813565238900_ocaf155-B23","doi-asserted-by":"publisher","author":"Zhao","year":"2023","DOI":"10.48550\/arXiv.2303.18223"},{"key":"2025111813565238900_ocaf155-B24","author":"Agrawal","year":"2022"},{"key":"2025111813565238900_ocaf155-B25","author":"OpenAI R","year":"2023"},{"key":"2025111813565238900_ocaf155-B26","doi-asserted-by":"publisher","first-page":"1134","DOI":"10.1038\/s41591-024-02855-5","article-title":"Adapted large language models can outperform medical experts in clinical text summarization","volume":"30","author":"Van Veen","year":"2024","journal-title":"Nat Med"},{"key":"2025111813565238900_ocaf155-B27","author":"Nori","year":"2023"},{"key":"2025111813565238900_ocaf155-B28","doi-asserted-by":"publisher","first-page":"1388","DOI":"10.1093\/jamia\/ocae041","article-title":"Why do users override alerts? Utilizing large language model to summarize comments and optimize clinical decision support","volume":"31","author":"Liu","year":"2024","journal-title":"J Am Med Inform Assoc"},{"key":"2025111813565238900_ocaf155-B29","doi-asserted-by":"publisher","DOI":"10.1056\/AIcs2300235","article-title":"GPT-4 for information retrieval and comparison of medical oncology guidelines","volume":"1","author":"Dyke","year":"2024","journal-title":"NEJM AI."},{"key":"2025111813565238900_ocaf155-B30","doi-asserted-by":"publisher","first-page":"24","DOI":"10.1186\/s11689-025-09612-w","article-title":"Automated extraction of functional biomarkers of verbal and ambulatory ability from multi-institutional clinical notes using large language models","volume":"17","author":"Kaster","year":"2025","journal-title":"J Neurodev Disord"},{"key":"2025111813565238900_ocaf155-B31","author":"Team","year":"2025"},{"key":"2025111813565238900_ocaf155-B32","author":"Guo","year":"2025"},{"key":"2025111813565238900_ocaf155-B33","doi-asserted-by":"publisher","first-page":"14","DOI":"10.1016\/j.jbi.2017.07.012","article-title":"Natural language processing systems for capturing and standardizing unstructured clinical information: a systematic review","volume":"73","author":"Kreimeyer","year":"2017","journal-title":"J Biomed Inform"},{"key":"2025111813565238900_ocaf155-B34","doi-asserted-by":"publisher","first-page":"34","DOI":"10.1016\/j.jbi.2017.11.011","article-title":"Clinical information extraction applications: a literature review","volume":"77","author":"Wang","year":"2018","journal-title":"J Biomed Inform"},{"key":"2025111813565238900_ocaf155-B35","doi-asserted-by":"publisher","first-page":"221","DOI":"10.1136\/amiajnl-2013-001935","article-title":"A review of approaches to identifying patient phenotype cohorts using electronic health records","volume":"21","author":"Shivade","year":"2014","journal-title":"J Am Med Inform Assoc"},{"key":"2025111813565238900_ocaf155-B36","doi-asserted-by":"publisher","first-page":"e12239","DOI":"10.2196\/12239","article-title":"Natural language processing of clinical notes on chronic diseases: systematic review","volume":"7","author":"Sheikhalishahi","year":"2019","journal-title":"JMIR Med Inform"},{"key":"2025111813565238900_ocaf155-B37","first-page":"438","article-title":"Launching into clinical space with medspaCy: a new clinical text processing toolkit in Python","volume":"2021","author":"Eyre","year":"2021","journal-title":"AMIA Annu Symp Proc"},{"key":"2025111813565238900_ocaf155-B38","doi-asserted-by":"publisher","author":"McKinney","year":"2010","DOI":"10.25080\/Majora-92bf1922-00a"},{"key":"2025111813565238900_ocaf155-B39","doi-asserted-by":"publisher","first-page":"357","DOI":"10.1038\/s41586-020-2649-2","article-title":"Array programming with NumPy","volume":"585","author":"Harris","year":"2020","journal-title":"Nature"},{"key":"2025111813565238900_ocaf155-B40","volume-title":"RuSH: A Rule-Based Segmentation Tool Using Hashing for Extremely Accurate Sentence Segmentation of Clinical Text","author":"Shi","year":"2016"},{"key":"2025111813565238900_ocaf155-B41","doi-asserted-by":"publisher","first-page":"e20171904","DOI":"10.1542\/peds.2017-1904","article-title":"Clinical practice guideline for screening and management of high blood pressure in children and adolescents","volume":"140","author":"Flynn","year":"2017","journal-title":"Pediatrics"},{"key":"2025111813565238900_ocaf155-B42","author":"Ollama","year":"2024"},{"key":"2025111813565238900_ocaf155-B43","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","year":"2023","journal-title":"Nature"},{"key":"2025111813565238900_ocaf155-B44","doi-asserted-by":"publisher","first-page":"52","DOI":"10.1080\/01621459.1961.10482090","article-title":"Multiple comparisons among means","volume":"56","author":"Dunn","year":"1961","journal-title":"J Am Stat Assoc"},{"key":"2025111813565238900_ocaf155-B45","first-page":"2825","article-title":"Scikit-learn: machine learning in Python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"J Machine Learning Res"},{"key":"2025111813565238900_ocaf155-B46","doi-asserted-by":"publisher","first-page":"261","DOI":"10.1038\/s41592-019-0686-2","article-title":"SciPy 1.0: fundamental algorithms for scientific computing in Python","volume":"17","author":"Virtanen","year":"2020","journal-title":"Nat Methods"},{"key":"2025111813565238900_ocaf155-B47","doi-asserted-by":"publisher","first-page":"377","DOI":"10.1016\/j.jbi.2008.08.010","article-title":"Research electronic data capture (REDCap)\u2014A metadata-driven methodology and workflow process for providing translational research informatics support","volume":"42","author":"Harris","year":"2009","journal-title":"J Biomed Inform"},{"key":"2025111813565238900_ocaf155-B48","doi-asserted-by":"publisher","first-page":"305","DOI":"10.1016\/j.cct.2012.12.002","article-title":"Development of an international internet-based neurofibromatosis Type 1 patient registry","volume":"34","author":"Johnson","year":"2013","journal-title":"Contemp Clin Trials"},{"key":"2025111813565238900_ocaf155-B49","doi-asserted-by":"publisher","first-page":"1686","DOI":"10.1002\/ajmg.a.36530","article-title":"Evaluation of participant recruitment methods to a rare disease online registry","volume":"164A","author":"Johnson","year":"2014","journal-title":"Am J Med Genet A"},{"key":"2025111813565238900_ocaf155-B50","doi-asserted-by":"publisher","first-page":"212","DOI":"10.1016\/j.cct.2014.12.006","article-title":"Validity of participant-reported diagnoses in an online patient registry: a report from the NF1 patient registry initiative","volume":"40","author":"Sharkey","year":"2015","journal-title":"Contemp Clin Trials"},{"key":"2025111813565238900_ocaf155-B51","doi-asserted-by":"publisher","first-page":"1891","DOI":"10.1093\/jamia\/ocae184","article-title":"The potential and pitfalls of using a large language model such as ChatGPT, GPT-4, or LLaMA as a clinical assistant","volume":"31","author":"Zhang","year":"2024","journal-title":"J Am Med Inform Assoc"},{"key":"2025111813565238900_ocaf155-B52","doi-asserted-by":"publisher","first-page":"1506","DOI":"10.1038\/s41436-021-01170-5","article-title":"Revised diagnostic criteria for neurofibromatosis type 1 and Legius syndrome: an international consensus recommendation","volume":"23","author":"Legius","year":"2021","journal-title":"Genetics Med"},{"key":"2025111813565238900_ocaf155-B53","doi-asserted-by":"publisher","first-page":"986","DOI":"10.1093\/jamia\/ocx039","article-title":"Challenges in adapting existing clinical natural language processing systems to multiple, diverse health care settings","volume":"24","author":"Carrell","year":"2017","journal-title":"J Am Med Inform Assoc"},{"key":"2025111813565238900_ocaf155-B54","doi-asserted-by":"publisher","first-page":"2355","DOI":"10.1007\/s11606-019-05025-3","article-title":"Variation in Physicians\u2019 electronic health record documentation and potential patient harm from that variation","volume":"34","author":"Cohen","year":"2019","journal-title":"J Gen Intern Med"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/32\/11\/1663\/64259798\/ocaf155.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/32\/11\/1663\/64259798\/ocaf155.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,11,18]],"date-time":"2025-11-18T18:57:01Z","timestamp":1763492221000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/32\/11\/1663\/8253274"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,9,12]]},"references-count":54,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2025,9,12]]},"published-print":{"date-parts":[[2025,11,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocaf155","relation":{},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2025,11]]},"published":{"date-parts":[[2025,9,12]]}}}