{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,27]],"date-time":"2026-08-27T03:13:25Z","timestamp":1787800405181,"version":"build-2784847793"},"reference-count":54,"publisher":"Public Library of Science (PLoS)","issue":"2","license":[{"start":{"date-parts":[[2026,2,12]],"date-time":"2026-02-12T00:00:00Z","timestamp":1770854400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["DMS-2413741"],"award-info":[{"award-number":["DMS-2413741"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.plosdigitalhealth.org"],"crossmark-restriction":false},"short-container-title":["PLOS Digit Health"],"abstract":"<jats:p>Named Entity Recognition (NER) in the rare disease domain poses unique challenges due to limited labeled data, semantic ambiguity between entity types, and long-tail distributions. In this study, we evaluate the capabilities of GPT-4o for rare disease NER under low-resource settings, using a range of prompt-based strategies including zero-shot prompting, few-shot in-context learning, retrieval-augmented generation (RAG), and task-level fine-tuning. We design a structured prompting framework that encodes domain-specific knowledge and disambiguation rules for four entity types. We further introduce two semantically guided few-shot example selection methods to improve in-context performance while reducing labeling effort. Experiments on the RareDis Corpus show that GPT-4o achieves competitive or superior performance compared to BioClinicalBERT, with task-level fine-tuning yielding the strongest performance among the evaluated approaches and improving upon the previously reported BioClinicalBERT baseline. Cost-performance analysis reveals that few-shot prompting delivers high returns at low token budgets. RAG provides limited overall gains but can improve recall for challenging entity types, especially signs and symptoms. An error taxonomy highlights common failure modes such as boundary drift and type confusion, suggesting opportunities for post-processing and hybrid refinement. Our results demonstrate that prompt-optimized LLMs can serve as effective, scalable alternatives to traditional supervised models in biomedical NER, particularly in rare disease applications where annotated data is scarce.<\/jats:p>","DOI":"10.1371\/journal.pdig.0001242","type":"journal-article","created":{"date-parts":[[2026,2,12]],"date-time":"2026-02-12T18:37:28Z","timestamp":1770921448000},"page":"e0001242","update-policy":"https:\/\/doi.org\/10.1371\/journal.pdig.corrections_policy","source":"Crossref","is-referenced-by-count":1,"title":["Leveraging large language models for rare disease named entity recognition"],"prefix":"10.1371","volume":"5","author":[{"given":"Nan Miles","family":"Xi","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yu","family":"Deng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3126-8838","authenticated-orcid":true,"given":"Lin","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2026,2,12]]},"reference":[{"issue":"5","key":"pdig.0001242.ref001","doi-asserted-by":"crossref","first-page":"101123","DOI":"10.1016\/j.isci.2020.101123","article-title":"Rare genetic diseases: nature\u2019s experiments on human development","volume":"23","author":"CE Lee","year":"2020","journal-title":"iScience"},{"issue":"1","key":"pdig.0001242.ref002","doi-asserted-by":"crossref","first-page":"263","DOI":"10.1186\/s12859-022-04810-y","article-title":"Exploring deep learning methods for recognizing rare diseases and their clinical manifestations from texts","volume":"23","author":"I Segura-Bedmar","year":"2022","journal-title":"BMC Bioinform"},{"issue":"2","key":"pdig.0001242.ref003","doi-asserted-by":"crossref","first-page":"438","DOI":"10.1007\/s41666-023-00155-0","article-title":"Identifying and extracting rare diseases and their phenotypes with large language models","volume":"8","author":"C Shyr","year":"2024","journal-title":"J Healthc Inform Res"},{"issue":"1","key":"pdig.0001242.ref004","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1007\/s41870-022-01145-y","article-title":"Knowledge graph enrichment from clinical narratives using NLP, NER, and biomedical ontologies for healthcare applications","volume":"15","author":"A Thukral","year":"2023","journal-title":"Int J Inf Tecnol"},{"issue":"4","key":"pdig.0001242.ref005","doi-asserted-by":"crossref","first-page":"286","DOI":"10.4258\/hir.2023.29.4.286","article-title":"Named entity recognition in electronic health records: a methodological review","volume":"29","author":"MC Durango","year":"2023","journal-title":"Healthc Inform Res"},{"key":"pdig.0001242.ref006","article-title":"An automatic and end-to-end system for rare disease knowledge graph construction based on ontology-enhanced large language models: development study","volume":"12","author":"L Cao","year":"2024","journal-title":"JMIR Med Inform"},{"issue":"1","key":"pdig.0001242.ref007","doi-asserted-by":"crossref","first-page":"11143","DOI":"10.1038\/s41598-022-15342-z","article-title":"Improving the diagnosis of thyroid cancer by machine learning and clinical data","volume":"12","author":"NM Xi","year":"2022","journal-title":"Sci Rep"},{"issue":"1","key":"pdig.0001242.ref008","doi-asserted-by":"crossref","first-page":"13721","DOI":"10.1038\/s41598-023-39986-7","article-title":"Identifying COVID-19 cases and extracting patient reported symptoms from Reddit using natural language processing","volume":"13","author":"M Guo","year":"2023","journal-title":"Sci Rep"},{"issue":"1","key":"pdig.0001242.ref009","doi-asserted-by":"crossref","first-page":"30","DOI":"10.1186\/s12911-024-02439-w","article-title":"An evaluation of GPT models for phenotype concept recognition","volume":"24","author":"T Groza","year":"2024","journal-title":"BMC Med Inform Decis Mak"},{"issue":"2","key":"pdig.0001242.ref010","doi-asserted-by":"crossref","first-page":"165","DOI":"10.1038\/s41431-019-0508-0","article-title":"Estimating cumulative point prevalence of rare diseases: analysis of the Orphanet database","volume":"28","author":"S Nguengang Wakap","year":"2020","journal-title":"Eur J Hum Genet"},{"issue":"4","key":"pdig.0001242.ref011","doi-asserted-by":"crossref","first-page":"1234","DOI":"10.1093\/bioinformatics\/btz682","article-title":"BioBERT: a pre-trained biomedical language representation model for biomedical text mining","volume":"36","author":"J Lee","year":"2020","journal-title":"Bioinformatics"},{"key":"pdig.0001242.ref012","article-title":"Publicly available clinical BERT embeddings","author":"E Alsentzer","year":"2019","journal-title":"arXiv"},{"key":"pdig.0001242.ref013","doi-asserted-by":"crossref","unstructured":"Portelli B, Scaboro S, Santus E, Sedghamiz H, Chersoni E, Serra G. Generalizing over long tail concepts for medical term normalization. In: Goldberg Y, Kozareva Z, Zhang Y, editors. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Stroudsburg (PA): Association for Computational Linguistics; 2022. p. 8580\u201391. doi: 10.18653\/v1\/2022.emnlp-main.588","DOI":"10.18653\/v1\/2022.emnlp-main.588"},{"key":"pdig.0001242.ref014","first-page":"1877","article-title":"Language models are few-shot learners","volume":"34","author":"TB Brown","year":"2020","journal-title":"Neural Inf Process Syst"},{"key":"pdig.0001242.ref015","article-title":"Large language models vote: prompting for rare disease identification","author":"D Oniani","year":"2023","journal-title":"arXiv"},{"issue":"1","key":"pdig.0001242.ref016","doi-asserted-by":"crossref","first-page":"289","DOI":"10.1186\/s12911-024-02698-7","article-title":"A hybrid framework with large language models for rare disease phenotyping","volume":"24","author":"J Wu","year":"2024","journal-title":"BMC Med Inform Decis Mak"},{"issue":"8","key":"pdig.0001242.ref017","doi-asserted-by":"crossref","DOI":"10.1093\/bioinformatics\/btad496","article-title":"Few-shot biomedical named entity recognition via knowledge-guided instance generation and prompt contrastive learning","volume":"39","author":"P Chen","year":"2023","journal-title":"Bioinformatics"},{"issue":"1","key":"pdig.0001242.ref018","doi-asserted-by":"crossref","first-page":"3280","DOI":"10.1038\/s41467-025-56989-2","article-title":"Benchmarking large language models for biomedical natural language processing applications and recommendations","volume":"16","author":"Q Chen","year":"2025","journal-title":"Nat Commun"},{"key":"pdig.0001242.ref019","article-title":"Large language models struggle in token-level clinical named entity recognition","author":"Q Lu","year":"2024","journal-title":"arXiv"},{"key":"pdig.0001242.ref020","first-page":"9459","article-title":"Retrieval-augmented generation for knowledge-intensive NLP tasks","volume":"20","author":"P Lewis","year":"2020","journal-title":"Neural Inf Process Syst"},{"issue":"9","key":"pdig.0001242.ref021","doi-asserted-by":"crossref","first-page":"1929","DOI":"10.1093\/jamia\/ocae095","article-title":"RT: a Retrieving and Chain-of-Thought framework for few-shot medical named entity recognition","volume":"31","author":"M Li","year":"2024","journal-title":"J Am Med Inform Assoc"},{"key":"pdig.0001242.ref022","article-title":"GPT-4 technical report","author":"OpenAI","year":"2023","journal-title":"arXiv"},{"key":"pdig.0001242.ref023","doi-asserted-by":"crossref","unstructured":"Kim Y, Riloff E. Stacked generalization for medical concept extraction from clinical notes. Proceedings of BioNLP 15. Stroudsburg (PA): Association for Computational Linguistics; 2015. p. 61\u201370. doi: 10.18653\/v1\/w15-3807","DOI":"10.18653\/v1\/W15-3807"},{"key":"pdig.0001242.ref024","doi-asserted-by":"crossref","first-page":"102083","DOI":"10.1016\/j.artmed.2021.102083","article-title":"Multi-domain clinical natural language processing with MedCAT: The Medical Concept Annotation Toolkit","volume":"117","author":"Z Kraljevic","year":"2021","journal-title":"Artif Intell Med"},{"key":"pdig.0001242.ref025","doi-asserted-by":"crossref","first-page":"103961","DOI":"10.1016\/j.jbi.2021.103961","article-title":"The RareDis corpus: a corpus annotated with rare diseases, their signs and symptoms","volume":"125","author":"C Mart\u00ednez-deMiguel","year":"2022","journal-title":"J Biomed Inform"},{"key":"pdig.0001242.ref026","first-page":"545","article-title":"The National Organization for Rare Disorders (NORD).","volume-title":"The grants register 2019","year":"2019"},{"key":"pdig.0001242.ref027","article-title":"GPT-4o system card","author":"OpenAI","year":"2024","journal-title":"arXiv"},{"key":"pdig.0001242.ref028","article-title":"SummQA at MEDIQA-chat 2023: in-context learning with GPT-4 for medical summarization","author":"Y Mathur","year":"2023","journal-title":"arXiv"},{"issue":"11","key":"pdig.0001242.ref029","doi-asserted-by":"crossref","first-page":"3126","DOI":"10.1093\/bioinformatics\/btac271","article-title":"scSampler: fast diversity-preserving subsampling of large-scale single-cell transcriptomic data","volume":"38","author":"D Song","year":"2022","journal-title":"Bioinformatics"},{"key":"pdig.0001242.ref030","first-page":"1","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","author":"C Raffel","year":"2019","journal-title":"arXiv"},{"key":"pdig.0001242.ref031","article-title":"Comprehensive and practical evaluation of retrieval-augmented generation systems for medical question answering","author":"NT Ngo","year":"2024","journal-title":"arXiv"},{"issue":"9","key":"pdig.0001242.ref032","first-page":"518","article-title":"Orphanet: a European database for rare diseases","volume":"152","author":"SS Weinreich","year":"2008","journal-title":"Ned Tijdschr Geneeskd"},{"key":"pdig.0001242.ref033","unstructured":"Merker JH, Bondarenko A, Hagen M, Viehweger A. MiBi at BioASQ 2024: retrieval-augmented generation for answering biomedical questions. CLEF; 2024. p. 176\u201387."},{"issue":"15","key":"pdig.0001242.ref034","doi-asserted-by":"crossref","first-page":"2894","DOI":"10.1002\/sim.10075","article-title":"A fast bootstrap algorithm for causal inference with large data","volume":"43","author":"M Kosko","year":"2024","journal-title":"Stat Med"},{"issue":"4","key":"pdig.0001242.ref035","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pone.0299811","article-title":"Analysis of learning curves in predictive modeling using exponential curve fitting with an asymptotic approach","volume":"19","author":"LS Vianna","year":"2024","journal-title":"PLoS One"},{"key":"pdig.0001242.ref036","doi-asserted-by":"crossref","first-page":"596","DOI":"10.1080\/01621459.1988.10478639","article-title":"Locally weighted regression: an approach to regression analysis by local fitting","volume":"83","author":"WS Cleveland","year":"1988","journal-title":"J Am Stat Assoc"},{"issue":"1","key":"pdig.0001242.ref037","doi-asserted-by":"crossref","first-page":"320","DOI":"10.1038\/s41746-024-01315-1","article-title":"A strategy for cost-effective large language model use at health system-scale","volume":"7","author":"E Klang","year":"2024","journal-title":"NPJ Digit Med"},{"issue":"20","key":"pdig.0001242.ref038","doi-asserted-by":"crossref","first-page":"3539","DOI":"10.1093\/bioinformatics\/bty356","article-title":"D3NER: biomedical named entity recognition using CRF-biLSTM improved with fine-tuned embeddings of various linguistic information","volume":"34","author":"TH Dang","year":"2018","journal-title":"Bioinformatics"},{"key":"pdig.0001242.ref039","article-title":"Large language models are few-shot clinical information extractors","author":"M Agrawal","year":"2022","journal-title":"arXiv"},{"key":"pdig.0001242.ref040","first-page":"1","article-title":"Understanding sarcoidosis using large language models and social media data","author":"NM Xi","year":"2024","journal-title":"J Healthc Inform Res"},{"issue":"1","key":"pdig.0001242.ref041","doi-asserted-by":"crossref","first-page":"157","DOI":"10.1186\/s12967-023-04011-y","article-title":"Precision information extraction for rare disease epidemiology at scale","volume":"21","author":"WZ Kariampuzha","year":"2023","journal-title":"J Transl Med"},{"issue":"9","key":"pdig.0001242.ref042","doi-asserted-by":"crossref","first-page":"1812","DOI":"10.1093\/jamia\/ocad259","article-title":"Improving large language models for clinical named entity recognition via prompt engineering","volume":"31","author":"Y Hu","year":"2024","journal-title":"J Am Med Inform Assoc"},{"key":"pdig.0001242.ref043","doi-asserted-by":"crossref","first-page":"6829","DOI":"10.18653\/v1\/2024.findings-emnlp.400","article-title":"Few-shot clinical entity recognition in English, French and Spanish: masked language models outperform generative model prompting.","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2024","author":"M Naguib","year":"2024"},{"issue":"3","key":"pdig.0001242.ref044","doi-asserted-by":"crossref","first-page":"586","DOI":"10.1093\/jamia\/ocaf005","article-title":"Development of secure infrastructure for advancing generative artificial intelligence research in healthcare at an academic medical center","volume":"32","author":"MY Ng","year":"2025","journal-title":"J Am Med Inform Assoc"},{"key":"pdig.0001242.ref045","doi-asserted-by":"crossref","first-page":"102845","DOI":"10.1016\/j.artmed.2024.102845","article-title":"De-identification of clinical free text using natural language processing: a systematic review of current approaches","volume":"151","author":"A Kova\u010devi\u0107","year":"2024","journal-title":"Artif Intell Med"},{"issue":"1","key":"pdig.0001242.ref046","doi-asserted-by":"crossref","first-page":"143","DOI":"10.1038\/s41746-025-01476-7","article-title":"Implementing large language models in healthcare while balancing control, collaboration, costs and security","volume":"8","author":"F Dennst\u00e4dt","year":"2025","journal-title":"NPJ Digit Med"},{"key":"pdig.0001242.ref047","unstructured":"Wang X, et al. Self-consistency improves chain of thought reasoning in language models. arXiv [cs.CL]. 2022."},{"key":"pdig.0001242.ref048","first-page":"24824","article-title":"Chain-of-thought prompting elicits reasoning in large language models.","volume-title":"Advances in neural information processing systems","author":"J Wei","year":"2022"},{"key":"pdig.0001242.ref049","doi-asserted-by":"crossref","DOI":"10.2196\/66279","article-title":"Using synthetic health care data to leverage large language models for named entity recognition: development and validation study","volume":"27","author":"H \u0160uvalov","year":"2025","journal-title":"J Med Internet Res"},{"key":"pdig.0001242.ref050","article-title":"Orthogonal subsampling for big data linear regression","author":"L Wang","year":"2021","journal-title":"arXiv"},{"key":"pdig.0001242.ref051","first-page":"1","article-title":"Independence-encouraging subsampling for nonparametric additive models","author":"Y Zhang","year":"2024","journal-title":"J Comput Graph Stat"},{"key":"pdig.0001242.ref052","article-title":"Efficient subsampling for high-dimensional data","author":"V Chasiotis","year":"2024","journal-title":"arXiv"},{"issue":"0","key":"pdig.0001242.ref053","article-title":"Tuning hyperparameters of doublet-detection methods for single-cell RNA sequencing data","volume":"0","year":"2023","journal-title":"Quant Biol"},{"issue":"1","key":"pdig.0001242.ref054","doi-asserted-by":"crossref","first-page":"38447","DOI":"10.1038\/s41598-025-24326-8","article-title":"Fraudulent account detection in social media using hybrid deep transformer model and hyperparameter optimization","volume":"15","author":"PK Shukla","year":"2025","journal-title":"Sci Rep"}],"container-title":["PLOS Digital Health"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pdig.0001242","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,2,12]],"date-time":"2026-02-12T18:37:37Z","timestamp":1770921457000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pdig.0001242"}},"subtitle":[],"editor":[{"given":"Xiangbin","family":"Meng","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2026,2,12]]},"references-count":54,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2026,2,12]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pdig.0001242","relation":{},"ISSN":["2767-3170"],"issn-type":[{"value":"2767-3170","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,2,12]]}}}