{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,9]],"date-time":"2026-07-09T08:09:29Z","timestamp":1783584569393,"version":"3.55.0"},"reference-count":40,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2024,10,8]],"date-time":"2024-10-08T00:00:00Z","timestamp":1728345600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,10,8]],"date-time":"2024-10-08T00:00:00Z","timestamp":1728345600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Med Inform Decis Mak"],"abstract":"<jats:title>Abstract<\/jats:title><jats:sec>\n                <jats:title>Purpose<\/jats:title>\n                <jats:p>Rare diseases pose significant challenges in diagnosis and treatment due to their low prevalence and heterogeneous clinical presentations. Unstructured clinical notes contain valuable information for identifying rare diseases, but manual curation is time-consuming and prone to subjectivity. This study aims to develop a hybrid approach combining dictionary-based natural language processing (NLP) tools with large language models (LLMs) to improve rare disease identification from unstructured clinical reports.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Methods<\/jats:title>\n                <jats:p>We propose a novel hybrid framework that integrates the Orphanet Rare Disease Ontology (ORDO) and the Unified Medical Language System (UMLS) to create a comprehensive rare disease vocabulary. SemEHR, a dictionary-based NLP tool, is employed to extract rare disease mentions from clinical notes. To refine the results and improve accuracy, we leverage various LLMs, including LLaMA3, Phi3-mini, and domain-specific models like OpenBioLLM and BioMistral. Different prompting strategies, such as zero-shot, few-shot, and knowledge-augmented generation, are explored to optimize the LLMs\u2019 performance.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Results<\/jats:title>\n                <jats:p>The proposed hybrid approach demonstrates superior performance compared to traditional NLP systems and standalone LLMs. LLaMA3 and Phi3-mini achieve the highest F1 scores in rare disease identification. Few-shot prompting with 1-3 examples yields the best results, while knowledge-augmented generation shows limited improvement. Notably, the approach uncovers a significant number of potential rare disease cases not documented in structured diagnostic records, highlighting its ability to identify previously unrecognized patients.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Conclusion<\/jats:title>\n                <jats:p>The hybrid approach combining dictionary-based NLP tools with LLMs shows great promise for improving rare disease identification from unstructured clinical reports. By leveraging the strengths of both techniques, the method demonstrates superior performance and the potential to uncover hidden rare disease cases. Further research is needed to address limitations related to ontology mapping and overlapping case identification, and to integrate the approach into clinical practice for early diagnosis and improved patient outcomes.<\/jats:p>\n              <\/jats:sec>","DOI":"10.1186\/s12911-024-02698-7","type":"journal-article","created":{"date-parts":[[2024,10,8]],"date-time":"2024-10-08T02:01:59Z","timestamp":1728352919000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":30,"title":["A hybrid framework with large language models for rare disease phenotyping"],"prefix":"10.1186","volume":"24","author":[{"given":"Jinge","family":"Wu","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hang","family":"Dong","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zexi","family":"Li","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Haowei","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Runci","family":"Li","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Arijit","family":"Patra","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chengliang","family":"Dai","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Waqar","family":"Ali","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Phil","family":"Scordis","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Honghan","family":"Wu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,10,8]]},"reference":[{"issue":"10","key":"2698_CR1","doi-asserted-by":"publisher","first-page":"2711","DOI":"10.1111\/apa.15974","volume":"110","author":"SC Groft","year":"2021","unstructured":"Groft SC, Posada M, Taruscio D. Progress, challenges and global approaches to rare diseases. Acta Paediatr. 2021;110(10):2711\u20136.","journal-title":"Acta Paediatr."},{"issue":"9629","key":"2698_CR2","doi-asserted-by":"publisher","first-page":"2039","DOI":"10.1016\/S0140-6736(08)60872-7","volume":"371","author":"A Schieppati","year":"2008","unstructured":"Schieppati A, Henter JI, Daina E, Aperia A. Why rare diseases are an important medical and social issue. Lancet. 2008;371(9629):2039\u201341.","journal-title":"Lancet."},{"issue":"1","key":"2698_CR3","doi-asserted-by":"publisher","first-page":"233","DOI":"10.1186\/s13023-022-02358-x","volume":"17","author":"A Bauskis","year":"2022","unstructured":"Bauskis A, Strange C, Molster C, Fisher C. The diagnostic odyssey: insights from parents of children living with an undiagnosed condition. Orphanet J Rare Dis. 2022;17(1):233.","journal-title":"Orphanet J Rare Dis."},{"key":"2698_CR4","doi-asserted-by":"crossref","unstructured":"Hampson C, Evans W, Menzies L, McKay L. Measuring the impact of the COVID-19 pandemic on diagnostic delay in rare disease. Innovations. 2022.","DOI":"10.33590\/emj\/21-00181"},{"issue":"6","key":"2698_CR5","doi-asserted-by":"publisher","first-page":"1309","DOI":"10.1016\/j.chest.2017.12.018","volume":"153","author":"JK Stoller","year":"2018","unstructured":"Stoller JK. The challenge of rare diseases. Chest. 2018;153(6):1309\u201314.","journal-title":"Chest."},{"key":"2698_CR6","doi-asserted-by":"crossref","unstructured":"Thygesen JH, Zhang H, Issa H, Wu J, Hama T, Pinho-Gomes AC, et\u00a0al. A nationwide study of 331 rare diseases among 58 million individuals: prevalence, demographics, and COVID-19 outcomes. medRxiv. 2023;2023\u201310.","DOI":"10.1101\/2023.10.12.23296948"},{"issue":"1","key":"2698_CR7","doi-asserted-by":"publisher","first-page":"45","DOI":"10.1186\/s13023-023-02648-y","volume":"18","author":"Z Zhang","year":"2023","unstructured":"Zhang Z. Diagnosing rare diseases and mental well-being: a family\u2019s story. Orphanet J Rare Dis. 2023;18(1):45.","journal-title":"Orphanet J Rare Dis."},{"issue":"1","key":"2698_CR8","doi-asserted-by":"publisher","first-page":"20","DOI":"10.1016\/j.ymgme.2008.10.003","volume":"96","author":"RC Griggs","year":"2009","unstructured":"Griggs RC, Batshaw M, Dunkle M, Gopal-Srivastava R, Kaye E, Krischer J, et al. Clinical research for rare disease: opportunities, challenges, and solutions. Mol Genet Metab. 2009;96(1):20\u20136.","journal-title":"Mol Genet Metab."},{"issue":"2","key":"2698_CR9","doi-asserted-by":"publisher","first-page":"e12596","DOI":"10.2196\/12596","volume":"7","author":"A Arbabi","year":"2019","unstructured":"Arbabi A, Adams DR, Fidler S, Brudno M, et al. Identifying clinical terms in medical text using ontology-guided machine learning. JMIR Med Inform. 2019;7(2):e12596.","journal-title":"JMIR Med Inform."},{"key":"2698_CR10","doi-asserted-by":"crossref","unstructured":"Cook HV, Jensen LJ. A guide to dictionary-based text mining. Bioinforma Drug Discov. 2019;73\u201389.","DOI":"10.1007\/978-1-4939-9089-4_5"},{"issue":"5","key":"2698_CR11","doi-asserted-by":"publisher","first-page":"530","DOI":"10.1093\/jamia\/ocx160","volume":"25","author":"H Wu","year":"2018","unstructured":"Wu H, Toti G, Morley KI, Ibrahim ZM, Folarin A, Jackson R, et al. SemEHR: A general-purpose semantic search system to surface semantic data from clinical notes for tailored care, trial recruitment, and clinical research. J Am Med Inform Assoc. 2018;25(5):530\u20137.","journal-title":"J Am Med Inform Assoc."},{"issue":"1","key":"2698_CR12","doi-asserted-by":"publisher","first-page":"86","DOI":"10.1186\/s12911-023-02181-9","volume":"23","author":"H Dong","year":"2023","unstructured":"Dong H, Su\u00e1rez-Paniagua V, Zhang H, Wang M, Casey A, Davidson E, et al. Ontology-driven and weakly supervised rare disease identification from clinical notes. BMC Med Inform Decis Mak. 2023;23(1):86.","journal-title":"BMC Med Inform Decis Mak."},{"key":"2698_CR13","unstructured":"Achiam J, Adler S, Agarwal S, Ahmad L, Akkaya I, Aleman FL, et\u00a0al. Gpt-4 technical report.\u00a0arXiv\u00a0preprint\u00a0arXiv:230308774.\u00a02023."},{"key":"2698_CR14","unstructured":"Touvron H, Martin L, Stone K, Albert P, Almahairi A, Babaei Y, et\u00a0al. Llama 2: Open foundation and fine-tuned chat models.\u00a0arXiv\u00a0preprint\u00a0arXiv:230709288.\u00a02023."},{"key":"2698_CR15","unstructured":"Wang C, Liu X, Yue Y, Tang X, Zhang T, Jiayang C, et\u00a0al. Survey on factuality in large language models: Knowledge, retrieval and domain-specificity.\u00a0arXiv\u00a0preprint\u00a0arXiv:231007521.\u00a02023."},{"issue":"5","key":"2698_CR16","doi-asserted-by":"publisher","first-page":"1007","DOI":"10.1093\/jamia\/ocv180","volume":"23","author":"E Ford","year":"2016","unstructured":"Ford E, Carroll JA, Smith HE, Scott D, Cassell JA. Extracting information from the text of electronic medical records to improve case detection: a systematic review. J Am Med Inform Assoc. 2016;23(5):1007\u201315.","journal-title":"J Am Med Inform Assoc."},{"issue":"5","key":"2698_CR17","doi-asserted-by":"publisher","first-page":"610","DOI":"10.1016\/j.ajhg.2008.09.017","volume":"83","author":"PN Robinson","year":"2008","unstructured":"Robinson PN, K\u00f6hler S, Bauer S, Seelow D, Horn D, Mundlos S. The Human Phenotype Ontology: a tool for annotating and analyzing human hereditary disease. Am J Hum Genet. 2008;83(5):610\u20135.","journal-title":"Am J Hum Genet."},{"issue":"9","key":"2698_CR18","first-page":"518","volume":"152","author":"SS Weinreich","year":"2008","unstructured":"Weinreich SS, Mangon R, Sikkens J, Teeuw ME, Cornel M. Orphanet: a European database for rare diseases. Ned Tijdschr Geneeskd. 2008;152(9):518\u20139.","journal-title":"Ned Tijdschr Geneeskd."},{"issue":"1","key":"2698_CR19","doi-asserted-by":"publisher","first-page":"57","DOI":"10.1002\/(SICI)1098-1004(200001)15:1<57::AID-HUMU12>3.0.CO;2-G","volume":"15","author":"A Hamosh","year":"2000","unstructured":"Hamosh A, Scott AF, Amberger J, Valle D, McKusick VA. Online Mendelian inheritance in man (OMIM). Hum Mutat. 2000;15(1):57\u201361.","journal-title":"Hum Mutat."},{"key":"2698_CR20","doi-asserted-by":"crossref","unstructured":"Bodenreider O. The unified medical language system (UMLS): integrating biomedical terminology. Nucleic Acids Res. 2004;32(suppl_1):D267\u2013D270.","DOI":"10.1093\/nar\/gkh061"},{"key":"2698_CR21","unstructured":"Organization WH. International Statistical Classification of Diseases and related health problems: Alphabetical index. World Health Organ. 2004;3."},{"issue":"5","key":"2698_CR22","doi-asserted-by":"publisher","first-page":"507","DOI":"10.1136\/jamia.2009.001560","volume":"17","author":"GK Savova","year":"2010","unstructured":"Savova GK, Masanz JJ, Ogren PV, Zheng J, Sohn S, Kipper-Schuler KC, et al. Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications. J Am Med Inform Assoc. 2010;17(5):507\u201313.","journal-title":"J Am Med Inform Assoc."},{"key":"2698_CR23","unstructured":"Kraljevic Z, Bean D, Mascio A, Roguski L, Folarin A, Roberts A, et\u00a0al. MedCAT\u2013medical concept annotation tool.\u00a0arXiv\u00a0preprint\u00a0arXiv:191210166.\u00a02019."},{"issue":"6","key":"2698_CR24","doi-asserted-by":"publisher","first-page":"1046","DOI":"10.1093\/jamia\/ocv202","volume":"23","author":"JC Kirby","year":"2016","unstructured":"Kirby JC, Speltz P, Rasmussen LV, Basford M, Gottesman O, Peissig PL, et al. PheKB: a catalog and workflow for creating electronic phenotype algorithms for transportability. J Am Med Inform Assoc. 2016;23(6):1046\u201352.","journal-title":"J Am Med Inform Assoc."},{"issue":"1","key":"2698_CR25","doi-asserted-by":"publisher","first-page":"54","DOI":"10.1093\/jamia\/ocx111","volume":"25","author":"S Yu","year":"2018","unstructured":"Yu S, Ma Y, Gronsbell J, Cai T, Ananthakrishnan AN, Gainer VS, et al. Enabling phenotypic big data with PheNorm. J Am Med Inform Assoc. 2018;25(1):54\u201360.","journal-title":"J Am Med Inform Assoc."},{"issue":"2","key":"2698_CR26","doi-asserted-by":"publisher","first-page":"e0192360","DOI":"10.1371\/journal.pone.0192360","volume":"13","author":"S Gehrmann","year":"2018","unstructured":"Gehrmann S, Dernoncourt F, Li Y, Carlson ET, Wu JT, Welt J, et al. Comparing deep learning and concept extraction based methods for patient phenotyping from clinical narratives. PLoS ONE. 2018;13(2):e0192360.","journal-title":"PLoS ONE."},{"issue":"7767","key":"2698_CR27","doi-asserted-by":"publisher","first-page":"116","DOI":"10.1038\/s41586-019-1390-1","volume":"572","author":"N Toma\u0161ev","year":"2019","unstructured":"Toma\u0161ev N, Glorot X, Rae JW, Zielinski M, Askham H, Saraiva A, et al. A clinically applicable approach to continuous prediction of future acute kidney injury. Nature. 2019;572(7767):116\u20139.","journal-title":"Nature."},{"issue":"1","key":"2698_CR28","doi-asserted-by":"publisher","first-page":"7155","DOI":"10.1038\/s41598-020-62922-y","volume":"10","author":"Y Li","year":"2020","unstructured":"Li Y, Rao S, Solares JRA, Hassaine A, Ramakrishnan R, Canoy D, et al. BEHRT: transformer for electronic health records. Sci Rep. 2020;10(1):7155.","journal-title":"Sci Rep."},{"key":"2698_CR29","unstructured":"Alonso\u00a0Casero \u00c1. Named entity recognition and normalization in biomedical literature: a practical case in SARS-CoV-2 literature. ETSI_Informatica. 2021. Unpublished. https:\/\/oa.upm.es\/67933\/."},{"issue":"4","key":"2698_CR30","doi-asserted-by":"publisher","first-page":"1234","DOI":"10.1093\/bioinformatics\/btz682","volume":"36","author":"J Lee","year":"2020","unstructured":"Lee J, Yoon W, Kim S, Kim D, Kim S, So CH, et al. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics. 2020;36(4):1234\u201340.","journal-title":"Bioinformatics."},{"issue":"2","key":"2698_CR31","doi-asserted-by":"publisher","first-page":"1269","DOI":"10.1109\/TCBB.2022.3170301","volume":"20","author":"Y Feng","year":"2022","unstructured":"Feng Y, Qi L, Tian W. PhenoBERT: a combined deep learning method for automated recognition of human phenotype ontology. IEEE\/ACM Trans Comput Biol Bioinforma. 2022;20(2):1269\u201377.","journal-title":"IEEE\/ACM Trans Comput Biol Bioinforma."},{"issue":"1","key":"2698_CR32","doi-asserted-by":"publisher","first-page":"86","DOI":"10.1038\/s41746-021-00455-y","volume":"4","author":"L Rasmy","year":"2021","unstructured":"Rasmy L, Xiang Y, Xie Z, Tao C, Zhi D. Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction. NPJ Digit Med. 2021;4(1):86.","journal-title":"NPJ Digit Med."},{"key":"2698_CR33","doi-asserted-by":"crossref","unstructured":"Shyr C, Hu Y, Bastarache L, Cheng A, Hamid R, Harris P, et al. Identifying and Extracting Rare Diseases and Their Phenotypes with Large Language Models. J Healthc Inform Res. 2024;8(2):438\u201361.","DOI":"10.1007\/s41666-023-00155-0"},{"key":"2698_CR34","doi-asserted-by":"crossref","unstructured":"Yang J, Liu C, Deng W, Wu D, Weng C, Zhou Y, et\u00a0al. Enhancing phenotype recognition in clinical notes using large language models: PhenoBCBERT and PhenoGPT. Patterns. 2024;5(1).","DOI":"10.1016\/j.patter.2023.100887"},{"key":"2698_CR35","doi-asserted-by":"crossref","unstructured":"Wang A, Liu C, Yang J, Weng C. Fine-tuning large language models for rare disease concept normalization. J Am Med Inform Assoc. 2024;31(5)","DOI":"10.1101\/2023.12.28.573586"},{"key":"2698_CR36","unstructured":"Thompson WE, Vidmar DM, De\u00a0Freitas JK, Pfeifer JM, Fornwalt BK, Chen R, et\u00a0al. Large language models with retrieval-augmented generation for zero-shot disease phenotyping.\u00a0arXiv\u00a0preprint\u00a0arXiv:231206457.\u00a02023."},{"key":"2698_CR37","unstructured":"Oniani D, Hilsman J, Dong H, Gao F, Verma S, Wang Y. Large language models vote: Prompting for rare disease identification.\u00a0arXiv\u00a0preprint\u00a0arXiv:230812890.\u00a02023."},{"issue":"2","key":"2698_CR38","doi-asserted-by":"publisher","first-page":"443","DOI":"10.1109\/TKDE.2014.2327028","volume":"27","author":"W Shen","year":"2014","unstructured":"Shen W, Wang J, Han J. Entity linking with a knowledge base: Issues, techniques, and solutions. IEEE Trans Knowl Data Eng. 2014;27(2):443\u201360.","journal-title":"IEEE Trans Knowl Data Eng."},{"issue":"45","key":"2698_CR39","first-page":"343","volume":"63","author":"WH Organization","year":"1988","unstructured":"Organization WH, et al. International classification of diseases-Ninth revision (ICD-9). Wkly Epidemiol Rec Relev\u00e9 \u00c9pid\u00e9miologique Hebdomadaire. 1988;63(45):343\u20134.","journal-title":"Wkly Epidemiol Rec Relev\u00e9 \u00c9pid\u00e9miologique Hebdomadaire."},{"issue":"1","key":"2698_CR40","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41597-022-01899-x","volume":"10","author":"AE Johnson","year":"2023","unstructured":"Johnson AE, Bulgarelli L, Shen L, Gayles A, Shammout A, Horng S, et al. MIMIC-IV, a freely accessible electronic health record dataset. Sci Data. 2023;10(1):1.","journal-title":"Sci Data."}],"container-title":["BMC Medical Informatics and Decision Making"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-024-02698-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12911-024-02698-7\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-024-02698-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,8]],"date-time":"2024-10-08T18:06:19Z","timestamp":1728410779000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcmedinformdecismak.biomedcentral.com\/articles\/10.1186\/s12911-024-02698-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,10,8]]},"references-count":40,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,12]]}},"alternative-id":["2698"],"URL":"https:\/\/doi.org\/10.1186\/s12911-024-02698-7","relation":{},"ISSN":["1472-6947"],"issn-type":[{"value":"1472-6947","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,10,8]]},"assertion":[{"value":"10 July 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"26 September 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"8 October 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"This work used the de-identified clinical notes in MIMIC-IV. We completed the Collaborative Institutional Training Initiative (CITI) Program\u2019s \u201cData or Specimens Only Research\u201d course () and signed the data use agreement to get access to the data. We strictly followed the guidelines and only used locally hosted LLMs with the data.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"All authors are consent for publication.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare no competing interests.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"289"}}