{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,13]],"date-time":"2026-07-13T20:08:17Z","timestamp":1783973297904,"version":"3.55.0"},"reference-count":32,"publisher":"Oxford University Press (OUP)","issue":"9","license":[{"start":{"date-parts":[[2024,6,3]],"date-time":"2024-06-03T00:00:00Z","timestamp":1717372800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/pages\/standard-publication-reuse-rights"}],"funder":[{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["LM012895"],"award-info":[{"award-number":["LM012895"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["HG012655"],"award-info":[{"award-number":["HG012655"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"NIH","doi-asserted-by":"publisher","award":["HG013031"],"award-info":[{"award-number":["HG013031"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,9,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Objective<\/jats:title>\n                    <jats:p>We aim to develop a novel method for rare disease concept normalization by fine-tuning Llama 2, an open-source large language model (LLM), using a domain-specific corpus sourced from the Human Phenotype Ontology (HPO).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We developed an in-house template-based script to generate two corpora for fine-tuning. The first (NAME) contains standardized HPO names, sourced from the HPO vocabularies, along with their corresponding identifiers. The second (NAME+SYN) includes HPO names and half of the concept\u2019s synonyms as well as identifiers. Subsequently, we fine-tuned Llama 2 (Llama2-7B) for each sentence set and conducted an evaluation using a range of sentence prompts and various phenotype terms.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>When the phenotype terms for normalization were included in the fine-tuning corpora, both models demonstrated nearly perfect performance, averaging over 99% accuracy. In comparison, ChatGPT-3.5 has only \u223c20% accuracy in identifying HPO IDs for phenotype terms. When single-character typos were introduced in the phenotype terms, the accuracy of NAME and NAME+SYN is 10.2% and 36.1%, respectively, but increases to 61.8% (NAME+SYN) with additional typo-specific fine-tuning. For terms sourced from HPO vocabularies as unseen synonyms, the NAME model achieved 11.2% accuracy, while the NAME+SYN model achieved 92.7% accuracy.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>Our fine-tuned models demonstrate ability to normalize phenotype terms unseen in the fine-tuning corpus, including misspellings, synonyms, terms from other ontologies, and laymen\u2019s terms. Our approach provides a solution for the use of LLMs to identify named medical entities from clinical narratives, while successfully normalizing them to standard concepts in a controlled vocabulary.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/jamia\/ocae133","type":"journal-article","created":{"date-parts":[[2024,5,23]],"date-time":"2024-05-23T01:44:13Z","timestamp":1716428653000},"page":"2076-2083","source":"Crossref","is-referenced-by-count":38,"title":["Fine-tuning large language models for rare disease concept normalization"],"prefix":"10.1093","volume":"31","author":[{"ORCID":"https:\/\/orcid.org\/0009-0007-2323-1505","authenticated-orcid":false,"given":"Andy","family":"Wang","sequence":"first","affiliation":[{"name":"Peddie School , Hightstown, NJ 08520, United States"},{"name":"Department of Biomedical Informatics, Columbia University , New York, NY 10032, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6024-3037","authenticated-orcid":false,"given":"Cong","family":"Liu","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Columbia University , New York, NY 10032, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jingye","family":"Yang","sequence":"additional","affiliation":[{"name":"Department of Mathematics, University of Pennsylvania , Philadelphia, PA 19104, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9624-0214","authenticated-orcid":false,"given":"Chunhua","family":"Weng","sequence":"additional","affiliation":[{"name":"Department of Biomedical Informatics, Columbia University , New York, NY 10032, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2024,6,3]]},"reference":[{"issue":"5","key":"2024082207521470900_ocae133-B1","first-page":"499","article-title":"Patient-centred standardization in interstitial cystitis\/bladder pain syndrome-a PLEA","volume":"4","author":"Meijlink","year":"2015","journal-title":"Transl Androl Urol"},{"issue":"4","key":"2024082207521470900_ocae133-B2","doi-asserted-by":"crossref","first-page":"947","DOI":"10.1016\/j.chest.2017.02.013","article-title":"A patient-based analysis of the geographic distribution of mycobacterium avium complex, Mycobacterium abscessus, and Mycobacterium kansasii infections in the United States","volume":"151","author":"Mirsaeidi","year":"2017","journal-title":"Chest"},{"key":"2024082207521470900_ocae133-B3","doi-asserted-by":"crossref","first-page":"S804","DOI":"10.1007\/s11606-014-2881-2","article-title":"Important role of translational science in rare disease innovation, discovery, and drug development","volume":"29(Suppl 3)","author":"Pariser","year":"2014","journal-title":"J Gen Intern Med"},{"issue":"1","key":"2024082207521470900_ocae133-B4","doi-asserted-by":"crossref","first-page":"104","DOI":"10.1186\/s13023-018-0851-1","article-title":"Using a meta-narrative literature review and focus groups with key stakeholders to identify perceived challenges and solutions for generating robust evidence on the effectiveness of treatments for rare diseases","volume":"13","author":"Tingley","year":"2018","journal-title":"Orphanet J Rare Dis"},{"issue":"3","key":"2024082207521470900_ocae133-B5","first-page":"54","article-title":"A new focus on process and measure. Raising data quality with a standard coding workflow and benchmarks","volume":"79","author":"Wilson","year":"2008","journal-title":"J AHIMA"},{"issue":"1","key":"2024082207521470900_ocae133-B6","doi-asserted-by":"crossref","first-page":"85","DOI":"10.1186\/s13023-018-0830-6","article-title":"Next generation phenotyping using narrative reports in a rare disease clinical data warehouse","volume":"13","author":"Garcelon","year":"2018","journal-title":"Orphanet J Rare Dis"},{"issue":"4","key":"2024082207521470900_ocae133-B7","doi-asserted-by":"crossref","first-page":"342","DOI":"10.1111\/cts.12556","article-title":"Global standards to expedite learning from medical research data","volume":"11","author":"Hudson","year":"2018","journal-title":"Clin Transl Sci"},{"issue":"1","key":"2024082207521470900_ocae133-B8","doi-asserted-by":"crossref","first-page":"214","DOI":"10.1111\/cts.12845","article-title":"Standardized data structures in rare diseases: CDISC user guides for duchenne muscular dystrophy and Huntington\u2019s disease","volume":"14","author":"Mullin","year":"2021","journal-title":"Clin Transl Sci"},{"issue":"8","key":"2024082207521470900_ocae133-B9","doi-asserted-by":"crossref","DOI":"10.3390\/ijerph15081644","article-title":"Recommendations for improving the quality of rare disease registries","volume":"15","author":"Kodra","year":"2018","journal-title":"Int J Environ Res Public Health"},{"issue":"10","key":"2024082207521470900_ocae133-B10","doi-asserted-by":"crossref","first-page":"1576","DOI":"10.1093\/jamia\/ocaa155","article-title":"Clinical concept normalization with a hybrid natural language processing system combining multilevel matching and machine learning ranking","volume":"27","author":"Chen","year":"2020","journal-title":"J Am Med Inform Assoc"},{"key":"2024082207521470900_ocae133-B11","first-page":"93","article-title":"Clinical concept normalization on medical records using word embeddings and heuristics","volume":"270","author":"Silva","year":"2020","journal-title":"Stud Health Technol Inform"},{"issue":"3","key":"2024082207521470900_ocae133-B12","doi-asserted-by":"crossref","first-page":"229","DOI":"10.1136\/jamia.2009.002733","article-title":"An overview of MetaMap: historical perspective and recent advances","volume":"17","author":"Aronson","year":"2010","journal-title":"J Am Med Inform Assoc"},{"key":"2024082207521470900_ocae133-B13","first-page":"17","article-title":"Effective mapping of biomedical text to the UMLS Metathesaurus: the MetaMap program","author":"Aronson","year":"2001","journal-title":"Proc AMIA Symp"},{"issue":"W1","key":"2024082207521470900_ocae133-B14","doi-asserted-by":"crossref","first-page":"W566","DOI":"10.1093\/nar\/gkz386","article-title":"Doc2Hpo: a web application for efficient and accurate HPO concept curation","volume":"47","author":"Liu","year":"2019","journal-title":"Nucleic Acids Res"},{"key":"2024082207521470900_ocae133-B15","author":"Gillioz","year":"2020"},{"key":"2024082207521470900_ocae133-B16","first-page":"5753","article-title":"Xlnet: Generalized autoregressive pretraining for language understanding","volume":"32","author":"Yang","year":"2019","journal-title":"Adv Neu Inf Proc Syst"},{"key":"2024082207521470900_ocae133-B17","author":"Lavril","year":"2023"},{"key":"2024082207521470900_ocae133-B18","article-title":"Llama 2: open foundation and fine-tuned chat models","author":"Touvron","year":"2023","journal-title":"ArXiv"},{"issue":"2","key":"2024082207521470900_ocae133-B19","first-page":"e35179","article-title":"Artificial Hallucinations in ChatGPT: Implications in Scientific Writing","volume":"15","author":"Alkaissi","year":"2023","journal-title":"Cureus"},{"key":"2024082207521470900_ocae133-B20","doi-asserted-by":"crossref","first-page":"102695","DOI":"10.1016\/j.artmed.2023.102695","article-title":"Multimodal fine-tuning of clinical language models for predicting COVID-19 outcomes","volume":"146","author":"Henriksson","year":"2023","journal-title":"Artif Intell Med"},{"issue":"4","key":"2024082207521470900_ocae133-B21","doi-asserted-by":"crossref","first-page":"100729","DOI":"10.1016\/j.patter.2023.100729","article-title":"Fine-tuning large neural language models for biomedical natural language processing","volume":"4","author":"Tinn","year":"2023","journal-title":"Patterns (N Y)"},{"key":"2024082207521470900_ocae133-B22","doi-asserted-by":"crossref","first-page":"102086","DOI":"10.1016\/j.artmed.2021.102086","article-title":"Med7: A transferable clinical natural language processing model for electronic health records","volume":"118","author":"Kormilitzin","year":"2021","journal-title":"Artif Intell Med"},{"issue":"1","key":"2024082207521470900_ocae133-B23","doi-asserted-by":"crossref","first-page":"100887","DOI":"10.1016\/j.patter.2023.100887","article-title":"Enhancing phenotype recognition in clinical notes using large language models: PhenoBCBERT and PhenoGPT","volume":"5","author":"Yang","year":"2024","journal-title":"Patterns (N Y)"},{"key":"2024082207521470900_ocae133-B24","author":"Hu"},{"key":"2024082207521470900_ocae133-B25","author":"Wolf"},{"issue":"1","key":"2024082207521470900_ocae133-B26","doi-asserted-by":"crossref","first-page":"76","DOI":"10.1186\/s12911-018-0651-5","article-title":"SNOMED CT standard ontology based on the ontology for general medical science","volume":"18","author":"El-Sappagh","year":"2018","journal-title":"BMC Med Inform Decis Mak"},{"key":"2024082207521470900_ocae133-B27","author":"McCandless","year":"2010"},{"issue":"4","key":"2024082207521470900_ocae133-B28","doi-asserted-by":"crossref","first-page":"333","DOI":"10.1561\/1500000019","article-title":"The probabilistic relevance framework: BM25 and beyond","volume":"3","author":"Robertson","year":"2009","journal-title":"FNT in Information Retrieval"},{"key":"2024082207521470900_ocae133-B29","doi-asserted-by":"crossref","first-page":"66","DOI":"10.1186\/s12920-014-0066-9","article-title":"\u201cGenotype-first\u201d approaches on a curious case of idiopathic progressive cognitive decline","volume":"7","author":"Shi","year":"2014","journal-title":"BMC Med Genomics"},{"issue":"4","key":"2024082207521470900_ocae133-B30","doi-asserted-by":"crossref","first-page":"457","DOI":"10.1016\/j.ajhg.2009.09.003","article-title":"Clinical diagnostics in human genetics with semantic similarity searches in ontologies","volume":"85","author":"Kohler","year":"2009","journal-title":"Am J Hum Genet"},{"issue":"2","key":"2024082207521470900_ocae133-B31","doi-asserted-by":"crossref","first-page":"lqaa032","DOI":"10.1093\/nargab\/lqaa032","article-title":"Phen2Gene: rapid phenotype-driven gene prioritization for rare diseases","volume":"2","author":"Zhao","year":"2020","journal-title":"NAR Genom Bioinform"},{"key":"2024082207521470900_ocae133-B32","author":"Touvron"}],"container-title":["Journal of the American Medical Informatics Association"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/9\/2076\/58868268\/ocae133.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/jamia\/article-pdf\/31\/9\/2076\/58868268\/ocae133.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,8,22]],"date-time":"2024-08-22T08:16:37Z","timestamp":1724314597000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/jamia\/article\/31\/9\/2076\/7686878"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6,3]]},"references-count":32,"journal-issue":{"issue":"9","published-online":{"date-parts":[[2024,6,3]]},"published-print":{"date-parts":[[2024,9,1]]}},"URL":"https:\/\/doi.org\/10.1093\/jamia\/ocae133","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2023.12.28.573586","asserted-by":"object"}]},"ISSN":["1067-5027","1527-974X"],"issn-type":[{"value":"1067-5027","type":"print"},{"value":"1527-974X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,9]]},"published":{"date-parts":[[2024,6,3]]}}}