{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"institution":[{"name":"medRxiv"}],"indexed":{"date-parts":[[2026,10,18]],"date-time":"2026-10-18T14:22:56Z","timestamp":1792333376442,"version":"4.4.0"},"posted":{"date-parts":[[2024,12,2]]},"group-title":"Genetic and Genomic Medicine","reference-count":29,"publisher":"openRxiv","license":[{"start":{"date-parts":[[2024,12,2]],"date-time":"2024-12-02T00:00:00Z","timestamp":1733097600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"accepted":{"date-parts":[[2024,12,2]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                <jats:sec>\n                  <jats:title>Background<\/jats:title>\n                  <jats:p>Diagnosing rare genetic disorders relies on precise phenotypic and genotypic analysis, with the Human Phenotype Ontology (HPO) providing a standardized language for capturing clinical phenotypes. Traditional HPO tools, such as Doc2HPO and ClinPhen, employ concept recognition to automate phenotype extraction but struggle with incomplete phenotype assignment, often requiring intensive manual review. While large language models (LLMs) hold promise for more context-driven phenotype extraction, they are prone to errors and \u201challucinations,\u201d making them less reliable without further refinement. We present RAG-HPO, a Python-based tool that leverages Retrieval-Augmented Generation (RAG) to elevate LLM accuracy in HPO term assignment, bypassing the limitations of baseline models while avoiding the time and resource intensive process of fine-tuning. RAG-HPO integrates a dynamic vector database, allowing real-time retrieval and contextual matching.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Methods<\/jats:title>\n                  <jats:p>The high-dimensional vector database utilized by RAG-HPO includes &gt;54,000 phenotypic phrases mapped to HPO IDs, derived from the HPO database and supplemented with additional validated phrases. The RAG-HPO workflow uses an LLM to first extract phenotypic phrases that are then matched via semantic similarity to entries within a vector database before providing best term matches back to the LLM as context for final HPO term assignment. A benchmarking dataset of 120 published case reports with 1,792 manually-assigned HPO terms was developed, and the performance of RAG-HPO measured against existing published tools Doc2HPO, ClinPhen, and FastHPOCR.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Results<\/jats:title>\n                  <jats:p>In evaluations, RAG-HPO, powered by Llama-3 70B and applied to a set of 120 case reports, achieved a mean precision of 0.84, recall of 0.78, and an F1 score of 0.80\u2014significantly surpassing conventional tools (p&lt;0.00001). False positive HPO term identification occurred for 15.8% (256\/1,624) of terms, of which only 2.7% (7\/256) represented hallucinations, and 33.6% (86\/256) unrelated terms; the remainder of false positives (63.7%, 163\/256) were relative terms of the target term.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>Conclusions<\/jats:title>\n                  <jats:p>RAG-HPO is a user-friendly, adaptable tool designed for secure evaluation of clinical text and outperforms standard HPO-matching tools in precision, recall, and F1. Its enhanced precision and recall represent a substantial advancement in phenotypic analysis, accelerating the identification of genetic mechanisms underlying rare diseases and driving progress in genetic research and clinical genomics.<\/jats:p>\n                <\/jats:sec>","DOI":"10.1101\/2024.12.01.24318253","type":"posted-content","created":{"date-parts":[[2024,12,2]],"date-time":"2024-12-02T12:55:13Z","timestamp":1733144113000},"source":"Crossref","is-referenced-by-count":8,"title":["Improving Automated Deep Phenotyping Through Large Language Models Using Retrieval Augmented Generation"],"prefix":"10.64898","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7734-8601","authenticated-orcid":false,"given":"Brandon T.","family":"Garcia","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lauren","family":"Westerfield","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Priya","family":"Yelemali","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nikhita","family":"Gogate","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"E. Andres","family":"Rivera-Munoz","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Haowei","family":"Du","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Moez","family":"Dawood","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Angad","family":"Jolly","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9907-9246","authenticated-orcid":false,"given":"James R.","family":"Lupski","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jennifer E.","family":"Posey","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"54368","reference":[{"key":"2024120420302968000_2024.12.01.24318253v1.1","doi-asserted-by":"publisher","DOI":"10.1056\/NEJMoa1516767"},{"key":"2024120420302968000_2024.12.01.24318253v1.2","doi-asserted-by":"crossref","unstructured":"Posey JE . Genome sequencing and implications for rare disorders. Orphanet J Rare Dis. 2019;14:153.","DOI":"10.1186\/s13023-019-1127-0"},{"key":"2024120420302968000_2024.12.01.24318253v1.3","doi-asserted-by":"publisher","DOI":"10.1056\/NEJMc1812033"},{"key":"2024120420302968000_2024.12.01.24318253v1.4","doi-asserted-by":"publisher","DOI":"10.1056\/NEJMsr1406261"},{"key":"2024120420302968000_2024.12.01.24318253v1.5","doi-asserted-by":"publisher","DOI":"10.1056\/NEJMoa2209046"},{"key":"2024120420302968000_2024.12.01.24318253v1.6","doi-asserted-by":"publisher","DOI":"10.1056\/NEJMe2302643"},{"key":"2024120420302968000_2024.12.01.24318253v1.7","doi-asserted-by":"crossref","first-page":"798","DOI":"10.1038\/s41436-018-0408-7","article-title":"Insights into genetics, human biology and disease gleaned from family based genomic studies","volume":"21","year":"2019","journal-title":"Genet Med"},{"key":"2024120420302968000_2024.12.01.24318253v1.8","doi-asserted-by":"crossref","first-page":"D938","DOI":"10.1093\/nar\/gkad1082","article-title":"The Monarch Initiative in 2024: an analytic platform integrating phenotypes, genes and diseases across species","volume":"52","year":"2024","journal-title":"Nucleic Acids Res"},{"key":"2024120420302968000_2024.12.01.24318253v1.9","doi-asserted-by":"publisher","DOI":"10.1016\/j.ajhg.2008.09.017"},{"key":"2024120420302968000_2024.12.01.24318253v1.10","doi-asserted-by":"publisher","DOI":"10.1016\/j.ajhg.2018.05.010"},{"key":"2024120420302968000_2024.12.01.24318253v1.11","doi-asserted-by":"publisher","DOI":"10.1093\/nar\/gkz386"},{"key":"2024120420302968000_2024.12.01.24318253v1.12","doi-asserted-by":"crossref","first-page":"1585","DOI":"10.1038\/s41436-018-0381-1","article-title":"ClinPhen extracts and prioritizes patient phenotypes directly from medical records to expedite genetic disease diagnosis","volume":"21","year":"2019","journal-title":"Genet Med"},{"key":"2024120420302968000_2024.12.01.24318253v1.13","doi-asserted-by":"crossref","unstructured":"Groza T , Gration D , Baynam G , Robinson PN . FastHPOCR: pragmatic, fast, and accurate concept recognition using the human phenotype ontology. Bioinformatics. 2024;40.","DOI":"10.1093\/bioinformatics\/btae406"},{"key":"2024120420302968000_2024.12.01.24318253v1.14","doi-asserted-by":"publisher","DOI":"10.1016\/j.ajhg.2024.08.010"},{"key":"2024120420302968000_2024.12.01.24318253v1.15","doi-asserted-by":"crossref","unstructured":"Groza T , Caufield H , Gration D , Baynam G , Haendel MA , Robinson PN , et al. An evaluation of GPT models for phenotype concept recognition. BMC Med Inform Decis Mak. 2024;24:30.","DOI":"10.1186\/s12911-024-02439-w"},{"key":"2024120420302968000_2024.12.01.24318253v1.16","doi-asserted-by":"crossref","unstructured":"Gargari OK , Fatehi F , Mohammadi I , Firouzabadi SR , Shafiee A , Habibi G . Diagnostic accuracy of large language models in psychiatry. Asian J Psychiatr. 2024;100:104168.","DOI":"10.1016\/j.ajp.2024.104168"},{"key":"2024120420302968000_2024.12.01.24318253v1.17","doi-asserted-by":"crossref","unstructured":"Young CC , Enichen E , Rivera C , Auger CA , Grant N , Rao A , et al. Diagnostic accuracy of a custom large language model on rare pediatric disease case reports. Am J Med Genet A. 2024;:e63878.","DOI":"10.1002\/ajmg.a.63878"},{"key":"2024120420302968000_2024.12.01.24318253v1.18","doi-asserted-by":"crossref","first-page":"2030","DOI":"10.1093\/jamia\/ocae129","article-title":"RefAI: a GPT-powered retrieval-augmented generative tool for biomedical literature recommendation and summarization","volume":"31","year":"2024","journal-title":"J Am Med Inform Assoc"},{"key":"2024120420302968000_2024.12.01.24318253v1.19","doi-asserted-by":"publisher","DOI":"10.1038\/s41592-024-02201-0"},{"key":"2024120420302968000_2024.12.01.24318253v1.20","first-page":"152","article-title":"Developing ChatGPT for biology and medicine: a complete review of biomedical question answering","volume":"10","year":"2024","journal-title":"Biophys Rep"},{"key":"2024120420302968000_2024.12.01.24318253v1.21","doi-asserted-by":"crossref","unstructured":"Yang J , Liu C , Deng W , Wu D , Weng C , Zhou Y , et al. Enhancing phenotype recognition in clinical notes using large language models: PhenoBCBERT and PhenoGPT. Patterns (N Y). 2024;5:100887.","DOI":"10.1016\/j.patter.2023.100887"},{"key":"2024120420302968000_2024.12.01.24318253v1.22","doi-asserted-by":"crossref","first-page":"1269","DOI":"10.1109\/TCBB.2022.3170301","article-title":"Phenobert: A combined deep learning method for automated recognition of human phenotype ontology","volume":"20","year":"2023","journal-title":"IEEE\/ACM Trans Comput Biol Bioinform"},{"key":"2024120420302968000_2024.12.01.24318253v1.23","doi-asserted-by":"crossref","first-page":"e0000568","DOI":"10.1371\/journal.pdig.0000568","article-title":"Improving accuracy of GPT-3\/4 results on biomedical data using a retrieval-augmented language model","volume":"3","year":"2024","journal-title":"PLOS Digit Health"},{"key":"2024120420302968000_2024.12.01.24318253v1.24","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btae238"},{"key":"2024120420302968000_2024.12.01.24318253v1.25","first-page":"652","article-title":"PubMed Retrieval with RAG Techniques","volume":"316","year":"2024","journal-title":"Stud Health Technol Inform"},{"key":"2024120420302968000_2024.12.01.24318253v1.26","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2005.11401"},{"key":"2024120420302968000_2024.12.01.24318253v1.27","doi-asserted-by":"publisher","DOI":"10.1002\/humu.24389"},{"key":"2024120420302968000_2024.12.01.24318253v1.28","doi-asserted-by":"publisher","DOI":"10.1093\/database\/bav005"},{"issue":"1","key":"2024120420302968000_2024.12.01.24318253v1.29","first-page":"e12984","article-title":"Case Study of a Rare Genetic Disorder: Congenital Insensitivity to Pain With Anhidrosis","volume":"13","year":"2021","journal-title":"Cureus"}],"container-title":[],"original-title":[],"link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.1101\/2024.12.01.24318253","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,15]],"date-time":"2026-01-15T20:21:45Z","timestamp":1768508505000},"score":1,"resource":{"primary":{"URL":"http:\/\/medrxiv.org\/lookup\/doi\/10.1101\/2024.12.01.24318253"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,2]]},"references-count":29,"URL":"https:\/\/doi.org\/10.1101\/2024.12.01.24318253","relation":{},"subject":[],"published":{"date-parts":[[2024,12,2]]},"subtype":"preprint"}}