{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,1]],"date-time":"2026-04-01T19:11:32Z","timestamp":1775070692625,"version":"3.50.1"},"reference-count":27,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2024,1,31]],"date-time":"2024-01-31T00:00:00Z","timestamp":1706659200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,1,31]],"date-time":"2024-01-31T00:00:00Z","timestamp":1706659200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["HG010860, OD011883"],"award-info":[{"award-number":["HG010860, OD011883"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["HG010860, OD011883"],"award-info":[{"award-number":["HG010860, OD011883"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["HG010860, OD011883"],"award-info":[{"award-number":["HG010860, OD011883"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["HG010860, OD011883"],"award-info":[{"award-number":["HG010860, OD011883"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["HG010860, OD011883"],"award-info":[{"award-number":["HG010860, OD011883"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"name":"The Director, Office of Science, Office of Basic Energy Sciences, of the US Department of Energy","award":["DE-AC0205CH1123"],"award-info":[{"award-number":["DE-AC0205CH1123"]}]},{"name":"The Director, Office of Science, Office of Basic Energy Sciences, of the US Department of Energy","award":["DE-AC0205CH1123"],"award-info":[{"award-number":["DE-AC0205CH1123"]}]},{"name":"The Director, Office of Science, Office of Basic Energy Sciences, of the US Department of Energy","award":["DE-AC0205CH1123"],"award-info":[{"award-number":["DE-AC0205CH1123"]}]},{"DOI":"10.13039\/501100020544","name":"Angela Wright Bennett Foundation","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100020544","id-type":"DOI","asserted-by":"publisher"}]},{"name":"The Stan Perron Charitable Foundation, Australia"},{"name":"The McCusker Charitable Foundation via Channel 7 Telethon Trust"},{"name":"Mineral Resources via the Perth Children's Hospital Foundation"},{"DOI":"10.13039\/100000051","name":"National Human Genome Research Institute","doi-asserted-by":"publisher","award":["RM1HG010860, U24HG011449"],"award-info":[{"award-number":["RM1HG010860, U24HG011449"]}],"id":[{"id":"10.13039\/100000051","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000051","name":"National Human Genome Research Institute","doi-asserted-by":"publisher","award":["RM1HG010860, U24HG011449"],"award-info":[{"award-number":["RM1HG010860, U24HG011449"]}],"id":[{"id":"10.13039\/100000051","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Med Inform Decis Mak"],"abstract":"<jats:title>Abstract<\/jats:title><jats:sec>\n                <jats:title>Objective<\/jats:title>\n                <jats:p>Clinical deep phenotyping and phenotype annotation play a critical role in both the diagnosis of patients with rare disorders as well as in building computationally-tractable knowledge in the rare disorders field. These processes rely on using ontology concepts, often from the Human Phenotype Ontology, in conjunction with a phenotype concept recognition task (supported usually by machine learning methods) to curate patient profiles or existing scientific literature. With the significant shift in the use of large language models (LLMs) for most NLP tasks, we examine the performance of the latest Generative Pre-trained Transformer (GPT) models underpinning ChatGPT as a foundation for the tasks of clinical phenotyping and phenotype annotation.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Materials and methods<\/jats:title>\n                <jats:p>The experimental setup of the study included seven prompts of various levels of specificity, two GPT models (gpt-3.5-turbo and gpt-4.0) and two established gold standard corpora for phenotype recognition, one consisting of publication abstracts and the other clinical observations.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Results<\/jats:title>\n                <jats:p>The best run, using in-context learning, achieved 0.58 document-level F1 score on publication abstracts and 0.75 document-level F1 score on clinical observations, as well as a mention-level F1 score of 0.7, which surpasses the current best in class tool. Without in-context learning, however, performance is significantly below the existing approaches.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Conclusion<\/jats:title>\n                <jats:p>Our experiments show that gpt-4.0 surpasses the state of the art performance if the task is constrained to a subset of the target ontology where there is prior knowledge of the terms that are expected to be matched. While the results are promising, the non-deterministic nature of the outcomes, the high cost and the lack of concordance between different runs using the same prompt and input make the use of these LLMs challenging for this particular task.<\/jats:p>\n              <\/jats:sec>","DOI":"10.1186\/s12911-024-02439-w","type":"journal-article","created":{"date-parts":[[2024,1,31]],"date-time":"2024-01-31T12:03:24Z","timestamp":1706702604000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":35,"title":["An evaluation of GPT models for phenotype concept recognition"],"prefix":"10.1186","volume":"24","author":[{"given":"Tudor","family":"Groza","sequence":"first","affiliation":[]},{"given":"Harry","family":"Caufield","sequence":"additional","affiliation":[]},{"given":"Dylan","family":"Gration","sequence":"additional","affiliation":[]},{"given":"Gareth","family":"Baynam","sequence":"additional","affiliation":[]},{"given":"Melissa A.","family":"Haendel","sequence":"additional","affiliation":[]},{"given":"Peter N.","family":"Robinson","sequence":"additional","affiliation":[]},{"given":"Christopher J.","family":"Mungall","sequence":"additional","affiliation":[]},{"given":"Justin T.","family":"Reese","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2024,1,31]]},"reference":[{"key":"2439_CR1","doi-asserted-by":"publisher","first-page":"223","DOI":"10.1016\/j.ymgme.2015.11.003","volume":"116","author":"D Taruscio","year":"2015","unstructured":"Taruscio D, Groft SC, Cederroth H, et al. Undiagnosed Diseases Network International (UDNI): White paper for global actions to meet patient needs. Mol Genet Metab. 2015;116:223\u20135.","journal-title":"Mol Genet Metab"},{"key":"2439_CR2","first-page":"659","volume":"43","author":"KM Boycott","year":"2022","unstructured":"Boycott KM, Azzariti DR, Hamosh A, Rehm HL. Seven years since the launch of the Matchmaker Exchange: The evolution of genomic matchmaking. Hum Mutat. 2022;43:659\u201367.","journal-title":"Hum Mutat"},{"key":"2439_CR3","doi-asserted-by":"publisher","first-page":"817","DOI":"10.1038\/s41587-022-01357-4","volume":"40","author":"JOB Jacobsen","year":"2022","unstructured":"Jacobsen JOB, Baudis M, Baynam GS, et al. The GA4GH Phenopacket schema defines a computable representation of clinical data. Nat Biotechnol. 2022;40:817\u201320.","journal-title":"Nat Biotechnol"},{"key":"2439_CR4","doi-asserted-by":"publisher","first-page":"595","DOI":"10.1016\/j.ajhg.2016.07.005","volume":"99","author":"D Smedley","year":"2016","unstructured":"Smedley D, Schubach M, Jacobsen JOB, et al. A whole-genome analysis framework for effective identification of pathogenic regulatory variants in Mendelian disease. Am J Hum Genet. 2016;99:595\u2013606.","journal-title":"Am J Hum Genet"},{"key":"2439_CR5","doi-asserted-by":"publisher","first-page":"58","DOI":"10.1016\/j.ajhg.2018.05.010","volume":"103","author":"JH Son","year":"2018","unstructured":"Son JH, Xie G, Yuan C, et al. Deep Phenotyping on electronic health records facilitates genetic diagnosis by clinical exomes. Am J Hum Genet. 2018;103:58\u201373.","journal-title":"Am J Hum Genet"},{"key":"2439_CR6","doi-asserted-by":"publisher","first-page":"16","DOI":"10.1038\/s41525-018-0053-8","volume":"3","author":"MM Clark","year":"2018","unstructured":"Clark MM, Stark Z, Farnaes L, et al. Meta-analysis of the diagnostic and clinical utility of genome and exome sequencing and chromosomal microarray in children with suspected genetic diseases. NPJ Genom Med. 2018;3:16.","journal-title":"NPJ Genom Med"},{"key":"2439_CR7","doi-asserted-by":"publisher","first-page":"610","DOI":"10.1016\/j.ajhg.2008.09.017","volume":"83","author":"PN Robinson","year":"2008","unstructured":"Robinson PN, K\u00f6hler S, Bauer S, et al. The Human Phenotype Ontology: a tool for annotating and analyzing human hereditary disease. Am J Hum Genet. 2008;83:610\u20135.","journal-title":"Am J Hum Genet"},{"key":"2439_CR8","doi-asserted-by":"publisher","first-page":"D1018","DOI":"10.1093\/nar\/gky1105","volume":"47","author":"S K\u00f6hler","year":"2019","unstructured":"K\u00f6hler S, Carmody L, Vasilevsky N, et al. Expansion of the Human Phenotype Ontology (HPO) knowledge base and resources. Nucleic Acids Res. 2019;47:D1018\u201327.","journal-title":"Nucleic Acids Res"},{"key":"2439_CR9","doi-asserted-by":"publisher","first-page":"D704","DOI":"10.1093\/nar\/gkz997","volume":"48","author":"KA Shefchek","year":"2020","unstructured":"Shefchek KA, Harris NL, Gargano M, et al. The Monarch Initiative in 2019: an integrative data and analytic platform connecting phenotypes to genotypes across species. Nucleic Acids Res. 2020;48:D704\u201315.","journal-title":"Nucleic Acids Res"},{"issue":"20","key":"2439_CR10","doi-asserted-by":"publisher","first-page":"1868","DOI":"10.1056\/NEJMoa2035790","volume":"385","author":"10,000 Genomes Project Pilot Investigators","year":"2021","unstructured":"10,000 Genomes Project Pilot Investigators, et al. 100,000 genomes pilot on rare-disease diagnosis in health care - preliminary report. N Engl J Med. 2021;385(20):1868\u201380.","journal-title":"N Engl J Med"},{"key":"2439_CR11","doi-asserted-by":"publisher","first-page":"e12596","DOI":"10.2196\/12596","volume":"7","author":"A Arbabi","year":"2019","unstructured":"Arbabi A, Adams DR, Fidler S, Brudno M. Identifying clinical terms in medical text using ontology-guided machine learning. JMIR Med Inform. 2019;7:e12596.","journal-title":"JMIR Med Inform"},{"key":"2439_CR12","doi-asserted-by":"publisher","first-page":"1884","DOI":"10.1093\/bioinformatics\/btab019","volume":"37","author":"L Luo","year":"2021","unstructured":"Luo L, Yan S, Lai P-T, et al. PhenoTagger: a hybrid method for phenotype concept recognition using human phenotype ontology. Bioinformatics. 2021;37:1884\u201390.","journal-title":"Bioinformatics"},{"key":"2439_CR13","doi-asserted-by":"publisher","first-page":"1346","DOI":"10.1038\/s41551-022-00914-1","volume":"6","author":"R Krishnan","year":"2022","unstructured":"Krishnan R, Rajpurkar P, Topol EJ. Self-supervised learning in medicine and healthcare. Nat Biomed Eng. 2022;6:1346\u201352.","journal-title":"Nat Biomed Eng"},{"key":"2439_CR14","doi-asserted-by":"publisher","first-page":"1930","DOI":"10.1038\/s41591-023-02448-8","volume":"29","author":"AJ Thirunavukarasu","year":"2023","unstructured":"Thirunavukarasu AJ, et al. Large language models in medicine. Nat Med. 2023;29:1930\u201340.","journal-title":"Nat Med"},{"key":"2439_CR15","doi-asserted-by":"publisher","first-page":"1234","DOI":"10.1093\/bioinformatics\/btz682","volume":"36","author":"J Lee","year":"2020","unstructured":"Lee J, et al. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics. 2020;36:1234\u201340.","journal-title":"Bioinformatics"},{"key":"2439_CR16","doi-asserted-by":"publisher","first-page":"259","DOI":"10.1038\/s41586-023-05881-4","volume":"616","author":"M Moor","year":"2023","unstructured":"Moor M, et al. Foundation models for generalist medical artificial intelligence. Nature. 2023;616:259\u201365.","journal-title":"Nature"},{"issue":"1","key":"2439_CR17","first-page":"1","volume":"3","author":"G Yu","year":"2021","unstructured":"Yu G, Tinn R, Cheng H, et al. Domain-specific language model pretraining for biomedical natural language processing. ACM Trans Comp Healthc (HEALTH). 2021;3(1):1\u201323.","journal-title":"ACM Trans Comp Healthc (HEALTH)"},{"issue":"6","key":"2439_CR18","doi-asserted-by":"publisher","first-page":"bbac409","DOI":"10.1093\/bib\/bbac409","volume":"23","author":"R Luo","year":"2022","unstructured":"Luo R, Sun L, Xia Y, Qin T, Zhang S, Poon H, Liu TY. BioGPT: generative pre-trained transformer for biomedical text generation and mining. Brief Bioinform. 2022;23(6):bbac409. https:\/\/doi.org\/10.1093\/bib\/bbac409.","journal-title":"Brief Bioinform"},{"key":"2439_CR19","doi-asserted-by":"crossref","unstructured":"Ding B, Qin C, Liu L, Chia YK, Joty S, Li B, Bing L. Is GPT-3 a Good Data Annotator? Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023;1:11173\u201311195.\u00a0LongPapers.\u00a0","DOI":"10.18653\/v1\/2023.acl-long.626"},{"key":"2439_CR20","volume-title":"Can GPT Alleviate the Burden of Annotation? Proceedings of JURIX 2023 36th International Conference on Legal Knowledge and Information Systems","author":"M Gray","year":"2023","unstructured":"Gray M, Savelka J, Oliver W, Ashley K. Can GPT Alleviate the Burden of Annotation? Proceedings of JURIX 2023 36th International Conference on Legal Knowledge and Information Systems. Maastricht: Maastricht University; 2023."},{"key":"2439_CR21","unstructured":"Chen Q, Du J, Hu Y, Keloth VK, Peng X, Raja K, Zhang R, Lu X, Xu H. Large language models in biomedical natural language processing: benchmarks, baselines, and recommendations.\u00a02023. arXiv preprint, arXiv:2305.16326."},{"key":"2439_CR22","doi-asserted-by":"publisher","first-page":"bav005","DOI":"10.1093\/database\/bav005","volume":"2015","author":"T Groza","year":"2015","unstructured":"Groza T, K\u00f6hler S, Doelken S, Collier N, Oellrich A, Smedley D, Couto FM, Baynam G, Zankl A, Robinson PN. Automatic concept recognition using the human phenotype ontology reference and test suite corpora. Database (Oxford). 2015;2015:bav005. https:\/\/doi.org\/10.1093\/database\/bav005. Print 2015.","journal-title":"Database (Oxford)"},{"key":"2439_CR23","first-page":"8565739","volume":"2017","author":"M Lobo","year":"2017","unstructured":"Lobo M, Lamurias A, Couto FM. Identifying Human phenotype terms by combining machine learning and validation rules. Biomed Res Inte. 2017;2017:8565739.","journal-title":"Biomed Res Inte"},{"key":"2439_CR24","doi-asserted-by":"publisher","unstructured":"Weissenbacher D, Rawal S, Zhao X, Priestley JRC, Szigety KM, Schmidt SF, Higgins MJ, Magge A, O\u2019Connor K, Gonzalez-Hernandez G, Campbell IM. PheNorm, a language model normalizer of physical examinations from genetics clinical notes. medRxiv 2023.10.16.23296894.\u00a0https:\/\/doi.org\/10.1101\/2023.10.16.23296894.","DOI":"10.1101\/2023.10.16.23296894"},{"key":"2439_CR25","doi-asserted-by":"publisher","first-page":"W566","DOI":"10.1093\/nar\/gkz386","volume":"47","author":"C Liu","year":"2019","unstructured":"Liu C, Kury FSP, Li Z, Ta C, Wang K, Weng C. Doc2Hpo: a web application for efficient and accurate HPO concept curation. Nucleic Acids Res. 2019;47:W566\u201370.","journal-title":"Nucleic Acids Res"},{"key":"2439_CR26","doi-asserted-by":"publisher","first-page":"1585","DOI":"10.1038\/s41436-018-0381-1","volume":"21","author":"CA Deisseroth","year":"2019","unstructured":"Deisseroth CA, Birgmeier J, Bodle EE, et al. ClinPhen extracts and prioritizes patient phenotypes directly from medical records to expedite genetic disease diagnosis. Genet Med. 2019;21:1585\u201393.","journal-title":"Genet Med"},{"key":"2439_CR27","first-page":"56","volume":"2009","author":"C Jonquet","year":"2009","unstructured":"Jonquet C, Shah NH, Musen MA. The Open Biomedical Annotator. AMIA Joint Summit Transl Bioinform. 2009;2009:56\u201360.","journal-title":"AMIA Joint Summit Transl Bioinform"}],"container-title":["BMC Medical Informatics and Decision Making"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-024-02439-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12911-024-02439-w\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12911-024-02439-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,1,31]],"date-time":"2024-01-31T12:06:13Z","timestamp":1706702773000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcmedinformdecismak.biomedcentral.com\/articles\/10.1186\/s12911-024-02439-w"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,1,31]]},"references-count":27,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,12]]}},"alternative-id":["2439"],"URL":"https:\/\/doi.org\/10.1186\/s12911-024-02439-w","relation":{},"ISSN":["1472-6947"],"issn-type":[{"value":"1472-6947","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,1,31]]},"assertion":[{"value":"23 November 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"24 January 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"31 January 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare no competing interests.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"30"}}