{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,18]],"date-time":"2026-03-18T01:03:22Z","timestamp":1773795802370,"version":"3.50.1"},"reference-count":33,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2024,1,5]],"date-time":"2024-01-05T00:00:00Z","timestamp":1704412800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,1,5]],"date-time":"2024-01-05T00:00:00Z","timestamp":1704412800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Healthc Inform Res"],"published-print":{"date-parts":[[2024,6]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:sec>\n                <jats:title>Purpose<\/jats:title>\n                <jats:p>Phenotyping is critical for informing rare disease diagnosis and treatment, but disease phenotypes are often embedded in unstructured text. While natural language processing (NLP) can automate extraction, a major bottleneck is developing annotated corpora. Recently, prompt learning with large language models (LLMs) has been shown to lead to generalizable results without any (zero-shot) or few annotated samples (few-shot), but none have explored this for rare diseases. Our work is the first to study prompt learning for identifying and extracting rare disease phenotypes in the zero- and few-shot settings.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Methods<\/jats:title>\n                <jats:p>We compared the performance of prompt learning with ChatGPT and fine-tuning with BioClinicalBERT. We engineered novel prompts for ChatGPT to identify and extract rare diseases and their phenotypes (e.g., diseases, symptoms, and signs), established a benchmark for evaluating its performance, and conducted an in-depth error analysis.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Results<\/jats:title>\n                <jats:p>Overall, fine-tuning BioClinicalBERT resulted in higher performance (F1 of 0.689) than ChatGPT (F1 of 0.472 and 0.610 in the zero- and few-shot settings, respectively). However, ChatGPT achieved higher accuracy for rare diseases and signs in the one-shot setting (F1 of 0.778 and 0.725). Conversational, sentence-based prompts generally achieved higher accuracy than structured lists.<\/jats:p>\n              <\/jats:sec><jats:sec>\n                <jats:title>Conclusion<\/jats:title>\n                <jats:p>Prompt learning using ChatGPT has the potential to match or outperform fine-tuning BioClinicalBERT at extracting rare diseases and signs with just one annotated sample. Given its accessibility, ChatGPT could be leveraged to extract these entities without relying on a large, annotated corpus. While LLMs can support rare disease phenotyping, researchers should critically evaluate model outputs to ensure phenotyping accuracy.<\/jats:p>\n              <\/jats:sec>","DOI":"10.1007\/s41666-023-00155-0","type":"journal-article","created":{"date-parts":[[2024,1,5]],"date-time":"2024-01-05T14:01:51Z","timestamp":1704463311000},"page":"438-461","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":49,"title":["Identifying and Extracting Rare Diseases and Their Phenotypes with Large Language Models"],"prefix":"10.1007","volume":"8","author":[{"given":"Cathy","family":"Shyr","sequence":"first","affiliation":[]},{"given":"Yan","family":"Hu","sequence":"additional","affiliation":[]},{"given":"Lisa","family":"Bastarache","sequence":"additional","affiliation":[]},{"given":"Alex","family":"Cheng","sequence":"additional","affiliation":[]},{"given":"Rizwan","family":"Hamid","sequence":"additional","affiliation":[]},{"given":"Paul","family":"Harris","sequence":"additional","affiliation":[]},{"given":"Hua","family":"Xu","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2024,1,5]]},"reference":[{"issue":"2","key":"155_CR1","doi-asserted-by":"publisher","first-page":"165","DOI":"10.1038\/s41431-019-0508-0","volume":"28","author":"S Nguengang Wakap","year":"2020","unstructured":"Nguengang Wakap S, Lambert DM, Olry A, Rodwell C, Gueydan C, Lanneau V, Murphy D, Le Cam Y, Rath A (2020) Estimating cumulative point prevalence of rare diseases: analysis of the Orphanet database. Eur J Hum Genet 28(2):165\u2013173","journal-title":"Eur J Hum Genet"},{"key":"155_CR2","doi-asserted-by":"publisher","first-page":"1028545","DOI":"10.3389\/fpubh.2022.1028545","volume":"10","author":"Chung CCY, Project HKG, Chu ATW, Chung BHY","year":"2022","unstructured":"Chung CCY, Project HKG, Chu ATW, Chung BHY (2022) Rare disease emerging as a global public health priority. Front Public Health 10:1028545","journal-title":"Front Public Health"},{"issue":"5","key":"155_CR3","doi-asserted-by":"publisher","first-page":"1136","DOI":"10.1002\/ajmg.a.33380","volume":"152","author":"JS Cohen","year":"2010","unstructured":"Cohen JS, Biesecker BB (2010) Quality of life in rare genetic conditions: a systematic review of the literature. Am J Med Genet A 152(5):1136\u20131156","journal-title":"Am J Med Genet A"},{"key":"155_CR4","doi-asserted-by":"publisher","first-page":"325","DOI":"10.1007\/s10897-014-9773-9","volume":"24","author":"N Carmichael","year":"2015","unstructured":"Carmichael N, Tsipis J, Windmueller G, Mandel L, Estrella E (2015) Is it going to hurt?: the impact of the diagnostic odyssey on children and their families. J Genet Couns 24:325\u2013335","journal-title":"J Genet Couns"},{"issue":"1","key":"155_CR5","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s13023-022-02299-5","volume":"17","author":"G Yang","year":"2022","unstructured":"Yang G, Cintina I, Pariser A, Oehrlein E, Sullivan J, Kennedy A (2022) The national economic burden of rare disease in the united states in 2019. Orphanet J Rare Dis 17(1):1\u201311","journal-title":"Orphanet J Rare Dis"},{"issue":"6","key":"155_CR6","doi-asserted-by":"publisher","first-page":"626","DOI":"10.1097\/MOP.0000000000000155","volume":"26","author":"CJ Tifft","year":"2014","unstructured":"Tifft CJ, Adams DR (2014) The national institutes of health undiagnosed diseases program. Curr Opin Pediatr 26(6):626","journal-title":"Curr Opin Pediatr"},{"issue":"3\u20134","key":"155_CR7","first-page":"179","volume":"4","author":"EF Macnamara","year":"2019","unstructured":"Macnamara EF, D\u2019Souza P, Tifft CJ et al (2019) The undiagnosed diseases program: approach to diagnosis. Trans Sci Rare Dis 4(3\u20134):179\u2013188","journal-title":"Trans Sci Rare Dis"},{"key":"155_CR8","doi-asserted-by":"crossref","unstructured":"Ahmad FS, Ricket IM, Hammill BG, Eskenazi L, Robertson HR, Curtis LH, Dobi CD, Girotra S, Haynes K, Kizer JR, et al (2020) Computable phenotype implementation for a national, multicenter pragmatic clinical trial: lessons learned from adaptable. Circ: Cardio Qual Outcomes 13(6):006292","DOI":"10.1161\/CIRCOUTCOMES.119.006292"},{"key":"155_CR9","doi-asserted-by":"crossref","unstructured":"Chapman M, Dom\u00ednguez J, Fairweather E, Delaney B, Curcin V (2021) Using computable phenotypes in point-of-care clinical trial recruitment. In: public health and informatics-proceedings of MIE 2021: Studies in health technology and informatics, pp 560\u2013564. IOS Press, ???","DOI":"10.3233\/SHTI210233"},{"key":"155_CR10","doi-asserted-by":"publisher","first-page":"34","DOI":"10.1016\/j.jbi.2017.11.011","volume":"77","author":"Y Wang","year":"2018","unstructured":"Wang Y, Wang L, Rastegar-Mojarad M, Moon S, Shen F, Afzal N, Liu S, Zeng Y, Mehrabi S, Sohn S et al (2018) Clinical information extraction applications: a literature review. J Biomed Inf 77:34\u201349","journal-title":"J Biomed Inf"},{"key":"155_CR11","unstructured":"Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser \u0141, Polosukhin I (2017) Attention is all you need. Advances in neural information processing systems, p 30"},{"key":"155_CR12","unstructured":"Devlin J, Chang M-W, Lee K, Toutanova K (2018) Bert: pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805"},{"key":"155_CR13","unstructured":"OpenAI: introducing ChatGPT (2022) https:\/\/openai.com\/blog\/chatgpt"},{"key":"155_CR14","doi-asserted-by":"crossref","unstructured":"Agrawal M, Hegselmann S, Lang H, Kim Y, Sontag D (2022) Large language models are few-shot clinical information extractors. In: Proceedings of the 2022 Conference on empirical methods in natural language processing, pp 1998\u20132022","DOI":"10.18653\/v1\/2022.emnlp-main.130"},{"key":"155_CR15","unstructured":"Hu Y, Ameer I, Zuo X, Peng X, Zhou Y, Li Z, Li Y, Li J, Jiang X, Xu H (2023) Zero-shot clinical entity recognition using ChatGPT. arXiv:2303.16416"},{"key":"155_CR16","unstructured":"Chen Q, Du J, Hu Y, Keloth VK, Peng X, Raja K, Zhang R, Lu Z, Xu H (2023) Large language models in biomedical natural language processing: benchmarks, baselines, and recommendations. arXiv:2305.16326"},{"key":"155_CR17","unstructured":"Lee P, Goldberg C, Kohane I (2023) The AI revolution in medicine: GPT-4 and beyond. Pearson"},{"key":"155_CR18","doi-asserted-by":"crossref","unstructured":"Mehnen L, Gruarin S, Vasileva M, Knapp B (2023) ChatGPT as a medical doctor? A diagnostic accuracy study on common and rare diseases. medRxiv, 2023\u201304","DOI":"10.1101\/2023.04.20.23288859"},{"key":"155_CR19","doi-asserted-by":"publisher","first-page":"103961","DOI":"10.1016\/j.jbi.2021.103961","volume":"125","author":"C Mart\u00ednez-deMiguel","year":"2022","unstructured":"Mart\u00ednez-deMiguel C, Segura-Bedmar I, Chac\u00f3n-Solano E, Guerrero-Aspizua S (2022) The RareDis corpus: a corpus annotated with rare diseases, their signs and symptoms. J Biomed Inf 125:103961","journal-title":"J Biomed Inf"},{"key":"155_CR20","doi-asserted-by":"crossref","unstructured":"Alsentzer E, Murphy JR, Boag W, Weng W-H, Jin D, Naumann T, McDermott M (2019) Publicly available clinical BERT embeddings. arXiv:1904.03323","DOI":"10.18653\/v1\/W19-1909"},{"issue":"1","key":"155_CR21","doi-asserted-by":"publisher","first-page":"263","DOI":"10.1186\/s12859-022-04810-y","volume":"23","author":"I Segura-Bedmar","year":"2022","unstructured":"Segura-Bedmar I, Camino-Perdones D, Guerrero-Aspizua S (2022) Exploring deep learning methods for recognizing rare diseases and their clinical manifestations from texts. BMC Bioinf 23(1):263","journal-title":"BMC Bioinf"},{"issue":"e2","key":"155_CR22","doi-asserted-by":"publisher","first-page":"334","DOI":"10.1136\/amiajnl-2013-001999","volume":"20","author":"MF Davis","year":"2013","unstructured":"Davis MF, Sriram S, Bush WS, Denny JC, Haines JL (2013) Automated extraction of clinical traits of multiple sclerosis in electronic medical records. J Am Med Inf Assoc 20(e2):334\u2013340","journal-title":"J Am Med Inf Assoc"},{"key":"155_CR23","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s13023-021-01936-9","volume":"16","author":"T Lo Barco","year":"2021","unstructured":"Lo Barco T, Kuchenbuch M, Garcelon N, Neuraz A, Nabbout R (2021) Improving early diagnosis of rare diseases using natural language processing in unstructured medical records: an illustration from Dravet syndrome. Orphanet J Rare Dis 16:1\u201312","journal-title":"Orphanet J Rare Dis"},{"issue":"7","key":"155_CR24","doi-asserted-by":"publisher","first-page":"1585","DOI":"10.1038\/s41436-018-0381-1","volume":"21","author":"CA Deisseroth","year":"2019","unstructured":"Deisseroth CA, Birgmeier J, Bodle EE, Kohler JN, Matalon DR, Nazarenko Y, Genetti CA, Brownstein CA, Schmitz-Abe K, Schoch K et al (2019) Clinphen extracts and prioritizes patient phenotypes directly from medical records to expedite genetic disease diagnosis. Genet Med 21(7):1585\u20131593","journal-title":"Genet Med"},{"key":"155_CR25","doi-asserted-by":"publisher","first-page":"724","DOI":"10.1007\/s11606-014-2910-1","volume":"29","author":"SU Nigwekar","year":"2014","unstructured":"Nigwekar SU, Solid CA, Ankers E, Malhotra R, Eggert W, Turchin A, Thadhani RI, Herzog CA (2014) Quantifying a rare disease in administrative data: the example of calciphylaxis. J Gener Int Med 29:724\u2013731","journal-title":"J Gener Int Med"},{"key":"155_CR26","doi-asserted-by":"publisher","first-page":"121","DOI":"10.1016\/j.cmpb.2018.07.007","volume":"164","author":"H Fabregat","year":"2018","unstructured":"Fabregat H, Araujo L, Martinez-Romo J (2018) Deep neural models for extracting entities and relationships in the new RDD corpus relating disabilities and rare diseases. Comput Methods Prog Biomed 164:121\u2013129","journal-title":"Comput Methods Prog Biomed"},{"key":"155_CR27","unstructured":"Stenetorp P, Pyysalo S, Topi\u0107 G, Ohta T, Ananiadou S, Tsujii J (2012) BRAT: a web-based tool for NLP-assisted text annotation. In: Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics, pp 102\u2013107"},{"issue":"1","key":"155_CR28","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/sdata.2016.35","volume":"3","author":"AE Johnson","year":"2016","unstructured":"Johnson AE, Pollard TJ, Shen L, Lehman L-WH, Feng M, Ghassemi M, Moody B, Szolovits P, Anthony Celi L, Mark RG (2016) Mimic-iii, a freely accessible critical care database. Sci Data 3(1):1\u20139","journal-title":"Sci Data"},{"key":"155_CR29","unstructured":"spaCy: industrial-strength natural language processing in python. https:\/\/spacy.io"},{"key":"155_CR30","doi-asserted-by":"crossref","unstructured":"Cui L, Wu Y, Liu J, Yang S, Zhang Y (2021) Template-based named entity recognition using BART. arXiv:2106.01760","DOI":"10.18653\/v1\/2021.findings-acl.161"},{"key":"155_CR31","unstructured":"Guti\u00e9rrez BJ, McNeal N, Washington C, Chen Y, Li L, Sun H, Su Y (2022) Thinking about GPT-3 in-context learning for biomedical IE? Think again. arXiv:2203.08410"},{"key":"155_CR32","doi-asserted-by":"crossref","unstructured":"Ma R, Zhou X, Gui T, Tan Y, Li L, Zhang Q, Huang X (2021) Template-free prompt tuning for few-shot NER. arXiv:2109.13532","DOI":"10.18653\/v1\/2022.naacl-main.420"},{"key":"155_CR33","unstructured":"Taylor N, Zhang Y, Joyce D, Nevado-Holgado A, Kormilitzin A (2022) Clinical prompt learning with frozen language models. arXiv:2205.05535"}],"container-title":["Journal of Healthcare Informatics Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s41666-023-00155-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s41666-023-00155-0\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s41666-023-00155-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,4,26]],"date-time":"2024-04-26T12:11:52Z","timestamp":1714133512000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s41666-023-00155-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,1,5]]},"references-count":33,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2024,6]]}},"alternative-id":["155"],"URL":"https:\/\/doi.org\/10.1007\/s41666-023-00155-0","relation":{},"ISSN":["2509-4971","2509-498X"],"issn-type":[{"value":"2509-4971","type":"print"},{"value":"2509-498X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,1,5]]},"assertion":[{"value":"9 August 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"24 October 2023","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"13 November 2023","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"5 January 2024","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing of interest"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical approval"}}]}}