{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T05:25:37Z","timestamp":1787030737076,"version":"build-2736575974"},"reference-count":23,"publisher":"Wiley","issue":"4","license":[{"start":{"date-parts":[[2026,8,17]],"date-time":"2026-08-17T00:00:00Z","timestamp":1786924800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2026,8,17]],"date-time":"2026-08-17T00:00:00Z","timestamp":1786924800000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/doi.wiley.com\/10.1002\/tdm_license_1.1"}],"content-domain":{"domain":["onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Quant. Biol."],"published-print":{"date-parts":[[2026,12]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    Manual curation of gene\u2013disease\u2013phenotype relationships from the human genetics literature is a persistent bottleneck for maintaining its bioinformatics databases. Whereas large language models (LLMs) offer a promising alternative, there is currently no systematic benchmark that evaluates whether state\u2010of\u2010the\u2010art commercial LLMs can perform this task reliably on the full\u2010text articles. To address this gap, we introduce a standardized benchmark comprising 406 full\u2010text articles covering 180 congenital heart disease\u2010associated genes, and a multi\u2010dimensional evaluation framework\u00a0that incorporates fuzzy matching to account for synonyms and partial\u00a0matches. We benchmarked seven state\u2010of\u2010the\u2010art LLMs, GPT\u20104o, Claude\u2010Opus\u20104, DeepSeek\u2010R1, Grok\u20104, Qwen\u20103.5, Gemini\u20102.5 (Pro), and GPT\u20105 on the extraction of structured gene, disease, and phenotype fields. The top\u2010performing model, Grok\u20104, achieved 97.6% overall accuracy, whereas the lowest\u2010performing model reached approximately 88%, still surpassing many prior benchmarks employing zero\u2010shot or\n                    <jats:italic>n<\/jats:italic>\n                    \u2010shot prompting in biomedical relation extraction (RE) tasks. Our results provide a rigorous characterization of current LLMs capabilities and limitations. This paper contains two components. First, we conducted a human genetics field benchmark study on LLMs against a curated database. Second we developed the evaluation framework for this task. The benchmark dataset, evaluation framework, and model benchmarking outputs are made available online to support future studies in a reproducible manner.\n                  <\/jats:p>","DOI":"10.1002\/qub2.70050","type":"journal-article","created":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T04:45:37Z","timestamp":1787028337000},"update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Benchmarking commercial large language models for gene\u2013disease\u2013phenotype extraction from full\u2010text human genetics literature"],"prefix":"10.1002","volume":"14","author":[{"given":"Danqing","family":"Yin","sequence":"first","affiliation":[{"name":"School of Biomedical Sciences Li Ka Shing Faculty of Medicine The University of Hong Kong  Hong Kong China"},{"name":"Laboratory of Data Discovery for Health Hong Kong Science Park  Hong Kong China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Matthew Ka Siu","family":"Leung","sequence":"additional","affiliation":[{"name":"ESF South Island School  Hong Kong China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Darren Wan Ho","family":"Pun","sequence":"additional","affiliation":[{"name":"ESF South Island School  Hong Kong China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fiona Haixin","family":"Chen","sequence":"additional","affiliation":[{"name":"ESF South Island School  Hong Kong China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Julie Yujin","family":"Kwon","sequence":"additional","affiliation":[{"name":"ESF South Island School  Hong Kong China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xinyi","family":"Lin","sequence":"additional","affiliation":[{"name":"School of Biomedical Sciences Li Ka Shing Faculty of Medicine The University of Hong Kong  Hong Kong China"},{"name":"Laboratory of Data Discovery for Health Hong Kong Science Park  Hong Kong China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Joshua W. K.","family":"Ho","sequence":"additional","affiliation":[{"name":"School of Biomedical Sciences Li Ka Shing Faculty of Medicine The University of Hong Kong  Hong Kong China"},{"name":"Laboratory of Data Discovery for Health Hong Kong Science Park  Hong Kong China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"311","published-online":{"date-parts":[[2026,8,17]]},"reference":[{"issue":"1","key":"e_1_2_11_2_1","doi-asserted-by":"crossref","DOI":"10.1038\/s41467-025-56989-2","article-title":"Benchmarking large language models for biomedical natural language processing applications and recommendations","volume":"16","author":"Chen Q","year":"2025","journal-title":"Nat Commun"},{"issue":"9","key":"e_1_2_11_3_1","doi-asserted-by":"crossref","DOI":"10.1093\/bioinformatics\/btad557","article-title":"An extensive benchmark study on biomedical text generation and mining with ChatGPT","volume":"39","author":"Chen Q","year":"2023","journal-title":"Bioinformatics"},{"key":"e_1_2_11_4_1","doi-asserted-by":"crossref","unstructured":"JahanI LaskarMTR PengC HuangJ.A comprehensive evaluation of large language models on benchmark biomedical text processing tasks.2024. Preprint at arXiv:2310.04270.","DOI":"10.1016\/j.compbiomed.2024.108189"},{"issue":"1","key":"e_1_2_11_5_1","article-title":"A large language model framework for literature\u2010based disease\u2013gene association prediction","volume":"26","author":"Li P\u2010H","year":"2025","journal-title":"Briefings Bioinf"},{"key":"e_1_2_11_6_1","doi-asserted-by":"crossref","DOI":"10.1093\/database\/baaf011","article-title":"Assessing the performance of generative artificial intelligence in retrieving information against manually curated genetic and genomic data","volume":"2025","author":"Poretsky E","year":"2025","journal-title":"Database"},{"key":"e_1_2_11_7_1","unstructured":"ChangJ WangS LingC QinZ ZhaoL.Gene\u2010associated disease discovery powered by large language models.2024. Preprint at arXiv:2401.09490."},{"issue":"1","key":"e_1_2_11_8_1","doi-asserted-by":"crossref","DOI":"10.1038\/s41746-025-01686-z","article-title":"Iterative refinement and goal articulation to optimize large language models for clinical information extraction","volume":"8","author":"Hein D","year":"2025","journal-title":"npj Digit Med"},{"issue":"4","key":"e_1_2_11_9_1","doi-asserted-by":"crossref","first-page":"1234","DOI":"10.1093\/bioinformatics\/btz682","article-title":"BioBERT: a pre\u2010trained biomedical language representation model for biomedical text mining","volume":"36","author":"Lee J","year":"2020","journal-title":"Bioinformatics"},{"issue":"12","key":"e_1_2_11_10_1","doi-asserted-by":"crossref","first-page":"829","DOI":"10.1038\/nrg3337","article-title":"Text\u2010mining solutions for biomedical research: enabling integrative biology","volume":"13","author":"Rebholz\u2010Schuhmann D","year":"2012","journal-title":"Nat Rev Genet"},{"issue":"1","key":"e_1_2_11_11_1","first-page":"1883","article-title":"Large language model in healthcare for the prediction of genetic variants from unstructured text medicine data using natural language processing","volume":"84","author":"Ayesha N","year":"2025","journal-title":"Comput Mater Contin"},{"key":"e_1_2_11_12_1","doi-asserted-by":"crossref","first-page":"58","DOI":"10.18653\/v1\/2024.futured-1.6","volume-title":"Proceedings of the workshop on the future of event detection (FuturED)","author":"Zheng H","year":"2024"},{"key":"e_1_2_11_13_1","doi-asserted-by":"crossref","first-page":"50","DOI":"10.1016\/B978-0-323-95502-7.00017-8","volume-title":"Encyclopedia of bioinformatics and computational biology","author":"Lamurias A","year":"2025"},{"key":"e_1_2_11_14_1","volume-title":"Proceedings of the 2024 ALTA Workshop","author":"Gill R","year":"2024"},{"key":"e_1_2_11_15_1","article-title":"Grounded LLMs for drug\u2013gene\u2013cancer triad extraction using literature snippets","volume":"16","author":"Lai A","year":"2025","journal-title":"Nat Commun"},{"issue":"1","key":"e_1_2_11_16_1","doi-asserted-by":"crossref","DOI":"10.1038\/s41598-025-91809-z","article-title":"An NLP\u2010based method to mine gene and function relationships from published articles","volume":"15","author":"Kumar N","year":"2025","journal-title":"Sci Rep"},{"issue":"1","key":"e_1_2_11_17_1","doi-asserted-by":"crossref","DOI":"10.1186\/s12911-024-02439-w","article-title":"An evaluation of GPT models for phenotype concept recognition","volume":"24","author":"Groza T","year":"2024","journal-title":"BMC Med Inf Decis Making"},{"key":"e_1_2_11_18_1","doi-asserted-by":"crossref","unstructured":"NeeleyM QiG WangG TangR MaoD LiuC et\u00a0al.Survey and improvement strategies for gene prioritization with large language models.2025. Preprint at arXiv:2501.18794.","DOI":"10.1093\/bioadv\/vbaf148"},{"issue":"10","key":"e_1_2_11_19_1","doi-asserted-by":"crossref","first-page":"2190","DOI":"10.1016\/j.ajhg.2024.08.010","article-title":"Assessing the utility of large language models for phenotype\u2010driven gene prioritization in the diagnosis of rare genetic disease","volume":"111","author":"Kim J","year":"2024","journal-title":"Am J Hum Genet"},{"issue":"5","key":"e_1_2_11_20_1","article-title":"Evaluating hallucination in multilingual biomedical LLMs","volume":"19","author":"Kodikara N","year":"2024","journal-title":"PLoS One"},{"issue":"3","key":"e_1_2_11_21_1","article-title":"CHDgene: a curated database for congenital heart disease genes","volume":"15","author":"Yang A","year":"2022","journal-title":"Circulation: Genomic and Precision Medicine"},{"issue":"1","key":"e_1_2_11_22_1","doi-asserted-by":"crossref","DOI":"10.1186\/s13073-025-01521-w","article-title":"Improving automated deep phenotyping through large language models using retrieval\u2010augmented generation","volume":"17","author":"Garcia BT","year":"2025","journal-title":"Genome Med"},{"issue":"1","key":"e_1_2_11_23_1","doi-asserted-by":"crossref","DOI":"10.1038\/s41598-025-99539-y","article-title":"The application of Large Language Models to the phenotype\u2010based prioritization of causative genes in rare disease patients","volume":"15","author":"Kafkas \u015e","year":"2025","journal-title":"Sci Rep"},{"issue":"8017","key":"e_1_2_11_24_1","doi-asserted-by":"crossref","first-page":"625","DOI":"10.1038\/s41586-024-07421-0","article-title":"Detecting hallucinations in large language models using semantic entropy","volume":"630","author":"Farquhar S","year":"2024","journal-title":"Nature"}],"container-title":["Quantitative Biology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/qub2.70050","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/full-xml\/10.1002\/qub2.70050","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/qub2.70050","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T04:45:48Z","timestamp":1787028348000},"score":1,"resource":{"primary":{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/10.1002\/qub2.70050"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,17]]},"references-count":23,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2026,12]]}},"alternative-id":["10.1002\/qub2.70050"],"URL":"https:\/\/doi.org\/10.1002\/qub2.70050","archive":["Portico"],"relation":{},"ISSN":["2095-4689","2095-4697"],"issn-type":[{"value":"2095-4689","type":"print"},{"value":"2095-4697","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,17]]},"assertion":[{"value":"2026-01-19","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-05-29","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-08-17","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"e70050"}}