{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,15]],"date-time":"2026-09-15T15:35:54Z","timestamp":1789486554170,"version":"build-2803163510"},"reference-count":48,"publisher":"Oxford University Press (OUP)","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2026,1,15]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Evaluating large language models (LLMs) in the biomedical domain requires benchmarks that can distinguish genuine reasoning from pattern matching and that remain discriminative as model capabilities improve. Existing biomedical question answering (QA) benchmarks are limited in this respect: multiple-choice formats allow models to succeed by answer elimination rather than inference, and widely circulated exam-style datasets are subject to performance saturation and training data contamination. Multi-hop reasoning, i.e. the ability to integrate information across multiple sources to derive an answer, is central to clinically meaningful tasks such as diagnosis support, literature-based discovery, and hypothesis generation, yet remains underrepresented in current biomedical QA benchmarks. We present MedHopQA, a disease-centred multi-hop reasoning benchmark of 1000 expert-curated question\u2013answer pairs introduced as a shared task at BioCreative IX. Each question requires synthesis of information across two distinct Wikipedia articles, and answers are provided in open-ended free-text format rather than as multiple-choice selections. Gold annotations are augmented with ontology-grounded synonym sets (MONDO, NCBI Gene, and NCBI Taxonomy) to support both lexical and concept-level evaluation. The dataset was constructed through a multi-stage human\u2013AI pipeline combining structured human annotation, triage, iterative verification, and LLM-as-a-judge validation. To reduce leaderboard gaming and contamination risk, the 1000 scored questions are embedded within a publicly downloadable set of 10\u2009000 questions, with answers withheld, on a CodaBench leaderboard. Evaluation of four frontier LLMs under a zero-shot setting (GPT-5.1, Gemini 2.5 Pro, Claude Sonnet 4.5, and GPT-4o) reveals performance variation across answer types, with overall accuracy ranging from 66.3% to 83.4%. Performance is strongest on chemical and anatomical questions and most variable on disease and gene\/protein categories, where fine-grained semantic discrimination is required. MedHopQA provides both a benchmark and a reusable framework for constructing future biomedical QA datasets that prioritize compositional reasoning, saturation resistance, and contamination mitigation as design constraints.<\/jats:p>","DOI":"10.1093\/database\/baag057","type":"journal-article","created":{"date-parts":[[2026,9,4]],"date-time":"2026-09-04T11:30:23Z","timestamp":1788521423000},"source":"Crossref","is-referenced-by-count":0,"title":["MedHopQA: a disease-centred multi-hop reasoning benchmark and evaluation framework for LLM-based biomedical question answering"],"prefix":"10.1093","volume":"2026","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5651-1860","authenticated-orcid":false,"given":"Rezarta","family":"Islamaj","sequence":"first","affiliation":[{"name":"Division of Intramural Research, National Library of Medicine, 8600 Rockville Pike , Bethesda, MD 20894 ,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3296-5766","authenticated-orcid":false,"given":"Robert","family":"Leaman","sequence":"additional","affiliation":[{"name":"Division of Intramural Research, National Library of Medicine, 8600 Rockville Pike , Bethesda, MD 20894 ,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Joey","family":"Chan","sequence":"additional","affiliation":[{"name":"University of Illinois at Urbana-Champaign, 201 North Goodwin Avenue Department of Computer Science, , Urbana, IL 61801 ,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nicholas","family":"Wan","sequence":"additional","affiliation":[{"name":"University of Michigan Medical School, 1301 Catherine St , Ann Arbor, MI 48109 ,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1268-7239","authenticated-orcid":false,"given":"Qiao","family":"Jin","sequence":"additional","affiliation":[{"name":"Division of Intramural Research, National Library of Medicine, 8600 Rockville Pike , Bethesda, MD 20894 ,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Natalie","family":"Xie","sequence":"additional","affiliation":[{"name":"Division of Intramural Research, National Library of Medicine, 8600 Rockville Pike , Bethesda, MD 20894 ,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"John","family":"Wilbur","sequence":"additional","affiliation":[{"name":"Division of Intramural Research, National Library of Medicine, 8600 Rockville Pike , Bethesda, MD 20894 ,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6415-1439","authenticated-orcid":false,"given":"Shubo","family":"Tian","sequence":"additional","affiliation":[{"name":"Division of Intramural Research, National Library of Medicine, 8600 Rockville Pike , Bethesda, MD 20894 ,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lana","family":"Yeganova","sequence":"additional","affiliation":[{"name":"Division of Intramural Research, National Library of Medicine, 8600 Rockville Pike , Bethesda, MD 20894 ,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Po-Ting","family":"Lai","sequence":"additional","affiliation":[{"name":"Division of Intramural Research, National Library of Medicine, 8600 Rockville Pike , Bethesda, MD 20894 ,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5094-7321","authenticated-orcid":false,"given":"Chih-Hsuan","family":"Wei","sequence":"additional","affiliation":[{"name":"Division of Intramural Research, National Library of Medicine, 8600 Rockville Pike , Bethesda, MD 20894 ,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yifan","family":"Yang","sequence":"additional","affiliation":[{"name":"Division of Intramural Research, National Library of Medicine, 8600 Rockville Pike , Bethesda, MD 20894 ,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yao","family":"Ge","sequence":"additional","affiliation":[{"name":"Division of Intramural Research, National Library of Medicine, 8600 Rockville Pike , Bethesda, MD 20894 ,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Qingqing","family":"Zhu","sequence":"additional","affiliation":[{"name":"Division of Intramural Research, National Library of Medicine, 8600 Rockville Pike , Bethesda, MD 20894 ,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhizheng","family":"Wang","sequence":"additional","affiliation":[{"name":"Division of Intramural Research, National Library of Medicine, 8600 Rockville Pike , Bethesda, MD 20894 ,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9998-916X","authenticated-orcid":false,"given":"Zhiyong","family":"Lu","sequence":"additional","affiliation":[{"name":"Division of Intramural Research, National Library of Medicine, 8600 Rockville Pike , Bethesda, MD 20894 ,","place":["United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2026,9,15]]},"reference":[{"key":"2026091509081456100_bib1","doi-asserted-by":"publisher","first-page":"91","DOI":"10.1038\/s41597-021-00875-1","article-title":"NLM-Chem, a new resource for chemical entity recognition in PubMed full text literature","volume-title":"Sci Data","author":"Islamaj","year":"2021"},{"key":"2026091509081456100_bib2","doi-asserted-by":"crossref","first-page":"baae071","DOI":"10.1093\/database\/baae071","article-title":"The biomedical relationship corpus of the BioRED track at the BioCreative VIII challenge and workshop","volume":"2024","author":"Islamaj","year":"2024","journal-title":"Database (Oxford)"},{"key":"2026091509081456100_bib3","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.jbi.2013.12.006","article-title":"NCBI disease corpus: a resource for disease name recognition and concept normalization","volume":"47","author":"Dogan","year":"2014","journal-title":"J Biomed Inform"},{"key":"2026091509081456100_bib4","doi-asserted-by":"publisher","first-page":"S1","DOI":"10.1186\/1471-2105-6-S1-S1","article-title":"Overview of BioCreAtIvE: critical assessment of information extraction for biology","volume":"6","author":"Hirschman","year":"2005","journal-title":"BMC Bioinf"},{"key":"2026091509081456100_bib5","doi-asserted-by":"publisher","first-page":"S1","DOI":"10.1186\/gb-2008-9-s2-s1","article-title":"Evaluation of text-mining systems for biology: overview of the Second BioCreative community challenge","volume":"9","author":"Krallinger","year":"2008","journal-title":"Genome Biol"},{"key":"2026091509081456100_bib6","doi-asserted-by":"publisher","first-page":"S3","DOI":"10.1186\/1471-2105-12-S8-S3","article-title":"The protein-protein interaction tasks of BioCreative III: classification\/ranking of articles and linking bio-ontology concepts to full text","volume":"12","author":"Krallinger","year":"2011","journal-title":"BMC Bioinf"},{"key":"2026091509081456100_bib7","first-page":"1","volume-title":"Overview of BioNLP\u201909 Shared Task on Event Extraction.","author":"Kim"},{"key":"2026091509081456100_bib8","first-page":"173","article-title":"Annotating chemicals, diseases, and their interactions in biomedical literature","volume-title":"Proceedings of the fifth BioCreative Challenge Evaluation Workshop","author":"Li","year":"2015"},{"key":"2026091509081456100_bib9","doi-asserted-by":"publisher","article-title":"On the opportunities and risks of foundation models","author":"Bommasani","DOI":"10.48550\/arXiv.2108.07258"},{"key":"2026091509081456100_bib10","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","year":"2023","journal-title":"Nature"},{"key":"2026091509081456100_bib11","doi-asserted-by":"publisher","first-page":"1930","DOI":"10.1038\/s41591-023-02448-8","article-title":"Large language models in medicine","volume":"29","author":"Thirunavukarasu","year":"2023","journal-title":"Nat Med"},{"key":"2026091509081456100_bib12","doi-asserted-by":"crossref","unstructured":"Jin D, Pan E, Oufattole N \u00a0et al. \u00a0What disease does this patient have? A large-scale open domain question answering dataset from medical exams. Appl Sci. 2021;11:6421. 10.3390\/app11146421","DOI":"10.3390\/app11146421"},{"key":"2026091509081456100_bib13","first-page":"248","article-title":"MedMCQA: a large-scale multi-subject multi-choice dataset for medical domain question answering","volume-title":"Proceedings of the Conference on Health, Inference, and Learning; Proceedings of Machine Learning Research","author":"Pal","year":"2022"},{"key":"2026091509081456100_bib14","first-page":"2019, 2567","volume-title":"PubMedQA: A Dataset for Biomedical Research Question Answering.","author":"Jin"},{"key":"2026091509081456100_bib15","article-title":"Measuring Massive Multitask Language Understanding","volume-title":"International Conference on Learning Representations","author":"Hendrycks","year":"2021"},{"key":"2026091509081456100_bib16","author":"MedQA: Evaluating language model bias in medical questions"},{"key":"2026091509081456100_bib17"},{"key":"2026091509081456100_bib18","article-title":"Holistic evaluation of language models","author":"Liang","year":"2023","journal-title":"Trans Mach Learn Res"},{"key":"2026091509081456100_bib19","doi-asserted-by":"publisher","article-title":"Llms outperform experts on challenging biology benchmarks","author":"Justen","DOI":"10.48550\/arXiv.2505.06108"},{"key":"2026091509081456100_bib20","article-title":"Time Travel in LLMs: Tracing Data Contamination in Large Language Models","volume-title":"The Twelfth International Conference on Learning Representations (ICLR)","author":"Golchin","year":"2024."},{"key":"2026091509081456100_bib21","first-page":"2023, 10776","article-title":"NLP Evaluation in trouble: on the Need to Measure LLM Data Contamination for each Benchmark","author":"Sainz"},{"key":"2026091509081456100_bib22","unstructured":"Islamaj R, Chan J, Leaman R \u00a0et al. \u00a0Overview of the MedHopQA track at BioCreative IX: track description, participation and evaluation of systems for multi-hop medical question answering. In: Proceedings of the BioCreative IX Challenge and Workshop (BC9): Large Language Models for Clinical and Biomedical NLP at the International Joint Conference on Artificial Intelligence (IJCAI): 2025; Montreal, Canada. zenodo.org"},{"key":"2026091509081456100_bib23","doi-asserted-by":"crossref","DOI":"10.1093\/database\/baag046","article-title":"Overview of the MedHopQA track at BioCreative IX: track description, participation and evaluation of systems for multi-hop medical question answering","author":"Islamaj","year":"2026","journal-title":"Database (Oxford)"},{"key":"2026091509081456100_bib24","doi-asserted-by":"publisher","first-page":"138","DOI":"10.1186\/s12859-015-0564-6","article-title":"An overview of the BIOASQ large-scale biomedical semantic indexing and question answering competition","volume":"16","author":"Tsatsaronis","year":"2015","journal-title":"BMC Bioinf"},{"key":"2026091509081456100_bib25","doi-asserted-by":"crossref","unstructured":"Pappas D, Androutsopoulos I, Papageorgiou H. BioRead: A New Dataset for Biomedical Reading Comprehension. In: Miyazaki, Japan: European Language Resources Association (ELRA), 2018.","DOI":"10.63317\/2k6jps9hh3ui"},{"key":"2026091509081456100_bib26","doi-asserted-by":"crossref","volume-title":"Overview of the MEDIQA 2021 Shared Task on Summarization in the Medical Domain","author":"Ben\u00a0Abacha","DOI":"10.18653\/v1\/2021.bionlp-1.8"},{"key":"2026091509081456100_bib27","doi-asserted-by":"crossref","volume-title":"Overview of the MEDIQA 2019 Shared Task on Textual Inference, Question Entailment and Question Answering","author":"Ben\u00a0Abacha","DOI":"10.18653\/v1\/W19-5039"},{"key":"2026091509081456100_bib28","doi-asserted-by":"publisher","first-page":"2357","DOI":"10.18653\/v1\/D18-1258","article-title":"emrqa: a large corpus for question answering on electronic medical records","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Pampari","year":"2018"},{"key":"2026091509081456100_bib29","first-page":"1586","volume-title":"Lessons from Natural Language Inference in the Clinical Domain","author":"Romanov","year":"2018"},{"key":"2026091509081456100_bib30","volume-title":"COVID-QA: A Question Answering Dataset for COVID-19","author":"M\u00f6ller","year":"2020"},{"key":"2026091509081456100_bib31","doi-asserted-by":"crossref","volume-title":"Question Answering with Long Multiple-Span Answers","author":"Zhu","DOI":"10.18653\/v1\/2020.findings-emnlp.342"},{"key":"2026091509081456100_bib32","volume-title":"Overview of the Medical Question Answering Task at TREC 2017 LiveQA","author":"Abacha"},{"key":"2026091509081456100_bib33","doi-asserted-by":"crossref","unstructured":"Soni S, Gudala M, Pajouhi A \u00a0et al. \u00a0RadQA: A Question Answering Dataset to Improve Comprehension of Radiology Reports. In Marseille, France: European Language Resources Association, 2022, 6250\u201359.","DOI":"10.63317\/2rkacs78gax6"},{"key":"2026091509081456100_bib34","volume-title":"RealMedQA: A Pilot Biomedical Question Answering Dataset Containing Realistic Clinical Questions","author":"Kell"},{"key":"2026091509081456100_bib35","doi-asserted-by":"crossref","unstructured":"Welbl J, Stenetorp P, Riedel S. Constructing datasets for multi-hop reading comprehension across documents. Trans Assoc Comput Linguist. 2018;6:287\u2013302. 10.1162\/tacl_a_00021","DOI":"10.1162\/tacl_a_00021"},{"key":"2026091509081456100_bib36","doi-asserted-by":"publisher","first-page":"20552076251390447","DOI":"10.1177\/20552076251390447","article-title":"HealthBench: advancing AI evaluation in healthcare, but not yet clinically ready","volume":"11","author":"Liu","year":"2025","journal-title":"Digit Health"},{"key":"2026091509081456100_bib37","first-page":"277","volume-title":"K-QA: A Real-World Medical Q&A Benchmark","author":"Manes"},{"key":"2026091509081456100_bib38","volume-title":"MedExQA: Medical Question Answering Benchmark with Multiple Explanations","author":"Kim"},{"key":"2026091509081456100_bib39","first-page":"14459","volume-title":"MedREQAL: Examining Medical Knowledge Recall of Large Language Models via Question Answering","author":"Vladika"},{"key":"2026091509081456100_bib40","doi-asserted-by":"crossref","unstructured":"Nimo C, Olatunji T, Owodunni AT \u00a0et al. \u00a0AfriMed-QA: A Pan-African, Multi-Specialty, Medical Question-Answering Benchmark Dataset. Vienna, Austria: Association for Computational Linguistics, 2025, 1948\u201373.","DOI":"10.18653\/v1\/2025.acl-long.96"},{"key":"2026091509081456100_bib41","doi-asserted-by":"publisher","first-page":"280","DOI":"10.1007\/s41666-025-00204-w","article-title":"LongHealth: a question answering benchmark with long clinical documents","volume":"9","author":"Adams","year":"2025","journal-title":"J Healthc Inform Res"},{"key":"2026091509081456100_bib42","first-page":"378","volume-title":"Overview of the ClinIQLink 2025 Shared Task on Medical Question-Answering","author":"Colelough"},{"key":"2026091509081456100_bib43","first-page":"12894","volume-title":"BioHopR: A Benchmark for Multi-Hop, Multi-Answer Reasoning in Biomedical Domain","author":"Kim"},{"key":"2026091509081456100_bib44","doi-asserted-by":"crossref","first-page":"258","DOI":"10.1038\/s41746-026-02465-0","article-title":"A large-scale benchmark for evaluating large language models on medical question answering in Romanian","volume":"9","author":"Rogoz","year":"2026","journal-title":"NPJ Digit Med"},{"key":"2026091509081456100_bib45","doi-asserted-by":"crossref","unstructured":"Yang Z, Qi P, Zhang S \u00a0et al. \u00a0HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. Brussels, Belgium: Association for Computational Linguistics, 2018, 2369\u201380.","DOI":"10.18653\/v1\/D18-1259"},{"key":"2026091509081456100_bib46","doi-asserted-by":"publisher","first-page":"539","DOI":"10.1162\/tacl_a_00475","article-title":"MuSiQue: multihop Questions via Single-hop Question Composition","volume":"10","author":"Trivedi","year":"2022","journal-title":"Trans Assoc Comput Linguist"},{"key":"2026091509081456100_bib47","doi-asserted-by":"crossref","unstructured":"Ho X, Duong\u00a0Nguyen A-K, Sugawara S \u00a0et al. \u00a0Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps. Barcelona, Spain: International Committee on Computational Linguistics, 2020, 6609\u201325.","DOI":"10.18653\/v1\/2020.coling-main.580"},{"key":"2026091509081456100_bib48","doi-asserted-by":"publisher","DOI":"10.1093\/bioinformatics\/btad651","article-title":"MedCPT: contrastive Pre-trained Transformers with large-scale PubMed search logs for zero-shot biomedical information retrieval","volume":"39","author":"Jin","year":"2023","journal-title":"Bioinformatics"}],"container-title":["Database"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/database\/article-pdf\/doi\/10.1093\/database\/baag057\/71089003\/baag057.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/database\/article-pdf\/doi\/10.1093\/database\/baag057\/71089003\/baag057.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,9,15]],"date-time":"2026-09-15T13:08:28Z","timestamp":1789477708000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/database\/article\/doi\/10.1093\/database\/baag057\/8796013"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026]]},"references-count":48,"URL":"https:\/\/doi.org\/10.1093\/database\/baag057","relation":{},"ISSN":["1758-0463"],"issn-type":[{"value":"1758-0463","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026]]},"published":{"date-parts":[[2026]]},"article-number":"baag057"}}