{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,6]],"date-time":"2026-03-06T14:33:07Z","timestamp":1772807587459,"version":"3.50.1"},"reference-count":51,"publisher":"Oxford University Press (OUP)","issue":"2","license":[{"start":{"date-parts":[[2026,3,6]],"date-time":"2026-03-06T00:00:00Z","timestamp":1772755200000},"content-version":"vor","delay-in-days":5,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100004807","name":"DFG","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100004807","id-type":"DOI","asserted-by":"publisher"}]},{"name":"TALENTS Marie Sk\u0142odowska-Curie COFUND-Action of the European Commission"},{"DOI":"10.13039\/501100000780","name":"European Union","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100000780","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2026,3,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Post-Acute Infection Syndromes (PAIS) are medical conditions that persist following acute infections from pathogens such as SARS-CoV-2, Epstein\u2013Barr virus, and Influenza virus. Despite growing global awareness of PAIS and the exponential increase in biomedical literature, only a small fraction of this literature pertains specifically to PAIS, making the identification of pathogen\u2013disease associations within such a vast, heterogeneous, and unstructured corpus a significant challenge for researchers. This study evaluated the effectiveness of large language models (LLMs) in extracting these associations through a binary classification task using a curated dataset of 1000 manually labeled PubMed abstracts. We benchmarked a wide range of open-source LLMs of varying sizes (4B\u201370B parameters), including generalist, reasoning, and biomedical-specific models. We also investigated the extent to which prompting strategies such as zero-shot, few-shot, and Chain of Thought (CoT) methods can improve classification performance. Our results indicate that model performance varied by size, architecture, and prompting strategy. Zero-shot prompting produced the most reliable results: Mistral-Small-Instruct-2409 and Llama-3.1-Nemotron-70B-Instruct achieved balanced accuracy scores of 0.81 and 0.80, respectively, along with macro-F1 scores of up to 0.80, while maintaining minimal invalid outputs. While few-shot and CoT prompting often degraded performance in generalist models, reasoning models such as DeepSeek-R1-Distill-Llama-70B and QwQ-32B demonstrated improved accuracy and consistency when provided with additional context.<\/jats:p>","DOI":"10.1093\/bib\/bbag089","type":"journal-article","created":{"date-parts":[[2026,2,13]],"date-time":"2026-02-13T12:44:33Z","timestamp":1770986673000},"source":"Crossref","is-referenced-by-count":0,"title":["Benchmarking large language models for pathogen\u2013disease classification in post-acute infection syndromes"],"prefix":"10.1093","volume":"27","author":[{"ORCID":"https:\/\/orcid.org\/0009-0007-6482-119X","authenticated-orcid":false,"given":"Syed","family":"Mohammed Khalid","sequence":"first","affiliation":[{"name":"Clinical Bioinformatics, Saarland University , 66123 Saarbr\u00fccken ,","place":["Germany"]}]},{"given":"Tom","family":"W\u00f6lker","sequence":"additional","affiliation":[{"name":"Clinical Bioinformatics, Saarland University , 66123 Saarbr\u00fccken ,","place":["Germany"]}]},{"given":"Leidy-Alejandra G","family":"Molano","sequence":"additional","affiliation":[{"name":"Clinical Bioinformatics, Saarland University , 66123 Saarbr\u00fccken ,","place":["Germany"]},{"name":"Helmholtz Institute for Pharmaceutical Research Saarland (HIPS)\u2013Helmholtz Centre for Infection Research (HZI), Saarland University Campus , 66123 Saarbr\u00fccken ,","place":["Germany"]}]},{"given":"Simon","family":"Graf","sequence":"additional","affiliation":[{"name":"Clinical Bioinformatics, Saarland University , 66123 Saarbr\u00fccken ,","place":["Germany"]},{"name":"Helmholtz Institute for Pharmaceutical Research Saarland (HIPS)\u2013Helmholtz Centre for Infection Research (HZI), Saarland University Campus , 66123 Saarbr\u00fccken ,","place":["Germany"]}]},{"given":"Andreas","family":"Keller","sequence":"additional","affiliation":[{"name":"Clinical Bioinformatics, Saarland University , 66123 Saarbr\u00fccken ,","place":["Germany"]},{"name":"Helmholtz Institute for Pharmaceutical Research Saarland (HIPS)\u2013Helmholtz Centre for Infection Research (HZI), Saarland University Campus , 66123 Saarbr\u00fccken ,","place":["Germany"]}]}],"member":"286","published-online":{"date-parts":[[2026,3,6]]},"reference":[{"key":"2026030605024897300_ref1","doi-asserted-by":"publisher","first-page":"911","DOI":"10.1038\/s41591-022-01810-6","article-title":"Unexplained post-acute infection syndromes","volume":"28","author":"Choutka","year":"2022","journal-title":"Nat Med"},{"key":"2026030605024897300_ref2","doi-asserted-by":"publisher","first-page":"133","DOI":"10.1038\/s41579-022-00846-2","article-title":"Long covid: major findings, mechanisms and recommendations","volume":"21","author":"Davis","year":"2023","journal-title":"Nat Rev Microbiol"},{"key":"2026030605024897300_ref3","doi-asserted-by":"publisher","first-page":"129","DOI":"10.1016\/S2213-2600(21)00031-X","article-title":"Nice guideline on long covid","volume":"9","author":"Venkatesan","year":"2021","journal-title":"Lancet Respir Med"},{"key":"2026030605024897300_ref4","doi-asserted-by":"publisher","first-page":"e21","DOI":"10.1016\/S0140-6736(23)01175-3","article-title":"6-month consequences of covid-19 in patients discharged from hospital: a cohort study","volume":"401","author":"Huang","year":"2023","journal-title":"Lancet"},{"key":"2026030605024897300_ref5","doi-asserted-by":"publisher","first-page":"296","DOI":"10.1126\/science.abj8222","article-title":"Longitudinal analysis reveals high prevalence of Epstein\u2013Barr virus associated with multiple sclerosis","volume":"375","author":"Bjornevik","year":"2022","journal-title":"Science"},{"key":"2026030605024897300_ref6","doi-asserted-by":"publisher","first-page":"806","DOI":"10.1001\/jamainternmed.2023.2228","article-title":"Comparison of medical and mental health sequelae following hospitalization for covid-19, influenza, and sepsis. JAMA","volume":"183","author":"Quinn","year":"2023","journal-title":"Intern Med"},{"key":"2026030605024897300_ref7","doi-asserted-by":"publisher","first-page":"1185","DOI":"10.1586\/14787210.2015.1079128","article-title":"Post-ebolavirus disease syndrome: what do we know","volume":"13","author":"Carod-Artal","year":"2015","journal-title":"Expert Rev Anti Infect Ther"},{"key":"2026030605024897300_ref8","doi-asserted-by":"publisher","first-page":"380","DOI":"10.1016\/j.cmi.2024.12.001","article-title":"Learning from post-covid-19 condition for epidemic preparedness: a variable catalogue for future post-acute infection syndromes","volume":"31","author":"G\u00f3rska","year":"2025","journal-title":"Clin Microbiol Infect"},{"key":"2026030605024897300_ref9","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s13326-019-0208-2","article-title":"Ontology based mining of pathogen\u2013disease associations from literature","volume":"10","author":"Kafkas","year":"2019","journal-title":"J Biomed Semant"},{"key":"2026030605024897300_ref10","doi-asserted-by":"publisher","first-page":"260","DOI":"10.3390\/fi15080260","article-title":"The power of generative ai: a review of requirements, models, input\u2013output formats, evaluation metrics, and challenges","volume":"15","author":"Bandi","year":"2023","journal-title":"Future Internet"},{"key":"2026030605024897300_ref11","doi-asserted-by":"publisher","DOI":"10.1109\/JBHI.2025.3561197","article-title":"Integrating transformer and graph attention network for circrna-mirna interaction prediction","volume":"29","author":"Wei","year":"2025","journal-title":"IEEE J Biomed Health Inform"},{"key":"2026030605024897300_ref12","doi-asserted-by":"publisher","first-page":"23","DOI":"10.1186\/s12915-025-02128-8","article-title":"Consensus representation of multiple cell\u2013cell graphs from gene signaling pathways for cell type annotation","volume":"23","author":"Huang","year":"2025","journal-title":"BMC Biol"},{"key":"2026030605024897300_ref13","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1706.03762","article-title":"Attention is all you need","volume":"30","author":"Vaswani","year":"2017","journal-title":"Adv Neural Inf Process Syst"},{"key":"2026030605024897300_ref14","doi-asserted-by":"publisher","first-page":"27730","DOI":"10.48550\/arXiv.2203.02155","article-title":"Training language models to follow instructions with human feedback","volume":"35","author":"Ouyang","year":"2022","journal-title":"Adv Neural Inf Process Syst"},{"key":"2026030605024897300_ref15","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2410.21276","article-title":"Gpt-4o system card","author":"Hurst","year":"2024"},{"key":"2026030605024897300_ref16","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2407.21783","article-title":"The llama 3 herd of models","author":"Grattafiori","year":"2024"},{"key":"2026030605024897300_ref17","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-025-09422-z","article-title":"DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning","volume-title":"Nature","author":"Guo"},{"key":"2026030605024897300_ref18","doi-asserted-by":"publisher","first-page":"915","DOI":"10.1007\/s40593-023-00372-z","article-title":"Can chatgpt pass high school exams on english language comprehension","volume":"34","author":"de Winter","year":"2024","journal-title":"International Journal of Artificial Intelligence in Education"},{"key":"2026030605024897300_ref19","doi-asserted-by":"publisher","first-page":"39","DOI":"10.1162\/tacl_a_00632","article-title":"Benchmarking large language models for news summarization","volume":"12","author":"Zhang","year":"2024","journal-title":"Trans Assoc Comput Linguist"},{"key":"2026030605024897300_ref20","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-41498-5_2","article-title":"Leveraging large language models for topic classification in the domain of public affairs","volume-title":"Document Analysis and Recognition \u2013 ICDAR 2023 Workshops","author":"Pe\u00f1a"},{"key":"2026030605024897300_ref21","doi-asserted-by":"publisher","first-page":"14918","DOI":"10.18653\/v1\/2023.emnlp-main.923","volume-title":"Proc Empir Methods Nat Lang Process (EMNLP)","author":"Sun","year":"2023"},{"key":"2026030605024897300_ref22","doi-asserted-by":"publisher","first-page":"1234","DOI":"10.1093\/bioinformatics\/btz682","article-title":"Biobert: a pre-trained biomedical language representation model for biomedical text mining","volume":"36","author":"Lee","year":"2020","journal-title":"Bioinformatics"},{"key":"2026030605024897300_ref23","doi-asserted-by":"publisher","first-page":"940","DOI":"10.1093\/jamia\/ocad256","article-title":"Evaluating the chatgpt family of models for biomedical reasoning and classification","volume":"31","author":"Chen","year":"2024","journal-title":"J Am Med Inform Assoc"},{"key":"2026030605024897300_ref24","doi-asserted-by":"publisher","first-page":"108189","DOI":"10.1016\/j.compbiomed.2024.108189","article-title":"A comprehensive evaluation of large language models on benchmark biomedical text processing tasks","volume":"171","author":"Israt Jahan","year":"2024","journal-title":"Comput Biol Med"},{"key":"2026030605024897300_ref25","doi-asserted-by":"publisher","first-page":"2024","DOI":"10.1101\/2024.05.17.24307411","article-title":"Evaluation of large language model performance on the biomedical language understanding and reasoning benchmark: comparative study","author":"Feng","year":"2024"},{"key":"2026030605024897300_ref26","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2311.16452","article-title":"Can generalist foundation models outcompete special-purpose tuning? Case study in medicine","author":"Nori","year":"2023"},{"key":"2026030605024897300_ref27","doi-asserted-by":"publisher","first-page":"1877","DOI":"10.48550\/arXiv.2005.14165","article-title":"Language models are few-shot learners","volume":"33","author":"Brown","year":"2020","journal-title":"Adv Neural Inf Process Syst"},{"key":"2026030605024897300_ref28","doi-asserted-by":"publisher","first-page":"219","DOI":"10.1186\/s13643-024-02609-x","article-title":"Evaluating the effectiveness of large language models in abstract screening: a comparative analysis","volume":"13","author":"Li","year":"2024","journal-title":"Syst Rev"},{"key":"2026030605024897300_ref29","doi-asserted-by":"publisher","first-page":"e64143","DOI":"10.2196\/64143","article-title":"Practical aspects of using large language models to screen abstracts for cardiovascular drug development: cross-sectional study","volume":"12","author":"Ronquillo","year":"2024","journal-title":"JMIR Med Inform"},{"key":"2026030605024897300_ref30","doi-asserted-by":"publisher","first-page":"1929","DOI":"10.1093\/jamia\/ocae095","article-title":"Rt: a retrieving and chain-of-thought framework for few-shot medical named entity recognition","volume":"31","author":"Li","year":"2024","journal-title":"J Am Med Inform Assoc"},{"key":"2026030605024897300_ref31","doi-asserted-by":"publisher","first-page":"106","DOI":"10.18653\/v1\/2025.insights-1.11","article-title":"LLMs are not zero-shot reasoners for biomedical information extraction","volume-title":"The Sixth Workshop on Insights from Negative Results in NLP; May 2025","author":"Nagar","year":"2025"},{"key":"2026030605024897300_ref32","doi-asserted-by":"publisher","first-page":"15012","DOI":"10.18653\/v1\/2023.emnlp-main.928","article-title":"Dr ChatGPT tell me what I want to hear: how different prompts impact health answer correctness","volume-title":"Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing","author":"Koopman","year":"2023"},{"key":"2026030605024897300_ref33","doi-asserted-by":"publisher","DOI":"10.1038\/s41597-019-0090-x","article-title":"Pathophenodb, linking human pathogens to their phenotypes in support of infectious disease research","volume":"6","author":"Kafkas","year":"2019","journal-title":"Sci Data"},{"key":"2026030605024897300_ref34","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s12866-018-1197-5","article-title":"Disbiome database: Linking the microbiome to disease","volume":"18","author":"Janssens","year":"2018","journal-title":"BMC Microbiol"},{"key":"2026030605024897300_ref35","doi-asserted-by":"publisher","first-page":"D714","DOI":"10.1093\/nar\/gkad875","article-title":"Gcpathogen: a comprehensive genomic resource of human pathogens for public health","volume":"52","author":"Guo","year":"2024","journal-title":"Nucleic Acids Res"},{"key":"2026030605024897300_ref36","doi-asserted-by":"publisher","first-page":"D1305","DOI":"10.1093\/nar\/gkad1051","article-title":"The do-kb knowledgebase: a 20-year journey developing the disease open science ecosystem","volume":"52","author":"Allen Baron","year":"2024","journal-title":"Nucleic Acids Res"},{"key":"2026030605024897300_ref37","doi-asserted-by":"publisher","first-page":"D136","DOI":"10.1093\/nar\/gkr1178","article-title":"The ncbi taxonomy database","volume":"40","author":"Federhen","year":"2011","journal-title":"Nucleic Acids Res"},{"key":"2026030605024897300_ref38","article-title":"Entrez programming utilities help","volume-title":"National Center for Biotechnology Information (US)","author":"National Center for Biotechnology Information (US)","year":"2010"},{"key":"2026030605024897300_ref40","doi-asserted-by":"publisher","first-page":"56548","DOI":"10.48550\/arXiv.2410.01257","article-title":"HelpSteer2-Preference: complementing ratings with preferences","volume-title":"Proc Int Conf Learn Represent (ICLR)","author":"Wang","year":"2025"},{"key":"2026030605024897300_ref41","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2412.15115","article-title":"Qwen2. 5 technical report","author":"Yang","year":"2024"},{"key":"2026030605024897300_ref42","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2404.14219","article-title":"Phi-3 technical report: a highly capable language model locally on your phone","author":"Abdin","year":"2024"},{"key":"2026030605024897300_ref43","author":"Mistral"},{"key":"2026030605024897300_ref44","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2412.08905","article-title":"Phi-4 technical report","author":"Abdin","year":"2024"},{"key":"2026030605024897300_ref46"},{"key":"2026030605024897300_ref47","doi-asserted-by":"publisher","first-page":"5848","DOI":"10.18653\/v1\/2024.findings-acl.348","article-title":"BioMistral: a collection of open-source pretrained large language models for medical domains","volume-title":"Findings of the Association for Computational Linguistics: ACL 2024; 2024 Aug","author":"Labrak","year":"2024"},{"key":"2026030605024897300_ref48","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2311.16079","article-title":"Meditron-70b: scaling medical pretraining for large language models","author":"Chen","year":"2023"},{"key":"2026030605024897300_ref49","doi-asserted-by":"publisher","first-page":"1833","DOI":"10.48550\/arXiv.2304.14454","article-title":"Pmc-llama: toward building open-source language models for medicine","volume":"31","author":"Chaoyi","year":"2024","journal-title":"J Am Med Inform Assoc"},{"key":"2026030605024897300_ref50","doi-asserted-by":"publisher","first-page":"2567","DOI":"10.18653\/v1\/D19-1259","article-title":"PubMedQA: a dataset for biomedical research question answering","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP); 2019 Nov","author":"Jin","year":"2019"},{"key":"2026030605024897300_ref51","first-page":"9","article-title":"Language models are unsupervised multitask learners","volume":"1","author":"Radford","year":"2019","journal-title":"OpenAI Blog"},{"key":"2026030605024897300_ref52","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2402.07927","article-title":"A systematic survey of prompt engineering in large language models: techniques and applications","author":"Sahoo","year":"2024"},{"key":"2026030605024897300_ref53","doi-asserted-by":"publisher","first-page":"2049","DOI":"10.48550\/arXiv.2307.12114","article-title":"A zero-shot and few-shot study of instruction-finetuned large language models applied to clinical and biomedical tasks","volume-title":"Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024); 2024 May","author":"Labrak","year":"2024"}],"container-title":["Briefings in Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bib\/article-pdf\/27\/2\/bbag089\/67256906\/bbag089.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bib\/article-pdf\/27\/2\/bbag089\/67256906\/bbag089.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,3,6]],"date-time":"2026-03-06T10:03:14Z","timestamp":1772791394000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bib\/article\/doi\/10.1093\/bib\/bbag089\/8509101"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,3,1]]},"references-count":51,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2026,3,1]]}},"URL":"https:\/\/doi.org\/10.1093\/bib\/bbag089","relation":{},"ISSN":["1467-5463","1477-4054"],"issn-type":[{"value":"1467-5463","type":"print"},{"value":"1477-4054","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026,3]]},"published":{"date-parts":[[2026,3,1]]},"article-number":"bbag089"}}