{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,24]],"date-time":"2026-06-24T17:52:57Z","timestamp":1782323577158,"version":"3.54.5"},"reference-count":51,"publisher":"Public Library of Science (PLoS)","issue":"6","license":[{"start":{"date-parts":[[2026,6,24]],"date-time":"2026-06-24T00:00:00Z","timestamp":1782259200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000009","name":"Foundation for the National Institutes of Health","doi-asserted-by":"publisher","award":["2R24AI13661806"],"award-info":[{"award-number":["2R24AI13661806"]}],"id":[{"id":"10.13039\/100000009","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.plosone.org"],"crossmark-restriction":false},"short-container-title":["PLoS One"],"abstract":"<jats:sec id=\"sec001\">\n                    <jats:title>Objectives<\/jats:title>\n                    <jats:p>The ability of large language models (LLMs) to answer targeted scientific questions by synthesizing information from research articles remains an open research challenge.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"sec002\">\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We evaluated the effects of fine-tuning and a question-specific prompting strategy to answer 16 pre-defined questions about HIV drug resistance studies, including whether viral genetic sequences were reported and the demographics and antiviral treatments of the individuals from whom sequences were obtained. For fine-tuning, we constructed an instruction set comprising 250 HIV drug resistance studies, with 16 questions per study and corresponding answers and explanations. For question-specific prompting, we developed a set of if-then rules tailored to each question. We compared the performance of three base models \u2013 GPT-4o-mini-2024-07-18 (GPT-4o), Meta Llama-3.1-70B-Instruct (Llama-3.1-70B), and Meta Llama-3.1-8B-Instruct (Llama-3.1-8B) \u2013 with their performance using fine-tuning, question specific prompting, and fine-tuning followed by question-specific prompting. Performance was assessed using accuracy, precision, recall, and F1 score, averaged over 150 held-out studies not used for fine-tuning. Comparisons were performed using Wilcoxon signed-rank tests.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"sec003\">\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Fine-tuning increased precision by 5% for GPT-4o, 16% for Llama-3.1-70B, and 8% for Llama-3.1-8B, although this increase reached statistical significance only for Llama-3.1-70B. Fine-tuning also significantly increased recall for GPT-4o by 11%. Question specific prompting increased recall for all three models (6% for GPT-4o, 7% for Llama-3.1-70B, and 18% for Llama-3.1-8B), with statistically significant improvements observed only for Llama-3.1-8B. Applying question specific prompting to each of the fine-tuned models did not yield additional improvements beyond fine-tuning alone. When pooled across the three models, fine-tuning was associated with a greater effect on precision than recall (OR = 4.35; p\u2009=\u20090.001; Fisher\u2019s exact test), whereas question-specific prompting led to a greater effect on recall than on precision (OR= 7.09; p\u2009=\u20090.0001; Fisher\u2019s exact test).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"sec004\">\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>In this domain-focused proof-of-concept study, fine-tuning and question-specific prompting each led to improvement in one or more metrics for each of the three models. Pooled analyses indicated that fine-tuning improved precision, whereas question specific prompting preferentially improved recall.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1371\/journal.pone.0351631","type":"journal-article","created":{"date-parts":[[2026,6,24]],"date-time":"2026-06-24T17:40:42Z","timestamp":1782322842000},"page":"e0351631","update-policy":"https:\/\/doi.org\/10.1371\/journal.pone.corrections_policy","source":"Crossref","is-referenced-by-count":0,"title":["Fine-tuning and structured prompting strategies for question answering over full-text biomedical research articles"],"prefix":"10.1371","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8731-1271","authenticated-orcid":true,"given":"Kaiming","family":"Tao","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rohit","family":"Satija","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jinru","family":"Zhou","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zachary A.","family":"Osman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vineet","family":"Ahluwalia","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chiara","family":"Sabatti","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2513-2643","authenticated-orcid":true,"given":"Robert W.","family":"Shafer","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2026,6,24]]},"reference":[{"key":"pone.0351631.ref001","doi-asserted-by":"crossref","first-page":"125","DOI":"10.1038\/s42256-020-00287-7","article-title":"An open source machine learning framework for efficient and transparent systematic reviews","volume":"3","author":"R van de Schoot","year":"2021","journal-title":"Nat Mach Intell"},{"issue":"1","key":"pone.0351631.ref002","doi-asserted-by":"crossref","first-page":"322","DOI":"10.1186\/s12874-022-01805-4","article-title":"Machine learning computational tools to assist the performance of systematic reviews: A mapping review","volume":"22","author":"R Cierco Jimenez","year":"2022","journal-title":"BMC Med Res Methodol"},{"issue":"3","key":"pone.0351631.ref003","doi-asserted-by":"crossref","first-page":"353","DOI":"10.1002\/jrsm.1553","article-title":"Using artificial intelligence methods for systematic review in health sciences: A systematic review","volume":"13","author":"A Blaizot","year":"2022","journal-title":"Res Synth Methods"},{"issue":"7","key":"pone.0351631.ref004","article-title":"Artificial intelligence in systematic reviews: promising when appropriately used","volume":"13","author":"SHB van Dijk","year":"2023","journal-title":"BMJ Open"},{"issue":"11","key":"pone.0351631.ref005","article-title":"MedCPT: Contrastive Pre-trained Transformers with large-scale PubMed search logs for zero-shot biomedical information retrieval","volume":"39","author":"Q Jin","year":"2023","journal-title":"Bioinformatics"},{"key":"pone.0351631.ref006","doi-asserted-by":"crossref","first-page":"104389","DOI":"10.1016\/j.jbi.2023.104389","article-title":"The use of artificial intelligence for automating or semi-automating biomedical literature analyses: A scoping review","volume":"142","author":"\u00c1OD Santos","year":"2023","journal-title":"J Biomed Inform"},{"key":"pone.0351631.ref007","doi-asserted-by":"crossref","first-page":"156","DOI":"10.1002\/jrsm.1589","article-title":"In-depth evaluation of machine learning methods for semi-automating article screening in a systematic review of mechanistic literature","volume":"14","author":"MM Kebede","year":"2023","journal-title":"Res Synth Methods"},{"issue":"7","key":"pone.0351631.ref008","doi-asserted-by":"crossref","first-page":"351","DOI":"10.3390\/systems11070351","article-title":"Harnessing the Power of ChatGPT for Automating Systematic Review Process: Methodology, Case Study, Limitations, and Future Directions","volume":"11","author":"A Alshami","year":"2023","journal-title":"Systems"},{"key":"pone.0351631.ref009","doi-asserted-by":"crossref","DOI":"10.2196\/48933","article-title":"Applications of the Natural Language Processing Tool ChatGPT in Clinical Practice: Comparative Study and Augmented Systematic Review","volume":"11","author":"N Schopow","year":"2023","journal-title":"JMIR Med Inform"},{"key":"pone.0351631.ref010","article-title":"Assessing the Ability of ChatGPT to Screen Articles for Systematic Reviews","author":"E Syriani","year":"2023","journal-title":"arXiv"},{"issue":"4","key":"pone.0351631.ref011","doi-asserted-by":"crossref","first-page":"616","DOI":"10.1002\/jrsm.1715","article-title":"Can large language models replace humans in systematic reviews? Evaluating GPT-4\u2019s efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languages","volume":"15","author":"Q Khraisha","year":"2024","journal-title":"Res Synth Methods"},{"key":"pone.0351631.ref012","article-title":"Automated Paper Screening for Clinical Reviews Using Large Language Models: Data Analysis Study","volume":"26","author":"E Guo","year":"2024","journal-title":"J Med Internet Res"},{"issue":"1","key":"pone.0351631.ref013","doi-asserted-by":"crossref","first-page":"1569","DOI":"10.1038\/s41467-024-45914-8","article-title":"Extracting accurate materials data from research papers with conversational language models and prompt engineering","volume":"15","author":"MP Polak","year":"2024","journal-title":"Nat Commun"},{"issue":"1","key":"pone.0351631.ref014","doi-asserted-by":"crossref","first-page":"158","DOI":"10.1186\/s13643-024-02575-4","article-title":"Title and abstract screening for literature reviews using large language models: an exploratory study in the biomedical domain","volume":"13","author":"F Dennst\u00e4dt","year":"2024","journal-title":"Syst Rev"},{"issue":"1","key":"pone.0351631.ref015","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1186\/s12874-024-02203-8","article-title":"Methodological insights into ChatGPT\u2019s screening performance in systematic reviews","volume":"24","author":"M Issaiy","year":"2024","journal-title":"BMC Med Res Methodol"},{"issue":"1","key":"pone.0351631.ref016","doi-asserted-by":"crossref","first-page":"1418","DOI":"10.1038\/s41467-024-45563-x","article-title":"Structured information extraction from scientific text with large language models","volume":"15","author":"J Dagdelen","year":"2024","journal-title":"Nat Commun"},{"issue":"6","key":"pone.0351631.ref017","doi-asserted-by":"crossref","first-page":"385","DOI":"10.1136\/bmjebm-2024-113199","article-title":"From promise to practice: challenges and pitfalls in the evaluation of large language models for data extraction in evidence synthesis","volume":"30","author":"G Gartlehner","year":"2025","journal-title":"BMJ Evid Based Med"},{"key":"pone.0351631.ref018","doi-asserted-by":"crossref","unstructured":"Hilgert L, Liu D, Niehues J. Evaluating and Training Long-Context Large Language Models for Question Answering on Scientific Papers. In: Proceedings of the 1st Workshop on Customizable NLP: Progress and Challenges in Customizing NLP for a Domain, Application, Group, or Individual (CustomNLP4U). Miami (FL): Association for Computational Linguistics; 2024. p. 220\u201336.","DOI":"10.18653\/v1\/2024.customnlp4u-1.17"},{"issue":"5","key":"pone.0351631.ref019","doi-asserted-by":"crossref","first-page":"818","DOI":"10.1002\/jrsm.1732","article-title":"Performance of two large language models for data extraction in evidence synthesis","volume":"15","author":"A Konet","year":"2024","journal-title":"Res Synth Methods"},{"key":"pone.0351631.ref020","article-title":"Automation of Systematic Reviews with Large Language Models","author":"C Cao","year":"2025","journal-title":"medRxiv"},{"issue":"1","key":"pone.0351631.ref021","doi-asserted-by":"crossref","first-page":"139","DOI":"10.1186\/s12874-024-02253-y","article-title":"GPT-4 performance on querying scientific publications: reproducibility, accuracy, and impact of an instruction sheet","volume":"24","author":"K Tao","year":"2024","journal-title":"BMC Med Res Methodol"},{"issue":"4","key":"pone.0351631.ref022","doi-asserted-by":"crossref","first-page":"100729","DOI":"10.1016\/j.patter.2023.100729","article-title":"Fine-tuning large neural language models for biomedical natural language processing","volume":"4","author":"R Tinn","year":"2023","journal-title":"Patterns (N Y)"},{"key":"pone.0351631.ref023","article-title":"LoRA: Low-Rank Adaptation of Large Language Models","author":"EJ Hu","year":"2021","journal-title":"arXiv"},{"key":"pone.0351631.ref024","article-title":"Finetuned Language Models Are Zero-Shot Learners","author":"J Wei","year":"2022","journal-title":"arXiv"},{"key":"pone.0351631.ref025","article-title":"Instruction Tuning for Large Language Models: A Survey","author":"S Zhang","year":"2023","journal-title":"arXiv"},{"issue":"1","key":"pone.0351631.ref026","doi-asserted-by":"crossref","first-page":"1608","DOI":"10.1038\/s41598-025-85715-7","article-title":"A scientific-article key-insight extraction system based on multi-actor of fine-tuned open-source large language models","volume":"15","author":"Z Song","year":"2025","journal-title":"Sci Rep"},{"issue":"9","key":"pone.0351631.ref027","doi-asserted-by":"crossref","first-page":"1821","DOI":"10.1093\/jamia\/ocae122","article-title":"BioInstruct: instruction tuning of large language models for biomedical natural language processing","volume":"31","author":"H Tran","year":"2024","journal-title":"J Am Med Inform Assoc"},{"key":"pone.0351631.ref028","doi-asserted-by":"crossref","first-page":"111746","DOI":"10.1016\/j.jclinepi.2025.111746","article-title":"Large language models for conducting systematic reviews: on the rise, but not yet ready for use-a scoping review","volume":"181","author":"J-L Lieberum","year":"2025","journal-title":"J Clin Epidemiol"},{"key":"pone.0351631.ref029","article-title":"Automating Research Synthesis with Domain-Specific Large Language Model Fine-Tuning","author":"T Susnjak","year":"2024","journal-title":"arXiv"},{"issue":"1","key":"pone.0351631.ref030","doi-asserted-by":"crossref","first-page":"239","DOI":"10.1038\/s41746-024-01239-w","article-title":"Closing the gap between open source and commercial large language models for medical evidence summarization","volume":"7","author":"G Zhang","year":"2024","journal-title":"NPJ Digit Med"},{"key":"pone.0351631.ref031","article-title":"Human Immunodeficiency Virus Reverse Transcriptase and Protease Sequence Database","author":"RW Shafer","year":"2000","journal-title":"Nucleic Acids Res"},{"issue":"9","key":"pone.0351631.ref032","article-title":"HIV-1 transmitted drug resistance surveillance: shifting trends in study design and prevalence estimates","volume":"23","author":"S-Y Rhee","year":"2020","journal-title":"J Int AIDS Soc"},{"issue":"5","key":"pone.0351631.ref033","article-title":"Public availability of HIV-1 drug resistance sequence and treatment data: a systematic review","volume":"3","author":"S-Y Rhee","year":"2022","journal-title":"Lancet Microbe"},{"issue":"3","key":"pone.0351631.ref034","doi-asserted-by":"crossref","first-page":"399","DOI":"10.3390\/v16030399","article-title":"Prevalence of Emergent Dolutegravir Resistance Mutations in People Living with HIV: A Rapid Scoping Review","volume":"16","author":"C Chu","year":"2024","journal-title":"Viruses"},{"key":"pone.0351631.ref035","author":"Huggingface","year":"2023"},{"key":"pone.0351631.ref036","article-title":"QLoRA: Efficient Finetuning of Quantized LLMs","author":"T Dettmers","year":"2023","journal-title":"arXiv"},{"key":"pone.0351631.ref037"},{"issue":"3","key":"pone.0351631.ref038","doi-asserted-by":"crossref","first-page":"389","DOI":"10.7326\/ANNALS-24-02189","article-title":"Development of Prompt Templates for Large Language Model-Driven Screening in Systematic Reviews","volume":"178","author":"C Cao","year":"2025","journal-title":"Ann Intern Med"},{"key":"pone.0351631.ref039","author":"OpenAI","year":"2026"},{"key":"pone.0351631.ref040","unstructured":"Lee Y, Lee K, Park S, Hwang D, Kim J, Lee H, et al. Qasa: Advanced Question Answering on Scientific Articles. In: Proceedings of the 40th International Conference on Machine Learning. 2023."},{"key":"pone.0351631.ref041","article-title":"Lost in the Middle: How Language Models Use Long Contexts","author":"NF Liu","year":"2023","journal-title":"arXiv"},{"key":"pone.0351631.ref042","article-title":"A Survey of Large Language Models","author":"WX Zhao","year":"2025","journal-title":"arXiv"},{"issue":"9","key":"pone.0351631.ref043","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3560815","article-title":"Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing","volume":"55","author":"P Liu","year":"2023","journal-title":"ACM Comput Surv"},{"key":"pone.0351631.ref044","article-title":"Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks","author":"P Lewis","year":"2021","journal-title":"arXiv"},{"key":"pone.0351631.ref045","doi-asserted-by":"crossref","unstructured":"Khan A, Ali Z, Aziz A, Kefalas P. Talk2Doc: A Patient Q&A system using Retrieval- Augmented Generation with Weighted Knowledge Graphs and LLMs. 2025 International Conference on Intelligent Computing. 2025.","DOI":"10.65286\/icic.v21i4.47743"},{"issue":"2","key":"pone.0351631.ref046","doi-asserted-by":"crossref","DOI":"10.1056\/AIoa2300068","article-title":"Almanac - Retrieval-Augmented Language Models for Clinical Medicine","volume":"1","author":"C Zakka","year":"2024","journal-title":"NEJM AI"},{"issue":"7","key":"pone.0351631.ref047","doi-asserted-by":"crossref","first-page":"697","DOI":"10.1016\/j.jclinepi.2005.11.010","article-title":"Single data extraction generated more errors than double data extraction in systematic reviews","volume":"59","author":"N Buscemi","year":"2006","journal-title":"J Clin Epidemiol"},{"issue":"4","key":"pone.0351631.ref048","doi-asserted-by":"crossref","first-page":"430","DOI":"10.1001\/jama.298.4.430","article-title":"Data extraction errors in meta-analyses that use standardized mean differences","volume":"298","author":"PC G\u00f8tzsche","year":"2007","journal-title":"JAMA"},{"issue":"1","key":"pone.0351631.ref049","doi-asserted-by":"crossref","first-page":"152","DOI":"10.1186\/s12874-017-0431-4","article-title":"Frequency of data extraction errors and methods to increase data extraction quality: a methodological review","volume":"17","author":"T Mathes","year":"2017","journal-title":"BMC Med Res Methodol"},{"key":"pone.0351631.ref050","doi-asserted-by":"crossref","first-page":"160018","DOI":"10.1038\/sdata.2016.18","article-title":"The FAIR Guiding Principles for scientific data management and stewardship","volume":"3","author":"MD Wilkinson","year":"2016","journal-title":"Sci Data"},{"key":"pone.0351631.ref051","author":"S Verhulst","year":"2025","journal-title":"Moving Toward the FAIR-R Principles: Advancing AI-Ready Data"}],"container-title":["PLOS One"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0351631","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,24]],"date-time":"2026-06-24T17:41:00Z","timestamp":1782322860000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0351631"}},"subtitle":[],"editor":[{"given":"Mehmooda","family":"Munazir","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2026,6,24]]},"references-count":51,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2026,6,24]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pone.0351631","relation":{},"ISSN":["1932-6203"],"issn-type":[{"value":"1932-6203","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,24]]}}}