{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,6]],"date-time":"2026-10-06T14:03:28Z","timestamp":1791295408666,"version":"4.1.0"},"reference-count":49,"publisher":"Cambridge University Press (CUP)","issue":"5","license":[{"start":{"date-parts":[[2026,4,14]],"date-time":"2026-04-14T00:00:00Z","timestamp":1776124800000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nd\/4.0"}],"content-domain":{"domain":["cambridge.org"],"crossmark-restriction":true},"short-container-title":["Res. synth. methods"],"published-print":{"date-parts":[[2026,9]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Interest in large language models (LLMs) as a tool for meta-analyses and systematic reviews (MA\/SRs) is growing. We prospectively developed 515 unique prompts by predefined screening-related categories and tested with open-access LLMs (Llama, Mistral) against four gold-standard MA\/SRs from different medical fields published after the LLMs\u2019 training cut-offs, using a Python-based pipeline. Heterogeneity between prompts was quantified, and hypothetical workload\/cost reduction with top-performing prompts calculated. Across 12,360 pipeline runs, LLMs versus MA\/SRs reached average recall\/sensitivity = 83.6 \u00b1 17.0%, precision = 18.5 \u00b1 15.6%, specificity = 36.6 \u00b1 23.7% F1-score = 27.6 \u00b1 17.2%, and accuracy = 61.1 \u00b1 11.0%. F1-scores were significantly higher when prompts focused on methods (0.78 \u00b1 0.40%), explicitly mentioned MA\/SR screening (0.81 \u00b1 0.37%), included the comparison MA\/SR\u2019s title (5.64 \u00b1 0.37%) or selection criteria (8.05 \u00b1 0.68%), and with more LLM parameters (70b = 4.48 \u00b1 0.31%, 123b = 7.77 \u00b1 0.31%), but lower when screening abstracts instead of titles (\u22123.67 \u00b1 0.28%). In LLM-base preselection, top-performing F1-score prompts (recall\/sensitivity = 72.2%, specificity = 66.1%, precision = 28.6%) would reduce screening demands by 34.5%\u221237.5%, saving 8.4\u20138.8 weeks of work and 17,592\u201318,552. Recall\/sensitivity increased with less MA\/SR information contrasting F1-score results, which highlights a recall\/sensitivity-precision\/specificity trade-off. F1-score increased with detailed MA\/SR information, while recall\/sensitivity increased with shorter, zeroshot prompts. We provide the first prospectively assessed prompt engineering framework for early-stage LLM-based paper screening across medical fields. The publicly available Python pipeline and full prompt list used here support further development of LLM-based evidence synthesis.<\/jats:p>","DOI":"10.1017\/rsm.2026.10093","type":"journal-article","created":{"date-parts":[[2026,4,14]],"date-time":"2026-04-14T02:29:37Z","timestamp":1776133777000},"page":"939-956","update-policy":"https:\/\/doi.org\/10.1017\/policypage","source":"Crossref","is-referenced-by-count":5,"title":["Prompt engineering of large language models for paper screening in medical meta-analyses and systematic reviews: A prospective comparative study"],"prefix":"10.1017","volume":"17","author":[{"ORCID":"https:\/\/orcid.org\/0009-0004-6609-3462","authenticated-orcid":false,"given":"Till J.","family":"Adam","sequence":"first","affiliation":[{"name":"Charit\u00e9 \u2013 Universit\u00e4tsmedizin Berlin"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Salma A. S.","family":"Abosabie","sequence":"additional","affiliation":[{"name":"Charite \u2013 Universitatsmedizin Berlin"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Max","family":"Dittmer","sequence":"additional","affiliation":[{"name":"Technische Universit\u00e4t Berlin"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-5130-4352","authenticated-orcid":false,"given":"Elise","family":"Wolf","sequence":"additional","affiliation":[{"name":"Universit\u00e4t Mannheim"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sara A.","family":"Abosabie","sequence":"additional","affiliation":[{"name":"Charite \u2013 Universitatsmedizin Berlin"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Clara","family":"Behnke","sequence":"additional","affiliation":[{"name":"Humboldt-Universitat zu Berlin"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Felix","family":"Baier","sequence":"additional","affiliation":[{"name":"Humboldt-Universitat zu Berlin"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Annabelle","family":"Weickmann","sequence":"additional","affiliation":[{"name":"Humboldt-Universitat zu Berlin"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ludwig","family":"K\u00f6ser","sequence":"additional","affiliation":[{"name":"University of Potsdam"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christoph U.","family":"Correll","sequence":"additional","affiliation":[{"name":"Charit\u00e9 \u2013 Universit\u00e4tsmedizin Berlin"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Niklas","family":"Rutsch","sequence":"additional","affiliation":[{"name":"Charit\u00e9 \u2013 Universit\u00e4tsmedizin Berlin"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"56","published-online":{"date-parts":[[2026,4,14]]},"reference":[{"key":"S1759287926100933_r24","doi-asserted-by":"publisher","DOI":"10.1186\/s12874-019-0782-0"},{"key":"S1759287926100933_r1","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-025-56989-2"},{"key":"S1759287926100933_r32","doi-asserted-by":"publisher","DOI":"10.7326\/0003-4819-149-12-200812160-00008"},{"key":"S1759287926100933_r4","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-01840-7"},{"key":"S1759287926100933_r22","unstructured":"[22] Web of Science Journal Info: Impact factor, Quartile, Ranking. https:\/\/wos-journal.info\/"},{"key":"S1759287926100933_r25","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-024-01029-4"},{"key":"S1759287926100933_r2","doi-asserted-by":"publisher","DOI":"10.1016\/j.patter.2024.100968"},{"key":"S1759287926100933_r41","doi-asserted-by":"publisher","DOI":"10.3390\/jcm13247697"},{"key":"S1759287926100933_r8","doi-asserted-by":"publisher","DOI":"10.1016\/j.conctc.2019.100443"},{"key":"S1759287926100933_r12","doi-asserted-by":"publisher","DOI":"10.1002\/jrsm.1335"},{"key":"S1759287926100933_r17","doi-asserted-by":"publisher","DOI":"10.1145\/3661167.3661172"},{"key":"S1759287926100933_r5","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijcard.2025.133423"},{"key":"S1759287926100933_r7","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2016-012545"},{"key":"S1759287926100933_r21","doi-asserted-by":"publisher","DOI":"10.1016\/j.clinimag.2024.110276"},{"key":"S1759287926100933_r20","doi-asserted-by":"publisher","DOI":"10.1136\/openhrt-2023-002455"},{"key":"S1759287926100933_r47","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-01651-w"},{"key":"S1759287926100933_r11","doi-asserted-by":"publisher","DOI":"10.7326\/ANNALS-24-02189"},{"key":"S1759287926100933_r46","doi-asserted-by":"publisher","DOI":"10.1002\/jrsm.1369"},{"key":"S1759287926100933_r37","unstructured":"[37] R Core Team. R: A Language and Environment for Statistical Computing. 2024. https:\/\/www.R-project.org\/"},{"key":"S1759287926100933_r44","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2020.01.005"},{"key":"S1759287926100933_r49","doi-asserted-by":"publisher","DOI":"10.1186\/s13643-020-01569-2"},{"key":"S1759287926100933_r14","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2411962122"},{"key":"S1759287926100933_r43","doi-asserted-by":"publisher","DOI":"10.1111\/pcn.13709"},{"key":"S1759287926100933_r39","doi-asserted-by":"publisher","DOI":"10.18637\/jss.v082.i13"},{"key":"S1759287926100933_r30","unstructured":"[30] Meta AI. Meta Llama 3-70B. 2024. https:\/\/huggingface.co\/meta-llama\/Meta-Llama-3-70B"},{"key":"S1759287926100933_r27","unstructured":"[27] Millard, J. markovify: A simple, extensible Markov chain generator. 2015. https:\/\/github.com\/jsvine\/markovify"},{"key":"S1759287926100933_r3","doi-asserted-by":"publisher","DOI":"10.1093\/nar\/gkac1032"},{"key":"S1759287926100933_r38","doi-asserted-by":"publisher","DOI":"10.18637\/jss.v067.i01"},{"key":"S1759287926100933_r34","doi-asserted-by":"publisher","DOI":"10.1161\/CIRCULATIONAHA.105.594929"},{"key":"S1759287926100933_r19","unstructured":"[19] OpenAI. ChatGPT. 2025. https:\/\/chat.openai.com\/chat"},{"key":"S1759287926100933_r9","doi-asserted-by":"publisher","DOI":"10.1016\/j.cola.2024.101287"},{"key":"S1759287926100933_r33","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.i6"},{"key":"S1759287926100933_r48","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-024-81370-6"},{"key":"S1759287926100933_r42","doi-asserted-by":"publisher","DOI":"10.1038\/s41372-024-02185-x"},{"key":"S1759287926100933_r16","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-023-02448-8"},{"key":"S1759287926100933_r15","doi-asserted-by":"publisher","DOI":"10.1038\/s44222-025-00279-5"},{"key":"S1759287926100933_r18","doi-asserted-by":"publisher","DOI":"10.1016\/j.hlpt.2024.100943"},{"key":"S1759287926100933_r40","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0306296"},{"key":"S1759287926100933_r13","article-title":"Novel AI applications in systematic review: GPT-4 assisted data extraction, analysis, review of bias","author":"Kim","journal-title":"BMJ Evid-Based Med"},{"key":"S1759287926100933_r45","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2023-076912"},{"key":"S1759287926100933_r28","unstructured":"[28] Van Rossum, G , Drake, FL Python. 2009. https:\/\/docs.python.org\/3\/reference\/"},{"key":"S1759287926100933_r31","unstructured":"[31] Mistral AI. Mistral Large 2407. 2024. https:\/\/mistral.ai\/news\/mistral-large-2407"},{"key":"S1759287926100933_r26","doi-asserted-by":"publisher","DOI":"10.1016\/j.patter.2025.101260"},{"key":"S1759287926100933_r23","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.n71"},{"key":"S1759287926100933_r10","doi-asserted-by":"publisher","DOI":"10.1111\/cobi.13231"},{"key":"S1759287926100933_r29","unstructured":"[29] Meta AI. Meta Llama 3-8B. 2024. https:\/\/huggingface.co\/meta-llama\/Meta-Llama-3-8B"},{"key":"S1759287926100933_r35","volume-title":"Evaluation: From Precision, Recall and F-Factor to ROC","author":"Powers"},{"key":"S1759287926100933_r6","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0227742"},{"key":"S1759287926100933_r36","doi-asserted-by":"publisher","DOI":"10.1037\/met0000159"}],"updated-by":[{"DOI":"10.1017\/rsm.2026.10104","type":"correction","label":"Correction","source":"publisher","updated":{"date-parts":[[2026,4,14]],"date-time":"2026-04-14T00:00:00Z","timestamp":1776124800000}}],"container-title":["Research Synthesis Methods"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.cambridge.org\/core\/services\/aop-cambridge-core\/content\/view\/S1759287926100933","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,27]],"date-time":"2026-08-27T12:02:56Z","timestamp":1787832176000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.cambridge.org\/core\/product\/identifier\/S1759287926100933\/type\/journal_article"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4,14]]},"references-count":49,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2026,9]]}},"alternative-id":["S1759287926100933"],"URL":"https:\/\/doi.org\/10.1017\/rsm.2026.10093","relation":{},"ISSN":["1759-2879","1759-2887"],"issn-type":[{"value":"1759-2879","type":"print"},{"value":"1759-2887","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,4,14]]},"assertion":[{"value":"\u00a9 The Author(s), 2026. Published by Cambridge University Press on behalf of The Society for Research Synthesis Methodology","name":"copyright","label":"Copyright","group":{"name":"copyright_and_licensing","label":"Copyright and Licensing"}},{"value":"This is an Open Access article, distributed under the terms of the Creative Commons Attribution-NoDerivatives licence (https:\/\/creativecommons.org\/licenses\/by-nd\/4.0), which permits re-use, distribution, and reproduction in any medium, provided that no alterations are made and the original article is properly cited.","name":"license","label":"License","group":{"name":"copyright_and_licensing","label":"Copyright and Licensing"}},{"value":"This content has been made available to all.","name":"free","label":"Free to read"}]}}