{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,6]],"date-time":"2026-10-06T14:03:28Z","timestamp":1791295408914,"version":"4.1.0"},"reference-count":19,"publisher":"Wiley","issue":"3","license":[{"start":{"date-parts":[[2026,4,11]],"date-time":"2026-04-11T00:00:00Z","timestamp":1775865600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2026,4,11]],"date-time":"2026-04-11T00:00:00Z","timestamp":1775865600000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/doi.wiley.com\/10.1002\/tdm_license_1.1"}],"content-domain":{"domain":["onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Cochrane Evidence Synthesis and Methods"],"published-print":{"date-parts":[[2026,5]]},"abstract":"<jats:title>ABSTRACT<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Manual abstract screening is a primary bottleneck in evidence synthesis. Emerging evidence suggests that large language models (LLMs) can automate this task, but their performance when processing multiple references simultaneously in \u201cbatches\u201d is uncertain.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Objectives<\/jats:title>\n                    <jats:p>To evaluate the classification performance of four state\u2010of\u2010the\u2010art LLMs (Gemini 2.5 Pro, Gemini 2.5 Flash, GPT\u20105, and GPT\u20105 mini) in predicting reference eligibility across a wide range of batch sizes for a systematic review of randomized controlled trials.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We used a gold\u2010standard dataset of 790 references (93 considered relevant) from a published Cochrane Review on stem cell treatment for acute myocardial infarction. Using the public APIs for each model, batches of 1 to 790 references were submitted to classify each as \u201cInclude\u201d or \u201cExclude.\u201d Performance was assessed using sensitivity and specificity, with internal validation conducted through 10 repeated runs for each model\u2010batch combination.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Gemini 2.5 Pro was the most robust model, successfully processing the full 790\u2010reference batch. In contrast, GPT\u20105 failed at batches \u2265400, while GPT\u20105 mini and Gemini 2.5 Flash failed at the 790\u2010reference batch. Overall, all models demonstrated strong performance within their operational ranges, with two notable exceptions: Gemini 2.5 Flash showed low initial sensitivity at batch 1, and GPT\u20105 mini's sensitivity degraded at higher batch sizes (from 0.88 at batch 200 to 0.48 at batch 400). At a practical batch size of 100, Gemini 2.5 Pro achieved the highest sensitivity (1.00, 95% CI 1.00\u20131.00), whereas GPT\u20105 delivered the highest specificity (0.98, 95% CI 0.98\u20130.98).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>State\u2010of\u2010the\u2010art LLMs can effectively screen multiple abstracts per prompt, moving beyond inefficient single\u2010reference processing. However, performance is model\u2010dependent, revealing trade\u2010offs between sensitivity and specificity. Therefore, batch size optimization and strategic model selection are important parameters for successful implementation.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1002\/cesm.70082","type":"journal-article","created":{"date-parts":[[2026,4,11]],"date-time":"2026-04-11T07:54:29Z","timestamp":1775894069000},"update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Batch Size Effects on Mid\u20102025 State\u2010of\u2010the\u2010Art Large Language Model Performance in Automated Title and Abstract Screening"],"prefix":"10.1002","volume":"4","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1443-3128","authenticated-orcid":false,"given":"Petter","family":"Fagerberg","sequence":"first","affiliation":[{"name":"The National Board of Health and Welfare Stockholm Region Stockholm Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Oscar","family":"Sallander","sequence":"additional","affiliation":[{"name":"The National Board of Health and Welfare Stockholm Region Stockholm Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kim Vikhe","family":"Patil","sequence":"additional","affiliation":[{"name":"The National Board of Health and Welfare Stockholm Region Stockholm Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anders","family":"Berg","sequence":"additional","affiliation":[{"name":"The National Board of Health and Welfare Stockholm Region Stockholm Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anastasia","family":"Nyman","sequence":"additional","affiliation":[{"name":"The National Board of Health and Welfare Stockholm Region Stockholm Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Natalia","family":"Borg","sequence":"additional","affiliation":[{"name":"The National Board of Health and Welfare Stockholm Region Stockholm Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thomas","family":"Lind\u00e9n","sequence":"additional","affiliation":[{"name":"The National Board of Health and Welfare Stockholm Region Stockholm Sweden"},{"name":"Institute of Neuroscience and Physiology, Sahlgrenska Academy Gothenburg University Gothenburg Region V\u00e4stra G\u00f6taland Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"311","published-online":{"date-parts":[[2026,4,11]]},"reference":[{"key":"e_1_2_15_2_1","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2016-012545"},{"key":"e_1_2_15_3_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.conctc.2019.100443"},{"key":"e_1_2_15_4_1","doi-asserted-by":"publisher","DOI":"10.1002\/jrsm.1354"},{"key":"e_1_2_15_5_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2025.111746"},{"key":"e_1_2_15_6_1","doi-asserted-by":"publisher","DOI":"10.7326\/ANNALS-24-02189"},{"key":"e_1_2_15_7_1","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocaf050"},{"key":"e_1_2_15_8_1","doi-asserted-by":"publisher","DOI":"10.1002\/jrsm.1715"},{"key":"e_1_2_15_9_1","unstructured":"LMArena.ai. \u201cLeader Board Overview\u2014LMArena\u201d2025 Accessed 2025\u2010Aug\u201006 from:https:\/\/lmarena.ai\/leaderboard."},{"key":"e_1_2_15_10_1","unstructured":"lmarena\u2011ai. LMArena Leaderboard\u2014Hugging Face Spaces.2025 Accessed 2025\u2010Aug\u201006 from:https:\/\/huggingface.co\/spaces\/lmarena-ai\/lmarena-leaderboard."},{"key":"e_1_2_15_11_1","unstructured":"DeepMind. Gemini Pro\u2014Google DeepMind.2025 Accessed 2025\u2010Aug\u201006 from:https:\/\/deepmind.google\/models\/gemini\/pro\/."},{"key":"e_1_2_15_12_1","unstructured":"DeepMind. Gemini Flash\u2014Google DeepMind.2025June 2025 Accessed 2025\u2010Aug\u201006 from:https:\/\/deepmind.google\/models\/gemini\/flash\/."},{"key":"e_1_2_15_13_1","unstructured":"OpenAi. Vi presenterar GPT\u20105\u2014v\u00e5rt hittills b\u00e4sta AI\u2010system. 2025 2025\/08\/11 Accessed from:https:\/\/openai.com\/sv-SE\/index\/introducing-gpt-5\/."},{"issue":"6","key":"e_1_2_15_14_1","article-title":"Stem Cell Treatment for Acute Myocardial Infarction","author":"Zwetsloot P. P.","year":"2025","journal-title":"Cochrane Database of Systematic Reviews"},{"key":"e_1_2_15_15_1","unstructured":"TheR.Foundation. The Comprehensive R Archive Network (CRAN)\u2014cloud.r\u2010project.org. 2025 2025\/08\/06 Accessed from:https:\/\/cloud.r-project.org\/."},{"key":"e_1_2_15_16_1","unstructured":"GoogleDeepThink. Gemini 2.5: Deep Think Is Now Rolling Out\u2014Google Blog.2025 Accessed 2025\u2010Aug\u201006 from:https:\/\/blog.google\/products\/gemini\/gemini-2-5-deep-think\/."},{"key":"e_1_2_15_17_1","unstructured":"OpenAI. o3\u2011Pro.2025 Accessed 2025\u2010Aug\u201006 from:https:\/\/openai.com\/sv-SE\/index\/o3-o4-mini-system-card\/."},{"key":"e_1_2_15_18_1","unstructured":"GoogleDeepThink. Gemini\u202f2.5: Our Newest Gemini Model With Thinking\u2014Google DeepMind2025Accessed 2025\u2010Aug\u201006 from:https:\/\/blog.google\/technology\/google-deepmind\/gemini-model-thinking-updates-march-2025\/#building-on-best-gemini."},{"key":"e_1_2_15_19_1","article-title":"Multimodal Needle in a Haystack: Benchmarking Long\u2011Context Capability of Multimodal Large Language Models","author":"Wang H.","year":"2025","journal-title":"arXiv Preprint"},{"key":"e_1_2_15_20_1","volume-title":"Context Rot: How Increasing Input Tokens Impacts LLM Performance","author":"Chroma Research","year":"2025"}],"container-title":["Cochrane Evidence Synthesis and Methods"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/cesm.70082","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/full-xml\/10.1002\/cesm.70082","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/cesm.70082","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,27]],"date-time":"2026-05-27T05:06:28Z","timestamp":1779858388000},"score":1,"resource":{"primary":{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/10.1002\/cesm.70082"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4,11]]},"references-count":19,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2026,5]]}},"alternative-id":["10.1002\/cesm.70082"],"URL":"https:\/\/doi.org\/10.1002\/cesm.70082","archive":["Portico"],"relation":{"has-review":[{"id-type":"doi","id":"10.1002\/CESM.70082\/v1\/decision1","asserted-by":"object"},{"id-type":"doi","id":"10.1002\/CESM.70082\/v1\/review4","asserted-by":"object"},{"id-type":"doi","id":"10.1002\/CESM.70082\/v1\/review3","asserted-by":"object"},{"id-type":"doi","id":"10.1002\/CESM.70082\/v1\/review2","asserted-by":"object"},{"id-type":"doi","id":"10.1002\/CESM.70082\/v1\/review1","asserted-by":"object"},{"id-type":"doi","id":"10.1002\/CESM.70082\/v2\/response1","asserted-by":"object"},{"id-type":"doi","id":"10.1002\/CESM.70082\/v2\/decision1","asserted-by":"object"}]},"ISSN":["2832-9023","2832-9023"],"issn-type":[{"value":"2832-9023","type":"print"},{"value":"2832-9023","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,4,11]]},"assertion":[{"value":"2025-11-10","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-03-27","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-04-11","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"e70082"}}