{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T01:05:26Z","timestamp":1791421526399,"version":"4.3.3"},"reference-count":31,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T00:00:00Z","timestamp":1777420800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100011923","name":"KDDI Foundation","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100011923","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Digit. Health"],"abstract":"<jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Systematic reviews depend on manual data extraction and synthesis, which are time-consuming and prone to human error. Although large language models (LLMs) have the potential to automate parts of this process, their accuracy, reproducibility, and efficiency across different models and prompt strategies remain insufficiently characterized.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>This study evaluated the performance of three LLMs, including ChatGPT-4o, Claude 3 Sonnet, and Gemini 1.5 Pro, for data extraction from trials addressing five clinical questions (CQs) in the Japanese Clinical Practice Guidelines for the Management of Sepsis and Septic Shock 2024 (J-SSCG 2024). Using portable document format files of eligible studies, LLMs extracted predefined background characteristics and clinical outcomes. Outputs generated using an original prompt were compared with those produced using chain-of-thought and self-reflection (SR) prompt strategies. Two independent reviewers assessed accuracy against a reference standard derived from manual extraction by the guideline members. Inter-session consistency across three sessions and processing time were also evaluated.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>For background data extraction, mean no-error proportions ranged from 81.6% (ChatGPT-4o) to 92.4% (Claude 3 Sonnet) across models. For outcome data extraction, mean no-error proportions ranged from 27.8% (Gemini 1.5 Pro) to 80.7% (Claude 3 Sonnet). Missing or incorrect values accounted for most extraction errors, whereas fabricated outputs were relatively uncommon. Prompt engineering strategies resulted in only modest changes in extraction accuracy across models. Inter-session consistency ranged from 76.3% (ChatGPT-4o) to 91.3% (Gemini 1.5 Pro) for background data extraction and from 44.8% (ChatGPT-4o) to 65.6% (Claude 3 Sonnet) for outcome data extraction. Mean processing times ranged from 29.2 to 39.7\u2005s per article for background data extraction and from 19.3 to 46.3\u2005s for outcome data extraction using standard prompts. When SR prompts were used, processing times increased to 59.0 to 97.7\u2005s for background data extraction and to 52.7 to 107.1\u2005s for outcome data extraction.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusions<\/jats:title>\n                    <jats:p>LLMs can reliably support background data extraction in systematic reviews. However, outcome data extraction remains challenging, emphasizing the continued need for human oversight. Extraction performance varied across models and prompt engineering strategies.<\/jats:p>\n                    <jats:p>\n                      <jats:bold>Clinical Trial Registration<\/jats:bold>\n                      : The study was registered in the University Hospital Medical Information Network (UMIN) clinical trials registry, identifier (UMIN000054461).\n                    <\/jats:p>\n                  <\/jats:sec>","DOI":"10.3389\/fdgth.2026.1799623","type":"journal-article","created":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T05:58:04Z","timestamp":1777442284000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":5,"title":["Performance of large language models and prompt engineering strategies for data extraction in systematic reviews"],"prefix":"10.3389","volume":"8","author":[{"given":"Takehiko","family":"Oami","sequence":"first","affiliation":[{"name":"Department of Emergency and Critical Care Medicine, Chiba University Graduate School of Medicine","place":["Chiba, Japan"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yohei","family":"Okada","sequence":"additional","affiliation":[{"name":"Department of Preventive Services, Kyoto University Graduate School of Medicine","place":["Kyoto, Japan"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kenjiro","family":"Maeda","sequence":"additional","affiliation":[{"name":"Department of Emergency and Critical Care Medicine, Chiba University Graduate School of Medicine","place":["Chiba, Japan"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Taka-aki","family":"Nakada","sequence":"additional","affiliation":[{"name":"Department of Emergency and Critical Care Medicine, Chiba University Graduate School of Medicine","place":["Chiba, Japan"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2026,4,29]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"312","DOI":"10.1002\/jrsm.1313","article-title":"Using systematic reviews in guideline development: the GRADE approach","volume":"10","author":"Zhang","year":"2019","journal-title":"Res Synth Methods"},{"key":"B2","doi-asserted-by":"publisher","first-page":"1049","DOI":"10.1136\/bmj.39493.646875.AE","article-title":"Going from evidence to recommendations","volume":"336","author":"Guyatt","year":"2008","journal-title":"Br Med J"},{"key":"B3","doi-asserted-by":"publisher","first-page":"e012545","DOI":"10.1136\/bmjopen-2016-012545","article-title":"Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry","volume":"7","author":"Borah","year":"2017","journal-title":"BMJ Open"},{"key":"B4","doi-asserted-by":"publisher","first-page":"119","DOI":"10.1002\/jrsm.42","article-title":"Precision of healthcare systematic review searches in a cross-sectional sample","volume":"2","author":"Sampson","year":"2011","journal-title":"Res Synth Methods"},{"key":"B5","doi-asserted-by":"publisher","first-page":"e0227742","DOI":"10.1371\/journal.pone.0227742","article-title":"Error rates of human reviewers during abstract screening in systematic reviews","volume":"15","author":"Wang","year":"2020","journal-title":"PLoS One"},{"key":"B6","doi-asserted-by":"publisher","first-page":"1201","DOI":"10.1056\/NEJMra2302038","article-title":"Artificial intelligence and machine learning in clinical medicine, 2023","volume":"388","author":"Haug","year":"2023","journal-title":"N Engl J Med"},{"key":"B7","doi-asserted-by":"publisher","first-page":"1233","DOI":"10.1056\/NEJMsr2214184","article-title":"Benefits, limits, and risks of GPT-4 as an AI chatbot for medicine","volume":"388","author":"Lee","year":"2023","journal-title":"N Engl J Med"},{"key":"B8","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","year":"2023","journal-title":"Nature"},{"key":"B9","doi-asserted-by":"publisher","first-page":"278","DOI":"10.1186\/s13643-019-1222-2","article-title":"Performance and usability of machine learning for screening in systematic reviews: a comparative evaluation of three tools","volume":"8","author":"Gates","year":"2019","journal-title":"Syst Rev"},{"key":"B10","doi-asserted-by":"publisher","first-page":"380","DOI":"10.2188\/jea.JE20230227","article-title":"Efficiency and workload reduction of semi-automated citation screening software for creating clinical practice guidelines: a prospective observational study","volume":"34","author":"Oami","year":"2023","journal-title":"J Epidemiol"},{"key":"B11","doi-asserted-by":"publisher","first-page":"143","DOI":"10.1186\/s13643-019-1062-0","article-title":"A question of trust: can we build an evidence base to gain trust in systematic review automation technologies?","volume":"8","author":"O'Connor","year":"2019","journal-title":"Syst Rev"},{"key":"B12","doi-asserted-by":"publisher","first-page":"576","DOI":"10.1002\/jrsm.1710","article-title":"Data extraction for evidence synthesis using a large language model: a proof-of-concept study","volume":"15","author":"Gartlehner","year":"2024","journal-title":"Res Synth Methods"},{"key":"B13","doi-asserted-by":"publisher","first-page":"347","DOI":"10.1002\/jrsm.1689","article-title":"Methods for using bing\u2019s AI-powered search engine for data extraction for a systematic review","volume":"15","author":"Hill","year":"2024","journal-title":"Res Synth Methods"},{"key":"B14","doi-asserted-by":"publisher","first-page":"616","DOI":"10.1002\/jrsm.1715","article-title":"Can large language models replace humans in systematic reviews? Evaluating GPT-4\u2019s efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languages","volume":"15","author":"Khraisha","year":"2024","journal-title":"Res Synth Methods"},{"key":"B15","doi-asserted-by":"publisher","first-page":"e68097","DOI":"10.2196\/68097","article-title":"Critical assessment of large language Models\u2019 (ChatGPT) performance in data extraction for systematic reviews: exploratory study","volume":"4","author":"Mahmoudi","year":"2025","journal-title":"JMIR AI"},{"key":"B16","doi-asserted-by":"publisher","first-page":"74","DOI":"10.1038\/s41746-025-01457-w","article-title":"Language models for data extraction and risk of bias assessment in complementary medicine","volume":"8","author":"Lai","year":"2025","journal-title":"NPJ Digit Med"},{"key":"B17","doi-asserted-by":"publisher","first-page":"e68666","DOI":"10.2196\/68666","article-title":"Evaluating a customized version of ChatGPT for systematic review data extraction in health research: development and usability study","volume":"9","author":"Sercombe","year":"2025","journal-title":"JMIR Form Res"},{"key":"B18","doi-asserted-by":"publisher","first-page":"1569","DOI":"10.1038\/s41467-024-45914-8","article-title":"Extracting accurate materials data from research papers with conversational language models and prompt engineering","volume":"15","author":"Polak","year":"2024","journal-title":"Nat Commun"},{"key":"B19","doi-asserted-by":"publisher","first-page":"2722","DOI":"10.1097\/JS9.0000000000002215","article-title":"AI-driven evidence synthesis: data extraction of randomized controlled trials with large language models","volume":"111","author":"Liu","year":"2025","journal-title":"Int J Surg"},{"key":"B20","doi-asserted-by":"publisher","first-page":"818","DOI":"10.1002\/jrsm.1732","article-title":"Performance of two large language models for data extraction in evidence synthesis","volume":"15","author":"Konet","year":"2024","journal-title":"Res Synth Methods"},{"key":"B21","doi-asserted-by":"publisher","first-page":"135","DOI":"10.1186\/s13643-024-02523-2","article-title":"Evaluating the OpenAI\u2019s GPT-3.5 Turbo\u2019s performance in extracting information from scientific articles on diabetic retinopathy","volume":"13","author":"Gue","year":"2024","journal-title":"Syst Rev"},{"key":"B22","doi-asserted-by":"publisher","first-page":"e70033","DOI":"10.1002\/cesm.70033","article-title":"Data extractions using a large language model (elicit) and human reviewers in randomized controlled trials: a systematic comparison","volume":"3","author":"Bianchi","year":"2025","journal-title":"Cochrane Evid Synth Methods"},{"key":"B23","doi-asserted-by":"publisher","first-page":"313","DOI":"10.1136\/bmjebm-2024-113066","article-title":"Novel AI applications in systematic review: gPT-4 assisted data extraction, analysis, review of bias","volume":"30","author":"Kim","year":"2025","journal-title":"BMJ Evid Based Med"},{"key":"B24","doi-asserted-by":"publisher","first-page":"e0313401","DOI":"10.1371\/journal.pone.0313401","article-title":"ChatGPT-4o can serve as the second rater for data extraction in systematic reviews","volume":"20","author":"Motzfeldt Jensen","year":"2025","journal-title":"PLoS One"},{"key":"B25","doi-asserted-by":"publisher","first-page":"15","DOI":"10.1186\/s40560-025-00776-0","article-title":"The Japanese clinical practice guidelines for management of sepsis and septic shock 2024","volume":"13","author":"Shime","year":"2025","journal-title":"J Intensive Care"},{"key":"B26","doi-asserted-by":"publisher","first-page":"53","DOI":"10.1186\/s40560-021-00555-7","article-title":"The Japanese clinical practice guidelines for management of sepsis and septic shock 2020 (J-SSCG 2020)","volume":"9","author":"Egi","year":"2021","journal-title":"J Intensive Care"},{"key":"B27","doi-asserted-by":"publisher","first-page":"24824","DOI":"10.5555\/3600270.3602070","article-title":"Chain-of-thought prompting elicits reasoning in large language models","volume":"35","author":"Wei","year":"2022","journal-title":"Adv Neural Inf Process Syst"},{"key":"B28","author":"Wang","year":"2022"},{"key":"B29","author":"Trikalinos","year":"2012"},{"key":"B30","doi-asserted-by":"publisher","first-page":"cjae011","DOI":"10.1093\/ejo\/cjae011","article-title":"Enhancing systematic reviews in orthodontics: a comparative examination of GPT-3.5 and GPT-4 for generating PICO-based queries with tailored prompts and configurations","volume":"46","author":"Demir","year":"2024","journal-title":"Eur J Orthod"},{"key":"B31","doi-asserted-by":"publisher","first-page":"1594","DOI":"10.1186\/s12903-025-06982-4","article-title":"AI-driven abstract generating: evaluating LLMs with a tailored prompt under the PRISMA-A framework","volume":"25","author":"Demir","year":"2025","journal-title":"BMC Oral Health"}],"container-title":["Frontiers in Digital Health"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fdgth.2026.1799623\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,29]],"date-time":"2026-04-29T05:58:05Z","timestamp":1777442285000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fdgth.2026.1799623\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4,29]]},"references-count":31,"alternative-id":["10.3389\/fdgth.2026.1799623"],"URL":"https:\/\/doi.org\/10.3389\/fdgth.2026.1799623","relation":{},"ISSN":["2673-253X"],"issn-type":[{"value":"2673-253X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,4,29]]},"article-number":"1799623"}}