{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,25]],"date-time":"2026-09-25T22:10:04Z","timestamp":1790374204888,"version":"4.1.0"},"reference-count":23,"publisher":"Cambridge University Press (CUP)","issue":"1","license":[{"start":{"date-parts":[[2025,9,17]],"date-time":"2025-09-17T00:00:00Z","timestamp":1758067200000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001691","name":"Japan Society for the Promotion of Science","doi-asserted-by":"publisher","award":["22K15664"],"award-info":[{"award-number":["22K15664"]}],"id":[{"id":"10.13039\/501100001691","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Res. synth. methods"],"published-print":{"date-parts":[[2026,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Large language models have shown promise for automating data extraction (DE) in systematic reviews (SRs), but most existing approaches require manual interaction. We developed an open-source system using GPT-4o to automatically extract data with no human intervention during the extraction process. We developed the system on a dataset of 290 randomized controlled trials (RCTs) from a published SR about cognitive behavioral therapy for insomnia. We evaluated the system on two other datasets: 5 RCTs from an updated search for the same review and 10 RCTs used in a separate published study that had also evaluated automated DE. We developed the best approach across all variables in the development dataset using GPT-4o. The performance in the updated-search dataset using o3 was 74.9% sensitivity, 76.7% specificity, 75.7 precision, 93.5% variable detection comprehensiveness, and 75.3% accuracy. In both datasets, accuracy was higher for string variables (e.g., country, study design, drug names, and outcome definitions) compared with numeric variables. In the third external validation dataset, GPT-4o showed a lower performance with a mean accuracy of 84.4% compared with the previous study. However, by adjusting our DE method, while maintaining the same prompting technique, we achieved a mean accuracy of 96.3%, which was comparable to the previous manual extraction study. Our system shows potential for assisting the DE of string variables alongside a human reviewer. However, it cannot yet replace humans for numeric DE. Further evaluation across diverse review contexts is needed to establish broader applicability.<\/jats:p>","DOI":"10.1017\/rsm.2025.10030","type":"journal-article","created":{"date-parts":[[2025,9,17]],"date-time":"2025-09-17T07:35:55Z","timestamp":1758094555000},"page":"42-62","source":"Crossref","is-referenced-by-count":11,"title":["Automating the data extraction process for systematic reviews using GPT-4o and o3"],"prefix":"10.1017","volume":"17","author":[{"given":"Yuki","family":"Kataoka","sequence":"first","affiliation":[{"name":"Kyoto Min-iren Asukai Hospital"},{"name":"Scientific Research WorkS Peer Support Group (SRWS-PSG)"},{"name":"Kyoto University Graduate School of Medicine\/School of Public Health"},{"name":"Tohoku University Graduate School of Dentistry"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tomohiro","family":"Takayama","sequence":"additional","affiliation":[{"name":"Kyoto University"},{"name":"Fitting Cloud Inc."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-3985-2197","authenticated-orcid":false,"given":"Keisuke","family":"Yoshimura","sequence":"additional","affiliation":[{"name":"Kyoto University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ryuhei","family":"So","sequence":"additional","affiliation":[{"name":"Scientific Research WorkS Peer Support Group (SRWS-PSG)"},{"id":[{"id":"https:\/\/ror.org\/02thzwy35","id-type":"ROR","asserted-by":"publisher"}],"name":"Okayama Psychiatric Medical Center"},{"name":"CureApp, Inc."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yasushi","family":"Tsujimoto","sequence":"additional","affiliation":[{"name":"Scientific Research WorkS Peer Support Group (SRWS-PSG)"},{"name":"Oku Medical Clinic"},{"name":"Kyoto University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yosuke","family":"Yamagishi","sequence":"additional","affiliation":[{"name":"The University of Tokyo"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shiro","family":"Takagi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yuki","family":"Furukawa","sequence":"additional","affiliation":[{"name":"University of Tokyo"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Masatsugu","family":"Sakata","sequence":"additional","affiliation":[{"name":"Kyoto University"},{"name":"Nagoya City University Graduate School of Medical Sciences"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"\u0110or\u0111e","family":"Ba\u0161i\u0107","sequence":"additional","affiliation":[{"name":"Vrije Universiteit Amsterdam"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andrea","family":"Cipriani","sequence":"additional","affiliation":[{"name":"University of Oxford"},{"name":"NIHR Oxford Health Biomedical Research Centre"},{"name":"Warneford Hospital"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pim","family":"Cuijpers","sequence":"additional","affiliation":[{"name":"Vrije Universiteit Amsterdam"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Eirini","family":"Karyotaki","sequence":"additional","affiliation":[{"name":"Vrije Universiteit Amsterdam"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mathias","family":"Harrer","sequence":"additional","affiliation":[{"name":"Vrije Universiteit Amsterdam"},{"name":"Technical University of Munich"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Stefan","family":"Leucht","sequence":"additional","affiliation":[{"name":"Technical University of Munich"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ava","family":"Homiar","sequence":"additional","affiliation":[{"name":"University of Oxford"},{"name":"NIHR Oxford Health Biomedical Research Centre"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Edoardo G.","family":"Ostinelli","sequence":"additional","affiliation":[{"name":"University of Oxford"},{"name":"NIHR Oxford Health Biomedical Research Centre"},{"name":"Warneford Hospital"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Clara","family":"Miguel","sequence":"additional","affiliation":[{"name":"Vrije Universiteit Amsterdam"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alessandro","family":"Rodolico","sequence":"additional","affiliation":[{"name":"Technical University of Munich"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Toshi A.","family":"Furukawa","sequence":"additional","affiliation":[{"name":"Kyoto University Office of Institutional Advancement and Communications"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"56","published-online":{"date-parts":[[2025,9,17]]},"reference":[{"key":"S1759287925100306_r4","unstructured":"[4] Kataoka, Y , So, R , Banno, M , et al. Development of meta-prompts for large language models to screen titles and abstracts for diagnostic test accuracy reviews [Internet]. medRxiv. 2023. Cited November 2, 2023. p. 2023.10.31.23297818. https:\/\/www.medrxiv.org\/content\/10.1101\/2023.10.31.23297818v1.abstract."},{"key":"S1759287925100306_r6","doi-asserted-by":"publisher","DOI":"10.12688\/f1000research.51117.2"},{"key":"S1759287925100306_r11","doi-asserted-by":"publisher","DOI":"10.1002\/jrsm.1732"},{"key":"S1759287925100306_r12","unstructured":"[12] Wu, S , Ma, X , Luo, D , et al. Automated review generation method based on large language models [Internet]. arXiv [cs.CL]. 2024. http:\/\/arxiv.org\/abs\/2407.20906."},{"key":"S1759287925100306_r1","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-psych-010418-102803"},{"key":"S1759287925100306_r16","unstructured":"[16] PDF Extract API [Internet]. Cited August 28, 2024. https:\/\/developer.adobe.com\/document-services\/docs\/overview\/pdf-extract-api\/."},{"key":"S1759287925100306_r7","doi-asserted-by":"publisher","DOI":"10.1016\/j.mcpdig.2024.05.023"},{"key":"S1759287925100306_r17","doi-asserted-by":"publisher","DOI":"10.1002\/jrsm.1749"},{"key":"S1759287925100306_r19","unstructured":"[19] Azure OpenAI service models [Internet]. Cited March 14, 2025. https:\/\/learn.microsoft.com\/en-us\/azure\/ai-services\/openai\/concepts\/models?tabs=global-standard,standard-chat-completions."},{"key":"S1759287925100306_r9","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-024-45914-8"},{"key":"S1759287925100306_r18","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2309.06275"},{"key":"S1759287925100306_r22","unstructured":"[22] Li, D , Kadav, A , Gao, A , Li, R , Bourgon, R. Automated clinical data extraction with knowledge conditioned LLMs [Internet]. arXiv [cs.CL]. 2024. Cited August 4, 2024. http:\/\/arxiv.org\/abs\/2406.18027."},{"key":"S1759287925100306_r23","unstructured":"[23] Ranjit, M , Ganapathy, G , Manuel, R , Ganu, T. Retrieval augmented chest X-ray report generation using OpenAI GPT models [Internet]. arXiv [cs.CL]. 2023. http:\/\/arxiv.org\/abs\/2305.03660."},{"key":"S1759287925100306_r13","unstructured":"[13] Wang, Z , Cao, L , Danek, B , et al. Accelerating clinical evidence synthesis with large language models [Internet]. arXiv [cs.CL]. 2024. Cited July 30, 2024. http:\/\/arxiv.org\/abs\/2406.17755."},{"key":"S1759287925100306_r20","unstructured":"[20] Model\u2014OpenAI API [Internet]. Cited May 6, 2025. https:\/\/platform.openai.com\/docs\/models\/o3."},{"key":"S1759287925100306_r10","doi-asserted-by":"publisher","DOI":"10.1002\/jrsm.1710"},{"key":"S1759287925100306_r14","doi-asserted-by":"publisher","DOI":"10.1001\/jamapsychiatry.2023.5060"},{"key":"S1759287925100306_r15","first-page":"1","article-title":"Pre-train, prompt, and predict: a systematic survey of prompting methods in natural language processing","volume":"55","author":"Liu","year":"2023","journal-title":"ACM Comput Surv."},{"key":"S1759287925100306_r2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2020.01.008"},{"key":"S1759287925100306_r5","doi-asserted-by":"publisher","DOI":"10.12688\/f1000research.151493.1"},{"key":"S1759287925100306_r3","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.m1328"},{"key":"S1759287925100306_r8","doi-asserted-by":"publisher","DOI":"10.2196\/56780"},{"key":"S1759287925100306_r21","volume-title":"RStudio: Integrated Development for R","year":"2024"}],"container-title":["Research Synthesis Methods"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.cambridge.org\/core\/services\/aop-cambridge-core\/content\/view\/S1759287925100306","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,9]],"date-time":"2026-01-09T10:30:35Z","timestamp":1767954635000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.cambridge.org\/core\/product\/identifier\/S1759287925100306\/type\/journal_article"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,9,17]]},"references-count":23,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2026,1]]}},"alternative-id":["S1759287925100306"],"URL":"https:\/\/doi.org\/10.1017\/rsm.2025.10030","relation":{},"ISSN":["1759-2879","1759-2887"],"issn-type":[{"value":"1759-2879","type":"print"},{"value":"1759-2887","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,9,17]]}}}