{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,2]],"date-time":"2026-11-02T16:39:39Z","timestamp":1793637579435,"version":"4.5.1"},"reference-count":27,"publisher":"American Medical Association (AMA)","issue":"7","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JAMA Netw Open"],"abstract":"Importance<\/jats:title>Large language models (LLMs) are promising as tools for citation screening in systematic reviews. However, their applicability has not yet been determined.<\/jats:p><\/jats:sec>Objective<\/jats:title>To evaluate the accuracy and efficiency of an LLM in title and abstract literature screening.<\/jats:p><\/jats:sec>Design, Setting, and Participants<\/jats:title>This prospective diagnostic study used the data from the title and abstract screening process for 5 clinical questions (CQs) in the development of the Japanese Clinical Practice Guidelines for Management of Sepsis and Septic Shock. The LLM decided to include or exclude citations based on the inclusion and exclusion criteria in terms of patient, population, problem; intervention; comparison; and study design of the selected CQ and was compared with the conventional method for title and abstract screening. This study was conducted from January 7 to 15, 2024.<\/jats:p><\/jats:sec>Exposures<\/jats:title>LLM (GPT-4 Turbo)\u2013assisted citation screening or the conventional method.<\/jats:p><\/jats:sec>Main Outcomes and Measures<\/jats:title>The sensitivity and specificity of the LLM-assisted screening process was calculated, and the full-text screening result using the conventional method was set as the reference standard in the primary analysis. Pooled sensitivity and specificity were also estimated, and screening times of the 2 methods were compared.<\/jats:p><\/jats:sec>Results<\/jats:title>In the conventional citation screening process, 8 of 5634 publications in CQ 1, 4 of 3418 in CQ 2, 4 of 1038 in CQ 3, 17 of 4326 in CQ 4, and 8 of 2253 in CQ 5 were selected. In the primary analysis of 5 CQs, LLM-assisted citation screening demonstrated an integrated sensitivity of 0.75 (95% CI, 0.43 to 0.92) and specificity of 0.99 (95% CI, 0.99 to 0.99). Post hoc modifications to the command prompt improved the integrated sensitivity to 0.91 (95% CI, 0.77 to 0.97) without substantially compromising specificity (0.98 [95% CI, 0.96 to 0.99]). Additionally, LLM-assisted screening was associated with reduced time for processing 100 studies (1.3 minutes vs 17.2 minutes for conventional screening methods; mean difference, \u221215.25 minutes [95% CI, \u221217.70 to \u221212.79 minutes]).<\/jats:p><\/jats:sec>Conclusions and Relevance<\/jats:title>In this prospective diagnostic study investigating the performance of LLM-assisted citation screening, the model demonstrated acceptable sensitivity and reasonably high specificity with reduced processing time. This novel method could potentially enhance efficiency and reduce workload in systematic reviews.<\/jats:p><\/jats:sec>","DOI":"10.1001\/jamanetworkopen.2024.20496","type":"journal-article","created":{"date-parts":[[2024,7,8]],"date-time":"2024-07-08T15:01:36Z","timestamp":1720450896000},"page":"e2420496","source":"Crossref","is-referenced-by-count":100,"title":["Performance of a Large Language Model in Screening Citations"],"prefix":"10.1001","volume":"7","author":[{"given":"Takehiko","family":"Oami","sequence":"first","affiliation":[{"name":"Department of Emergency and Critical Care Medicine, Chiba University Graduate School of Medicine, Chiba, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yohei","family":"Okada","sequence":"additional","affiliation":[{"name":"Department of Preventive Services, Kyoto University Graduate School of Medicine, Kyoto, Japan"},{"name":"Health Services and Systems Research, Duke-NUS Medical School, National University of Singapore, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Taka-aki","family":"Nakada","sequence":"additional","affiliation":[{"name":"Department of Emergency and Critical Care Medicine, Chiba University Graduate School of Medicine, Chiba, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"10","published-online":{"date-parts":[[2024,7,8]]},"reference":[{"issue":"2","key":"zoi240660r1","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2016-012545","article-title":"Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry.","volume":"7","author":"Borah","year":"2017","journal-title":"BMJ Open"},{"issue":"2","key":"zoi240660r2","doi-asserted-by":"publisher","first-page":"119","DOI":"10.1002\/jrsm.v2.2","article-title":"Precision of healthcare systematic review searches in a cross-sectional sample.","volume":"2","author":"Sampson","year":"2011","journal-title":"Res Synth Methods"},{"issue":"1","key":"zoi240660r3","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0227742","article-title":"Error rates of human reviewers during abstract screening in systematic reviews.","volume":"15","author":"Wang","year":"2020","journal-title":"PLoS One"},{"key":"zoi240660r4","doi-asserted-by":"publisher","first-page":"125","DOI":"10.1038\/s42256-020-00287-7","article-title":"An open source machine learning framework for efficient and transparent systematic reviews.","volume":"3","author":"van de Schoot","year":"2021","journal-title":"Nat Mach Intell"},{"issue":"1","key":"zoi240660r5","doi-asserted-by":"publisher","first-page":"163","DOI":"10.1186\/s13643-019-1074-9","article-title":"Toward systematic review automation: a practical guide to using machine learning tools in research synthesis.","volume":"8","author":"Marshall","year":"2019","journal-title":"Syst Rev"},{"issue":"1","key":"zoi240660r6","doi-asserted-by":"publisher","first-page":"7","DOI":"10.1186\/s12874-020-0897-3","article-title":"Software tools to support title and abstract screening for systematic reviews in healthcare: an evaluation.","volume":"20","author":"Harrison","year":"2020","journal-title":"BMC Med Res Methodol"},{"issue":"1","key":"zoi240660r7","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1186\/2046-4053-4-5","article-title":"Using text mining for study identification in systematic reviews: a systematic review of current approaches.","volume":"4","author":"O\u2019Mara-Eves","year":"2015","journal-title":"Syst Rev"},{"key":"zoi240660r8","doi-asserted-by":"publisher","first-page":"55","DOI":"10.1186\/1471-2105-11-55","article-title":"Semi-automated screening of biomedical citations for systematic reviews.","volume":"11","author":"Wallace","year":"2010","journal-title":"BMC Bioinformatics"},{"issue":"1","key":"zoi240660r9","doi-asserted-by":"publisher","first-page":"278","DOI":"10.1186\/s13643-019-1222-2","article-title":"Performance and usability of machine learning for screening in systematic reviews: a comparative evaluation of three tools.","volume":"8","author":"Gates","year":"2019","journal-title":"Syst Rev"},{"key":"zoi240660r10","doi-asserted-by":"publisher","DOI":"10.2188\/jea.JE20230227","article-title":"Efficiency and workload reduction of semi-automated citation screening software for creating clinical practice guidelines: a prospective observational study.","author":"Oami","year":"2023","journal-title":"J Epidemiol"},{"issue":"1","key":"zoi240660r11","doi-asserted-by":"publisher","first-page":"143","DOI":"10.1186\/s13643-019-1062-0","article-title":"A question of trust: can we build an evidence base to gain trust in systematic review automation technologies?","volume":"8","author":"O\u2019Connor","year":"2019","journal-title":"Syst Rev"},{"issue":"13","key":"zoi240660r12","doi-asserted-by":"publisher","first-page":"1201","DOI":"10.1056\/NEJMra2302038","article-title":"Artificial intelligence and machine learning in clinical medicine, 2023.","volume":"388","author":"Haug","year":"2023","journal-title":"N Engl J Med"},{"issue":"13","key":"zoi240660r13","doi-asserted-by":"publisher","first-page":"1233","DOI":"10.1056\/NEJMsr2214184","article-title":"Benefits, limits, and risks of GPT-4 as an AI chatbot for medicine.","volume":"388","author":"Lee","year":"2023","journal-title":"N Engl J Med"},{"issue":"7972","key":"zoi240660r14","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge.","volume":"620","author":"Singhal","year":"2023","journal-title":"Nature"},{"issue":"9","key":"zoi240660r15","doi-asserted-by":"publisher","first-page":"866","DOI":"10.1001\/jama.2023.14217","article-title":"Creation and adoption of large language models in medicine.","volume":"330","author":"Shah","year":"2023","journal-title":"JAMA"},{"key":"zoi240660r16","article-title":"Enhancing title and abstract screening for systematic reviews with GPT-3.5 turbo.","author":"Kohandel Gargari","journal-title":"BMJ Evid Based Med"},{"issue":"1","key":"zoi240660r17","doi-asserted-by":"publisher","first-page":"72","DOI":"10.1186\/s13643-023-02243-z","article-title":"Are ChatGPT and large language models \u201cthe answer\u201d to bringing us closer to systematic review automation?","volume":"12","author":"Qureshi","year":"2023","journal-title":"Syst Rev"},{"issue":"12","key":"zoi240660r19","doi-asserted-by":"publisher","first-page":"2629","DOI":"10.1007\/s10439-023-03272-4","article-title":"Prompt engineering with ChatGPT: a guide for academic writers.","volume":"51","author":"Giray","year":"2023","journal-title":"Ann Biomed Eng"},{"key":"zoi240660r20","doi-asserted-by":"publisher","DOI":"10.2196\/50638","article-title":"Prompt engineering as an important emerging skill for medical professionals: tutorial.","volume":"25","author":"Mesk\u00f3","year":"2023","journal-title":"J Med Internet Res"},{"issue":"3","key":"zoi240660r23","doi-asserted-by":"publisher","first-page":"177","DOI":"10.1016\/0197-2456(86)90046-2","article-title":"Meta-analysis in clinical trials.","volume":"7","author":"DerSimonian","year":"1986","journal-title":"Control Clin Trials"},{"key":"zoi240660r24","doi-asserted-by":"publisher","DOI":"10.2196\/51391","article-title":"Learning to make rare and complex diagnoses with generative AI assistance: qualitative study of popular large language models.","volume":"10","author":"Abdullahi","year":"2024","journal-title":"JMIR Med Educ"},{"issue":"4","key":"zoi240660r26","doi-asserted-by":"publisher","first-page":"608","DOI":"10.1002\/jrsm.v14.4","article-title":"A real-world evaluation of the implementation of NLP technology in abstract screening of a systematic review.","volume":"14","author":"Perlman-Arrow","year":"2023","journal-title":"Res Synth Methods"},{"issue":"1","key":"zoi240660r27","doi-asserted-by":"publisher","first-page":"45","DOI":"10.1186\/s13643-018-0707-8","article-title":"Technology-assisted title and abstract screening for systematic reviews: a retrospective evaluation of the Abstrackr machine learning tool.","volume":"7","author":"Gates","year":"2018","journal-title":"Syst Rev"},{"key":"zoi240660r22","doi-asserted-by":"crossref","DOI":"10.1002\/9781119536604","volume-title":"Cochrane Handbook for Systematic Reviews of Interventions version 6.0 (updated July 2019)","author":"Higgins","year":"2019"},{"key":"zoi240660r18","doi-asserted-by":"crossref","unstructured":"Oami? T, Okada? Y, Nakada? Ta. Citation screening using large language models for creating clinical practice guidelines: a protocol for a prospective study.? medRxiv. Preprint posted online December 31, 2023. doi:10.1101\/2023.12.29.23300652","DOI":"10.1101\/2023.12.29.23300652"},{"key":"zoi240660r21","unstructured":"GPT-assisted citation screening. GitHub. Accessed June 3, 2024. https:\/\/github.com\/seveneleven711thanks39\/gpt-assisted_citation_screening"},{"key":"zoi240660r25","unstructured":"Wang? X, Wei? J, Schuurmans? D, Le? Q, Chi? EH-h, Zhou? D. Self-consistency improves chain of thought reasoning in language models.? arXiv. Preprint updated March 7, 2023. doi:10.48550\/arXiv.2203.11171"}],"container-title":["JAMA Network Open"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/jamanetwork.com\/journals\/jamanetworkopen\/articlepdf\/2820861\/oami_2024_oi_240660_1719580325.87415.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,7,8]],"date-time":"2024-07-08T15:01:41Z","timestamp":1720450901000},"score":1,"resource":{"primary":{"URL":"https:\/\/jamanetwork.com\/journals\/jamanetworkopen\/fullarticle\/2820861"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,8]]},"references-count":27,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2024,7,1]]}},"URL":"https:\/\/doi.org\/10.1001\/jamanetworkopen.2024.20496","relation":{},"ISSN":["2574-3805"],"issn-type":[{"value":"2574-3805","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,7,8]]}}}