{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,1]],"date-time":"2026-08-01T00:44:18Z","timestamp":1785545058797,"version":"3.56.0"},"reference-count":47,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR AI"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Automated systematic literature review (SLR) may reduce the workload and errors associated with manual review, enabling faster, up-to-date reviews even with increasing publication volumes. Large language models (LLMs) have demonstrated strong capabilities in understanding unstructured languages. However, few studies have explored the potential of a comprehensive LLM platform to streamline the entire SLR process from article screening to data extraction.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aimed to investigate the feasibility of applying an LLM-based system to assist with SLR development.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n \n We developed the Intelligent Systematic Literature Review (ISLaR 2.0) platform, powered by an LLM, and applied it to a use case of the economic burden of pneumococcal disease (PD) literature. First, we established the inclusion and exclusion criteria for the SLR. Second, we defined data elements related to economic burden and domain knowledge, along with guidelines for applying these definitions. Finally, we used the criteria and data element specifications to develop LLM prompts for screening and data extraction. For data extraction, we identified relevant study characteristics and economic burden outcomes. We evaluated ISLaR 2.0\u2019s performance against a gold standard of 50 expert-curated PD articles, using standard metrics (accuracy, precision, recall, and\n F<\/jats:italic>\n 1<\/jats:sub>\n -score). We also conducted a qualitative analysis to describe errors made by the system.\n <\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n ISLaR 2.0 performed well in abstract and full-text screening (\n F<\/jats:italic>\n 1<\/jats:sub>\n -scores of 86.27 for abstract screening and 87.18 for full-text screening) and data extraction from text (\n F<\/jats:italic>\n 1<\/jats:sub>\n -scores of 92.83 for study details and 79.76 for economic burden outcomes). The\n F<\/jats:italic>\n 1<\/jats:sub>\n -score for data extraction of tabular economic burden outcome data was 94.83. The qualitative analysis revealed 2 main challenges in extracting economic burden details: misclassification of cost categories and failure to extract relevant information.\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n ISLaR 2.0 enabled efficient execution of an SLR regarding the economic burden of PD. The platform allowed users to flexibly define and modify criteria and data elements, supporting its use across a broad range of health research topics.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/81049","type":"journal-article","created":{"date-parts":[[2026,6,15]],"date-time":"2026-06-15T21:11:14Z","timestamp":1781557874000},"page":"e81049-e81049","source":"Crossref","is-referenced-by-count":0,"title":["AI-Assisted Systematic Literature Review of the Economic Burden of Pneumococcal Disease: Development and Validation Study"],"prefix":"10.2196","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0009-0003-3322-6284","authenticated-orcid":false,"given":"Dong","family":"Wang","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5634-2005","authenticated-orcid":false,"given":"Surabhi","family":"Datta","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-2552-9754","authenticated-orcid":false,"given":"Julie","family":"Glasgow","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6937-9931","authenticated-orcid":false,"given":"Kyeryoung","family":"Lee","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-9916-5654","authenticated-orcid":false,"given":"Hunki","family":"Paek","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-8918-5464","authenticated-orcid":false,"given":"Jun","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9553-5670","authenticated-orcid":false,"given":"Yi","family":"Zheng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5515-5080","authenticated-orcid":false,"given":"Yi-Ling","family":"Huang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-8842-1841","authenticated-orcid":false,"given":"Long","family":"He","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6962-3554","authenticated-orcid":false,"given":"Majid","family":"Rastegar-Mojarad","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0369-4186","authenticated-orcid":false,"given":"Kelsie","family":"Cassell","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4193-4120","authenticated-orcid":false,"given":"Xiaoyan","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-4982-4547","authenticated-orcid":false,"given":"Nicole","family":"Cossrow","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,6,15]]},"reference":[{"key":"R1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2021.03.001","article-title":"The PRISMA 2020 statement: an updated guideline for reporting systematic reviews","volume":"134","author":"Page","journal-title":"J Clin Epidemiol"},{"issue":"1","key":"R2","doi-asserted-by":"publisher","DOI":"10.1186\/s12874-017-0468-4","article-title":"What kind of systematic review should I conduct? A proposed typology and guidance for systematic reviewers in the medical and health sciences","volume":"18","author":"Munn","journal-title":"BMC Med Res Methodol"},{"key":"R3","doi-asserted-by":"publisher","DOI":"10.1016\/j.conctc.2019.100443","article-title":"The significant cost of systematic reviews and meta-analyses: a call for greater involvement of machine learning to assess the promise of clinical trials","volume":"16","author":"Michelson","journal-title":"Contemp Clin Trials Commun"},{"key":"R4","doi-asserted-by":"publisher","DOI":"10.1186\/2046-4053-3-74","article-title":"Systematic review automation technologies","volume":"3","author":"Tsafnat","journal-title":"Syst Rev"},{"issue":"9","key":"R5","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pmed.1000326","article-title":"Seventy-five trials and eleven systematic reviews a day: how will we ever keep up?","volume":"7","author":"Bastian","journal-title":"PLoS Med"},{"issue":"2","key":"R6","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2016-012545","article-title":"Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry","volume":"7","author":"Borah","journal-title":"BMJ Open"},{"key":"R7","doi-asserted-by":"crossref","unstructured":"Wang S Potthast M Scells H Koopman B Zhuang S Zuccon G et al. Zero-shot generative large language models for systematic review screening automation. In: Goharian N Tonellotto N He Y Lipani A McDonald G Macdonald C Ounis I , editors. Advances in Information Retrieval. Springer; 2024:403-420.","DOI":"10.1007\/978-3-031-56027-9_25"},{"key":"R8","doi-asserted-by":"publisher","DOI":"10.2196\/48996","article-title":"Automated paper screening for clinical reviews using large language models: data analysis study","volume":"26","author":"Guo","journal-title":"J Med Internet Res"},{"issue":"1","key":"R9","doi-asserted-by":"publisher","DOI":"10.1186\/s13643-024-02609-x","article-title":"Evaluating the effectiveness of large language models in abstract screening: a comparative analysis","volume":"13","author":"Li","journal-title":"Syst Rev"},{"issue":"4","key":"R10","doi-asserted-by":"publisher","first-page":"616","DOI":"10.1002\/jrsm.1715","article-title":"Can large language models replace humans in systematic reviews? Evaluating GPT-4\u2019s efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languages","volume":"15","author":"Khraisha","journal-title":"Res Synth Methods"},{"issue":"4","key":"R11","doi-asserted-by":"publisher","first-page":"576","DOI":"10.1002\/jrsm.1710","article-title":"Data extraction for evidence synthesis using a large language model: a proof-of-concept study","volume":"15","author":"Gartlehner","journal-title":"Res Synth Methods"},{"issue":"1","key":"R12","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-01840-7","article-title":"Accelerating clinical evidence synthesis with large language models","volume":"8","author":"Wang","journal-title":"NPJ Digit Med"},{"issue":"1","key":"R13","doi-asserted-by":"publisher","DOI":"10.1186\/s12874-024-02224-3","article-title":"Machine learning models for abstract screening task - a systematic literature review application for health economics and outcome research","volume":"24","author":"Du","journal-title":"BMC Med Res Methodol"},{"issue":"9","key":"R14","doi-asserted-by":"publisher","first-page":"1049","DOI":"10.1080\/14737167.2023.2234639","article-title":"Can artificial intelligence separate the wheat from the chaff in systematic reviews of health economic articles?","volume":"23","author":"Oude Wolcherink","journal-title":"Expert Rev Pharmacoecon Outcomes Res"},{"key":"R15","doi-asserted-by":"publisher","DOI":"10.2196\/54653","article-title":"Accelerating evidence synthesis in observational studies: development of a living natural language processing-assisted intelligent systematic literature review system","volume":"12","author":"Manion","journal-title":"JMIR Med Inform"},{"issue":"4","key":"R16","doi-asserted-by":"publisher","first-page":"874","DOI":"10.1080\/21645515.2018.1564444","article-title":"Invasive pneumococcal disease in relation to vaccine type serotypes","volume":"15","author":"Wantuch","journal-title":"Hum Vaccin Immunother"},{"issue":"1","key":"R17","doi-asserted-by":"publisher","DOI":"10.1186\/s41479-022-00097-y","article-title":"The remarkable history of pneumococcal vaccination: an ongoing challenge","volume":"14","author":"Musher","journal-title":"Pneumonia (Nathan)"},{"issue":"2","key":"R18","doi-asserted-by":"publisher","first-page":"135","DOI":"10.1007\/s10719-023-10100-3","article-title":"Strengths and weaknesses of pneumococcal conjugate vaccines","volume":"40","author":"Micoli","journal-title":"Glycoconj J"},{"issue":"1","key":"R19","doi-asserted-by":"publisher","DOI":"10.1186\/s12913-018-3529-4","article-title":"Trends in healthcare utilization and costs associated with pneumonia in the United States during 2008-2014","volume":"18","author":"Tong","journal-title":"BMC Health Serv Res"},{"issue":"4","key":"R20","doi-asserted-by":"publisher","first-page":"362","DOI":"10.1001\/jamapediatrics.2018.5273","article-title":"Long-term association of 13-valent pneumococcal conjugate vaccine implementation with rates of community-acquired pneumonia in children","volume":"173","author":"Ouldali","journal-title":"JAMA Pediatr"},{"issue":"3","key":"R21","doi-asserted-by":"publisher","DOI":"10.3390\/vaccines8030387","article-title":"Impact of the 13-valent conjugated pneumococcal vaccine on the direct costs of invasive pneumococcal disease requiring hospital admission in children aged <5 years: a prospective study","volume":"8","author":"Hern\u00e1ndez","journal-title":"Vaccines (Basel)"},{"issue":"5","key":"R22","doi-asserted-by":"publisher","DOI":"10.1080\/21645515.2022.2046433","article-title":"Health and economic burden associated with 15-valent pneumococcal conjugate vaccine serotypes in Korea and Hong Kong","volume":"18","author":"Mohanty","journal-title":"Hum Vaccin Immunother"},{"issue":"30","key":"R23","doi-asserted-by":"publisher","first-page":"3733","DOI":"10.1016\/j.vaccine.2017.05.055","article-title":"Decrease in the incidence and in hospital mortality of community-acquired pneumonia among children in Spain (2001-2014)","volume":"35","author":"Jimenez Trujillo","journal-title":"Vaccine"},{"issue":"4","key":"R24","doi-asserted-by":"publisher","first-page":"541","DOI":"10.1093\/cid\/cix850","article-title":"Impact of pneumococcal vaccination on pneumonia hospitalizations and related costs in Ontario: a population-based ecological study","volume":"66","author":"Luca","journal-title":"Clin Infect Dis"},{"issue":"Supplement 3","key":"R25","doi-asserted-by":"publisher","first-page":"C26","DOI":"10.1016\/j.vaccine.2011.06.104","article-title":"Prevenar experience","volume":"29 Suppl 3","author":"Arguedas","journal-title":"Vaccine (Auckl)"},{"issue":"2","key":"R26","doi-asserted-by":"publisher","first-page":"187","DOI":"10.1097\/QCO.0000000000000246","article-title":"Pneumococcal vaccination","volume":"29","author":"Cill\u00f3niz","journal-title":"Curr Opin Infect Dis"},{"issue":"5","key":"R27","doi-asserted-by":"publisher","first-page":"540","DOI":"10.1016\/j.jnma.2019.04.011","article-title":"Racial disparities in adult pneumococcal vaccination indications and pneumococcal hospitalizations in the U.S","volume":"111","author":"Nowalk","journal-title":"J Natl Med Assoc"},{"issue":"27","key":"R28","doi-asserted-by":"publisher","first-page":"3608","DOI":"10.1016\/j.vaccine.2021.05.037","article-title":"Is further research on adult pneumococcal vaccine uptake improvement programs worthwhile? \u0391 value of information analysis","volume":"39","author":"Wateska","journal-title":"Vaccine"},{"issue":"3","key":"R29","first-page":"1","volume":"12","author":"Sehatzadeh","journal-title":"Ont Health Technol Assess Ser"},{"issue":"37","key":"R30","doi-asserted-by":"publisher","first-page":"3950","DOI":"10.1016\/j.vaccine.2013.06.037","article-title":"Cost-effectiveness of pneumococcal conjugate vaccination in immunocompromised adults","volume":"31","author":"Smith","journal-title":"Vaccine"},{"key":"R31","doi-asserted-by":"publisher","DOI":"10.1186\/s12913-016-1432-4","article-title":"Rates and costs of invasive pneumococcal disease and pneumonia in persons with underlying medical conditions","volume":"16","author":"Weycker","journal-title":"BMC Health Serv Res"},{"issue":"1","key":"R32","doi-asserted-by":"publisher","DOI":"10.1186\/s13054-020-03442-z","article-title":"Burden of pneumococcal pneumonia requiring ICU admission in France: 1-year prognosis, resources use, and costs","volume":"25","author":"Dupuis","journal-title":"Crit Care"},{"issue":"5","key":"R33","doi-asserted-by":"publisher","first-page":"473","DOI":"10.1001\/jama.1981.03310300027014","article-title":"A cost-benefit analysis of immunization for pneumococcal pneumonia","volume":"245","author":"Patrick","journal-title":"JAMA"},{"issue":"22","key":"R34","doi-asserted-by":"publisher","first-page":"2910","DOI":"10.1001\/jama.264.22.2910","article-title":"Pneumococcal vaccine. Efficacy and associated cost savings","volume":"264","author":"Gable","journal-title":"JAMA"},{"issue":"4","key":"R35","doi-asserted-by":"publisher","first-page":"392","DOI":"10.1007\/s12325-014-0115-y","article-title":"Cost-effectiveness of pneumococcal vaccines for adults in the United States","volume":"31","author":"Chen","journal-title":"Adv Ther"},{"issue":"11","key":"R36","doi-asserted-by":"publisher","first-page":"1389","DOI":"10.1017\/S0950268818000262","article-title":"Estimating the burden on general practitioner services in England from increases in respiratory disease associated with seasonal respiratory pathogen activity","volume":"146","author":"Morbey","journal-title":"Epidemiol Infect"},{"key":"R37","doi-asserted-by":"publisher","DOI":"10.1016\/j.ymeth.2024.04.005","article-title":"AlpaPICO: extraction of PICO frames from clinical trial documents using LLMs","volume":"226","author":"Ghosh","journal-title":"Methods"},{"key":"R38","doi-asserted-by":"publisher","DOI":"10.1016\/j.zefq.2023.06.007","article-title":"A narrative review of recent tools and innovations toward automating living systematic reviews and evidence syntheses","volume":"181","author":"Schmidt","journal-title":"Z Evid Fortbild Qual Gesundhwes"},{"issue":"12","key":"R39","doi-asserted-by":"publisher","first-page":"1903","DOI":"10.1093\/jamia\/ocaa163","article-title":"Trialstreamer: a living, automatically updated database of clinical trial reports","volume":"27","author":"Marshall","journal-title":"J Am Med Inform Assoc"},{"key":"R40","doi-asserted-by":"publisher","DOI":"10.1186\/s13643-016-0263-z","article-title":"SWIFT-Review: a text-mining workbench for systematic review","volume":"5","author":"Howard","journal-title":"Syst Rev"},{"issue":"7","key":"R41","doi-asserted-by":"publisher","DOI":"10.1016\/j.jval.2022.04.1277","article-title":"MSR70 pilot study to evaluate efficiency of DISTILLERSR\u00ae\u2019S artificial intelligence (AI) tool over manual screening process in literature review","volume":"25","author":"Kamra","journal-title":"Value Health"},{"issue":"12","key":"R42","doi-asserted-by":"publisher","DOI":"10.1016\/j.jval.2023.09.2044","article-title":"HTA361 Living systematic review (LSR) in health technology assessment (HTA): current guidance, methods, and challenges","volume":"26","author":"Sauca","journal-title":"Value Health"},{"key":"R43","doi-asserted-by":"publisher","DOI":"10.1016\/j.envint.2020.105623","article-title":"SWIFT-Active Screener: accelerated document screening through active learning and integrated recall estimation","volume":"138","author":"Howard","journal-title":"Environ Int"},{"key":"R44","doi-asserted-by":"crossref","unstructured":"Wallace BC Small K Brodley CE Lau J Trikalinos TA . Deploying an interactive machine learning system in an evidence-based practice center. Presented at: IHI \u201912: ACM International Health Informatics Symposium; Jan 28-30, 2012; Miami, FL. [doi: 10.1145\/2110363.2110464]","DOI":"10.1145\/2110363.2110464"},{"key":"R45","unstructured":"Thomas J Graziosi S Brunton J Ghouze Z O\u2019Driscoll P Bond M et al. EPPI-Reviewer: advanced software for systematic reviews, maps and evidence synthesis. EPPI Centre, UCL Social Research Institute, University College London. 2023. URL: https:\/\/eppi.ioe.ac.uk\/cms\/resources\/tools\/eppi-reviewer-introduction [Accessed 08-05-2026]"},{"issue":"1","key":"R46","doi-asserted-by":"publisher","first-page":"193","DOI":"10.1093\/jamia\/ocv044","article-title":"RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials","volume":"23","author":"Marshall","journal-title":"J Am Med Inform Assoc"},{"key":"R47","doi-asserted-by":"publisher","DOI":"10.3389\/frai.2025.1662202","article-title":"Analysis of article screening and data extraction performance by an AI systematic literature review platform","volume":"8","author":"Cassell","journal-title":"Front Artif Intell"}],"container-title":["JMIR AI"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/ai.jmir.org\/2026\/1\/e81049","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T22:26:35Z","timestamp":1785536795000},"score":1,"resource":{"primary":{"URL":"https:\/\/ai.jmir.org\/2026\/1\/e81049"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,15]]},"references-count":47,"URL":"https:\/\/doi.org\/10.2196\/81049","relation":{},"ISSN":["2817-1705"],"issn-type":[{"value":"2817-1705","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,15]]},"article-number":"v5i1e81049"}}