{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,7]],"date-time":"2026-08-07T17:26:27Z","timestamp":1786123587999,"version":"build-2736575974"},"reference-count":36,"publisher":"The Journal of Health Economics and Outcomes Research","issue":"2","license":[{"start":{"date-parts":[[2026,8,7]],"date-time":"2026-08-07T00:00:00Z","timestamp":1786060800000},"content-version":"unspecified","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["jheor.org"],"crossmark-restriction":true},"short-container-title":[],"abstract":"<jats:sec id=\"background\">\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Systematic literature reviews (SLRs) are central to evidence generation in health economics and outcomes research, but traditional SLR workflows are labor-intensive. Artificial intelligence (AI) is increasingly being explored to support evidence synthesis tasks.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"objectives\">\n                    <jats:title>Objectives<\/jats:title>\n                    <jats:p>To describe methods used to assess AI performance in screening and data extraction in SLRs, and to provide recommendations on how to ensure appropriate use of AI in literature reviews.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"methods\">\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>We conducted an AI-assisted SLR that mirrored a traditional SLR performed by humans only and assessed the performance of the AI tools employed. AI was used to screen titles\/abstracts, screen full texts, and conduct data extraction. Using the traditional SLR as the benchmark, AI performance for title\/abstract screening was evaluated in terms of accuracy, recall, and precision at predefined screening milestones, followed by the accuracy assessment of full-text screening by AI. Data items extracted by AI were verified by humans and categorized as correct, incomplete, missing, incorrect, or requiring human checks. The average accuracy rate calculated on the basis of correct data items extracted was used as an indicator of AI performance in data extraction.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec id=\"results\">\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>During title\/abstract screening, accuracy and recall remained high but precision was low, increasing false positives. Full-text screening identified a minority of studies included in the traditional SLR. Mean extraction accuracy was 72.93% (range, 57.69%-88.46%). No data were hallucinated by the AI model used.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.36469\/001c.165173","type":"journal-article","created":{"date-parts":[[2026,8,7]],"date-time":"2026-08-07T16:11:08Z","timestamp":1786119068000},"update-policy":"https:\/\/doi.org\/10.36469\/00001","source":"Crossref","is-referenced-by-count":0,"title":["Evaluating AI Performance in Systematic Literature Reviews for HEOR: A Case Study"],"prefix":"10.36469","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0009-0009-8669-1415","authenticated-orcid":true,"given":"Jing","family":"Wang-Silvanto","sequence":"first","affiliation":[{"name":"Astellas Pharma A\/S, filial I Finland, Espoo; Observational Health Data Sciences and Informatics (OHDSI), Finland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mansee","family":"Jajoo","sequence":"additional","affiliation":[{"name":"Astellas Pharma, Astellas Pharma Global Development Inc. Northbrook, Illinois, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"He","family":"Guo","sequence":"additional","affiliation":[{"name":"Astellas Pharma, Astellas Pharma Global Development Inc. Northbrook, Illinois, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rishi","family":"Ohri","sequence":"additional","affiliation":[{"name":"Astellas Pharma, Astellas Pharma Global Development Inc. Northbrook, Illinois, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Judith","family":"Nelissen","sequence":"additional","affiliation":[{"name":"Astellas Pharma Europe BV, Leiden, The Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Carolina","family":"Casa\u00f1as i Comabella","sequence":"additional","affiliation":[{"name":"PPD Evidera Health Economics & Market Access, Thermo Fisher Scientific, London, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"22273","published-online":{"date-parts":[[2026,8,7]]},"reference":[{"issue":"11","key":"ref1","doi-asserted-by":"publisher","first-page":"1665","DOI":"10.1016\/j.jval.2025.07.001","article-title":"Role of generative artificial intelligence in assisting systematic review process in health research: a systematic review","volume":"28","author":"M. Rashid","year":"2025","journal-title":"Value Health"},{"issue":"23-24","key":"ref2","doi-asserted-by":"publisher","first-page":"e70263","DOI":"10.1002\/sim.70263","article-title":"ChatGPT as a tool for biostatisticians: a tutorial on applications, opportunities, and limitations","volume":"44","author":"D. Dobler","year":"2025","journal-title":"Stat Med"},{"issue":"6","key":"ref3","doi-asserted-by":"publisher","first-page":"e70058","DOI":"10.1002\/cesm.70058","article-title":"Harnessing large-language models for efficient data extraction in systematic reviews: the role of prompt engineering","volume":"3","author":"M. Murton","year":"2025","journal-title":"Cochrane Evid Synth Methods"},{"issue":"7","key":"ref4","doi-asserted-by":"publisher","first-page":"3851","DOI":"10.1080\/10447318.2024.2344142","article-title":"A systematic review of the limitations and associated opportunities of ChatGPT","volume":"41","author":"N. Cong-Lem","year":"2025","journal-title":"Int J Hum Comput Interact"},{"issue":"10","key":"ref5","doi-asserted-by":"publisher","first-page":"ED000178","DOI":"10.1002\/14651858.ED000178","article-title":"Position statement on artificial intelligence (AI) use in evidence synthesis across Cochrane, the Campbell Collaboration, JBI and the Collaboration for Environmental Evidence 2025","author":"E. Flemyng","year":"2025","journal-title":"Cochrane Database Syst Rev"},{"issue":"5","key":"ref6","doi-asserted-by":"publisher","first-page":"e33219","DOI":"10.2196\/33219","article-title":"Web-based software tools for systematic literature review in medicine: systematic search and feature analysis","volume":"10","author":"K. Cowie","year":"2022","journal-title":"JMIR Med Inform"},{"key":"ref7","doi-asserted-by":"publisher","first-page":"22","DOI":"10.1016\/j.jclinepi.2021.12.005","article-title":"Tools to support the automation of systematic reviews: a scoping review","volume":"144","author":"H. Khalil","year":"2022","journal-title":"J Clin Epidemiol"},{"issue":"3","key":"ref8","doi-asserted-by":"publisher","first-page":"e70100","DOI":"10.1111\/jep.70100","article-title":"Digital tools to support the systematic review process: an introduction","volume":"31","author":"L. Schmidt","year":"2025","journal-title":"J Eval Clin Pract"},{"issue":"1","key":"ref9","doi-asserted-by":"publisher","first-page":"7","DOI":"10.1186\/s12874-020-0897-3","article-title":"Software tools to support title and abstract screening for systematic reviews in healthcare: an evaluation","volume":"20","author":"H. Harrison","year":"2020","journal-title":"BMC Med Res Methodol"},{"key":"ref10","doi-asserted-by":"publisher","first-page":"140","DOI":"10.1016\/j.jclinepi.2020.11.003","article-title":"Machine learning reduced workload with minimal risk of missing studies: development and evaluation of a randomized controlled trial classifier for Cochrane Reviews","volume":"133","author":"J. Thomas","year":"2021","journal-title":"J Clin Epidemiol"},{"key":"ref11","article-title":"Use of AI in evidence generation: NICE position statement","author":"National Institute for Health and Care Excellence","year":"2024"},{"issue":"11","key":"ref12","doi-asserted-by":"publisher","first-page":"1655","DOI":"10.1016\/j.jval.2025.08.002","article-title":"A4SLR: An agentic artificial intelligence-assisted systematic literature review framework to augment evidence synthesis for health economics and outcomes research and health technology assessment","volume":"28","author":"K. Lee","year":"2025","journal-title":"Value Health"},{"key":"ref13","doi-asserted-by":"publisher","first-page":"e80247","DOI":"10.2196\/80247","article-title":"Transparent reporting of AI in systematic literature reviews: development of the PRISMA-trAIce checklist","volume":"4","author":"D. Holst","year":"2025","journal-title":"JMIR AI"},{"key":"ref14","article-title":"Canada's Drug Agency position statement on the use of artificial intelligence in the generation and reporting of evidence","author":"Canada's Drug Agency","year":"2025"},{"issue":"11","key":"ref15","doi-asserted-by":"publisher","first-page":"1611","DOI":"10.1016\/j.jval.2025.06.018","article-title":"ELEVATE-GenAI: reporting guidelines for the use of large language models in health economics and outcomes research: an ISPOR Working Group Report","volume":"28","author":"R. L. Fleurence","year":"2025","journal-title":"Value Health"},{"key":"ref16","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2308.06610","article-title":"Bio-sieve: exploring instruction tuning large language models for systematic review automation","author":"A. Robinson","year":"2023","journal-title":"arXiv"},{"issue":"6 suppl 1","key":"ref17","doi-asserted-by":"publisher","first-page":"S306","DOI":"10.1016\/j.jval.2025.04.2141","article-title":"PT3 an artificial intelligence (AI)-assisted systematic literature review (SLR) of the economic burden in metastatic pancreatic adenocarcinoma: a proof-of-concept study","volume":"28","author":"C. Casanas i Comabella","year":"2025","journal-title":"Value Health"},{"key":"ref18","article-title":"Adaptive Smart Tags","author":"Nested Knowledge","year":"2025"},{"issue":"1","key":"ref19","doi-asserted-by":"publisher","first-page":"69","DOI":"10.1136\/bmjebm-2023-112678","article-title":"Enhancing title and abstract screening for systematic reviews with GPT-3.5 turbo","volume":"29","author":"O. K. Gargari","year":"2024","journal-title":"BMJ Evid Based Med"},{"issue":"6","key":"ref20","doi-asserted-by":"publisher","first-page":"791","DOI":"10.7326\/M23-3389","article-title":"Sensitivity and specificity of using GPT-3.5 turbo models for title and abstract screening in systematic reviews and meta-analyses","volume":"177","author":"V. T. Tran","year":"2024","journal-title":"Ann Intern Med"},{"issue":"1","key":"ref21","doi-asserted-by":"publisher","first-page":"e53164","DOI":"10.2196\/53164","article-title":"Hallucination rates and reference accuracy of ChatGPT and Bard for systematic reviews: comparative analysis","volume":"26","author":"M. Chelli","year":"2024","journal-title":"J Med Internet Res"},{"key":"ref22","article-title":"Oxford Centre for Evidence-Based Medicine: levels of evidence (March 2009)","author":"Oxford Centre for Evidence-Based Medicine","year":"2009"},{"issue":"1","key":"ref23","doi-asserted-by":"publisher","first-page":"78","DOI":"10.1186\/s12874-024-02203-8","article-title":"Methodological insights into ChatGPT's screening performance in systematic reviews","volume":"24","author":"M. Issaiy","year":"2024","journal-title":"BMC Med Res Methodol"},{"key":"ref24","article-title":"Responsible use of AI in evidence SynthEsis (RAISE) 2: building and evaluating AI evidence synthesis tools","author":"J. Thomas","year":"2025","journal-title":"Open Science Framework"},{"issue":"6 suppl 1","key":"ref25","doi-asserted-by":"publisher","first-page":"S295","DOI":"10.1016\/j.jval.2025.04.1257","article-title":"MSR105 Stability of a large language model for data extraction in systematic literature reviews","volume":"28","author":"A. Shree","year":"2025","journal-title":"Value Health"},{"key":"ref26","doi-asserted-by":"publisher","first-page":"n71","DOI":"10.1136\/bmj.n71","article-title":"The PRISMA 2020 statement: an updated guideline for reporting systematic reviews","volume":"372","author":"M. J. Page","year":"2021","journal-title":"BMJ"},{"issue":"8","key":"ref27","doi-asserted-by":"publisher","first-page":"e0329349","DOI":"10.1371\/journal.pone.0329349","article-title":"An open-source interactive AI framework for assisting automatic literature review in forensic medicine: focus on brain injury mechanisms","volume":"20","author":"Y. W. Liu","year":"2025","journal-title":"PLoS One"},{"issue":"8","key":"ref28","doi-asserted-by":"publisher","first-page":"3360","DOI":"10.1007\/s12325-023-02527-9","article-title":"Artificial intelligence in head and neck cancer: a systematic review of systematic reviews","volume":"40","author":"A. A. Makitie","year":"2023","journal-title":"Adv Ther"},{"key":"ref29","article-title":"Cochrane launches innovative study to assess AI tools for evidence synthesis","author":"M. Powell","year":"2026","journal-title":"Cochrane Collaboration"},{"key":"ref30","article-title":"Cochrane Handbook for Systematic Reviews of Interventions","author":"J. P. T. Higgins","year":"2024"},{"key":"ref31","doi-asserted-by":"publisher","first-page":"100443","DOI":"10.1016\/j.conctc.2019.100443","article-title":"The significant cost of systematic reviews and meta-analyses: a call for greater involvement of machine learning to assess the promise of clinical trials","volume":"16","author":"M. Michelson","year":"2019","journal-title":"Contemp Clin Trials Commun"},{"issue":"2","key":"ref32","doi-asserted-by":"publisher","first-page":"e012545","DOI":"10.1136\/bmjopen-2016-012545","article-title":"Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry","volume":"7","author":"R. Borah","year":"2017","journal-title":"BMJ Open"},{"key":"ref33","doi-asserted-by":"publisher","first-page":"78","DOI":"10.1186\/s13643-015-0066-7","article-title":"Automating data extraction in systematic reviews: a systematic review","volume":"4","author":"S. R. Jonnalagadda","year":"2015","journal-title":"Syst Rev"},{"key":"ref34","doi-asserted-by":"publisher","first-page":"80","DOI":"10.1016\/j.jclinepi.2021.06.030","article-title":"Systematic review automation tools improve efficiency but lack of knowledge impedes their adoption: a survey","volume":"138","author":"A. M. Scott","year":"2021","journal-title":"J Clin Epidemiol"},{"issue":"suppl 3","key":"ref35","doi-asserted-by":"publisher","first-page":"S367","DOI":"10.1016\/j.jval.2018.09.2193","article-title":"PRM72 - Performance of automated screening of citations compared to human reviewers in systematic literature reviews: a systematic literature review","volume":"21","author":"R. Lopes","year":"2018","journal-title":"Value Health"},{"issue":"1","key":"ref36","doi-asserted-by":"publisher","first-page":"269","DOI":"10.1186\/s13643-024-02682-2","article-title":"Leveraging artificial intelligence to enhance systematic reviews in health research: advanced tools and challenges","volume":"13","author":"L. Ge","year":"2024","journal-title":"Syst Rev"}],"container-title":["Journal of Health Economics and Outcomes Research"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/jheor.org\/article\/165173-evaluating-ai-performance-in-systematic-literature-reviews-for-heor-a-case-study","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,7]],"date-time":"2026-08-07T16:11:09Z","timestamp":1786119069000},"score":1,"resource":{"primary":{"URL":"https:\/\/jheor.org\/article\/165173-evaluating-ai-performance-in-systematic-literature-reviews-for-heor-a-case-study"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,7]]},"references-count":36,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2026]]}},"URL":"https:\/\/doi.org\/10.36469\/001c.165173","relation":{},"ISSN":["2327-2236"],"issn-type":[{"value":"2327-2236","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,7]]}}}