{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:48:11Z","timestamp":1782841691046,"version":"3.54.5"},"reference-count":41,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Res Protoc"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Artificial intelligence (AI), including large language models (LLMs), is increasingly integrated into systematic review (SR) workflows. AI tools may accelerate searching, screening, data extraction, and reporting, but their effects on methodological quality, reporting completeness, transparency, and reproducibility remain uncertain. Existing evaluations largely examine isolated tasks, and inconsistent disclosure of AI use limits reproducibility and oversight.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This 4-phase mixed methods meta-research study will (1) compare the methodological quality of AI-assisted versus traditional SRs; (2) refine, finalize, and apply a preliminary AI Transparency and Disclosure Index (AITDI); (3) evaluate reproducibility by comparing outputs across repeated runs of the same AI model, across different AI models, and between AI models and human reviewers at multiple SR stages; and (4) explore knowledge user perspectives on rigor, transparency, and trust in AI-assisted SRs.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n We will conduct a matched cohort analysis of SRs published from 2023 to 2025 in biomedical journals. Each AI-assisted SR will be matched 1:2 with traditional SRs by publication year, clinical domain, review type, and meta-analysis status. Two independent reviewers will apply A Measurement Tool to Assess Systematic Reviews, version 2 (AMSTAR 2; methodological quality), PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) 2020 (reporting completeness), and, when applicable, Risk of Bias in SRs (ROBIS; risk-of-bias rigor). A preliminary AITDI will be refined and then applied to all AI-assisted SRs. Reproducibility will be assessed using SR-derived task sets to compare outputs across repeated runs of the same model, across different models, and between AI and human reviewers at key SR stages. Semistructured interviews with authors, editors, clinicians, policymakers, and patient partners will be analyzed using reflexive thematic analysis.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n As of December 2025, the study has been preregistered on the Open Science Framework (OSF; DOI: 10.17605\/OSF.IO\/Q5JRW), the search strategy has been finalized, and title\/abstract screening has begun. Data extraction is planned for March-May 2026, followed by AITDI refinement and reproducibility testing from May 2026 to October 2026. Qualitative interviews are anticipated from October 2026 to February 2027, with final analyses by April 2027 and dissemination planned for mid-2027.<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n This study will provide one of the first empirical comparisons of methodological quality, transparency, and reproducibility of AI-assisted versus traditional SRs in the LLM era. Findings will inform expectations for responsible AI integration and support refinement of reporting and methodological best practices, including future development of AI-specific reporting and appraisal extensions (eg, PRISMA-LLM [Preferred Reporting Items for Systematic Reviews and Meta-Analyses\u2013large language model] and AMSTAR-LLM [A Measurement Tool to Assess Systematic Reviews-large language model]).<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/90588","type":"journal-article","created":{"date-parts":[[2026,3,17]],"date-time":"2026-03-17T13:50:06Z","timestamp":1773755406000},"page":"e90588-e90588","source":"Crossref","is-referenced-by-count":0,"title":["Evaluating the Methodological Quality of Artificial Intelligence\u2013Assisted Systematic Reviews: Protocol for a Mixed Methods Meta-Research Study"],"prefix":"10.2196","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0009-0002-7299-4342","authenticated-orcid":false,"given":"Mohammad","family":"Jay","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-5171-4932","authenticated-orcid":false,"given":"Mary","family":"Morgan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6106-832X","authenticated-orcid":false,"given":"Sharon Elizabeth","family":"Straus","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-1462-885X","authenticated-orcid":false,"given":"Emma","family":"Wilson","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0320-6042","authenticated-orcid":false,"given":"Rinku","family":"Sutradhar","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2355-1046","authenticated-orcid":false,"given":"Catherine","family":"Yu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3769-5590","authenticated-orcid":false,"given":"Lesley","family":"Gotlib Conn","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5223-5755","authenticated-orcid":false,"given":"Christoffer","family":"Dharma","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7412-7993","authenticated-orcid":false,"given":"Lorraine","family":"Lipscombe","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4326-1393","authenticated-orcid":false,"given":"Antoine","family":"Eskander","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,5,14]]},"reference":[{"issue":"1","key":"R1","doi-asserted-by":"publisher","DOI":"10.56570\/jimgs.v2i1.76","article-title":"Systematic review methodology: conducting high-quality reviews and understanding their significance in evidence-based practice","volume":"2","journal-title":"J Int Med Graduates"},{"issue":"6","key":"R2","doi-asserted-by":"publisher","first-page":"359","DOI":"10.1701\/4042.40229","article-title":"Evidence synthesis 2.0: how artificial intelligence is making systematic reviews more efficient","volume":"114","author":"Petri","journal-title":"Recenti Prog Med"},{"issue":"1","key":"R3","doi-asserted-by":"publisher","DOI":"10.1186\/s12911-025-03138-w","article-title":"A comparative study of screening performance between abstrackr and GPT models: systematic review and contextual analysis","volume":"25","author":"Xu","journal-title":"BMC Med Inform Decis Mak"},{"issue":"1","key":"R4","doi-asserted-by":"publisher","DOI":"10.1186\/s13643-016-0384-4","article-title":"Rayyan-a web and mobile app for systematic reviews","volume":"5","author":"Ouzzani","journal-title":"Syst Rev"},{"issue":"1","key":"R5","doi-asserted-by":"publisher","DOI":"10.1186\/s13643-019-1221-3","article-title":"Assessing the accuracy of machine-assisted abstract screening with DistillerAI: a user study","volume":"8","author":"Gartlehner","journal-title":"Syst Rev"},{"issue":"12","key":"R6","doi-asserted-by":"publisher","first-page":"1416","DOI":"10.4103\/ija.ija_1008_25","article-title":"Automated tools in systematic reviews: current trends","volume":"69","author":"Maurya","journal-title":"Indian J Anaesth"},{"key":"R7","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2022.04.027","article-title":"Artificial intelligence in COVID-19 evidence syntheses was underutilized, but impactful: a methodological study","volume":"148","author":"Tercero-Hidalgo","journal-title":"J Clin Epidemiol"},{"issue":"8","key":"R8","doi-asserted-by":"publisher","first-page":"460","DOI":"10.22038\/ABJS.2025.84896.3874","article-title":"Evaluation of a popular large language model in orthopedic literature review: comparison to previously published reviews","volume":"13","author":"Yao","journal-title":"Arch Bone Jt Surg"},{"key":"R9","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2024.111533","article-title":"Benchmarking human-AI collaboration for common evidence appraisal tools","volume":"175","author":"Woelfle","journal-title":"J Clin Epidemiol"},{"issue":"1","key":"R10","doi-asserted-by":"publisher","DOI":"10.1186\/s12874-021-01451-2","article-title":"Guidance for using artificial intelligence for title and abstract screening while conducting knowledge syntheses","volume":"21","author":"Hamel","journal-title":"BMC Med Res Methodol"},{"issue":"1","key":"R11","doi-asserted-by":"publisher","first-page":"63","DOI":"10.1017\/rsm.2025.10032","article-title":"Exploring the methodological quality and risk of bias in 200 systematic reviews: a comparative study of ROBIS and AMSTAR-2 tools","volume":"17","author":"Lunny","journal-title":"Res Synth Methods"},{"key":"R12","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2015.06.005","article-title":"ROBIS: a new tool to assess risk of bias in systematic reviews was developed","volume":"69","author":"Whiting","journal-title":"J Clin Epidemiol"},{"key":"R13","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.n71","article-title":"The PRISMA 2020 statement: an updated guideline for reporting systematic reviews","volume":"372","author":"Page","journal-title":"BMJ"},{"key":"R14","doi-asserted-by":"crossref","unstructured":"Movva R Balachandar S Peng K Agostini G Garg N Pierson E . Topics, authors, and institutions in large language model research: trends from 17K arxiv papers. Presented at: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics; Jun 16-21, 2024; Mexico City, Mexico. URL: https:\/\/aclanthology.org\/2024.naacl-long [Accessed 06-05-2026] [doi: 10.18653\/v1\/2024.naacl-long.67]","DOI":"10.18653\/v1\/2024.naacl-long.67"},{"issue":"2","key":"R15","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pmed.1000217","article-title":"Guidance for developers of health research reporting guidelines","volume":"7","author":"Moher","journal-title":"PLoS Med"},{"issue":"2","key":"R16","doi-asserted-by":"publisher","first-page":"195","DOI":"10.1093\/oxfordjournals.aje.a009786","article-title":"Factors influencing the optimal control-to-case ratio in matched case-control studies","volume":"149","author":"Hennessy","journal-title":"Am J Epidemiol"},{"key":"R17","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.j4008","article-title":"AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both","volume":"358","author":"Shea","journal-title":"BMJ"},{"key":"R18","unstructured":"SR \/ MA \/ HTA \/ ITC \u2013 CINAHL canada\u2019s drug agency. Canada\u2019s Drug Agency. URL: https:\/\/www.cda-amc.ca\/ [Accessed 08-12-2025]"},{"key":"R19","unstructured":"Blackwood D . Peer review of electronic search strategies (PRESS): \u201cCan you check my systematic review search strategy. HLA News; 2015:9-10. URL: https:\/\/search.informit.org\/doi\/abs\/10.3316\/informit.314342355541565 [Accessed 06-05-2026]"},{"issue":"1","key":"R20","doi-asserted-by":"publisher","DOI":"10.1186\/s13643-024-02682-2","article-title":"Leveraging artificial intelligence to enhance systematic reviews in health research: advanced tools and challenges","volume":"13","author":"Ge","journal-title":"Syst Rev"},{"issue":"2","key":"R21","first-page":"e49","volume":"2","author":"Altman","journal-title":"Open Med"},{"issue":"1","key":"R22","doi-asserted-by":"publisher","DOI":"10.1186\/s13643-020-01542-z","article-title":"PRISMA-S: an extension to the PRISMA statement for reporting literature searches in systematic reviews","volume":"10","author":"Rethlefsen","journal-title":"Syst Rev"},{"issue":"1","key":"R23","doi-asserted-by":"publisher","DOI":"10.1186\/s13063-020-04951-6","article-title":"Reporting guidelines for clinical trials of artificial intelligence interventions: the SPIRIT-AI and CONSORT-AI guidelines","volume":"22","author":"Ibrahim","journal-title":"Trials"},{"issue":"2","key":"R24","doi-asserted-by":"publisher","first-page":"83","DOI":"10.56012\/ugaa9986","article-title":"ICMJE recommendations update: what\u2019s new?","volume":"33","author":"Dyer","journal-title":"Med Writ"},{"issue":"6","key":"R25","doi-asserted-by":"publisher","first-page":"390","DOI":"10.1136\/bmjebm-2025-113825","article-title":"Reporting guideline for the use of generative artificial intelligence tools in medical research: the GAMER Statement","volume":"30","author":"Luo","journal-title":"BMJ Evid Based Med"},{"key":"R26","doi-asserted-by":"publisher","DOI":"10.2196\/80247","article-title":"Transparent reporting of AI in systematic literature reviews: development of the PRISMA-trAIce checklist","volume":"4","author":"Holst","journal-title":"JMIR AI"},{"key":"R27","doi-asserted-by":"publisher","DOI":"10.1016\/j.zefq.2022.04.025","article-title":"Reporting guidelines for Delphi techniques in health sciences: a methodological review","volume":"172","author":"Spranger","journal-title":"Z Evid Fortbild Qual Gesundhwes"},{"issue":"1","key":"R28","doi-asserted-by":"publisher","DOI":"10.1186\/s12913-023-09618-x","article-title":"Assessing the sustainability and scalability of a diabetes eHealth innovation: a mixed-methods study","volume":"23","author":"Sivakumar","journal-title":"BMC Health Serv Res"},{"issue":"6","key":"R29","doi-asserted-by":"publisher","first-page":"859","DOI":"10.1017\/rsm.2025.10014","article-title":"Optimal large language models to screen citations for systematic reviews","volume":"16","author":"Oami","journal-title":"Res Synth Methods"},{"key":"R30","doi-asserted-by":"publisher","DOI":"10.2196\/53164","article-title":"Hallucination rates and reference accuracy of ChatGPT and Bard for systematic reviews: comparative analysis","volume":"26","author":"Chelli","journal-title":"J Med Internet Res"},{"key":"R31","unstructured":"Son G Ko H Lee H Kim Y Hong S . LLM-as-a-judge and reward model: what they can and cannot do. arXiv. Preprint posted online on Sep 17, 2024. [doi: 10.48550\/arXiv.2409.11239]"},{"key":"R32","doi-asserted-by":"publisher","DOI":"10.1177\/16094069221077758","article-title":"Constructivist grounded theory or interpretive phenomenology? methodological choices within specific study contexts","volume":"21","author":"Burns","journal-title":"Int J Qual Methods"},{"issue":"7","key":"R33","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2023-072254","article-title":"Artificial intelligence in systematic reviews: promising when appropriately used","volume":"13","author":"van Dijk","journal-title":"BMJ Open"},{"issue":"3","key":"R34","doi-asserted-by":"publisher","first-page":"1386","DOI":"10.20319\/pijss.2019.43.13861394","article-title":"Research paradigm and the philosophical foundations of a qualitative study","volume":"4","author":"Abdullah Kamal","journal-title":"PEOPLE Int J Soc Sci"},{"key":"R35","doi-asserted-by":"publisher","DOI":"10.1016\/j.glmedi.2024.100171","article-title":"Sample size for saturation in qualitative research: debates, definitions, and strategies","volume":"5","author":"Ahmed","journal-title":"J Med Surg Public Health"},{"key":"R36","doi-asserted-by":"publisher","DOI":"10.1177\/16094069241229777","article-title":"Demystification and actualisation of data saturation in qualitative research through thematic analysis","volume":"23","author":"Naeem","journal-title":"Int J Qual Methods"},{"key":"R37","doi-asserted-by":"publisher","DOI":"10.1093\/swr\/svy026","article-title":"An examination of the sufficiency of small qualitative samples","author":"Young","journal-title":"Soc Work Res"},{"key":"R38","doi-asserted-by":"crossref","unstructured":"Braun V Clarke V Hayfield N Davey L Jenkinson E . Doing reflexive thematic analysis. In: Supporting Research in Counselling and Psychotherapy: Qualitative, Quantitative, and Mixed Methods Research. Springer; 2023:19-38. [doi: 10.1007\/978-3-031-13942-0_2] ISBN: 978-3-031-13941-3","DOI":"10.1007\/978-3-031-13942-0_2"},{"key":"R39","doi-asserted-by":"crossref","unstructured":"Booth A Hannes K Harden A Noyes J Harris J Tong A . COREQ (consolidated criteria for reporting qualitative studies). In: Moher D Altman DG Schulz KF Simera I Wager E , editors. Guidelines for Reporting Health Research: A User\u2019s Manual. 2014:214-226. [doi: 10.1002\/9781118715598]","DOI":"10.1002\/9781118715598.ch21"},{"key":"R40","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.j3453","article-title":"GRIPP2 reporting checklists: tools to improve reporting of patient and public involvement in research","volume":"358","author":"Staniszewska","journal-title":"BMJ"},{"issue":"4","key":"R41","doi-asserted-by":"publisher","first-page":"380","DOI":"10.1016\/j.jclinepi.2008.08.008","article-title":"Non-Cochrane vs. Cochrane reviews were twice as likely to have positive conclusion statements: cross-sectional study","volume":"62","author":"Tricco","journal-title":"J Clin Epidemiol"}],"container-title":["JMIR Research Protocols"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.researchprotocols.org\/2026\/1\/e90588","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:15:38Z","timestamp":1782839738000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.researchprotocols.org\/2026\/1\/e90588"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,5,14]]},"references-count":41,"URL":"https:\/\/doi.org\/10.2196\/90588","relation":{},"ISSN":["1929-0748"],"issn-type":[{"value":"1929-0748","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,5,14]]},"article-number":"v15i1e90588"}}