{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,20]],"date-time":"2026-08-20T18:24:53Z","timestamp":1787250293459,"version":"build-2736575974"},"reference-count":47,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Form Res"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Literature reviews rely on rigorous title and abstract screening by researchers, which is time-consuming. AI-assisted literature screening tools have been proposed to improve efficiency by prioritizing titles and abstracts with the highest likelihood of meeting the inclusion criteria, thereby reducing the need to screen all records.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This study aims to evaluate the performance of two AI-assisted screening approaches in ASReview (version 1.3; Department of Methodology and Statistics, Utrecht University) compared with manual title and abstract screening in a previously completed and published scoping review on how AI can support the quality of life in people with dementia.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n This study used a dataset of 4690 titles and abstracts from a published scoping review. The manual screening decisions of the scoping review served as the reference standard. Both ASReview approaches were applied by the same author who conducted the majority of the original manual title and abstract screening. Approach A used a simpler model with minimal prior input, whereas approach B used a more advanced model with a larger training set. Both ASReview approaches applied predefined stopping rules: (1) more than 10% of the dataset to be screened; and (2) 50 consecutive irrelevant titles and abstracts. Performance was evaluated in terms of sensitivity, specificity, precision, accuracy, and screening time. 95% CIs were calculated for sensitivity, specificity, precision, and accuracy. Agreement between manual and ASReview approaches was assessed using the Cohen \u03ba, and differences in how manual and both ASReview approaches classified titles and abstracts were examined using the McNemar test. Performance and agreement were calculated at two levels: (1) after title and abstract screening and (2) after full-text inclusion.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n \n Manual screening identified 283 titles and abstracts for full-text review and resulted in 30 final included studies, requiring 19 hours. Of the 4690 titles and abstracts, approach A screened 830 (17.7%) in 4.3 hours and retrieved 16 of the 30 (sensitivity 0.53, 95% CI 0.36\u20100.70) final included studies, whereas approach B screened 798 (17.0%) in 5.5 hours and retrieved 21 of the 30 (sensitivity 0.70, 95% CI 0.52\u20100.83) final included studies. Although both ASReview approaches showed high specificity and accuracy, these metrics should be interpreted cautiously because the dataset was highly imbalanced and contained relatively few relevant titles and abstracts. McNemar tests showed significant directional imbalance (\n P<\/jats:italic>\n <.001): ASReview missed more manually selected titles and abstracts at level 1, whereas at level 2, ASReview more often labeled titles and abstracts not included in the final review as relevant.\n <\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n ASReview can support workload reduction in title and abstract screening, but the evaluated ASReview approaches did not retrieve all final included studies from the original dementia care scoping review. These findings suggest that the evaluated ASReview configurations may be insufficient for reviews in which near-complete retrieval of relevant evidence is required.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/84989","type":"journal-article","created":{"date-parts":[[2026,8,20]],"date-time":"2026-08-20T17:41:14Z","timestamp":1787247674000},"page":"e84989-e84989","source":"Crossref","is-referenced-by-count":0,"title":["Performance of Two AI Approaches in ASReview Compared With Manual Screening for Dementia Care Literature Screening: Comparative Analysis"],"prefix":"10.2196","volume":"10","author":[{"ORCID":"https:\/\/orcid.org\/0009-0004-0889-2194","authenticated-orcid":false,"given":"Dirk","family":"Steijger","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-2736-7375","authenticated-orcid":false,"given":"Stella","family":"Thissen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3121-412X","authenticated-orcid":false,"given":"Sil","family":"Aarts","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3740-5162","authenticated-orcid":false,"given":"Hilde","family":"Verbeek","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2113-4134","authenticated-orcid":false,"given":"Marjolein E","family":"de Vugt","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2838-2202","authenticated-orcid":false,"given":"Hannah","family":"Christie","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,8,20]]},"reference":[{"key":"R1","unstructured":"Higgins JPT Thomas J Chandler J et al, editors. Cochrane Handbook for Systematic Reviews of Interventions. Wiley; 2019. URL: https:\/\/dariososafoula.wordpress.com\/wp-content\/uploads\/2017\/01\/cochrane-handbook-for-systematic-reviews-of-interventions-2019-1.pdf [Accessed 22-07-2026]"},{"issue":"3","key":"R2","doi-asserted-by":"publisher","first-page":"330","DOI":"10.1002\/jrsm.1354","article-title":"Best practice guidelines for abstract screening large\u2010evidence systematic reviews and meta\u2010analyses","volume":"10","author":"Polanin","journal-title":"Res Synth Methods"},{"key":"R3","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2021.05.019","article-title":"Resource use during systematic review production varies widely: a scoping review","volume":"139","author":"Nussbaumer-Streit","journal-title":"J Clin Epidemiol"},{"issue":"3","key":"R4","doi-asserted-by":"publisher","first-page":"353","DOI":"10.1002\/jrsm.1553","article-title":"Using artificial intelligence methods for systematic review in health sciences: a systematic review","volume":"13","author":"Blaizot","journal-title":"Res Synth Methods"},{"issue":"1","key":"R5","doi-asserted-by":"publisher","first-page":"870","DOI":"10.51519\/journalisi.v7i1.1035","article-title":"Evaluating the efficacy of AI tools in systematic literature reviews: a comprehensive analysis","volume":"7","author":"Mogoale","journal-title":"J Inf Syst Inform"},{"key":"R6","unstructured":"van de Schoot R de Bruin J Schram R et al. Open source software for efficient and transparent reviews. Preprint posted online on Jun 22, 2020. [doi: 10.48550\/arXiv.2006.12166]"},{"issue":"1","key":"R7","doi-asserted-by":"publisher","DOI":"10.1186\/s13643-024-02682-2","article-title":"Leveraging artificial intelligence to enhance systematic reviews in health research: advanced tools and challenges","volume":"13","author":"Ge","journal-title":"Syst Rev"},{"key":"R8","doi-asserted-by":"publisher","DOI":"10.2196\/48996","article-title":"Automated paper screening for clinical reviews using large language models: data analysis study","volume":"26","author":"Guo","journal-title":"J Med Internet Res"},{"issue":"1","key":"R9","doi-asserted-by":"publisher","DOI":"10.1186\/s13643-015-0067-6","article-title":"Faster title and abstract screening? Evaluating Abstrackr, a semi-automated online screening program for systematic reviewers","volume":"4","author":"Rathbone","journal-title":"Syst Rev"},{"issue":"1","key":"R10","doi-asserted-by":"publisher","DOI":"10.1186\/s12874-020-01129-1","article-title":"An evaluation of DistillerSR\u2019s machine learning-based prioritization tool for title\/abstract screening - impact on reviewer-relevant outcomes","volume":"20","author":"Hamel","journal-title":"BMC Med Res Methodol"},{"issue":"2","key":"R11","doi-asserted-by":"publisher","first-page":"68","DOI":"10.5596\/c14-016","article-title":"Product review: covidence (systematic review software)","volume":"35","author":"Babineau","journal-title":"J Can Health Libr Assoc"},{"issue":"1","key":"R12","doi-asserted-by":"publisher","DOI":"10.1186\/s13643-016-0384-4","article-title":"Rayyan-a web and mobile app for systematic reviews","volume":"5","author":"Ouzzani","journal-title":"Syst Rev"},{"issue":"7","key":"R13","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2023-072254","article-title":"Artificial intelligence in systematic reviews: promising when appropriately used","volume":"13","author":"van Dijk","journal-title":"BMJ Open"},{"issue":"4","key":"R14","doi-asserted-by":"publisher","DOI":"10.1136\/rmdopen-2024-005024","article-title":"Is AI-assisted active learning software able to reliably speed-up systematic literature reviews in rheumatology? A real-time comparison of AI-assisted and manual abstract selection","volume":"10","author":"van der Pol","journal-title":"RMD Open"},{"issue":"2","key":"R15","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2411962122","article-title":"Transforming literature screening: the emerging role of large language models in systematic reviews","volume":"122","author":"Delgado-Chaves","journal-title":"Proc Natl Acad Sci U S A"},{"key":"R16","doi-asserted-by":"crossref","unstructured":"Akaraci S Jones SM Tate C et al. Evaluating the use of artificial intelligence (AI) in systematic review abstract screening: a comparative study of AI-aided tools. Research Square. Preprint posted online on Jan 21, 2026. [doi: 10.21203\/rs.3.rs-7915337\/v1]","DOI":"10.21203\/rs.3.rs-7915337\/v1"},{"key":"R17","doi-asserted-by":"publisher","DOI":"10.1080\/13645579.2026.2668099","article-title":"Manual versus AI-assisted document screening (ASReview): a comparative analysis within a rapid systematized review in the social sciences","author":"Gauthier Mongeon","journal-title":"Int J Soc Res Methodol"},{"key":"R18","unstructured":"Kataoka Y Banno M Kyo M et al. TiAb review plugin: a browser-based tool for AI-assisted title and abstract screening. arXiv. Preprint posted online on Apr 8, 2026. [doi: 10.48550\/arXiv.2604.08602]"},{"key":"R19","doi-asserted-by":"crossref","unstructured":"Boesen K Hemkens L Janiaud P Hirt J . Machine-learning assisted screening for evidence synthesis: a case study of using the asreview tool. SSRN. Preprint posted online on Feb 6, 2025. [doi: 10.2139\/ssrn.5125681]","DOI":"10.2139\/ssrn.5125681"},{"issue":"9","key":"R20","doi-asserted-by":"publisher","first-page":"1049","DOI":"10.1080\/14737167.2023.2234639","article-title":"Can artificial intelligence separate the wheat from the chaff in systematic reviews of health economic articles?","volume":"23","author":"Oude Wolcherink","journal-title":"Expert Rev Pharmacoecon Outcomes Res"},{"key":"R21","doi-asserted-by":"publisher","DOI":"10.1016\/j.jdent.2024.105259","article-title":"Assessing the article screening efficiency of artificial intelligence for systematic reviews","volume":"149","author":"Chan","journal-title":"J Dent"},{"key":"R22","unstructured":"Scherhag J Burgard T . Performance of semi-automated screening using rayyan and asreview: a retrospective analysis of potential work reduction and different stopping rules. PsychArchives. Preprint posted online on May 3, 2023. [doi: 10.23668\/psycharchives.12843]"},{"key":"R23","doi-asserted-by":"publisher","DOI":"10.1177\/01655515251379058","article-title":"Simulation-based active learning for systematic reviews: a scoping review of literature","author":"Teijema","journal-title":"J Inf Sci"},{"key":"R24","unstructured":"Navigating the maze of models in ASReview. ASReview. 2025. URL: https:\/\/asreview.nl\/blog\/asreview-model-selection-guide\/ [Accessed 22-07-2026]"},{"issue":"1","key":"R25","doi-asserted-by":"publisher","DOI":"10.1186\/s13643-024-02502-7","article-title":"The SAFE procedure: a practical stopping heuristic for active learning-based screening in systematic reviews and meta-analyses","volume":"13","author":"Boetje","journal-title":"Syst Rev"},{"key":"R26","doi-asserted-by":"publisher","DOI":"10.1016\/j.arr.2025.102741","article-title":"Use of artificial intelligence to support quality of life of people with dementia: a scoping review","volume":"108","author":"Steijger","journal-title":"Ageing Res Rev"},{"key":"R27","doi-asserted-by":"publisher","DOI":"10.2196\/80247","article-title":"Transparent reporting of AI in systematic literature reviews: development of the PRISMA-trAIce checklist","volume":"4","author":"Holst","journal-title":"JMIR AI"},{"issue":"5","key":"R28","doi-asserted-by":"publisher","first-page":"513","DOI":"10.1016\/0306-4573(88)90021-0","article-title":"Term-weighting approaches in automatic text retrieval","volume":"24","author":"Salton","journal-title":"Inf Process Manag"},{"key":"R29","unstructured":"McCallum A Nigam K . A comparison of event models for naive bayes text classification. Presented at: AAAI-98 Workshop on Learning for Text Categorization; Jul 27, 1998; Madison, Wisconsin, USA. URL: https:\/\/cdn.aaai.org\/Workshops\/1998\/WS-98-05\/WS98-05-007.pdf [Accessed 22-07-2026]"},{"issue":"2","key":"R30","doi-asserted-by":"publisher","first-page":"125","DOI":"10.1038\/s42256-020-00287-7","article-title":"An open source machine learning framework for efficient and transparent systematic reviews","volume":"3","author":"van de Schoot","journal-title":"Nat Mach Intell"},{"issue":"1","key":"R31","doi-asserted-by":"publisher","first-page":"37","DOI":"10.1177\/001316446002000104","article-title":"A coefficient of agreement for nominal scales","volume":"20","author":"Cohen","journal-title":"Educ Psychol Meas"},{"issue":"2","key":"R32","doi-asserted-by":"publisher","first-page":"153","DOI":"10.1007\/BF02295996","article-title":"Note on the sampling error of the difference between correlated proportions or percentages","volume":"12","author":"McNEMAR","journal-title":"Psychometrika"},{"issue":"1","key":"R33","doi-asserted-by":"publisher","DOI":"10.1007\/s10648-024-09862-5","article-title":"Screening smarter, not harder: a comparative analysis of machine learning screening algorithms and heuristic stopping criteria for systematic reviews in educational research","volume":"36","author":"Campos","journal-title":"Educ Psychol Rev"},{"issue":"2","key":"R34","doi-asserted-by":"publisher","first-page":"101","DOI":"10.1214\/ss\/1009213286","article-title":"Interval estimation for a binomial proportion","volume":"16","author":"Brown","journal-title":"Statist Sci"},{"issue":"3","key":"R35","doi-asserted-by":"publisher","first-page":"136","DOI":"10.7142\/igakutoshokan.72.3_136","article-title":"Expanding the feasibility of systematic reviews with AI support: a practical case using ASReview","volume":"72","author":"Rai","journal-title":"Igaku Toshokan"},{"key":"R36","doi-asserted-by":"publisher","DOI":"10.1016\/j.canep.2023.102511","article-title":"Evaluating the efficacy of artificial intelligence tools for the automation of systematic reviews in cancer research: a systematic review","volume":"88","author":"Yao","journal-title":"Cancer Epidemiol"},{"issue":"6","key":"R37","doi-asserted-by":"publisher","first-page":"5435","DOI":"10.1007\/s41060-025-00777-0","article-title":"Large-scale simulation study of active learning models for systematic reviews","volume":"20","author":"Teijema","journal-title":"Int J Data Sci Anal"},{"issue":"1","key":"R38","doi-asserted-by":"publisher","DOI":"10.1186\/s12874-026-02866-5","article-title":"When to stop reviewing: validation of stop criteria in ASReview","volume":"26","author":"Kempny","journal-title":"BMC Med Res Methodol"},{"issue":"1","key":"R39","doi-asserted-by":"publisher","DOI":"10.1186\/2046-4053-4-5","article-title":"Using text mining for study identification in systematic reviews: a systematic review of current approaches","volume":"4","author":"O\u2019Mara-Eves","journal-title":"Syst Rev"},{"issue":"5","key":"R40","doi-asserted-by":"publisher","first-page":"893","DOI":"10.1093\/jamia\/ocaf050","article-title":"High-performance automated abstract screening with large language model ensembles","volume":"32","author":"Sanghera","journal-title":"J Am Med Inform Assoc"},{"issue":"10","key":"R41","doi-asserted-by":"publisher","DOI":"10.1002\/14651858.ED000142","article-title":"Updated guidance for trusted systematic reviews: a new edition of the Cochrane Handbook for Systematic Reviews of Interventions","volume":"10","author":"Cumpston","journal-title":"Cochrane Database Syst Rev"},{"key":"R42","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2020.01.005","article-title":"Single-reviewer abstract screening missed 13 percent of relevant studies: a crowd-based, randomized controlled trial","volume":"121","author":"Gartlehner","journal-title":"J Clin Epidemiol"},{"issue":"2","key":"R43","doi-asserted-by":"crossref","first-page":"363","DOI":"10.2307\/2529786","volume":"33","author":"Landis","journal-title":"Biometrics"},{"key":"R44","doi-asserted-by":"publisher","DOI":"10.2174\/1874434601711010211","article-title":"High agreement and high prevalence: the paradox of Cohen\u2019s kappa","volume":"11","author":"Zec","journal-title":"Open Nurs J"},{"key":"R45","unstructured":"Agresti A . Categorical Data Analysis. John Wiley & Sons; 2013. ISBN: 978-0-470-46363-5"},{"issue":"1","key":"R46","doi-asserted-by":"publisher","DOI":"10.1186\/s13750-025-00374-5","article-title":"Position statement on artificial intelligence (AI) use in evidence synthesis across Cochrane, the Campbell Collaboration, JBI and the Collaboration for Environmental Evidence 2025","volume":"14","author":"Flemyng","journal-title":"Environ Evid"},{"issue":"1","key":"R47","doi-asserted-by":"publisher","DOI":"10.1186\/s13643-023-02257-7","article-title":"Performance of active learning models for screening prioritization in systematic reviews: a simulation study into the average time to discover relevant records","volume":"12","author":"Ferdinands","journal-title":"Syst Rev"}],"container-title":["JMIR Formative Research"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,8,20]],"date-time":"2026-08-20T17:41:18Z","timestamp":1787247678000},"score":1,"resource":{"primary":{"URL":"https:\/\/formative.jmir.org\/2026\/1\/e84989"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,20]]},"references-count":47,"URL":"https:\/\/doi.org\/10.2196\/84989","relation":{},"ISSN":["2561-326X"],"issn-type":[{"value":"2561-326X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,20]]},"article-number":"v10i3e84989"}}