{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,23]],"date-time":"2026-06-23T00:00:08Z","timestamp":1782172808797,"version":"3.54.5"},"reference-count":41,"publisher":"Wiley","issue":"6","license":[{"start":{"date-parts":[[2025,10,25]],"date-time":"2025-10-25T00:00:00Z","timestamp":1761350400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2025,10,25]],"date-time":"2025-10-25T00:00:00Z","timestamp":1761350400000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/doi.wiley.com\/10.1002\/tdm_license_1.1"}],"content-domain":{"domain":["onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["Cochrane Evidence Synthesis and Methods"],"published-print":{"date-parts":[[2025,11]]},"abstract":"<jats:title>ABSTRACT<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Introduction<\/jats:title>\n                    <jats:p>While artificial intelligence (AI) tools have been utilized for individual stages within the systematic literature review (SLR) process, no tool has previously been shown to support each critical SLR step. In addition, the need for expert oversight has been recognized to ensure the quality of SLR findings. Here, we describe a complete methodology for utilizing our AI SLR tool with human\u2010in\u2010the\u2010loop curation workflows, as well as AI validations, time savings, and approaches to ensure compliance with best review practices.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Methods<\/jats:title>\n                    <jats:p>SLRs require completing Search, Screening, and Extraction from relevant studies, with meta\u2010analysis and critical appraisal as relevant. We present a full methodological framework for completing SLRs utilizing our AutoLit software (Nested Knowledge). This system integrates AI models into the central steps in SLR: Search strategy generation, Dual Screening of Titles\/Abstracts and Full Texts, and Extraction of qualitative and quantitative evidence. The system also offers manual Critical Appraisal and Insight drafting and fully\u2010automated Network Meta\u2010analysis. Validations comparing AI performance to experts are reported, and where relevant, time savings and \u2018rapid review\u2019 alternatives to the SLR workflow.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>Search strategy generation with the Smart Search AI can turn a Research Question into full Boolean strings with 76.8% and 79.6% Recall in two validation sets. Supervised machine learning tools can achieve 82\u201397% Recall in reviewer\u2010level Screening. Population, Interventions\/Comparators, and Outcomes (PICOs) extraction achieved F1 of 0.74; accuracy for study type, location, and size were 74%, 78%, and 91%, respectively. Time savings of 50% in Abstract Screening and 70\u201380% in qualitative extraction were reported. Extraction of user\u2010specified qualitative and quantitative tags and data elements remains exploratory and requires human curation for SLRs.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>AI systems can support high\u2010quality, human\u2010in\u2010the\u2010loop execution of key SLR stages. Transparency, replicability, and expert oversight are central to the use of AI SLR tools.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1002\/cesm.70059","type":"journal-article","created":{"date-parts":[[2025,10,25]],"date-time":"2025-10-25T08:00:31Z","timestamp":1761379231000},"update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["RETRACTED: Human\u2010in\u2010the\u2010Loop Artificial Intelligence System for Systematic Literature Review: Methods and Validations for the AutoLit Review Software"],"prefix":"10.1002","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5439-4074","authenticated-orcid":false,"given":"Kevin M.","family":"Kallmes","sequence":"first","affiliation":[{"name":"Nested Knowledge St. Paul Minnesota USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jade","family":"Thurnham","sequence":"additional","affiliation":[{"name":"Nested Knowledge St. Paul Minnesota USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marius","family":"Sauca","sequence":"additional","affiliation":[{"name":"Nested Knowledge St. Paul Minnesota USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ranita","family":"Tarchand","sequence":"additional","affiliation":[{"name":"Nested Knowledge St. Paul Minnesota USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Keith R.","family":"Kallmes","sequence":"additional","affiliation":[{"name":"Nested Knowledge St. Paul Minnesota USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Karl J.","family":"Holub","sequence":"additional","affiliation":[{"name":"Nested Knowledge St. Paul Minnesota USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"311","published-online":{"date-parts":[[2025,10,25]]},"reference":[{"key":"e_1_2_15_2_1","doi-asserted-by":"publisher","DOI":"10.1186\/s13643-024-02682-2"},{"key":"e_1_2_15_3_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2025.111746"},{"key":"e_1_2_15_4_1","doi-asserted-by":"publisher","DOI":"10.1186\/s12911-020-01332-6"},{"key":"e_1_2_15_5_1","unstructured":"National Institute for Health and Care Excellence (NICE). Use of AI in Evidence Generation\u2013 NICE Position Statement. NICE. Published October 2023 accessed May 29 2025 https:\/\/www.nice.org.uk\/about\/what-we-do\/our-research-work\/use-of-ai-in-evidence-generation--nice-position-statement."},{"key":"e_1_2_15_6_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jval.2013.08.1236"},{"key":"e_1_2_15_7_1","unstructured":"Guidance on Review Types. Nested Knowledge accessed May 29 2025 https:\/\/about.nested-knowledge.com\/docs\/guidance-on-review-types\/."},{"key":"e_1_2_15_8_1","unstructured":"J.Thurnham K.Holub J.Johnson et al. AutoLit Documentation. Nested Knowledge. Last edited May 15 2025 accessed May 29 2025 https:\/\/about.nested-knowledge.com\/docs\/autolit\/."},{"key":"e_1_2_15_9_1","unstructured":"J.Twaites K.Holub J.Johnson et al. Model Cards. Nested Knowledge. Last edited January 17 2025 accessed May 29 2025 https:\/\/about.nested-knowledge.com\/docs-category\/model-cards\/."},{"key":"e_1_2_15_10_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jval.2022.03.022"},{"key":"e_1_2_15_11_1","unstructured":"Crossref. Crossref Metadata Search. Crossref. Published 2023 accessed May 30 2025 https:\/\/search.crossref.org."},{"key":"e_1_2_15_12_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10032-015-0249-8"},{"key":"e_1_2_15_13_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2017.08.010"},{"key":"e_1_2_15_14_1","unstructured":"Cochrane Training. Artificial Intelligence Technologies in Cochrane. Cochrane. Published January 18 2023 accessed May 29 2025 https:\/\/training.cochrane.org\/resource\/artificial-intelligence-technologies-in-cochrane\/."},{"key":"e_1_2_15_15_1","unstructured":"ISPOR. Revolutionizing Systematic Reviews: Harnessing the Power of AI. ISPOR. Published May 20 2024 accessed May 29 2025 https:\/\/www.ispor.org\/education-training\/webinars\/webinar\/revolutionizing-systematic-reviews--harnessing-the-power-of-ai\/."},{"key":"e_1_2_15_16_1","unstructured":"R.Fleurence J.Bian X.Wang et al. Generative AI for Health Technology Assessment: Opportunities Challenges and Policy Considerations. arXiv. Published July 9 2024 accessed May 29 2025 https:\/\/arxiv.org\/abs\/2407.11054."},{"key":"e_1_2_15_17_1","unstructured":"National Institute for Health and Care Excellence (NICE). Developing NICE Guidelines: The Manual (PMG20). NICE. Published October 31 2014. Last updated May 29 2024 accessed May 29 2025 https:\/\/www.nice.org.uk\/process\/pmg20\/."},{"issue":"1","key":"e_1_2_15_18_1","first-page":"P22","article-title":"Assessment of Reasoning Agents for Building Literature Search Strategies","volume":"28","author":"Twaites J.","year":"2025","journal-title":"Value in Health"},{"key":"e_1_2_15_19_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jval.2024.03.1524"},{"issue":"1","key":"e_1_2_15_20_1","first-page":"SA72","article-title":"Repeatable Auto\u2010Extraction Frameworks in Clinical Systematic Literature Review: Validating a Multi\u2010Model Human\u2010In\u2010The\u2010Loop Artificial Intelligence System for Extracting Study PICOs, Location, Size, and Type","volume":"28","author":"Twaites J.","year":"2025","journal-title":"Value in Health"},{"key":"e_1_2_15_21_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.bionlp-1.16"},{"key":"e_1_2_15_22_1","doi-asserted-by":"publisher","DOI":"10.1007\/11495772_68"},{"key":"e_1_2_15_23_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0227742"},{"key":"e_1_2_15_24_1","unstructured":"T.Li J. P. T.Higgins andJ. J.DeeksChapter 5: Collecting data. In: Higgins JPT Thomas J Chandler J Cumpston M Li T Page MJ Welch VA eds. Cochrane Handbook for Systematic Reviews of Interventions. Version 6.5. Cochrane; 2024 accessed May 31 2025 https:\/\/training.cochrane.org\/handbook\/current\/chapter-05."},{"key":"e_1_2_15_25_1","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.n71"},{"key":"e_1_2_15_26_1","doi-asserted-by":"crossref","unstructured":"T.ChenandC.GuestrinXGBoost: A scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD' 16). ACM;2016:785\u2013794 https:\/\/doi.org\/10.1145\/2939672.2939785.","DOI":"10.1145\/2939672.2939785"},{"key":"e_1_2_15_27_1","unstructured":"M.SaucaandK. J.HolubAI for Screening: How Is It Used and how can We Tell how Accurate It Is? Nested Knowledge https:\/\/about.nested-knowledge.com\/2024\/08\/02\/ai-for-screening-how-is-it-used-and-how-can-we-tell-how-accurate-it-is\/. Published August 2 2024 accessed May 31 2025."},{"key":"e_1_2_15_28_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jval.2024.03.1455"},{"key":"e_1_2_15_29_1","unstructured":"K. M.KallmesTag Tables Explained. Nested Knowledge https:\/\/about.nested-knowledge.com\/2023\/07\/10\/tag-tables-explained. Published July 10 2023 accessed May 31 2025."},{"key":"e_1_2_15_30_1","doi-asserted-by":"crossref","unstructured":"B.Nye J. J.Li R.Patel et al. A Corpus With Multi\u2010level Annotations of Patients Interventions and Outcomes to Support Language Processing for Medical Literature. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics;2018:197\u2013207 https:\/\/doi.org\/10.18653\/v1\/P18-1019.","DOI":"10.18653\/v1\/P18-1019"},{"key":"e_1_2_15_31_1","unstructured":"A.Ciapponi A.Bardach D.Glujovsky andR.TarchandThe Efficiency of \u201cNested Knowledge\u201d to Facilitate the Conduction of Systematic Reviews and Meta\u2010analyses. Poster Presented at: Cochrane Global Evidence Summit; September 4\u20136 2024; Prague Czech Republic https:\/\/abstracts.cochrane.org\/2024-prague-global-evidence-summit\/efficiency-nested-knowledge-facilitate-conduction-systematic."},{"key":"e_1_2_15_32_1","unstructured":"Y.ShinyamaPDFMiner: Python PDF Parser and Analyzer. Released June 29 2008 accessed May 31 2025 https:\/\/pypi.org\/project\/pdfminer\/."},{"key":"e_1_2_15_33_1","unstructured":"K. J.HolubShukra: Tools for Meta\u2010analytical Statistics Including Network Meta\u2010analysis (NMA) in NodeJS. Version 3.2.4. Published May 16 2023 accessed May 31 2025 https:\/\/github.com\/holub008\/shukra."},{"key":"e_1_2_15_34_1","doi-asserted-by":"publisher","DOI":"10.1148\/ryct.230022"},{"key":"e_1_2_15_35_1","unstructured":"K. M.KallmesRapid Reviews in Nested Knowledge. Nested Knowledge https:\/\/about.nested-knowledge.com\/docs\/rapid-reviews-in-nested-knowledge\/. Published March 19 2025 accessed May 30 2025."},{"key":"e_1_2_15_36_1","unstructured":"K. J.HolubThe Data Is In: Deciding When to Automate Screening in Your SLR. Nested Knowledge https:\/\/about.nested-knowledge.com\/2023\/11\/10\/the-data-is-in-deciding-when-to-automate-screening-in-your-slr\/. Published November 10 2023 accessed May 30 2025."},{"issue":"1","key":"e_1_2_15_37_1","first-page":"MSR56","article-title":"Comparison of AI\u2010Enhanced Tools for Automating Scientific Literature Reviews","volume":"28","author":"Grys M.","year":"2025","journal-title":"Value in Health"},{"key":"e_1_2_15_38_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.conctc.2019.100443"},{"key":"e_1_2_15_39_1","doi-asserted-by":"publisher","DOI":"10.1136\/bmjopen-2016-012545"},{"key":"e_1_2_15_40_1","doi-asserted-by":"publisher","DOI":"10.1007\/s40273-022-01229-4"},{"key":"e_1_2_15_41_1","doi-asserted-by":"publisher","DOI":"10.1186\/s12916-016-0555-0"},{"key":"e_1_2_15_42_1","doi-asserted-by":"publisher","DOI":"10.2196\/33219"}],"updated-by":[{"DOI":"10.1002\/cesm.70086","type":"retraction","label":"Retraction","source":"publisher","updated":{"date-parts":[[2026,6,1]],"date-time":"2026-06-01T00:00:00Z","timestamp":1780272000000}},{"DOI":"10.1002\/cesm.70086","type":"retraction","label":"Retraction","source":"retraction-watch","updated":{"date-parts":[[2026,6,1]],"date-time":"2026-06-01T00:00:00Z","timestamp":1780272000000},"record-id":"70982"}],"container-title":["Cochrane Evidence Synthesis and Methods"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/cesm.70059","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/full-xml\/10.1002\/cesm.70059","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/cesm.70059","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,17]],"date-time":"2026-06-17T04:27:29Z","timestamp":1781670449000},"score":1,"resource":{"primary":{"URL":"https:\/\/onlinelibrary.wiley.com\/doi\/10.1002\/cesm.70059"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,25]]},"references-count":41,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2025,11]]}},"alternative-id":["10.1002\/cesm.70059"],"URL":"https:\/\/doi.org\/10.1002\/cesm.70059","archive":["Portico"],"relation":{"has-review":[{"id-type":"doi","id":"10.1002\/CESM.70059\/v1\/review3","asserted-by":"object"},{"id-type":"doi","id":"10.1002\/CESM.70059\/v2\/response1","asserted-by":"object"},{"id-type":"doi","id":"10.1002\/CESM.70059\/v2\/decision1","asserted-by":"object"},{"id-type":"doi","id":"10.1002\/CESM.70059\/v1\/review1","asserted-by":"object"},{"id-type":"doi","id":"10.1002\/CESM.70059\/v1\/review2","asserted-by":"object"},{"id-type":"doi","id":"10.1002\/CESM.70059\/v1\/decision1","asserted-by":"object"}],"has-preprint":[{"id-type":"doi","id":"10.22541\/au.174897563.38129748\/v1","asserted-by":"object"}]},"ISSN":["2832-9023","2832-9023"],"issn-type":[{"value":"2832-9023","type":"print"},{"value":"2832-9023","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,10,25]]},"assertion":[{"value":"2025-05-31","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-10-14","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-10-25","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"e70059"}}