{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,14]],"date-time":"2026-09-14T22:04:01Z","timestamp":1789423441329,"version":"build-2803163510"},"reference-count":26,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Res Protoc"],"abstract":"Abstract<\/jats:title>\n \n Background<\/jats:title>\n Digital mental health interventions (DMHIs) can help close persistent gaps in access to assessment, prevention, and treatment. Recent advances in generative AI, particularly large language models (LLMs), further expand this promise by enabling natural language understanding, personalization, and empathic interaction across assessment, support, and therapeutic contexts. However, significant evaluation challenges persist, including a lack of standardized constructs and validated instruments, which limit the comparability, reproducibility, and generalizability of the findings. No systematic synthesis currently documents the frameworks, methodologies, and tools used to evaluate LLMs in DMHIs, thereby hampering the development of a comprehensive evidence base to guide future evaluation efforts.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This scoping review aims to systematically map and synthesize the available evidence on frameworks, methodologies, and tools used to evaluate LLMs applied to DMHIs. Specifically, it aims to identify the constructs assessed, the instruments used, and the evaluation procedures and stages addressed.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n Following the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guidelines, this scoping review will search 5 electronic databases (PubMed, Scopus, Web of Science, IEEE Xplore, and ACM Digital Library) from January 1, 2019, to September 15, 2025. Eligibility criteria will encompass both empirical studies and theoretical proposals evaluating LLMs embedded within DMHIs. Studies limited to risk detection or decision support systems without an intervention component will be excluded. Data extraction will capture information on conceptual frameworks, methodological designs, evaluation procedures and tools, measured constructs, and other relevant contextual information.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n The systematic search was conducted between September 1 and 15, 2025, yielding 4273 records across the 5 databases. After duplicate removal, of the 4273 records, 2980 (69.7%) remained for screening. A pilot screening exercise involving 4 independent reviewers achieved high interrater reliability (free-marginal Randolph \u03ba=0.81), with 76% (19\/25 of the pilot sample) unanimous agreement, indicating adequate calibration of selection criteria. These figures are interim process indicators rather than final review findings as title and abstract screening of the remaining records is currently underway.<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n This scoping review is expected to provide one of the first systematic syntheses of frameworks, methodologies, and tools used to evaluate LLMs in DMHIs. By identifying prevailing patterns and gaps, the resulting evidence map is intended to serve as a practical reference for researchers, developers, and policymakers working toward a scientifically grounded, safe, ethical, and effective deployment of LLMs in mental health interventions.<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/91677","type":"journal-article","created":{"date-parts":[[2026,9,14]],"date-time":"2026-09-14T21:21:00Z","timestamp":1789420860000},"page":"e91677-e91677","source":"Crossref","is-referenced-by-count":0,"title":["Frameworks, Methodologies, and Tools for Evaluating Large Language Models in Digital Mental Health Interventions: Protocol for a Scoping Review"],"prefix":"10.2196","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0009-0009-2270-6948","authenticated-orcid":false,"given":"Antonio","family":"Salinas-Layana","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5621-9322","authenticated-orcid":false,"given":"\u00c1lvaro","family":"Jim\u00e9nez-Molina","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9417-0059","authenticated-orcid":false,"given":"Daniela","family":"Lira","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-1788-2649","authenticated-orcid":false,"given":"F\u00e9lix Alberto","family":"V\u00e9liz-Montoya","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-5944-4407","authenticated-orcid":false,"given":"Alexi","family":"Venegas","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-6718-4998","authenticated-orcid":false,"given":"Nicol\u00e1s","family":"Mu\u00f1oz","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-1670-8427","authenticated-orcid":false,"given":"Mario","family":"Chand\u00eda","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-3202-2086","authenticated-orcid":false,"given":"Rigoberto","family":"Rojas","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5980-7122","authenticated-orcid":false,"given":"Vania","family":"Mart\u00ednez","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,9,14]]},"reference":[{"issue":"10157","key":"R1","doi-asserted-by":"publisher","first-page":"1553","DOI":"10.1016\/S0140-6736(18)31612-X","article-title":"The Lancet Commission on global mental health and sustainable development","volume":"392","author":"Patel","journal-title":"Lancet"},{"key":"R2","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyt.2019.00664","article-title":"Internet-based interventions for the prevention and treatment of mental disorders in Latin America: a scoping review","volume":"10","author":"Jim\u00e9nez-Molina","journal-title":"Front Psychiatry"},{"issue":"5","key":"R3","doi-asserted-by":"publisher","DOI":"10.2196\/35159","article-title":"Digital health interventions for delivery of mental health care: systematic and comprehensive meta-review","volume":"9","author":"Philippe","journal-title":"JMIR Ment Health"},{"issue":"2","key":"R4","doi-asserted-by":"publisher","first-page":"156","DOI":"10.1002\/wps.21299","article-title":"The evolving field of digital mental health: current evidence and implementation issues for smartphone apps, generative artificial intelligence, and virtual reality","volume":"24","author":"Torous","journal-title":"World Psychiatry"},{"issue":"2","key":"R5","doi-asserted-by":"publisher","DOI":"10.1037\/tmb0000163","article-title":"Readiness Evaluation for AI-Mental Health Deployment and Implementation (READI): a review and proposed framework","volume":"6","author":"Stade","journal-title":"Technol Mind Behav"},{"issue":"7","key":"R6","doi-asserted-by":"publisher","first-page":"1285","DOI":"10.1038\/s41562-024-01882-z","article-title":"Testing theory of mind in large language models and humans","volume":"8","author":"Strachan","journal-title":"Nat Hum Behav"},{"key":"R7","doi-asserted-by":"publisher","DOI":"10.2196\/67365","article-title":"Safety and user experience of a generative artificial intelligence digital mental health intervention: exploratory randomized controlled trial","volume":"27","author":"Campellone","journal-title":"J Med Internet Res"},{"issue":"1","key":"R8","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-025-01611-4","article-title":"A scoping review of large language models for generative tasks in mental health care","volume":"8","author":"Hua","journal-title":"NPJ Digit Med"},{"issue":"4","key":"R9","doi-asserted-by":"publisher","first-page":"e282","DOI":"10.1016\/S2589-7500(24)00255-3","article-title":"Large language models for the mental health community: framework for translating code to care","volume":"7","author":"Malgaroli","journal-title":"Lancet Digit Health"},{"issue":"4","key":"R10","doi-asserted-by":"publisher","DOI":"10.1056\/AIoa2400802","article-title":"Randomized trial of a generative AI chatbot for mental health treatment","volume":"2","author":"Heinz","journal-title":"NEJM AI"},{"issue":"1","key":"R11","doi-asserted-by":"publisher","first-page":"19","DOI":"10.1097\/YCO.0000000000001048","article-title":"Efficacy and risks of artificial intelligence chatbots for anxiety and depression: a narrative review of recent clinical studies","volume":"39","author":"Bodner","journal-title":"Curr Opin Psychiatry"},{"issue":"5","key":"R12","doi-asserted-by":"publisher","first-page":"1269","DOI":"10.1038\/s41591-024-02943-6","article-title":"The health risks of generative AI-based wellness apps","volume":"30","author":"De Freitas","journal-title":"Nat Med"},{"key":"R13","doi-asserted-by":"publisher","DOI":"10.3389\/frai.2023.1229805","article-title":"A review of the explainability and safety of conversational agents for mental health to identify avenues for improvement","volume":"6","author":"Sarkar","journal-title":"Front Artif Intell"},{"key":"R14","doi-asserted-by":"publisher","DOI":"10.2196\/69284","article-title":"The applications of large language models in mental health: scoping review","volume":"27","author":"Jin","journal-title":"J Med Internet Res"},{"issue":"7","key":"R15","doi-asserted-by":"publisher","first-page":"467","DOI":"10.7326\/M18-0850","article-title":"PRISMA extension for Scoping Reviews (PRISMA-ScR): checklist and explanation","volume":"169","author":"Tricco","journal-title":"Ann Intern Med"},{"key":"R16","doi-asserted-by":"crossref","unstructured":"Peters MD Godfrey C McInerney P Munn Z Tricco AC Khalil H . Scoping reviews. In: Aromataris E Lockwood C Porritt K Pilla B Jordan Z , editors. JBI Manual for Evidence Synthesis. JBI; 2024. [doi: 10.46658\/JBIMES-24-09]","DOI":"10.46658\/JBIMES-24-09"},{"key":"R17","doi-asserted-by":"publisher","DOI":"10.1007\/s13412-023-00833-w","article-title":"What is a framework? Understanding their purpose, value, development and use","volume":"13","author":"Partelow","journal-title":"J Environ Stud Sci"},{"issue":"1","key":"R18","doi-asserted-by":"publisher","first-page":"223","DOI":"10.1109\/TSMC.2020.3042898","article-title":"Internet of things as system of systems: a review of methodologies, frameworks, platforms, and tools","volume":"51","author":"Fortino","journal-title":"IEEE Trans Syst Man Cybern Syst"},{"key":"R19","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2024.3365742","article-title":"A review on large language models: architectures, applications, taxonomies, open issues and challenges","volume":"12","author":"Raiaan","journal-title":"IEEE Access"},{"issue":"3","key":"R20","doi-asserted-by":"publisher","first-page":"746","DOI":"10.1093\/jamia\/ocad222","article-title":"Evaluation framework for conversational agents with artificial intelligence in health interventions: a systematic scoping review","volume":"31","author":"Ding","journal-title":"J Am Med Inform Assoc"},{"key":"R21","unstructured":"Vaswani A Shazeer N Parmar N et al. Attention is all you need. In: NIPS\u201917: Proceedings of the 31st International Conference on Neural Information Processing Systems. Curran Associates Inc; 2017:6000-6010. [doi: 10.5555\/3295222.3295349]"},{"key":"R22","doi-asserted-by":"publisher","DOI":"10.2196\/59069","article-title":"Revolutionizing health care: the transformative impact of large language models in medicine","volume":"27","author":"Zhang","journal-title":"J Med Internet Res"},{"key":"R23","doi-asserted-by":"publisher","DOI":"10.2196\/62865","article-title":"Exploring the credibility of large language models for mental health support: protocol for a scoping review","volume":"14","author":"Gautam","journal-title":"JMIR Res Protoc"},{"issue":"1","key":"R24","doi-asserted-by":"publisher","first-page":"345","DOI":"10.1108\/EMJB-03-2024-0055","article-title":"AI meets academia: transforming systematic literature reviews","volume":"21","author":"Tomczyk","journal-title":"EuroMed J Bus"},{"key":"R25","unstructured":"Randolph JJ . Free-marginal multirater kappa (multirater \u03bafree): an alternative to Fleiss\u2019 fixed-marginal multirater kappa. Presented at: Joensuu Learning and Instruction Symposium 2005; Oct 14-15, 2005; Joensuu, Finland. URL: https:\/\/www.academia.edu\/2439155\/Free_Marginal_Multirater_Kappa_multiraterfree_An_Alternative_to_Fleiss_Fixed_Marginal_Multirater_Kappa [Accessed 23-08-2026]"},{"key":"R26","doi-asserted-by":"publisher","DOI":"10.1017\/neu.2024.50","article-title":"Use of generative artificial intelligence (AI) in psychiatry and mental health care: a systematic review","volume":"37","author":"Kolding","journal-title":"Acta Neuropsychiatr"}],"container-title":["JMIR Research Protocols"],"original-title":[],"language":"en","deposited":{"date-parts":[[2026,9,14]],"date-time":"2026-09-14T21:21:05Z","timestamp":1789420865000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.researchprotocols.org\/2026\/1\/e91677"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,14]]},"references-count":26,"URL":"https:\/\/doi.org\/10.2196\/91677","relation":{},"ISSN":["1929-0748"],"issn-type":[{"value":"1929-0748","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,14]]},"article-number":"v15i10157e91677"}}