{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,11]],"date-time":"2026-08-11T22:30:42Z","timestamp":1786487442405,"version":"build-2736575974"},"reference-count":38,"publisher":"JMIR Publications Inc.","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMIR Res Protoc"],"abstract":"\n Background<\/jats:title>\n Large language models (LLMs) are increasingly used in health care by nonprofessionals (ie, individuals without formal training in health-related professions). These applications must be evaluated in an appropriate manner to prevent misinformation and harmful decisions. To date, guidance to evaluate LLM-based applications for nonprofessional users remains limited and fragmented, leaving researchers and developers without a scientifically grounded set of quality dimensions, metrics, and measurement tools to guide them.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n This protocol outlines a scoping review that maps approaches for evaluation of LLM-based applications used for health purposes by nonprofessionals. It identifies current methods and maps them thematically by assigning them to evaluation dimensions, metrics, and measurement instruments. The review will provide a comprehensive overview of evaluation methods currently in use.<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n The study follows the Joana Briggs Institute approach for conducting scoping reviews and reports. The protocol is reported in accordance with the PRISMA-P (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Protocols) guidelines, and the scoping review will be reported in accordance with the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guidelines. The inclusion criteria comprise studies that evaluate LLM-based applications that are used in the context of health care by nonprofessionals. The search was conducted in PubMed, CINAHL, PsycInfo, and IEEE Xplore. Results since 2021 were considered. Data will be summarized and interpreted qualitatively. Publication screening was conducted by 2 independent reviewers in a blinded manner, with discrepancies settled through discussion. Data extraction and charting will be performed by 1 reviewer. To ensure quality, a random 10% sample of the publications will be independently charted by a second reviewer. Disagreements in the double-extracted subset will be resolved through discussion.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n As of July 2026, a steering committee of 6 researchers has been chosen for the conduct of the review. An initial search resulted in 8538 records after removing duplicates. After screening of these 8538 publications, 17.8% (1524\/8538) were eligible for retrieval, of which 88.3% (1345\/1524) were retrieved. Full-text screening (completed by 1 reviewer) excluded publications due to nonmatching populations (155\/1345, 11.5%), concepts (246\/1345, 18.3%), and contexts (24\/1345, 1.8%), as well as secondary work (14\/1345, 1%), leaving 67.4% (906\/1345) of these publications for data extraction. We plan to perform final full-text screening, data extraction, coding, and synthesis of results in the fourth quarter of 2026.<\/jats:p>\n <\/jats:sec>\n \n Conclusions<\/jats:title>\n The scoping review aims to identify and map current evaluation methods for LLM-based applications used in health care by nonprofessionals. It will provide a systematic overview of the current state of research and insights into quality dimensions, metrics, and measurement instruments. The findings will provide directional guidance for further research and development in the field of quality assurance for LLM-based applications used by nonprofessionals.<\/jats:p>\n <\/jats:sec>\n \n International Registered Report Identifier (IRRID)<\/jats:title>\n DERR1-10.2196\/93509<\/jats:p>\n <\/jats:sec>","DOI":"10.2196\/93509","type":"journal-article","created":{"date-parts":[[2026,8,3]],"date-time":"2026-08-03T09:05:06Z","timestamp":1785747906000},"page":"e93509","source":"Crossref","is-referenced-by-count":0,"title":["Methods of Evaluating Large Language Model\u2013Based Health Care Applications Used by Nonprofessionals: Protocol for a Scoping Review"],"prefix":"10.2196","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0009-0009-1611-6933","authenticated-orcid":false,"given":"Maren","family":"Keuchel","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5354-6766","authenticated-orcid":false,"given":"Pinar","family":"Bisgin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2216-894X","authenticated-orcid":false,"given":"Tom","family":"Strube","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-8706-9314","authenticated-orcid":false,"given":"Niklas","family":"Tschorn","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-4187-0229","authenticated-orcid":false,"given":"Leoni","family":"Weltermann","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0522-986X","authenticated-orcid":false,"given":"Sven","family":"Meister","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1010","published-online":{"date-parts":[[2026,8,11]]},"reference":[{"key":"ref1","doi-asserted-by":"publisher","DOI":"10.3390\/bioengineering12060631"},{"key":"ref2","doi-asserted-by":"publisher","DOI":"10.3390\/fi17020076"},{"key":"ref3","doi-asserted-by":"publisher","DOI":"10.1038\/s43856-024-00717-2"},{"key":"ref4","doi-asserted-by":"publisher","DOI":"10.1038\/s43856-025-00927-2"},{"key":"ref5","doi-asserted-by":"publisher","DOI":"10.3233\/SHTI251302"},{"key":"ref6","unstructured":"Classifying health workers: mapping occupations to the International Standard ClassificationWorld Health Organization201907312026-07-31https:\/\/www.who.int\/publications\/m\/item\/classifying-health-workers"},{"key":"ref7","doi-asserted-by":"publisher","DOI":"10.3389\/fmed.2024.1477898"},{"key":"ref8","unstructured":"Regulation (EU) 2024\/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence and amending Regulations (EC) No 300\/2008, (EU) No 167\/2013, (EU) No 168\/2013, (EU) 2018\/858, (EU) 2018\/1139 and (EU) 2019\/2144 and Directives 2014\/90\/EU, (EU) 2016\/797 and (EU) 2020\/1828 (Artificial Intelligence Act) (Text with EEA relevance)European Union20242026-01-07https:\/\/eur-lex.europa.eu\/eli\/reg\/2024\/1689\/oj\/eng"},{"key":"ref9","doi-asserted-by":"publisher","DOI":"10.1186\/s12911-025-02954-4"},{"key":"ref10","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2024.21700"},{"key":"ref11","doi-asserted-by":"publisher","DOI":"10.3115\/1073083.1073135"},{"key":"ref12","author":"Lin, CY","year":"2004","journal-title":"Text Summarization Branches Out"},{"key":"ref13","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1904.09675"},{"key":"ref14","doi-asserted-by":"publisher","DOI":"10.3390\/app11146421"},{"key":"ref15","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-025-31010-4"},{"key":"ref16","unstructured":"About the USMLEUnited States Medical Licensing Examination2026-01-12https:\/\/www.usmle.org\/about-usmle"},{"key":"ref17","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-023-06291-2"},{"key":"ref18","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-1259"},{"key":"ref19","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2505.11733"},{"key":"ref20","doi-asserted-by":"publisher","DOI":"10.1186\/s12911-024-02459-6"},{"key":"ref21","doi-asserted-by":"publisher","DOI":"10.1186\/s12911-024-02709-7"},{"key":"ref22","doi-asserted-by":"publisher","DOI":"10.1038\/s41746-024-01258-7"},{"key":"ref23","author":"Gupta, M","year":"2025","journal-title":"Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies"},{"key":"ref24","author":"Yao, Z","year":"2026","journal-title":"arXiv. Preprint posted online on October 2, 2024"},{"key":"ref25","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2407.19098"},{"key":"ref26","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2409.07314"},{"key":"ref27","doi-asserted-by":"publisher","DOI":"10.11124\/JBIES-20-00167"},{"key":"ref28","doi-asserted-by":"publisher","DOI":"10.1186\/2046-4053-4-1"},{"key":"ref29","doi-asserted-by":"publisher","DOI":"10.7326\/M18-0850"},{"key":"ref30","unstructured":"JMIR research protocolsJMIR Publications2025-12-18https:\/\/www.researchprotocols.org\/"},{"key":"ref31","doi-asserted-by":"publisher","DOI":"10.1016\/j.zefq.2019.05.004"},{"key":"ref32","unstructured":"Welcome to Medical Subject HeadingsNational Institutes of Health National Library of Medicine2026-01-16https:\/\/www.nlm.nih.gov\/mesh\/meshhome.html"},{"key":"ref33","doi-asserted-by":"crossref","unstructured":"Rayyan2025-12-19https:\/\/new.rayyan.ai\/","DOI":"10.5339\/jist.2025.15"},{"key":"ref34","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.m3164"},{"key":"ref35","unstructured":"MayringPQualitative content analysis: theoretical foundation, basic procedures and software solutionSocial Science Open Access Repository20142026-07-31https:\/\/www.ssoar.info\/ssoar\/handle\/document\/39517"},{"key":"ref36","unstructured":"Artificial Intelligence Risk Management Framework (AI RMF 1.0)National Institute of Standards and Technology2023012026-08-04https:\/\/nvlpubs.nist.gov\/nistpubs\/ai\/NIST.AI.100-1.pdf"},{"key":"ref37","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.n71"},{"key":"ref38","unstructured":"DeepL2026-08-06https:\/\/www.deepl.com\/de"}],"container-title":["JMIR Research Protocols"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.researchprotocols.org\/2026\/1\/e93509","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,11]],"date-time":"2026-08-11T22:05:11Z","timestamp":1786485911000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.researchprotocols.org\/2026\/1\/e93509"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,11]]},"references-count":38,"URL":"https:\/\/doi.org\/10.2196\/93509","relation":{"has-preprint":[{"id-type":"doi","id":"10.2196\/preprints.93509","asserted-by":"object"}]},"ISSN":["1929-0748"],"issn-type":[{"value":"1929-0748","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,11]]}}}