{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,28]],"date-time":"2026-04-28T04:56:50Z","timestamp":1777352210848,"version":"3.51.4"},"reference-count":29,"publisher":"Public Library of Science (PLoS)","issue":"3","license":[{"start":{"date-parts":[[2026,3,18]],"date-time":"2026-03-18T00:00:00Z","timestamp":1773792000000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["www.plosone.org"],"crossmark-restriction":false},"short-container-title":["PLoS One"],"abstract":"\n Background<\/jats:title>\n Conversational agents based on large language models (LLMs) have shown moderate efficacy in reducing depressive and anxiety symptoms. However, most existing evaluations lack methodological transparency, rely on closed-source models, and show limited standardization in performance and safety assessment.<\/jats:p>\n <\/jats:sec>\n \n Objective<\/jats:title>\n We have two study objectives: (1) to develop an LLM-based conversational agent through system design analysis and initial functionality testing, and (2) to evaluate its safety and performance through standardized assessment in controlled simulated interactions focused on depression and anxiety of two LLMs (GPT-4o and Llama 3.1-8B).<\/jats:p>\n <\/jats:sec>\n \n Methods<\/jats:title>\n We conducted a cross-sectional study in two phases. First, we developed a mental health platform integrating a conversational agent with functionalities including personalized context, pretrained therapeutic modules, self-assessment tools, and an emergency alert system. Second, we evaluated the agent\u2019s responses in simulated interactions based on predefined user personas for each LLM. Four expert raters assessed 816 interaction pairs using a 5-point Likert scale evaluating tone, clarity, domain accuracy (correctness), robustness, completeness, boundaries, target language, and safety. In addition, we use performance metrics based on numerical criteria such as cost, response length, and number of tokens. Mixed-effects models were used to compare LLM performance and assess metric interrelations.<\/jats:p>\n <\/jats:sec>\n \n Results<\/jats:title>\n First, we developed a web-based mental health platform using a user-centered design, structured into frontend, backend, and database layers. The system integrates therapeutic chat (GPT-4o and Llama 3.1-8B), psychological assessments (PHQ-9, GAD-7), CBT-based tasks, and an emergency alert system. The platform supports secure user authentication, data encryption, multilingual access, and session tracking. Second, GPT-4o outperformed Llama 3.1-8B in both performance metrics based on numerical criteria and Likert scale criteria, generating longer and more lexically diverse responses, using more tokens, and scoring higher in clarity, robustness, completeness, boundaries, and target language. However, it incurred higher costs, with no significant differences in tone, accuracy, or safety.<\/jats:p>\n <\/jats:sec>\n \n Conclusion<\/jats:title>\n Our study presents a conversational agent with multiple functionalities and shows that GPT-4o outperforms Llama 3.1-8B in performance, although at a higher cost. This platform could be used in future clinical trials or real-world implementation studies.<\/jats:p>\n <\/jats:sec>","DOI":"10.1371\/journal.pone.0344939","type":"journal-article","created":{"date-parts":[[2026,3,18]],"date-time":"2026-03-18T17:29:07Z","timestamp":1773854947000},"page":"e0344939","update-policy":"https:\/\/doi.org\/10.1371\/journal.pone.corrections_policy","source":"Crossref","is-referenced-by-count":1,"title":["Development, system design, safety, and performance metrics of a conversational agent for reducing depressive and anxious symptoms based on a large language model: The MHAI study"],"prefix":"10.1371","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2222-4764","authenticated-orcid":true,"given":"David","family":"Villarreal-Zegarra","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yscenia","family":"Paredes-Gonzales","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Andrea","family":"D\u00e1maso-Rom\u00e1n","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Judith","family":"Qui\u00f1ones-Inga","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Gianfranco","family":"Centeno-Terrazas","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9278-0654","authenticated-orcid":true,"given":"Yan Pieer Alexis-Montalban","family":"Lozada","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"340","published-online":{"date-parts":[[2026,3,18]]},"reference":[{"issue":"1","key":"pone.0344939.ref001","doi-asserted-by":"crossref","first-page":"236","DOI":"10.1038\/s41746-023-00979-5","article-title":"Systematic review and meta-analysis of AI-based conversational agents for promoting mental health and well-being","volume":"6","author":"H Li","year":"2023","journal-title":"NPJ Digit Med"},{"issue":"4","key":"pone.0344939.ref002","doi-asserted-by":"crossref","DOI":"10.1056\/AIoa2400802","article-title":"Randomized Trial of a Generative AI Chatbot for Mental Health Treatment","volume":"2","author":"MV Heinz","year":"2025","journal-title":"NEJM AI"},{"key":"pone.0344939.ref003","doi-asserted-by":"crossref","first-page":"598","DOI":"10.1016\/j.jad.2022.09.028","article-title":"Evaluation of chatbot-delivered interventions for self-management of depression: content analysis","volume":"319","author":"L Martinengo","year":"2022","journal-title":"J Affect Disord"},{"issue":"2","key":"pone.0344939.ref004","article-title":"Large language models for chatbot health advice studies: a systematic review","volume":"8","author":"B Huo","year":"2025","journal-title":"JAMA Netw Open"},{"key":"pone.0344939.ref005","doi-asserted-by":"crossref","DOI":"10.2196\/44548","article-title":"Evaluating conversational agents for mental health: scoping review of outcomes and outcome measurement instruments","volume":"25","author":"AI Jabir","year":"2023","journal-title":"J Med Internet Res"},{"key":"pone.0344939.ref006","doi-asserted-by":"crossref","DOI":"10.2196\/58974","article-title":"Empathic conversational agent platform designs and their evaluation in the context of mental health: systematic review","volume":"11","author":"R Sanjeewa","year":"2024","journal-title":"JMIR Ment Health"},{"key":"pone.0344939.ref007","doi-asserted-by":"crossref","first-page":"171","DOI":"10.1055\/s-0041-1736664","article-title":"Evaluation Metrics for Health Chatbots: A Delphi Study","volume":"60","author":"K Denecke","year":"2021","journal-title":"Methods Inf Med"},{"key":"pone.0344939.ref008","doi-asserted-by":"crossref","DOI":"10.2196\/63538","article-title":"Development and evaluation of a mental health chatbot using ChatGPT 4.0: mixed methods user experience study with Korean users","volume":"13","author":"B Kang","year":"2025","journal-title":"JMIR Med Inform"},{"key":"pone.0344939.ref009","first-page":"890","article-title":"AI Chatbot for Cancer Patient Support: Development and Evaluation Using Llama 3.1, Mistral 7B, and PHI 3B","volume":"327","author":"S McInerney","year":"2025","journal-title":"Stud Health Technol Inform"},{"key":"pone.0344939.ref010","volume-title":"Clinical handbook of psychological disorders: a step-by-step treatment manual","author":"DH Barlow","year":"2021","edition":"6"},{"key":"pone.0344939.ref011","volume-title":"Skills training manual for treating borderline personality disorder","author":"MM Linehan","year":"2014","edition":"2"},{"key":"pone.0344939.ref012","article-title":"Scope, characteristics, behavior change techniques, and quality of conversational agents for mental health and well-being: systematic assessment of apps","volume":"25","author":"X Lin","year":"2023","journal-title":"J Med Internet Res"},{"key":"pone.0344939.ref013","doi-asserted-by":"crossref","first-page":"1737","DOI":"10.1001\/jama.282.18.1737","article-title":"Validation and utility of a self-report version of PRIME-MD: the PHQ primary care study","volume":"282","author":"RL Spitzer","year":"1999","journal-title":"JAMA"},{"issue":"10","key":"pone.0344939.ref014","doi-asserted-by":"crossref","first-page":"1092","DOI":"10.1001\/archinte.166.10.1092","article-title":"A brief measure for assessing generalized anxiety disorder: the GAD-7","volume":"166","author":"RL Spitzer","year":"2006","journal-title":"Arch Intern Med"},{"key":"pone.0344939.ref015","doi-asserted-by":"crossref","DOI":"10.2196\/43293","article-title":"Vickybot, a chatbot for anxiety-depressive symptoms and work-related burnout in primary care and health care professionals: development, feasibility, and potential effectiveness studies","volume":"25","author":"G Anmella","year":"2023","journal-title":"J Med Internet Res"},{"key":"pone.0344939.ref016","article-title":"Does daily happiness and nervousness differ in adolescents and young adults in three Latin American cities: an experience sampling study","author":"AL Vilela-Estrada","year":"2024","journal-title":"JMIR Preprints"},{"key":"pone.0344939.ref017","article-title":"Chatbot for the return of positive genetic screening results for hereditary cancer syndromes: a prompt engineering study","author":"E Coen","year":"2024","journal-title":"Res Sq"},{"issue":"6","key":"pone.0344939.ref018","doi-asserted-by":"crossref","first-page":"347","DOI":"10.1111\/pcn.13656","article-title":"Comparing the performance of ChatGPT GPT-4, Bard, and Llama-2 in the Taiwan psychiatric licensing examination and in differential diagnosis with multi-center psychiatrists","volume":"78","author":"D-J Li","year":"2024","journal-title":"Psychiatry Clin Neurosci"},{"issue":"1","key":"pone.0344939.ref019","doi-asserted-by":"crossref","first-page":"332","DOI":"10.1038\/s41746-025-01746-4","article-title":"Racial bias in AI-mediated psychiatric diagnosis and treatment: a qualitative comparison of four large language models","volume":"8","author":"A Bouguettaya","year":"2025","journal-title":"NPJ Digit Med"},{"key":"pone.0344939.ref020","doi-asserted-by":"crossref","DOI":"10.2196\/57986","article-title":"Exploring biases of large language models in the field of mental health: comparative questionnaire study of the effect of gender and sexual orientation in anorexia nervosa and bulimia nervosa case vignettes","volume":"12","author":"R Schnepper","year":"2025","journal-title":"JMIR Ment Health"},{"key":"pone.0344939.ref021","doi-asserted-by":"crossref","first-page":"104148","DOI":"10.1016\/j.ajp.2024.104148","article-title":"Comparative analysis of large language models in psychiatry and mental health: a focus on GPT, AYA, and Nemotron-3-8B","volume":"99","author":"OK Gargari","year":"2024","journal-title":"Asian J Psychiatr"},{"key":"pone.0344939.ref022","doi-asserted-by":"crossref","first-page":"665","DOI":"10.1016\/j.mayocp.2023.02.005","article-title":"Advancing translation of clinical research into practice and population health impact through implementation science","volume":"99","author":"LJF Rutten","year":"2024","journal-title":"Mayo Clinic Proceed"},{"issue":"4","key":"pone.0344939.ref023","doi-asserted-by":"crossref","first-page":"319","DOI":"10.1001\/jama.2024.21700","article-title":"Testing and evaluation of health care applications of large language models: a systematic review","volume":"333","author":"S Bedi","year":"2025","journal-title":"JAMA"},{"key":"pone.0344939.ref024","doi-asserted-by":"crossref","DOI":"10.2196\/38204","article-title":"Cost-effectiveness of internet interventions compared with treatment as usual for people with mental disorders: systematic review and meta-analysis of randomized controlled trials","volume":"25","author":"PJ Rohrbach","year":"2023","journal-title":"J Med Internet Res"},{"issue":"3","key":"pone.0344939.ref025","doi-asserted-by":"crossref","DOI":"10.2196\/34760","article-title":"Development of a framework for the implementation of synchronous digital mental health: realist synthesis of systematic reviews","volume":"9","author":"D Villarreal-Zegarra","year":"2022","journal-title":"JMIR Ment Health"},{"key":"pone.0344939.ref026","article-title":"Ethical and regulatory gaps in using generative AI for mental health support in low- and middle-income countries","volume":"12","author":"L Rojas-Mezarina","year":"2026","journal-title":"Interacciones"},{"issue":"8081","key":"pone.0344939.ref027","doi-asserted-by":"crossref","first-page":"633","DOI":"10.1038\/s41586-025-09422-z","article-title":"DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning","volume":"645","author":"D Guo","year":"2025","journal-title":"Nature"},{"issue":"6","key":"pone.0344939.ref028","doi-asserted-by":"crossref","first-page":"1015","DOI":"10.1093\/jamia\/ocaf045","article-title":"Evaluating the effectiveness of biomedical fine-tuning for large language models on clinical tasks","volume":"32","author":"FJ Dorfner","year":"2025","journal-title":"J Am Med Inform Assoc"},{"key":"pone.0344939.ref029","doi-asserted-by":"crossref","DOI":"10.2196\/63538","article-title":"Development and evaluation of a mental health chatbot using ChatGPT 4.0: mixed methods user experience study with Korean users","volume":"13","author":"B Kang","year":"2025","journal-title":"JMIR Med Inform"}],"container-title":["PLOS One"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0344939","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,3,18]],"date-time":"2026-03-18T17:29:12Z","timestamp":1773854952000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pone.0344939"}},"subtitle":[],"editor":[{"given":"Issa","family":"Atoum","sequence":"first","affiliation":[],"role":[{"role":"editor","vocabulary":"crossref"}]}],"short-title":[],"issued":{"date-parts":[[2026,3,18]]},"references-count":29,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2026,3,18]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pone.0344939","relation":{},"ISSN":["1932-6203"],"issn-type":[{"value":"1932-6203","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,3,18]]}}}