{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,7]],"date-time":"2026-04-07T08:39:13Z","timestamp":1775551153853,"version":"3.50.1"},"reference-count":76,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2025,3,20]],"date-time":"2025-03-20T00:00:00Z","timestamp":1742428800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Hub Life Science\u2014Digital Health (LSH-DH)","award":["PNC-E3-2022-23683267"],"award-info":[{"award-number":["PNC-E3-2022-23683267"]}]},{"name":"Hub Life Science\u2014Digital Health (LSH-DH)","award":["PNC-E.3"],"award-info":[{"award-number":["PNC-E.3"]}]},{"name":"Ministry of Health (Italy) under the Piano Nazionale Complementare al PNRR Ecosistema Innovativo della Salute","award":["PNC-E3-2022-23683267"],"award-info":[{"award-number":["PNC-E3-2022-23683267"]}]},{"name":"Ministry of Health (Italy) under the Piano Nazionale Complementare al PNRR Ecosistema Innovativo della Salute","award":["PNC-E.3"],"award-info":[{"award-number":["PNC-E.3"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Informatics"],"abstract":"<jats:p>Background\/Objectives: With advancements in Large Language Models (LLMs), counseling chatbots are becoming essential tools for delivering scalable and accessible mental health support. Traditional evaluation scales, however, fail to adequately capture the sophisticated capabilities of these systems, such as personalized interactions, empathetic responses, and memory retention. This study aims to design a robust and comprehensive evaluation scale, the Comprehensive Evaluation Scale for LLM-Powered Counseling Chatbots (CES-LCC), using the eDelphi method to address this gap. Methods: A panel of 16 experts in psychology, artificial intelligence, human-computer interaction, and digital therapeutics participated in two iterative eDelphi rounds. The process focused on refining dimensions and items based on qualitative and quantitative feedback. Initial validation, conducted after assembling the final version of the scale, involved 49 participants using the CES-LCC to evaluate an LLM-powered chatbot delivering Self-Help Plus (SH+), an Acceptance and Commitment Therapy-based intervention for stress management. Results: The final version of the CES-LCC features 27 items grouped into nine dimensions: Understanding Requests, Providing Helpful Information, Clarity and Relevance of Responses, Language Quality, Trust, Emotional Support, Guidance and Direction, Memory, and Overall Satisfaction. Initial real-world validation revealed high internal consistency (Cronbach\u2019s alpha = 0.94), although minor adjustments are required for specific dimensions, such as Clarity and Relevance of Responses. Conclusions: The CES-LCC fills a critical gap in the evaluation of LLM-powered counseling chatbots, offering a standardized tool for assessing their multifaceted capabilities. While preliminary results are promising, further research is needed to validate the scale across diverse populations and settings.<\/jats:p>","DOI":"10.3390\/informatics12010033","type":"journal-article","created":{"date-parts":[[2025,3,20]],"date-time":"2025-03-20T07:59:54Z","timestamp":1742457594000},"page":"33","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Development of a Comprehensive Evaluation Scale for LLM-Powered Counseling Chatbots (CES-LCC) Using the eDelphi Method"],"prefix":"10.3390","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7789-2535","authenticated-orcid":false,"given":"Marco","family":"Bolpagni","sequence":"first","affiliation":[{"name":"Department of General Psychology, University of Padova, 35121 Padova, Italy"},{"name":"Digital Health Research, Centre for Digital Health and Wellbeing, Fondazione Bruno Kessler, 38123 Trento, Italy"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7080-0615","authenticated-orcid":false,"given":"Silvia","family":"Gabrielli","sequence":"additional","affiliation":[{"name":"Digital Health Research, Centre for Digital Health and Wellbeing, Fondazione Bruno Kessler, 38123 Trento, Italy"}]}],"member":"1968","published-online":{"date-parts":[[2025,3,20]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"64","DOI":"10.1159\/000501812","article-title":"The Next Generation: Chatbots in Clinical Psychology and Psychotherapy to Foster Mental Health\u2014A Scoping Review","volume":"32","author":"Bendig","year":"2022","journal-title":"Verhaltenstherapie"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"e56930","DOI":"10.2196\/56930","article-title":"Roles, Users, Benefits, and Limitations of Chatbots in Health Care: Rapid Review","volume":"26","author":"Laymouna","year":"2024","journal-title":"J. Med. Internet Res."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Balcombe, L. (2023). AI Chatbots in Digital Mental Health. Informatics, 10.","DOI":"10.3390\/informatics10040082"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"2179","DOI":"10.1001\/jama.2022.8661","article-title":"Clinician Shortage Exacerbates Pandemic-Fueled \u201cMental Health Crisis\u201d","volume":"327","author":"Kuehn","year":"2022","journal-title":"JAMA"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1080\/17434440.2021.2013200","article-title":"Artificially intelligent chatbots in digital mental health interventions: A review","volume":"18","author":"Boucher","year":"2021","journal-title":"Expert. Rev. Med. Devices"},{"key":"ref_6","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017). Attention Is All You Need. arXiv."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Skjuve, M., F\u00f8lstad, A., and Brandtzaeg, P.B. (2023, January 19\u201321). The User Experience of ChatGPT: Findings from a Questionnaire Study of Early Users. Proceedings of the 5th International Conference on Conversational User Interfaces, Eindhoven, The Netherlands.","DOI":"10.1145\/3571884.3597144"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"100081","DOI":"10.1016\/j.chbah.2024.100081","article-title":"User engagement, attitudes, and the effectiveness of chatbots as a mental health intervention: A systematic review","volume":"2","author":"Limpanopparat","year":"2024","journal-title":"Comput. Hum. Behav. Artif. Hum."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"938","DOI":"10.1002\/asi.20801","article-title":"What is user engagement? A conceptual framework for defining user engagement with technology","volume":"59","author":"Toms","year":"2008","journal-title":"J. Am. Soc. Inf. Sci. Technol."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1080\/01449290500330331","article-title":"User experience-a research agenda","volume":"25","author":"Hassenzahl","year":"2006","journal-title":"Behav. Inf. Technol."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"339","DOI":"10.1016\/j.intcom.2009.04.007","article-title":"Usability\u2014Context, framework, definition, design and evaluation","volume":"21","author":"Shackel","year":"2009","journal-title":"Interact. Comput."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Moilanen, J., Visuri, A., Suryanarayana, S.A., Alorwu, A., Yatani, K., and Hosio, S. (2022, January 27\u201330). Measuring the Effect of Mental Health Chatbot Personality on User Engagement. Proceedings of the 21st International Conference on Mobile and Ubiquitous Multimedia, Lisbon, Portugal.","DOI":"10.1145\/3568444.3568464"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"e27965","DOI":"10.2196\/27965","article-title":"Engagement and Effectiveness of a Healthy-Coping Intervention via Chatbot for University Students During the COVID-19 Pandemic: Mixed Methods Proof-of-Concept Study","volume":"9","author":"Gabrielli","year":"2021","journal-title":"JMIR Mhealth Uhealth"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"50","DOI":"10.1002\/asi.21229","article-title":"The development and evaluation of a survey to measure user engagement","volume":"61","author":"Toms","year":"2010","journal-title":"J. Am. Soc. Inf. Sci. Technol."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1170","DOI":"10.1109\/TETC.2020.2974478","article-title":"A Mental Health Chatbot for Regulating Emotions (SERMO)\u2014Concept and Usability Test","volume":"9","author":"Denecke","year":"2021","journal-title":"IEEE Trans. Emerg. Top. Comput."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"100668","DOI":"10.1016\/j.invent.2023.100668","article-title":"A chatbot-delivered intervention for optimizing social media use and reducing perceived isolation among rural-living LGBTQ+ youth: Development, acceptability, usability, satisfaction, and utility","volume":"34","author":"Porta","year":"2023","journal-title":"Internet Interv."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Lima, M.R., Wairagkar, M., Natarajan, N., Vaitheswaran, S., and Vaidyanathan, R. (2021). Robotic Telemedicine for Mental Health: A Multimodal Approach to Improve Human-Robot Engagement. Front. Robot. AI, 8.","DOI":"10.3389\/frobt.2021.618866"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Holzinger, A. (2008). Construction and Evaluation of a User Experience Questionnaire. HCI and Usability for Education and Work, Springer.","DOI":"10.1007\/978-3-540-89350-9"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"1229","DOI":"10.1002\/eat.23798","article-title":"Development and usability testing of a chatbot to promote mental health services use among individuals with eating disorders following screening","volume":"55","author":"Shah","year":"2022","journal-title":"Int. J. Eat. Disord."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Boyd, K., Potts, C., Bond, R., Mulvenna, M., Broderick, T., Burns, C., Bickerdike, A., Mctear, M., Kostenius, C., and Vakaloudis, A. (2022, January 4\u20137). Usability testing and trust analysis of a mental health and wellbeing chatbot. Proceedings of the 33rd European Conference on Cognitive Ergonomics, Kaiserslautern, Germany.","DOI":"10.1145\/3552327.3552348"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Islam, M.N., Khan, S.R., Islam, N.N., Rezwan-A-Rownok Md Zaman, S.R., and Zaman, S.R. (2021). A Mobile Application for Mental Health Care During COVID-19 Pandemic: Development and Usability Evaluation with System Usability Scale, Springer.","DOI":"10.1007\/978-3-030-68133-3_4"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Valtolina, S., Zanotti, P., and Mandelli, S. (2024, January 16\u201319). Designing Conversational Agents to Empower Active Aging. Proceedings of the ACM International Conference on Intelligent Virtual Agents, Glasgow, UK.","DOI":"10.1145\/3652988.3673939"},{"key":"ref_23","unstructured":"Brooke, J. (1996). SUS: A \u201cQuick and Dirty\u201d Usability Scale. Usability Evaluation In Industry, CRC Press."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Holmes, S., Moorhead, A., Bond, R., Zheng, H., Coates, V., and Mctear, M. (2019, January 10\u201313). Usability testing of a healthcare chatbot: Can we use conventional methods to assess conversational user interfaces?. Proceedings of the 31st European Conference on Cognitive Ergonomics, Belfast, UK.","DOI":"10.1145\/3335082.3335094"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1007\/s00779-021-01582-9","article-title":"The Chatbot Usability Scale: The Design and Pilot of a Usability Scale for Interaction with AI-Based Conversational Agents","volume":"26","author":"Borsci","year":"2022","journal-title":"Pers. Ubiquitous Comput."},{"key":"ref_26","unstructured":"Henkel, T., Linn, A.J., and van der Goot, M.J. (2022, January 22\u201323). Understanding the Intention to Use Mental Health Chatbots Among LGBTQIA+\u2009Individuals: Testing and Extending the UTAUT. Proceedings of the 6th International Workshop, CONVERSATIONS 2022, Amsterdam, The Netherlands."},{"key":"ref_27","first-page":"1","article-title":"A Chatbot System for Mental Healthcare Based on SAT Counseling Method","volume":"2019","author":"Kamita","year":"2019","journal-title":"Mob. Inf. Syst."},{"key":"ref_28","first-page":"425","article-title":"User Acceptance of Information Technology: Toward a Unified View","volume":"27","author":"Venkatesh","year":"2003","journal-title":"MIS Qarterly"},{"key":"ref_29","first-page":"319","article-title":"Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology","volume":"13","author":"Davis","year":"1989","journal-title":"MIS Qarterly"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"24lr03782","DOI":"10.4088\/PCC.24lr03782","article-title":"Measuring Empathy in Artificial Intelligence: Insights From Psychodermatology and Implications for General Practice","volume":"26","author":"Ahuja","year":"2024","journal-title":"Prim. Care Companion CNS Disord."},{"key":"ref_31","unstructured":"Zhao, J., Plaza-del-Arco, F.M., Genchel, B., and Curry, A.C. (2024). Language Model Council: Democratically Benchmarking Foundation Models on Highly Subjective Tasks. arXiv."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Schmidmaier, M., Rupp, J., Cvetanova, D., and Mayer, S. (2024, January 11\u201316). Perceived Empathy of Technology Scale (PETS): Measuring Empathy of Systems Toward the User. Proceedings of the CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA.","DOI":"10.1145\/3613904.3642035"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"2233","DOI":"10.1007\/s00146-023-01715-z","article-title":"Measuring perceived empathy in dialogue systems","volume":"39","author":"Concannon","year":"2024","journal-title":"AI Soc."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"e16660","DOI":"10.2196\/16660","article-title":"Measuring Alliance Toward Embodied Virtual Therapists in the Era of Automated Treatments With the Virtual Therapist Alliance Scale (VTAS): Development and Psychometric Evaluation","volume":"22","author":"Miloff","year":"2020","journal-title":"J. Med. Internet Res."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Wei, S., Freeman, D., and Rovira, A. (2023). A randomised controlled test of emotional attributes of a virtual coach within a virtual reality (VR) mental health treatment. Sci. Rep., 13.","DOI":"10.1038\/s41598-023-38499-7"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"16","DOI":"10.21037\/jmai-23-136","article-title":"An experimental study of integrating fine-tuned large language models and prompts for enhancing mental health support chatbot system","volume":"7","author":"Yu","year":"2024","journal-title":"J. Med. Artif. Intell."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Crasto, R., Dias, L., Miranda, D., and Kayande, D. (2021, January 21\u201323). CareBot: A Mental Health ChatBot. Proceedings of the 2021 2nd International Conference for Emerging Technology (INCET), Belagavi, India.","DOI":"10.1109\/INCET51464.2021.9456326"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Srivastava, A., Pandey, I., Akhtar, M.S., and Chakraborty, T. (May, January 30). Response-act Guided Reinforced Dialogue Generation for Mental Health Counseling. Proceedings of the ACM Web Conference 2023, Austin, TX, USA.","DOI":"10.1145\/3543507.3583380"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Kaysar, M.N., and Shiramatsu, S. (2024). Mental State-Based Dialogue System for Mental Health Care by Using GPT-3. Proceedings of Eighth International Congress on Information and Communication Technology, Springer.","DOI":"10.1007\/978-981-99-3043-2_74"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. (2002, January 7\u201312). BLEU. Proceedings of the 40th Annual Meeting on Association for Computational Linguistics\u2014ACL \u201902, Philadelphia, PA, USA.","DOI":"10.3115\/1073083.1073135"},{"key":"ref_41","unstructured":"Lin, C.Y. (2004). ROUGE: A Package for Automatic Evaluation of Summaries. Text Summarization Branches Out, Association for Computational Linguistics."},{"key":"ref_42","unstructured":"Radziwill, N.M., and Benton, M.C. (2017). Evaluating Quality of Chatbots and Intelligent Conversational Agents. arXiv."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"746","DOI":"10.1093\/jamia\/ocad222","article-title":"Evaluation framework for conversational agents with artificial intelligence in health interventions: A systematic scoping review","volume":"31","author":"Ding","year":"2024","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"38","DOI":"10.1080\/19325037.2012.10599216","article-title":"Advantages and Limitations of the e-Delphi Technique","volume":"43","author":"Donohoe","year":"2012","journal-title":"Am. J. Health Educ."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"72","DOI":"10.1016\/j.techfore.2019.07.002","article-title":"Improving the practical application of the Delphi method in group-based judgment: A six-step prescription for a well-founded and defensible process","volume":"147","author":"Belton","year":"2019","journal-title":"Technol. Forecast. Soc. Change"},{"key":"ref_46","first-page":"655","article-title":"How to use the nominal group and Delphi techniques","volume":"38","author":"McMillan","year":"2016","journal-title":"Int. J. Clin. Pharm."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"684","DOI":"10.1177\/0269216317690685","article-title":"Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: Recommendations based on a methodological systematic review","volume":"31","author":"Payne","year":"2017","journal-title":"Palliat. Med."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"e52399","DOI":"10.2196\/52399","article-title":"Potential of Large Language Models in Health Care: Delphi Study","volume":"26","author":"Denecke","year":"2024","journal-title":"J. Med. Internet Res."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Maroengsit, W., Piyakulpinyo, T., Phonyiam, K., Pongnumkul, S., Chaovalit, P., and Theeramunkong, T. (2019, January 29\u201331). A Survey on Evaluation Methods for Chatbots. Proceedings of the 2019 7th International Conference on Information and Education Technology, Aizu-Wakamatsu, Japan.","DOI":"10.1145\/3323771.3323824"},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"171","DOI":"10.1055\/s-0041-1736664","article-title":"Evaluation Metrics for Health Chatbots: A Delphi Study","volume":"60","author":"Denecke","year":"2021","journal-title":"Methods Inf. Med."},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Guo, Z., Lai, A., Thygesen, J.H., Farrington, J., Keen, T., and Li, K. (2024). Large Language Model for Mental Health: A Systematic Review. arXiv.","DOI":"10.2196\/preprints.57400"},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41746-024-01258-7","article-title":"A Framework for Human Evaluation of Large Language Models in Healthcare Derived from Literature Review","volume":"7","author":"Tam","year":"2024","journal-title":"npj Digit. Med."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1145\/3641289","article-title":"A Survey on Evaluation of Large Language Models","volume":"15","author":"Chang","year":"2024","journal-title":"ACM Trans. Intell. Syst. Technol."},{"key":"ref_54","unstructured":"Peng, J.-L., Cheng, S., Diau, E., Shih, Y.-Y., Chen, P.-H., Lin, Y.-T., and Chen, Y.-N. (2024). A Survey of Useful LLM Evaluation. arXiv."},{"key":"ref_55","unstructured":"Qualtrics (2025, March 11). Qualtrics XM. Provo (UT): Qualtrics., Available online: https:\/\/www.qualtrics.com."},{"key":"ref_56","unstructured":"Mistral AI (2025, March 11). Mistral Large. Version 2407. Mistral AI: Paris, France. Available online: https:\/\/mistral.ai."},{"key":"ref_57","unstructured":"McKinney, W. (July, January 28). Data Structures for Statistical Computing in Python. Proceedings of the 9th Python in Science Conference, Austin, TX, USA."},{"key":"ref_58","doi-asserted-by":"crossref","first-page":"357","DOI":"10.1007\/s10602-022-09380-y","article-title":"Selecting a voting method: The case for the Borda count","volume":"34","author":"Saari","year":"2023","journal-title":"Const. Political Econ."},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"297","DOI":"10.1080\/17439760.2016.1262613","article-title":"Thematic analysis","volume":"12","author":"Clarke","year":"2017","journal-title":"J. Posit. Psychol."},{"key":"ref_60","doi-asserted-by":"crossref","first-page":"66","DOI":"10.1097\/LGT.0000000000000361","article-title":"Development of the Adult Vulvar Lichen Sclerosus Severity Scale\u2014A Delphi Consensus Exercise for Item Generation","volume":"22","author":"Sheinis","year":"2018","journal-title":"J. Low. Genit. Tract. Dis."},{"key":"ref_61","doi-asserted-by":"crossref","first-page":"247","DOI":"10.1007\/s40519-018-0556-4","article-title":"The Barcelona Orthorexia Scale (BOS): Development process using the Delphi method","volume":"24","author":"Bauer","year":"2019","journal-title":"Eat. Weight. Disord.\u2014Stud. Anorex. Bulim. Obes."},{"key":"ref_62","doi-asserted-by":"crossref","unstructured":"Xin, T., Ding, X., Gao, H., Li, C., Jiang, Y., and Chen, X. (2022). Using Delphi method to develop Chinese women\u2019s cervical cancer screening intention scale based on planned behavior theory. BMC Womens Health, 22.","DOI":"10.1186\/s12905-022-02113-1"},{"key":"ref_63","doi-asserted-by":"crossref","first-page":"84","DOI":"10.1186\/s43058-024-00618-4","article-title":"Development of the Technical Assistance Engagement Scale: A modified Delphi study","volume":"5","author":"Scott","year":"2024","journal-title":"Implement. Sci. Commun."},{"key":"ref_64","unstructured":"World Health Organization (2020). Doing What Matters in Times of Stress: An Illustrated Guide, World Health Organization."},{"key":"ref_65","doi-asserted-by":"crossref","first-page":"297","DOI":"10.1007\/BF02310555","article-title":"Coefficient Alpha and the Internal Structure of Tests","volume":"16","author":"Cronbach","year":"1951","journal-title":"Psychometrika"},{"key":"ref_66","doi-asserted-by":"crossref","first-page":"87","DOI":"10.1177\/001316445301300109","article-title":"The Correlation of an Item With a Composite of the Remaining Items in a Test","volume":"13","author":"Guilford","year":"1953","journal-title":"Educ. Psychol. Meas."},{"key":"ref_67","doi-asserted-by":"crossref","first-page":"53","DOI":"10.5116\/ijme.4dfb.8dfd","article-title":"Making sense of Cronbach\u2019s alpha","volume":"2","author":"Tavakol","year":"2011","journal-title":"Int. J. Med. Educ."},{"key":"ref_68","doi-asserted-by":"crossref","unstructured":"R\u00f6schel, A., Wagner, C., and D\u00fcr, M. (2021). Examination of validity, reliability, and interpretability of a self-reported questionnaire on Occupational Balance in Informal Caregivers (OBI-Care)\u2014A Rasch analysis. PLoS ONE, 16.","DOI":"10.1371\/journal.pone.0261815"},{"key":"ref_69","doi-asserted-by":"crossref","first-page":"104419","DOI":"10.1016\/j.brat.2023.104419","article-title":"Cognitive Therapy-as-Usual versus Cognitive Therapy plus the Memory Support Intervention for adults with depression: 12-month outcomes and opportunities for improved efficacy in a secondary analysis of a randomized controlled trial","volume":"170","author":"Zieve","year":"2023","journal-title":"Behav. Res. Ther."},{"key":"ref_70","doi-asserted-by":"crossref","first-page":"104167","DOI":"10.1016\/j.brat.2022.104167","article-title":"Can integrating the Memory Support Intervention into cognitive therapy improve depression outcome? A randomized controlled trial","volume":"157","author":"Dong","year":"2022","journal-title":"Behav. Res. Ther."},{"key":"ref_71","doi-asserted-by":"crossref","unstructured":"Jo, E., Jeong, Y., Park, S., Epstein, D.A., and Kim, Y.H. (2024, January 11\u201316). Understanding the Impact of Long-Term Memory on Self-Disclosure with Large Language Model-Driven Chatbots for Public Health Intervention. Proceedings of the CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA.","DOI":"10.1145\/3613904.3642420"},{"key":"ref_72","unstructured":"(2023). Systems and Software Engineering\u2014Systems and Software Quality Requirements and Evaluation (SQuaRE): Product Quality Model (Standard No. ISO\/IEC TS 25010:2023(en))."},{"key":"ref_73","doi-asserted-by":"crossref","unstructured":"Ouhbi, S., Idri, A., Fern\u00e1ndez-Alem\u00e1n, J.L., Toval, A., and Benjelloun, H. (2015, January 12\u201315). Applying ISO\/IEC 25010 on Mobile Personal Health Records. Proceedings of the BIOSTEC 2015: Proceedings of the International Joint Conference on Biomedical Engineering Systems and Technologies, Lisbon, Portugal.","DOI":"10.5220\/0005216604050412"},{"key":"ref_74","doi-asserted-by":"crossref","first-page":"632","DOI":"10.1007\/s11747-020-00762-y","article-title":"Understanding anthropomorphism in service provision: A meta-analysis of physical robots, chatbots, and other AI","volume":"49","author":"Blut","year":"2021","journal-title":"J. Acad. Mark. Sci."},{"key":"ref_75","doi-asserted-by":"crossref","unstructured":"Eyssel, F., and Reich, N. (2013, January 3\u20136). Loneliness makes the heart grow fonder (of robots)\u2014On the effects of loneliness on psychological anthropomorphism. Proceedings of the 2013 8th ACM\/IEEE International Conference on Human-Robot Interaction (HRI), Tokyo, Japan.","DOI":"10.1109\/HRI.2013.6483531"},{"key":"ref_76","unstructured":"Jessup, S., Schneider, T., Alarcon, G., Ryan, T., and Capiola, A. (2018). The Measurement of the Propensity to Trust Technology. [Master\u2019s Thesis, Wright University,]."}],"container-title":["Informatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2227-9709\/12\/1\/33\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T16:57:13Z","timestamp":1760029033000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2227-9709\/12\/1\/33"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,3,20]]},"references-count":76,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2025,3]]}},"alternative-id":["informatics12010033"],"URL":"https:\/\/doi.org\/10.3390\/informatics12010033","relation":{},"ISSN":["2227-9709"],"issn-type":[{"value":"2227-9709","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,3,20]]}}}