{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,2]],"date-time":"2026-06-02T15:24:10Z","timestamp":1780413850076,"version":"3.54.1"},"reference-count":40,"publisher":"MDPI AG","issue":"11","license":[{"start":{"date-parts":[[2025,11,11]],"date-time":"2025-11-11T00:00:00Z","timestamp":1762819200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000181","name":"Air Force Office of Scientific Research","doi-asserted-by":"crossref","award":["FA2386-23-1-4098"],"award-info":[{"award-number":["FA2386-23-1-4098"]}],"id":[{"id":"10.13039\/100000181","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>With the rapid advancement of large language models (LLMs), recent systems have demonstrated increasing capability in understanding and expressing human emotions. However, no objective and standardized metric currently exists to evaluate how empathetic an LLM\u2019s response is. To address this gap, we propose a novel evaluation framework that measures both sentiment-level and emotion-level alignment between a user query and a model-generated response. The proposed metric consists of two components. The sentiment component evaluates overall affective polarity through Sentlink and the naturalness of emotional expression via NEmpathySort. The emotion component measures fine-grained emotional correspondence using Emosight. Additionally, a semantic component, based on RAGAS, assesses the contextual relevance and coherence of the response. Experimental results demonstrate that our metric effectively captures both the intensity and nuance of empathy in LLM-generated responses, providing a solid foundation for the development of emotionally intelligent conversational AI.<\/jats:p>","DOI":"10.3390\/info16110977","type":"journal-article","created":{"date-parts":[[2025,11,11]],"date-time":"2025-11-11T14:28:59Z","timestamp":1762871339000},"page":"977","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Performance Evaluation Metrics for Empathetic LLMs"],"prefix":"10.3390","volume":"16","author":[{"given":"Yuna","family":"Hong","sequence":"first","affiliation":[{"name":"Department of Electrical and Computer Engineering, Korea University, Seoul 02841, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bonhwa","family":"Ku","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, Korea University, Seoul 02841, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8744-4514","authenticated-orcid":false,"given":"Hanseok","family":"Ko","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, Korea University, Seoul 02841, Republic of Korea"},{"name":"Department of Electrical and Computer Engineering, Catholic University of America, Washington, DC 20064, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2025,11,11]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"583","DOI":"10.1109\/TCE.2006.1649683","article-title":"Dual channel based speech enhancement using novelty filter for robust speech recognition in automobile environment","volume":"52","author":"Beh","year":"2006","journal-title":"IEEE Trans. Consum. Electron."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"22","DOI":"10.1109\/TCE.2005.1405694","article-title":"Background noise reduction via dual-channel scheme for speech recognition in vehicular environment","volume":"51","author":"Ahn","year":"2005","journal-title":"IEEE Trans. Consum. Electron."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"131","DOI":"10.1109\/LSP.2019.2959917","article-title":"Spectro-temporal attention-based voice activity detection","volume":"27","author":"Lee","year":"2019","journal-title":"IEEE Signal Process. Lett."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Kwak, J.g., Dong, E., Jin, Y., Ko, H., Mahajan, S., and Yi, K.M. (2024, January 16\u201322). Vivid-1-to-3: Novel view synthesis with video diffusion models. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR52733.2024.00647"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Kwak, J.g., Li, Y., Yoon, D., Kim, D., Han, D., and Ko, H. (2022, January 23\u201327). Injecting 3d perception of controllable nerf-gan into stylegan for editable portrait image synthesis. Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel.","DOI":"10.1007\/978-3-031-19790-1_15"},{"key":"ref_6","unstructured":"Vaswani, A. (2017). Attention is all you need. Adv. Neural Inf. Process. Syst., 5998\u20136008."},{"key":"ref_7","unstructured":"Kenton, J.D.M.W.C., and Toutanova, L.K. (2019, January 6\u20137). Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT), Minneapolis, MN, USA."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1016\/j.iotcps.2023.04.003","article-title":"ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope","volume":"3","author":"Ray","year":"2023","journal-title":"Internet Things Cyber-Phys. Syst."},{"key":"ref_9","unstructured":"Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., and Anadkat, S. (2023). Gpt-4 technical report. arXiv."},{"key":"ref_10","unstructured":"Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozi\u00e8re, B., Goyal, N., Hambro, E., and Azhar, F. (2023). Llama: Open and efficient foundation language models. arXiv."},{"key":"ref_11","unstructured":"Team, G., Anil, R., Borgeaud, S., Alayrac, J.B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A.M., Hauth, A., and Millican, K. (2023). Gemini: A family of highly capable multimodal models. arXiv."},{"key":"ref_12","unstructured":"Jiang, A.Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D.S., Casas, D.d.l., Bressand, F., Lengyel, G., Lample, G., and Saulnier, L. (2023). Mistral 7B. arXiv."},{"key":"ref_13","first-page":"9459","article-title":"Retrieval-augmented generation for knowledge-intensive nlp tasks","volume":"33","author":"Lewis","year":"2020","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Fu, J., Ng, S.K., Jiang, Z., and Liu, P. (2023). Gptscore: Evaluate as you desire. arXiv.","DOI":"10.18653\/v1\/2024.naacl-long.365"},{"key":"ref_15","unstructured":"Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., and Artzi, Y. (2019). Bertscore: Evaluating text generation with bert. arXiv."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Lewis, M. (2019). Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv.","DOI":"10.18653\/v1\/2020.acl-main.703"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Es, S., James, J., Espinosa-Anke, L., and Schockaert, S. (2023). Ragas: Automated evaluation of retrieval augmented generation. arXiv.","DOI":"10.18653\/v1\/2024.eacl-demo.16"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Liu, Z., Yang, K., Xie, Q., Zhang, T., and Ananiadou, S. (2024, January 25\u201329). Emollms: A series of emotional large language models and annotation tools for comprehensive affective analysis. Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Barcelona, Spain.","DOI":"10.1145\/3637528.3671552"},{"key":"ref_19","unstructured":"Jia, Y., Cao, S., Niu, C., Ma, Y., Zan, H., Chao, R., and Zhang, W. (2021). EmoDialoGPT: Enhancing DialoGPT with emotion. Proceedings Part II 10, Proceedings of the Natural Language Processing and Chinese Computing: 10th CCF International Conference (NLPCC 2021), Qingdao, China, 13\u201317 October 2021, ACM Digital Library."},{"key":"ref_20","unstructured":"Yuan, W., Neubig, G., and Liu, P. (2021, January 6\u201314). BARTScore: Evaluating Generated Text as Text Generation. Proceedings of the Advances in Neural Information Processing Systems, Virtual."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Hutto, C., and Gilbert, E. (2014, January 1\u20134). Vader: A parsimonious rule-based model for sentiment analysis of social media text. Proceedings of the International AAAI Conference on Web and Social Media, Ann Arbor, MI, USA.","DOI":"10.1609\/icwsm.v8i1.14550"},{"key":"ref_22","unstructured":"Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Demszky, D., Movshovitz-Attias, D., Ko, J., Cowen, A., Nemade, G., and Ravi, S. (2020). GoEmotions: A dataset of fine-grained emotions. arXiv.","DOI":"10.18653\/v1\/2020.acl-main.372"},{"key":"ref_24","unstructured":"Xiao, S., Liu, Z., Zhang, P., and Muennighoff, N. (2023). C-Pack: Packaged Resources To Advance General Chinese Embedding. arXiv."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Huang, L., Bras, R.L., Bhagavatula, C., and Choi, Y. (2019). Cosmos QA: Machine reading comprehension with contextual commonsense reasoning. arXiv.","DOI":"10.18653\/v1\/D19-1243"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. (2016). Squad: 100,000+ questions for machine comprehension of text. arXiv.","DOI":"10.18653\/v1\/D16-1264"},{"key":"ref_27","unstructured":"Li, Y., Su, H., Shen, X., Li, W., Cao, Z., and Niu, S. (December, January 27). DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset. Proceedings of the Proceedings of the 8th International Joint Conference on Natural Language Processing (IJCNLP 2017), Taipei, Taiwan."},{"key":"ref_28","unstructured":"Valero, L.A.M. (2025, August 20). Empathetic_counseling_Dataset. Available online: https:\/\/huggingface.co\/datasets\/LuangMV97\/Empathetic_counseling_Dataset."},{"key":"ref_29","unstructured":"Rashkin, H., Smith, E.M., Li, M., and Boureau, Y.L. (August, January 28). Towards Empathetic Open-domain Conversation Models: A New Benchmark and Dataset. Proceedings of the ACL, Florence, Italy."},{"key":"ref_30","unstructured":"Amod (2025, August 20). Amod\/mental_health_counseling_conversations. Available online: https:\/\/huggingface.co\/datasets\/Amod\/mental_health_counseling_conversations."},{"key":"ref_31","unstructured":"EmoCareAI (2025, August 20). ChatPsychiatrist. Available online: https:\/\/github.com\/EmoCareAI\/ChatPsychiatrist."},{"key":"ref_32","unstructured":"Bertagnolli, N. (2025, August 20). Counsel Chat: Bootstrapping High-Quality Therapy Data. Available online: https:\/\/medium.com\/data-science\/counsel-chat-bootstrapping-high-quality-therapy-data-971b419f33da."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Poria, S., Hazarika, D., Majumder, N., Naik, G., Cambria, E., and Mihalcea, R. (2018). Meld: A multimodal multi-party dataset for emotion recognition in conversations. arXiv.","DOI":"10.18653\/v1\/P19-1050"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Lee, B., Hong, J., Shin, H., Ku, B., and Ko, H. (March, January 26). Dropout Connects Transformers and CNNs: Transfer General Knowledge for Knowledge Distillation. Proceedings of the 2025 IEEE\/CVF Winter Conference on Applications of Computer Vision (WACV), Tucson, AZ, USA.","DOI":"10.1109\/WACV61041.2025.00809"},{"key":"ref_35","first-page":"12449","article-title":"wav2vec 2.0: A framework for self-supervised learning of speech representations","volume":"33","author":"Baevski","year":"2020","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Livingstone, S.R., and Russo, F.A. (2018). The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in North American English. PLoS ONE, 13.","DOI":"10.1371\/journal.pone.0196391"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"377","DOI":"10.1109\/TAFFC.2014.2336244","article-title":"Crema-d: Crowd-sourced emotional multimodal actors dataset","volume":"5","author":"Cao","year":"2014","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_38","unstructured":"ONNX Runtime (2025, August 20). Available online: https:\/\/onnxruntime.ai\/."},{"key":"ref_39","unstructured":"Google LLC (2025, August 20). Speech-to-Text AI: Speech Recognition and Transcription. Available online: https:\/\/cloud.google.com\/speech-to-text."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"82","DOI":"10.1037\/h0059730","article-title":"The attitude and orientation of the counselor in client-centered therapy","volume":"13","author":"Rogers","year":"1949","journal-title":"J. Consult. Psychol."}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/16\/11\/977\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,11,13]],"date-time":"2025-11-13T05:25:44Z","timestamp":1763011544000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/16\/11\/977"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,11,11]]},"references-count":40,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2025,11]]}},"alternative-id":["info16110977"],"URL":"https:\/\/doi.org\/10.3390\/info16110977","relation":{},"ISSN":["2078-2489"],"issn-type":[{"value":"2078-2489","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,11,11]]}}}