{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,6]],"date-time":"2026-06-06T01:14:54Z","timestamp":1780708494410,"version":"3.54.1"},"reference-count":53,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2022,1,27]],"date-time":"2022-01-27T00:00:00Z","timestamp":1643241600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"JST, Moonshot R&amp;D , Japan","award":["JPMJMS2011"],"award-info":[{"award-number":["JPMJMS2011"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Mental health issues are receiving more and more attention in society. In this paper, we introduce a preliminary study on human\u2013robot mental comforting conversation, to make an android robot (ERICA) present an understanding of the user\u2019s situation by sharing similar emotional experiences to enhance the perception of empathy. Specifically, we create the emotional speech for ERICA by using CycleGAN-based emotional voice conversion model, in which the pitch and spectrogram of the speech are converted according to the user\u2019s mental state. Then, we design dialogue scenarios for the user to talk about his\/her predicament with ERICA. In the dialogue, ERICA shares other people\u2019s similar predicaments and adopts a low-spirit voice to express empathy to the interlocutor\u2019s situation. At the end of the dialogue, ERICA tries to encourage with a positive voice. Subsequently, questionnaire-based evaluation experiments were conducted with the recorded conversation. In the questionnaire, we use the Big Five scale to evaluate ERICA\u2019s personality. In addition, the perception of emotion, empathy, and encouragement in the dialogue are evaluated. The results show that the proposed emotional expression strategy helps the android robot better present low-spirit emotion, empathy, the personality of extroversion, while making the user better feel the encouragement.<\/jats:p>","DOI":"10.3390\/s22030991","type":"journal-article","created":{"date-parts":[[2022,1,27]],"date-time":"2022-01-27T22:01:57Z","timestamp":1643320917000},"page":"991","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":22,"title":["A Preliminary Study on Realizing Human\u2013Robot Mental Comforting Dialogue via Sharing Experience Emotionally"],"prefix":"10.3390","volume":"22","author":[{"given":"Changzeng","family":"Fu","sequence":"first","affiliation":[{"name":"Graduate School of Engineering Science, Osaka University, Toyonaka 560-8531, Japan"},{"name":"Interactive Robot Research Team, Robotics Project, RIKEN, Kyoto 619-0237, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Qi","family":"Deng","sequence":"additional","affiliation":[{"name":"Graduate School of Intercultural Studies, Kobe University, Kobe 657-8501, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jingcheng","family":"Shen","sequence":"additional","affiliation":[{"name":"Graduate School of Information Science, Osaka University, Suita 565-0871, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6066-259X","authenticated-orcid":false,"given":"Hamed","family":"Mahzoon","sequence":"additional","affiliation":[{"name":"Graduate School of Engineering Science, Osaka University, Toyonaka 560-8531, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hiroshi","family":"Ishiguro","sequence":"additional","affiliation":[{"name":"Graduate School of Engineering Science, Osaka University, Toyonaka 560-8531, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,1,27]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Jeong, S., Logan, D.E., Goodwin, M.S., Graca, S., O\u2019Connell, B., Goodenough, H., Anderson, L., Stenquist, N., Fitzpatrick, K., and Zisook, M. (2015, January 2\u20135). A social robot to mitigate stress, anxiety, and pain in hospital pediatric care. Proceedings of the Tenth Annual ACM\/IEEE International Conference on Human-Robot Interaction Extended Abstracts, Portland, OR, USA.","DOI":"10.1145\/2701973.2702028"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Wada, K., and Shibata, T. (2006, January 6\u20138). Robot therapy in a care house-results of case studies. Proceedings of the ROMAN 2006\u2014The 15th IEEE International Symposium on Robot and Human Interactive Communication, Hatfield, UK.","DOI":"10.1109\/ROMAN.2006.314452"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1290","DOI":"10.1109\/TSMCB.2012.2228851","article-title":"Robotic emotional expression generation based on mood transition and personality model","volume":"43","author":"Han","year":"2012","journal-title":"IEEE Trans. Cybern."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Boccanfuso, L., Wang, Q., Leite, I., Li, B., Torres, C., Chen, L., Salomons, N., Foster, C., Barney, E., and Ahn, Y.A. (2016, January 26\u201331). A thermal emotion classifier for improved human-robot interaction. Proceedings of the 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), New York, NY, USA.","DOI":"10.1109\/ROMAN.2016.7745198"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"445","DOI":"10.1016\/j.neunet.2005.03.003","article-title":"Emotion understanding from the perspective of autonomous robots research","volume":"18","year":"2005","journal-title":"Neural Netw."},{"key":"ref_6","unstructured":"Miller, D. (2017). The Comfort of People, John Wiley & Sons."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"417","DOI":"10.1111\/j.1741-3737.2001.00417.x","article-title":"Daily variation in paternal engagement and negative mood: Implications for emotionally supportive and conflictual interactions","volume":"63","author":"Almeida","year":"2001","journal-title":"J. Marriage Fam."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"82","DOI":"10.1016\/j.learninstruc.2017.08.003","article-title":"Emotionally supportive classroom interactions and students\u2019 perceptions of their teachers as caring and just","volume":"54","author":"Gasser","year":"2018","journal-title":"Learn. Instr."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"463","DOI":"10.1007\/s11031-014-9394-5","article-title":"Motivational systems and preferences for social support strategies","volume":"38","author":"High","year":"2014","journal-title":"Motiv. Emot."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1093\/joc\/jqz040","article-title":"How the comforting process fails: Psychological reactance to support messages","volume":"70","author":"Tian","year":"2020","journal-title":"J. Commun."},{"key":"ref_11","unstructured":"Lee, D., Oh, K.J., and Choi, H.J. (2017, January 13\u201316). The chatbot feels you-a counseling service using emotional response generation. Proceedings of the 2017 IEEE International Conference on Big Data and Smart Computing (BigComp), Jeju, Korea."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"712","DOI":"10.1093\/joc\/jqy026","article-title":"Psychological, relational, and emotional effects of self-disclosure after conversations with a chatbot","volume":"68","author":"Ho","year":"2018","journal-title":"J. Commun."},{"key":"ref_13","first-page":"179","article-title":"Explaining recipient responses to supportive messages","volume":"159","author":"Burleson","year":"2010","journal-title":"New Dir. Interpers. Commun. Res."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1080\/10510974.2011.598208","article-title":"A review and meta-analysis of person-centered messages and social support outcomes","volume":"63","author":"High","year":"2012","journal-title":"Commun. Stud."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Sabelli, A.M., Kanda, T., and Hagita, N. (2011, January 8\u201311). A conversational robot in an elderly care center: An ethnographic study. Proceedings of the 2011 6th ACM\/IEEE International Conference on Human-Robot Interaction (HRI), Lausanne, Switzerland.","DOI":"10.1145\/1957656.1957669"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"341","DOI":"10.1007\/s12369-020-00643-y","article-title":"Sharing experiences to help a robot present its mind and sociability","volume":"13","author":"Fu","year":"2021","journal-title":"Int. J. Soc. Robot."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"6670","DOI":"10.1109\/LRA.2021.3094779","article-title":"Using an Android Robot to Improve Social Connectedness by Sharing Recent Experiences of Group Members in Human-Robot Conversations","volume":"6","author":"Fu","year":"2021","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"250","DOI":"10.1016\/j.ijhcs.2012.09.005","article-title":"The influence of empathy in human\u2013robot relations","volume":"71","author":"Leite","year":"2013","journal-title":"Int. J. Hum.-Comput. Stud."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"385","DOI":"10.1016\/j.robot.2006.02.004","article-title":"Using perspective taking to learn from ambiguous demonstrations","volume":"54","author":"Breazeal","year":"2006","journal-title":"Robot. Auton. Syst."},{"key":"ref_20","unstructured":"Torrey, C., Fussell, S.R., and Kiesler, S. (2009). What robots could teach us about perspective-taking. Expressing Oneself\/Expressing One\u2019s Self: Communication, Cognition, Language, and Identity, Psychology Press."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"335","DOI":"10.1007\/s10579-008-9076-6","article-title":"IEMOCAP: Interactive emotional dyadic motion capture database","volume":"42","author":"Busso","year":"2008","journal-title":"Lang. Resour. Eval."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Poria, S., Hazarika, D., Majumder, N., Naik, G., Cambria, E., and Mihalcea, R. (2018). Meld: A multimodal multi-party dataset for emotion recognition in conversations. arXiv.","DOI":"10.18653\/v1\/P19-1050"},{"key":"ref_23","first-page":"40","article-title":"Ten Item Personality Inventory (TIPI-J)","volume":"21","author":"Koshio","year":"2012","journal-title":"Panasonic Res."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1109\/MEMB.2008.919496","article-title":"Robot therapy for elders affected by dementia","volume":"27","author":"Wada","year":"2008","journal-title":"IEEE Eng. Med. Biol. Mag."},{"key":"ref_25","unstructured":"Pipitpukdee, J., and Phantachat, W. (2011, January 21\u201323). The study of the pet robot therapy in Thai autistic children. Proceedings of the 5th International Conference on Rehabilitation Engineering & Assistive Technology, Bangkok, Thailand."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Aminuddin, R., Sharkey, A., and Levita, L. (2016, January 7\u201310). Interaction with the Paro robot may reduce psychophysiological stress responses. Proceedings of the 2016 11th ACM\/IEEE International Conference on Human-Robot Interaction (HRI), Christchurch, New Zealand.","DOI":"10.1109\/HRI.2016.7451872"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"765","DOI":"10.1007\/s12369-019-00616-w","article-title":"Emotional and behavioural distraction by a social robot for children anxiety reduction during vaccination","volume":"12","author":"Rossi","year":"2020","journal-title":"Int. J. Soc. Robot."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Baecker, A.N., Geiskkovitch, D.Y., Gonz\u00e1lez, A.L., and Young, J.E. (2020, January 23\u201326). Emotional support domestic robots for healthy older adults: Conversational prototypes to help with loneliness. Proceedings of the Companion of the 2020 ACM\/IEEE International Conference on Human-Robot Interaction, Cambridge, UK.","DOI":"10.1145\/3371382.3378279"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"569","DOI":"10.1007\/s12369-018-0466-7","article-title":"The effects of humanlike and robot-specific affective nonverbal behavior on perception, emotion, and behavior","volume":"10","author":"Herrmann","year":"2018","journal-title":"Int. J. Soc. Robot."},{"key":"ref_30","unstructured":"Paiva, A., Leite, I., and Ribeiro, T. (2014). Emotion modeling for social robots. Oxf. Handb. Affect. Comput., 296\u2013308."},{"key":"ref_31","unstructured":"Xie, B., and Park, C.H. (2021). A MultiModal Social Robot Toward Personalized Emotion Interaction. arXiv."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Graterol, W., Diaz-Amado, J., Cardinale, Y., Dongo, I., Lopes-Silva, E., and Santos-Libarino, C. (2021). Emotion detection for social robots based on NLP transformers and an emotion ontology. Sensors, 21.","DOI":"10.3390\/s21041322"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Fu, C., Liu, C., Ishi, C.T., and Ishiguro, H. (2020). Multi-modality emotion recognition model with GAT-based multi-head inter-modality attention. Sensors, 20.","DOI":"10.3390\/s20174894"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Hegel, F., Spexard, T., Wrede, B., Horstmann, G., and Vogt, T. (2006, January 4\u20136). Playing a different imitation game: Interaction with an Empathic Android Robot. Proceedings of the 2006 6th IEEE-RAS International Conference on Humanoid Robots, Genova, Italy.","DOI":"10.1109\/ICHR.2006.321363"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1007\/s12193-009-0028-2","article-title":"When my robot smiles at me: Enabling human-robot rapport via real-time head gesture mimicry","volume":"3","author":"Riek","year":"2010","journal-title":"J. Multimodal User Interfaces"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"511","DOI":"10.2307\/1128810","article-title":"The early development of conceptual perspective taking: Distinguishing among multiple perspectives","volume":"47","author":"Marvin","year":"1976","journal-title":"Child Dev."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Crumpton, J., and Bethel, C. (2014, January 25\u201329). Conveying emotion in robotic speech: Lessons learned. Proceedings of the 23rd IEEE International Symposium on Robot and Human Interactive Communication, Edinburgh, UK.","DOI":"10.1109\/ROMAN.2014.6926265"},{"key":"ref_38","unstructured":"Nass, C., Foehr, U., Brave, S., and Somoza, M. (2001, January 2\u20134). The effects of emotion of voice in synthesized and recorded speech. Proceedings of the AAAI Symposium Emotional and Intelligent II: The Tangled Knot of Social Cognition, North Falmouth, MA, USA."},{"key":"ref_39","unstructured":"Roehling, S., MacDonald, B., and Watson, C. (2006). Towards expressive speech synthesis in english on a robotic platform. Proceedings of the Australasian International Conference on Speech Science and Technology, Auckland, New Zealand, 6\u20138 December 2006, Citeseer."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Lee, J. (2021). Generating Robotic Speech Prosody for Human Robot Interaction: A Preliminary Study. Appl. Sci., 11.","DOI":"10.3390\/app11083468"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"405","DOI":"10.1016\/j.ijnurstu.2005.06.004","article-title":"Emotional comfort: The patient\u2019s perspective of a therapeutic context","volume":"43","author":"Williams","year":"2006","journal-title":"Int. J. Nurs. Stud."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"123","DOI":"10.1002\/jclp.22486","article-title":"Emotion reactivity, comfort expressing emotions, and future suicidal ideation in emerging adults","volume":"74","author":"Moore","year":"2018","journal-title":"J. Clin. Psychol."},{"key":"ref_43","unstructured":"Asai, S., Yoshino, K., Shinagawa, S., Sakti, S., and Nakamura, S. (2020, January 11\u201316). Emotional speech corpus for persuasive dialogue system. Proceedings of the 12th Language Resources and Evaluation Conference, Marseille, France."},{"key":"ref_44","unstructured":"Liu, S., Cao, Y., and Meng, H. (2020). Emotional Voice Conversion With Cycle-consistent Adversarial Network. arXiv."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Kaneko, T., and Kameoka, H. (2018, January 3\u20137). Cyclegan-vc: Non-parallel voice conversion using cycle-consistent adversarial networks. Proceedings of the 2018 IEEE 26th European Signal Processing Conference (EUSIPCO), Rome, Italy.","DOI":"10.23919\/EUSIPCO.2018.8553236"},{"key":"ref_46","unstructured":"Fu, C., Liu, C., Ishi Toshinori, C., and Ishiguro, H. (2021). CycleTransGAN-EVC: A CycleGAN-based Emotional Voice Conversion Model with Transformer. arXiv."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Zhou, K., Sisman, B., and Li, H. (2020). Transforming spectrum and prosody for emotional voice conversion with non-parallel training data. arXiv.","DOI":"10.21437\/Odyssey.2020-33"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.specom.2014.09.003","article-title":"CheapTrick, a spectral envelope estimator for high-quality speech synthesis","volume":"67","author":"Morise","year":"2015","journal-title":"Speech Commun."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Andrist, S., Mutlu, B., and Tapus, A. (2015, January 18\u201323). Look like me: Matching robot personality via gaze to increase motivation. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, Seoul, Korea.","DOI":"10.1145\/2702123.2702592"},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Goetz, J., and Kiesler, S. (2002, January 20\u201325). Cooperation with a robotic assistant. Proceedings of the CHI\u201902 Extended Abstracts on Human Factors in Computing Systems, Minneapolis, MN, USA.","DOI":"10.1145\/506443.506492"},{"key":"ref_51","unstructured":"Robert, L. (2018, January 16\u201318). Personality in the human robot interaction literature: A review and brief critique. Proceedings of the 24th Americas Conference on Information Systems, New Orleans, LA, USA. Available online: https:\/\/ssrn.com\/abstract=3308191."},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Uchida, T., Takahashi, H., Ban, M., Shimaya, J., Minato, T., Ogawa, K., Yoshikawa, Y., and Ishiguro, H. (2020). Japanese Young Women did not discriminate between robots and humans as listeners for their self-disclosure-pilot study. Multimodal Technol. Interact., 4.","DOI":"10.3390\/mti4030035"},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1007\/s12369-016-0340-4","article-title":"The influence of prior expectations of a robot\u2019s lifelikeness on users\u2019 intentions to treat a zoomorphic robot as a companion","volume":"9","author":"Allouch","year":"2017","journal-title":"Int. J. Soc. Robot."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/3\/991\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:09:01Z","timestamp":1760134141000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/3\/991"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,1,27]]},"references-count":53,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2022,2]]}},"alternative-id":["s22030991"],"URL":"https:\/\/doi.org\/10.3390\/s22030991","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,1,27]]}}}