{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T03:50:18Z","timestamp":1760241018829,"version":"build-2065373602"},"reference-count":55,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2019,11,9]],"date-time":"2019-11-09T00:00:00Z","timestamp":1573257600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Seventh Framework Programme","award":["611391"],"award-info":[{"award-number":["611391"]}]},{"name":"imec.icon","award":["ROBO-CURE"],"award-info":[{"award-number":["ROBO-CURE"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Robotics"],"abstract":"<jats:p>Maintaining engagement is challenging in human\u2013human interaction. When disengagements happen, people try to adapt their behavior with an expectation that engagement will be regained. In human\u2013robot interaction, although socially interactive robots are engaging, people can easily drop engagement while interacting with robots. This paper proposes a multi-layer re-engagement system that applies different strategies through human-like verbal and non-verbal behaviors to regain user engagement, taking into account the user\u2019s attention level and affective states. We conducted a usability test in a robot storytelling scenario to demonstrate technical operation of the system as well as to investigate how people react when interacting with a robot with re-engagement ability. Our usability test results reveal that the system has the potential to maintain a user\u2019s engagement. Our selected users gave positive comments, through open-ended questions, to the robot with this ability. They also rated the robot with the re-engagement ability higher on several dimensions, i.e., animacy, likability, and perceived intelligence.<\/jats:p>","DOI":"10.3390\/robotics8040095","type":"journal-article","created":{"date-parts":[[2019,11,12]],"date-time":"2019-11-12T04:07:07Z","timestamp":1573531627000},"page":"95","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["\u201cHmm, Did You Hear What I Just Said?\u201d: Development of a Re-Engagement System for Socially Interactive Robots"],"prefix":"10.3390","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2851-5527","authenticated-orcid":false,"given":"Hoang-Long","family":"Cao","sequence":"first","affiliation":[{"name":"Brussels Human Robotics Research Center, Vrije Universiteit Brussel and Flanders Make, 1050 Brussels, Belgium"},{"name":"College of Engineering Technology, Can Tho University, Can Tho 90000, Vietnam"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Paola Cecilia","family":"Torrico Moron","sequence":"additional","affiliation":[{"name":"Brussels Human Robotics Research Center, Vrije Universiteit Brussel and Flanders Make, 1050 Brussels, Belgium"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Pablo G.","family":"Esteban","sequence":"additional","affiliation":[{"name":"Brussels Human Robotics Research Center, Vrije Universiteit Brussel and Flanders Make, 1050 Brussels, Belgium"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8570-6613","authenticated-orcid":false,"given":"Albert","family":"De Beir","sequence":"additional","affiliation":[{"name":"Brussels Human Robotics Research Center, Vrije Universiteit Brussel and Flanders Make, 1050 Brussels, Belgium"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Elahe","family":"Bagheri","sequence":"additional","affiliation":[{"name":"Brussels Human Robotics Research Center, Vrije Universiteit Brussel and Flanders Make, 1050 Brussels, Belgium"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dirk","family":"Lefeber","sequence":"additional","affiliation":[{"name":"Brussels Human Robotics Research Center, Vrije Universiteit Brussel and Flanders Make, 1050 Brussels, Belgium"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4881-9341","authenticated-orcid":false,"given":"Bram","family":"Vanderborght","sequence":"additional","affiliation":[{"name":"Brussels Human Robotics Research Center, Vrije Universiteit Brussel and Flanders Make, 1050 Brussels, Belgium"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2019,11,9]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"143","DOI":"10.1016\/S0921-8890(02)00372-X","article-title":"A survey of socially interactive robots","volume":"42","author":"Fong","year":"2003","journal-title":"Robot. Auton. Syst."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"261","DOI":"10.1007\/s12369-013-0226-7","article-title":"Recognizing emotional body language displayed by a human-like social robot","volume":"6","author":"McColl","year":"2014","journal-title":"Int. J. Soc. Robot."},{"key":"ref_3","unstructured":"Belpaeme, T., Kennedy, J., Baxter, P., Vogt, P., Krahmer, E.E., Kopp, S., Bergmann, K., Leseman, P., K\u00fcntay, A.C., and G\u00f6ksun, T. (2015, January 26\u201330). L2TOR-second language tutoring using social robots. Proceedings of the ICSR 2015 WONDER Workshop, Paris, France."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"73","DOI":"10.3389\/fnhum.2017.00073","article-title":"Child-robot interactions for second language tutoring to preschool children","volume":"11","author":"Vogt","year":"2017","journal-title":"Front. Hum. Neurosci."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"18","DOI":"10.1515\/pjbr-2017-0002","article-title":"How to build a supervised autonomous system for robot-enhanced therapy for children with autism spectrum disorder","volume":"8","author":"Esteban","year":"2017","journal-title":"Paladyn J. Behav. Robot."},{"key":"ref_6","first-page":"33","article-title":"Multimodal child-robot interaction: Building social bonds","volume":"1","author":"Belpaeme","year":"2012","journal-title":"J. Hum. Robot. Interact."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1109\/MRA.2019.2904121","article-title":"Robot-enhanced therapy: Development and validation of a supervised autonomous robotic system for autism spectrum disorders therapy","volume":"26","author":"Cao","year":"2019","journal-title":"IEEE Robot. Autom. Mag."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Loza-Matovelle, D., Verdugo, A., Zalama, E., and G\u00f3mez-Garc\u00eda-Bermejo, J. (2019). An Architecture for the Integration of Robots and Sensors for the Care of the Elderly in an Ambient Assisted Living Environment. Robotics, 8.","DOI":"10.3390\/robotics8030076"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Palac\u00edn, J., Clotet, E., Mart\u00ednez, D., Mart\u00ednez, D., and Moreno, J. (2019). Extending the Application of an Assistant Personal Robot as a Walk-Helper Tool. Robotics, 8.","DOI":"10.3390\/robotics8020027"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1016\/S0004-3702(99)00070-3","article-title":"Experiences with an interactive museum tour-guide robot","volume":"114","author":"Burgard","year":"1999","journal-title":"Artif. Intell."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Yamazaki, A., Yamazaki, K., Ohyama, T., Kobayashi, Y., and Kuno, Y. (2012, January 5\u20138). A techno-sociological solution for designing a museum guide robot: Regarding choosing an appropriate visitor. Proceedings of the 2012 7th ACM\/IEEE International Conference on Human-Robot Interaction (HRI), Boston, MA, USA.","DOI":"10.1145\/2157689.2157800"},{"key":"ref_12","unstructured":"Kidd, C.D., and Breazeal, C. (October, January 28). Effect of a robot on user perceptions. Proceedings of the IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS 2004), Sendai, Japan."},{"key":"ref_13","unstructured":"Xu, J. (2015). Affective Body Language of Humanoid Robots: Perception and Effects in Human Robot Interaction. [Ph.D. Thesis, Delft University of Technology]."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Mower, E., Feil-Seifer, D.J., Mataric, M.J., and Narayanan, S. (2007, January 26\u201329). Investigating implicit cues for user state estimation in human-robot interaction using physiological measurements. Proceedings of the 16th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN 2007), Jeju, Korea.","DOI":"10.1109\/ROMAN.2007.4415249"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Pitsch, K., Kuzuoka, H., Suzuki, Y., Sussenbach, L., Luff, P., and Heath, C. (October, January 27). \u201cThe first five seconds\u201d: Contingent stepwise entry into an interaction as a means to secure sustained engagement in HRI. Proceedings of the 18th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN 2009), Toyama, Japan.","DOI":"10.1109\/ROMAN.2009.5326167"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"943","DOI":"10.1080\/10447318.2017.1300750","article-title":"Adaptive social robot for sustaining social engagement during long-term children\u2013robot interaction","volume":"33","author":"Ahmad","year":"2017","journal-title":"Int. J. Hum. Comput. Interact."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"32","DOI":"10.5898\/JHRI.5.1.Coninx","article-title":"Towards long-term social child-robot interaction: using multi-activity switching to engage young users","volume":"5","author":"Coninx","year":"2016","journal-title":"J. Hum. Robot. Interact."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Komatsubara, T., Shiomi, M., Kanda, T., Ishiguro, H., and Hagita, N. (2014, January 29\u201331). Can a social robot help children\u2019s understanding of science in classrooms?. Proceedings of the Second International Conference on Human-Agent Interaction, Tsukuba, Japan.","DOI":"10.1145\/2658861.2658881"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1515\/jaiscr-2015-0018","article-title":"An emotional expression model for educational-support robots","volume":"5","author":"Jimenez","year":"2015","journal-title":"J. Artif. Intell. Soft Comput. Res."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Moshkina, L., Trickett, S., and Trafton, J.G. (2014, January 3\u20136). Social engagement in public places: A tale of one robot. Proceedings of the 2014 ACM\/IEEE international conference on Human-Robot Interaction, Bielefeld, Germany.","DOI":"10.1145\/2559636.2559678"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"63","DOI":"10.1007\/s12369-016-0357-8","article-title":"Towards engagement models that consider individual factors in HRI: On the relation of extroversion and negative attitude towards robots to gaze and speech during a human\u2013robot assembly task","volume":"9","author":"Ivaldi","year":"2017","journal-title":"Int. J. Soc. Robot."},{"key":"ref_22","unstructured":"Kuno, Y., Sadazuka, K., Kawashima, M., Yamazaki, K., Yamazaki, A., and Kuzuoka, H. (May, January 28). Museum guide robot based on sociological interaction analysis. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, San Jose, CA, USA."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"140","DOI":"10.1016\/j.artint.2005.03.005","article-title":"Explorations in engagement for humans and robots","volume":"166","author":"Sidner","year":"2005","journal-title":"Artif. Intell."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"2398","DOI":"10.1016\/j.pragma.2009.12.023","article-title":"Coordination of verbal and non-verbal actions in human-robot interaction at museums and exhibitions","volume":"42","author":"Yamazaki","year":"2010","journal-title":"J. Pragmat."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Corrigan, L.J., Basedow, C., K\u00fcster, D., Kappas, A., Peters, C., and Castellano, G. (September, January 31). Perception matters! Engagement in task orientated social robotics. Proceedings of the 2015 24th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), Kobe, Japan.","DOI":"10.1109\/ROMAN.2015.7333665"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Sidner, C.L., Kidd, C.D., Lee, C., and Lesh, N. (2004, January 13\u201316). Where to look: A study of human-robot engagement. Proceedings of the 9th International Conference on Intelligent User Interfaces, Funchal, Portugal.","DOI":"10.1145\/964442.964458"},{"key":"ref_27","unstructured":"Bohus, D., and Horvitz, E. (2014, January 12\u201316). Managing human-robot engagement with forecasts and... um... hesitations. Proceedings of the 16th International Conference on Multimodal Interaction, Istanbul, Turkey."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Chan, J., and Nejat, G. (2010, January 6\u20139). Promoting engagement in cognitively stimulating activities using an intelligent socially assistive robot. Proceedings of the 2010 IEEE\/ASME International Conference on Advanced Intelligent Mechatronics (AIM), Montreal, ON, Canada.","DOI":"10.1109\/AIM.2010.5695806"},{"key":"ref_29","first-page":"13","article-title":"A review of the applicability of robots in education","volume":"1","author":"Mubin","year":"2013","journal-title":"J. Technol. Educ. Learn."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"329","DOI":"10.1007\/s12369-014-0227-1","article-title":"Empathic robots for long-term interaction","volume":"6","author":"Leite","year":"2014","journal-title":"Int. J. Soc. Robot."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"40","DOI":"10.1109\/MRA.2018.2833157","article-title":"A Mass-Produced Sociable Humanoid Robot: Pepper: The First Machine of Its Kind","volume":"25","author":"Pandey","year":"2018","journal-title":"IEEE Robot. Autom. Mag."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"368","DOI":"10.1177\/001872089403600215","article-title":"Sample sizes for usability studies: Additional considerations","volume":"36","author":"Lewis","year":"1994","journal-title":"Hum. Factors"},{"key":"ref_33","unstructured":"Nielsen, J. (2019, November 08). Usability 101: Introduction to Usability. Available online: https:\/\/www.nngroup.com\/articles\/usability-101-introduction-to-usability\/."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"457","DOI":"10.1177\/001872089203400407","article-title":"Refining the test phase of usability evaluation: How many subjects is enough?","volume":"34","author":"Virzi","year":"1992","journal-title":"Hum. Factors"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"380","DOI":"10.1177\/0956797616685870","article-title":"The novelty penalty: Why do people like talking about new experiences but hearing about old ones?","volume":"28","author":"Cooney","year":"2017","journal-title":"Psychol. Sci."},{"key":"ref_36","unstructured":"Ortony, A., Clore, G.L., and Collins, A. (1990). The Cognitive Structure of Emotions, Cambridge University Press."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Ortony, A., Norman, D., and Revelle, W. (2005). Affect and Proto-Affect in Effective Functioning. Who Needs Emotions, The Brain Meets the Robot, Oxford University Press.","DOI":"10.1093\/acprof:oso\/9780195166194.003.0007"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Sloman, A. (2011). Varieties of meta-cognition in natural and artificial systems. Metareasoning: Thinking about Thinking, MIT Press.","DOI":"10.7551\/mitpress\/9780262014809.003.0020"},{"key":"ref_39","unstructured":"Sloman, A., and Logan, B. (2000). Evolvable Architectures for Human-Like Minds. Affective Minds, Elsevier."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Mutlu, B., Forlizzi, J., and Hodgins, J. (2006, January 4\u20136). A storytelling robot: Modeling and evaluation of human-like gaze behavior. Proceedings of the 2006 6th IEEE-RAS International Conference on Humanoid Robots, Genova, Italy.","DOI":"10.1109\/ICHR.2006.321322"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Yoshikawa, Y., Shinozawa, K., Ishiguro, H., Hagita, N., and Miyamoto, T. (2006, January 16\u201319). Responsive Robot Gaze to Interaction Partner. Proceedings of the Robotics: Science and Systems, Philadelphia, PA, USA.","DOI":"10.15607\/RSS.2006.II.037"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Lazzeri, N., Mazzei, D., Zaraki, A., and De Rossi, D. (2013). Towards a believable social robot. Biomimetic and Biohybrid Systems, Springer.","DOI":"10.1007\/978-3-642-39802-5_45"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Saldien, J., Vanderborght, B., Goris, K., Van Damme, M., and Lefeber, D. (2014). A motion system for social and animated robots. Int. J. Adv. Robot. Syst., 11.","DOI":"10.5772\/58402"},{"key":"ref_44","unstructured":"G\u00f3mez Esteban, P., Cao, H.L., De Beir, A., Van de Perre, G., Lefeber, D., and Vanderborght, B. (2016, January 5\u20136). A multilayer reactive system for robots interacting with children with autism. Proceedings of the 5th International Symposium on New Frontiers in Human-Robot Interaction, Sheffield, UK."},{"key":"ref_45","first-page":"574","article-title":"The relationship between selected immediacy behaviors and cognitive learning","volume":"10","author":"Richmond","year":"1987","journal-title":"Ann. Int. Commun. Assoc."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Shim, J., and Arkin, R.C. (2014, January 5\u201310). Other-oriented robot deception: A computational approach for deceptive action generation to benefit the mark. Proceedings of the 2014 IEEE International Conference on Robotics and Biomimetics (ROBIO 2014), Bali, Indonesia.","DOI":"10.1109\/ROBIO.2014.7090385"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Szafir, D., and Mutlu, B. (2012, January 5\u201310). Pay attention!: Designing adaptive agents that monitor and improve user engagement. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Austin, TX, USA.","DOI":"10.1145\/2207676.2207679"},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Baroni, I., Nalin, M., Zelati, M.C., Oleari, E., and Sanna, A. (2014, January 25\u201329). Designing motivational robot: How robots might motivate children to eat fruits and vegetables. Proceedings of the 23rd IEEE International Symposium on Robot and Human Interactive Communication (2014 RO-MAN), Edinburgh, UK.","DOI":"10.1109\/ROMAN.2014.6926350"},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Chidambaram, V., Chiang, Y.H., and Mutlu, B. (2012, January 5\u20138). Designing persuasive robots: How robots might persuade people using vocal and nonverbal cues. Proceedings of the Seventh Annual ACM\/IEEE International Conference on Human-Robot Interaction, Boston, MA, USA.","DOI":"10.1145\/2157689.2157798"},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Brown, L., Kerwin, R., and Howard, A.M. (2013, January 13\u201316). Applying behavioral strategies for student engagement using a robotic educational agent. Proceedings of the 2013 IEEE International Conference on Systems, Man, and Cybernetics, Manchester, UK.","DOI":"10.1109\/SMC.2013.744"},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"271","DOI":"10.1007\/s12369-015-0329-4","article-title":"A survey of using vocal prosody to convey emotion in robot speech","volume":"8","author":"Crumpton","year":"2016","journal-title":"Int. J. Soc. Robot."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"126","DOI":"10.1109\/TAMD.2014.2317513","article-title":"The mei robot: Towards using motherese to develop multimodal emotional intelligence","volume":"6","author":"Lim","year":"2014","journal-title":"IEEE Trans. Auton. Ment. Dev."},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Bennewitz, M., Faber, F., Joho, D., and Behnke, S. (2007). Intuitive multimodal interaction with communication robot Fritz. Humanoid Robots, Human-Like Machines, Itech.","DOI":"10.5772\/4826"},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Gonsior, B., Sosnowski, S., Bu\u00df, M., Wollherr, D., and K\u00fchnlenz, K. (2012, January 7\u201312). An emotional adaption approach to increase helpfulness towards a robot. Proceedings of the 2012 IEEE\/RSJ International Conference on Intelligent Robots and Systems, Vilamoura, Portugal.","DOI":"10.1109\/IROS.2012.6385941"},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1007\/s12369-008-0001-3","article-title":"Measurement instruments for the anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of robots","volume":"1","author":"Bartneck","year":"2009","journal-title":"Int. J. Soc. Robot."}],"container-title":["Robotics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2218-6581\/8\/4\/95\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T13:33:17Z","timestamp":1760189597000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2218-6581\/8\/4\/95"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,11,9]]},"references-count":55,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2019,12]]}},"alternative-id":["robotics8040095"],"URL":"https:\/\/doi.org\/10.3390\/robotics8040095","relation":{},"ISSN":["2218-6581"],"issn-type":[{"type":"electronic","value":"2218-6581"}],"subject":[],"published":{"date-parts":[[2019,11,9]]}}}