{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,9]],"date-time":"2026-03-09T05:16:13Z","timestamp":1773033373875,"version":"3.50.1"},"reference-count":81,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2020,10,20]],"date-time":"2020-10-20T00:00:00Z","timestamp":1603152000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100000155","name":"Social Sciences and Humanities Research Council of Canada","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100000155","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["J. Hum.-Robot Interact."],"published-print":{"date-parts":[[2021,3,31]]},"abstract":"<jats:p>Physical agents that can autonomously generate engaging, life-like behavior will lead to more responsive and user-friendly robots and other autonomous systems. Although many advances have been made for one-to-one interactions in well-controlled settings, physical agents should be capable of interacting with humans in natural settings, including group interaction. To generate engaging behaviors, the autonomous system must first be able to estimate its human partners\u2019 engagement level. In this article, we propose an approach for estimating engagement during group interaction by simultaneously taking into account active and passive interaction, and use the measure as the reward signal within a reinforcement learning framework to learn engaging interactive behaviors. The proposed approach is implemented in an interactive sculptural system in a museum setting. We compare the learning system to a baseline using pre-scripted interactive behaviors. Analysis based on sensory data and survey data shows that adaptable behaviors within an expert-designed action space can achieve higher engagement and likeability.<\/jats:p>","DOI":"10.1145\/3408876","type":"journal-article","created":{"date-parts":[[2020,10,20]],"date-time":"2020-10-20T12:20:19Z","timestamp":1603196419000},"page":"1-29","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":8,"title":["Learning to Engage with Interactive Systems"],"prefix":"10.1145","volume":"10","author":[{"given":"Lingheng","family":"Meng","sequence":"first","affiliation":[{"name":"University of Waterloo, Waterloo, Ontario, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Daiwei","family":"Lin","sequence":"additional","affiliation":[{"name":"University of Waterloo, Waterloo, Ontario, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Adam","family":"Francey","sequence":"additional","affiliation":[{"name":"University of Waterloo, Waterloo, Ontario, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Rob","family":"Gorbet","sequence":"additional","affiliation":[{"name":"University of Waterloo, Waterloo, Ontario, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Philip","family":"Beesley","sequence":"additional","affiliation":[{"name":"University of Waterloo, Waterloo, Ontario, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dana","family":"Kuli\u0107","sequence":"additional","affiliation":[{"name":"University of Waterloo and Monash University, VIC, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2020,10,20]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1007\/s12369-015-0298-7"},{"key":"e_1_2_1_2_1","volume-title":"Proceedings of the 18th Annual ACM-SIAM Symposium on Discrete Algorithms. 1027--1035","author":"Arthur David","year":"2007","unstructured":"David Arthur and Sergei Vassilvitskii. 2007. k-means++: The advantages of careful seeding. In Proceedings of the 18th Annual ACM-SIAM Symposium on Discrete Algorithms. 1027--1035."},{"key":"e_1_2_1_3_1","volume-title":"Adaptive Control","author":"\u00c5str\u00f6m Karl J.","unstructured":"Karl J. \u00c5str\u00f6m and Bj\u00f6rn Wittenmark. 2013. Adaptive Control. Courier Corporation."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/MRA.2013.2275693"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1007\/s12369-008-0001-3"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.7564\/15-NGBJ20"},{"key":"e_1_2_1_7_1","volume-title":"Hylozoic Ground: Liminal Responsive Architecture: Philip Beesley","author":"Beesley Philip","year":"2010","unstructured":"Philip Beesley, Pernilla Ohrstedt, and Rob Gorbet. 2010. Hylozoic Ground: Liminal Responsive Architecture: Philip Beesley. Riverside Architectural Press. https:\/\/books.google.ca\/books?id=Ad6gAQAACAAJ."},{"key":"e_1_2_1_8_1","unstructured":"Marc G. Bellemare Yavar Naddaf Joel Veness and Michael Bowling. 2012. The arcade learning environment: An evaluation platform for general agents. arXiv:1207.4708. http:\/\/arxiv.org\/abs\/1207.4708."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.3115\/1708376.1708409"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2663204.2663241"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","unstructured":"Cynthia Breazeal Kerstin Dautenhahn and Takayuki Kanda. 2016. Social Robotics. Springer International Cham Switzerland 1935--1972. DOI:https:\/\/doi.org\/10.1007\/978-3-319-32552-1_72","DOI":"10.1007\/978-3-319-32552-1_72"},{"key":"e_1_2_1_13_1","unstructured":"Greg Brockman Vicki Cheung Ludwig Pettersson Jonas Schneider John Schulman Jie Tang and Wojciech Zaremba. 2016. OpenAI Gym. arXiv:1606.01540. http:\/\/arxiv.org\/abs\/1606.01540."},{"key":"e_1_2_1_14_1","doi-asserted-by":"crossref","unstructured":"Zhe Cao Tomas Simon Shih-En Wei and Yaser Sheikh. 2016. Realtime multi-person 2D pose estimation using part affinity fields. arXiv:1611.08050 http:\/\/arxiv.org\/abs\/1611.08050.","DOI":"10.1109\/CVPR.2017.143"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/1647314.1647336"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2015.7353856"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/ROMAN.2016.7745104"},{"key":"e_1_2_1_18_1","volume-title":"Proceedings of the 2017 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS\u201917)","author":"Chen Yu Fan","unstructured":"Yu Fan Chen, Michael Everett, Miao Liu, and Jonathan P. How. 2017. Socially aware motion planning with deep reinforcement learning. In Proceedings of the 2017 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS\u201917). IEEE, Los Alamitos, CA, 1343--1350."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/ROMAN.2018.8525604"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3212721.3212888"},{"key":"e_1_2_1_21_1","volume-title":"Scale Development: Theory and Applications.","author":"DeVellis Robert F.","year":"2016","unstructured":"Robert F. DeVellis. 2016. Scale Development: Theory and Applications. Vol. 26. Sage Publications."},{"key":"e_1_2_1_22_1","unstructured":"Prafulla Dhariwal Christopher Hesse Oleg Klimov Alex Nichol Matthias Plappert Alec Radford John Schulman Szymon Sidor Yuhuai Wu and Peter Zhokhov. 2017. OpenAI Baselines. Retrieved August 1 2020 from https:\/\/github.com\/openai\/baselines."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3234149"},{"key":"e_1_2_1_24_1","unstructured":"Scott Fujimoto Herke van Hoof and David Meger. 2018. Addressing function approximation error in actor-critic methods. arXiv:1802.09477."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACII.2015.7344688"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1561\/1100000005"},{"key":"e_1_2_1_27_1","volume-title":"Proceedings of the 30th AAAI Conference on Artificial Intelligence.","author":"Gordon Goren","year":"2016","unstructured":"Goren Gordon, Samuel Spaulding, Jacqueline Kory Westlund, Jin Joo Lee, Luke Plummer, Marayna Martinez, Madhurima Das, and Cynthia Breazeal. 2016. Affective personalization of a social robot tutor for children\u2019s second language skills. In Proceedings of the 30th AAAI Conference on Artificial Intelligence."},{"key":"e_1_2_1_28_1","volume-title":"Proceedings of the 26th International Conference on Neural Information Processing Systems\u2014Volume 2 (NIPS\u201913)","author":"Griffith Shane","year":"2013","unstructured":"Shane Griffith, Kaushik Subramanian, Jonathan Scholz, Charles L. Isbell, and Andrea Thomaz. 2013. Policy shaping: Integrating human feedback with reinforcement learning. In Proceedings of the 26th International Conference on Neural Information Processing Systems\u2014Volume 2 (NIPS\u201913). 2625--2633. http:\/\/dl.acm.org\/citation.cfm?id=2999792.2999905."},{"key":"e_1_2_1_29_1","unstructured":"Tuomas Haarnoja Aurick Zhou Pieter Abbeel and Sergey Levine. 2018. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv:1801.01290."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-32552-1_69"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/1228716.1228735"},{"key":"e_1_2_1_32_1","volume-title":"Proceedings of the 32nd AAAI Conference on Artificial Intelligence.","author":"Henderson Peter","year":"2018","unstructured":"Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. 2018. Deep reinforcement learning that matters. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/375735.376334"},{"key":"e_1_2_1_34_1","unstructured":"Serena Ivaldi S\u00e9bastien Lefort Jan Peters Mohamed Chetouani Joelle Provasi and Elisabetta Zibetti. 2015. Towards engagement models that consider individual factors in HRI: on the relation of extroversion and negative attitude towards robots to gaze and speech during a human-robot assembly task. arXiv:1508.04603. http:\/\/arxiv.org\/abs\/1508.04603"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3029798.3038389"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3275116.3275137"},{"key":"e_1_2_1_37_1","volume-title":"Human-Robotic Interaction in Social Robotics","author":"Kanda Takayuki","unstructured":"Takayuki Kanda and Hiroshi Ishiguro. 2012. Human-Robotic Interaction in Social Robotics. Boca Raton, FL."},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/2600021"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCDS.2018.2843122"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1080\/10447318.2016.1275435"},{"key":"e_1_2_1_41_1","volume-title":"Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems (AAMAS\u201910)","author":"Bradley Knox W.","year":"2010","unstructured":"W. Bradley Knox and Peter Stone. 2010. Combining manual feedback with subsequent MDP reward signals for reinforcement learning. In Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems (AAMAS\u201910)."},{"key":"e_1_2_1_42_1","volume-title":"Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems\u2014Volume 1 (AAMAS\u201912)","author":"Bradley Knox W.","year":"2012","unstructured":"W. Bradley Knox and Peter Stone. 2012. Reinforcement learning from simultaneous human and MDP reward. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems\u2014Volume 1 (AAMAS\u201912). 475--482. http:\/\/dl.acm.org\/citation.cfm?id=2343576.2343644."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3277904"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/TRO.2007.904899"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/ROMAN.2018.8525679"},{"key":"e_1_2_1_46_1","volume-title":"Proceedings of the International Conference on User Modeling, Adaptation, and Personalization. 135--147","author":"Leite Iolanda","year":"2011","unstructured":"Iolanda Leite, Andr\u00e9 Pereira, Ginevra Castellano, Samuel Mascarenhas, Carlos Martinho, and Ana Paiva. 2011. Modelling empathy in social robotic companions. In Proceedings of the International Conference on User Modeling, Adaptation, and Personalization. 135--147."},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1007\/s12369-016-0381-8"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/TRO.2016.2626479"},{"key":"e_1_2_1_49_1","unstructured":"Timothy P. Lillicrap Jonathan J. Hunt Alexander Pritzel Nicolas Heess Tom Erez Yuval Tassa David Silver and Daan Wierstra. 2015. Continuous control with deep reinforcement learning. arXiv:1509.02971. http:\/\/arxiv.org\/abs\/1509.02971."},{"key":"e_1_2_1_50_1","article-title":"Visualizing data using t-SNE","author":"van der Maaten Laurens","year":"2008","unstructured":"Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of Machine Learning Research 9 (Nov. 2008), 2579--2605.","journal-title":"Journal of Machine Learning Research 9"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2013.6696660"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177730491"},{"key":"e_1_2_1_53_1","unstructured":"Lingheng Meng Rob Gorbet and Dana Kuli\u0107. 2020. The effect of multi-step methods on overestimation in deep reinforcement learning. arXiv:2006.12692."},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1109\/AMC.2006.1631755"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/1349822.1349860"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1109\/HRI.2013.6483570"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.14288\/1.0107445"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.3389\/neuro.12.006.2007"},{"key":"e_1_2_1_59_1","unstructured":"Matthias Plappert Rein Houthooft Prafulla Dhariwal Szymon Sidor Richard Y. Chen Xi Chen Tamim Asfour Pieter Abbeel and Marcin Andrychowicz. 2017. Parameter space noise for exploration. arXiv:1706.01905. http:\/\/arxiv.org\/abs\/1706.01905."},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1109\/HUMANOIDS.2016.7803357"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1109\/HRI.2010.5453163"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/1957656.1957781"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/52.3-4.591"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1177\/0018720816644364"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1007\/s12369-017-0442-7"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/1121241.1121293"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/964442.964458"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSMC.2003.1244506"},{"key":"e_1_2_1_69_1","volume-title":"Proceedings of the 31st International Conference on Machine Learning (Proceedings of Machine Learning Research), E. P. Xing and T. Jebara (Eds.)","volume":"32","author":"Silver David","year":"2014","unstructured":"David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. 2014. Deterministic policy gradient algorithms. In Proceedings of the 31st International Conference on Machine Learning (Proceedings of Machine Learning Research), E. P. Xing and T. Jebara (Eds.), Vol. 32. PMLR, Bejing, China, 387--395. http:\/\/proceedings.mlr.press\/v32\/silver14.html."},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.robot.2016.10.013"},{"key":"e_1_2_1_71_1","volume-title":"Proceedings of the 2010 IEEE\/ACM International Conference on Human Robots Interaction.","author":"St David","unstructured":"David St.-Onge and Nicolas Reeves. 2010. Human interaction with flying cubic automata. In Proceedings of the 2010 IEEE\/ACM International Conference on Human Robots Interaction."},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1109\/ROMAN.2011.6005223"},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1561\/2300000049"},{"key":"e_1_2_1_74_1","volume-title":"Proceedings of the 21st National Conference on Artificial Intelligence\u2014Volume 1 (AAAI\u201906)","author":"Andrea","unstructured":"Andrea L. Thomaz and Cynthia Breazeal. 2006. Reinforcement learning with human teachers: Evidence of feedback and guidance with implications for learning performance. In Proceedings of the 21st National Conference on Artificial Intelligence\u2014Volume 1 (AAAI\u201906). 1000--1005. http:\/\/dl.acm.org\/citation.cfm?id=1597538.1597696."},{"key":"e_1_2_1_75_1","volume-title":"Proceedings of the AAAI 2005 Workshop on Human Comprehensible Machine Learning.","author":"Thomaz Andrea Lockerd","year":"2005","unstructured":"Andrea Lockerd Thomaz, Guy Hoffman, and Cynthia Breazeal. 2005. Real-time interactive reinforcement learning for robots. In Proceedings of the AAAI 2005 Workshop on Human Comprehensible Machine Learning."},{"key":"e_1_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.1080\/0951192X.2015.1130251"},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1515\/pjbr-2018-0016"},{"key":"e_1_2_1_78_1","doi-asserted-by":"publisher","DOI":"10.1109\/FUZZ-IEEE.2017.8015647"},{"key":"e_1_2_1_79_1","doi-asserted-by":"publisher","DOI":"10.1109\/MCG.2013.99"},{"key":"e_1_2_1_80_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2016.7487791"},{"key":"e_1_2_1_81_1","doi-asserted-by":"publisher","DOI":"10.1109\/TRO.2015.2392451"},{"key":"e_1_2_1_82_1","doi-asserted-by":"publisher","DOI":"10.1007\/s12369-014-0267-6"}],"container-title":["ACM Transactions on Human-Robot Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3408876","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3408876","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T21:31:36Z","timestamp":1750195896000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3408876"}},"subtitle":["A Field Study on Deep Reinforcement Learning in a Public Museum"],"short-title":[],"issued":{"date-parts":[[2020,10,20]]},"references-count":81,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2021,3,31]]}},"alternative-id":["10.1145\/3408876"],"URL":"https:\/\/doi.org\/10.1145\/3408876","relation":{},"ISSN":["2573-9522"],"issn-type":[{"value":"2573-9522","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,10,20]]},"assertion":[{"value":"2019-04-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-06-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-10-20","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}