{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,1]],"date-time":"2026-02-01T02:38:58Z","timestamp":1769913538803,"version":"3.49.0"},"reference-count":63,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2022,2,17]],"date-time":"2022-02-17T00:00:00Z","timestamp":1645056000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100003725","name":"National Research Foundation of Korea","doi-asserted-by":"publisher","award":["2021R1G1A1003801"],"award-info":[{"award-number":["2021R1G1A1003801"]}],"id":[{"id":"10.13039\/501100003725","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["MTI"],"abstract":"<jats:p>The development of new approaches for creating more \u201clife-like\u201d artificial intelligence (AI) capable of natural social interaction is of interest to a number of scientific fields, from virtual reality to human\u2013robot interaction to natural language speech systems. Yet how such \u201cSocial AI\u201d agents might be manifested remains an open question. Previous research has shown that both behavioral factors related to the artificial agent itself as well as contextual factors beyond the agent (i.e., interaction context) play a critical role in how people perceive interactions with interactive technology. As such, there is a need for customizable agents and customizable environments that allow us to explore both sides in a simultaneous manner. To that end, we describe here the development of a cooperative game environment and Social AI using a data-driven approach, which allows us to simultaneously manipulate different components of the social interaction (both behavioral and contextual). We conducted multiple human\u2013human and human\u2013AI interaction experiments to better understand the components necessary for creation of a Social AI virtual avatar capable of autonomously speaking and interacting with humans in multiple languages during cooperative gameplay (in this case, a social survival video game) in context-relevant ways.<\/jats:p>","DOI":"10.3390\/mti6020016","type":"journal-article","created":{"date-parts":[[2022,2,17]],"date-time":"2022-02-17T20:25:45Z","timestamp":1645129545000},"page":"16","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":11,"title":["Exploring Data-Driven Components of Socially Intelligent AI through Cooperative Game Paradigms"],"prefix":"10.3390","volume":"6","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2012-9250","authenticated-orcid":false,"given":"Casey","family":"Bennett","sequence":"first","affiliation":[{"name":"Department of Data Science, Hanyang University, Seoul 04763, Korea"},{"name":"College of Computing and Digital Media, DePaul University, Chicago 60601, IL, USA"}]},{"given":"Benjamin","family":"Weiss","sequence":"additional","affiliation":[{"name":"Quality and Usability Lab, Technische Universit\u00e4t, 10623 Berlin, Germany"}]},{"given":"Jaeyoung","family":"Suh","sequence":"additional","affiliation":[{"name":"Department of Data Science, Hanyang University, Seoul 04763, Korea"}]},{"given":"Eunseo","family":"Yoon","sequence":"additional","affiliation":[{"name":"Department of Data Science, Hanyang University, Seoul 04763, Korea"}]},{"given":"Jihong","family":"Jeong","sequence":"additional","affiliation":[{"name":"Department of Data Science, Hanyang University, Seoul 04763, Korea"}]},{"given":"Yejin","family":"Chae","sequence":"additional","affiliation":[{"name":"Department of Data Science, Hanyang University, Seoul 04763, Korea"}]}],"member":"1968","published-online":{"date-parts":[[2022,2,17]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"V\u00f6lkel, S.T., Schneegass, C., Eiband, M., and Buschek, D. (2020, January 17\u201320). What is \u201cintelligent\u201d in intelligent user interfaces? A meta-analysis of 25 years of IUI. Proceedings of the 25th International Conference on Intelligent User Interfaces (IUI), Cagliari, Italy.","DOI":"10.1145\/3377325.3377500"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Gero, K.I., Ashktorab, Z., Dugan, C., Pan, W., Johnson, J., Geyer, W., Ruiz, M., Miller, S., Millen, D.R., and Campbell, M. (2020, January 25\u201330). Mental models of AI agents in a cooperative game setting. Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI), Oahu, HI, USA.","DOI":"10.1145\/3313831.3376316"},{"key":"ref_3","unstructured":"Fussell, S.R., and Kreuz, R.J. (1991). The grounding problem in conversations with and through computers. Social and Cognitive Psychological Approaches to Interpersonal Communication, Lawrence Erlbaum."},{"key":"ref_4","unstructured":"Enfield, N. (2017). How we talk. The Inner Workings of Conversation, BasicBooks."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Esposito, A., Esposito, A.M., and Jain, L. (2019). Speech pause patterns in collaborative dialogs. Innovations in Big Data Mining and Embedded Knowledge, Springer.","DOI":"10.1007\/978-3-030-15939-9"},{"key":"ref_6","unstructured":"Knapp, M., and Hall, J. (2010). Nonverbal Communication in Human Interaction, Thomas Learning."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Tseng, S.H., Hsu, Y.H., Chiang, Y.S., Wu, T.Y., and Fu, L.C. (2014, January 25\u201329). Multi-human spatial social pattern understanding for a multi-modal robot through nonverbal social signals. Proceedings of the 23rd IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), Edinburgh, Scotland.","DOI":"10.1109\/ROMAN.2014.6926307"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.artint.2018.07.007","article-title":"Explanation in artificial intelligence: Insights from the social sciences","volume":"267","author":"Miller","year":"2019","journal-title":"Artif. Intell."},{"key":"ref_9","first-page":"17","article-title":"Automation, algorithms, and politics| talking to Bots: Symbiotic agency and the case of Tay","volume":"10","author":"Neff","year":"2016","journal-title":"Int. J. Commun."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"589","DOI":"10.1007\/s11023-020-09535-6","article-title":"Building thinking machines by solving animal cognition tasks","volume":"30","author":"Crosby","year":"2020","journal-title":"Minds Mach."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"861","DOI":"10.3389\/fpsyg.2018.00861","article-title":"Understanding and resolving failures in human-robot interaction: Literature review and model development","volume":"9","author":"Honig","year":"2018","journal-title":"Front. Psychol."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"114","DOI":"10.3389\/frobt.2018.00114","article-title":"A systematic review of social presence: Definition, antecedents, and implications","volume":"5","author":"Oh","year":"2018","journal-title":"Front. Robot. AI"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Doyle, P.R., Clark, L., and Cowan, B.R. (2021, January 8\u201313). What do we see in them? identifying dimensions of partner models for speech interfaces using a psycholexical approach. Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI), Yokohama, Japan.","DOI":"10.1145\/3411764.3445206"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"87","DOI":"10.2307\/2025382","article-title":"Intentional systems","volume":"68","author":"Dennett","year":"1971","journal-title":"J. Philos."},{"key":"ref_15","first-page":"105","article-title":"Computational human-robot interaction","volume":"4","author":"Thomaz","year":"2016","journal-title":"Found. Trends Robot."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"83","DOI":"10.1007\/s12369-020-00623-2","article-title":"Robotic faciality: The philosophy, science and art of robot faces","volume":"13","author":"Chesher","year":"2021","journal-title":"Int. J. Soc. Robot."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Bennett, C.C., Sabanovic, S., Fraune, M.R., and Shaw, K. (2014, January 25\u201329). Context congruency and robotic facial expressions: Do effects on human perceptions vary across culture?. Proceedings of the 23rd IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), Edinburgh, Scotland.","DOI":"10.1109\/ROMAN.2014.6926296"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"195","DOI":"10.1109\/TSMCC.2004.826278","article-title":"A social informatics approach to human-robot interaction with a service social robot","volume":"34","author":"Lisetti","year":"2004","journal-title":"IEEE Trans. Syst. Man Cybern. Part C"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"26435","DOI":"10.1109\/ACCESS.2017.2777823","article-title":"Enhancing user satisfaction by adapting Robot\u2019s perception of uncertain information based on environment and user feedback","volume":"5","author":"Muthugala","year":"2017","journal-title":"IEEE Access"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"250","DOI":"10.1016\/j.ijhcs.2012.09.005","article-title":"The influence of empathy in human\u2013robot relations","volume":"71","author":"Leite","year":"2013","journal-title":"Int. J. Hum. Comput. Stud."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Correia, F., Alves-Oliveira, P., Maia, N., Ribeiro, T., Petisca, S., Melo, F.S., and Paiva, A. (2016, January 26\u201331). Just follow the suit! trust in human-robot interactions during card game playing. Proceedings of the 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), New York, NY, USA.","DOI":"10.1109\/ROMAN.2016.7745165"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"106220","DOI":"10.1016\/j.chb.2019.106220","article-title":"Effects of robot-human versus robot-robot behavior and entitativity on anthropomorphism and willingness to interact","volume":"105","author":"Fraune","year":"2020","journal-title":"Comput. Hum. Behav."},{"key":"ref_23","unstructured":"Pearl, C. (2016). Designing Voice User Interfaces: Principles of Conversational Experiences, O\u2019Reilly."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Bernsen, N.O., Dybkjaer, H., and Dybkjaer, L. (1998). Designing Interactive Speech Systems. From First Ideas to User Testing, Springer.","DOI":"10.1007\/978-1-4471-0897-9"},{"key":"ref_25","unstructured":"International Telecommunication Union (2011). Parameters Describing the Interaction with Multimodal Dialogue Systems, International Telecommunication Union."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"484","DOI":"10.1016\/j.intcom.2007.04.002","article-title":"The effects of speech-gesture cooperation in animated agents\u2019 behavior in multimedia presentations","volume":"19","author":"Buisine","year":"2007","journal-title":"Interact. Comput."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"419","DOI":"10.1016\/j.evolhumbehav.2013.08.001","article-title":"Convergence of speech rate in conversation predicts cooperation","volume":"34","author":"Manson","year":"2013","journal-title":"Evol. Hum. Behav."},{"key":"ref_28","unstructured":"Reitter, D., and Moore, J.D. (2007, January 25\u201327). Predicting success in dialogue. Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, Prague, Czech Republic."},{"key":"ref_29","unstructured":"Lai, C., Carletta, J., and Renals, S. (2013, January 22\u201323). Modelling participant affect in meetings with turn-taking features. Proceedings of the Workshop of Affective Social Speech Signals, Grenoble, France."},{"key":"ref_30","unstructured":"M\u00f6ller, S. (2004). Quality of Telephone-Based Spoken Dialogue Systems, Springer Science & Business Media."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Munteanu, C., Clark, L., Cowan, B., Schl\u00f6gl, S., Torres, M.I., Edwards, J., Murad, C., Aylett, M., Porcheron, M., and Candello, H. (2020, January 17\u201320). CUI: Conversational user interfaces: A workshop on new theoretical and methodological perspectives for researching speech-based conversational interactions. Proceedings of the 25th International Conference on Intelligent User Interfaces Companion (IUI), Cagliari, Italy.","DOI":"10.1145\/3379336.3379358"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"351","DOI":"10.1177\/002383099103400404","article-title":"The HCRC map task corpus","volume":"34","author":"Anderson","year":"1991","journal-title":"Lang Speech"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"3549","DOI":"10.1098\/rstb.2009.0138","article-title":"Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments","volume":"364","author":"Slater","year":"2009","journal-title":"Phil. Trans. Biol. Sci."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1125","DOI":"10.3389\/fpsyg.2017.01125","article-title":"Model of illusions and virtual reality","volume":"8","author":"Lanier","year":"2017","journal-title":"Front. Psychol."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"357","DOI":"10.1016\/j.chb.2018.08.027","article-title":"Evaluations of an artificial intelligence instructor\u2019s voice: Social Identity Theory in human-robot interactions","volume":"90","author":"Edwards","year":"2019","journal-title":"Comput. Hum. Behav."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Slater, M., Antley, A., Davison, A., Swapp, D., Guger, C., Barker, C., Pistrang, N., and Sanchez-Vives, M.V. (2006). A virtual reprise of the Stanley Milgram obedience experiments. PLoS ONE, 1.","DOI":"10.1371\/journal.pone.0000039"},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"20180033","DOI":"10.1098\/rstb.2018.0033","article-title":"Brain activity during reciprocal social interaction investigated using conversational robots as control condition","volume":"374","author":"Rauchbauer","year":"2019","journal-title":"Phil. Trans. Roy. Soc. Lond. B"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Bennett, C.C. (2021, January 8\u201312). Evoking an intentional stance during human-agent social interaction: Appearances can be deceiving. Proceedings of the 30th IEEE International Conference on Robot & Human Interactive Communication (RO-MAN), Vancouver, BC, Canada.","DOI":"10.1109\/RO-MAN50785.2021.9515420"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"2","DOI":"10.3389\/frvir.2020.00002","article-title":"The role of plausibility in the experience of spatial presence in virtual environments","volume":"1","author":"Hofer","year":"2020","journal-title":"Front. Virtual Real."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Abdul, A., Vermeulen, J., Wang, D., Lim, B.Y., and Kankanhalli, M. (2018, January 21\u201326). Trends and trajectories for explainable, accountable and intelligible systems: An HCI research agenda. Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI), Montreal, QC, Canada.","DOI":"10.1145\/3173574.3174156"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Kolokoltsov, V.N., and Malafeyev, O.A. (2020). Understanding Game Theory: Introduction to the Analysis of Many Agent Systems with Competition and Cooperation, World Scientific Publishing.","DOI":"10.1142\/11681"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"57","DOI":"10.1016\/j.ijhcs.2009.09.008","article-title":"Computer agents versus avatars: Responses to interactive game characters controlled by a computer or other player","volume":"68","author":"Lim","year":"2010","journal-title":"Int. J. Hum. Comput. Stud."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"1401","DOI":"10.1007\/s11192-018-2825-4","article-title":"The peer review game: An agent-based model of scientists facing resource constraints and institutional pressures","volume":"116","author":"Bianchi","year":"2018","journal-title":"Scientometrics"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"531805","DOI":"10.3389\/frobt.2020.531805","article-title":"Behavioral cues of humanness in complex environments: How people engage with human and artificially intelligent agents in a multiplayer videogame","volume":"7","author":"Jesso","year":"2020","journal-title":"Front. Robot. AI"},{"key":"ref_45","unstructured":"Correia, F., Alves-Oliveira, P., Ribeiro, T., Melo, F., and Paiva, A. (2017, January 5\u20139). A social robot as a card game player. Proceedings of the 13th AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, Salt Lake City, UT, USA."},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"V\u00f6lkel, S.T., Sch\u00f6del, R., Buschek, D., Stachl, C., Winterhalter, V., B\u00fchner, M., and Hussmann, H. (2020, January 25\u201330). Developing a personality model for speech-based conversational agents using the psycholexical approach. Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI), Oahu, HI, USA.","DOI":"10.1145\/3313831.3376210"},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"104108","DOI":"10.1016\/j.cognition.2019.104108","article-title":"The sociality of social inhibition of return","volume":"195","author":"Nafcha","year":"2020","journal-title":"Cognition"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"346","DOI":"10.1111\/1467-9280.00166","article-title":"Inhibition of return is a foraging facilitator in visual search","volume":"10","author":"Klein","year":"1999","journal-title":"Psychol. Sci."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"56855","DOI":"10.1109\/ACCESS.2020.2982225","article-title":"LSTM-CNN architecture for human activity recognition","volume":"8","author":"Xia","year":"2020","journal-title":"IEEE Access"},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1613\/jair.953","article-title":"SMOTE: Synthetic minority over-sampling technique","volume":"16","author":"Chawla","year":"2002","journal-title":"J. Artif. Intell. Res."},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Siebert, J., Joeckel, L., Heidrich, J., Nakamichi, K., Ohashi, K., Namba, I., Yamamoto, R., and Aoyama, M. (2020, January 8\u201311). Towards guidelines for assessing qualities of machine learning systems. Proceedings of the International Conference on the Quality of Information and Communications Technology (QUATIC), Online Conference.","DOI":"10.1007\/978-3-030-58793-2_2"},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"105","DOI":"10.1016\/j.hlpt.2012.03.001","article-title":"EHRs connect research and practice: Where predictive modeling, artificial intelligence, and clinical decision support intersect","volume":"1","author":"Bennett","year":"2012","journal-title":"Health Policy Technol."},{"key":"ref_53","unstructured":"Ekman, P., and Friesen, W.V. (2003). Unmasking the Face: A Guide to Recognizing Emotions from Facial Clues, Malor Books."},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"100307","DOI":"10.1016\/j.entcom.2019.100307","article-title":"Modeling and adjusting in-game difficulty based on facial expression analysis","volume":"31","author":"Blom","year":"2019","journal-title":"Entertain. Comput."},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Mistry, K., Jasekar, J., Issac, B., and Zhang, L. (2018, January 8\u201313). Extended LBP based facial expression recognition system for adaptive AI agent behaviour. Proceedings of the IEEE International Joint Conference on Neural Networks (IJCNN), Rio de Janeiro, Brazil.","DOI":"10.1109\/IJCNN.2018.8489415"},{"key":"ref_56","unstructured":"M\u00f6ller, S. (September, January 29). Perceptual quality dimensions of spoken dialogue systems: A review and new experimental results. Proceedings of the 4th European Congress on Acoustics (Forum Acusticum Budapest 2005), Budapest, Hungary."},{"key":"ref_57","unstructured":"Witz, J.M. (2015). From talking and listening robots to intelligent communicative machines. Robots That Talk and Listen, De Gruyter."},{"key":"ref_58","unstructured":"Bowden, K., Wu, J., Oraby, S., Misra, A., and Walker, M. (2018, January 7\u201312). SlugNERDS: A Named Entity Recognition Tool for Open Domain Dialogue Systems. Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC), Miyazaki, Japan."},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1049\/iet-cvi.2017.0052","article-title":"Review of constraints on vision-based gesture recognition for human\u2013computer interaction","volume":"12","author":"Chakraborty","year":"2018","journal-title":"IET Comput. Vis."},{"key":"ref_60","doi-asserted-by":"crossref","first-page":"119","DOI":"10.5898\/JHRI.1.1.Riek","article-title":"Wizard of oz studies in HRI: A systematic review and new reporting guidelines","volume":"1","author":"Riek","year":"2012","journal-title":"J. Hum.-Robot. Interact."},{"key":"ref_61","doi-asserted-by":"crossref","unstructured":"Lee, H.R., and \u0160abanovi\u0107, S. (2014, January 3\u20136). Culturally variable preferences for robot design and use in South Korea, Turkey, and the United States. Proceedings of the ACM\/IEEE International Conference on Human-Robot Interaction (HRI), Bielefeld, Germany.","DOI":"10.1145\/2559636.2559676"},{"key":"ref_62","doi-asserted-by":"crossref","unstructured":"Anagnostis, A., Benos, L., Tsaopoulos, D., Tagarakis, A., Tsolakis, N., and Bochtis, D. (2021). Human activity recognition through recurrent neural networks for human\u2013robot interaction in agriculture. Appl. Sci., 11.","DOI":"10.3390\/app11052188"},{"key":"ref_63","doi-asserted-by":"crossref","unstructured":"Lee, H.R., \u0160abanovi\u0107, S., Chang, W.L., Nagata, S., Piatt., J., Bennett, C., and Hakken, D. (2017, January 6\u20139). Steps toward participatory design of social robots: Mutual learning with older adults with depression. Proceedings of the ACM\/IEEE International Conference on Human-Robot Interaction (HRI), Vienna, Austria.","DOI":"10.1145\/2909824.3020237"}],"container-title":["Multimodal Technologies and Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2414-4088\/6\/2\/16\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:21:19Z","timestamp":1760134879000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2414-4088\/6\/2\/16"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,2,17]]},"references-count":63,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2022,2]]}},"alternative-id":["mti6020016"],"URL":"https:\/\/doi.org\/10.3390\/mti6020016","relation":{},"ISSN":["2414-4088"],"issn-type":[{"value":"2414-4088","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,2,17]]}}}