{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,9]],"date-time":"2026-05-09T10:54:02Z","timestamp":1778324042772,"version":"3.51.4"},"reference-count":45,"publisher":"Springer Science and Business Media LLC","issue":"4","license":[{"start":{"date-parts":[[2022,4,7]],"date-time":"2022-04-07T00:00:00Z","timestamp":1649289600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,4,7]],"date-time":"2022-04-07T00:00:00Z","timestamp":1649289600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100006041","name":"Innovate UK","doi-asserted-by":"publisher","award":["TS\/S02221X\/1"],"award-info":[{"award-number":["TS\/S02221X\/1"]}],"id":[{"id":"10.13039\/501100006041","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000266","name":"Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["EP\/L015846\/1"],"award-info":[{"award-number":["EP\/L015846\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Virtual Reality"],"published-print":{"date-parts":[[2022,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>People can understand how human interaction unfolds and can pinpoint social attitudes such as showing interest or social engagement with a conversational partner. However, summarising this with a set of rules is difficult, as our judgement is sometimes subtle and subconscious. Hence, it is challenging to program Non-Player Characters (NPCs) to react towards social signals appropriately, which is important for immersive narrative games in Virtual Reality (VR). We collaborated with two game studios to develop an immersive machine learning (ML) pipeline for detecting social engagement. We collected data from participants-NPC interaction in VR, which was then annotated in the same immersive environment. Game design is a creative process and it is vital to respect designer\u2019s creative vision and judgement. We therefore view annotation as a key part of the creative process. We trained a reinforcement learning algorithm (PPO) with imitation learning rewards using raw data (e.g. head position) and socially meaningful derived data (e.g. proxemics); we compared different ML configurations including pre-training and a temporal memory (LSTM). The pre-training and LSTM configuration using derived data performed the best (84% F1-score, 83% accuracy). The models using raw data did not generalise. Overall, this work introduces an immersive ML pipeline for detecting social engagement and demonstrates how creatives could use ML and VR to expand their ability to design more engaging experiences. Given the pipeline\u2019s results for social engagement detection, we generalise it for detecting human-defined social attitudes.<\/jats:p>","DOI":"10.1007\/s10055-022-00644-4","type":"journal-article","created":{"date-parts":[[2022,4,7]],"date-time":"2022-04-07T13:03:41Z","timestamp":1649336621000},"page":"1519-1538","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":30,"title":["Immersive machine learning for social attitude detection in virtual reality narrative games"],"prefix":"10.1007","volume":"26","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9284-9954","authenticated-orcid":false,"given":"Georgiana Cristina","family":"Dobre","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Marco","family":"Gillies","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xueni","family":"Pan","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2022,4,7]]},"reference":[{"key":"644_CR1","doi-asserted-by":"crossref","unstructured":"Ahuja C, Ma S, Morency LP, Sheikh Y (2019) To react or not to react: end-to-end visual pose forecasting for personalized avatar during dyadic conversations. In: 2019 International conference on multimodal interaction, pp 74\u201384","DOI":"10.1145\/3340555.3353725"},{"key":"644_CR2","unstructured":"Bailenson J (2018) If a possible mass shooter wants to hone his craft, don\u2019t hand him a virtual boot camp. https:\/\/edition.cnn.com\/2018\/03\/05\/opinions\/video-games-shooting-opinion-bailenson\/index.html, https:\/\/edition.cnn.com\/2018\/03\/05\/opinions\/video-games-shooting-opinion-bailenson\/index.html"},{"key":"644_CR3","doi-asserted-by":"crossref","unstructured":"Bee N, Franke S, Andre\u00e9 E (2009) Relations between facial display, eye gaze and head tilt: dominance perception variations of virtual agents. In: 2009 3rd international conference on affective computing and intelligent interaction and workshops. IEEE, pp 1\u20137","DOI":"10.1109\/ACII.2009.5349573"},{"issue":"8","key":"644_CR4","doi-asserted-by":"publisher","first-page":"1798","DOI":"10.1109\/TPAMI.2013.50","volume":"35","author":"Y Bengio","year":"2013","unstructured":"Bengio Y, Courville A, Vincent P (2013) Representation learning: a review and new perspectives. IEEE Trans Pattern Anal Mach Intell 35(8):1798\u20131828","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"key":"644_CR5","doi-asserted-by":"publisher","unstructured":"Bohus D, Horvitz E (2014) Managing human\u2013robot engagement with forecasts and... um... hesitations. In: Proceedings of the 16th international conference on multimodal interaction, association for computing machinery, New York, NY, USA, ICMI\u201914, pp 2\u20139, https:\/\/doi.org\/10.1145\/2663204.2663241","DOI":"10.1145\/2663204.2663241"},{"key":"644_CR6","doi-asserted-by":"publisher","unstructured":"Brugel S, Postma-Nilsenov\u00e1 M, Tates K (2015) The link between perception of clinical empathy and nonverbal behavior: The effect of a doctor\u2019s gaze and body orientation. Patient Educ Counsel 98(10):1260\u20131265. https:\/\/doi.org\/10.1016\/j.pec.2015.08.007 communication in Healthcare: Best papers from the International Conference on Communication in Healthcare, Amsterdam, The Netherlands, 28 September\u20131 October 2014","DOI":"10.1016\/j.pec.2015.08.007"},{"key":"644_CR7","doi-asserted-by":"publisher","first-page":"295","DOI":"10.1111\/j.1468-2885.1993.tb00076.x","volume":"3","author":"J Burgoon","year":"2006","unstructured":"Burgoon J, Dillman L, Stem L (2006) Adaptation in dyadic interaction: defining and operationalizing patterns of reciprocity and compensation. Commun Theory 3:295\u2013316. https:\/\/doi.org\/10.1111\/j.1468-2885.1993.tb00076.x","journal-title":"Commun Theory"},{"issue":"2","key":"644_CR8","doi-asserted-by":"publisher","first-page":"2914796","DOI":"10.1145\/2914796","volume":"6","author":"A Cafaro","year":"2016","unstructured":"Cafaro A, Ravenet B, Ochs M, Vilhj\u00e1lmsson HH, Pelachaud C (2016) The effects of interpersonal attitude of a group of agents on user\u2019s presence and proxemics behavior. ACM Trans Interact Intell Syst 6(2):2914796. https:\/\/doi.org\/10.1145\/2914796","journal-title":"ACM Trans Interact Intell Syst"},{"key":"644_CR9","doi-asserted-by":"publisher","unstructured":"Chinchor N (1992) Muc-4 evaluation metrics. In: Proceedings of the 4th conference on message understanding, association for computational linguistics, USA, MUC4\u201992, pp 22\u201329, https:\/\/doi.org\/10.3115\/1072064.1072067","DOI":"10.3115\/1072064.1072067"},{"key":"644_CR10","doi-asserted-by":"crossref","unstructured":"Christensen JV, Mathiesen M, Poulsen JH, Ustrup EE, Kraus M (2018) Player experience in a vr and non-vr multiplayer game. In: Proceedings of the virtual reality international conference-Laval virtual, pp 1\u20134","DOI":"10.1145\/3234253.3234297"},{"key":"644_CR11","doi-asserted-by":"crossref","unstructured":"Dermouche S, Pelachaud C (2019a) Engagement modeling in dyadic interaction. In: 2019 international conference on multimodal interaction, pp 440\u2013445","DOI":"10.1145\/3340555.3353765"},{"key":"644_CR12","doi-asserted-by":"crossref","unstructured":"Dermouche S, Pelachaud C (2019b) Generative model of agent\u2019s behaviors in human-agent interaction. In: 2019 international conference on multimodal interaction, pp 375\u2013384","DOI":"10.1145\/3340555.3353758"},{"key":"644_CR13","doi-asserted-by":"crossref","unstructured":"Dhamija S, Boult TE (2017) Automated mood-aware engagement prediction. In: 2017 seventh international conference on affective computing and intelligent interaction (ACII). IEEE, pp 1\u20138","DOI":"10.1109\/ACII.2017.8273571"},{"key":"644_CR14","doi-asserted-by":"crossref","unstructured":"Feng W, Kannan A, Gkioxari G, Zitnick CL (2017) Learn2smile: learning non-verbal interaction through observation. In: 2017 IEEE\/RSJ international conference on intelligent robots and systems (IROS), pp 4131\u20134138","DOI":"10.1109\/IROS.2017.8206272"},{"key":"644_CR15","doi-asserted-by":"crossref","unstructured":"Ferstl Y, McDonnell R (2018) Investigating the use of recurrent motion modelling for speech gesture generation. In: Proceedings of the 18th international conference on intelligent virtual agents. ACM, pp 93\u201398","DOI":"10.1145\/3267851.3267898"},{"key":"644_CR16","unstructured":"Forbes-Riley K, Litman D, Friedberg H, Drummond J (2012) Intrinsic and extrinsic evaluation of an automatic user disengagement detector for an uncertainty-adaptive spoken dialogue system. In: Proceedings of the 2012 conference of the North American chapter of the association for computational linguistics: human language technologies, Association for Computational Linguistics, Montr\u00e9al, Canada, pp 91\u2013102, https:\/\/www.aclweb.org\/anthology\/N12-1010"},{"key":"644_CR17","doi-asserted-by":"publisher","unstructured":"Gillies M, Kleinsmith A, Brenton H (2015) Applying the CASSM framework to improving end user debugging of interactive machine learning. In: International conference on intelligent user interfaces, proceedings IUI, vol 2015, https:\/\/doi.org\/10.1145\/2678025.2701373","DOI":"10.1145\/2678025.2701373"},{"key":"644_CR18","doi-asserted-by":"crossref","unstructured":"Glas N, Pelachaud C (2015) Definitions of engagement in human-agent interaction. In: 2015 international conference on affective computing and intelligent interaction (ACII), IEEE, pp 944\u2013949","DOI":"10.1109\/ACII.2015.7344688"},{"key":"644_CR19","doi-asserted-by":"crossref","unstructured":"Gordon G, Spaulding S, Westlund JK, Lee JJ, Plummer L, Martinez M, Das M, Breazeal C (2016) Affective personalization of a social robot tutor for children\u2019s second language skills. In: Thirtieth AAAI conference on artificial intelligence","DOI":"10.1609\/aaai.v30i1.9914"},{"key":"644_CR20","doi-asserted-by":"crossref","unstructured":"Greenwood D, Laycock S, Matthews I (2017) Predicting head pose in dyadic conversation. In: International conference on intelligent virtual agents. Springer, pp 160\u2013169","DOI":"10.1007\/978-3-319-67401-8_18"},{"issue":"1","key":"644_CR21","doi-asserted-by":"publisher","first-page":"63","DOI":"10.1007\/s10919-019-00320-3","volume":"44","author":"J Hale","year":"2020","unstructured":"Hale J, Ward JA, Buccheri F, Oliver D, Hamilton AFdC (2020) Are you on my wavelength? Interpersonal coordination in dyadic conversations. J Nonverbal Behav 44(1):63\u201383","journal-title":"J Nonverbal Behav"},{"key":"644_CR22","volume-title":"The hidden dimension","author":"ET Hall","year":"1966","unstructured":"Hall ET (1966) The hidden dimension, vol 609. Doubleday, Garden City"},{"key":"644_CR23","unstructured":"Ho J, Ermon S (2016) Generative adversarial imitation learning. In: Advances in neural information processing systems, pp 4565\u20134573"},{"issue":"8","key":"644_CR24","doi-asserted-by":"publisher","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","volume":"9","author":"S Hochreiter","year":"1997","unstructured":"Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735\u20131780","journal-title":"Neural Comput"},{"issue":"2","key":"644_CR25","doi-asserted-by":"publisher","first-page":"103","DOI":"10.1177\/1555412010364982","volume":"6","author":"B Ip","year":"2011","unstructured":"Ip B (2011) Narrative structures in computer and video games: Part 1: context, definitions, and initial findings. Games Cult 6(2):103\u2013134","journal-title":"Games Cult"},{"issue":"2","key":"644_CR26","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3340250","volume":"2","author":"A Jin","year":"2019","unstructured":"Jin A, Deng Q, Zhang Y, Deng Z (2019) A deep learning-based model for head and eye motion generation in three-party conversations. Proc ACM Comput Graph Interact Tech 2(2):1\u201319","journal-title":"Proc ACM Comput Graph Interact Tech"},{"key":"644_CR27","doi-asserted-by":"crossref","unstructured":"Khaki H, Bozkurt E, Erzin E (2016) Agreement and disagreement classification of dyadic interactions using vocal and gestural cues. In: 2016 IEEE international conference on acoustics. Speech and signal processing (ICASSP). IEEE, pp 2762\u20132766","DOI":"10.1109\/ICASSP.2016.7472180"},{"key":"644_CR28","doi-asserted-by":"crossref","unstructured":"Lee SP, Badler JB, Badler NI (2002) Eyes alive. In: Proceedings of the 29th annual conference on Computer graphics and interactive techniques, pp 637\u2013644","DOI":"10.1145\/566654.566629"},{"key":"644_CR29","doi-asserted-by":"crossref","unstructured":"Marsella S, Xu Y, Lhommet M, Feng A, Scherer S, Shapiro A (2013) Virtual character performance from speech. In: Proceedings of the 12th ACM SIGGRAPH\/Eurographics symposium on computer animation, pp 25\u201335","DOI":"10.1145\/2485895.2485900"},{"key":"644_CR30","unstructured":"Max\u00a0Roser CA, Ritchie H (2013) Human height. Our World in Data https:\/\/ourworldindata.org\/human-height"},{"key":"644_CR31","doi-asserted-by":"crossref","unstructured":"Mota S, Picard RW (2003) Automated posture analysis for detecting learner\u2019s interest level. In: 2003 Conference on computer vision and pattern recognition workshop, vol\u00a05, pp 49\u201349","DOI":"10.1109\/CVPRW.2003.10047"},{"issue":"3","key":"644_CR32","doi-asserted-by":"publisher","first-page":"395","DOI":"10.1111\/bjop.12290","volume":"109","author":"X Pan","year":"2018","unstructured":"Pan X, Hamilton AFdC (2018) Why and how to use virtual reality to study human social interaction: the challenges of exploring a new research landscape. Br J Psychol 109(3):395\u2013417","journal-title":"Br J Psychol"},{"key":"644_CR33","doi-asserted-by":"publisher","first-page":"80","DOI":"10.3389\/frobt.2018.00080","volume":"5","author":"X Pan","year":"2018","unstructured":"Pan X, Collingwoode-Williams T, Antley A, Brenton H, Congdon B, Drewett O, Gillies MFP, Swapp D, Pleasence P, Fertleman C et al (2018) A study of professional awareness using immersive virtual reality: the responses of general practitioners to child safeguarding concerns. Front Robot AI 5:80","journal-title":"Front Robot AI"},{"key":"644_CR34","doi-asserted-by":"publisher","unstructured":"Sanghvi J, Castellano G, Leite I, Pereira A, McOwan PW, Paiva A (2011) Automatic analysis of affective postures and body motion to detect engagement with a game companion. In: Proceedings of the 6th international conference on human\u2013robot interaction, association for computing machinery, New York, NY, USA, HRI\u201911, pp 305\u2013312, https:\/\/doi.org\/10.1145\/1957656.1957781","DOI":"10.1145\/1957656.1957781"},{"key":"644_CR35","doi-asserted-by":"publisher","first-page":"393","DOI":"10.1017\/S0140525X12000660","volume":"36","author":"L Schilbach","year":"2013","unstructured":"Schilbach L, Timmermans B, Reddy V, Costall A, Bente G, Schlicht T, Vogeley K (2013) Toward a second-person neuroscience. Behav Brain Sci 36:393\u2013414. https:\/\/doi.org\/10.1017\/S0140525X12000660","journal-title":"Behav Brain Sci"},{"issue":"2","key":"644_CR36","doi-asserted-by":"publisher","first-page":"191","DOI":"10.1007\/BF01324126","volume":"4","author":"A Schmidt","year":"2000","unstructured":"Schmidt A (2000) Implicit human computer interaction through context. Pers Technol 4(2):191\u2013199","journal-title":"Pers Technol"},{"key":"644_CR37","unstructured":"Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:170706347"},{"key":"644_CR38","unstructured":"Shao K, Tang Z, Zhu Y, Li N, Zhao D (2019) A survey of deep reinforcement learning in video games. arXiv:1912.10944"},{"issue":"1535","key":"644_CR39","doi-asserted-by":"publisher","first-page":"3549","DOI":"10.1098\/rstb.2009.0138","volume":"364","author":"M Slater","year":"2009","unstructured":"Slater M (2009) Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments. Philos Trans R Soc B: Biol Sci 364(1535):3549\u20133557","journal-title":"Philos Trans R Soc B: Biol Sci"},{"issue":"5","key":"644_CR40","doi-asserted-by":"publisher","first-page":"413","DOI":"10.1162\/105474600566925","volume":"9","author":"M Slater","year":"2000","unstructured":"Slater M, Steed A (2000) A virtual presence counter. Presence Teleoper Virtual Environ 9(5):413\u2013434","journal-title":"Presence Teleoper Virtual Environ"},{"key":"644_CR41","doi-asserted-by":"crossref","unstructured":"Steed A, Schroeder R (2015) Collaboration in immersive and non-immersive virtual environments. In: Immersed in media. Springer, pp 263\u2013282","DOI":"10.1007\/978-3-319-10190-3_11"},{"issue":"1","key":"644_CR42","doi-asserted-by":"publisher","first-page":"69","DOI":"10.1109\/T-AFFC.2011.27","volume":"3","author":"A Vinciarelli","year":"2011","unstructured":"Vinciarelli A, Pantic M, Heylen D, Pelachaud C, Poggi I, D\u2019Errico F, Schroeder M (2011) Bridging the gap between social animal and unsocial machine: a survey of social signal processing. IEEE Trans Affect Comput 3(1):69\u201387","journal-title":"IEEE Trans Affect Comput"},{"key":"644_CR43","doi-asserted-by":"crossref","unstructured":"Wilson G, McGill M (2018) Violent video games in virtual reality: re-evaluating the impact and rating of interactive experiences. In: Proceedings of the 2018 annual symposium on computer\u2013human interaction in Play, pp 535\u2013548","DOI":"10.1145\/3242671.3242684"},{"issue":"3\u20134","key":"644_CR44","doi-asserted-by":"publisher","first-page":"129","DOI":"10.1504\/IJLT.2009.028804","volume":"4","author":"B Woolf","year":"2009","unstructured":"Woolf B, Burleson W, Arroyo I, Dragon T, Cooper D, Picard R (2009) Affect-aware tutors: recognising and responding to student affect. Int J Learn Technol 4(3\u20134):129\u2013164","journal-title":"Int J Learn Technol"},{"key":"644_CR45","doi-asserted-by":"crossref","unstructured":"Yu C, Aoki PM, Woodruff A (2004) Detecting user engagement in everyday conversations. arXiv preprint arXiv:cs\/0410027","DOI":"10.21437\/Interspeech.2004-327"}],"container-title":["Virtual Reality"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10055-022-00644-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10055-022-00644-4\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10055-022-00644-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,10,22]],"date-time":"2022-10-22T17:15:20Z","timestamp":1666458920000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10055-022-00644-4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,4,7]]},"references-count":45,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2022,12]]}},"alternative-id":["644"],"URL":"https:\/\/doi.org\/10.1007\/s10055-022-00644-4","relation":{},"ISSN":["1359-4338","1434-9957"],"issn-type":[{"value":"1359-4338","type":"print"},{"value":"1434-9957","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,4,7]]},"assertion":[{"value":"19 March 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 February 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"7 April 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors have no conflicts of interest to declare that are relevant to the content of this article.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}