{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,24]],"date-time":"2026-06-24T18:45:46Z","timestamp":1782326746226,"version":"3.54.5"},"reference-count":85,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2023,3,9]],"date-time":"2023-03-09T00:00:00Z","timestamp":1678320000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Interact. Intell. Syst."],"published-print":{"date-parts":[[2023,3,31]]},"abstract":"<jats:p>\n            We developed a method to synthesize game levels that accounts for the degree of collaboration required by two players to finish a given game level. We first asked a game level designer to create playable game level chunks. Then, two\n            <jats:bold>artificial intelligence (AI)<\/jats:bold>\n            virtual agents driven by behavior trees played each game level chunk. We recorded the degree of collaboration required to accomplish each game level chunk by the AI virtual agents and used it to characterize each game level chunk. To synthesize a game level, we assigned to the total cost function cost terms that encode both the degree of collaboration and game level design decisions. Then, we used a Markov-chain Monte Carlo optimization method, called simulated annealing, to solve the total cost function and proposed a design for a game level. We synthesized three game levels (low, medium, and high degrees of collaboration game levels) to evaluate our implementation. We then recruited groups of participants to play the game levels to explore whether they would experience a certain degree of collaboration and validate whether the AI virtual agents provided sufficient data that described the collaborative behavior of players in each game level chunk. By collecting both in-game objective measurements and self-reported subjective ratings, we found that the three game levels indeed impacted the collaboration gameplay behavior of our participants. Moreover, by analyzing our collected data, we found moderate and strong correlations between the participants and the AI virtual agents. These results show that game developers can consider AI virtual agents as an alternative method for evaluating the degree of collaboration required to finish a game level.\n          <\/jats:p>","DOI":"10.1145\/3558773","type":"journal-article","created":{"date-parts":[[2022,8,23]],"date-time":"2022-08-23T11:12:17Z","timestamp":1661253137000},"page":"1-36","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":27,"title":["Synthesizing Game Levels for Collaborative Gameplay in a Shared Virtual Environment"],"prefix":"10.1145","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2620-9585","authenticated-orcid":false,"given":"Huimin","family":"Liu","sequence":"first","affiliation":[{"name":"Purdue University, West Lafayette, Indiana, U.S.A."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9459-4070","authenticated-orcid":false,"given":"Minsoo","family":"Choi","sequence":"additional","affiliation":[{"name":"Purdue University, West Lafayette, Indiana, U.S.A."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7732-6258","authenticated-orcid":false,"given":"Dominic","family":"Kao","sequence":"additional","affiliation":[{"name":"Purdue University, West Lafayette, Indiana, U.S.A."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0955-7959","authenticated-orcid":false,"given":"Christos","family":"Mousas","sequence":"additional","affiliation":[{"name":"Purdue University, West Lafayette, Indiana, U.S.A."}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,3,9]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.5465\/amr.2002.5922314"},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/3445792"},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173888"},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-53088-8_2"},{"key":"e_1_3_3_6_2","first-page":"811","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment","author":"Bauer Aaron","year":"2012","unstructured":"Aaron Bauer and Zoran Popovi\u0107. 2012. RRT-based game level analysis, visualization, and visual refinement. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment. AAAI, Palo Alto, California, 811\u2013813."},{"key":"e_1_3_3_7_2","first-page":"153","volume-title":"International Conference on Motion in Games","author":"Berseth Glen","year":"2014","unstructured":"Glen Berseth, M. Brandon Haworth, Mubbasir Kapadia, and Petros Faloutsos. 2014. Characterizing and optimizing game level difficulty. In International Conference on Motion in Games. ACM, New York, NY, USA, 153\u2013160."},{"key":"e_1_3_3_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/2668084.2668100"},{"key":"e_1_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/2522628.2522650"},{"key":"e_1_3_3_10_2","first-page":"1","volume-title":"4th Annual International Workshop on Presence, Philadelphia, PA","author":"Biocca Frank","year":"2001","unstructured":"Frank Biocca, Chad Harms, and Jenn Gregg. 2001. The networked minds measure of social presence: Pilot test of the factor structure and concurrent validity. In 4th Annual International Workshop on Presence, Philadelphia, PA. PBWorks, San Mateo, CA, 1\u20139."},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.5555\/3198482.3198618"},{"key":"e_1_3_3_12_2","doi-asserted-by":"publisher","DOI":"10.1037\/a0028821"},{"key":"e_1_3_3_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145291"},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","DOI":"10.1080\/00031305.1995.10476177"},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0360-1315(01)00028-8"},{"key":"e_1_3_3_16_2","doi-asserted-by":"publisher","DOI":"10.4324\/9780203771587"},{"key":"e_1_3_3_17_2","doi-asserted-by":"publisher","DOI":"10.1201\/9780429489105"},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10111-014-0289-8"},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","DOI":"10.1111\/bjet.12636"},{"key":"e_1_3_3_20_2","first-page":"1","volume-title":"International Conference on the Foundations of Digital Games","author":"Silva Fernando de Mesentier","year":"2017","unstructured":"Fernando de Mesentier Silva, Scott Lee, Julian Togelius, and Andy Nealen. 2017. AI-based playtesting of contemporary board games. In International Conference on the Foundations of Digital Games. ACM, New York, NY, USA, 1\u201310."},{"key":"e_1_3_3_21_2","doi-asserted-by":"publisher","DOI":"10.1109\/BigData.2014.7004282"},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.1201\/b10905-4"},{"key":"e_1_3_3_23_2","doi-asserted-by":"publisher","DOI":"10.3758\/BRM.41.4.1149"},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.3758\/BF03193146"},{"key":"e_1_3_3_25_2","doi-asserted-by":"publisher","DOI":"10.5555\/358242.358264"},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.1111\/1467-8535.00173"},{"key":"e_1_3_3_27_2","first-page":"309","volume-title":"Game AI Pro 360: Guide to Architecture","author":"Francis Anthony","year":"2019","unstructured":"Anthony Francis. 2019. Overcoming pitfalls in behavior tree design. In Game AI Pro 360: Guide to Architecture. CRC Press, Boca Raton, Florida, 309\u2013320."},{"key":"e_1_3_3_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/CoG52621.2021.9619053"},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/CoG47356.2020.9231548"},{"key":"e_1_3_3_30_2","article-title":"Improving playtesting coverage via curiosity driven reinforcement learning agents","author":"Gordillo Camilo","year":"2021","unstructured":"Camilo Gordillo, Joakim Bergdahl, Konrad Tollmar, and Linus Gissl\u00e9n. 2021. Improving playtesting coverage via curiosity driven reinforcement learning agents. arXiv preprint arXiv:2103.13798.","journal-title":"arXiv preprint arXiv:2103.13798"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/82.4.711"},{"key":"e_1_3_3_32_2","first-page":"1","volume-title":"International Conference on Computer Supported Collaborative Learning","author":"Greenwald Scott W.","year":"2017","unstructured":"Scott W. Greenwald, Wiley Corning, and Pattie Maes. 2017. Multi-user framework for collaboration and co-creation in virtual reality. In International Conference on Computer Supported Collaborative Learning. ACM, 1\u20132."},{"key":"e_1_3_3_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/963900.963915"},{"key":"e_1_3_3_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/2422956.2422957"},{"key":"e_1_3_3_35_2","first-page":"1","volume-title":"SIGGRAPH Asia Emerging Technologies","author":"Ibayashi Hikaru","year":"2015","unstructured":"Hikaru Ibayashi, Yuta Sugiura, Daisuke Sakamoto, Natsuki Miyata, Mitsunori Tada, Takashi Okuma, Takeshi Kurata, Masaaki Mochimaru, and Takeo Igarashi. 2015. Dollhouse VR: A multi-view, multi-user collaborative design workspace with VR technology. In SIGGRAPH Asia Emerging Technologies. ACM, New York, NY, USA, 1\u20132."},{"key":"e_1_3_3_36_2","first-page":"1","article-title":"Evaluating tutorial-based instructions for controllers in virtual reality games","volume":"5","author":"Kao Dominic","year":"2021","unstructured":"Dominic Kao, Alejandra J. Magana, and Christos Mousas. 2021. Evaluating tutorial-based instructions for controllers in virtual reality games. Proceedings of the ACM on Human-Computer Interaction 5, CHI PLAY (2021), 1\u201328.","journal-title":"Proceedings of the ACM on Human-Computer Interaction"},{"key":"e_1_3_3_37_2","volume-title":"Latin Squares and their Applications","author":"Keedwell A. Donald","year":"2015","unstructured":"A. Donald Keedwell and J\u00f3zsef D\u00e9nes. 2015. Latin Squares and their Applications. Elsevier, Amsterdam, Netherlands."},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/3386569.3392393"},{"key":"e_1_3_3_39_2","first-page":"30","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment","author":"Liapis Antonios","year":"2013","unstructured":"Antonios Liapis, Georgios Yannakakis, and Julian Togelius. 2013. Towards a generic method of evaluating game levels. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment. AAAI, Palo Alto, California, 30\u201336."},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858039"},{"key":"e_1_3_3_41_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.gvc.2021.200020"},{"key":"e_1_3_3_42_2","first-page":"566","volume-title":"IEEE International Symposium on Mixed and Augmented Reality","author":"Liu Huimin","year":"2020","unstructured":"Huimin Liu, Zhiquan Wang, Christos Mousas, and Dominic Kao. 2020. Virtual reality racket sports: Virtual drills for exercise and training. In IEEE International Symposium on Mixed and Augmented Reality. IEEE, New York, NY, USA, 566\u2013576."},{"key":"e_1_3_3_43_2","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.12314"},{"key":"e_1_3_3_44_2","doi-asserted-by":"publisher","DOI":"10.1080\/0951192X.2019.1690685"},{"key":"e_1_3_3_45_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.1123007"},{"key":"e_1_3_3_46_2","first-page":"1","volume-title":"IEEE VR Workshop on Sonic Interactions for Virtual Environments","author":"Men Liang","year":"2018","unstructured":"Liang Men and Nick Bryan-Kinns. 2018. LeMo: Supporting collaborative music making in virtual reality. In IEEE VR Workshop on Sonic Interactions for Virtual Environments. IEEE, New York, NY, USA, 1\u20136."},{"key":"e_1_3_3_47_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.entcom.2012.09.006"},{"key":"e_1_3_3_48_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.entcom.2018.02.005"},{"key":"e_1_3_3_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/3458844"},{"key":"e_1_3_3_50_2","first-page":"1","volume-title":"SIGGRAPH Asia Emerging Technologies","author":"Piumsomboon Thammathip","year":"2017","unstructured":"Thammathip Piumsomboon, Youngho Lee, Gun Lee, and Mark Billinghurst. 2017. CoVAR: A collaborative virtual and augmented reality system for remote collaboration. In SIGGRAPH Asia Emerging Technologies. ACM, New York, NY, USA, 1\u20132."},{"issue":"2","key":"e_1_3_3_51_2","first-page":"101","article-title":"Can I say something? The effects of digital game play on willingness to communicate.","volume":"18","author":"Reinders Hayo","year":"2014","unstructured":"Hayo Reinders and Sorada Wattana. 2014. Can I say something? The effects of digital game play on willingness to communicate.Language Learning & Technology 18, 2 (2014), 101\u2013123.","journal-title":"Language Learning & Technology"},{"key":"e_1_3_3_52_2","volume-title":"Authoring Collaborative Multiplayer Games-Game Design Patterns, Structural Verification, Collaborative Balancing and Rapid Prototyping","author":"Reuter Christian","year":"2016","unstructured":"Christian Reuter. 2016. Authoring Collaborative Multiplayer Games-Game Design Patterns, Structural Verification, Collaborative Balancing and Rapid Prototyping. Ph.D. Dissertation. Technische Universit\u00e4t Darmstadt."},{"key":"e_1_3_3_53_2","first-page":"1","volume-title":"DiGRA","author":"Reuter Christian","year":"2014","unstructured":"Christian Reuter, Viktor Wendel, Stefan G\u00f6bel, and Ralf Steinmetz. 2014. Game design patterns for collaborative player interactions. In DiGRA. DiGRA, 1\u201316."},{"key":"e_1_3_3_54_2","first-page":"72","volume-title":"ZON Digital Games","author":"Rocha Jos\u00e9 Bernardo","year":"2008","unstructured":"Jos\u00e9 Bernardo Rocha, Samuel Mascarenhas, and Rui Prada. 2008. Game mechanics for cooperative games. In ZON Digital Games. ZON, 72\u201380."},{"key":"e_1_3_3_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/3474658"},{"key":"e_1_3_3_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/3410404.3414235"},{"key":"e_1_3_3_57_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jesp.2004.08.008"},{"key":"e_1_3_3_58_2","doi-asserted-by":"publisher","DOI":"10.1201\/9780080919171"},{"key":"e_1_3_3_59_2","doi-asserted-by":"publisher","DOI":"10.1109\/CTS.2013.6567257"},{"key":"e_1_3_3_60_2","doi-asserted-by":"publisher","DOI":"10.1145\/1753326.1753363"},{"key":"e_1_3_3_61_2","doi-asserted-by":"publisher","DOI":"10.1142\/S0218213017300010"},{"key":"e_1_3_3_62_2","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1145\/2522628.2522629","volume-title":"Motion on Games","author":"Shoulson Alexander","year":"2013","unstructured":"Alexander Shoulson, Max L. Gilbert, Mubbasir Kapadia, and Norman I. Badler. 2013. An event-centric planning approach for dynamic real-time narrative. In Motion on Games. ACM, New York, NY, USA, 121\u2013130."},{"key":"e_1_3_3_63_2","doi-asserted-by":"publisher","DOI":"10.7205\/MILMED-D-15-00164"},{"key":"e_1_3_3_64_2","doi-asserted-by":"publisher","DOI":"10.1162\/pres.1994.3.2.130"},{"key":"e_1_3_3_65_2","doi-asserted-by":"publisher","DOI":"10.1109\/TCIAIG.2011.2158545"},{"key":"e_1_3_3_66_2","doi-asserted-by":"publisher","DOI":"10.1515\/jso-2014-0006"},{"key":"e_1_3_3_67_2","doi-asserted-by":"publisher","DOI":"10.1145\/1124772.1124950"},{"key":"e_1_3_3_68_2","doi-asserted-by":"publisher","DOI":"10.1093\/jopart\/mum036"},{"key":"e_1_3_3_69_2","volume-title":"Procedural Content Generation in Games: A Textbook and an Overview of Current Research","author":"Togelius Julian","year":"2014","unstructured":"Julian Togelius, Noor Shaker, and Mark J. Nelson. 2014. Procedural Content Generation in Games: A Textbook and an Overview of Current Research. Springer, Berlin, Germany."},{"key":"e_1_3_3_70_2","doi-asserted-by":"publisher","DOI":"10.1109\/TCIAIG.2011.2148116"},{"key":"e_1_3_3_71_2","doi-asserted-by":"publisher","DOI":"10.2307\/257085"},{"key":"e_1_3_3_72_2","first-page":"313","volume-title":"ACM Annual Symposium on Computer-Human Interaction in Play","author":"Tyack April","year":"2016","unstructured":"April Tyack, Peta Wyeth, and Daniel Johnson. 2016. The appeal of MOBA games: What makes people start, stay, and stop. In ACM Annual Symposium on Computer-Human Interaction in Play. ACM, New York, NY, USA, 313\u2013325."},{"key":"e_1_3_3_73_2","doi-asserted-by":"publisher","DOI":"10.1177\/0894486515568974"},{"key":"e_1_3_3_74_2","first-page":"87","volume-title":"International Conference on Intelligent Games and Simulation","author":"Arkel Benjamin van","year":"2015","unstructured":"Benjamin van Arkel, Daniel Karavolos, Anders Bouwer, Sander Bakkes, and Frank Nack. 2015. Procedural generation of collaborative puzzle-platform game levels. In International Conference on Intelligent Games and Simulation. Eurosis, 87\u201393."},{"key":"e_1_3_3_75_2","first-page":"1","volume-title":"International Conference on the Foundations of Digital Games","author":"Rozen Riemer van","year":"2018","unstructured":"Riemer van Rozen and Quinten Heijn. 2018. Measuring quality of grammars for procedural level generation. In International Conference on the Foundations of Digital Games. ACM, New York, NY, USA, 1\u20138."},{"key":"e_1_3_3_76_2","doi-asserted-by":"publisher","DOI":"10.1109\/CoG47356.2020.9231754"},{"key":"e_1_3_3_77_2","doi-asserted-by":"publisher","DOI":"10.1109\/SMC.2015.485"},{"key":"e_1_3_3_78_2","first-page":"1","volume-title":"Dipresentasikan Dalam Computer Game Technology Conference","author":"Winn Brian M.","year":"2004","unstructured":"Brian M. Winn and J. W. Fisher. 2004. Design of communication, competition, and collaboration in online games. In Dipresentasikan Dalam Computer Game Technology Conference. Citeseer, 1\u201313."},{"key":"e_1_3_3_79_2","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173817"},{"key":"e_1_3_3_80_2","doi-asserted-by":"publisher","DOI":"10.3389\/frvir.2021.645153"},{"key":"e_1_3_3_81_2","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2018.2793618"},{"key":"e_1_3_3_82_2","doi-asserted-by":"publisher","DOI":"10.1177\/1046878105282279"},{"key":"e_1_3_3_83_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.advengsoft.2009.01.023"},{"key":"e_1_3_3_84_2","doi-asserted-by":"publisher","DOI":"10.1080\/01443410500341262"},{"key":"e_1_3_3_85_2","first-page":"5","volume-title":"Annual Symposium on Computer-Human Interaction in Play","author":"Zhou Zhuoming","year":"2019","unstructured":"Zhuoming Zhou, Elena M\u00e1rquez Segura, Jared Duval, Michael John, and Katherine Isbister. 2019. Astaire: A collaborative mixed reality dance game for collocated players. In Annual Symposium on Computer-Human Interaction in Play. ACM, New York, NY, USA, 5\u201318."},{"key":"e_1_3_3_86_2","doi-asserted-by":"publisher","DOI":"10.1155\/2018\/5681652"}],"container-title":["ACM Transactions on Interactive Intelligent Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3558773","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3558773","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:00:28Z","timestamp":1750186828000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3558773"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,3,9]]},"references-count":85,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2023,3,31]]}},"alternative-id":["10.1145\/3558773"],"URL":"https:\/\/doi.org\/10.1145\/3558773","relation":{},"ISSN":["2160-6455","2160-6463"],"issn-type":[{"value":"2160-6455","type":"print"},{"value":"2160-6463","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,3,9]]},"assertion":[{"value":"2021-11-24","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-07-18","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-03-09","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}