{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,23]],"date-time":"2026-02-23T22:31:35Z","timestamp":1771885895946,"version":"3.50.1"},"reference-count":68,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2022,7,26]],"date-time":"2022-07-26T00:00:00Z","timestamp":1658793600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Interact. Intell. Syst."],"published-print":{"date-parts":[[2022,9,30]]},"abstract":"<jats:p>Designing games is a complicated and time-consuming process, where developing new levels for existing games can take weeks. Procedural content generation offers the potential to shorten this timeframe, however, automated design tools are not adopted widely in the game industry. This article presents an expert evaluation of a human-in-the-loop generative design approach for commercial game maps that incorporates multiple computational agents. The evaluation aims to gauge the extent to which such an approach could support and be accepted by human game designers and to determine whether the computational agents improve the overall design. To evaluate the approach, 11 game designers utilized the approach to design game levels with the computational agents both active and inactive. Eye-tracking, observational, and think-aloud data was collected to determine whether designers favored levels suggested by the computational agents. This data was triangulated with qualitative data from semi-structured interviews that were used to gather overall opinions of the approach. The eye-tracking data indicates that the participating game level designers showed a clear preference for levels suggested by the computational agents, however, expert designers in particular appeared to reject the idea that the computational agents are helpful. The perception of computational tools not being useful needs to be addressed if procedural content generation approaches are to fulfill their potential for the game industry.<\/jats:p>","DOI":"10.1145\/3531009","type":"journal-article","created":{"date-parts":[[2022,4,30]],"date-time":"2022-04-30T11:12:37Z","timestamp":1651317157000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["Evaluation of a Multi-agent \u201cHuman-in-the-loop\u201d Game Design System"],"prefix":"10.1145","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1460-1547","authenticated-orcid":false,"given":"Jan","family":"Kruse","sequence":"first","affiliation":[{"name":"Mediadesign University of Applied Sciences, Berlin, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4046-5959","authenticated-orcid":false,"given":"Andy M.","family":"Connor","sequence":"additional","affiliation":[{"name":"Auckland University of Technology, Auckland, New Zealand"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5997-3136","authenticated-orcid":false,"given":"Stefan","family":"Marks","sequence":"additional","affiliation":[{"name":"Auckland University of Technology, Auckland, New Zealand"}]}],"member":"320","published-online":{"date-parts":[[2022,7,26]]},"reference":[{"key":"e_1_3_1_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/3067695.3075614"},{"key":"e_1_3_1_3_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00163-002-0023-z"},{"key":"e_1_3_1_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/3235765.3235815"},{"key":"e_1_3_1_5_2","doi-asserted-by":"publisher","DOI":"10.1109\/CIG.2017.8080411"},{"key":"e_1_3_1_6_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4471-0819-1_8"},{"key":"e_1_3_1_7_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.destud.2015.08.001"},{"key":"e_1_3_1_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/2001576.2001631"},{"key":"e_1_3_1_9_2","first-page":"63","volume-title":"Proceedings of the European Conference on the Applications of Evolutionary Computation","author":"Cardamone L.","year":"2001","unstructured":"L. Cardamone, G. N. Yannakakis, J. Togelius, and P. L. Lanzi. 2001. Evolving interesting maps for a first person shooter. In Proceedings of the European Conference on the Applications of Evolutionary Computation. Springer, 63\u201372."},{"key":"e_1_3_1_10_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.asoc.2014.11.010"},{"key":"e_1_3_1_11_2","doi-asserted-by":"publisher","DOI":"10.26522\/brocked.v12i2.38"},{"key":"e_1_3_1_12_2","unstructured":"F. Cifaldi. 2006. Analysts: FPS \u201cMost Attractive\u201d genre for publishers. Retrieved from http:\/\/www.gamasutra.com\/php-bin\/news_index.php?story=8241."},{"key":"e_1_3_1_13_2","doi-asserted-by":"publisher","DOI":"10.1007\/s40869-017-0043-6"},{"issue":"15","key":"e_1_3_1_14_2","first-page":"e4","article-title":"Evolutionary generation of game levels","volume":"5","author":"Connor A. M.","year":"2018","unstructured":"A. M. Connor, T. J. Grieg, and J. Kruse. 2018. Evolutionary generation of game levels. EAI Trans. Creat. Technol. 5, 15 (2018), e4.","journal-title":"EAI Trans. Creat. Technol."},{"key":"e_1_3_1_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/TCIAIG.2016.2520305"},{"key":"e_1_3_1_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/2967934.2968086"},{"key":"e_1_3_1_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/SBGAMES.2011.15"},{"key":"e_1_3_1_18_2","first-page":"23","volume-title":"Proceedings of the Annual Symposium on Computer-Human Interaction in Play","author":"Denisova Alena","year":"2015","unstructured":"Alena Denisova and Paul Cairns. 2015. The placebo effect in digital games: Phantom perception of adaptive artificial intelligence. In Proceedings of the Annual Symposium on Computer-Human Interaction in Play. ACM, 23\u201333."},{"key":"e_1_3_1_19_2","first-page":"1","article-title":"An approach to level design using procedural content generation and difficulty curves","author":"Hector Adrian Diaz-Furlong","year":"2013","unstructured":"Adrian Diaz-Furlong Hector and Luisa Solis-Gonzalez Cosio Ana. 2013. An approach to level design using procedural content generation and difficulty curves. In Proceedings of the Conference on Computational Intelligence in Games. IEEE, 1\u20138.","journal-title":"Proceedings of the Conference on Computational Intelligence in Games"},{"key":"e_1_3_1_20_2","doi-asserted-by":"publisher","DOI":"10.5555\/3134162"},{"key":"e_1_3_1_21_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-31727-9_10"},{"key":"e_1_3_1_22_2","first-page":"44","volume-title":"Proceedings of the International Conference on Augmented Reality, Virtual Reality and Computer Graphics","author":"Galdieri Riccardo","unstructured":"Riccardo Galdieri, Alessandro Longobardi, Michele De Bonis, and Marcello Carrozzino. 2021. Users\u2019 evaluation of procedurally generated game levels. In Proceedings of the International Conference on Augmented Reality, Virtual Reality and Computer Graphics. Springer, 44\u201352."},{"key":"e_1_3_1_23_2","doi-asserted-by":"publisher","DOI":"10.1109\/GEM.2018.8516539"},{"key":"e_1_3_1_24_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0169-8141(98)00068-7"},{"key":"e_1_3_1_25_2","first-page":"31","volume-title":"Proceedings of the 14th Conference on Artificial Intelligence and Interactive Digital Entertainment. AAAI","author":"Guzdial Matthew","year":"2018","unstructured":"Matthew Guzdial and Mark Riedl. 2018. Automated game design via conceptual expansion. In Proceedings of the 14th Conference on Artificial Intelligence and Interactive Digital Entertainment. AAAI, 31\u201337."},{"key":"e_1_3_1_26_2","doi-asserted-by":"publisher","DOI":"10.1109\/TCIAIG.2009.2038365"},{"key":"e_1_3_1_27_2","volume-title":"SAGE Research Methods Foundations","author":"Hennink Monique M.","year":"2020","unstructured":"Monique M. Hennink and Bonnie N. Kaiser. 2020. Saturation in qualitative research. In SAGE Research Methods Foundations, Paul Atkinson, Sara Delamont, Alexandru Cernat, Joseph W. Sakshaug and Richard A. Williams (Eds.). SAGE Publications Limited, Thousand Oaks, CA. Retrieved from https:\/\/methods.sagepub.com\/foundations\/saturation-in-qualitative-research."},{"key":"e_1_3_1_28_2","doi-asserted-by":"publisher","DOI":"10.1080\/10447311003781300"},{"key":"e_1_3_1_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/1822348.1822359"},{"key":"e_1_3_1_30_2","volume-title":"Games without Frontiers: Methods for Game Studies and Design","author":"J\u00e4rvinen Aki","year":"2009","unstructured":"Aki J\u00e4rvinen. 2009. Games without Frontiers: Methods for Game Studies and Design. VDM Verlag, Saarbr\u00fccken, Germany."},{"key":"e_1_3_1_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/IJCNN52387.2021.9533847"},{"key":"e_1_3_1_32_2","first-page":"77","volume-title":"Proceedings of the 7th International Conference on Computational Creativity","author":"Kantosalo Anna","year":"2016","unstructured":"Anna Kantosalo and Hannu Toivonen. 2016. Modes for creative human-computer collaboration: Alternating and task-divided co-creativity. In Proceedings of the 7th International Conference on Computational Creativity. Association for Computational Creativity, 77\u201384."},{"key":"e_1_3_1_33_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0142-694X(01)00021-7"},{"key":"e_1_3_1_34_2","first-page":"95","volume-title":"Proceedings of the 16th Conference on Artificial Intelligence and Interactive Digital Entertainment","author":"Khalifa Ahmed","year":"2020","unstructured":"Ahmed Khalifa, Philip Bontrager, Sam Earle, and Julian Togelius. 2020. PCGRL: Procedural content generation via reinforcement learning. In Proceedings of the 16th Conference on Artificial Intelligence and Interactive Digital Entertainment. AAAI, 95\u2013101."},{"key":"e_1_3_1_35_2","first-page":"216","volume-title":"Proceedings of the 8th Annual International Conference on Industrial Engineering \u2013 Theory, Applications and Practice","author":"Khasawneh Mohammad T.","year":"2003","unstructured":"Mohammad T. Khasawneh, Shannon R. Bowling, Xiaochun Jiang, Anand K. Gramopadhye, and Brian J. Melloy. 2003. A model for predicting human trust in automated systems. In Proceedings of the 8th Annual International Conference on Industrial Engineering \u2013 Theory, Applications and Practice. Citeseer, 216\u2013222."},{"key":"e_1_3_1_36_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSMC.2001.972056"},{"key":"e_1_3_1_37_2","doi-asserted-by":"publisher","DOI":"10.4108\/eai.20-10-2015.150099"},{"key":"e_1_3_1_38_2","doi-asserted-by":"publisher","DOI":"10.1007\/s40869-020-00119-z"},{"key":"e_1_3_1_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/2843043.2843479"},{"key":"e_1_3_1_40_2","volume-title":"Interactive Evolutionary Computation in Design Applications for Virtual Worlds","author":"Kruse Jan","year":"2014","unstructured":"Jan Kruse. 2014. Interactive Evolutionary Computation in Design Applications for Virtual Worlds. Thesis. Auckland University of Technology. Retrieved from http:\/\/aut.researchgateway.ac.nz\/handle\/10292\/8593."},{"key":"e_1_3_1_41_2","first-page":"1","article-title":"Evolving maps for match balancing in first person shooters","author":"Lanzi P. L.","year":"2014","unstructured":"P. L. Lanzi, D. Lomacono, and R. Stucchi. 2014. Evolving maps for match balancing in first person shooters. In Proceedings of the Conference on Computational Intelligence and Games. IEEE, 1\u20138.","journal-title":"Proceedings of the Conference on Computational Intelligence and Games"},{"key":"e_1_3_1_42_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-42716-4_11"},{"issue":"2","key":"e_1_3_1_43_2","first-page":"136","article-title":"Can computers foster human users\u2019 creativity? Theory and praxis of mixed-initiative co-creativity","volume":"8","author":"Liapis Antonios","year":"2016","unstructured":"Antonios Liapis, Georgios N. Yannakakis, Constantine Alexopoulos, and Phil Lopes. 2016. Can computers foster human users\u2019 creativity? Theory and praxis of mixed-initiative co-creativity. Digit. Cult. Educ. 8, 2 (2016), 136\u2013153.","journal-title":"Digit. Cult. Educ."},{"key":"e_1_3_1_44_2","first-page":"213","volume-title":"Proceedings of the 8th International Conference on the Foundations of Digital Games","author":"Liapis Antonios","year":"2013","unstructured":"Antonios Liapis, Georgios N. Yannakakis, and Julian Togelius. 2013. Sentient sketchbook: Computer-assisted game level authoring. In Proceedings of the 8th International Conference on the Foundations of Digital Games. ACM, 213\u2013220."},{"key":"e_1_3_1_45_2","first-page":"30","volume-title":"Proceedings of the 9th Conference on Artificial Intelligence and Interactive Digital Entertainment. AAAI","author":"Liapis Antonios","year":"2013","unstructured":"Antonios Liapis, Georgios Yannakakis, and Julian Togelius. 2013. Towards a generic method of evaluating game levels. In Proceedings of the 9th Conference on Artificial Intelligence and Interactive Digital Entertainment. AAAI, 30\u201336."},{"key":"e_1_3_1_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/2614106.2614136"},{"key":"e_1_3_1_47_2","first-page":"1527","article-title":"Interactive evolution of levels for a competitive multiplayer FPS","author":"\u00d8lsted P. T.","year":"2015","unstructured":"P. T. \u00d8lsted, B. Ma, and S. Risi. 2015. Interactive evolution of levels for a competitive multiplayer FPS. In Proceedings of the Congress on Evolutionary Computation. IEEE, 1527\u20131534.","journal-title":"Proceedings of the Congress on Evolutionary Computation"},{"key":"e_1_3_1_48_2","doi-asserted-by":"publisher","DOI":"10.1037\/qup0000194"},{"key":"e_1_3_1_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/1274000.1274098"},{"key":"e_1_3_1_50_2","doi-asserted-by":"publisher","DOI":"10.1007\/s12652-018-0861-9"},{"key":"e_1_3_1_51_2","first-page":"1","volume-title":"Proceedings of the Games Entertainment Media Conference","author":"Pirovano Michele","year":"2015","unstructured":"Michele Pirovano, Renato Mainetti, and Daniele Loiacono. 2015. Volcano: An interactive sword generator. In Proceedings of the Games Entertainment Media Conference. IEEE, 1\u20138."},{"key":"e_1_3_1_52_2","doi-asserted-by":"publisher","DOI":"10.1109\/ECAI50035.2020.9223203"},{"key":"e_1_3_1_53_2","doi-asserted-by":"publisher","DOI":"10.1109\/CIG.2015.7317906"},{"key":"e_1_3_1_54_2","doi-asserted-by":"publisher","DOI":"10.22456\/1679-1916.95952"},{"key":"e_1_3_1_55_2","author":"Salda\u00f1a Johnny","year":"2016","unstructured":"Johnny Salda\u00f1a. 2016. The Coding Manual for Qualitative Researchers (2nd ed.). Sage.","journal-title":"The Coding Manual for Qualitative Researchers"},{"key":"e_1_3_1_56_2","first-page":"148","volume-title":"Proceedings of the Genetic and Evolutionary Computation Conference","author":"Schrum Jacob","year":"2020","unstructured":"Jacob Schrum, Jake Gutierrez, Vanessa Volz, Jialin Liu, Simon Lucas, and Sebastian Risi. 2020. Interactive evolution and exploration within latent level-design space of generative adversarial networks. In Proceedings of the Genetic and Evolutionary Computation Conference. ACM, 148\u2013156."},{"key":"e_1_3_1_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/3210753"},{"key":"e_1_3_1_58_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-42716-4_12"},{"key":"e_1_3_1_59_2","doi-asserted-by":"publisher","DOI":"10.5555\/3161477"},{"key":"e_1_3_1_60_2","unstructured":"Mike Stout. 2016. A beginner's guide to designing video game levels. Retrieved from https:\/\/gamedevelopment.tutsplus.com\/tutorials\/a-beginners-guide-to-designing-video-game-levels\u2013cms-25662."},{"key":"e_1_3_1_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/3102071.3102080"},{"key":"e_1_3_1_62_2","doi-asserted-by":"publisher","DOI":"10.1109\/5.949485"},{"key":"e_1_3_1_63_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10710-012-9174-5"},{"key":"e_1_3_1_64_2","volume-title":"Proceedings of the Workshop on Adaptive Approaches for Optimizing Player Satisfaction in Computer and Physical Games","author":"Togelius Julian","year":"2006","unstructured":"Julian Togelius, Renzo De Nardi, and Simon M. Lucas. 2006. Making racing fun through player modeling and track evolution. In Proceedings of the Workshop on Adaptive Approaches for Optimizing Player Satisfaction in Computer and Physical Games."},{"key":"e_1_3_1_65_2","doi-asserted-by":"publisher","DOI":"10.1145\/2000919.2000922"},{"key":"e_1_3_1_66_2","doi-asserted-by":"publisher","DOI":"10.1109\/CSCI46756.2018.00205"},{"key":"e_1_3_1_67_2","volume-title":"Thought and Language","author":"Vygotski\u012d Lev. S.","year":"2012","unstructured":"Lev. S. Vygotski\u012d. 2012. Thought and Language. The MIT Press."},{"key":"e_1_3_1_68_2","doi-asserted-by":"publisher","DOI":"10.1109\/T-AFFC.2011.6"},{"key":"e_1_3_1_69_2","doi-asserted-by":"publisher","DOI":"10.1109\/CIG.2018.8490433"}],"container-title":["ACM Transactions on Interactive Intelligent Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3531009","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3531009","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T18:09:27Z","timestamp":1750183767000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3531009"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,7,26]]},"references-count":68,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2022,9,30]]}},"alternative-id":["10.1145\/3531009"],"URL":"https:\/\/doi.org\/10.1145\/3531009","relation":{},"ISSN":["2160-6455","2160-6463"],"issn-type":[{"value":"2160-6455","type":"print"},{"value":"2160-6463","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,7,26]]},"assertion":[{"value":"2021-06-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-03-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-07-26","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}