{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,11]],"date-time":"2026-03-11T17:09:30Z","timestamp":1773248970441,"version":"3.50.1"},"reference-count":49,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2024,7,11]],"date-time":"2024-07-11T00:00:00Z","timestamp":1720656000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100002848","name":"National Agency for Research and Development (Agencia Nacional de Investigaci\u00f3n y Desarrollo, ANID Chile)","doi-asserted-by":"publisher","award":["2023-21230824"],"award-info":[{"award-number":["2023-21230824"]}],"id":[{"id":"10.13039\/501100002848","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100002848","name":"National Agency for Research and Development (Agencia Nacional de Investigaci\u00f3n y Desarrollo, ANID Chile)","doi-asserted-by":"publisher","award":["11220438"],"award-info":[{"award-number":["11220438"]}],"id":[{"id":"10.13039\/501100002848","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100002848","name":"FONDECYT Iniciaci\u00f3n","doi-asserted-by":"publisher","award":["2023-21230824"],"award-info":[{"award-number":["2023-21230824"]}],"id":[{"id":"10.13039\/501100002848","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100002848","name":"FONDECYT Iniciaci\u00f3n","doi-asserted-by":"publisher","award":["11220438"],"award-info":[{"award-number":["11220438"]}],"id":[{"id":"10.13039\/501100002848","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>Procedural Content Generation for video games (PCG) is widely used by today\u2019s video game industry to create huge open worlds or enhance replayability. However, there is little scientific evidence that these systems produce high-quality content. In this document, we evaluate three open-source automated level generators for Super Mario Bros in addition to the original levels used for training. These are based on Genetic Algorithms, Generative Adversarial Networks, and Markov Chains. The evaluation was performed through an Expressive Range Analysis (ERA) on 200 levels with nine metrics. The results show how analyzing the algorithms\u2019 expressive range can help us evaluate the generators as a preliminary measure to study whether they respond to users\u2019 needs. This method allows us to recognize potential problems early in the content generation process, in addition to taking action to guarantee quality content when a generator is used.<\/jats:p>","DOI":"10.3390\/a17070307","type":"journal-article","created":{"date-parts":[[2024,7,11]],"date-time":"2024-07-11T11:33:22Z","timestamp":1720697602000},"page":"307","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Evaluating the Expressive Range of Super Mario Bros Level Generators"],"prefix":"10.3390","volume":"17","author":[{"ORCID":"https:\/\/orcid.org\/0009-0004-8175-4396","authenticated-orcid":false,"given":"Hans","family":"Schaa","sequence":"first","affiliation":[{"name":"Doctoral Program in Engineering Systems, Faculty of Engineering, Universidad de Talca, Curic\u00f3 3340000, Chile"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1914-3840","authenticated-orcid":false,"given":"Nicolas A.","family":"Barriga","sequence":"additional","affiliation":[{"name":"Department of Interactive Visualization and Virtual Reality, Faculty of Engineering, Universidad de Talca, Talca 3460000, Chile"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2024,7,11]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Shaker, N., Togelius, J., and Nelson, M.J. (2016). Procedural Content Generation in Games, Springer.","DOI":"10.1007\/978-3-319-42716-4"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2974026","article-title":"Procedural content generation for game props? A study on the effects on user experience","volume":"15","author":"Korn","year":"2017","journal-title":"Comput. Entertain. (CIE)"},{"key":"ref_3","unstructured":"Scirea, M., Barros, G.A., Shaker, N., and Togelius, J. (July, January 29). SMUG: Scientific Music Generator. Proceedings of the ICCC, Park City, UT, USA."},{"key":"ref_4","unstructured":"Gandikota, R., and Brown, N.B. (2022). DC-Art-GAN: Stable Procedural Content Generation using DC-GANs for Digital Art. arXiv."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Schaa, H., and Barriga, N.A. (2021, January 15\u201319). Generating Entertaining Human-Like Sokoban Initial States. Proceedings of the 2021 40th International Conference of the Chilean Computer Science Society (SCCC), La Serena, Chile.","DOI":"10.1109\/SCCC54552.2021.9650431"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Amato, A. (2017). Procedural content generation in the game industry. Game Dynamics: Best Practices in Procedural and Dynamic Game Content Generation, Springer.","DOI":"10.1007\/978-3-319-53088-8_2"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Tyni, H., and Sotamaa, O. (2011, January 28\u201330). Extended or exhausted: How console DLC keeps the player on the rail. Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments, Tampere, Finland.","DOI":"10.1145\/2181037.2181094"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Ferreira, L., Pereira, L., and Toledo, C. (2014, January 12\u201316). A multi-population genetic algorithm for procedural generation of levels for platform games. Proceedings of the Companion Publication of the 2014 Annual Conference on Genetic and Evolutionary Computation, Vancouver, BC, Canada.","DOI":"10.1145\/2598394.2598489"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Volz, V., Schrum, J., Liu, J., Lucas, S.M., Smith, A., and Risi, S. (2018, January 15\u201319). Evolving mario levels in the latent space of a deep convolutional generative adversarial network. Proceedings of the Genetic and Evolutionary Computation Conference, Kyoto, Japan.","DOI":"10.1145\/3205455.3205517"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Smith, G., and Whitehead, J. (2010, January 18). Analyzing the expressive range of a level generator. Proceedings of the 2010 Workshop on Procedural Content Generation in Games, Monterey, CA, USA.","DOI":"10.1145\/1814256.1814260"},{"key":"ref_11","unstructured":"(2024, July 10). The SuperTux Team, SuperTux. Available online: https:\/\/www.supertux.org\/."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Kerssemakers, M., Tuxen, J., Togelius, J., and Yannakakis, G.N. (2012, January 11\u201314). A procedural procedural level generator generator. Proceedings of the 2012 IEEE Conference on Computational Intelligence and Games (CIG), Granada, Spain.","DOI":"10.1109\/CIG.2012.6374174"},{"key":"ref_13","unstructured":"Togelius, J., Champandard, A.J., Lanzi, P.L., Mateas, M., Paiva, A., Preuss, M., and Stanley, K.O. (2013). Procedural content generation: Goals, challenges and actionable steps. Artificial and Computational Intelligence in Games. Dagstuhl Follow-Ups, Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Gravina, D., Khalifa, A., Liapis, A., Togelius, J., and Yannakakis, G.N. (2019, January 20\u201323). Procedural content generation through quality diversity. Proceedings of the 2019 IEEE Conference on Games (CoG), London, UK.","DOI":"10.1109\/CIG.2019.8848053"},{"key":"ref_15","unstructured":"Craveirinha, R., Santos, L., and Roque, L. (2013, January 12\u201315). An author-centric approach to procedural content generation. Proceedings of the Advances in Computer Entertainment: 10th International Conference, ACE 2013, Boekelo, The Netherlands. Proceedings 10."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Togelius, J., Preuss, M., and Yannakakis, G.N. (2010, January 18). Towards multiobjective procedural map generation. Proceedings of the 2010 Workshop on Procedural Content Generation in Games, Monterey, CA, USA.","DOI":"10.1145\/1814256.1814259"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"19","DOI":"10.1007\/s00521-020-05383-8","article-title":"Deep learning for procedural content generation","volume":"33","author":"Liu","year":"2021","journal-title":"Neural Comput. Appl."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Guzdial, M., Sturtevant, N., and Li, B. (2016, January 8\u201312). Deep static and dynamic level analysis: A study on infinite mario. Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, Burlingame, CA, USA.","DOI":"10.1609\/aiide.v12i2.12894"},{"key":"ref_19","unstructured":"Wikipedia (2023, December 27). Super Mario Bros. Available online: https:\/\/en.wikipedia.org\/wiki\/Super_Mario."},{"key":"ref_20","unstructured":"Compton, K., and Mateas, M. (2016, January 8\u201312). Procedural level design for platform games. Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, Burlingame, CA, USA."},{"key":"ref_21","unstructured":"Iyer, V., Bilmes, J., Wright, M., and Wessel, D. (1997, January 25\u201330). A novel representation for rhythmic structure. Proceedings of the 23rd International Computer Music Conference, Thessaloniki, Greece."},{"key":"ref_22","unstructured":"Mourato, F., and Santos, M.P.d. (2010, January 13\u201315). Measuring difficulty in platform videogames. Proceedings of the 4\u00aa Confer\u00eancia Nacional Interac\u00e7\u00e3o Humano-Computador, Aveiro, Portugal."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Pedersen, C., Togelius, J., and Yannakakis, G.N. (2009, January 7\u201310). Modeling player experience in super mario bros. Proceedings of the 2009 IEEE Symposium on Computational Intelligence and Games, Milan, Italy.","DOI":"10.1109\/CIG.2009.5286482"},{"key":"ref_24","unstructured":"Summerville, A., and Mateas, M. (2016). Super mario as a string: Platformer level generation via lstms. arXiv."},{"key":"ref_25","unstructured":"Sudhakaran, S., Gonz\u00e1lez-Duque, M., Freiberger, M., Glanois, C., Najarro, E., and Risi, S. (2023, January 10\u201316). Mariogpt: Open-ended text2level generation through large language models. Proceedings of the Advances in Neural Information Processing Systems, New Orleans, LA, USA."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Todd, G., Earle, S., Nasir, M.U., Green, M.C., and Togelius, J. (2023, January 12\u201314). Level Generation Through Large Language Models. Proceedings of the 18th International Conference on the Foundations of Digital Games (FDG 2023), Lisbon, Portugal.","DOI":"10.1145\/3582437.3587211"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Gallotta, R., Todd, G., Zammit, M., Earle, S., Liapis, A., Togelius, J., and Yannakakis, G.N. (2024). Large Language Models and Games: A Survey and Roadmap. arXiv.","DOI":"10.1109\/TG.2024.3461510"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Nasir, M.U., James, S., and Togelius, J. (2024). Word2World: Generating Stories and Worlds through Large Language Models. arXiv.","DOI":"10.1109\/CoG57401.2023.10333197"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Nasir, M.U., and Togelius, J. (2023). Practical PCG Through Large Language Models. arXiv.","DOI":"10.1109\/CoG57401.2023.10333197"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Shu, T., Liu, J., and Yannakakis, G.N. (2021, January 17\u201320). Experience-driven PCG via reinforcement learning: A Super Mario Bros study. Proceedings of the 2021 IEEE Conference on Games (CoG), Copenhagen, Denmark.","DOI":"10.1109\/CoG52621.2021.9619124"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Hauck, E., and Aranha, C. (2020, January 24\u201327). Automatic generation of Super Mario levels via graph grammars. Proceedings of the 2020 IEEE Conference on Games (CoG), Osaka, Japan.","DOI":"10.1109\/CoG47356.2020.9231526"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Shaker, N., Yannakakis, G.N., Togelius, J., Nicolau, M., and O\u2019neill, M. (2012, January 8\u201312). Evolving personalized content for super mario bros using grammatical evolution. Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, Stanford, CA, USA.","DOI":"10.1609\/aiide.v8i1.12501"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Shi, P., and Chen, K. (2016, January 20\u201323). Online level generation in Super Mario Bros via learning constructive primitives. Proceedings of the 2016 IEEE Conference on Computational Intelligence and Games (CIG), Santorini, Greece.","DOI":"10.1109\/CIG.2016.7860397"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"de Pontes, R.G., and Gomes, H.M. (2020, January 7\u201310). Evolutionary procedural content generation for an endless platform game. Proceedings of the 2020 19th Brazilian Symposium on Computer Games and Digital Entertainment (SBGames), Recife, Brazil.","DOI":"10.1109\/SBGames51465.2020.00021"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Dahlskog, S., and Togelius, J. (2014, January 23\u201325). Procedural content generation using patterns as objectives. Proceedings of the European Conference on the Applications of Evolutionary Computation, Granada, Spain.","DOI":"10.1007\/978-3-662-45523-4_27"},{"key":"ref_36","unstructured":"Sarkar, A., and Cooper, S. (2021). Procedural content generation using behavior trees (PCGBT). arXiv."},{"key":"ref_37","unstructured":"Snodgrass, S. (2018). Markov Models for Procedural Content Generation, Drexel University."},{"key":"ref_38","unstructured":"Murphy, K.P. (2024, June 01). Markov Models. Available online: https:\/\/citeseerx.ist.psu.edu\/document?repid=rep1&type=pdf&doi=6c4d1f04f5c5004370f03f2e759e6a4b1115cb8c."},{"key":"ref_39","unstructured":"Snodgrass, S., and Onta\u00f1\u00f3n, S. (2014, January 3\u20137). A hierarchical approach to generating maps using markov chains. Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, Raleigh, NC, USA."},{"key":"ref_40","unstructured":"Snodgrass, S., and Ontanon, S. (2015, January 14\u201318). A hierarchical mdmc approach to 2D video game map generation. Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, Santa Cruz, CA, USA."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"410","DOI":"10.1109\/TCIAIG.2016.2623560","article-title":"Learning to generate video game maps using markov models","volume":"9","author":"Snodgrass","year":"2016","journal-title":"IEEE Trans. Comput. Intell. AI Games"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Biemer, C.F., and Cooper, S. (2022, January 21\u201324). On linking level segments. Proceedings of the 2022 IEEE Conference on Games (CoG), Beijing, China.","DOI":"10.1109\/CoG51982.2022.9893705"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Withington, O., and Tokarchuk, L. (2023, January 12\u201314). The Right Variety: Improving Expressive Range Analysis with Metric Selection Methods. Proceedings of the 18th International Conference on the Foundations of Digital Games, Lisbon, Portugal.","DOI":"10.1145\/3582437.3582453"},{"key":"ref_44","unstructured":"Horn, B., Dahlskog, S., Shaker, N., Smith, G., and Togelius, J. (2014, January 3\u20137). A comparative evaluation of procedural level generators in the mario AI framework. Proceedings of the Foundations of Digital Games 2014, Ft. Lauderdale, FL, USA."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Summerville, A., Mari\u00f1o, J.R., Snodgrass, S., Onta\u00f1\u00f3n, S., and Lelis, L.H. (2017, January 14\u201317). Understanding mario: An evaluation of design metrics for platformers. Proceedings of the 12th International Conference on the Foundations of Digital Games, Hyannis, MA, USA.","DOI":"10.1145\/3102071.3102080"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"379","DOI":"10.1002\/j.1538-7305.1948.tb01338.x","article-title":"A mathematical theory of communication","volume":"27","author":"Shannon","year":"1948","journal-title":"Bell Syst. Tech. J."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Cook, M., and Colton, S. (September, January 31). Multi-faceted evolution of simple arcade games. Proceedings of the 2011 IEEE Conference on Computational Intelligence and Games (CIG\u201911), Seoul, Republic of Korea.","DOI":"10.1109\/CIG.2011.6032019"},{"key":"ref_48","unstructured":"Arjovsky, M., Chintala, S., and Bottou, L. (2017, January 6\u201311). Wasserstein generative adversarial networks. Proceedings of the International Conference on Machine Learning, ICML, Sidney, Australia."},{"key":"ref_49","unstructured":"Di Liello, L., Ardino, P., Gobbi, J., Morettin, P., Teso, S., and Passerini, A. (2020, January 6\u201312). Efficient generation of structured objects with constrained adversarial networks. Proceedings of the Advances in Neural Information Processing Systems, Online."}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/17\/7\/307\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T15:13:14Z","timestamp":1760109194000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/17\/7\/307"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,11]]},"references-count":49,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2024,7]]}},"alternative-id":["a17070307"],"URL":"https:\/\/doi.org\/10.3390\/a17070307","relation":{},"ISSN":["1999-4893"],"issn-type":[{"value":"1999-4893","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,7,11]]}}}