{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,16]],"date-time":"2026-07-16T15:37:21Z","timestamp":1784216241892,"version":"3.55.0"},"reference-count":69,"publisher":"ASME International","issue":"5","license":[{"start":{"date-parts":[[2020,2,11]],"date-time":"2020-02-11T00:00:00Z","timestamp":1581379200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.asme.org\/publications-submissions\/publishing-information\/legal-policies"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["1834465"],"award-info":[{"award-number":["1834465"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["asmedigitalcollection.asme.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2020,10,1]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>This work presents a deep reinforcement learning (DRL) approach for procedural content generation (PCG) to automatically generate three-dimensional (3D) virtual environments that users can interact with. The primary objective of PCG methods is to algorithmically generate new content in order to improve user experience. Researchers have started exploring the use of machine learning (ML) methods to generate content. However, these approaches frequently implement supervised ML algorithms that require initial datasets to train their generative models. In contrast, RL algorithms do not require training data to be collected a priori since they take advantage of simulation to train their models. Considering the advantages of RL algorithms, this work presents a method that generates new 3D virtual environments by training an RL agent using a 3D simulation platform. This work extends the authors\u2019 previous work and presents the results of a case study that supports the capability of the proposed method to generate new 3D virtual environments. The ability to automatically generate new content has the potential to maintain users\u2019 engagement in a wide variety of applications such as virtual reality applications for education and training, and engineering conceptual design.<\/jats:p>","DOI":"10.1115\/1.4046293","type":"journal-article","created":{"date-parts":[[2020,2,11]],"date-time":"2020-02-11T10:36:30Z","timestamp":1581417390000},"update-policy":"https:\/\/doi.org\/10.1115\/crossmarkpolicy-asme","source":"Crossref","is-referenced-by-count":17,"title":["Deep Reinforcement Learning for Procedural Content Generation of 3D Virtual Environments"],"prefix":"10.1115","volume":"20","author":[{"given":"Christian E.","family":"L\u00f3pez","sequence":"first","affiliation":[{"name":"Department of Computer Science, Afiliated to Mechanical Engineering, Lafayette College, Easton, PA 18042"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"James","family":"Cunningham","sequence":"additional","affiliation":[{"name":"Department of Mechanical Engineering, Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA 15213"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Omar","family":"Ashour","sequence":"additional","affiliation":[{"name":"Department of Industrial Engineering, The Pennsylvania State University, Erie, PA 16563"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Conrad S.","family":"Tucker","sequence":"additional","affiliation":[{"name":"Department of Mechanical Engineering, Courtesy Appointment, Machine Learning, Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA 15213"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"33","published-online":{"date-parts":[[2020,6,3]]},"reference":[{"key":"2021060300032749700_CIT0001","article-title":"Super Mario as a String: Platformer Level Generation Via LSTMs","author":"Summerville","year":"2016"},{"key":"2021060300032749700_CIT0002","first-page":"175","article-title":"Navigating Through the RLATES Interface: A Web-based Adaptive and Intelligent Educational System","author":"Iglesias","year":"2003"},{"key":"2021060300032749700_CIT0003","first-page":"460","article-title":"Building Adaptive Tutoring Model Using Artificial Neural Networks and Reinforcement Learning","author":"Fenza","year":"2017"},{"issue":"3","key":"2021060300032749700_CIT0004","doi-asserted-by":"crossref","first-page":"172","DOI":"10.1109\/TCIAIG.2011.2148116","article-title":"Search-Based Procedural Content Generation: A Taxonomy and Survey","volume":"3","author":"Togelius","year":"2011","journal-title":"IEEE Trans. Comput. Intell. AI Games"},{"key":"2021060300032749700_CIT0005","article-title":"Integrating Semantics and Procedural Generation: Key Enabling Factors for Declarative Modeling of Virtual Worlds","author":"Bidarra","year":"2010"},{"issue":"3","key":"2021060300032749700_CIT0006","doi-asserted-by":"crossref","first-page":"201","DOI":"10.1109\/TCIAIG.2011.2159716","article-title":"Tanagra: Reactive Planning and Constraint Solving for Mixed-Initiative Level Design","volume":"3","author":"Smith","year":"2011","journal-title":"IEEE Trans. Comput. Intell. AI Games"},{"issue":"2","key":"2021060300032749700_CIT0007","doi-asserted-by":"crossref","first-page":"293","DOI":"10.1177\/0735633117706909","article-title":"A Procedural Content Generation-Based Framework for Educational Games: Toward a Tailored Data-Driven Game for Developing Early English Reading Skills","volume":"56","author":"Hooshyar","year":"2018","journal-title":"J. Educ. Comput. Res."},{"key":"2021060300032749700_CIT0008","first-page":"99","article-title":"Scenario Generation for Emergency Rescue Training Games","author":"Hullett","year":"2009"},{"key":"2021060300032749700_CIT0009","first-page":"323","article-title":"Balancing Learning and Engagement in Game-Based Learning Environments with Multi-Objective Reinforcement Learning","author":"Sawyer","year":"2017"},{"issue":"1","key":"2021060300032749700_CIT0010","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2422956.2422957","article-title":"Procedural Content Generation for Games: A Survey","volume":"9","author":"Hendrikx","year":"2013","journal-title":"ACM Trans. Multimed. Comput. Commun. Appl."},{"key":"2021060300032749700_CIT0011","first-page":"285","article-title":"Game AI Revisited","author":"Yannakakis","year":"2012"},{"issue":"6","key":"2021060300032749700_CIT0012","doi-asserted-by":"crossref","first-page":"26","DOI":"10.1109\/MSP.2017.2743240","article-title":"Deep Reinforcement Learning: A Brief Survey","volume":"34","author":"Arulkumaran","year":"2017","journal-title":"IEEE Signal Process. Mag."},{"key":"2021060300032749700_CIT0013","doi-asserted-by":"crossref","DOI":"10.1115\/DETC2019-97711","article-title":"Reinforcement Learning Content Generation for Virtual Reality Applications","author":"Lopez","year":"2019"},{"issue":"4","key":"2021060300032749700_CIT0014","doi-asserted-by":"crossref","first-page":"332","DOI":"10.1109\/TCIAIG.2011.2166267","article-title":"The 2010 Mario AI Championship: Level Generation Track","volume":"3","author":"Shaker","year":"2011","journal-title":"IEEE Trans. Comput. Intell. AI Games"},{"issue":"2","key":"2021060300032749700_CIT0015","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1109\/TCIAIG.2017.2740210","article-title":"Learning Constructive Primitives for Real-Time Dynamic Difficulty Adjustment in Super Mario Bros","volume":"10","author":"Shi","year":"2018","journal-title":"IEEE Trans. Comput. Intell. AI Games"},{"key":"2021060300032749700_CIT0016","first-page":"31","article-title":"Deep Static and Dynamic Level Analysis: A Study on Infinite Mario","author":"Guzdial","year":"2016"},{"key":"2021060300032749700_CIT0017","first-page":"31","article-title":"The Learning of Zelda: Datadriven Learning of Level Topology","author":"Summerville","year":"2015"},{"key":"2021060300032749700_CIT0018","article-title":"Illuminating Generalization in Deep Reinforcement Learning Through Procedural Level Generation","author":"Justesen","year":"2018"},{"key":"2021060300032749700_CIT0019","first-page":"200","article-title":"Linear Levels Through n-Grams","author":"Dahlskog","year":"2014"},{"key":"2021060300032749700_CIT0020","first-page":"25","article-title":"Generating Maps Using Markov Chains","author":"Snodgrass","year":"2013"},{"key":"2021060300032749700_CIT0021","first-page":"167","article-title":"Alone We Can Do So Little, Together We Can Do So Much: A Combinatorial Approach for Generating Game Content","author":"Shaker","year":"2014"},{"issue":"1","key":"2021060300032749700_CIT0022","doi-asserted-by":"crossref","first-page":"47","DOI":"10.1515\/jaiscr-2017-0004","article-title":"A Survey of Artificial Intelligence Techniques Employed for Adaptive Educational Systems Within e-Learning Platforms","volume":"7","author":"Almohammadi","year":"2017","journal-title":"J. Artif. Intell. Soft Comput. Res."},{"key":"2021060300032749700_CIT0023","first-page":"1","article-title":"A Hybrid Machine Learning Approach to Automated Scenario Generation (ASG) to Support Adaptive Instruction in Virtual Simulations and Games","author":"Sottilare","year":"2018"},{"issue":"6","key":"2021060300032749700_CIT0024","doi-asserted-by":"crossref","first-page":"2091","DOI":"10.1016\/j.eswa.2012.10.014","article-title":"Comparing Strategies for Modeling Students Learning Styles Through Reinforcement Learning in Adaptive and Intelligent Educational Systems: An Experimental Analysis","volume":"40","author":"Dor\u00e7a","year":"2013","journal-title":"Expert Syst. Appl."},{"issue":"1","key":"2021060300032749700_CIT0025","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1007\/s10489-008-0115-1","article-title":"Learning Teaching Strategies in an Adaptive and Intelligent Educational System Through Reinforcement Learning","volume":"31","author":"Iglesias","year":"2009","journal-title":"Appl. Intell."},{"issue":"6","key":"2021060300032749700_CIT0026","doi-asserted-by":"crossref","first-page":"731","DOI":"10.1111\/jcal.12280","article-title":"A Data-Driven Procedural-Content-Generation Approach for Educational Games","volume":"34","author":"Hooshyar","year":"2018","journal-title":"J. Comput. Assist. Learn."},{"issue":"3","key":"2021060300032749700_CIT0027","first-page":"1","article-title":"A Systematic Review of Data-Driven Approaches in Player Modeling of Educational Games","volume":"52","author":"Hooshyar","year":"2017","journal-title":"Artif. Intell. Rev."},{"key":"2021060300032749700_CIT0028","first-page":"156","article-title":"A Case Study of Expressively Constrainable Level Design Automation Tools for a Puzzle Game","author":"Smith","year":"2012"},{"key":"2021060300032749700_CIT0029","first-page":"756","article-title":"A Math Educational Computer Game Using Procedural Content Generation","author":"Rodrigues","year":"2017"},{"key":"2021060300032749700_CIT0030","first-page":"192","article-title":"Towards Player Adaptivity in a Serious Game for Conflict Resolution","author":"Grappiolo","year":"2011"},{"key":"2021060300032749700_CIT0031","first-page":"2595","article-title":"Generative Adversarial Networks for Increasing the Veracity of Big Data","author":"Dering","year":"2017"},{"key":"2021060300032749700_CIT0032","first-page":"2834","article-title":"Gradient-Based Signatures for Big Multimedia Data","author":"Beecks","year":"2015"},{"issue":"2)","key":"2021060300032749700_CIT0033","doi-asserted-by":"crossref","first-page":"021104","DOI":"10.1115\/1.4041857","article-title":"Exploring Biases Between Human and Machine Generated Designs","volume":"141","author":"Lopez","year":"2019","journal-title":"ASME J. Mech. Des."},{"key":"2021060300032749700_CIT0034","doi-asserted-by":"crossref","DOI":"10.1115\/DETC2018-85698","article-title":"Human Validation of Computer vs Human Generated Design Sketches","author":"Lopez","year":"2018"},{"key":"2021060300032749700_CIT0035","article-title":"Sketch-pix2seq: A Model to Generate Sketches of Multiple Categories","author":"Chen","year":"2017"},{"key":"2021060300032749700_CIT0036","first-page":"237","article-title":"Reinforcement Learning: A Survey","volume":"4","author":"Kaelbling","year":"1996","journal-title":"Int. J. Artif. Intell. Res."},{"key":"2021060300032749700_CIT0037","volume-title":"Reinforcement Learning: An Introduction","author":"Sutton","year":"1998"},{"key":"2021060300032749700_CIT0038","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.ins.2013.08.037","article-title":"Reinforcement Learning Algorithms With Function Approximation: Recent Advances and Applications","volume":"261","author":"Xu","year":"2014","journal-title":"Inf. Sci."},{"key":"2021060300032749700_CIT0039","article-title":"Playing Atari With Deep Reinforcement Learning","author":"Mnih","year":"2013"},{"issue":"7676","key":"2021060300032749700_CIT0040","doi-asserted-by":"crossref","first-page":"354","DOI":"10.1038\/nature24270","article-title":"Mastering the Game of Go Without Human Knowledge","volume":"550","author":"Silver","year":"2017","journal-title":"Nature"},{"issue":"7540","key":"2021060300032749700_CIT0041","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-Level Control Through Deep Reinforcement Learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"2021060300032749700_CIT0042","first-page":"4246","article-title":"The Malmo Platform for Artificial Intelligence Experimentation","author":"Johnson","year":"2016"},{"key":"2021060300032749700_CIT0043","first-page":"5026","article-title":"MuJoCo: A Physics Engine for Model-Based Control","author":"Todorov","year":"2012"},{"key":"2021060300032749700_CIT0044","article-title":"Unity: A General Platform for Intelligent Agents","author":"Juliani","year":"2018"},{"key":"2021060300032749700_CIT0045","first-page":"3852","article-title":"Interactive Narrative Personalization With Deep Reinforcement Learning","author":"Wang","year":"2017"},{"key":"2021060300032749700_CIT0046","first-page":"65","article-title":"Toward Automated Scenario Generation With Deep Reinforcement Learning in GIFT","author":"Rowe","year":"2018"},{"issue":"1\u20134","key":"2021060300032749700_CIT0047","doi-asserted-by":"crossref","first-page":"957","DOI":"10.1007\/s00170-017-0895-8","article-title":"Classification of Facility Layout Problems: A Review Study","volume":"94","author":"Hosseini-Nasab","year":"2018","journal-title":"Int. J. Adv. Manuf. Technol."},{"key":"2021060300032749700_CIT0048","doi-asserted-by":"crossref","DOI":"10.1007\/978-1-4419-7997-1","volume-title":"Handbook of Combinatorial Optimization","author":"Pardalos","year":"2013"},{"issue":"2","key":"2021060300032749700_CIT0049","doi-asserted-by":"crossref","first-page":"255","DOI":"10.1016\/j.arcontrol.2007.04.001","article-title":"Facility Layout Problems: A Survey","volume":"31","author":"Drira","year":"2007","journal-title":"Annu. Rev. Control"},{"key":"2021060300032749700_CIT0050","first-page":"319","article-title":"Learning-Based Multi-Agent System for Solving Combinatorial Optimization Problems: A New Architecture","author":"Lotfi","year":"2015"},{"issue":"1","key":"2021060300032749700_CIT0051","doi-asserted-by":"crossref","first-page":"169","DOI":"10.1016\/j.ejor.2016.02.045","article-title":"A Multi-Agent Based Cooperative Approach to Scheduling and Routing","volume":"254","author":"Martin","year":"2016","journal-title":"Eur. J. Oper. Res."},{"key":"2021060300032749700_CIT0052","doi-asserted-by":"crossref","first-page":"276","DOI":"10.1016\/j.asoc.2016.01.006","article-title":"A New Reinforcement Learning-Based Memetic Particle Swarm Optimizer","volume":"43","author":"Samma","year":"2016","journal-title":"Appl. Soft Comput. J."},{"key":"2021060300032749700_CIT0053","first-page":"104","article-title":"A Multi-Agent Metaheuristic Optimization Framework With Cooperation","author":"Silva","year":"2015"},{"issue":"2\u20133","key":"2021060300032749700_CIT0054","doi-asserted-by":"crossref","first-page":"169","DOI":"10.1016\/S0921-8890(00)00087-7","article-title":"Dynamic Job-Shop Scheduling Using Reinforcement Learning Agents","volume":"33","author":"Aydin","year":"2000","journal-title":"Rob. Autom. Syst."},{"issue":"1","key":"2021060300032749700_CIT0055","doi-asserted-by":"crossref","first-page":"73","DOI":"10.1016\/j.engappai.2004.08.018","article-title":"Application of Reinforcement Learning for Agent-Based Production Scheduling","volume":"18","author":"Wang","year":"2005","journal-title":"Eng. Appl. Artif. Intell."},{"key":"2021060300032749700_CIT0056","doi-asserted-by":"crossref","first-page":"148","DOI":"10.1016\/j.eswa.2019.04.056","article-title":"A Reinforcement Learning-Based Multi-Agent Framework Applied for Solving Routing and Scheduling Problems","volume":"131","author":"Silva","year":"2019","journal-title":"Expert Syst. Appl."},{"key":"2021060300032749700_CIT0057","doi-asserted-by":"crossref","first-page":"75","DOI":"10.1016\/j.cie.2017.05.026","article-title":"A Reinforcement Learning Approach to Parameter Estimation in Dynamic Job Shop Scheduling","volume":"110","author":"Shahrabi","year":"2017","journal-title":"Comput. Ind. Eng."},{"key":"2021060300032749700_CIT0058","first-page":"9839","article-title":"Reinforcement Learning for Solving the Vehicle Routing Problem","author":"Nazari","year":"2018"},{"key":"2021060300032749700_CIT0059","first-page":"16","article-title":"Applying Reinforcement Learning to Plan Manufacturing Material Handling Part 2: Experimentation and Results","author":"Govindaiah","year":"2019"},{"key":"2021060300032749700_CIT0060","first-page":"168","article-title":"Applying Reinforcement Learning to Plan Manufacturing Material Handling Part 1: Background and Formal Problem Specification","author":"Govindaiah","year":"2019"},{"key":"2021060300032749700_CIT0061","article-title":"Proximal Policy Optimization Algorithms","author":"Schulman","year":"2017"},{"key":"2021060300032749700_CIT0062","first-page":"1889","article-title":"Trust Region Policy Optimization","author":"Schulman","year":"2015"},{"key":"2021060300032749700_CIT0063","first-page":"1","article-title":"An Introduction to CLICK: Leveraging Virtual Reality to Integrate the Industrial Engineering Curriculum","author":"Lopez","year":"2019"},{"key":"2021060300032749700_CIT0064","doi-asserted-by":"crossref","DOI":"10.18260\/1-2--30895","article-title":"Product-Based Learning: Bundling Goods and Services for an Integrated Context-Rich Industrial Engineering Curriculum","author":"Terpenny","year":"2018"},{"key":"2021060300032749700_CIT0065","article-title":"Unity_Game_Engine","author":"","year":"2011"},{"key":"2021060300032749700_CIT0066","first-page":"27","article-title":"An Engine Selection Methodology for High Fidelity Serious Games","author":"Petridis","year":"2010"},{"key":"2021060300032749700_CIT0067","first-page":"454","article-title":"Making Learning Fun: Educational Concepts & Logics Through Game","author":"Alsubaie","year":"2018"},{"key":"2021060300032749700_CIT0068","doi-asserted-by":"crossref","DOI":"10.1115\/DETC2018-86299","article-title":"A Validation Neural Network (VNN) Metamodel for Predicting the Performance of Deep Generative Designs","author":"Cunningham","year":"2018"},{"key":"2021060300032749700_CIT0069","article-title":"Large-Scale Study of Curiosity-Driven Learning","author":"Burda","year":"2018"}],"container-title":["Journal of Computing and Information Science in Engineering"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/asmedigitalcollection.asme.org\/computingengineering\/article-pdf\/20\/5\/051005\/6647470\/jcise_20_5_051005.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/asmedigitalcollection.asme.org\/computingengineering\/article-pdf\/20\/5\/051005\/6647470\/jcise_20_5_051005.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,10,15]],"date-time":"2022-10-15T07:13:09Z","timestamp":1665817989000},"score":1,"resource":{"primary":{"URL":"https:\/\/asmedigitalcollection.asme.org\/computingengineering\/article\/20\/5\/051005\/1074423\/Deep-Reinforcement-Learning-for-Procedural-Content"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,6,3]]},"references-count":69,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2020,10,1]]}},"URL":"https:\/\/doi.org\/10.1115\/1.4046293","relation":{},"ISSN":["1530-9827","1944-7078"],"issn-type":[{"value":"1530-9827","type":"print"},{"value":"1944-7078","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,6,3]]},"article-number":"051005"}}