{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,14]],"date-time":"2025-11-14T10:03:31Z","timestamp":1763114611208,"version":"3.41.0"},"reference-count":48,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2023,7,26]],"date-time":"2023-07-26T00:00:00Z","timestamp":1690329600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Graph."],"published-print":{"date-parts":[[2023,8]]},"abstract":"<jats:p>Training AI agents to perform complex tasks in simulated worlds requires millions to billions of steps of experience. To achieve high performance, today's fastest simulators for training AI agents adopt the idea of batch simulation: using a single simulation engine to simultaneously step many environments in parallel. We introduce a framework for productively authoring novel training environments (including custom logic for environment generation, environment time stepping, and generating agent observations and rewards) that execute as high-performance, GPU-accelerated batched simulators. Our key observation is that the entity-component-system (ECS) design pattern, popular for expressing CPU-side game logic today, is also well-suited for providing the structure needed for high-performance batched simulators. We contribute the first fully-GPU accelerated ECS implementation that natively supports batch environment simulation. We demonstrate how ECS abstractions impose structure on a training environment's logic and state that allows the system to efficiently manage state, amortize work, and identify GPU-friendly coherent parallel computations within and across different environments. We implement several learning environments in this framework, and demonstrate GPU speedups of two to three orders of magnitude over open source CPU baselines and 5-33\u00d7 over strong baselines running on a 32-thread CPU. An implementation of the OpenAI hide and seek 3D environment written in our framework, which performs rigid body physics and ray tracing in each simulator step, achieves over 1.9 million environment steps per second on a single GPU.<\/jats:p>","DOI":"10.1145\/3592427","type":"journal-article","created":{"date-parts":[[2023,7,26]],"date-time":"2023-07-26T14:29:21Z","timestamp":1690381761000},"page":"1-13","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":7,"title":["An Extensible, Data-Oriented Architecture for High-Performance, Many-World Simulation"],"prefix":"10.1145","volume":"42","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2894-1158","authenticated-orcid":false,"given":"Brennan","family":"Shacklett","sequence":"first","affiliation":[{"name":"Stanford University, Stanford, CA, United States of America"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-5818-8400","authenticated-orcid":false,"given":"Luc Guy","family":"Rosenzweig","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, CA, United States of America"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0214-6439","authenticated-orcid":false,"given":"Zhiqiang","family":"Xie","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, CA, United States of America"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0584-3504","authenticated-orcid":false,"given":"Bidipta","family":"Sarkar","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, CA, United States of America"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-5419-0302","authenticated-orcid":false,"given":"Andrew","family":"Szot","sequence":"additional","affiliation":[{"name":"Georgia Institute of Technology, Atlanta, GA, United States of America"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4254-3751","authenticated-orcid":false,"given":"Erik","family":"Wijmans","sequence":"additional","affiliation":[{"name":"Georgia Institute of Technology, Atlanta, GA, United States of America"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0858-0970","authenticated-orcid":false,"given":"Vladlen","family":"Koltun","sequence":"additional","affiliation":[{"name":"Vladlen Koltun, Jackson, WY, United States of America"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1358-0011","authenticated-orcid":false,"given":"Dhruv","family":"Batra","sequence":"additional","affiliation":[{"name":"Georgia Institute of Technology, Atlanta, GA, United States of America"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8754-0429","authenticated-orcid":false,"given":"Kayvon","family":"Fatahalian","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, CA, United States of America"}]}],"member":"320","published-online":{"date-parts":[[2023,7,26]]},"reference":[{"key":"e_1_2_2_1_1","volume-title":"Game Developers Conference.","author":"Acton Mike","year":"2019","unstructured":"Mike Acton. 2019. Connecting the DOTS: The Unity Data-Oriented Technology Stack. https:\/\/www.gdcvault.com\/play\/1026171\/Connecting-the-DOTS-The-Unity. Game Developers Conference."},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/1572769.1572792"},{"key":"e_1_2_2_3_1","volume-title":"Emergent Tool Use From Multi-Agent Autocurricula. In 8th International Conference on Learning Representations.","author":"Baker Bowen","year":"2020","unstructured":"Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. 2020. Emergent Tool Use From Multi-Agent Autocurricula. In 8th International Conference on Learning Representations."},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2019.103216"},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSMC.1983.6313077"},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/CoG47356.2020.9231552"},{"key":"e_1_2_2_7_1","volume-title":"Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang.","author":"Bradbury James","year":"2018","unstructured":"James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. 2018. JAX: composable transformations of Python+NumPy programs. http:\/\/github.com\/google\/jax"},{"key":"e_1_2_2_8_1","unstructured":"Greg Brockman Vicki Cheung Ludwig Pettersson Jonas Schneider John Schulman Jie Tang and Wojciech Zaremba. 2016. OpenAI Gym. arXiv:1606.01540"},{"key":"e_1_2_2_9_1","unstructured":"Michele Caini. 2019. ECS Back and Forth (Part 1). https:\/\/skypjack.github.io\/2019-02-14-ecs-baf-part-1\/."},{"key":"e_1_2_2_10_1","volume-title":"Advances in Neural Information Processing Systems","volume":"32","author":"Carroll Micah","year":"2019","unstructured":"Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. 2019. On the Utility of Learning about Humans for Human-AI Coordination. In Advances in Neural Information Processing Systems, Vol. 32."},{"key":"e_1_2_2_11_1","unstructured":"Erwin Coumans and Yunfei Bai. 2016--2021. PyBullet a Python module for physics simulation for games robotics and machine learning. http:\/\/pybullet.org."},{"key":"e_1_2_2_12_1","volume-title":"Advances in Neural Information Processing Systems","volume":"33","author":"Dalton Steven","year":"2020","unstructured":"Steven Dalton, Iuri Frosio, and Michael Garland. 2020. Accelerating Reinforcement Learning through GPU Atari Emulation. In Advances in Neural Information Processing Systems, Vol. 33."},{"key":"e_1_2_2_13_1","volume-title":"Proceedings of the 1st Annual Conference on Robot Learning (Proceedings of Machine Learning Research","volume":"16","author":"Dosovitskiy Alexey","year":"2017","unstructured":"Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. 2017. CARLA: An Open Urban Driving Simulator. In Proceedings of the 1st Annual Conference on Robot Learning (Proceedings of Machine Learning Research, Vol. 78). 1--16."},{"key":"e_1_2_2_14_1","unstructured":"Epic Games. 2022. Unreal Engine 5. https:\/\/www.unrealengine.com"},{"key":"e_1_2_2_15_1","volume-title":"SEED RL: Scalable and Efficient Deep-RL with Accelerated Central Inference. In 8th International Conference on Learning Representations.","author":"Espeholt Lasse","year":"2020","unstructured":"Lasse Espeholt, Rapha\u00ebl Marinier, Piotr Stanczyk, Ke Wang, and Marcin Michalski. 2020. SEED RL: Scalable and Efficient Deep-RL with Accelerated Central Inference. In 8th International Conference on Learning Representations."},{"key":"e_1_2_2_16_1","volume-title":"Proceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research","volume":"1416","author":"Espeholt Lasse","year":"2018","unstructured":"Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, and Koray Kavukcuoglu. 2018. IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures. In Proceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 80). 1407--1416."},{"key":"e_1_2_2_17_1","volume-title":"Game Developers Conference.","author":"Ford Timothy","year":"2017","unstructured":"Timothy Ford. 2017. Overwatch Gameplay Architecture and Netcode. https:\/\/www.gdcvault.com\/play\/1024001\/-Overwatch-Gameplay-Architecture-and. Game Developers Conference."},{"key":"e_1_2_2_18_1","volume-title":"Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks","volume":"1","author":"Freeman C. Daniel","year":"2021","unstructured":"C. Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. 2021. Brax - A Differentiable Physics Engine for Large Scale Rigid Body Simulation. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, Vol. 1."},{"key":"e_1_2_2_19_1","unstructured":"Matteo Hessel Manuel Kroiss Aidan Clark Iurii Kemaev John Quan Thomas Keck Fabio Viola and Hado van Hasselt. 2021. Podracer architectures for scalable Reinforcement Learning. (2021). arXiv:2104.06272"},{"key":"e_1_2_2_20_1","volume-title":"Distributed Prioritized Experience Replay. In 6th International Conference on Learning Representations.","author":"Horgan Dan","year":"2018","unstructured":"Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado van Hasselt, and David Silver. 2018. Distributed Prioritized Experience Replay. In 6th International Conference on Learning Representations."},{"key":"e_1_2_2_21_1","volume-title":"Game Developers Conference.","author":"Johansson Tim","year":"2018","unstructured":"Tim Johansson. 2018. Unity at GDC - Job System & Entity Component System. https:\/\/www.youtube.com\/watch?v=kwnb9Clh2Is. Game Developers Conference."},{"key":"e_1_2_2_22_1","volume-title":"Unity: A General Platform for Intelligent Agents. arXiv:1809.02627","author":"Juliani Arthur","year":"2018","unstructured":"Arthur Juliani, Vincent-Pierre Berges, Ervin Teng, Andrew Cohen, Jonathan Harper, Chris Elion, Chris Goy, Yuan Gao, Hunter Henry, Marwan Mattar, and Danny Lange. 2018. Unity: A General Platform for Intelligent Agents. arXiv:1809.02627"},{"key":"e_1_2_2_23_1","unstructured":"Eric Kolve Roozbeh Mottaghi Winson Han Eli VanderBilt Luca Weihs Alvaro Herrasti Matt Deitke Kiana Ehsani Daniel Gordon Yuke Zhu Aniruddha Kembhavi Abhinav Gupta and Ali Farhadi. 2017. AI2-THOR: An Interactive 3D Environment for Visual AI. arXiv:1712.05474"},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2833157.2833162"},{"key":"e_1_2_2_25_1","first-page":"1","article-title":"WarpDrive: Fast End-to-End Deep Multi-Agent Reinforcement Learning on a GPU","volume":"23","author":"Lan Tian","year":"2022","unstructured":"Tian Lan, Sunil Srinivasa, Huan Wang, and Stephan Zheng. 2022. WarpDrive: Fast End-to-End Deep Multi-Agent Reinforcement Learning on a GPU. Journal of Machine Learning Research 23, 316 (2022), 1--6.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_2_2_26_1","volume-title":"Proceedings of the 6th Conference on Robot Learning (Proceedings of Machine Learning Research","volume":"80","author":"Li Chengshu","year":"2022","unstructured":"Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Mart\u00edn-Mart\u00edn, Chen Wang, Gabrael Levine, Michael Lingelbach, Jiankai Sun, Mona Anvari, Minjune Hwang, Manasi Sharma, Arman Aydin, Dhruva Bansal, Samuel Hunter, Kyu-Young Kim, Alan Lou, Caleb R Matthews, Ivan Villa-Renteria, Jerry Huayang Tang, Claire Tang, Fei Xia, Silvio Savarese, Hyowon Gweon, Karen Liu, Jiajun Wu, and Li Fei-Fei. 2022. BEHAVIOR-1K: A Benchmark for Embodied AI with 1,000 Everyday Activities and Realistic Simulation. In Proceedings of the 6th Conference on Robot Learning (Proceedings of Machine Learning Research, Vol. 205). 80 -- 93."},{"key":"e_1_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3386569.3392422"},{"key":"e_1_2_2_28_1","volume-title":"NVIDIA GPU Technology Conference.","author":"Macklin Miles","year":"2022","unstructured":"Miles Macklin. 2022. Warp: A High-performance Python Framework for GPU Simulation and Graphics. https:\/\/github.com\/nvidia\/warp. NVIDIA GPU Technology Conference."},{"key":"e_1_2_2_29_1","volume-title":"Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks","volume":"1","author":"Makoviychuk Viktor","year":"2021","unstructured":"Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, and Gavriel State. 2021. Isaac Gym: High Performance GPU Based Physics Simulation For Robot Learning. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, Vol. 1."},{"key":"e_1_2_2_30_1","unstructured":"Sam McCandlish Jared Kaplan Dario Amodei and OpenAI Dota Team. 2018. An empirical model of large-batch training. arXiv:1812.06162"},{"key":"e_1_2_2_31_1","unstructured":"Sander Mertens. 2020. Flecs 2.0 Is Out! https:\/\/ajmmertens.medium.com\/flecs-2-0-is-out-1255a2443fbd."},{"key":"e_1_2_2_32_1","volume-title":"Virtual Texturing in Software and Hardware. In ACM SIGGRAPH 2012 Courses (SIGGRAPH '12)","author":"Obert Juraj","year":"2012","unstructured":"Juraj Obert, J. M. P. van Waveren, and Graham Sellers. 2012. Virtual Texturing in Software and Hardware. In ACM SIGGRAPH 2012 Courses (SIGGRAPH '12). Article 5, 29 pages."},{"key":"e_1_2_2_33_1","unstructured":"OpenAI. 2018. OpenAI Five. https:\/\/blog.openai.com\/openai-five\/."},{"key":"e_1_2_2_34_1","volume-title":"Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang.","author":"Christopher Berner AI","year":"2019","unstructured":"OpenAI, Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal J\u00f3zefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pond\u00e9 de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. 2019. Dota 2 with Large Scale Deep Reinforcement Learning. arXiv:1912.06680"},{"key":"e_1_2_2_35_1","unstructured":"Mario Palermo. 2022. Large Numbers of Entities with Mass in UE5. https:\/\/www.youtube.com\/watch?v=f9q8A-9DvPo."},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/1778765.1778803"},{"key":"e_1_2_2_37_1","volume-title":"Advances in Neural Information Processing Systems","volume":"32","author":"Paszke Adam","year":"2019","unstructured":"Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, Vol. 32."},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3528223.3530110"},{"key":"e_1_2_2_39_1","volume-title":"Proceedings of the 37th International Conference on Machine Learning (Proceedings of Machine Learning Research","volume":"7662","author":"Petrenko Aleksei","year":"2020","unstructured":"Aleksei Petrenko, Zhehui Huang, Tushar Kumar, Gaurav S. Sukhatme, and Vladlen Koltun. 2020. Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learning. In Proceedings of the 37th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 119). 7652--7662."},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00943"},{"key":"e_1_2_2_41_1","volume-title":"Bagdanov","author":"Sestini Alessandro","year":"2022","unstructured":"Alessandro Sestini, Linus Gissl\u00e9n, Joakim Bergdahl, Konrad Tollmar, and Andrew D. Bagdanov. 2022. Automated Gameplay Testing and Validation with Curiosity-Conditioned Proximal Trajectories. IEEE Transactions on Games (2022), 1--14."},{"key":"e_1_2_2_42_1","volume-title":"Large Batch Simulation for Deep Reinforcement Learning. In 9th International Conference on Learning Representations.","author":"Shacklett Brennan","year":"2021","unstructured":"Brennan Shacklett, Erik Wijmans, Aleksei Petrenko, Manolis Savva, Dhruv Batra, Vladlen Koltun, and Kayvon Fatahalian. 2021. Large Batch Simulation for Deep Reinforcement Learning. In 9th International Conference on Learning Representations."},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1109\/IROS51168.2021.9636667"},{"key":"e_1_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/IROS.2012.6386109"},{"key":"e_1_2_2_45_1","doi-asserted-by":"crossref","unstructured":"Oriol Vinyals Igor Babuschkin Wojciech M. Czarnecki Micha\u00ebl Mathieu Andrew Dudzik Junyoung Chung David H. Choi Richard Powell Timo Ewalds Petko Georgiev et al. 2019. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature 575 7782 (2019).","DOI":"10.1038\/s41586-019-1724-z"},{"key":"e_1_2_2_46_1","volume-title":"Advances in Neural Information Processing Systems","volume":"35","author":"Weng Jiayi","year":"2022","unstructured":"Jiayi Weng, Min Lin, Shengyi Huang, Bo Liu, Denys Makoviichuk, Viktor Makoviychuk, Zichen Liu, Yufan Song, Ting Luo, Yukun Jiang, Zhongwen Xu, and Shuicheng YAN. 2022. EnvPool: A Highly Parallel Reinforcement Learning Environment Execution Engine. In Advances in Neural Information Processing Systems, Vol. 35."},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3450626.3459761"},{"key":"e_1_2_2_48_1","volume-title":"Advances in Neural Information Processing Systems","volume":"35","author":"Yu Chao","year":"2022","unstructured":"Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. 2022. The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games. In Advances in Neural Information Processing Systems, Vol. 35."}],"container-title":["ACM Transactions on Graphics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3592427","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3592427","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T17:48:59Z","timestamp":1750182539000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3592427"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,7,26]]},"references-count":48,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2023,8]]}},"alternative-id":["10.1145\/3592427"],"URL":"https:\/\/doi.org\/10.1145\/3592427","relation":{},"ISSN":["0730-0301","1557-7368"],"issn-type":[{"type":"print","value":"0730-0301"},{"type":"electronic","value":"1557-7368"}],"subject":[],"published":{"date-parts":[[2023,7,26]]},"assertion":[{"value":"2023-07-26","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}