{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,30]],"date-time":"2026-10-30T16:41:21Z","timestamp":1793378481343,"version":"4.5.0"},"reference-count":44,"publisher":"Springer Science and Business Media LLC","issue":"11","license":[{"start":{"date-parts":[[2026,9,21]],"date-time":"2026-09-21T00:00:00Z","timestamp":1789948800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,9,21]],"date-time":"2026-09-21T00:00:00Z","timestamp":1789948800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"NSF AI Institute for Societal Decision Making","award":["2229881"],"award-info":[{"award-number":["2229881"]}]},{"DOI":"10.13039\/100000185","name":"Defense Advanced Research Projects Agency","doi-asserted-by":"publisher","award":["W911NF-20-1-0006"],"award-info":[{"award-number":["W911NF-20-1-0006"]}],"id":[{"id":"10.13039\/100000185","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Behav Res"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    The increasing integration of artificial intelligence (AI) in everyday life brings with it new challenges and questions regarding how humans interact with autonomous agents. Multi-agent experiments, where humans and AI act together, can offer important opportunities to study social decision-making, but there is a lack of accessible tooling available to researchers to run such experiments. We introduce two tools designed to reduce these barriers. The first,\n                    <jats:sc>CoGrid<\/jats:sc>\n                    , is a multi-agent grid-based simulation library with dual NumPy and JAX backends. The second, Multi-User Gymnasium (\n                    <jats:sc>MUG<\/jats:sc>\n                    ), translates such simulation environments directly into interactive web-based experiments.\n                    <jats:sc>MUG<\/jats:sc>\n                    supports interactions with arbitrary numbers of humans and AI, utilizing either server-authoritative or peer-to-peer networking with rollback netcode to account for latency. Together, these tools can enable researchers to deploy studies of human\u2013AI interaction, facilitating inquiry into core questions of psychology, cognition, and decision-making, and their relationship to human\u2013AI interaction. Both tools are open source and available to the broader research community. Documentation and source code is available at {}. This paper details the functionality of these tools and presents several case studies to illustrate their utility in human\u2013AI multi-agent experimentation.\n                  <\/jats:p>","DOI":"10.3758\/s13428-026-03078-x","type":"journal-article","created":{"date-parts":[[2026,9,21]],"date-time":"2026-09-21T17:46:15Z","timestamp":1790012775000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["CoGrid &amp; the Multi-User Gymnasium: A framework for multi-agent experimentation"],"prefix":"10.3758","volume":"58","author":[{"given":"Chase","family":"McDonald","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cleotilde","family":"Gonzalez","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,9,21]]},"reference":[{"key":"3078_CR1","unstructured":"Agapiou, J. P., Vezhnevets, A. S., Du\u00e9\u00f1ez-Guzm\u00e1n, E. A., Matyas, J., Mao, Y., Sunehag, P.,... Comanescu, R., et al. (2022). Melting pot 2.0. arXiv preprint arXiv:2211.13746."},{"key":"3078_CR2","unstructured":"Ayd\u0131n, H., Godin-Dubois, K., Braz, L. G., Hengst, F. d., Baraka, K., \u00c7elikok, M. M.,... Oliehoek, F. A. (2025). Sharpie: A modular framework for reinforcement learning and human-ai interaction experiments. arXiv preprint arXiv:2501.19245."},{"key":"3078_CR3","doi-asserted-by":"publisher","first-page":"1696","DOI":"10.3758\/s13428-016-0824-z","volume":"49","author":"S Balietti","year":"2017","unstructured":"Balietti, S. (2017). Nodegame: Real-time, synchronous, online experiments in the browser. Behavior Research Methods, 49, 1696\u20131715.","journal-title":"Behavior Research Methods"},{"key":"3078_CR4","unstructured":"Bamford, C., Huang, S., & Lucas, S. (2022). Griddly: A platform for AI research in games. Number: arXiv:2011.06363arXiv:2011.06363 [cs]."},{"key":"3078_CR5","volume-title":"Hippo gym: Facilitating human-in-the-loop reinforcement learning experiments","author":"T Bewley","year":"2021","unstructured":"Bewley, T., Lawry, J., & Richards, A. (2021). Hippo gym: Facilitating human-in-the-loop reinforcement learning experiments. AAAI: Workshop on Reinforcement Learning Tools and Environments."},{"key":"3078_CR6","unstructured":"Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D.,... Zhang, Q. (2018). JAX: Composable transformations of Python+NumPy programs (Version 0.3.13). Retrieved from http:\/\/github.com\/jax-ml\/jax"},{"key":"3078_CR7","unstructured":"Cannon, T. (2019). GGPO: Good game, peace out rollback network sdk. https:\/\/github.com\/pond3r\/ggpo."},{"key":"3078_CR8","unstructured":"Carroll, M., Shah, R., Ho, M. K., Griffiths, T. L., Seshia, S. A., Abbeel, P., & Dragan, A. (2020). On the Utility of Learning about Humans for Human-AI Coordination. arXiv:1910.05789 [cs, stat]. arXiv: 1910.05789. Retrieved July 6, 2021"},{"key":"3078_CR9","doi-asserted-by":"crossref","unstructured":"Chevalier-Boisvert, M., Dai, B., Towers, M., de Lazcano, R., Willems, L., Lahlou, S.,... Terry, J. (2023). Minigrid & Miniworld: Modular & customizable reinforcement learning environments for goaloriented tasks. Number: arXiv:2306.13831 [cs].","DOI":"10.52202\/075280-3209"},{"issue":"CSCW2","key":"3078_CR10","first-page":"1","volume":"8","author":"W Duan","year":"2024","unstructured":"Duan, W., Zhou, S., Scalia, M. J., Yin, X., Weng, N., Zhang, R., & Tolston, M. (2024). Understanding the evolvement of trust over time within human-ai teams. Proceedings of the ACM on Human-Computer Interaction, 8(CSCW2), 1\u201331.","journal-title":"Proceedings of the ACM on Human-Computer Interaction"},{"key":"3078_CR11","doi-asserted-by":"crossref","unstructured":"Ellis, B., Cook, J., Moalla, S., Samvelyan, M., Sun, M., Mahajan, A.,... Whiteson, S. (2024). Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 36.","DOI":"10.52202\/075280-1634"},{"key":"3078_CR12","unstructured":"Fickinger, A. (2020). Multi-agent gridworld environment for openai gym. https:\/\/github.com\/ArnaudFickinger\/gym-multigrid. GitHub."},{"key":"3078_CR13","unstructured":"Gerstgrasser, M., Trivedi, R., & Parkes, D. C. (2021). CrowdPlay: Crowdsourcing human demonstration data for offline learning in Atari games. Retrieved February 20, 2022, from https:\/\/openreview.net\/forum?id=qyTBxTztIpQ"},{"key":"3078_CR14","unstructured":"Glazer, J., & Madhav, S. (2015). Multiplayer game programming: Architecting networked games. Addison-Wesley Professional."},{"key":"3078_CR15","unstructured":"Ha, D. (2020). Slime volleyball gym environment. https:\/\/github.com\/hardmaru\/slimevolleygym.GitHub."},{"key":"3078_CR16","unstructured":"Hafner, D. (2021). Benchmarking the spectrum of agent capabilities. arXiv preprint arXiv:2109.06780."},{"issue":"7825","key":"3078_CR17","doi-asserted-by":"publisher","first-page":"357","DOI":"10.1038\/s41586-020-2649-2","volume":"585","author":"CR Harris","year":"2020","unstructured":"Harris, C. R., Millman, K. J., van der Walt, S. J., Gommers, R., Virtanen, P., Cournapeau, D., & Oliphant, T. E. (2020). Array programming with NumPy. Nature, 585(7825), 357\u2013362. https:\/\/doi.org\/10.1038\/s41586-020-2649-2","journal-title":"Nature"},{"key":"3078_CR18","unstructured":"Hughes, E., Leibo, J. Z., Phillips, M., Tuyls, K., Due\u00f1ez-Guzman, E., Garc\u00eda Casta\u00f1eda, A.,... Koster, R., et al. (2018). Inequity aversion improves cooperation in intertemporal social dilemmas. Advances in Neural Information Processing Systems, 31."},{"key":"3078_CR19","unstructured":"Jaques, N., Lazaridou, A., Hughes, E., Gulcehre, C., Ortega, P., Strouse, D.,... De Freitas, N. (2019). Social influence as intrinsic motivation for multi-agent deep reinforcement learning. International Conference on Machine Learning, 3040\u20133049."},{"key":"3078_CR20","doi-asserted-by":"crossref","unstructured":"Leibo, J. Z., Zambaldi, V., Lanctot, M., Marecki, J., & Graepel, T. (2017). Multi-agent reinforcement learning in sequential social dilemmas. arXiv preprint arXiv:1702.03037.","DOI":"10.65109\/QMSU8421"},{"key":"3078_CR21","unstructured":"Liang, E., Liaw, R., Nishihara, R., Moritz, P., Fox, R., Goldberg, K.,... Stoica, I. (2018). Rllib: Abstractions for distributed reinforcement learning. International Conference on Machine Learning, 3053\u20133062."},{"key":"3078_CR22","doi-asserted-by":"crossref","unstructured":"Meta Fundamental AI Research Diplomacy Team (FAIR), Bakhtin, A., Brown, N., Dinan, E., Farina, G., Flaherty, C., Hu, H., et al. (2022). Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science, 378(6624), 1067\u20131074.","DOI":"10.1126\/science.ade9097"},{"key":"3078_CR23","unstructured":"Ndousse, K. (2020). Marlgrid. https:\/\/github.com\/kandouss\/marlgrid. GitHub."},{"key":"3078_CR24","doi-asserted-by":"crossref","unstructured":"Nguyen, T. N., & Gonzalez, C. (2023). Minimap: An interactive dynamic decision making game for search and rescue missions. Behavior Research Methods, 1\u201322.","DOI":"10.2139\/ssrn.4123762"},{"key":"3078_CR25","unstructured":"Nor\u00e9n, J. F. W. (2020). Derk gym environment. https:\/\/gym.derkgame.com. Mount Rouke."},{"key":"3078_CR26","doi-asserted-by":"publisher","first-page":"27730","DOI":"10.52202\/068431-2011","volume":"35","author":"L Ouyang","year":"2022","unstructured":"Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Ray, A., et al. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35, 27730\u201327744.","journal-title":"Advances in Neural Information Processing Systems"},{"issue":"6623","key":"3078_CR27","doi-asserted-by":"publisher","first-page":"990","DOI":"10.1126\/science.add4679","volume":"378","author":"J Perolat","year":"2022","unstructured":"Perolat, J., De Vylder, B., Hennes, D., Tarassov, E., Strub, F., de Boer, V., Anthony, T., et al. (2022). Mastering the game of stratego with model-free multiagent reinforcement learning. Science, 378(6623), 990\u2013996.","journal-title":"Science"},{"key":"3078_CR28","doi-asserted-by":"crossref","unstructured":"Pignatelli, E., Liesen, J., Lange, R. T., Lu, C., Castro, P. S., & Toni, L. (2024). Navix: Scaling minigrid environments with jax. arXiv preprint arXiv:2407.19396.","DOI":"10.52202\/085713-4437"},{"key":"3078_CR29","unstructured":"Russell, S. (2019). Human compatible: Ai and the problem of control. Penguin UK."},{"key":"3078_CR30","doi-asserted-by":"crossref","unstructured":"Rutherford, A., Ellis, B., Gallici, M., Cook, J., Lupu, A., Ingvarsson, G.,... Foerster, J. N. (2024). Jaxmarl: Multi-agent rl environments and algorithms in jax. The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track.","DOI":"10.52202\/079017-1612"},{"key":"3078_CR31","doi-asserted-by":"crossref","unstructured":"Salikutluk, V., Sch\u00f6pper, J., Herbert, F., Scheuermann, K., Frodl, E., Balfanz, D.,... Koert, D. (2024). An evaluation of situational autonomy for human-ai collaboration in a shared workspace setting. Proceedings of the CHI Conference on Human Factors in Computing Systems, 1\u201317.","DOI":"10.1145\/3613904.3642564"},{"key":"3078_CR32","unstructured":"Samvelyan, M., Kirk, R., Kurin, V., Parker-Holder, J., Jiang, M., Hambro, E.,... Rockt\u00e4schel, T. (2021). Minihack the planet: A sandbox for open-ended reinforcement learning research. arXiv preprint arXiv:2109.13202."},{"key":"3078_CR33","doi-asserted-by":"crossref","unstructured":"Schelble, B. G., Flathmann, C., McNeese, N. J., Freeman, G., & Mallick, R. (2022). Let\u2019s think together! assessing shared mental models, performance, and trust in human-agent teams. Proceedings of the ACM on Human-Computer Interaction, 6(GROUP), 1\u201329.","DOI":"10.1145\/3492832"},{"key":"3078_CR34","doi-asserted-by":"crossref","unstructured":"Schmutz, J. B., Outland, N., Kerstan, S., Georganta, E., & Ulfert, A.-S. (2024). Ai-teaming: Redefining collaboration in the digital era. Current Opinion in Psychology, 101837.","DOI":"10.1016\/j.copsyc.2024.101837"},{"key":"3078_CR35","unstructured":"Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal policy optimization algorithms. arXiv:1707.06347 [cs]. Retrieved August 24, 2021."},{"key":"3078_CR36","doi-asserted-by":"publisher","unstructured":"Silver, D., Singh, S., Precup, D., & Sutton, R. S. (2021). Reward Is Enough. Artificial Intelligence, 103535. https:\/\/doi.org\/10.1016\/j.artint.2021.103535","DOI":"10.1016\/j.artint.2021.103535"},{"key":"3078_CR37","doi-asserted-by":"publisher","unstructured":"Skrynnik, A., Andreychuk, A., Yakovlev, K., & Panov, A. I. (2022). Pogema: Partially observable grid environment for multiple agents. https:\/\/doi.org\/10.48550\/ARXIV.2206.10944","DOI":"10.48550\/ARXIV.2206.10944"},{"key":"3078_CR38","unstructured":"Strouse, D. J., McKee, K. R., Botvinick, M., Hughes, E., & Everett, R. (2021). Collaborating with humans without human data. arXiv:2110.08176 [cs]. Retrieved November 29, 2021."},{"key":"3078_CR39","unstructured":"Sukhbaatar, S., Szlam, A., Synnaeve, G., Chintala, S., & Fergus, R. (2015). Mazebase: A sandbox for learning from games. arXiv preprint arXiv:1511.07401."},{"key":"3078_CR40","doi-asserted-by":"publisher","unstructured":"Team, T. P. D. (2021). Pyodide\/pyodide (Version 0.29.3). https:\/\/doi.org\/10.5281\/zenodo.5156931","DOI":"10.5281\/zenodo.5156931"},{"key":"3078_CR41","first-page":"15032","volume":"34","author":"J Terry","year":"2021","unstructured":"Terry, J., Black, B., Grammel, N., Jayakumar, M., Hari, A., Sullivan, R., Perez-Vicente, R., et al. (2021). Pettingzoo: Gym for multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34, 15032\u201315043.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"3078_CR42","doi-asserted-by":"publisher","unstructured":"Towers, M., Terry, J. K., Kwiatkowski, A., Balis, J. U., Cola, G., & d., Deleu, T., Younis, O. G. (2023). Gymnasium. https:\/\/doi.org\/10.5281\/zenodo.8127026","DOI":"10.5281\/zenodo.8127026"},{"key":"3078_CR43","doi-asserted-by":"crossref","unstructured":"Wilder, B., Horvitz, E., & Kamar, E. (2020). Learning to complement humans. arXiv preprint arXiv:2005.00582.","DOI":"10.24963\/ijcai.2020\/212"},{"key":"3078_CR44","doi-asserted-by":"crossref","unstructured":"Zhang, R., McNeese, N. J., Freeman, G., & Musick, G. (2021). \u201can ideal human\u201d expectations of ai teammates in human-ai teaming. Proceedings of the ACM on Human-Computer Interaction, 4(CSCW3), 1\u201325.","DOI":"10.1145\/3432945"}],"container-title":["Behavior Research Methods"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.3758\/s13428-026-03078-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.3758\/s13428-026-03078-x","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.3758\/s13428-026-03078-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,30]],"date-time":"2026-10-30T16:14:22Z","timestamp":1793376862000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/article\/10.3758\/s13428-026-03078-x"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,9,21]]},"references-count":44,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2026,11]]}},"alternative-id":["3078"],"URL":"https:\/\/doi.org\/10.3758\/s13428-026-03078-x","relation":{},"ISSN":["1554-3528"],"issn-type":[{"value":"1554-3528","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,9,21]]},"assertion":[{"value":"8 July 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"7 May 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"21 September 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors have no competing interests to declare that are relevant to the content of this article.","order":1,"name":"Ethics","label":"Conflicts of Interest","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Ethics approval was received by the Institutional Review Board at Carnegie Mellon University under the project\n                      Building Human\u2013Machine Shared Mental Models\n                      , study\n                      STUDY2021\n                      _\n                      00000324\n                      .","order":2,"name":"Ethics","label":"Ethics Approval","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Informed consent was obtained from all individual participants.","order":3,"name":"Ethics","label":"Consent to Participate","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The participants gave their consent for the publication of their behavioral data.","order":4,"name":"Ethics","label":"Consent for Publication","group":{"name":"EthicsHeading","label":"Declarations"}}],"article-number":"295"}}