{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,18]],"date-time":"2025-11-18T10:59:41Z","timestamp":1763463581596,"version":"3.45.0"},"reference-count":51,"publisher":"Association for Computing Machinery (ACM)","issue":"1","funder":[{"DOI":"10.13039\/100023827","name":"Under Secretary of Defense for Research and Engineering","doi-asserted-by":"crossref","award":["FA8702-15-D-0001 or FA8702-25-D-B002"],"award-info":[{"award-number":["FA8702-15-D-0001 or FA8702-25-D-B002"]}],"id":[{"id":"10.13039\/100023827","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["J. Hum.-Robot Interact."],"published-print":{"date-parts":[[2026,1,31]]},"abstract":"<jats:p>\n                    In cooperative training, humans within a team coordinate on complex tasks, building mental models of their teammates and learning to adapt to teammates\u2019 actions in real-time. To reduce the often prohibitive scheduling constraints associated with cooperative training, this article introduces a paradigm for\n                    <jats:italic toggle=\"yes\">cooperative asynchronous training of human teams<\/jats:italic>\n                    in which trainees practice coordination with autonomous teammates rather than humans. We introduce a novel experimental design for evaluating autonomous teammates for use as training partners in cooperative training. We apply this design to a human-subjects experiment where humans are trained with either another human or an autonomous teammate and are evaluated with a new human subject in a new, partially observable, cooperative game developed for this study. Importantly, we employ an unsupervised sequential clustering methodology to partition teammate trajectories from demonstrations performed in the experiment to form a smaller number of training conditions. This results in a simpler experiment design, enabling us to conduct a complex cooperative training human-subjects study in a reasonable amount of time. Through a demonstration of the proposed experimental design, we provide takeaways and design recommendations for future research in the development of cooperative asynchronous training systems utilizing robot surrogates for human teammates.\n                  <\/jats:p>","DOI":"10.1145\/3766892","type":"journal-article","created":{"date-parts":[[2025,9,17]],"date-time":"2025-09-17T16:39:02Z","timestamp":1758127142000},"page":"1-18","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Asynchronous Training of Mixed-Role Human Actors in a Partially Observable Environment"],"prefix":"10.1145","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0009-0000-3614-9980","authenticated-orcid":false,"given":"Kimberlee","family":"Chestnut Chang","sequence":"first","affiliation":[{"name":"MIT Lincoln Laboratory, Lexington, Massachusetts, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-5791-2639","authenticated-orcid":false,"given":"Reed","family":"Jensen","sequence":"additional","affiliation":[{"name":"MIT Lincoln Laboratory, Lexington, Massachusetts, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0773-8054","authenticated-orcid":false,"given":"Rohan","family":"Paleja","sequence":"additional","affiliation":[{"name":"MIT Lincoln Laboratory, Lexington, Massachusetts, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9482-0258","authenticated-orcid":false,"given":"Sam L.","family":"Polk","sequence":"additional","affiliation":[{"name":"MIT Lincoln Laboratory, Lexington, Massachusetts, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1193-3821","authenticated-orcid":false,"given":"Rob","family":"Seater","sequence":"additional","affiliation":[{"name":"MIT Lincoln Laboratory, Lexington, Massachusetts, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6646-9667","authenticated-orcid":false,"given":"Jackson","family":"Steilberg","sequence":"additional","affiliation":[{"name":"MIT Lincoln Laboratory, Lexington, Massachusetts, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9334-1370","authenticated-orcid":false,"given":"Curran","family":"Schiefelbein","sequence":"additional","affiliation":[{"name":"MIT Lincoln Laboratory, Lexington, Massachusetts, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7053-9448","authenticated-orcid":false,"given":"Melissa","family":"Scheldrup","sequence":"additional","affiliation":[{"name":"MetaTeq Inc., Eugene, Oregon, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5321-6038","authenticated-orcid":false,"given":"Matthew","family":"Gombolay","sequence":"additional","affiliation":[{"name":"Georgia Institute of Technology, Atlanta, Georgia, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-5309-5473","authenticated-orcid":false,"given":"Mabel D.","family":"Ramirez","sequence":"additional","affiliation":[{"name":"MIT Lincoln Laboratory, Lexington, Massachusetts, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2025,11,18]]},"reference":[{"key":"e_1_3_4_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/1015330.1015430"},{"key":"e_1_3_4_3_2","doi-asserted-by":"publisher","DOI":"10.31446\/JCP.2021.1.12"},{"key":"e_1_3_4_4_2","doi-asserted-by":"publisher","DOI":"10.3390\/electronics10233001"},{"key":"e_1_3_4_5_2","doi-asserted-by":"publisher","DOI":"10.1080\/1463922X.2022.2061080"},{"key":"e_1_3_4_6_2","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511611483"},{"key":"e_1_3_4_7_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-89738-3_24-1"},{"issue":"2","key":"e_1_3_4_8_2","first-page":"32","article-title":"The Likert scale: What it is and how to use it","volume":"50","author":"Batterton Katherine A.","year":"2017","unstructured":"Katherine A. Batterton and Kimberly N. Hale. 2017. The Likert scale: What it is and how to use it. Phalanx 50, 2 (2017), 32\u201339.","journal-title":"Phalanx"},{"key":"e_1_3_4_9_2","first-page":"213","volume-title":"Human Factors and Ergonomics Society Annual Meeting","author":"Bolstad Cheryl A.","year":"1999","unstructured":"Cheryl A. Bolstad and Mica R. Endsley. 1999. Shared mental models and shared displays: An empirical evaluation of team performance. In Human Factors and Ergonomics Society Annual Meeting. SAGE Publications, Santa Monica, CA, 213\u2013217."},{"key":"e_1_3_4_10_2","doi-asserted-by":"publisher","DOI":"10.1109\/TMMS.1970.299942"},{"key":"e_1_3_4_11_2","first-page":"5174","article-title":"On the utility of learning about humans for human-AI coordination","volume":"32","author":"Carroll Micah","year":"2019","unstructured":"Micah Carroll, Rohin Shah, Mark K. Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. 2019. On the utility of learning about humans for human-AI coordination. In Advances in Neural Information Processing Systems, Vol. 32, 5174\u20135185.","journal-title":"Advances in Neural Information Processing Systems"},{"issue":"27","key":"e_1_3_4_12_2","article-title":"The impact of simulation-based training in medical education: A review","volume":"103","author":"Elendu Chukwuka","year":"2024","unstructured":"Chukwuka Elendu, Dependable C. Amaechi, Alexander U. Okatta, Emmanuel C. Amaechi, Tochi C. Elendu, Chiamaka P. Ezeh, and Ijeoma D. Elendu. 2024. The impact of simulation-based training in medical education: A review. Medicine 103, 27 (2024), 14 pages.","journal-title":"Medicine"},{"key":"e_1_3_4_13_2","first-page":"61","volume-title":"Human Factors and Ergonomics Society Annual Meeting","author":"Endsley Mica R.","year":"2000","unstructured":"Mica R. Endsley. 2000. Situation models: An avenue to the modeling of mental models. In Human Factors and Ergonomics Society Annual Meeting. SAGE Publications, San Diego, CA, 61\u201364."},{"key":"e_1_3_4_14_2","first-page":"4650","article-title":"Unsupervised scalable representation learning for multivariate time series","volume":"32","author":"Franceschi Jean-Yves","year":"2019","unstructured":"Jean-Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. 2019. Unsupervised scalable representation learning for multivariate time series. In Advances in Neural Information Processing Systems, Vol. 32, 4650\u20134661.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_4_15_2","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511815867"},{"issue":"2","key":"e_1_3_4_16_2","doi-asserted-by":"crossref","first-page":"1458","DOI":"10.1007\/s00464-022-09386-5","article-title":"Remote and asynchronous training network: From a SAGES grant to an eight-country remote laparoscopic simulation training program","volume":"37","author":"Gaete Mar\u00eda In\u00e9s","year":"2023","unstructured":"Mar\u00eda In\u00e9s Gaete, Francisca Belmar, Mat\u00edas Cort\u00e9s, Adnan Alseidi, Domenech Asbun, Valentina Dur\u00e1n, Gabriel Escalona, Pablo Achurra, Ignacio Villagr\u00e1n, Fernando Crovari, et al. 2023. Remote and asynchronous training network: From a SAGES grant to an eight-country remote laparoscopic simulation training program. Surgical Endoscopy 37, 2 (2023), 1458\u20131465.","journal-title":"Surgical Endoscopy"},{"key":"e_1_3_4_17_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10864-019-09332-x"},{"key":"e_1_3_4_18_2","doi-asserted-by":"crossref","unstructured":"Ashok K. Goel and Lalith Polepeddi. 2018. Jill Watson: A virtual teaching assistant for online education. In Learning Engineering for Online Education. C. Dede J. Richards and B. Saxberg (Eds.) Routledge New York NY Chapter 7 120\u2013143.","DOI":"10.4324\/9781351186193-7"},{"key":"e_1_3_4_19_2","unstructured":"Edward Gu Ho Chit Siu Melanie Platt Isabelle Hurley Jaime Pe\u00f1a and Rohan Paleja. 2025. Enabling rapid shared human-AI mental model alignment via the after-action review. arXiv:2503.19607. Retrieved from https:\/\/arxiv.org\/abs\/2503.19607"},{"key":"e_1_3_4_20_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-387-84858-7"},{"key":"e_1_3_4_21_2","article-title":"Generative adversarial imitation learning","volume":"29","author":"Ho Jonathan","year":"2016","unstructured":"Jonathan Ho and Stefano Ermon. 2016. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, Vol. 29, 9 pages.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_4_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/THMS.2019.2904558"},{"issue":"1","key":"e_1_3_4_23_2","article-title":"The potential transformation of higher education through computer-based adaptive learning systems","volume":"2016","author":"Johnson David","year":"2016","unstructured":"David Johnson and Dina Samora. 2016. The potential transformation of higher education through computer-based adaptive learning systems. Global Education Journal 2016, 1 (2016), 7 pages.","journal-title":"Global Education Journal"},{"key":"e_1_3_4_24_2","first-page":"282","volume-title":"European Conference on Machine Learning","author":"Kocsis Levente","year":"2006","unstructured":"Levente Kocsis and Csaba Szepesv\u00e1ri. 2006. Bandit-based Monte-Carlo planning. In European Conference on Machine Learning. Springer, Berlin, 282\u2013293."},{"key":"e_1_3_4_25_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00521-022-07665-9"},{"key":"e_1_3_4_26_2","doi-asserted-by":"publisher","DOI":"10.1109\/THMS.2021.3107675"},{"key":"e_1_3_4_27_2","article-title":"InfoGAIL: Interpretable imitation learning from visual demonstrations","volume":"30","author":"Li Yunzhu","year":"2017","unstructured":"Yunzhu Li, Jiaming Song, and Stefano Ermon. 2017. InfoGAIL: Interpretable imitation learning from visual demonstrations. In Advances in Neural Information Processing Systems, Vol. 30, 11 pages.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_4_28_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10758-017-9326-z"},{"issue":"160","key":"e_1_3_4_29_2","first-page":"1","article-title":"Learning by unsupervised nonlinear diffusion","volume":"20","author":"Maggioni Mauro","year":"2019","unstructured":"Mauro Maggioni and James M. Murphy. 2019. Learning by unsupervised nonlinear diffusion. Journal of Machine Learning Research 20, 160 (2019), 1\u201356.","journal-title":"Journal of Machine Learning Research"},{"issue":"4","key":"e_1_3_4_30_2","doi-asserted-by":"crossref","first-page":"604","DOI":"10.1002\/bin.1727","article-title":"A review of the literature on staff training strategies that minimize trainer involvement","volume":"35","author":"Marano Kathleen E.","year":"2020","unstructured":"Kathleen E. Marano, Jason C. Vladescu, Kenneth F. Reeve, Tina M. Sidener, and David J. Cox. 2020. A review of the literature on staff training strategies that minimize trainer involvement. Behavioral Interventions 35, 4 (2020), 604\u2013641.","journal-title":"Behavioral Interventions"},{"key":"e_1_3_4_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2018.2851790"},{"key":"e_1_3_4_32_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.acha.2021.11.004"},{"key":"e_1_3_4_33_2","first-page":"197","volume-title":"International Conference on Human-Robot Interaction","author":"Murphy Robin R.","year":"2013","unstructured":"Robin R. Murphy and Debra Schreckenghost. 2013. Survey of metrics for human-robot interaction. In International Conference on Human-Robot Interaction. IEEE\/ACM, 197\u2013198."},{"key":"e_1_3_4_34_2","unstructured":"Akos Nagy Yannis Spyridis Gregory Mills and Vasileios Argyriou. 2025. MemoryPods: Enhancing asynchronous communication in extended reality. arXiv:2502.15622. Retrieved from https:\/\/arxiv.org\/abs\/2502.15622"},{"key":"e_1_3_4_35_2","first-page":"26","volume-title":"AAAI Workshop on Challenges in Game Artificial Intelligence","author":"Orkin Jeff","year":"2004","unstructured":"Jeff Orkin. 2004. Symbolic representation of game world state: Toward real-time planning in games. In AAAI Workshop on Challenges in Game Artificial Intelligence, Vol. 5. AAAI, San Jose, CA, 26\u201330."},{"key":"e_1_3_4_36_2","first-page":"610","article-title":"The utility of explainable AI in ad hoc human-machine teaming","volume":"34","author":"Paleja Rohan","year":"2021","unstructured":"Rohan Paleja, Muyleng Ghuy, Nadun Ranawaka Arachchige, Reed Jensen, and Matthew Gombolay. 2021. The utility of explainable AI in ad hoc human-machine teaming. In Advances in Neural Information Processing Systems, Vol. 34, 610\u2013623.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_4_37_2","first-page":"64942","article-title":"Designs for enabling collaboration in human-machine teaming via interactive and explainable systems","volume":"37","author":"Paleja Rohan","year":"2024","unstructured":"Rohan Paleja, Michael Munje, Kimberlee Chang, Reed Jensen, and Matthew Gombolay. 2024. Designs for enabling collaboration in human-machine teaming via interactive and explainable systems. In Advances in Neural Information Processing Systems, Vol. 37, 64942\u201364969.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_4_38_2","first-page":"6417","article-title":"Interpretable and personalized apprenticeship scheduling: Learning interpretable scheduling policies from heterogeneous user demonstrations","volume":"33","author":"Paleja Rohan","year":"2020","unstructured":"Rohan Paleja, Andrew Silva, Letian Chen, and Matthew Gombolay. 2020. Interpretable and personalized apprenticeship scheduling: Learning interpretable scheduling policies from heterogeneous user demonstrations. In Advances in Neural Information Processing Systems, Vol. 33, 6417\u20136428.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_4_39_2","first-page":"1","volume-title":"IEEE Conference on Games","author":"Pfau Johannes","year":"2024","unstructured":"Johannes Pfau. 2024. The real MVP: Quantifying individual performances in multiplayer online games. In IEEE Conference on Games. IEEE, 1\u20138."},{"key":"e_1_3_4_40_2","first-page":"1","volume-title":"IEEE Conference on Games","author":"Polk Sam L.","year":"2024","unstructured":"Sam L. Polk, Eric J. Pabon-Cancel, Rohan Paleja, Kimberlee Chestnut Chang, Reed Jensen, and Mabel D. Ramirez. 2024. Unsupervised behavior inference from human action sequences. In IEEE Conference on Games. IEEE, 1\u20138."},{"key":"e_1_3_4_41_2","doi-asserted-by":"publisher","DOI":"10.7812\/TPP\/17-055"},{"key":"e_1_3_4_42_2","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2024.3403155"},{"key":"e_1_3_4_43_2","doi-asserted-by":"publisher","DOI":"10.1016\/0377-0427(87)90125-7"},{"issue":"1","key":"e_1_3_4_44_2","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1177\/0162643416633334","article-title":"Teaching discrete trial training: The effects of asynchronous computer-based instruction on live implementation","volume":"31","author":"Serna Richard W.","year":"2016","unstructured":"Richard W. Serna, Michelle M. Foran, Caroline Cooke, Kaitlyn E. Hurd, Ashley J. Tello, Rishi Vangapalli, and Charles D. Hamad. 2016. Teaching discrete trial training: The effects of asynchronous computer-based instruction on live implementation. Journal of Special Education Technology 31, 1 (2016), 39\u201349.","journal-title":"Journal of Special Education Technology"},{"key":"e_1_3_4_45_2","first-page":"16183","article-title":"Evaluation of human-AI teams for learned and rule-based agents in Hanabi","volume":"34","author":"Siu Ho Chit","year":"2021","unstructured":"Ho Chit Siu, Jaime Pe\u00f1a, Edenna Chen, Yutai Zhou, Victor Lopez, Kyle Palko, Kimberlee Chang, and Ross Allen. 2021. Evaluation of human-AI teams for learned and rule-based agents in Hanabi. In Advances in Neural Information Processing Systems, Vol. 34, 16183\u201316195.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_4_46_2","doi-asserted-by":"publisher","DOI":"10.1177\/15485129211028317"},{"key":"e_1_3_4_47_2","first-page":"2607","volume-title":"IEEE International Conference on Intelligent Transportation Systems","author":"V\u00f6lz B.","year":"2016","unstructured":"B. V\u00f6lz, K. Behrendt, H. Mielenz, I. Gilitschenski, R. Siegwart, and J. Nieto. 2016. A data-driven approach for pedestrian intention estimation. In IEEE International Conference on Intelligent Transportation Systems. IEEE, 2607\u20132612."},{"key":"e_1_3_4_48_2","doi-asserted-by":"publisher","DOI":"10.3390\/app9235152"},{"key":"e_1_3_4_49_2","doi-asserted-by":"publisher","DOI":"10.1155\/2019\/3679174"},{"issue":"3","key":"e_1_3_4_50_2","first-page":"1","article-title":"\u201cAn ideal human\u201d: Expectations of AI teammates in human-AI teaming","volume":"4","author":"Zhang Rui","year":"2021","unstructured":"Rui Zhang, Nathan J. McNeese, Guo Freeman, and Geoff Musick. 2021. \u201cAn ideal human\u201d: Expectations of AI teammates in human-AI teaming. Human-Computer Interaction 4, CSCW3 (2021), 1\u201325.","journal-title":"Human-Computer Interaction"},{"key":"e_1_3_4_51_2","unstructured":"Weixiang Zhao Xingyu Sui Yulin Hu Jiahe Guo Haixiao Liu Biye Li Yanyan Zhao Bing Qin and Ting Liu. 2025. Teaching language models to evolve with users: Dynamic profile modeling for personalized alignment. arXiv:2505.15456. Retrieved from https:\/\/arxiv.org\/abs\/2505.15456"},{"key":"e_1_3_4_52_2","doi-asserted-by":"publisher","DOI":"10.1109\/TCIAIG.2014.2342934"}],"container-title":["ACM Transactions on Human-Robot Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3766892","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,11,18]],"date-time":"2025-11-18T10:57:43Z","timestamp":1763463463000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3766892"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,11,18]]},"references-count":51,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2026,1,31]]}},"alternative-id":["10.1145\/3766892"],"URL":"https:\/\/doi.org\/10.1145\/3766892","relation":{},"ISSN":["2573-9522"],"issn-type":[{"type":"electronic","value":"2573-9522"}],"subject":[],"published":{"date-parts":[[2025,11,18]]},"assertion":[{"value":"2024-12-23","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-07-25","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-11-18","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}