{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,16]],"date-time":"2026-05-16T19:32:00Z","timestamp":1778959920453,"version":"3.51.4"},"reference-count":48,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2022,2,12]],"date-time":"2022-02-12T00:00:00Z","timestamp":1644624000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Dutch Ministry of Defense","award":["060.38570"],"award-info":[{"award-number":["060.38570"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Systems"],"abstract":"<jats:p>The rapid progress in artificial intelligence enables technology to more and more become a partner of humans in a team, rather than being a tool. Even more than in human teams, partners of human\u2013agent teams have different strengths and weaknesses, and they must acknowledge and utilize their respective capabilities. Coordinated team collaboration can be accomplished by smartly designing the interactions within human\u2013agent teams. Such designs are called Team Design Patterns (TDPs). We investigated the effects of a specific TDP on proactive task reassignment. This TDP supports team members to dynamically allocate tasks by utilizing their knowledge about the task demands and about the capabilities of team members. In a pilot study, agent\u2013agent teams were used to study the effectiveness of proactive task reassignment. Results showed that this TDP improves a team\u2019s performance, provided that partners have accurate knowledge representations of each member\u2019s skill level. The main study of this paper addresses the effects of task reassignments in a human\u2013agent team. It was hypothesized that when agents provide explanations when issuing and responding to task reassignment requests, this will enhance the quality of the human\u2019s mental model. Results confirmed that participants developed more accurate mental models when agent-partners provide explanations. This did not result in a higher performance of the human\u2013agent team, however. The study contributes to our understanding of designing effective collaboration in human\u2013agent teams.<\/jats:p>","DOI":"10.3390\/systems10010015","type":"journal-article","created":{"date-parts":[[2022,2,13]],"date-time":"2022-02-13T20:35:28Z","timestamp":1644784528000},"page":"15","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":11,"title":["\u201cI\u2019m Afraid I Can\u2019t Do That, Dave\u201d; Getting to Know Your Buddies in a Human\u2013Agent Team"],"prefix":"10.3390","volume":"10","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1518-6314","authenticated-orcid":false,"given":"Maarten P. D.","family":"Schadd","sequence":"first","affiliation":[{"name":"TNO, Oude Waalsdorperweg 63, 2597 AK The Hague, P.O. Box 96864, 2509 JG Den Haag, The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3698-2849","authenticated-orcid":false,"given":"Tjeerd A. J.","family":"Schoonderwoerd","sequence":"additional","affiliation":[{"name":"TNO, Kampweg 55, 3769 DE Soesterberg, P.O. Box 23, 3769 ZG Soesterberg, The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0427-8275","authenticated-orcid":false,"given":"Karel","family":"van den Bosch","sequence":"additional","affiliation":[{"name":"TNO, Kampweg 55, 3769 DE Soesterberg, P.O. Box 23, 3769 ZG Soesterberg, The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Olaf H.","family":"Visker","sequence":"additional","affiliation":[{"name":"TNO, Oude Waalsdorperweg 63, 2597 AK The Hague, P.O. Box 96864, 2509 JG Den Haag, The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tjalling","family":"Haije","sequence":"additional","affiliation":[{"name":"TNO, Kampweg 55, 3769 DE Soesterberg, P.O. Box 23, 3769 ZG Soesterberg, The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kim H. J.","family":"Veltman","sequence":"additional","affiliation":[{"name":"TNO, Kampweg 55, 3769 DE Soesterberg, P.O. Box 23, 3769 ZG Soesterberg, The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,2,12]]},"reference":[{"key":"ref_1","unstructured":"Chakraborti, T., Kambhampati, S., Scheutz, M., and Zhang, Y. (2017). AI Challenges in Human-Robot Cognitive Teaming. arXiv."},{"key":"ref_2","unstructured":"Boella, G., Elkind, E., Savarimuthu, B., Dignum, F., and Purvis, M. (2013). Evaluating the impact of the human-agent teamwork communication model (HAT-CoM) on the development of a shared mental model. International Conference on Principles and Practice of Multi-Agent Systems, Springer. LNCS 8291."},{"key":"ref_3","unstructured":"Castellan, N. (1993). Shared mental models in expert team decision making. Individual and Group Decision Making: Current Issues, Lawrence Erlbaum Associates Publishers."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"273","DOI":"10.1037\/0021-9010.85.2.273","article-title":"The influence of shared mental models on team process and performance","volume":"85","author":"Mathieu","year":"2000","journal-title":"J. Appl. Psychol."},{"key":"ref_5","unstructured":"Zsambok, C., and Klein, G. (1997). How can you turn a team of experts into an expert team? Emerging training strategies. Naturalistic Decision Making, Psychology Press."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"593","DOI":"10.1037\/amp0000334","article-title":"The science of teamwork: Progress, reflections, and the road ahead","volume":"73","author":"Salas","year":"2018","journal-title":"Am. Psychol."},{"key":"ref_7","unstructured":"Bergstein, B. (2021, December 20). AI Isn\u2019t Very Smart Yet. But We Need to Get Moving to Make Sure Automation Works for More People. MIT Technology Review. Available online: https:\/\/www.technologyreview.com\/2017\/12\/15\/146836\/the-great-ai-paradox\/."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Harris, D. (2016). Design patterns for human-cognitive agent teaming. Engineering Psychology and Cognitive Ergonomics, Springer. LNCS 9736.","DOI":"10.1007\/978-3-319-40030-3"},{"key":"ref_9","first-page":"15","article-title":"Developing effective and resilient human-agent teamwork using team design patterns","volume":"34","author":"Neerincx","year":"2018","journal-title":"IEEE Intell. Syst."},{"key":"ref_10","unstructured":"Oka, N., Koda, T., Obaid, M., Nakanishi, H., Mubin, O., and Tanaka, K. (2019). Team Design Patterns. Proceedings of the 7th International Conference on Human-Agent Interaction (HAI\u201919), Kyoto, Japan, 6\u201310 October 2019, Association for Computing Machinery."},{"key":"ref_11","unstructured":"Van Diggelen, J., Jorritsma, W., and van der Vecht, B. (2021). Teaming up with information agents. arXiv."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"102684","DOI":"10.1016\/j.ijhcs.2021.102684","article-title":"Human-Centered XAI: Developing Design Patterns for Explanations of Clinical Decision Support Systems","volume":"154","author":"Schoonderwoerd","year":"2021","journal-title":"Int. J. Hum.-Comput. Stud."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"640647","DOI":"10.3389\/frobt.2021.640647","article-title":"Moral decision making in human-agent teams: Human control and the role of explanations","volume":"8","author":"Verdult","year":"2021","journal-title":"Front. Robot. AI"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Sottilare, R., and Schwarz, J. (2019). Six challenges for human-AI Co-learning. Adaptive Instructional Systems, Springer. LNCS 11597.","DOI":"10.1007\/978-3-030-22341-0"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.artint.2018.07.007","article-title":"Explanation in artificial intelligence: Insights from the social sciences","volume":"267","author":"Miller","year":"2019","journal-title":"Artif. Intell."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Harris, D. (2018). Using Perceptual and Cognitive Explanations for Enhanced Human-Agent Team Performance. Engineering Psychology and Cognitive Ergonomics, Springer. LNCS 10906.","DOI":"10.1007\/978-3-319-91122-9"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1109\/MIS.2004.74","article-title":"Ten Challenges for Making Automation a \u201cTeam Player\u201d in Joint Human-Agent Activity","volume":"19","author":"Klein","year":"2004","journal-title":"IEEE Intell. Syst."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"334","DOI":"10.1037\/amp0000241","article-title":"Foundations of teamwork and collaboration","volume":"73","author":"Driskell","year":"2018","journal-title":"Am. Psychol."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"540","DOI":"10.1518\/001872008X288457","article-title":"On teams, teamwork, and team performance: Discoveries and developments","volume":"50","author":"Salas","year":"2008","journal-title":"Hum. Factors"},{"key":"ref_20","unstructured":"Law, E., and Wortman Vaughan, J. (2019). Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance. Proceedings of the Seventh AAAI Conference on Human Computation and Crowdsourcing, Skamania Lodge, WA, USA, 28\u201330 October 2019, AAAI Press."},{"key":"ref_21","unstructured":"Bernhaupt, R., Mueller, F., McGrenere, J., Cockburn, A., Bj\u00f8rn, P., and Zhao, S. Mental Models of AI Agents in a Cooperative Game Setting. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA, 25\u201330 April 2020."},{"key":"ref_22","unstructured":"Hwang, J.H., Lee, K., and Kwon, D.S. The role of mental model and shared grounds in human-robot interaction. Proceedings of the ROMAN 2005. IEEE International Workshop on Robot and Human Interactive Communication, Nashville, TN, USA, 13\u201315 August 2005."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Hanna, N., and Richards, D. (2018). The impact of multimodal communication on a shared mental model, trust, and commitment in human-intelligent virtual agent teams. Multimodal Technol. Interact., 2.","DOI":"10.3390\/mti2030048"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"340","DOI":"10.1080\/14639221003729193","article-title":"Merging internal and external processes: Examining the mental model convergence process through team communication","volume":"11","author":"Kennedy","year":"2010","journal-title":"Theor. Issues Ergon. Sci."},{"key":"ref_25","unstructured":"Gentner, D., and Stevens, A. (1983). Some observations on mental models. Mental Models, Psychology Press."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"289","DOI":"10.1177\/155534341000400403","article-title":"Implicit learning, tacit knowledge, expertise development, and naturalistic decision making","volume":"4","author":"Patterson","year":"2010","journal-title":"J. Cogn. Eng. Decis. Mak."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"587","DOI":"10.1006\/imms.1993.1028","article-title":"Mental models: Concepts for human-computer interaction research","volume":"38","author":"Staggers","year":"1993","journal-title":"Int. J. Man-Mach. Stud."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Bignold, A., Cruz, F., Taylor, M.E., Brys, T., Dazeley, R., Vamplew, P., and Foale, C. (2021). A conceptual framework for externally-influenced agents: An assisted reinforcement learning review. J. Ambient. Intell. Humaniz. Comput.","DOI":"10.1007\/s12652-021-03489-y"},{"key":"ref_29","unstructured":"De Graaf, M.M., and Malle, B.F. (2017). How People Explain Action (and Autonomous Intelligent Systems Should Too), AAAI Press."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"227","DOI":"10.1146\/annurev.psych.57.102904.190100","article-title":"Explanation and Understanding","volume":"57","author":"Keil","year":"2006","journal-title":"Annu. Rev. Psychol."},{"key":"ref_31","unstructured":"Bignold, A., Cruz, F., Dazeley, R., Vamplew, P., and Foale, C. (2020). Human Engagement Providing Evaluative and Informative Advice for Interactive Reinforcement Learning. arXiv."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"103525","DOI":"10.1016\/j.artint.2021.103525","article-title":"Levels of explainable artificial intelligence for human-aligned conversational explanations","volume":"299","author":"Dazeley","year":"2021","journal-title":"Artif. Intell."},{"key":"ref_33","unstructured":"Johnson-Laird, P.N., and Byrne, R.M. (1991). Deduction, Lawrence Erlbaum Associates, Inc."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"412","DOI":"10.1037\/0033-295X.94.4.412","article-title":"Pretense and representation: The origins of \u201ctheory of mind\u201d","volume":"94","author":"Leslie","year":"1987","journal-title":"Psychol. Rev."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"367","DOI":"10.1109\/TSMCB.2003.811794","article-title":"Human-robot interactions during the robot-assisted urban search and rescue response at the world trade center","volume":"33","author":"Casper","year":"2003","journal-title":"IEEE Trans. Syst. Man Cybern. Part B"},{"key":"ref_36","unstructured":"Behnke, S., Veloso, M., Visser, A., and Xiong, R. (2013). Hector open source modules for autonomous mapping and navigation with rescue robots. Robot Soccer World Cup: Robot World Cup XVII, Springer. LNCS 8371."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"147","DOI":"10.1007\/s10846-013-9822-x","article-title":"Robotic urban search and rescue: A survey from the control perspective","volume":"72","author":"Liu","year":"2013","journal-title":"J. Intell. Robot. Syst."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"De Vos, M., Fornara, N., Pitt, J.V., and Vouros, G. (2011). Shared mental models\u2014A Conceptual Analysis. Coordination, Organizations, Institutions, and Norms in Agent Systems VI, Springer.","DOI":"10.1007\/978-3-642-21268-0"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"301","DOI":"10.1007\/s12564-010-9142-9","article-title":"Effects of role division, interaction, and shared mental model on team performance in project-based learning environment","volume":"12","author":"Jo","year":"2011","journal-title":"Asia Pac. Educ. Rev."},{"key":"ref_40","first-page":"130","article-title":"A review and integration of team composition models: Moving toward a dynamic and temporal framework","volume":"40","author":"Mathieu","year":"2014","journal-title":"J. Manag."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Salas, E. (2015). Team Training Essentials: A Research-Based Guide, Routledge.","DOI":"10.4324\/9781315747644"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"209","DOI":"10.1109\/THMS.2019.2904558","article-title":"Evaluating fluency in human\u2013robot collaboration","volume":"49","author":"Hoffman","year":"2019","journal-title":"IEEE Trans. Hum.-Mach. Syst."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"211","DOI":"10.18869\/acadpub.johe.5.4.211","article-title":"Evaluation of Rating Scale Mental Effort (RSME) effectiveness for mental workload assessment in nurses","volume":"5","author":"Ashnagar","year":"2016","journal-title":"J. Occup. Health Epidemiol."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"277","DOI":"10.1142\/S2705078520500150","article-title":"Unexplainability and Incomprehensibility of AI","volume":"7","author":"Yampolskiy","year":"2020","journal-title":"J. Artif. Intell. Conscious."},{"key":"ref_45","unstructured":"Van den Bosch, K., and Bronkhorst, A. (June, January 30). Human-AI Cooperation to Benefit Military Decision Making. Proceedings of the NATO IST-160 Specialist\u2019 Meeting on Big Data and Artificial Intelligence for Military Decision Making, STO-MP-IST-160, Bordeaux, France."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"1669","DOI":"10.3389\/fpsyg.2021.590290","article-title":"Improving Teamwork Competencies in Human-Machine Teams: Perspectives From Team Science","volume":"12","author":"Stowers","year":"2021","journal-title":"Front. Psychol."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"1503","DOI":"10.1177\/1071181319631433","article-title":"Developing Human-Robot Team Interdependence in a Synthetic Task Environment","volume":"63","author":"Lematta","year":"2019","journal-title":"Proc. Hum. Factors Ergon. Soc. Annu. Meet."},{"key":"ref_48","first-page":"44","article-title":"DARPA\u2019s Explainable Artificial Intelligence (XAI) Program","volume":"40","author":"Gunning","year":"2019","journal-title":"AI Mag."}],"container-title":["Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2079-8954\/10\/1\/15\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:18:06Z","timestamp":1760134686000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2079-8954\/10\/1\/15"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,2,12]]},"references-count":48,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2022,2]]}},"alternative-id":["systems10010015"],"URL":"https:\/\/doi.org\/10.3390\/systems10010015","relation":{},"ISSN":["2079-8954"],"issn-type":[{"value":"2079-8954","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,2,12]]}}}