{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,10]],"date-time":"2026-03-10T11:21:24Z","timestamp":1773141684887,"version":"3.50.1"},"reference-count":37,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T00:00:00Z","timestamp":1750118400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Robot. AI"],"abstract":"<jats:p>This paper focuses on the problem of collaborative task execution by teams comprising of people and multiple heterogeneous robots. In particular, the problem is motivated by the need for the team members to dynamically coordinate their execution, in order to avoid overlapping actions (i.e. multiple team members working on the same part of the task) and to ensure a correct execution of the task. This paper expands on our own prior work on collaborative task execution by single human-robot and single robot-robot teams, by taking an approach inspired by simulation Theory of Mind (ToM) to develop a real-time distributed architecture that enables collaborative execution of tasks with hierarchical representations and multiple types of execution constraints by teams of people and multiple robots with variable heterogeneity. First, the architecture presents a novel approach for concurrent coordination of task execution with both human and robot teammates. Second, a novel pipeline is developed in order to handle automatic grasping of objects with unknown initial locations. Furthermore, the architecture relies on a novel continuous-valued metric which accounts for a robot\u2019s capability to perform tasks during the dynamic, on-line task allocation process. To assess the proposed approach, the architecture is validated with: 1) a heterogeneous team of two humanoid robots and 2) a heterogeneous team of one human and two humanoid robots, performing a household task in different environmental conditions. The results support the proposed approach, as different environmental conditions result in different and continuously changing values for the robots\u2019 task execution abilities. Thus, the proposed architecture enables adaptive, real-time collaborative task execution through dynamic task allocation by a heterogeneous human-robot team, for tasks with hierarchical representations and multiple types of constraints.<\/jats:p>","DOI":"10.3389\/frobt.2025.1533054","type":"journal-article","created":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T08:59:56Z","timestamp":1750150796000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Simulation theory of mind for heterogeneous human-robot teams"],"prefix":"10.3389","volume":"12","author":[{"given":"Monica","family":"Nicolescu","sequence":"first","affiliation":[]},{"given":"Janelle","family":"Blankenburg","sequence":"additional","affiliation":[]},{"given":"Bashira Akter","family":"Anima","sequence":"additional","affiliation":[]},{"given":"Mariya","family":"Zagainova","sequence":"additional","affiliation":[]},{"given":"Pourya","family":"Hoseini","sequence":"additional","affiliation":[]},{"given":"Mircea","family":"Nicolescu","sequence":"additional","affiliation":[]},{"given":"David","family":"Feil-Seifer","sequence":"additional","affiliation":[]}],"member":"1965","published-online":{"date-parts":[[2025,6,17]]},"reference":[{"key":"B1","first-page":"301","article-title":"Collaborative human-robot hierarchical task execution with an activation spreading architecture","author":"Anima","year":"2019"},{"key":"B2","first-page":"283","article-title":"Design of an autonomous and distributed robot system: actress","volume-title":"Iros","author":"Asama","year":"1989"},{"key":"B3","first-page":"585","article-title":"A distributed control architecture for collaborative multi-robot task allocation","author":"Blankenburg","year":"2017"},{"key":"B4","first-page":"2992","article-title":"An architecture for tightly coupled multi-robot cooperation","author":"Chaimowicz","year":"2001"},{"key":"B5","first-page":"1198","article-title":"Simultaneous allocations of multiple tightly-coupled multi-robot tasks to coalitions of heterogeneous robots","author":"Das","year":"2014"},{"key":"B6","doi-asserted-by":"publisher","first-page":"1257","DOI":"10.1109\/JPROC.2006.876939","article-title":"Market-based multirobot coordination: a survey and analysis","volume":"94","author":"Dias","year":"2006","journal-title":"Proc. IEEE"},{"key":"B7","first-page":"697","article-title":"A compact task representation for hierarchical robot control","author":"Fraser","year":"2016"},{"key":"B8","first-page":"522","article-title":"A human-aware method to plan complex cooperative and autonomous tasks using behavior trees","author":"Fusaro","year":"2021"},{"key":"B9","first-page":"1119","article-title":"Joint mind modeling for explanation generation in complex human-robot collaborative tasks","author":"Gao","year":"2020"},{"key":"B10","first-page":"203","article-title":"Murdoch: publish\/subscribe task allocation for heterogeneous agents","author":"Gerkey","year":"2000"},{"key":"B11","doi-asserted-by":"crossref","first-page":"464","DOI":"10.1109\/ROBOT.2002.1013403","article-title":"Pusher-watcher: an approach to fault-tolerant tightly-coupled robot coordination","volume-title":"Proc. 2002 IEEE Int. Conf. Robotics Automation (Cat. No.02CH37292)","author":"Gerkey","year":"2002"},{"key":"B12","doi-asserted-by":"publisher","first-page":"939","DOI":"10.1177\/0278364904045564","article-title":"A formal analysis and taxonomy of task allocation in multi-robot systems","volume":"23","author":"Gerkey","year":"2004","journal-title":"Int. J. Robotics Res."},{"key":"B13","doi-asserted-by":"publisher","first-page":"293","DOI":"10.1007\/s10514-015-9457-9","article-title":"Decision-making authority, team efficiency and human worker satisfaction in mixed human\u2014robot teams","volume":"39","author":"Gombolay","year":"2015","journal-title":"Aut. Robots"},{"key":"B14","doi-asserted-by":"publisher","first-page":"220","DOI":"10.1109\/TRO.2018.2795034","article-title":"Fast scheduling of robot teams performing tasks with temporospatial constraints","volume":"34","author":"Gombolay","year":"2018","journal-title":"IEEE Trans. Robotics"},{"key":"B15","first-page":"598","article-title":"High precision grasp pose detection in dense clutter","author":"Gualtieri","year":"2016"},{"key":"B16","first-page":"993","article-title":"Robots with theory of mind for humans: a survey","author":"Gurney","year":"2022"},{"key":"B17","first-page":"2215","article-title":"Anticipating human actions for collaboration in the presence of task and sensor uncertainty","author":"Hawkins","year":"2014"},{"key":"B18","first-page":"499","article-title":"Probabilistic human action prediction and wait-sensitive planning for responsive human-robot collaboration","author":"Hawkins","year":"2013"},{"key":"B19","first-page":"4826","article-title":"Behavior tree capabilities for dynamic multi-robot task allocation with heterogeneous robot teams","author":"Heppner","year":"2024"},{"key":"B20","article-title":"Bayesian intention for enhanced human robot collaboration","author":"Hernandez-Cruz","year":"2024"},{"key":"B21","article-title":"A dual-camera robotic vision system based on the concept of active perception","author":"Hoseini","year":"2019"},{"key":"B22","doi-asserted-by":"publisher","first-page":"550","DOI":"10.1109\/TSMCA.2003.817398","article-title":"Campout: a control architecture for tightly coupled coordination of multirobot systems for planetary surface exploration","volume":"33","author":"Huntsberger","year":"2003","journal-title":"IEEE Trans. Syst. Man, Cybern. - Part A Syst. Humans"},{"key":"B23","doi-asserted-by":"publisher","first-page":"79","DOI":"10.1023\/b:auro.0000032939.08597.62","article-title":"Distributed control of multi-robot systems engaged in tightly coupled tasks","volume":"17","author":"Huntsberger","year":"2004","journal-title":"Aut. Robots"},{"key":"B24","doi-asserted-by":"publisher","first-page":"203","DOI":"10.1142\/s0219843608001418","article-title":"An architecture for understanding intent using a novel hidden Markov formulation","volume":"5","author":"Kelley","year":"2008","journal-title":"Int. J. Humanoid Robotics, Special Issue Cognitive Humanoid Robots"},{"key":"B25","first-page":"1292","article-title":"Heterogeneous multirobot coordination with spatial and temporal constraints","author":"Koes","year":"2005"},{"key":"B26","first-page":"113","article-title":"A model of generating and predicting intention toward human-robot cooperation","author":"Nguyen","year":"2018"},{"key":"B27","first-page":"776","article-title":"Alliance: an architecture for fault tolerant, cooperative control of heterogeneous mobile robots","author":"Parker","year":"1994"},{"key":"B28","doi-asserted-by":"publisher","first-page":"305","DOI":"10.1163\/156855397x00344","article-title":"L-alliance: task-oriented multi-robot learning in behavior-based systems","volume":"11","author":"Parker","year":"1996","journal-title":"Adv. Robot."},{"key":"B29","doi-asserted-by":"crossref","first-page":"921","DOI":"10.1007\/978-3-540-30301-5_41","article-title":"Multiple mobile robot systems","volume-title":"Springer handbook of robotics","author":"Parker","year":"2008"},{"key":"B30","article-title":"ROS: an open-source robot operating system","volume-title":"ICRA workshop on open source software","author":"Quigley","year":"2009"},{"key":"B31","doi-asserted-by":"publisher","first-page":"284","DOI":"10.3390\/robotics4030284","article-title":"Intent understanding using an activation spreading architecture","volume":"4","author":"Saffar","year":"2015","journal-title":"Robotics"},{"key":"B32","article-title":"Moveit!","author":"Sucan","year":"2013"},{"key":"B33","first-page":"150","article-title":"A task allocation algorithm based on market mechanism for multiple robot systems","author":"Wang","year":"2016"},{"key":"B34","first-page":"21","article-title":"Broadcast of local eligibility: behavior-based control for strongly cooperative robot teams","author":"Werger","year":"2000"},{"key":"B35","first-page":"5595","article-title":"Iq-asymtre: synthesizing coalition formation and execution for tightly-coupled multirobot tasks","author":"Zhang","year":"2010"},{"key":"B36","first-page":"1090","article-title":"Coalition coordination for tightly coupled multirobot tasks with sensor constraints","author":"Zhang","year":"2014"},{"key":"B37","doi-asserted-by":"publisher","first-page":"344","DOI":"10.1007\/s10458-010-9146-1","article-title":"Petri net plans - a framework for collaboration and coordination in multi-robot systems","volume":"23","author":"Ziparo","year":"2011","journal-title":"Aut. Agents Multi-Agent Syst."}],"container-title":["Frontiers in Robotics and AI"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2025.1533054\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T08:59:58Z","timestamp":1750150798000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frobt.2025.1533054\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,17]]},"references-count":37,"alternative-id":["10.3389\/frobt.2025.1533054"],"URL":"https:\/\/doi.org\/10.3389\/frobt.2025.1533054","relation":{},"ISSN":["2296-9144"],"issn-type":[{"value":"2296-9144","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,17]]},"article-number":"1533054"}}