{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,21]],"date-time":"2026-10-21T18:39:58Z","timestamp":1792607998867,"version":"4.5.0"},"reference-count":46,"publisher":"American Association for the Advancement of Science (AAAS)","issue":"119","funder":[{"DOI":"10.13039\/501100002661","name":"Fonds De La Recherche Scientifique - FNRS","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100002661","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100002661","name":"Fonds De La Recherche Scientifique - FNRS","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100002661","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100002910","name":"F\u00e9d\u00e9ration Wallonie-Bruxelles","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100002910","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.science.org"],"crossmark-restriction":true},"short-container-title":["Sci. Robot."],"published-print":{"date-parts":[[2026,10,21]]},"abstract":"<jats:p>Reinforcement learning offers a powerful framework for automatically generating control policies for robot swarms, yet its application to physical multirobot systems has been limited by the simulation-to-reality gap. Despite substantial advances in multiagent reinforcement learning, systematic demonstrations of gradient-based sim-to-real transfer to physical robot swarms remain scarce. Most work remains confined to simulation, and the few real-robot demonstrations lack evaluation across multiple runs and tasks. We show that this barrier can be overcome by embedding structural constraints into policy learning. We introduce attention-based counterfactual policy gradients for behavior selection in robot swarms (SwarmACB), a multiagent actor-critic method in which robots arbitrate among predefined behavior modules\u2014reactive sensorimotor routines such as phototaxis or aggregation\u2014rather than learning low-level motor commands. This architectural choice reduces the variance of learned solutions, which we argue mitigates overfitting to simulation artifacts. Through systematic evaluation on five collective missions using 20 physical e-puck robots (250 real-robot runs), we demonstrate that SwarmACB transfers successfully from simulation to reality, substantially outperforming continuous-action baselines. By progressively constraining policy architecture, perception, and model capacity, we show that smaller, structured policies transfer more reliably than large, unconstrained ones. These findings challenge the assumption that scale is necessary for successful transfer, suggesting instead that principled structural design enables reinforcement learning to produce control software for robot swarms that is both learned automatically and directly deployable in the physical world.<\/jats:p>","DOI":"10.1126\/scirobotics.adw4624","type":"journal-article","created":{"date-parts":[[2026,10,21]],"date-time":"2026-10-21T17:58:08Z","timestamp":1792605488000},"update-policy":"https:\/\/doi.org\/10.34133\/aaas_crossmark","source":"Crossref","is-referenced-by-count":0,"title":["Structural bias enables reliable sim-to-real transfer in reinforcement learning for robot swarms"],"prefix":"10.1126","volume":"11","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2329-6484","authenticated-orcid":true,"given":"Ilyes","family":"Gharbi","sequence":"first","affiliation":[{"name":"IRIDIA, Universit\u00e9 libre de Bruxelles, Brussels, Belgium."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9018-4995","authenticated-orcid":true,"given":"Miquel","family":"Kegeleirs","sequence":"additional","affiliation":[{"name":"IRIDIA, Universit\u00e9 libre de Bruxelles, Brussels, Belgium."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4948-9049","authenticated-orcid":true,"given":"Guillermo","family":"Legarda Herranz","sequence":"additional","affiliation":[{"name":"IRIDIA, Universit\u00e9 libre de Bruxelles, Brussels, Belgium."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-8334-8223","authenticated-orcid":true,"given":"Jeanne","family":"Szpirer","sequence":"additional","affiliation":[{"name":"IRIDIA, Universit\u00e9 libre de Bruxelles, Brussels, Belgium."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8196-9889","authenticated-orcid":true,"given":"Ken","family":"Hasselmann","sequence":"additional","affiliation":[{"name":"IRIDIA, Universit\u00e9 libre de Bruxelles, Brussels, Belgium."},{"name":"Royal Military Academy, Brussels, Belgium."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3309-2194","authenticated-orcid":true,"given":"Mauro","family":"Birattari","sequence":"additional","affiliation":[{"name":"IRIDIA, Universit\u00e9 libre de Bruxelles, Brussels, Belgium."}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"221","reference":[{"key":"e_1_3_2_2_2","unstructured":"R. S. Sutton A. G. Barto Reinforcement Learning: An Introduction (MIT Press ed. 2 2018)."},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11721-012-0075-2"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.aar7650"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.abe4385"},{"key":"e_1_3_2_6_2","unstructured":"K. Soma Y. Bouteiller H. Hamann G. Beltrame The hive mind is a single reinforcement learning agent. arXiv:2410.17517 [cs.MA] (2024)."},{"key":"e_1_3_2_7_2","unstructured":"M. H\u00fcttenrauch A. \u0160o\u0161i\u0107 G. Neumann \u201cGuided deep reinforcement learning for swarm systems \u201d paper presented at AAMAS 2017 Autonomous Robots and Multirobot Systems (ARMS) Workshop S\u00e3o Paulo Brazil 8 to 12 May 2017."},{"key":"e_1_3_2_8_2","doi-asserted-by":"crossref","unstructured":"M. H\u00fcttenrauch A. \u0160o\u0161i\u0107 G. Neumann \u201cLocal communication protocols for learning complex swarm behaviors with deep reinforcement learning\u201d in Swarm Intelligence\u2014ANTS 2018 vol. 11172 of Lecture Notes in Computer Science M. Dorigo M. Birattari C. Blum A. L. Christensen A. Reina V. Trianni Eds. (Springer 2018) pp. 71\u201383.","DOI":"10.1007\/978-3-030-00533-7_6"},{"key":"e_1_3_2_9_2","first-page":"1966","article-title":"Deep reinforcement learning for swarm systems","volume":"20","author":"H\u00fcttenrauch M.","year":"2019","unstructured":"M. H\u00fcttenrauch, A. \u0160o\u0161i\u0107, G. Neumann, Deep reinforcement learning for swarm systems. J. Mach. Learn. Res. 20, 1966\u20131996 (2019).","journal-title":"J. Mach. Learn. Res."},{"key":"e_1_3_2_10_2","doi-asserted-by":"crossref","unstructured":"W. Zhao J. P. Queralta T. Westerlund \u201cSim-to-real transfer in deep reinforcement learning for robotics: A survey\u201d in 2020 IEEE Symposium Series on Computational Intelligence (SSCI) (IEEE 2020) pp. 737\u2013744.","DOI":"10.1109\/SSCI47803.2020.9308468"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-022-00573-6"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1177\/10597123231202593"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.ado5888"},{"key":"e_1_3_2_14_2","doi-asserted-by":"crossref","unstructured":"J. Bloom J. Poindexter C. Pinciroli Enhancing learning of collective transport with global state prediction under local bandwidth-limited communication constraints. Research Square [Preprint] (2025); https:\/\/researchsquare.com\/article\/rs-6597379\/v1.","DOI":"10.21203\/rs.3.rs-6597379\/v1"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/TMC.2025.3530438"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-020-0215-0"},{"key":"e_1_3_2_17_2","doi-asserted-by":"crossref","unstructured":"S. Nolfi D. Floreano Evolutionary Robotics: The Biology Intelligence and Technology of Self-Organizing Machines (MIT Press 2000).","DOI":"10.7551\/mitpress\/2889.001.0001"},{"key":"e_1_3_2_18_2","doi-asserted-by":"crossref","unstructured":"V. Trianni Evolutionary Swarm Robotics (Springer 2008).","DOI":"10.1007\/978-3-540-77612-3"},{"key":"e_1_3_2_19_2","unstructured":"S. Nolfi Behavioral and Cognitive Robotics: An Adaptive Perspective (Institute of Cognitive Sciences and Technologies National Research Council 2021)."},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-021-24642-3"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11721-014-0092-4"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11721-015-0107-9"},{"key":"e_1_3_2_23_2","doi-asserted-by":"crossref","unstructured":"M. Birattari A. Ligot G. Francesca \u201cAutoMoDe: A modular approach to the automatic off-line design and fine-tuning of control software for robot swarms\u201d in Automated Design of Machine Learning and Search Algorithms N. Pillay R. Qu Eds. (Springer 2021) pp. 73\u201390.","DOI":"10.1007\/978-3-030-72069-8_5"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1992.4.1.1"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.3389\/frobt.2019.00059"},{"key":"e_1_3_2_26_2","doi-asserted-by":"crossref","unstructured":"J. N. Foerster G. Farquhar T. Afouras N. Nardelli S. Whiteson \u201cCounterfactual multi-agent policy gradients\u201d in Proceedings of the AAAI Conference on Artificial Intelligence (AAAI Press 2018) vol. 32 pp. 2974\u20132982.","DOI":"10.1609\/aaai.v32i1.11794"},{"key":"e_1_3_2_27_2","unstructured":"A. Cohen E. Teng V.-P. Berges R.-P. Dong H. Henry M. Mattar A. Zook S. Ganguly On the use and misuse of absorbing states in multi-agent reinforcement learning. arXiv:2111.05992 [cs.LG] (2021)."},{"key":"e_1_3_2_28_2","unstructured":"F. Mondada M. Bonani X. Raemy J. Pugh C. Cianci A. Klaptocz S. Magnenat J.-C. Zufferey D. Floreano A. Martinoli \u201cThe e-puck a robot designed for education in engineering\u201d in Proceedings of ROBOTICA 2009: The 9th Conference on Autonomous Robot Systems and Competitions P. Gon\u00e7alves P. Torres C. Alves Eds. (Instituto Polit\u00e9cnico de Castelo Branco 2009) pp. 59\u201365."},{"key":"e_1_3_2_29_2","unstructured":"K. Hasselmann A. Ligot G. Francesca D. Garz\u00f3n Ramos M. Salman J. Kuckling F. J. Mendiburu M. Birattari \u201cReference Models for AutoMoDe\u201d (TR\/IRIDIA\/2018-002 IRIDIA Universit\u00e9 libre de Bruxelles 2018)."},{"key":"e_1_3_2_30_2","doi-asserted-by":"crossref","unstructured":"A. Ligot K. Hasselmann M. Birattari \u201cAutoMoDe-Arlequin: Neural networks as behavioral modules for the automatic design of probabilistic finite state machines\u201d in Swarm Intelligence\u2014ANTS 2020 vol. 12421 of Lecture Notes in Computer Science M. Dorigo T. St\u00fctzle M. J. Blesa C. Blum H. Hamann M. K. Heinrich V. Strobel Eds. (Springer 2020) pp. 271\u2013281.","DOI":"10.1007\/978-3-030-60376-2_21"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.swevo.2023.101395"},{"key":"e_1_3_2_32_2","unstructured":"L. Garattoni G. Francesca A. Brutschy C. Pinciroli M. Birattari \u201cSoftware Infrastructure for e-puck (and TAM)\u201d (TR\/IRIDIA\/2015-004 IRIDIA Universit\u00e9 libre de Bruxelles 2015)."},{"key":"e_1_3_2_33_2","unstructured":"GCtronic Overo extension (2018); https:\/\/gctronic.com\/doc\/index.php\/Overo_Extension."},{"key":"e_1_3_2_34_2","doi-asserted-by":"crossref","unstructured":"\u00c1. Guti\u00e9rrez A. Campo M. Dorigo J. Donate F. Monasterio-Huelin L. Magdalena \u201cOpen e-puck range & bearing miniaturized board for local communication in swarm robotics\u201d in 2009 IEEE International Conference on Robotics and Automation K. Kosuge Ed. (IEEE 2009) pp. 3111\u20133116.","DOI":"10.1109\/ROBOT.2009.5152456"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11721-012-0072-5"},{"key":"e_1_3_2_36_2","unstructured":"A. Juliani V.-P. Berges E. Teng A. Cohen J. Harper C. Elion C. Goy Y. Gao H. Henry M. Mattar Unity: A general platform for intelligent agents. arXiv:1809.02627 [cs.LG] (2018)."},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1023\/A:1022672621406"},{"key":"e_1_3_2_38_2","unstructured":"B. Baker I. Kanitscheider T. Markov Y. Wu G. Powell B. McGrew I. Mordatch Emergent tool use from multi-agent autocurricula. arXiv:1909.07528 [cs.LG] (2019)."},{"key":"e_1_3_2_39_2","unstructured":"A. Vaswani N. Shazeer N. Parmar J. Uszkoreit L. Jones A. N. Gomez \u0141. Kaiser I. Polosukhin \u201cAttention is all you need\u201d in Advances in Neural Information Processing Systems I. Guyon U. von Luxburg S. Bengio H. Wallach R. Fergus S. V. N. Vishwanathan R. Garnett Eds. (Curran Associates 2017) vol. 30 pp. 5998\u20136008."},{"key":"e_1_3_2_40_2","unstructured":"J. Schulman F. Wolski P. Dhariwal A. Radford O. Klimov Proximal policy optimization algorithms. arXiv:1707.06347 [cs.LG] (2017)."},{"key":"e_1_3_2_41_2","doi-asserted-by":"crossref","unstructured":"M. Birattari Z. Yuan P. Balaprakash T. St\u00fctzle \u201cF-Race and iterated F-Race: An overview\u201d in Experimental Methods for the Analysis of Optimization Algorithms T. Bartz-Beielstein M. Chiarandini L. Paquete M. Preuss Eds. (Springer 2010) pp. 311\u2013336.","DOI":"10.1007\/978-3-642-02538-9_13"},{"key":"e_1_3_2_42_2","first-page":"43","article-title":"The irace package: Iterated racing for automatic algorithm configuration","volume":"3","author":"L\u00f3pez-Ib\u00e1\u00f1ez M.","year":"2016","unstructured":"M. L\u00f3pez-Ib\u00e1\u00f1ez, J. Dubois-Lacoste, L. P\u00e9rez C\u00e1ceres, M. Birattari, T. St\u00fctzle, The irace package: Iterated racing for automatic algorithm configuration. Oper. Res. Perspect. 3, 43\u201358 (2016).","journal-title":"Oper. Res. Perspect."},{"key":"e_1_3_2_43_2","doi-asserted-by":"crossref","unstructured":"N. Cambier E. Ferrante \u201cAutoMoDe-Pomodoro: An evolutionary class of modular designs\u201d in Proceedings of the Genetic and Evolutionary Computation Conference Companion J. E. Fieldsend Ed. (ACM 2022) pp. 100\u2013103.","DOI":"10.1145\/3520304.3529031"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2022.3144848"},{"key":"e_1_3_2_45_2","unstructured":"G. Legarda Herranz D. Garz\u00f3n Ramos J. Kuckling M. Kegeleirs M. Birattari \u201cTycho: A Robust ROS-Based Tracking System for Robot Swarms\u201d (TR\/IRIDIA\/2022-009 IRIDIA Universit\u00e9 libre de Bruxelles 2022)."},{"key":"e_1_3_2_46_2","unstructured":"I. Gharbi ilgha\/SwarmACB_Experiment_Code: SwarmACB_Experiment_Code version 1 Zenodo (2026); https:\/\/doi.org\/10.5281\/zenodo.21536300."},{"key":"e_1_3_2_47_2","unstructured":"I. Gharbi ilgha\/SwarmACB_Unity: SwarmACB_Unity version 1 Zenodo (2026); https:\/\/doi.org\/10.5281\/zenodo.21536363."}],"container-title":["Science Robotics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.science.org\/doi\/pdf\/10.1126\/scirobotics.adw4624","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,21]],"date-time":"2026-10-21T17:58:12Z","timestamp":1792605492000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.science.org\/doi\/10.1126\/scirobotics.adw4624"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,10,21]]},"references-count":46,"journal-issue":{"issue":"119","published-print":{"date-parts":[[2026,10,21]]}},"alternative-id":["10.1126\/scirobotics.adw4624"],"URL":"https:\/\/doi.org\/10.1126\/scirobotics.adw4624","relation":{},"ISSN":["2470-9476"],"issn-type":[{"value":"2470-9476","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,10,21]]},"assertion":[{"value":"2025-11-05","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-09-22","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-10-21","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"eadw4624"}}