{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,12]],"date-time":"2026-04-12T03:50:09Z","timestamp":1775965809959,"version":"3.50.1"},"reference-count":61,"publisher":"American Association for the Advancement of Science (AAAS)","issue":"97","content-domain":{"domain":["www.science.org"],"crossmark-restriction":true},"short-container-title":["Sci. Robot."],"published-print":{"date-parts":[[2024,12,18]]},"abstract":"<jats:p>Swarm robots offer fascinating opportunities to perform complex tasks beyond the capabilities of individual machines. Just as a swarm of ants collectively moves large objects, similar functions can emerge within a group of robots through individual strategies based on local sensing. However, realizing collective functions with individually controlled microrobots is particularly challenging because of their micrometer size, large number of degrees of freedom, strong thermal noise relative to the propulsion speed, and complex physical coupling between neighboring microrobots. Here, we implemented multiagent reinforcement learning (MARL) to generate a control strategy for up to 200 microrobots whose motions are individually controlled by laser spots. During the learning process, we used so-called counterfactual rewards that automatically assign credit to the individual microrobots, which allows fast and unbiased training. With the help of this efficient reward scheme, swarm microrobots learn to collectively transport a large cargo object to an arbitrary position and orientation, similar to ant swarms. We show that this flexible and versatile swarm robotic system is robust to variations in group size, the presence of malfunctioning units, and environmental noise. In addition, we let the robot swarms manipulate multiple objects simultaneously in a demonstration experiment, highlighting the benefits of distributed control and independent microrobot motion. Control strategies such as ours can potentially enable complex and automated assembly of mobile micromachines, programmable drug delivery capsules, and other advanced lab-on-a-chip applications.<\/jats:p>","DOI":"10.1126\/scirobotics.ado5888","type":"journal-article","created":{"date-parts":[[2024,12,18]],"date-time":"2024-12-18T19:00:44Z","timestamp":1734548444000},"update-policy":"https:\/\/doi.org\/10.34133\/aaas_crossmark","source":"Crossref","is-referenced-by-count":33,"title":["Counterfactual rewards promote collective transport using individually controlled swarm microrobots"],"prefix":"10.1126","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1248-4770","authenticated-orcid":true,"given":"Veit-Lorenz","family":"Heuthe","sequence":"first","affiliation":[{"name":"Department of Physics, University of Konstanz, Universitaetsstrasse 10, Konstanz, 78464, Germany."},{"name":"Centre for the Advanced Study of Collective Behaviour, Universitaetsstrasse 10, Konstanz, 78464, Germany."}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6367-0082","authenticated-orcid":true,"given":"Emanuele","family":"Panizon","sequence":"additional","affiliation":[{"name":"Abdus Salam International Centre for Theoretical Physics (ICTP), Strada Costiera 11 Trieste, 34151, Italy."},{"name":"Data Engineering Laboratory, Area Science Park, Localit\u00e0 Padriciano 99, Trieste, 34149, Italy."}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6396-3408","authenticated-orcid":true,"given":"Hongri","family":"Gu","sequence":"additional","affiliation":[{"name":"Department of Physics, University of Konstanz, Universitaetsstrasse 10, Konstanz, 78464, Germany."}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5496-5268","authenticated-orcid":true,"given":"Clemens","family":"Bechinger","sequence":"additional","affiliation":[{"name":"Department of Physics, University of Konstanz, Universitaetsstrasse 10, Konstanz, 78464, Germany."},{"name":"Centre for the Advanced Study of Collective Behaviour, Universitaetsstrasse 10, Konstanz, 78464, Germany."}]}],"member":"221","reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1121\/1.2967480"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1818759116"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.228.4707.1489"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.crvi.2004.07.002"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jocs.2019.101049"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.3390\/drones7040269"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.3389\/frobt.2020.00036"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1109\/TRO.2018.2857475"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.aau8479"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-021-00341-y"},{"key":"e_1_3_2_12_2","doi-asserted-by":"crossref","unstructured":"J. W. Romanishin K. Gilpin S. Claici D. Rus 3D M-Blocks: Self-reconfiguring robots capable of locomotion via pivoting in three dimensions in 2015 IEEE International Conference on Robotics and Automation (ICRA) (IEEE 2015) pp. 1925\u20131932.","DOI":"10.1109\/ICRA.2015.7139450"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.1245842"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.abd8668"},{"key":"e_1_3_2_15_2","doi-asserted-by":"crossref","unstructured":"M. Kouzehgar M. Meghjani R. Bouffanais Multi-agent reinforcement learning for dynamic ocean monitoring by a swarm of buoys in Global Oceans 2020: Singapore\u2013US Gulf Coast (IEEE 2020) pp. 1\u20138.","DOI":"10.1109\/IEEECONF38699.2020.9389128"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-control-042920-013322"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/TRO.2021.3111788"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1038\/nmat4696"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1038\/nature12673"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.aav8006"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-018-05749-6"},{"key":"e_1_3_2_22_2","doi-asserted-by":"crossref","unstructured":"S. Martel M. Mohammadi Using a swarm of self-propelled natural microrobots in the form of flagellated bacteria to perform complex micro-assembly tasks in 2010 IEEE International Conference on Robotics and Automation (IEEE 2010) pp. 500\u2013505.","DOI":"10.1109\/ROBOT.2010.5509752"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-020-2626-9"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.abe3950"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.abn0602"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1002\/adfm.202304937"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.abd3557"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.aau5347"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1103\/RevModPhys.88.045006"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41563-019-0407-3"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1038\/ncomms4124"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1088\/1367-2630\/acf33a"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cogr.2023.07.004"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1007\/s43154-022-00091-8"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-017-14126-0"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-018-05675-7"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1039\/D0SM00964D"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-019-12360-w"},{"key":"e_1_3_2_39_2","unstructured":"P. Sunehag G. Lever A. Gruslys W. M. Czarnecki V. Zambaldi M. Jaderberg M. Lanctot N. Sonnerat J. Z. Leibo K. Tuyls T. Graepel Value-decomposition networks for cooperative multi-agent learning. arXiv:1706.05296 [cs.AI] (2017)."},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1142\/S0219525901000188"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1021\/acs.accounts.8b00237"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1088\/1361-648X\/ac3a86"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-023-42119-9"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.abd9285"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-022-00482-8"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.adh3073"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSSC.2017.2745818"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10439-013-0791-9"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-022-05408-3"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1021\/ac4006414"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1126\/scirobotics.aah6451"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1126\/sciadv.adl1549"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.3390\/s24082583"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0030-4018(02)01524-9"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-020-16161-4"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1006\/jtbi.2002.3025"},{"key":"e_1_3_2_57_2","unstructured":"A. Y. Ng D. Harada S. Russell Policy invariance under reward transformations: Theory and application to reward shaping in Proceedings of the Sixteenth International Conference on Machine Learning (ICML 1999) pp. 278\u2013287."},{"key":"e_1_3_2_58_2","unstructured":"M. Abadi A. Agarwal P. Barham E. Brevdo Z. Chen C. Citro G. S. Corrado A. Davis J. Dean M. Devin S. Ghemawat I. Goodfellow A. Harp G. Irving M. Isard Y. Jia R. Jozefowicz L. Kaiser M. Kudlur J. Levenberg D. Mane R. Monga S. Moore D. Murray C. Olah M. Schuster J. Shlens B. Steiner I. Sutskever K. Talwar P. Tucker V. Vanhoucke V. Vasudevan F. Viegas O. Vinyals P. Warden M. Wattenberg M. Wicke Y. Yu X. Zheng TensorFlow: Large-scale machine learning on heterogeneous systems. arXiv:1603.04467 [cs.DC] (2016)."},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1016\/0262-8856(90)90059-E"},{"key":"e_1_3_2_60_2","unstructured":"J. Schulman S. Levine P. Abbeel M. Jordan P. Moritz Trust region policy optimization in Proceedings of the 32nd International Conference on Machine Learning (ICML 2015) pp. 1889\u20131897."},{"key":"e_1_3_2_61_2","unstructured":"J. Schulman F. Wolski P. Dhariwal A. Radford O. Klimov Proximal policy optimization algorithms. arXiv:1707.06347 [cs.LG] (2017)."},{"key":"e_1_3_2_62_2","doi-asserted-by":"crossref","unstructured":"K. Zhang Z. Yang T. Ba\u015far Multi-agent reinforcement learning: A selective overview of theories and algorithms in Handbook of Reinforcement Learning and Control K. G. Vamvoudakis Y. Wan F. L. Lewis D. Cansever Eds. (Springer 2021) pp. 321\u2013384.","DOI":"10.1007\/978-3-030-60990-0_12"}],"container-title":["Science Robotics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.science.org\/doi\/pdf\/10.1126\/scirobotics.ado5888","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,12,18]],"date-time":"2024-12-18T19:00:56Z","timestamp":1734548456000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.science.org\/doi\/10.1126\/scirobotics.ado5888"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,18]]},"references-count":61,"journal-issue":{"issue":"97","published-print":{"date-parts":[[2024,12,18]]}},"alternative-id":["10.1126\/scirobotics.ado5888"],"URL":"https:\/\/doi.org\/10.1126\/scirobotics.ado5888","relation":{},"ISSN":["2470-9476"],"issn-type":[{"value":"2470-9476","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,12,18]]},"assertion":[{"value":"2024-02-14","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-11-19","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-12-18","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"eado5888"}}