{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,31]],"date-time":"2026-01-31T01:58:29Z","timestamp":1769824709168,"version":"3.49.0"},"reference-count":38,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2017,9,12]],"date-time":"2017-09-12T00:00:00Z","timestamp":1505174400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Robotics"],"abstract":"<jats:p>While the robotics community agrees that the benchmarking is of high importance to objectively compare different solutions, there are only few and limited tools to support it. To address this issue in the context of multi-robot systems, we have defined a benchmarking process based on experimental designs, which aimed at improving the reproducibility of experiments by making explicit all elements of a benchmark such as parameters, measurements and metrics. We have also developed a ROS (Robot Operating System)-based testbed with the goal of making it easy for users to validate, benchmark, and compare different algorithms including coordination strategies. Our testbed uses the MORSE (Modular OpenRobots Simulation Engine) simulator for realistic simulation and a computer cluster for decentralized computation. In this paper, we present our testbed in details with the architecture and infrastructure, the issues encountered in implementing the infrastructure, and the automation of the deployment. We also report a series of experiments on multi-robot exploration, in order to demonstrate the capabilities of our testbed.<\/jats:p>","DOI":"10.3390\/robotics6030021","type":"journal-article","created":{"date-parts":[[2017,9,12]],"date-time":"2017-09-12T10:40:04Z","timestamp":1505212804000},"page":"21","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":26,"title":["Building a ROS-Based Testbed for Realistic Multi-Robot Simulation: Taking the Exploration as an Example"],"prefix":"10.3390","volume":"6","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8251-9786","authenticated-orcid":false,"given":"Zhi","family":"Yan","sequence":"first","affiliation":[{"name":"Le2i Laboratory, University of Technology of Belfort-Montb\u00e9liard (UTBM), 90010 Belfort, France"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Luc","family":"Fabresse","sequence":"additional","affiliation":[{"name":"CAR Team, IMT Lille Douai, 59500 Douai, France"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jannik","family":"Laval","sequence":"additional","affiliation":[{"name":"DISP Laboratory, University Lyon 2, 69007 Lyon, France"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Noury","family":"Bouraqadi","sequence":"additional","affiliation":[{"name":"CAR Team, IMT Lille Douai, 59500 Douai, France"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2017,9,12]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Parker, L.E. (2008). Multiple Mobile Robot Systems, Springer.","DOI":"10.1007\/978-3-540-30301-5_41"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"399","DOI":"10.5772\/57313","article-title":"A Survey and Analysis of Multi-Robot Coordination","volume":"10","author":"Yan","year":"2013","journal-title":"Int. J. Adv. Robot. Syst."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Yamauchi, B. (1998, January 10\u201313). Frontier-Based Exploration Using Multiple Robots. Proceedings of the 2nd International Conference on Autonomous Agents, Minneapolis, MN, USA.","DOI":"10.1145\/280765.280773"},{"key":"ref_4","unstructured":"Burgard, W., Moors, M., Fox, D., Simmons, R., and Thrun, S. (2000, January 24\u201328). Collaborative Multi-Robot Exploration. Proceedings of the 2000 IEEE International Conference on Robotics and Automation (ICRA), San Francisco, CA, USA."},{"key":"ref_5","unstructured":"Vazquez, J., and Malcolm, C. (2004, January 22\u201324). Distributed Multirobot Exploration Maintaining a Mobile Network. Proceedings of the 6th IEEE International Conference on Intelligent Systems (IS), Varna, Bulgaria."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1243","DOI":"10.1177\/0278364906072250","article-title":"Multi-robot Simultaneous Localization and Mapping using Particle Filters","volume":"25","author":"Howard","year":"2006","journal-title":"Int. J. Robot. Res."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"945","DOI":"10.1016\/j.robot.2006.06.003","article-title":"Distributed multi-robot coordination in area exploration","volume":"54","author":"Sheng","year":"2006","journal-title":"Robot. Auton. Syst."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"435","DOI":"10.1016\/j.conengprac.2006.08.007","article-title":"Multi-robot exploration under the constraints of wireless networking","volume":"15","author":"Rooker","year":"2007","journal-title":"Control Eng. Pract."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Doniec, A., Bouraqadi, N., Defoort, M., Le, V.T., and Stinckwich, S. (2009, January 2\u20134). Distributed Constraint Reasoning Applied to Multi-robot Exploration. Proceedings of the 21st IEEE International Conference on Tools with Artificial Intelligence (ICTAI), Newark, NJ, USA.","DOI":"10.1109\/ICTAI.2009.104"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Stachniss, C. (2009). Robotic Mapping and Exploration, Springer.","DOI":"10.1007\/978-3-642-01097-2"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"850","DOI":"10.1109\/TSMCB.2011.2179799","article-title":"A Flooding Algorithm for Multirobot Exploration","volume":"42","author":"Xiao","year":"2012","journal-title":"IEEE Trans. Syst. Man Cybern. Part B"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1109\/MRA.2014.2312841","article-title":"RFS Collaborative Multivehicle SLAM: SLAM in Dynamic High-Clutter Environments","volume":"21","author":"Moratuwage","year":"2014","journal-title":"IEEE Robot. Autom. Mag."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1109\/MRA.2014.2304091","article-title":"Group Mapping: A Topological Approach to Map Merging for Multiple Robots","volume":"21","author":"Saeedi","year":"2014","journal-title":"IEEE Robot. Autom. Mag."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Yan, Z., Fabresse, L., Laval, J., and Bouraqadi, N. (2014, January 20\u201323). Team Size Optimization for Multi-robot Exploration. Proceedings of the 4th International Conference on Simulation, Modeling, and Programming for Autonomous Robots (SIMPAR), Bergamo, Italy.","DOI":"10.1007\/978-3-319-11900-7_37"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Faigl, J., and Kulich, M. (2015, January 2\u20134). On benchmarking of frontier-based multi-robot exploration strategies. Proceedings of the 2015 European Conference on Mobile Robots (ECMR), Lincoln, UK.","DOI":"10.1109\/ECMR.2015.7324183"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"200","DOI":"10.1016\/j.robot.2013.10.004","article-title":"Benchmark of swarm robotics distributed techniques in a search task","volume":"62","author":"Couceiro","year":"2014","journal-title":"Robot. Auton. Syst."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"22","DOI":"10.1109\/MRA.2014.2298363","article-title":"Fostering Progress in Performance Evaluation and Benchmarking of Robotic and Automation Systems","volume":"21","author":"Bonsignorio","year":"2014","journal-title":"IEEE Robot. Autom. Mag."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Taylor, J.R., Drumwright, E.M., and Parmer, G. (2014, January 20\u201323). Making Time Make Sense in Robotic Simulation. Proceedings of the 4th International Conference on Simulation, Modeling, and Programming for Autonomous Robots (SIMPAR), Bergamo, Italy.","DOI":"10.1007\/978-3-319-11900-7_1"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"483","DOI":"10.1016\/j.robot.2013.01.003","article-title":"RoboNetSim: An integrated framework for multi-robot and network simulation","volume":"61","author":"Kudelski","year":"2013","journal-title":"Robot. Auton. Syst."},{"key":"ref_20","unstructured":"Gerkey, B.P., Vaughan, R.T., and Howard, A. (July, January 30). The Player\/Stage Project: Tools for Multi-Robot and Distributed Sensor Systems. Proceedings of the 11th International Conference on Advanced Robotics (ICAR), Coimbra, Portugal."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"39","DOI":"10.5772\/5618","article-title":"Cyberbotics Ltd. WebotsTM: Professional Mobile Robot Simulation","volume":"1","author":"Michel","year":"2004","journal-title":"Int. J. Adv. Robot. Syst."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Freese, M., Singh, S.P.N., Ozaki, F., and Matsuhira, N. (2010, January 15\u201318). Virtual Robot Experimentation Platform V-REP: A Versatile 3D Robot Simulator. Proceedings of the 2nd International Conference on Simulation, Modeling, and Programming for Autonomous Robots (SIMPAR), Darmstadt, Germany.","DOI":"10.1007\/978-3-642-17319-6_8"},{"key":"ref_23","unstructured":"Koenig, N.P., and Howard, A. (2004, January 14\u201318). Design and use paradigms for Gazebo, an open-source multi-robot simulator. Proceedings of the 2004 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Chicago, IL, USA."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Echeverria, G., Lemaignan, S., Degroote, A., Lacroix, S., Karg, M., Koch, P., Lesire, C., and Stinckwich, S. (2012, January 5\u20138). Simulating Complex Robotic Scenarios with MORSE. Proceedings of the 3rd International Conference on Simulation, Modeling, and Programming for Autonomous Robots (SIMPAR), Tsukuba, Japan.","DOI":"10.1007\/978-3-642-34327-8_20"},{"key":"ref_25","unstructured":"Quigley, M., Conley, K., Gerkey, B., Faust, J., Foote, T., Leibs, J., Wheeler, R., and Ng, A.Y. (2009, January 18\u201320). ROS: An open-source Robot Operating System. Proceedings of the ICRA\u201909 Workshop on Open Source Software, Guiyang, China."},{"key":"ref_26","unstructured":"Del Pobil, A.P., Madhavan, R., and Messina, E. (2006, January 9\u201315). Benchmarks in robotics research. Proceedings of the 2006 IEEE\/RSJ International Conference on Intelligent Robots and Systems, Beijing, China."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1109\/MRA.2011.941627","article-title":"Recasting Robotics Challenges as Experiments","volume":"18","author":"Anderson","year":"2011","journal-title":"IEEE Robot. Autom. Mag."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1109\/M-RA.2007.914924","article-title":"Experimental Testbed for Large Multirobot Teams","volume":"15","author":"Michael","year":"2008","journal-title":"IEEE Robot. Autom. Mag."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Okada, T., abd Razvan Beuran, J.N., Tan, Y., and Shinoda, Y. (2008, January 15\u201316). Large-scale Simulation Method of Mobile Robots. Proceedings of the IEEE 2nd International Symposium on Universal Communication (ISUC), Osaka, Japan.","DOI":"10.1109\/ISUC.2008.42"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Yan, Z., Fabresse, L., Laval, J., and Bouraqadi, N. (October, January 28). Metrics for Performance Benchmarking of Multi-robot Exploration. Proceedings of the 2015 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Hamburg, Germany.","DOI":"10.1109\/IROS.2015.7353852"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Le, V.T., Bouraqadi, N., Stinckwich, S., Moraru, V., and Doniec, A. (2009, January 12\u201317). Making networked robots connectivity-aware. Proceedings of the 2009 IEEE International Conference on Robotics and Automation (ICRA), Kobe, Japan.","DOI":"10.1109\/ROBOT.2009.5152471"},{"key":"ref_32","unstructured":"Lampe, A., and Chatila, R. (2006, January 15\u201319). Performance Measure for the Evaluation of Mobile Robot Autonomy. Proceedings of the 2006 IEEE International Conference on Robotics and Automation (ICRA), Orlando, FL, USA."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Zlot, R., Stentz, A.T., Dias, M.B., and Thayer, S. (2002, January 11\u201315). Multi-Robot Exploration Controlled by a Market Economy. Proceedings of the 2002 IEEE International Conference on Robotics and Automation (ICRA), Washington, DC, USA.","DOI":"10.21236\/ADA529530"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Frank, S., Listmann, K.D., Haumann, A.D., and Willert, V. (2010, January 15\u201318). Performance Analysis for Multi-robot Exploration Strategies. Proceedings of the 2nd International Conference on Simulation, Modeling, and Programming for Autonomous Robots (SIMPAR), Darmstadt, Germany.","DOI":"10.1007\/978-3-642-17319-6_37"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Scrapper, C., Madhavan, R., Lakaemper, R., Censi, A., Godil, A., Wagan, A., and Jacoff, A. (2009). Quantitative Assessment of Robot-Generated Maps. Performance Evaluation and Benchmarking of Intelligent Systems, Springer.","DOI":"10.1007\/978-1-4419-0492-8_10"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Henderson, R.L. (1995, January 5). Job scheduling under the Portable Batch System. Proceedings of the Workshop on Job Scheduling Strategies for Parallel Processing, Santa Barbara, CA, USA.","DOI":"10.1007\/3-540-60153-8_34"},{"key":"ref_37","unstructured":"Zhu, J., Zheng, Q., Liang, Y., and Liang, M. (2009, January 22\u201324). A component-based hybrid testbed for multi-mobile robots. Proceedings of the 2009 International Conference on Information and Automation, Macau, China."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Laval, J., Fabresse, L., and Bouraqadi, N. (2013, January 3\u20137). A methodology for testing mobile autonomous robots. Proceedings of the 2013 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Tokyo, Japan.","DOI":"10.1109\/IROS.2013.6696599"}],"container-title":["Robotics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2218-6581\/6\/3\/21\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T18:44:42Z","timestamp":1760208282000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2218-6581\/6\/3\/21"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,9,12]]},"references-count":38,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2017,9]]}},"alternative-id":["robotics6030021"],"URL":"https:\/\/doi.org\/10.3390\/robotics6030021","relation":{},"ISSN":["2218-6581"],"issn-type":[{"value":"2218-6581","type":"electronic"}],"subject":[],"published":{"date-parts":[[2017,9,12]]}}}