{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T01:41:52Z","timestamp":1760146912449,"version":"build-2065373602"},"reference-count":66,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2024,12,31]],"date-time":"2024-12-31T00:00:00Z","timestamp":1735603200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"H2020 EU Marine Robots project","award":["731103"],"award-info":[{"award-number":["731103"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["JMSE"],"abstract":"<jats:p>According to Martec\u2019s Law, technology is evolving exponentially, while organizations are evolving logarithmically. These two trends present unprecedent challenges to the development of future maritime unmanned vehicle systems (MUS). This is exacerbated by the fact that we are also witnessing a paradigm change in MUS, from the operation of vehicles to the operation of systems of systems (SoS), delivering unprecedented capabilities and powered by recent developments in artificial intelligence and machine learning. Here, we discuss the impact of these trends on technological and organizational development from the perspective of the increasing role of evaluation and testing in operational environments. This is carried out in the framework of systems engineering, with reference to three illustrative case studies and required advances, to highlight the impact of the paradigm change on the development and deployment of SoS. SoS will have both and fast development scales, thus requiring fast development models that can only be achieved by streamlining modeling, simulation, and operational experimentation processes.<\/jats:p>","DOI":"10.3390\/jmse13010051","type":"journal-article","created":{"date-parts":[[2024,12,31]],"date-time":"2024-12-31T14:21:12Z","timestamp":1735654872000},"page":"51","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Evaluation and Testing in Operational Environments: Accelerating Development Cycles for Maritime Unmanned Systems"],"prefix":"10.3390","volume":"13","author":[{"given":"Jo\u00e3o","family":"Borges de Sousa","sequence":"first","affiliation":[{"name":"Laborat\u00f3rio de Sistemas e Tecnologias Subaqu\u00e1ticas (LSTS), Institute of Science and Innovation in Mechanical and Industrial Engineering (INEGI), Porto University, Rua Roberto Frias s\/n, 4200-465 Porto, Portugal"}]}],"member":"1968","published-online":{"date-parts":[[2024,12,31]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Berge, J., Johnsen, G., and Cohen, J.H. (2020). Sensor-Carrying Platforms. Marine Ecology\u2014Life and Light in the Dead of Night, Springer.","DOI":"10.1007\/978-3-030-33208-2"},{"key":"ref_2","first-page":"1","article-title":"Towards integrated autonomous underwater operations for ocean mapping and monitoring","volume":"42","author":"Ludvigsen","year":"2016","journal-title":"J. Annu. Rev. Control"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1098","DOI":"10.1126\/science.1146230","article-title":"Robotics in Remote and Hostile Environments","volume":"318","author":"Bellingham","year":"2007","journal-title":"Science"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"935","DOI":"10.1109\/JOE.2006.880429","article-title":"Multi-AUV Control and Adaptive Sampling in Monterey Bay","volume":"31","author":"Fiorelli","year":"2006","journal-title":"IEEE J. Ocean. Eng."},{"key":"ref_5","unstructured":"Skaugset, K., and S\u00f8rensen, A.J. (,  2019). Game of Drones. Proceedings of the Keynote Lecture at 25th Anniversary Jubilee of Equinor Research Centre 2019, Trondheim, Norway."},{"key":"ref_6","unstructured":"(2024, May 12). Martec\u2019s Law: The Greatest Management Challenge of the 21st Century. Available online: https:\/\/chiefmartec.com\/2016\/11\/martecs-law-great-management-challenge-21st-century\/."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Teuscher, C. (2004). The Law of Accelerating Returns. Alan Turing: Life and Legacy of a Great Thinker, Springer.","DOI":"10.1007\/978-3-662-05642-4"},{"key":"ref_8","unstructured":"(2024, May 12). Underwater Systems and Technologies Laboratory. Available online: https:\/\/lsts.fe.up.pt\/."},{"key":"ref_9","unstructured":"(2024, May 12). Exploring Fronts with Multiple Robots. Available online: https:\/\/schmidtocean.org\/cruise\/exploring_fronts_with_multiple_aerial-surface-underwater-vehicles\/."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"843","DOI":"10.55417\/fr.2022028","article-title":"Coordinated Robotic Exploration of Dynamic Open Ocean Phenomena","volume":"2","author":"Pinto","year":"2022","journal-title":"J. Field Robot."},{"key":"ref_11","unstructured":"(2024, May 12). EU Marine Robots. Available online: https:\/\/www.eumarinerobots.eu\/."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Alcaraz, D., Antonelli, G., Caccia, M., Dooly, G., Flavin, N., Kopf, A., Ludvigsen, M., Opderbecke, J., Palmer, M., and Pascoal, A. (October, January 30). The Marine Robotics Research Infrastructure Network (EUMarine Robots): An Overview. Proceedings of the 2020 IEEE\/OES Autonomous Underwater Vehicles Symposium (AUV), St. Johns, NL, Canada.","DOI":"10.1109\/AUV50043.2020.9267940"},{"key":"ref_13","unstructured":"(2024, May 12). Liquid Robotics. Available online: https:\/\/www.liquid-robotics.com\/wave-glider\/how-it-works\/."},{"key":"ref_14","unstructured":"(2024, May 12). Saildrone: Defense, Commercial & Science Data Solutions. Available online: https:\/\/www.saildrone.com\/."},{"key":"ref_15","unstructured":"Molina, M., L\u00f3pez-Castej\u00f3n, F., Gilabert, J., Lucas, F., and Gonz\u00e1lez, A. (2015, January 15\u201317). IVER2 improvement for oil in water sampling. Proceedings of the Sixth International Workshop on Marine Technology, Martech, Cartagena, Spain."},{"key":"ref_16","unstructured":"(2024, May 12). Laborat\u00f3rio de Sistemas e Tecnologias Subaqu\u00e1ticas (LSTS). Available online: http:\/\/www.lsts.pt\/vehicles\/lauv."},{"key":"ref_17","unstructured":"(2024, May 12). Manta Communication Gateways. Available online: http:\/\/www.lsts.pt\/support_systems\/manta."},{"key":"ref_18","unstructured":"(2024, May 12). LSTS Software Toolchain. Available online: https:\/\/lsts.fe.up.pt\/toolchain."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Belkin, I.M., Borges de Sousa, J., Pinto, J., Mendes, R., and L\u00f3pez-Castej\u00f3n, F. (2018, January 6\u20139). Marine robotics exploration of a large-scale open-ocean fron. Proceedings of the IEEE\/OES Autonomous Underwater Vehicle Symposium, Porto, Portugal.","DOI":"10.1109\/AUV.2018.8729725"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1007\/s11263-019-01217-w","article-title":"Robust Attentional Aggregation of Deep Feature Sets for Multi-view 3D Reconstruction","volume":"128","author":"Yang","year":"2019","journal-title":"Int. J. Comput. Vis."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"205","DOI":"10.1016\/j.ifacol.2019.12.308","article-title":"Multi-Representation Multi-Heuristic A* Motion Planning for a Dual-Arm Underwater Vehicle Manipulation System","volume":"52","author":"Pi","year":"2019","journal-title":"IFAC-PapersOnLine"},{"key":"ref_22","unstructured":"Franchi, M., Bucci, A., Zacchini, L., Topini, E., Ridolfi, A., and Allotta, B. (October, January 30). A Probabilistic 3D Map Representation for Forward-Looking SONAR Reconstructions. Proceedings of the IEEE\/OES Autonomous Underwater Vehicles Symposium, St. John, NL, Canada."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.oceaneng.2019.106840","article-title":"Vision based autonomous docking for work class ROVs","volume":"196","author":"Trslic","year":"2020","journal-title":"Ocean Eng."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"14608","DOI":"10.1016\/j.ifacol.2020.12.1469","article-title":"The Value Function as a Decision Support Tool in Unmanned Vehicle Operations","volume":"52","author":"Aguiar","year":"2020","journal-title":"IFAC-PapersOnLine"},{"key":"ref_25","unstructured":"Gao, C., Zhao, G., and Konopka, N. (2019). Range-based Navigation Algorithms for Marine Applications. Cooperative Localization and Navigation: Theory, Research and Practice, CRC Press & Routledge-Taylor & Francis Group."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1771","DOI":"10.1016\/j.jfranklin.2018.12.011","article-title":"Circular formation control for cooperative target tracking with limited information","volume":"356","author":"Brinon","year":"2019","journal-title":"J. Frankl. Inst."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Quintas, J., Petroccia, R., Pascoal, A., Cruz, J., Gois, P., Morlando, L., and Stipanov, M. (2021). Hybrid acoustic-optical underwater communication networks for next-generation cooperative systems: The EUMR experience. OCEANS 2021 San Diego Porto, IEEE.","DOI":"10.23919\/OCEANS44145.2021.9706085"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Mo-Bj\u00f8rkelund, T., Norgren, P., and Ludvigsen, M. (2020, January 30). Simulation and forecasting of ice drift as a tool for autonomous under ice operations. Proceedings of the IEEE\/OES Autonomous Underwater Vehicles Symposium, St. John, NL, Canada.","DOI":"10.1109\/AUV50043.2020.9267921"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Norgren, P., Mo-Bj\u00f8rkelund, T., Gade, K., Hegren\u00e6s, \u00d8., and Ludvigsen, M. (2020, January 30). Intelligent Buoys for Aiding AUV Navigation Under the Ice. Proceedings of the IEEE\/OES Autonomous Underwater Vehicles Symposium, St. John, NL, Canada.","DOI":"10.1109\/AUV50043.2020.9267889"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Bachmayer, R., Kausche, A., Riesen, P., and Gutierrez, P. (2020, January 30). Concept development for a minimally invasive Autonomous Underwater Vehicle. Proceedings of the IEEE\/OES Autonomous Underwater Vehicles Symposium, St. John, NL, Canada.","DOI":"10.1109\/AUV50043.2020.9267943"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Kvasi\u0107, I., Mi\u0161kovi\u0107, N., and Vuki\u0107, Z. (2019, January 17\u201320). Convolutional Neural Network Architectures for Sonar-Based Diver Detection and Tracking. Proceedings of the OCEANS 2019 Marseille, Marseille, France.","DOI":"10.1109\/OCEANSE.2019.8867461"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1016\/j.ifacol.2019.12.307","article-title":"Towards Advancing Diver-Robot Interaction Capabilities","volume":"52","author":"Walker","year":"2019","journal-title":"IFAC-PapersOnLine"},{"key":"ref_33","unstructured":"(2024, May 12). ZeroTier. Available online: https:\/\/my.zerotier.com."},{"key":"ref_34","unstructured":"(2024, May 12). Ripples. Available online: https:\/\/ripples.lsts.pt\/."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Martins, R., Borges de Sousa, J., and Afonso, C.C. (2011). The REP-AUV10 Experiment Shallow water surveys with a fleet of heterogeneous autonomous vehicles. Sea Technol., 27\u201331.","DOI":"10.1109\/Oceans-Spain.2011.6003568"},{"key":"ref_36","unstructured":"McGuillivary, P., Borges de Sousa, J., and Martins, R. (Marine Technology Reporter, 2012). Connecting the dots. Networking Maritime Fleets of Autonomous Systems for Science and Surveillance, Marine Technology Reporter."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Martins, R., Borges de Sousa, J., Afonso, C.C., and Incze, M. (2011, January 19\u201322). REP10 AUV: Shallow water survey with a fleet of heterogeneous autonomous vehicles. Proceedings of the MTS-IEEE Oceans Conference, Santander, Spain.","DOI":"10.1109\/Oceans-Spain.2011.6003568"},{"key":"ref_38","unstructured":"Rodrigues, R., Afonso, C.C., Borges de Sousa, J., Caldas, R., Martins, R., and Passos Morgado, J. (2013, January 10\u201314). Experiments in multivehicle operations: The REP12 exercise. Proceedings of the MAST Conference, Gdansk, Poland."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Borges de Sousa, J., Pereira, J., Pinto, J., Louren\u00e7o, P.C., Galocha, J.M., Fontes, J., Silva, M., Rajan, K., Johansson, T., and Alves, J. (2016, January 19\u201323). Rapid environmental picture Atlantic exercise 2015: A field report. Proceedings of the OCEANS MTS\/IEEE, Monterey, CA, USA.","DOI":"10.1109\/OCEANS.2016.7761496"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Ferreira, A.S., Pereira, J., Pinto, J., Dias, P., Borges de Sousa, J., and Lourenco, P.C. (2017, January 19\u201322). Rapid environmental picture atlantic exercise 2016: Field report. Proceedings of the OCEANS MTS\/IEEE, Aberdeen, Scotland.","DOI":"10.1109\/OCEANSE.2017.8084800"},{"key":"ref_41","first-page":"9","article-title":"REP18 Atlantic\u2013A Large Scale Exercise Using Unmanned Systems","volume":"20","author":"Dias","year":"2018","journal-title":"Instrum. ViewPoint"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Costa, J., Dias, P., Santos, R., Neiva, J., Ven\u00e2ncio, L., Gon\u00e7alves, P., Almeida, M.F., Pereira, J., Ribeiro, M., and Choi, W. (2023, January 25\u201328). 2022 Edition of the Large Scale Exercise REPMUS: Field Report. Proceedings of the MTS\/IEEE OCEANS, Limerick, Biloxi, MS, USA.","DOI":"10.1109\/OCEANSLimerick52467.2023.10244531"},{"key":"ref_43","unstructured":"(2024, May 12). Enabling Federate Collaborative Autonomy. Available online: https:\/\/www.sto.nato.int\/Lists\/test1\/activitydetails.aspx?ID=16932."},{"key":"ref_44","unstructured":"Riet, M., Alves, J., Arcieri, G., Bazzarello, L., Borges de Sousa, J., Colin, M., Cormack, A., Dias, P., Dinale, J., and Ferri, G. (2023, January 25\u201328). Experimental Demonstrations of CATL, the Collaborative Autonomy Tasking Layer\u2019. Proceedings of the MTS\/IEEE OCEANS, Limerick, Biloxi, MS, USA."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Potter, J., Alves, J., Green, D., Zappa, G., Nissen, I., and McCoy, K. (2014, January 3\u20135). The JANUS underwater communications standard. Proceedings of the 2014 Underwater Communications and Networking (UComms), Sestri Levante, Italy.","DOI":"10.1109\/UComms.2014.7017134"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Lima, K., Marques, E., Pinto, J., and Borges de Sousa, J. (2018, January 22\u201325). Programming Networked Vehicle Systems Using Dolphin-Field Tests at REP17. Proceedings of the OCEANS-MTS\/IEEE, Kobe Techno-Oceans, Charleston, SC, USA.","DOI":"10.1109\/OCEANSKOBE.2018.8559474"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Marques, E., Ribeiro, M., Pinto, J., Borges de Sousa, J., and Martins, F. (2015, January 13\u201317). NVL: A coordination language for unmanned vehicle networks. Proceedings of the 30th Annual ACM Symposium on Applied Computing, Salamanca, Spain.","DOI":"10.1145\/2695664.2696029"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"1555","DOI":"10.1007\/s10514-018-9810-x","article-title":"Advancing multi-vehicle deployments in oceanographic field experiments","volume":"43","author":"Ferreira","year":"2019","journal-title":"Auton. Robot."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Bowen, M.F., and Peters, D.B. (1998). A Deep-Sea Docking Station for ODYSSEY Class Autonomous Underwater Vehicles, a Technical Report Woods Hole Oceanographic Institution.","DOI":"10.1575\/1912\/319"},{"key":"ref_50","unstructured":"Hydroid (2012, January 22\u201324). Underwater mobile docking of autonomous underwater vehicles. Proceedings of the IEEE\/MTS 2012 Oceans, Hampton Roads, VA, USA."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"550","DOI":"10.1109\/JOE.2008.2005348","article-title":"Docking Control System for a 54-cm-Diameter (21-in) AUV","volume":"33","author":"McEwen","year":"2008","journal-title":"IEEE J. Ocean. Eng."},{"key":"ref_52","doi-asserted-by":"crossref","first-page":"147","DOI":"10.1016\/j.nahs.2019.05.007","article-title":"Position paper on the challenges posed by modern applications to cyber-physical systems theory","volume":"34","author":"Kapinski","year":"2019","journal-title":"Nonlinear Anal. Hybrid Syst."},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"eabe4385","DOI":"10.1126\/scirobotics.abe4385","article-title":"Reflections on the future of swarm robotics","volume":"5","author":"Dorigo","year":"2020","journal-title":"Sci. Robot."},{"key":"ref_54","unstructured":"(2024, May 12). Incose. Available online: https:\/\/www.incose.org\/about-systems-engineering."},{"key":"ref_55","unstructured":"(2024, May 12). Carnegie Mellon University, Software Enginnering Institute, an Introduction to Model-Based Systems Enginnering (MBSE). Available online: https:\/\/insights.sei.cmu.edu\/blog\/introduction-model-based-systems-engineering-mbse\/."},{"key":"ref_56","unstructured":"(2024, May 12). Wikipedia, Systems Development Life Ccyle. Available online: https:\/\/en.wikipedia.org\/wiki\/Systems_development_life_cycle."},{"key":"ref_57","doi-asserted-by":"crossref","unstructured":"Danjou, C., Harik, R., Nyffenegger, F., Rivest, L., and Bouras, A. (2024). SE Based Development Framework for Changeable Maritime Systems. Product Lifecycle Management. Leveraging Digital Twins, Circular Economy, and Knowledge Management for Sustainable Innovation, Springer. PLM 2023.","DOI":"10.1007\/978-3-031-62582-4"},{"key":"ref_58","doi-asserted-by":"crossref","first-page":"e2202075119","DOI":"10.1073\/pnas.2202075119","article-title":"Are general circulation models obsolete?","volume":"119","author":"Balaji","year":"2022","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_59","unstructured":"(2024, May 12). The Nobel Prize in Physics 2021. Available online: https:\/\/www.nobelprize.org\/prizes\/physics\/2021\/summary\/."},{"key":"ref_60","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1109\/MCS.2024.3382388","article-title":"A Quantitative Framework for Layered Multirate Control: Toward a Theory of Control Architecture","volume":"44","author":"Matni","year":"2024","journal-title":"IEEE Control Syst."},{"key":"ref_61","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1002\/acs.920","article-title":"A verified hierarchical control architecture for coordinated multi-vehicle operations","volume":"21","author":"Johansson","year":"2007","journal-title":"J. Adapt. Control Signal Process."},{"key":"ref_62","doi-asserted-by":"crossref","unstructured":"\u015eim\u015fek, T., Varaiya, P., and Borges de Sousa, J. (2001, January 25\u201327). Communication and control of distributed hybrid systems. Proceedings of the 2001 American Control Conference.(Cat. No. 01CH37148), Arlington, MA, USA.","DOI":"10.1109\/ACC.2001.945772"},{"key":"ref_63","doi-asserted-by":"crossref","first-page":"459","DOI":"10.1016\/S0951-8339(00)00035-6","article-title":"A hierarchical control architecture for Mobile Offshore Bases","volume":"13","author":"Girard","year":"2000","journal-title":"Mar. Struct. J."},{"key":"ref_64","unstructured":"Borges de Sousa, J., Simsek, T., and Varaiya, P. (2004, January 14\u201317). Task planning and execution for UAV teams. Proceedings of the IEEE Conference on Decision and Control, Atlantis, Paradise Island, The Bahamas."},{"key":"ref_65","doi-asserted-by":"crossref","unstructured":"Sousa, L.L., L\u00f3pez-Castej\u00f3n, F., Gilabert, J., Relvas, P., Couto, A., Queiroz, N., Caldas, R., Dias, P., Dias, P., and Faria, M. (2016). Integrated monitoring of Mola mola behavior in space and time. PLoS ONE, 11.","DOI":"10.1371\/journal.pone.0160404"},{"key":"ref_66","unstructured":"Djaferis, T.E., and Schick, I.C. (2000). Question about Hierarchical Systems. System Theory, Springer International Series in Engineering and Computer Science."}],"container-title":["Journal of Marine Science and Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2077-1312\/13\/1\/51\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T16:57:50Z","timestamp":1760115470000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2077-1312\/13\/1\/51"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,31]]},"references-count":66,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2025,1]]}},"alternative-id":["jmse13010051"],"URL":"https:\/\/doi.org\/10.3390\/jmse13010051","relation":{},"ISSN":["2077-1312"],"issn-type":[{"type":"electronic","value":"2077-1312"}],"subject":[],"published":{"date-parts":[[2024,12,31]]}}}