{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,13]],"date-time":"2026-07-13T22:00:53Z","timestamp":1783980053069,"version":"3.55.0"},"reference-count":49,"publisher":"MDPI AG","issue":"8","license":[{"start":{"date-parts":[[2021,4,19]],"date-time":"2021-04-19T00:00:00Z","timestamp":1618790400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>The monitoring of water resources using Autonomous Surface Vehicles with water-quality sensors has been a recent approach due to the advances in unmanned transportation technology. The Ypacara\u00ed Lake, the biggest water resource in Paraguay, suffers from a major contamination problem because of cyanobacteria blooms. In order to supervise the blooms using these on-board sensor modules, a Non-Homogeneous Patrolling Problem (a NP-hard problem) must be solved in a feasible amount of time. A dimensionality study is addressed to compare the most common methodologies, Evolutionary Algorithm and Deep Reinforcement Learning, in different map scales and fleet sizes with changes in the environmental conditions. The results determined that Deep Q-Learning overcomes the evolutionary method in terms of sample-efficiency by 50\u201370% in higher resolutions. Furthermore, it reacts better than the Evolutionary Algorithm in high space-state actions. In contrast, the evolutionary approach shows a better efficiency in lower resolutions and needs fewer parameters to synthesize robust solutions. This study reveals that Deep Q-learning approaches exceed in efficiency for the Non-Homogeneous Patrolling Problem but with many hyper-parameters involved in the stability and convergence.<\/jats:p>","DOI":"10.3390\/s21082862","type":"journal-article","created":{"date-parts":[[2021,4,19]],"date-time":"2021-04-19T21:59:49Z","timestamp":1618869589000},"page":"2862","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":20,"title":["A Dimensional Comparison between Evolutionary Algorithm and Deep Reinforcement Learning Methodologies for Autonomous Surface Vehicles with Water Quality Sensors"],"prefix":"10.3390","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7796-3599","authenticated-orcid":false,"given":"Samuel","family":"Yanes Luis","sequence":"first","affiliation":[{"name":"Department of Electronic Engineering, University of Seville, 41009 Seville, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2481-5058","authenticated-orcid":false,"given":"Daniel","family":"Guti\u00e9rrez-Reina","sequence":"additional","affiliation":[{"name":"Department of Electronic Engineering, University of Seville, 41009 Seville, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2612-0388","authenticated-orcid":false,"given":"Sergio","family":"Toral Mar\u00edn","sequence":"additional","affiliation":[{"name":"Department of Electronic Engineering, University of Seville, 41009 Seville, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,4,19]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Moreira, M.G.A.L., Hinegk, L., Salvadore, A., Zolezzi, G., H\u00f6lker, F., Monte Domecq S., R.A., Bocci, M., Carrer, S., Nat, L.D., and Escrib\u00e1, J. (2018). Eutrophication, research and management history of the shallow Ypacara\u00ed Lake (Paraguay). Sustainability, 10.","DOI":"10.3390\/su10072426"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1495","DOI":"10.1007\/s12652-018-0920-2","article-title":"Comparison of Eulerian and Hamiltonian circuits for evolutionary-based path planning of an autonomous surface vehicle for monitoring Ypacarai Lake","volume":"10","author":"Espartza","year":"2019","journal-title":"J. Ambient. Intell. Humaniz. Comput."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"9163","DOI":"10.1109\/ACCESS.2021.3050934","article-title":"A Bayesian Optimization Approach for Water Resources Monitoring through an Autonomous Surface Vehicle: The Ypacarai Lake Case Study","volume":"9","author":"Reina","year":"2021","journal-title":"IEEE Access"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1016\/j.comcom.2018.02.002","article-title":"A survey on unmanned aerial and aquatic vehicle multi-hop networks: Wireless communications, evaluation tools and applications","volume":"119","author":"Arzamendia","year":"2018","journal-title":"Comput. Commun."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Peralta, F., Arzamendia, M.E., Gregor, D., Reina, D.G., and Toral, S. (2020). A Comparison of Local Path Planning Techniques of Autonomous Surface Vehicles for Monitoring Applications: The Ypacarai Lake Case-study. Sensors, 20.","DOI":"10.3390\/s20051488"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"204076","DOI":"10.1109\/ACCESS.2020.3036938","article-title":"A Deep Reinforcement Learning Approach for the Patrolling Problem of Water Resources Through Autonomous Surface Vehicles: The Ypacarai Lake Case","volume":"8","author":"Yanes","year":"2020","journal-title":"IEEE Access"},{"key":"ref_7","unstructured":"Chevaleyre, Y. (2004, January 20\u201324). Theoretical analysis of the multi-agent patrolling problem. Proceedings of the IEEE\/WIC\/ACM International Conference on Intelligent Agent Technology, Beijing, China."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Arzamendia, M., Gregor, D., Reina, D., and Toral, S. (2016, January 7\u20139). Evolutionary path planning of an autonomous surface vehicle for water quality monitoring. Proceedings of the 2016 9th International Conference on Developments in eSystems Engineering (DeSE), Brussels, Belgium.","DOI":"10.1109\/DeSE.2016.39"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Ferreira, H., Almeida, C., Martins, A., Almeida, J., Dias, N., Dias, A., and Silva, E. (2009, January 11\u201314). Autonomous bathymetry for risk assessment with ROAZ robotic surface vehicle. Proceedings of the Oceans 2009-Europe, Bremen, Germany.","DOI":"10.1109\/OCEANSE.2009.5278235"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Eichhorn, M., Ament, C., Jacobi, M., Pfuetzenreuter, T., Karimanzira, D., Bley, K., Boer, M., and Wehde, H. (2018). Modular AUV System with Integrated Real-Time Water Quality Analysis. Sensors, 18.","DOI":"10.3390\/s18061837"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Chang, H.C., Hsu, Y.L., Hung, S.S., Ou, G.R., Wu, J.R., and Hsu, C. (2021). Autonomous Water Quality Monitoring and Water Surface Cleaning for Unmanned Surface Vehicle. Sensors, 21.","DOI":"10.3390\/s21041102"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Theile, M., Bayerlein, H., Nai, R., Gesbert, D., and Caccamo, M. (2020). UAV Coverage Path Planning under Varying Power Constraints using Deep Reinforcement Learning. arXiv.","DOI":"10.1109\/IROS45743.2020.9340934"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Piciarelli, C., and Foresti, G.L. (2019, January 9\u201311). Drone patrolling with reinforcement learning. Proceedings of the 13th International Conference on Distributed Smart Cameras, Trento, Italy.","DOI":"10.1145\/3349801.3349805"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"180","DOI":"10.1016\/j.procs.2018.01.113","article-title":"Genetic Algorithm Based Approach for Autonomous Mobile Robot Path Planning","volume":"127","author":"Lamini","year":"2018","journal-title":"Procedia Comput. Sci."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1748","DOI":"10.3182\/20140824-6-ZA-1003.01073","article-title":"Optimization of Robotic Arm Trajectory Using Genetic Algorithm","volume":"47","author":"Sekaj","year":"2014","journal-title":"IFAC Proc. Vol."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1723","DOI":"10.1007\/s00500-017-2895-x","article-title":"An evolutionary approach to constrained path planning of an autonomous surface vehicle for maximizing the covered area of Ypacarai Lake","volume":"23","author":"Arzamendia","year":"2019","journal-title":"Soft Comput."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"103078","DOI":"10.1016\/j.autcon.2020.103078","article-title":"Complete coverage path planning using reinforcement learning for Tetromino based cleaning and maintenance robot","volume":"112","author":"Ramalingam","year":"2020","journal-title":"Autom. Constr."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Van Hasselt, H., Guez, A., and Silver, D. (2016, January 12\u201317). Deep reinforcement learning with double Q-Learning. Proceedings of the 30th AAAI Conference on Artificial Intelligence, Phoenix, AZ, USA. Available online: http:\/\/xxx.lanl.gov\/abs\/1509.06461.","DOI":"10.1609\/aaai.v30i1.10295"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Apuroop, K., Le, A.V., Elara, M.R., and Sheu, B.J. (2021). Reinforcement Learning-Based Complete Area Coverage Path Planning for a Modified hTrihex Robot. Sensors, 21.","DOI":"10.3390\/s21041067"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Shi, W., Li, J., Wu, H., Zhou, C., Cheng, N., and Shen, X. (2020). Drone-cell Trajectory Planning and Resource Allocation for Highly Mobile Networks: A Hierarchical DRL Approach. IEEE Internet Things J.","DOI":"10.1109\/JIOT.2020.3020067"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Anderlini, E., Parker, G.G., and Thomas, G. (2019). Docking Control of an Autonomous Underwater Vehicle Using Reinforcement Learning. Appl. Sci., 9.","DOI":"10.3390\/app9173456"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"12786","DOI":"10.1109\/TVT.2020.3025627","article-title":"Battery-Involved Energy Management for Hybrid Electric Bus Based on Expert-Assistance Deep Deterministic Policy Gradient Algorithm","volume":"69","author":"Wu","year":"2020","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"3751","DOI":"10.1109\/TII.2020.3014599","article-title":"Battery Thermal- and Health-Constrained Energy Management for Hybrid Electric Bus Based on Soft Actor-Critic DRL Algorithm","volume":"17","author":"Wu","year":"2021","journal-title":"IEEE Trans. Ind. Inform."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"17084","DOI":"10.1109\/ACCESS.2021.3053348","article-title":"A Multiagent Deep Reinforcement Learning Approach for Path Planning in Autonomous Surface Vehicles: The Ypacarai-Lake Patrolling Case","volume":"9","author":"Luis","year":"2021","journal-title":"IEEE Access"},{"key":"ref_26","first-page":"6380","article-title":"Multi-agent actor-critic for mixed cooperative-competitive environments","volume":"2017","author":"Lowe","year":"2017","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Chu, T., Qu, S., and Wang, J. (2016, January 12\u201314). Large-scale multi-agent reinforcement learning using image-based state representation. Proceedings of the 2016 IEEE 55th Conference on Decision and Control, Las Vegas, NV, USA.","DOI":"10.1109\/CDC.2016.7799442"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"146264","DOI":"10.1109\/ACCESS.2019.2943253","article-title":"Joint Optimization of Multi-UAV Target Assignment and Path Planning Based on Multi-Agent Reinforcement Learning","volume":"7","author":"Qie","year":"2019","journal-title":"IEEE Access"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Tran, Q., and Bae, S. (2021). An Efficiency Enhancing Methodology for Multiple Autonomous Vehicles in an Urban Network Adopting Deep Reinforcement Learning. Appl. Sci., 11.","DOI":"10.3390\/app11041514"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Dastbaz, M., Arabnia, H., and Akhgar, B. (2018). A Path Planning Approach of an Autonomous Surface Vehicle for Water Quality Monitoring Using Evolutionary Computation. Technology for Smart Futures, Springer International Publishing.","DOI":"10.1007\/978-3-319-60137-3"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"106209","DOI":"10.1016\/j.knosys.2020.106209","article-title":"A constrained differential evolution algorithm to solve UAV path planning in disaster scenarios","volume":"204","author":"Yu","year":"2020","journal-title":"Knowl. Based Syst."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Wang, L., and Guo, Q. (2019). Coordination of Multiple Autonomous Agents Using Naturally Generated Languages in Task Planning. Appl. Sci., 9.","DOI":"10.3390\/app9173571"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Cabreira, T.M., de Aguiar, M.S., and Dimuro, G.P. (2013, January 20\u201323). An Extended Evolutionary Learning Approach For Multiple Robot Path Planning In A Multi-Agent Environment. Proceedings of the 2013 IEEE Congress on Evolutionary Computation, Cancun, Mexico.","DOI":"10.1109\/CEC.2013.6557982"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"103","DOI":"10.1016\/0967-0661(94)90578-9","article-title":"Coordination in evolutionary multi-agent-robotic system using fuzzy and genetic algorithm","volume":"2","author":"Shibata","year":"1994","journal-title":"Control Eng. Pract."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Li, H., Yi, F., Elgazzar, K., and Paull, L. (2009, January 16\u201317). Path planning for multiple Unmanned Aerial Vehicles using genetic algorithms. Proceedings of the 2009 Canadian Conference on Electrical and Computer Engineering, Vancouver, BC, Canada.","DOI":"10.1109\/CCECE.2009.5090303"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"228","DOI":"10.1016\/j.swevo.2018.03.011","article-title":"Reinforcement learning versus evolutionary computation: A survey on hybrid algorithms","volume":"44","author":"Drugan","year":"2019","journal-title":"Swarm Evol. Comput."},{"key":"ref_37","first-page":"1609","article-title":"A Convergent O(n) Temporal-difference Algorithm for Off-policy Learning with Linear Function Approximation","volume":"Volume 21","author":"Koller","year":"2009","journal-title":"Advances in Neural Information Processing Systems"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Taylor, M.E., Whiteson, S., and Stone, P. (2006, January 8\u201312). Comparing Evolutionary and Temporal Difference Methods in a Reinforcement Learning Domain. Proceedings of the 8th Annual Conference on Genetic and Evolutionary Computation, Washington, DC, USA.","DOI":"10.1145\/1143997.1144202"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"717","DOI":"10.2478\/v10006-011-0057-3","article-title":"Evolving Small-Board Go Players using Coevolutionary Temporal Difference Learning with Archive","volume":"21","author":"Krawiec","year":"2011","journal-title":"Int. J. Appl. Math. Comput. Sci."},{"key":"ref_40","unstructured":"Ter-Sarkisov, A., Marsland, S.R., and Convergence Properties of Two (\u03bc+\u03bb) Evolutionary Algorithms On OneMax and Royal Roads Test Functions (2021, January 22). CoRR 2011, Available online: http:\/\/xxx.lanl.gov\/abs\/1108.4080."},{"key":"ref_41","unstructured":"Wu, Y.H., Sun, F.Y., Chang, Y.Y., and Lin, S.D. (2018). ANS: Adaptive network scaling for deep rectifier reinforcement learning models. arXiv."},{"key":"ref_42","unstructured":"Luke, S. (2013). Essentials of Metaheuristics, Lulu. [2nd ed.]. Available online: http:\/\/cs.gmu.edu\/sean\/book\/metaheuristics\/."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Hassanat, A., Almohammadi, K., Alkafaween, E., Abunawas, E., Hammouri, A., and Prasath, V.B.S. (2019). Choosing mutation and crossover ratios for genetic algorithms-a review with a new dynamic approach. Information, 10.","DOI":"10.3390\/info10120390"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Birattari, M. (2009). Tuning Metaheuristics: A Machine Learning Perspective, Springer Publishing Company. [2nd ed.].","DOI":"10.1007\/978-3-642-00483-4_4"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Liessner, R., Schmitt, J., Dietermann, A., and B\u00e4ker, B. (2019, January 19\u201321). Hyperparameter Optimization for Deep Reinforcement Learning in Vehicle Energy Management. Proceedings of the 11th International Conference on Agents and Artificial Intelligence, Prague, Czech Republic.","DOI":"10.5220\/0007364701340144"},{"key":"ref_46","unstructured":"Li, H., Chaudhari, P., Yang, H., Lam, M., Ravichandran, A., Bhotika, R., and Soatto, S. (2021, April 10). Rethinking the Hyperparameters for Fine-Tuning, Available online: http:\/\/xxx.lanl.gov\/abs\/2002.11770."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"156","DOI":"10.1109\/TSMCC.2007.913919","article-title":"A comprehensive survey of multiagent reinforcement learning","volume":"38","year":"2008","journal-title":"IEEE Trans. Syst. Man Cybern. Part C Appl. Rev."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1016\/j.jpdc.2019.07.008","article-title":"Distributed Bayesian optimization of deep reinforcement learning algorithms","volume":"139","author":"Young","year":"2020","journal-title":"J. Parallel Distrib. Comput."},{"key":"ref_49","unstructured":"Vincenzo, N., and Gasparetto, A. (2018). Working Cycle Sequence Optimization for Industrial Robots. Advances in Italian Mechanism Sciences, Springer International Publishing."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/8\/2862\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:49:40Z","timestamp":1760161780000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/8\/2862"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,4,19]]},"references-count":49,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2021,4]]}},"alternative-id":["s21082862"],"URL":"https:\/\/doi.org\/10.3390\/s21082862","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,4,19]]}}}