{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,10]],"date-time":"2026-06-10T16:37:45Z","timestamp":1781109465709,"version":"3.54.1"},"reference-count":36,"publisher":"MDPI AG","issue":"11","license":[{"start":{"date-parts":[[2018,11,9]],"date-time":"2018-11-09T00:00:00Z","timestamp":1541721600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["CNS-1453886, CNS-1551067, and ECCS-1651135"],"award-info":[{"award-number":["CNS-1453886, CNS-1551067, and ECCS-1651135"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>This work studies online learning-based trajectory planning for multiple autonomous underwater vehicles (AUVs) to estimate a water parameter field of interest in the under-ice environment. A centralized system is considered, where several fixed access points on the ice layer are introduced as gateways for communications between the AUVs and a remote data fusion center. We model the water parameter field of interest as a Gaussian process with unknown hyper-parameters. The AUV trajectories for sampling are determined on an epoch-by-epoch basis. At the end of each epoch, the access points relay the observed field samples from all the AUVs to the fusion center, which computes the posterior distribution of the field based on the Gaussian process regression and estimates the field hyper-parameters. The optimal trajectories of all the AUVs in the next epoch are determined to maximize a long-term reward that is defined based on the field uncertainty reduction and the AUV mobility cost, subject to the kinematics constraint, the communication constraint and the sensing area constraint. We formulate the adaptive trajectory planning problem as a Markov decision process (MDP). A reinforcement learning-based online learning algorithm is designed to determine the optimal AUV trajectories in a constrained continuous space. Simulation results show that the proposed learning-based trajectory planning algorithm has performance similar to a benchmark method that assumes perfect knowledge of the field hyper-parameters.<\/jats:p>","DOI":"10.3390\/s18113859","type":"journal-article","created":{"date-parts":[[2018,11,13]],"date-time":"2018-11-13T03:27:31Z","timestamp":1542079651000},"page":"3859","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":30,"title":["Reinforcement Learning-Based Multi-AUV Adaptive Trajectory Planning for Under-Ice Field Estimation"],"prefix":"10.3390","volume":"18","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4049-977X","authenticated-orcid":false,"given":"Chaofeng","family":"Wang","sequence":"first","affiliation":[{"name":"Department of Electrical and Computer Engineering, Michigan Technological University, Houghton, MI 49931, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Li","family":"Wei","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, Michigan Technological University, Houghton, MI 49931, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8771-0819","authenticated-orcid":false,"given":"Zhaohui\u00a0","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, Michigan Technological University, Houghton, MI 49931, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Min","family":"Song","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, Stevens Institute of Technology, Hoboken, NJ 07030, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nina","family":"Mahmoudian","sequence":"additional","affiliation":[{"name":"Department of Mechanical Engineering-Engineering Mechanics, Michigan Technological University, Houghton, MI 49931, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2018,11,9]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Thompson, D., Caress, D., Thomas, H., and Conlin, D. (2015, January 19\u201322). MBARI Mapping AUV Operations in the Gulf of California 2015. Proceedings of the Conference on MTS\/IEEE OCEANS, Washington, DC, USA.","DOI":"10.23919\/OCEANS.2015.7401816"},{"key":"ref_2","unstructured":"Thompson, D., Caress, D., Clague, D., Conlin, D., Harvey, J., Martin, E., Paduan, J., Paull, C., Ryan, J., and Thomas, H. (2013, January 23\u201327). MBARI Dorado AUV\u2019s Scientific Results. Proceedings of the Conference on MTS\/IEEE OCEANS, San Diego, CA, USA."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Kukulya, A., Plueddemann, A., Austin, T., Stokey, R., Purcell, M., Allen, B., Littlefield, R., Freitag, L., Koski, P., and Gallimore, E. (2010, January 1\u20133). Under-ice operations with a REMUS-100 AUV in the Arctic. Proceedings of the IEEE\/OES Autonomous Underwater Vehicles, Monterey, CA, USA.","DOI":"10.1109\/AUV.2010.5779661"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"48","DOI":"10.1109\/JPROC.2006.887295","article-title":"Collective motion, sensor networks, and ocean sampling","volume":"Volume 95","author":"Leonard","year":"2007","journal-title":"Proceedings of the IEEE;"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"522","DOI":"10.1109\/JOE.2008.2002105","article-title":"Path planning of autonomous underwater vehicles for adaptive sampling using mixed integer linear programming","volume":"33","author":"Yilmaz","year":"2008","journal-title":"IEEE J. Ocean. Eng."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"504","DOI":"10.1109\/TSMCB.2012.2210212","article-title":"Dynamic task assignment and path planning of multi-AUV system based on an improved self-organizing map and velocity synthesis method in three-dimensional underwater workspace","volume":"43","author":"Zhu","year":"2013","journal-title":"IEEE Trans. Cybern."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"10","DOI":"10.1109\/JOE.2012.2236491","article-title":"Trend and bounds for error growth in controlled Lagrangian particle tracking","volume":"39","author":"Szwaykowska","year":"2014","journal-title":"IEEE J. Ocean. Eng."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1118","DOI":"10.1109\/TRO.2011.2162766","article-title":"Mobile sensor network navigation using Gaussian processes with truncated observations","volume":"27","author":"Xu","year":"2011","journal-title":"IEEE Trans. Robot."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Marchant, R., and Ramos, F. (June, January 31). Bayesian optimisation for informative continuous path planning. Proceedings of the 2014 IEEE International Conference on Robotics and Automation (ICRA), Hong Kong, China.","DOI":"10.1109\/ICRA.2014.6907763"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"372","DOI":"10.1109\/TCST.2015.2435657","article-title":"Information-driven adaptive sampling strategy for mobile robotic wireless sensor network","volume":"24","author":"Nguyen","year":"2016","journal-title":"IEEE Trans. Control Syst. Technol."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"93","DOI":"10.1007\/s10514-009-9130-2","article-title":"A Bayesian exploration-exploitation approach for optimal online sensing and planning with a visually guided mobile robot","volume":"27","author":"Freitas","year":"2009","journal-title":"Auton. Robot."},{"key":"ref_12","unstructured":"Singh, A., Krause, A., and Kaiser, W. (2009, January 11\u201317). Nonmyopic adaptive informative path planning for multiple robots. Proceedings of the Twenty-First International Joint Conference on Artificial Intelligence (IJCAI-09), Pasadena, CA, USA."},{"key":"ref_13","unstructured":"Marchant, R., Ramos, F., and Sanner, S. (2014, January 23\u201317). Sequential Bayesian optimization for spatial-temporal monitoring. Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI), Quebec City, QC, Canada."},{"key":"ref_14","unstructured":"Morere, P., Marchant, R., and Ramos, F. (June, January 29). Sequential Bayesian optimization as a POMDP for environment monitoring with UAVs. Proceedings of the Conference on Robotics and Automation (ICRA), Singapore."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Binney, J., Krause, A., and Sukhatme, G.S. (2010, January 3\u20137). Informative path planning for an autonomous underwater vehicle. Proceedings of the Conference on Robotics and Automation (ICRA), Anchorage, AK, USA.","DOI":"10.1109\/ROBOT.2010.5509714"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Hollinger, G., Englot, B., Hover, F., Mitra, U., and Sukhatme, G. (2012, January 14\u201318). Uncertainty-driven view planning for underwater inspection. Proceedings of the Conference on Robotics and Automation (ICRA), Saint Paul, MN, USA.","DOI":"10.1109\/ICRA.2012.6224726"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"313","DOI":"10.1109\/TCST.2014.2312550","article-title":"A decentralized strategy for multirobot sampling\/patrolling: Theory and experiments","volume":"23","author":"Marino","year":"2015","journal-title":"IEEE Trans. Control Syst. Technol."},{"key":"ref_18","unstructured":"Kemna, S., Rogers, J.G., Nieto-Granda, C., Young, S., and Sukhatme, G.S. (June, January 29). Multi-robot coordination through dynamic Voronoi partitioning for informative adaptive sampling in communication-constrained environments. Proceedings of the Conference on Robotics and Automation (ICRA), Singapore."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Marino, A., and Antonelli, G. (2013, January 6\u201310). Experimental results of coordinated sampling\/patrolling by autonomous underwater vehicles. Proceedings of the Conference on IEEE Robotics and Automation (ICRA), Karlsruhe, Germany.","DOI":"10.1109\/ICRA.2013.6631161"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Rasmussen, C.E. (2004). Gaussian processes in machine learning. Advanced Lectures on Machine Learning, Springer Berlin Heidelberg.","DOI":"10.7551\/mitpress\/3206.001.0001"},{"key":"ref_21","unstructured":"Williams, C.K., and Rasmussen, C.E. (1996). Gaussian processes for regression. Advances in Neural Information Processing Systems, MIT Press."},{"key":"ref_22","first-page":"679","article-title":"A Markovian decision process","volume":"6","author":"Bellman","year":"1957","journal-title":"J. Math. Mech."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Karl, H., and Willig, A. (2005). Protocols and Architectures for Wireless Sensor Networks, John Wiley & Sons. [1st ed.].","DOI":"10.1002\/0470095121"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Brito, M.P., Lewis, R.S., Bose, N., and Griffiths, G. (2018). Adaptive autonomous underwater vehicles: An assessment of their effectiveness for oceanographic applications. IEEE Trans. Eng. Manage., 1\u201314.","DOI":"10.1109\/TEM.2018.2805159"},{"key":"ref_25","unstructured":"Mertikas, S.P. (1985). Error Distributions and Accuracy Measures in Navigation: An Overview, Geodesy and Geomatics Engineering. Technical Report."},{"key":"ref_26","unstructured":"Kay, S.M. (1993). Fundamentals of Statistical Signal Processing: Estimation Theory, Prentice Hall."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1190","DOI":"10.1137\/0916069","article-title":"A limited memory algorithm for bound constrained optimization","volume":"16","author":"Byrd","year":"1995","journal-title":"SIAM J. Sci. Comput."},{"key":"ref_28","unstructured":"Sutton, R.S., and Barto, A.G. (2017). Reinforcement Learning: An Introduction, The MIT Press."},{"key":"ref_29","unstructured":"Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2016, January 16\u201321). Continuous control with deep reinforcement learning. Proceedings of the Conference on Robotics and Automation (ICRA), Stockholm, Sweden."},{"key":"ref_30","unstructured":"Bishop, C.M. (2006). Pattern Recognition and Machine Learning, Springer. [6th ed.]."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Le Gall, F. (2014, January 23\u201325). Powers of tensors and fast matrix multiplication. Proceedings of the Symposium on Symbolic and Algebraic Computation, Kobe, Japan.","DOI":"10.1145\/2608628.2608664"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"367","DOI":"10.1080\/03610910601161298","article-title":"O(N2)-operation approximation of covariance matrix inverse in Gaussian process regression based on quasi-Newton BFGS Method","volume":"36","author":"Leithead","year":"2007","journal-title":"Commun. Stat. Simul. Comput."},{"key":"ref_33","unstructured":"Takefuji, Y. (2012). Neural Network Parallel Computing, Springer Science & Business Media."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Schmidt, V. (2015). Spatial process simulation. Stochastic Geometry, Spatial Statistics and Random Fields: Models and Algorithms, Springer.","DOI":"10.1007\/978-3-319-10064-7"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"443","DOI":"10.1109\/JOE.2004.838336","article-title":"Performance of an AUV navigation system at Arctic latitudes","volume":"30","author":"McEwen","year":"2005","journal-title":"IEEE J. Ocean. Eng."},{"key":"ref_36","first-page":"269","article-title":"Using autonomous underwater vehicles as sensor platforms for ice-monitoring","volume":"35","author":"Norgre","year":"2014","journal-title":"Model. Identif. Control"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/18\/11\/3859\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T15:28:57Z","timestamp":1760196537000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/18\/11\/3859"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,11,9]]},"references-count":36,"journal-issue":{"issue":"11","published-online":{"date-parts":[[2018,11]]}},"alternative-id":["s18113859"],"URL":"https:\/\/doi.org\/10.3390\/s18113859","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,11,9]]}}}