{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,4]],"date-time":"2026-04-04T04:31:37Z","timestamp":1775277097529,"version":"3.50.1"},"reference-count":65,"publisher":"MDPI AG","issue":"21","license":[{"start":{"date-parts":[[2022,11,4]],"date-time":"2022-11-04T00:00:00Z","timestamp":1667520000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000001","name":"Geotechnical Engineering and Materials Program of the National Science Foundation","doi-asserted-by":"publisher","award":["1742656"],"award-info":[{"award-number":["1742656"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"Geotechnical Engineering and Materials Program of the National Science Foundation","doi-asserted-by":"publisher","award":["ACI-1548562"],"award-info":[{"award-number":["ACI-1548562"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["1742656"],"award-info":[{"award-number":["1742656"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["ACI-1548562"],"award-info":[{"award-number":["ACI-1548562"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>The occurrence of landslides has been increasing in recent years due to intense and prolonged rainfall events. Lowering the groundwater in natural and man-made slopes can help to mitigate the hazards. Subsurface drainage systems equipped with pumps have traditionally been regarded as a temporary remedy for lowering the groundwater in geosystems, whereas long-term usage of pumping-based techniques is uncommon due to the associated high operational costs in labor and energy. This study investigates the intelligent control of groundwater in slopes enabled by deep reinforcement learning (DRL), a subfield of machine learning for automated decision-making. The purpose is to develop an autonomous geosystem that can minimize the operating cost and enhance the system\u2019s safety without introducing human errors and interventions. To prove the concept, a seepage analysis model was implemented using a partial differential equation solver, FEniCS, to simulate the geosystem (i.e., a slope equipped with a pump and subjected to rainfall events). A Deep Q-Network (i.e., a DRL learning agent) was trained to learn the optimal control policy for regulating the pump\u2019s flow rate. The objective is to enable intermittent control of the pump\u2019s flow rate (i.e., 0%, 25%, 50%, 75%, and 100% of the pumping capacity) to keep the groundwater close to the target level during rainfall events and consequently help to prevent slope failure. A comparison of the results with traditional proportional-integral-derivative-controlled and uncontrolled water tables showed that the geosystem integrated with DRL can dynamically adapt its response to diverse weather events by adjusting the pump\u2019s flow rate and improve the adopted control policy by gaining more experience over time. In addition, it was observed that the DRL control helped to mitigate slope failure during rainfall events.<\/jats:p>","DOI":"10.3390\/s22218503","type":"journal-article","created":{"date-parts":[[2022,11,4]],"date-time":"2022-11-04T09:26:00Z","timestamp":1667553960000},"page":"8503","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["Intelligent Control of Groundwater in Slopes with Deep Reinforcement Learning"],"prefix":"10.3390","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8610-6185","authenticated-orcid":false,"given":"Aynaz","family":"Biniyaz","sequence":"first","affiliation":[{"name":"Department of Civil, Environmental, and Geospatial Engineering, Michigan Technological University, Houghton, MI 49931, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1734-2176","authenticated-orcid":false,"given":"Behnam","family":"Azmoon","sequence":"additional","affiliation":[{"name":"Department of Civil, Environmental, and Geospatial Engineering, Michigan Technological University, Houghton, MI 49931, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8013-3804","authenticated-orcid":false,"given":"Zhen","family":"Liu","sequence":"additional","affiliation":[{"name":"Department of Civil, Environmental, and Geospatial Engineering, Michigan Technological University, Houghton, MI 49931, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,11,4]]},"reference":[{"key":"ref_1","unstructured":"Schuster, R.L. (1996, January 27\u201328). The 25 most catastrophic landslides of the 20th century. Proceedings of the 8th International Conference and Field Trip on Landslides, Granada, Spain."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"927","DOI":"10.1130\/G33217.1","article-title":"Global patterns of loss of life from landslides","volume":"40","author":"Petley","year":"2012","journal-title":"Geology"},{"key":"ref_3","first-page":"69","article-title":"The impact of landslide on environment and socio-economy: GIS based study on Badulla district in Sri Lanka","volume":"88","author":"Kaleel","year":"2017","journal-title":"World Sci. News"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"93","DOI":"10.1016\/j.gsf.2020.07.012","article-title":"Landslide susceptibility modeling based on ANFIS with teaching-learning-based optimization and Satin bowerbird optimizer","volume":"12","author":"Chen","year":"2021","journal-title":"Geosci. Front."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1821","DOI":"10.5194\/nhess-15-1821-2015","article-title":"Regional trends and controlling factors of fatal landslides in Latin America and the Caribbean","volume":"15","author":"Petley","year":"2015","journal-title":"Nat. Hazards Earth Syst. Sci."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"921","DOI":"10.14419\/ijet.v7i2.29.14284","article-title":"The Rise of groundwater due to rainfall and the control of landslide by zero-energy groundwater withdrawal system","volume":"7","author":"Alsubal","year":"2018","journal-title":"Int. J. Eng. Technol."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"547","DOI":"10.1016\/j.ijthermalsci.2008.04.003","article-title":"Approximate analytical solution for one-dimensional tissue freezing around cylindrical cryoprobes","volume":"48","author":"Jiji","year":"2009","journal-title":"Int. J. Therm. Sci."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"184","DOI":"10.1016\/j.enggeo.2016.07.008","article-title":"Stability analysis of unsaturated soil slopes considering water-air flow caused by rainfall infiltration","volume":"211","author":"Cho","year":"2016","journal-title":"Eng. Geol."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"821","DOI":"10.1007\/s11242-015-0469-x","article-title":"Effects of airflow induced by rainfall infiltration on unsaturated soil slope stability","volume":"107","author":"Sun","year":"2015","journal-title":"Transp. Porous Media"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"e2019GL085347","DOI":"10.1029\/2019GL085347","article-title":"Changes in extreme precipitation and landslides over High Mountain Asia","volume":"47","author":"Kirschbaum","year":"2020","journal-title":"Geophys. Res. Lett."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"258","DOI":"10.1016\/j.iswcr.2017.07.001","article-title":"Effect of variations in rainfall intensity on slope stability in Singapore","volume":"5","author":"Kristo","year":"2017","journal-title":"Int. Soil Water Conserv. Res."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Cotterill, D., Stott, P., and Kendon, E. (2020, January 4\u20138). Increase in the frequency of heavy rainfall events over the UK in the light of climate change. Proceedings of the EGU General Assembly Conference Abstracts, Online.","DOI":"10.5194\/egusphere-egu2020-11025"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Nicholson, P.G. (2014). Soil Improvement and Ground Modification Methods, Butterworth-Heinemann.","DOI":"10.1016\/B978-0-12-408076-8.00012-1"},{"key":"ref_14","unstructured":"Turner, A.K., and Schuster, R.L. (1996). Landslides: Investigation and Mitigation, National Academy Press."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1583","DOI":"10.1007\/s10346-019-01202-x","article-title":"Drainage controls on the Donglingxing landslide (China) induced by rainfall and fluctuation in reservoir water levels","volume":"16","author":"Yan","year":"2019","journal-title":"Landslides"},{"key":"ref_16","first-page":"439","article-title":"Landslides: Investigation and Mitigation","volume":"247","author":"Holtz","year":"1996","journal-title":"Transp. Res. Board Spec. Rep."},{"key":"ref_17","unstructured":"Curden, D., and Varnes, D. (1992). Landslides: Investigation and Mitigation, Transportation Research Board."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Cashman, P.M., and Preene, M. (2001). Groundwater Lowering in Construction: A Practical Guide, CRC Press.","DOI":"10.4324\/9780203476321"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"351","DOI":"10.1016\/j.gsf.2020.02.012","article-title":"Landslide identification using machine learning","volume":"12","author":"Wang","year":"2021","journal-title":"Geosci. Front."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"150623","DOI":"10.1109\/ACCESS.2021.3123501","article-title":"Image-Data-Driven Slope Stability Analysis for Preventing Landslides Using Deep Learning","volume":"9","author":"Azmoon","year":"2021","journal-title":"IEEE Access"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"24112","DOI":"10.1038\/s41598-021-03585-1","article-title":"Deep learning-based landslide susceptibility mapping","volume":"11","author":"Azarafza","year":"2021","journal-title":"Sci. Rep."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"104426","DOI":"10.1016\/j.catena.2019.104426","article-title":"Comparing the prediction performance of a Deep Learning Neural Network model with conventional machine learning models in landslide susceptibility assessment","volume":"188","author":"Bui","year":"2020","journal-title":"Catena"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"395","DOI":"10.1016\/j.gsf.2020.04.003","article-title":"Deep learning based classification of rock structure of tunnel face","volume":"12","author":"Chen","year":"2021","journal-title":"Geosci. Front."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"14887","DOI":"10.1007\/s11042-021-10544-5","article-title":"A comprehensive review on soil classification using deep learning and computer vision techniques","volume":"80","author":"Srivastava","year":"2021","journal-title":"Multimed. Tools Appl."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"64186","DOI":"10.1109\/ACCESS.2019.2916330","article-title":"Real-time tunnel crack analysis system via deep learning","volume":"7","author":"Song","year":"2019","journal-title":"IEEE Access"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"638","DOI":"10.1111\/mice.12367","article-title":"A fast detection method via region-based fully convolutional neural networks for shield tunnel lining defects","volume":"33","author":"Xue","year":"2018","journal-title":"Comput. -Aided Civ. Infrastruct. Eng."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1237","DOI":"10.1007\/s12145-020-00508-y","article-title":"Predicting groundwater depth fluctuations using deep learning, extreme learning machine and Gaussian process: A comparative study","volume":"13","author":"Kumar","year":"2020","journal-title":"Earth Sci. Inform."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Shin, M.-J., Moon, S.-H., Kang, K.G., Moon, D.-C., and Koh, H.-J. (2020). Analysis of groundwater level variations caused by the changes in groundwater withdrawals using long short-term memory network. Hydrology, 7.","DOI":"10.3390\/hydrology7030064"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"3904","DOI":"10.1038\/s41598-020-60698-9","article-title":"A comparative study among machine learning and numerical models for simulating groundwater dynamics in the Heihe River Basin, northwestern China","volume":"10","author":"Chen","year":"2020","journal-title":"Sci. Rep."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"918","DOI":"10.1016\/j.jhydrol.2018.04.065","article-title":"Developing a Long Short-Term Memory (LSTM) based model for predicting water table depth in agricultural areas","volume":"561","author":"Zhang","year":"2018","journal-title":"J. Hydrol."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"101097","DOI":"10.1016\/j.aei.2020.101097","article-title":"Reinforcement learning based optimizer for improvement of predicting tunneling-induced ground responses","volume":"45","author":"Zhang","year":"2020","journal-title":"Adv. Eng. Inform."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"990","DOI":"10.1016\/j.undsp.2022.01.005","article-title":"The application of reinforcement learning to NATM tunnel design","volume":"7","author":"Soranzo","year":"2022","journal-title":"Undergr. Space"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"103701","DOI":"10.1016\/j.autcon.2021.103701","article-title":"Reinforcement learning based process optimization and strategy development in conventional tunneling","volume":"127","author":"Erharter","year":"2021","journal-title":"Autom. Constr."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Mysore, S., Mabsout, B., Mancuso, R., and Saenko, K. (June, January 30). Regularizing action policies for smooth control with reinforcement learning. Proceedings of the 2021 IEEE International Conference on Robotics and Automation (ICRA), Xi\u2019an, China.","DOI":"10.1109\/ICRA48506.2021.9561138"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Fran\u00e7ois-Lavet, V., Henderson, P., Islam, R., Bellemare, M.G., and Pineau, J. (2018). An introduction to deep reinforcement learning. arXiv.","DOI":"10.1561\/9781680835397"},{"key":"ref_37","unstructured":"Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., and Graepel, T. (2017). Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"354","DOI":"10.1038\/nature24270","article-title":"Mastering the game of go without human knowledge","volume":"550","author":"Silver","year":"2017","journal-title":"Nature"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"418","DOI":"10.1126\/science.aao1733","article-title":"Superhuman AI for heads-up no-limit poker: Libratus beats top professionals","volume":"359","author":"Brown","year":"2018","journal-title":"Science"},{"key":"ref_40","unstructured":"Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2015). Continuous control with deep reinforcement learning. arXiv."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"70","DOI":"10.2352\/ISSN.2470-1173.2017.19.AVM-023","article-title":"Deep reinforcement learning framework for autonomous driving","volume":"2017","author":"Sallab","year":"2017","journal-title":"Electron. Imaging"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"103600","DOI":"10.1016\/j.advwatres.2020.103600","article-title":"Deep reinforcement learning for the real time control of stormwater systems","volume":"140","author":"Mullapudi","year":"2020","journal-title":"Adv. Water Resour."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"115660","DOI":"10.1016\/j.apenergy.2020.115660","article-title":"Optimal carbon storage reservoir management through deep reinforcement learning","volume":"278","author":"Sun","year":"2020","journal-title":"Appl. Energy"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Biniyaz, A., Azmoon, B., and Liu, Z. (2022, January 20\u201323). Deep Reinforcement Learning for Controlling the Groundwater in Slopes. Proceedings of the Geo-Congress 2022, Charlotte, NC, USA.","DOI":"10.1061\/9780784484036.065"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"458","DOI":"10.1061\/(ASCE)0733-9496(2003)129:6(458)","article-title":"Neural networks and reinforcement learning in control of water systems","volume":"129","author":"Bhattacharya","year":"2003","journal-title":"J. Water Resour. Plan. Manag."},{"key":"ref_46","unstructured":"Sutton, R.S., and Barto, A.G. (1998). Introduction to Reinforcement Learning, MIT Press Cambridge."},{"key":"ref_47","doi-asserted-by":"crossref","first-page":"26","DOI":"10.1109\/MSP.2017.2743240","article-title":"Deep reinforcement learning: A brief survey","volume":"34","author":"Arulkumaran","year":"2017","journal-title":"IEEE Signal Process. Mag."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/1731022.1731030","article-title":"DOLFIN: Automated finite element computing","volume":"37","author":"Logg","year":"2010","journal-title":"ACM Trans. Math. Softw. (TOMS)"},{"key":"ref_49","first-page":"100","article-title":"The FEniCS project version 1.5","volume":"3","author":"Blechta","year":"2015","journal-title":"Arch. Numer. Softw."},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"2139","DOI":"10.1007\/s11440-021-01359-7","article-title":"Coupled transient saturated\u2013unsaturated seepage and limit equilibrium analysis for slopes: Influence of rapid water level changes","volume":"17","author":"Biniyaz","year":"2021","journal-title":"Acta Geotech."},{"key":"ref_51","doi-asserted-by":"crossref","first-page":"892","DOI":"10.2136\/sssaj1980.03615995004400050002x","article-title":"A closed-form equation for predicting the hydraulic conductivity of unsaturated soils","volume":"44","year":"1980","journal-title":"Soil Sci. Soc. Am. J."},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Wartalska, K., Ka\u017amierczak, B., Nowakowska, M., and Kotowski, A. (2020). Analysis of hyetographs for drainage system modeling. Water, 12.","DOI":"10.3390\/w12010149"},{"key":"ref_53","doi-asserted-by":"crossref","first-page":"113820","DOI":"10.1016\/j.eswa.2020.113820","article-title":"Multi-DQN: An ensemble of Deep Q-learning agents for stock market forecasting","volume":"164","author":"Carta","year":"2021","journal-title":"Expert Syst. Appl."},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"186","DOI":"10.1007\/s10916-018-1045-z","article-title":"Maintaining security and privacy in health care system using learning based deep-Q-networks","volume":"42","author":"Baskar","year":"2018","journal-title":"J. Med. Syst."},{"key":"ref_55","unstructured":"Zhou, N. (2020, January 18\u201319). Intelligent control of agricultural irrigation based on reinforcement learning. Proceedings of the Journal of Physics: Conference Series, Xi\u2019an, China."},{"key":"ref_56","doi-asserted-by":"crossref","first-page":"86886","DOI":"10.1109\/ACCESS.2020.2992480","article-title":"Crop yield prediction using deep reinforcement learning model for sustainable agrarian applications","volume":"8","author":"Elavarasan","year":"2020","journal-title":"IEEE Access"},{"key":"ref_57","doi-asserted-by":"crossref","unstructured":"Li, W., Wang, X., Zhang, R., Cui, Y., Mao, J., and Jin, R. (2010, January 24\u201328). Exploitation and exploration in a performance based contextual advertising system. Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Washington, DC, USA.","DOI":"10.1145\/1835804.1835811"},{"key":"ref_58","doi-asserted-by":"crossref","first-page":"933","DOI":"10.1098\/rstb.2007.2098","article-title":"Should I stay or should I go? How the human brain manages the trade-off between exploitation and exploration","volume":"362","author":"Cohen","year":"2007","journal-title":"Philos. Trans. R. Soc. B Biol. Sci."},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"1247","DOI":"10.5194\/gmd-7-1247-2014","article-title":"Root mean square error (RMSE) or mean absolute error (MAE)?\u2013Arguments against avoiding RMSE in the literature","volume":"7","author":"Chai","year":"2014","journal-title":"Geosci. Model Dev."},{"key":"ref_60","first-page":"1","article-title":"The design of the PID controller","volume":"8","author":"Paz","year":"2001","journal-title":"Klipsch Sch. Electr. Comput. Eng."},{"key":"ref_61","doi-asserted-by":"crossref","first-page":"559","DOI":"10.1109\/TCST.2005.847331","article-title":"PID control system analysis, design, and technology","volume":"13","author":"Ang","year":"2005","journal-title":"IEEE Trans. Control. Syst. Technol."},{"key":"ref_62","doi-asserted-by":"crossref","first-page":"142","DOI":"10.17485\/ijst\/2015\/v8i2\/58441","article-title":"Coordination of PSS and PID controller for power system stability enhancement\u2013overview","volume":"8","author":"Kasilingam","year":"2015","journal-title":"Indian J. Sci. Technol."},{"key":"ref_63","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1680\/geot.1955.5.1.7","article-title":"The use of the slip circle in the stability analysis of slopes","volume":"5","author":"Bishop","year":"1955","journal-title":"Geotechnique"},{"key":"ref_64","doi-asserted-by":"crossref","first-page":"379","DOI":"10.1139\/t96-060","article-title":"Model for the prediction of shear strength with respect to soil suction","volume":"33","author":"Vanapalli","year":"1996","journal-title":"Can. Geotech. J."},{"key":"ref_65","unstructured":"Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning, MIT Press."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/21\/8503\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T01:10:50Z","timestamp":1760145050000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/21\/8503"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,11,4]]},"references-count":65,"journal-issue":{"issue":"21","published-online":{"date-parts":[[2022,11]]}},"alternative-id":["s22218503"],"URL":"https:\/\/doi.org\/10.3390\/s22218503","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,11,4]]}}}