{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,18]],"date-time":"2026-06-18T15:36:55Z","timestamp":1781797015238,"version":"3.54.5"},"reference-count":49,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2023,6,15]],"date-time":"2023-06-15T00:00:00Z","timestamp":1686787200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100002855","name":"National Key R&amp;D Program of China","doi-asserted-by":"publisher","award":["2022YFE0114200"],"award-info":[{"award-number":["2022YFE0114200"]}],"id":[{"id":"10.13039\/501100002855","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>This paper presents a multi-agent reinforcement learning (MARL) algorithm to address the scheduling and routing problems of multiple automated guided vehicles (AGVs), with the goal of minimizing overall energy consumption. The proposed algorithm is developed based on the multi-agent deep deterministic policy gradient (MADDPG) algorithm, with modifications made to the action and state space to fit the setting of AGV activities. While previous studies overlooked the energy efficiency of AGVs, this paper develops a well-designed reward function that helps to optimize the overall energy consumption required to fulfill all tasks. Moreover, we incorporate the e-greedy exploration strategy into the proposed algorithm to balance exploration and exploitation during training, which helps it converge faster and achieve better performance. The proposed MARL algorithm is equipped with carefully selected parameters that aid in avoiding obstacles, speeding up path planning, and achieving minimal energy consumption. To demonstrate the effectiveness of the proposed algorithm, three types of numerical experiments including the \u03f5-greedy MADDPG, MADDPG, and Q-Learning methods were conducted. The results show that the proposed algorithm can effectively solve the multi-AGV task assignment and path planning problems, and the energy consumption results show that the planned routes can effectively improve energy efficiency.<\/jats:p>","DOI":"10.3390\/s23125615","type":"journal-article","created":{"date-parts":[[2023,6,16]],"date-time":"2023-06-16T02:54:33Z","timestamp":1686884073000},"page":"5615","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":27,"title":["Toward Energy-Efficient Routing of Multiple AGVs with Multi-Agent Reinforcement Learning"],"prefix":"10.3390","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4168-1183","authenticated-orcid":false,"given":"Xianfeng","family":"Ye","sequence":"first","affiliation":[{"name":"School of Mechanical Science and Engineering, Huazhong University of Science and Technology, Wuhan 430074, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhiyun","family":"Deng","sequence":"additional","affiliation":[{"name":"School of Mechanical Science and Engineering, Huazhong University of Science and Technology, Wuhan 430074, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yanjun","family":"Shi","sequence":"additional","affiliation":[{"name":"Department of Mechanical Engineering, Dalian University of Technology, Dalian 116023, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5204-7992","authenticated-orcid":false,"given":"Weiming","family":"Shen","sequence":"additional","affiliation":[{"name":"School of Mechanical Science and Engineering, Huazhong University of Science and Technology, Wuhan 430074, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2023,6,15]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"152","DOI":"10.1016\/j.jmsy.2019.12.002","article-title":"Automated guided vehicle systems, state-of-the-art control algorithms and techniques","volume":"54","author":"Versteyhe","year":"2020","journal-title":"J. Manuf. Syst."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"109","DOI":"10.1049\/iet-cim.2019.0007","article-title":"Potential applications of 5G communication technologies in collaborative intelligent manufacturing","volume":"1","author":"Shi","year":"2019","journal-title":"IET Collab. Intell. Manuf."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1045","DOI":"10.1109\/LRA.2019.2894001","article-title":"New automated guided vehicle system using real-time holonic scheduling for warehouse picking","volume":"4","author":"Yoshitake","year":"2019","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"96","DOI":"10.1049\/iet-cim.2020.0041","article-title":"Address business crisis caused by COVID-19 with collaborative intelligent manufacturing technologies","volume":"2","author":"Shen","year":"2020","journal-title":"IET Collab. Intell. Manuf."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Ahmed, S.U., Affan, M., Raza, M.I., and Hashmi, M.H. (2022, January 12\u201313). Inspecting Mega Solar Plants through Computer Vision and Drone Technologies. Proceedings of the 2022 International Conference on Frontiers of Information Technology (FIT), Islamabad, Pakistan.","DOI":"10.1109\/FIT57066.2022.00014"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"120640","DOI":"10.1016\/j.apenergy.2023.120640","article-title":"Real-time energy optimization and scheduling of buildings integrated with renewable microgrid","volume":"335","author":"Alzahrani","year":"2023","journal-title":"Appl. Energy"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"136472","DOI":"10.1016\/j.jclepro.2023.136472","article-title":"Flexible time\u2013space network formulation and hybrid metaheuristic for conflict-free and energy-efficient path planning of automated guided vehicles","volume":"398","author":"Xin","year":"2023","journal-title":"J. Clean. Prod."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1080\/00207543.2021.1998695","article-title":"Anti-conflict AGV path planning in automated container terminals based on multi-agent reinforcement learning","volume":"61","author":"Hu","year":"2023","journal-title":"Int. J. Prod. Res."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"7882","DOI":"10.1109\/TII.2020.3034280","article-title":"Cyber-physical system-based heuristic planning and scheduling method for multiple automatic guided vehicles in logistics systems","volume":"17","author":"Lian","year":"2020","journal-title":"IEEE Trans. Ind. Inform."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"3686","DOI":"10.1109\/TFUZZ.2021.3053838","article-title":"Fuzzy integrated cell formation and production scheduling considering automated guided vehicles and human factors","volume":"29","author":"Goli","year":"2021","journal-title":"IEEE Trans. Fuzzy Syst."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"90","DOI":"10.1016\/j.cie.2019.01.023","article-title":"A three-stage decomposition method for the joint vehicle dispatching and storage allocation problem in automated container terminals","volume":"129","author":"Hu","year":"2019","journal-title":"Comput. Ind. Eng."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"126019","DOI":"10.1016\/j.jclepro.2021.126019","article-title":"Optimizing configuration and scheduling of double 40 ft dual-trolley quay cranes and AGVs for improving container terminal services","volume":"292","author":"Yue","year":"2021","journal-title":"J. Clean. Prod."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"707","DOI":"10.1080\/00207543.2021.2015806","article-title":"Efficient path planning for automated guided vehicles using A*(Astar) algorithm incorporating turning costs in search heuristic","volume":"61","author":"Fransen","year":"2023","journal-title":"Int. J. Prod. Res."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"966","DOI":"10.1109\/TASE.2019.2952920","article-title":"Cell-based local search heuristics for guide path design of automated guided vehicle systems with dynamic multicommodity flow","volume":"17","author":"Nishi","year":"2019","journal-title":"IEEE Trans. Autom. Sci. Eng."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"624","DOI":"10.1080\/00207543.2018.1475761","article-title":"Comparative analysis of different routing heuristics for the battery management of automated guided vehicles","volume":"57","author":"Kabir","year":"2019","journal-title":"Int. J. Prod. Res."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"106371","DOI":"10.1016\/j.cie.2020.106371","article-title":"Multi-AGV scheduling for conflict-free path planning in automated container terminals","volume":"142","author":"Zhong","year":"2020","journal-title":"Comput. Ind. Eng."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"113675","DOI":"10.1016\/j.eswa.2020.113675","article-title":"An effective discrete artificial bee colony algorithm for multi-AGVs dispatching problem in a matrix manufacturing workshop","volume":"161","author":"Zou","year":"2020","journal-title":"Expert Syst. Appl."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"93","DOI":"10.1049\/cim2.12016","article-title":"Scheduling multi\u2013mode resource\u2013constrained tasks of automated guided vehicles with an improved particle swarm optimization algorithm","volume":"3","author":"Xiao","year":"2021","journal-title":"IET Collab. Intell. Manuf."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"67","DOI":"10.1049\/iet-cim.2018.0009","article-title":"Review on flexible job shop scheduling","volume":"1","author":"Xie","year":"2019","journal-title":"IET Collab. Intell. Manuf."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"106749","DOI":"10.1016\/j.cie.2020.106749","article-title":"Deep reinforcement learning based AGVs real-time scheduling with mixed rule for flexible shop floor in industry 4.0","volume":"149","author":"Hu","year":"2020","journal-title":"Comput. Ind. Eng."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1049\/cim2.12010","article-title":"Digital Twin models in industrial operations: State-of-the-art and future research directions","volume":"3","author":"Melesse","year":"2021","journal-title":"IET Collab. Intell. Manuf."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"102202","DOI":"10.1016\/j.rcim.2021.102202","article-title":"Multi-agent reinforcement learning for online scheduling in smart factories","volume":"72","author":"Zhou","year":"2021","journal-title":"Robot. Comput.-Integr. Manuf."},{"key":"ref_23","unstructured":"Russell, S.J., and Norvig, P. (2016). Artificial Intelligence: A Modern Approach, Pearson Education Limited."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Lu, C., Long, J., Xing, Z., Wu, W., Gu, Y., Luo, J., and Huang, Y. (2020, January 26\u201327). Deep Reinforcement Learning for Solving AGVs Routing Problem. Proceedings of the International Conference on Verification and Evaluation of Computer and Communication Systems, Xi\u2019an, China.","DOI":"10.1007\/978-3-030-65955-4_16"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"2252015","DOI":"10.1142\/S0218001422520152","article-title":"Multi-AGV Task allocation with Attention Based on Deep Reinforcement Learning","volume":"36","author":"Yin","year":"2022","journal-title":"Int. J. Pattern Recognit. Artif. Intell."},{"key":"ref_26","first-page":"V001T04A001","article-title":"A Conflict-Free Routing Method for Automated Guided Vehicles Using Reinforcement Learning","volume":"Volume 83617","author":"Chujo","year":"2020","journal-title":"Proceedings of the International Symposium on Flexible Automation"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Yan, J., Liu, Z., Zhang, T., and Zhang, Y. (2021, January 9\u201311). Autonomous decision-making method of transportation process for flexible job shop scheduling problem based on reinforcement learning. Proceedings of the 2021 International Conference on Machine Learning and Intelligent Systems Engineering (MLISE), Chongqing, China.","DOI":"10.1109\/MLISE54096.2021.00049"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Zhang, H., Luo, J., Lin, X., Tan, K., and Pan, C. (2021, January 3\u20135). Dispatching and Path Planning of Automated Guided Vehicles based on Petri Nets and Deep Reinforcement Learning. Proceedings of the 2021 IEEE International Conference on Networking, Sensing and Control (ICNSC), Xiamen, China.","DOI":"10.1109\/ICNSC52481.2021.9702196"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Liu, H., Hyodo, A., Akai, A., Sakaniwa, H., and Suzuki, S. (2021, January 14\u201316). Action-limited, Multimodal Deep Q Learning for AGV Fleet Route Planning. Proceedings of the 5th International Conference on Control Engineering and Artificial Intelligence, Sanya, China.","DOI":"10.1145\/3448218.3448219"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Nagayoshi, M., Elderton, S., Sakakibara, K., and Tamaki, H. (2017, January 19\u201322). Adaptive Negotiation-rules Acquisition Methods in Decentralized AGV Transportation Systems by Reinforcement Learning with a State Space Filter. Proceedings of the International Conference on Artificial Life and Robotics, ICAROB 2017, Miyazaki, Japan.","DOI":"10.5954\/ICAROB.2017.GS2-1"},{"key":"ref_31","unstructured":"Li, M.P. (2021). Task Assignment and Path Planning for Autonomous Mobile Robots in Stochastic Warehouse Systems. [Ph.D. Thesis, Rochester Institute of Technology]."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Xue, T., Zeng, P., and Yu, H. (2018, January 19\u201322). A reinforcement learning method for multi-AGV scheduling in manufacturing. Proceedings of the 2018 IEEE International Conference on Industrial Technology (ICIT), Lyon, France.","DOI":"10.1109\/ICIT.2018.8352413"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"948","DOI":"10.20965\/jaciii.2017.p0948","article-title":"Reinforcement Learning Approach for Adaptive Negotiation-Rules Acquisition in AGV Transportation Systems","volume":"21","author":"Nagayoshi","year":"2017","journal-title":"J. Adv. Comput. Intell. Intell. Inform."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Sierra-Garcia, J.E., and Santos, M. (2022). Combining reinforcement learning and conventional control to improve automatic guided vehicles tracking of complex trajectories. Expert Syst., e13076.","DOI":"10.1111\/exsy.13076"},{"key":"ref_35","unstructured":"Zhang, Y., Qian, Y., Yao, Y., Hu, H., and Xu, Y. (2020, January 9\u201313). Learning to cooperate: Application of deep reinforcement learning for online AGV path finding. Proceedings of the 19th International Conference on Autonomous Agents and Multiagent Systems, Auckland, New Zealand."},{"key":"ref_36","first-page":"53","article-title":"Online optimization of AGV transport systems using deep reinforcement learning","volume":"9","author":"Takahashi","year":"2020","journal-title":"Bull. Netw. Comput. Syst. Softw."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"1523","DOI":"10.1016\/j.procir.2021.11.257","article-title":"Simultaneous production and agv scheduling using multi-agent deep reinforcement learning","volume":"104","author":"Popper","year":"2021","journal-title":"Procedia CIRP"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Li, M., Guo, B., Zhang, J., Liu, J., Liu, S., Yu, Z., Li, Z., and Xiang, L. (2021, January 4\u20137). Decentralized Multi-AGV Task Allocation based on Multi-Agent Reinforcement Learning with Information Potential Field Rewards. Proceedings of the 2021 IEEE 18th International Conference on Mobile Ad Hoc and Smart Systems (MASS), Denver, CO, USA.","DOI":"10.1109\/MASS52906.2021.00066"},{"key":"ref_39","unstructured":"Zhang, K., Yang, Z., and Ba\u015far, T. (2021). Handbook of Reinforcement Learning and Control, Springer."},{"key":"ref_40","unstructured":"Littman, M.L. (1994). Machine Learning Proceedings 1994, Elsevier."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"354","DOI":"10.1109\/TRO.2004.838026","article-title":"Fast replanning for navigation in unknown terrain","volume":"21","author":"Koenig","year":"2005","journal-title":"IEEE Trans. Robot."},{"key":"ref_42","unstructured":"Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., and Mordatch, I. (2017, January 6\u20139). Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments. Proceedings of the Neural Information Processing Systems (NIPS), Long Beach, CA, USA."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Mordatch, I., and Abbeel, P. (2017). Emergence of Grounded Compositional Language in Multi-Agent Populations. arXiv.","DOI":"10.1609\/aaai.v32i1.11492"},{"key":"ref_44","unstructured":"Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M. (2014, January 21\u201326). Deterministic policy gradient algorithms. Proceedings of the International Conference on Machine Learning, Beijing, China."},{"key":"ref_45","unstructured":"Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2015). Continuous control with deep reinforcement learning. arXiv."},{"key":"ref_46","unstructured":"Wunder, M., Littman, M.L., and Babes, M. (2010, January 21\u201324). Classes of multiagent q-learning dynamics with epsilon-greedy exploration. Proceedings of the 27th International Conference on Machine Learning (ICML-10), Haifa, Israel."},{"key":"ref_47","unstructured":"Dann, C., Mansour, Y., Mohri, M., Sekhari, A., and Sridharan, K. (2022, January 17\u201323). Guarantees for epsilon-greedy reinforcement learning with function approximation. Proceedings of the International Conference on Machine Learning, Baltimore, MD, USA."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Wadhwania, S., Kim, D.K., Omidshafiei, S., and How, J.P. (2019, January 3\u20138). Policy distillation and value matching in multiagent reinforcement learning. Proceedings of the 2019 IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS), Macau, China.","DOI":"10.1109\/IROS40897.2019.8967849"},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3543846","article-title":"Reinforcement learning based recommender systems: A survey","volume":"55","author":"Afsar","year":"2022","journal-title":"Acm Comput. Surv."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/12\/5615\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T19:55:48Z","timestamp":1760126148000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/12\/5615"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,15]]},"references-count":49,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2023,6]]}},"alternative-id":["s23125615"],"URL":"https:\/\/doi.org\/10.3390\/s23125615","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,6,15]]}}}