{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,2,21]],"date-time":"2025-02-21T22:09:23Z","timestamp":1740175763896,"version":"3.37.3"},"reference-count":40,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2024,1,10]],"date-time":"2024-01-10T00:00:00Z","timestamp":1704844800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,1,10]],"date-time":"2024-01-10T00:00:00Z","timestamp":1704844800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Jilin Province Major Science and  Technology Special Project \u201cResearch on Repeat Positioning Accuracy  Technology of AGV\u201d","award":["20210301028GX"],"award-info":[{"award-number":["20210301028GX"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Complex Intell. Syst."],"published-print":{"date-parts":[[2024,4]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>To address the challenges of traffic congestion and suboptimal operational efficiency in the context of large-scale applications like production plants and warehouses that utilize multiple automatic guided vehicles (multi-AGVs), this article proposed using an Improved Q-learning (IQL) algorithm and Macroscopic Fundamental Diagram (MFD) for the purposes of load balancing and congestion discrimination on road networks. Traditional Q-learning converges slowly, which is why we have proposed the use of an updated <jats:italic>Q<\/jats:italic> value of the previous iteration step as the maximum <jats:italic>Q<\/jats:italic> value of the next state to reduce the number of <jats:italic>Q<\/jats:italic> value comparisons and improve the algorithm\u2019s convergence speed. When calculating the cost of AGV operation, the traditional Q-learning algorithm only considers the evaluation function of a single distance and introduces an improved reward and punishment mechanism to combine the operating distance of AGV and the road network load, which finally equalizes the road network load. MFD is the basic property of road networks and is based on MFD, which is combined with the Markov Chain (MC) model. Road network traffic congestion state discrimination method was proposed to classify the congestion state according to the detected number of vehicles on the road network. The MC model accurately discriminated the range near the critical point. Finally, the scale of the road network and the load factor were changed for several simulations. The findings indicated that the improved algorithm showed a notable ability to achieve equilibrium in the load distribution of the road network. This led to a substantial enhancement in AGV operational efficiency.<\/jats:p>","DOI":"10.1007\/s40747-023-01278-y","type":"journal-article","created":{"date-parts":[[2024,1,10]],"date-time":"2024-01-10T09:02:11Z","timestamp":1704877331000},"page":"3025-3039","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Load balancing of multi-AGV road network based on improved Q-learning algorithm and macroscopic fundamental diagram"],"prefix":"10.1007","volume":"10","author":[{"given":"Xiumei","family":"Zhang","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0009-0007-3115-7819","authenticated-orcid":false,"given":"Wensong","family":"Li","sequence":"additional","affiliation":[]},{"given":"Hui","family":"Li","sequence":"additional","affiliation":[]},{"given":"Yue","family":"Liu","sequence":"additional","affiliation":[]},{"given":"Fang","family":"Liu","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2024,1,10]]},"reference":[{"issue":"11","key":"1278_CR1","doi-asserted-by":"publisher","DOI":"10.1088\/1361-6501\/ac8368","volume":"33","author":"H Tao","year":"2022","unstructured":"Tao H, Cheng L, Qiu J et al (2022) Few shot cross equipment fault diagnosis method based on parameter optimization and feature metric. Meas Sci Technol 33(11):115005","journal-title":"Meas Sci Technol"},{"issue":"1","key":"1278_CR2","doi-asserted-by":"publisher","first-page":"440","DOI":"10.1287\/ijoc.2021.1060","volume":"34","author":"M L\u00f6ffler","year":"2022","unstructured":"L\u00f6ffler M, Boysen N, Schneider M (2022) Picker routing in AGV-assisted order picking systems. INFORMS J Comput 34(1):440\u2013462. https:\/\/doi.org\/10.1287\/ijoc.2021.1060","journal-title":"INFORMS J Comput"},{"key":"1278_CR3","doi-asserted-by":"crossref","unstructured":"Pan F, Sun Q (2019) A traffic control strategy of the heavy-duty AGVS in a square topology. IEEE International Conference on Mechatronics and Automation (ICMA). IEEE, pp 263\u2013268","DOI":"10.1109\/ICMA.2019.8816435"},{"key":"1278_CR4","doi-asserted-by":"publisher","DOI":"10.1007\/s12204-022-2561-z","author":"Y Chen","year":"2022","unstructured":"Chen Y, Jiang Z (2022) Multi-AGVs scheduling with vehicle conflict consideration in ship outfitting items warehouse. J Shanghai Jiaotong Univ (Science). https:\/\/doi.org\/10.1007\/s12204-022-2561-z","journal-title":"J Shanghai Jiaotong Univ (Science)."},{"key":"1278_CR5","doi-asserted-by":"publisher","DOI":"10.3233\/ATDE220672","author":"BR Moser","year":"2022","unstructured":"Moser BR (2022) Machine learning and digital twin-sed path planning for AGVs at automated container terminals. Adv Transdisciplinary Eng. https:\/\/doi.org\/10.3233\/ATDE220672","journal-title":"Adv Transdisciplinary Eng"},{"key":"1278_CR6","doi-asserted-by":"crossref","unstructured":"Zheng T, Xu Y, Zheng D (2019) AGV path planning based on improved A-star algorithm. IEEE 3rd Advanced Information Management, Communicates, Electronic and Automation Control Conference (IMCEC). IEEE, pp 1534\u20131538","DOI":"10.1109\/IMCEC46724.2019.8983841"},{"issue":"10","key":"1278_CR7","doi-asserted-by":"publisher","first-page":"3003","DOI":"10.1080\/00207543.2018.1521532","volume":"57","author":"C Chen","year":"2019","unstructured":"Chen C, Tiong LK, Chen IM (2019) Using a genetic algorithm to schedule the space-constrained AGV-based prefabricated bathroom units manufacturing system. Int J Prod Res 57(10):3003\u20133019. https:\/\/doi.org\/10.1080\/00207543.2018.1521532","journal-title":"Int J Prod Res"},{"issue":"12","key":"1278_CR8","doi-asserted-by":"publisher","first-page":"1439","DOI":"10.3390\/jmse9121439","volume":"9","author":"C Chen","year":"2021","unstructured":"Chen C, Hu ZH, Wang L (2021) Scheduling of AGVs in automated container terminal based on the deep deterministic policy gradient (DDPG) using the convolutional neural network (CNN). J Marine Sci Eng 9(12):1439. https:\/\/doi.org\/10.3390\/jmse9121439","journal-title":"J Marine Sci Eng"},{"key":"1278_CR9","doi-asserted-by":"publisher","first-page":"106749","DOI":"10.1016\/j.cie.2020.106749","volume":"149","author":"H Hu","year":"2020","unstructured":"Hu H, Jia X, He Q et al (2020) Deep reinforcement learning based AGVs real-time scheduling with mixed rule for flexible shop floor in industry 40. Comput Ind Eng 149:106749. https:\/\/doi.org\/10.1016\/j.cie.2020.106749","journal-title":"Comput Ind Eng"},{"issue":"5","key":"1278_CR10","doi-asserted-by":"publisher","first-page":"1224","DOI":"10.1109\/TCYB.2016.2542923","volume":"47","author":"Q Wei","year":"2016","unstructured":"Wei Q, Lewis FL, Sun Q et al (2016) Discrete-time deterministic $ Q $-learning: a novel convergence analysis. IEEE Trans Cybern 47(5):1224\u20131237. https:\/\/doi.org\/10.1109\/TCYB.2016.2542923","journal-title":"IEEE Trans Cybern"},{"key":"1278_CR11","doi-asserted-by":"publisher","unstructured":"Devraj AM, Meyn SP (2017) Fastest convergence for Q-learning. arXiv preprint arXiv:1707.03770. https:\/\/doi.org\/10.48550\/arXiv.1707.03770. Accessed 23 Mar 2018","DOI":"10.48550\/arXiv.1707.03770"},{"key":"1278_CR12","doi-asserted-by":"publisher","first-page":"143","DOI":"10.1016\/j.robot.2019.02.013","volume":"115","author":"ES Low","year":"2019","unstructured":"Low ES, Ong P, Cheah KC (2019) Solving the optimal path planning of a mobile robot using improved Q-learning. Robot Auton Syst 115:143\u2013161. https:\/\/doi.org\/10.1016\/j.robot.2019.02.013","journal-title":"Robot Auton Syst"},{"key":"1278_CR13","doi-asserted-by":"publisher","unstructured":"Yu N, Li T, Wang B (2021) Multi-load AGVs scheduling algorithm in automated sorting warehouse. 14th International Symposium on Computational Intelligence and Design (ISCID). IEEE. 126\u2013129. https:\/\/doi.org\/10.1109\/ISCID52796.2021.00037","DOI":"10.1109\/ISCID52796.2021.00037"},{"issue":"19","key":"1278_CR14","doi-asserted-by":"publisher","first-page":"5685","DOI":"10.3390\/s20195685","volume":"20","author":"BS Roh","year":"2020","unstructured":"Roh BS, Han MH, Ham JH et al (2020) Q-LBR: Q-learning based load balancing routing for UAV-assisted VANET. Sensors 20(19):5685. https:\/\/doi.org\/10.3390\/s20195685","journal-title":"Sensors"},{"key":"1278_CR15","doi-asserted-by":"publisher","first-page":"96","DOI":"10.1016\/j.future.2022.11.012","volume":"141","author":"V Sethi","year":"2023","unstructured":"Sethi V, Pal S (2023) FedDOVe: a federated deep q-learning-based offloading for vehicular fog computing. Futur Gener Comput Syst 141:96\u2013105. https:\/\/doi.org\/10.1016\/j.future.2022.11.012","journal-title":"Futur Gener Comput Syst"},{"issue":"24","key":"1278_CR16","doi-asserted-by":"publisher","first-page":"17508","DOI":"10.1109\/JIOT.2021.3081694","volume":"8","author":"J Chen","year":"2021","unstructured":"Chen J, Xing H, Xiao Z et al (2021) A DRL agent for jointly optimizing computation offloading and resource allocation in MEC. IEEE Internet Things J 8(24):17508\u201317524. https:\/\/doi.org\/10.1109\/JIOT.2021.3081694","journal-title":"IEEE Internet Things J"},{"key":"1278_CR17","doi-asserted-by":"publisher","unstructured":"Xiao Z, et al. (2023) Deep Contrastive Representation Learning With Self-Distillation. In: IEEE transactions on emerging topics in computational intelligence. https:\/\/doi.org\/10.1109\/tetci.2023.3304948","DOI":"10.1109\/tetci.2023.3304948"},{"key":"1278_CR18","doi-asserted-by":"publisher","unstructured":"Song F, Xing H, Wang X, et al. (2022) Evolutionary multi-objective reinforcement learning based trajectory control and task offloading in UAV-assisted mobile edge computing. arXiv e-prints. DOI:https:\/\/doi.org\/10.48550\/arXiv.2202.12028","DOI":"10.48550\/arXiv.2202.12028"},{"issue":"1","key":"1278_CR19","doi-asserted-by":"publisher","first-page":"40","DOI":"10.3141\/2161-05","volume":"2161","author":"Y Ji","year":"2010","unstructured":"Ji Y, Daamen W, Hoogendoorn S et al (2010) Investigating the shape of the macroscopic fundamental diagram using simulation data. Transp Res Rec 2161(1):40\u201348. https:\/\/doi.org\/10.3141\/2161-05","journal-title":"Transp Res Rec"},{"key":"1278_CR20","doi-asserted-by":"publisher","first-page":"119","DOI":"10.1016\/j.trb.2018.10.013","volume":"137","author":"L Amb\u00fchl","year":"2020","unstructured":"Amb\u00fchl L, Loder A, Bliemer MCJ et al (2020) A functional form with a physical meaning for the macroscopic fundamental diagram. Transp Res Part B: Methodol 137:119\u2013132. https:\/\/doi.org\/10.1016\/j.trb.2018.10.013","journal-title":"Transp Res Part B: Methodol"},{"key":"1278_CR21","doi-asserted-by":"publisher","DOI":"10.1088\/1361-6501\/acb075","author":"L Shen","year":"2023","unstructured":"Shen L, Tao H, Ni Y et al (2023) Improved YOLOv3 model with feature map cropping for multi-scale road object detection. Meas Sci Technol. https:\/\/doi.org\/10.1088\/1361-6501\/acb075","journal-title":"Meas Sci Technol"},{"key":"1278_CR22","doi-asserted-by":"publisher","first-page":"168","DOI":"10.1016\/j.trc.2014.03.004","volume":"42","author":"N Geroliminis","year":"2014","unstructured":"Geroliminis N, Zheng N, Ampountolas K (2014) A three-dimensional macroscopic fundamental diagram for mixed bi-modal urban networks. Transp Res Part C Emerg Technol 42:168\u2013181. https:\/\/doi.org\/10.1016\/j.trc.2014.03.004","journal-title":"Transp Res Part C Emerg Technol"},{"key":"1278_CR23","doi-asserted-by":"publisher","first-page":"255","DOI":"10.1016\/j.trb.2014.09.010","volume":"70","author":"VV Gayah","year":"2014","unstructured":"Gayah VV, Gao XS, Nagle AS (2014) On the impacts of locally adaptive signal control on urban network stability and the macroscopic fundamental diagram. Transp Res Part B Methodol 70:255\u2013268. https:\/\/doi.org\/10.1016\/j.trb.2014.09.010","journal-title":"Transp Res Part B Methodol"},{"key":"1278_CR24","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.trb.2019.09.004","volume":"129","author":"A Loder","year":"2019","unstructured":"Loder A, Dakic I, Bressan L et al (2019) Capturing network properties with a functional form for the multi-modal macroscopic fundamental diagram. Transp Res Part B Methodol 129:1\u201319. https:\/\/doi.org\/10.1016\/j.trb.2019.09.004","journal-title":"Transp Res Part B Methodol"},{"issue":"2","key":"1278_CR25","doi-asserted-by":"publisher","first-page":"1653","DOI":"10.3390\/su15021653","volume":"15","author":"M Halakoo","year":"2023","unstructured":"Halakoo M, Yang H, Abdulsattar H (2023) Heterogeneity aware emission macroscopic fundamental diagram (e-MFD). Sustainability 15(2):1653. https:\/\/doi.org\/10.3390\/su15021653","journal-title":"Sustainability"},{"key":"1278_CR26","doi-asserted-by":"publisher","first-page":"425","DOI":"10.1016\/j.proeng.2016.01.277","volume":"137","author":"F He","year":"2016","unstructured":"He F, Yan X, Liu Y et al (2016) A traffic congestion assessment method for urban road networks based on speed performance index. Proc Eng 137:425\u2013433. https:\/\/doi.org\/10.1016\/j.proeng.2016.01.277","journal-title":"Proc Eng"},{"key":"1278_CR27","unstructured":"Amb\u00fchl L, Loder A, Menendez M, et al. (2017) Empirical macroscopic fundamental diagrams: new insights from loop detector and floating car data. TRB 96th Annual Meeting Compendium of Papers. Transportation Research Board, pp 17\u201303331"},{"key":"1278_CR28","doi-asserted-by":"publisher","first-page":"479","DOI":"10.1007\/978-3-319-48896-7_47","volume-title":"Automatic extraction and construction algorithm of overpass from raster maps. Pacific Rim conference on multimedia","author":"X Zhao","year":"2016","unstructured":"Zhao X, Liu Y, Wang Y (2016) Automatic extraction and construction algorithm of overpass from raster maps. Pacific Rim conference on multimedia. Springer, Cham, pp 479\u2013489. https:\/\/doi.org\/10.1007\/978-3-319-48896-7_47"},{"key":"1278_CR29","doi-asserted-by":"publisher","first-page":"1060","DOI":"10.48550\/arXiv.2007.08794","volume":"33","author":"J Oh","year":"2020","unstructured":"Oh J, Hessel M, Czarnecki WM et al (2020) Discovering reinforcement learning algorithms. Adv Neural Inform Process Syst. 33:1060\u20131070. https:\/\/doi.org\/10.48550\/arXiv.2007.08794","journal-title":"Adv Neural Inform Process Syst."},{"key":"1278_CR30","doi-asserted-by":"publisher","first-page":"331","DOI":"10.1002\/9780470316887","volume":"2","author":"ML Puterman","year":"1990","unstructured":"Puterman ML (1990) Markov decision processes. Handbooks Oper Res Manag Sci 2:331\u2013434. https:\/\/doi.org\/10.1002\/9780470316887","journal-title":"Handbooks Oper Res Manag Sci"},{"key":"1278_CR31","first-page":"528","volume-title":"Multi-step reinforcement learning algorithm of mobile robot path planning based on virtual potential field. International Conference of Pioneering Computer Scientists","author":"J Liu","year":"2017","unstructured":"Liu J, Qi W, Lu X (2017) Multi-step reinforcement learning algorithm of mobile robot path planning based on virtual potential field. International Conference of Pioneering Computer Scientists. Engineers and Educators. Springer, Singapore, pp 528\u2013538"},{"issue":"2","key":"1278_CR32","doi-asserted-by":"publisher","first-page":"1454","DOI":"10.1016\/j.jfranklin.2022.11.004","volume":"360","author":"H Tao","year":"2023","unstructured":"Tao H, Qiu J, Chen Y et al (2023) Unsupervised cross-domain rolling bearing fault diagnosis based on time-frequency information fusion. J Franklin Inst 360(2):1454\u20131477. https:\/\/doi.org\/10.1016\/j.jfranklin.2022.11.004","journal-title":"J Franklin Inst"},{"key":"1278_CR33","doi-asserted-by":"publisher","DOI":"10.1108\/EC-11-2022-0672","author":"Y Shang","year":"2023","unstructured":"Shang Y, Liu F, Qin P et al (2023) Research on path planning of autonomous vehicle based on RRT algorithm of Q-learning and obstacle distribution. Eng Comput. https:\/\/doi.org\/10.1108\/EC-11-2022-0672","journal-title":"Eng Comput"},{"key":"1278_CR34","doi-asserted-by":"publisher","DOI":"10.1016\/j.conengprac.2023.105513","volume":"135","author":"X Song","year":"2023","unstructured":"Song X, Wu C, Stojanovic V et al (2023) 1 bit encoding\u2013decoding-based event-triggered fixed-time adaptive control for unmanned surface vehicle with guaranteed tracking performance. Control Eng Pract 135:105513. https:\/\/doi.org\/10.1016\/j.conengprac.2023.105513","journal-title":"Control Eng Pract"},{"issue":"2","key":"1278_CR35","doi-asserted-by":"publisher","first-page":"261","DOI":"10.1049\/itr2.12020","volume":"15","author":"G Hu","year":"2021","unstructured":"Hu G, Lu W, Whalin RW et al (2021) Analytical approximation for macroscopic fundamental diagram of urban corridor with mixed human and connected and autonomous traffic [J]. IET Intel Transport Syst 15(2):261\u2013272. https:\/\/doi.org\/10.1049\/itr2.12020","journal-title":"IET Intel Transport Syst"},{"key":"1278_CR36","doi-asserted-by":"publisher","first-page":"91","DOI":"10.1016\/j.trb.2015.01.001","volume":"73","author":"X Qu","year":"2015","unstructured":"Qu X, Wang S, Zhang J (2015) On the fundamental diagram for freeway traffic: a novel calibration approach for single-regime models. Transp Res Part B Methodol 73:91\u2013102. https:\/\/doi.org\/10.1016\/j.trb.2015.01.001","journal-title":"Transp Res Part B Methodol"},{"issue":"14","key":"1278_CR37","doi-asserted-by":"publisher","first-page":"11378","DOI":"10.3390\/su151411378","volume":"15","author":"K Ji","year":"2023","unstructured":"Ji K, Tang J, Li M et al (2023) Distributed traffic control based on road network partitioning using normalization algorithm. Sustainability 15(14):11378. https:\/\/doi.org\/10.3390\/su151411378","journal-title":"Sustainability"},{"key":"1278_CR38","doi-asserted-by":"publisher","DOI":"10.1007\/0-387-29337-X","volume-title":"Markov chains. Models, algorithms and applications","author":"WK Ching","year":"2006","unstructured":"Ching WK, Ng MK (2006) Markov chains. Models, algorithms and applications. Kluwer Academic Publishers, Boston. https:\/\/doi.org\/10.1007\/0-387-29337-X"},{"key":"1278_CR39","doi-asserted-by":"publisher","first-page":"116272","DOI":"10.1016\/j.eswa.2021.116272","volume":"192","author":"Z Sun","year":"2022","unstructured":"Sun Z, Wang G, Jin L et al (2022) Noise-suppressing zeroing neural network for online solving time-varying matrix square roots problems: a control-theoretic approach. Expert Syst Appl 192:116272. https:\/\/doi.org\/10.1016\/j.eswa.2021.116272","journal-title":"Expert Syst Appl"},{"key":"1278_CR40","doi-asserted-by":"publisher","first-page":"107284","DOI":"10.1016\/j.ijfatigue.2022.107284","volume":"166","author":"E Bellec","year":"2023","unstructured":"Bellec E, Doudard C, Facchinetti ML et al (2023) Loading classification proposal for fatigue design of automotive chassis-parts: a relevant process for variable amplitude and multi-input load cases. Int J Fatigue 166:107284. https:\/\/doi.org\/10.1016\/j.ijfatigue.2022.107284","journal-title":"Int J Fatigue"}],"container-title":["Complex &amp; Intelligent Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s40747-023-01278-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s40747-023-01278-y\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s40747-023-01278-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,3,30]],"date-time":"2024-03-30T15:38:12Z","timestamp":1711813092000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s40747-023-01278-y"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,1,10]]},"references-count":40,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2024,4]]}},"alternative-id":["1278"],"URL":"https:\/\/doi.org\/10.1007\/s40747-023-01278-y","relation":{},"ISSN":["2199-4536","2198-6053"],"issn-type":[{"type":"print","value":"2199-4536"},{"type":"electronic","value":"2198-6053"}],"subject":[],"published":{"date-parts":[[2024,1,10]]},"assertion":[{"value":"12 April 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"4 November 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 January 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"We declare that we have no known competing financial interests or personal relationships or organizations that could have appeared to influence the work reported in this paper.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}