{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,20]],"date-time":"2026-04-20T15:41:42Z","timestamp":1776699702900,"version":"3.51.2"},"reference-count":28,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2022,6,4]],"date-time":"2022-06-04T00:00:00Z","timestamp":1654300800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Ministry of Industry and Information Technology of China for Cruise Program","award":["2018-473"],"award-info":[{"award-number":["2018-473"]}]},{"name":"Ministry of Industry and Information Technology of China for Cruise Program","award":["2019YFB1704403"],"award-info":[{"award-number":["2019YFB1704403"]}]},{"DOI":"10.13039\/501100012166","name":"National Key Research and Development Program of China","doi-asserted-by":"publisher","award":["2018-473"],"award-info":[{"award-number":["2018-473"]}],"id":[{"id":"10.13039\/501100012166","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100012166","name":"National Key Research and Development Program of China","doi-asserted-by":"publisher","award":["2019YFB1704403"],"award-info":[{"award-number":["2019YFB1704403"]}],"id":[{"id":"10.13039\/501100012166","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>This study proposes a digital twin (DT) application framework that integrates deep reinforcement learning (DRL) algorithms for the dynamic scheduling of crane transportation in workshops. DT is used to construct the connection between the workshop service system, logical simulation environment, 3D visualization model and physical workshop, and DRL is used to support the core decision in scheduling. First, the dynamic scheduling problem of crane transportation is constructed as a Markov decision process (MDP), and the corresponding double deep Q-network (DDQN) is designed to interact with the logic simulation environment to complete the offline training of the algorithm. Second, the trained DDQN is embedded into the DT framework, and then connected with the physical workshop and the workshop service system to realize online dynamic crane scheduling based on the real-time states of the workshop. Finally, case studies of crane scheduling under dynamic job arrival and equipment failure scenarios are presented to demonstrate the effectiveness of the proposed framework. The numerical analysis shows that the proposed method is superior to the traditional dynamic scheduling method, and it is also suitable for large-scale problems.<\/jats:p>","DOI":"10.3390\/info13060286","type":"journal-article","created":{"date-parts":[[2022,6,4]],"date-time":"2022-06-04T09:42:32Z","timestamp":1654335752000},"page":"286","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":15,"title":["Dynamic Scheduling of Crane by Embedding Deep Reinforcement Learning into a Digital Twin Framework"],"prefix":"10.3390","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5321-6943","authenticated-orcid":false,"given":"Zhenyu","family":"Xu","sequence":"first","affiliation":[{"name":"Institute of Logistics Science and Engineering, Shanghai Maritime University, Shanghai 201306, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Daofang","family":"Chang","sequence":"additional","affiliation":[{"name":"Institute of Logistics Science and Engineering, Shanghai Maritime University, Shanghai 201306, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Miaomiao","family":"Sun","sequence":"additional","affiliation":[{"name":"Institute of Logistics Science and Engineering, Shanghai Maritime University, Shanghai 201306, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tian","family":"Luo","sequence":"additional","affiliation":[{"name":"School of Economics & Management, Shanghai Maritime University, Shanghai 201306, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,6,4]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"265","DOI":"10.1016\/j.jmsy.2021.09.011","article-title":"Smart manufacturing scheduling: A literature review","volume":"61","author":"Mula","year":"2021","journal-title":"J. Manuf. Syst."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"102","DOI":"10.1016\/j.cor.2014.03.005","article-title":"Scheduling multiple factory cranes on a common track","volume":"48","author":"Peterson","year":"2014","journal-title":"Comput. Oper. Res."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"2853","DOI":"10.1016\/j.cor.2008.12.014","article-title":"Scheduling of a single crane in batch annealing process","volume":"36","author":"Tang","year":"2009","journal-title":"Comput. Oper. Res."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"765","DOI":"10.1016\/j.jclepro.2018.11.231","article-title":"Integrated green scheduling optimization of flexible job shop and crane transportation considering comprehensive energy consumption","volume":"211","author":"Liu","year":"2019","journal-title":"J. Clean. Prod."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"10719","DOI":"10.1007\/s00521-019-04608-9","article-title":"Decomposition-based 2-echelon multi-objective evolutionary algorithm with energy-efficient local search strategies for shop floor multi-crane scheduling problems","volume":"32","author":"Zhou","year":"2020","journal-title":"Neural Comput. Appl."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"106217","DOI":"10.1016\/j.asoc.2020.106217","article-title":"Particle filter and Levy flight-based decomposed multi-objective evolution hybridized particle swarm for flexible job shop greening scheduling with crane transportation","volume":"91","author":"Zhou","year":"2020","journal-title":"Appl. Soft Comput."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"6970","DOI":"10.1080\/00207543.2019.1687952","article-title":"Simulation-based solution for a dynamic multi-crane-scheduling problem in a steelmaking shop","volume":"58","author":"Li","year":"2020","journal-title":"Int. J. Prod. Res."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"100861","DOI":"10.1016\/j.swevo.2021.100861","article-title":"A hybrid estimation of distribution algorithm for distributed flexible job shop scheduling with crane transportations","volume":"62","author":"Du","year":"2021","journal-title":"Swarm Evol. Comput."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"55","DOI":"10.1016\/j.procir.2016.11.011","article-title":"Optimized Adaptive Scheduling of a Manufacturing Process System with Multi-skill Workforce and Multiple Machine Types: An Ontology-based, Multi-agent Reinforcement Learning Approach","volume":"57","author":"Qu","year":"2016","journal-title":"Procedia CIRP"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"75","DOI":"10.1016\/j.cie.2017.05.026","article-title":"A reinforcement learning approach to parameter estimation in dynamic job shop scheduling","volume":"110","author":"Shahrabi","year":"2017","journal-title":"Comput. Ind. Eng."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"417","DOI":"10.1007\/s10845-018-1454-3","article-title":"Adaptive job shop scheduling strategy based on weighted Q-learning algorithm","volume":"31","author":"Wang","year":"2020","journal-title":"J. Intell. Manuf."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"5867","DOI":"10.1080\/00207543.2020.1794075","article-title":"Adaptive scheduling for assembly job shop with uncertain assembly times based on dual Q-learning","volume":"59","author":"Wang","year":"2021","journal-title":"Int. J. Prod. Res."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"4276","DOI":"10.1109\/TII.2019.2908210","article-title":"Smart Manufacturing Scheduling With Edge Computing Using Multiclass Deep Q Network","volume":"15","author":"Lin","year":"2019","journal-title":"IEEE Trans. Ind. Inform."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"3362","DOI":"10.1080\/00207543.2020.1717008","article-title":"Intelligent scheduling of discrete automated production line via deep reinforcement learning","volume":"58","author":"Shi","year":"2020","journal-title":"Int. J. Prod. Res."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"71752","DOI":"10.1109\/ACCESS.2020.2987820","article-title":"Actor-Critic Deep Reinforcement Learning for Solving Job Shop Scheduling Problems","volume":"8","author":"Liu","year":"2020","journal-title":"IEEE Access"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"106208","DOI":"10.1016\/j.asoc.2020.106208","article-title":"Dynamic scheduling for flexible job shop with new job insertions by deep reinforcement learning","volume":"91","author":"Luo","year":"2020","journal-title":"Appl. Soft Comput."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.jmsy.2020.02.004","article-title":"Petri-net-based dynamic scheduling of flexible manufacturing system via deep reinforcement learning with graph convolutional network","volume":"55","author":"Hu","year":"2020","journal-title":"J. Manuf. Syst."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"186474","DOI":"10.1109\/ACCESS.2020.3029868","article-title":"Research on Adaptive Job Shop Scheduling Problems Based on Dueling Double DQN","volume":"8","author":"Han","year":"2020","journal-title":"IEEE Access"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"107969","DOI":"10.1016\/j.comnet.2021.107969","article-title":"Dynamic job-shop scheduling in smart manufacturing using deep reinforcement learning","volume":"190","author":"Wang","year":"2021","journal-title":"Comput. Netw."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Yang, S., Xu, Z., and Wang, J. (2021). Intelligent Decision-Making of Scheduling for Dynamic Permutation Flowshop via Deep Reinforcement Learning. Sensors, 21.","DOI":"10.3390\/s21031019"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"6425","DOI":"10.1109\/TII.2019.2938572","article-title":"Digital-Twin-Based Job Shop Scheduling Toward Smart Manufacturing","volume":"15","author":"Fang","year":"2019","journal-title":"IEEE Trans. Ind. Inform."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1034","DOI":"10.1080\/00207543.2019.1607978","article-title":"Knowledge-driven digital twin manufacturing cell towards intelligent manufacturing","volume":"58","author":"Zhou","year":"2020","journal-title":"Int. J. Prod. Res."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"2189","DOI":"10.1007\/s00170-020-05779-9","article-title":"Model construction of planning and scheduling system based on digital twin","volume":"109","author":"Wang","year":"2020","journal-title":"Int. J. Adv. Manuf. Technol."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"59","DOI":"10.1016\/j.jmsy.2021.05.007","article-title":"Bi-level dynamic scheduling architecture based on service unit digital twin agents","volume":"60","author":"Zhang","year":"2021","journal-title":"J. Manuf. Syst."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"102198","DOI":"10.1016\/j.rcim.2021.102198","article-title":"Research on flexible job shop scheduling under finite transportation conditions for digital twin workshop","volume":"72","author":"Yan","year":"2021","journal-title":"Robot. Comput. -Integr. Manuf."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"328","DOI":"10.1016\/j.jpdc.2017.12.010","article-title":"Event-based sensor data exchange and fusion in the Internet of Things environments","volume":"118","author":"Esposito","year":"2018","journal-title":"J. Parallel Distrib. Comput."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"106749","DOI":"10.1016\/j.cie.2020.106749","article-title":"Deep reinforcement learning based AGVs real-time scheduling with mixed rule for flexible shop floor in industry 4.0","volume":"149","author":"Hu","year":"2020","journal-title":"Comput. Ind. Eng."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1350014","DOI":"10.1142\/S0217595913500140","article-title":"Flow Shop Scheduling with Reinforcement Learning","volume":"30","author":"Zhang","year":"2013","journal-title":"Asia Pac. J. Oper. Res."}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/13\/6\/286\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T23:24:30Z","timestamp":1760138670000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/13\/6\/286"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,6,4]]},"references-count":28,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2022,6]]}},"alternative-id":["info13060286"],"URL":"https:\/\/doi.org\/10.3390\/info13060286","relation":{},"ISSN":["2078-2489"],"issn-type":[{"value":"2078-2489","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,6,4]]}}}