{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,7]],"date-time":"2026-04-07T07:11:47Z","timestamp":1775545907886,"version":"3.50.1"},"reference-count":33,"publisher":"Springer Science and Business Media LLC","issue":"6","license":[{"start":{"date-parts":[[2022,9,2]],"date-time":"2022-09-02T00:00:00Z","timestamp":1662076800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,9,2]],"date-time":"2022-09-02T00:00:00Z","timestamp":1662076800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100012166","name":"national key r &d program of china","doi-asserted-by":"crossref","award":["2018AAA0101704"],"award-info":[{"award-number":["2018AAA0101704"]}],"id":[{"id":"10.13039\/501100012166","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["71690230\/71690234"],"award-info":[{"award-number":["71690230\/71690234"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61973237"],"award-info":[{"award-number":["61973237"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61873191"],"award-info":[{"award-number":["61873191"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004543","name":"China Scholarship Council","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100004543","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Complex Intell. Syst."],"published-print":{"date-parts":[[2022,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The dynamic scheduling problem of semiconductor manufacturing systems (SMSs) is becoming more complicated and challenging due to internal uncertainties and external demand changes. To this end, this paper addresses integrated release control and production scheduling problems with uncertain processing times and urgent orders and proposes a convolutional neural network and asynchronous advanced actor critic-based method (CNN-A3C) that involves a training phase and a deployment phase. In the training phase, actor\u2013critic networks are trained to predict the evaluation of scheduling decisions and to output the optimal scheduling decision. In the deployment phase, the most appropriate release control and scheduling decisions are periodically generated according to the current production status based on the networks. Furthermore, we improve the four key points in the deep reinforcement learning (DRL) algorithm, state space, action space, reward function, and network structure and design four mechanisms: a slide-window-based two-dimensional state perception mechanism, an adaptive reward function that considers multiple objectives and automatically adjusts to dynamic events, a continuous action space based on composite dispatching rules (CDR) and release strategies, and actor\u2013critic networks based on convolutional neural networks (CNNs). To verify the feasibility and effectiveness of the proposed dynamic scheduling method, it is implemented on a simplified SMS. The simulation experimental results show that the proposed method outperforms the unimproved A3C-based method and the common dispatching rules under the new uncertain scenarios.<\/jats:p>","DOI":"10.1007\/s40747-022-00844-0","type":"journal-article","created":{"date-parts":[[2022,9,2]],"date-time":"2022-09-02T06:02:43Z","timestamp":1662098563000},"page":"4641-4662","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":31,"title":["Dynamic scheduling for semiconductor manufacturing systems with uncertainties using convolutional neural networks and reinforcement learning"],"prefix":"10.1007","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8934-2127","authenticated-orcid":false,"given":"Juan","family":"Liu","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1513-8753","authenticated-orcid":false,"given":"Fei","family":"Qiao","sequence":"additional","affiliation":[]},{"given":"Minjie","family":"Zou","sequence":"additional","affiliation":[]},{"given":"Jonas","family":"Zinn","sequence":"additional","affiliation":[]},{"given":"Yumin","family":"Ma","sequence":"additional","affiliation":[]},{"given":"Birgit","family":"Vogel-Heuser","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2022,9,2]]},"reference":[{"key":"844_CR1","doi-asserted-by":"publisher","unstructured":"Altenm\u00fcller T, St\u00fcker T, Waschneck B, Kuhnle A, Lanza G (2020) Reinforcement learning for an intelligent and autonomous production control of complex job-shops under time constraints. Prod Eng 14(3):319\u2013328. https:\/\/doi.org\/10.1007\/s11740-020-00967-8https:\/\/link.springer.com\/10.1007\/s11740-020-00967-8","DOI":"10.1007\/s11740-020-00967-8"},{"key":"844_CR2","doi-asserted-by":"publisher","unstructured":"Bahaji N, Kuhl ME (2008) A simulation study of new multi-objective composite dispatching rules, CONWIP, and push lot release in semiconductor fabrication. Int J Prod Res 46(14):3801\u20133824. https:\/\/doi.org\/10.1080\/00207540600711879http:\/\/www.tandfonline.com\/doi\/abs\/10.1080\/00207540600711879","DOI":"10.1080\/00207540600711879"},{"key":"844_CR3","doi-asserted-by":"publisher","unstructured":"Bhatt N, Chauhan, NR (2015) Genetic algorithm applications on Job Shop Scheduling Problem: A review. In: 2015 International Conference on Soft Computing Techniques and Implementations (ICSCTI), pp. 7\u201314 https:\/\/doi.org\/10.1109\/ICSCTI.2015.7489556","DOI":"10.1109\/ICSCTI.2015.7489556"},{"key":"844_CR4","doi-asserted-by":"publisher","unstructured":"Chen Y, Zhou H, Huang P, Chou F, Huang S (2019) A refined order release method for achieving robustness of non-repetitive dynamic manufacturing system performance. Ann Oper Res https:\/\/doi.org\/10.1007\/s10479-019-03484-9http:\/\/link.springer.com\/10.1007\/s10479-019-03484-9","DOI":"10.1007\/s10479-019-03484-9"},{"key":"844_CR5","doi-asserted-by":"publisher","unstructured":"Chiang TC (2013) Enhancing rule-based scheduling in wafer fabrication facilities by evolutionary algorithms: Review and opportunity. Comput Ind Eng 64(1):524\u2013535. https:\/\/doi.org\/10.1016\/j.cie.2012.08.009https:\/\/linkinghub.elsevier.com\/retrieve\/pii\/S0360835212002082","DOI":"10.1016\/j.cie.2012.08.009"},{"key":"844_CR6","doi-asserted-by":"publisher","unstructured":"Gabel T, Riedmiller M (2012) Distributed policy search reinforcement learning for job-shop scheduling tasks. Int J Prod Res 50(1):41\u201361. https:\/\/doi.org\/10.1080\/00207543.2011.571443http:\/\/www.tandfonline.com\/doi\/abs\/10.1080\/00207543.2011.571443","DOI":"10.1080\/00207543.2011.571443"},{"key":"844_CR7","doi-asserted-by":"publisher","unstructured":"He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770\u2013778. Las Vegas, USA (2016). https:\/\/doi.org\/10.1109\/CVPR.2016.90. https:\/\/ieeexplore.ieee.org\/document\/7780459","DOI":"10.1109\/CVPR.2016.90"},{"key":"844_CR8","doi-asserted-by":"publisher","unstructured":"Hu L, Liu Z, Hu W, Wang Y, Tan J, Wu F (2020) Petri-net-based dynamic scheduling of flexible manufacturing system via deep reinforcement learning with graph convolutional network. J Manuf Syst 55:1\u201314. https:\/\/doi.org\/10.1016\/j.jmsy.2020.02.004https:\/\/linkinghub.elsevier.com\/retrieve\/pii\/S0278612520300145","DOI":"10.1016\/j.jmsy.2020.02.004"},{"key":"844_CR9","doi-asserted-by":"publisher","unstructured":"Krizhevsky A, Sutskever I, Hinton GE (2017) Imagenet classification with deep convolutional neural networks. Commun. ACM 60(6):84\u201390. https:\/\/doi.org\/10.1145\/3065386https:\/\/doi.org\/10.1145\/3065386","DOI":"10.1145\/3065386"},{"key":"844_CR10","doi-asserted-by":"publisher","unstructured":"Kuhnle A, Sch\u00e4fer L, Stricker N, Lanza G (2019) Design, Implementation and Evaluation of Reinforcement Learning for an Adaptive Order Dispatching in Job Shop Manufacturing Systems. Procedia CIRP 81:234\u2013239. https:\/\/doi.org\/10.1016\/j.procir.2019.03.041https:\/\/linkinghub.elsevier.com\/retrieve\/pii\/S2212827119303464","DOI":"10.1016\/j.procir.2019.03.041"},{"key":"844_CR11","doi-asserted-by":"publisher","unstructured":"Lecun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436\u2013444. https:\/\/doi.org\/10.1038\/nature14539https:\/\/www.nature.com\/articles\/nature14539","DOI":"10.1038\/nature14539"},{"key":"844_CR12","doi-asserted-by":"publisher","unstructured":"Lecun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278\u20132324. https:\/\/doi.org\/10.1109\/5.726791https:\/\/ieeexplore.ieee.org\/document\/726791","DOI":"10.1109\/5.726791"},{"key":"844_CR13","doi-asserted-by":"publisher","unstructured":"Li, H., Kumar, N., Chen, R., Georgiou, P.: A deep reinforcement learning framework for identifying funny scenes in movies. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3116\u20133120 (2018). https:\/\/doi.org\/10.1109\/ICASSP.2018.8462686. https:\/\/ieeexplore.ieee.org\/document\/8462686","DOI":"10.1109\/ICASSP.2018.8462686"},{"key":"844_CR14","doi-asserted-by":"publisher","unstructured":"Lin CC, Deng DJ, Chih YL, Chiu HT (2019) Smart Manufacturing Scheduling With Edge Computing Using Multiclass Deep Q Network. IEEE Trans Ind Inform 15(7):4276\u20134284. https:\/\/doi.org\/10.1109\/TII.2019.2908210https:\/\/ieeexplore.ieee.org\/document\/8676376\/","DOI":"10.1109\/TII.2019.2908210"},{"key":"844_CR15","doi-asserted-by":"publisher","unstructured":"Lowe JJ, Mason SJ (2016) Integrated Semiconductor Supply Chain Production Planning. IEEE Trans Semicond Manuf 29(2):116\u2013126. https:\/\/doi.org\/10.1109\/TSM.2016.2544202http:\/\/ieeexplore.ieee.org\/document\/7436801\/","DOI":"10.1109\/TSM.2016.2544202"},{"key":"844_CR16","doi-asserted-by":"publisher","unstructured":"Luo S (2020) Dynamic scheduling for flexible job shop with new job insertions by deep reinforcement learning. Appl Soft Comput 91:106208. https:\/\/doi.org\/10.1016\/j.asoc.2020.106208https:\/\/linkinghub.elsevier.com\/retrieve\/pii\/S1568494620301484","DOI":"10.1016\/j.asoc.2020.106208"},{"key":"844_CR17","unstructured":"Mnih V, Badia A, Mirza M, Graves A, Lillicrap T, Harley T, Silver D, Kavukcuoglu K (2016) Asynchronous methods for deep reinforcement learning. In: Proceedings of the 33rd International Conference on Machine Learning pp. 1928\u20131937"},{"key":"844_CR18","doi-asserted-by":"publisher","unstructured":"O\u2019Neill D, Xue B, Zhang M (2021) Evolutionary neural architecture search for high-dimensional skip-connection structures on densenet style networks. In: IEEE Transactions on Evolutionary Computation. https:\/\/doi.org\/10.1109\/TEVC.2021.3083315https:\/\/ieeexplore.ieee.org\/document\/9439793","DOI":"10.1109\/TEVC.2021.3083315"},{"key":"844_CR19","doi-asserted-by":"publisher","unstructured":"Palombarini JA, Mart\u00ednez EC (2019) Closed-loop Rescheduling using Deep Reinforcement Learning. IFAC-PapersOnLine 52(1):231\u2013236. https:\/\/doi.org\/10.1016\/j.ifacol.2019.06.067https:\/\/linkinghub.elsevier.com\/retrieve\/pii\/S2405896319301521","DOI":"10.1016\/j.ifacol.2019.06.067"},{"key":"844_CR20","doi-asserted-by":"publisher","unstructured":"Park, I.B., Huh, J., Kim, J., Park, J.: A Reinforcement Learning Approach to Robust Scheduling of Semiconductor Manufacturing Facilities. IEEE Trans Autom Sci Eng pp. 1\u201312 (2020). https:\/\/doi.org\/10.1109\/TASE.2019.2956762. https:\/\/ieeexplore.ieee.org\/document\/8946870\/","DOI":"10.1109\/TASE.2019.2956762"},{"key":"844_CR21","doi-asserted-by":"publisher","unstructured":"Priore P, G\u00f3mez A, Pino R, Rosillo R (2014) Dynamic scheduling of manufacturing systems using machine learning: An updated review. Artif Intell Eng Des Anal Manuf 28(1):83\u201397. https:\/\/doi.org\/10.1017\/S0890060413000516https:\/\/www.cambridge.org\/core\/product\/identifier\/S0890060413000516\/type\/journal_article","DOI":"10.1017\/S0890060413000516"},{"key":"844_CR22","doi-asserted-by":"publisher","unstructured":"Rossit DA, Tohm\u00e9 F, Frutos M (2019) A data-driven scheduling approach to smart manufacturing. J Ind Inf Integr 15:69\u201379. https:\/\/doi.org\/10.1016\/j.jii.2019.04.003https:\/\/linkinghub.elsevier.com\/retrieve\/pii\/S2452414X18300475","DOI":"10.1016\/j.jii.2019.04.003"},{"key":"844_CR23","doi-asserted-by":"publisher","unstructured":"Shiue YR, Lee KC, Su CT (2018) Real-time scheduling for a smart factory using a reinforcement learning approach. Comput Ind Eng 125:604\u2013614. https:\/\/doi.org\/10.1016\/j.cie.2018.03.039https:\/\/linkinghub.elsevier.com\/retrieve\/pii\/S036083521830130X","DOI":"10.1016\/j.cie.2018.03.039"},{"key":"844_CR24","unstructured":"Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. Comput Sci (2014). arXiv:1409.1556pdf"},{"key":"844_CR25","doi-asserted-by":"publisher","unstructured":"Stricker N, Kuhnle A, Sturm R, Friess S (2018) Reinforcement learning for adaptive order dispatching in the semiconductor industry. CIRP Ann 67(1):511\u2013514. https:\/\/doi.org\/10.1016\/j.cirp.2018.04.041https:\/\/linkinghub.elsevier.com\/retrieve\/pii\/S0007850618300659","DOI":"10.1016\/j.cirp.2018.04.041"},{"key":"844_CR26","doi-asserted-by":"crossref","unstructured":"Szegedy C, Wei L, Jia Y, Sermanet P, Rabinovich A (2015) Going deeper with convolutions. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1\u20139. Boston, USA. https:\/\/ieeexplore.ieee.org\/stamp\/stamp.jsp?arnumber=7298594","DOI":"10.1109\/CVPR.2015.7298594"},{"key":"844_CR27","doi-asserted-by":"publisher","unstructured":"Vargas-Villamil F, Rivera D, Kempf K (2003) A hierarchical approach to production control of reentrant semiconductor manufacturing lines. IEEE Trans Control Syst Technol 11(4):578\u2013587. https:\/\/doi.org\/10.1109\/TCST.2003.813368http:\/\/ieeexplore.ieee.org\/document\/1208336\/","DOI":"10.1109\/TCST.2003.813368"},{"key":"844_CR28","doi-asserted-by":"publisher","unstructured":"Waschneck B, Reichstaller A, Belzner L, Altenm\u00fcller T, Bauernhansl T, Knapp A, Kyek A (2018) Optimization of global production scheduling with deep reinforcement learning. Procedia CIRP 72:1264\u20131269. https:\/\/doi.org\/10.1016\/j.procir.2018.03.212https:\/\/linkinghub.elsevier.com\/retrieve\/pii\/S221282711830372X","DOI":"10.1016\/j.procir.2018.03.212"},{"key":"844_CR29","doi-asserted-by":"publisher","unstructured":"Xue Y, Jiang P, Neri F, Liang J (2021) A multiobjective evolutionary approach based on graph-in-graph for neural architecture search of convolutional neural networks. Int J Neural Syst https:\/\/doi.org\/10.1142\/S0129065721500350","DOI":"10.1142\/S0129065721500350"},{"key":"844_CR30","doi-asserted-by":"publisher","unstructured":"Xue Y, Wang Y, Liang J, Slowik A (2021) A self-adaptive mutation neural architecture search algorithm based on blocks. IEEE Comput Intell Mag 16(3):67\u201378. https:\/\/doi.org\/10.1109\/MCI.2021.3084435https:\/\/ieeexplore.ieee.org\/document\/9492170","DOI":"10.1109\/MCI.2021.3084435"},{"key":"844_CR31","doi-asserted-by":"publisher","unstructured":"Yugma C, Blue J, Dauz\u00e8re-P\u00e9r\u00e9s S, Obeid A (2015) Integration of scheduling and advanced process control in semiconductor manufacturing: review and outlook. J Sched 18(2):195\u2013205. https:\/\/doi.org\/10.1007\/s10951-014-0381-1http:\/\/link.springer.com\/10.1007\/s10951-014-0381-1","DOI":"10.1007\/s10951-014-0381-1"},{"key":"844_CR32","doi-asserted-by":"publisher","unstructured":"Zhang J, Ding G, Zou Y, Qin S, Fu J (2019) Review of job shop scheduling research and its new perspectives under Industry 4.0. J Intell Manuf 30(4), 1809\u20131830 . https:\/\/doi.org\/10.1007\/s10845-017-1350-2. http:\/\/link.springer.com\/10.1007\/s10845-017-1350-2","DOI":"10.1007\/s10845-017-1350-2"},{"key":"844_CR33","unstructured":"Zhang W, Dietterich, TG (1995) A reinforcement learning approach to job-shop scheduling. In: Proceedings of the 14th International Joint Conference on Artificial Intelligence pp. 1114\u20131120"}],"container-title":["Complex &amp; Intelligent Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s40747-022-00844-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s40747-022-00844-0\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s40747-022-00844-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,10,27]],"date-time":"2022-10-27T12:28:09Z","timestamp":1666873689000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s40747-022-00844-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,9,2]]},"references-count":33,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2022,12]]}},"alternative-id":["844"],"URL":"https:\/\/doi.org\/10.1007\/s40747-022-00844-0","relation":{},"ISSN":["2199-4536","2198-6053"],"issn-type":[{"value":"2199-4536","type":"print"},{"value":"2198-6053","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,9,2]]},"assertion":[{"value":"8 July 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 October 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 September 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"This work was supported in part by the National Key R &D Program of China under Grant No. 2018AAA0101704, in part by the National Natural Science Foundation of China under Grant No. 71690230\/71690234, 61973237, 61873191, and in part by the China Scholarship Council scholarship.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Funding"}},{"value":"All of the authors declare that they have no conflict of interest.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"The data used to support the findings of this study are available from the corresponding author upon request.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Availability of data and materials"}},{"value":"The code is available from the corresponding author upon request.","order":5,"name":"Ethics","group":{"name":"EthicsHeading","label":"Code availability"}},{"value":"This paper does not contain any studies with human participants or animals performed by any of the authors.","order":6,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval"}},{"value":"Not applicable.","order":7,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent to participate"}}]}}