{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,15]],"date-time":"2026-05-15T20:10:26Z","timestamp":1778875826143,"version":"3.51.4"},"reference-count":54,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2024,1,24]],"date-time":"2024-01-24T00:00:00Z","timestamp":1706054400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Systems"],"abstract":"<jats:p>In the context of Industry 4.0 and smart manufacturing, production factories are increasingly focusing on process optimization, high product customization, quality improvement, cost reduction, and energy saving by implementing a new type of digital solutions that are mainly driven by Internet of Things (IoT), artificial intelligence, big data, and cloud computing. By the adoption of the cyber\u2013physical systems (CPSs) concept, today\u2019s factories are gaining in synergy between the physical and the cyber worlds. As a fast-spreading concept, a digital twin is considered today as a robust solution for decision-making support and optimization. Alongside these benefits, sectors are still working to adopt this technology because of the complexity of modeling manufacturing operations as digital twins. In addition, attempting to use a digital twin for fully automatic decision-making adds yet another layer of complexity. This paper presents our framework for the implementation of a full-duplex (data and decisions) specific-purpose digital twin system for autonomous process control, with plastic injection molding as a practical use-case. Our approach is based on a combination of supervised learning and deep reinforcement learning models that allows for an automated updating of the virtual representation of the system, in addition to an intelligent decision-making process for operational metrics optimization. The suggested method allows for improvements in the product quality while lowering costs. The outcomes demonstrate how the suggested structure can produce high-quality output with the least amount of human involvement. This study shows how the digital twin technology can improve the productivity and effectiveness of production processes and advances the use of the technology in the industrial sector.<\/jats:p>","DOI":"10.3390\/systems12020038","type":"journal-article","created":{"date-parts":[[2024,1,24]],"date-time":"2024-01-24T07:42:16Z","timestamp":1706082136000},"page":"38","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":66,"title":["A Deep-Reinforcement-Learning-Based Digital Twin for Manufacturing Process Optimization"],"prefix":"10.3390","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5440-6741","authenticated-orcid":false,"given":"Abdelmoula","family":"Khdoudi","sequence":"first","affiliation":[{"name":"Artificial Intelligence for Engineering Science Team, Moulay Ismail University, Meknes 50050, Morocco"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7761-6300","authenticated-orcid":false,"given":"Tawfik","family":"Masrour","sequence":"additional","affiliation":[{"name":"Artificial Intelligence for Engineering Science Team, Moulay Ismail University, Meknes 50050, Morocco"},{"name":"Mathematics, Computer Science and Engineering Department, University of Quebec at Rimouski, Rimouski, QC G5L 3A1, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4073-6904","authenticated-orcid":false,"given":"Ibtissam","family":"El Hassani","sequence":"additional","affiliation":[{"name":"Artificial Intelligence for Engineering Science Team, Moulay Ismail University, Meknes 50050, Morocco"},{"name":"Mathematics, Computer Science and Engineering Department, University of Quebec at Rimouski, Rimouski, QC G5L 3A1, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Choumicha","family":"El Mazgualdi","sequence":"additional","affiliation":[{"name":"Artificial Intelligence for Engineering Science Team, Moulay Ismail University, Meknes 50050, Morocco"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2024,1,24]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Cheng, K. (2022). Digital-Twins-Driven Semi-Physical Simulation for Testing and Evaluation of Industrial Software in a Smart Manufacturing System. Machines, 10.","DOI":"10.3390\/machines10050388"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"119","DOI":"10.1016\/j.jmsy.2021.05.011","article-title":"Digital twins-based smart manufacturing system design in Industry 4.0: A review","volume":"60","author":"Leng","year":"2021","journal-title":"J. Manuf. Syst."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"217","DOI":"10.1007\/978-3-319-54660-5_20","article-title":"Virtual twins as integrative components of smart products","volume":"Volume 13","author":"Abramovici","year":"2016","journal-title":"Product Lifecycle Management for Digital Transformation of Industries: 13th IFIP WG 5.1 International Conference, PLM 2016, Columbia, SC, USA, 11\u201313 July 2016, Revised Selected Papers"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"567","DOI":"10.1016\/j.ifacol.2015.06.141","article-title":"About the importance of autonomy and digital twins for the future of manufacturing","volume":"48","author":"Rosen","year":"2015","journal-title":"IFAC-Paper"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Zhang, Q., Liu, Z., Duan, J., and Qin, J. (2023). A Novel Method of Digital Twin-Based Manufacturing Process State Modeling and Incremental Anomaly Detection. Machines, 11.","DOI":"10.3390\/machines11020151"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Lee, D., Kim, C.-K., Yang, J., Cho, K.-Y., Choi, J., Noh, S.-D., and Nam, S. (2022). Digital Twin-Based Analysis and Optimization for Design and Planning of Production Lines. Machines, 10.","DOI":"10.3390\/machines10121147"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Xiao, W., He, M., Wei, Z., and Wang, N. (2022). SWLC-DT: An Architecture for Ship Whole Life Cycle Digital Twin Based on Vertical\u2013Horizontal Design. Machines, 10.","DOI":"10.3390\/machines10110998"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Tang, Y.-M., Ho, G.T.S., Lau, Y.-Y., and Tsui, S.-Y. (2022). Integrated Smart Warehouse and Manufacturing Management with Demand Forecasting in Small-Scale Cyclical Industries. Machines, 10.","DOI":"10.3390\/machines10060472"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"724","DOI":"10.1016\/j.procir.2022.05.052","article-title":"Using the Process Digital Twin as a tool for companies to evaluate the Return on Investment of manufacturing automation","volume":"107","author":"Caccamo","year":"2022","journal-title":"Procedia CIRP"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"752","DOI":"10.1016\/j.procir.2022.05.057","article-title":"Digital twin-driven multi-variable process control of thermal manufacturing processes","volume":"107","author":"Stavropoulos","year":"2022","journal-title":"Procedia CIRP"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"103586","DOI":"10.1016\/j.compind.2021.103586","article-title":"A framework for data-driven digital twins of smart manufacturing systems","volume":"136","author":"Friederich","year":"2022","journal-title":"Comput. Ind."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1016\/j.jmsy.2021.01.005","article-title":"Automated manufacturing system discovery and digital twin generation","volume":"59","author":"Lugaresi","year":"2021","journal-title":"J. Manuf. Syst."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"1125","DOI":"10.1016\/j.promfg.2020.10.158","article-title":"Availability of Manufacturing data resources in Digital Twin","volume":"51","author":"Bazaz","year":"2020","journal-title":"Procedia Manuf."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"164","DOI":"10.1016\/j.procir.2021.10.026","article-title":"Manufacturing resilience and agility through processes digital twin: Design and testing applied in the LPBF case","volume":"103","author":"Papacharalampopoulos","year":"2021","journal-title":"Procedia CIRP"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"359","DOI":"10.1016\/j.ifacol.2022.09.413","article-title":"Reinforcement learning and digital twin-based real-time scheduling method in intelligent manufacturing systems","volume":"55","author":"Zhang","year":"2022","journal-title":"IFAC-Paper"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"242","DOI":"10.1016\/j.jmsy.2023.02.008","article-title":"Digital twin-enabled automated anomaly detection and bottleneck identification in complex manufacturing systems using a multi-agent approach","volume":"67","author":"Latsou","year":"2023","journal-title":"J. Manuf. Syst."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"100661","DOI":"10.1016\/j.measen.2022.100661","article-title":"Digital twin and IOT technology for secure manufacturing systems","volume":"25","author":"Gopal","year":"2023","journal-title":"Meas. Sens."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"102456","DOI":"10.1016\/j.rcim.2022.102456","article-title":"Digital twin driven production progress prediction for discrete manufacturing workshop","volume":"80","author":"Qian","year":"2023","journal-title":"Robot. Comput. Integr. Manuf."},{"key":"ref_19","unstructured":"Balakrishnan, P., Babu, K.R., Naiju, C.D., and Madiajagan, M. (2019). SAE Technical Paper, SAE International."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"1388","DOI":"10.1016\/j.procir.2019.04.049","article-title":"Digital Twin for Machining Tool Condition Prediction","volume":"81","author":"Qianzhe","year":"2019","journal-title":"Procedia CIRP"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"898","DOI":"10.1016\/j.procir.2019.03.223","article-title":"Visualisation of the digital twin data in manufacturing by using augmented reality","volume":"81","author":"Zhu","year":"2019","journal-title":"Procedia CIRP"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"746","DOI":"10.1016\/j.procir.2019.01.063","article-title":"A digital twin-based approach for dynamic clamping and positioning of the flexible tooling system","volume":"80","author":"Liu","year":"2019","journal-title":"Procedia CIRP"},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"117","DOI":"10.1016\/j.rcim.2018.09.005","article-title":"Design of a machine tool control system for function reconfiguration and reuse in network environment","volume":"56","author":"Yao","year":"2019","journal-title":"Robot. Comput. Integr. Manuf."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"611","DOI":"10.1016\/j.vrih.2019.10.003","article-title":"Three-dimensional virtual-real mapping of aircraft automatic spray operation and online simulation monitoring","volume":"1","author":"Qiu","year":"2019","journal-title":"Virtual Real. Intell. Hardw."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"012077","DOI":"10.1088\/1757-899X\/324\/1\/012077","article-title":"Digital Twin concept for smart injection molding","volume":"324","author":"Liau","year":"2018","journal-title":"IOP Conf. Ser. Mater. Sci. Eng."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"103568","DOI":"10.1016\/j.compind.2021.103568","article-title":"Application of the Digital Twin for in process monitoring of the micro injection moulding process quality","volume":"135","author":"Modoni","year":"2022","journal-title":"Comput. Ind."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"8838914","DOI":"10.1155\/2021\/8838914","article-title":"A Study on Intelligent Manufacturing Industrial Internet for Injection Molding Industry Based on Digital Twin","volume":"2021","author":"Wang","year":"2021","journal-title":"Complexity"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"874","DOI":"10.1016\/j.ifacol.2021.08.103","article-title":"Designing the Digital Twins of Reconfigurable Manufacturing Systems: Application on a smart factory","volume":"54","author":"Cardin","year":"2021","journal-title":"IFAC-Paper"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"458","DOI":"10.1016\/j.procir.2022.05.008","article-title":"Concept for Digital Twin Based Virtual Part Inspection for Additive Manufacturing","volume":"107","author":"Anderl","year":"2022","journal-title":"Procedia CIRP"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"786","DOI":"10.1016\/j.mfglet.2022.07.097","article-title":"A machining digital twin for hybrid manufacturing","volume":"33","author":"Dvorak","year":"2022","journal-title":"Manuf. Lett."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"549","DOI":"10.1016\/j.procir.2022.09.098","article-title":"Concept of hybrid modeled digital twins and its application for an energy management of manufacturing systems","volume":"112","author":"Langlotz","year":"2022","journal-title":"Procedia CIRP"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"102545","DOI":"10.1016\/j.rcim.2023.102545","article-title":"Hybrid learning-based digital twin for manufacturing process: Modeling framework and implementation","volume":"82","author":"Huang","year":"2023","journal-title":"Robot. Comput. Integr. Manuf."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"176","DOI":"10.1016\/j.jmsy.2021.05.010","article-title":"A digital-twin visualized architecture for Flexible Manufacturing System","volume":"60","author":"Fan","year":"2021","journal-title":"J. Manuf. Syst."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"108463","DOI":"10.1016\/j.cie.2022.108463","article-title":"Towards a connected Digital Twin Learning Ecosystem in manufacturing: Enablers and challenges","volume":"171","author":"Bregon","year":"2022","journal-title":"Comput. Ind. Eng."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Li, S.E. (2023). Reinforcement Learning for Sequential Decision and Optimal Control, Springer Nature.","DOI":"10.1007\/978-981-19-7784-8"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Puterman, M.L. (1994). Markov Decision Processes\u2014Discrete Stochastic Dynamic Programming, John Wiley & Sons, Inc.","DOI":"10.1002\/9780470316887"},{"key":"ref_37","unstructured":"Szepesvari, C., Sutton, R.S., Modayil, J., Bhatnagar, S., and Models, U.O. (2014, January 8\u201313). In Proceedings of the 28th Annual Conference Processing Systems, Montreal, QC, Canada. Available online: https:\/\/search.worldcat.org\/zh-cn\/title\/advances-in-neural-information-processing-systems-27-28th-annual-conference-on-neural-information-processing-systems-2014-nips-december-8-13-2014-montreal-canada-proceedings-of-the-2014-conference-vol-2\/oclc\/931952337."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"341","DOI":"10.1007\/BF00992701","article-title":"Temporal differences: TD(\u03bb) for general \u03bb","volume":"8","author":"Dayan","year":"1992","journal-title":"Mach. Learn."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"279","DOI":"10.1007\/BF00992698","article-title":"Q-learning","volume":"8","author":"Watkins","year":"1989","journal-title":"Machine Learning"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"682","DOI":"10.1016\/j.neunet.2008.02.003","article-title":"Reinforcement learning of motor skills with policy gradients","volume":"21","author":"Peters","year":"2008","journal-title":"Neural Netw."},{"key":"ref_42","first-page":"1058","article-title":"Policy gradient methods for reinforcement learning with function approximation","volume":"12","author":"Sutton","year":"1999","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_43","unstructured":"Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M. (2014). International Conference on Machine Learning, PMLR."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Dong, H., Ding, Z., and Zhang, S. (2020). Deep Reinforcement Learning, Springer.","DOI":"10.1007\/978-981-15-4095-0"},{"key":"ref_45","unstructured":"Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. (2015). International Conference on Machine Learning, PMLR."},{"key":"ref_46","unstructured":"Kurutach, T., Clavera, I., Duan, Y., Tamar, A., and Abbeel, P. (2018). Model-ensemble trust-region policy optimization. arXiv."},{"key":"ref_47","first-page":"32","article-title":"Trust region-guided proximal policy optimization","volume":"32","author":"Wang","year":"2019","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_48","first-page":"7","article-title":"Actor-critic algorithms","volume":"12","author":"Konda","year":"1999","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_49","doi-asserted-by":"crossref","first-page":"1180","DOI":"10.1016\/j.neucom.2007.11.026","article-title":"Natural actor-critic","volume":"71","author":"Peters","year":"2008","journal-title":"Neurocomputing"},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"2471","DOI":"10.1016\/j.automatica.2009.07.008","article-title":"Natural actor\u2013critic algorithms","volume":"45","author":"Bhatnagar","year":"2009","journal-title":"Automatica"},{"key":"ref_51","unstructured":"Mnih, V., Badia, A.P., Mirza, M., Graves, A., Lillicrap, T.P., Harley, T., Silver, D., and Kavukcuoglu, K. (2016). International Conference on Machine Learning, PMLR."},{"key":"ref_52","unstructured":"Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2015). Continuous control with deep reinforcement learning. arXiv."},{"key":"ref_53","unstructured":"Fujimoto, S., Hoof, H., and Meger, D. (2018). International Conference on Machine Learning, PMLR."},{"key":"ref_54","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.jmsy.2022.12.010","article-title":"Application of automation for in-line quality inspection, a zero-defect manufacturing approach","volume":"67","author":"Azamfirei","year":"2023","journal-title":"J. Manuf. Syst."}],"container-title":["Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2079-8954\/12\/2\/38\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T13:48:27Z","timestamp":1760104107000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2079-8954\/12\/2\/38"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,1,24]]},"references-count":54,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2024,2]]}},"alternative-id":["systems12020038"],"URL":"https:\/\/doi.org\/10.3390\/systems12020038","relation":{},"ISSN":["2079-8954"],"issn-type":[{"value":"2079-8954","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,1,24]]}}}