{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,12]],"date-time":"2026-05-12T22:19:04Z","timestamp":1778624344805,"version":"3.51.4"},"reference-count":37,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2024,12,23]],"date-time":"2024-12-23T00:00:00Z","timestamp":1734912000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,12,23]],"date-time":"2024-12-23T00:00:00Z","timestamp":1734912000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"German Federal Ministry for Economic Affairs and Climate Action","award":["01MD22001C"],"award-info":[{"award-number":["01MD22001C"]}]},{"name":"German Federal Ministry for Economic Affairs and Climate Action","award":["01MD22001C"],"award-info":[{"award-number":["01MD22001C"]}]},{"name":"German Federal Ministry for Economic Affairs and Climate Action","award":["01MD22001C"],"award-info":[{"award-number":["01MD22001C"]}]},{"name":"German Federal Ministry for Economic Affairs and Climate Action","award":["01MD22001C"],"award-info":[{"award-number":["01MD22001C"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Discov Internet Things"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The significance of digital technologies in the context of digitizing production processes, such as Artificial Intelligence (AI) and Digital Twins, is on the rise. A promising avenue of research is the optimization of digital twins through Reinforcement Learning (RL). This necessitates a simulation environment that can be integrated with RL. One is introduced in this paper as the Digital Model Playground (DMPG). The paper outlines the implementation of the DMPG, followed by demonstrating its application in optimizing production scheduling through RL within a sample process. Although there is potential for further development, the DMPG already enables the modeling and optimization of production processes using RL and is comparable to commercial discrete event simulation software regarding the simulation-speed. Furthermore, it is highly flexible and adaptable, as shown by two projects, which distribute the DMPG to a high-performance cluster or generate 2D\/3D-Visualization of the simulation model with Unreal. This establishes the DMPG as a valuable tool for advancing the digital transformation of manufacturing systems, affirming its potential impact on the future of production optimization. Currently, planned extensions include the integration of more optimization algorithms and Process Mining techniques, to further enhance the usability of the framework.<\/jats:p>","DOI":"10.1007\/s43926-024-00087-0","type":"journal-article","created":{"date-parts":[[2024,12,23]],"date-time":"2024-12-23T14:30:08Z","timestamp":1734964208000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["Reinforcement learning and digital twin-driven optimization of production scheduling with the digital model playground"],"prefix":"10.1007","volume":"4","author":[{"given":"Arne","family":"Seipolt","sequence":"first","affiliation":[]},{"given":"Ralf","family":"Buscherm\u00f6hle","sequence":"additional","affiliation":[]},{"given":"Vladislav","family":"Haag","sequence":"additional","affiliation":[]},{"given":"Wilhelm","family":"Hasselbring","sequence":"additional","affiliation":[]},{"given":"Maximilian","family":"H\u00f6finghoff","sequence":"additional","affiliation":[]},{"given":"Marcel","family":"Schumacher","sequence":"additional","affiliation":[]},{"given":"Henrik","family":"Wilbers","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2024,12,23]]},"reference":[{"key":"87_CR1","unstructured":"Joshi S, 5 major benefits of data collection for manufacturing companies manufacturing tomorrow. 2022. https:\/\/manufacturingtomorrow.com\/article\/2022\/08\/5-major-benefits-of-data-collection-for-manufacturing-companies\/19116."},{"key":"87_CR2","doi-asserted-by":"publisher","first-page":"75337","DOI":"10.1109\/ACCESS.2024.3406510","volume":"12","author":"A Barbie","year":"2024","unstructured":"Barbie A, Hasselbring W. From digital twins to digital twin prototypes: concepts, formalization, and applications. IEEE Access. 2024;12:75337\u201365. https:\/\/doi.org\/10.1109\/ACCESS.2024.3406510.","journal-title":"IEEE Access"},{"key":"87_CR3","unstructured":"Minevich M, How to reinvent continuous improvement with intelligent digital twins in manufacturing. https:\/\/www.forbes.com\/sites\/markminevich\/2022\/01\/28\/how-to-reinvent-continuous-improvement-with-intelligent-digital-twins-in-manufacturing\/. Accessed 21 Sep 2023."},{"key":"87_CR4","unstructured":"Gya R, et al., Digital twins: adding intelligence to the real world\u2019. 2022. https:\/\/www.capgemini.com\/gb-en\/wp-content\/uploads\/sites\/3\/2022\/05\/Capgemini-Research-Institute_DigitalTwins_Web.pdf"},{"key":"87_CR5","doi-asserted-by":"publisher","first-page":"343","DOI":"10.1016\/S0167-5060(08)70743-X","volume-title":"Studies in integer programming, in annals of discrete mathematics","author":"JK Lenstra","year":"1977","unstructured":"Lenstra JK, Rinnooy Kan AHG, Brucker P. Complexity of machine scheduling problems. In: Hammer PL, Johnson EL, Korte BH, Nemhauser GL, editors. Studies in integer programming, in annals of discrete mathematics. North-Holland: Elsevier; 1977. p. 343\u201362. https:\/\/doi.org\/10.1016\/S0167-5060(08)70743-X."},{"key":"87_CR6","doi-asserted-by":"publisher","DOI":"10.1016\/j.cor.2021.105400","author":"N Mazyavkina","year":"2021","unstructured":"Mazyavkina N, Sviridov S, Ivanov S, Burnaev E. Reinforcement learning for combinatorial optimization: a survey. Comput Oper Res. 2021. https:\/\/doi.org\/10.1016\/j.cor.2021.105400.","journal-title":"Comput Oper Res"},{"key":"87_CR7","unstructured":"Crites RH, Large-scale dynamic optimization using teams of reinforcement learning agents."},{"key":"87_CR8","unstructured":"Crites RH, Barto G, \u2018Elevator group control using multiple reinforcement learning agents."},{"key":"87_CR9","doi-asserted-by":"publisher","unstructured":"Feldkamp N, Bergmann S, Strassburger S, Simulation-Based Deep Reinforcement Learning For Modular Production Systems, in 2020 Winter Simulation Conference (WSC), Orlando, FL, USA: IEEE, Dec. 2020, pp. 1596\u20131607. https:\/\/doi.org\/10.1109\/WSC48552.2020.9384089.","DOI":"10.1109\/WSC48552.2020.9384089"},{"issue":"10","key":"87_CR10","doi-asserted-by":"publisher","first-page":"8262","DOI":"10.3390\/su15108262","volume":"15","author":"AF \u0130nal","year":"2023","unstructured":"\u0130nal AF, Sel \u00c7, Aktepe A, T\u00fcrker AK, Ers\u00f6z S. A multi-agent reinforcement learning approach to the dynamic job shop scheduling problem. Sustainability. 2023;15(10):8262. https:\/\/doi.org\/10.3390\/su15108262.","journal-title":"Sustainability"},{"key":"87_CR11","doi-asserted-by":"publisher","first-page":"743","DOI":"10.1016\/j.jmsy.2022.10.019","volume":"65","author":"M Panzer","year":"2022","unstructured":"Panzer M, Bender B, Gronau N. Neural agent-based production planning and control: an architectural review. J Manuf Syst. 2022;65:743\u201366. https:\/\/doi.org\/10.1016\/j.jmsy.2022.10.019.","journal-title":"J Manuf Syst"},{"issue":"16","key":"87_CR12","doi-asserted-by":"publisher","first-page":"5772","DOI":"10.1080\/00207543.2022.2104180","volume":"61","author":"A Esteso","year":"2023","unstructured":"Esteso A, Peidro D, Mula J, D\u00edaz-Madro\u00f1ero M. Reinforcement learning applied to production planning and control. Int J Prod Res. 2023;61(16):5772\u201389. https:\/\/doi.org\/10.1080\/00207543.2022.2104180.","journal-title":"Int J Prod Res"},{"key":"87_CR13","doi-asserted-by":"publisher","first-page":"32030","DOI":"10.1109\/ACCESS.2021.3060863","volume":"9","author":"MM Rathore","year":"2021","unstructured":"Rathore MM, Shah SA, Shukla D, Bentafat E, Bakiras S. The role of AI, machine learning, and big data in digital twinning: a systematic literature review, challenges, and opportunities. IEEE Access. 2021;9:32030\u201352. https:\/\/doi.org\/10.1109\/ACCESS.2021.3060863.","journal-title":"IEEE Access"},{"key":"87_CR14","doi-asserted-by":"publisher","DOI":"10.1016\/j.rcim.2024.102778","volume":"89","author":"N Ouahabi","year":"2024","unstructured":"Ouahabi N, Chebak A, Kamach O, Laayati O, Zegrari M. Leveraging digital twin into dynamic production scheduling: A review. Robot Comput-Integr Manuf. 2024;89: 102778. https:\/\/doi.org\/10.1016\/j.rcim.2024.102778.","journal-title":"Robot Comput-Integr Manuf"},{"issue":"10","key":"87_CR15","doi-asserted-by":"publisher","first-page":"359","DOI":"10.1016\/j.ifacol.2022.09.413","volume":"55","author":"L Zhang","year":"2022","unstructured":"Zhang L, Yan Y, Hu Y, Ren W. Reinforcement learning and digital twin-based real-time scheduling method in intelligent manufacturing systems. IFAC-PapersOnLine. 2022;55(10):359\u201364. https:\/\/doi.org\/10.1016\/j.ifacol.2022.09.413.","journal-title":"IFAC-PapersOnLine"},{"key":"87_CR16","doi-asserted-by":"publisher","first-page":"210","DOI":"10.1016\/j.jmsy.2020.06.012","volume":"58","author":"K Xia","year":"2021","unstructured":"Xia K, et al. A digital twin to train deep reinforcement learning agent for smart manufacturing plants: environment, interfaces and intelligence. J Manuf Syst. 2021;58:210\u201330. https:\/\/doi.org\/10.1016\/j.jmsy.2020.06.012.","journal-title":"J Manuf Syst"},{"issue":"1","key":"87_CR17","doi-asserted-by":"publisher","first-page":"408","DOI":"10.1016\/j.ifacol.2021.08.046","volume":"54","author":"Z Mueller-Zhang","year":"2021","unstructured":"Mueller-Zhang Z, Antonino PO, Kuhn T. Integrated planning and scheduling for customized production using digital twins and reinforcement learning. IFAC-PapersOnLine. 2021;54(1):408\u201313. https:\/\/doi.org\/10.1016\/j.ifacol.2021.08.046.","journal-title":"IFAC-PapersOnLine"},{"issue":"11","key":"87_CR18","doi-asserted-by":"publisher","first-page":"1016","DOI":"10.1016\/j.ifacol.2018.08.474","volume":"51","author":"W Kritzinger","year":"2018","unstructured":"Kritzinger W, Karner M, Traar G, Henjes J, Sihn W. Digital twin in manufacturing: a categorical literature review and classification. IFAC-PapersOnLine. 2018;51(11):1016\u201322. https:\/\/doi.org\/10.1016\/j.ifacol.2018.08.474.","journal-title":"IFAC-PapersOnLine"},{"issue":"4","key":"87_CR19","doi-asserted-by":"publisher","first-page":"2569","DOI":"10.1109\/COMST.2023.3297395","volume":"25","author":"H Xu","year":"2023","unstructured":"Xu H, Wu J, Pan Q, Guan X, Guizani M. A survey on digital twin for industrial internet of things: applications, technologies and tools. IEEE Commun Surv Tutor. 2023;25(4):2569\u201398. https:\/\/doi.org\/10.1109\/COMST.2023.3297395.","journal-title":"IEEE Commun Surv Tutor"},{"issue":"2","key":"87_CR20","doi-asserted-by":"publisher","first-page":"39","DOI":"10.1109\/MS.2021.3130755","volume":"39","author":"R Eramo","year":"2022","unstructured":"Eramo R, Bordeleau F, Combemale B, Brand MVD, Wimmer M, Wortmann A. Conceptualizing digital twins. IEEE Softw. 2022;39(2):39\u201346. https:\/\/doi.org\/10.1109\/MS.2021.3130755.","journal-title":"IEEE Softw"},{"key":"87_CR21","doi-asserted-by":"crossref","unstructured":"Blazewicz J, Ecker KH, Pesch E, Schmidt G, Sterna M, Weglarz J, Handbook on scheduling: from theory to applications, 2. Auflage. Springer Nature, 2019.","DOI":"10.1007\/978-3-319-99849-7"},{"key":"87_CR22","doi-asserted-by":"publisher","DOI":"10.5772\/1392","author":"R da Righi","year":"2012","unstructured":"da Righi R. \u2018Preface\u2019, in production scheduling. InTech. 2012. https:\/\/doi.org\/10.5772\/1392.","journal-title":"InTech"},{"issue":"6","key":"87_CR23","doi-asserted-by":"publisher","first-page":"1531","DOI":"10.1007\/s10845-019-01531-7","volume":"31","author":"JP Usuga Cadavid","year":"2020","unstructured":"Usuga Cadavid JP, Lamouri S, Grabot B, Pellerin R, Fortin A. Machine learning applied in production planning and control: a state-of-the-art in the era of industry 4.0. J Intell Manuf. 2020;31(6):1531\u201358. https:\/\/doi.org\/10.1007\/s10845-019-01531-7.","journal-title":"J Intell Manuf"},{"issue":"3","key":"87_CR24","doi-asserted-by":"publisher","first-page":"193","DOI":"10.1057\/jos.2015.9","volume":"10","author":"G Dagkakis","year":"2016","unstructured":"Dagkakis G, Heavey C. A review of open source discrete event simulation software for operations research. J Simul. 2016;10(3):193\u2013206. https:\/\/doi.org\/10.1057\/jos.2015.9.","journal-title":"J Simul"},{"key":"87_CR25","unstructured":"Kuhnle A, SimPyRLFab, 2020. https:\/\/github.com\/AndreasKuhnle\/SimRLFab."},{"key":"87_CR26","unstructured":"Rinciog A, Meyer A, FabricatioRL-v2, GitHub repository. GitHub, 2023. https:\/\/github.com\/malerinc\/fabricatio-rl.git."},{"key":"87_CR27","unstructured":"Hubbs CD, Perez HD, Sarwar O, Sahinidis NV, Grossmann IE, Wassick JM, OR-Gym: a reinforcement learning library for operations research problems. 2020."},{"key":"87_CR28","unstructured":"\u2018DMPG\u2014Digital model playground\u2019. https:\/\/gitlab.com\/digitaltwinml\/DMPG"},{"key":"87_CR29","unstructured":"\u2018SimPy\u2014Discrete event simulation for Python\u2019. https:\/\/simpy.readthedocs.io\/en\/latest\/"},{"key":"87_CR30","unstructured":"Staib T, SEP_DMPG, 2024. https:\/\/github.com\/Tomstaib\/SEP_DMPG."},{"key":"87_CR31","unstructured":"DMPG Animated, 2024. https:\/\/gitlab.com\/digitaltwinml\/DMPG\/-\/tree\/Animated-stable?ref_type=heads."},{"key":"87_CR32","doi-asserted-by":"publisher","unstructured":"Belsare S, Badilla ED, Dehghanimohammadabadi M. Reinforcement learning with discrete event simulation: the premise, reality, and promise\u2019, in 2022 Winter Simulation Conference (WSC), Singapore: IEEE, Dec. 2022, pp. 2724\u20132735. https:\/\/doi.org\/10.1109\/WSC57314.2022.10015503.","DOI":"10.1109\/WSC57314.2022.10015503"},{"key":"87_CR33","unstructured":"TensorFlow Developers, TensorFlow. (Jul. 11, 2024). Zenodo."},{"issue":"7540","key":"87_CR34","doi-asserted-by":"publisher","first-page":"529","DOI":"10.1038\/nature14236","volume":"518","author":"V Mnih","year":"2015","unstructured":"Mnih V, et al. Human-level control through deep reinforcement learning. Nature. 2015;518(7540):529\u201333. https:\/\/doi.org\/10.1038\/nature14236.","journal-title":"Nature"},{"key":"87_CR35","unstructured":"\u2018TIOBE Index\u2019. https:\/\/www.tiobe.com\/tiobe-index\/."},{"key":"87_CR36","doi-asserted-by":"publisher","first-page":"100287","DOI":"10.1016\/j.jii.2021.100287","volume":"27","author":"E Guzman","year":"2022","unstructured":"Guzman E, Andres B, Poler R. Models and algorithms for production planning, scheduling and sequencing problems: a holistic framework and a systematic review. J Ind Inf Integr. 2022;27:100287. https:\/\/doi.org\/10.1016\/j.jii.2021.100287.","journal-title":"J Ind Inf Integr"},{"key":"87_CR37","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-662-49851-4","volume-title":"Process mining","author":"W Van Der Aalst","year":"2016","unstructured":"Van Der Aalst W. Process mining. Berlin: Springer, Berlin Heidelberg; 2016."}],"container-title":["Discover Internet of Things"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s43926-024-00087-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s43926-024-00087-0\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s43926-024-00087-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,12,23]],"date-time":"2024-12-23T15:12:23Z","timestamp":1734966743000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s43926-024-00087-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,23]]},"references-count":37,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,12]]}},"alternative-id":["87"],"URL":"https:\/\/doi.org\/10.1007\/s43926-024-00087-0","relation":{},"ISSN":["2730-7239"],"issn-type":[{"value":"2730-7239","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,12,23]]},"assertion":[{"value":"22 August 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"13 December 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 December 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"34"}}