{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,19]],"date-time":"2026-01-19T08:29:30Z","timestamp":1768811370435,"version":"3.49.0"},"reference-count":48,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2022,2,28]],"date-time":"2022-02-28T00:00:00Z","timestamp":1646006400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>Discrete event modeling and simulation and reinforcement learning are two frameworks suited for cyberphysical system design, which, when combined, can give powerful tools for system optimization or decision making process for example. This paper describes how discrete event modeling and simulation could be integrated into reinforcement learning concepts and tools in order to assist in the realization of reinforcement learning systems, more specially considering the temporal, hierarchical, and multi-agent aspects. An overview of these different improvements are given based on the implementation of the Q-Learning reinforcement learning algorithm in the framework of the Discrete Event system Specification (DEVS) and System Entity Structure (SES) formalisms.<\/jats:p>","DOI":"10.3390\/info13030121","type":"journal-article","created":{"date-parts":[[2022,2,28]],"date-time":"2022-02-28T20:11:14Z","timestamp":1646079074000},"page":"121","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":15,"title":["Discrete Event Modeling and Simulation for Reinforcement Learning System Design"],"prefix":"10.3390","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0793-8742","authenticated-orcid":false,"given":"Laurent","family":"Capocchi","sequence":"first","affiliation":[{"name":"SPE UMR CNRS 6134, University of Corsica, 20250 Corte, France"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9143-532X","authenticated-orcid":false,"given":"Jean-Fran\u00e7ois","family":"Santucci","sequence":"additional","affiliation":[{"name":"SPE UMR CNRS 6134, University of Corsica, 20250 Corte, France"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,2,28]]},"reference":[{"key":"ref_1","unstructured":"Alpaydin, E. (2016). Machine Learning: The New AI, The MIT Press."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Busoniu, L., Babuska, R., and Schutter, B.D. (2006, January 5\u20138). Multi-Agent Reinforcement Learning: A Survey. Proceedings of the Ninth International Conference on Control, Automation, Robotics and Vision, ICARCV 2006, Singapore.","DOI":"10.1109\/ICARCV.2006.345353"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Zeigler, B.P., Muzy, A., and Kofman, E. (2019). Theory of Modeling and Simulation, Academic Press. [3rd ed.].","DOI":"10.1016\/B978-0-12-813370-5.00010-9"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1340006","DOI":"10.1142\/S1793962313400060","article-title":"System entity structures for suites of simulation models","volume":"04","author":"Zeigler","year":"2013","journal-title":"Int. J. Model. Simul. Sci. Comput."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Puterman, M.L. (1994). Markov Decision Processes: Discrete Stochastic Dynamic Programming, John Wiley & Sons, Inc.. [1st ed.].","DOI":"10.1002\/9780470316887"},{"key":"ref_6","unstructured":"Sutton, R.S., and Barto, A.G. (2018). Reinforcement Learning: An Introduction, A Bradford Book."},{"key":"ref_7","unstructured":"Bellman, R.E. (2003). Dynamic Programming, Dover Publications, Inc."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Yu, H., Mahmood, A.R., and Sutton, R.S. (2017, January 16\u201319). On Generalized Bellman Equations and Temporal-Difference Learning. Proceedings of the Advances in Artificial Intelligence\u201430th Canadian Conference on Artificial Intelligence, Canadian AI 2017, Edmonton, AB, Canada.","DOI":"10.1007\/978-3-319-57351-9_1"},{"key":"ref_9","unstructured":"Watkins, C.J.C.H. (1989). Learning from Delayed Rewards. [Ph.D. Thesis, King\u2019s College]."},{"key":"ref_10","first-page":"1","article-title":"Learning Rates for Q-learning","volume":"5","author":"Mansour","year":"2004","journal-title":"J. Mach. Learn. Res."},{"key":"ref_11","unstructured":"Russell, S., and Norvig, P. (2009). Artificial Intelligence: A Modern Approach, Prentice Hall Press. [3rd ed.]."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"7363","DOI":"10.1109\/TSMC.2020.2967936","article-title":"Deep Q-Learning With Q-Matrix Transfer Learning for Novel Fire Evacuation Environment","volume":"51","author":"Sharma","year":"2021","journal-title":"IEEE Trans. Syst. Man Cybern. Syst."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"177734","DOI":"10.1109\/ACCESS.2020.3020590","article-title":"An Improved DDPG and Its Application Based on the Double-Layer BP Neural Network","volume":"8","author":"Zhang","year":"2020","journal-title":"IEEE Access"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Fishwick, P.A., and Modjeski, R.B. (1991). Application of Artificial Intelligence Techniques to Simulation. Knowledge-Based Simulation: Methodology and Application, Springer.","DOI":"10.1007\/978-1-4612-3040-3"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Wallis, L., and Paich, M. (2017, January 3\u20136). Integrating artifical intelligence with anylogic simulation. Proceedings of the 2017 Winter Simulation Conference (WSC), Las Vegas, NV, USA.","DOI":"10.1109\/WSC.2017.8248156"},{"key":"ref_16","unstructured":"Foo, N.Y., and Peppas, P. (2004). Systems Theory: Melding the AI and Simulation Perspectives. Artificial Intelligence and Simulation, Proceedings of the 13th International Conference on AI, Simulation, and Planning in High Autonomy Systems, AIS 2004, Jeju Island, Korea, 4\u20136 October 2004, Springer. Revised Selected Papers."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Meraji, S., and Tropper, C. (2010, January 13\u201316). A Machine Learning Approach for Optimizing Parallel Logic Simulation. Proceedings of the 2010 39th International Conference on Parallel Processing, San Diego, CA, USA.","DOI":"10.1109\/ICPP.2010.62"},{"key":"ref_18","unstructured":"Floyd, M.W., and Wainer, G.A. (2010, January 11\u201314). Creation of DEVS Models Using Imitation Learning. Proceedings of the 2010 Summer Computer Simulation Conference, SCSC \u201910, Ottawa, ON, Canada."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Belousov, B., Abdulsamad, H., Klink, P., Parisi, S., and Peters, J. (2021). Reward Function Design in Reinforcement Learning. Reinforcement Learning Algorithms: Analysis and Applications, Springer International Publishing.","DOI":"10.1007\/978-3-030-41188-6"},{"key":"ref_20","unstructured":"Zhao, S., Song, J., and Ermon, S. (2017, January 6\u201311). Learning Hierarchical Features from Deep Generative Models. Proceedings of the 34th International Conference on Machine Learning, Sydney, NSW, Australia."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Canese, L., Cardarilli, G.C., Di Nunzio, L., Fazzolari, R., Giardino, D., Re, M., and Span\u00f2, S. (2021). Multi-Agent Reinforcement Learning: A Review of Challenges and Applications. Appl. Sci., 11.","DOI":"10.3390\/app11114948"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Zeigler, B.P., and Sarjoughian, H.S. (2013). System Entity Structure Basics. Guide to Modeling and Simulation of Systems of Systems, Springer. Simulation Foundations, Methods and Applications.","DOI":"10.1007\/978-0-85729-865-2"},{"key":"ref_23","unstructured":"Pardo, F., Tavakoli, A., Levdik, V., and Kormushev, P. (2018, January 10\u201315). Time Limits in Reinforcement Learning. Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholm, Sweden."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"28","DOI":"10.1049\/iet-csr.2018.0001","article-title":"Time-in-Action Reinforcement Learning","volume":"1","author":"Zhu","year":"2019","journal-title":"IET Cyber-Syst. Robot."},{"key":"ref_25","unstructured":"Bradtke, S., and Duff, M. (1994). Reinforcement Learning Methods for Continuous-Time Markov Decision Problems. Advances in Neural Information Processing Systems 7, MIT Press."},{"key":"ref_26","unstructured":"Mahadevan, S., Marchalleck, N., Das, T., and Gosavi, A. (1997, January 8\u201312). Self-Improving Factory Simulation using Continuous-time Average-Reward Reinforcement Learning. Proceedings of the 14th International Conference on Machine Learning, Nashville, TN, USA."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"181","DOI":"10.1016\/S0004-3702(99)00052-1","article-title":"Between MDPs and semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning","volume":"112","author":"Sutton","year":"1999","journal-title":"Artif. Intell."},{"key":"ref_28","unstructured":"Rachelson, E., Quesnel, G., Garcia, F., and Fabiani, P. (2008, January 21\u201325). A Simulation-based Approach for Solving Generalized Semi-Markov Decision Processes. Proceedings of the 2008 Conference on ECAI 2008: 18th European Conference on Artificial Intelligence, Patras, Greece."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Seo, C., Zeigler, B.P., and Kim, D. (2018, January 15\u201318). DEVS Markov Modeling and Simulation: Formal Definition and Implementation. Proceedings of the Theory of Modeling and Simulation Symposium, TMS \u201918, Baltimore, MD, USA.","DOI":"10.1145\/3213187.3213188"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"227","DOI":"10.1613\/jair.639","article-title":"Hierarchical reinforcement learning with the MAXQ value function decomposition","volume":"13","author":"Dietterich","year":"2000","journal-title":"J. Artif. Intell. Res."},{"key":"ref_31","unstructured":"Vezhnevets, A.S., Osindero, S., Schaul, T., Heess, N., Jaderberg, M., Silver, D., and Kavukcuoglu, K. (2017, January 6\u201311). FeUdal Networks for Hierarchical Reinforcement Learning. Proceedings of the 34th International Conference on Machine Learning, Sydney, NSW, Australia."},{"key":"ref_32","unstructured":"Parr, R., and Russell, S. (1997, January 1\u20136). Reinforcement Learning with Hierarchies of Machines. Proceedings of the 1997 Conference on Advances in Neural Information Processing Systems 10, NIPS \u201997, Denver, CO, USA."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Kessler, C., Capocchi, L., Santucci, J.F., and Zeigler, B. (2017, January 3\u20136). Hierarchical Markov Decision Process Based on Devs Formalism. Proceedings of the 2017 Winter Simulation Conference, WSC \u201917, Las Vegas, NV, USA.","DOI":"10.1109\/WSC.2017.8247850"},{"key":"ref_34","unstructured":"Bonaccorso, G. (2017). Machine Learning Algorithms: A Reference Guide to Popular Algorithms for Data Science and Machine Learning, Packt Publishing."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Yoshizawa, A., Nishiyama, H., Iwasaki, H., and Mizoguchi, F. (2016, January 22\u201323). Machine-learning approach to analysis of driving simulation data. Proceedings of the 2016 IEEE 15th International Conference on Cognitive Informatics Cognitive Computing (ICCI*CC), Palo Alto, CA, USA.","DOI":"10.1109\/ICCI-CC.2016.7862067"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Malakar, P., Balaprakash, P., Vishwanath, V., Morozov, V., and Kumaran, K. (2018, January 12). Benchmarking Machine Learning Methods for Performance Modeling of Scientific Applications. Proceedings of the 2018 IEEE\/ACM Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS), Dallas, TX, USA.","DOI":"10.1109\/PMBS.2018.8641686"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Elbattah, M., and Molloy, O. (2017, January 3\u20136). Learning about systems using machine learning: Towards more data-driven feedback loops. Proceedings of the 2017 Winter Simulation Conference (WSC), Las Vegas, NV, USA.","DOI":"10.1109\/WSC.2017.8247895"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Elbattah, M., and Molloy, O. (2018, January 23\u201325). ML-Aided Simulation: A Conceptual Framework for Integrating Simulation Models with Machine Learning. Proceedings of the 2018 ACM SIGSIM Conference on Principles of Advanced Discrete Simulation, SIGSIM-PADS \u201918, Rome, Italy.","DOI":"10.1145\/3200921.3200933"},{"key":"ref_39","unstructured":"Saadawi, H., Wainer, G., and Pliego, G. (2016, January 3\u20136). DEVS execution acceleration with machine learning. Proceedings of the 2016 Symposium on Theory of Modeling and Simulation (TMS-DEVS), Pasadena, CA, USA."},{"key":"ref_40","unstructured":"Toma, S. (2014). Detection and Identication Methodology for Multiple Faults in Complex Systems Using Discrete-Events and Neural Networks: Applied to the Wind Turbines Diagnosis. [Ph.D. Thesis, University of Corsica]."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Bin Othman, M.S., and Tan, G. (2018, January 15\u201317). Machine Learning Aided Simulation of Public Transport Utilization. Proceedings of the 2018 IEEE\/ACM 22nd International Symposium on Distributed Simulation and Real Time Applications (DS-RT), Madrid, Spain.","DOI":"10.1109\/DISTRA.2018.8601011"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"De la Fuente, R., Erazo, I., and Smith, R.L. (2018, January 9\u201312). Enabling Intelligent Processes in Simulation Utilizing the Tensorflow Deep Learning Resources. Proceedings of the 2018 Winter Simulation Conference (WSC), Gothenburg, Sweden.","DOI":"10.1109\/WSC.2018.8632539"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Feng, K., Chen, S., and Lu, W. (2018, January 9\u201312). Machine Learning Based Construction Simulation and Optimization. Proceedings of the 2018 Winter Simulation Conference (WSC), Gothenburg, Sweden.","DOI":"10.1109\/WSC.2018.8632290"},{"key":"ref_44","unstructured":"Liu, F., Ma, P., and Yang, M. (2005, January 18\u201321). A validation methodology for AI simulation models. Proceedings of the 2005 International Conference on Machine Learning and Cybernetics, Guangzhou, China."},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Elbattah, M., Molloy, O., and Zeigler, B.P. (2018, January 9\u201312). Designing Care Pathways Using Simulation Modeling and Machine Learning. Proceedings of the 2018 Winter Simulation Conference (WSC), Gothenburg, Sweden.","DOI":"10.1109\/WSC.2018.8632360"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Arumugam, K., Ranjan, D., Zubair, M., Terzi\u0107, B., Godunov, A., and Islam, T. (2017, January 14\u201317). A Machine Learning Approach for Efficient Parallel Simulation of Beam Dynamics on GPUs. Proceedings of the 2017 46th International Conference on Parallel Processing (ICPP), Bristol, UK.","DOI":"10.1109\/ICPP.2017.55"},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Batata, O., Augusto, V., and Xie, X. (2018, January 9\u201312). Mixed Machine Learning and Agent-Based Simulation for Respite Care Evaluation. Proceedings of the 2018 Winter Simulation Conference (WSC), Gothenburg, Sweden.","DOI":"10.1109\/WSC.2018.8632385"},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"John, L.K. (2017, January 24\u201325). Machine learning for performance and power modeling\/prediction. Proceedings of the 2017 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), Santa Rosa, CA, USA.","DOI":"10.1109\/ISPASS.2017.7975264"}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/13\/3\/121\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:29:32Z","timestamp":1760135372000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/13\/3\/121"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,2,28]]},"references-count":48,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2022,3]]}},"alternative-id":["info13030121"],"URL":"https:\/\/doi.org\/10.3390\/info13030121","relation":{},"ISSN":["2078-2489"],"issn-type":[{"value":"2078-2489","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,2,28]]}}}