{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,3]],"date-time":"2026-07-03T17:26:35Z","timestamp":1783099595992,"version":"3.54.6"},"reference-count":43,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2022,11,30]],"date-time":"2022-11-30T00:00:00Z","timestamp":1669766400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Natural Science and Engineering Research Council (NSERC), Canada"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>Powered by advances in information and internet technologies, network-based applications have developed rapidly, and cybersecurity has grown more critical. Inspired by Reinforcement Learning (RL) success in many domains, this paper proposes an Intrusion Detection System (IDS) to improve cybersecurity. The IDS based on two RL algorithms, i.e., Deep Q-Learning and Policy Gradient, is carefully formulated, strategically designed, and thoroughly evaluated at the packet-level and flow-level using the CICDDoS2019 dataset. Compared to other research work in a similar line of research, this paper is focused on providing a systematic and complete design paradigm of IDS based on RL algorithms, at both the packet and flow levels. For the packet-level RL-based IDS, first, the session data are transformed into images via an image embedding method proposed in this work. A comparison between 1D-Convolutional Neural Networks (1D-CNN) and CNN for extracting features from these images (for further RL agent training) is drawn from the quantitative results. In addition, an anomaly detection module is designed to detect unknown network traffic. For flow-level IDS, a Conditional Generative Adversarial Network (CGAN) and the \u03b5-greedy strategy are adopted in designing the exploration module for RL agent training. To improve the robustness of the intrusion detection, a sample agent with a complement reward policy of the RL agent is introduced for the purpose of adversarial training. The experimental results of the proposed RL-based IDS show improved results over the state-of-the-art algorithms presented in the literature for packet-level and flow-level IDS.<\/jats:p>","DOI":"10.3390\/a15120453","type":"journal-article","created":{"date-parts":[[2022,11,30]],"date-time":"2022-11-30T04:32:53Z","timestamp":1669782773000},"page":"453","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":16,"title":["Packet-Level and Flow-Level Network Intrusion Detection Based on Reinforcement Learning and Adversarial Training"],"prefix":"10.3390","volume":"15","author":[{"given":"Bin","family":"Yang","sequence":"first","affiliation":[{"name":"Department of Electrical and Computer Engineering, University of Alberta, Edmonton, AB T6G 2R3, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2036-1642","authenticated-orcid":false,"given":"Muhammad Haseeb","family":"Arshad","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, University of Alberta, Edmonton, AB T6G 2R3, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8205-9708","authenticated-orcid":false,"given":"Qing","family":"Zhao","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, University of Alberta, Edmonton, AB T6G 2R3, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,11,30]]},"reference":[{"key":"ref_1","unstructured":"Smaha, S. (1988, January 12\u201316). Haystack: An intrusion detection system. Proceedings of the [Proceedings 1988] Fourth Aerospace Computer Security Applications, Orlando, FL, USA."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Hwang, R.H., Peng, M.C., Nguyen, V.L., and Chang, Y.L. (2019). An LSTM-based deep learning approach for classifying malicious traffic at the packet level. Appl. Sci., 9.","DOI":"10.3390\/app9163414"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Ge, M., Fu, X., Syed, N., Baig, Z., Teo, G., and Robles-Kelly, A. (2019, January 1\u20133). Deep learning-based intrusion detection for iot networks. Proceedings of the 2019 IEEE 24th Pacific Rim International Symposium on Dependable Computing (PRDC), Kyoto, Japan.","DOI":"10.1109\/PRDC47002.2019.00056"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Doshi, R., Apthorpe, N., and Feamster, N. (2018, January 24). Machine learning ddos detection for consumer internet of things devices. Proceedings of the 2018 IEEE Security and Privacy Workshops (SPW), San Francisco, CA, USA.","DOI":"10.1109\/SPW.2018.00013"},{"key":"ref_5","unstructured":"Radford, B., Apolonio, L., Trias, A., and Simpson, J. (2018). Network Traffic Anomaly Detection Using Recurrent Neural Networks. arXiv."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"705","DOI":"10.1016\/j.neucom.2020.07.138","article-title":"Intrusion detection approach based on optimised artificial neural network","volume":"452","author":"Pawlicki","year":"2021","journal-title":"Neurocomputing"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1016\/j.neucom.2019.11.016","article-title":"A novel statistical analysis and autoencoder driven intelligent intrusion detection approach","volume":"387","author":"Ieracitano","year":"2020","journal-title":"Neurocomputing"},{"key":"ref_8","unstructured":"Goodfellow, I., Bengio, Y., Courville, A., and Bengio, Y. (2016). Deep Learning, MIT Press."},{"key":"ref_9","unstructured":"Bodeau, D., and Graubart, R. (2017). Cyber Resiliency Design Principles Selective Use Throughout the Lifecycle and in Conjunction with Related Disciplines, MITRE CORP. Technical Report."},{"key":"ref_10","unstructured":"Beal, M., Ghahramani, Z., and Rasmussen, C. (2001, January 3\u20138). The infinite hidden Markov model. Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Toyoshima, K., Oda, T., Hirota, M., Katayama, K., and Barolli, L. (2020, January 24\u201326). A DQN based mobile actor node control in WSAN: Simulation results of different distributions of events considering three-dimensional environment. Proceedings of the International Conference on Emerging Internetworking, Data & Web Technologies, Kitakyushu, Japan.","DOI":"10.1007\/978-3-030-39746-3_21"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Saito, N., Oda, T., Hirata, A., Hirota, Y., Hirota, M., and Katayama, K. (2020, January 28\u201330). Design and implementation of a DQN based AAV. Proceedings of the International Conference on Broadband and Wireless Computing, Communication and Applications, Yonago, Japan.","DOI":"10.1007\/978-3-030-61108-8_32"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"102091","DOI":"10.1016\/j.cose.2020.102091","article-title":"Evaluating the effectiveness of shuffle and redundancy mtd techniques in the cloud","volume":"102","author":"Alavizadeh","year":"2021","journal-title":"Comput. Secur."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Sethi, K., Kumar, R., Mohanty, D., and Bera, P. (2020, January 17\u201321). Robust adaptive cloud intrusion detection system using advanced deep reinforcement learning. Proceedings of the International Conference on Security, Privacy, and Applied Cryptography Engineering, Kolkata, India.","DOI":"10.1007\/978-3-030-66626-2_4"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Sethi, K., Kumar, R., Prajapati, N., and Bera, P. (2020, January 7\u201311). Deep reinforcement learning based intrusion detection system for cloud infrastructure. Proceedings of the 2020 International Conference on COMmunication Systems & NETworkS (COMSNETS), Bengaluru, India.","DOI":"10.1109\/COMSNETS48256.2020.9027452"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"657","DOI":"10.1007\/s10207-019-00482-7","article-title":"A context-aware robust intrusion detection system: A reinforcement learning-based approach","volume":"19","author":"Sethi","year":"2020","journal-title":"Int. J. Inf. Secur."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Cappart, Q., Moisan, T., Rousseau, L.M., Pr\u00e9mont-Schwarz, I., and Cire, A.A. (2021, January 2\u20139). Combining reinforcement learning and constraint programming for combinatorial optimization. Proceedings of the AAAI Conference on Artificial Intelligence, Virtual.","DOI":"10.1609\/aaai.v35i5.16484"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"943","DOI":"10.1109\/TNSE.2020.3004312","article-title":"Aesmote: Adversarial reinforcement learning with smote for anomaly detection","volume":"8","author":"Ma","year":"2020","journal-title":"IEEE Trans. Netw. Sci. Eng."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"112963","DOI":"10.1016\/j.eswa.2019.112963","article-title":"Application of deep reinforcement learning to intrusion detection for supervised problems","volume":"141","author":"Carro","year":"2020","journal-title":"Expert Syst. Appl."},{"key":"ref_20","first-page":"266","article-title":"Off-policy q-learning technique for intrusion response in network security","volume":"12","author":"Stefanova","year":"2018","journal-title":"Int. J. Comput. Inf. Eng."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"219","DOI":"10.1561\/2200000071","article-title":"An introduction to deep reinforcement learning","volume":"11","author":"Henderson","year":"2018","journal-title":"Found. Trends Mach. Learn."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"8198","DOI":"10.1109\/TII.2021.3063489","article-title":"Shifting deep reinforcement learning algorithm toward training directly in transient real-world environment: A case study in powertrain control","volume":"17","author":"Hu","year":"2021","journal-title":"IEEE Trans. Ind. Inform."},{"key":"ref_23","first-page":"102923","article-title":"Attention based multi-agent intrusion detection systems using reinforcement learning","volume":"61","author":"Sethi","year":"2021","journal-title":"J. Inf. Secur. Appl."},{"key":"ref_24","unstructured":"Nguyen, T.T., and Reddi, V.J. (2021). Deep reinforcement learning for cyber security. IEEE Trans. Neural Netw. Learn. Syst., 1\u201317."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"96","DOI":"10.1016\/j.comnet.2019.05.013","article-title":"Adversarial environment reinforcement learning algorithm for intrusion detection","volume":"159","author":"Caminero","year":"2019","journal-title":"Comput. Netw."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Tavallaee, M., Bagheri, E., Lu, W., and Ghorbani, A.A. (2009, January 8\u201310). A detailed analysis of the KDD CUP 99 data set. Proceedings of the 2009 IEEE Symposium on Computational Intelligence for Security and Defense Applications, Ottawa, ON, Canada.","DOI":"10.1109\/CISDA.2009.5356528"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"184","DOI":"10.1109\/COMST.2015.2402161","article-title":"Intrusion detection in 802.11 networks: Empirical evaluation of threats and a public dataset","volume":"18","author":"Kolias","year":"2015","journal-title":"IEEE Commun. Surv. Tutor."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Iannucci, S., Barba, O.D., Cardellini, V., and Banicescu, I. (2019, January 16\u201320). A performance evaluation of deep reinforcement learning for model-based intrusion response. Proceedings of the 2019 IEEE 4th International Workshops on Foundations and Applications of Self* Systems (FAS*W), Umea, Sweden.","DOI":"10.1109\/FAS-W.2019.00047"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1016\/j.future.2020.03.018","article-title":"A hybrid model-free approach for the near-optimal intrusion response control of non-stationary systems","volume":"109","author":"Iannucci","year":"2020","journal-title":"Future Gener. Comput. Syst."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"270","DOI":"10.1016\/j.engappai.2015.01.013","article-title":"Distributed response to network intrusions using multiagent reinforcement learning","volume":"41","author":"Malialis","year":"2015","journal-title":"Eng. Appl. Artif. Intell."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"134","DOI":"10.1109\/TDSC.2017.2751478","article-title":"Real-time multistep attack prediction based on hidden markov models","volume":"17","author":"Holgado","year":"2017","journal-title":"IEEE Trans. Dependable Secur. Comput."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"525","DOI":"10.1016\/j.cose.2011.06.002","article-title":"Toward cost-sensitive self-optimizing anomaly detection and response in autonomic networks","volume":"30","author":"Zhang","year":"2011","journal-title":"Comput. Secur."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"237","DOI":"10.1016\/j.ins.2014.02.139","article-title":"A multi-attribute decision model for intrusion response system","volume":"270","author":"Fessi","year":"2014","journal-title":"Inf. Sci."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Alavizadeh, H., Alavizadeh, H., and Jang-Jaccard, J. (2022). Deep Q-Learning based Reinforcement Learning Approach for Network Intrusion Detection. Computers, 11.","DOI":"10.3390\/computers11030041"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"106460","DOI":"10.1016\/j.compeleceng.2019.106460","article-title":"Designing online network intrusion detection using deep auto-encoder Q-learning","volume":"79","author":"Kim","year":"2019","journal-title":"Comput. Electr. Eng."},{"key":"ref_36","unstructured":"Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013, January 2\u20134). Efficient estimation of word representations in vector space. Proceedings of the Workshop at International Conference on Learning Representations (ICLR), Scottsdale, AZ, USA."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"16548","DOI":"10.1109\/JIOT.2021.3074830","article-title":"Agents-Based Algorithm for a Distributed Information System in Internet of Things","volume":"8","author":"Forestiero","year":"2021","journal-title":"IEEE Internet Things J."},{"key":"ref_38","unstructured":"Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. (2016). Openai gym. arXiv."},{"key":"ref_39","unstructured":"Dhariwal, P., Hesse, C., Klimov, O., Nichol, A., Plappert, M., Radford, A., Schulman, J., Sidor, S., Wu, Y., and Zhokhov, P. (2022, November 24). OpenAI Baselines, a Set of High-Quality Reinforcement Learning Algorithms. Available online: https:\/\/github.com\/openai\/baselines."},{"key":"ref_40","unstructured":"Hasselt, H. (2010, January 6\u20139). Double Q-learning. Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada."},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Wang, W., Zhu, M., Wang, J., Zeng, X., and Yang, Z. (2017, January 22\u201324). End-to-end encrypted traffic classification with one-dimensional convolution neural networks. Proceedings of the 2017 IEEE International Conference on Intelligence and Security Informatics (ISI), Beijing, China.","DOI":"10.1109\/ISI.2017.8004872"},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Sharafaldin, I., Lashkari, A.H., Hakak, S., and Ghorbani, A.A. (2019, January 1\u20133). Developing realistic distributed denial of service (DDoS) attack dataset and taxonomy. Proceedings of the 2019 International Carnahan Conference on Security Technology (ICCST), Chennai, India.","DOI":"10.1109\/CCST.2019.8888419"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Sch\u00f6lkopf, B., Luo, Z., and Vovk, V. (2013). Explaining AdaBoost. Empirical Inference: Festschrift in Honor of Vladimir N. Vapnik, Springer.","DOI":"10.1007\/978-3-642-41136-6"}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/15\/12\/453\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T01:29:55Z","timestamp":1760146195000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/15\/12\/453"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,11,30]]},"references-count":43,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2022,12]]}},"alternative-id":["a15120453"],"URL":"https:\/\/doi.org\/10.3390\/a15120453","relation":{},"ISSN":["1999-4893"],"issn-type":[{"value":"1999-4893","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,11,30]]}}}