{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T20:14:28Z","timestamp":1776111268395,"version":"3.50.1"},"reference-count":40,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2022,3,11]],"date-time":"2022-03-11T00:00:00Z","timestamp":1646956800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100003524","name":"Ministry of Business, Innovation and Employment","doi-asserted-by":"publisher","award":["MAUX1912"],"award-info":[{"award-number":["MAUX1912"]}],"id":[{"id":"10.13039\/501100003524","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computers"],"abstract":"<jats:p>The rise of the new generation of cyber threats demands more sophisticated and intelligent cyber defense solutions equipped with autonomous agents capable of learning to make decisions without the knowledge of human experts. Several reinforcement learning methods (e.g., Markov) for automated network intrusion tasks have been proposed in recent years. In this paper, we introduce a new generation of the network intrusion detection method, which combines a Q-learning based reinforcement learning with a deep feed forward neural network method for network intrusion detection. Our proposed Deep Q-Learning (DQL) model provides an ongoing auto-learning capability for a network environment that can detect different types of network intrusions using an automated trial-error approach and continuously enhance its detection capabilities. We provide the details of fine-tuning different hyperparameters involved in the DQL model for more effective self-learning. According to our extensive experimental results based on the NSL-KDD dataset, we confirm that the lower discount factor, which is set as 0.001 under 250 episodes of training, yields the best performance results. Our experimental results also show that our proposed DQL is highly effective in detecting different intrusion classes and outperforms other similar machine learning approaches.<\/jats:p>","DOI":"10.3390\/computers11030041","type":"journal-article","created":{"date-parts":[[2022,3,11]],"date-time":"2022-03-11T12:58:36Z","timestamp":1647003516000},"page":"41","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":190,"title":["Deep Q-Learning Based Reinforcement Learning Approach for Network Intrusion Detection"],"prefix":"10.3390","volume":"11","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0033-6706","authenticated-orcid":false,"given":"Hooman","family":"Alavizadeh","sequence":"first","affiliation":[{"name":"UNSW Institute for Cyber Security, University of New South Wales, Canberra 2612, Australia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0344-4494","authenticated-orcid":false,"given":"Hootan","family":"Alavizadeh","sequence":"additional","affiliation":[{"name":"Computer Engineering Department, Imam Reza International University, Mashhad 553-91735, Iran"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1002-057X","authenticated-orcid":false,"given":"Julian","family":"Jang-Jaccard","sequence":"additional","affiliation":[{"name":"Cybersecurity Laboratory, School of Information Technology and Electrical Engineering, Massey University, Auckland 0632, New Zealand"}]}],"member":"1968","published-online":{"date-parts":[[2022,3,11]]},"reference":[{"key":"ref_1","unstructured":"Stoecklin, M.P. (2018). Deeplocker: How AI Can Power a Stealthy New Breed of Malware. Secur. Intell., 8, Available online: https:\/\/securityintelligence.com\/deeplocker-how-ai-can-power-a-stealthy-new-breed-of-malware\/."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"8450","DOI":"10.1007\/s11227-019-03122-y","article-title":"Low-cohesion differential privacy protection for industrial Internet","volume":"76","author":"Hou","year":"2020","journal-title":"J. Supercomput."},{"key":"ref_3","unstructured":"Brundage, M., Avin, S., Clark, J., Toner, H., Eckersley, P., Garfinkel, B., Dafoe, A., Scharre, P., Zeitzoff, T., and Filar, B. (2018). The malicious use of artificial intelligence: Forecasting, prevention, and mitigation. arXiv."},{"key":"ref_4","unstructured":"Bodeau, D., and Graubart, R. (2017). Cyber Resiliency Design Principles: Selective Use throughout the Lifecycle and in Conjunction with Related Disciplines, The MITRE Corporation."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Toyoshima, K., Oda, T., Hirota, M., Katayama, K., and Barolli, L. (2020, January 24\u201326). A DQN based mobile actor node control in WSAN: Simulation results of different distributions of events considering three-dimensional environment. Proceedings of the International Conference on Emerging Internetworking, Data & Web Technologies, Kitakyushu, Japan.","DOI":"10.1007\/978-3-030-39746-3_21"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Saito, N., Oda, T., Hirata, A., Hirota, Y., Hirota, M., and Katayama, K. (2020, January 28\u201330). Design and Implementation of a DQN Based AAV. Proceedings of the International Conference on Broadband and Wireless Computing, Communication and Applications, Yonago, Japan.","DOI":"10.1007\/978-3-030-61108-8_32"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"102091","DOI":"10.1016\/j.cose.2020.102091","article-title":"Evaluating the effectiveness of shuffle and redundancy mtd techniques in the cloud","volume":"102","author":"Alavizadeh","year":"2021","journal-title":"Comput. Secur."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Sethi, K., Kumar, R., Mohanty, D., and Bera, P. (2020, January 17\u201321). Robust Adaptive Cloud Intrusion Detection System Using Advanced Deep Reinforcement Learning. Proceedings of the International Conference on Security, Privacy, and Applied Cryptography Engineering, Kolkata, India.","DOI":"10.1007\/978-3-030-66626-2_4"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Sethi, K., Kumar, R., Prajapati, N., and Bera, P. (2020, January 7\u201311). Deep reinforcement learning based intrusion detection system for cloud infrastructure. Proceedings of the 2020 International Conference on COMmunication Systems & NETworkS (COMSNETS), Bangalore, India.","DOI":"10.1109\/COMSNETS48256.2020.9027452"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"657","DOI":"10.1007\/s10207-019-00482-7","article-title":"A context-aware robust intrusion detection system: A reinforcement learning-based approach","volume":"19","author":"Sethi","year":"2020","journal-title":"Int. J. Inf. Secur."},{"key":"ref_11","unstructured":"Dang, Q.V., and Vo, T.H. (2022, January 25\u201326). Reinforcement learning for the problem of detecting intrusion in a computer system. Proceedings of the Sixth International Congress on Information and Communication Technology, Online."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Cappart, Q., Moisan, T., Rousseau, L.M., Pr\u00e9mont-Schwarz, I., and Cire, A. (2020). Combining reinforcement learning and constraint programming for combinatorial optimization. arXiv.","DOI":"10.1609\/aaai.v35i5.16484"},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"943","DOI":"10.1109\/TNSE.2020.3004312","article-title":"Aesmote: Adversarial reinforcement learning with smote for anomaly detection","volume":"8","author":"Ma","year":"2020","journal-title":"IEEE Trans. Netw. Sci. Eng."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"112963","DOI":"10.1016\/j.eswa.2019.112963","article-title":"Application of deep reinforcement learning to intrusion detection for supervised problems","volume":"141","author":"Carro","year":"2020","journal-title":"Expert Syst. Appl."},{"key":"ref_15","first-page":"262","article-title":"Off-Policy Q-learning Technique for Intrusion Response in Network Security","volume":"136","author":"Stefanova","year":"2018","journal-title":"World Acad. Sci. Eng. Technol. Int. Sci. Index"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Fran\u00e7ois-Lavet, V., Henderson, P., Islam, R., Bellemare, M.G., and Pineau, J. (2018). An introduction to deep reinforcement learning. arXiv.","DOI":"10.1561\/9781680835397"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"8198","DOI":"10.1109\/TII.2021.3063489","article-title":"Shifting Deep Reinforcement Learning Algorithm towards Training Directly in Transient Real-World Environment: A Case Study in Powertrain Control","volume":"17","author":"Hu","year":"2021","journal-title":"IEEE Trans. Ind. Inform."},{"key":"ref_18","first-page":"102923","article-title":"Attention based multi-agent intrusion detection systems using reinforcement learning","volume":"61","author":"Sethi","year":"2021","journal-title":"J. Inf. Secur. Appl."},{"key":"ref_19","unstructured":"Nguyen, T.T., and Reddi, V.J. (2019). Deep Reinforcement Learning for Cyber Security. arXiv."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"96","DOI":"10.1016\/j.comnet.2019.05.013","article-title":"Adversarial environment reinforcement learning algorithm for intrusion detection","volume":"159","author":"Caminero","year":"2019","journal-title":"Comput. Netw."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Tavallaee, M., Bagheri, E., Lu, W., and Ghorbani, A.A. (2009, January 8\u201310). A detailed analysis of the KDD CUP 99 data set. Proceedings of the 2009 IEEE Symposium on Computational Intelligence for Security and Defense Applications, Ottawa, ON, Canada.","DOI":"10.1109\/CISDA.2009.5356528"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"184","DOI":"10.1109\/COMST.2015.2402161","article-title":"Intrusion Detection in 802.11 Networks: Empirical Evaluation of Threats and a Public Dataset","volume":"18","author":"Kolias","year":"2016","journal-title":"IEEE Commun. Surv. Tutorials"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Iannucci, S., Barba, O.D., Cardellini, V., and Banicescu, I. (2019, January 16\u201320). A performance evaluation of deep reinforcement learning for model-based intrusion response. Proceedings of the 2019 IEEE 4th International Workshops on Foundations and Applications of Self* Systems (FAS* W), Umea, Sweden.","DOI":"10.1109\/FAS-W.2019.00047"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1016\/j.future.2020.03.018","article-title":"A hybrid model-free approach for the near-optimal intrusion response control of non-stationary systems","volume":"109","author":"Iannucci","year":"2020","journal-title":"Future Gener. Comput. Syst."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"270","DOI":"10.1016\/j.engappai.2015.01.013","article-title":"Distributed response to network intrusions using multiagent reinforcement learning","volume":"41","author":"Malialis","year":"2015","journal-title":"Eng. Appl. Artif. Intell."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"134","DOI":"10.1109\/TDSC.2017.2751478","article-title":"Real-time multistep attack prediction based on hidden markov models","volume":"17","author":"Holgado","year":"2017","journal-title":"IEEE Trans. Dependable Secur. Comput."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"525","DOI":"10.1016\/j.cose.2011.06.002","article-title":"Toward cost-sensitive self-optimizing anomaly detection and response in autonomic networks","volume":"30","author":"Zhang","year":"2011","journal-title":"Comput. Secur."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"237","DOI":"10.1016\/j.ins.2014.02.139","article-title":"A multi-attribute decision model for intrusion response system","volume":"270","author":"Fessi","year":"2014","journal-title":"Inf. Sci."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"16605","DOI":"10.1007\/s00500-020-04963-z","article-title":"DeepBot: A time-based botnet detection with deep learning","volume":"24","author":"Shi","year":"2020","journal-title":"Soft Comput."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Ganju, K., Wang, Q., Yang, W., Gunter, C.A., and Borisov, N. (2018, January 15\u201319). Property inference attacks on fully connected neural networks using permutation invariant representations. Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, Toronto, ON, Canada.","DOI":"10.1145\/3243734.3243834"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"147","DOI":"10.1016\/j.comnet.2019.01.023","article-title":"Internet of Things: A survey on machine learning-based intrusion detection approaches","volume":"151","author":"Papa","year":"2019","journal-title":"Comput. Netw."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"386","DOI":"10.1016\/j.ins.2019.10.069","article-title":"A hybrid deep learning model for efficient intrusion detection in big data environment","volume":"513","author":"Hassan","year":"2020","journal-title":"Inf. Sci."},{"key":"ref_33","first-page":"107","article-title":"A comparison study for intrusion database (Kdd99, Nsl-Kdd) based on self organization map (SOM) artificial neural network","volume":"8","author":"Ibrahim","year":"2013","journal-title":"J. Eng. Sci. Technol."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"32464","DOI":"10.1109\/ACCESS.2020.2973730","article-title":"Network intrusion detection combined hybrid sampling with deep hierarchical network","volume":"8","author":"Jiang","year":"2020","journal-title":"IEEE Access"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Yang, K., Liu, J., Zhang, C., and Fang, Y. (2018, January 29\u201331). Adversarial examples against the deep learning based network intrusion detection systems. Proceedings of the MILCOM 2018\u20142018 IEEE Military Communications Conference (MILCOM), Los Angeles, CA, USA.","DOI":"10.1109\/MILCOM.2018.8599759"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Alavizadeh, H., Alavizadeh, H., Kim, D.S., Jang-Jaccard, J., and Torshiz, M.N. (2019, January 4\u20136). An automated security analysis framework and implementation for MTD techniques on cloud. Proceedings of the International Conference on Information Security and Cryptology, Seoul, Korea.","DOI":"10.1007\/978-3-030-40921-0_9"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Alavizadeh, H., Alavizadeh, H., and Jang-Jaccard, J. (January, January 29). Cyber situation awareness monitoring and proactive response for enterprises on the cloud. Proceedings of the 2020 IEEE 19th International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom), Guangzhou, China.","DOI":"10.1109\/TrustCom50675.2020.00171"},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"171542","DOI":"10.1109\/ACCESS.2020.3024991","article-title":"Multi-Loss Siamese Neural Network with Batch Normalization Layer for Malware Detection","volume":"8","author":"Zhu","year":"2020","journal-title":"IEEE Access"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"McIntosh, T., Jang-Jaccard, J., Watters, P., and Susnjak, T. (2019, January 12\u201315). The inadequacy of entropy-based ransomware detection. Proceedings of the International Conference on Neural Information Processing, Sydney, Australia.","DOI":"10.1007\/978-3-030-36802-9_20"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"229033","DOI":"10.1109\/ACCESS.2020.3043421","article-title":"Large-Scale Outlier Detection for Low-Cost PM10 Sensors","volume":"8","author":"Wei","year":"2020","journal-title":"IEEE Access"}],"container-title":["Computers"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2073-431X\/11\/3\/41\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:35:08Z","timestamp":1760135708000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2073-431X\/11\/3\/41"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3,11]]},"references-count":40,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2022,3]]}},"alternative-id":["computers11030041"],"URL":"https:\/\/doi.org\/10.3390\/computers11030041","relation":{},"ISSN":["2073-431X"],"issn-type":[{"value":"2073-431X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,3,11]]}}}