{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,25]],"date-time":"2025-11-25T06:57:10Z","timestamp":1764053830813,"version":"build-2065373602"},"reference-count":50,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2022,11,25]],"date-time":"2022-11-25T00:00:00Z","timestamp":1669334400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Major Program for Technological Innovation 2030-New Generation Artifical Intelligence","award":["2020AAA0107700","62172244","ZR2020YQ06","ZR2021MF132","2021KJ001","2022JBZ01-01"],"award-info":[{"award-number":["2020AAA0107700","62172244","ZR2020YQ06","ZR2021MF132","2021KJ001","2022JBZ01-01"]}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["2020AAA0107700","62172244","ZR2020YQ06","ZR2021MF132","2021KJ001","2022JBZ01-01"],"award-info":[{"award-number":["2020AAA0107700","62172244","ZR2020YQ06","ZR2021MF132","2021KJ001","2022JBZ01-01"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Shandong Provincial Natural Science Foundation","award":["2020AAA0107700","62172244","ZR2020YQ06","ZR2021MF132","2021KJ001","2022JBZ01-01"],"award-info":[{"award-number":["2020AAA0107700","62172244","ZR2020YQ06","ZR2021MF132","2021KJ001","2022JBZ01-01"]}]},{"name":"Young innovation team of colleges and universities in 308 Shandong province","award":["2020AAA0107700","62172244","ZR2020YQ06","ZR2021MF132","2021KJ001","2022JBZ01-01"],"award-info":[{"award-number":["2020AAA0107700","62172244","ZR2020YQ06","ZR2021MF132","2021KJ001","2022JBZ01-01"]}]},{"name":"Pilot Project for Integrated Innovation of Science, Education and Industry of Qilu University of Technology (Shandong Academy of Sciences)","award":["2020AAA0107700","62172244","ZR2020YQ06","ZR2021MF132","2021KJ001","2022JBZ01-01"],"award-info":[{"award-number":["2020AAA0107700","62172244","ZR2020YQ06","ZR2021MF132","2021KJ001","2022JBZ01-01"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>With the rapid development of Industrial Internet of Things technology, the industrial control system (ICS) faces more and more security threats, which may lead to serious risks and extensive damage. Naturally, it is particularly important to construct efficient, robust, and low-cost protection strategies for ICS. However, how to construct an objective function of optimal security protection strategy considering both the security risk and protection cost, and to find the optimal solution, are all significant challenges. In this paper, we propose an optimal security protection strategy selection model and develop an optimization framework based on Q-Learning particle swarm optimization (QLPSO). The model performs security risk assessment of ICS by introducing the protection strategy into the Bayesian attack graph. The QLPSO adopts the Q-Learning to improve the local optimum, insufficient diversity, and low precision of the PSO algorithm. Simulations are performed on a water distribution ICS, and the results verify the validity and feasibility of our proposed model and the QLPSO algorithm.<\/jats:p>","DOI":"10.3390\/e24121727","type":"journal-article","created":{"date-parts":[[2022,11,28]],"date-time":"2022-11-28T03:28:49Z","timestamp":1669606129000},"page":"1727","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Optimal Security Protection Strategy Selection Model Based on Q-Learning Particle Swarm Optimization"],"prefix":"10.3390","volume":"24","author":[{"given":"Xin","family":"Gao","sequence":"first","affiliation":[{"name":"Shandong Provincial Key Laboratory of Computer Networks, Shandong Computer Science Center (National Supercomputer Center in Jinan), Qilu University of Technology (Shandong Academy of Sciences), Jinan 250014, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yang","family":"Zhou","sequence":"additional","affiliation":[{"name":"Shandong Provincial Key Laboratory of Computer Networks, Shandong Computer Science Center (National Supercomputer Center in Jinan), Qilu University of Technology (Shandong Academy of Sciences), Jinan 250014, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lijuan","family":"Xu","sequence":"additional","affiliation":[{"name":"Shandong Provincial Key Laboratory of Computer Networks, Shandong Computer Science Center (National Supercomputer Center in Jinan), Qilu University of Technology (Shandong Academy of Sciences), Jinan 250014, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1812-1316","authenticated-orcid":false,"given":"Dawei","family":"Zhao","sequence":"additional","affiliation":[{"name":"Shandong Provincial Key Laboratory of Computer Networks, Shandong Computer Science Center (National Supercomputer Center in Jinan), Qilu University of Technology (Shandong Academy of Sciences), Jinan 250014, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,11,25]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"24","DOI":"10.1109\/MSP.2008.150","article-title":"Security for process control systems: An overview","volume":"6","author":"Naedele","year":"2008","journal-title":"IEEE Secur. Priv."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Fan, X., Fan, K., Wang, Y., and Zhou, R. (2015, January 5\u20137). Overview of cyber-security of industrial control system. Proceedings of the 2015 international conference on cyber security of smart cities, industrial control system and communications (SSIC), Shanghai, China.","DOI":"10.1109\/SSIC.2015.7245324"},{"key":"ref_3","unstructured":"Wilhoit, K. (2013). Who\u2019s really attacking your ICS equipment?. Trend Micro, 10."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Clarke, G., Reynders, D., and Wright, E. (2004). Practical Modern SCADA Protocols: DNP3, 60870.5 and Related Systems, Elsevier.","DOI":"10.1016\/B978-075065799-0\/50019-X"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"579","DOI":"10.3166\/ejc.17.579-602","article-title":"Control of distributed systems: Tutorial and overview","volume":"17","author":"Boutin","year":"2011","journal-title":"Eur. J. Control"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Babu, B., Ijyas, T., Muneer, P., and Varghese, J. (2017, January 26\u201327). Security issues in SCADA based industrial control systems. Proceedings of the 2017 2nd International Conference on Anti-Cyber Crimes (ICACC), Abha, Saudi Arabia.","DOI":"10.1109\/Anti-Cybercrime.2017.7905261"},{"key":"ref_7","unstructured":"Wang, Y. (2010, January 29\u201331). SCM\/ERP\/MES\/PCS integration for process enterprise. Proceedings of the 29th Chinese Control Conference, Beijing, China."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1367","DOI":"10.1109\/JPROC.2017.2687865","article-title":"Cybersecurity in distributed power systems","volume":"105","author":"Li","year":"2017","journal-title":"Proc. IEEE"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"2236","DOI":"10.1109\/TII.2016.2599841","article-title":"A cybersecurity detection framework for supervisory control and data acquisition systems","volume":"12","author":"Cruz","year":"2016","journal-title":"IEEE Trans. Ind. Inform."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"91","DOI":"10.1109\/MC.2011.115","article-title":"Lessons from stuxnet","volume":"44","author":"Chen","year":"2011","journal-title":"Computer"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1016\/j.ijepes.2017.12.020","article-title":"Cyber security of a power grid: State-of-the-art","volume":"99","author":"Sun","year":"2018","journal-title":"Int. J. Electr. Power Energy Syst."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1755","DOI":"10.1109\/TIFS.2018.2885254","article-title":"Virus propagation and patch distribution in multiplex networks: Modeling, analysis, and optimal allocation","volume":"14","author":"Zhao","year":"2019","journal-title":"IEEE Trans. Inf. Forensics Secur."},{"key":"ref_13","first-page":"16","article-title":"Guide to industrial control systems (ICS) security","volume":"800","author":"Stouffer","year":"2011","journal-title":"NIST Spec. Publ."},{"key":"ref_14","unstructured":"David, A. (2007). Multiple Efforts to Secure Control Systems Are under Way, But Challenges Remain, US Government Accountability Office (US GAO). Technical report."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"7823","DOI":"10.1109\/TSMC.2020.2987163","article-title":"Minimum dominating set of multiplex networks: Definition, application, and identification","volume":"51","author":"Zhao","year":"2021","journal-title":"IEEE Trans. Syst. Man Cybern. Syst."},{"key":"ref_16","first-page":"134760.1","article-title":"A Retroactive-Burst Framework for Automated Intrusion Response System","volume":"2013","author":"Desfossez","year":"2013","journal-title":"J. Comput. Netw. Commun."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"182","DOI":"10.1109\/4235.996017","article-title":"A fast and elitist multiobjective genetic algorithm: NSGA-II","volume":"6","author":"Deb","year":"2002","journal-title":"IEEE Trans. Evol. Comput."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1016\/j.compeleceng.2015.07.023","article-title":"Selecting optimal countermeasures for attacks against critical systems using the attack volume model and the RORI index","volume":"47","author":"Alvarez","year":"2015","journal-title":"Comput. Electr. Eng."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Miehling, E., Rasouli, M., and Teneketzis, D. (2015, January 12). Optimal defense policies for partially observable spreading processes on Bayesian attack graphs. Proceedings of the Second ACM Workshop on Moving Target Defense, Denver, CO, USA.","DOI":"10.1145\/2808475.2808482"},{"key":"ref_20","unstructured":"Jaquith, A. (2007). Security Metrics: Replacing Fear, Uncertainty, and Doubt, Pearson Education."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Bandyopadhyay, S., and Saha, S. (2013). Some single-and multiobjective optimization techniques. Unsupervised Classification, Springer.","DOI":"10.1007\/978-3-642-32451-2_2"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"61","DOI":"10.1109\/TDSC.2011.34","article-title":"Dynamic security risk management using bayesian attack graphs","volume":"9","author":"Poolsappasit","year":"2011","journal-title":"IEEE Trans. Dependable Secur. Comput."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Yigit, B., G\u00fcr, G., and Alag\u00f6z, F. (2014, January 6\u20138). Cost-aware network hardening with limited budget using compact attack graphs. Proceedings of the 2014 IEEE Military Communications Conference, Baltimore, MD, USA.","DOI":"10.1109\/MILCOM.2014.31"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"156","DOI":"10.1109\/ACCESS.2016.2633983","article-title":"Optimal strategy selection for moving target defense based on Markov game","volume":"5","author":"Lei","year":"2017","journal-title":"IEEE Access"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Herold, N., Wachs, M., Posselt, S.A., and Carle, G. (2016, January 24\u201325). An optimal metric-aware response selection strategy for intrusion response systems. Proceedings of the International Symposium on Foundations and Practice of Security, Quebec City, QC, Canada.","DOI":"10.1007\/978-3-319-51966-1_5"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Butler, S.A. (2002, January 19\u201325). Security attribute evaluation method: A cost-benefit approach. Proceedings of the 24th International Conference on Software Engineering, Orlando, FL, USA.","DOI":"10.1145\/581368.581370"},{"key":"ref_27","unstructured":"Butler, S.A., and Fischbeck, P. (2002, January 16). Multi-attribute risk assessment. Proceedings of the Symposium on Requirements Engineering for Information Security, Raleigh, NC, USA."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Roy, A., Kim, D.S., and Trivedi, K.S. (2012, January 25\u201328). Scalable optimal countermeasure selection using implicit enumeration on attack countermeasure trees. Proceedings of the IEEE\/IFIP International Conference on Dependable Systems and Networks (DSN 2012), Boston, MA, USA.","DOI":"10.1109\/DSN.2012.6263940"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"599","DOI":"10.1016\/j.dss.2012.04.001","article-title":"A novel risk assessment and optimisation model for a multi-objective network security countermeasure selection problem","volume":"53","author":"Viduto","year":"2012","journal-title":"Decis. Support Syst."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"167","DOI":"10.1007\/s10207-012-0160-y","article-title":"Optimal security hardening on attack tree models of networks: A cost-benefit analysis","volume":"11","author":"Dewri","year":"2012","journal-title":"Int. J. Inf. Secur."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"158","DOI":"10.1016\/j.cose.2012.09.013","article-title":"Exploring attack graph for cost-benefit security hardening: A probabilistic approach","volume":"32","author":"Wang","year":"2013","journal-title":"Comput. Secur."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Kordy, B., and Wide\u0142, W. (2017, January 20\u201322). How well can I secure my system?. Proceedings of the International Conference on Integrated Formal Methods, Turin, Italy.","DOI":"10.1007\/978-3-319-66845-1_22"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Fila, B., and Wide\u0142, W. (2020, January 22\u201326). Exploiting attack\u2013defense trees to find an optimal set of countermeasures. Proceedings of the 2020 IEEE 33rd Computer Security Foundations Symposium (CSF), Boston, MA, USA.","DOI":"10.1109\/CSF49147.2020.00035"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Speicher, P., Steinmetz, M., K\u00fcnnemann, R., Simeonovski, M., Pellegrino, G., Hoffmann, J., and Backes, M. (2018, January 24\u201326). Formally reasoning about the cost and efficacy of securing the email infrastructure. Proceedings of the 2018 IEEE European Symposium on Security and Privacy (EuroS&P), London, UK.","DOI":"10.1109\/EuroSP.2018.00014"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"813","DOI":"10.1007\/s10207-022-00586-7","article-title":"A multi-objective cost\u2013benefit optimization algorithm for network hardening","volume":"21","author":"Zenitani","year":"2022","journal-title":"Int. J. Inf. Secur."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Frigault, M., and Wang, L. (August, January 28). Measuring network security using bayesian network-based attack graphs. Proceedings of the 2008 32nd Annual IEEE International Computer Software and Applications Conference, Turku, Finland.","DOI":"10.1109\/COMPSAC.2008.88"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Liu, Y., and Man, H. (2005, January 28\u201329). Network vulnerability assessment using Bayesian networks. Proceedings of the Data Mining, Intrusion Detection, Information Assurance, and Data Networks Security 2005, Orlando, FL, USA.","DOI":"10.1117\/12.604240"},{"key":"ref_38","unstructured":"Ou, X., Govindavajhala, S., and Appel, A.W. (August, January 31). MulVAL: A Logic-based Network Security Analyzer. Proceedings of the USENIX Security Symposium, Baltimore, MD, USA."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"85","DOI":"10.1109\/MSP.2006.145","article-title":"Common vulnerability scoring system","volume":"4","author":"Mell","year":"2006","journal-title":"IEEE Secur. Priv."},{"key":"ref_40","first-page":"111","article-title":"Dynamic security risk assessment model based on Bayesian attack graph","volume":"48","author":"Gao","year":"2016","journal-title":"J. Sichuan Univ. (Eng. Sci. Ed.)"},{"key":"ref_41","first-page":"125","article-title":"Optimal security hardening measures selection model based on Bayesian attack graph","volume":"52","author":"Gao","year":"2016","journal-title":"Comput. Eng. Appl."},{"key":"ref_42","unstructured":"Kennedy, J., and Eberhart, R. (December, January 27). Particle swarm optimization. Proceedings of the ICNN\u201995-International Conference on Neural Networks, Perth, WA, Australia."},{"key":"ref_43","unstructured":"Clerc, M. (2010). Particle Swarm Optimization, John Wiley & Sons."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"133653","DOI":"10.1109\/ACCESS.2019.2941229","article-title":"Q-learning algorithms: A comprehensive classification and applications","volume":"7","author":"Jang","year":"2019","journal-title":"IEEE Access"},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"279","DOI":"10.1007\/BF00992698","article-title":"Q-learning","volume":"8","author":"Watkins","year":"1992","journal-title":"Mach. Learn."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"279","DOI":"10.1146\/annurev-statistics-031219-041220","article-title":"Q-learning: Theory and applications","volume":"7","author":"Clifton","year":"2020","journal-title":"Annu. Rev. Stat. Its Appl."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Liu, Y., Lu, H., Cheng, S., and Shi, Y. (2019, January 10\u201313). An adaptive online parameter control algorithm for particle swarm optimization based on reinforcement learning. Proceedings of the 2019 IEEE Congress on Evolutionary Computation (CEC), Wellington, New Zealand.","DOI":"10.1109\/CEC.2019.8790035"},{"key":"ref_48","first-page":"71","article-title":"A comparison study of cooperative Q-learning algorithms for independent learners","volume":"14","author":"Paul","year":"2016","journal-title":"Int. J. Artif. Intell."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Meerza, S.I.A., Islam, M., and Uzzal, M.M. (2019, January 3\u20135). Q-learning based particle swarm optimization algorithm for optimal path planning of swarm of mobile robots. Proceedings of the 2019 1st International Conference on Advances in Science, Engineering and Robotics Technology (ICASERT), Dhaka, Bangladesh.","DOI":"10.1109\/ICASERT.2019.8934450"},{"key":"ref_50","doi-asserted-by":"crossref","first-page":"703","DOI":"10.1109\/TNSE.2021.3130602","article-title":"Detecting Semantic Attack in SCADA System: A Behavioral Model Based on Secondary Labeling of States-Duration Evolution Graph","volume":"9","author":"Xu","year":"2021","journal-title":"IEEE Trans. Netw. Sci. Eng."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/24\/12\/1727\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T01:26:54Z","timestamp":1760146014000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/24\/12\/1727"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,11,25]]},"references-count":50,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2022,12]]}},"alternative-id":["e24121727"],"URL":"https:\/\/doi.org\/10.3390\/e24121727","relation":{},"ISSN":["1099-4300"],"issn-type":[{"type":"electronic","value":"1099-4300"}],"subject":[],"published":{"date-parts":[[2022,11,25]]}}}