{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,11]],"date-time":"2026-07-11T16:21:16Z","timestamp":1783786876849,"version":"3.55.0"},"reference-count":56,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2020,11,8]],"date-time":"2020-11-08T00:00:00Z","timestamp":1604793600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000148","name":"Division of Electrical, Communications and Cyber Systems","doi-asserted-by":"publisher","award":["1710859"],"award-info":[{"award-number":["1710859"]}],"id":[{"id":"10.13039\/100000148","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000144","name":"Division of Computer and Network Systems","doi-asserted-by":"publisher","award":["1422594"],"award-info":[{"award-number":["1422594"]}],"id":[{"id":"10.13039\/100000144","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000183","name":"Army Research Office","doi-asserted-by":"publisher","award":["W911NF-13-1-0421"],"award-info":[{"award-number":["W911NF-13-1-0421"]}],"id":[{"id":"10.13039\/100000183","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100009226","name":"National Security Agency","doi-asserted-by":"crossref","award":["H98230-15-1-0289"],"award-info":[{"award-number":["H98230-15-1-0289"]}],"id":[{"id":"10.13039\/100009226","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Priv. Secur."],"published-print":{"date-parts":[[2021,2,28]]},"abstract":"<jats:p>Growing multi-stage attacks in computer networks impose significant security risks and necessitate the development of effective defense schemes that are able to autonomously respond to intrusions during vulnerability windows. However, the defender faces several real-world challenges, e.g., unknown likelihoods and unknown impacts of successful exploits. In this article, we leverage reinforcement learning to develop an innovative adaptive cyber defense to maximize the cost-effectiveness subject to the aforementioned challenges. In particular, we use Bayesian attack graphs to model the interactions between the attacker and networks. Then we formulate the defense problem of interest as a partially observable Markov decision process problem where the defender maintains belief states to estimate system states, leverages Thompson sampling to estimate transition probabilities, and utilizes reinforcement learning to choose optimal defense actions using measured utility values. The algorithm performance is verified via numerical simulations based on real-world attacks.<\/jats:p>","DOI":"10.1145\/3418897","type":"journal-article","created":{"date-parts":[[2020,11,9]],"date-time":"2020-11-09T01:42:26Z","timestamp":1604886146000},"page":"1-25","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":26,"title":["Adaptive Cyber Defense Against Multi-Stage Attacks Using Learning-Based POMDP"],"prefix":"10.1145","volume":"24","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1940-9829","authenticated-orcid":false,"given":"Zhisheng","family":"Hu","sequence":"first","affiliation":[{"name":"Baidu Security, Sunnyvale, CA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Minghui","family":"Zhu","sequence":"additional","affiliation":[{"name":"Pennsylvania State University, PA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Peng","family":"Liu","sequence":"additional","affiliation":[{"name":"Pennsylvania State University, PA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2020,11,8]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2017.2743240"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1016\/0022-247X(65)90154-X"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/TAC.1985.1103963"},{"key":"e_1_2_1_4_1","volume-title":"Dreyfus","author":"Bellman Richard E.","year":"1962"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2810103.2813691"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1186\/s42400-018-0003-x"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-25594-1_1"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-13841-1_1"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1049\/iet-ifs.2014.0272"},{"key":"e_1_2_1_10_1","volume-title":"Proceedings of the 1998 Conference on Advances in Neural Information Processing Systems II (NIPS\u201998)","author":"Friedman Nir","year":"1999"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/1456362.1456368"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.automatica.2019.02.032"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3140549.3140556"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2008.295"},{"key":"e_1_2_1_16_1","volume-title":"Proceedings of the 2016 25th International Conference on Computer Communication and Networks (ICCCN\u201916)","author":"Iannucci S."},{"key":"e_1_2_1_17_1","volume-title":"FirePatch: Secure and Time-Critical Dissemination of Software Patches","author":"Johansen H\u00e5vard"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2014.25"},{"key":"e_1_2_1_19_1","volume-title":"Proceedings of the 2006 IEEE Military Communications Conference (MILCOM\u201906)","author":"Lippmann R."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1117\/12.604240"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2808475.2808482"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIFS.2018.2819967"},{"key":"e_1_2_1_23_1","volume-title":"et\u00a0al","author":"Mnih Volodymyr","year":"2015"},{"key":"e_1_2_1_24_1","first-page":"1938","article-title":"A brief study of wannacry threat: Ransomware attack 2017","volume":"8","author":"Mohurle Savita","year":"2017","journal-title":"International Journal of Advanced Research in Computer Science"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3140549.3140562"},{"key":"e_1_2_1_26_1","volume-title":"Proceedings of the 2015 9th International Conference on IT Security Incident Management IT Forensics (IMF\u201915)","author":"Ossenbuhl S."},{"key":"e_1_2_1_27_1","volume-title":"Proceedings of the 13th ACM Conference on Computer and Communications Security (CCS\u201906)","author":"Ou Xinming"},{"key":"e_1_2_1_28_1","unstructured":"Yi Ouyang Mukul Gagrani Ashutosh Nayyar and Rahul Jain. 2017. Learning unknown Markov decision processes: A Thompson sampling approach. In Advances in Neural Information Processing Systems 30 (NIPS\u201917). 1333--1342.  Yi Ouyang Mukul Gagrani Ashutosh Nayyar and Rahul Jain. 2017. Learning unknown Markov decision processes: A Thompson sampling approach. In Advances in Neural Information Processing Systems 30 (NIPS\u201917). 1333--1342."},{"key":"e_1_2_1_29_1","volume-title":"Proceedings of the 2018 IEEE European Symposium on Security and Privacy (EuroSP\u201918)","author":"Papernot N."},{"key":"e_1_2_1_30_1","volume-title":"Proceedings of the 18th International Joint Conference on Artificial Intelligence (IJCAI\u201903)","author":"Pineau Joelle","year":"2003"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/TDSC.2011.34"},{"key":"e_1_2_1_32_1","unstructured":"Pascal Poupart and Craig Boutilier. 2003. Value-directed compression of POMDPs. In Advances in Neural Information Processing Systems (NIPS\u201902). 1579--1586.  Pascal Poupart and Craig Boutilier. 2003. Value-directed compression of POMDPs. In Advances in Neural Information Processing Systems (NIPS\u201902). 1579--1586."},{"key":"e_1_2_1_33_1","volume-title":"VDCBPI: An approximate scalable algorithm for large POMDPs. In Advances in Neural Information Processing Systems (NIPS\u201904). 1081--1088.","author":"Poupart Pascal","year":"2004"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/1813654.1813655"},{"key":"e_1_2_1_35_1","volume-title":"Abbas Kazerouni, and Ian Osband.","author":"Russo Daniel","year":"2017"},{"key":"e_1_2_1_36_1","volume-title":"Proceedings of the 26th AAAI Conference on Artificial Intelligence (AAAI\u201912)","author":"Sarraute Carlos","year":"2012"},{"key":"e_1_2_1_37_1","volume-title":"Retrieved","author":"Schiffman Mike","year":"2017"},{"key":"e_1_2_1_38_1","volume-title":"Learning and Solving Partially Observable Markov Decision Processes","author":"Shani Guy"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10458-012-9200-2"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1287\/opre.26.2.282"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.5555\/1622519.1622525"},{"key":"e_1_2_1_42_1","volume-title":"Proceedings of the 17th International Conference on Machine Learning (ICML\u201900)","author":"Strens Malcolm J. A.","year":"2000"},{"key":"e_1_2_1_43_1","volume-title":"Retrieved","year":"2015"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/25.3-4.285"},{"key":"e_1_2_1_45_1","series-title":"Lecture Notes in Computer Science","volume-title":"KI 2010: Advances in Artificial Intelligence","author":"Tokic Michel"},{"key":"e_1_2_1_46_1","volume-title":"Proceedings of the National Conference on Artificial Intelligence (AAAI\u201907)","volume":"22","author":"Virin Yan"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF00992698"},{"key":"e_1_2_1_48_1","volume-title":"Proceedings of the 2010 IEEE\/IFIP International Conference on Dependable Systems Networks (DSN\u201910)","author":"Xie Peng"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.5555\/1949317.1949338"},{"key":"e_1_2_1_50_1","volume-title":"Proceedings of the 8th Annual Cyber Security and Information Intelligence Research Workshop (CSIIRW\u201913)","author":"Yu Lu"},{"key":"e_1_2_1_51_1","volume-title":"Retrieved","author":"Zambon Emmanuele","year":"2006"},{"key":"e_1_2_1_52_1","volume-title":"A survey of coordinated attacks and collaborative intrusion detection. Computers 8 Security 29, 1","author":"Zhou Chenfeng Vincent","year":"2010"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/2663474.2663481"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1137\/110843332"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1109\/CDC.2009.5399894"},{"key":"e_1_2_1_56_1","first-page":"305","article-title":"Hybrid learning in stochastic games and its applications in network security","volume":"17","author":"Zhu Quanyan","year":"2013","journal-title":"Reinforcement Learning and Approximate Dynamic Programming for Feedback Control"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/586110.586130"}],"container-title":["ACM Transactions on Privacy and Security"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3418897","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3418897","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3418897","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:02:28Z","timestamp":1750197748000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3418897"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,11,8]]},"references-count":56,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2021,2,28]]}},"alternative-id":["10.1145\/3418897"],"URL":"https:\/\/doi.org\/10.1145\/3418897","relation":{},"ISSN":["2471-2566","2471-2574"],"issn-type":[{"value":"2471-2566","type":"print"},{"value":"2471-2574","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,11,8]]},"assertion":[{"value":"2019-10-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-08-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-11-08","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}