{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,26]],"date-time":"2026-02-26T16:13:46Z","timestamp":1772122426787,"version":"3.50.1"},"reference-count":45,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2021,6,8]],"date-time":"2021-06-08T00:00:00Z","timestamp":1623110400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,6,8]],"date-time":"2021-06-08T00:00:00Z","timestamp":1623110400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100005366","name":"University of Oslo","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100005366","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Int. J. Inf. Secur."],"published-print":{"date-parts":[[2022,4]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Website hacking is a frequent attack type used by malicious actors to obtain confidential information, modify the integrity of web pages or make websites unavailable. The tools used by attackers are becoming more and more automated and sophisticated, and malicious machine learning agents seem to be the next development in this line. In order to provide ethical hackers with similar tools, and to understand the impact and the limitations of artificial agents, we present in this paper a model that formalizes web hacking tasks for reinforcement learning agents. Our model, named<jats:italic>Agent Web Model<\/jats:italic>, considers web hacking as a capture-the-flag style challenge, and it defines reinforcement learning problems at seven different levels of abstraction. We discuss the complexity of these problems in terms of actions and states an agent has to deal with, and we show that such a model allows to represent most of the relevant web vulnerabilities. Aware that the driver of advances in reinforcement learning is the availability of standardized challenges, we provide an implementation for the first three abstraction layers, in the hope that the community would consider these challenges in order to develop intelligent web hacking agents.<\/jats:p>","DOI":"10.1007\/s10207-021-00554-7","type":"journal-article","created":{"date-parts":[[2021,6,8]],"date-time":"2021-06-08T07:02:55Z","timestamp":1623135775000},"page":"293-309","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":8,"title":["The Agent Web Model: modeling web hacking for reinforcement learning"],"prefix":"10.1007","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4910-4228","authenticated-orcid":false,"given":"L\u00e1szl\u00f3","family":"Erd\u0151di","sequence":"first","affiliation":[]},{"given":"Fabio Massimo","family":"Zennaro","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2021,6,8]]},"reference":[{"key":"554_CR1","unstructured":"Anley, C.: Advanced SQL injection in SQL server applications. https:\/\/crypto.stanford.edu\/cs155old\/cs155-spring11\/papers\/sql_injection.pdf. Accessed: 2020-09-12 (2002)"},{"key":"554_CR2","doi-asserted-by":"crossref","unstructured":"Antunes, N., Vieira, M.: Designing vulnerability testing tools for web services: approach, components, and tools. Int. J. Inf. Secur. (2017)","DOI":"10.1007\/s10207-016-0334-0"},{"key":"554_CR3","doi-asserted-by":"crossref","unstructured":"Appelt, D., Nguyen, C.D., Panichella, A., Briand, L.C.: A machine-learning-driven evolutionary approach for testing web application firewalls. IEEE Trans. Reliab. 67(3) (2018)","DOI":"10.1109\/TR.2018.2805763"},{"key":"554_CR4","volume-title":"Weaving the Web: The Original Design and Ultimate Destiny of the World Wide Web by Its Inventor","author":"T Berners-Lee","year":"2001","unstructured":"Berners-Lee, T., Fischetti, M.: Weaving the Web: The Original Design and Ultimate Destiny of the World Wide Web by Its Inventor. DIANE Publishing Company, Darby (2001)"},{"key":"554_CR5","unstructured":"Blasco, J.: Introduction to XPath injection techniques. http:\/\/repository.root-me.org\/Exploitation%20-%20Web\/EN%20-%20Introduction%20to%20Xpath%20injection%20techniques.pdf. Accessed 2020-09-12 (2007)"},{"key":"554_CR6","unstructured":"Boddy, M.S., Gohde, J., Haigh, T., Harp, S.A.: Course of action generation for cyber security using classical planning. In: ICAPS, pp. 12\u201321 (2005)"},{"key":"554_CR7","unstructured":"Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., Zaremba, W.: Openai gym. arXiv preprint arXiv:1606.01540 (2016)"},{"key":"554_CR8","doi-asserted-by":"crossref","unstructured":"Chowdary, A., Huang, D., Mahendran, J.S., Romo, D., Deng, Y., Sabur, A.: Autonomous security analysis and penetration testing. In: The 16th International Conference on Mobility, Sensing and Networking (MSN 2020) (2020)","DOI":"10.1109\/MSN50589.2020.00086"},{"key":"554_CR9","unstructured":"Damele, B., Stampar, M.: sqlmap user\u2019s manual. http:\/\/www.it-docs.net\/ddata\/4956.pdf. Accessed 2020-05-09 (2011)"},{"key":"554_CR10","doi-asserted-by":"crossref","unstructured":"Elderman, R., Pater, L.J., Thie, A.S.: Adversarial reinforcement learning in a cyber security simulation. PhD thesis, Faculty of Science and Engineering (2016)","DOI":"10.5220\/0006197105590566"},{"key":"554_CR11","doi-asserted-by":"crossref","unstructured":"Fielding, R., Irivine, U.C., Gettys, J., Mogul, J., Frystyk, H.: Request for comments 2616, hypertext transfer protocol \u2013 http 1\/1. https:\/\/tools.ietf.org\/html\/rfc2616. Accessed: 2020-09-05 (1999)","DOI":"10.17487\/rfc2616"},{"key":"554_CR12","doi-asserted-by":"crossref","unstructured":"Fonseca, J., Vieira, M., Madeira, H.: Testing and comparing web vulnerability scanning tools for SQL injection and XSS attacks. In: 13th Pacific Rim International Symposium on Dependable Computing (PRDC 2007) (2020)","DOI":"10.1109\/PRDC.2007.55"},{"key":"554_CR13","unstructured":"Fraze, D.: Cyber grand challenge (CGC). https:\/\/www.darpa.mil\/program\/cyber-grand-challenge. Accessed 2020-05-09 (2016)"},{"key":"554_CR14","doi-asserted-by":"crossref","unstructured":"Ghanem, M.C., Chen, T.M.: Reinforcement learning for intelligent penetration testing. In: 2018 Second World Conference on Smart Trends in Systems, Security and Sustainability (WorldS4) (2018)","DOI":"10.1109\/WorldS4.2018.8611595"},{"issue":"1","key":"554_CR15","doi-asserted-by":"publisher","first-page":"6","DOI":"10.3390\/info11010006","volume":"11","author":"MC Ghanem","year":"2020","unstructured":"Ghanem, M.C., Chen, T.M.: Reinforcement learning for efficient network penetration testing. Information 11(1), 6 (2020)","journal-title":"Information"},{"key":"554_CR16","doi-asserted-by":"crossref","unstructured":"Grossman, J., Hansen, R., Petkov, D., Rager, A., Fogie, S.: XSS Attacks: Cross Site Scripting Exploits and Defense. Syngress (2007)","DOI":"10.1016\/B978-159749154-9\/50005-6"},{"key":"554_CR17","doi-asserted-by":"crossref","unstructured":"Hoffmann, J.: Simulated penetration testing: From \u201cDijkstra\u201d to \u201cTuring test++\u201d. In: Twenty-Fifth International Conference on Automated Planning and Scheduling (2015)","DOI":"10.1609\/icaps.v25i1.13684"},{"key":"554_CR18","unstructured":"Howard, M., LeBlanc, D., Viega, J.: 24 deadly sins of software security, sin 2: Web server-related vulnerabilities (XSS, XSRF, and response splitting). http:\/\/index-of.es\/Miscellanous\/24-DEADLY-SINS-OF-SOFTWARE-SECURITY-2010.pdf. Accessed 2020-09-12 (2010)"},{"key":"554_CR19","unstructured":"Howard, M., LeBlanc, D., Viega, J.: 24 deadly sins of software security, sin 4: Use of magic URLs, predictable cookies, and hidden form fields. http:\/\/index-of.es\/Miscellanous\/24-DEADLY-SINS-OF-SOFTWARE-SECURITY-2010.pdf. Accessed 2020-09-12 (2010)"},{"key":"554_CR20","unstructured":"Johnson, G.: Remote and local file inclusion explained. http:\/\/repository.root-me.org\/Exploitation%20-%20Web\/EN%20-%20Remote%20File%20Inclusion%20and%20Local%20File%20Inclusion%20explained.pdf. Accessed 2020-09-12 (2008)"},{"key":"554_CR21","unstructured":"Kettle, J.: Server-side template injection: RCE for the modern webapp. https:\/\/www.blackhat.com\/docs\/us-15\/materials\/us-15-Kettle-Server-Side-Template-%Injection-RCE-For-The-Modern-Web-App-wp.pdf. Accessed 2020-09-12 (2015)"},{"key":"554_CR22","unstructured":"Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097\u20131105 (2012)"},{"key":"554_CR23","unstructured":"Lison, P., Mavroeidis, V.: Automatic detection of malware-generated domains with recurrent neural models. arXiv preprint arXiv:1709.07102 (2017)"},{"key":"554_CR24","doi-asserted-by":"publisher","first-page":"102108","DOI":"10.1016\/j.cose.2020.102108","volume":"100","author":"R Maeda","year":"2021","unstructured":"Maeda, R., Mimura, M.: Automating post-exploitation with deep reinforcement learning. Comput. Secur. 100, 102108 (2021)","journal-title":"Comput. Secur."},{"key":"554_CR25","doi-asserted-by":"crossref","unstructured":"McDaniel, L., Talvi, E., Ba, H.: Capture the flag as cyber security introduction. In: Annual Hawaii International Conference on System Sciences (HICSS) (2016)","DOI":"10.1109\/HICSS.2016.677"},{"issue":"7540","key":"554_CR26","doi-asserted-by":"publisher","first-page":"529","DOI":"10.1038\/nature14236","volume":"518","author":"V Mnih","year":"2015","unstructured":"Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski, G., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529\u2013533 (2015)","journal-title":"Nature"},{"key":"554_CR27","unstructured":"Mnih, V., Badia, A.P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., Kavukcuoglu, K.: Asynchronous methods for deep reinforcement learning. In: International Conference on Machine Learning, PMLR, pp 1928\u20131937 (2016)"},{"key":"554_CR28","unstructured":"Niculae, S., Dichiu, D., Yang, K., B\u00e4ck, T.: Automating penetration testing using reinforcement learning. https:\/\/stefann.eu\/files\/Automating%20Penetration%20Testing%20using%20Reinforcement%20Learning.pdf (2020)"},{"key":"554_CR29","unstructured":"Pettersson, A., Fjordefalk, O.: Using Markov decision processes and reinforcement learning to guide penetration testers in the search for web vulnerabilities (2019)"},{"key":"554_CR30","doi-asserted-by":"crossref","unstructured":"Pozdniakov, K., Alonso, E., Stankovic, V., Tam, K., Jones, K.: Smart security audit: reinforcement learning with a deep neural network approximator. In: 2020 International Conference on Cyber Situational Awareness, Data Analytics and Assessment (CyberSA), pp. 1\u20138 (2020)","DOI":"10.1109\/CyberSA49311.2020.9139683"},{"key":"554_CR31","volume-title":"Nessus Network Auditing","author":"R Rogers","year":"2011","unstructured":"Rogers, R.: Nessus Network Auditing, 2nd edn. Syngress, Burlington (2011)","edition":"2"},{"key":"554_CR32","doi-asserted-by":"crossref","unstructured":"Russell, R., Kim, L., Hamilton, L., Lazovich, T., Harer, J., Ozdemir, O., Ellingwood, P., McConley, M.: Automated vulnerability detection in source code using deep representation learning. In: 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), IEEE, pp. 757\u2013762 (2018)","DOI":"10.1109\/ICMLA.2018.00120"},{"key":"554_CR33","unstructured":"Sarraute, C., Buffet, O., Hoffmann, J.: Penetration testing== POMDP solving? arXiv preprint arXiv:1306.4714 (2013)"},{"key":"554_CR34","unstructured":"Schwartz, J., Kurniawati, H.: Autonomous penetration testing using reinforcement learning. arXiv preprint arXiv:1905.05965v1 (2019)"},{"key":"554_CR35","unstructured":"Shalev-Shwartz, S., Shammah, S., Shashua, A.: Safe, multi-agent, reinforcement learning for autonomous driving. arXiv preprint arXiv:1610.03295 (2016)"},{"key":"554_CR36","doi-asserted-by":"crossref","unstructured":"Siddiqui, S., Verma, D.: Cross site request forgery: a common web application weakness. In: International Conference on Communication Software and Networks, ICCSN (2011)","DOI":"10.1109\/ICCSN.2011.6014783"},{"issue":"7676","key":"554_CR37","doi-asserted-by":"publisher","first-page":"354","DOI":"10.1038\/nature24270","volume":"550","author":"D Silver","year":"2017","unstructured":"Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al.: Mastering the game of Go without human knowledge. Nature 550(7676), 354 (2017)","journal-title":"Nature"},{"key":"554_CR38","doi-asserted-by":"crossref","unstructured":"Speicher, P., Steinmetz, M., Hoffmann, J., Backes, M., K\u00fcnnemann, R.: Towards automated network mitigation analysis. In: Proceedings of the 34th ACM\/SIGAPP Symposium on Applied Computing, pp. 1971\u20131978 (2019)","DOI":"10.1145\/3297280.3297473"},{"key":"554_CR39","doi-asserted-by":"crossref","unstructured":"Stasinopoulos, A., Ntantogian, C., Xenakis, C.: Commix: automating evaluation and exploitation of command injection vulnerabilities in web applications. Int. J. Inf. Secur. (2019)","DOI":"10.1007\/s10207-018-0399-z"},{"key":"554_CR40","volume-title":"Reinforcement Learning: An Introduction","author":"RS Sutton","year":"2018","unstructured":"Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (2018)"},{"key":"554_CR41","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, \u0141., Polosukhin, I.: Attention is all you need. In: Advances in Neural Information Processing Systems, pp. 5998\u20136008 (2017)"},{"issue":"7782","key":"554_CR42","doi-asserted-by":"publisher","first-page":"350","DOI":"10.1038\/s41586-019-1724-z","volume":"575","author":"O Vinyals","year":"2019","unstructured":"Vinyals, O., Babuschkin, I., Czarnecki, W.M., Mathieu, M., Dudzik, A., Chung, J., Choi, D.H., Powell, R., Ewalds, T., Georgiev, P., et al.: Grandmaster level in Starcraft II using multi-agent reinforcement learning. Nature 575(7782), 350\u2013354 (2019)","journal-title":"Nature"},{"key":"554_CR43","doi-asserted-by":"crossref","unstructured":"Vlsaggio, C.A., Blasio, L.C.: Session management vulnerabilities in today\u2019s web. IEEE Security and Privacy (2010)","DOI":"10.1109\/MSP.2010.114"},{"key":"554_CR44","unstructured":"Wichers, D., Williams, J.: Owasp - top 10 web application security risks. https:\/\/owasp.org\/www-project-top-ten\/. Accessed 2020-09-05 (2017)"},{"key":"554_CR45","unstructured":"Zennaro, F.M., Erdodi, L.: Modeling penetration testing with reinforcement learning using capture-the-flag challenges and tabular Q-learning. arXiv preprint arXiv:2005.12632 (2020)"}],"container-title":["International Journal of Information Security"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10207-021-00554-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10207-021-00554-7\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10207-021-00554-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,30]],"date-time":"2022-12-30T07:55:18Z","timestamp":1672386918000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10207-021-00554-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,6,8]]},"references-count":45,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2022,4]]}},"alternative-id":["554"],"URL":"https:\/\/doi.org\/10.1007\/s10207-021-00554-7","relation":{},"ISSN":["1615-5262","1615-5270"],"issn-type":[{"value":"1615-5262","type":"print"},{"value":"1615-5270","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,6,8]]},"assertion":[{"value":"8 June 2021","order":1,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"All authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"This article does not contain any studies with human participants or animals performed by any of the authors.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Human and animals rights"}}]}}