{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,4]],"date-time":"2026-11-04T20:40:06Z","timestamp":1793824806815,"version":"4.6.1"},"reference-count":71,"publisher":"Association for Computing Machinery (ACM)","issue":"6","license":[{"start":{"date-parts":[[2026,5,13]],"date-time":"2026-05-13T00:00:00Z","timestamp":1778630400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2026,6,30]]},"abstract":"<jats:p>Traditional enterprise penetration-testing, while critical for validating defenses and uncovering vulnerabilities, is often limited by high operational costs and the scarcity of human expertise. This article investigates the feasibility and effectiveness of using Large Language Model (LLM)-driven autonomous systems to address these challenges in real-world Active Directory (AD) enterprise networks.<\/jats:p>\n                  <jats:p>\n                    We introduce a novel prototype,\n                    <jats:italic toggle=\"yes\">cochise<\/jats:italic>\n                    , designed to employ LLMs to autonomously perform Assumed Breach penetration-testing against enterprise networks. Our system represents the first demonstration of a fully autonomous, LLM-driven framework capable of compromising accounts within a real-life Microsoft Active Directory testbed, the\n                    <jats:italic toggle=\"yes\">Game of Active Directory<\/jats:italic>\n                    (GOAD). The evaluation deliberately utilizes GOAD to capture the intricate interactions and sometimes nondeterministic outcomes of live network penetration-testing, moving beyond the limitations of synthetic benchmarks.\n                  <\/jats:p>\n                  <jats:p>We perform our empirical evaluation using five LLMs, comparing reasoning to non-reasoning models as well as including open-weight models. Through comprehensive quantitative and qualitative analysis, incorporating insights from cybersecurity experts, we demonstrate that autonomous LLMs can effectively conduct Assumed Breach simulations. Key findings highlight their ability to dynamically adapt attack strategies, perform inter-context attacks (e.g., web application audits, social engineering, and unstructured data analysis for credentials), and generate scenario-specific attack parameters like realistic password candidates. The prototype also exhibits robust self-correction mechanisms, automatically installing missing tools and rectifying invalid command generations.<\/jats:p>\n                  <jats:p>Critically, we find that the associated costs are competitive with, and often significantly lower than, those incurred by professional human penetration testers, suggesting a path toward democratizing access to essential security testing for organizations with budgetary constraints. However, our research also illuminates existing limitations, including instances of LLM \u201cgoing down rabbit holes,\u201d challenges in comprehensive information transfer between planning and execution modules, and critical safety concerns that necessitate human oversight. Our findings lay foundational groundwork for future software engineering research into LLM-driven cybersecurity automation, emphasizing that the prototype\u2019s underlying LLM-driven architecture and techniques are domain-agnostic and hold promise for improving autonomous LLM usage in broader software engineering domains. The source code, traces, and analyzed logs are open-sourced to foster collective cybersecurity and future research.<\/jats:p>","DOI":"10.1145\/3766895","type":"journal-article","created":{"date-parts":[[2025,9,10]],"date-time":"2025-09-10T14:04:24Z","timestamp":1757513064000},"page":"1-54","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":17,"title":["Can LLMs Hack Enterprise Networks? Autonomous Assumed Breach Penetration-Testing Active Directory Networks"],"prefix":"10.1145","volume":"35","author":[{"ORCID":"https:\/\/orcid.org\/0009-0000-2484-0109","authenticated-orcid":false,"given":"Andreas","family":"Happe","sequence":"first","affiliation":[{"name":"Information Systems Engineering, TU Wien Faculty of Informatics, Vienna, Austria"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8619-1271","authenticated-orcid":false,"given":"J\u00fcrgen","family":"Cito","sequence":"additional","affiliation":[{"name":"Information Systems Engineering, TU Wien Faculty of Informatics, Vienna, Austria"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,5,13]]},"reference":[{"key":"e_1_3_3_2_2","unstructured":"Abdulrahman Alamri and Lexie Mooney. 2025. Dragos Industrial Ransomware Analysis: Q1 2025. Retrieved June 2 2025 from https:\/\/www.dragos.com\/blog\/dragos-industrial-ransomware-analysis-q1-2025\/"},{"key":"e_1_3_3_3_2","volume-title":"Caldera: A Red-Blue Cyber Operations Automation Platform","author":"Alford Ron","year":"2022","unstructured":"Ron Alford, Dean Lawrence, and Michael Kouremetis. 2022. Caldera: A Red-Blue Cyber Operations Automation Platform. MITRE, Bedford, MA."},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1109\/NCG.2018.8593188"},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.1191\/1478088706qp063oa"},{"key":"e_1_3_3_6_2","volume-title":"Constructing Grounded Theory: A Practical Guide through Qualitative Analysis","author":"Charmaz Kathy","year":"2006","unstructured":"Kathy Charmaz. 2006. Constructing Grounded Theory: A Practical Guide through Qualitative Analysis. Sage."},{"key":"e_1_3_3_7_2","unstructured":"dair ai. 2025. Reasoning LLMs Guide. Retrieved June 11 2025 from https:\/\/www.promptingguide.ai\/guides\/reasoning-llms"},{"key":"e_1_3_3_8_2","unstructured":"Gelei Deng Yi Liu V\u00edctor Mayoral-Vilches Peng Liu Yuekang Li Yuan Xu Tianwei Zhang Yang Liu Martin Pinzger and Stefan Rass. 2024. PentestGPT: An LLM-empowered automatic penetration testing tool. arXiv:2308.06782. Retrieved from https:\/\/arxiv.org\/abs\/2308.06782"},{"key":"e_1_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.4324\/9781315129945"},{"key":"e_1_3_3_10_2","unstructured":"Richard Fang Rohan Bindu Akul Gupta and Daniel Kang. 2024. LLM agents can autonomously exploit one-day vulnerabilities. arXiv:2404.08144. Retrieved from https:\/\/arxiv.org\/abs\/2404.08144"},{"key":"e_1_3_3_11_2","unstructured":"Richard Fang Rohan Bindu Akul Gupta Qiusi Zhan and Daniel Kang. 2024. LLM agents can autonomously hack websites. arXiv:2402.06664. Retrieved from https:\/\/arxiv.org\/abs\/2402.06664"},{"key":"e_1_3_3_12_2","unstructured":"Richard Fang Rohan Bindu Akul Gupta Qiusi Zhan and Daniel Kang. 2024. Teams of LLM agents can exploit zero-day vulnerabilities. arXiv:2406.01637. Retrieved from https:\/\/arxiv.org\/abs\/2406.01637"},{"key":"e_1_3_3_13_2","unstructured":"Luca Gioacchini Marco Mellia Idilio Drago Alexander Delsanto Giuseppe Siracusano and Roberto Bifulco. 2024. AutoPenBench: Benchmarking generative agents for penetration testing. arXiv:2410.03225. Retrieved from https:\/\/arxiv.org\/abs\/2410.03225"},{"key":"e_1_3_3_14_2","unstructured":"Daya Guo Dejian Yang Haowei Zhang Junxiao Song Ruoyu Zhang Runxin Xu Qihao Zhu Shirong Ma Peiyi Wang Xiao Bi et al. 2025. Deepseek-r1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv:2501.12948. Retrieved from https:\/\/arxiv.org\/abs\/2501.12948"},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3611643.3613083"},{"key":"e_1_3_3_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/3611643.3613900"},{"key":"e_1_3_3_17_2","unstructured":"Andreas Happe and J\u00fcrgen Cito. 2025. Benchmarking practices in LLM-driven offensive security: testbeds metrics and experiment design. arXiv:2504.10112. Retrieved from https:\/\/arxiv.org\/abs\/2504.10112."},{"key":"e_1_3_3_18_2","unstructured":"Andreas Happe and J\u00fcrgen Cito. 2025. On the ethics of using LLMs for offensive security. arXiv:2506.08693. Retrieved from https:\/\/arxiv.org\/abs\/2506.08693"},{"key":"e_1_3_3_19_2","unstructured":"Andreas Happe Aaron Kaplan and Juergen Cito. 2024. Llms as hackers: Autonomous Linux privilege escalation attacks. arXiv:2310.11409. Retrieved from https:\/\/arxiv.org\/abs\/2310.11409"},{"key":"e_1_3_3_20_2","unstructured":"Fred Heiding Simon Lermen Andrew Kao Bruce Schneier and Arun Vishwanath. 2024. Evaluating large language models\u2019 capability to launch fully automated spear phishing campaigns: Validated on human subjects. arXiv:2412.00586. Retrieved from https:\/\/arxiv.org\/abs\/2412.00586"},{"key":"e_1_3_3_21_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.socscimed.2021.114523"},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/3689933.3690831"},{"key":"e_1_3_3_23_2","unstructured":"Isamu Isozaki Manil Shrestha Rick Console and Edward Kim. 2024. Towards automated penetration testing: Introducing LLM benchmark analysis and improvements. arXiv:2410.17141. Retrieved from https:\/\/arxiv.org\/abs\/2410.17141"},{"key":"e_1_3_3_24_2","unstructured":"Aaron Jaech Adam Kalai Adam Lerer Adam Richardson Ahmed El-Kishky Aiden Low Alec Helyar Aleksander Madry Alex Beutel Alex Carney et al. 2024. OpenAI O1 system card. arXiv:2412.16720. Retrieved from https:\/\/arxiv.org\/abs\/2412.16720"},{"key":"e_1_3_3_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICBATS54253.2022.9759000"},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2021.116198"},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.1051\/e3sconf\/202455601006"},{"key":"e_1_3_3_28_2","volume-title":"Mastering Active Directory for Windows Server 2003","author":"King Robert R.","year":"2006","unstructured":"Robert R. King. 2006. Mastering Active Directory for Windows Server 2003. John Wiley & Sons."},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2002.1027796"},{"key":"e_1_3_3_30_2","first-page":"22199","article-title":"Large language models are zero-shot reasoners","volume":"35","author":"Kojima Takeshi","year":"2022","unstructured":"Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in Neural Information Processing Systems 35 (2022), 22199\u201322213.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_3_31_2","unstructured":"He Kong Die Hu Jingguo Ge Liangxiong Li Tong Li and Bingzhen Wu. 2025. VulnBot: Autonomous penetration testing for a multi-agent collaborative framework. arXiv:2501.13411. Retrieved from https:\/\/arxiv.org\/abs\/2501.13411"},{"key":"e_1_3_3_32_2","unstructured":"Swetha Krishnamoorthi and Jarad Carleton. 2020. Active Directory Holds the Keys to your Kingdom but is it Secure? Retrieved June 2 2025 from https:\/\/www.frost.com\/growth-opportunity-news\/active-directory-holds-the-keys-to-your-kingdom-but-is-it-secure"},{"key":"e_1_3_3_33_2","unstructured":"Xiaomin Li Zhou Yu Zhiwei Zhang Xupeng Chen Ziji Zhang Yingying Zhuang Narayanan Sadagopan and Anurag Beniwal. 2025. When thinking fails: The pitfalls of reasoning for instruction-following in LLMs. arXiv:2505.11423. Retrieved from https:\/\/arxiv.org\/abs\/2505.11423"},{"key":"e_1_3_3_34_2","unstructured":"Kamil\u0117 Luko\u0161i\u016bt\u0117 and Adam Swanda. 2025. LLM cyber evaluations don\u2019t capture real-world risk. arXiv250200072 [csCR]. Retrieved from https:\/\/arxiv.org\/abs\/2502.00072"},{"key":"e_1_3_3_35_2","unstructured":"Steve Morgan. 2025. Global Ransomware Damage Costs Predicted To Exceed $275 Billion By 2031. Retrieved June 02 2025 from https:\/\/cybersecurityventures.com\/global-ransomware-damage-costs-predicted-to-reach-250-billion-usd-by-2031\/"},{"key":"e_1_3_3_36_2","unstructured":"Lajos Muzsai David Imolai and Andr\u00e1s Luk\u00e1cs. 2024. HackSynth: LLM agent and evaluation framework for autonomous penetration testing. arXiv:2412.01778. Retrieved from https:\/\/arxiv.org\/abs\/2412.01778"},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSE54508.2022.10005490"},{"key":"e_1_3_3_38_2","unstructured":"Sho Nakatani. 2025. RapidPen: Fully automated IP-to-shell penetration testing with LLM-based agents. arXiv:2502.16730. Retrieved from https:\/\/arxiv.org\/abs\/2502.16730"},{"key":"e_1_3_3_39_2","unstructured":"Part of the National Cyber Security Centre (NCSC) New Zealand\u2019s CERT (Computer Emergency Response Team). 2023. How ransomware happens and how to stop it. Retrieved June 2 2025 from https:\/\/www.cert.govt.nz\/information-and-advice\/guides\/how-ransomware-happens-and-how-to-stop-it\/"},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/97243.97281"},{"key":"e_1_3_3_41_2","unstructured":"OpenAI. 2024. Introducing OpenAI o1-Preview. Retrieved June 2 2025 from https:\/\/openai.com\/index\/introducing-openai-o1-preview\/"},{"key":"e_1_3_3_42_2","unstructured":"OpenAI. 2024. Learning to Reason with LLMs. Retrieved June 6 2025 from https:\/\/openai.com\/index\/learning-to-reason-with-llms\/"},{"key":"e_1_3_3_43_2","unstructured":"OpenAI. 2025. As Some of You Have Noticed Avoid \u201cBoomer Prompts\u201d with O-Series Models. Instead Be Simple and Direct with Specific Guidelines. Retrieved June 11 2025 from https:\/\/x.com\/OpenAIDevs\/status\/1890147300493914437"},{"key":"e_1_3_3_44_2","unstructured":"OpenAI. 2025. Reasoning Best Practices. Retrieved June 10 2025 from https:\/\/platform.openai.com\/docs\/guides\/reasoning-best-practices"},{"key":"e_1_3_3_45_2","first-page":"5913","volume-title":"33rd USENIX Security Symposium (USENIX Security 24)","author":"De Pasquale Giulio","year":"2024","unstructured":"Giulio De Pasquale, Ilya Grishchenko, Riccardo Iesari, Gabriel Pizarro, Lorenzo Cavallaro, Christopher Kruegel, and Giovanni Vigna. 2024. ChainReactor: Automated privilege escalation chain discovery via AI planning. In 33rd USENIX Security Symposium (USENIX Security 24). USENIX Association, Philadelphia, PA, 5913\u20135929. Retrieved from https:\/\/www.usenix.org\/conference\/usenixsecurity24\/presentation\/de-pasquale"},{"key":"e_1_3_3_46_2","unstructured":"Ivo Petrov Jasper Dekoninck Lyuben Baltadzhiev Maria Drencheva Kristian Minchev Mislav Balunovi\u0107 Nikola Jovanovi\u0107 and Martin Vechev. 2025. Proof or Bluff? Evaluating LLMs on 2025 USA math olympiad. arXiv:2503.21934. Retrieved from https:\/\/arxiv.org\/abs\/2503.21934"},{"key":"e_1_3_3_47_2","unstructured":"Boomer Prompts. 2025. BoomerPrompts. Retrieved June 11 2025 from https:\/\/boomerprompts.com\/"},{"key":"e_1_3_3_48_2","doi-asserted-by":"crossref","unstructured":"Pat Rondon Renyao Wei Jos\u00e9 Cambronero J\u00fcrgen Cito Aaron Sun Siddhant Sanyam Michele Tufano and Satish Chandra. 2025. Evaluating agent-based program repair at Google. arXiv:2501.07531. Retrieved from https:\/\/arxiv.org\/abs\/2501.07531","DOI":"10.1109\/ICSE-SEIP66354.2025.00038"},{"key":"e_1_3_3_49_2","unstructured":"Shanto Roy Emmanouil Panaousis Cameron Noakes Aron Laszka Sakshyam Panda and George Loukas. 2023. SoK: The MITRE ATT&CK framework in research and practice. arXiv:2304.07411. Retrieved from https:\/\/arxiv.org\/abs\/2304.07411"},{"key":"e_1_3_3_50_2","first-page":"1816","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","volume":"26","author":"Sarraute Carlos","year":"2012","unstructured":"Carlos Sarraute, Olivier Buffet, and J\u00f6rg Hoffmann. 2012. POMDPs make better hackers: Accounting for uncertainty in penetration testing. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 26, 1816\u20131824."},{"key":"e_1_3_3_51_2","unstructured":"Carlos Sarraute Olivier Buffet and J\u00f6rg Hoffmann. 2013. Penetration testing== POMDP solving? arXiv:1306.4714. Retrieved from https:\/\/arxiv.org\/abs\/1306.4714"},{"key":"e_1_3_3_52_2","unstructured":"Minghao Shao Boyuan Chen Sofija Jancheska Brendan Dolan-Gavitt Siddharth Garg Ramesh Karri and Muhammad Shafique. 2024. An empirical evaluation of LLMs for solving offensive security challenges. arXiv:2402.11814. Retrieved from https:\/\/arxiv.org\/abs\/2402.11814"},{"key":"e_1_3_3_53_2","unstructured":"Minghao Shao Sofija Jancheska Meet Udeshi Brendan Dolan-Gavitt Haoran Xi Kimberly Milner Boyuan Chen Max Yin Siddharth Garg Prashanth Krishnamurthy et al. 2024. NYU CTF Dataset: A scalable open-source benchmark dataset for evaluating LLMs in offensive security. arXiv:2406.05590. Retrieved from https:\/\/arxiv.org\/abs\/2406.05590"},{"key":"e_1_3_3_54_2","unstructured":"Venu Shastri. 2022. Attackers Set Sights on Active Directory: Understanding Your Identity Exposure. Retrieved June 2 2025 from https:\/\/www.crowdstrike.com\/en-us\/blog\/attackers-set-sights-on-active-directory-understanding-your-identity-exposure\/"},{"key":"e_1_3_3_55_2","article-title":"Reflexion: Language agents with verbal reinforcement learning","volume":"36","author":"Shinn Noah","year":"2024","unstructured":"Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2024. Reflexion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, Vol. 36.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_3_56_2","doi-asserted-by":"crossref","unstructured":"Parshin Shojaee Iman Mirzadeh Keivan Alizadeh Maxwell Horton Samy Bengio and Mehrdad Farajtabar. 2025. The illusion of thinking: Understanding the strengths and limitations of reasoning models via the lens of problem complexity. arXiv:2506.06941. Retrieved from https:\/\/arxiv.org\/abs\/2506.06941","DOI":"10.70777\/si.v2i6.15919"},{"key":"e_1_3_3_57_2","unstructured":"Brian Singer Keane Lucas Lakshmi Adiga Meghna Jain Lujo Bauer and Vyas Sekar. 2025. On the feasibility of using LLMs to execute multistage network attacks. arXiv:2501.16466. Retrieved from https:\/\/arxiv.org\/abs\/2501.16466"},{"key":"e_1_3_3_58_2","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2010.25"},{"issue":"207","key":"e_1_3_3_59_2","first-page":"800","article-title":"Zero trust architecture","volume":"800","author":"Stafford V.","year":"2020","unstructured":"V. Stafford. 2020. Zero trust architecture. NIST Special Publication 800, 207 (2020), 800\u2013207.","journal-title":"NIST Special Publication"},{"key":"e_1_3_3_60_2","unstructured":"Jai Vijayan. 2025. 25 Years On Active Directory Is Still a Prime Attack Target. Retrieved June 2 2025 from https:\/\/www.darkreading.com\/identity-access-management-security\/25-years-active-directory-prime-attack-target"},{"key":"e_1_3_3_61_2","doi-asserted-by":"crossref","unstructured":"Lei Wang Wanyu Xu Yihuai Lan Zhiqiang Hu Yunshi Lan Roy Ka-Wei Lee and Ee-Peng Lim. 2023. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. arXiv:2305.04091. Retrieved from https:\/\/arxiv.org\/abs\/2305.04091","DOI":"10.18653\/v1\/2023.acl-long.147"},{"key":"e_1_3_3_62_2","first-page":"24824","article-title":"Chain-of-thought prompting elicits reasoning in large language models","volume":"35","author":"Wei Jason","year":"2022","unstructured":"Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V. Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems 35 (2022), 24824\u201324837.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_3_63_2","unstructured":"Benlong Wu Guoqiang Chen Kejiang Chen Xiuwei Shang Jiapeng Han Yanru He Weiming Zhang and Nenghai Yu. 2024. AutoPT: How far are we from the End2End automated web penetration testing? arXiv:2411.01236. Retrieved from https:\/\/arxiv.org\/abs\/2411.01236"},{"key":"e_1_3_3_64_2","unstructured":"Siwei Wu Zhongyuan Peng Xinrun Du Tuney Zheng Minghao Liu Jialong Wu Jiachen Ma Yizhi Li Jian Yang Wangchunshu Zhou et al. 2024. A comparative study on reasoning patterns of OpenAI\u2019s o1 model. arXiv:2410.13639. Retrieved from https:\/\/arxiv.org\/abs\/2410.13639"},{"key":"e_1_3_3_65_2","unstructured":"Jiacen Xu Jack W. Stokes Geoff McDonald Xuesong Bai David Marshall Siyue Wang Adith Swaminathan and Zhou Li. 2024. Autoattacker: A large language model guided system to implement automatic cyber-attacks. arXiv:2403.01038. Retrieved from https:\/\/arxiv.org\/abs\/2403.01038"},{"key":"e_1_3_3_66_2","unstructured":"An Yang Anfeng Li Baosong Yang Beichen Zhang Binyuan Hui Bo Zheng Bowen Yu Chang Gao Chengen Huang Chenxu Lv et al. 2025. Qwen3 technical report. arXiv:2505.09388. Retrieved from https:\/\/arxiv.org\/abs\/2505.09388"},{"key":"e_1_3_3_67_2","unstructured":"John Yang Carlos E. Jimenez Alexander Wettig Kilian Lieret Shunyu Yao Karthik Narasimhan and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering. arXiv:2405.15793. Retrieved from https:\/\/arxiv.org\/abs\/2405.15793"},{"key":"e_1_3_3_68_2","doi-asserted-by":"publisher","DOI":"10.3724\/SP.J.1042.2022.00511"},{"key":"e_1_3_3_69_2","unstructured":"Shunyu Yao Jeffrey Zhao Dian Yu Nan Du Izhak Shafran Karthik Narasimhan and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. arXiv:2210.03629. Retrieved from https:\/\/arxiv.org\/abs\/2210.03629"},{"key":"e_1_3_3_70_2","unstructured":"Andy K. Zhang Neil Perry Riya Dulepet Joey Ji Justin W. Lin Eliot Jones Celeste Menders Gashon Hussein Samantha Liu Donovan Jasper et al. 2024. Cybench: A framework for evaluating cybersecurity capabilities and risks of language models. arXiv:2408.08926. Retrieved from https:\/\/arxiv.org\/abs\/2408.08926"},{"key":"e_1_3_3_71_2","unstructured":"Zhuosheng Zhang Aston Zhang Mu Li and Alex Smola. 2022. Automatic chain of thought prompting in large language models. arXiv:2210.03493. Retrieved from https:\/\/arxiv.org\/abs\/2210.03493"},{"key":"e_1_3_3_72_2","unstructured":"Tianyang Zhong Zhengliang Liu Yi Pan Yutong Zhang Yifan Zhou Shizhe Liang Zihao Wu Yanjun Lyu Peng Shu Xiaowei Yu et al. 2024. Evaluation of openai o1: Opportunities and challenges of agi. arXiv:2409.18486. Retrieved from https:\/\/arxiv.org\/abs\/2409.18486"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3766895","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,14]],"date-time":"2026-05-14T02:34:39Z","timestamp":1778726079000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3766895"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,5,13]]},"references-count":71,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2026,6,30]]}},"alternative-id":["10.1145\/3766895"],"URL":"https:\/\/doi.org\/10.1145\/3766895","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,5,13]]},"assertion":[{"value":"2025-02-16","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-08-24","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-05-13","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}