{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,23]],"date-time":"2026-10-23T23:48:42Z","timestamp":1792799322065,"version":"4.5.0"},"publisher-location":"New York, NY, USA","reference-count":54,"publisher":"ACM","funder":[{"name":"Google-BAIR Commons","award":["Year 6"],"award-info":[{"award-number":["Year 6"]}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["2229876"],"award-info":[{"award-number":["2229876"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"name":"OpenAI","award":[""],"award-info":[{"award-number":[""]}],"id":[{"id":"https:\/\/ror.org\/05wx9n238","id-type":"ROR","asserted-by":"crossref"}]},{"DOI":"10.13039\/100014895","name":"Open Philanthropy Project","doi-asserted-by":"publisher","award":[""],"award-info":[{"award-number":[""]}],"id":[{"id":"10.13039\/100014895","id-type":"DOI","asserted-by":"publisher"}]},{"name":"the Department of Homeland Security","award":[""],"award-info":[{"award-number":[""]}]},{"name":"IBM","award":[""],"award-info":[{"award-number":[""]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2025,10,13]]},"DOI":"10.1145\/3733799.3762982","type":"proceedings-article","created":{"date-parts":[[2025,12,30]],"date-time":"2025-12-30T11:38:49Z","timestamp":1767094729000},"page":"242-252","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":25,"title":["Defending Against Prompt Injection With a Few DefensiveTokens"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3274-6926","authenticated-orcid":false,"given":"Sizhe","family":"Chen","sequence":"first","affiliation":[{"name":"UC Berkeley, Berkeley, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-9626-0290","authenticated-orcid":false,"given":"Yizhu","family":"Wang","sequence":"additional","affiliation":[{"name":"UC Berkeley, Berkeley, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1463-3461","authenticated-orcid":false,"given":"Nicholas","family":"Carlini","sequence":"additional","affiliation":[{"name":"Google, Mountain View, CA, USA and Anthropic, San Francisco, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4949-9661","authenticated-orcid":false,"given":"Chawin","family":"Sitawarin","sequence":"additional","affiliation":[{"name":"Google, Mountain View, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9944-9232","authenticated-orcid":false,"given":"David","family":"Wagner","sequence":"additional","affiliation":[{"name":"UC Berkeley, Berkeley, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,12,30]]},"reference":[{"key":"e_1_3_3_1_2_2","doi-asserted-by":"crossref","unstructured":"Sahar Abdelnabi Aideen Fay Giovanni Cherubin Ahmed Salem Mario Fritz and Andrew Paverd. 2025. Get my drift? Catching LLM Task Drift with Activation Deltas. arxiv:https:\/\/arXiv.org\/abs\/2406.00799https:\/\/arxiv.org\/abs\/2406.00799","DOI":"10.1109\/SaTML64287.2025.00011"},{"key":"e_1_3_3_1_3_2","unstructured":"Manish Bhatt Sahana Chennabasappa Yue Li Cyrus Nikolaidis Daniel Song Shengye Wan Faizan Ahmad Cornelius Aschermann Yaohui Chen Dhaval Kapil et\u00a0al. 2024. Cyberseceval 2: A wide-ranging cybersecurity evaluation suite for large language models. arxiv:https:\/\/arXiv.org\/abs\/2404.13161https:\/\/arxiv.org\/abs\/2404.13161"},{"key":"e_1_3_3_1_4_2","volume-title":"USENIX Security Symposium","author":"Chen Sizhe","year":"2025","unstructured":"Sizhe Chen, Julien Piet, Chawin Sitawarin, and David Wagner. 2025. StruQ: Defending against prompt injection with structured queries. In USENIX Security Symposium. https:\/\/arxiv.org\/abs\/2402.06363"},{"key":"e_1_3_3_1_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/3719027.3744836"},{"key":"e_1_3_3_1_6_2","unstructured":"Sizhe Chen Arman Zharmagambetov David Wagner and Chuan Guo. 2025. Meta SecAlign: A Secure Foundation LLM Against Prompt Injection Attacks. arXiv:https:\/\/arXiv.org\/abs\/2507.02735 (2025). https:\/\/arxiv.org\/abs\/2507.02735"},{"key":"e_1_3_3_1_7_2","doi-asserted-by":"publisher","DOI":"10.5555\/3692070.3692401"},{"key":"e_1_3_3_1_8_2","unstructured":"Edoardo Debenedetti Ilia Shumailov Tianqi Fan Jamie Hayes Nicholas Carlini Daniel Fabian Christoph Kern Chongyang Shi Andreas Terzis and Florian Tram\u00e8r. 2025. Defeating prompt injections by design. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2503.18813 (2025). https:\/\/arxiv.org\/abs\/2503.18813"},{"key":"e_1_3_3_1_9_2","doi-asserted-by":"publisher","DOI":"10.5555\/3666122.3667430"},{"key":"e_1_3_3_1_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3605764.3623985"},{"key":"e_1_3_3_1_11_2","volume-title":"International Conference on Learning Representations (ICLR)","author":"Hu Edward\u00a0J","year":"2022","unstructured":"Edward\u00a0J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et\u00a0al. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations (ICLR). https:\/\/arxiv.org\/abs\/2106.09685"},{"key":"e_1_3_3_1_12_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2025.findings-naacl.123"},{"key":"e_1_3_3_1_13_2","unstructured":"Omar Khattab Arnav Singhvi Paridhi Maheshwari Zhiyuan Zhang Keshav Santhanam Sri Vardhamanan Saiful Haq Ashutosh Sharma Thomas\u00a0T. Joshi Hanna Moazam Heather Miller Matei Zaharia and Christopher Potts. 2024. DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines. https:\/\/openreview.net\/pdf?id=sY5N0zY5Od"},{"key":"e_1_3_3_1_14_2","unstructured":"Learn Prompting. 2023. Learn Prompting: Your guide to communicating with AI. https:\/\/learnprompting.org."},{"key":"e_1_3_3_1_15_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.243"},{"key":"e_1_3_3_1_16_2","unstructured":"Xuechen Li Tianyi Zhang Yann Dubois Rohan Taori Ishaan Gulrajani Carlos Guestrin Percy Liang and Tatsunori\u00a0B. Hashimoto. 2023. AlpacaEval: An Automatic Evaluator of Instruction-following Models. https:\/\/github.com\/tatsu-lab\/alpaca_eval"},{"key":"e_1_3_3_1_17_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.acl-long.353"},{"key":"e_1_3_3_1_18_2","unstructured":"Huawei Lin Yingjie Lao Tong Geng Tan Yu and Weijie Zhao. 2025. UniGuardian: A Unified Defense for Detecting Prompt Injection Backdoor Attacks and Adversarial Attacks in Large Language Models. https:\/\/arxiv.org\/abs\/2502.13141"},{"key":"e_1_3_3_1_19_2","unstructured":"Xiaogeng Liu Zhiyuan Yu Yizhe Zhang Ning Zhang and Chaowei Xiao. 2024. Automatic and Universal Prompt Injection Attacks against Large Language Models. https:\/\/arxiv.org\/abs\/2403.04957"},{"key":"e_1_3_3_1_20_2","first-page":"1831","volume-title":"USENIX Security Symposium","author":"Liu Yupei","year":"2024","unstructured":"Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, and Neil\u00a0Zhenqiang Gong. 2024. Formalizing and benchmarking prompt injection attacks and defenses. In USENIX Security Symposium. 1831\u20131847. https:\/\/www.usenix.org\/conference\/usenixsecurity24\/presentation\/liu-yupei"},{"key":"e_1_3_3_1_21_2","doi-asserted-by":"publisher","DOI":"10.1109\/SP61157.2025.00250"},{"key":"e_1_3_3_1_22_2","unstructured":"Sourab Mangrulkar Sylvain Gugger Lysandre Debut Younes Belkada Sayak Paul and Benjamin Bossan. 2022. PEFT: State-of-the-art Parameter-Efficient Fine-Tuning methods. ttps:\/\/github.com\/huggingface\/peft"},{"key":"e_1_3_3_1_23_2","unstructured":"Meta. 2024. Prompt Guard. https:\/\/llama.meta.com\/docs\/model-cards-and-prompt-formats\/prompt-guard"},{"key":"e_1_3_3_1_24_2","volume-title":"Annual Conference on Neural Information Processing Systems (NeurIPS)","author":"Mo Yichuan","year":"2024","unstructured":"Yichuan Mo, Yuji Wang, Zeming Wei, and Yisen Wang. 2024. Fight back against jailbreaking via prompt adversarial tuning. In Annual Conference on Neural Information Processing Systems (NeurIPS). https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2024\/file\/759ca99a82e2a9137c6bef4811c8d378-Paper-Conference.pdf"},{"key":"e_1_3_3_1_25_2","unstructured":"OpenAI. 2024. Safety evaluations hub. https:\/\/openai.com\/safety\/evaluations-hub."},{"key":"e_1_3_3_1_26_2","unstructured":"OpenAI. 2025. ChatGPT Agent System Card. (2025). https:\/\/cdn.openai.com\/pdf\/839e66fc-602c-48bf-81d3-b21eacc3459d\/chatgpt_agent_system_card.pdf"},{"key":"e_1_3_3_1_27_2","unstructured":"OWASP. 2023. OWASP Top 10 for LLM Applications. https:\/\/llmtop10.com"},{"key":"e_1_3_3_1_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/3689932.3694764"},{"key":"e_1_3_3_1_29_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-70879-46"},{"key":"e_1_3_3_1_30_2","unstructured":"PromptArmor. 2024. Data Exfiltration from Slack AI via indirect prompt injection. https:\/\/promptarmor.substack.com\/p\/data-exfiltration-from-slack-ai-via"},{"key":"e_1_3_3_1_31_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.emnlp-main.494"},{"key":"e_1_3_3_1_32_2","unstructured":"Embrace\u00a0The Red. 2025. ChatGPT Operator: Prompt Injection Exploits & Defenses. https:\/\/embracethered.com\/blog\/posts\/2025\/chatgpt-operator-prompt-injection-exploits"},{"key":"e_1_3_3_1_33_2","unstructured":"Johann Rehberger. 2023. Hacking Google Bard - From Prompt Injection to Data Exfiltration. https:\/\/embracethered.com\/blog\/posts\/2023\/google-bard-data-exfiltration"},{"key":"e_1_3_3_1_34_2","unstructured":"Johann Rehberger. 2024. ZombAIs: From Prompt Injection to C2 with Claude Computer Use. https:\/\/embracethered.com\/blog\/posts\/2024\/claude-computer-use-c2-the-zombais-are-coming."},{"key":"e_1_3_3_1_35_2","unstructured":"Gene Ruebsamen. 2024. Cleaned Alpaca Dataset. https:\/\/github.com\/gururise\/AlpacaDataCleaned"},{"key":"e_1_3_3_1_36_2","unstructured":"Sander Schulhoff. 2024. Instruction Defense. https:\/\/learnprompting.org\/docs\/prompt_hacking\/defensive_measures\/instruction"},{"key":"e_1_3_3_1_37_2","unstructured":"Sander Schulhoff. 2024. Sandwich Defense. https:\/\/learnprompting.org\/docs\/prompt_hacking\/defensive_measures\/sandwich_defense"},{"key":"e_1_3_3_1_38_2","unstructured":"Chongyang Shi Sharon Lin Shuang Song Jamie Hayes Ilia Shumailov Itay Yona Juliette Pluto Aneesh Pappu Christopher\u00a0A Choquette-Choo Milad Nasr et\u00a0al. 2025. Lessons from Defending Gemini Against Indirect Prompt Injections. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2505.14534 (2025). https:\/\/arxiv.org\/abs\/2505.14534"},{"key":"e_1_3_3_1_39_2","unstructured":"Eric Wallace Kai Xiao Reimar Leike Lilian Weng Johannes Heidecke and Alex Beutel. 2024. The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions. arxiv:https:\/\/arXiv.org\/abs\/2404.13208https:\/\/arxiv.org\/abs\/2404.13208"},{"key":"e_1_3_3_1_40_2","volume-title":"Neural Information Processing Systems (NeurIPS)","author":"Wei Alexander","year":"2023","unstructured":"Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023. Jailbroken: How Does LLM Safety Training Fail?. In Neural Information Processing Systems (NeurIPS). https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2023\/hash\/fd6613131889a4b656206c50a8bd7790-Abstract-Conference.html"},{"key":"e_1_3_3_1_41_2","volume-title":"International Conference on Machine Learning (ICML)","author":"Wei Zeming","year":"2024","unstructured":"Zeming Wei, Yifei Wang, and Yisen Wang. 2024. Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations. In International Conference on Machine Learning (ICML). https:\/\/arxiv.org\/abs\/2310.06387"},{"key":"e_1_3_3_1_42_2","unstructured":"Simon Willison. 2022. Prompt Injection Attacks against GPT-3. https:\/\/simonwillison.net\/2022\/Sep\/12\/prompt-injection\/"},{"key":"e_1_3_3_1_43_2","unstructured":"Tong Wu Chong Xiang Jiachen\u00a0T. Wang and Prateek Mittal. 2025. Effectively Controlling Reasoning Models through Thinking Intervention. arxiv:https:\/\/arXiv.org\/abs\/2503.24370https:\/\/arxiv.org\/abs\/2503.24370"},{"key":"e_1_3_3_1_44_2","volume-title":"International Conference on Learning Representations (ICLR)","author":"Wu Tong","year":"2025","unstructured":"Tong Wu, Shujian Zhang, Kaiqiang Song, Silei Xu, Sanqiang Zhao, Ravi Agrawal, Sathish\u00a0Reddy Indurthi, Chong Xiang, Prateek Mittal, and Wenxuan Zhou. 2025. Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchy. In International Conference on Learning Representations (ICLR). https:\/\/arxiv.org\/abs\/2410.09102"},{"key":"e_1_3_3_1_45_2","unstructured":"Lingling Xu Haoran Xie Si-Zhao\u00a0Joe Qin Xiaohui Tao and Fu\u00a0Lee Wang. 2023. Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment. https:\/\/arxiv.org\/abs\/2312.12148"},{"key":"e_1_3_3_1_46_2","volume-title":"International Conference on Learning Representations (ICLR)","author":"Yao Shunyu","year":"2023","unstructured":"Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR). https:\/\/par.nsf.gov\/servlets\/purl\/10451467"},{"key":"e_1_3_3_1_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/3690624.3709179"},{"key":"e_1_3_3_1_48_2","unstructured":"Li Yin and Zhangyang Wang. 2025. LLM-AutoDiff: Auto-Differentiate Any LLM Workflow. https:\/\/ui.adsabs.harvard.edu\/abs\/2025arXiv250116673Y\/abstract"},{"key":"e_1_3_3_1_49_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-025-08661-4"},{"key":"e_1_3_3_1_50_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.findings-acl.624"},{"key":"e_1_3_3_1_51_2","doi-asserted-by":"publisher","unstructured":"Yanli Zhao Andrew Gu Rohan Varma Liang Luo Chien-Chin Huang Min Xu Less Wright Hamid Shojanazeri Myle Ott Sam Shleifer Alban Desmaison Can Balioglu Pritam Damania Bernard Nguyen Geeta Chauhan Yuchen Hao Ajit Mathews and Shen Li. 2023. Pytorch FSDP: experiences on scaling fully sharded data parallel. Proc. VLDB Endow. 16 12 (2023) 3848\u20133860. 10.14778\/3611540.3611569","DOI":"10.14778\/3611540.3611569"},{"key":"e_1_3_3_1_52_2","first-page":"61593","volume-title":"International Conference on Machine Learning (ICML)","author":"Zheng Chujie","year":"2024","unstructured":"Chujie Zheng, Fan Yin, Hao Zhou, Fandong Meng, Jie Zhou, Kai-Wei Chang, Minlie Huang, and Nanyun Peng. 2024. On Prompt-Driven Safeguarding for Large Language Models. In International Conference on Machine Learning (ICML). 61593\u201361613. https:\/\/arxiv.org\/abs\/2401.18018"},{"key":"e_1_3_3_1_53_2","volume-title":"Annual Conference on Neural Information Processing Systems (NeurIPS)","author":"Zhou Andy","year":"2024","unstructured":"Andy Zhou, Bo Li, and Haohan Wang. 2024. Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks. In Annual Conference on Neural Information Processing Systems (NeurIPS). https:\/\/arxiv.org\/abs\/2401.17263"},{"key":"e_1_3_3_1_54_2","unstructured":"Andy Zou Zifan Wang Nicholas Carlini Milad Nasr J.\u00a0Zico Kolter and Matt Fredrikson. 2023. Universal and Transferable Adversarial Attacks on Aligned Language Models. https:\/\/arxiv.org\/abs\/2307.15043"},{"key":"e_1_3_3_1_55_2","volume-title":"International Conference on Learning Representations (ICLR)","author":"Zverev Egor","year":"2025","unstructured":"Egor Zverev, Sahar Abdelnabi, Mario Fritz, and Christoph\u00a0H Lampert. 2025. Can LLMs Separate Instructions From Data? And What Do We Even Mean By That?. In International Conference on Learning Representations (ICLR). https:\/\/openreview.net\/pdf?id=8EtSBX41mt"}],"event":{"name":"AISec '25: Proceedings of the 2025 Workshop on Artificial Intelligence and Security","location":"Taipei , Taiwan","acronym":"AISec '25","sponsor":["SIGSAC ACM Special Interest Group on Security, Audit, and Control"]},"container-title":["Proceedings of the 18th ACM Workshop on Artificial Intelligence and Security"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3733799.3762982","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,12,30]],"date-time":"2025-12-30T11:52:23Z","timestamp":1767095543000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3733799.3762982"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,13]]},"references-count":54,"alternative-id":["10.1145\/3733799.3762982","10.1145\/3733799"],"URL":"https:\/\/doi.org\/10.1145\/3733799.3762982","relation":{},"subject":[],"published":{"date-parts":[[2025,10,13]]},"assertion":[{"value":"2025-12-30","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}