{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T01:56:03Z","timestamp":1787018163693,"version":"build-2736575974"},"reference-count":53,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2024,12,24]],"date-time":"2024-12-24T00:00:00Z","timestamp":1734998400000},"content-version":"vor","delay-in-days":358,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,12,18]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Retrieval Augmented Language Models (RALMs) have gained significant attention for their ability to generate accurate answers and improve efficiency. However, RALMs are inherently vulnerable to imperfect information due to their reliance on the imperfect retriever or knowledge source. We identify three common scenarios\u2014unanswerable, adversarial, conflicting\u2014where retrieved document sets can confuse RALMs with plausible real-world examples. We present the first comprehensive investigation to assess how well RALMs detect and handle such problematic scenarios. Among these scenarios, to systematically examine adversarial robustness we propose a new adversarial attack method, Generative model-based ADVersarial attack (GenADV) and a novel metric Robustness under Additional Document (RAD). Our findings reveal that RALMs often fail to identify the unanswerability or contradiction of a document set, which frequently leads to hallucinations. Moreover, we show that the addition of an adversary significantly degrades RALM\u2019s performance, with the model becoming even more vulnerable when the two scenarios overlap (adversarial+ unanswerable). Our research identifies critical areas for assessing and enhancing the robustness of RALMs, laying the foundation for the development of more robust models.1<\/jats:p>","DOI":"10.1162\/tacl_a_00724","type":"journal-article","created":{"date-parts":[[2024,12,24]],"date-time":"2024-12-24T11:39:12Z","timestamp":1735040352000},"page":"1686-1702","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":5,"title":["Toward Robust RALMs: Revealing the Impact of Imperfect Retrieval on Retrieval-Augmented Language Models"],"prefix":"10.1162","volume":"12","author":[{"given":"Seong-Il","family":"Park","sequence":"first","affiliation":[{"name":"Graduate School of Data Science Seoul National University Seoul, Republic of Korea. athjk3@snu.ac.kr"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jay-Yoon","family":"Lee","sequence":"additional","affiliation":[{"name":"Graduate School of Data Science Seoul National University Seoul, Republic of Korea. lee.jayyoon@snu.ac.kr"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2024,12,18]]},"reference":[{"key":"2024122416390489900_bib1","article-title":"Gpt-4 technical report","author":"Achiam","year":"2023","journal-title":"arXiv preprint arXiv:2303.08774"},{"key":"2024122416390489900_bib2","unstructured":"Rohan\n              Anil\n            , Andrew MDai, OrhanFirat, MelvinJohnson, DmitryLepikhin, AlexandrePassos, SiamakShakeri, EmanuelTaropa, PaigeBailey, ZhifengChen, \n          2023. Palm 2 technical report. arXiv preprint arXiv:2305.10403."},{"key":"2024122416390489900_bib3","doi-asserted-by":"publisher","first-page":"1492","DOI":"10.18653\/v1\/2021.acl-long.118","article-title":"Challenges in information-seeking qa: Unanswerable questions and paragraph retrieval","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)","author":"Asai","year":"2021"},{"key":"2024122416390489900_bib4","article-title":"Self-rag: Learning to retrieve, generate, and critique through self-reflection","author":"Asai","year":"2023","journal-title":"arXiv preprint arXiv:2310.11511"},{"key":"2024122416390489900_bib5","unstructured":"Jinze\n              Bai\n            , ShuaiBai, YunfeiChu, ZeyuCui, KaiDang, XiaodongDeng, YangFan, WenbinGe, YuHan, FeiHuang, BinyuanHui, LuoJi, MeiLi, JunyangLin, RunjiLin, DayihengLiu, GaoLiu, ChengqiangLu, KemingLu, JianxinMa, RuiMen, XingzhangRen, XuanchengRen, ChuanqiTan, SinanTan, JianhongTu, PengWang, ShijieWang, WeiWang, ShengguangWu, BenfengXu, JinXu, AnYang, HaoYang, JianYang, ShushengYang, YangYao, BowenYu, HongyiYuan, ZhengYuan, JianweiZhang, XingxuanZhang, YichangZhang, ZhenruZhang, ChangZhou, JingrenZhou, XiaohuanZhou, and TianhangZhu. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609."},{"key":"2024122416390489900_bib6","doi-asserted-by":"crossref","first-page":"1533","DOI":"10.18653\/v1\/D13-1160","article-title":"Semantic parsing on freebase from question-answer pairs","volume-title":"Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing","author":"Berant","year":"2013"},{"key":"2024122416390489900_bib7","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"Brown","year":"2020","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2024122416390489900_bib8","doi-asserted-by":"publisher","first-page":"11975","DOI":"10.18653\/v1\/2022.emnlp-main.821","article-title":"TASA: Deceiving question answering models by twin answer sentences attack","volume-title":"Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing","author":"Cao","year":"2022"},{"key":"2024122416390489900_bib9","doi-asserted-by":"publisher","first-page":"1870","DOI":"10.18653\/v1\/P17-1171","article-title":"Reading wikipedia to answer open-domain questions","volume-title":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Chen","year":"2017"},{"key":"2024122416390489900_bib10","doi-asserted-by":"crossref","first-page":"2292","DOI":"10.18653\/v1\/2022.emnlp-main.146","article-title":"Rich knowledge sources bring complex knowledge conflicts: Recalibrating models to reflect conflicting evidence","volume-title":"Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing","author":"Chen","year":"2022"},{"key":"2024122416390489900_bib11","article-title":"Selection-inference: Exploiting large language models for interpretable logical reasoning","author":"Creswell","year":"2022","journal-title":"arXiv preprint arXiv:2205.09712"},{"key":"2024122416390489900_bib12","doi-asserted-by":"publisher","first-page":"10581","DOI":"10.1609\/aaai.v36i10.21302","article-title":"Synthetic disinformation attacks on automated fact verification systems","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","author":"Yibing","year":"2022"},{"key":"2024122416390489900_bib13","first-page":"3929","article-title":"Retrieval augmented language model pre-training","volume-title":"International Conference on Machine Learning","author":"Guu","year":"2020"},{"key":"2024122416390489900_bib14","article-title":"Won\u2019t get fooled again: Answering questions with false premises","author":"Shengding","year":"2023","journal-title":"arXiv preprint arXiv:2307.02394"},{"key":"2024122416390489900_bib15","doi-asserted-by":"publisher","first-page":"874","DOI":"10.18653\/v1\/2021.eacl-main.74","article-title":"Leveraging passage retrieval with generative models for open domain question answering","volume-title":"Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume","author":"Izacard","year":"2021"},{"key":"2024122416390489900_bib16","article-title":"Few-shot learning with retrieval augmented language models","author":"Izacard","year":"2022","journal-title":"arXiv preprint arXiv:2208.03299"},{"key":"2024122416390489900_bib17","doi-asserted-by":"publisher","first-page":"2021","DOI":"10.18653\/v1\/D17-1215","article-title":"Adversarial examples for evaluating reading comprehension systems","volume-title":"Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing","author":"Jia","year":"2017"},{"key":"2024122416390489900_bib18","article-title":"Mistral 7b","author":"Jiang","year":"2023","journal-title":"arXiv preprint arXiv:2310.06825"},{"key":"2024122416390489900_bib19","doi-asserted-by":"publisher","first-page":"8018","DOI":"10.1609\/aaai.v34i05.6311","article-title":"Is bert really robust? A strong baseline for natural language attack on text classification and entailment","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","author":"Di","year":"2020"},{"key":"2024122416390489900_bib20","doi-asserted-by":"publisher","first-page":"1601","DOI":"10.18653\/v1\/P17-1147","article-title":"Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension","volume-title":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Joshi","year":"2017"},{"key":"2024122416390489900_bib21","doi-asserted-by":"publisher","first-page":"5591","DOI":"10.18653\/v1\/2023.acl-long.307","article-title":"Evaluating open- domain question answering in the era of large language models","volume-title":"Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Kamalloo","year":"2023"},{"key":"2024122416390489900_bib22","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.550","article-title":"Dense passage retrieval for open-domain question answering","author":"Karpukhin","year":"2020","journal-title":"arXiv preprint arXiv:2004.04906"},{"key":"2024122416390489900_bib23","doi-asserted-by":"publisher","first-page":"611","DOI":"10.1145\/3600006.3613165","article-title":"Efficient memory management for large language model serving with pagedattention","volume-title":"Proceedings of the 29th Symposium on Operating Systems Principles","author":"Kwon","year":"2023"},{"key":"2024122416390489900_bib24","article-title":"Latent retrieval for weakly supervised open domain question answering","author":"Lee","year":"2019","journal-title":"arXiv preprint arXiv:1906.00300"},{"key":"2024122416390489900_bib25","doi-asserted-by":"publisher","first-page":"1095","DOI":"10.18653\/v1\/2022.findings-naacl.83","article-title":"Phrase-level textual adversarial attack with label preservation","volume-title":"Findings of the Association for Computational Linguistics: NAACL 2022","author":"Lei","year":"2022"},{"key":"2024122416390489900_bib26","article-title":"Standing on the shoulders of giant frozen language models","author":"Levine","year":"2022","journal-title":"arXiv preprint arXiv:2204.10019"},{"key":"2024122416390489900_bib27","article-title":"Huge frozen language models as readers for open-domain question answering","volume-title":"ICML 2022 Workshop on Knowledge Retrieval and Language Models","author":"Levine","year":"2022"},{"key":"2024122416390489900_bib28","first-page":"9459","article-title":"Retrieval-augmented generation for knowledge- intensive nlp tasks","volume":"33","author":"Lewis","year":"2020","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2024122416390489900_bib29","doi-asserted-by":"publisher","first-page":"1774","DOI":"10.18653\/v1\/2023.findings-acl.112","article-title":"Large language models with controllable working memory","volume-title":"Findings of the Association for Computational Linguistics: ACL 2023","author":"Li","year":"2023"},{"key":"2024122416390489900_bib30","first-page":"5053","article-title":"Contextualized perturbation for textual adversarial attack","volume-title":"Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Li","year":"2021"},{"key":"2024122416390489900_bib31","article-title":"Ra-dit: Retrieval-augmented dual instruction tuning","author":"Xi","year":"2023","journal-title":"arXiv preprint arXiv:2310.01352"},{"key":"2024122416390489900_bib32","doi-asserted-by":"publisher","first-page":"7052","DOI":"10.18653\/v1\/2021.emnlp-main.565","article-title":"Entity-based knowledge conflicts in question answering","volume-title":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing","author":"Longpre","year":"2021"},{"key":"2024122416390489900_bib33","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.acl-long.546","article-title":"When not to trust language models: Investigating effectiveness of parametric and non-parametric memories","volume-title":"The 61st Annual Meeting Of The Association For Computational Linguistics","author":"Mallen","year":"2023"},{"key":"2024122416390489900_bib34","article-title":"Orca 2: Teaching small language models how to reason","author":"Mitra","year":"2023","journal-title":"arXiv preprint arXiv:2311.11045"},{"key":"2024122416390489900_bib35","doi-asserted-by":"publisher","first-page":"525","DOI":"10.18653\/v1\/2023.ijcnlp-main.35","article-title":"Attacking open-domain question answering by injecting misinformation","volume-title":"Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Pan","year":"2023"},{"key":"2024122416390489900_bib36","article-title":"How context affects language models\u2019 factual predictions","author":"Petroni","year":"2020","journal-title":"arXiv preprint arXiv:2005.04611"},{"key":"2024122416390489900_bib37","doi-asserted-by":"publisher","first-page":"5582","DOI":"10.18653\/v1\/P19-1561","article-title":"Combating adversarial misspellings with robust word recognition","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Pruthi","year":"2019"},{"key":"2024122416390489900_bib38","doi-asserted-by":"crossref","first-page":"1339","DOI":"10.18653\/v1\/2023.emnlp-main.85","article-title":"Is chatgpt a general-purpose natural language processing task solver?","volume-title":"Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing","author":"Qin","year":"2023"},{"key":"2024122416390489900_bib39","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00605","article-title":"In-context retrieval-augmented language models","author":"Ram","year":"2023","journal-title":"arXiv preprint arXiv:2302.00083"},{"key":"2024122416390489900_bib40","doi-asserted-by":"publisher","first-page":"3982","DOI":"10.18653\/v1\/D19-1410","article-title":"Sentence-bert: Sentence embeddings using siamese bert-networks","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Reimers","year":"2019"},{"key":"2024122416390489900_bib41","article-title":"Out-of-distribution detection and selective generation for conditional language models","volume-title":"The Eleventh International Conference on Learning Representations","author":"Ren","year":"2022"},{"key":"2024122416390489900_bib42","article-title":"Investigating the factual knowledge boundary of large language models with retrieval augmentation","author":"Ren","year":"2023","journal-title":"arXiv preprint arXiv:2307.11019"},{"key":"2024122416390489900_bib43","doi-asserted-by":"publisher","first-page":"3715","DOI":"10.18653\/v1\/2022.naacl-main.272","article-title":"Colbertv2: Effective and efficient retrieval via lightweight late interaction","volume-title":"Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Santhanam","year":"2022"},{"key":"2024122416390489900_bib44","first-page":"31210","article-title":"Large language models can be easily distracted by irrelevant context","volume-title":"International Conference on Machine Learning","author":"Shi","year":"2023"},{"key":"2024122416390489900_bib45","article-title":"Replug: Retrieval-augmented black-box language models","author":"Shi","year":"2023","journal-title":"arXiv preprint arXiv:2301.12652"},{"key":"2024122416390489900_bib46","doi-asserted-by":"publisher","first-page":"1075","DOI":"10.18653\/v1\/2022.naacl-main.79","article-title":"Yes, no or idk: The challenge of unanswerable yes\/no questions","volume-title":"Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Sulem","year":"2022"},{"key":"2024122416390489900_bib47","article-title":"Gemma: Open models based on gemini research and technology","author":"Team","year":"2024","journal-title":"arXiv preprint arXiv:2403.08295"},{"key":"2024122416390489900_bib48","article-title":"Llama 2: Open foundation and fine-tuned chat models","author":"Touvron","year":"2023","journal-title":"arXiv preprint arXiv:2307.09288"},{"key":"2024122416390489900_bib49","article-title":"Defending against poisoning attacks in open-domain question answering","author":"Weller","year":"2022","journal-title":"arXiv preprint arXiv:2212.10002"},{"key":"2024122416390489900_bib50","article-title":"Adaptive chameleon or stubborn sloth: Unraveling the behavior of large language models in knowledge conflicts","author":"Xie","year":"2023","journal-title":"arXiv preprint arXiv:2305.13300"},{"key":"2024122416390489900_bib51","article-title":"Recomp: Improving retrieval-augmented lms with compression and selective augmentation","author":"Fangyuan","year":"2023","journal-title":"arXiv preprint arXiv:2310.04408"},{"key":"2024122416390489900_bib52","article-title":"Making retrieval- augmented language models robust to irrelevant context","author":"Yoran","year":"2023","journal-title":"arXiv preprint arXiv:2310.01558"},{"key":"2024122416390489900_bib53","article-title":"Chain-of-note: Enhancing robustness in retrieval-augmented language models","author":"Wenhao","year":"2023","journal-title":"arXiv preprint arXiv:2311.09210"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00724\/2487355\/tacl_a_00724.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00724\/2487355\/tacl_a_00724.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,12,24]],"date-time":"2024-12-24T11:39:19Z","timestamp":1735040359000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00724\/125985\/Toward-Robust-RALMs-Revealing-the-Impact-of"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024]]},"references-count":53,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00724","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024]]},"published":{"date-parts":[[2024]]}}}