{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,5]],"date-time":"2026-10-05T18:03:26Z","timestamp":1791223406855,"version":"4.1.0"},"reference-count":39,"publisher":"Association for Computing Machinery (ACM)","issue":"3","funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["U23B2057, 62176185, 62276063"],"award-info":[{"award-number":["U23B2057, 62176185, 62276063"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Shanghai Pilot Program for Basic Research","award":["22TQ1400300"],"award-info":[{"award-number":["22TQ1400300"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2026,3,31]]},"abstract":"<jats:p>\n                    Recent advancements in Large Language Models (LLMs) have significantly extended context windows, igniting discussions about the necessity of Retrieval-Augmented Generation (RAG). U-NIAH, a unified Needle-in-a-Haystack (NIAH) framework, systematically evaluates LLMs and RAG methods in controlled long-context settings. It extends beyond traditional NIAH by incorporating more practical and complex scenarios like multi-needle, long-needle, and needle-in-needle configurations and leveraging the synthetic dataset to mitigate LLM biases. The experiments aim to address three research questions in long-context scenarios: (1) performance tradeoffs between LLMs and RAG, (2) error patterns in RAG, and (3) RAG\u2019s limitations in complex settings. Results show that smaller LLMs benefit more from RAG. In all settings, RAG achieves a win rate of 82.58% over direct answers. Additionally, it is found that retrieval noise and chunk ordering degrade RAG performance, and we further summarized typical error patterns, including omissions due to noise, hallucinations under high noise critical conditions, and self-doubt behaviors, as well as how these phenomena vary with context length. Finally, in some challenging scenarios, experiments show that deep reasoning models are more easily affected by distractors. These findings highlight the complementary roles of RAG and LLMs and offer actionable insights for optimizing deployment strategies (\n                    <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"uri\" xlink:href=\"https:\/\/github.com\/Tongji-KGLLM\/U-NIAH\">https:\/\/github.com\/Tongji-KGLLM\/U-NIAH<\/jats:ext-link>\n                    ).\n                  <\/jats:p>","DOI":"10.1145\/3786609","type":"journal-article","created":{"date-parts":[[2025,12,24]],"date-time":"2025-12-24T13:57:22Z","timestamp":1766584642000},"page":"1-30","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":6,"title":["U-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-in-a-Haystack"],"prefix":"10.1145","volume":"44","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7932-2752","authenticated-orcid":false,"given":"Yunfan","family":"Gao","sequence":"first","affiliation":[{"name":"Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University, Shanghai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8575-5415","authenticated-orcid":false,"given":"Yun","family":"Xiong","sequence":"additional","affiliation":[{"name":"Shanghai Key Laboratory of Data Science, School of Computer Science, Fudan University, Shanghai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-6590-6126","authenticated-orcid":false,"given":"Wenlong","family":"Wu","sequence":"additional","affiliation":[{"name":"College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3408-9037","authenticated-orcid":false,"given":"Bohan","family":"Li","sequence":"additional","affiliation":[{"name":"College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2351-8799","authenticated-orcid":false,"given":"Yijie","family":"Zhong","sequence":"additional","affiliation":[{"name":"College of Design and Innovation, Tongji University, Shanghai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3018-3824","authenticated-orcid":false,"given":"Haofen","family":"Wang","sequence":"additional","affiliation":[{"name":"College of Design and Innovation, Tongji University, Shanghai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,2,16]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.acl-long.172"},{"key":"e_1_3_2_3_2","doi-asserted-by":"crossref","unstructured":"Yushi Bai Shangqing Tu Jiajie Zhang Hao Peng Xiaozhi Wang Xin Lv Shulin Cao Jiazheng Xu Lei Hou Yuxiao Dong et al. 2024. LongBench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. arXiv:2412.15204. Retrieved from https:\/\/arxiv.org\/abs\/2412.15204","DOI":"10.18653\/v1\/2025.acl-long.183"},{"key":"e_1_3_2_4_2","unstructured":"Victoria Basmov Yoav Goldberg and Reut Tsarfaty. 2024. LLMs\u2019 reading comprehension is affected by parametric knowledge and struggles with hypothetical statements. arXiv:2404.06283. Retrieved from https:\/\/arxiv.org\/abs\/2404.06283"},{"key":"e_1_3_2_5_2","unstructured":"Aydar Bulatov Yuri Kuratov Yermek Kapushev and Mikhail S. Burtsev. 2023. Scaling transformer to 1M tokens and beyond with RMT. arXiv:2304.11062. Retrieved from https:\/\/arxiv.org\/abs\/2304.11062"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.emnlp-main.845"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/3626772.3657834"},{"key":"e_1_3_2_8_2","first-page":"11091","volume-title":"Proceedings of the 41st International Conference on Machine Learning","author":"Ding Yiran","year":"2024","unstructured":"Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. 2024. LongRoPE: Extending LLM context window beyond 2 million tokens. In Proceedings of the 41st International Conference on Machine Learning, 11091\u201311104."},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.findings-acl.306"},{"key":"e_1_3_2_10_2","unstructured":"Yunfan Gao Yun Xiong Xinyu Gao Kangxiang Jia Jinliu Pan Yuxi Bi Yi Dai Jiawei Sun and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv:2312.10997. Retrieved from https:\/\/arxiv.org\/abs\/2312.10997"},{"key":"e_1_3_2_11_2","unstructured":"Yunfan Gao Yun Xiong Meng Wang and Haofen Wang. 2024. Modular RAG: Transforming RAG systems into LEGO-like reconfigurable frameworks. arXiv:2407.21059. Retrieved from https:\/\/arxiv.org\/abs\/2407.21059"},{"key":"e_1_3_2_12_2","unstructured":"Michael G\u00fcnther Isabelle Mohr Daniel James Williams Bo Wang and Han Xiao. 2024. Late chunking: Contextual chunk embeddings using long-context embedding models. arXiv:2409.04701. Retrieved from https:\/\/arxiv.org\/abs\/2409.04701"},{"key":"e_1_3_2_13_2","first-page":"1270","article-title":"KVQuant: Towards 10 million context length LLM inference with KV cache quantization","volume":"37","author":"Hooper Coleman","year":"2024","unstructured":"Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun S. Shao, Kurt Keutzer, and Amir Gholami. 2024. KVQuant: Towards 10 million context length LLM inference with KV cache quantization. In Advances in Neural Information Processing Systems, Vol. 37, 1270\u20131303.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_14_2","unstructured":"Cheng-Ping Hsieh Simeng Sun Samuel Kriman Shantanu Acharya Dima Rekesh Fei Jia Yang Zhang and Boris Ginsburg. 2024. RULER: What\u2019s the real context size of your long-context language models? arXiv:2404.06654. Retrieved from https:\/\/arxiv.org\/abs\/2404.06654"},{"key":"e_1_3_2_15_2","unstructured":"Yunpeng Huang Jingwei Xu Junyu Lai Zixu Jiang Taolue Chen Zenan Li Yuan Yao Xiaoxing Ma Lijuan Yang Hao Chen et al. 2023. Advancing transformer architecture in long-context large language models: A comprehensive survey. arXiv:2311.12351. Retrieved from https:\/\/arxiv.org\/abs\/2311.12351"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/3701716.3716889"},{"key":"e_1_3_2_17_2","first-page":"106519","article-title":"BABILong: Testing the limits of LLMs with long context reasoning-in-a-haystack","volume":"37","author":"Kuratov Yury","year":"2024","unstructured":"Yury Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. 2024. BABILong: Testing the limits of LLMs with long context reasoning-in-a-haystack. In Advances in Neural Information Processing Systems, Vol. 37, 106519\u2013106554.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_18_2","doi-asserted-by":"crossref","unstructured":"Philippe Laban Alexander R. Fabbri Caiming Xiong and Chien-Sheng Wu. 2024. Summary of a haystack: A challenge to long-context LLMs and RAG systems. arXiv:2407.01370. Retrieved from https:\/\/arxiv.org\/abs\/2407.01370","DOI":"10.18653\/v1\/2024.emnlp-main.552"},{"key":"e_1_3_2_19_2","unstructured":"Mo Li Songyang Zhang Yunxin Liu and Kai Chen. 2024. NeedleBench: Can LLMs do retrieval and reasoning in 1 million context window? arXiv:2407.11963. Retrieved from https:\/\/arxiv.org\/abs\/2407.11963"},{"key":"e_1_3_2_20_2","unstructured":"Xinze Li Yixin Cao Yubo Ma and Aixin Sun. 2024. Long context vs. RAG for LLMs: An evaluation and revisits. arXiv:2501.01880. Retrieved from https:\/\/arxiv.org\/abs\/2501.01880"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.emnlp-industry.66"},{"key":"e_1_3_2_22_2","unstructured":"Aixin Liu Bei Feng Bing Xue Bingxuan Wang Bochao Wu Chengda Lu Chenggang Zhao Chengqi Deng Chenyu Zhang Chong Ruan et al. 2024. Deepseek-v3 technical report. arXiv:2412.19437. Retrieved from https:\/\/arxiv.org\/abs\/2412.19437"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00638"},{"key":"e_1_3_2_24_2","unstructured":"Ali Modarressi Hanieh Deilamsalehy Franck Dernoncourt Trung Bui Ryan A. Rossi Seunghyun Yoon and Hinrich Sch\u00fctze. 2025. NoLiMa: Long-context evaluation beyond literal matching. arXiv:2502.05167. Retrieved from https:\/\/arxiv.org\/abs\/2502.05167"},{"key":"e_1_3_2_25_2","unstructured":"Tsendsuren Munkhdalai Manaal Faruqui and Siddharth Gopal. 2024. Leave no context behind: Efficient infinite context transformers with infini-attention. arXiv:2404.07143 101. Retrieved from https:\/\/arxiv.org\/abs\/2404.07143"},{"key":"e_1_3_2_26_2","unstructured":"David Rein Betty Li Hou Asa Cooper Stickland Jackson Petty Richard Yuanzhe Pang Julien Dirani Julian Michael and Samuel R. Bowman. 2023. GPQA: A graduate-level google-proof Q&A benchmark. arXiv:2311.12022. Retrieved from https:\/\/arxiv.org\/abs\/2311.12022"},{"key":"e_1_3_2_27_2","unstructured":"Jon Saad-Falcon Daniel Y. Fu Simran Arora Neel Guha and Christopher R\u00e9. 2024. Benchmarking and building long-context retrieval models with loco and m2-bert. arXiv:2402.07440. Retrieved from https:\/\/arxiv.org\/abs\/2402.07440"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.emnlp-main.322"},{"key":"e_1_3_2_29_2","first-page":"20540","article-title":"Needle in a multimodal haystack","volume":"37","author":"Wang Weiyun","year":"2024","unstructured":"Weiyun Wang, Shuibo Zhang, Yiming Ren, Yuchen Duan, Tiantong Li, Shuo Liu, Mengkang Hu, Zhe Chen, Kaipeng Zhang, Lewei Lu, et al. 2024. Needle in a multimodal haystack. In Advances in Neural Information Processing Systems, Vol. 37, 20540\u201320565.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2024\/917SurveyTrack"},{"key":"e_1_3_2_31_2","unstructured":"Jules White Quchen Fu Sam Hays Michael Sandborn Carlos Olea Henry Gilbert Ashraf Elnashar Jesse Spencer-Smith and Douglas C. Schmidt. 2023. A prompt pattern catalog to enhance prompt engineering with ChatGPT. arXiv:2302.11382. Retrieved from https:\/\/arxiv.org\/abs\/2302.11382"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.acl-long.79"},{"key":"e_1_3_2_33_2","unstructured":"Chong Xiang Tong Wu Zexuan Zhong David Wagner Danqi Chen and Prateek Mittal. 2024. Certifiably robust RAG against retrieval corruption. arXiv:2405.15556. Retrieved from https:\/\/arxiv.org\/abs\/2405.15556"},{"key":"e_1_3_2_34_2","unstructured":"Peng Xu Wei Ping Xianchao Wu Lawrence McAfee Chen Zhu Zihan Liu Sandeep Subramanian Evelina Bakhturina Mohammad Shoeybi and Bryan Catanzaro. 2023. Retrieval meets long context large language models. arXiv:2310.03025. Retrieved from https:\/\/arxiv.org\/abs\/2310.03025"},{"key":"e_1_3_2_35_2","unstructured":"An Yang Baosong Yang Beichen Zhang Binyuan Hui Bo Zheng Bowen Yu Chengyuan Li Dayiheng Liu Fei Huang Haoran Wei et al. 2024. Qwen2.5 technical report. arXiv:2412.15115. Retrieved from https:\/\/arxiv.org\/abs\/2412.15115"},{"key":"e_1_3_2_36_2","unstructured":"Tan Yu Anbang Xu and Rama Akkiraju. 2024. In defense of RAG in the era of long-context language models. arXiv:2409.01666. Retrieved from https:\/\/arxiv.org\/abs\/2409.01666"},{"key":"e_1_3_2_37_2","first-page":"121156","article-title":"RankRAG: Unifying context ranking with retrieval-augmented generation in LLMs","volume":"37","author":"Yu Yue","year":"2024","unstructured":"Yue Yu, Wei Ping, Zihan Liu, Boxin Wang, Jiaxuan You, Chao Zhang, Mohammad Shoeybi, and Bryan Catanzaro. 2024. RankRAG: Unifying context ranking with retrieval-augmented generation in LLMs. In Advances in Neural Information Processing Systems, Vol. 37, 121156\u2013121184.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE65448.2025.00108"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.acl-long.814"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.emnlp-main.1259"}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3786609","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,3,12]],"date-time":"2026-03-12T12:16:24Z","timestamp":1773317784000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3786609"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,2,16]]},"references-count":39,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2026,3,31]]}},"alternative-id":["10.1145\/3786609"],"URL":"https:\/\/doi.org\/10.1145\/3786609","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"value":"1046-8188","type":"print"},{"value":"1558-2868","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,2,16]]},"assertion":[{"value":"2025-06-27","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-11-03","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-02-16","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}