{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,7]],"date-time":"2026-04-07T21:01:02Z","timestamp":1775595662533,"version":"3.50.1"},"reference-count":72,"publisher":"Association for Computing Machinery (ACM)","issue":"1","funder":[{"name":"Development Program of China","award":["2024YFB4505600"],"award-info":[{"award-number":["2024YFB4505600"]}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["92467102"],"award-info":[{"award-number":["92467102"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Tsinghua University Initiative Scientific Research Program, Young Elite Scientists Sponsorship Program by CAST","award":["2022QNRC001"],"award-info":[{"award-number":["2022QNRC001"]}]},{"name":"Beijing Natural Science Foundation","award":["L252014"],"award-info":[{"award-number":["L252014"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Manag. Data"],"published-print":{"date-parts":[[2026,4,2]]},"abstract":"<jats:p>Retrieval-Augmented Generation enhances Large Language Models by integrating external knowledge, which reduces hallucinations but increases prompt length. This increase leads to higher computational costs and longer Time to First Token. To mitigate this issue, existing solutions aim to reuse the preprocessed KVCache of each retrieved chunk to accelerate RAG. However, the lack of cross-chunk contextual information leads to a significant drop in generation quality, leaving the potential benefits of KVCache reuse largely unfulfilled.<\/jats:p>\n                  <jats:p>The challenge lies in how to reuse the precomputed KVCache chunk while preserving generation quality. We propose FusionRAG, a novel inference framework that optimizes both the preprocessing and reprocessing stages of RAG. In the offline preprocessing stage, we embed information from other related text chunks into each chunk, while in the online reprocessing stage, we recompute the KVCache for tokens that the model focuses on. As a result, we achieve a better trade-off between generation quality and efficiency. According to our experiments, FusionRAG significantly improves generation quality at the same recomputation ratio compared to previous state-of-the-art solutions. By recomputing fewer than 15% of the tokens, FusionRAG achieves up to 70% higher normalized-F1 scores than baselines and reduces TTFT by 2.66-9.39\u00d7 compared to Full Attention.<\/jats:p>","DOI":"10.1145\/3786655","type":"journal-article","created":{"date-parts":[[2026,4,7]],"date-time":"2026-04-07T17:54:13Z","timestamp":1775584453000},"page":"1-28","source":"Crossref","is-referenced-by-count":0,"title":["From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generation"],"prefix":"10.1145","volume":"4","author":[{"ORCID":"https:\/\/orcid.org\/0009-0001-8030-5976","authenticated-orcid":false,"given":"Jiahao","family":"Wang","sequence":"first","affiliation":[{"name":"Hangzhou Dianzi University, Hangzhou, China and Approaching.AI, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0173-1027","authenticated-orcid":false,"given":"Weiyu","family":"Xie","sequence":"additional","affiliation":[{"name":"Tsinghua University, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7518-0753","authenticated-orcid":false,"given":"Mingxing","family":"Zhang","sequence":"additional","affiliation":[{"name":"Tsinghua University, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-2857-3713","authenticated-orcid":false,"given":"Boxin","family":"Zhang","sequence":"additional","affiliation":[{"name":"Tsinghua University, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-3175-1607","authenticated-orcid":false,"given":"Jianwei","family":"Dong","sequence":"additional","affiliation":[{"name":"Tsinghua University, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-6716-5709","authenticated-orcid":false,"given":"Yuening","family":"Zhu","sequence":"additional","affiliation":[{"name":"Tsinghua University, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-8554-0411","authenticated-orcid":false,"given":"Chen","family":"Lin","sequence":"additional","affiliation":[{"name":"Tsinghua University, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-3300-9425","authenticated-orcid":false,"given":"Jingqi","family":"Tang","sequence":"additional","affiliation":[{"name":"Approaching.AI, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-7744-4707","authenticated-orcid":false,"given":"Yaochen","family":"Han","sequence":"additional","affiliation":[{"name":"Approaching.AI, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-8050-6288","authenticated-orcid":false,"given":"Zhiyuan","family":"Ai","sequence":"additional","affiliation":[{"name":"Approaching.AI, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-4975-4856","authenticated-orcid":false,"given":"Xianglin","family":"Chen","sequence":"additional","affiliation":[{"name":"Approaching.AI, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6651-7032","authenticated-orcid":false,"given":"Yongwei","family":"Wu","sequence":"additional","affiliation":[{"name":"Tsinghua University, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3592-0328","authenticated-orcid":false,"given":"Congfeng","family":"Jiang","sequence":"additional","affiliation":[{"name":"Hangzhou Dianzi University, Hangzhou, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2026,4,7]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/3725273"},{"key":"e_1_2_1_2_1","volume-title":"Multitask Benchmark for Long Context Understanding. ArXiv","author":"Bai Yushi","year":"2023","unstructured":"Yushi Bai, Xin Lv, Jiajie Zhang, Hong Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2023. LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding. ArXiv, Vol. abs\/2308.14508 (2023). https:\/\/api.semanticscholar.org\/CorpusID:261245264"},{"key":"e_1_2_1_3_1","unstructured":"Tom Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared D Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell et al. 2020. Language models are few-shot learners. Advances in neural information processing systems Vol. 33 (2020) 1877-1901."},{"key":"e_1_2_1_4_1","volume-title":"PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling. ArXiv","author":"Cai Zefan","year":"2069","unstructured":"Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, and Wen Xiao. 2024. PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling. ArXiv, Vol. abs\/2406.02069 (2024). https:\/\/api.semanticscholar.org\/CorpusID:270226243"},{"key":"e_1_2_1_5_1","volume-title":"Real-world Use Cases for Large Language Models (LLMs). Retrieved","year":"2025","unstructured":"CellStrat. 2023. Real-world Use Cases for Large Language Models (LLMs). Retrieved March 2, 2025 from https:\/\/cellstrat.medium.com\/real-world-use-cases-for-large-language-models-llms-d71c3a577bf2"},{"key":"e_1_2_1_6_1","volume-title":"Pangu Embedded: An Efficient Dual-system LLM Reasoner with Metacognition. ArXiv","author":"Chen Hanting","year":"2025","unstructured":"Hanting Chen, Yasheng Wang, Kai Han, Dong Li, Lin Li, Zhenni Bi, Jinpeng Li, Haoyu Wang, Fei Mi, Mingjian Zhu, Bin Wang, Kaikai Song, Yifei Fu, Xu He, Yu-Mei Luo, Chong Zhu, Quan He, Xue Wu, Wei He, Hailin Hu, Yehui Tang, Dacheng Tao, Xinghao Chen, and Yunhe Wang. 2025. Pangu Embedded: An Efficient Dual-system LLM Reasoner with Metacognition. ArXiv, Vol. abs\/2505.22375 (2025). https:\/\/api.semanticscholar.org\/CorpusID:278959233"},{"key":"e_1_2_1_7_1","volume-title":"Multi-Granularity Text Embeddings Through Self-Knowledge Distillation. In Annual Meeting of the Association for Computational Linguistics. https:\/\/api.semanticscholar.org\/CorpusID:267413218","author":"Chen Jianlv","year":"2024","unstructured":"Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation. In Annual Meeting of the Association for Computational Linguistics. https:\/\/api.semanticscholar.org\/CorpusID:267413218"},{"key":"e_1_2_1_8_1","first-page":"1","article-title":"Palm: Scaling language modeling with pathways","volume":"24","author":"Chowdhery Aakanksha","year":"2023","unstructured":"Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al., 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, Vol. 24, 240 (2023), 1-113.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_2_1_9_1","volume-title":"7 Top Large Language Model Use Cases and Applications. Retrieved","year":"2025","unstructured":"Daivi. 2024. 7 Top Large Language Model Use Cases and Applications. Retrieved March 2, 2025 from https:\/\/www.projectpro.io\/article\/large-language-model-use-cases-and-applications\/887"},{"key":"e_1_2_1_10_1","volume-title":"Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in neural information processing systems","author":"Dao Tri","year":"2022","unstructured":"Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher R\u00e9. 2022. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in neural information processing systems, Vol. 35 (2022), 16344-16359."},{"key":"e_1_2_1_11_1","volume-title":"Flex Attention: A Programming Model for Generating Optimized Attention Kernels. ArXiv","author":"Dong Juechu","year":"2024","unstructured":"Juechu Dong, Boyuan Feng, Driss Guessous, Yanbo Liang, and Horace He. 2024. Flex Attention: A Programming Model for Generating Optimized Attention Kernels. ArXiv, Vol. abs\/2412.05496 (2024). https:\/\/api.semanticscholar.org\/CorpusID:274598006"},{"key":"e_1_2_1_12_1","volume-title":"From Local to Global: A Graph RAG Approach to Query-Focused Summarization. ArXiv","author":"Edge Darren","year":"2024","unstructured":"Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. 2024. From Local to Global: A Graph RAG Approach to Query-Focused Summarization. ArXiv, Vol. abs\/2404.16130 (2024). https:\/\/api.semanticscholar.org\/CorpusID:269363075"},{"key":"e_1_2_1_13_1","volume-title":"Ragas: Supercharge Your LLM Application Evaluations. https:\/\/github.com\/explodinggradients\/ragas.","year":"2024","unstructured":"ExplodingGradients. 2024. Ragas: Supercharge Your LLM Application Evaluations. https:\/\/github.com\/explodinggradients\/ragas."},{"key":"e_1_2_1_14_1","volume-title":"Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference. ArXiv","author":"Feng Yuan","year":"2024","unstructured":"Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and S. Kevin Zhou. 2024. Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference. ArXiv, Vol. abs\/2407.11550 (2024). https:\/\/api.semanticscholar.org\/CorpusID:271218006"},{"key":"e_1_2_1_15_1","volume-title":"Identify Critical KV Cache in LLM Inference from an Output Perturbation Perspective. ArXiv","author":"Feng Yuan","year":"2025","unstructured":"Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and S. Kevin Zhou. 2025. Identify Critical KV Cache in LLM Inference from an Output Perturbation Perspective. ArXiv, Vol. abs\/2502.03805 (2025). https:\/\/api.semanticscholar.org\/CorpusID:276161406"},{"key":"e_1_2_1_16_1","volume-title":"Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning. ArXiv","author":"Fu Yu","year":"1925","unstructured":"Yu Fu, Zefan Cai, Abedelkadir Asi, Wayne Xiong, Yue Dong, and Wen Xiao. 2024. Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning. ArXiv, Vol. abs\/2410.19258 (2024). https:\/\/api.semanticscholar.org\/CorpusID:273638229"},{"key":"e_1_2_1_17_1","volume-title":"Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention. In USENIX Annual Technical Conference. https:\/\/api.semanticscholar.org\/CorpusID:268793498","author":"Gao Bin","year":"2024","unstructured":"Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, and Pengfei Zuo. 2024. Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention. In USENIX Annual Technical Conference. https:\/\/api.semanticscholar.org\/CorpusID:268793498"},{"key":"e_1_2_1_18_1","volume-title":"Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997","author":"Gao Yunfan","year":"2023","unstructured":"Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, Vol. 2 (2023)."},{"key":"e_1_2_1_19_1","first-page":"325","article-title":"Prompt cache: Modular attention reuse for low-latency inference","volume":"6","author":"Gim In","year":"2024","unstructured":"In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong. 2024. Prompt cache: Modular attention reuse for low-latency inference. Proceedings of Machine Learning and Systems, Vol. 6 (2024), 325-338.","journal-title":"Proceedings of Machine Learning and Systems"},{"key":"e_1_2_1_20_1","unstructured":"Team GLM Aohan Zeng Bin Xu Bowen Wang Chenhui Zhang Da Yin Diego Rojas Guanyu Feng Hanlin Zhao Hanyu Lai Hao Yu Hongning Wang Jiadai Sun Jiajie Zhang Jiale Cheng Jiayi Gui Jie Tang Jing Zhang Juanzi Li Lei Zhao Lindong Wu Lucen Zhong Mingdao Liu Minlie Huang Peng Zhang Qinkai Zheng Rui Lu Shuaiqi Duan Shudan Zhang Shulin Cao Shuxun Yang Weng Lam Tam Wenyi Zhao Xiao Liu Xiao Xia Xiaohan Zhang Xiaotao Gu Xin Lv Xinghan Liu Xinyi Liu Xinyue Yang Xixuan Song Xunkai Zhang Yifan An Yifan Xu Yilin Niu Yuantao Yang Yueyan Li Yushi Bai Yuxiao Dong Zehan Qi Zhaoyu Wang Zhen Yang Zhengxiao Du Zhenyu Hou and Zihan Wang. 2024. ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools. arXiv:2406.12793"},{"key":"e_1_2_1_21_1","volume-title":"LightRAG: Simple and Fast Retrieval-Augmented Generation. ArXiv","author":"Guo Zirui","year":"2024","unstructured":"Zirui Guo, Lianghao Xia, Yanhua Yu, Tu Ao, and Chao Huang. 2024. LightRAG: Simple and Fast Retrieval-Augmented Generation. ArXiv, Vol. abs\/2410.05779 (2024). https:\/\/api.semanticscholar.org\/CorpusID:273227829"},{"key":"e_1_2_1_22_1","volume-title":"Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps. ArXiv","author":"Ho Xanh","year":"2020","unstructured":"Xanh Ho, A. Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps. ArXiv, Vol. abs\/2011.01060 (2020). https:\/\/api.semanticscholar.org\/CorpusID:226236740"},{"key":"e_1_2_1_23_1","volume-title":"GRAG: Graph Retrieval-Augmented Generation. ArXiv","author":"Hu Yuntong","year":"2024","unstructured":"Yuntong Hu, Zhihan Lei, Zhengwu Zhang, Bo Pan, Chen Ling, and Liang Zhao. 2024. GRAG: Graph Retrieval-Augmented Generation. ArXiv, Vol. abs\/2405.16506 (2024). https:\/\/api.semanticscholar.org\/CorpusID:270062608"},{"key":"e_1_2_1_24_1","volume-title":"Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering. ArXiv","author":"Izacard Gautier","year":"2020","unstructured":"Gautier Izacard and Edouard Grave. 2020. Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering. ArXiv, Vol. abs\/2007.01282 (2020). https:\/\/api.semanticscholar.org\/CorpusID:220302360"},{"key":"e_1_2_1_25_1","unstructured":"Albert Qiaochu Jiang Alexandre Sablayrolles Arthur Mensch Chris Bamford Devendra Singh Chaplot Diego de Las Casas Florian Bressand Gianna Lengyel Guillaume Lample Lucile Saulnier L'elio Renard Lavaud Marie-Anne Lachaux Pierre Stock Teven Le Scao Thibaut Lavril Thomas Wang Timoth\u00e9e Lacroix and William El Sayed. 2023. Mistral 7B. ArXiv Vol. abs\/2310.06825 (2023). https:\/\/api.semanticscholar.org\/CorpusID:263830494"},{"key":"e_1_2_1_26_1","first-page":"52481","volume-title":"Advances in Neural Information Processing Systems","volume":"37","author":"Jiang Huiqiang","year":"2024","unstructured":"Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir Abdi, Dongsheng Li, Chin-Yew Lin, et al., 2024. Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention. Advances in Neural Information Processing Systems, Vol. 37 (2024), 52481-52515."},{"key":"e_1_2_1_27_1","volume-title":"RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation. ArXiv","author":"Jin Chao","year":"2024","unstructured":"Chao Jin, Zili Zhang, Xuanlin Jiang, Fangyue Liu, Xin Liu, Xuanzhe Liu, and Xin Jin. 2024. RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation. ArXiv, Vol. abs\/2404.12457 (2024). https:\/\/api.semanticscholar.org\/CorpusID:269283058"},{"key":"e_1_2_1_28_1","volume-title":"TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. ArXiv","author":"Joshi Mandar","year":"2017","unstructured":"Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. ArXiv, Vol. abs\/1705.03551 (2017). https:\/\/api.semanticscholar.org\/CorpusID:26501419"},{"key":"e_1_2_1_29_1","volume-title":"How to Stop Off-the-Shelf Deep Neural Networks from Overthinking. ArXiv","author":"Kaya Yigitcan","year":"2018","unstructured":"Yigitcan Kaya and Tudor Dumitras. 2018. How to Stop Off-the-Shelf Deep Neural Networks from Overthinking. ArXiv, Vol. abs\/1810.07052 (2018). https:\/\/api.semanticscholar.org\/CorpusID:53113950"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3600006.3613165"},{"key":"e_1_2_1_31_1","unstructured":"Benjamin Lefaudeux Francisco Massa Diana Liskovich Wenhan Xiong Vittorio Caggiano Sean Naren Min Xu Jieru Hu Marta Tintore Susan Zhang Patrick Labatut Daniel Haziza Luca Wehrstedt Jeremy Reizenstein and Grigory Sizov. 2022. xFormers: A modular and hackable Transformer modelling library. https:\/\/github.com\/facebookresearch\/xformers."},{"key":"e_1_2_1_32_1","unstructured":"Patrick Lewis Ethan Perez Aleksandra Piktus Fabio Petroni Vladimir Karpukhin Naman Goyal Heinrich K\u00fcttler Mike Lewis Wen-tau Yih Tim Rockt\u00e4schel et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems Vol. 33 (2020) 9459-9474."},{"key":"e_1_2_1_33_1","first-page":"22947","article-title":"Snapkv: Llm knows what you are looking for before generation","volume":"37","author":"Li Yuhong","year":"2024","unstructured":"Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024. Snapkv: Llm knows what you are looking for before generation. Advances in Neural Information Processing Systems, Vol. 37 (2024), 22947-22970.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_1_34_1","unstructured":"Chien-Yu Lin Keisuke Kamahori Yiyu Liu Xiaoxiang Shi Madhav Kashyap Yile Gu Rulin Shao Zihao Ye Kan Zhu Stephanie Wang et al. 2025. TeleRAG: Efficient Retrieval-Augmented Generation Inference with Lookahead Retrieval. arXiv preprint arXiv:2502.20969 (2025)."},{"key":"e_1_2_1_35_1","volume-title":"MiniCache: KV Cache Compression in Depth Dimension for Large Language Models. ArXiv","author":"Liu Akide","year":"2024","unstructured":"Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Gholamreza Haffari, and Bohan Zhuang. 2024. MiniCache: KV Cache Compression in Depth Dimension for Large Language Models. ArXiv, Vol. abs\/2405.14366 (2024). https:\/\/api.semanticscholar.org\/CorpusID:269982665"},{"key":"e_1_2_1_36_1","first-page":"52342","article-title":"Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time","volume":"36","author":"Liu Zichang","year":"2023","unstructured":"Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. 2023. Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time. Advances in Neural Information Processing Systems, Vol. 36 (2023), 52342-52364.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_1_37_1","unstructured":"Songshuo Lu Hua Wang Yutian Rong Zhi Chen and Yaohua Tang. 2024. TurboRAG: Accelerating Retrieval-Augmented Generation with Precomputed KV Caches for Chunked Text. https:\/\/api.semanticscholar.org\/CorpusID:273233795"},{"key":"e_1_2_1_38_1","volume-title":"Block-Attention for Efficient Prefilling. In International Conference on Learning Representations. https:\/\/api.semanticscholar.org\/CorpusID:272832445","author":"Ma Dongyang","year":"2024","unstructured":"Dongyang Ma, Yan Wang, and Tian Lan. 2024. Block-Attention for Efficient Prefilling. In International Conference on Learning Representations. https:\/\/api.semanticscholar.org\/CorpusID:272832445"},{"key":"e_1_2_1_39_1","unstructured":"Piotr Nawrot Adrian La'ncucki Marcin Chochowski David Tarjan and E. Ponti. 2024. Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference. ArXiv Vol. abs\/2403.09636 (2024). https:\/\/api.semanticscholar.org\/CorpusID:268384862"},{"key":"e_1_2_1_40_1","volume-title":"EE-Tuning: An Economical yet Scalable Solution for Tuning Early-Exit Large Language Models. ArXiv","author":"Pan Xuchen","year":"2024","unstructured":"Xuchen Pan, Yanxi Chen, Yaliang Li, Bolin Ding, and Jingren Zhou. 2024. EE-Tuning: An Economical yet Scalable Solution for Tuning Early-Exit Large Language Models. ArXiv, Vol. abs\/2402.00518 (2024). https:\/\/api.semanticscholar.org\/CorpusID:267365444"},{"key":"e_1_2_1_41_1","volume-title":"PyTorch: An Imperative Style","author":"Paszke Adam","year":"2019","unstructured":"Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas K\u00f6pf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. ArXiv, Vol. abs\/1912.01703 (2019). https:\/\/api.semanticscholar.org\/CorpusID:202786778"},{"key":"e_1_2_1_42_1","first-page":"155","volume-title":"23rd USENIX Conference on File and Storage Technologies (FAST 25)","author":"Qin Ruoyu","year":"2025","unstructured":"Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2025. Mooncake: Trading More Storage for Less Computation\u2014A Architecture for Serving Chatbot. In 23rd USENIX Conference on File and Storage Technologies (FAST 25). 155-170."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D16-1264"},{"key":"e_1_2_1_44_1","volume-title":"Parallel Context Windows for Large Language Models. In Annual Meeting of the Association for Computational Linguistics. https:\/\/api.semanticscholar.org\/CorpusID:258686160","author":"Ratner Nir","year":"2022","unstructured":"Nir Ratner, Yoav Levine, Yonatan Belinkov, Ori Ram, Inbal Magar, Omri Abend, Ehud D. Karpas, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2022. Parallel Context Windows for Large Language Models. In Annual Meeting of the Association for Computational Linguistics. https:\/\/api.semanticscholar.org\/CorpusID:258686160"},{"key":"e_1_2_1_45_1","volume-title":"RoFormer: Enhanced Transformer with Rotary Position Embedding. ArXiv","author":"Su Jianlin","year":"2021","unstructured":"Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. 2021. RoFormer: Enhanced Transformer with Rotary Position Embedding. ArXiv, Vol. abs\/2104.09864 (2021). https:\/\/api.semanticscholar.org\/CorpusID:233307138"},{"key":"e_1_2_1_46_1","volume-title":"RazorAttention: Efficient KV Cache Compression Through Retrieval Heads. ArXiv","author":"Tang Hanlin","year":"2024","unstructured":"Hanlin Tang, Yang Lin, Jing Lin, Qingsen Han, Shikuan Hong, Yiwu Yao, and Gongyi Wang. 2024. RazorAttention: Efficient KV Cache Compression Through Retrieval Heads. ArXiv, Vol. abs\/2407.15891 (2024). https:\/\/api.semanticscholar.org\/CorpusID:271334610"},{"key":"e_1_2_1_47_1","unstructured":"Techopedia. 2023. 12 Practical Large Language Model (LLM) Applications. https:\/\/www.techopedia.com\/12-practical-large-language-model-llm-applications."},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/3315508.3329973"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00475"},{"key":"e_1_2_1_50_1","volume-title":"Attention is all you need. Advances in neural information processing systems","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, Vol. 30 (2017)."},{"key":"e_1_2_1_51_1","unstructured":"Zhongwei Wan Xinjian Wu Yu Zhang Yi Xin Chaofan Tao Zhihong Zhu Xin Wang Siqi Luo Jing Xiong Longyue Wang and Mi Zhang. 2024b. D2O: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Models. https:\/\/api.semanticscholar.org\/CorpusID:276961235"},{"key":"e_1_2_1_52_1","volume-title":"LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference. arXiv preprint arXiv:2406.18139","author":"Wan Zhongwei","year":"2024","unstructured":"Zhongwei Wan, Ziang Wu, Che Liu, Jinfa Huang, Zhihong Zhu, Peng Jin, Longyue Wang, and Li Yuan. 2024a. LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference. arXiv preprint arXiv:2406.18139 (2024)."},{"key":"e_1_2_1_53_1","unstructured":"Yumeng Wang and Zhenyang Xiao. 2024. LoMA: Lossless Compressed Memory Attention. ArXiv Vol. abs\/2401.09486 (2024). https:\/\/api.semanticscholar.org\/CorpusID:267035186"},{"key":"e_1_2_1_54_1","volume-title":"Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Tasks. ArXiv","author":"Wang Zheng","year":"2024","unstructured":"Zheng Wang, Boxiao Jin, Zhongzhi Yu, and Minjia Zhang. 2024. Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Tasks. ArXiv, Vol. abs\/2407.08454 (2024). https:\/\/api.semanticscholar.org\/CorpusID:271097687"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-demos.6"},{"key":"e_1_2_1_56_1","volume-title":"DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads. ArXiv","author":"Xiao Guangxuan","year":"2024","unstructured":"Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. 2024. DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads. ArXiv, Vol. abs\/2410.10819 (2024). https:\/\/api.semanticscholar.org\/CorpusID:273345166"},{"key":"e_1_2_1_57_1","volume-title":"Efficient Streaming Language Models with Attention Sinks. ArXiv","author":"Xiao Guangxuan","year":"2023","unstructured":"Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2023. Efficient Streaming Language Models with Attention Sinks. ArXiv, Vol. abs\/2309.17453 (2023). https:\/\/api.semanticscholar.org\/CorpusID:263310483"},{"key":"e_1_2_1_58_1","volume-title":"arXiv preprint arXiv:2412.15115","author":"Yang An","year":"2024","unstructured":"An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. 2024c. Qwen2.5 Technical Report. arXiv preprint arXiv:2412.15115 (2024)."},{"key":"e_1_2_1_59_1","volume-title":"PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference. ArXiv","author":"Yang Dongjie","year":"2024","unstructured":"Dongjie Yang, Xiaodong Han, Yan Gao, Yao Hu, Shilin Zhang, and Hai Zhao. 2024b. PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference. ArXiv, Vol. abs\/2405.12532 (2024). https:\/\/api.semanticscholar.org\/CorpusID:269930254"},{"key":"e_1_2_1_60_1","volume-title":"KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse. ArXiv","author":"Yang Jingbo","year":"2025","unstructured":"Jingbo Yang, Bairu Hou, Wei Wei, Yujia Bao, and Shiyu Chang. 2025. KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse. ArXiv, Vol. abs\/2502.16002 (2025). https:\/\/api.semanticscholar.org\/CorpusID:276576000"},{"key":"e_1_2_1_61_1","volume-title":"KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing. ArXiv","author":"Yang Yifei","year":"1851","unstructured":"Yifei Yang, Zouying Cao, Qiguang Chen, Libo Qin, Dongjie Yang, Hai Zhao, and Zhi Chen. 2024a. KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing. ArXiv, Vol. abs\/2410.18517 (2024). https:\/\/api.semanticscholar.org\/CorpusID:273549721"},{"key":"e_1_2_1_62_1","volume-title":"Explainable Multi-hop Question Answering. In Conference on Empirical Methods in Natural Language Processing. https:\/\/api.semanticscholar.org\/CorpusID:52822214","author":"Yang Zhilin","unstructured":"Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. In Conference on Empirical Methods in Natural Language Processing. https:\/\/api.semanticscholar.org\/CorpusID:52822214"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/3689031.3696098"},{"key":"e_1_2_1_64_1","volume-title":"Infinite Retrieval: Attention Enhanced LLMs in Long-Context Processing. ArXiv","author":"Ye Xiaoju","year":"2025","unstructured":"Xiaoju Ye, Zhichun Wang, and Jingyuan Wang. 2025. Infinite Retrieval: Attention Enhanced LLMs in Long-Context Processing. ArXiv, Vol. abs\/2502.12962 (2025). https:\/\/api.semanticscholar.org\/CorpusID:276422377"},{"key":"e_1_2_1_65_1","first-page":"521","volume-title":"16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22)","author":"Yu Gyeong-In","year":"2022","unstructured":"Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A distributed serving system for generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). 521-538."},{"key":"e_1_2_1_66_1","volume-title":"Inference Scaling for Long-Context Retrieval Augmented Generation. ArXiv","author":"Yue Zhenrui","year":"2024","unstructured":"Zhenrui Yue, Honglei Zhuang, Aijun Bai, Kai Hui, Rolf Jagerman, Hansi Zeng, Zhen Qin, Dong Wang, Xuanhui Wang, and Michael Bendersky. 2024. Inference Scaling for Long-Context Retrieval Augmented Generation. ArXiv, Vol. abs\/2410.04343 (2024). https:\/\/api.semanticscholar.org\/CorpusID:273185794"},{"key":"e_1_2_1_67_1","doi-asserted-by":"crossref","unstructured":"Xuan Zhang Cunxiao Du Chao Du Tianyu Pang Wei Gao and Min Lin. 2024a. SimLayerKV: A Simple Framework for Layer-Level KV Cache Reduction. arXiv:2410.13846 [cs.CL] https:\/\/arxiv.org\/abs\/2410.13846","DOI":"10.32388\/EPAI4T"},{"key":"e_1_2_1_68_1","volume-title":"Unifying kv cache compression for large language models with leankv. arXiv preprint arXiv:2412.03131","author":"Zhang Yanqi","year":"2024","unstructured":"Yanqi Zhang, Yuwei Hu, Runyuan Zhao, John Lui, and Haibo Chen. 2024b. Unifying kv cache compression for large language models with leankv. arXiv preprint arXiv:2412.03131 (2024)."},{"key":"e_1_2_1_69_1","volume-title":"Unifying KV Cache Compression for Large Language Models with LeanKV. ArXiv","author":"Zhang Yanqi","year":"2024","unstructured":"Yanqi Zhang, Yuwei Hu, Runyuan Zhao, John C.S. Lui, and Haibo Chen. 2024c. Unifying KV Cache Compression for Large Language Models with LeanKV. ArXiv, Vol. abs\/2412.03131 (2024). https:\/\/api.semanticscholar.org\/CorpusID:274464890"},{"key":"e_1_2_1_70_1","first-page":"34661","article-title":"H2o: Heavy-hitter oracle for efficient generative inference of large language models","volume":"36","author":"Zhang Zhenyu","year":"2023","unstructured":"Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher R\u00e9, Clark Barrett, et al., 2023. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, Vol. 36 (2023), 34661-34710.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_1_71_1","volume-title":"Best applications of large language models. Retrieved","author":"Zharovskikh Anastasiya","year":"2025","unstructured":"Anastasiya Zharovskikh. 2023. Best applications of large language models. Retrieved March 2, 2025 from https:\/\/indatalabs.com\/blog\/large-language-model-apps"},{"key":"e_1_2_1_72_1","volume-title":"DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMs. ArXiv","author":"Zhou Xiabin","year":"2024","unstructured":"Xiabin Zhou, Wenbin Wang, Minyan Zeng, Jiaxian Guo, Xuebo Liu, Li Shen, Min Zhang, and Liang Ding. 2024. DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMs. ArXiv, Vol. abs\/2412.14838 (2024). https:\/\/api.semanticscholar.org\/CorpusID:274860109"}],"container-title":["Proceedings of the ACM on Management of Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3786655","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,7]],"date-time":"2026-04-07T20:00:11Z","timestamp":1775592011000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3786655"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4,2]]},"references-count":72,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2026,4,2]]}},"alternative-id":["10.1145\/3786655"],"URL":"https:\/\/doi.org\/10.1145\/3786655","relation":{},"ISSN":["2836-6573"],"issn-type":[{"value":"2836-6573","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,4,2]]}}}