{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,11]],"date-time":"2026-07-11T15:42:18Z","timestamp":1783784538125,"version":"3.55.0"},"reference-count":71,"publisher":"Association for Computing Machinery (ACM)","issue":"1","funder":[{"name":"National Key Research and Development Program of China","award":["2022YFB4500700"],"award-info":[{"award-number":["2022YFB4500700"]}]},{"name":"Scientific Research Innovation Capability Support Project for Young Faculty","award":["ZYGXQNJSKYCXNLZCXM-I1"],"award-info":[{"award-number":["ZYGXQNJSKYCXNLZCXM-I1"]}]},{"name":"Fundamental Research Funds for the Central Universities, Peking University, and the National Natural Science Foundation of China","award":["62172008 and 62325201"],"award-info":[{"award-number":["62172008 and 62325201"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Comput. Syst."],"published-print":{"date-parts":[[2026,2,28]]},"abstract":"<jats:p>Retrieval-Augmented Generation (RAG) has demonstrated substantial advancements in various natural language processing tasks by integrating the strengths of large language models (LLMs) and external knowledge databases. However, the retrieval step introduces long sequence generation and extra data dependency, resulting in long end-to-end latency.<\/jats:p>\n                  <jats:p>\n                    Our analysis benchmarks current RAG systems and reveals that, while the retrieval step poses performance challenges, it also offers optimization opportunities through its retrieval pattern and streaming search behavior. We propose RAGCache, a latency-optimized serving system tailored for RAG. RAGCache leverages the retrieval pattern to organize and cache the intermediate states of retrieved knowledge in a\n                    <jats:italic toggle=\"yes\">knowledge tree<\/jats:italic>\n                    across the GPU and host memory hierarchy, reducing LLM generation time. RAGCache employs\n                    <jats:italic toggle=\"yes\">dynamic speculative pipelining<\/jats:italic>\n                    to exploit the streaming search behavior, overlapping retrieval with LLM generation to minimize end-to-end latency. We implement RAGCache based on vLLM and Faiss, and evaluate it on both open-source and production datasets. Experimental results demonstrate that RAGCache reduces the time to first token (TTFT) by up to 4\u00d7 and improves the throughput by up to 2.1\u00d7 compared to vLLM integrated with Faiss.\n                  <\/jats:p>","DOI":"10.1145\/3768628","type":"journal-article","created":{"date-parts":[[2025,9,20]],"date-time":"2025-09-20T11:00:23Z","timestamp":1758366023000},"page":"1-27","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":39,"title":["RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation"],"prefix":"10.1145","volume":"44","author":[{"ORCID":"https:\/\/orcid.org\/0009-0006-1355-4995","authenticated-orcid":false,"given":"Chao","family":"Jin","sequence":"first","affiliation":[{"name":"School of Computer Science, Peking University","place":["Beijing, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4209-9451","authenticated-orcid":false,"given":"Zili","family":"Zhang","sequence":"additional","affiliation":[{"name":"School of Computer Science, Peking University","place":["Beijing, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-2265-1877","authenticated-orcid":false,"given":"Xuanlin","family":"Jiang","sequence":"additional","affiliation":[{"name":"School of Electronics Engineering and Computer Science, Peking University","place":["Beijing, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-9791-046X","authenticated-orcid":false,"given":"Fangyue","family":"Liu","sequence":"additional","affiliation":[{"name":"School of Computer Science, Peking University","place":["Beijing, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-1978-4983","authenticated-orcid":false,"given":"Shufan","family":"Liu","sequence":"additional","affiliation":[{"name":"ByteDance Seed","place":["Beijing, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7908-8484","authenticated-orcid":false,"given":"Xuanzhe","family":"Liu","sequence":"additional","affiliation":[{"name":"School of Computer Science, Peking University","place":["Beijing, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8741-5847","authenticated-orcid":false,"given":"Xin","family":"Jin","sequence":"additional","affiliation":[{"name":"School of Computer Science, Peking University","place":["Beijing, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,11,7]]},"reference":[{"key":"e_1_3_2_2_2","first-page":"9459","article-title":"Retrieval-augmented generation for knowledge-intensive nlp tasks","volume":"33","author":"Lewis Patrick","year":"2020","unstructured":"Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K\u00fcttler, Mike Lewis, Wen-tau Yih, Tim Rockt\u00e4schel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems 33 (2020), 9459\u20139474.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_3_2","unstructured":"2024. LangChain. Retrieved from https:\/\/python.langchain.com\/docs\/get_started\/introduction"},{"key":"e_1_3_2_4_2","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"Brown Tom","year":"2020","unstructured":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. Advances in Neural Information Processing Systems 33 (2020), 1877\u20131901.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v38i16.29728"},{"key":"e_1_3_2_6_2","unstructured":"Lianmin Zheng Liangsheng Yin Zhiqiang Xie Jeff Huang Chuyue Sun Cody Hao Yu Shiyi Cao Christos Kozyrakis Ion Stoica Joseph E. Gonzalez Clark Barrett and Ying Sheng. 2023. Efficiently programming large language models using sglang. arXiv:2312.07104. Retrieved from https:\/\/arxiv.org\/abs\/arXiv:2312.07104"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/3600006.3613165"},{"key":"e_1_3_2_8_2","first-page":"11809","article-title":"Tree of thoughts: Deliberate problem solving with large language models","volume":"36","author":"Yao Shunyu","year":"2023","unstructured":"Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems 36 (2023), 11809\u201311822.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_9_2","article-title":"Attention is all you need","volume":"30","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Processing Systems 30 (2017).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_10_2","unstructured":"Hung-Ting Chen Fangyuan Xu Shane A. Arora and Eunsol Choi. 2023. Understanding retrieval augmentation for long-form question answering. arXiv:2310.12150. Retrieved from https:\/\/arxiv.org\/abs\/arXiv:2310.12150"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00638"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2014.2361319"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2018.2889473"},{"key":"e_1_3_2_14_2","unstructured":"2024. Pinecone: Introduction to Facebook AI Similarity Search (Faiss). (2024). Retrieved from https:\/\/www.pinecone.io\/learn\/series\/faiss\/faiss-tutorial\/"},{"key":"e_1_3_2_15_2","unstructured":"Wenqi Jiang Shuai Zhang Boran Han Jie Wang Bernie Wang and Tim Kraska. 2024. Piperag: Fast retrieval-augmented generation via algorithm-system co-design. arXiv:2403.05676. Retrieved from https:\/\/arxiv.org\/abs\/arXiv:2403.05676"},{"key":"e_1_3_2_16_2","first-page":"2206","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Borgeaud Sebastian","year":"2022","unstructured":"Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and Laurent Sifre. 2022. Improving language models by retrieving from trillions of tokens. In Proceedings of the International Conference on Machine Learning. PMLR, 2206\u20132240."},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00605"},{"key":"e_1_3_2_18_2","doi-asserted-by":"crossref","unstructured":"Harsh Trivedi Niranjan Balasubramanian Tushar Khot and Ashish Sabharwal. 2022. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. arXiv:2212.10509. Retrieved from https:\/\/arxiv.org\/abs\/arXiv:2212.10509","DOI":"10.18653\/v1\/2023.acl-long.557"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00530"},{"key":"e_1_3_2_20_2","unstructured":"Daniel Adiwardana Minh-Thang Luong David R. So Jamie Hall Noah Fiedel Romal Thoppilan Zi Yang Apoorv Kulshreshtha Gaurav Nemade Yifeng Lu and Quoc V. Le. 2020. Towards a human-like open-domain chatbot. arXiv:2001.09977. Retrieved from https:\/\/arxiv.org\/abs\/arXiv:2001.09977"},{"key":"e_1_3_2_21_2","unstructured":"Omar Khattab Keshav Santhanam Xiang Lisa Li David Hall Percy Liang Christopher Potts and Matei Zaharia. 2022. Demonstrate-search-predict: Composing retrieval and language models for knowledge-intensive nlp. arXiv:2212.14024. Retrieved from https:\/\/arxiv.org\/abs\/arXiv:2212.14024"},{"key":"e_1_3_2_22_2","unstructured":"Shuai Lu Nan Duan Hojae Han Daya Guo Seung-won Hwang and Alexey Svyatkovskiy. 2022. Reacc: A retrieval-augmented code completion framework. arXiv:2203.07722. Retrieved from https:\/\/arxiv.org\/abs\/arXiv:2203.07722"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/3491101.3519665"},{"key":"e_1_3_2_24_2","first-page":"995","volume-title":"Proceedings of the 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23)","author":"Zhang Zili","year":"2023","unstructured":"Zili Zhang, Chao Jin, Linpeng Tang, Xuanzhe Liu, and Xin Jin. 2023. Fast, approximate vector queries on very large unstructured datasets. In Proceedings of the 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). 995\u20131011."},{"key":"e_1_3_2_25_2","first-page":"23","volume-title":"Proceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24)","author":"Zhang Zili","year":"2024","unstructured":"Zili Zhang, Fangyue Liu, Gang Huang, Xuanzhe Liu, and Xin Jin. 2024. Fast vector query processing for large datasets beyond \\(\\lbrace\\) GPU \\(\\rbrace\\) memory with reordered pipelining. In Proceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24). 23\u201340."},{"key":"e_1_3_2_26_2","unstructured":"Hugo Touvron Thibaut Lavril Gautier Izacard Xavier Martinet Marie-Anne Lachaux Timoth\u00e9e Lacroix Baptiste Rozi\u00e8re Naman Goyal Eric Hambro Faisal Azhar Aurelien Rodriguez Armand Joulin Edouard Grave and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. arXiv:2302.13971. Retrieved from https:\/\/arxiv.org\/abs\/arXiv:2302.13971"},{"key":"e_1_3_2_27_2","unstructured":"2024. Wikipedia (en) embedded with cohere.ai multilingual-22-12 encoder. Retrieved from https:\/\/huggingface.co\/datasets\/Cohere\/wikipedia-22-12-en-embeddings\/"},{"key":"e_1_3_2_28_2","unstructured":"Dan Hendrycks Collin Burns Steven Basart Andy Zou Mantas Mazeika Dawn Song and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv:2009.03300. Retrieved from https:\/\/arxiv.org\/abs\/arXiv:2009.03300"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00276"},{"key":"e_1_3_2_30_2","volume-title":"Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Yang Zhilin","year":"2018","unstructured":"Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)."},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-1147"},{"key":"e_1_3_2_32_2","unstructured":"2024. OpenAI text-embedding-3 model. Retrieved from https:\/\/openai.com\/blog\/new-embedding-models-and-api-updates\/"},{"key":"e_1_3_2_33_2","unstructured":"2024. OpenAI. Retrieved from https:\/\/openai.com\/"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3318464.3380600"},{"key":"e_1_3_2_35_2","volume-title":"Improving WWW proxies performance with greedy-dual-size-frequency caching policy","author":"Cherkasova Ludmila","year":"1998","unstructured":"Ludmila Cherkasova. 1998. Improving WWW proxies performance with greedy-dual-size-frequency caching policy. Hewlett-Packard Laboratories Palo Alto, CA, USA."},{"key":"e_1_3_2_36_2","first-page":"521","volume-title":"Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22)","author":"Yu Gyeong-In","year":"2022","unstructured":"Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A distributed serving system for \\(\\lbrace\\) Transformer-Based \\(\\rbrace\\) generative models. In Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). 521\u2013538."},{"key":"e_1_3_2_37_2","first-page":"117","volume-title":"Proceedings of the 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24)","author":"Agrawal Amey","year":"2024","unstructured":"Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. 2024. Taming throughput-latency tradeoff in LLM inference with Sarathi-serve. In Proceedings of the 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). 117\u2013134."},{"key":"e_1_3_2_38_2","first-page":"193","volume-title":"Proceedings of the 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24)","author":"Zhong Yinmin","year":"2024","unstructured":"Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. 2024. \\(\\lbrace\\) DistServe \\(\\rbrace\\) : Disaggregating prefill and decoding for goodput-optimized large language model serving. In Proceedings of the 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). 193\u2013210."},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISCA59077.2024.00019"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/3651890.3672274"},{"key":"e_1_3_2_41_2","unstructured":"DeepSeek-AI Aixin Liu Bei Feng Bing Xue Bingxuan Wang Bochao Wu Chengda Lu Chenggang Zhao Chengqi Deng Chenyu Zhang2024. DeepSeek-V3 technical report. arXiv:2412.19437. Retrieved from https:\/\/arxiv.org\/abs\/arXiv:2412.19437"},{"key":"e_1_3_2_42_2","first-page":"155","volume-title":"Proceedings of the 23rd USENIX Conference on File and Storage Technologies (FAST 25)","author":"Qin Ruoyu","year":"2025","unstructured":"Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. 2025. Mooncake: Trading more storage for less computation \u2014 a KVCache-centric architecture for serving LLM chatbot. In Proceedings of the 23rd USENIX Conference on File and Storage Technologies (FAST 25). 155\u2013170."},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/3695053.3731076"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11432-023-3894-4"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/3651890.3672252"},{"key":"e_1_3_2_46_2","unstructured":"2025. Introducing Deep Research. Retrieved from https:\/\/openai.com\/index\/introducing-deep-research\/"},{"key":"e_1_3_2_47_2","first-page":"69","volume-title":"Proceedings of the 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23)","author":"Wu Bingyang","year":"2023","unstructured":"Bingyang Wu, Zili Zhang, Zhihao Bai, Xuanzhe Liu, and Xin Jin. 2023. Transparent \\(\\lbrace\\) GPU \\(\\rbrace\\) sharing in container clouds for deep learning workloads. In Proceedings of the 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). 69\u201385."},{"issue":"12","key":"e_1_3_2_48_2","doi-asserted-by":"crossref","first-page":"222101","DOI":"10.1007\/s11432-024-4227-2","article-title":"MuxFlow: Efficient GPU sharing in production-level clusters with more than 10000 GPUs","volume":"67","author":"Liu Xuanzhe","year":"2024","unstructured":"Xuanzhe Liu, Yihao Zhao, Shufan Liu, Xiang Li, Yibo Zhu, Xin Liu, and Xin Jin. 2024. MuxFlow: Efficient GPU sharing in production-level clusters with more than 10000 GPUs. Science China Information Sciences 67, 12 (2024), 222101.","journal-title":"Science China Information Sciences"},{"key":"e_1_3_2_49_2","unstructured":"Adam Paszke Sam Gross Francisco Massa Adam Lerer James Bradbury Gregory Chanan Trevor Killeen Zeming Lin Natalia Gimelshein Luca Antiga Alban Desmaison Andreas K\u00f6pf Edward Yang Zach DeVito Martin Raison Alykhan Tejani Sasank Chilamkurthy Benoit Steiner Lu Fang Junjie Bai and Soumith Chintala. 2019. Pytorch: An imperative style high-performance deep learning library. Advances in Neural Information Processing Systems 32 (2019)."},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1145\/3315508.3329973"},{"key":"e_1_3_2_51_2","doi-asserted-by":"crossref","unstructured":"Joshua Ainslie James Lee-Thorp Michiel de Jong Yury Zemlyanskiy Federico Lebr\u00f3n and Sumit Sanghai. 2023. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv:2305.13245. Retrieved from https:\/\/arxiv.org\/abs\/arXiv:2305.13245","DOI":"10.18653\/v1\/2023.emnlp-main.298"},{"key":"e_1_3_2_52_2","unstructured":"Albert Q. Jiang Alexandre Sablayrolles Arthur Mensch Chris Bamford Devendra Singh Chaplot Diego de las Casas Florian Bressand Gianna Lengyel Guillaume Lample Lucile Saulnier L\u00e9lio Renard Lavaud Marie-Anne Lachaux Pierre Stock Teven Le Scao Thibaut Lavril Thomas Wang Timoth\u00e9e Lacroix and William El Sayed. 2023. Mistral 7B. arXiv:2310.06825. Retrieved from https:\/\/arxiv.org\/abs\/arXiv:2310.06825"},{"key":"e_1_3_2_53_2","unstructured":"Albert Q. Jiang Alexandre Sablayrolles Antoine Roux Arthur Mensch Blanche Savary Chris Bamford Devendra Singh Chaplot Diego de las Casas Emma Bou Hanna Florian Bressand Gianna Lengyel Guillaume Bour Guillaume Lample L\u00e9lio Renard Lavaud Lucile Saulnier Marie-Anne Lachaux Pierre Stock Sandeep Subramanian Sophia Yang Szymon Antoniak Teven Le Scao Th\u00e9ophile Gervet Thibaut Lavril Thomas Wang Timoth\u00e9e Lacroix and William El Sayed. 2024. Mixtral of experts. arXiv:2401.04088. Retrieved from https:\/\/arxiv.org\/abs\/arXiv:2401.04088"},{"key":"e_1_3_2_54_2","unstructured":"Bingyang Wu Yinmin Zhong Zili Zhang Gang Huang Xuanzhe Liu and Xin Jin. 2023. Fast distributed inference serving for large language models. arXiv:2305.05920. Retrieved from https:\/\/arxiv.org\/abs\/arXiv:2305.05920"},{"key":"e_1_3_2_55_2","first-page":"663","volume-title":"Proceedings of the 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23)","author":"Li Zhuohan","year":"2023","unstructured":"Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng, Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. \\(\\lbrace\\) AlpaServe \\(\\rbrace\\) : Statistical multiplexing with model parallelism for deep learning serving. In Proceedings of the 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23). 663\u2013679."},{"key":"e_1_3_2_56_2","unstructured":"Charlie Chen Sebastian Borgeaud Geoffrey Irving Jean-Baptiste Lespiau Laurent Sifre and John Jumper. 2023. Accelerating large language model decoding with speculative sampling. arXiv:2302.01318. Retrieved from https:\/\/arxiv.org\/abs\/arXiv:2302.01318"},{"key":"e_1_3_2_57_2","unstructured":"Zhihao Zhang Alan Zhu Lijie Yang Yihua Xu Lanting Li Phitchaya Mangpo Phothilimthana and Zhihao Jia. 2024. Accelerating retrieval-augmented language model serving with speculation. arXiv:2401.14021. Retrieved from https:\/\/arxiv.org\/abs\/arXiv:2401.14021"},{"key":"e_1_3_2_58_2","first-page":"5199","article-title":"Spann: Highly-efficient billion-scale approximate nearest neighborhood search","volume":"34","author":"Chen Qi","year":"2021","unstructured":"Qi Chen, Bing Zhao, Haidong Wang, Mingqin Li, Chuanjie Liu, Zengzhong Li, Mao Yang, and Jingdong Wang. 2021. Spann: Highly-efficient billion-scale approximate nearest neighborhood search. Advances in Neural Information Processing Systems 34 (2021), 5199\u20135212.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.14778\/3303753.3303754"},{"key":"e_1_3_2_60_2","article-title":"Diskann: Fast accurate billion-point nearest neighbor search on a single node","volume":"32","author":"Subramanya Suhas Jayaram","year":"2019","unstructured":"Suhas Jayaram Subramanya, Fnu Devvrit, Harsha Vardhan Simhadri, Ravishankar Krishnawamy, and Rohan Kadekodi. 2019. Diskann: Fast accurate billion-point nearest neighbor search on a single node. Advances in Neural Information Processing Systems 32 (2019).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_61_2","unstructured":"Guangxuan Xiao Yuandong Tian Beidi Chen Song Han and Mike Lewis. 2023. Efficient streaming language models with attention sinks. arXiv:2309.17453. Retrieved from https:\/\/arxiv.org\/abs\/arXiv:2309.17453"},{"key":"e_1_3_2_62_2","first-page":"30318","article-title":"Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale","volume":"35","author":"Dettmers Tim","year":"2022","unstructured":"Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems 35 (2022), 30318\u201330332.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_63_2","first-page":"5958","volume-title":"Proceedings of the International Conference on machine learning","author":"Li Zhuohan","year":"2020","unstructured":"Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joey Gonzalez. 2020. Train big, then compress: Rethinking model size for efficient training and inference of transformers. In Proceedings of the International Conference on machine learning. PMLR, 5958\u20135968."},{"key":"e_1_3_2_64_2","first-page":"52342","article-title":"Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time","volume":"36","author":"Liu Zichang","year":"2023","unstructured":"Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. 2023. Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time. Advances in Neural Information Processing Systems 36 (2023), 52342\u201352364.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_65_2","unstructured":"Suyu Ge Yunan Zhang Liyuan Liu Minjia Zhang Jiawei Han and Jianfeng Gao. 2023. Model tells you what to discard: Adaptive kv cache compression for llms. arXiv:2310.01801. Retrieved from https:\/\/arxiv.org\/abs\/arXiv:2310.01801"},{"key":"e_1_3_2_66_2","first-page":"34661","article-title":"H2o: Heavy-hitter oracle for efficient generative inference of large language models","volume":"36","author":"Zhang Zhenyu","year":"2023","unstructured":"Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher R\u00e9, Clark Barrett, Zhangyang Wang, and Beidi Chen. 2023. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems 36 (2023), 34661\u201334710.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_67_2","first-page":"325","article-title":"Prompt cache: Modular attention reuse for low-latency inference","volume":"6","author":"Gim In","year":"2024","unstructured":"In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong. 2024. Prompt cache: Modular attention reuse for low-latency inference. Proceedings of Machine Learning and Systems 6 (2024), 325\u2013338.","journal-title":"Proceedings of Machine Learning and Systems"},{"key":"e_1_3_2_68_2","unstructured":"Lu Ye Ze Tao Yong Huang and Yang Li. 2024. ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition. arXiv:2402.15220. Retrieved from https:\/\/arxiv.org\/abs\/arXiv:2402.15220 (2024)."},{"key":"e_1_3_2_69_2","first-page":"94","volume-title":"Proceedings of the 20th European Conference on Computer Systems","author":"Yao Jiayi","year":"2025","unstructured":"Jiayi Yao, Hanchen Li, Yuhan Liu, Siddhant Ray, Yihua Cheng, Qizheng Zhang, Kuntai Du, Shan Lu, and Junchen Jiang. 2025. CacheBlend: Fast large language model serving for RAG with cached knowledge fusion. In Proceedings of the 20th European Conference on Computer Systems. 94\u2013109."},{"key":"e_1_3_2_70_2","unstructured":"Chien-Yu Lin Keisuke Kamahori Yiyu Liu Xiaoxiang Shi Madhav Kashyap Yile Gu Rulin Shao Zihao Ye Kan Zhu Stephanie Wang Arvind Krishnamurthy Rohan Kadekodi Luis Ceze and Baris Kasikci. 2025. TeleRAG: Efficient retrieval-augmented generation inference with lookahead retrieval. arXiv:2502.20969. Retrieved from https:\/\/arxiv.org\/abs\/arXiv:2502.20969"},{"key":"e_1_3_2_71_2","doi-asserted-by":"publisher","DOI":"10.1145\/3695053.3731089"},{"key":"e_1_3_2_72_2","doi-asserted-by":"publisher","DOI":"10.1145\/3695053.3731032"}],"container-title":["ACM Transactions on Computer Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3768628","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,11,7]],"date-time":"2025-11-07T14:00:45Z","timestamp":1762524045000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3768628"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,11,7]]},"references-count":71,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2026,2,28]]}},"alternative-id":["10.1145\/3768628"],"URL":"https:\/\/doi.org\/10.1145\/3768628","relation":{},"ISSN":["0734-2071","1557-7333"],"issn-type":[{"value":"0734-2071","type":"print"},{"value":"1557-7333","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,11,7]]},"assertion":[{"value":"2025-04-21","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-09-05","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-11-07","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}