{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,7]],"date-time":"2026-10-07T18:09:59Z","timestamp":1791396599558,"version":"4.3.3"},"reference-count":52,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2026,8,17]],"date-time":"2026-08-17T00:00:00Z","timestamp":1786924800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"funder":[{"name":"National Key Research & Development Program of China","award":["022YFB4502004"],"award-info":[{"award-number":["022YFB4502004"]}]},{"name":"Beijing Natural Science Foundation","award":["L252014"],"award-info":[{"award-number":["L252014"]}]},{"name":"Tsinghua University Initiative Scientific Research Program"},{"name":"Approaching.AI Co., Ltd."}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Storage"],"published-print":{"date-parts":[[2026,12,30]]},"abstract":"<jats:p>\n                    <jats:sc>Mooncake<\/jats:sc>\n                    is the serving platform for Kimi, an LLM chatbot service developed by Moonshot AI. This platform features a KVCache-centric disaggregated architecture that not only separates prefill and decoding clusters but also efficiently utilizes the underexploited CPU, DRAM, SSD and NIC resources of the GPU cluster to establish a disaggregated KVCache. At the core of\n                    <jats:sc>Mooncake<\/jats:sc>\n                    is its KVCache-centric global cache and a scheduler designed to maximize throughput while adhering to stringent latency-related Service Level Objectives (SLOs). Our experiments demonstrate that\n                    <jats:sc>Mooncake<\/jats:sc>\n                    excels in scenarios involving long-context inputs. In tests using real traces,\n                    <jats:sc>Mooncake<\/jats:sc>\n                    increases the effective request capacity by 59%\u223c498% when compared to baseline methods, all while complying with SLOs. Currently,\n                    <jats:sc>Mooncake<\/jats:sc>\n                    is operational across thousands of nodes, processing over 100 billion tokens daily. In practical deployments,\n                    <jats:sc>Mooncake<\/jats:sc>\n                    \u2019s innovative architecture enables Kimi to handle 115% and 107% more requests on NVIDIA A800 and H800 clusters, respectively, compared to previous systems.\n                  <\/jats:p>","DOI":"10.1145\/3773772","type":"journal-article","created":{"date-parts":[[2025,11,15]],"date-time":"2025-11-15T09:11:56Z","timestamp":1763197916000},"page":"1-38","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":27,"title":["Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving"],"prefix":"10.1145","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0009-0001-7794-4400","authenticated-orcid":false,"given":"Ruoyu","family":"Qin","sequence":"first","affiliation":[{"name":"Computer Science and Technology, Tsinghua University","place":["Beijing, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-5091-3114","authenticated-orcid":false,"given":"Zheming","family":"Li","sequence":"additional","affiliation":[{"name":"Moonshot AI","place":["Beijing, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-7951-6854","authenticated-orcid":false,"given":"Weiran","family":"He","sequence":"additional","affiliation":[{"name":"Moonshot AI","place":["Beijing, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-0038-100X","authenticated-orcid":false,"given":"Jialei","family":"Cui","sequence":"additional","affiliation":[{"name":"Moonshot AI","place":["Beijing, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4284-1089","authenticated-orcid":false,"given":"Heyi","family":"Tang","sequence":"additional","affiliation":[{"name":"Moonshot AI","place":["Beijing, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-3599-4086","authenticated-orcid":false,"given":"Feng","family":"Ren","sequence":"additional","affiliation":[{"name":"Computer Science and Technology, Tsinghua University","place":["Beijing, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7104-1526","authenticated-orcid":false,"given":"Teng","family":"Ma","sequence":"additional","affiliation":[{"name":"Alibaba Cloud Computing","place":["Hangzhou, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0902-7774","authenticated-orcid":false,"given":"Shangming","family":"Cai","sequence":"additional","affiliation":[{"name":"Alibaba Cloud Computing","place":["Hangzhou, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-7743-2508","authenticated-orcid":false,"given":"Yineng","family":"Zhang","sequence":"additional","affiliation":[{"name":"Independent Researcher","place":["Berkeley, United States"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7518-0753","authenticated-orcid":false,"given":"Mingxing","family":"Zhang","sequence":"additional","affiliation":[{"name":"Tsinghua University","place":["Beijing, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6651-7032","authenticated-orcid":false,"given":"Yongwei","family":"Wu","sequence":"additional","affiliation":[{"name":"Computer Science and Technology, Tsinghua University","place":["Beijing, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-3026-2278","authenticated-orcid":false,"given":"Weimin","family":"Zheng","sequence":"additional","affiliation":[{"name":"department of computer science and technology, Tsinghua University","place":["Beijing, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-8504-978X","authenticated-orcid":false,"given":"Xinran","family":"Xu","sequence":"additional","affiliation":[{"name":"Moonshot AI","place":["Beijing, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,8,17]]},"reference":[{"key":"e_1_3_3_2_2","first-page":"117","volume-title":"Proceedings of the 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24)","author":"Agrawal Amey","year":"2024","unstructured":"Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. 2024. Taming throughput-latency tradeoff in LLM inference with sarathi-serve. In Proceedings of the 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). 117\u2013134."},{"key":"e_1_3_3_3_2","first-page":"1","volume-title":"Proceedings of the SC22: International Conference for High Performance Computing, Networking, Storage and Analysis","author":"Aminabadi Reza Yazdani","year":"2022","unstructured":"Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, et\u00a0al. 2022. Deepspeed-inference: Enabling efficient inference of transformer models at unprecedented scale. In Proceedings of the SC22: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 1\u201315."},{"key":"e_1_3_3_4_2","first-page":"14388","article-title":"L-eval: Instituting standardized evaluation for long context language models","author":"An Chenxin","year":"2024","unstructured":"Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2024. L-eval: Instituting standardized evaluation for long context language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 14388\u201314411.","journal-title":"Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"},{"key":"e_1_3_3_5_2","doi-asserted-by":"crossref","unstructured":"William Brandon Mayank Mishra Aniruddha Nrusimha Rameswar Panda and Jonathan Ragan-Kelley. 2024. Reducing transformer key-value cache size with cross-layer attention. Advances in Neural Information Processing Systems 37 (2024) 86927\u201386957.","DOI":"10.52202\/079017-2758"},{"key":"e_1_3_3_6_2","unstructured":"William Brandon Aniruddha Nrusimha Kevin Qian Zachary Ankner Tian Jin Zhiye Song and Jonathan Ragan-Kelley. 2023. Striped Attention: Faster ring attention for causal transformers. arXiv:2311.09431 [cs.LG] https:\/\/arxiv.org\/abs\/2311.09431"},{"key":"e_1_3_3_7_2","unstructured":"Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde de Oliveira Pinto Jared Kaplan Harri Edwards Yuri Burda Nicholas Joseph Greg Brockman Alex Ray Raul Puri Gretchen Krueger Michael Petrov Heidy Khlaaf Girish Sastry Pamela Mishkin Brooke Chan Scott Gray Nick Ryder Mikhail Pavlov Alethea Power Lukasz Kaiser Mohammad Bavarian Clemens Winter Philippe Tillet Felipe Petroski Such Dave Cummings Matthias Plappert Fotios Chantzis Elizabeth Barnes Ariel Herbert-Voss William Hebgen Guss Alex Nichol Alex Paino Nikolas Tezak Jie Tang Igor Babuschkin Suchir Balaji Shantanu Jain William Saunders Christopher Hesse Andrew N. Carr Jan Leike Josh Achiam Vedant Misra Evan Morikawa Alec Radford Matthew Knight Miles Brundage Mira Murati Katie Mayer Peter Welinder Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever and Wojciech Zaremba. 2021. Evaluating large language models trained on code. arXiv:2107.03374 [cs.LG] https:\/\/arxiv.org\/abs\/2107.03374"},{"key":"e_1_3_3_8_2","unstructured":"Alibaba Cloud. 2025. Elastic Compute Service:eRDMA. Retrieved from https:\/\/www.alibabacloud.com\/help\/en\/ecs\/user-guide\/elastic-rdma-erdma\/"},{"key":"e_1_3_3_9_2","unstructured":"LMDeploy Contributors. 2023. LMDeploy: A Toolkit for Compressing Deploying and Serving LLM. Retrieved from https:\/\/github.com\/InternLM\/lmdeploy"},{"key":"e_1_3_3_10_2","unstructured":"NVIDIA Corporation. 2019. FasterTransformer. Retrieved from https:\/\/github.com\/NVIDIA\/FasterTransformer"},{"key":"e_1_3_3_11_2","unstructured":"NVIDIA Corporation. 2020. NCCL. Retrieved from https:\/\/github.com\/NVIDIA\/nccl"},{"key":"e_1_3_3_12_2","unstructured":"NVIDIA Corporation. 2023. TensorRT-LLM. Retrieved from https:\/\/github.com\/NVIDIA\/TensorRT-LLM"},{"key":"e_1_3_3_13_2","unstructured":"DeepSeek. 2025. DeepEP. Retrieved from https:\/\/github.com\/deepseek-ai\/DeepEP"},{"key":"e_1_3_3_14_2","unstructured":"Jiarui Fang and Shangchun Zhao. 2024. Usp: A unified sequence parallelism approach for long context generative ai. arXiv:2405.07719. Retrieved from https:\/\/arxiv.org\/abs\/2405.07719"},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.5555\/3691992.3691999"},{"key":"e_1_3_3_16_2","unstructured":"In Gim Guojun Chen Seung-seob Lee Nikhil Sarda Anurag Khandelwal and Lin Zhong. 2024. Prompt cache: Modular attention reuse for low-latency inference. In Proceedings of Machine Learning and Systems P. Gibbons G. Pekhimenko and C. De Sa (Eds.). Vol. 6. 325\u2013338. https:\/\/proceedings.mlsys.org\/paper_files\/paper\/2024\/file\/a66caa1703fe34705a4368c3014c1966-Paper-Conference.pdf"},{"key":"e_1_3_3_17_2","unstructured":"Google. 2024. Our next-generation model: Gemini 1.5. Retrieved from https:\/\/blog.google\/technology\/ai\/google-gemini-next-generation-model-february-2024"},{"key":"e_1_3_3_18_2","unstructured":"DeepSeek-AI Daya Guo Dejian Yang Haowei Zhang Junxiao Song Ruoyu Zhang Runxin Xu Qihao Zhu Shirong Ma Peiyi Wang Xiao Bi Xiaokang Zhang Xingkai Yu YuWu Z. F.Wu Zhibin Gou Zhihong Shao Zhuoshu Li Ziyi Gao Aixin Liu Bing Xue Bingxuan Wang Bochao Wu Bei Feng Chengda Lu Chenggang Zhao Chengqi Deng Chenyu Zhang Chong Ruan Damai Dai Deli Chen Dongjie Ji Erhang Li Fangyun Lin Fucong Dai Fuli Luo Guangbo Hao Guanting Chen Guowei Li H. Zhang Han Bao Hanwei Xu Haocheng Wang Honghui Ding Huajian Xin Huazuo Gao Hui Qu Hui Li Jianzhong Guo Jiashi Li Jiawei Wang Jingchang Chen Jingyang Yuan Junjie Qiu Junlong Li J. L. Cai Jiaqi Ni Jian Liang Jin Chen Kai Dong Kai Hu Kaige Gao Kang Guan Kexin Huang Kuai Yu Lean Wang Lecong Zhang Liang Zhao Litong Wang Liyue Zhang Lei Xu Leyi Xia Mingchuan Zhang Minghua Zhang Minghui Tang Meng Li Miaojun Wang Mingming Li Ning Tian Panpan Huang Peng Zhang Qiancheng Wang Qinyu Chen Qiushi Du Ruiqi Ge Ruisong Zhang Ruizhe Pan Runji Wang R. J. Chen R. L. Jin Ruyi Chen Shanghao Lu Shangyan Zhou Shanhuang Chen Shengfeng Ye Shiyu Wang Shuiping Yu Shunfeng Zhou Shuting Pan S. S. Li Shuang Zhou Shaoqing Wu Shengfeng Ye Tao Yun Tian Pei Tianyu Sun T. Wang Wangding Zeng Wanjia Zhao Wen Liu Wenfeng Liang Wenjun Gao Wenqin Yu Wentao Zhang W. L. Xiao Wei An Xiaodong Liu Xiaohan Wang Xiaokang Chen Xiaotao Nie Xin Cheng Xin Liu Xin Xie Xingchao Liu Xinyu Yang Xinyuan Li Xuecheng Su Xuheng Lin X. Q. Li Xiangyue Jin Xiaojin Shen Xiaosha Chen Xiaowen Sun Xiaoxiang Wang Xinnan Song Xinyi Zhou Xianzu Wang Xinxia Shan Y. K. Li Y. Q. Wang Y. X. Wei Yang Zhang Yanhong Xu Yao Li Yao Zhao Yaofeng Sun Yaohui Wang Yi Yu Yichao Zhang Yifan Shi Yiliang Xiong Ying He Yishi Piao Yisong Wang Yixuan Tan Yiyang Ma Yiyuan Liu Yongqiang Guo Yuan Ou Yuduan Wang Yue Gong Yuheng Zou Yujia He Yunfan Xiong Yuxiang Luo Yuxiang You Yuxuan Liu Yuyang Zhou Y. X. Zhu Yanhong Xu Yanping Huang Yaohui Li Yi Zheng Yuchen Zhu Yunxian Ma Ying Tang Yukun Zha Yuting Yan Z. Z. Ren Zehui Ren Zhangli Sha Zhe Fu Zhean Xu Zhenda Xie Zhengyan Zhang Zhewen Hao Zhicheng Ma Zhigang Yan Zhiyu Wu Zihui Gu Zijia Zhu Zijun Liu Zilin Li Ziwei Xie Ziyang Song Zizheng Pan Zhen Huang Zhipeng Xu Zhongyu Zhang and Zhen Zhang. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv:2501.12948 [cs.CL] https:\/\/arxiv.org\/abs\/2501.12948"},{"key":"e_1_3_3_19_2","doi-asserted-by":"crossref","unstructured":"Coleman Hooper Sehoon Kim Hiva Mohammadzadeh Michael W. Mahoney Yakun S. Shao Kurt Keutzer and Amir Gholami. 2024. Kvquant: Towards 10 million context length LLM inference with kv cache quantization. Advances in Neural Information Processing Systems 37 (2024) 1270\u20131303.","DOI":"10.52202\/079017-0040"},{"key":"e_1_3_3_20_2","unstructured":"Cunchen Hu Heyang Huang Liangliang Xu Xusheng Chen Jiang Xu Shuang Chen Hao Feng Chenxi Wang Sa Wang Yungang Bao Ninghui Sun and Yizhou Shan. 2024. Inference without interference: disaggregate LLM inference for mixed downstream workloads. arXiv:2401.11181 [cs.DC] https:\/\/arxiv.org\/abs\/2401.11181"},{"key":"e_1_3_3_21_2","doi-asserted-by":"crossref","unstructured":"Sam Ade Jacobs Masahiro Tanaka Chengming Zhang Minjia Zhang Reza Yazdani Aminadabi Shuaiwen Leon Song Samyam Rajbhandari and Yuxiong He. 2024. System optimizations for enabling training of extreme long sequence transformer models. In Proceedings of the 43rd ACM Symposium on Principles of Distributed Computing. 121\u2013130.","DOI":"10.1145\/3662158.3662806"},{"key":"e_1_3_3_22_2","unstructured":"Vijay Anand Korthikanti Jared Casper Sangkug Lym Lawrence McAfee Michael Andersch Mohammad Shoeybi and Bryan Catanzaro. 2023. Reducing activation recomputation in large transformer models. In Proceedings of Machine Learning and Systems D. Song M. Carbin and T. Chen (Eds.). Vol. 5. Curan 341\u2013353. https:\/\/proceedings.mlsys.org\/paper_files\/paper\/2023\/file\/80083951326cf5b35e5100260d64ed81-Paper-mlsys2023.pdf"},{"key":"e_1_3_3_23_2","unstructured":"KVCache.ai. 2025. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving. Retrieved from https:\/\/github.com\/kvcache-ai\/Mooncake"},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/3600006.3613165"},{"key":"e_1_3_3_25_2","volume-title":"Proceedings of the Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WANT@ NeurIPS 2023)","author":"Li Dacheng","year":"2023","unstructured":"Dacheng Li, Rulin Shao, Anze Xie, Eric P. Xing, Joseph E. Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. 2023. LightSeq:: Sequence level parallelism for distributed training of long context transformers. In Proceedings of the Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WANT@ NeurIPS 2023)."},{"key":"e_1_3_3_26_2","doi-asserted-by":"crossref","unstructured":"Jiaqi Li MengmengWang Zilong Zheng and Muhan Zhang. 2024. Loogle: Can long-context language models understand long contexts? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 16304\u201316333.","DOI":"10.18653\/v1\/2024.acl-long.859"},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.acl-long.134"},{"key":"e_1_3_3_28_2","first-page":"6543","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Li Zhuohan","year":"2021","unstructured":"Zhuohan Li, Siyuan Zhuang, Shiyuan Guo, Danyang Zhuo, Hao Zhang, Dawn Song, and Ion Stoica. 2021. Terapipe: Token-level pipeline parallelism for training large-scale language models. In Proceedings of the International Conference on Machine Learning. PMLR, 6543\u20136552."},{"key":"e_1_3_3_29_2","unstructured":"DeepSeek-AI Aixin Liu Bei Feng Bin Wang Bingxuan Wang Bo Liu Chenggang Zhao Chengqi Dengr Chong Ruan Damai Dai Daya Guo Dejian Yang Deli Chen Dongjie Ji Erhang Li Fangyun Lin Fuli Luo Guangbo Hao Guanting Chen Guowei Li H. Zhang Hanwei Xu Hao Yang Haowei Zhang Honghui Ding Huajian Xin Huazuo Gao Hui Li Hui Qu J. L. Cai Jian Liang Jianzhong Guo Jiaqi Ni Jiashi Li Jin Chen Jingyang Yuan Junjie Qiu Junxiao Song Kai Dong Kaige Gao Kang Guan Lean Wang Lecong Zhang Lei Xu Leyi Xia Liang Zhao Liyue Zhang Meng Li Miaojun Wang Mingchuan Zhang Minghua Zhang Minghui Tang Mingming Li Ning Tian Panpan Huang Peiyi Wang Peng Zhang Qihao Zhu Qinyu Chen Qiushi Du R. J. Chen R. L. Jin Ruiqi Ge Ruizhe Pan Runxin Xu Ruyi Chen S. S. Li Shanghao Lu Shangyan Zhou Shanhuang Chen Shaoqing Wu Shengfeng Ye Shirong Ma Shiyu Wang Shuang Zhou Shuiping Yu Shunfeng Zhou Size Zheng T. Wang Tian Pei Tian Yuan Tianyu Sun W. L. Xiao Wangding Zeng Wei An Wen Liu Wenfeng Liang Wenjun Gao Wentao Zhang X. Q. Li Xiangyue Jin Xianzu Wang Xiao Bi Xiaodong Liu Xiaohan Wang Xiaojin Shen Xiaokang Chen Xiaosha Chen Xiaotao Nie Xiaowen Sun Xiaoxiang Wang Xin Liu Xin Xie Xingkai Yu Xinnan Song Xinyi Zhou Xinyu Yang Xuan Lu Xuecheng Su Y. Wu Y. K. Li Y. X. Wei Y. X. Zhu Yanhong Xu Yanping Huang Yao Li Yao Zhao Yaofeng Sun Yaohui Li Yaohui Wang Yi Zheng Yichao Zhang Yiliang Xiong Yilong Zhao Ying He Ying Tang Yishi Piao Yixin Dong Yixuan Tan Yiyuan Liu Yongji Wang Yongqiang Guo Yuchen Zhu Yuduan Wang Yuheng Zou Yukun Zha Yunxian Ma Yuting Yan Yuxiang You Yuxuan Liu Z. Z. Ren Zehui Ren Zhangli Sha Zhe Fu Zhen Huang Zhen Zhang Zhenda Xie Zhewen Hao Zhihong Shao Zhiniu Wen Zhipeng Xu Zhongyu Zhang Zhuoshu Li Zihan Wang Zihui Gu Zilin Li and Ziwei Xie. 2024. DeepSeek-V2: A strong economical and efficient mixture-of-experts language model. arXiv:2405.04434 [cs.CL] https:\/\/arxiv.org\/abs\/2405.04434"},{"key":"e_1_3_3_30_2","volume-title":"Proceedings of the NeurIPS 2023 Foundation Models for Decision Making Workshop","author":"Liu Hao","year":"2023","unstructured":"Hao Liu, Matei Zaharia, and Pieter Abbeel. 2023. Ring attention with blockwise transformers for near-infinite context. In Proceedings of the NeurIPS 2023 Foundation Models for Decision Making Workshop."},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/3651890.3672274"},{"key":"e_1_3_3_32_2","unstructured":"Zirui Liu Jiayi Yuan Hongye Jin Shaochen Zhong Zhaozhuo Xu Vladimir Braverman Beidi Chen and Xia Hu. 2024. KIVI: A tuning-free asymmetric 2bit quantization for KV cache. In Proceedings of the 41st International Conference on Machine Learning. 32332\u201332344."},{"key":"e_1_3_3_33_2","unstructured":"Moonshot AI. 2023. Kimi. Retrieved from https:\/\/kimi.moonshot.cn"},{"key":"e_1_3_3_34_2","unstructured":"MoonshotAI. 2025. Kimi K2. Retrieved from https:\/\/github.com\/MoonshotAI\/Kimi-K2"},{"key":"e_1_3_3_35_2","unstructured":"NVIDIA. 2022. NVIDIA H100 Tensor Core GPU Architecture. Retrieved from https:\/\/resources.nvidia.com\/en-us-tensor-core"},{"key":"e_1_3_3_36_2","unstructured":"OpenAI. 2022. Introducing ChatGPT. Retrieved from https:\/\/openai.com\/blog\/chatgpt"},{"key":"e_1_3_3_37_2","unstructured":"LMSYS Org. 2025. Deploying DeepSeek with PD Disaggregation and Large-Scale Expert Parallelism on 96 H100 GPUs. Retrieved from https:\/\/lmsys.org\/blog\/2025-05-05-large-scale-ep\/"},{"key":"e_1_3_3_38_2","unstructured":"Charles Packer Vivian Fang Shishir G. Patil Kevin Lin Sarah Wooders and Joseph E. Gonzalez. 2024. Memgpt: Towards llms as operating systems. arXiv:2310.08560 [cs.AI] https:\/\/arxiv.org\/abs\/2310.08560"},{"key":"e_1_3_3_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/3586183.3606763"},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISCA59077.2024.00019"},{"issue":"8","key":"e_1_3_3_41_2","first-page":"9","article-title":"Language models are unsupervised multitask learners","volume":"1","year":"2019","unstructured":"Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9.","journal-title":"OpenAI Blog"},{"key":"e_1_3_3_42_2","unstructured":"Facebook AI Research. 2017. Gloo. Retrieved from https:\/\/github.com\/pytorch\/gloo"},{"key":"e_1_3_3_43_2","unstructured":"Ying Sheng Lianmin Zheng Binhang Yuan Zhuohan Li Max Ryabinin Beidi Chen Percy Liang Christopher R\u00e9 Ion Stoica and Ce Zhang. 2023. Flexgen: High-throughput generative inference of large language models with a single GPU. In International Conference on Machine Learning. PMLR 31094\u201331116."},{"key":"e_1_3_3_44_2","first-page":"31094","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Sheng Ying","year":"2023","unstructured":"Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher R\u00e9, Ion Stoica, and Ce Zhang. 2023. Flexgen: High-throughput generative inference of large language models with a single gpu. In Proceedings of the International Conference on Machine Learning. PMLR, 31094\u201331116."},{"key":"e_1_3_3_45_2","unstructured":"Vikranth Srivatsa Zijian He Reyna Abhyankar Dongming Li and Yiying Zhang. 2024. Preble: Efficient distributed prompt scheduling for LLM serving. arXiv preprint arXiv:2407.00023 (2024)."},{"key":"e_1_3_3_46_2","unstructured":"Hugo Touvron Thibaut Lavril Gautier Izacard Xavier Martinet Marie-Anne Lachaux Timoth\u00e9e Lacroix Baptiste Rozi\u00e8re Naman Goyal Eric Hambro Faisal Azhar Aurelien Rodriguez Armand Joulin Edouard Grave and Guillaume Lample. 2023. LLaMA: Open and efficient foundation language models. arXiv:2302.13971 [cs.CL] https:\/\/arxiv.org\/abs\/2302.13971"},{"key":"e_1_3_3_47_2","unstructured":"Hugo Touvron Louis Martin Kevin Stone Peter Albert Amjad Almahairi Yasmine Babaei Nikolay Bashlykov Soumya Batra Prajjwal Bhargava Shruti Bhosale Dan Bikel Lukas Blecher Cristian Canton Ferrer Moya Chen Guillem Cucurull David Esiobu Jude Fernandes Jeremy Fu Wenyin Fu Brian Fuller Cynthia Gao Vedanuj Goswami Naman Goyal Anthony Hartshorn Saghar Hosseini Rui Hou Hakan Inan Marcin Kardas Viktor Kerkez Madian Khabsa Isabel Kloumann Artem Korenev Punit Singh Koura Marie-Anne Lachaux Thibaut Lavril Jenya Lee Diana Liskovich Yinghai Lu Yuning Mao Xavier Martinet Todor Mihaylov Pushkar Mishra Igor Molybog Yixin Nie Andrew Poulton Jeremy Reizenstein Rashi Rungta Kalyan Saladi Alan Schelten Ruan Silva Eric Michael Smith Ranjan Subramanian Xiaoqing Ellen Tan Binh Tang Ross Taylor Adina Williams Jian Xiang Kuan Puxin Xu Zheng Yan Iliyan Zarov Yuchen Zhang Angela Fan Melanie Kambadur Sharan Narang Aurelien Rodriguez Robert Stojnic Sergey Edunov and Thomas Scialom. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288 [cs.CL] https:\/\/arxiv.org\/abs\/2307.09288"},{"key":"e_1_3_3_48_2","doi-asserted-by":"publisher","DOI":"10.1145\/3694715.3695948"},{"key":"e_1_3_3_49_2","unstructured":"Bingyang Wu Yinmin Zhong Zili Zhang Gang Huang Xuanzhe Liu and Xin Jin. 2024. Fast distributed inference serving for large language models. arXiv:2305.05920 [cs.LG] https:\/\/arxiv.org\/abs\/2305.05920"},{"key":"e_1_3_3_50_2","first-page":"521","volume-title":"Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22)","author":"Yu Gyeong-In","year":"2022","unstructured":"Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. 2022. Orca: A distributed serving system for Transformer-Based generative models. In Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). 521\u2013538."},{"key":"e_1_3_3_51_2","first-page":"1229","volume-title":"Proceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24)","author":"Zhang Yazhuo","year":"2024","unstructured":"Yazhuo Zhang, Juncheng Yang, Yao Yue, Ymir Vigfusson, and K. V. Rashmi. 2024. SIEVE is simpler than LRU: An efficient turn-key eviction algorithm for web caches. In Proceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24). 1229\u20131246."},{"key":"e_1_3_3_52_2","unstructured":"Lianmin Zheng Liangsheng Yin Zhiqiang Xie Chuyue Sun Jeff Huang Cody Hao Yu Shiyi Cao Christos Kozyrakis Ion Stoica Joseph E. Gonzalez Clark Barrett and Ying Sheng. 2024. SGLang: Efficient execution of structured language model programs. In Proceedings of the 38th International Conference on Neural Information Processing Systems (Vancouver BC Canada) (NIPS\u201924). Curran Associates Inc. Red Hook NY USA Article 2000 27 pages."},{"key":"e_1_3_3_53_2","first-page":"193","volume-title":"Proceedings of the 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24)","author":"Zhong Yinmin","year":"2024","unstructured":"Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. 2024. DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving. In Proceedings of the 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). 193\u2013210."}],"container-title":["ACM Transactions on Storage"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3773772","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,17]],"date-time":"2026-08-17T16:40:26Z","timestamp":1786984826000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3773772"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,17]]},"references-count":52,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2026,12,30]]}},"alternative-id":["10.1145\/3773772"],"URL":"https:\/\/doi.org\/10.1145\/3773772","relation":{},"ISSN":["1553-3077","1553-3093"],"issn-type":[{"value":"1553-3077","type":"print"},{"value":"1553-3093","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,17]]},"assertion":[{"value":"2025-07-28","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-10-23","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-08-17","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}