{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,15]],"date-time":"2026-03-15T15:30:26Z","timestamp":1773588626619,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":70,"publisher":"ACM","funder":[{"name":"National Key R&D Program of China","award":["2022ZD0160300"],"award-info":[{"award-number":["2022ZD0160300"]}]},{"name":"Center of Excellence, Peking University","award":["NA"],"award-info":[{"award-number":["NA"]}]},{"name":"CCF-Tencent Rhino-Bird Open Research Fund","award":["NA"],"award-info":[{"award-number":["NA"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2026,3,22]]},"DOI":"10.1145\/3779212.3790237","type":"proceedings-article","created":{"date-parts":[[2026,3,10]],"date-time":"2026-03-10T13:55:26Z","timestamp":1773150926000},"page":"2048-2062","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill & Decode Inference"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0009-0002-0486-8651","authenticated-orcid":false,"given":"Xiaojuan","family":"Tang","sequence":"first","affiliation":[{"name":"Peking University, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-2382-556X","authenticated-orcid":false,"given":"Fanxu","family":"Meng","sequence":"additional","affiliation":[{"name":"Peking University, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-7958-7144","authenticated-orcid":false,"given":"Pingzhi","family":"Tang","sequence":"additional","affiliation":[{"name":"Peking University, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-7552-307X","authenticated-orcid":false,"given":"Yuxuan","family":"Wang","sequence":"additional","affiliation":[{"name":"Peking University, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-7246-3102","authenticated-orcid":false,"given":"Di","family":"Yin","sequence":"additional","affiliation":[{"name":"Tencent YouTu Lab, Shanghai, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8132-9083","authenticated-orcid":false,"given":"Xing","family":"Sun","sequence":"additional","affiliation":[{"name":"Tencent YouTu Lab, Shanghai, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7680-6401","authenticated-orcid":false,"given":"Muhan","family":"Zhang","sequence":"additional","affiliation":[{"name":"Peking University, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2026,3,22]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"AI@Meta. 2024. Llama 3 Model Card. https:\/\/github.com\/meta-llama\/llama3\/blob\/main\/MODEL_CARD.md"},{"key":"e_1_3_2_1_2_1","volume-title":"Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245","author":"Ainslie Joshua","year":"2023","unstructured":"Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebr\u00f3n, and Sumit Sanghai. 2023. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245 (2023)."},{"key":"e_1_3_2_1_3_1","unstructured":"Anthropic. 2024. Claude 3.5 Sonnet. https:\/\/www.anthropic.com\/news\/claude-3-5-sonnet"},{"key":"e_1_3_2_1_4_1","volume-title":"Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508","author":"Bai Yushi","year":"2023","unstructured":"Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al., 2023. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508 (2023)."},{"key":"e_1_3_2_1_5_1","volume-title":"https:\/\/huggingface.co\/datasets\/HuggingFaceTB\/smollm-corpus","author":"Allal Loubna Ben","year":"2024","unstructured":"Loubna Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, and Leandro von Werra. 2024. SmolLM-Corpus. (2024). https:\/\/huggingface.co\/datasets\/HuggingFaceTB\/smollm-corpus"},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1609\/AAAI.V34I05.6239"},{"key":"e_1_3_2_1_7_1","volume-title":"MatryoshkaKV: Adaptive KV Compression via Trainable Orthogonal Projection. arXiv preprint arXiv:2410.14731","author":"Bokai Lin","year":"2024","unstructured":"Lin Bokai, Zeng Zihao, Xiao Zipeng, Kou Siqi, Hou Tianqi, Gao Xiaofeng, Zhang Hao, and Deng Zhijie. 2024. MatryoshkaKV: Adaptive KV Compression via Trainable Orthogonal Projection. arXiv preprint arXiv:2410.14731 (2024). https:\/\/www.arxiv.org\/abs\/2410.14731"},{"key":"e_1_3_2_1_8_1","unstructured":"Tom Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared D Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell et al. 2020. Language models are few-shot learners. Advances in neural information processing systems Vol. 33 (2020) 1877-1901."},{"key":"e_1_3_2_1_9_1","volume-title":"xKV: Cross-Layer SVD for KV-Cache Compression. arXiv preprint arXiv:2503.18893","author":"Chi-Chih Chang","year":"2025","unstructured":"Chang Chi-Chih, Lin Chien-Yu, Akhauri Yash, Lin Wei-Cheng, Wu Kai-Chiang, Ceze Luis, and Abdelfattah Mohamed, S., 2025. xKV: Cross-Layer SVD for KV-Cache Compression. arXiv preprint arXiv:2503.18893 (2025). https:\/\/www.arxiv.org\/abs\/2503.18893"},{"key":"e_1_3_2_1_10_1","volume-title":"Palu: Compressing KV-Cache with Low-Rank Projection. arXiv preprint arXiv:2407.21118","author":"Chi-Chih Chang","year":"2024","unstructured":"Chang Chi-Chih, Lin Wei-Cheng, Lin Chien-Yu, Chen Chong-Yan, Hu Yu-Fang, Wang Pei-Shuo, Huang Ning-Chi, Ceze Luis, Abdelfattah Mohamed, S., and Wu and, Kai-Chiang. 2024. Palu: Compressing KV-Cache with Low-Rank Projection. arXiv preprint arXiv:2407.21118 (2024). https:\/\/www.arxiv.org\/abs\/2407.21118"},{"key":"e_1_3_2_1_11_1","volume-title":"Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. CoRR","author":"Clark Peter","year":"2018","unstructured":"Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. CoRR, Vol. abs\/1803.05457 (2018). arXiv:1803.05457 http:\/\/arxiv.org\/abs\/1803.05457"},{"key":"e_1_3_2_1_12_1","volume-title":"KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization. arXiv preprint arXiv:2401.18079","author":"Coleman Hooper","year":"2024","unstructured":"Hooper Coleman, Kim Sehoon, Mohammadzadeh Hiva, Mahoney Michael, W., Shao Yakun, Sophia, Keutzer Kurt, and Gholami Amir. 2024. KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization. arXiv preprint arXiv:2401.18079 (2024). https:\/\/www.arxiv.org\/abs\/2401.18079"},{"key":"e_1_3_2_1_13_1","unstructured":"Jeffrey Dean Greg Corrado Rajat Monga Kai Chen Matthieu Devin Mark Mao Marc'aurelio Ranzato Andrew Senior Paul Tucker Ke Yang et al. 2012. Large scale distributed deep networks. Advances in neural information processing systems Vol. 25 (2012)."},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2401.02954"},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2405.04434"},{"key":"e_1_3_2_1_16_1","volume-title":"HGCA: Hybrid GPU-CPU Attention for Long Context LLM Inference. arXiv:2507.03153 [cs.LG] https:\/\/arxiv.org\/abs\/2507.03153","author":"Deng Weishu","year":"2025","unstructured":"Weishu Deng, Yujie Yang, Peiran Du, Lingfeng Xiang, Zhen Lin, Chen Zhong, Song Jiang, Hui Lu, and Jia Rao. 2025. HGCA: Hybrid GPU-CPU Attention for Long Context LLM Inference. arXiv:2507.03153 [cs.LG] https:\/\/arxiv.org\/abs\/2507.03153"},{"key":"e_1_3_2_1_17_1","volume-title":"TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization. arXiv preprint arXiv:2505.19586","author":"Dingyu Yao","year":"2025","unstructured":"Yao Dingyu, Shen Bowen, Lin Zheng, Liu Wei, Luan Jian, Wang Bin, and Wang Weiping. 2025. TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization. arXiv preprint arXiv:2505.19586 (2025). https:\/\/www.arxiv.org\/abs\/2505.19586"},{"key":"e_1_3_2_1_18_1","unstructured":"Amr Elmeleegy Harry Kim David Zier Kyle Kranen Neelay Shah Ryan Olson and Omri Kahalon. 2025. NVIDIA Dynamo A Low-Latency Distributed Inference Framework for Scaling Reasoning AI Models. NVIDIA Developer Blog. https:\/\/developer.nvidia.com\/blog\/introducing-nvidia-dynamo-a-low-latency-distributed-inference-framework-for-scaling-reasoning-ai-models Published March 18 2025."},{"key":"e_1_3_2_1_19_1","volume-title":"SPD: Sync-Point Drop for efficient tensor parallelism of Large Language Models. arXiv preprint arXiv:2502.20727","author":"Han-Byul Kim","year":"2025","unstructured":"Kim Han-Byul, Hoang Duc, Kundu Arnav, Samragh Mohammad, and Cho Minsik. 2025. SPD: Sync-Point Drop for efficient tensor parallelism of Large Language Models. arXiv preprint arXiv:2502.20727 (2025). https:\/\/www.arxiv.org\/abs\/2502.20727"},{"key":"e_1_3_2_1_20_1","volume-title":"Effectively Compress KV Heads for LLM. arXiv preprint arXiv:2406.07056","author":"Hao Yu","year":"2024","unstructured":"Yu Hao, Yang Zelan, Li Shen, Li Yong, and Wu Jianxin. 2024. Effectively Compress KV Heads for LLM. arXiv preprint arXiv:2406.07056 (2024). https:\/\/www.arxiv.org\/abs\/2406.07056"},{"key":"e_1_3_2_1_21_1","volume-title":"Layer-Condensed KV Cache for Efficient Inference of Large Language Models. arXiv preprint arXiv:2405.10637","author":"Haoyi Wu","year":"2024","unstructured":"Wu Haoyi and Tu Kewei. 2024. Layer-Condensed KV Cache for Efficient Inference of Large Language Models. arXiv preprint arXiv:2405.10637 (2024). https:\/\/www.arxiv.org\/abs\/2405.10637"},{"key":"e_1_3_2_1_22_1","volume-title":"Measuring Massive Multitask Language Understanding. In 9th International Conference on Learning Representations, ICLR 2021","author":"Hendrycks Dan","year":"2021","unstructured":"Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. https:\/\/openreview.net\/forum?id=d7KBjmI3GmQ"},{"key":"e_1_3_2_1_23_1","volume-title":"RULER: What's the Real Context Size of Your Long-Context Language Models? arXiv:2404.06654 [cs.CL] https:\/\/arxiv.org\/abs\/2404.06654","author":"Hsieh Cheng-Ping","year":"2024","unstructured":"Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. RULER: What's the Real Context Size of Your Long-Context Language Models? arXiv:2404.06654 [cs.CL] https:\/\/arxiv.org\/abs\/2404.06654"},{"key":"e_1_3_2_1_24_1","volume-title":"Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems","author":"Huang Yanping","year":"2019","unstructured":"Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al., 2019. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, Vol. 32 (2019)."},{"key":"e_1_3_2_1_25_1","volume-title":"Tensor-Parallelism with Partially Synchronized Activations. arXiv preprint arXiv:2506.19645v1","author":"Itay Lamprecht","year":"2025","unstructured":"Lamprecht Itay, Karnieli Asaf, Hanani Yair, Giladi Niv, and Soudry Daniel. 2025. Tensor-Parallelism with Partially Synchronized Activations. arXiv preprint arXiv:2506.19645v1 (2025). https:\/\/www.arxiv.org\/abs\/2506.19645v1"},{"key":"e_1_3_2_1_26_1","volume-title":"Efficient Long-Context LLM Inference via KV Cache Clustering. arXiv preprint arXiv:2506.11418","author":"Jie Hu","year":"2025","unstructured":"Hu Jie, Wang Shengnan, He Yutong, Gong Ping, Yi Jiawei, Zhang Juncheng, Bai Youhui, Chen Renhai, Zhang Gong, Li Cheng, and Yuan Kun. 2025. Efficient Long-Context LLM Inference via KV Cache Clustering. arXiv preprint arXiv:2506.11418 (2025). https:\/\/www.arxiv.org\/abs\/2506.11418"},{"key":"e_1_3_2_1_27_1","unstructured":"Jinhyuk Lee Anthony Chen Zhuyun Dai Dheeru Dua Devendra Singh Sachan Michael Boratko Yi Luan S\u00e9bastien M. R. Arnold Vincent Perot Siddharth Dalmia Hexiang Hu Xudong Lin Panupong Pasupat Aida Amini Jeremy R. Cole Sebastian Riedel Iftekhar Naim Ming-Wei Chang and Kelvin Guu. 2024. Can Long-Context Language Models Subsume Retrieval RAG SQL and More? arXiv:2406.13121 [cs.CL] https:\/\/arxiv.org\/abs\/2406.13121"},{"key":"e_1_3_2_1_28_1","unstructured":"Aixin Liu Bei Feng Bing Xue Bingxuan Wang Bochao Wu Chengda Lu Chenggang Zhao Chengqi Deng Chenyu Zhang Chong Ruan et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 (2024)."},{"key":"e_1_3_2_1_29_1","volume-title":"Transformers are Multi-State RNNs. arXiv preprint arXiv:2401.06104","author":"Matanel Oren","year":"2024","unstructured":"Oren Matanel, Hassid Michael, Yarden Nir, Adi Yossi, and Schwartz Roy. 2024. Transformers are Multi-State RNNs. arXiv preprint arXiv:2401.06104 (2024). https:\/\/www.arxiv.org\/abs\/2401.06104"},{"key":"e_1_3_2_1_30_1","volume-title":"TransMLA: Multi-head Latent Attention Is All You Need. arXiv preprint arXiv:2502.07864","author":"Meng Fanxu","year":"2025","unstructured":"Fanxu Meng, Pingzhi Tang, Zengwei Yao, and Muhan Zhang. 2025. TransMLA: Multi-head Latent Attention Is All You Need. arXiv preprint arXiv:2502.07864 (2025)."},{"key":"e_1_3_2_1_31_1","unstructured":"Stephen Merity Caiming Xiong James Bradbury and Richard Socher. 2016. Pointer Sentinel Mixture Models. arXiv:1609.07843 [cs.CL]"},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/D18-1260"},{"key":"e_1_3_2_1_33_1","volume-title":"Ladder-residual: parallelism-aware architecture for accelerating large model inference with communication overlapping. arXiv preprint arXiv:2501.06589","author":"Muru Zhang","year":"2025","unstructured":"Zhang Muru, Mishra Mayank, Zhou Zhongzhu, Brandon William, Wang Jue, Kim Yoon, Ragan-Kelley Jonathan, Song Shuaiwen, Leon, Athiwaratkun Ben, and Dao Tri. 2025. Ladder-residual: parallelism-aware architecture for accelerating large model inference with communication overlapping. arXiv preprint arXiv:2501.06589 (2025). https:\/\/www.arxiv.org\/abs\/2501.06589"},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3341301.3359646"},{"key":"e_1_3_2_1_35_1","unstructured":"OpenAI. 2024. Hello GPT-4o. https:\/\/openai.com\/index\/hello-gpt-4o\/"},{"key":"e_1_3_2_1_36_1","volume-title":"LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inference. arXiv preprint arXiv:2407.14057","author":"Qichen Fu","year":"2024","unstructured":"Fu Qichen, Cho Minsik, Merth Thomas, Mehta Sachin, Rastegari Mohammad, and Najibi Mahyar. 2024. LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inference. arXiv preprint arXiv:2407.14057 (2024). https:\/\/www.arxiv.org\/abs\/2407.14057"},{"key":"e_1_3_2_1_37_1","volume-title":"An Efficient 2D Method for Training Super-Large Deep Learning Models. arXiv preprint arXiv:2104.05343","author":"Qifan Xu","year":"2021","unstructured":"Xu Qifan, Li Shenggui, Gong Chaoyu, and You Yang. 2021. An Efficient 2D Method for Training Super-Large Deep Learning Models. arXiv preprint arXiv:2104.05343 (2021). https:\/\/www.arxiv.org\/abs\/2104.05343"},{"key":"e_1_3_2_1_38_1","volume-title":"Flash Communication: Reducing Tensor Parallelization Bottleneck for Fast Large Language Model Inference. arXiv preprint arXiv:2412.04964","author":"Qingyuan Li","year":"2024","unstructured":"Li Qingyuan, Zhang Bo, Ye Liang, Zhang Yifan, Wu Wei, Sun Yerui, Ma Lin, and Xie Yuchen. 2024. Flash Communication: Reducing Tensor Parallelization Bottleneck for Fast Large Language Model Inference. arXiv preprint arXiv:2412.04964 (2024). https:\/\/www.arxiv.org\/abs\/2412.04964"},{"key":"e_1_3_2_1_39_1","unstructured":"Alec Radford. 2018. Improving language understanding by generative pre-training. (2018)."},{"key":"e_1_3_2_1_40_1","volume-title":"LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy. arXiv preprint arXiv:2410.03111","author":"Rongzhi Zhang","year":"2024","unstructured":"Zhang Rongzhi, Wang Kuang, Liu Liyuan, Wang Shuohang, Cheng Hao, Zhang Chao, and Shen Yelong. 2024. LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy. arXiv preprint arXiv:2410.03111 (2024). https:\/\/www.arxiv.org\/abs\/2410.03111"},{"key":"e_1_3_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3474381"},{"key":"e_1_3_2_1_42_1","volume-title":"Horovod: fast and easy distributed deep learning in TensorFlow. arXiv preprint arXiv:1802.05799","author":"Sergeev Alexander","year":"2018","unstructured":"Alexander Sergeev and Mike Del Balso. 2018. Horovod: fast and easy distributed deep learning in TensorFlow. arXiv preprint arXiv:1802.05799 (2018)."},{"key":"e_1_3_2_1_43_1","volume-title":"A Large-Scale Generative Language Model. arXiv preprint arXiv:2201.11990","author":"Shaden Smith","year":"2022","unstructured":"Smith Shaden, Patwary Mostofa, Norick Brandon, LeGresley Patrick, Rajbhandari Samyam, Casper Jared, Liu Zhun, Prabhumoye Shrimai, Zerveas George, Korthikanti Vijay, Zhang Elton, Child Rewon, Aminabadi Reza, Yazdani, Bernauer Julie, Song Xia, Shoeybi Mohammad, He Yuxiong, Houston Michael, Tiwary Saurabh, and Catanzaro and, Bryan. 2022. Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model. arXiv preprint arXiv:2201.11990 (2022). https:\/\/www.arxiv.org\/abs\/2201.11990"},{"key":"e_1_3_2_1_44_1","volume-title":"Inference-Friendly Models With MixAttention. arXiv preprint arXiv:2409.15012","author":"Shashank Rajput","year":"2024","unstructured":"Rajput Shashank, Sheng Ying, Owen Sean, and Chiley Vitaliy. 2024. Inference-Friendly Models With MixAttention. arXiv preprint arXiv:2409.15012 (2024). https:\/\/www.arxiv.org\/abs\/2409.15012"},{"key":"e_1_3_2_1_45_1","volume-title":"Sequence Parallelism: Long Sequence Training from System Perspective. arXiv preprint arXiv:2105.13120","author":"Shenggui Li","year":"2021","unstructured":"Li Shenggui, Xue Fuzhao, Baranwal Chaitanya, Li Yongbin, and You Yang. 2021. Sequence Parallelism: Long Sequence Training from System Perspective. arXiv preprint arXiv:2105.13120 (2021). https:\/\/www.arxiv.org\/abs\/2105.13120"},{"key":"e_1_3_2_1_46_1","volume-title":"QAQ: Quality Adaptive Quantization for LLM KV Cache. arXiv preprint arXiv:2403.04643","author":"Shichen Dong","year":"2024","unstructured":"Dong Shichen, Cheng Wen, Qin Jiayu, and Wang Wei. 2024. QAQ: Quality Adaptive Quantization for LLM KV Cache. arXiv preprint arXiv:2403.04643 (2024). https:\/\/www.arxiv.org\/abs\/2403.04643"},{"key":"e_1_3_2_1_47_1","volume-title":"Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053","author":"Shoeybi Mohammad","year":"2019","unstructured":"Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053 (2019)."},{"key":"e_1_3_2_1_48_1","volume-title":"Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs. arXiv preprint arXiv:2310.01801","author":"Suyu Ge","year":"2023","unstructured":"Ge Suyu, Zhang Yunan, Liu Liyuan, Zhang Minjia, Han Jiawei, and Gao Jianfeng. 2023. Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs. arXiv preprint arXiv:2310.01801 (2023). https:\/\/www.arxiv.org\/abs\/2310.01801"},{"key":"e_1_3_2_1_49_1","volume-title":"Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al.","author":"Team Gemini","year":"2024","unstructured":"Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al., 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530 (2024)."},{"key":"e_1_3_2_1_50_1","unstructured":"Kimi Team Yifan Bai Yiping Bao Guanduo Chen Jiahao Chen Ningxin Chen Ruijue Chen Yanru Chen Yuankun Chen Yutian Chen et al. 2025. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534 (2025)."},{"key":"e_1_3_2_1_51_1","volume-title":"Hardware-Efficient Attention for Fast Decoding. arXiv preprint arXiv:2505.21487v1","author":"Ted Zadouri","year":"2025","unstructured":"Zadouri Ted, Strauss Hubert, and Dao Tri. 2025. Hardware-Efficient Attention for Fast Decoding. arXiv preprint arXiv:2505.21487v1 (2025). https:\/\/www.arxiv.org\/abs\/2505.21487v1"},{"key":"e_1_3_2_1_52_1","volume-title":"Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention. arXiv preprint arXiv:2404.07143","author":"Tsendsuren Munkhdalai","year":"2024","unstructured":"Munkhdalai Tsendsuren and and Siddharth Gopal Manaal, Faruqui. 2024. Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention. arXiv preprint arXiv:2404.07143 (2024). https:\/\/www.arxiv.org\/abs\/2404.07143"},{"key":"e_1_3_2_1_53_1","volume-title":"Reducing Transformer Key-Value Cache Size with Cross-Layer Attention. arXiv preprint arXiv:2405.12981","author":"William Brandon","year":"2024","unstructured":"Brandon William, Mishra Mayank, Nrusimha Aniruddha, Panda Rameswar, and Kelly Jonathan, Ragan. 2024. Reducing Transformer Key-Value Cache Size with Cross-Layer Attention. arXiv preprint arXiv:2405.12981 (2024). https:\/\/www.arxiv.org\/abs\/2405.12981"},{"key":"e_1_3_2_1_54_1","volume-title":"DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMs. arXiv preprint arXiv:2412.14838","author":"Xiabin Zhou","year":"2024","unstructured":"Zhou Xiabin, Wang Wenbin, Zeng Minyan, Guo Jiaxian, Liu Xuebo, Shen Li, Zhang Min, and Ding Liang. 2024. DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMs. arXiv preprint arXiv:2412.14838 (2024). https:\/\/www.arxiv.org\/abs\/2412.14838"},{"key":"e_1_3_2_1_55_1","volume-title":"CompressKV: Semantic Retrieval Heads Know What Tokens are Not Important Before Generation. arXiv preprint arXiv:2508.02401v1","author":"Xiaolin Lin","year":"2025","unstructured":"Lin Xiaolin, Wang Jingcun, Kondrateva Olga, Shi Yiyu, Li Bing, and Zhang Grace, Li. 2025. CompressKV: Semantic Retrieval Heads Know What Tokens are Not Important Before Generation. arXiv preprint arXiv:2508.02401v1 (2025). https:\/\/www.arxiv.org\/abs\/2508.02401v1"},{"key":"e_1_3_2_1_56_1","volume-title":"ZSMerge: Zero-Shot KV Cache Compression for Memory-Efficient Long-Context LLMs. arXiv preprint arXiv:2503.10714","author":"Xin Liu","year":"2025","unstructured":"Liu Xin, Liu Pei, and Tang Guoming. 2025. ZSMerge: Zero-Shot KV Cache Compression for Memory-Efficient Long-Context LLMs. arXiv preprint arXiv:2503.10714 (2025). https:\/\/www.arxiv.org\/abs\/2503.10714"},{"key":"e_1_3_2_1_57_1","unstructured":"Amy Yang Jingyi Yang Aya Ibrahim Xinfeng Xie Bangsheng Tang Grigory Sizov Jeremy Reizenstein Jongsoo Park and Jianyu Huang. 2025. Context Parallelism for Scalable Million-Token Inference. arXiv:2411.01783 [cs.DC] https:\/\/arxiv.org\/abs\/2411.01783"},{"key":"e_1_3_2_1_58_1","volume-title":"KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing. arXiv preprint arXiv:2410.18517","author":"Yifei Yang","year":"2024","unstructured":"Yang Yifei, Cao Zouying, Chen Qiguang, Qin Libo, Yang Dongjie, Zhao Hai, and Chen Zhi. 2024. KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing. arXiv preprint arXiv:2410.18517 (2024). https:\/\/www.arxiv.org\/abs\/2410.18517"},{"key":"e_1_3_2_1_59_1","volume-title":"A Systematic Study of Cross-Layer KV Sharing for Efficient LLM Inference. arXiv preprint arXiv:2410.14442","author":"You Wu","year":"2024","unstructured":"Wu You, Wu Haoyi, and Tu Kewei. 2024. A Systematic Study of Cross-Layer KV Sharing for Efficient LLM Inference. arXiv preprint arXiv:2410.14442 (2024). https:\/\/www.arxiv.org\/abs\/2410.14442"},{"key":"e_1_3_2_1_60_1","volume-title":"SnapKV: LLM Knows What You are Looking for Before Generation. arXiv preprint arXiv:2404.14469","author":"Yuhong Li","year":"2024","unstructured":"Li Yuhong, Huang Yingbing, Yang Bowen, Venkitesh Bharat, Locatelli Acyr, Ye Hanchen, Cai Tianle, Lewis Patrick, and Chen Deming. 2024. SnapKV: LLM Knows What You are Looking for Before Generation. arXiv preprint arXiv:2404.14469 (2024). https:\/\/www.arxiv.org\/abs\/2404.14469"},{"key":"e_1_3_2_1_61_1","volume-title":"Jongmin Kim, Hyungyo Kim, Juhwan Cho, Seungmin Baek, and Jung Ho Ahn.","author":"Yun Sungmin","year":"2025","unstructured":"Sungmin Yun, Seonyong Park, Hwayong Nam, Younjoo Lee, Gunjun Lee, Kwanhee Kyung, Sangpyo Kim, Nam Sung Kim, Jongmin Kim, Hyungyo Kim, Juhwan Cho, Seungmin Baek, and Jung Ho Ahn. 2025. The New LLM Bottleneck: A Systems Perspective on Latent Attention and Mixture-of-Experts. arXiv:2507.15465 [cs.AR] https:\/\/arxiv.org\/abs\/2507.15465"},{"key":"e_1_3_2_1_62_1","volume-title":"MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decoding. arXiv preprint arXiv:2406.09297","author":"Zayd Kawakibi Zuhri","year":"2024","unstructured":"Kawakibi Zuhri Zayd, Muhammad, Adilazuarda Muhammad, Farid, Purwarianti Ayu, and Aji Alham, Fikri. 2024. MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decoding. arXiv preprint arXiv:2406.09297 (2024). https:\/\/www.arxiv.org\/abs\/2406.09297"},{"key":"e_1_3_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/P19-1472"},{"key":"e_1_3_2_1_64_1","volume-title":"Sandwich: Separating Prefill-Decode Compilation for Efficient CPU LLM Serving. arXiv preprint arXiv:2507.18454","author":"Zhao Juntao","year":"2025","unstructured":"Juntao Zhao, Jiuru Li, and Chuan Wu. 2025. Sandwich: Separating Prefill-Decode Compilation for Efficient CPU LLM Serving. arXiv preprint arXiv:2507.18454 (2025)."},{"key":"e_1_3_2_1_65_1","volume-title":"Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Tasks. arXiv preprint arXiv:2407.08454","author":"Zheng Wang","year":"2024","unstructured":"Wang Zheng, Jin Boxiao, Yu Zhongzhi, and Zhang Minjia. 2024. Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Tasks. arXiv preprint arXiv:2407.08454 (2024). https:\/\/www.arxiv.org\/abs\/2407.08454"},{"key":"e_1_3_2_1_66_1","volume-title":"Maximizing Parallelism in Distributed Training for Huge Neural Networks. arXiv preprint arXiv:2105.14450","author":"Zhengda Bian","year":"2021","unstructured":"Bian Zhengda, Xu Qifan, Wang Boxiang, and You Yang. 2021. Maximizing Parallelism in Distributed Training for Huge Neural Networks. arXiv preprint arXiv:2105.14450 (2021). https:\/\/www.arxiv.org\/abs\/2105.14450"},{"key":"e_1_3_2_1_67_1","volume-title":"HO: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models. arXiv preprint arXiv:2306.14048","author":"Zhenyu Zhang","year":"2023","unstructured":"Zhang Zhenyu, Sheng Ying, Zhou Tianyi, Chen Tianlong, Zheng Lianmin, Cai Ruisi, Song Zhao, Tian Yuandong, R\u00e9 Christopher, Barrett Clark, Wang Zhangyang, and Chen Beidi. 2023. HO: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models. arXiv preprint arXiv:2306.14048 (2023). https:\/\/www.arxiv.org\/abs\/2306.14048"},{"key":"e_1_3_2_1_68_1","first-page":"193","volume-title":"18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24)","author":"Zhong Yinmin","year":"2024","unstructured":"Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. 2024. : Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). 193-210."},{"key":"e_1_3_2_1_69_1","volume-title":"KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache. arXiv preprint arXiv:2402.02750","author":"Zirui Liu","year":"2024","unstructured":"Liu Zirui, Yuan Jiayi, Jin Hongye, Zhong Shaochen, Xu Zhaozhuo, Braverman Vladimir, Chen Beidi, and Hu Xia. 2024. KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache. arXiv preprint arXiv:2402.02750 (2024). https:\/\/www.arxiv.org\/abs\/2402.02750"},{"key":"e_1_3_2_1_70_1","volume-title":"MILLION: Mastering Long-Context LLM Inference Via Outlier-Immunized KV Product Quantization. arXiv preprint arXiv:2504.03661","author":"Zongwu Wang","year":"2025","unstructured":"Wang Zongwu, Xu Peng, Liu Fangxin, Hu Yiwei, Sun Qingxiao, Li Gezi, Li Cheng, Wang Xuan, Jiang Li, and Guan Haibing. 2025. MILLION: Mastering Long-Context LLM Inference Via Outlier-Immunized KV Product Quantization. arXiv preprint arXiv:2504.03661 (2025). https:\/\/www.arxiv.org\/abs\/2504.03661"}],"event":{"name":"ASPLOS '26: 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems","location":"Pittsburgh PA USA","sponsor":["SIGOPS ACM Special Interest Group on Operating Systems","SIGPLAN ACM Special Interest Group on Programming Languages","SIGARCH ACM Special Interest Group on Computer Architecture","SIGBED ACM Special Interest Group on Embedded Systems"]},"container-title":["Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2"],"original-title":[],"deposited":{"date-parts":[[2026,3,15]],"date-time":"2026-03-15T13:58:41Z","timestamp":1773583121000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3779212.3790237"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,3,22]]},"references-count":70,"alternative-id":["10.1145\/3779212.3790237","10.1145\/3779212"],"URL":"https:\/\/doi.org\/10.1145\/3779212.3790237","relation":{},"subject":[],"published":{"date-parts":[[2026,3,22]]},"assertion":[{"value":"2026-03-22","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}