{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T15:53:16Z","timestamp":1782834796139,"version":"3.54.5"},"reference-count":72,"publisher":"Association for Computing Machinery (ACM)","issue":"4","funder":[{"name":"Nanjing U35 Talent Cultivation Program","award":["No. U (2024) 001"],"award-info":[{"award-number":["No. U (2024) 001"]}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["NO.62325205, 62072230"],"award-info":[{"award-number":["NO.62325205, 62072230"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Postgraduate Practice & Research Innovation Program of Jiangsu Province","award":["NJU KYCX24_025"],"award-info":[{"award-number":["NJU KYCX24_025"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Manag. Data"],"published-print":{"date-parts":[[2025,9,22]]},"abstract":"<jats:p>Prompt engineering techniques are widely used to enhance the generation quality of large language models (LLMs). However, the long prompts significantly increase inference latency and reduce inference throughput. Since many prompts share common prefixes, prefix sharing has been proposed to reuse shared prefix KV caches during inference. Nevertheless, the large number of prefix KV caches and the limited GPU memory capacity make it impractical to store all prefix KV caches in GPU memory. This limitation necessitates the use of external memory storage strategies, which often suffer from high I\/O overhead and frequent cache misses with traditional approaches. To address these challenges, this paper proposes HotPrefix, a hotness-aware KV cache scheduling framework designed for efficient prefix sharing in LLM inference systems. HotPrefix introduces three core innovations: (1) Dynamic Hotness Tracking, which dynamically monitors and updates the hotness of prefix tree nodes over time; (2) Selective KV Cache Admission, which evaluates evicted KV caches from GPU memory, retaining only high-hotness caches in CPU memory to expand GPU memory capacity and reduce KV cache transfer overhead; (3) Hotness Promotion, which periodically promotes high-hotness prefix tree KV caches from CPU memory to GPU memory. This is combined with an efficient pipeline strategy for I\/O and computation, ensuring GPU memory is allocated to the most critical prefixes while masking the I\/O overhead associated with KV cache transmission. These mechanisms significantly improve cache hit rates, reduce inference latency, and enhance throughput. Implemented in the SGLang framework, HotPrefix reduces inference latency and increases throughput by up to 2.25\u00d7 compared with vLLM with prefix sharing enabled. Against SGLang, it achieves up to 2\u00d7 latency reduction and throughput increase.<\/jats:p>","DOI":"10.1145\/3749168","type":"journal-article","created":{"date-parts":[[2025,9,23]],"date-time":"2025-09-23T17:17:03Z","timestamp":1758647823000},"page":"1-27","source":"Crossref","is-referenced-by-count":5,"title":["HotPrefix: Hotness-Aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems"],"prefix":"10.1145","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0009-0003-1994-8133","authenticated-orcid":false,"given":"Yuhang","family":"Li","sequence":"first","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, Jiangsu, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1565-9997","authenticated-orcid":false,"given":"Rong","family":"Gu","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, Jiangsu, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3154-3580","authenticated-orcid":false,"given":"Chengying","family":"Huan","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, Jiangsu, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9204-4075","authenticated-orcid":false,"given":"Zhibin","family":"Wang","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, Jiangsu, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-9556-5773","authenticated-orcid":false,"given":"Renjie","family":"Yao","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, Jiangsu, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2710-7628","authenticated-orcid":false,"given":"Chen","family":"Tian","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, Jiangsu, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6934-1685","authenticated-orcid":false,"given":"Guihai","family":"Chen","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, Jiangsu, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,9,23]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"Reyna Abhyankar Zijian He Vikranth Srivatsa Hao Zhang and Yiying Zhang. 2024. APIServe: Efficient API Support for Large-Language Model Inferencing. In arXiv preprint arXiv:2402.01869."},{"key":"e_1_2_1_2_1","volume-title":"Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.","author":"Achiam Josh","year":"2023","unstructured":"Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. In arXiv preprint arXiv:2303.08774."},{"key":"e_1_2_1_3_1","volume-title":"Proceedings of Machine Learning and Systems (MLSys). 114--127","author":"Adnan Muhammad","year":"2024","unstructured":"Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant Nair, Ilya Soloveychik, and Purushotham Kamath. 2024. Keyformer: Kv cache reduction through key tokens selection for efficient generative inference. In Proceedings of Machine Learning and Systems (MLSys). 114--127."},{"key":"e_1_2_1_4_1","unstructured":"Anthropic. 2025. https:\/\/www.anthropic.com"},{"key":"e_1_2_1_5_1","unstructured":"Appleby and Austin. 2008. https:\/\/github.com\/aappleby\/smhasher"},{"key":"e_1_2_1_6_1","unstructured":"Xiao Bi Deli Chen Guanting Chen Shanhuang Chen Damai Dai Chengqi Deng Honghui Ding Kai Dong Qiushi Du Zhe Fu et al. 2024. Deepseek llm: Scaling open-source language models with longtermism. In arXiv preprint arXiv:2401.02954."},{"key":"e_1_2_1_7_1","volume-title":"Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). 1877--1901","author":"Brown Tom","year":"2020","unstructured":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). 1877--1901."},{"key":"e_1_2_1_8_1","unstructured":"Josiah L. Carlson. 2013. Redis in Action."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v38i16.29728"},{"key":"e_1_2_1_10_1","volume-title":"Chatlaw: Open-source legal large language model with integrated external knowledge bases. In arXiv preprint arXiv:2306.16092.","author":"Cui Jiaxi","year":"2023","unstructured":"Jiaxi Cui, Zongjian Li, Yang Yan, Bohua Chen, and Li Yuan. 2023. Chatlaw: Open-source legal large language model with integrated external knowledge bases. In arXiv preprint arXiv:2306.16092."},{"key":"e_1_2_1_11_1","volume-title":"QAQ: Quality Adaptive Quantization for LLM KV Cache. In arXiv preprint arXiv:2403.04643.","author":"Dong Shichen","year":"2024","unstructured":"Shichen Dong, Wen Cheng, Jiayu Qin, and Wei Wang. 2024. QAQ: Quality Adaptive Quantization for LLM KV Cache. In arXiv preprint arXiv:2403.04643."},{"key":"e_1_2_1_12_1","unstructured":"Abhimanyu Dubey Abhinav Jauhri Abhinav Pandey Abhishek Kadian Ahmad Al-Dahle Aiesha Letman Akhil Mathur Alan Schelten Amy Yang Angela Fan et al. 2024. The llama 3 herd of models. In arXiv preprint arXiv:2407.21783."},{"key":"e_1_2_1_13_1","volume-title":"Proceedings of ACM SIGCOMM Conference (SIGCOMM). 38--56","author":"Yuhan","unstructured":"Yuhan Liu et al. 2024. Cachegen: Kv cache compression and streaming for fast large language model serving. In Proceedings of ACM SIGCOMM Conference (SIGCOMM). 38--56."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/2674005.2674994"},{"key":"e_1_2_1_15_1","unstructured":"Yuan Feng Hyeran Jeon Filip Blagojevic Cyril Guyot Qing Li and Dong Li. 2023. AttMEMO:Accelerating Transformers with Memoization on Big Memory Systems. In arXiv preprint arXiv:2301.09262."},{"key":"e_1_2_1_16_1","unstructured":"Brad Fitzpatrick. 2003. Memcached Official Documentation. https:\/\/memcached.org"},{"key":"e_1_2_1_17_1","volume-title":"Proceedings of the USENIX Annual Technical Conference (USENIX ATC). 111--126","author":"Gao Bin","year":"2024","unstructured":"Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, and Pengfei Zuo. 2024. Cost-Efficient large language model serving for multi-turn conversations with CachedAttention. In Proceedings of the USENIX Annual Technical Conference (USENIX ATC). 111--126."},{"key":"e_1_2_1_18_1","unstructured":"Suyu Ge Yunan Zhang Liyuan Liu Minjia Zhang Jiawei Han and Jianfeng Gao. 2023. Model tells you what to discard: Adaptive kv cache compression for llms. In arXiv preprint arXiv:2310.01801."},{"key":"e_1_2_1_19_1","volume-title":"Proceedings of Machine Learning and Systems (MLSys). 325--338","author":"Gim In","year":"2024","unstructured":"In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong. 2024. Prompt cache: Modular attention reuse for low-latency inference. In Proceedings of Machine Learning and Systems (MLSys). 325--338."},{"key":"e_1_2_1_20_1","volume-title":"Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). 15908--15919","author":"Han Kai","year":"2021","unstructured":"Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. 2021. Transformer in transformer. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). 15908--15919."},{"key":"e_1_2_1_21_1","unstructured":"Dan Hendrycks Collin Burns Steven Basart Andy Zou Mantas Mazeika Dawn Song and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Understanding. In arXiv preprint arXiv:2009.03300."},{"key":"e_1_2_1_22_1","volume-title":"Kurt Keutzer, and Amir Gholami.","author":"Hooper Coleman","year":"2024","unstructured":"Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. 2024. Kvquant: Towards 10 million context length llm inference with kv cache quantization. In arXiv preprint arXiv:2401.18079."},{"key":"e_1_2_1_23_1","unstructured":"Yuzhen Huang Yuzhuo Bai Zhihao Zhu Junlei Zhang Jinghan Zhang Tangjun Su Junteng Liu Chuancheng Lv Yikai Zhang Jiayi Lei Yao Fu Maosong Sun and Junxian He. 2023. C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models. In arXiv preprint arXiv:2305.08322."},{"key":"e_1_2_1_24_1","volume-title":"Hossein Entezari Zarch, and Murali Annavaram","author":"Jiang Chaoyi","year":"2024","unstructured":"Chaoyi Jiang, Lei Gao, Hossein Entezari Zarch, and Murali Annavaram. 2024. Efficient llm inference with i\/o-aware partial kv cache recomputation. In arXiv preprint arXiv:2411.17089."},{"key":"e_1_2_1_25_1","doi-asserted-by":"crossref","unstructured":"Chao Jin Zili Zhang Xuanlin Jiang Fangyue Liu Xin Liu Xuanzhe Liu and Xin Jin. 2024. RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation. In arXiv preprint arXiv:2404.12457.","DOI":"10.1145\/3768628"},{"key":"e_1_2_1_26_1","unstructured":"Shuowei Jin Xueshen Liu Qingzhao Zhang and Z Morley Mao. 2024. Compute or load kv cache? why not both?. In arXiv preprint arXiv:2410.03065."},{"key":"e_1_2_1_27_1","volume-title":"Proceedings of the International Conference on Very Large Data Bases (VLDB). 439--450","author":"Johnson Theodore","year":"1994","unstructured":"Theodore Johnson, Dennis Shasha, et al. 1994. 2Q: a low overhead high performance bu er management replacement algorithm. In Proceedings of the International Conference on Very Large Data Bases (VLDB). 439--450."},{"key":"e_1_2_1_28_1","volume-title":"Collection of leaked system prompts","year":"2023","unstructured":"jujumilk3. Collection of leaked system prompts. 2023. https:\/\/github.com\/jujumilk3\/leaked-system-prompts."},{"key":"e_1_2_1_29_1","volume-title":"Gear: An efficient kv cache compression recipefor near-lossless generative inference of llm. In arXiv preprint arXiv:2403.05527.","author":"Kang Hao","year":"2024","unstructured":"Hao Kang, Qingru Zhang, Souvik Kundu, Geonhwa Jeong, Zaoxing Liu, Tushar Krishna, and Tuo Zhao. 2024. Gear: An efficient kv cache compression recipefor near-lossless generative inference of llm. In arXiv preprint arXiv:2403.05527."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3600006.3613165"},{"key":"e_1_2_1_31_1","volume-title":"proceedings of USENIX Symposium on Operating Systems Design and Implementation (SOSP). 155--172","author":"Lee Wonbeom","year":"2024","unstructured":"Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. 2024. InfiniGen: Efficient generative inference of large language models with dynamic KV cache management. In proceedings of USENIX Symposium on Operating Systems Design and Implementation (SOSP). 155--172."},{"key":"e_1_2_1_32_1","volume-title":"Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). 9459--9474","author":"Lewis Patrick","year":"2020","unstructured":"Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K\u00fcttler, Mike Lewis, Wen-tau Yih, Tim Rockt\u00e4schel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). 9459--9474."},{"key":"e_1_2_1_33_1","unstructured":"Chengshu Li Jacky Liang Andy Zeng Xinyun Chen Karol Hausman Dorsa Sadigh Sergey Levine Li Fei-Fei Fei Xia and Brian Ichter. 2023. Chain of code: Reasoning with a language model-augmented code emulator. In arXiv preprint arXiv:2312.04474."},{"key":"e_1_2_1_34_1","volume-title":"Snapkv: Llm knows what you are looking for before generation. In arXiv preprint arXiv:2404.14469.","author":"Li Yuhong","year":"2024","unstructured":"Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. 2024. Snapkv: Llm knows what you are looking for before generation. In arXiv preprint arXiv:2404.14469."},{"key":"e_1_2_1_35_1","unstructured":"Shu Liu Asim Biswal Audrey Cheng Xiangxi Mo Shiyi Cao Joseph E Gonzalez Ion Stoica and Matei Zaharia. 2024. Optimizing llm queries in relational workloads. In arXiv preprint arXiv:2403.05821."},{"key":"e_1_2_1_36_1","volume-title":"Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). 52342--52364","author":"Liu Zichang","year":"2024","unstructured":"Zichang Liu, Aditya Desai, Fangshuo Liao,WeitaoWang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. 2024. Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). 52342--52364."},{"key":"e_1_2_1_37_1","volume-title":"Kivi: A tuning-free asymmetric 2bit quantization for kv cache. In arXiv preprint arXiv:2402.02750.","author":"Liu Zirui","year":"2024","unstructured":"Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. 2024. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. In arXiv preprint arXiv:2402.02750."},{"key":"e_1_2_1_38_1","unstructured":"Robert Love. 2010. Linux kernel development."},{"key":"e_1_2_1_39_1","volume-title":"Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). 43447--43478","author":"Lu Pan","year":"2023","unstructured":"Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. 2023. Chameleon: Plug-and-play compositional reasoning with large language models. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). 43447--43478."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2004.1297303"},{"key":"e_1_2_1_41_1","unstructured":"Yuhong Mo Hao Qin Yushan Dong Ziyi Zhu and Zhenglin Li. 2024. Large language model (llm) ai text generation detection based on transformer deep learning algorithm. In arXiv preprint arXiv:2405.06652."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2024.3402809"},{"key":"e_1_2_1_43_1","volume-title":"Collection of chain-of-thought prompts","year":"2023","unstructured":"FranxYao. Collection of chain-of-thought prompts. 2023. https:\/\/github.com\/FranxYao\/chain-of-thought-hub"},{"key":"e_1_2_1_44_1","unstructured":"OpenAI. 2025. https:\/\/openai.com\/blog\/chatgpt"},{"key":"e_1_2_1_45_1","unstructured":"Matthew Prince. 2012. Cloudflare Architecture. https:\/\/www.cloudflare.com\/"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3387514.3405881"},{"key":"e_1_2_1_47_1","volume-title":"Mathbert: A pre-trained language model for general nlp tasks in mathematics education. In arXiv preprint arXiv:2106.07340.","author":"Shen Jia Tracy","year":"2021","unstructured":"Jia Tracy Shen, Michiharu Yamashita, Ethan Prihar, Neil Heffernan, XintaoWu, Ben Graff, and Dongwon Lee. 2021. Mathbert: A pre-trained language model for general nlp tasks in mathematics education. In arXiv preprint arXiv:2106.07340."},{"key":"e_1_2_1_48_1","volume-title":"Proceedings of International Conference on Machine Learning (ICML). 31094--31116","author":"Sheng Ying","year":"2023","unstructured":"Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher R\u00e9, Ion Stoica, and Ce Zhang. 2023. Flexgen: High-throughput generative inference of large language models with a single gpu. In Proceedings of International Conference on Machine Learning (ICML). 31094--31116."},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00530"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbi.2020.103637"},{"key":"e_1_2_1_51_1","volume-title":"Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, and Jason Wei.","author":"Suzgun Mirac","year":"2022","unstructured":"Mirac Suzgun, Nathan Scales, Nathanael Sch\u00e4rli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, and Jason Wei. 2022. Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them. In arXiv preprint arXiv:2210.09261."},{"key":"e_1_2_1_52_1","volume-title":"Juliette Love, et al.","author":"Team Gemma","year":"2024","unstructured":"Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi\u00e8re, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. In arXiv preprint arXiv:2403.08295."},{"key":"e_1_2_1_53_1","unstructured":"Hugo Touvron Louis Martin Kevin Stone Peter Albert Amjad Almahairi Yasmine Babaei Nikolay Bashlykov Soumya Batra Prajjwal Bhargava Shruti Bhosale et al. 2023. Llama 2: Open foundation and fine-tuned chat models. In arXiv preprint arXiv:2307.09288."},{"key":"e_1_2_1_54_1","unstructured":"Hugo Touvron Louis Martin Kevin Stone Peter Albert Amjad Almahairi Yasmine Babaei Nikolay Bashlykov Soumya Batra Prajjwal Bhargava Shruti Bhosale et al. 2023. Llama 2: Open foundation and fine-tuned chat models. In arXiv preprint arXiv:2307.09288."},{"key":"e_1_2_1_55_1","unstructured":"Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N Gomez Lukasz Kaiser and Illia Polosukhin. 2017. Attention is all you need. In arXiv preprint arXiv:1706.03762."},{"key":"e_1_2_1_56_1","volume-title":"Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). 24824--24837","author":"Wei Jason","year":"2022","unstructured":"Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). 24824--24837."},{"key":"e_1_2_1_57_1","volume-title":"Jesse Spencer- Smith, and Douglas C Schmidt","author":"White Jules","year":"2023","unstructured":"Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer- Smith, and Douglas C Schmidt. 2023. A prompt pattern catalog to enhance prompt engineering with chatgpt. In arXiv preprint arXiv:2302.11382."},{"key":"e_1_2_1_58_1","unstructured":"xAI. 2025. https:\/\/x.ai\/"},{"key":"e_1_2_1_59_1","unstructured":"Guangxuan Xiao Yuandong Tian Beidi Chen Song Han and Mike Lewis. 2023. Efficient streaming language models with attention sinks. In arXiv preprint arXiv:2309.17453."},{"key":"e_1_2_1_60_1","unstructured":"An Yang Baosong Yang Binyuan Hui Bo Zheng Bowen Yu Chang Zhou and Chengpeng Li. 2024. Qwen2 Technical Report. In arXiv preprint arXiv:2407.10671."},{"key":"e_1_2_1_61_1","doi-asserted-by":"crossref","unstructured":"Dongjie Yang XiaoDong Han Yan Gao Yao Hu Shilin Zhang and Hai Zhao. 2024. PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference. In arXiv preprint arXiv:2405.12532.","DOI":"10.18653\/v1\/2024.findings-acl.195"},{"key":"e_1_2_1_62_1","volume-title":"Se Jung Kwon, and Dongsoo Lee","author":"Yang June Yong","year":"2024","unstructured":"June Yong Yang, Byeongwook Kim, Jeongin Bae, Beomseok Kwon, Gunho Park, Eunho Yang, Se Jung Kwon, and Dongsoo Lee. 2024. No token left behind: Reliable kv cache compression via importance-aware mixed precision quantization. In arXiv preprint arXiv:2402.18096."},{"key":"e_1_2_1_63_1","unstructured":"Sherry Yang Ofir Nachum Yilun Du Jason Wei Pieter Abbeel and Dale Schuurmans. 2023. Foundation models for decision making: Problems methods and opportunities. In arXiv preprint arXiv:2303.04129."},{"key":"e_1_2_1_64_1","volume-title":"Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). 11809--11822","author":"Yao Shunyu","year":"2024","unstructured":"Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2024. Tree of thoughts: Deliberate problem solving with large language models. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). 11809--11822."},{"key":"e_1_2_1_65_1","volume-title":"React: Synergizing reasoning and acting in language models. In arXiv preprint arXiv:2210.03629.","author":"Yao Shunyu","year":"2022","unstructured":"Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. In arXiv preprint arXiv:2210.03629."},{"key":"e_1_2_1_66_1","volume-title":"Chunkattention: Efficient self-attention with prefix-aware kv cache and two-phase partition. In arXiv preprint arXiv:2402.15220.","author":"Ye Lu","year":"2024","unstructured":"Lu Ye, Ze Tao, Yong Huang, and Yang Li. 2024. Chunkattention: Efficient self-attention with prefix-aware kv cache and two-phase partition. In arXiv preprint arXiv:2402.15220."},{"key":"e_1_2_1_67_1","unstructured":"Lingfan Yu Jinkun Lin and Jinyang Li. 2023. Stateful large language model serving with pensieve. In arXiv preprint arXiv:2312.05516."},{"key":"e_1_2_1_68_1","volume-title":"Hellaswag: Can a machine really finish your sentence?. In arXiv preprint arXiv:1905.07830.","author":"Zellers Rowan","year":"2019","unstructured":"Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence?. In arXiv preprint arXiv:1905.07830."},{"key":"e_1_2_1_69_1","unstructured":"Tianyi Zhang Jonah Yi Zhaozhuo Xu and Anshumali Shrivastava. 2024. KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization. In arXiv preprint arXiv:2405.03917."},{"key":"e_1_2_1_70_1","volume-title":"Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). 34661--34710","author":"Zhang Zhenyu","year":"2023","unstructured":"Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher R\u00e9, Clark Barrett, et al. 2023. H2o: Heavy-hitter oracle for efficient generative inference of large language models. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). 34661--34710."},{"key":"e_1_2_1_71_1","volume-title":"Kun Chu, and Stefan Wermter.","author":"Zhao Xufeng","year":"2023","unstructured":"Xufeng Zhao, Mengdi Li, Wenhao Lu, Cornelius Weber, Jae Hee Lee, Kun Chu, and Stefan Wermter. 2023. Enhancing zero-shot chain-of-thought reasoning in large language models through logic. In arXiv preprint arXiv:2309.13339."},{"key":"e_1_2_1_72_1","volume-title":"Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al.","author":"Zheng Lianmin","year":"2024","unstructured":"Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. 2024. Sglang: Efficient execution of structured language model programs. In arXiv preprint arXiv:2312.07104."}],"container-title":["Proceedings of the ACM on Management of Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3749168","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,13]],"date-time":"2026-06-13T04:41:56Z","timestamp":1781325716000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3749168"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,9,22]]},"references-count":72,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2025,9,22]]}},"alternative-id":["10.1145\/3749168"],"URL":"https:\/\/doi.org\/10.1145\/3749168","relation":{},"ISSN":["2836-6573"],"issn-type":[{"value":"2836-6573","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,9,22]]}}}