{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,15]],"date-time":"2026-06-15T11:56:22Z","timestamp":1781524582535,"version":"3.54.1"},"reference-count":50,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2026,6,15]],"date-time":"2026-06-15T00:00:00Z","timestamp":1781481600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Intell. Syst. Technol."],"published-print":{"date-parts":[[2026,8,31]]},"abstract":"<jats:p>\n                    Large language models (LLMs) are critical in natural language processing but face challenges in inference speed and computational efficiency, hindering real-time applications. The key-value (KV) Cache mechanism helps reduce computational overhead in transformer models; however, efficient contextual understanding remains problematic. In this article, we introduce BUZZ, an innovative KV caching algorithm that leverages structured contextual information to optimize cache memory usage while enhancing inference speed. The core concept of BUZZ involves interval sampling of historically significant tokens to maintain sentence structure information, ensuring that KV Cache historical tokens are consistently distributed at nearly equal intervals. Tokens recently removed from the sliding window undergo local-max sampling based on attention values, preserving crucial contextual information. Additionally, we propose BUZZ with\n                    <jats:inline-formula content-type=\"math\/tex\">\n                      <jats:tex-math notation=\"LaTeX\" version=\"MathJax\">\\(\\log n\\)<\/jats:tex-math>\n                    <\/jats:inline-formula>\n                    , an extension that enhances performance under extreme compression and long-context settings. Evaluations on five real-world datasets\u2014CNN\/Daily Mail, XSUM, LongBench, Wikitext, and 10-QA\u2014demonstrate that BUZZ (1) achieves a\n                    <jats:italic toggle=\"yes\">\n                      2.5\n                      <jats:inline-formula content-type=\"math\/tex\">\n                        <jats:tex-math notation=\"LaTeX\" version=\"MathJax\">\\(\\times\\)<\/jats:tex-math>\n                      <\/jats:inline-formula>\n                    <\/jats:italic>\n                    reduction in cache memory usage for LLM inference while maintaining over 99% accuracy in long-text summarization and (2) surpasses state-of-the-art multi-document question answering by\n                    <jats:italic toggle=\"yes\">7.69%<\/jats:italic>\n                    under equivalent memory constraints, avoiding out-of-memory issues faced by full cache approaches. Furthermore, BUZZ achieves substantial inference speed improvements with a\n                    <jats:inline-formula content-type=\"math\/tex\">\n                      <jats:tex-math notation=\"LaTeX\" version=\"MathJax\">\\(\\log{n}\\)<\/jats:tex-math>\n                    <\/jats:inline-formula>\n                    time complexity. The implementation of BUZZ is available at:\n                    <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"uri\" xlink:href=\"https:\/\/github.com\/JunqiZhao888\/buzz-llm\">https:\/\/github.com\/JunqiZhao888\/buzz-llm<\/jats:ext-link>\n                    .\n                  <\/jats:p>","DOI":"10.1145\/3810942","type":"journal-article","created":{"date-parts":[[2026,5,4]],"date-time":"2026-05-04T13:32:08Z","timestamp":1777901528000},"page":"1-22","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["BUZZ: Beehive-structured Sparse KV Cache with Segmented Heavy Hitters for Efficient LLM Inference"],"prefix":"10.1145","volume":"17","author":[{"ORCID":"https:\/\/orcid.org\/0009-0003-4909-7851","authenticated-orcid":false,"given":"Junqi","family":"Zhao","sequence":"first","affiliation":[{"name":"Department of Computer Science, New York University, New York, New York, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-2573-5033","authenticated-orcid":false,"given":"Zhijin","family":"Fang","sequence":"additional","affiliation":[{"name":"Department of Statistics and Data Science, University of California Los Angeles, Los Angeles, California, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-0262-1248","authenticated-orcid":false,"given":"Shu","family":"Li","sequence":"additional","affiliation":[{"name":"Department of Mathematical Science, Beihang University, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-6389-0790","authenticated-orcid":false,"given":"Shaohui","family":"Yang","sequence":"additional","affiliation":[{"name":"School of Computer Science, University of Leeds, Leeds, United Kingdom of Great Britain and Northern Ireland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-0673-527X","authenticated-orcid":false,"given":"Shichao","family":"He","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Applied Math, Brandeis University, Waltham, Massachusetts, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,6,15]]},"reference":[{"key":"e_1_3_3_2_2","article-title":"Emergent abilities of large language models","author":"Wei Jason","year":"2022","unstructured":"Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022. Emergent abilities of large language models. Transactions on Machine Learning Research 2022 (2022). Retrieved from https:\/\/openreview.net\/forum?id=yzkSU5zdwD","journal-title":"Transactions on Machine Learning Research"},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639187"},{"key":"e_1_3_3_4_2","volume-title":"Advances in Neural Information Processing Systems","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Processing Systems 30 (2017)."},{"key":"e_1_3_3_5_2","unstructured":"Hugo Touvron Louis Martin Kevin Stone Peter Albert Amjad Almahairi Yasmine Babaei Nikolay Bashlykov Soumya Batra Prajjwal Bhargava Shruti Bhosale et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288. Retrieved from https:\/\/arxiv.org\/abs\/2307.09288"},{"key":"e_1_3_3_6_2","unstructured":"Aaron Grattafiori Abhimanyu Dubey Abhinav Jauhri Abhinav Pandey Abhishek Kadian Ahmad Al-Dahle Aiesha Letman Akhil Mathur Alan Schelten Alex Vaughan et al. 2024. The Llama 3 herd of models. arXiv:2407.21783. Retrieved from https:\/\/arxiv.org\/abs\/2407.21783"},{"key":"e_1_3_3_7_2","first-page":"1877","volume-title":"Advances in Neural Information Processing systems","volume":"33","author":"Brown Tom","year":"2020","unstructured":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In Advances in Neural Information Processing systems, Vol. 33, 1877\u20131901."},{"key":"e_1_3_3_8_2","unstructured":"Josh Achiam Steven Adler Sandhini Agarwal Lama Ahmad Ilge Akkaya Florencia Leoni Aleman Diogo Almeida Janko Altenschmidt Sam Altman Shyamal Anadkat et al. 2023. GPT-4 technical report. arXiv:2303.08774. Retrieved from https:\/\/arxiv.org\/abs\/2303.08774"},{"key":"e_1_3_3_9_2","unstructured":"Gemini Team Rohan Anil Sebastian Borgeaud Yonghui Wu Jean-Baptiste Alayrac Jiahui Yu Radu Soricut Johan Schalkwyk Andrew M. Dai Anja Hauth et al. 2023. Gemini: A family of highly capable multimodal models. arXiv:2312.11805. Retrieved from https:\/\/arxiv.org\/abs\/2312.11805"},{"key":"e_1_3_3_10_2","unstructured":"Albert Q. Jiang Alexandre Sablayrolles Arthur Mensch Chris Bamford Devendra Singh Chaplot Diego de las Casas Florian Bressand Gianna Lengyel Guillaume Lample Lucile Saulnier et al. 2023. Mistral 7b. arXiv:2310.06825. Retrieved from https:\/\/arxiv.org\/abs\/2310.06825"},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N19-4009"},{"key":"e_1_3_3_12_2","first-page":"15908","volume-title":"Advances in Neural Information Processing Systems","volume":"34","author":"Han Kai","year":"2021","unstructured":"Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. 2021. Transformer in transformer. In Advances in Neural Information Processing Systems, Vol. 34, 15908\u201315919."},{"key":"e_1_3_3_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52729.2023.00599"},{"key":"e_1_3_3_14_2","first-page":"139997","volume-title":"Advances in Neural Information Processing Systems","volume":"37","author":"Liu Akide","year":"2024","unstructured":"Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Gholamreza Haffari, and Bohan Zhuang. 2024. Minicache: KV cache compression in depth dimension for large language models. In Advances in Neural Information Processing Systems, Vol. 37, 139997\u2013140031."},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/HPCA51647.2021.00018"},{"key":"e_1_3_3_16_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2021.3092574"},{"key":"e_1_3_3_17_2","volume-title":"Proceedings of the 12th International Conference on Learning Representations","author":"Han Insu","year":"2024","unstructured":"Insu Han, Rajesh Jayaram, Amin Karbasi, Vahab Mirrokni, David Woodruff, and Amir Zandieh. 2024. Hyperattention: Long-context attention in near-linear time. In Proceedings of the 12th International Conference on Learning Representations."},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","DOI":"10.52202\/068431-1189"},{"key":"e_1_3_3_19_2","volume-title":"Proceedings of the 1st Conference on Language Modeling","author":"Duanmu Haojie","year":"2024","unstructured":"Haojie Duanmu, Zhihang Yuan, Xiuhong Li, Jiangfei Duan, Xingcheng Zhang, and Dahua Lin. 2024. SKVQ: Sliding-window key and value cache quantization for large language models. In Proceedings of the 1st Conference on Language Modeling."},{"key":"e_1_3_3_20_2","volume-title":"Advances in Neural Information Processing Systems","volume":"36","author":"Zhang Zhenyu","year":"2024","unstructured":"Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Re, Clark Barrett, et al. 2024. H2O: Heavy-hitter oracle for efficient generative inference of large language models. In Advances in Neural Information Processing Systems, Vol. 36."},{"key":"e_1_3_3_21_2","volume-title":"Proceedings of the 12th International Conference on Learning Representations","author":"Xiao Guangxuan","year":"2024","unstructured":"Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. Efficient streaming language models with attention sinks. In Proceedings of the 12th International Conference on Learning Representations."},{"issue":"2","key":"e_1_3_3_22_2","first-page":"78","article-title":"Simple random sampling","volume":"1","author":"Noor Shagofah","year":"2022","unstructured":"Shagofah Noor, Omid Tajik, and Jawad Golzar. 2022. Simple random sampling. International Journal of Education & Language Studies 1, 2 (2022), 78\u201382.","journal-title":"International Journal of Education & Language Studies"},{"key":"e_1_3_3_23_2","unstructured":"Rewon Child Scott Gray Alec Radford and Ilya Sutskever. 2019. Generating long sequences with sparse transformers. arXiv:1904.10509. Retrieved from https:\/\/arxiv.org\/abs\/1904.10509"},{"key":"e_1_3_3_24_2","unstructured":"Iz Beltagy Matthew E. Peters and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv:2004.05150. Retrieved from https:\/\/arxiv.org\/abs\/2004.05150"},{"key":"e_1_3_3_25_2","unstructured":"Chi Han Qifan Wang Wenhan Xiong Yu Chen Heng Ji and Sinong Wang. 2023. Lm-infinite: Simple on-the-fly length generalization for large language models. arXiv:2308.16137. Retrieved from https:\/\/arxiv.org\/abs\/2308.16137"},{"key":"e_1_3_3_26_2","first-page":"52481","volume-title":"Advances in Neural Information Processing Systems","volume":"37","author":"Jiang Huiqiang","year":"2024","unstructured":"Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H. Abdi, Dongsheng Li, Chin-Yew Lin, et al. 2024. Minference 1.0: Accelerating pre-filling for long-context LLMS via dynamic sparse attention. In Advances in Neural Information Processing Systems, Vol. 37, 52481\u201352515."},{"key":"e_1_3_3_27_2","volume-title":"Proceedings of the 37th Conference on Neural Information Processing Systems","author":"Liu Zichang","year":"2023","unstructured":"Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. 2023. Scissorhands: Exploiting the persistence of importance hypothesis for LLM KV cache compression at test time. In Proceedings of the 37th Conference on Neural Information Processing Systems."},{"key":"e_1_3_3_28_2","first-page":", 114","volume-title":"Proceedings of Machine Learning and Systems","volume":"6","author":"Adnan Muhammad","year":"2024","unstructured":"Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant Nair, Ilya Soloveychik, and Purushotham Kamath. 2024. Keyformer: KV cache reduction through key tokens selection for efficient generative inference. In Proceedings of Machine Learning and Systems, Vol. 6, 114\u2013127."},{"key":"e_1_3_3_29_2","unstructured":"Amir Zandieh Insu Han Vahab Mirrokni and Amin Karbasi. Subgen. 2024. Token generation in sublinear time and memory. arXiv:2402.06082. Retrieved from https:\/\/arxiv.org\/abs\/2402.06082"},{"key":"e_1_3_3_30_2","volume-title":"Proceedings of the 12th International Conference on Learning Representations","author":"Ge Suyu","year":"2024","unstructured":"Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. 2024. Model tells you what to discard: Adaptive KV cache compression for LLMs. In Proceedings of the 12th International Conference on Learning Representations."},{"key":"e_1_3_3_31_2","unstructured":"Shichen Dong Wen Cheng Jiayu Qin and Wei Wang. 2024. QAQ: Quality adaptive quantization for LLM KV cache. arXiv:2403.04643. Retrieved from https:\/\/arxiv.org\/abs\/2403.04643"},{"key":"e_1_3_3_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52729.2023.00196"},{"key":"e_1_3_3_33_2","unstructured":"Yuxuan Yue Zhihang Yuan Haojie Duanmu Sifan Zhou Jianlong Wu and Liqiang Nie. 2024. Wkvquant: Quantizing weight and key\/value cache for large language models gains more. arXiv:2402.12065. Retrieved from https:\/\/arxiv.org\/abs\/2402.12065"},{"key":"e_1_3_3_34_2","first-page":"1270","volume-title":"Advances in Neural Information Processing Systems","volume":"37","author":"Hooper Coleman","year":"2024","unstructured":"Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun S. Shao, Kurt Keutzer, and Amir Gholami. 2024. KVQUANT: Towards 10 million context length LLM inference with KV cache quantization. In Advances in Neural Information Processing Systems, Vol. 37, 1270\u20131303."},{"key":"e_1_3_3_35_2","unstructured":"Hao Kang Qingru Zhang Souvik Kundu Geonhwa Jeong Zaoxing Liu Tushar Krishna and Tuo Zhao. 2024. GEAR: An efficient KV cache compression recipe for near-lossless generative inference of LLM. arXiv:2403.05527. Retrieved from Retrieved from https:\/\/arxiv.org\/abs\/2403.05527"},{"key":"e_1_3_3_36_2","first-page":"5156","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Katharopoulos Angelos","year":"2020","unstructured":"Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Fran\u00e7ois Fleuret. 2020. Transformers are RNNs: Fast autoregressive transformers with linear attention. In Proceedings of the International Conference on Machine Learning. PMLR, 5156\u20135165."},{"key":"e_1_3_3_37_2","unstructured":"Piotr Nawrot Adrian \u0141a\u0144cucki Marcin Chochowski David Tarjan and Edoardo M. Ponti. 2024. Dynamic memory compression: Retrofitting LLMS for accelerated inference. arXiv:2403.09636. Retrieved from https:\/\/arxiv.org\/abs\/2403.09636"},{"key":"e_1_3_3_38_2","unstructured":"Harry Dong Xinyu Yang Zhenyu Zhang Zhangyang Wang Yuejie Chi and Beidi Chen. 2024. Get more with less: Synthesizing recurrence with KV cache compression for efficient LLM inference. arXiv:2402.09398. Retrieved from https:\/\/arxiv.org\/abs\/2402.09398"},{"key":"e_1_3_3_39_2","volume-title":"Proceedings of the 1st Conference on Language Modeling","author":"Gu Albert","year":"2024","unstructured":"Albert Gu and Tri Dao. 2024. Mamba: Linear-time sequence modeling with selective state spaces. In Proceedings of the 1st Conference on Language Modeling."},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/3564246.3585100"},{"key":"e_1_3_3_41_2","unstructured":"Praneeth Kacham and David P. Woodruff. 2024. Faster algorithms for Schatten-p low rank approximation. arXiv:2407.11959. Retrieved from https:\/\/arxiv.org\/abs\/2407.11959"},{"key":"e_1_3_3_42_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.acl-long.602"},{"key":"e_1_3_3_43_2","unstructured":"Deep Seek-Ai Aixin Liu Bei Feng Bin Wang Bingxuan Wang Bo Liu Chenggang Zhao Chengqi Deng Chong Ruan Damai Dai Daya Guo Dejian Yang et al. 2024. Deepseek-v2: A strong economical and efficient mixture-of-experts language model. arXiv:2405.04434. Retrieved from https:\/\/arxiv.org\/abs\/2405.04434"},{"key":"e_1_3_3_44_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.acl-long.527"},{"key":"e_1_3_3_45_2","unstructured":"Aiyuan Yang Bin Xiao Bingning Wang Borong Zhang Ce Bian Chao Yin Chenxu Lv Da Pan Dian Wang Dong Yan et al. 2023. Baichuan 2: Open large-scale language models. arXiv:2309.10305. Retrieved from https:\/\/arxiv.org\/abs\/2309.10305"},{"key":"e_1_3_3_46_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-1099"},{"key":"e_1_3_3_47_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1206"},{"key":"e_1_3_3_48_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.acl-long.172"},{"key":"e_1_3_3_49_2","unstructured":"Stephen Merity Caiming Xiong James Bradbury and Richard Socher. 2016. Pointer sentinel mixture models. arXiv:1609.07843. Retrieved from https:\/\/arxiv.org\/abs\/1609.07843"},{"key":"e_1_3_3_50_2","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00638"},{"key":"e_1_3_3_51_2","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Holtzman Ari","year":"2020","unstructured":"Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In Proceedings of the International Conference on Learning Representations."}],"container-title":["ACM Transactions on Intelligent Systems and Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3810942","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,15]],"date-time":"2026-06-15T11:39:36Z","timestamp":1781523576000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3810942"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,15]]},"references-count":50,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2026,8,31]]}},"alternative-id":["10.1145\/3810942"],"URL":"https:\/\/doi.org\/10.1145\/3810942","relation":{},"ISSN":["2157-6904","2157-6912"],"issn-type":[{"value":"2157-6904","type":"print"},{"value":"2157-6912","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,15]]},"assertion":[{"value":"2024-11-13","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-03-26","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-06-15","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}