{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,24]],"date-time":"2026-07-24T13:06:37Z","timestamp":1784898397962,"version":"3.55.0"},"reference-count":49,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2026,7,24]],"date-time":"2026-07-24T00:00:00Z","timestamp":1784851200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62502170"],"award-info":[{"award-number":["62502170"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Special High-Quality Program of MIIT","award":["2540STC62590"],"award-info":[{"award-number":["2540STC62590"]}]},{"DOI":"10.13039\/501100007129","name":"Shandong Provincial Natural Science Foundation","doi-asserted-by":"crossref","award":["ZR2024LZH004"],"award-info":[{"award-number":["ZR2024LZH004"]}],"id":[{"id":"10.13039\/501100007129","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100002858","name":"China Postdoctoral Science Foundation","doi-asserted-by":"crossref","award":["2024M751011"],"award-info":[{"award-number":["2024M751011"]}],"id":[{"id":"10.13039\/501100002858","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Postdoctor Project of Hubei Province","award":["2024HBBHCXA027"],"award-info":[{"award-number":["2024HBBHCXA027"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Archit. Code Optim."],"published-print":{"date-parts":[[2026,9,30]]},"abstract":"<jats:p>Recently, video language models (VLMs) have been applied in various fields. However, the visual token sequence of the VLM is too long, which may cause intolerant inference latency and GPU memory usage. Existing methods propose mixed-precision quantization to the key-value (KV) cache in VLMs based on token granularity, which is time-consuming in the search process and hardware inefficient during computation. This article introduces a novel approach called WindowQuant, which employs window-adaptive mixed-precision quantization to optimize the KV cache. WindowQuant consists of two modules: window-level quantization search and window-level KV cache computation. Window-level quantization search quickly determines the optimal bit-width configuration of the KV cache windows based on the similarity scores between the corresponding visual token windows and the text prompt, maintaining the model accuracy. Furthermore, window-level KV cache computation reorders the KV cache windows before quantization, avoiding the hardware inefficiency caused by mixed-precision quantization in inference computation. Extensive experiments demonstrate that WindowQuant outperforms state-of-the-art VLM models and KV cache quantization methods on various datasets.<\/jats:p>","DOI":"10.1145\/3817053","type":"journal-article","created":{"date-parts":[[2026,7,24]],"date-time":"2026-07-24T12:32:17Z","timestamp":1784896337000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["WindowQuant: Mixed-Precision KV Cache Quantization Based on Window-Level Similarity for VLMs Inference Optimization"],"prefix":"10.1145","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0009-0003-1262-5729","authenticated-orcid":false,"given":"Wei","family":"Tao","sequence":"first","affiliation":[{"name":"Huazhong University of Science and Technology","place":["Wuhan, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8353-4064","authenticated-orcid":false,"given":"Xiaoyang","family":"Qu","sequence":"additional","affiliation":[{"name":"Ping An Technology (Shenzhen) Co., Ltd.","place":["Shenzhen, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-4504-4721","authenticated-orcid":false,"given":"Peiqiang","family":"Wang","sequence":"additional","affiliation":[{"name":"Tsinghua Shenzhen International Graduate School","place":["Shenzhen, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-7998-5520","authenticated-orcid":false,"given":"Guokuan","family":"Li","sequence":"additional","affiliation":[{"name":"Huazhong University of Science and Technology","place":["Wuhan, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3440-4460","authenticated-orcid":false,"given":"Jiguang","family":"Wan","sequence":"additional","affiliation":[{"name":"Huazhong University of Science and Technology","place":["Wuhan, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7757-4083","authenticated-orcid":false,"given":"Kai","family":"Lu","sequence":"additional","affiliation":[{"name":"Huazhong University of Science and Technology","place":["Wuhan, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9237-4231","authenticated-orcid":false,"given":"Jianzong","family":"Wang","sequence":"additional","affiliation":[{"name":"Ping An Technology (Shenzhen) Co., Ltd.","place":["Shenzhen, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,7,24]]},"reference":[{"key":"e_1_3_1_2_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.emnlp-main.298"},{"key":"e_1_3_1_3_2","first-page":"46254","volume-title":"International Conference on Learning Representations","author":"Cai Mu","year":"2025","unstructured":"Mu Cai, Jianwei Yang, Jianfeng Gao, and Yong Jae Lee. 2025. Matryoshka multimodal models. In International Conference on Learning Representations. 46254\u201346272."},{"key":"e_1_3_1_4_2","first-page":"5209","volume-title":"Proceedings of the 41st International Conference on Machine Learning","author":"Cai Tianle","year":"2024","unstructured":"Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. 2024. MEDUSA: Simple LLM inference acceleration framework with multiple decoding heads. In Proceedings of the 41st International Conference on Machine Learning. 5209\u20135235."},{"key":"e_1_3_1_5_2","first-page":"578","volume-title":"Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18)","author":"Chen Tianqi","year":"2018","unstructured":"Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, et\u00a0al. 2018. \\(\\lbrace\\) TVM \\(\\rbrace\\) : An automated \\(\\lbrace\\) End-to-End \\(\\rbrace\\) optimizing compiler for deep learning. In Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18). 578\u2013594."},{"key":"e_1_3_1_6_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11432-024-4231-5"},{"key":"e_1_3_1_7_2","unstructured":"Zesen Cheng Sicong Leng Hang Zhang Yifei Xin Xin Li Guanzheng Chen Yongxin Zhu Wenqi Zhang Ziyang Luo Deli Zhao et\u00a0al. 2024. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-LLMs. arXiv:2406.07476. Retrieved from https:\/\/arxiv.org\/abs\/2406.07476"},{"key":"e_1_3_1_8_2","first-page":"35549","volume-title":"International Conference on Learning Representations","volume":"2024","author":"Dao Tri","year":"2024","unstructured":"Tri Dao. 2024. FlashAttention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations 2024 (2024), 35549\u201335562."},{"key":"e_1_3_1_9_2","doi-asserted-by":"publisher","DOI":"10.52202\/068431-1189"},{"key":"e_1_3_1_10_2","doi-asserted-by":"publisher","DOI":"10.52202\/068431-2198"},{"key":"e_1_3_1_11_2","volume-title":"The Twelfth International Conference on Learning Representations","author":"Dettmers Tim","year":"2024","unstructured":"Tim Dettmers, Ruslan A. Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. 2024. SpQR: A sparse-quantized representation for near-lossless LLM weight compression. In The Twelfth International Conference on Learning Representations."},{"key":"e_1_3_1_12_2","unstructured":"Shichen Dong Wen Cheng Jiayu Qin and Wei Wang. 2024. QAQ: Quality Adaptive Quantization for LLM KV Cache. arXiv:2403.04643. Retrieved from https:\/\/arxiv.org\/abs\/2403.04643. (2024)."},{"key":"e_1_3_1_13_2","volume-title":"First Conference on Language Modeling","author":"Duanmu Haojie","year":"2024","unstructured":"Haojie Duanmu, Zhihang Yuan, Xiuhong Li, Jiangfei Duan, Xingcheng Zhang, and Dahua Lin. 2024. SKVQ: Sliding-window key and value cache quantization for large language models. In First Conference on Language Modeling."},{"key":"e_1_3_1_14_2","volume-title":"The Eleventh International Conference on Learning Representations","author":"Frantar Elias","year":"2023","unstructured":"Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. OPTQ: Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations."},{"key":"e_1_3_1_15_2","first-page":"22975","volume-title":"International Conference on Learning Representations","volume":"2024","author":"Ge Suyu","year":"2024","unstructured":"Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. 2024. Model tells you what to discard: Adaptive KV cache compression for LLMs. In International Conference on Learning Representations 2024 (2024), 22975\u201322988."},{"key":"e_1_3_1_16_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.naacl-long.222"},{"key":"e_1_3_1_17_2","doi-asserted-by":"publisher","DOI":"10.52202\/079017-0040"},{"key":"e_1_3_1_18_2","first-page":"23901","volume-title":"Proceedings of the 41st International Conference on Machine Learning","author":"Kim Sehoon","year":"2024","unstructured":"Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W. Mahoney, and Kurt Keutzer. 2024. SqueezeLLM: Dense-and-sparse quantization. In Proceedings of the 41st International Conference on Machine Learning. 23901\u201323923."},{"key":"e_1_3_1_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2024.3517625"},{"key":"e_1_3_1_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/3600006.3613165"},{"key":"e_1_3_1_21_2","article-title":"LLaVA-OneVision: Easy visual task transfer","author":"Li Bo","year":"2025","unstructured":"Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et\u00a0al. 2025. LLaVA-OneVision: Easy visual task transfer. Transactions on Machine Learning Research.","journal-title":"Transactions on Machine Learning Research"},{"key":"e_1_3_1_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52733.2024.02095"},{"key":"e_1_3_1_23_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.emnlp-main.342"},{"key":"e_1_3_1_24_2","first-page":"87","article-title":"AWQ: Activation-aware weight quantization for on-device LLM compression and acceleration","volume":"6","author":"Lin Ji","year":"2024","unstructured":"Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. AWQ: Activation-aware weight quantization for on-device LLM compression and acceleration. Proceedings of Machine Learning and Systems 6 (2024), 87\u2013100.","journal-title":"Proceedings of Machine Learning and Systems"},{"key":"e_1_3_1_25_2","article-title":"QServe: W4A8KV4 quantization and system co-design for efficient LLM serving","volume":"7","author":"Lin Yujun","year":"2025","unstructured":"Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. 2025. QServe: W4A8KV4 quantization and system co-design for efficient LLM serving. Proceedings of Machine Learning and Systems 7 (2025).","journal-title":"Proceedings of Machine Learning and Systems"},{"key":"e_1_3_1_26_2","doi-asserted-by":"publisher","DOI":"10.52202\/075280-1516"},{"key":"e_1_3_1_27_2","doi-asserted-by":"publisher","DOI":"10.52202\/075280-1516"},{"key":"e_1_3_1_28_2","first-page":"1","volume-title":"Proceedings of the European Conference on Computer Vision","author":"Liu Ruyang","year":"2024","unstructured":"Ruyang Liu, Chen Li, Haoran Tang, Yixiao Ge, Ying Shan, and Ge Li. 2024. St-LLM: Large language models are effective temporal learners. In Proceedings of the European Conference on Computer Vision. Springer, 1\u201318."},{"key":"e_1_3_1_29_2","first-page":"32332","volume-title":"Proceedings of the 41st International Conference on Machine Learning","author":"Liu Zirui","year":"2024","unstructured":"Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. 2024. KIVI: A tuning-free asymmetric 2-bit quantization for KV cache. In Proceedings of the 41st International Conference on Machine Learning. 32332\u201332344."},{"key":"e_1_3_1_30_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52733.2024.01249"},{"key":"e_1_3_1_31_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.acl-long.679"},{"key":"e_1_3_1_32_2","doi-asserted-by":"publisher","DOI":"10.52202\/079017-0739"},{"key":"e_1_3_1_33_2","unstructured":"OpenAI. 2023. GPT-4V(ision) System Card. Retrieved from https:\/\/cdn.openai.com\/papers\/GPTV_System_Card.pdf"},{"key":"e_1_3_1_34_2","unstructured":"Tingyu Qu Mingxiao Li Tinne Tuytelaars and Marie-Francine Moens. 2024. TS-LLaVA: Constructing visual tokens through thumbnail-and-sampling for training-free video large language models. arXiv:2411.11066. Retrieved from https:\/\/arxiv.org\/abs\/2411.11066"},{"key":"e_1_3_1_35_2","unstructured":"Noam Shazeer. 2019. Fast transformer decoding: One write-head is all you need. arXiv:1911.02150. Retrieved from https:\/\/arxiv.org\/abs\/1911.02150"},{"key":"e_1_3_1_36_2","first-page":"31094","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Sheng Ying","year":"2023","unstructured":"Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher R\u00e9, Ion Stoica, and Ce Zhang. 2023. Flexgen: High-throughput generative inference of large language models with a single GPU. In Proceedings of the International Conference on Machine Learning. PMLR, 31094\u201331116."},{"key":"e_1_3_1_37_2","volume-title":"First Conference on Language Modeling","author":"Song Dingjie","year":"2024","unstructured":"Dingjie Song, Shunian Chen, Guiming Hardy Chen, Fei Yu, Xiang Wan, and Benyou Wang. 2024. MileBench: Benchmarking MLLMs in long context. In First Conference on Language Modeling."},{"key":"e_1_3_1_38_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2025.3604614"},{"key":"e_1_3_1_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICME59968.2025.11209367"},{"key":"e_1_3_1_40_2","article-title":"Attention is all you need","volume":"30","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Processing Systems 30 (2017).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_41_2","unstructured":"Peng Wang Shuai Bai Sinan Tan Shijie Wang Zhihao Fan Jinze Bai Keqin Chen Xuejing Liu Jialin Wang Wenbin Ge et\u00a0al. 2024. Qwen2-vl: Enhancing vision-language model\u2019s perception of the world at any resolution. arXiv:2409.12191. Retrieved from https:\/\/arxiv.org\/abs\/2409.12191"},{"key":"e_1_3_1_42_2","first-page":"23593","volume-title":"International Conference on Learning Representations","volume":"2025","author":"Wang Zihao","year":"2025","unstructured":"Zihao Wang, Bin Cui, and Shaoduo Gan. 2025. SqueezeAttention: 2D management of KV-cache in LLM inference via layer-wise optimal budget. In International Conference on Learning Representations 2025 (2025), 23593\u201323606."},{"key":"e_1_3_1_43_2","first-page":"38087","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Xiao Guangxuan","year":"2023","unstructured":"Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2023. Smoothquant: Accurate and efficient post-training quantization for large language models. In Proceedings of the International Conference on Machine Learning. PMLR, 38087\u201338099."},{"key":"e_1_3_1_44_2","first-page":"21875","volume-title":"International Conference on Learning Representations","volume":"2024","author":"Xiao Guangxuan","year":"2024","unstructured":"Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. Efficient streaming language models with attention sinks. In International Conference on Learning Representations 2024 (2024), 21875\u201321895."},{"key":"e_1_3_1_45_2","unstructured":"Lin Xu Yilin Zhao Daquan Zhou Zhijie Lin See Kiong Ng and Jiashi Feng. 2024. Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv:2404.16994. Retrieved from https:\/\/arxiv.org\/abs\/2404.16994. (2024)."},{"key":"e_1_3_1_46_2","unstructured":"Mingze Xu Mingfei Gao Zhe Gan Hong-You Chen Zhengfeng Lai Haiming Gang Kai Kang and Afshin Dehghan. 2024. Slowfast-llava: A strong training-free baseline for video large language models. arXiv:2407.15841. Retrieved from https:\/\/arxiv.org\/abs\/2407.15841"},{"key":"e_1_3_1_47_2","unstructured":"June Yong Yang Byeongwook Kim Jeongin Bae Beomseok Kwon Gunho Park Eunho Yang Se Jung Kwon and Dongsoo Lee. 2024. No token left behind: Reliable kv cache compression via importance-aware mixed precision quantization. arXiv:2402.18096. Retrieved from https:\/\/arxiv.org\/abs\/2402.18096"},{"key":"e_1_3_1_48_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.emnlp-demo.49"},{"key":"e_1_3_1_49_2","article-title":"H2o: Heavy-hitter oracle for efficient generative inference of large language models","volume":"36","author":"Zhang Zhenyu","year":"2024","unstructured":"Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher R\u00e9, Clark Barrett, et\u00a0al. 2024. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems 36 (2024).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_50_2","first-page":"196","article-title":"Atom: Low-bit quantization for efficient and accurate LLM serving","volume":"6","author":"Zhao Yilong","year":"2024","unstructured":"Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. 2024. Atom: Low-bit quantization for efficient and accurate LLM serving. Proceedings of Machine Learning and Systems 6 (2024), 196\u2013209.","journal-title":"Proceedings of Machine Learning and Systems"}],"container-title":["ACM Transactions on Architecture and Code Optimization"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3817053","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,24]],"date-time":"2026-07-24T12:57:36Z","timestamp":1784897856000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3817053"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,24]]},"references-count":49,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2026,9,30]]}},"alternative-id":["10.1145\/3817053"],"URL":"https:\/\/doi.org\/10.1145\/3817053","relation":{},"ISSN":["1544-3566","1544-3973"],"issn-type":[{"value":"1544-3566","type":"print"},{"value":"1544-3973","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,24]]},"assertion":[{"value":"2025-10-11","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-05-01","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-07-24","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}