{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,9,24]],"date-time":"2025-09-24T00:15:08Z","timestamp":1758672908184,"version":"3.44.0"},"publisher-location":"California","reference-count":0,"publisher":"International Joint Conferences on Artificial Intelligence Organization","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025,9]]},"abstract":"<jats:p>Deploying Large Language Models (LLMs) on edge devices is increasingly important, as it eliminates reliance on network connections, reduces expensive API calls, and enhances user privacy. However, on-device deployment is challenging due to the limited computational resources of edge devices. In particular, the key bottleneck stems from memory bandwidth constraints related to weight loading. \n\nWeight-only quantization effectively reduces memory access, yet often induces significant accuracy degradation. \n\nRecent efforts to incorporate sub-branches have shown promise for mitigating quantization errors, but these methods either lack robust optimization strategies or rely on suboptimal objectives. To address these gaps, we propose FeedBack Quantization (FBQuant), a novel approach inspired by negative feedback mechanisms in automatic control.\n\nFBQuant inherently ensures that the reconstructed weights remain bounded by the quantization process, thereby reducing the risk of overfitting.\n\nTo further offset the additional latency introduced by sub-branches, we develop an efficient CUDA kernel that decreases 60% of extra inference time. \n\nComprehensive experiments demonstrate the efficiency and effectiveness of FBQuant across various LLMs. Notably, for 3-bit Llama2-7B, FBQuant improves zero-shot accuracy by 1.2%.<\/jats:p>","DOI":"10.24963\/ijcai.2025\/844","type":"proceedings-article","created":{"date-parts":[[2025,9,19]],"date-time":"2025-09-19T08:10:40Z","timestamp":1758269440000},"page":"7589-7597","source":"Crossref","is-referenced-by-count":0,"title":["FBQuant: FeedBack Quantization for Large Language Models"],"prefix":"10.24963","author":[{"given":"Yijiang","family":"Liu","sequence":"first","affiliation":[{"name":"School of Electronic Science and Engineering, Nanjing University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hengyu","family":"Fang","sequence":"additional","affiliation":[{"name":"School of Electronic Science and Engineering, Nanjing University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Liulu","family":"He","sequence":"additional","affiliation":[{"name":"School of Electronic Science and Engineering, Nanjing University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Rongyu","family":"Zhang","sequence":"additional","affiliation":[{"name":"School of Electronic Science and Engineering, Nanjing University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yichuan","family":"Bai","sequence":"additional","affiliation":[{"name":"School of Electronic Science and Engineering, Nanjing University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yuan","family":"Du","sequence":"additional","affiliation":[{"name":"School of Electronic Science and Engineering, Nanjing University"},{"name":"Interdisciplinary Research Center for Future Intelligent Chips (Chip-X), Nanjing University, Suzhou"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Li","family":"Du","sequence":"additional","affiliation":[{"name":"School of Electronic Science and Engineering, Nanjing University"},{"name":"Interdisciplinary Research Center for Future Intelligent Chips (Chip-X), Nanjing University, Suzhou"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"10584","event":{"number":"34","sponsor":["International Joint Conferences on Artificial Intelligence Organization (IJCAI)"],"acronym":"IJCAI-2025","name":"Thirty-Fourth International Joint Conference on Artificial Intelligence {IJCAI-25}","start":{"date-parts":[[2025,8,16]]},"theme":"Artificial Intelligence","location":"Montreal, Canada","end":{"date-parts":[[2025,8,22]]}},"container-title":["Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence"],"original-title":[],"deposited":{"date-parts":[[2025,9,23]],"date-time":"2025-09-23T11:35:17Z","timestamp":1758627317000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.ijcai.org\/proceedings\/2025\/844"}},"subtitle":[],"proceedings-subject":"Artificial Intelligence Research Articles","short-title":[],"issued":{"date-parts":[[2025,9]]},"references-count":0,"URL":"https:\/\/doi.org\/10.24963\/ijcai.2025\/844","relation":{},"subject":[],"published":{"date-parts":[[2025,9]]}}}