{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T19:00:58Z","timestamp":1782846058931,"version":"3.54.5"},"reference-count":77,"publisher":"Association for Computing Machinery (ACM)","issue":"FSE","license":[{"start":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T00:00:00Z","timestamp":1782777600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/legalcode"}],"funder":[{"name":"2nd CSIRO and A*STAR: ResearchIndustry (2+2) Partnership Program","award":["R24I5IR047"],"award-info":[{"award-number":["R24I5IR047"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Softw. Eng."],"published-print":{"date-parts":[[2026,6,30]]},"abstract":"<jats:p>The demand for better prediction accuracy and higher execution performance in neural networks continues to grow. The emergence and success of Large Language Models (LLMs) have produced many cloud-based tools for software engineering tasks such as code suggestion. Although effective, cloud deployment raises concerns over privacy, latency, and reliance on network connectivity. Running LLMs locally on personal devices such as laptops would address these issues, because it enables offline use and reduces response time. However, local deployment is challenging, since commodity devices lack high-performance accelerators such as GPUs and are constrained by limited memory and compute capacity, which makes it hard to execute large models efficiently.<\/jats:p>\n                  <jats:p>We present Ditto, a framework that optimizes both the model size of Code LLMs and the inference programs that execute them, with a focus on statically-typed languages such as C. Our approach integrates two components. The first is a quantization technique inspired by product quantization, which groups model parameters into per-block codebooks via K-Means clustering and stores each weight as a bit-packed low-bitwidth index. The quantizer further supports a mixed-precision mode that keeps a small number of sensitivity-critical tensors in float32. The second component is a compilation pass integrated into LLVM that automatically detects and replaces unoptimized General Matrix-Vector Multiplication (GEMV) operations, which are the most computationally intensive part of code models, with calls into Basic Linear Algebra Subprograms (BLAS) libraries that are highly optimized for the target hardware. The output of Ditto is a compiled executable that runs the selected Code LLM on commodity hardware.<\/jats:p>\n                  <jats:p>We evaluate Ditto on three popular Code LLMs, namely Code Llama, MagicCoder, and OpenCodeInterpreter, achieving up to 10.5\u00d7 faster inference, 6.4\u00d7 lower memory usage, and 10.5\u00d7 lower energy consumption compared with their original inference pipelines, while preserving accuracy close to the full-precision models, with an average loss of only 0.27% in pass@1. Ditto also outperforms the state-of-the-art int8 quantization baseline, achieving up to 6.96% higher pass@1 accuracy, 2.2\u00d7 speedup, and 1.6\u00d7 memory reduction, which demonstrates the effectiveness of our approach.<\/jats:p>","DOI":"10.1145\/3808196","type":"journal-article","created":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:06:14Z","timestamp":1782839174000},"page":"4299-4322","source":"Crossref","is-referenced-by-count":0,"title":["Compiling Code LLMs into Lightweight Executables"],"prefix":"10.1145","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0799-5018","authenticated-orcid":false,"given":"Jieke","family":"Shi","sequence":"first","affiliation":[{"name":"Singapore Management University, Singapore, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3370-8585","authenticated-orcid":false,"given":"Junda","family":"He","sequence":"additional","affiliation":[{"name":"Singapore Management University, Singapore, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5938-1918","authenticated-orcid":false,"given":"Zhou","family":"Yang","sequence":"additional","affiliation":[{"name":"University of Alberta, Edmonton, Canada"},{"name":"Alberta Machine Intelligence Institute (Amii), Edmonton, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6100-8127","authenticated-orcid":false,"given":"Chengran","family":"Yang","sequence":"additional","affiliation":[{"name":"Singapore Management University, Singapore, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4641-8977","authenticated-orcid":false,"given":"Mykhailo","family":"Klymenko","sequence":"additional","affiliation":[{"name":"CSIRO's Data61, Eveleigh, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5096-4834","authenticated-orcid":false,"given":"Thong (James)","family":"Hoang","sequence":"additional","affiliation":[{"name":"CSIRO's Data61, Eveleigh, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2273-1862","authenticated-orcid":false,"given":"Sherry (Xiwei)","family":"Xu","sequence":"additional","affiliation":[{"name":"CSIRO's Data61, Eveleigh, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7663-1421","authenticated-orcid":false,"given":"Zhenchang","family":"Xing","sequence":"additional","affiliation":[{"name":"CSIRO's Data61, Eveleigh, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4367-7201","authenticated-orcid":false,"given":"David","family":"Lo","sequence":"additional","affiliation":[{"name":"Singapore Management University, Singapore, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,6,30]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME64153.2025.00049"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639183"},{"key":"e_1_2_1_3_1","unstructured":"Apple. 2025. Apple's Accelerate framework. https:\/\/developer.apple.com\/documentation\/accelerate\/blas\/."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/1985793.1985795"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","unstructured":"Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski David Dohan Ellen Jiang Carrie Cai Michael Terry Quoc Le et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 (2021). doi:10.48550\/arXiv.2108.07732 10.48550\/arXiv.2108.07732","DOI":"10.48550\/arXiv.2108.07732"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","unstructured":"Jinze Bai Shuai Bai Yunfei Chu Zeyu Cui Kai Dang and et al. 2023. Qwen Technical Report. doi:10.48550\/arXiv.2309. 16609 arXiv:2309.16609 [cs.CL] 10.48550\/arXiv.2309.16609","DOI":"10.48550\/arXiv.2309"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/567806.567807"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3372266"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2107.03374"},{"key":"e_1_2_1_10_1","volume-title":"Manning","author":"Clark Kevin","year":"2020","unstructured":"Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators. (2020). https:\/\/openreview.net\/forum?id=r1xMH1BtvB"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10773-006-9199-8"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3459010"},{"key":"e_1_2_1_13_1","unstructured":"Hugging Face. 2025. Hugging Face -The AI community building the future. https:\/\/huggingface.co."},{"key":"e_1_2_1_14_1","unstructured":"Hugging Face. 2026. GPTQ. https:\/\/huggingface.co\/docs\/transformers\/en\/quantization\/gptq."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/icse-fose59343.2023.00008"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","unstructured":"Sen Fang Weiyuan Ding Antonio Mastropaolo and Bowen Xu. 2025. Smaller = Weaker? Benchmarking Robustness of Quantized LLMs in Code Generation. doi:10.48550\/arXiv.2506.22776 arXiv:2506.22776 [cs.SE] 10.48550\/arXiv.2506.22776","DOI":"10.48550\/arXiv.2506.22776"},{"key":"e_1_2_1_17_1","volume-title":"OPTQ: Accurate Quantization for Generative Pre-trained Transformers. In The Eleventh International Conference on Learning Representations. https:\/\/openreview. net\/forum?id=tcbBPnfwxS","author":"Frantar Elias","year":"2023","unstructured":"Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. OPTQ: Accurate Quantization for Generative Pre-trained Transformers. In The Eleventh International Conference on Learning Representations. https:\/\/openreview. net\/forum?id=tcbBPnfwxS"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00413"},{"key":"e_1_2_1_19_1","unstructured":"Georgi Gerganov. 2023. ggerganov\/llama.cpp: LLM inference in C\/C++. https:\/\/github.com\/ggerganov\/llama.cpp."},{"key":"e_1_2_1_20_1","unstructured":"GitHub. 2025. GitHub Copilot \u2022 Your AI pair programmer. https:\/\/github.com\/features\/copilot\/."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","unstructured":"Md Nazmul Haque Hua Yang Zhou Yang and Bowen Xu. 2025. How Quantization Impacts Privacy Risk on LLMs for Code? doi:10.48550\/arXiv.2508.00128 arXiv:2508.00128 [cs.SE] 10.48550\/arXiv.2508.00128","DOI":"10.48550\/arXiv.2508.00128"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","unstructured":"D\u00e1vid Hidv\u00e9gi Khashayar Etemadi Sofia Bobadilla and Martin Monperrus. 2024. CigaR: Cost-efficient Program Repair with LLMs. doi:10.48550\/arXiv.2402.06598 arXiv:2402.06598 [cs.SE] 10.48550\/arXiv.2402.06598","DOI":"10.48550\/arXiv.2402.06598"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1503.02531"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3695988"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/dac63849.2025.11133264"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3719276.3725197"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3747588"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/icse48619.2023.00125"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3611643.3613892"},{"key":"e_1_2_1_30_1","unstructured":"Tom Jobbins. 2026. TheBloke (Tom Jobbins). https:\/\/huggingface.co\/TheBloke."},{"key":"e_1_2_1_31_1","unstructured":"Andrej karpathy. 2025. karpathy\/llama2.c: Inference Llama 2 in one file of pure C. https:\/\/github.com\/karpathy\/llama2. c.git."},{"key":"e_1_2_1_32_1","unstructured":"Andrej Karpathy. 2026. karpathy\/tinyllamas \u2022 Hugging Face. https:\/\/huggingface.co\/karpathy\/tinyllamas."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2309.14021"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/PAIS66004.2025.11126523"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/ojcs.2025.3587005"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3643916.3644434"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3669940.3707268"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3714983.3714987"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.52202\/075280-0943"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/HPCA61900.2025.00112"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIT"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/icse-fose59343.2023.00010"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1109\/ase63991.2025.00024"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177730491"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.3169\/mta.6.2"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3605943"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3695053.3731057"},{"key":"e_1_2_1_48_1","volume-title":"The Twelfth International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=gLARhFLE0F","author":"Park Gunho","year":"2024","unstructured":"Gunho Park, Baeseong park, Minsub Kim, Sungjae Lee, Jeonghoon Kim, Beomseok Kwon, Se Jung Kwon, Byeongwook Kim, Youngjoo Lee, and Dongsoo Lee. 2024. LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models. In The Twelfth International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=gLARhFLE0F"},{"key":"e_1_2_1_49_1","unstructured":"Qwen. 2026. Qwen\/CodeQwen1.5-7B. https:\/\/huggingface.co\/Qwen\/CodeQwen1.5-7B."},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1109\/icst60714.2024.00018"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3623402"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","unstructured":"Baptiste Rozi\u00e8re Jonas Gehring Fabian Gloeckle Sten Sootla Itai Gat Xiaoqing Ellen Tan Yossi Adi Jingyu Liu Romain Sauvestre Tal Remez J\u00e9r\u00e9my Rapin Artyom Kozhevnikov Ivan Evtimov Joanna Bitton Manish Bhatt Cristian Canton Ferrer Aaron Grattafiori Wenhan Xiong Alexandre D\u00e9fossez Jade Copet Faisal Azhar Hugo Touvron Louis Martin Nicolas Usunier Thomas Scialom and Gabriel Synnaeve. 2024. Code Llama: Open Foundation Models for Code. doi:10.48550\/arXiv.2308.12950 arXiv:2308.12950 [cs.CL] 10.48550\/arXiv.2308.12950","DOI":"10.48550\/arXiv.2308.12950"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2024.107610"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/3639475.3640097"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3708525"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3556964"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10515-024-00421-4"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/3650212.3680347"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/3564625.3567985"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","unstructured":"Hugo Touvron Louis Martin Kevin Stone Peter Albert and et al. 2023. Llama 2: Open Foundation and Fine-Tuned Chat Models. doi:10.48550\/arXiv.2307.09288 arXiv:2307.09288 [cs.CL] 10.48550\/arXiv.2307.09288","DOI":"10.48550\/arXiv.2307.09288"},{"key":"e_1_2_1_61_1","volume-title":"The Thirteenth International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=2zMHHZ569S","author":"Vallaeys Th\u00e9ophane","year":"2025","unstructured":"Th\u00e9ophane Vallaeys, Matthew J. Muckley, Jakob Verbeek, and Matthijs Douze. 2025. Qinco2: Vector Compression and Search with Improved Implicit Neural Codebooks. In The Thirteenth International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=2zMHHZ569S"},{"key":"e_1_2_1_62_1","volume-title":"\u0141 ukasz Kaiser, and Illia Polosukhin","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, \u0141 ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems, I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30. Curran Associates, Inc. https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2017\/file\/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1109\/tse.2024.3368208"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/3689031.3696099"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/3611643.3616302"},{"key":"e_1_2_1_66_1","volume-title":"Proceedings of the 41st International Conference on Machine Learning (Proceedings of Machine Learning Research","volume":"52657","author":"Wei Yuxiang","year":"2024","unstructured":"Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. 2024. Magicoder: Empowering Code Generation with OSS-Instruct. In Proceedings of the 41st International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 235), Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (Eds.). PMLR, 52632-52657. https:\/\/proceedings.mlr.press\/v235\/wei24h.html"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1109\/icse48619.2023.00129"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/3725843.3756078"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/3520312.3534862"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v39i21.34385"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1007\/S11432-"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549094"},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1007\/S11704-026-60308-3"},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.findings-acl.762"},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","unstructured":"Zibin Zheng Kaiwen Ning Yanlin Wang Jingwen Zhang Dewu Zheng Mingxi Ye and Jiachi Chen. 2024. A Survey of Large Language Models for Code: Evolution Benchmarking and Future Trends. doi:10.48550\/arXiv.2311.10372 arXiv:2311.10372 [cs.SE] 10.48550\/arXiv.2311.10372","DOI":"10.48550\/arXiv.2311.10372"},{"key":"e_1_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.1145\/3639476.3639762"},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2406.15877"}],"container-title":["Proceedings of the ACM on Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3808196","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:04:09Z","timestamp":1782842649000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3808196"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,30]]},"references-count":77,"journal-issue":{"issue":"FSE","published-print":{"date-parts":[[2026,6,30]]}},"alternative-id":["10.1145\/3808196"],"URL":"https:\/\/doi.org\/10.1145\/3808196","relation":{},"ISSN":["2994-970X"],"issn-type":[{"value":"2994-970X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,30]]}}}