{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,24]],"date-time":"2026-08-24T23:16:18Z","timestamp":1787613378417,"version":"build-2736575974"},"reference-count":111,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2024,12,2]],"date-time":"2024-12-02T00:00:00Z","timestamp":1733097600000},"content-version":"vor","delay-in-days":336,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,11,27]]},"abstract":"Abstract<\/jats:title>\n Large Language Models (LLMs) have transformed natural language processing tasks successfully. Yet, their large size and high computational needs pose challenges for practical use, especially in resource-limited settings. Model compression has emerged as a key research area to address these challenges. This paper presents a survey of model compression techniques for LLMs. We cover methods like quantization, pruning, and knowledge distillation, highlighting recent advancements. We also discuss benchmarking strategies and evaluation metrics crucial for assessing compressed LLMs. This survey offers valuable insights for researchers and practitioners, aiming to enhance efficiency and real-world applicability of LLMs while laying a foundation for future advancements.<\/jats:p>","DOI":"10.1162\/tacl_a_00704","type":"journal-article","created":{"date-parts":[[2024,12,2]],"date-time":"2024-12-02T11:32:06Z","timestamp":1733139126000},"page":"1556-1577","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":201,"title":["A Survey on Model Compression for Large Language\n Models"],"prefix":"10.1162","volume":"12","author":[{"given":"Xunyu","family":"Zhu","sequence":"first","affiliation":[{"name":"Institute of Information Engineering, Chinese Academy of Sciences, China"},{"name":"School of Cyber Security, University of Chinese Academy of Sciences, China. zhuxunyu@iie.ac.cn"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jian","family":"Li","sequence":"additional","affiliation":[{"name":"School of Artificial Intelligence, Beijing Normal University, China. lijian9026@iie.ac.cn; jli@bnu.edu.cn"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yong","family":"Liu","sequence":"additional","affiliation":[{"name":"Gaoling School of Artificial Intelligence, Renmin University of China, China. liuyonggsai@ruc.edu.cn"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Can","family":"Ma","sequence":"additional","affiliation":[{"name":"Institute of Information Engineering, Chinese Academy of Sciences, China"},{"name":"School of Cyber Security, University of Chinese Academy of Sciences, China. macan@iie.ac.cn"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Weiping","family":"Wang","sequence":"additional","affiliation":[{"name":"Institute of Information Engineering, Chinese Academy of Sciences, China"},{"name":"School of Cyber Security, University of Chinese Academy of Sciences, China. wangweiping@iie.ac.cn"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2024,11,27]]},"reference":[{"key":"2024120216444397500_bib1","article-title":"Generalized knowledge distillation for\n auto-regressive language models","volume-title":"The Twelfth\n International Conference on Learning\n Representations","author":"Agarwal","year":"2024"},{"key":"2024120216444397500_bib2","doi-asserted-by":"publisher","first-page":"10865","DOI":"10.1609\/aaai.v38i10.28960","article-title":"Fluctuation-based adaptive structured pruning for large\n language models","volume-title":"Thirty-Eighth AAAI Conference on\n Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative\n Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on\n Educational Advances in Artificial Intelligence, EAAI 2014, February\n 20\u201327, 2024, Vancouver, Canada","author":"An","year":"2024"},{"key":"2024120216444397500_bib3","article-title":"SliceGPT: Compress large language models\n by deleting rows and columns","volume-title":"The Twelfth\n International Conference on Learning\n Representations","author":"Ashkboos","year":"2024"},{"key":"2024120216444397500_bib4","doi-asserted-by":"publisher","first-page":"18392","DOI":"10.1109\/CVPR52729.2023.01764","article-title":"Instructpix2pix: Learning to follow image\n editing instructions","volume-title":"IEEE\/CVF Conference on\n Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada,\n June 17\u201324, 2023","author":"Brooks","year":"2023"},{"key":"2024120216444397500_bib5","article-title":"Language models are few-shot\n learners","volume-title":"Advances in Neural Information\n Processing Systems 33: Annual Conference on Neural Information Processing\n Systems 2020, NeurIPS 2020, December 6\u201312, 2020,\n virtual","author":"Brown","year":"2020"},{"key":"2024120216444397500_bib6","article-title":"QuIP: 2-bit quantization of large language\n models with guarantees","volume-title":"Thirty-seventh Conference\n on Neural Information Processing Systems","author":"Chee","year":"2023"},{"issue":"2","key":"2024120216444397500_bib7","doi-asserted-by":"publisher","first-page":"29","DOI":"10.1109\/MM.2021.3061394","article-title":"NVIDIA A100 tensor core GPU: Performance\n and innovation","volume":"41","author":"Choquette","year":"2021","journal-title":"IEEE Micro"},{"issue":"70","key":"2024120216444397500_bib8","first-page":"1","article-title":"Scaling instruction-finetuned language\n models","volume":"25","author":"Chung","year":"2024","journal-title":"Journal of Machine Learning Research"},{"key":"2024120216444397500_bib9","article-title":"Training verifiers to solve math word\n problems","volume":"abs\/2110.14168","author":"Cobbe","year":"2021","journal-title":"CoRR"},{"key":"2024120216444397500_bib10","article-title":"Gpt3.int8(): 8-bit matrix multiplication\n for transformers at scale","volume-title":"Advances in Neural\n Information Processing Systems 35: Annual Conference on Neural Information\n Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28\n \u2013 December 9, 2022","author":"Dettmers","year":"2022"},{"key":"2024120216444397500_bib11","article-title":"Qlora: Efficient finetuning of quantized\n llms","volume-title":"Advances in Neural Information Processing\n Systems 36: Annual Conference on Neural Information Processing Systems 2023,\n NeurIPS 2023, New Orleans, LA, USA, December 10 \u2013 16,\n 2023","author":"Dettmers","year":"2023"},{"key":"2024120216444397500_bib12","article-title":"SpQR: A sparse-quantized representation\n for near-lossless LLM weight compression","volume-title":"The\n Twelfth International Conference on Learning\n Representations","author":"Dettmers","year":"2024"},{"key":"2024120216444397500_bib13","article-title":"A survey for in-context learning","volume":"abs\/2301.00234","author":"Dong","year":"2023","journal-title":"CoRR"},{"key":"2024120216444397500_bib14","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.acl-long.7","article-title":"Bitdistiller: Unleashing the potential of\n sub-4-bit llms via self-distillation","volume":"abs\/2402.10631","author":"Dayou","year":"2024","journal-title":"CoRR"},{"key":"2024120216444397500_bib15","first-page":"1126","article-title":"Model-agnostic meta-learning for fast\n adaptation of deep networks","volume-title":"Proceedings of the\n 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW,\n Australia, 6\u201311 August 2017","author":"Finn","year":"2017"},{"key":"2024120216444397500_bib16","article-title":"The lottery ticket hypothesis: Finding\n sparse, trainable neural networks","volume-title":"7th\n International Conference on Learning Representations, ICLR 2019, New\n Orleans, LA, USA, May 6\u20139, 2019","author":"Frankle","year":"2019"},{"key":"2024120216444397500_bib17","article-title":"Optimal brain compression: A framework for\n accurate post-training quantization and pruning","volume-title":"Advances in Neural Information Processing\n Systems","author":"Frantar","year":"2022"},{"key":"2024120216444397500_bib18","first-page":"10323","article-title":"Sparsegpt: Massive language models can be\n accurately pruned in one-shot","volume-title":"International\n Conference on Machine Learning, ICML 2023, 23\u201329 July 2023, Honolulu,\n Hawaii, USA","author":"Frantar","year":"2023"},{"key":"2024120216444397500_bib19","article-title":"OPTQ: Accurate quantization for generative\n pre-trained transformers","volume-title":"The Eleventh\n International Conference on Learning\n Representations","author":"Frantar","year":"2023"},{"key":"2024120216444397500_bib20","first-page":"10421","article-title":"Specializing smaller language models towards multi-step\n reasoning","volume-title":"Proceedings of the 40th International\n Conference on Machine Learning","author":"Yao","year":"2023"},{"key":"2024120216444397500_bib21","article-title":"A framework for few-shot language model\n evaluation","author":"Gao","year":"2023"},{"key":"2024120216444397500_bib22","doi-asserted-by":"publisher","first-page":"346","DOI":"10.1162\/tacl_a_00370","article-title":"Did aristotle use a laptop? A question\n answering benchmark with implicit reasoning strategies","volume":"9","author":"Geva","year":"2021","journal-title":"Transactions of the Association for Computational\n Linguistics"},{"issue":"6","key":"2024120216444397500_bib23","doi-asserted-by":"publisher","first-page":"2325","DOI":"10.1109\/18.720541","article-title":"Quantization","volume":"44","author":"Gray","year":"1998","journal-title":"IEEE\n Transactions on Information Theory"},{"key":"2024120216444397500_bib24","article-title":"MiniLLM: Knowledge distillation of large\n language models","volume-title":"The Twelfth International\n Conference on Learning Representations","author":"Yuxian","year":"2024"},{"key":"2024120216444397500_bib25","doi-asserted-by":"publisher","first-page":"3:1\u20133:15","DOI":"10.1145\/3579371.3589038","article-title":"Olive: Accelerating large language models via\n hardware-friendly outlier-victim pair quantization","volume-title":"Proceedings of the 50th Annual International Symposium on Computer\n Architecture, ISCA 2023, Orlando, FL, USA, June 17\u201321, 2023","author":"Guo","year":"2023"},{"key":"2024120216444397500_bib26","article-title":"Deep compression: Compressing deep neural\n network with pruning, trained quantization and huffman\n coding","volume-title":"4th International Conference on Learning\n Representations, ICLR 2016, San Juan, Puerto Rico, May 2\u20134, 2016,\n Conference Track Proceedings","author":"Han","year":"2016"},{"key":"2024120216444397500_bib27","article-title":"Learning both weights and connections for\n efficient neural network","volume-title":"Advances in Neural\n Information Processing Systems","author":"Han","year":"2015"},{"key":"2024120216444397500_bib28","article-title":"Distilling the knowledge in a neural network","volume":"abs\/1503.02531","author":"Hinton","year":"2015","journal-title":"CoRR"},{"key":"2024120216444397500_bib29","doi-asserted-by":"publisher","first-page":"14852","DOI":"10.18653\/v1\/2023.acl-long.830","article-title":"Large language models are reasoning teachers","volume-title":"Proceedings of the 61st Annual Meeting of the Association for\n Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto,\n Canada, July 9\u201314, 2023","author":"Ho","year":"2023"},{"key":"2024120216444397500_bib30","article-title":"Kvquant: Towards 10 million context length\n LLM inference with KV cache quantization","author":"Hooper","year":"2024","journal-title":"CoRR"},{"key":"2024120216444397500_bib31","doi-asserted-by":"publisher","first-page":"8003","DOI":"10.18653\/v1\/2023.findings-acl.507","article-title":"Distilling step-by-step! Outperforming\n larger language models with less training data and smaller model\n sizes","volume-title":"Findings of the Association for\n Computational Linguistics: ACL 2023, Toronto, Canada, July 9\u201314,\n 2023","author":"Hsieh","year":"2023"},{"key":"2024120216444397500_bib32","article-title":"In-context learning distillation:\n Transferring few-shot learning ability of pre-trained language\n models","author":"Huang","year":"2022","journal-title":"CoRR"},{"key":"2024120216444397500_bib33","article-title":"L4Q: Parameter efficient quantization-aware training on large\n language models via lora-wise LSQ","author":"Jeon","year":"2024","journal-title":"CoRR"},{"key":"2024120216444397500_bib34","doi-asserted-by":"publisher","first-page":"3134","DOI":"10.18653\/v1\/2023.emnlp-main.189","article-title":"Lion: Adversarial distillation of proprietary large language\n models","volume-title":"Proceedings of the 2023 Conference on\n Empirical Methods in Natural Language Processing","author":"Jiang","year":"2023"},{"key":"2024120216444397500_bib35","article-title":"Scaling laws for neural language\n models","author":"Kaplan","year":"2020","journal-title":"CoRR"},{"key":"2024120216444397500_bib36","article-title":"Shortened llama: A simple depth pruning for large language\n models","author":"Kim","year":"2024","journal-title":"ICLR Workshop on Mathematical and Empirical\n Understanding of Foundation Models (ME-FoMo)"},{"key":"2024120216444397500_bib37","article-title":"Memory-efficient fine-tuning of compressed large language\n models via sub-4-bit integer quantization","volume-title":"Thirty-seventh Conference on Neural Information Processing\n Systems","author":"Kim","year":"2023"},{"key":"2024120216444397500_bib38","article-title":"Squeezellm: Dense-and-sparse\n quantization","author":"Kim","year":"2023","journal-title":"CoRR"},{"key":"2024120216444397500_bib39","first-page":"598","article-title":"Optimal brain damage","volume-title":"Advances in Neural Information Processing Systems 2, [NIPS\n Conference, Denver, Colorado, USA, November 27\u201330, 1989]","author":"LeCun","year":"1989"},{"key":"2024120216444397500_bib40","doi-asserted-by":"publisher","first-page":"13355","DOI":"10.1609\/aaai.v38i12.29237","article-title":"OWQ: Outlier-aware weight quantization for efficient\n fine-tuning and inference of large language models","volume-title":"Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024,\n Thirty-Sixth Conference on Innovative Applications of Artificial\n Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in\n Artificial Intelligence, EAAI 2014, February 20\u201327, 2024, Vancouver,\n Canada","author":"Lee","year":"2024"},{"key":"2024120216444397500_bib41","article-title":"Selective reflection-tuning: Student-selected\n data recycling for LLM instruction-tuning","author":"Li","year":"2024","journal-title":"CoRR"},{"key":"2024120216444397500_bib42","article-title":"Explanations from large language models make small reasoners\n better","volume-title":"2nd Workshop on Sustainable\n AI","author":"Li","year":"2024"},{"key":"2024120216444397500_bib43","doi-asserted-by":"publisher","first-page":"18591","DOI":"10.1609\/aaai.v38i17.29821","article-title":"Turning dust into gold: Distilling complex reasoning\n capabilities from llms by leveraging negative data","volume-title":"Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024,\n Thirty-Sixth Conference on Innovative Applications of Artificial\n Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in\n Artificial Intelligence, EAAI 2014, February 20\u201327, 2024, Vancouver,\n Canada","author":"Li","year":"2024"},{"key":"2024120216444397500_bib44","article-title":"Loftq: Lora-fine-tuning-aware quantization\n for large language models","author":"Li","year":"2023","journal-title":"CoRR"},{"key":"2024120216444397500_bib45","article-title":"E-sparse: Boosting the large language model\n inference through entropy-based N: M sparsity","author":"Li","year":"2023","journal-title":"CoRR"},{"issue":"3","key":"2024120216444397500_bib46","doi-asserted-by":"publisher","first-page":"60","DOI":"10.3390\/computers12030060","article-title":"Model compression for deep neural networks: A\n survey","volume":"12","author":"Li","year":"2023","journal-title":"Computers"},{"key":"2024120216444397500_bib47","first-page":"20852","article-title":"Less is more: Task-aware layer-wise distillation for language\n model compression","volume-title":"International Conference on\n Machine Learning, ICML 2023, 23\u201329 July 2023, Honolulu, Hawaii,\n USA","author":"Liang","year":"2023"},{"key":"2024120216444397500_bib48","article-title":"AWQ: Activation-aware weight quantization for LLM compression\n and acceleration","author":"Ji","year":"2023","journal-title":"CoRR"},{"key":"2024120216444397500_bib49","doi-asserted-by":"publisher","first-page":"592","DOI":"10.18653\/v1\/2023.emnlp-main.39","article-title":"LLM-FP4: 4-bit floating-point quantized\n transformers","volume-title":"Proceedings of the 2023 Conference\n on Empirical Methods in Natural Language Processing","author":"Liu","year":"2023"},{"key":"2024120216444397500_bib50","article-title":"Learning to reason with autoregressive in-context\n distillation","volume-title":"The Second Tiny Papers Track at\n ICLR 2024","author":"Liu","year":"2024"},{"key":"2024120216444397500_bib51","article-title":"LLM-QAT: Data-free quantization aware\n training for large language models","author":"Liu","year":"2023","journal-title":"CoRR"},{"key":"2024120216444397500_bib52","article-title":"KIVI: A tuning-free asymmetric 2bit\n quantization for KV cache","author":"Liu","year":"2024","journal-title":"CoRR"},{"key":"2024120216444397500_bib53","article-title":"LLM-pruner: On the structural pruning of large language\n models","volume-title":"Thirty-seventh Conference on Neural\n Information Processing Systems","author":"Ma","year":"2023"},{"key":"2024120216444397500_bib54","doi-asserted-by":"publisher","first-page":"1773","DOI":"10.18653\/v1\/2023.acl-short.151","article-title":"Teaching small language models to\n reason","volume-title":"Proceedings of the 61st Annual Meeting of\n the Association for Computational Linguistics (Volume 2: Short Papers), ACL\n 2023, Toronto, Canada, July 9\u201314, 2023","author":"Magister","year":"2023"},{"issue":"2","key":"2024120216444397500_bib55","doi-asserted-by":"publisher","first-page":"313","DOI":"10.21236\/ADA273556","article-title":"Building a large annotated corpus of\n English: The Penn Treebank","volume":"19","author":"Marcus","year":"1993","journal-title":"Computational\n Linguistics"},{"key":"2024120216444397500_bib56","article-title":"Pointer sentinel mixture\n models","volume-title":"5th International Conference on Learning\n Representations, ICLR 2017, Toulon, France, April 24\u201326, 2017,\n Conference Track Proceedings","author":"Merity","year":"2017"},{"key":"2024120216444397500_bib57","doi-asserted-by":"publisher","first-page":"2381","DOI":"10.18653\/v1\/D18-1260","article-title":"Can a suit of armor conduct electricity? A\n new dataset for open book question answering","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural\n Language Processing, Brussels, Belgium, October 31 \u2013 November 4,\n 2018","author":"Mihaylov","year":"2018"},{"key":"2024120216444397500_bib58","doi-asserted-by":"publisher","first-page":"11264","DOI":"10.1109\/CVPR.2019.01152","article-title":"Importance estimation for neural network\n pruning","volume-title":"IEEE Conference on Computer Vision and\n Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16\u201320,\n 2019","author":"Molchanov","year":"2019"},{"key":"2024120216444397500_bib59","unstructured":"OpenAI. 2024. Gpt-4 technical\n report."},{"key":"2024120216444397500_bib60","article-title":"Training language models to follow instructions with human\n feedback","volume-title":"NeurIPS","author":"Ouyang","year":"2022"},{"key":"2024120216444397500_bib61","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P16-1144","article-title":"The LAMBADA dataset: Word prediction\n requiring a broad discourse context","volume-title":"Proceedings\n of the 54th Annual Meeting of the Association for Computational Linguistics,\n ACL 2016, August 7\u201312, 2016, Berlin, Germany, Volume 1: Long\n Papers","author":"Paperno","year":"2016"},{"key":"2024120216444397500_bib62","article-title":"LUT-GEMM: Quantized matrix multiplication based on LUTs for\n efficient inference in large-scale generative language\n models","volume-title":"The Twelfth International Conference on\n Learning Representations","author":"Park","year":"2024"},{"issue":"8","key":"2024120216444397500_bib63","first-page":"9","article-title":"Language models are unsupervised multitask\n learners","volume":"1","author":"Radford","year":"2019","journal-title":"OpenAI blog"},{"key":"2024120216444397500_bib64","first-page":"140:1\u2013140:67","article-title":"Exploring the limits of transfer learning with a unified\n text-to-text transformer","volume":"21","author":"Raffel","year":"2020","journal-title":"Journal of Machine\n Learning Research"},{"key":"2024120216444397500_bib65","doi-asserted-by":"publisher","first-page":"525","DOI":"10.1007\/978-3-319-46493-0_32","article-title":"Xnor-net: Imagenet classification using\n binary convolutional neural networks","volume-title":"Computer\n Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands,\n October 11\u201314, 2016, Proceedings, Part IV","author":"Rastegari","year":"2016"},{"key":"2024120216444397500_bib66","doi-asserted-by":"publisher","first-page":"842","DOI":"10.1162\/tacl_a_00349","article-title":"A primer in BERTology: What we know about\n how BERT works","volume":"8","author":"Rogers","year":"2020","journal-title":"Transactions of the Association for\n Computational Linguistics"},{"key":"2024120216444397500_bib67","article-title":"Matrix compression via randomized low rank\n and low precision factorization","volume-title":"Advances in\n Neural Information Processing Systems 36: Annual Conference on Neural\n Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA,\n December 10\u201316, 2023","author":"Saha","year":"2023"},{"key":"2024120216444397500_bib68","article-title":"Learning implicitly recurrent cnns through\n parameter sharing","volume-title":"7th International Conference\n on Learning Representations, ICLR 2019, New Orleans, LA, USA, May\n 6\u20139, 2019","author":"Savarese","year":"2019"},{"key":"2024120216444397500_bib69","article-title":"BLOOM: A 176b- parameter open-access\n multilingual language model","author":"Scao","year":"2022","journal-title":"CoRR"},{"key":"2024120216444397500_bib70","doi-asserted-by":"publisher","first-page":"11296","DOI":"10.1109\/ICASSP48485.2024.10445737","article-title":"One-shot sensitivity-aware mixed sparsity\n pruning for large language models","volume-title":"ICASSP 2024\n \u2013 2024 IEEE International Conference on Acoustics, Speech and Signal\n Processing (ICASSP)","author":"Shao","year":"2024"},{"key":"2024120216444397500_bib71","article-title":"Omniquant: Omnidirectionally calibrated quantization for\n large language models","volume-title":"The Twelfth International\n Conference on Learning Representations","author":"Shao","year":"2024"},{"key":"2024120216444397500_bib72","article-title":"The truth is in there: Improving reasoning\n with layer-selective rank reduction","volume-title":"The Twelfth\n International Conference on Learning\n Representations","author":"Sharma","year":"2024"},{"key":"2024120216444397500_bib73","doi-asserted-by":"publisher","first-page":"7059","DOI":"10.18653\/v1\/2023.findings-acl.441","article-title":"Distilling reasoning capabilities into\n smaller language models","volume-title":"Findings of the\n Association for Computational Linguistics: ACL 2023, Toronto, Canada, July\n 9\u201314, 2023","author":"Shridhar","year":"2023"},{"key":"2024120216444397500_bib74","first-page":"720","article-title":"Weighted low-rank\n approximations","volume-title":"Machine Learning, Proceedings of\n the Twentieth International Conference (ICML 2003), August 21\u201324,\n 2003, Washington, DC, USA","author":"Srebro","year":"2003"},{"key":"2024120216444397500_bib75","article-title":"Beyond the imitation game: Quantifying and\n extrapolating the capabilities of language models","author":"Srivastava","year":"2023","journal-title":"Transactions on Machine Learning Research"},{"key":"2024120216444397500_bib76","first-page":"6906","article-title":"Does knowledge distillation really\n work?","volume-title":"Advances in Neural Information Processing\n Systems 34: Annual Conference on Neural Information Processing Systems 2021,\n NeurIPS 2021, December 6\u201314, 2021, virtual","author":"Stanton","year":"2021"},{"key":"2024120216444397500_bib77","article-title":"A simple and effective pruning approach\n for large language models","volume-title":"The Twelfth\n International Conference on Learning\n Representations","author":"Sun","year":"2024"},{"key":"2024120216444397500_bib78","doi-asserted-by":"publisher","first-page":"4149","DOI":"10.18653\/v1\/N19-1421","article-title":"CommonsenseQA: A question answering\n challenge targeting commonsense knowledge","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of\n the Association for Computational Linguistics: Human Language Technologies,\n Volume 1 (Long and Short Papers)","author":"Talmor","year":"2019"},{"key":"2024120216444397500_bib79","doi-asserted-by":"publisher","first-page":"141","DOI":"10.1109\/SSDM.2003.1214975","article-title":"Piqa: An algebra for querying protein data\n sets","volume-title":"Proceedings of the 15th International\n Conference on Scientific and Statistical Database Management (SSDBM 2003),\n 9\u201311 July 2003, Cambridge, MA, USA","author":"Tata","year":"2003"},{"key":"2024120216444397500_bib80","article-title":"Llama: Open and efficient foundation\n language models","author":"Touvron","year":"2023","journal-title":"CoRR"},{"key":"2024120216444397500_bib81","article-title":"Llama 2: Open foundation and fine-tuned\n chat models","author":"Touvron","year":"2023","journal-title":"CoRR"},{"key":"2024120216444397500_bib82","article-title":"GPT-J-6B: A 6 Billion Parameter\n Autoregressive Language Model","author":"Wang","year":"2021"},{"key":"2024120216444397500_bib83","doi-asserted-by":"publisher","first-page":"5546","DOI":"10.18653\/v1\/2023.acl-long.304","article-title":"SCOTT: Self-consistent chain-of-thought\n distillation","volume-title":"Proceedings of the 61st Annual\n Meeting of the Association for Computational Linguistics (Volume 1: Long\n Papers), ACL 2023, Toronto, Canada, July 9\u201314, 2023","author":"Wang","year":"2023"},{"key":"2024120216444397500_bib84","article-title":"Large language models are latent variable\n models: Explaining and finding good demonstrations for in-context\n learning","volume-title":"Thirty-seventh Conference on Neural\n Information Processing Systems","author":"Wang","year":"2023"},{"key":"2024120216444397500_bib85","article-title":"Self-consistency improves chain of thought\n reasoning in language models","volume-title":"The Eleventh\n International Conference on Learning Representations, ICLR 2023, Kigali,\n Rwanda, May 1\u20135, 2023","author":"Wang","year":"2023"},{"key":"2024120216444397500_bib86","doi-asserted-by":"publisher","first-page":"13484","DOI":"10.18653\/v1\/2023.acl-long.754","article-title":"Self-instruct: Aligning language models\n with self-generated instructions","volume-title":"Proceedings of\n the 61st Annual Meeting of the Association for Computational Linguistics\n (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9\u201314,\n 2023","author":"Wang","year":"2023"},{"key":"2024120216444397500_bib87","doi-asserted-by":"publisher","first-page":"8696","DOI":"10.18653\/v1\/2021.emnlp-main.685","article-title":"Codet5: Identifier- aware unified pre-trained encoder-decoder\n models for code understanding and generation","volume-title":"Proceedings of the 2021 Conference on Empirical Methods in Natural\n Language Processing, EMNLP 2021, Virtual Event \/ Punta Cana, Dominican\n Republic, 7\u201311 November, 2021","author":"Wang","year":"2021"},{"key":"2024120216444397500_bib88","doi-asserted-by":"publisher","first-page":"1948","DOI":"10.18653\/v1\/2023.emnlp-main.120","article-title":"Democratizing reasoning ability: Tailored\n learning from large language model","volume-title":"Proceedings\n of the 2023 Conference on Empirical Methods in Natural Language Processing,\n EMNLP 2023, Singapore, December 6\u201310, 2023","author":"Wang","year":"2023"},{"key":"2024120216444397500_bib89","article-title":"Chain-of-thought prompting elicits reasoning in large\n language models","volume-title":"NeurIPS","author":"Wei","year":"2022"},{"key":"2024120216444397500_bib90","doi-asserted-by":"publisher","first-page":"1648","DOI":"10.18653\/v1\/2023.emnlp-main.102","article-title":"Outlier suppression+: Accurate quantization of large language\n models by equivalent and effective shifting and scaling","volume-title":"Proceedings of the 2023 Conference on Empirical Methods in Natural\n Language Processing","author":"Wei","year":"2023"},{"key":"2024120216444397500_bib91","article-title":"Learning structured sparsity in deep neural\n networks","volume-title":"Advances in Neural Information\n Processing Systems","author":"Wen","year":"2016"},{"key":"2024120216444397500_bib92","article-title":"How does calibration data affect the\n post-training pruning and quantization of large language\n models?","author":"Williams","year":"2023","journal-title":"CoRR"},{"issue":"4","key":"2024120216444397500_bib93","doi-asserted-by":"publisher","first-page":"65","DOI":"10.1145\/1498765.1498785","article-title":"Roofline: An insightful visual performance\n model for multicore architectures","volume":"52","author":"Williams","year":"2009","journal-title":"Communications of\n the ACM"},{"key":"2024120216444397500_bib94","first-page":"944","article-title":"LaMini-LM: A diverse herd of distilled models from\n large-scale instructions","volume-title":"Proceedings of the 18th\n Conference of the European Chapter of the Association for Computational\n Linguistics (Volume 1: Long Papers)","author":"Minghao","year":"2024"},{"issue":"2","key":"2024120216444397500_bib95","doi-asserted-by":"publisher","first-page":"211","DOI":"10.14778\/3626292.3626303","article-title":"Flash-llm: Enabling cost-effective and highly-efficient large\n generative model inference with unstructured sparsity","volume":"17","author":"Xia","year":"2023","journal-title":"Proceedings of the VLDB Endowment"},{"key":"2024120216444397500_bib96","article-title":"Sheared LLaMA: Accelerating language model pre-training via\n structured pruning","volume-title":"The Twelfth International\n Conference on Learning Representations","author":"Xia","year":"2024"},{"key":"2024120216444397500_bib97","doi-asserted-by":"publisher","first-page":"7516","DOI":"10.18653\/v1\/2020.emnlp-main.608","article-title":"Which *bert? A survey organizing\n contextualized encoders","volume-title":"Proceedings of the 2020\n Conference on Empirical Methods in Natural Language Processing, EMNLP 2020,\n Online, November 16\u201320, 2020","author":"Xia","year":"2020"},{"key":"2024120216444397500_bib98","first-page":"38087","article-title":"Smoothquant: Accurate and efficient post-training\n quantization for large language models","volume-title":"International Conference on Machine Learning, ICML 2023,\n 23\u201329 July 2023, Honolulu, Hawaii, USA","author":"Xiao","year":"2023"},{"key":"2024120216444397500_bib99","doi-asserted-by":"publisher","first-page":"10653","DOI":"10.18653\/v1\/2021.emnlp-main.832","article-title":"Beyond preserved accuracy: Evaluating loyalty and robustness\n of BERT compression","volume-title":"Proceedings of the 2021\n Conference on Empirical Methods in Natural Language Processing, EMNLP 2021,\n Virtual Event \/ Punta Cana, Dominican Republic, 7\u201311 November,\n 2021","author":"Canwen","year":"2021"},{"key":"2024120216444397500_bib100","article-title":"Onebit: Towards extremely low-bit large language\n models","author":"Yuzhuang","year":"2024","journal-title":"CoRR"},{"key":"2024120216444397500_bib101","article-title":"Zeroquant: Efficient and affordable post-training\n quantization for large-scale transformers","volume-title":"NeurIPS","author":"Yao","year":"2022"},{"key":"2024120216444397500_bib102","article-title":"Zeroquant-v2: Exploring post-training quantization in llms\n from comprehensive study to low rank compensation","author":"Yao","year":"2023","journal-title":"CoRR"},{"key":"2024120216444397500_bib103","article-title":"RPTQ: Reorder-based post-training\n quantization for large language models","author":"Yuan","year":"2023","journal-title":"CoRR"},{"key":"2024120216444397500_bib104","article-title":"ASVD: Activation-aware singular value decomposition for\n compressing large language models","author":"Yuan","year":"2023","journal-title":"CoRR"},{"key":"2024120216444397500_bib105","article-title":"Wkvquant: Quantizing weight and key\/value cache for large\n language models gains more","author":"Yue","year":"2024","journal-title":"CoRR"},{"key":"2024120216444397500_bib106","article-title":"OPT: Open pre-trained transformer language\n models","author":"Zhang","year":"2022","journal-title":"CoRR"},{"key":"2024120216444397500_bib107","article-title":"Dynamic sparse no training: Training-free fine-tuning for\n sparse LLMs","volume-title":"The Twelfth International Conference\n on Learning Representations","author":"Zhang","year":"2024"},{"key":"2024120216444397500_bib108","article-title":"A survey of large language models","author":"Zhao","year":"2023","journal-title":"CoRR"},{"key":"2024120216444397500_bib109","article-title":"Judging llm-as-a-judge with mt-bench and\n chatbot arena","volume-title":"Advances in Neural Information\n Processing Systems 36: Annual Conference on Neural Information Processing\n Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 \u2013 16,\n 2023","author":"Zheng","year":"2023"},{"key":"2024120216444397500_bib110","doi-asserted-by":"publisher","first-page":"2571","DOI":"10.18653\/v1\/2024.naacl-long.142","article-title":"PaD: Program-aided distillation can teach small models\n reasoning better than chain-of-thought fine-tuning","volume-title":"Proceedings of the 2024 Conference of the North American Chapter of\n the Association for Computational Linguistics: Human Language Technologies\n (Volume 1: Long Papers)","author":"Zhu","year":"2024"},{"key":"2024120216444397500_bib111","article-title":"Neural architecture search with reinforcement\n learning","volume-title":"5th International Conference on\n Learning Representations, ICLR 2017, Toulon, France, April 24\u201326,\n 2017, Conference Track Proceedings","author":"Zoph","year":"2017"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00704\/2482209\/tacl_a_00704.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00704\/2482209\/tacl_a_00704.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,12,2]],"date-time":"2024-12-02T11:45:09Z","timestamp":1733139909000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00704\/125482\/A-Survey-on-Model-Compression-for-Large-Language"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024]]},"references-count":111,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00704","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024]]},"published":{"date-parts":[[2024]]}}}