{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,7]],"date-time":"2026-07-07T12:06:36Z","timestamp":1783425996365,"version":"3.54.6"},"reference-count":82,"publisher":"MIT Press - Journals","license":[{"start":{"date-parts":[[2021,9,23]],"date-time":"2021-09-23T00:00:00Z","timestamp":1632355200000},"content-version":"vor","delay-in-days":265,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,9,21]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Pre-trained Transformer-based models have achieved state-of-the-art performance for various Natural Language Processing (NLP) tasks. However, these models often have billions of parameters, and thus are too resource- hungry and computation-intensive to suit low- capability devices or applications with strict latency requirements. One potential remedy for this is model compression, which has attracted considerable research attention. Here, we summarize the research in compressing Transformers, focusing on the especially popular BERT model. In particular, we survey the state of the art in compression for BERT, we clarify the current best practices for compressing large-scale Transformer models, and we provide insights into the workings of various methods. Our categorization and analysis also shed light on promising future research directions for achieving lightweight, accurate, and generic NLP models.<\/jats:p>","DOI":"10.1162\/tacl_a_00413","type":"journal-article","created":{"date-parts":[[2021,9,24]],"date-time":"2021-09-24T03:51:45Z","timestamp":1632455505000},"page":"1061-1080","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":119,"title":["Compressing Large-Scale Transformer-Based Models: A Case Study on BERT"],"prefix":"10.1162","volume":"9","author":[{"given":"Prakhar","family":"Ganesh","sequence":"first","affiliation":[{"name":"Advanced Digital Sciences Center, Singapore. prakhar.g@adsc-create.edu.sg"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yao","family":"Chen","sequence":"additional","affiliation":[{"name":"Advanced Digital Sciences Center, Singapore. yao.chen@adsc-create.edu.sg"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xin","family":"Lou","sequence":"additional","affiliation":[{"name":"Advanced Digital Sciences Center, Singapore. lou.xin@adsc-create.edu.sg"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mohammad Ali","family":"Khan","sequence":"additional","affiliation":[{"name":"Advanced Digital Sciences Center, Singapore. mohammad.k@adsc-create.edu.sg"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yin","family":"Yang","sequence":"additional","affiliation":[{"name":"College of Science and Engineering, Hamad Bin Khalifa University, Qatar. yyang@hbku.edu.qa"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hassan","family":"Sajjad","sequence":"additional","affiliation":[{"name":"Qatar Computing Research Institute, Hamad Bin Khalifa University, Qatar. hsajjad@hbku.edu.qa"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Preslav","family":"Nakov","sequence":"additional","affiliation":[{"name":"Qatar Computing Research Institute, Hamad Bin Khalifa University, Qatar. pnakov@hbku.edu.qa"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Deming","family":"Chen","sequence":"additional","affiliation":[{"name":"University of Illinois at Urbana-Champaign, USA. dchen@illinois.edu"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marianne","family":"Winslett","sequence":"additional","affiliation":[{"name":"University of Illinois at Urbana-Champaign, USA. winslett@illinois.edu"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2021,9,21]]},"reference":[{"key":"2021121611065257400_bib1","first-page":"1753","article-title":"Fixed- point optimization of transformer neural network","volume-title":"Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing","author":"Boo","year":"2020"},{"key":"2021121611065257400_bib2","first-page":"1877","article-title":"Language models are few-shot learners","volume-title":"Advances in Neural Information Processing Systems","author":"Brown","year":"2020"},{"key":"2021121611065257400_bib3","doi-asserted-by":"crossref","first-page":"4487","DOI":"10.18653\/v1\/2020.acl-main.411","article-title":"DeFormer: Decomposing pre-trained transformers for faster question answering","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Cao","year":"2020"},{"key":"2021121611065257400_bib4","doi-asserted-by":"publisher","first-page":"2463","DOI":"10.24963\/ijcai.2020\/341","article-title":"AdaBERT: Task-adaptive BERT compression with differentiable neural architecture search","volume-title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020","author":"Chen","year":"2020"},{"key":"2021121611065257400_bib5","first-page":"1753","article-title":"The lottery ticket hypothesis for pre-trained BERT networks","volume-title":"Proceedings of the 34th Conference on Neural Information Processing Systems","author":"Chen","year":"2020"},{"issue":"1","key":"2021121611065257400_bib6","doi-asserted-by":"publisher","first-page":"126","DOI":"10.1109\/MSP.2017.2765695","article-title":"Model compression and acceleration for deep neural networks: The principles, progress, and challenges","volume":"35","author":"Cheng","year":"2018","journal-title":"IEEE Signal Processing Magazine"},{"key":"2021121611065257400_bib7","article-title":"Transformer to CNN: Label- scarce distillation for efficient text classification","volume-title":"Proceedings of the Compact Deep Neural Network Representation with Industrial Applications Workshop","author":"Chia","year":"2018"},{"key":"2021121611065257400_bib8","article-title":"Weight squeezing: Reparameterization for extreme compression and fast inference","author":"Chumachenko","year":"2020","journal-title":"arXiv:2010.06993"},{"key":"2021121611065257400_bib9","doi-asserted-by":"publisher","first-page":"276","DOI":"10.18653\/v1\/W19-4828","article-title":"What does BERT look at? An analysis of BERT\u2019s attention","volume-title":"Proceedings of the ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP","author":"Clark","year":"2019"},{"key":"2021121611065257400_bib10","doi-asserted-by":"publisher","first-page":"403","DOI":"10.1007\/978-1-4614-6940-7_15","article-title":"Multi-objective optimization","volume-title":"Search Methodologies","author":"Deb","year":"2014"},{"key":"2021121611065257400_bib11","first-page":"4171","article-title":"BERT: Pre-training of deep bidirectional transformers for language understanding","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Devlin","year":"2019"},{"key":"2021121611065257400_bib12","doi-asserted-by":"publisher","first-page":"420","DOI":"10.1109\/ICBK50248.2020.00066","article-title":"SDSK2BERT: Explore the specific depth with specific knowledge to compress BERT","volume-title":"Proceedings of the IEEE International Conference on Knowledge Graph","author":"Ding","year":"2020"},{"key":"2021121611065257400_bib13","article-title":"Reducing transformer depth on demand with structured dropout","volume-title":"Proceedings of the 8th International Conference on Learning Representations","author":"Fan","year":"2020"},{"key":"2021121611065257400_bib14","article-title":"Training with quantization noise for extreme model compression","volume-title":"Proceedings of the 10th International Conference on Learning Representations","author":"Fan","year":"2021"},{"issue":"4","key":"2021121611065257400_bib15","doi-asserted-by":"publisher","first-page":"681","DOI":"10.1007\/s11023-020-09548-1","article-title":"GPT-3: Its nature, scope, limits, and consequences","volume":"30","author":"Floridi","year":"2020","journal-title":"Minds and Machines"},{"key":"2021121611065257400_bib16","article-title":"The lottery ticket hypothesis: Finding sparse, trainable neural networks","volume-title":"Proceedings of the 7th International Conference on Learning Representations","author":"Frankle","year":"2019"},{"key":"2021121611065257400_bib17","doi-asserted-by":"publisher","first-page":"143","DOI":"10.18653\/v1\/2020.repl4nlp-1.18","article-title":"Compressing BERT: Studying the effects of weight pruning on transfer learning","volume-title":"Proceedings of the 5th Workshop on Representation Learning for NLP","author":"Gordon","year":"2020"},{"key":"2021121611065257400_bib18","first-page":"3690","article-title":"PoWER-BERT: Accelerating BERT inference via progressive word-vector elimination","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Goyal","year":"2020"},{"key":"2021121611065257400_bib19","article-title":"Reweighted proximal pruning for large-scale language representation","author":"Guo","year":"2019","journal-title":"arXiv:1909.12486"},{"key":"2021121611065257400_bib20","article-title":"DynaBERT: Dynamic BERT with adaptive width and depth","volume-title":"Advances in Neural Information Processing Systems","author":"Hou","year":"2020"},{"key":"2021121611065257400_bib21","doi-asserted-by":"publisher","first-page":"328","DOI":"10.18653\/v1\/P18-1031","article-title":"Universal language model fine-tuning for text classification","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics","author":"Howard","year":"2018"},{"issue":"1","key":"2021121611065257400_bib22","first-page":"6869","article-title":"Quantized neural networks: Training neural networks with low precision weights and activations","volume":"18","author":"Hubara","year":"2017","journal-title":"The Journal of Machine Learning Research"},{"key":"2021121611065257400_bib23","doi-asserted-by":"publisher","first-page":"4163","DOI":"10.18653\/v1\/2020.findings-emnlp.372","article-title":"TinyBERT: Distilling BERT for natural language understanding","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings","author":"Jiao","year":"2020"},{"key":"2021121611065257400_bib24","doi-asserted-by":"publisher","first-page":"2807","DOI":"10.18653\/v1\/2020.acl-main.250","article-title":"schuBERT: Optimizing elements of BERT","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Khetan","year":"2020"},{"key":"2021121611065257400_bib25","article-title":"Reformer: The efficient transformer","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Kitaev","year":"2020"},{"key":"2021121611065257400_bib26","first-page":"4365","article-title":"Revealing the dark secrets of BERT","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing","author":"Kovaleva","year":"2019"},{"key":"2021121611065257400_bib27","article-title":"ALBERT: A lite BERT for self-supervised learning of language representations","volume-title":"Proceedings of the 8th International Conference on Learning Representations","author":"Lan","year":"2020"},{"key":"2021121611065257400_bib28","doi-asserted-by":"crossref","first-page":"3187","DOI":"10.18653\/v1\/2020.findings-emnlp.286","article-title":"Efficient transformer-based large scale language representations using hardware- friendly block structured pruning","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2020","author":"Li","year":"2020"},{"key":"2021121611065257400_bib29","first-page":"3009","article-title":"BERT-EMD: Many-to-many layer mapping for BERT compression with earth mover\u2019s distance","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing","author":"Li","year":"2020"},{"key":"2021121611065257400_bib30","first-page":"5958","article-title":"Train big, then compress: Rethinking model size for efficient training and inference of transformers","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Li","year":"2020"},{"key":"2021121611065257400_bib31","doi-asserted-by":"publisher","first-page":"719","DOI":"10.18653\/v1\/2020.findings-emnlp.64","article-title":"Pruning redundant mappings in transformer models via spectral-normalized identity prior","volume-title":"Findings of the 2020 Conference on Empirical Methods in Natural Language Processing","author":"Zi","year":"2020"},{"key":"2021121611065257400_bib32","article-title":"MKD: A multi-task knowledge distillation approach for pretrained language models","author":"Liu","year":"2019","journal-title":"arXiv:1911.03588"},{"key":"2021121611065257400_bib33","doi-asserted-by":"crossref","first-page":"6035","DOI":"10.18653\/v1\/2020.acl-main.537","article-title":"FastBERT: A self-distilling BERT with adaptive inference time","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Liu","year":"2020"},{"key":"2021121611065257400_bib34","article-title":"RoBERTa: A robustly optimized BERT pretraining approach","author":"Liu","year":"2019","journal-title":"arXiv:1907. 11692"},{"key":"2021121611065257400_bib35","doi-asserted-by":"publisher","first-page":"293","DOI":"10.3115\/v1\/W14-3336","article-title":"Results of the WMT14 metrics shared task","volume-title":"Proceedings of the Ninth Workshop on Statistical Machine Translation","author":"Machacek","year":"2014"},{"key":"2021121611065257400_bib36","doi-asserted-by":"crossref","first-page":"3225","DOI":"10.18653\/v1\/2020.coling-main.287","article-title":"LadaBERT: Lightweight adaptation of BERT through hybrid model compression","volume-title":"Proceedings of the 28th International Conference on Computational Linguistics","author":"Mao","year":"2020"},{"key":"2021121611065257400_bib37","first-page":"14014","article-title":"Are sixteen heads really better than one?","volume-title":"Advances in Neural Information Processing Systems","author":"Michel","year":"2019"},{"key":"2021121611065257400_bib38","doi-asserted-by":"publisher","first-page":"2221","DOI":"10.18653\/v1\/2020.acl-main.202","article-title":"XtremeDistil: Multi-stage distillation for massive multilingual models","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Mukherjee","year":"2020"},{"key":"2021121611065257400_bib39","doi-asserted-by":"publisher","first-page":"1797","DOI":"10.18653\/v1\/D18-1206","article-title":"Don\u2019t give me the details, just the summary! Topic-aware convolutional neural networks for extreme summarization","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Narayan","year":"2018"},{"key":"2021121611065257400_bib40","first-page":"884","article-title":"Compressing pre-trained language models by matrix decomposition","volume-title":"Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing","author":"Noach","year":"2020"},{"key":"2021121611065257400_bib41","doi-asserted-by":"publisher","first-page":"2227","DOI":"10.18653\/v1\/N18-1202","article-title":"Deep contextualized word representations","volume-title":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Peters","year":"2018"},{"key":"2021121611065257400_bib42","doi-asserted-by":"publisher","first-page":"4711","DOI":"10.18653\/v1\/2020.findings-emnlp.423","article-title":"Compressing transformer-based semantic parsing models using compositional code embeddings","volume-title":"Findings of the 2020 Conference on Empirical Methods in Natural Language Processing","author":"Prakash","year":"2020"},{"key":"2021121611065257400_bib43","doi-asserted-by":"publisher","first-page":"3208","DOI":"10.18653\/v1\/2020.emnlp-main.259","article-title":"When BERT plays the lottery, all tickets are winning","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing","author":"Prasanna","year":"2020"},{"issue":"10","key":"2021121611065257400_bib44","doi-asserted-by":"publisher","first-page":"1872","DOI":"10.1007\/s11431-020-1647-3","article-title":"Pre-trained models for natural language processing: A survey","volume":"63","author":"Qiu","year":"2020","journal-title":"Science China Technological Sciences"},{"issue":"8","key":"2021121611065257400_bib45","first-page":"9","article-title":"Language models are unsupervised multitask learners","volume":"1","author":"Radford","year":"2019","journal-title":"OpenAI Blog"},{"key":"2021121611065257400_bib46","first-page":"1","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","volume":"21","author":"Raffel","year":"2020","journal-title":"Journal of Machine Learning Research"},{"key":"2021121611065257400_bib47","doi-asserted-by":"publisher","first-page":"556","DOI":"10.18653\/v1\/2020.findings-emnlp.49","article-title":"Fixed encoder self- attention patterns in transformer-based machine translation","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2020","author":"Raganato","year":"2020"},{"key":"2021121611065257400_bib48","doi-asserted-by":"publisher","first-page":"784","DOI":"10.18653\/v1\/P18-2124","article-title":"Know what you don\u2019t know: Unanswerable questions for SQuAD","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics","author":"Rajpurkar","year":"2018"},{"key":"2021121611065257400_bib49","doi-asserted-by":"publisher","first-page":"2383","DOI":"10.18653\/v1\/D16-1264","article-title":"SQuAD: 100,000+ questions for machine comprehension of text","volume-title":"Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing","author":"Rajpurkar","year":"2016"},{"key":"2021121611065257400_bib50","doi-asserted-by":"publisher","first-page":"842","DOI":"10.1162\/tacl_a_00349","article-title":"A primer in BERTology: What we know about how BERT works","volume":"8","author":"Rogers","year":"2020","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2021121611065257400_bib51","first-page":"13","article-title":"Turing-NLG: A 17-billion- parameter language model by Microsoft","volume":"2","author":"Rosset","year":"2020","journal-title":"Microsoft Research Blog"},{"key":"2021121611065257400_bib52","article-title":"Poor man\u2019s BERT: Smaller and faster transformer models","author":"Sajjad","year":"2020","journal-title":"arXiv: 2004.03844"},{"key":"2021121611065257400_bib53","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i05.6409","article-title":"DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter","volume-title":"Proceedings of the 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing","author":"Sanh","year":"2019"},{"key":"2021121611065257400_bib54","article-title":"Movement pruning: Adaptive sparsity by fine-tuning","volume-title":"Advances in Neural Information Processing Systems","author":"Sanh","year":"2020"},{"key":"2021121611065257400_bib55","first-page":"8815","article-title":"Q-BERT: Hessian based ultra low precision quantization of BERT","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","author":"Shen","year":"2020"},{"key":"2021121611065257400_bib56","article-title":"Megatron-LM: Training multi-billion parameter language models using model parallelism","author":"Shoeybi","year":"2019","journal-title":"arXiv:1909.08053"},{"key":"2021121611065257400_bib57","article-title":"LightPAFF: A two-stage distillation framework for pre-training and fine-tuning","author":"Song","year":"2020","journal-title":"arXiv:2004.12817"},{"key":"2021121611065257400_bib58","doi-asserted-by":"publisher","first-page":"4314","DOI":"10.18653\/v1\/D19-1441","article-title":"Patient knowledge distillation for BERT model compression","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing","author":"Sun","year":"2019"},{"key":"2021121611065257400_bib59","doi-asserted-by":"publisher","first-page":"498","DOI":"10.18653\/v1\/2020.emnlp-main.36","article-title":"Contrastive distillation on intermediate representations for language model compression","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing","author":"Sun","year":"2020"},{"key":"2021121611065257400_bib60","doi-asserted-by":"crossref","first-page":"2158","DOI":"10.18653\/v1\/2020.acl-main.195","article-title":"MobileBERT: A compact task-agnostic BERT for resource-limited devices","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Sun","year":"2020"},{"key":"2021121611065257400_bib61","article-title":"EdgeBERT: Sentence-level energy optimizations for latency-aware multi- task NLP inference","author":"Tambe","year":"2020","journal-title":"arXiv:2011.14203"},{"key":"2021121611065257400_bib62","doi-asserted-by":"publisher","first-page":"202","DOI":"10.18653\/v1\/D19-6122","article-title":"Natural language generation for effective knowledge distillation","volume-title":"Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP","author":"Tang","year":"2019"},{"key":"2021121611065257400_bib63","article-title":"Synthesizer: Rethinking self-attention in transformer models","author":"Yi","year":"2020","journal-title":"arXiv:2005.00743"},{"key":"2021121611065257400_bib64","article-title":"WaLDORf: Wasteless language-model distillation on reading-comprehension","author":"Tian","year":"2019","journal-title":"arXiv:1912.06638"},{"key":"2021121611065257400_bib65","article-title":"Finding fast transformers: One-shot neural architecture search by component composition","author":"Tsai","year":"2020","journal-title":"arXiv:2008.06808"},{"key":"2021121611065257400_bib66","first-page":"6000","article-title":"Attention is all you need","volume-title":"Proceedings of the 31st International Conference on Neural Information Processing Systems","author":"Vaswani","year":"2017"},{"key":"2021121611065257400_bib67","doi-asserted-by":"publisher","first-page":"353","DOI":"10.18653\/v1\/W18-5446","article-title":"GLUE: A multi-task benchmark and analysis platform for natural language understanding","volume-title":"Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP","author":"Wang","year":"2018"},{"issue":"3","key":"2021121611065257400_bib68","doi-asserted-by":"crossref","first-page":"717","DOI":"10.1007\/s00607-019-00768-7","article-title":"A survey of word embeddings based on deep learning","volume":"102","author":"Wang","year":"2020","journal-title":"Computing"},{"key":"2021121611065257400_bib69","doi-asserted-by":"publisher","DOI":"10.1007\/s00607-019-00768-7","article-title":"Linformer: Self-attention with linear complexity","author":"Wang","year":"2020","journal-title":"arXiv: 2006.04768"},{"key":"2021121611065257400_bib70","article-title":"MiniLM: Deep self-attention distillation for task-agnostic compression of pre-trained transformers","volume-title":"Advances in Neural Information Processing Systems","author":"Wang","year":"2020"},{"key":"2021121611065257400_bib71","doi-asserted-by":"publisher","first-page":"35","DOI":"10.18653\/v1\/2020.sustainlp-1.5","article-title":"Exploring the boundaries of low-resource BERT distillation","volume-title":"Proceedings of the Workshop on Simple and Efficient Natural Language Processing","author":"Wasserblat","year":"2020"},{"key":"2021121611065257400_bib72","article-title":"Google\u2019s neural machine translation system: Bridging the gap between human and machine translation","author":"Yonghui","year":"2016","journal-title":"arXiv:1609.08144"},{"key":"2021121611065257400_bib73","doi-asserted-by":"publisher","first-page":"2246","DOI":"10.18653\/v1\/2020.acl-main.204","article-title":"DeeBERT: Dynamic early exiting for accelerating BERT inference","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Ji","year":"2020"},{"key":"2021121611065257400_bib74","first-page":"7859","article-title":"BERT-of-Theseus: Compressing BERT by progressive module replacing","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing","author":"Canwen","year":"2020"},{"key":"2021121611065257400_bib75","first-page":"5753","article-title":"XLNet: Generalized autoregressive pretraining for language understanding","volume-title":"Advances in Neural Information Processing Systems","author":"Yang","year":"2019"},{"key":"2021121611065257400_bib76","doi-asserted-by":"publisher","first-page":"811","DOI":"10.1109\/MICRO50266.2020.00071","article-title":"GOBO: Quantizing attention-based NLP models for low latency and energy efficient inference","volume-title":"Proceedings of the 53rd Annual IEEE\/ACM International Symposium on Microarchitecture","author":"Zadeh","year":"2020"},{"key":"2021121611065257400_bib77","doi-asserted-by":"publisher","DOI":"10.1109\/EMC2-NIPS53020.2019.00016","article-title":"Q8BERT: Quantized 8bit BERT","volume-title":"Proceedings of the 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing","author":"Zafrir","year":"2019"},{"key":"2021121611065257400_bib78","article-title":"Explicit sparse transformer: Concentrated attention through explicit selection","author":"Zhao","year":"2019","journal-title":"arXiv:1912.11637"},{"key":"2021121611065257400_bib79","article-title":"Extreme language model compression with optimal subwords and shared projections","author":"Zhao","year":"2019","journal-title":"arXiv:1909.11687"},{"key":"2021121611065257400_bib80","first-page":"11546","article-title":"Go wide, then narrow: Efficient training of deep thin networks","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Zhou","year":"2020"},{"key":"2021121611065257400_bib81","article-title":"BERT loses patience: Fast and robust inference with early exit","volume-title":"Advances in Neural Information Processing Systems","author":"Zhou","year":"2020"},{"key":"2021121611065257400_bib82","doi-asserted-by":"publisher","first-page":"19","DOI":"10.1109\/ICCV.2015.11","article-title":"Aligning books and movies: Towards story-like visual explanations by watching movies and reading books","volume-title":"Proceedings of the IEEE International Conference on Computer Vision","author":"Zhu","year":"2015"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00413\/1964006\/tacl_a_00413.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00413\/1964006\/tacl_a_00413.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,12,16]],"date-time":"2021-12-16T11:12:53Z","timestamp":1639653173000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00413\/107387\/Compressing-Large-Scale-Transformer-Based-Models-A"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021]]},"references-count":82,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00413","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021]]},"published":{"date-parts":[[2021]]}}}