{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,6]],"date-time":"2026-05-06T18:17:30Z","timestamp":1778091450930,"version":"3.51.4"},"reference-count":102,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2026,5,6]],"date-time":"2026-05-06T00:00:00Z","timestamp":1778025600000},"content-version":"vor","delay-in-days":125,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2026,5,4]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>To enhance the efficiency of the attention mechanism within large language models (LLMs), previous works primarily compress the Key-Value cache or group attention heads, while largely overlooking redundancy between layers. Our comprehensive analyses across various LLMs show that highly similar attention patterns persist within most layers. It\u2019s intuitive to reduce the redundancy by sharing attention weights across layers. However, further analysis reveals two challenges: (1) Directly sharing the weight matrix without carefully rearranging the attention heads proves to be ineffective; (2) Shallow layers are vulnerable to small deviations in attention weights. Driven by these insights, we introduce LiSA, a lightweight substitute for self-attention in well-trained LLMs. LiSA employs tiny feed-forward networks to align attention heads between adjacent layers and low-rank matrices to approximate differences in layer-wise attention weights. Evaluations encompassing 13 typical benchmarks demonstrate that LiSA maintains high response quality in terms of accuracy and perplexity while reducing redundant attention calculations within 53% \u221284% of the total layers. Our implementations of LiSA achieve a 6 \u00d7 compression of Q and K matrices within the attention mechanism, with maximum throughput improvements 19.5%, 32.3%, and 40.1% for LLaMA3-8B, LLaMA2-7B, and LLaMA2-13B, respectively. Our code is available at https:\/\/github.com\/takagi97\/lisa.<\/jats:p>","DOI":"10.1162\/tacl.a.616","type":"journal-article","created":{"date-parts":[[2026,5,6]],"date-time":"2026-05-06T18:00:55Z","timestamp":1778090455000},"page":"656-688","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":0,"title":["Cross-layer Attention Sharing for Pre-trained Large Language Models"],"prefix":"10.1162","volume":"14","author":[{"given":"Yongyu","family":"Mu","sequence":"first","affiliation":[{"name":"NLP Lab, School of Computer Science and Engineering, Northeastern University, Shenyang, China. lixiaoyumu9@gmail.com"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yuzhang","family":"Wu","sequence":"additional","affiliation":[{"name":"NLP Lab, School of Computer Science and Engineering, Northeastern University, Shenyang, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yuchun","family":"Fan","sequence":"additional","affiliation":[{"name":"NLP Lab, School of Computer Science and Engineering, Northeastern University, Shenyang, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chenglong","family":"Wang","sequence":"additional","affiliation":[{"name":"NLP Lab, School of Computer Science and Engineering, Northeastern University, Shenyang, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hengyu","family":"Li","sequence":"additional","affiliation":[{"name":"NLP Lab, School of Computer Science and Engineering, Northeastern University, Shenyang, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jiali","family":"Zeng","sequence":"additional","affiliation":[{"name":"Pattern Recognition Center, WeChat AI, Tencent Inc, China. lemonzeng@tencent.com"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Qiaozhi","family":"He","sequence":"additional","affiliation":[{"name":"NLP Lab, School of Computer Science and Engineering, Northeastern University, Shenyang, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Murun","family":"Yang","sequence":"additional","affiliation":[{"name":"NLP Lab, School of Computer Science and Engineering, Northeastern University, Shenyang, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Fandong","family":"Meng","sequence":"additional","affiliation":[{"name":"Pattern Recognition Center, WeChat AI, Tencent Inc, China. fandongmeng@tencent.com"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jie","family":"Zhou","sequence":"additional","affiliation":[{"name":"Pattern Recognition Center, WeChat AI, Tencent Inc, China. withtomzhou@tencent.com"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tong","family":"Xiao","sequence":"additional","affiliation":[{"name":"NLP Lab, School of Computer Science and Engineering, Northeastern University, Shenyang, China. xiaotong@mail.neu.edu.cn"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jingbo","family":"Zhu","sequence":"additional","affiliation":[{"name":"NLP Lab, School of Computer Science and Engineering, Northeastern University, Shenyang, China. zhujingbo@mail.neu.edu.cn"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","published-online":{"date-parts":[[2026,5,4]]},"reference":[{"key":"2026050614005001800_bib1","article-title":"Keyformer: KV cache reduction through key tokens selection for efficient generative inference","volume-title":"Proceedings of the Seventh Annual Conference on Machine Learning and Systems, MLSys 2024, Santa Clara, CA, USA, May 13\u201316, 2024","author":"Adnan","year":"2024"},{"key":"2026050614005001800_bib2","article-title":"CHAI: Clustered head attention for efficient LLM inference","volume-title":"Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21\u201327, 2024","author":"Agarwal","year":"2024"},{"key":"2026050614005001800_bib3","unstructured":"AI@Meta. 2024. Llama 3 model card."},{"key":"2026050614005001800_bib4","doi-asserted-by":"publisher","first-page":"4895","DOI":"10.18653\/v1\/2023.emnlp-main.298","article-title":"GQA: Training generalized multi-query transformer models from multi-head checkpoints","volume-title":"Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6\u201310, 2023","author":"Ainslie","year":"2023"},{"key":"2026050614005001800_bib5","doi-asserted-by":"publisher","first-page":"268","DOI":"10.18653\/v1\/2020.emnlp-main.19","article-title":"ETC: Encoding long and structured inputs in transformers","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16\u201320, 2020","author":"Ainslie","year":"2020"},{"key":"2026050614005001800_bib6","article-title":"Think you have solved direct-answer question answering? Try ARC-DA, the Direct-Answer AI2 reasoning challenge","author":"Bhakthavatsalam","year":"2021","journal-title":"CoRR"},{"key":"2026050614005001800_bib7","doi-asserted-by":"publisher","first-page":"7432","DOI":"10.1609\/aaai.v34i05.6239","article-title":"PIQA: Reasoning about physical commonsense in natural language","volume-title":"The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7\u201312, 2020","author":"Bisk","year":"2020"},{"key":"2026050614005001800_bib8","doi-asserted-by":"crossref","DOI":"10.52202\/079017-2758","article-title":"Reducing transformer key-value cache size with cross-layer attention","volume-title":"Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10\u201315, 2024","author":"Brandon","year":"2024"},{"key":"2026050614005001800_bib9","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2406.02069","article-title":"PyramidKV: Dynamic KV cache compression based on pyramidal information funneling","author":"Cai","year":"2024","journal-title":"CoRR"},{"key":"2026050614005001800_bib10","article-title":"Rethinking attention with performers","volume-title":"9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3\u20137, 2021","author":"Choromanski","year":"2021"},{"key":"2026050614005001800_bib11","first-page":"2924","article-title":"BoolQ: Exploring the surprising difficulty of natural yes\/no questions","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2\u20137, 2019, Volume 1 (Long and Short Papers)","author":"Clark","year":"2019"},{"key":"2026050614005001800_bib12","article-title":"Training verifiers to solve math word problems","author":"Cobbe","year":"2021","journal-title":"CoRR"},{"key":"2026050614005001800_bib13","article-title":"RedPajama: An open source recipe to reproduce LLaMA training dataset","author":"Computer","year":"2023"},{"key":"2026050614005001800_bib14","article-title":"FlashAttention-2: Faster attention with better parallelism and work partitioning","volume-title":"The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\u201311, 2024","author":"Dao","year":"2024"},{"key":"2026050614005001800_bib15","doi-asserted-by":"publisher","DOI":"10.52202\/068431-1189","article-title":"FlashAttention: Fast and memory-efficient exact attention with IO-awareness","volume-title":"Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28\u2013December 9, 2022","author":"Dao","year":"2022"},{"key":"2026050614005001800_bib16","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2405.04434","article-title":"DeepSeek-V2: A strong, economical, and efficient mixture-of-experts language model","author":"DeepSeek-AI","year":"2024","journal-title":"CoRR"},{"key":"2026050614005001800_bib17","article-title":"GPT3.int8(): 8-bit matrix multiplication for transformers at scale","volume-title":"Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28\u2013December 9, 2022","author":"Dettmers","year":"2022"},{"key":"2026050614005001800_bib18","first-page":"4171","article-title":"BERT: Pre-training of deep bidirectional transformers for language understanding","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2\u20137, 2019, Volume 1 (Long and Short Papers)","author":"Devlin","year":"2019"},{"key":"2026050614005001800_bib19","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.emnlp-main.1027","article-title":"A simple and effective L_2 norm-based strategy for KV cache compression","author":"Devoto","year":"2024","journal-title":"arXiv preprint arXiv: 2406.11430"},{"key":"2026050614005001800_bib20","doi-asserted-by":"publisher","first-page":"2498","DOI":"10.18653\/v1\/2021.naacl-main.198","article-title":"WEC: Deriving a large-scale cross- document event coreference dataset from Wikipedia","volume-title":"Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6\u201311, 2021","author":"Eirew","year":"2021"},{"key":"2026050614005001800_bib21","article-title":"Reducing transformer depth on demand with structured dropout","volume-title":"8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26\u201330, 2020","author":"Fan","year":"2020"},{"key":"2026050614005001800_bib22","doi-asserted-by":"publisher","first-page":"5083","DOI":"10.24963\/ijcai.2025\/566","article-title":"Not all layers of LLMs are necessary during inference","volume-title":"Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI 2025, Montreal, Canada, August 16\u201322, 2025","author":"Fan","year":"2025"},{"issue":"11","key":"2026050614005001800_bib23","doi-asserted-by":"publisher","first-page":"1573","DOI":"10.1109\/TVLSI.2022.3197282","article-title":"An algorithm-hardware co-optimized framework for accelerating N: M sparse transformers","volume":"30","author":"Fang","year":"2022","journal-title":"IEEE Transactions on Very Large Scale Integration Systems"},{"key":"2026050614005001800_bib24","article-title":"A framework for few-shot language model evaluation","author":"Gao","year":"2023"},{"key":"2026050614005001800_bib25","article-title":"Model tells you what to discard: adaptive KV cache compression for LLMs","volume-title":"Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\u201311, 2024","author":"Ge","year":"2024"},{"key":"2026050614005001800_bib26","first-page":"2214","article-title":"The reversible residual network: Backpropagation without storing activations","volume-title":"Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4\u20139, 2017, Long Beach, CA, USA","author":"Gomez","year":"2017"},{"key":"2026050614005001800_bib27","doi-asserted-by":"publisher","first-page":"143","DOI":"10.18653\/v1\/2020.repl4nlp-1.18","article-title":"Compressing BERT: Studying the effects of weight pruning on transfer learning","volume-title":"Proceedings of the 5th Workshop on Representation Learning for NLP, RepL4NLP@ACL 2020, Online, July 9, 2020","author":"Gordon","year":"2020"},{"key":"2026050614005001800_bib28","article-title":"The unreasonable ineffectiveness of the deeper layers","volume-title":"The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24\u201328, 2025","author":"Gromov","year":"2025"},{"key":"2026050614005001800_bib29","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.emnlp-main.1178","article-title":"Attention score is not all you need for token importance indicator in KV cache reduction: Value also matters","author":"Guo","year":"2024","journal-title":"arXiv preprint arXiv:2406.12335"},{"key":"2026050614005001800_bib30","doi-asserted-by":"publisher","first-page":"328","DOI":"10.1109\/HPCA47549.2020.00035","article-title":"A3: Accelerating attention mechanisms in neural networks with approximation","volume-title":"IEEE International Symposium on High Performance Computer Architecture, HPCA 2020, San Diego, CA, USA, February 22\u201326, 2020","author":"Ham","year":"2020"},{"key":"2026050614005001800_bib31","doi-asserted-by":"publisher","first-page":"3610","DOI":"10.21437\/Interspeech.2020-2059","article-title":"ContextNet: Improving convolutional neural networks for automatic speech recognition with global context","volume-title":"21st Annual Conference of the International Speech Communication Association, Interspeech 2020, Virtual Event, Shanghai, China, October 25\u201329, 2020","author":"Han","year":"2020"},{"key":"2026050614005001800_bib32","article-title":"Measuring massive multitask language understanding","volume-title":"9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3\u20137, 2021","author":"Hendrycks","year":"2021"},{"key":"2026050614005001800_bib33","article-title":"LoRA: Low-rank adaptation of large language models","volume-title":"The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25\u201329, 2022","author":"Edward","year":"2022"},{"key":"2026050614005001800_bib34","doi-asserted-by":"publisher","first-page":"492","DOI":"10.1007\/978-1-4612-4380-9_35","article-title":"Robust estimation of a location parameter","volume-title":"Breakthroughs in Statistics: Methodology and Distribution","author":"Huber","year":"1992"},{"key":"2026050614005001800_bib35","article-title":"Perceiver IO: A General architecture for structured inputs & outputs","volume-title":"Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25\u201329, 2022","author":"Jaegle","year":"2022"},{"key":"2026050614005001800_bib36","doi-asserted-by":"publisher","first-page":"4163","DOI":"10.18653\/v1\/2020.findings-emnlp.372","article-title":"TinyBERT: Distilling BERT for natural language understanding","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16\u201320 November 2020","author":"Jiao","year":"2020"},{"key":"2026050614005001800_bib37","doi-asserted-by":"publisher","first-page":"1601","DOI":"10.18653\/v1\/P17-1147","article-title":"TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension","volume-title":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30\u2013August 4, Volume 1: Long Papers","author":"Joshi","year":"2017"},{"key":"2026050614005001800_bib38","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2403.05527","article-title":"GEAR: An efficient KV Cache compression recipe for near-lossless generative inference of LLM","author":"Kang","year":"2024","journal-title":"CoRR"},{"key":"2026050614005001800_bib39","first-page":"5156","article-title":"Transformers are RNNs: Fast autoregressive transformers with linear attention","volume-title":"Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13\u201318 July 2020, Virtual Event","author":"Katharopoulos","year":"2020"},{"key":"2026050614005001800_bib40","first-page":"2765","article-title":"Paraphrasing complex network: Network compression via factor transfer","volume-title":"Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3\u20138, 2018, Montr\u00e9al, Canada","author":"Kim","year":"2018"},{"key":"2026050614005001800_bib41","first-page":"5506","article-title":"I-BERT: Integer-only BERT quantization","volume-title":"Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18\u201324 July 2021, Virtual Event","author":"Kim","year":"2021"},{"key":"2026050614005001800_bib42","article-title":"Reformer: The efficient transformer","volume-title":"8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26\u201330, 2020","author":"Kitaev","year":"2020"},{"key":"2026050614005001800_bib43","doi-asserted-by":"publisher","first-page":"1","DOI":"10.18653\/v1\/2024.wmt-1.1","article-title":"Findings of the WMT24 general machine translation shared task: The LLM era is here but MT is not solved yet","volume-title":"Proceedings of the Ninth Conference on Machine Translation, WMT 2024, Miami, FL, USA, November 15\u201316, 2024","author":"Kocmi","year":"2024"},{"key":"2026050614005001800_bib44","doi-asserted-by":"publisher","first-page":"452","DOI":"10.1162\/TACL_A_00276","article-title":"Natural questions: A benchmark for question answering research","volume":"7","author":"Kwiatkowski","year":"2019","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2026050614005001800_bib45","article-title":"ALBERT: A lite BERT for self-supervised learning of language representations","volume-title":"8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26\u201330, 2020","author":"Lan","year":"2020"},{"key":"2026050614005001800_bib46","article-title":"Alpacaeval: An automatic evaluator of instruction-following models","author":"Li","year":"2023"},{"issue":"3","key":"2026050614005001800_bib47","doi-asserted-by":"publisher","first-page":"4037","DOI":"10.1007\/S11042-020-09276-9","article-title":"An efficient framework for counting pedestrians crossing a line using low-cost devices: the benefits of distilling the knowledge in a neural network","volume":"80","author":"Lin","year":"2021","journal-title":"Multimedia Tools and Applications"},{"key":"2026050614005001800_bib48","article-title":"MiniCache: KV cache compression in depth dimension for large language models","volume-title":"Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10\u201315, 2024","author":"Liu","year":"2024"},{"key":"2026050614005001800_bib49","article-title":"Generating Wikipedia by summarizing long sequences","volume-title":"6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30\u2013May 3, 2018, Conference Track Proceedings","author":"Liu","year":"2018"},{"key":"2026050614005001800_bib50","article-title":"Scissorhands: Exploiting the persistence of importance hypothesis for LLM KV cache compression at test time","volume-title":"Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10\u201316, 2023","author":"Liu","year":"2023"},{"key":"2026050614005001800_bib51","first-page":"22137","article-title":"Deja Vu: Contextual sparsity for efficient LLMs at inference time","volume-title":"International Conference on Machine Learning, ICML 2023, 23\u201329 July 2023, Honolulu, Hawaii, USA","author":"Liu","year":"2023"},{"key":"2026050614005001800_bib52","doi-asserted-by":"publisher","first-page":"1412","DOI":"10.18653\/v1\/D15-1166","article-title":"Effective approaches to attention-based neural machine translation","volume-title":"Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17\u201321, 2015","author":"Luong","year":"2015"},{"key":"2026050614005001800_bib53","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2503.23924","article-title":"Model hemorrhage and the robustness limits of large language models","author":"Ma","year":"2025","journal-title":"CoRR"},{"key":"2026050614005001800_bib54","doi-asserted-by":"publisher","first-page":"3225","DOI":"10.18653\/v1\/2020.coling-main.287","article-title":"LadaBERT: Lightweight adaptation of BERT through hybrid model compression","volume-title":"Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8\u201313, 2020","author":"Mao","year":"2020"},{"issue":"5","key":"2026050614005001800_bib55","doi-asserted-by":"publisher","first-page":"90:1\u201390:30","DOI":"10.1145\/3527155","article-title":"Split computing and early exiting for deep learning applications: Survey and research challenges","volume":"55","author":"Matsubara","year":"2023","journal-title":"ACM Computing Surveys"},{"key":"2026050614005001800_bib56","doi-asserted-by":"publisher","first-page":"20192","DOI":"10.18653\/v1\/2025.findings-acl.1035","article-title":"ShortGPT: Layers in large language models are more redundant than you expect","volume-title":"Findings of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, July 27\u2013August 1, 2025","author":"Men","year":"2025"},{"key":"2026050614005001800_bib57","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2403.08295","article-title":"Gemma: Open models based on gemini research and technology","author":"Mesnard","year":"2024","journal-title":"CoRR"},{"key":"2026050614005001800_bib58","first-page":"14014","article-title":"Are sixteen heads really better than one?","volume-title":"Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8\u201314, 2019, Vancouver, BC, Canada","author":"Michel","year":"2019"},{"key":"2026050614005001800_bib59","doi-asserted-by":"publisher","first-page":"2381","DOI":"10.18653\/v1\/D18-1260","article-title":"Can a suit of armor conduct electricity? A new dataset for open book question answering","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31\u2013November 4, 2018","author":"Mihaylov","year":"2018"},{"key":"2026050614005001800_bib60","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P16-1144","article-title":"The LAMBADA dataset: Word prediction requiring a broad discourse context","volume-title":"Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7\u201312, 2016, Berlin, Germany, Volume 1: Long Papers","author":"Paperno","year":"2016"},{"key":"2026050614005001800_bib61","first-page":"4052","article-title":"Image transformer","volume-title":"Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsm\u00e4ssan, Stockholm, Sweden, July 10\u201315, 2018","author":"Parmar","year":"2018"},{"key":"2026050614005001800_bib62","doi-asserted-by":"publisher","first-page":"283","DOI":"10.1007\/978-3-030-01252-6_17","article-title":"Learning deep representations with probabilistic knowledge transfer","volume-title":"Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8\u201314, 2018, Proceedings, Part XI","author":"Passalis","year":"2018"},{"key":"2026050614005001800_bib63","doi-asserted-by":"publisher","first-page":"529","DOI":"10.18653\/v1\/2021.blackboxnlp-1.42","article-title":"Fine-tuned transformers show clusters of similar representations across layers","volume-title":"Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, BlackboxNLP@EMNLP 2021, Punta Cana, Dominican Republic, November 11, 2021","author":"Phang","year":"2021"},{"key":"2026050614005001800_bib64","doi-asserted-by":"publisher","first-page":"1031","DOI":"10.18653\/v1\/2023.wmt-1.98","article-title":"One wide feedforward is all you need","volume-title":"Proceedings of the Eighth Conference on Machine Translation, WMT 2023, Singapore, December 6\u20137, 2023","author":"Pires","year":"2023"},{"issue":"8","key":"2026050614005001800_bib65","first-page":"9","article-title":"Language models are unsupervised multitask learners","volume":"1","author":"Radford","year":"2019","journal-title":"OpenAI blog"},{"key":"2026050614005001800_bib66","article-title":"ZeRO: Memory optimization towards training a trillion parameter models","author":"Rajbhandari","year":"2019","journal-title":"CoRR"},{"key":"2026050614005001800_bib67","doi-asserted-by":"publisher","first-page":"249","DOI":"10.1162\/TACL_a_00266","article-title":"CoQA: A conversational question answering challenge","volume":"7","author":"Reddy","year":"2019","journal-title":"Transactions of the Association for Computational Linguistics,"},{"key":"2026050614005001800_bib68","article-title":"FitNets: Hints for thin deep nets","volume-title":"3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7\u20139, 2015, Conference Track Proceedings","author":"Romero","year":"2015"},{"key":"2026050614005001800_bib69","doi-asserted-by":"publisher","first-page":"53","DOI":"10.1162\/TACL_a_00353","article-title":"Efficient content-based sparse attention with routing transformers","volume":"9","author":"Roy","year":"2021","journal-title":"Transactions of the Association for Computational Linguistics"},{"issue":"9","key":"2026050614005001800_bib70","doi-asserted-by":"publisher","first-page":"99","DOI":"10.1145\/3474381","article-title":"WinoGrande: An adversarial Winograd schema challenge at scale","volume":"64","author":"Sakaguchi","year":"2021","journal-title":"Communications of the ACM"},{"key":"2026050614005001800_bib71","article-title":"Movement Pruning: Adaptive sparsity by fine-tuning","volume-title":"Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6\u201312, 2020, virtual","author":"Sanh","year":"2020"},{"key":"2026050614005001800_bib72","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2211.05100","article-title":"BLOOM: A 176B-parameter open-access multilingual language model","author":"Scao","year":"2022","journal-title":"CoRR"},{"key":"2026050614005001800_bib73","article-title":"Fast transformer decoding: One write-head is all you need","author":"Shazeer","year":"2019","journal-title":"CoRR"},{"key":"2026050614005001800_bib74","doi-asserted-by":"publisher","first-page":"8815","DOI":"10.1609\/aaai.v34i05.6409","article-title":"Q-BERT: Hessian based ultra low precision quantization of BERT","volume-title":"The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7\u201312, 2020","author":"Shen","year":"2020"},{"key":"2026050614005001800_bib75","doi-asserted-by":"publisher","first-page":"3723","DOI":"10.21437\/Interspeech.2018-1910","article-title":"Self-attentional acoustic models","volume-title":"19th Annual Conference of the International Speech Communication Association, Interspeech 2018, Hyderabad, India, September 2\u20136, 2018","author":"Sperber","year":"2018"},{"key":"2026050614005001800_bib76","article-title":"You only cache once: Decoder-decoder architectures for language models","volume-title":"Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10\u201315, 2024","author":"Sun","year":"2024"},{"key":"2026050614005001800_bib77","doi-asserted-by":"publisher","first-page":"2158","DOI":"10.18653\/v1\/2020.acl-main.195","article-title":"MobileBERT: A compact task-agnostic BERT for resource-limited devices","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5\u201310, 2020","author":"Sun","year":"2020"},{"key":"2026050614005001800_bib78","article-title":"Stanford alpaca: An instruction-following llama model","author":"Taori","year":"2023"},{"issue":"6","key":"2026050614005001800_bib79","doi-asserted-by":"publisher","first-page":"109:1","DOI":"10.1145\/3530811","article-title":"Efficient transformers: A survey","volume":"55","author":"Yi","year":"2023","journal-title":"ACM Computing Surveys"},{"key":"2026050614005001800_bib80","doi-asserted-by":"publisher","first-page":"2464","DOI":"10.1109\/ICPR.2016.7900006","article-title":"BranchyNet: Fast inference via early exiting from deep neural networks","volume-title":"23rd International Conference on Pattern Recognition, ICPR 2016, Canc\u00fan, Mexico, December 4\u20138, 2016","author":"Teerapittayanon","year":"2016"},{"key":"2026050614005001800_bib81","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2302.13971","article-title":"LLaMA: Open and efficient foundation language models","author":"Touvron","year":"2023","journal-title":"CoRR"},{"key":"2026050614005001800_bib82","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2307.09288","article-title":"Llama 2: Open foundation and fine-tuned chat models","author":"Touvron","year":"2023","journal-title":"CoRR"},{"key":"2026050614005001800_bib83","article-title":"Revisiting Linformer with a modified self-attention with linear complexity","author":"Verma","year":"2021","journal-title":"CoRR"},{"key":"2026050614005001800_bib84","doi-asserted-by":"publisher","first-page":"5797","DOI":"10.18653\/v1\/P19-1580","article-title":"Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned","volume-title":"Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28\u2013August 2, 2019, Volume 1: Long Papers","author":"Voita","year":"2019"},{"key":"2026050614005001800_bib85","doi-asserted-by":"publisher","first-page":"11389","DOI":"10.18653\/v1\/2024.findings-acl.676","article-title":"Hybrid alignment training for large language models","volume-title":"Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11\u201316, 2024","author":"Wang","year":"2024"},{"key":"2026050614005001800_bib86","doi-asserted-by":"publisher","first-page":"7675","DOI":"10.18653\/v1\/2020.acl-main.686","article-title":"HAT: Hardware-aware transformers for efficient natural language processing","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5\u201310, 2020","author":"Wang","year":"2020"},{"key":"2026050614005001800_bib87","article-title":"Linformer: Self-attention with linear complexity","author":"Wang","year":"2020","journal-title":"CoRR"},{"key":"2026050614005001800_bib88","article-title":"Chain-of-thought prompting elicits reasoning in large language models","volume-title":"Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28\u2013December 9, 2022","author":"Wei","year":"2022"},{"key":"2026050614005001800_bib89","doi-asserted-by":"publisher","first-page":"11175","DOI":"10.18653\/v1\/2024.acl-long.602","article-title":"Layer-condensed KV cache for efficient inference of large language models","volume-title":"Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11\u201316, 2024","author":"Haoyi","year":"2024"},{"key":"2026050614005001800_bib90","article-title":"Sheared LLaMA: Accelerating language model pre-training via structured pruning","volume-title":"The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\u201311, 2024","author":"Xia","year":"2024"},{"key":"2026050614005001800_bib91","article-title":"Efficient streaming language models with attention sinks","volume-title":"The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\u201311, 2024","author":"Xiao","year":"2024"},{"key":"2026050614005001800_bib92","doi-asserted-by":"publisher","first-page":"5292","DOI":"10.24963\/ijcai.2019\/735","article-title":"Sharing attention weights for fast transformer","volume-title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10\u201316, 2019","author":"Xiao","year":"2019"},{"key":"2026050614005001800_bib93","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2311.17633","article-title":"Introduction to transformers: An NLP perspective","author":"Xiao","year":"2023","journal-title":"CoRR"},{"key":"2026050614005001800_bib94","article-title":"Few-shot task-agnostic neural architecture search for distilling large language models","volume-title":"Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28\u2013December 9, 2022","author":"Dongkuan","year":"2022"},{"key":"2026050614005001800_bib95","doi-asserted-by":"publisher","first-page":"1933","DOI":"10.1145\/3447548.3467262","article-title":"NAS-BERT: Task-agnostic and adaptive-size BERT compression with neural architecture search","volume-title":"KDD \u201921: The 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, Singapore, August 14\u201318, 2021","author":"Jin","year":"2021"},{"key":"2026050614005001800_bib96","doi-asserted-by":"publisher","first-page":"3258","DOI":"10.18653\/v1\/2024.findings-acl.195","article-title":"PyramidInfer: Pyramid KV cache compression for high-throughput LLM inference","volume-title":"Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11\u201316, 2024","author":"Yang","year":"2024"},{"key":"2026050614005001800_bib97","doi-asserted-by":"publisher","first-page":"4791","DOI":"10.18653\/v1\/P19-1472","article-title":"HellaSwag: Can a machine really finish your sentence?","volume-title":"Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28\u2013August 2, 2019, Volume 1: Long Papers","author":"Zellers","year":"2019"},{"key":"2026050614005001800_bib98","doi-asserted-by":"publisher","first-page":"1789","DOI":"10.18653\/v1\/P18-1166","article-title":"Accelerating neural transformer via an average attention network","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15\u201320, 2018, Volume 1: Long Papers","author":"Zhang","year":"2018"},{"key":"2026050614005001800_bib99","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2205.01068","article-title":"OPT: Open pre-trained transformer language models","author":"Zhang","year":"2022","journal-title":"CoRR"},{"key":"2026050614005001800_bib100","article-title":"H2O: Heavy-hitter oracle for efficient generative inference of large language models","volume-title":"Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10\u201316, 2023","author":"Zhang","year":"2023"},{"key":"2026050614005001800_bib101","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2310.14928","article-title":"Unveiling A core linguistic region in large language models","author":"Zhao","year":"2023","journal-title":"CoRR"},{"key":"2026050614005001800_bib102","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.acl-demos.38","article-title":"Llamafactory: Unified efficient fine-tuning of 100+ language models","volume-title":"Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)","author":"Zheng","year":"2024"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/TACL.a.616\/2598700\/tacl.a.616.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/TACL.a.616\/2598700\/tacl.a.616.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,6]],"date-time":"2026-05-06T18:01:09Z","timestamp":1778090469000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/TACL.a.616\/136548\/Cross-layer-Attention-Sharing-for-Pre-trained"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026]]},"references-count":102,"URL":"https:\/\/doi.org\/10.1162\/tacl.a.616","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026]]},"published":{"date-parts":[[2026]]}}}