{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,25]],"date-time":"2026-10-25T03:45:57Z","timestamp":1792899957970,"version":"4.5.0"},"reference-count":96,"publisher":"Association for Computing Machinery (ACM)","issue":"6","license":[{"start":{"date-parts":[[2022,12,7]],"date-time":"2022-12-07T00:00:00Z","timestamp":1670371200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nd\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Comput. Surv."],"published-print":{"date-parts":[[2023,6,30]]},"abstract":"<jats:p>\n                    Transformer model architectures have garnered immense interest lately due to their effectiveness across a range of domains like language, vision, and reinforcement learning. In the field of natural language processing for example, Transformers have become an indispensable staple in the modern deep learning stack. Recently, a dizzying number of\n                    <jats:italic>\u201cX-former\u201d<\/jats:italic>\n                    models have been proposed\u2014Reformer, Linformer, Performer, Longformer, to name a few\u2014which improve upon the original Transformer architecture, many of which make improvements around computational and memory\n                    <jats:italic>efficiency<\/jats:italic>\n                    . With the aim of helping the avid researcher navigate this flurry, this article characterizes a large and thoughtful selection of recent efficiency-flavored \u201cX-former\u201d models, providing an organized and comprehensive overview of existing work and models across multiple domains.\n                  <\/jats:p>\n                  <jats:p\/>","DOI":"10.1145\/3530811","type":"journal-article","created":{"date-parts":[[2022,4,22]],"date-time":"2022-04-22T11:28:19Z","timestamp":1650626899000},"page":"1-28","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":890,"title":["Efficient Transformers: A Survey"],"prefix":"10.1145","volume":"55","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6896-4496","authenticated-orcid":false,"given":"Yi","family":"Tay","sequence":"first","affiliation":[{"name":"Google Research, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9772-1095","authenticated-orcid":false,"given":"Mostafa","family":"Dehghani","sequence":"additional","affiliation":[{"name":"Google Research, Amsterdam"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0144-2911","authenticated-orcid":false,"given":"Dara","family":"Bahri","sequence":"additional","affiliation":[{"name":"Google Research, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4276-6269","authenticated-orcid":false,"given":"Donald","family":"Metzler","sequence":"additional","affiliation":[{"name":"Google Research, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2022,12,7]]},"reference":[{"key":"e_1_3_2_2_2","article-title":"Ranking via sinkhorn propagation","author":"Adams Ryan Prescott","year":"2011","unstructured":"Ryan Prescott Adams and Richard S. Zemel. 2011. Ranking via sinkhorn propagation. arXiv preprint arXiv:1106.1925 (2011).","journal-title":"arXiv preprint arXiv:1106.1925"},{"key":"e_1_3_2_3_2","article-title":"Weighted transformer network for machine translation","author":"Ahmed Karim","year":"2017","unstructured":"Karim Ahmed, Nitish Shirish Keskar, and Richard Socher. 2017. Weighted transformer network for machine translation. arXiv preprint arXiv:1711.02132 (2017).","journal-title":"arXiv preprint arXiv:1711.02132"},{"key":"e_1_3_2_4_2","article-title":"ETC: Encoding long and structured data in transformers","author":"Ainslie Joshua","year":"2020","unstructured":"Joshua Ainslie, Santiago Ontanon, Chris Alberti, Philip Pham, Anirudh Ravula, and Sumit Sanghai. 2020. ETC: Encoding long and structured data in transformers. In Proceedings of EMNLP (2020).","journal-title":"Proceedings of EMNLP"},{"key":"e_1_3_2_5_2","article-title":"Layer normalization","author":"Ba Jimmy Lei","year":"2016","unstructured":"Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450 (2016).","journal-title":"arXiv preprint arXiv:1607.06450"},{"key":"e_1_3_2_6_2","article-title":"Longformer: The long-document transformer","author":"Beltagy Iz","year":"2020","unstructured":"Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150 (2020).","journal-title":"arXiv preprint arXiv:2004.05150"},{"key":"e_1_3_2_7_2","article-title":"Language models are few-shot learners","author":"Brown Tom B.","year":"2020","unstructured":"Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et\u00a0al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 (2020).","journal-title":"arXiv preprint arXiv:2005.14165"},{"key":"e_1_3_2_8_2","article-title":"End-to-end object detection with transformers","author":"Carion Nicolas","year":"2020","unstructured":"Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020. End-to-end object detection with transformers. arXiv preprint arXiv:2005.12872 (2020).","journal-title":"arXiv preprint arXiv:2005.12872"},{"key":"e_1_3_2_9_2","volume-title":"Thirty-Fifth Conference on Neural Information Processing Systems","author":"Chen Beidi","year":"2021","unstructured":"Beidi Chen, Tri Dao, Eric Winsor, Zhao Song, Atri Rudra, and Christopher R\u00e9. 2021. Scatterbrain: Unifying sparse and low-rank attention. In Thirty-Fifth Conference on Neural Information Processing Systems."},{"key":"e_1_3_2_10_2","article-title":"Generating long sequences with sparse transformers","author":"Child Rewon","year":"2019","unstructured":"Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509 (2019).","journal-title":"arXiv preprint arXiv:1904.10509"},{"key":"e_1_3_2_11_2","article-title":"Masked language modeling for proteins via linearly scalable long-context transformers","author":"Choromanski Krzysztof","year":"2020","unstructured":"Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Jared Davis, Tamas Sarlos, David Belanger, Lucy Colwell, and Adrian Weller. 2020. Masked language modeling for proteins via linearly scalable long-context transformers. In Proceedings of ICLR (2020).","journal-title":"In Proceedings of ICLR"},{"key":"e_1_3_2_12_2","article-title":"Rethinking attention with performers","author":"Choromanski Krzysztof","year":"2020","unstructured":"Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et\u00a0al. 2020. Rethinking attention with performers. In Proceedings of ICLR (2020).","journal-title":"Proceedings of ICLR"},{"key":"e_1_3_2_13_2","article-title":"PaLM: Scaling language modeling with pathways","author":"Chowdhery Aakanksha","year":"2022","unstructured":"Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et\u00a0al. 2022. PaLM: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 (2022).","journal-title":"arXiv preprint arXiv:2204.02311"},{"key":"e_1_3_2_14_2","article-title":"Fast and accurate deep network learning by exponential linear units (ELUS)","author":"Clevert Djork-Arn\u00e9","year":"2015","unstructured":"Djork-Arn\u00e9 Clevert, Thomas Unterthiner, and Sepp Hochreiter. 2015. Fast and accurate deep network learning by exponential linear units (ELUS). In Proceedings of ICLR 2016 (2015).","journal-title":"Proceedings of ICLR 2016"},{"key":"e_1_3_2_15_2","article-title":"Adaptively sparse transformers","author":"Correia Gon\u00e7alo M.","year":"2019","unstructured":"Gon\u00e7alo M. Correia, Vlad Niculae, and Andr\u00e9 F. T. Martins. 2019. Adaptively sparse transformers. In Proceedings of EMNLP (2019).","journal-title":"Proceedings of EMNLP"},{"key":"e_1_3_2_16_2","article-title":"Funnel-transformer: Filtering out sequential redundancy for efficient language processing","author":"Dai Zihang","year":"2020","unstructured":"Zihang Dai, Guokun Lai, Yiming Yang, and Quoc V. Le. 2020. Funnel-transformer: Filtering out sequential redundancy for efficient language processing. In Proceedings of NeurIPS (2020).","journal-title":"Proceedings of NeurIPS"},{"key":"e_1_3_2_17_2","article-title":"Transformer-XL: Attentive language models beyond a fixed-length context","author":"Dai Zihang","year":"2019","unstructured":"Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of ACL (2019).","journal-title":"Proceedings of ACL"},{"key":"e_1_3_2_18_2","article-title":"The efficiency misnomer","author":"Dehghani Mostafa","year":"2021","unstructured":"Mostafa Dehghani, Anurag Arnab, Lucas Beyer, Ashish Vaswani, and Yi Tay. 2021. The efficiency misnomer. arXiv preprint arXiv:2110.12894 (2021).","journal-title":"arXiv preprint arXiv:2110.12894"},{"key":"e_1_3_2_19_2","article-title":"Universal transformers","author":"Dehghani Mostafa","year":"2018","unstructured":"Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and \u0141ukasz Kaiser. 2018. Universal transformers. In Proceedings of ICLR (2018).","journal-title":"Proceedings of ICLR"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"e_1_3_2_21_2","article-title":"BERT:: Pre-training of deep bidirectional transformers for language understanding","author":"Devlin Jacob","year":"2018","unstructured":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT:: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL (2018).","journal-title":"Proceedings of NAACL"},{"key":"e_1_3_2_22_2","unstructured":"Nan Du Yanping Huang Andrew M. Dai Simon Tong Dmitry Lepikhin Yuanzhong Xu Maxim Krikun Yanqi Zhou Wei Yu Adams Orhan Firat Barret Zoph Liam Fedus Maarten Bosma Zongwei Zhou Tao Wang Yu Emma Wang Kellie Webster Marie Pellat Kevin Robinson Kathy Meier-Hellstern Toju Duke Lucas Dixon Kun Zhang Quoc V. Le Yonghui Wu Zhifeng Chen and Claire Cui. 2021. GLaM: Efficient scaling of language models with mixture-of-experts. arxiv:2112.06905 [cs.CL]."},{"key":"e_1_3_2_23_2","article-title":"Training with quantization noise for extreme fixed-point compression","author":"Fan Angela","year":"2020","unstructured":"Angela Fan, Pierre Stock, Benjamin Graham, Edouard Grave, Remi Gribonval, Herve Jegou, and Armand Joulin. 2020. Training with quantization noise for extreme fixed-point compression. arXiv preprint arXiv:2004.07320 (2020).","journal-title":"arXiv preprint arXiv:2004.07320"},{"key":"e_1_3_2_24_2","article-title":"Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity","author":"Fedus William","year":"2021","unstructured":"William Fedus, Barret Zoph, and Noam Shazeer. 2021. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961 (2021).","journal-title":"arXiv preprint arXiv:2101.03961"},{"key":"e_1_3_2_25_2","article-title":"Efficiently modeling long sequences with structured state spaces","author":"Gu Albert","year":"2021","unstructured":"Albert Gu, Karan Goel, and Christopher R\u00e9. 2021. Efficiently modeling long sequences with structured state spaces. In Proceedings of NeurIPS (2021).","journal-title":"Proceedings of NeurIPS"},{"key":"e_1_3_2_26_2","article-title":"Star-transformer","author":"Guo Qipeng","year":"2019","unstructured":"Qipeng Guo, Xipeng Qiu, Pengfei Liu, Yunfan Shao, Xiangyang Xue, and Zheng Zhang. 2019. Star-transformer. In Proceedings of NAACL (2019).","journal-title":"Proceedings of NAACL"},{"key":"e_1_3_2_27_2","first-page":"737","volume-title":"Advances in Neural Information Processing Systems","author":"Guo Yong","year":"2019","unstructured":"Yong Guo, Yin Zheng, Mingkui Tan, Qi Chen, Jian Chen, Peilin Zhao, and Junzhou Huang. 2019. NAT: Neural architecture transformer for accurate and compact architectures. In Advances in Neural Information Processing Systems. 737\u2013748."},{"key":"e_1_3_2_28_2","article-title":"Distilling the knowledge in a neural network","author":"Hinton Geoffrey","year":"2015","unstructured":"Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015).","journal-title":"arXiv preprint arXiv:1503.02531"},{"key":"e_1_3_2_29_2","article-title":"Axial attention in multidimensional transformers","author":"Ho Jonathan","year":"2019","unstructured":"Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. 2019. Axial attention in multidimensional transformers. arXiv preprint arXiv:1912.12180 (2019).","journal-title":"arXiv preprint arXiv:1912.12180"},{"key":"e_1_3_2_30_2","article-title":"Parameter-efficient transfer learning for NLP","author":"Houlsby Neil","year":"2019","unstructured":"Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of ICML (2019).","journal-title":"Proceedings of ICML"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00069"},{"key":"e_1_3_2_32_2","article-title":"Perceiver IO: A general architecture for structured inputs & outputs","author":"Jaegle Andrew","year":"2021","unstructured":"Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et\u00a0al. 2021. Perceiver IO: A general architecture for structured inputs & outputs. arXiv preprint arXiv:2107.14795 (2021).","journal-title":"arXiv preprint arXiv:2107.14795"},{"key":"e_1_3_2_33_2","article-title":"Sparse is enough in scaling transformers","volume":"34","author":"Jaszczur Sebastian","year":"2021","unstructured":"Sebastian Jaszczur, Aakanksha Chowdhery, Afroz Mohiuddin, \u0141ukasz Kaiser, Wojciech Gajewski, Henryk Michalewski, and Jonni Kanerva. 2021. Sparse is enough in scaling transformers. Advances in Neural Information Processing Systems 34 (2021).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_34_2","article-title":"TINYBERT: Distilling BERT for natural language understanding","author":"Jiao Xiaoqi","year":"2019","unstructured":"Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2019. TINYBERT: Distilling BERT for natural language understanding. arXiv preprint arXiv:1909.10351 (2019).","journal-title":"arXiv preprint arXiv:1909.10351"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-1147"},{"key":"e_1_3_2_36_2","article-title":"Depthwise separable convolutions for neural machine translation","author":"Kaiser Lukasz","year":"2017","unstructured":"Lukasz Kaiser, Aidan N. Gomez, and Francois Chollet. 2017. Depthwise separable convolutions for neural machine translation. In Proceedings of ICLR (2017).","journal-title":"Proceedings of ICLR"},{"key":"e_1_3_2_37_2","article-title":"Transformers are RNNs: Fast autoregressive transformers with linear attention","author":"Katharopoulos Angelos","year":"2020","unstructured":"Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Fran\u00e7ois Fleuret. 2020. Transformers are RNNs: Fast autoregressive transformers with linear attention. arXiv preprint arXiv:2006.16236 (2020).","journal-title":"arXiv preprint arXiv:2006.16236"},{"key":"e_1_3_2_38_2","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Kitaev Nikita","year":"2020","unstructured":"Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. In Proceedings of the International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=rkgNKkHtvB."},{"key":"e_1_3_2_39_2","unstructured":"Alex Krizhevsky Geoffrey Hinton et\u00a0al. 2009. Learning multiple layers of features from tiny images. Master thesis."},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00276"},{"key":"e_1_3_2_41_2","article-title":"Block pruning for faster transformers","author":"Lagunas Fran\u00e7ois","year":"2021","unstructured":"Fran\u00e7ois Lagunas, Ella Charlaix, Victor Sanh, and Alexander M. Rush. 2021. Block pruning for faster transformers. In Proceedings of EMNLP 2021 (2021).","journal-title":"Proceedings of EMNLP 2021"},{"key":"e_1_3_2_42_2","article-title":"Large memory layers with product keys","author":"Lample Guillaume","year":"2019","unstructured":"Guillaume Lample, Alexandre Sablayrolles, Marc\u2019Aurelio Ranzato, Ludovic Denoyer, and Herv\u00e9 J\u00e9gou. 2019. Large memory layers with product keys. arXiv preprint arXiv:1907.05242 (2019).","journal-title":"arXiv preprint arXiv:1907.05242"},{"key":"e_1_3_2_43_2","article-title":"ALBERT: A lite BERTt for self-supervised learning of language representations","author":"Lan Zhenzhong","year":"2019","unstructured":"Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. ALBERT: A lite BERTt for self-supervised learning of language representations. In Proceedings of ICLR (2019).","journal-title":"Proceedings of ICLR"},{"key":"e_1_3_2_44_2","first-page":"3744","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Lee Juho","year":"2019","unstructured":"Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. 2019. Set transformer: A framework for attention-based permutation-invariant neural networks. In Proceedings of the International Conference on Machine Learning. 3744\u20133753."},{"key":"e_1_3_2_45_2","article-title":"FNet: Mixing tokens with fourier transforms","author":"Lee-Thorp James","year":"2021","unstructured":"James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, and Santiago Ontanon. 2021. FNet: Mixing tokens with fourier transforms. arXiv preprint arXiv:2105.03824 (2021).","journal-title":"arXiv preprint arXiv:2105.03824"},{"key":"e_1_3_2_46_2","article-title":"GSHARD: Scaling giant models with conditional computation and automatic sharding","author":"Lepikhin Dmitry","year":"2020","unstructured":"Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2020. GSHARD: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668 (2020).","journal-title":"arXiv preprint arXiv:2006.16668"},{"key":"e_1_3_2_47_2","article-title":"Base layers: Simplifying training of large, sparse models","author":"Lewis Mike","year":"2021","unstructured":"Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. 2021. Base layers: Simplifying training of large, sparse models. arXiv preprint arXiv:2103.16716 (2021).","journal-title":"arXiv preprint arXiv:2103.16716"},{"key":"e_1_3_2_48_2","article-title":"Pay attention to MLPs","author":"Liu Hanxiao","year":"2021","unstructured":"Hanxiao Liu, Zihang Dai, David R. So, and Quoc V. Le. 2021. Pay attention to MLPs. In Proceedings of NeurIPS (2021).","journal-title":"Proceedings of NeurIPS"},{"key":"e_1_3_2_49_2","article-title":"Generating Wikipedia by summarizing long sequences","author":"Liu Peter J.","year":"2018","unstructured":"Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating Wikipedia by summarizing long sequences. In Proceedings of ICLR (2018).","journal-title":"Proceedings of ICLR"},{"key":"e_1_3_2_50_2","article-title":"Swin transformer: Hierarchical vision transformer using shifted windows","author":"Liu Ze","year":"2021","unstructured":"Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103.14030 (2021).","journal-title":"arXiv preprint arXiv:2103.14030"},{"key":"e_1_3_2_51_2","volume-title":"Proceedings of NeurIPS 2021","author":"Ma Xuezhe","year":"2021","unstructured":"Xuezhe Ma, Xiang Kong, Sinong Wang, Chunting Zhou, Jonathan May, Hao Ma, and Luke Zettlemoyer. 2021. LUNA: Linear unified nested attention. In Proceedings of NeurIPS 2021."},{"key":"e_1_3_2_52_2","article-title":"Pointer sentinel mixture models","author":"Merity Stephen","year":"2017","unstructured":"Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture models. In Proceedings of ICLR (2017).","journal-title":"Proceedings of ICLR"},{"key":"e_1_3_2_53_2","article-title":"Are sixteen heads really better than one? In","author":"Michel Paul","year":"2019","unstructured":"Paul Michel, Omer Levy, and Graham Neubig. 2019. Are sixteen heads really better than one? InProceedings of NeurIPS (2019).","journal-title":"Proceedings of NeurIPS"},{"key":"e_1_3_2_54_2","article-title":"FAIRSEQ: A fast, extensible toolkit for sequence modeling","author":"Ott Myle","year":"2019","unstructured":"Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. FAIRSEQ: A fast, extensible toolkit for sequence modeling. arXiv preprint arXiv:1904.01038 (2019).","journal-title":"arXiv preprint arXiv:1904.01038"},{"key":"e_1_3_2_55_2","first-page":"68","volume-title":"Advances in Neural Information Processing Systems","author":"Parmar Niki","year":"2019","unstructured":"Niki Parmar, Prajit Ramachandran, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens. 2019. Stand-alone self-attention in vision models. In Advances in Neural Information Processing Systems. 68\u201380."},{"key":"e_1_3_2_56_2","article-title":"Image transformer","author":"Parmar Niki","year":"2018","unstructured":"Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, \u0141ukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. 2018. Image transformer. In Proceedings of ICML 2018 (2018).","journal-title":"Proceedings of ICML 2018"},{"key":"e_1_3_2_57_2","article-title":"Random feature attention","author":"Peng Hao","year":"2021","unstructured":"Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A. Smith, and Lingpeng Kong. 2021. Random feature attention. In Proceedings of ICLR (2021).","journal-title":"Proceedings of ICLR"},{"key":"e_1_3_2_58_2","article-title":"MAD-X: An adapter-based framework for multi-task cross-lingual transfer","author":"Pfeiffer Jonas","year":"2020","unstructured":"Jonas Pfeiffer, Ivan Vuli\u0107, Iryna Gurevych, and Sebastian Ruder. 2020. MAD-X: An adapter-based framework for multi-task cross-lingual transfer. In Proceedings of EMNLP (2020).","journal-title":"Proceedings of EMNLP"},{"key":"e_1_3_2_59_2","article-title":"Blockwise self-attention for long document understanding","author":"Qiu Jiezhong","year":"2019","unstructured":"Jiezhong Qiu, Hao Ma, Omer Levy, Scott Wen-tau Yih, Sinong Wang, and Jie Tang. 2019. Blockwise self-attention for long document understanding. arXiv preprint arXiv:1911.02972 (2019).","journal-title":"arXiv preprint arXiv:1911.02972"},{"key":"e_1_3_2_60_2","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Rae Jack W.","year":"2020","unstructured":"Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. 2020. Compressive transformers for long-range sequence modelling. In Proceedings of the International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=SylKikSYDH."},{"key":"e_1_3_2_61_2","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","author":"Raffel Colin","year":"2019","unstructured":"Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 2020 (2019).","journal-title":"Journal of Machine Learning Research, 2020"},{"key":"e_1_3_2_62_2","article-title":"Hash layers for large sparse models","author":"Roller Stephen","year":"2021","unstructured":"Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston. 2021. Hash layers for large sparse models. arXiv preprint arXiv:2106.04426 (2021).","journal-title":"arXiv preprint arXiv:2106.04426"},{"key":"e_1_3_2_63_2","article-title":"Efficient content-based sparse attention with routing transformers","author":"Roy Aurko","year":"2020","unstructured":"Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. 2020. Efficient content-based sparse attention with routing transformers. In Proceedings of TACL (2020).","journal-title":"Proceedings of TACL"},{"key":"e_1_3_2_64_2","volume-title":"Advances in Neural Information Processing Systems (NeurIPS)","author":"Ryoo Michael S.","year":"2021","unstructured":"Michael S. Ryoo, A. J. Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova. 2021. TokenLearner: Adaptive space-time tokenization for videos. Advances in Neural Information Processing Systems (NeurIPS)."},{"key":"e_1_3_2_65_2","article-title":"DistilBERT, A distilled version of BERT: Smaller, faster, cheaper and lighter","author":"Sanh Victor","year":"2019","unstructured":"Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, A distilled version of BERT: Smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 (2019).","journal-title":"arXiv preprint arXiv:1910.01108"},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.406"},{"key":"e_1_3_2_67_2","doi-asserted-by":"crossref","unstructured":"Sheng Shen Zhen Dong Jiayu Ye Linjian Ma Zhewei Yao Amir Gholami Michael W. Mahoney and Kurt Keutzer. 2020. Q-BERT: Hessian based ultra low precision quantization of BERT. In Proceedings of AAAI.","DOI":"10.1609\/aaai.v34i05.6409"},{"key":"e_1_3_2_68_2","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177703591"},{"key":"e_1_3_2_69_2","article-title":"The evolved transformer","author":"So David R.","year":"2019","unstructured":"David R. So, Chen Liang, and Quoc V. Le. 2019. The evolved transformer. Proceedings of ICML (2019).","journal-title":"Proceedings of ICML"},{"key":"e_1_3_2_70_2","article-title":"BERT and pals: Projected attention layers for efficient adaptation in multi-task learning","author":"Stickland Asa Cooper","year":"2019","unstructured":"Asa Cooper Stickland and Iain Murray. 2019. BERT and pals: Projected attention layers for efficient adaptation in multi-task learning. Proceedings of ICML (2019).","journal-title":"Proceedings of ICML"},{"key":"e_1_3_2_71_2","article-title":"Adaptive attention span in transformers","author":"Sukhbaatar Sainbayar","year":"2019","unstructured":"Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. 2019. Adaptive attention span in transformers. arXiv preprint arXiv:1905.07799 (2019).","journal-title":"arXiv preprint arXiv:1905.07799"},{"key":"e_1_3_2_72_2","article-title":"Augmenting self-attention with persistent memory","author":"Sukhbaatar Sainbayar","year":"2019","unstructured":"Sainbayar Sukhbaatar, Edouard Grave, Guillaume Lample, Herve Jegou, and Armand Joulin. 2019. Augmenting self-attention with persistent memory. arXiv preprint arXiv:1907.01470 (2019).","journal-title":"arXiv preprint arXiv:1907.01470"},{"key":"e_1_3_2_73_2","article-title":"Distilling task-specific knowledge from bert into simple neural networks","author":"Tang Raphael","year":"2019","unstructured":"Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. 2019. Distilling task-specific knowledge from bert into simple neural networks. arXiv preprint arXiv:1903.12136 (2019).","journal-title":"arXiv preprint arXiv:1903.12136"},{"key":"e_1_3_2_74_2","article-title":"Synthesizer: Rethinking self-attention in transformer models","author":"Tay Yi","year":"2020","unstructured":"Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. 2020. Synthesizer: Rethinking self-attention in transformer models. In Proceedings of ICML, 2021 (2020).","journal-title":"Proceedings of ICML, 2021"},{"key":"e_1_3_2_75_2","article-title":"Sparse sinkhorn attention","author":"Tay Yi","year":"2020","unstructured":"Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan. 2020. Sparse sinkhorn attention. In Proceedings of ICML (2020).","journal-title":"Proceedings of ICML"},{"key":"e_1_3_2_76_2","article-title":"Long range arena: A benchmark for efficient transformers","author":"Tay Yi","year":"2021","unstructured":"Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. 2021. Long range arena: A benchmark for efficient transformers. Proceedings of ICLR (2021).","journal-title":"Proceedings of ICLR"},{"key":"e_1_3_2_77_2","article-title":"Are pre-trained convolutions better than pre-trained transformers?","author":"Tay Yi","year":"2021","unstructured":"Yi Tay, Mostafa Dehghani, Jai Gupta, Dara Bahri, Vamsi Aribandi, Zhen Qin, and Donald Metzler. 2021. Are pre-trained convolutions better than pre-trained transformers?arXiv preprint arXiv:2105.03322 (2021).","journal-title":"arXiv preprint arXiv:2105.03322"},{"key":"e_1_3_2_78_2","article-title":"Transformer memory as a differentiable search index","author":"Tay Yi","year":"2022","unstructured":"Yi Tay, Vinh Q. Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, et\u00a0al. 2022. Transformer memory as a differentiable search index. arXiv preprint arXiv:2202.06991 (2022).","journal-title":"arXiv preprint arXiv:2202.06991"},{"key":"e_1_3_2_79_2","article-title":"Charformer: Fast character transformers via gradient-based subword tokenization","author":"Tay Yi","year":"2021","unstructured":"Yi Tay, Vinh Q. Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler. 2021. Charformer: Fast character transformers via gradient-based subword tokenization. arXiv preprint arXiv:2106.12672 (2021).","journal-title":"arXiv preprint arXiv:2106.12672"},{"key":"e_1_3_2_80_2","article-title":"Lightweight and efficient neural natural language processing with quaternion networks","author":"Tay Yi","year":"2019","unstructured":"Yi Tay, Aston Zhang, Luu Anh Tuan, Jinfeng Rao, Shuai Zhang, Shuohang Wang, Jie Fu, and Siu Cheung Hui. 2019. Lightweight and efficient neural natural language processing with quaternion networks. In Proceedings of ACL (2019).","journal-title":"Proceedings of ACL"},{"key":"e_1_3_2_81_2","article-title":"HyperGrid: Efficient multi-task transformers with grid-wise decomposable hyper projections","author":"Tay Yi","year":"2020","unstructured":"Yi Tay, Zhe Zhao, Dara Bahri, Donald Metzler, and Da-Cheng Juan. 2020. HyperGrid: Efficient multi-task transformers with grid-wise decomposable hyper projections. In Proceedings of ICLR (2020).","journal-title":"Proceedings of ICLR"},{"key":"e_1_3_2_82_2","volume-title":"Thirty-Fifth Conference on Neural Information Processing Systems","author":"Tolstikhin Ilya","year":"2021","unstructured":"Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Peter Steiner, Daniel Keysers, Jakob Uszkoreit, et\u00a0al. 2021. Mlp-mixer: An all-mlp architecture for vision. In Thirty-Fifth Conference on Neural Information Processing Systems."},{"key":"e_1_3_2_83_2","first-page":"5998","volume-title":"Advances in Neural Information Processing Systems","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems. 5998\u20136008."},{"key":"e_1_3_2_84_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1580"},{"key":"e_1_3_2_85_2","article-title":"Fast transformers with clustered attention","author":"Vyas Apoorv","year":"2020","unstructured":"Apoorv Vyas, Angelos Katharopoulos, and Fran\u00e7ois Fleuret. 2020. Fast transformers with clustered attention. In Proceedings of NeurIPS (2020).","journal-title":"Proceedings of NeurIPS"},{"key":"e_1_3_2_86_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W18-5446"},{"key":"e_1_3_2_87_2","article-title":"HAT: Hardware-aware transformers for efficient natural language processing","author":"Wang Hanrui","year":"2020","unstructured":"Hanrui Wang, Zhanghao Wu, Zhijian Liu, Han Cai, Ligeng Zhu, Chuang Gan, and Song Han. 2020. HAT: Hardware-aware transformers for efficient natural language processing. arXiv preprint arXiv:2005.14187 (2020).","journal-title":"arXiv preprint arXiv:2005.14187"},{"key":"e_1_3_2_88_2","article-title":"Linformer: Self-attention with linear complexity","author":"Wang Sinong","year":"2020","unstructured":"Sinong Wang, Belinda Li, Madian Khabsa, Han Fang, and Hao Ma. 2020. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768 (2020).","journal-title":"arXiv preprint arXiv:2006.04768"},{"key":"e_1_3_2_89_2","article-title":"Cluster-former: Clustering-based sparse transformer for long-range dependency encoding","author":"Wang Shuohang","year":"2020","unstructured":"Shuohang Wang, Luowei Zhou, Zhe Gan, Yen-Chun Chen, Yuwei Fang, Siqi Sun, Yu Cheng, and Jingjing Liu. 2020. Cluster-former: Clustering-based sparse transformer for long-range dependency encoding. In Proceedings of ACL-IJCNLP (Findings) (2020).","journal-title":"Proceedings of ACL-IJCNLP (Findings)"},{"key":"e_1_3_2_90_2","article-title":"Scaling autoregressive video models","author":"Weissenborn Dirk","year":"2019","unstructured":"Dirk Weissenborn, Oscar T\u00e4ckstr\u00f6m, and Jakob Uszkoreit. 2019. Scaling autoregressive video models. In Proceedings of ICLR (2019).","journal-title":"Proceedings of ICLR"},{"key":"e_1_3_2_91_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP40776.2020.9053878"},{"key":"e_1_3_2_92_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v35i16.17664"},{"key":"e_1_3_2_93_2","article-title":"\\( O (n) \\)  Connections are expressive enough: Universal approximability of sparse transformers","author":"Yun Chulhee","year":"2020","unstructured":"Chulhee Yun, Yin-Wen Chang, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank J. Reddi, and Sanjiv Kumar. 2020. \\( O (n) \\) Connections are expressive enough: Universal approximability of sparse transformers. In Proceedings of NeurIPS (2020).","journal-title":"Proceedings of NeurIPS"},{"key":"e_1_3_2_94_2","article-title":"Big bird: Transformers for longer sequences","author":"Zaheer Manzil","year":"2020","unstructured":"Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et\u00a0al. 2020. Big bird: Transformers for longer sequences. In Proceedings of NeurIPS (2020).","journal-title":"Proceedings of NeurIPS"},{"key":"e_1_3_2_95_2","first-page":"3391","volume-title":"Advances in Neural Information Processing Systems","author":"Zaheer Manzil","year":"2017","unstructured":"Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R. Salakhutdinov, and Alexander J. Smola. 2017. Deep sets. In Advances in Neural Information Processing Systems. 3391\u20133401."},{"key":"e_1_3_2_96_2","article-title":"Poolingformer: Long document modeling with pooling attention","author":"Zhang Hang","year":"2021","unstructured":"Hang Zhang, Yeyun Gong, Yelong Shen, Weisheng Li, Jiancheng Lv, Nan Duan, and Weizhu Chen. 2021. Poolingformer: Long document modeling with pooling attention. In Proceedings of ICML (2021).","journal-title":"Proceedings of ICML"},{"key":"e_1_3_2_97_2","article-title":"Long-short transformer: Efficient transformers for language and vision","volume":"34","author":"Zhu Chen","year":"2021","unstructured":"Chen Zhu, Wei Ping, Chaowei Xiao, Mohammad Shoeybi, Tom Goldstein, Anima Anandkumar, and Bryan Catanzaro. 2021. Long-short transformer: Efficient transformers for language and vision. Advances in Neural Information Processing Systems 34 (2021).","journal-title":"Advances in Neural Information Processing Systems"}],"container-title":["ACM Computing Surveys"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3530811","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3530811","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T14:09:25Z","timestamp":1750169365000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3530811"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,12,7]]},"references-count":96,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2023,6,30]]}},"alternative-id":["10.1145\/3530811"],"URL":"https:\/\/doi.org\/10.1145\/3530811","relation":{},"ISSN":["0360-0300","1557-7341"],"issn-type":[{"value":"0360-0300","type":"print"},{"value":"1557-7341","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,12,7]]},"assertion":[{"value":"2021-04-06","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-04-06","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-12-07","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}