{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,29]],"date-time":"2026-01-29T03:44:15Z","timestamp":1769658255153,"version":"3.49.0"},"publisher-location":"New York, NY, USA","reference-count":83,"publisher":"ACM","funder":[{"name":"National Natural Science Foundation of China","award":["62402525"],"award-info":[{"award-number":["62402525"]}]},{"name":"National Natural Science Foundation of China","award":["62322201"],"award-info":[{"award-number":["62322201"]}]},{"name":"National Natural Science Foundation of China","award":["U23B2020"],"award-info":[{"award-number":["U23B2020"]}]},{"name":"National Natural Science Foundation of China","award":["62402526"],"award-info":[{"award-number":["62402526"]}]},{"name":"Beijing Natural Science Foundation","award":["4244086"],"award-info":[{"award-number":["4244086"]}]},{"name":"CCF-Baidu Open Fund","award":["None"],"award-info":[{"award-number":["None"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2026,1,28]]},"DOI":"10.1145\/3774934.3786434","type":"proceedings-article","created":{"date-parts":[[2026,1,28]],"date-time":"2026-01-28T15:25:57Z","timestamp":1769613957000},"page":"620-634","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Accelerating Sparse Transformer Inference on GPU"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0009-0000-5551-7603","authenticated-orcid":false,"given":"Wenhao","family":"Dai","sequence":"first","affiliation":[{"name":"China University of Petroleum-Beijing, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-5627-6918","authenticated-orcid":false,"given":"Haodong","family":"Deng","sequence":"additional","affiliation":[{"name":"China University of Petroleum-Beijing, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-2202-2327","authenticated-orcid":false,"given":"Mengfei","family":"Rong","sequence":"additional","affiliation":[{"name":"China University of Petroleum-Beijing, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-9931-2486","authenticated-orcid":false,"given":"Xinyu","family":"Yang","sequence":"additional","affiliation":[{"name":"Beihang University, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-6128-3619","authenticated-orcid":false,"given":"Hongyu","family":"Liu","sequence":"additional","affiliation":[{"name":"Baidu, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8769-293X","authenticated-orcid":false,"given":"Fangxin","family":"Liu","sequence":"additional","affiliation":[{"name":"Shanghai Jiao Tong University, Shanghai, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1101-7927","authenticated-orcid":false,"given":"Hailong","family":"Yang","sequence":"additional","affiliation":[{"name":"Beihang University, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7491-5201","authenticated-orcid":false,"given":"Qianwen","family":"Cao","sequence":"additional","affiliation":[{"name":"China University of Petroleum-Beijing, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2927-362X","authenticated-orcid":false,"given":"Qingxiao","family":"Sun","sequence":"additional","affiliation":[{"name":"Beihang University, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2026,1,28]]},"reference":[{"key":"e_1_3_2_1_1_1","volume-title":"Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.","author":"Achiam Josh","year":"2023","unstructured":"Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. GPT-4 technical report. arXiv preprint arXiv:2303.08774."},{"key":"e_1_3_2_1_2_1","volume-title":"Conference on Machine Learning and Systems (MLSys \u201924)","author":"Adnan Muhammad","year":"2024","unstructured":"Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant Nair, Ilya Soloveychik, and Purushotham Kamath. 2024. Keyformer: KV cache reduction through key tokens selection for efficient generative inference. In Conference on Machine Learning and Systems (MLSys \u201924)."},{"key":"e_1_3_2_1_3_1","volume-title":"International Conference on High Performance Computing, Networking, Storage and Analysis (SC \u201922)","author":"Aminabadi Reza Yazdani","year":"2022","unstructured":"Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, et al. 2022. DeepSpeed Inference: Enabling efficient inference of Transformer models at unprecedented scale. In International Conference on High Performance Computing, Networking, Storage and Analysis (SC \u201922)."},{"key":"e_1_3_2_1_4_1","volume-title":"International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS \u201924)","author":"Ansel Jason","year":"2024","unstructured":"Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, et al. 2024. PyTorch 2: Faster machine learning through dynamic python bytecode transformation and graph compilation. In International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS \u201924)."},{"key":"e_1_3_2_1_5_1","volume-title":"Design Automation Conference (DAC \u201924)","author":"Bai Zhenyu","year":"2024","unstructured":"Zhenyu Bai, Pranav Dangi, Huize Li, and Tulika Mitra. 2024. SWAT: Scalable and efficient window attention-based Transformers acceleration on FPGAs. In Design Automation Conference (DAC \u201924)."},{"key":"e_1_3_2_1_6_1","volume-title":"Longformer: The long-document Transformer. arXiv preprint arXiv:2004.05150.","author":"Beltagy Iz","year":"2020","unstructured":"Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document Transformer. arXiv preprint arXiv:2004.05150."},{"key":"e_1_3_2_1_7_1","unstructured":"Lefaudeux Benjamin Massa Francisco Liskovich Diana Xiong Wenhan Caggiano Vittorio Naren Sean Xu Min Hu Jieru Tintore Marta Zhang Susan Labatut Patrick Haziza Daniel Wehrstedt Luca Reizenstein Jeremy and Sizov Grigory. 2022. xFormers: A modular and hackable Transformer modelling library. https:\/\/github.com\/facebookresearch\/xformers"},{"key":"e_1_3_2_1_8_1","article-title":"A survey on mixture of experts in large language models","author":"Cai Weilin","year":"2025","unstructured":"Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. 2025. A survey on mixture of experts in large language models. IEEE Transactions on Knowledge and Data Engineering.","journal-title":"IEEE Transactions on Knowledge and Data Engineering."},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3641289"},{"key":"e_1_3_2_1_10_1","volume-title":"USENIX Symposium on Operating Systems Design and Implementations (OSDI \u201918)","author":"Chen Tianqi","year":"2018","unstructured":"Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, et al. 2018. TVM: An automated end-to-end optimizing compiler for deep learning. In USENIX Symposium on Operating Systems Design and Implementations (OSDI \u201918)."},{"key":"e_1_3_2_1_11_1","volume-title":"International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS \u201924)","author":"Chen Zhaodong","year":"2024","unstructured":"Zhaodong Chen, Andrew Kerr, Richard Cai, Jack Kosaian, Haicheng Wu, Yufei Ding, and Yuan Xie. 2024. EVT: Accelerating deep learning training with Epilogue Visitor Tree. In International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS \u201924)."},{"key":"e_1_3_2_1_12_1","volume-title":"PipeThreader: Software-Defined Pipelining for Efficient DNN Execution. In 19th USENIX Symposium on Operating Systems Design and Implementation (OSDI \u201925)","author":"Cheng Yu","year":"2025","unstructured":"Yu Cheng, Lei Wang, Yining Shi, Yuqing Xia, Lingxiao Ma, Jilong Xue, Yang Wang, Zhiwen Mo, Feiyang Chen, Fan Yang, et al. 2025. PipeThreader: Software-Defined Pipelining for Efficient DNN Execution. In 19th USENIX Symposium on Operating Systems Design and Implementation (OSDI \u201925)."},{"key":"e_1_3_2_1_13_1","unstructured":"Rewon Child Scott Gray Alec Radford and Ilya Sutskever. 2019. Generating long sequences with sparse Transformers. arXiv preprint arXiv:1904.10509."},{"key":"e_1_3_2_1_14_1","volume-title":"International Parallel & Distributed Processing Symposiumm (IPDPS \u201923)","author":"Cho Younghyun","year":"2023","unstructured":"Younghyun Cho, James W Demmel, Jacob King, Xiaoye S Li, Yang Liu, and Hengrui Luo. 2023. Harnessing the crowd for autotuning high-performance computing applications. In International Parallel & Distributed Processing Symposiumm (IPDPS \u201923)."},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","unstructured":"Wenhao Dai. 2025. PPoPP26_AE_STOF_CODE. https:\/\/doi.org\/10.5281\/zenodo.17705801 10.5281\/zenodo.17705801","DOI":"10.5281\/zenodo.17705801"},{"key":"e_1_3_2_1_16_1","volume-title":"International Conference on Learning Representations (ICLR \u201923)","author":"Dao Tri","year":"2024","unstructured":"Tri Dao. 2024. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. In International Conference on Learning Representations (ICLR \u201923)."},{"key":"e_1_3_2_1_17_1","unstructured":"Tri Dao Daniel Y. Fu Stefano Ermon Atri Rudra and Christopher R\u00e9. 2022. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. In Advances in Neural Information Processing Systems (NeurIPS \u201922)."},{"key":"e_1_3_2_1_18_1","volume-title":"Annual Conference of the North American chapter of the association for computational linguistics: human language technologies.","author":"Devlin Jacob","year":"2019","unstructured":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional Transformers for language understanding. In Annual Conference of the North American chapter of the association for computational linguistics: human language technologies."},{"key":"e_1_3_2_1_19_1","volume-title":"Flex Attention: A programming model for generating optimized attention kernels. arXiv preprint arXiv:2412.05496.","author":"Dong Juechu","year":"2024","unstructured":"Juechu Dong, Boyuan Feng, Driss Guessous, Yanbo Liang, and Horace He. 2024. Flex Attention: A programming model for generating optimized attention kernels. arXiv preprint arXiv:2412.05496."},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/JPROC.2018.2868961"},{"key":"e_1_3_2_1_21_1","unstructured":"Alexey Dosovitskiy Lucas Beyer Alexander Kolesnikov Dirk Weissenborn Xiaohua Zhai Thomas Unterthiner Mostafa Dehghani Matthias Minderer Georg Heigold Sylvain Gelly et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929."},{"key":"e_1_3_2_1_22_1","volume-title":"International Conference on Learning Representations (ICLR \u201921)","author":"Dosovitskiy Alexey","year":"2021","unstructured":"Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR \u201921)."},{"key":"e_1_3_2_1_23_1","volume-title":"IEEE\/ACM International Symposium on Microarchitecture (MICRO \u201922)","author":"Fan Hongxiang","year":"2022","unstructured":"Hongxiang Fan, Thomas Chau, Stylianos I Venieris, Royson Lee, Alexandros Kouris, Wayne Luk, Nicholas D Lane, and Mohamed S Abdelfattah. 2022. Adaptable butterfly accelerator for attention-based NNs via hardware and algorithm co-design. In IEEE\/ACM International Symposium on Microarchitecture (MICRO \u201922)."},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3689031.3717481"},{"key":"e_1_3_2_1_25_1","volume-title":"ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP \u201921)","author":"Fang Jiarui","year":"2021","unstructured":"Jiarui Fang, Yang Yu, Chengduo Zhao, and Jie Zhou. 2021. TurboTransformers: An efficient GPU serving system for Transformer models. In ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP \u201921)."},{"key":"e_1_3_2_1_26_1","volume-title":"Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS \u201925)","author":"Gu Yufeng","year":"2025","unstructured":"Yufeng Gu, Alireza Khadem, Sumanth Umesh, Ning Liang, Xavier Servot, Onur Mutlu, Ravi Iyer, and Reetuparna Das. 2025. PIM is all you need: A CXL-enabled GPU-free system for large language model inference. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS \u201925)."},{"key":"e_1_3_2_1_27_1","unstructured":"Daya Guo Dejian Yang Haowei Zhang Junxiao Song Ruoyu Zhang Runxin Xu Qihao Zhu Shirong Ma Peiyi Wang Xiao Bi et al. 2025. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948."},{"key":"e_1_3_2_1_28_1","volume-title":"Proceedings of the ACM on Programming Languages (OOPSLA \u201925)","author":"Gupta Ahan","year":"2025","unstructured":"Ahan Gupta, Yueming Yuan, Devansh Jain, Yuhao Ge, David Aponte, Yanqi Zhou, and Charith Mendis. 2025. SPLAT: A framework for optimised GPU code-generation for SParse reguLar ATtention. In Proceedings of the ACM on Programming Languages (OOPSLA \u201925)."},{"key":"e_1_3_2_1_29_1","volume-title":"Seonghak Kim, Young H Oh, Yeonhong Park, Yoonho Song, Jung-Hun Park, Sanghee Lee, Kyoung Park, Jae W Lee, et al.","author":"Ham Tae Jun","year":"2020","unstructured":"Tae Jun Ham, Sung Jun Jung, Seonghak Kim, Young H Oh, Yeonhong Park, Yoonho Song, Jung-Hun Park, Sanghee Lee, Kyoung Park, Jae W Lee, et al. 2020. A^3: Accelerating attention mechanisms in neural networks with approximation. In High Performance Computer Architecture (HPCA \u201920)."},{"key":"e_1_3_2_1_30_1","volume-title":"International Symposium on Computer Architecture (ISCA \u201921)","author":"Ham Tae Jun","year":"2021","unstructured":"Tae Jun Ham, Yejin Lee, Seong Hoon Seo, Soosung Kim, Hyunji Choi, Sung Jun Jung, and Jae W Lee. 2021. ELSA: Hardware-software co-design for efficient, lightweight self-attention mechanism in neural networks. In International Symposium on Computer Architecture (ISCA \u201921)."},{"key":"e_1_3_2_1_31_1","volume-title":"Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS \u201925)","author":"He Yintao","year":"2025","unstructured":"Yintao He, Haiyu Mao, Christina Giannoula, Mohammad Sadrosadati, Juan G\u00f3mez-Luna, Huawei Li, Xiaowei Li, Ying Wang, and Onur Mutlu. 2025. Papi: Exploiting dynamic parallelism in large language model decoding with a processing-in-memory-enabled computing system. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS \u201925)."},{"key":"e_1_3_2_1_32_1","volume-title":"Conference on Neural Information Processing Systems (NeurIPS \u201924)","author":"Jiang Huiqiang","year":"2024","unstructured":"Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H Abdi, Dongsheng Li, Chin-Yew Lin, et al. 2024. Minference 1.0: Accelerating pre-filling for long-context llms via dynamic sparse attention. In Conference on Neural Information Processing Systems (NeurIPS \u201924)."},{"key":"e_1_3_2_1_33_1","volume-title":"International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS \u201923)","author":"Kao Sheng-Chun","year":"2023","unstructured":"Sheng-Chun Kao, Suvinay Subramanian, Gaurav Agrawal, Amir Yazdanbakhsh, and Tushar Krishna. 2023. FLAT: An optimized dataflow for mitigating attention bottlenecks. In International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS \u201923)."},{"key":"e_1_3_2_1_34_1","volume-title":"International Conference on Supercomputing (ICS \u201924)","author":"Li Chendi","year":"2024","unstructured":"Chendi Li, Yufan Xu, Sina Mahdipour Saravani, and Ponnuswamy Sadayappan. 2024. Accelerated auto-tuning of GPU kernels for tensor computations. In International Conference on Supercomputing (ICS \u201924)."},{"key":"e_1_3_2_1_35_1","volume-title":"International Conference on Parallel Processing (ICPP \u201923)","author":"Li Mingzhen","year":"2023","unstructured":"Mingzhen Li, Hailong Yang, Shanjun Zhang, Fengwei Yu, Ruihao Gong, Yi Liu, Zhongzhi Luan, and Depei Qian. 2023. Exploiting subgraph similarities for efficient auto-tuning of tensor programs. In International Conference on Parallel Processing (ICPP \u201923)."},{"key":"e_1_3_2_1_36_1","volume-title":"A survey of Transformers. AI open, 3","author":"Lin Tianyang","year":"2022","unstructured":"Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu. 2022. A survey of Transformers. AI open, 3 (2022), 111\u2013132."},{"key":"e_1_3_2_1_37_1","volume-title":"International Conference on Machine Learning (ICML \u201923)","author":"Liu Zichang","year":"2023","unstructured":"Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, et al. 2023. Deja Vu: Contextual sparsity for efficient LLMs at inference time. In International Conference on Machine Learning (ICML \u201923)."},{"key":"e_1_3_2_1_38_1","volume-title":"International System-on-Chip Conference (SOCC \u201920)","author":"Lu Siyuan","year":"2020","unstructured":"Siyuan Lu, Meiqi Wang, Shuang Liang, Jun Lin, and Zhongfeng Wang. 2020. Hardware accelerator for multi-head attention and position-wise feed-forward in the Transformer. In International System-on-Chip Conference (SOCC \u201920)."},{"key":"e_1_3_2_1_39_1","volume-title":"USENIX Symposium on Operating Systems Design and Implementation (OSDI \u201920)","author":"Ma Lingxiao","year":"2020","unstructured":"Lingxiao Ma, Zhiqiang Xie, Zhi Yang, Jilong Xue, Youshan Miao, Wei Cui, Wenxiang Hu, Fan Yang, Lintao Zhang, and Lidong Zhou. 2020. Rammer: Enabling holistic deep learning compiler optimizations with rTasks. In USENIX Symposium on Operating Systems Design and Implementation (OSDI \u201920)."},{"key":"e_1_3_2_1_40_1","first-page":"105","article-title":"PaddlePaddle: An open-source deep learning platform from industrial practice","volume":"1","author":"Ma Yanjun","year":"2019","unstructured":"Yanjun Ma, Dianhai Yu, Tian Wu, and Haifeng Wang. 2019. PaddlePaddle: An open-source deep learning platform from industrial practice. Frontiers of Data and Computing, 1, 1 (2019), 105\u2013115.","journal-title":"Frontiers of Data and Computing"},{"key":"e_1_3_2_1_41_1","volume-title":"International Conference on Programming Language Design and Implementation (PLDI \u201921)","author":"Niu Wei","year":"2021","unstructured":"Wei Niu, Jiexiong Guan, Yanzhi Wang, Gagan Agrawal, and Bin Ren. 2021. DNNFusion: Accelerating deep neural networks execution with advanced operator fusion. In International Conference on Programming Language Design and Implementation (PLDI \u201921)."},{"key":"e_1_3_2_1_42_1","volume-title":"ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP \u201922)","author":"Niu Yuyao","year":"2022","unstructured":"Yuyao Niu, Zhengyang Lu, Haonan Ji, Shuhui Song, Zhou Jin, and Weifeng Liu. 2022. TileSpGEMM: A tiled algorithm for parallel sparse general matrix-matrix multiplication on GPUs. In ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP \u201922)."},{"key":"e_1_3_2_1_43_1","unstructured":"NVIDIA. 2022. https:\/\/github.com\/NVIDIA\/FasterTransformer."},{"key":"e_1_3_2_1_44_1","unstructured":"NVIDIA. 2022. https:\/\/github.com\/NVIDIA\/cutlass."},{"key":"e_1_3_2_1_45_1","volume-title":"Conference on Neural Information Processing Systems (NeurIPS \u201922)","author":"Ouyang Long","year":"2022","unstructured":"Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. In Conference on Neural Information Processing Systems (NeurIPS \u201922)."},{"key":"e_1_3_2_1_46_1","volume-title":"International Conference on Supercomputing (ICS \u201919)","author":"Pfaffe Philip","year":"2019","unstructured":"Philip Pfaffe, Tobias Grosser, and Martin Tillmann. 2019. Efficient hierarchical online-autotuning: A case study on polyhedral accelerator mapping. In International Conference on Supercomputing (ICS \u201919). 354\u2013366."},{"key":"e_1_3_2_1_47_1","volume-title":"International Symposium on Computer Architecture (ISCA \u201923)","author":"Qin Yubin","year":"2023","unstructured":"Yubin Qin, Yang Wang, Dazheng Deng, Zhiren Zhao, Xiaolong Yang, Leibo Liu, Shaojun Wei, Yang Hu, and Shouyi Yin. 2023. FACT: FFN-attention co-optimized Transformer architecture with eager correlation prediction. In International Symposium on Computer Architecture (ISCA \u201923)."},{"key":"e_1_3_2_1_48_1","unstructured":"Alec Radford Jeffrey Wu Rewon Child David Luan Dario Amodei Ilya Sutskever et al. 2019. Language models are unsupervised multitask learners. OpenAI blog 1 8 (2019) 9."},{"key":"e_1_3_2_1_49_1","first-page":"1","article-title":"Exploring the limits of transfer learning with a unified text-to-text Transformer","volume":"21","author":"Raffel Colin","year":"2020","unstructured":"Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, ichael MMatena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text Transformer. Journal of Machine Learning Research, 21, 140 (2020), 1\u201367.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/2499370.2462176"},{"key":"e_1_3_2_1_51_1","volume-title":"International Conference on Supercomputing (ICS \u201923)","author":"Randall Thomas","year":"2023","unstructured":"Thomas Randall, Jaehoon Koo, Brice Videau, Michael Kruse, Xingfu Wu, Paul Hovland, Mary Hall, Rong Ge, and Prasanna Balaprakash. 2023. Transfer-learning-based autotuning using Gaussian copula. In International Conference on Supercomputing (ICS \u201923)."},{"key":"e_1_3_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00353"},{"key":"e_1_3_2_1_53_1","first-page":"68658","article-title":"FlashAttention-3: Fast and accurate attention with asynchrony and low-precision","volume":"37","author":"Shah Jay","year":"2024","unstructured":"Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, and Tri Dao. 2024. FlashAttention-3: Fast and accurate attention with asynchrony and low-precision. Advances in Neural Information Processing Systems, 37 (2024), 68658\u201368685.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_1_54_1","volume-title":"USENIX Symposium on Operating Systems Design and Implementations (OSDI \u201923)","author":"Shi Yining","year":"2023","unstructured":"Yining Shi, Zhi Yang, Jilong Xue, Lingxiao Ma, Yuqing Xia, Ziming Miao, Yuxiao Guo, Fan Yang, and Lidong Zhou. 2023. Welder: Scheduling deep learning memory access via tile-graph. In USENIX Symposium on Operating Systems Design and Implementations (OSDI \u201923)."},{"key":"e_1_3_2_1_55_1","volume-title":"IEEE International Conference on Cluster Computing (CLUSTER \u201921)","author":"Sun Qingxiao","year":"2021","unstructured":"Qingxiao Sun, Yi Liu, Hailong Yang, Zhonghui Jiang, Xiaoyan Liu, Ming Dun, Zhongzhi Luan, and Depei Qian. 2021. csTuner: Scalable auto-tuning framework for complex stencil computation on GPUs. In IEEE International Conference on Cluster Computing (CLUSTER \u201921)."},{"key":"e_1_3_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2023.3325630"},{"key":"e_1_3_2_1_57_1","volume-title":"Asia and South Pacific Design Automation Conference (ASP-DAC \u201922)","author":"Sun Qi","year":"2022","unstructured":"Qi Sun, Xinyun Zhang, Hao Geng, Yuxuan Zhao, Yang Bai, Haisheng Zheng, and Bei Yu. 2022. GTuner: Tuning DNN computations on GPU via graph attention network. In Asia and South Pacific Design Automation Conference (ASP-DAC \u201922)."},{"key":"e_1_3_2_1_58_1","volume-title":"Seer: Predictive runtime kernel selection for irregular problems. In Code Generation and Optimization (CGO \u201924).","author":"Swann Ryan","year":"2024","unstructured":"Ryan Swann, Muhammad Osama, Karthik Sangaiah, and Jalal Mahmud. 2024. Seer: Predictive runtime kernel selection for irregular problems. In Code Generation and Optimization (CGO \u201924)."},{"key":"e_1_3_2_1_59_1","volume-title":"ACM SIGPLAN International Workshop on Machine Learning and Programming Languages.","author":"Tillet Philippe","year":"2019","unstructured":"Philippe Tillet, Hsiang-Tsung Kung, and David Cox. 2019. Triton: An intermediate language and compiler for tiled neural network computationsf. In ACM SIGPLAN International Workshop on Machine Learning and Programming Languages."},{"key":"e_1_3_2_1_60_1","unstructured":"Hugo Touvron Thibaut Lavril Gautier Izacard Xavier Martinet Marie-Anne Lachaux Timoth\u00e9e Lacroix Baptiste Rozi\u00e8re Naman Goyal Eric Hambro Faisal Azhar et al. 2023. LLaMA: Open and Efficient Foundation Language Models. arXiv preprint arXiv:2302.13971."},{"key":"e_1_3_2_1_61_1","volume-title":"Conference on Neural Information Processing Systems (NeurIPS \u201917)","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, \u0141 ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Conference on Neural Information Processing Systems (NeurIPS \u201917)."},{"key":"e_1_3_2_1_62_1","volume-title":"FlashMask: Efficient and Rich Mask Extension of FlashAttention. In International Conference on Learning Representations (ICLR \u201924)","author":"Wang Guoxia","year":"2024","unstructured":"Guoxia Wang, Jinle Zeng, Xiyuan Xiao, Siming Wu, Jiabin Yang, Lujing Zheng, Zeyu Chen, Jiang Bian, Dianhai Yu, and Haifeng Wang. 2024. FlashMask: Efficient and Rich Mask Extension of FlashAttention. In International Conference on Learning Representations (ICLR \u201924)."},{"key":"e_1_3_2_1_63_1","volume-title":"CTA: Hardware-software co-design for compressed token attention mechanism. In High Performance Computer Architecture (HPCA \u201923).","author":"Wang Haoran","year":"2023","unstructured":"Haoran Wang, Haobo Xu, Ying Wang, and Yinhe Han. 2023. CTA: Hardware-software co-design for compressed token attention mechanism. In High Performance Computer Architecture (HPCA \u201923)."},{"key":"e_1_3_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1109\/TC.2024.3389507"},{"key":"e_1_3_2_1_65_1","volume-title":"Proceedings of the 39th ACM International Conference on Supercomputing (ICS \u201925)","author":"Wang Siqi","year":"2025","unstructured":"Siqi Wang, Hailong Yang, Pengbo Wang, Shaokang Du, Yufan Xu, Qingxiao Sun, Xiaoyan Liu, Xuezhu Wang, Xuning Liang, Zhongzhi Luan, et al. 2025. Accelerating Complex Stencil Computations with Adaptive Fusion Strategy. In Proceedings of the 39th ACM International Conference on Supercomputing (ICS \u201925). 265\u2013278."},{"key":"e_1_3_2_1_66_1","volume-title":"International Conference on High Performance Computing, Networking, Storage and Analysis (SC \u201922)","author":"Wang Xiaohui","year":"2022","unstructured":"Xiaohui Wang, Yang Wei, Ying Xiong, Guyue Huang, Xian Qian, Yufei Ding, Mingxuan Wang, and Lei Li. 2022. LightSeq2: Accelerated training for Transformer-based models on GPUs. In International Conference on High Performance Computing, Networking, Storage and Analysis (SC \u201922)."},{"key":"e_1_3_2_1_67_1","first-page":"204","article-title":"Bolt: Bridging the gap between auto-tuners and hardware-native performance","volume":"4","author":"Xing Jiarong","year":"2022","unstructured":"Jiarong Xing, Leyuan Wang, Shang Zhang, Jack Chen, Ang Chen, and Yibo Zhu. 2022. Bolt: Bridging the gap between auto-tuners and hardware-native performance. Proceedings of Machine Learning and Systems, 4 (2022), 204\u2013216.","journal-title":"Proceedings of Machine Learning and Systems"},{"key":"e_1_3_2_1_68_1","volume-title":"Enabling efficient sparse multiplications on GPUs with heuristic adaptability","author":"Xu Jiaming","year":"2024","unstructured":"Jiaming Xu, Shan Huang, Jinhao Li, Guyue Huang, Yuan Xie, Yu Wang, and Guohao Dai. 2024. Enabling efficient sparse multiplications on GPUs with heuristic adaptability. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, PP (2024), 1\u20131."},{"key":"e_1_3_2_1_69_1","volume-title":"European Conference on Computer Systems (EuroSys \u201923)","author":"Xu Zhiying","year":"2023","unstructured":"Zhiying Xu, Jiafan Xu, Hongding Peng, Wei Wang, Xiaoliang Wang, Haoran Wan, Haipeng Dai, Yixu Xu, Hao Cheng, Kun Wang, et al. 2023. ALT: Breaking the wall between data layout and loop optimizations for deep learning compilation. In European Conference on Computer Systems (EuroSys \u201923)."},{"key":"e_1_3_2_1_70_1","unstructured":"Haoran You Zhanyi Sun Huihong Shi Zhongzhi Yu Yang Zhao Yongan Zhang Chaojian Li Baopu Li and Yingyan Lin. 2023. ViTCoD: Vision Transformer acceleration via dedicated algorithm and accelerator co-design. In High Performance Computer Architecture (HPCA \u201923)."},{"key":"e_1_3_2_1_71_1","volume-title":"Conference on Neural Information Processing Systems (NeurIPS \u201920)","author":"Zaheer Manzil","year":"2020","unstructured":"Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020. Big Bird: Transformers for longer sequences. In Conference on Neural Information Processing Systems (NeurIPS \u201920)."},{"key":"e_1_3_2_1_72_1","volume-title":"International Parallel & Distributed Processing Symposium (IPDPS \u201923)","author":"Zhai Yujia","year":"2023","unstructured":"Yujia Zhai, Chengquan Jiang, Leyuan Wang, Xiaoying Jia, Shang Zhang, Zizhong Chen, Xin Liu, and Yibo Zhu. 2023. ByteTransformer: A high-performance Transformer boosted for variable-length inputs. In International Parallel & Distributed Processing Symposium (IPDPS \u201923)."},{"key":"e_1_3_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1109\/SC41406.2024.00040"},{"key":"e_1_3_2_1_74_1","volume-title":"ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI \u201921)","author":"Zhao Jie","year":"2021","unstructured":"Jie Zhao, Bojie Li, Wang Nie, Zhen Geng, Renwei Zhang, Xiong Gao, Bin Cheng, Chen Wu, Yun Cheng, Zheng Li, et al. 2021. AKG: Automatic kernel generation for neural processing units using polyhedral transformations. In ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI \u201921)."},{"key":"e_1_3_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCAD.2024.3373592"},{"key":"e_1_3_2_1_76_1","unstructured":"Wayne Xin Zhao Kun Zhou Junyi Li Tianyi Tang Xiaolei Wang Yupeng Hou Yingqian Min Beichen Zhang Junjie Zhang Zican Dong et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223 1 2 (2023)."},{"key":"e_1_3_2_1_77_1","volume-title":"USENIX Symposium on Operating Systems Design and Implementation (OSDI \u201920)","author":"Zheng Lianmin","year":"2020","unstructured":"Lianmin Zheng, Chengfan Jia, Minmin Sun, Zhao Wu, Cody Hao Yu, Ameer Haj-Ali, Yida Wang, Jun Yang, Danyang Zhuo, Koushik Sen, et al. 2020. Ansor: Generating high-performance tensor programs for deep learning. In USENIX Symposium on Operating Systems Design and Implementation (OSDI \u201920)."},{"key":"e_1_3_2_1_78_1","volume-title":"Conference on Neural Information Processing Systems (NeurIPS \u201921)","author":"Zheng Lianmin","year":"2021","unstructured":"Lianmin Zheng, Ruochen Liu, Junru Shao, Tianqi Chen, Joseph E Gonzalez, Ion Stoica, and Ameer Haj Ali. 2021. TenSet: A large-scale program performance dataset for learned tensor compilers. In Conference on Neural Information Processing Systems (NeurIPS \u201921)."},{"key":"e_1_3_2_1_79_1","first-page":"3220","article-title":"NeoFlow: A flexible framework for enabling efficient compilation for high performance DNN training","volume":"33","author":"Zheng Size","year":"2021","unstructured":"Size Zheng, Renze Chen, Yicheng Jin, Anjiang Wei, Bingyang Wu, Xiuhong Li, Shengen Yan, and Yun Liang. 2021. NeoFlow: A flexible framework for enabling efficient compilation for high performance DNN training. IEEE Transactions on Parallel and Distributed Systems, 33, 11 (2021), 3220\u20133232.","journal-title":"IEEE Transactions on Parallel and Distributed Systems"},{"key":"e_1_3_2_1_80_1","volume-title":"Proceedings of the 29th IEEE International Symposium on High Performance Computer Architecture (HPCA \u201923)","author":"Zheng Size","year":"2023","unstructured":"Size Zheng, Siyuan Chen, Peidi Song, Renze Chen, Xiuhong Li, Shengen Yan, Dahua Lin, Jingwen Leng, and Yun Liang. 2023. Chimera: An analytical optimizing framework for effective compute-intensive operators fusion. In Proceedings of the 29th IEEE International Symposium on High Performance Computer Architecture (HPCA \u201923)."},{"key":"e_1_3_2_1_81_1","volume-title":"International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS \u201920)","author":"Zheng Size","year":"2020","unstructured":"Size Zheng, Yun Liang, Shuo Wang, Renze Chen, and Kaiwen Sheng. 2020. FlexTensor: An automatic schedule exploration and optimization framework for tensor computation on heterogeneous system. In International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS \u201920)."},{"key":"e_1_3_2_1_82_1","volume-title":"International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS \u201918)","author":"Zheng Zhen","year":"2022","unstructured":"Zhen Zheng, Xuanda Yang, Pengzhan Zhao, Guoping Long, Kai Zhu, Feiwen Zhu, Wenyi Zhao, Xiaoyong Liu, Jun Yang, Jidong Zhai, et al. 2022. AStitch: Enabling a new multi-dimensional optimization space for memory-intensive ML training and inference on modern SIMT architectures. In International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS \u201918)."},{"key":"e_1_3_2_1_83_1","doi-asserted-by":"crossref","unstructured":"Minxuan Zhou Weihong Xu Jaeyoung Kang and Tajana Rosing. 2022. TransPIM: A memory-based acceleration via software-hardware co-design for Transformer. In High Performance Computer Architecture (HPCA \u201922).","DOI":"10.1109\/HPCA53966.2022.00082"}],"event":{"name":"PPoPP '26: 31st ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming","location":"Sydney NSW Australia","acronym":"PPoPP '26","sponsor":["SIGHPC ACM Special Interest Group on High Performance Computing, Special Interest Group on High Performance Computing","SIGPLAN ACM Special Interest Group on Programming Languages"]},"container-title":["Proceedings of the 31st ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3774934.3786434","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,1,28]],"date-time":"2026-01-28T15:28:31Z","timestamp":1769614111000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3774934.3786434"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,1,28]]},"references-count":83,"alternative-id":["10.1145\/3774934.3786434","10.1145\/3774934"],"URL":"https:\/\/doi.org\/10.1145\/3774934.3786434","relation":{},"subject":[],"published":{"date-parts":[[2026,1,28]]},"assertion":[{"value":"2026-01-28","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}