{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,2]],"date-time":"2026-07-02T12:47:41Z","timestamp":1782996461065,"version":"3.54.5"},"publisher-location":"New York, NY, USA","reference-count":44,"publisher":"ACM","license":[{"start":{"date-parts":[[2026,7,5]],"date-time":"2026-07-05T00:00:00Z","timestamp":1783209600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2026,7,6]]},"DOI":"10.1145\/3797905.3800530","type":"proceedings-article","created":{"date-parts":[[2026,7,2]],"date-time":"2026-07-02T11:50:37Z","timestamp":1782993037000},"page":"502-514","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["InferFast: Bridging the Gap Between Unstructured LLM Sparsity and Practical GPU Throughput"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0009-0008-9886-905X","authenticated-orcid":false,"given":"Zijie","family":"Shen","sequence":"first","affiliation":[{"name":"Guangzhou Institute of Technology, Xidian University, Guangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-3676-0210","authenticated-orcid":false,"given":"Weifeng","family":"Bu","sequence":"additional","affiliation":[{"name":"Guangzhou Institute of Technology, Xidian University, Guangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6602-7212","authenticated-orcid":false,"given":"Kai","family":"Sheng","sequence":"additional","affiliation":[{"name":"Guangzhou Institute of Technology, Xidian University, Guangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9857-6283","authenticated-orcid":false,"given":"Hao","family":"Chen","sequence":"additional","affiliation":[{"name":"College of Computer Science and Electronic Engineering, Hunan University, Changsha, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-1481-3179","authenticated-orcid":false,"given":"Xin","family":"He","sequence":"additional","affiliation":[{"name":"Guangzhou Institute of Technology, Xidian University, Guangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,7,5]]},"reference":[{"key":"e_1_3_3_1_2_2","series-title":"(SC \u201922)","volume-title":"Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis","author":"Aminabadi Reza\u00a0Yazdani","year":"2022","unstructured":"Reza\u00a0Yazdani Aminabadi, Samyam Rajbhandari, Ammar\u00a0Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, and Yuxiong He. 2022. DeepSpeed-inference: enabling efficient inference of transformer models at unprecedented scale. In Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis (Dallas, Texas) (SC \u201922). IEEE Press, Article 46, 15\u00a0pages."},{"key":"e_1_3_3_1_3_2","doi-asserted-by":"publisher","DOI":"10.52202\/079017-1468"},{"key":"e_1_3_3_1_4_2","unstructured":"Florentin Beck William Rudman and Carsten Eickhoff. 2025. TRIM: Achieving Extreme Sparsity with Targeted Row-wise Iterative Metric-driven Pruning. arxiv:https:\/\/arXiv.org\/abs\/2505.16743\u00a0[cs.CL] https:\/\/arxiv.org\/abs\/2505.16743"},{"key":"e_1_3_3_1_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/3650200.3656623"},{"key":"e_1_3_3_1_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/3581784.3607087"},{"key":"e_1_3_3_1_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/3458817.3476182"},{"key":"e_1_3_3_1_8_2","volume-title":"The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024","author":"Dao Tri","year":"2024","unstructured":"Tri Dao. 2024. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. https:\/\/openreview.net\/forum?id=mZn2Xyh9Ec"},{"key":"e_1_3_3_1_9_2","doi-asserted-by":"publisher","DOI":"10.52202\/068431-1189"},{"key":"e_1_3_3_1_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3620666.3651378"},{"key":"e_1_3_3_1_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/3689031.3717481"},{"key":"e_1_3_3_1_12_2","series-title":"(ICML\u201923)","volume-title":"Proceedings of the 40th International Conference on Machine Learning","author":"Frantar Elias","year":"2023","unstructured":"Elias Frantar and Dan Alistarh. 2023. SparseGPT: massive language models can be accurately pruned in one-shot. In Proceedings of the 40th International Conference on Machine Learning (Honolulu, Hawaii, USA) (ICML\u201923). JMLR.org, Article 414, 15\u00a0pages."},{"key":"e_1_3_3_1_13_2","series-title":"(SC \u201920)","volume-title":"Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis","author":"Gale Trevor","year":"2020","unstructured":"Trevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen. 2020. Sparse GPU kernels for deep learning. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (Atlanta, Georgia) (SC \u201920). IEEE Press, Article 17, 14\u00a0pages."},{"key":"e_1_3_3_1_14_2","unstructured":"Aaron Grattafiori Abhimanyu Dubey Abhinav Jauhri Abhinav Pandey Abhishek Kadian Ahmad Al-Dahle Aiesha Letman Akhil Mathur Alan Schelten Alex Vaughan Amy Yang Angela Fan et\u00a0al. 2024. The Llama 3 Herd of Models. arxiv:https:\/\/arXiv.org\/abs\/2407.21783\u00a0[cs.AI] https:\/\/arxiv.org\/abs\/2407.21783"},{"key":"e_1_3_3_1_15_2","unstructured":"Albert\u00a0Q. Jiang Alexandre Sablayrolles Antoine Roux Arthur Mensch Blanche Savary Chris Bamford Devendra\u00a0Singh Chaplot Diego de\u00a0las Casas Emma\u00a0Bou Hanna Florian Bressand Gianna Lengyel Guillaume Bour et\u00a0al. 2024. Mixtral of Experts. arxiv:https:\/\/arXiv.org\/abs\/2401.04088\u00a0[cs.LG] https:\/\/arxiv.org\/abs\/2401.04088"},{"key":"e_1_3_3_1_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/3725843.3756065"},{"key":"e_1_3_3_1_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/3600006.3613165"},{"key":"e_1_3_3_1_18_2","unstructured":"Kwanhee Lee Hyeondo Jang Dongyeop Lee Dan Alistarh and Namhoon Lee. 2025. The Unseen Frontier: Pushing the Limits of LLM Sparsity with Surrogate-Free ADMM. arxiv:https:\/\/arXiv.org\/abs\/2510.01650\u00a0[cs.LG] https:\/\/arxiv.org\/abs\/2510.01650"},{"key":"e_1_3_3_1_19_2","unstructured":"Andrea Matarazzo and Riccardo Torlone. 2025. A Survey on Large Language Models with some Insights on their Capabilities and Limitations. (2025). arxiv:https:\/\/arXiv.org\/abs\/2501.04040\u00a0[cs.CL] https:\/\/arxiv.org\/abs\/2501.04040"},{"key":"e_1_3_3_1_20_2","volume-title":"GPU Technology Conference","author":"Naumov Maxim","year":"2010","unstructured":"Maxim Naumov, L. Chien, Philippe Vandermersch, and Ujval Kapasi. 2010. cuSPARSE Library. In GPU Technology Conference."},{"key":"e_1_3_3_1_21_2","unstructured":"NVIDIA. 2023. FasterTransformer. https:\/\/github.com\/NVIDIA\/FasterTransformer."},{"key":"e_1_3_3_1_22_2","unstructured":"NVIDIA. 2025. cuBLAS. https:\/\/developer.nvidia.com\/cublas."},{"key":"e_1_3_3_1_23_2","unstructured":"NVIDIA. 2025. Nsight compute. https:\/\/developer.nvidia.com\/nsight-compute."},{"key":"e_1_3_3_1_24_2","unstructured":"NVIDIA. 2025. Nsight systems. https:\/\/developer.nvidia.com\/nsight-systems."},{"key":"e_1_3_3_1_25_2","unstructured":"OpenAI Josh Achiam Steven Adler Sandhini Agarwal Lama Ahmad Ilge Akkaya Florencia\u00a0Leoni Aleman Diogo Almeida Janko Altenschmidt Sam Altman Shyamal Anadkat Red Avila et\u00a0al. 2024. GPT-4 Technical Report. arxiv:https:\/\/arXiv.org\/abs\/2303.08774\u00a0[cs.CL] https:\/\/arxiv.org\/abs\/2303.08774"},{"key":"e_1_3_3_1_26_2","doi-asserted-by":"publisher","DOI":"10.1109\/CLOUD67622.2025.00036"},{"key":"e_1_3_3_1_27_2","unstructured":"Baptiste Rozi\u00e8re Jonas Gehring Fabian Gloeckle Sten Sootla Itai Gat Xiaoqing\u00a0Ellen Tan Yossi Adi Jingyu Liu Romain Sauvestre Tal Remez J\u00e9r\u00e9my Rapin Artyom Kozhevnikov Ivan Evtimov Joanna Bitton Manish Bhatt Cristian\u00a0Canton Ferrer Aaron Grattafiori Wenhan Xiong Alexandre D\u00e9fossez Jade Copet Faisal Azhar Hugo Touvron Louis Martin Nicolas Usunier Thomas Scialom and Gabriel Synnaeve. 2024. Code Llama: Open Foundation Models for Code. arxiv:https:\/\/arXiv.org\/abs\/2308.12950\u00a0[cs.CL] https:\/\/arxiv.org\/abs\/2308.12950"},{"key":"e_1_3_3_1_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/3721145.3730427"},{"key":"e_1_3_3_1_29_2","doi-asserted-by":"publisher","DOI":"10.52202\/075280-2997"},{"key":"e_1_3_3_1_30_2","doi-asserted-by":"publisher","DOI":"10.52202\/079017-2193"},{"key":"e_1_3_3_1_31_2","unstructured":"Mohammad Shoeybi Mostofa Patwary Raul Puri Patrick LeGresley Jared Casper and Bryan Catanzaro. 2019. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. CoRR abs\/1909.08053 (2019). arXiv:https:\/\/arXiv.org\/abs\/1909.08053http:\/\/arxiv.org\/abs\/1909.08053"},{"key":"e_1_3_3_1_32_2","volume-title":"The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024","author":"Sun Mingjie","year":"2024","unstructured":"Mingjie Sun, Zhuang Liu, Anna Bair, and J.\u00a0Zico Kolter. 2024. A Simple and Effective Pruning Approach for Large Language Models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. https:\/\/openreview.net\/forum?id=PxoFut3dWW"},{"key":"e_1_3_3_1_33_2","unstructured":"Hugo Touvron Louis Martin Kevin Stone Peter Albert Amjad Almahairi Yasmine Babaei Nikolay Bashlykov Soumya Batra Prajjwal Bhargava Shruti Bhosale Dan Bikel Lukas Blecher et\u00a0al. 2023. Llama 2: Open Foundation and Fine-Tuned Chat Models. arxiv:https:\/\/arXiv.org\/abs\/2307.09288\u00a0[cs.CL] https:\/\/arxiv.org\/abs\/2307.09288"},{"key":"e_1_3_3_1_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3721145.3725770"},{"key":"e_1_3_3_1_35_2","doi-asserted-by":"publisher","DOI":"10.52202\/068431-1800"},{"key":"e_1_3_3_1_36_2","doi-asserted-by":"publisher","unstructured":"Samuel Williams Andrew Waterman and David Patterson. 2009. Roofline: an insightful visual performance model for multicore architectures. Commun. ACM 52 4 (April 2009) 65\u201376. 10.1145\/1498765.1498785","DOI":"10.1145\/1498765.1498785"},{"key":"e_1_3_3_1_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3721145.3725773"},{"key":"e_1_3_3_1_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/3689031.3717455"},{"key":"e_1_3_3_1_39_2","doi-asserted-by":"publisher","unstructured":"Haojun Xia Zhen Zheng Yuchao Li Donglin Zhuang Zhongzhu Zhou Xiafei Qiu Yong Li Wei Lin and Shuaiwen\u00a0Leon Song. 2023. Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity. Proc. VLDB Endow. 17 2 (Oct. 2023) 211\u2013224. 10.14778\/3626292.3626303","DOI":"10.14778\/3626292.3626303"},{"key":"e_1_3_3_1_40_2","unstructured":"An Yang Baosong Yang Binyuan Hui Bo Zheng Bowen Yu Chang Zhou Chengpeng Li Chengyuan Li Dayiheng Liu Fei Huang Guanting Dong Haoran Wei et\u00a0al. 2024. Qwen2 Technical Report. arxiv:https:\/\/arXiv.org\/abs\/2407.10671\u00a0[cs.CL] https:\/\/arxiv.org\/abs\/2407.10671"},{"key":"e_1_3_3_1_41_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2025.findings-acl.224"},{"key":"e_1_3_3_1_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/3721145.3730425"},{"key":"e_1_3_3_1_43_2","unstructured":"Susan Zhang Stephen Roller Naman Goyal Mikel Artetxe Moya Chen Shuohui Chen Christopher Dewan Mona Diab Xian Li Xi\u00a0Victoria Lin Todor Mihaylov Myle Ott Sam Shleifer Kurt Shuster Daniel Simig Punit\u00a0Singh Koura Anjali Sridhar Tianlu Wang and Luke Zettlemoyer. 2022. OPT: Open Pre-trained Transformer Language Models. arxiv:https:\/\/arXiv.org\/abs\/2205.01068\u00a0[cs.CL] https:\/\/arxiv.org\/abs\/2205.01068"},{"key":"e_1_3_3_1_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/3710848.3710888"},{"key":"e_1_3_3_1_45_2","first-page":"213","volume-title":"16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22)","author":"Zheng Ningxin","year":"2022","unstructured":"Ningxin Zheng, Bin Lin, Quanlu Zhang, Lingxiao Ma, Yuqing Yang, Fan Yang, Yang Wang, Mao Yang, and Lidong Zhou. 2022. SparTA: Deep-Learning Model Sparsity via Tensor-with-Sparsity-Attribute. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). USENIX Association, Carlsbad, CA, 213\u2013232. https:\/\/www.usenix.org\/conference\/osdi22\/presentation\/zheng-ningxin"}],"event":{"name":"ICS '26: 2026 International Conference on Supercomputing","location":"Belfast United Kingdom","acronym":"ICS '26","sponsor":["SIGHPC ACM Special Interest Group on High Performance Computing, Special Interest Group on High Performance Computing","SIGARCH ACM Special Interest Group on Computer Architecture"]},"container-title":["Proceedings of the 40th ACM International Conference on Supercomputing"],"original-title":[],"deposited":{"date-parts":[[2026,7,2]],"date-time":"2026-07-02T12:28:30Z","timestamp":1782995310000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3797905.3800530"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,5]]},"references-count":44,"alternative-id":["10.1145\/3797905.3800530","10.1145\/3797905"],"URL":"https:\/\/doi.org\/10.1145\/3797905.3800530","relation":{},"subject":[],"published":{"date-parts":[[2026,7,5]]},"assertion":[{"value":"2026-07-05","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}