{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,11]],"date-time":"2026-07-11T05:12:14Z","timestamp":1783746734995,"version":"3.55.0"},"publisher-location":"New York, NY, USA","reference-count":56,"publisher":"ACM","license":[{"start":{"date-parts":[[2026,7,13]],"date-time":"2026-07-13T00:00:00Z","timestamp":1783900800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2026,7,13]]},"DOI":"10.1145\/3806645.3807815","type":"proceedings-article","created":{"date-parts":[[2026,7,11]],"date-time":"2026-07-11T04:21:11Z","timestamp":1783743671000},"page":"179-192","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["ResiHP: Taming LLM Training Failures with Dynamic Hybrid Parallelism"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0009-0001-3995-8596","authenticated-orcid":false,"given":"Tenghui","family":"Ma","sequence":"first","affiliation":[{"name":"Fudan University, Shanghai, China and Shanghai AI Laboratory, Shanghai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-9811-0234","authenticated-orcid":false,"given":"Jihu","family":"Guo","sequence":"additional","affiliation":[{"name":"Fudan University, Shanghai, China and Shanghai AI Laboratory, Shanghai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7048-1722","authenticated-orcid":false,"given":"Wei","family":"Gao","sequence":"additional","affiliation":[{"name":"Hong Kong University of Science and Technology, Hong Kong, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-3469-7136","authenticated-orcid":false,"given":"Sitian","family":"Lu","sequence":"additional","affiliation":[{"name":"Shanghai Jiao Tong University, Shanghai, China and Shanghai AI Laboratory, Shanghai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4568-0244","authenticated-orcid":false,"given":"Zhisheng","family":"Ye","sequence":"additional","affiliation":[{"name":"Independent Researcher, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-3897-9820","authenticated-orcid":false,"given":"Hanjing","family":"Wang","sequence":"additional","affiliation":[{"name":"Shanghai AI Laboratory, Shanghai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8865-7896","authenticated-orcid":false,"given":"Dahua","family":"Lin","sequence":"additional","affiliation":[{"name":"The Chinese University of Hong Kong, Hong Kong, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,7,13]]},"reference":[{"key":"e_1_3_3_2_2_2","first-page":"320","volume-title":"Conference on uncertainty in artificial intelligence","author":"Agudelo-Espa\u00f1a Diego","year":"2020","unstructured":"Diego Agudelo-Espa\u00f1a, Sebastian Gomez-Gonzalez, Stefan Bauer, Bernhard Sch\u00f6lkopf, and Jan Peters. 2020. Bayesian online prediction of change points. In Conference on uncertainty in artificial intelligence. PMLR, 320\u2013329."},{"key":"e_1_3_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/3492321.3519584"},{"key":"e_1_3_3_2_4_2","doi-asserted-by":"crossref","unstructured":"Zhenkun Cai Xiao Yan Kaihao Ma Yidi Wu Yuzhen Huang James Cheng Teng Su and Fan Yu. 2022. TensorOpt: Exploring the Tradeoffs in Distributed DNN Training With Auto-Parallelism. IEEE Transactions on Parallel and Distributed Systems 33 8 (2022) 1967\u20131981.","DOI":"10.1109\/TPDS.2021.3132413"},{"key":"e_1_3_3_2_5_2","first-page":"545","volume-title":"18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24)","author":"Chow Mike","year":"2024","unstructured":"Mike Chow, Yang Wang, William Wang, Ayichew Hailu, Rohan Bopardikar, Bin Zhang, Jialiang Qu, David Meisner, Santosh Sonawane, Yunqi Zhang, et\u00a0al. 2024. { ServiceLab} : Preventing tiny performance regressions at hyperscale through { Pre-Production} testing. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24). 545\u2013562."},{"key":"e_1_3_3_2_6_2","unstructured":"DeepSeek-AI Aixin Liu Bei Feng Bing Xue Bingxuan Wang Bochao Wu Chengda Lu Chenggang Zhao Chengqi Deng Chenyu Zhang Chong Ruan Damai Dai Daya Guo Dejian Yang Deli Chen Dongjie Ji Erhang Li Fangyun Lin Fucong Dai Fuli Luo Guangbo Hao Guanting Chen Guowei Li H. Zhang Han Bao Hanwei Xu Haocheng Wang Haowei Zhang Honghui Ding Huajian Xin Huazuo Gao Hui Li Hui Qu J.\u00a0L. Cai Jian Liang Jianzhong Guo Jiaqi Ni Jiashi Li Jiawei Wang Jin Chen Jingchang Chen Jingyang Yuan Junjie Qiu Junlong Li Junxiao Song Kai Dong Kai Hu Kaige Gao Kang Guan Kexin Huang Kuai Yu Lean Wang Lecong Zhang Lei Xu Leyi Xia Liang Zhao Litong Wang Liyue Zhang Meng Li Miaojun Wang Mingchuan Zhang Minghua Zhang Minghui Tang Mingming Li Ning Tian Panpan Huang Peiyi Wang Peng Zhang Qiancheng Wang Qihao Zhu Qinyu Chen Qiushi Du R.\u00a0J. Chen R.\u00a0L. Jin Ruiqi Ge Ruisong Zhang Ruizhe Pan Runji Wang Runxin Xu Ruoyu Zhang Ruyi Chen S.\u00a0S. Li Shanghao Lu Shangyan Zhou Shanhuang Chen Shaoqing Wu Shengfeng Ye Shengfeng Ye Shirong Ma Shiyu Wang Shuang Zhou Shuiping Yu Shunfeng Zhou Shuting Pan T. Wang Tao Yun Tian Pei Tianyu Sun W.\u00a0L. Xiao Wangding Zeng Wanjia Zhao Wei An Wen Liu Wenfeng Liang Wenjun Gao Wenqin Yu Wentao Zhang X.\u00a0Q. Li Xiangyue Jin Xianzu Wang Xiao Bi Xiaodong Liu Xiaohan Wang Xiaojin Shen Xiaokang Chen Xiaokang Zhang Xiaosha Chen Xiaotao Nie Xiaowen Sun Xiaoxiang Wang Xin Cheng Xin Liu Xin Xie Xingchao Liu Xingkai Yu Xinnan Song Xinxia Shan Xinyi Zhou Xinyu Yang Xinyuan Li Xuecheng Su Xuheng Lin Y.\u00a0K. Li Y.\u00a0Q. Wang Y.\u00a0X. Wei Y.\u00a0X. Zhu Yang Zhang Yanhong Xu Yanhong Xu Yanping Huang Yao Li Yao Zhao Yaofeng Sun Yaohui Li Yaohui Wang Yi Yu Yi Zheng Yichao Zhang Yifan Shi Yiliang Xiong Ying He Ying Tang Yishi Piao Yisong Wang Yixuan Tan Yiyang Ma Yiyuan Liu Yongqiang Guo Yu Wu Yuan Ou Yuchen Zhu Yuduan Wang Yue Gong Yuheng Zou Yujia He Yukun Zha Yunfan Xiong Yunxian Ma Yuting Yan Yuxiang Luo Yuxiang You Yuxuan Liu Yuyang Zhou Z.\u00a0F. Wu Z.\u00a0Z. Ren Zehui Ren Zhangli Sha Zhe Fu Zhean Xu Zhen Huang Zhen Zhang Zhenda Xie Zhengyan Zhang Zhewen Hao Zhibin Gou Zhicheng Ma Zhigang Yan Zhihong Shao Zhipeng Xu Zhiyu Wu Zhongyu Zhang Zhuoshu Li Zihui Gu Zijia Zhu Zijun Liu Zilin Li Ziwei Xie Ziyang Song Ziyi Gao and Zizheng Pan. 2025. DeepSeek-V3 Technical Report. arxiv:https:\/\/arXiv.org\/abs\/2412.19437\u00a0[cs.CL] https:\/\/arxiv.org\/abs\/2412.19437"},{"key":"e_1_3_3_2_7_2","series-title":"(USENIX ATC \u201925)","volume-title":"Proceedings of the 2025 USENIX Conference on Usenix Annual Technical Conference","author":"Dong Gen","year":"2025","unstructured":"Gen Dong, Yu Hua, Yongle Zhang, Zhangyu Chen, and Menglei Chen. 2025. Understanding and detecting fail-slow hardware failure bugs in cloud systems. In Proceedings of the 2025 USENIX Conference on Usenix Annual Technical Conference (Boston, MA, USA) (USENIX ATC \u201925). USENIX Association, USA, Article 66, 16\u00a0pages."},{"key":"e_1_3_3_2_8_2","unstructured":"Abhimanyu Dubey Abhinav Jauhri Abhinav Pandey Abhishek Kadian Ahmad Al-Dahle Aiesha Letman Akhil Mathur Alan Schelten Amy Yang Angela Fan et\u00a0al. 2024. The llama 3 herd of models. arXiv e-prints (2024) arXiv\u20132407."},{"key":"e_1_3_3_2_9_2","unstructured":"Yu Gan Mingyu Liang Sundar Dev David Lo and Christina Delimitrou. 2021. Sage: Leveraging ml to diagnose unpredictable performance in cloud microservices. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2112.06263 (2021)."},{"key":"e_1_3_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3297858.3304004"},{"key":"e_1_3_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/3694715.3695960"},{"key":"e_1_3_3_2_12_2","unstructured":"Wei Gao Yuheng Zhao Dakai An Tianyuan Wu Lunxi Cao Shaopan Xiong Ju Huang Weixun Wang Siran Yang Wenbo Su et\u00a0al. 2025. Rollpacker: Mitigating long-tail rollouts for fast synchronous rl post-training. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2509.21009 (2025)."},{"key":"e_1_3_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/3718958.3754352"},{"key":"e_1_3_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/1297027.1297033"},{"key":"e_1_3_3_2_15_2","doi-asserted-by":"crossref","unstructured":"Haryadi\u00a0S Gunawi Riza\u00a0O Suminto Russell Sears Casey Golliher Swaminathan Sundararaman Xing Lin Tim Emami Weiguang Sheng Nematollah Bidokhti Caitie McCaffrey et\u00a0al. 2018. Fail-slow at scale: Evidence of hardware performance faults in large production systems. ACM Transactions on Storage (TOS) 14 3 (2018) 1\u201326.","DOI":"10.1145\/3242086"},{"key":"e_1_3_3_2_16_2","doi-asserted-by":"publisher","unstructured":"Haryadi\u00a0S. Gunawi Riza\u00a0O. Suminto Russell Sears Casey Golliher Swaminathan Sundararaman Xing Lin Tim Emami Weiguang Sheng Nematollah Bidokhti Caitie McCaffrey Deepthi Srinivasan Biswaranjan Panda Andrew Baptist Gary Grider Parks\u00a0M. Fields Kevin Harms Robert\u00a0B. Ross Andree Jacobson Robert Ricci Kirk Webb Peter Alvaro H.\u00a0Birali Runesha Mingzhe Hao and Huaicheng Li. 2018. Fail-Slow at Scale: Evidence of Hardware Performance Faults in Large Production Systems. ACM Trans. Storage 14 3 Article 23 (Oct. 2018) 26\u00a0pages. 10.1145\/3242086","DOI":"10.1145\/3242086"},{"key":"e_1_3_3_2_17_2","unstructured":"Jihu Guo Tenghui Ma Wei Gao Peng Sun Jiaxing Li Xun Chen Yuyang Jin and Dahua Lin. 2025. AdaPtis: Reducing Pipeline Bubbles with Adaptive Pipeline Parallelism on Heterogeneous Models. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2509.23722 (2025)."},{"key":"e_1_3_3_2_18_2","series-title":"(NSDI\u201924)","volume-title":"Proceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation","author":"Hu Qinghao","year":"2024","unstructured":"Qinghao Hu, Zhisheng Ye, Zerui Wang, Guoteng Wang, Meng Zhang, Qiaoling Chen, Peng Sun, Dahua Lin, Xiaolin Wang, Yingwei Luo, Yonggang Wen, and Tianwei Zhang. 2024. Characterization of large language model development in the datacenter. In Proceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation (Santa Clara, CA, USA) (NSDI\u201924). USENIX Association, USA, Article 39, 21\u00a0pages."},{"key":"e_1_3_3_2_19_2","volume-title":"GPipe: efficient training of giant neural networks using pipeline parallelism","author":"Huang Yanping","year":"2019","unstructured":"Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Mia\u00a0Xu Chen, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc\u00a0V. Le, Yonghui Wu, and Zhifeng Chen. 2019. GPipe: efficient training of giant neural networks using pipeline parallelism. Curran Associates Inc., Red Hook, NY, USA."},{"key":"e_1_3_3_2_20_2","unstructured":"Sam\u00a0Ade Jacobs Masahiro Tanaka Chengming Zhang Minjia Zhang Shuaiwen\u00a0Leon Song Samyam Rajbhandari and Yuxiong He. 2023. DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models. arxiv:https:\/\/arXiv.org\/abs\/2309.14509\u00a0[cs.LG] https:\/\/arxiv.org\/abs\/2309.14509"},{"key":"e_1_3_3_2_21_2","volume-title":"The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling","author":"Jain Raj","year":"1991","unstructured":"Raj Jain. 1991. The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling. Wiley."},{"key":"e_1_3_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/3600006.3613152"},{"key":"e_1_3_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/3627703.3629585"},{"key":"e_1_3_3_2_24_2","first-page":"745","volume-title":"21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24)","author":"Jiang Ziheng","year":"2024","unstructured":"Ziheng Jiang, Haibin Lin, Yinmin Zhong, Qi Huang, Yangrui Chen, Zhi Zhang, Yanghua Peng, Xiang Li, Cong Xie, Shibiao Nong, et\u00a0al. 2024. { MegaScale} : Scaling large language model training to more than 10,000 { GPUs}. In 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24). 745\u2013760."},{"key":"e_1_3_3_2_25_2","unstructured":"Xueze Kang Guangyu Xiang Yuxin Wang Hao Zhang Yuchu Fang Yuhang Zhou Zhenheng Tang Youhui Lv Eliran Maman Mark Wasserman et\u00a0al. 2025. ElasWave: An Elastic-Native System for Scalable Hybrid-Parallel Training. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2510.00606 (2025)."},{"key":"e_1_3_3_2_26_2","unstructured":"Vijay\u00a0Anand Korthikanti Jared Casper Sangkug Lym Lawrence McAfee Michael Andersch Mohammad Shoeybi and Bryan Catanzaro. 2023. Reducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems 5 (2023) 341\u2013353."},{"key":"e_1_3_3_2_27_2","unstructured":"Mario\u00a0Michael Krell Matej Kosec Sergio\u00a0P. Perez and Andrew Fitzgibbon. 2022. Efficient Sequence Packing without Cross-contamination: Accelerating Large Language Models without Impacting Performance. arxiv:https:\/\/arXiv.org\/abs\/2107.02027\u00a0[cs.CL] https:\/\/arxiv.org\/abs\/2107.02027"},{"key":"e_1_3_3_2_28_2","unstructured":"Dmitry Lepikhin HyoukJoong Lee Yuanzhong Xu Dehao Chen Orhan Firat Yanping Huang Maxim Krikun Noam Shazeer and Zhifeng Chen. 2020. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2006.16668 (2020)."},{"key":"e_1_3_3_2_29_2","doi-asserted-by":"publisher","unstructured":"Haoyang Li Fangcheng Fu Sheng Lin Hao Ge Xuanyu Wang Jiawen Niu Jinbao Xue Yangyu Tao Di Wang Jie Jiang and Bin Cui. 2025. Hydraulis: Balancing Large Transformer Model Training via Co-designing Parallel Strategies and Data Assignment. Proc. ACM Manag. Data 3 6 Article 337 (Dec. 2025) 30\u00a0pages. 10.1145\/3769802","DOI":"10.1145\/3769802"},{"key":"e_1_3_3_2_30_2","unstructured":"Jinkun Lin Ziheng Jiang Zuquan Song Sida Zhao Menghan Yu Zhanghan Wang Chenyuan Wang Zuocheng Shi Xiang Shi Wei Jia et\u00a0al. 2025. Understanding Stragglers in Large Model Training Using What-if Analysis. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2505.05713 (2025)."},{"key":"e_1_3_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/HPCA57654.2024.00067"},{"key":"e_1_3_3_2_32_2","series-title":"(FAST\u201923)","volume-title":"Proceedings of the 21st USENIX Conference on File and Storage Technologies","author":"Lu Ruiming","year":"2023","unstructured":"Ruiming Lu, Erci Xu, Yiming Zhang, Fengyi Zhu, Zhaosheng Zhu, Mengtian Wang, Zongpeng Zhu, Guangtao Xue, Jiwu Shu, Minglu Li, and Jiesheng Wu. 2023. PERSEUS: a fail-slow detection framework for cloud storage systems. In Proceedings of the 21st USENIX Conference on File and Storage Technologies (Santa Clara, CA, USA) (FAST\u201923). USENIX Association, USA, Article 4, 15\u00a0pages."},{"key":"e_1_3_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2025.acl-long.996"},{"key":"e_1_3_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3341301.3359646"},{"key":"e_1_3_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/3458817.3476209"},{"key":"e_1_3_3_2_36_2","unstructured":"OpenAI Josh Achiam Steven Adler Sandhini Agarwal Lama Ahmad Ilge Akkaya Florencia\u00a0Leoni Aleman Diogo Almeida Janko Altenschmidt Sam Altman Shyamal Anadkat Red Avila Igor Babuschkin Suchir Balaji Valerie Balcom Paul Baltescu Haiming Bao Mohammad Bavarian Jeff Belgum Irwan Bello Jake Berdine Gabriel Bernadett-Shapiro Christopher Berner Lenny Bogdonoff Oleg Boiko Madelaine Boyd Anna-Luisa Brakman Greg Brockman Tim Brooks Miles Brundage Kevin Button Trevor Cai Rosie Campbell Andrew Cann Brittany Carey Chelsea Carlson Rory Carmichael Brooke Chan Che Chang Fotis Chantzis Derek Chen Sully Chen Ruby Chen Jason Chen Mark Chen Ben Chess Chester Cho Casey Chu Hyung\u00a0Won Chung Dave Cummings Jeremiah Currier Yunxing Dai Cory Decareaux Thomas Degry Noah Deutsch Damien Deville Arka Dhar David Dohan Steve Dowling Sheila Dunning Adrien Ecoffet Atty Eleti Tyna Eloundou David Farhi Liam Fedus Niko Felix Sim\u00f3n\u00a0Posada Fishman Juston Forte Isabella Fulford Leo Gao Elie Georges Christian Gibson Vik Goel Tarun Gogineni Gabriel Goh Rapha Gontijo-Lopes Jonathan Gordon Morgan Grafstein Scott Gray Ryan Greene Joshua Gross Shixiang\u00a0Shane Gu Yufei Guo Chris Hallacy Jesse Han Jeff Harris Yuchen He Mike Heaton Johannes Heidecke Chris Hesse Alan Hickey Wade Hickey Peter Hoeschele Brandon Houghton Kenny Hsu Shengli Hu Xin Hu Joost Huizinga Shantanu Jain Shawn Jain Joanne Jang Angela Jiang Roger Jiang Haozhun Jin Denny Jin Shino Jomoto Billie Jonn Heewoo Jun Tomer Kaftan \u0141ukasz Kaiser Ali Kamali Ingmar Kanitscheider Nitish\u00a0Shirish Keskar Tabarak Khan Logan Kilpatrick Jong\u00a0Wook Kim Christina Kim Yongjik Kim Jan\u00a0Hendrik Kirchner Jamie Kiros Matt Knight Daniel Kokotajlo \u0141ukasz Kondraciuk Andrew Kondrich Aris Konstantinidis Kyle Kosic Gretchen Krueger Vishal Kuo Michael Lampe Ikai Lan Teddy Lee Jan Leike Jade Leung Daniel Levy Chak\u00a0Ming Li Rachel Lim Molly Lin Stephanie Lin Mateusz Litwin Theresa Lopez Ryan Lowe Patricia Lue Anna Makanju Kim Malfacini Sam Manning Todor Markov Yaniv Markovski Bianca Martin Katie Mayer Andrew Mayne Bob McGrew Scott\u00a0Mayer McKinney Christine McLeavey Paul McMillan Jake McNeil David Medina Aalok Mehta Jacob Menick Luke Metz Andrey Mishchenko Pamela Mishkin Vinnie Monaco Evan Morikawa Daniel Mossing Tong Mu Mira Murati Oleg Murk David M\u00e9ly Ashvin Nair Reiichiro Nakano Rajeev Nayak Arvind Neelakantan Richard Ngo Hyeonwoo Noh Long Ouyang Cullen O\u2019Keefe Jakub Pachocki Alex Paino Joe Palermo Ashley Pantuliano Giambattista Parascandolo Joel Parish Emy Parparita Alex Passos Mikhail Pavlov Andrew Peng Adam Perelman Filipe de Avila Belbute\u00a0Peres Michael Petrov Henrique\u00a0Ponde de Oliveira\u00a0Pinto Michael Pokorny Michelle Pokrass Vitchyr\u00a0H. Pong Tolly Powell Alethea Power Boris Power Elizabeth Proehl Raul Puri Alec Radford Jack Rae Aditya Ramesh Cameron Raymond Francis Real Kendra Rimbach Carl Ross Bob Rotsted Henri Roussez Nick Ryder Mario Saltarelli Ted Sanders Shibani Santurkar Girish Sastry Heather Schmidt David Schnurr John Schulman Daniel Selsam Kyla Sheppard Toki Sherbakov Jessica Shieh Sarah Shoker Pranav Shyam Szymon Sidor Eric Sigler Maddie Simens Jordan Sitkin Katarina Slama Ian Sohl Benjamin Sokolowsky Yang Song Natalie Staudacher Felipe\u00a0Petroski Such Natalie Summers Ilya Sutskever Jie Tang Nikolas Tezak Madeleine\u00a0B. Thompson Phil Tillet Amin Tootoonchian Elizabeth Tseng Preston Tuggle Nick Turley Jerry Tworek Juan Felipe\u00a0Cer\u00f3n Uribe Andrea Vallone Arun Vijayvergiya Chelsea Voss Carroll Wainwright Justin\u00a0Jay Wang Alvin Wang Ben Wang Jonathan Ward Jason Wei CJ Weinmann Akila Welihinda Peter Welinder Jiayi Weng Lilian Weng Matt Wiethoff Dave Willner Clemens Winter Samuel Wolrich Hannah Wong Lauren Workman Sherwin Wu Jeff Wu Michael Wu Kai Xiao Tao Xu Sarah Yoo Kevin Yu Qiming Yuan Wojciech Zaremba Rowan Zellers Chong Zhang Marvin Zhang Shengjia Zhao Tianhao Zheng Juntang Zhuang William Zhuk and Barret Zoph. 2024. GPT-4 Technical Report. arxiv:https:\/\/arXiv.org\/abs\/2303.08774\u00a0[cs.CL] https:\/\/arxiv.org\/abs\/2303.08774"},{"key":"e_1_3_3_2_37_2","first-page":"47","volume-title":"2019 USENIX Annual Technical Conference (USENIX ATC 19)","author":"Panda Biswaranjan","year":"2019","unstructured":"Biswaranjan Panda, Deepthi Srinivasan, Huan Ke, Karan Gupta, Vinayak Khot, and Haryadi\u00a0S. Gunawi. 2019. IASO: A Fail-Slow Detection and Mitigation Framework for Distributed Storage Services. In 2019 USENIX Annual Technical Conference (USENIX ATC 19). USENIX Association, Renton, WA, 47\u201362. https:\/\/www.usenix.org\/conference\/atc19\/presentation\/panda"},{"key":"e_1_3_3_2_38_2","volume-title":"The Twelfth International Conference on Learning Representations","author":"Qi Penghui","year":"2024","unstructured":"Penghui Qi, Xinyi Wan, Guangxing Huang, and Min Lin. 2024. Zero bubble (almost) pipeline parallelism. In The Twelfth International Conference on Learning Representations."},{"key":"e_1_3_3_2_39_2","unstructured":"Mohammad Shoeybi Mostofa Patwary Raul Puri Patrick LeGresley Jared Casper and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/1909.08053 (2019)."},{"key":"e_1_3_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/3731545.3731570"},{"key":"e_1_3_3_2_41_2","first-page":"497","volume-title":"20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23)","author":"Thorpe John","year":"2023","unstructured":"John Thorpe, Pengzhan Zhao, Jonathan Eyolfson, Yifan Qiao, Zhihao Jia, Minjia Zhang, Ravi Netravali, and Guoqing\u00a0Harry Xu. 2023. Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNs. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). USENIX Association, Boston, MA, 497\u2013513. https:\/\/www.usenix.org\/conference\/nsdi23\/presentation\/thorpe"},{"key":"e_1_3_3_2_42_2","unstructured":"Hugo Touvron Louis Martin Kevin Stone Peter Albert Amjad Almahairi Yasmine Babaei Nikolay Bashlykov Soumya Batra Prajjwal Bhargava Shruti Bhosale et\u00a0al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2307.09288 (2023)."},{"key":"e_1_3_3_2_43_2","doi-asserted-by":"publisher","unstructured":"Leslie\u00a0G. Valiant. 1990. A bridging model for parallel computation. Commun. ACM 33 8 (Aug. 1990) 103\u2013111. 10.1145\/79173.79181","DOI":"10.1145\/79173.79181"},{"key":"e_1_3_3_2_44_2","unstructured":"Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan\u00a0N. Gomez Lukasz Kaiser and Illia Polosukhin. 2023. Attention Is All You Need. arxiv:https:\/\/arXiv.org\/abs\/1706.03762\u00a0[cs.CL] https:\/\/arxiv.org\/abs\/1706.03762"},{"key":"e_1_3_3_2_45_2","unstructured":"Yujie Wang Shiju Wang Shenhan Zhu Fangcheng Fu Xinyi Liu Xuefeng Xiao Huixia Li Jiashi Li Faming Wu and Bin Cui. 2025. FlexSP: Accelerating Large Language Model Training via Flexible Sequence Parallelism. arxiv:https:\/\/arXiv.org\/abs\/2412.01523\u00a0[cs.DC] https:\/\/arxiv.org\/abs\/2412.01523"},{"key":"e_1_3_3_2_46_2","series-title":"(OSDI \u201925)","volume-title":"Proceedings of the 19th USENIX Conference on Operating Systems Design and Implementation","author":"Wang Zheng","year":"2025","unstructured":"Zheng Wang, Anna Cai, Xinfeng Xie, Zaifeng Pan, Yue Guan, Weiwei Chu, Jie Wang, Shikai Li, Jianyu Huang, Chris Cai, Yuchen Hao, and Yufei Ding. 2025. WLB-LLM: workload-balanced 4D parallelism for large language model training. In Proceedings of the 19th USENIX Conference on Operating Systems Design and Implementation (Boston, MA, USA) (OSDI \u201925). USENIX Association, USA, Article 43, 17\u00a0pages."},{"key":"e_1_3_3_2_47_2","unstructured":"BigScience Workshop Teven\u00a0Le Scao Angela Fan Christopher Akiki Ellie Pavlick Suzana Ili\u0107 Daniel Hesslow Roman Castagn\u00e9 Alexandra\u00a0Sasha Luccioni Fran\u00e7ois Yvon et\u00a0al. 2022. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2211.05100 (2022)."},{"key":"e_1_3_3_2_48_2","unstructured":"Tianyuan Wu Lunxi Cao Hanfeng Lu Xiaoxiao Jiang Yinghao Yu Siran Yang Guodong Yang Jiamang Wang Lin Qu Liping Zhang et\u00a0al. 2025. Adaptra: Straggler-Resilient Hybrid-Parallel Training with Pipeline Adaptation. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2504.19232 (2025)."},{"key":"e_1_3_3_2_49_2","first-page":"731","volume-title":"2025 USENIX Annual Technical Conference (USENIX ATC 25)","author":"Wu Tianyuan","year":"2025","unstructured":"Tianyuan Wu, Wei Wang, Yinghao Yu, Siran Yang, Wenchao Wu, Qinkai Duan, Guodong Yang, Jiamang Wang, Lin Qu, and Liping Zhang. 2025. { GREYHOUND} : Hunting { Fail-Slows} in { Hybrid-Parallel} Training at Scale. In 2025 USENIX Annual Technical Conference (USENIX ATC 25). 731\u2013747."},{"key":"e_1_3_3_2_50_2","series-title":"(USENIX ATC\u201924)","volume-title":"Proceedings of the 2024 USENIX Conference on Usenix Annual Technical Conference","author":"Xiong Yifan","year":"2024","unstructured":"Yifan Xiong, Yuting Jiang, Ziyue Yang, Lei Qu, Guoshuai Zhao, Shuguang Liu, Dong Zhong, Boris Pinzur, Jie Zhang, Yang Wang, Jithin Jose, Hossein Pourreza, Jeff Baxter, Kushal Datta, Prabhat Ram, Luke Melton, Joe Chau, Peng Cheng, Yongqiang Xiong, and Lidong Zhou. 2024. SuperBench: improving cloud AI infrastructure reliability with proactive validation. In Proceedings of the 2024 USENIX Conference on Usenix Annual Technical Conference (Santa Clara, CA, USA) (USENIX ATC\u201924). USENIX Association, USA, Article 51, 16\u00a0pages."},{"key":"e_1_3_3_2_51_2","unstructured":"An Yang Anfeng Li Baosong Yang Beichen Zhang Binyuan Hui Bo Zheng Bowen Yu Chang Gao Chengen Huang Chenxu Lv Chujie Zheng Dayiheng Liu Fan Zhou Fei Huang Feng Hu Hao Ge Haoran Wei Huan Lin Jialong Tang Jian Yang Jianhong Tu Jianwei Zhang Jianxin Yang Jiaxi Yang Jing Zhou Jingren Zhou Junyang Lin Kai Dang Keqin Bao Kexin Yang Le Yu Lianghao Deng Mei Li Mingfeng Xue Mingze Li Pei Zhang Peng Wang Qin Zhu Rui Men Ruize Gao Shixuan Liu Shuang Luo Tianhao Li Tianyi Tang Wenbiao Yin Xingzhang Ren Xinyu Wang Xinyu Zhang Xuancheng Ren Yang Fan Yang Su Yichang Zhang Yinger Zhang Yu Wan Yuqiong Liu Zekun Wang Zeyu Cui Zhenru Zhang Zhipeng Zhou and Zihan Qiu. 2025. Qwen3 Technical Report. arxiv:https:\/\/arXiv.org\/abs\/2505.09388\u00a0[cs.CL] https:\/\/arxiv.org\/abs\/2505.09388"},{"key":"e_1_3_3_2_52_2","unstructured":"An Yang Baosong Yang Binyuan Hui Bo Zheng Bowen Yu Chang Zhou Chengpeng Li Chengyuan Li Dayiheng Liu Fei Huang Guanting Dong Haoran Wei Huan Lin Jialong Tang Jialin Wang Jian Yang Jianhong Tu Jianwei Zhang Jianxin Ma Jianxin Yang Jin Xu Jingren Zhou Jinze Bai Jinzheng He Junyang Lin Kai Dang Keming Lu Keqin Chen Kexin Yang Mei Li Mingfeng Xue Na Ni Pei Zhang Peng Wang Ru Peng Rui Men Ruize Gao Runji Lin Shijie Wang Shuai Bai Sinan Tan Tianhang Zhu Tianhao Li Tianyu Liu Wenbin Ge Xiaodong Deng Xiaohuan Zhou Xingzhang Ren Xinyu Zhang Xipin Wei Xuancheng Ren Xuejing Liu Yang Fan Yang Yao Yichang Zhang Yu Wan Yunfei Chu Yuqiong Liu Zeyu Cui Zhenru Zhang Zhifang Guo and Zhihao Fan. 2024. Qwen2 Technical Report. arxiv:https:\/\/arXiv.org\/abs\/2407.10671\u00a0[cs.CL] https:\/\/arxiv.org\/abs\/2407.10671"},{"key":"e_1_3_3_2_53_2","doi-asserted-by":"crossref","unstructured":"Zhisheng Ye Wei Gao Qinghao Hu Peng Sun Xiaolin Wang Yingwei Luo Tianwei Zhang and Yonggang Wen. 2024. Deep Learning Workload Scheduling in GPU Datacenters: A Survey. ACM Comput. Surv. 56 6 Article 146 (Jan. 2024) 38\u00a0pages.","DOI":"10.1145\/3638757"},{"key":"e_1_3_3_2_54_2","unstructured":"Susan Zhang Stephen Roller Naman Goyal Mikel Artetxe Moya Chen Shuohui Chen Christopher Dewan Mona Diab Xian Li Xi\u00a0Victoria Lin Todor Mihaylov Myle Ott Sam Shleifer Kurt Shuster Daniel Simig Punit\u00a0Singh Koura Anjali Sridhar Tianlu Wang and Luke Zettlemoyer. 2022. OPT: Open Pre-trained Transformer Language Models. arxiv:https:\/\/arXiv.org\/abs\/2205.01068\u00a0[cs.CL] https:\/\/arxiv.org\/abs\/2205.01068"},{"key":"e_1_3_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/3663529.3663834"},{"key":"e_1_3_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/3718958.3750472"},{"key":"e_1_3_3_2_57_2","volume-title":"OSDI","author":"Zheng Lianmin","year":"2022","unstructured":"Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Joseph\u00a0E. Gonzalez, and Ion Stoica. 2022. Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning. In OSDI."}],"event":{"name":"HPDC '26: 35th International Symposium on High-Performance Parallel and Distributed Computing","location":"Cleveland USA","acronym":"HPDC '26","sponsor":["SIGHPC ACM Special Interest Group on High Performance Computing, Special Interest Group on High Performance Computing","SIGARCH ACM Special Interest Group on Computer Architecture"]},"container-title":["Proceedings of the 35th International Symposium on High-Performance Parallel and Distributed Computing"],"original-title":[],"deposited":{"date-parts":[[2026,7,11]],"date-time":"2026-07-11T04:22:36Z","timestamp":1783743756000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3806645.3807815"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,13]]},"references-count":56,"alternative-id":["10.1145\/3806645.3807815","10.1145\/3806645"],"URL":"https:\/\/doi.org\/10.1145\/3806645.3807815","relation":{},"subject":[],"published":{"date-parts":[[2026,7,13]]},"assertion":[{"value":"2026-07-13","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}