{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,9,17]],"date-time":"2025-09-17T03:18:46Z","timestamp":1758079126720,"version":"3.44.0"},"reference-count":28,"publisher":"Association for Computing Machinery (ACM)","issue":"12","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2025,8]]},"abstract":"<jats:p>In large-scale industrial recommendation systems, model checkpoints are instrumental in maintaining training goodput and numerical correctness during system failures and job preemptions. The increasing prevalence of multi-terabyte models has rendered frequent regular model checkpoints impractical, resulting in substantial lost progress when recovering from failures. As model sizes continue to grow, researchers and practitioners are compelled to investigate more efficient and scalable solutions. This paper presents DECK, a novel approach to delta model checkpointing designed for real-world industrial systems. Specifically, DECK focuses on extracting delta states with near-zero overhead, staging and streaming delta checkpoints without interrupting the training process, and merging delta checkpoints in an optimal and decoupled manner. Experimental results demonstrate that DECK achieves a 12-fold increase in checkpoint frequency while maintaining negligible impact on training throughput, thereby attaining state-of-the-art (SOTA) production performance.<\/jats:p>","DOI":"10.14778\/3750601.3750621","type":"journal-article","created":{"date-parts":[[2025,9,16]],"date-time":"2025-09-16T13:38:05Z","timestamp":1758029885000},"page":"4978-4990","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["DECK: Experiences on Delta Checkpointing for Industrial Recommendation Systems"],"prefix":"10.14778","volume":"18","author":[{"given":"Xin","family":"Gao","sequence":"first","affiliation":[{"name":"Meta Inc."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sibasish","family":"Acharya","sequence":"additional","affiliation":[{"name":"Meta Inc."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sihui","family":"Han","sequence":"additional","affiliation":[{"name":"Meta Inc."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yongxiong","family":"Ren","sequence":"additional","affiliation":[{"name":"Meta Inc."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yanli","family":"Zhao","sequence":"additional","affiliation":[{"name":"Meta Inc."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Liang","family":"Luo","sequence":"additional","affiliation":[{"name":"Meta Inc."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chucheng","family":"Wang","sequence":"additional","affiliation":[{"name":"Meta Inc."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Pradeep","family":"Fernando","sequence":"additional","affiliation":[{"name":"Meta Inc."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Saurabh","family":"Mishra","sequence":"additional","affiliation":[{"name":"Meta Inc."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Siqi","family":"Yan","sequence":"additional","affiliation":[{"name":"Meta Inc."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yicong","family":"Du","sequence":"additional","affiliation":[{"name":"Meta Inc."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Elzbieta","family":"Krepska","sequence":"additional","affiliation":[{"name":"Meta Inc."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Intaik","family":"Park","sequence":"additional","affiliation":[{"name":"Meta Inc."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Min","family":"Ni","sequence":"additional","affiliation":[{"name":"Meta Inc."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Qunshu","family":"Zhang","sequence":"additional","affiliation":[{"name":"Meta Inc."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shen","family":"Li","sequence":"additional","affiliation":[{"name":"Meta Inc."}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2025,9,16]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"[n.d.]. GGUF \u2014 huggingface.co. https:\/\/huggingface.co\/docs\/hub\/en\/gguf. [Accessed 17-03-2025]."},{"key":"e_1_2_1_2_1","unstructured":"Tom Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared D Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020) 1877\u20131901."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE55515.2023.00228"},{"key":"e_1_2_1_4_1","volume-title":"11th USENIX symposium on operating systems design and implementation (OSDI 14)","author":"Chilimbi Trishul","year":"2014","unstructured":"Trishul Chilimbi, Yutaka Suzue, Johnson Apacible, and Karthik Kalyanaraman. 2014. Project adam: Building an efficient and scalable deep learning training system. In 11th USENIX symposium on operating systems design and implementation (OSDI 14). 571\u2013582."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2959100.2959190"},{"key":"e_1_2_1_6_1","volume-title":"19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 22)","author":"Eisenman Assaf","year":"2022","unstructured":"Assaf Eisenman, Kiran Kumar Matam, Steven Ingram, Dheevatsa Mudigere, Raghuraman Krishnamoorthi, Krishnakumar Nair, Misha Smelyanskiy, and Murali Annavaram. 2022. {Check-N-Run}: A checkpointing system for training deep learning recommendation models. In 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 22). 929\u2013943."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3523227.3547387"},{"key":"e_1_2_1_8_1","unstructured":"Shen Li Yanli Zhao Rohan Varma Omkar Salpekar Pieter Noordhuis Teng Li Adam Paszke Jeff Smith Brian Vaughan Pritam Damania and Soumith Chintala. 2020. PyTorch Distributed: Experiences on Accelerating Data Parallel Training. arXiv:2006.15704 [cs.DC] https:\/\/arxiv.org\/abs\/2006.15704"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3534678.3539070"},{"key":"e_1_2_1_10_1","volume-title":"Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameters. arXiv:2111.05897 [cs.LG] https:\/\/arxiv.org\/abs\/2111.05897","author":"Lian Xiangru","year":"2021","unstructured":"Xiangru Lian, Binhang Yuan, Xuefeng Zhu, Yulong Wang, Yongjun He, Honghuan Wu, Lei Sun, Haodong Lyu, Chengjun Liu, Xing Dong, Yiqiao Liao, Mingnan Luo, Congfei Zhang, Jingru Xie, Haonan Li, Lei Chen, Renjie Huang, Jianying Lin, Chengchun Shu, Xuezhong Qiu, Zhishan Liu, Dongying Kong, Lei Yuan, Hai Yu, Sen Yang, Ce Zhang, and Ji Liu. 2021. Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameters. arXiv:2111.05897 [cs.LG] https:\/\/arxiv.org\/abs\/2111.05897"},{"key":"e_1_2_1_11_1","unstructured":"Zhuoran Liu Leqi Zou Xuan Zou Caihua Wang Biao Zhang Da Tang Bolin Zhu Yijie Zhu Peng Wu Ke Wang et al. 2022. Monolith: real time recommendation system with collisionless embedding table. arXiv preprint arXiv:2209.07663 (2022)."},{"key":"e_1_2_1_12_1","volume-title":"Jongsoo Park, Dheevatsa Mudigere, and Maxim Naumov.","author":"Luo Liang","year":"2024","unstructured":"Liang Luo, Buyun Zhang, Michael Tsang, Yinbin Ma, Ching-Hsiang Chu, Yuxin Chen, Shen Li, Yuchen Hao, Yanli Zhao, Guna Lakshminarayanan, Ellie Dingqiao Wen, Jongsoo Park, Dheevatsa Mudigere, and Maxim Naumov. 2024. Disaggregated Multi-Tower: Topology-aware Modeling Technique for Efficient Large-Scale Recommendation. arXiv:2403.00877 [cs.LG] https:\/\/arxiv.org\/abs\/2403.00877"},{"key":"e_1_2_1_13_1","volume-title":"MARM: Unlocking the Future of Recommendation Systems through Memory Augmentation and Scalable Complexity. arXiv preprint arXiv:2411.09425","author":"Lv Xiao","year":"2024","unstructured":"Xiao Lv, Jiangxia Cao, Shijie Guan, Xiaoyou Zhou, Zhiguang Qi, Yaqiang Zang, Ming Li, Ben Wang, Kun Gai, and Guorui Zhou. 2024. MARM: Unlocking the Future of Recommendation Systems through Memory Augmentation and Scalable Complexity. arXiv preprint arXiv:2411.09425 (2024)."},{"key":"e_1_2_1_14_1","volume-title":"21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24)","author":"Matam Kiran Kumar","year":"2024","unstructured":"Kiran Kumar Matam, Hani Ramezani, Fan Wang, Zeliang Chen, Yue Dong, Maomao Ding, Zhiwei Zhao, Zhengyu Zhang, Ellie Wen, and Assaf Eisenman. 2024. QuickUpdate: a Real-Time Personalization System for Large-Scale Recommendation Models. In 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24). USENIX Association, Santa Clara, CA, 731\u2013744. https:\/\/www.usenix.org\/conference\/nsdi24\/presentation\/matam"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3470496.3533727"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","unstructured":"Maxim Naumov Dheevatsa Mudigere Hao-Jun Michael Shi Jianyu Huang Narayanan Sundaraman Jongsoo Park Xiaodong Wang Udit Gupta Carole-Jean Wu Alisson G. Azzolini Dmytro Dzhulgakov Andrey Mallevich Ilia Cherniavskii Yinghai Lu Raghuraman Krishnamoorthi Ansha Yu Volodymyr Kondratenko Stephanie Pereira Xianjie Chen Wenlin Chen Vijay Rao Bill Jia Liang Xiong and Misha Smelyanskiy. 2019. Deep Learning Recommendation Model for Personalization and Recommendation Systems. 10.48550\/ARXIV.1906.00091","DOI":"10.48550\/ARXIV.1906.00091"},{"key":"e_1_2_1_17_1","unstructured":"Nvidia. 2025. Cooperative primitives for CUDA C++. https:\/\/docs.nvidia.com\/cuda\/cub\/index.html."},{"key":"e_1_2_1_18_1","unstructured":"Nvidia. 2025. cuCollections: an open-source header-only library of GPU-accelerated concurrent data structures. https:\/\/github.com\/NVIDIA\/cuCollections."},{"key":"e_1_2_1_19_1","volume-title":"19th USENIX Conference on File and Storage Technologies (FAST 21)","author":"Pan Satadru","year":"2021","unstructured":"Satadru Pan, Theano Stavrinos, Yunqiao Zhang, Atul Sikaria, Pavel Zakharov, Abhinav Sharma, Mike Shuey, Richard Wareing, Monika Gangapuram, Guanglei Cao, et al. 2021. Facebook's tectonic filesystem: Efficiency from exascale. In 19th USENIX Conference on File and Storage Technologies (FAST 21). 217\u2013231."},{"key":"e_1_2_1_20_1","volume-title":"Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971","author":"Touvron Hugo","year":"2023","unstructured":"Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth\u00e9e Lacroix, Baptiste Rozi\u00e8re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3600006.3613145"},{"key":"e_1_2_1_22_1","volume-title":"Conference on Knowledge Discovery and Data Mining (KDD). 95","author":"Yang Jie Amy","year":"2020","unstructured":"Jie Amy Yang, Jongsoo Park, Srinivas Sridharan, and Ping Tak Peter Tang. 2020. Training deep learning recommendation model with quantized collective communications. In Conference on Knowledge Discovery and Data Mining (KDD). 95."},{"key":"e_1_2_1_23_1","unstructured":"Jiaqi Zhai Lucy Liao Xing Liu Yueming Wang Rui Li Xuan Cao Leon Gao Zhaojie Gong Fangda Gu Michael He Yinghai Lu and Yu Shi. 2024. Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations. arXiv:2402.17152 [cs.LG] https:\/\/arxiv.org\/abs\/2402.17152"},{"key":"e_1_2_1_24_1","volume-title":"Wukong: Towards a scaling law for large-scale recommendation. arXiv preprint arXiv:2403.02545","author":"Zhang Buyun","year":"2024","unstructured":"Buyun Zhang, Liang Luo, Yuxin Chen, Jade Nie, Xi Liu, Daifeng Guo, Yanli Zhao, Shen Li, Yuchen Hao, Yantao Yao, et al. 2024. Wukong: Towards a scaling law for large-scale recommendation. arXiv preprint arXiv:2403.02545 (2024)."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2203.11014"},{"key":"e_1_2_1_26_1","first-page":"412","article-title":"Distributed hierarchical gpu parameter server for massive scale deep learning ads systems","volume":"2","author":"Zhao Weijie","year":"2020","unstructured":"Weijie Zhao, Deping Xie, Ronglai Jia, Yulei Qian, Ruiquan Ding, Mingming Sun, and Ping Li. 2020. Distributed hierarchical gpu parameter server for massive scale deep learning ads systems. Proceedings of Machine Learning and Systems 2 (2020), 412\u2013428.","journal-title":"Proceedings of Machine Learning and Systems"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3357384.3358045"},{"key":"e_1_2_1_28_1","doi-asserted-by":"crossref","unstructured":"Yanli Zhao Andrew Gu Rohan Varma Liang Luo Chien-Chin Huang Min Xu Less Wright Hamid Shojanazeri Myle Ott Sam Shleifer Alban Desmaison Can Balioglu Pritam Damania Bernard Nguyen Geeta Chauhan Yuchen Hao Ajit Mathews and Shen Li. 2023. PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel. arXiv:2304.11277 [cs.DC] https:\/\/arxiv.org\/abs\/2304.11277","DOI":"10.14778\/3611540.3611569"}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3750601.3750621","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,16]],"date-time":"2025-09-16T13:41:24Z","timestamp":1758030084000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3750601.3750621"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8]]},"references-count":28,"journal-issue":{"issue":"12","published-print":{"date-parts":[[2025,8]]}},"alternative-id":["10.14778\/3750601.3750621"],"URL":"https:\/\/doi.org\/10.14778\/3750601.3750621","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2025,8]]},"assertion":[{"value":"2025-09-16","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}