{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T03:06:08Z","timestamp":1787022368646,"version":"build-2736575974"},"reference-count":55,"publisher":"Association for Computing Machinery (ACM)","issue":"2","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2021,10]]},"abstract":"<jats:p>\n                    Embedding models have been an effective learning paradigm for high-dimensional data. However, one open issue of embedding models is that their representations (latent factors) often result in large parameter space. We observe that existing distributed training frameworks face a scalability issue of embedding models since updating and retrieving the shared embedding parameters from servers usually dominates the training cycle. In this paper, we propose HET, a new system framework that significantly improves the scalability of huge embedding model training. We embrace skewed popularity distributions of embeddings as a performance opportunity and leverage it to address the communication bottleneck with an\n                    <jats:italic>embedding cache.<\/jats:italic>\n                    To ensure consistency across the caches, we incorporate a new consistency model into HET design, which provides fine-grained consistency guarantees on a per-embedding basis. Compared to previous work that only allows staleness for read operations, HET also utilizes staleness for write operations. Evaluations on six representative tasks show that HET achieves up to 88% embedding communication reductions and up to 20.68\u00d7performance speedup over the state-of-the-art baselines.\n                  <\/jats:p>","DOI":"10.14778\/3489496.3489511","type":"journal-article","created":{"date-parts":[[2022,2,4]],"date-time":"2022-02-04T19:28:36Z","timestamp":1644002916000},"page":"312-320","source":"Crossref","is-referenced-by-count":54,"title":["HET"],"prefix":"10.14778","volume":"15","author":[{"given":"Xupeng","family":"Miao","sequence":"first","affiliation":[{"name":"Peking University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hailin","family":"Zhang","sequence":"additional","affiliation":[{"name":"Peking University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yining","family":"Shi","sequence":"additional","affiliation":[{"name":"Peking University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xiaonan","family":"Nie","sequence":"additional","affiliation":[{"name":"Peking University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhi","family":"Yang","sequence":"additional","affiliation":[{"name":"Peking University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yangyu","family":"Tao","sequence":"additional","affiliation":[{"name":"Tencent Inc."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bin","family":"Cui","sequence":"additional","affiliation":[{"name":"Peking University"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2022,2,4]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"2014. Criteo Kaggle Ad. https:\/\/www.kaggle.com\/c\/criteo-display-ad-challenge. 2014. Criteo Kaggle Ad. https:\/\/www.kaggle.com\/c\/criteo-display-ad-challenge."},{"key":"e_1_2_1_2_1","unstructured":"2020. MLPerf Benchmark. https:\/\/mlperf.org. 2020. MLPerf Benchmark. https:\/\/mlperf.org."},{"key":"e_1_2_1_3_1","unstructured":"2021. HET Appendix. https:\/\/github.com\/Hsword\/HET\/blob\/main\/vldb2021_het_appendix.pdf. 2021. HET Appendix. https:\/\/github.com\/Hsword\/HET\/blob\/main\/vldb2021_het_appendix.pdf."},{"key":"e_1_2_1_4_1","unstructured":"2021. NVIDIA collective communications library (NCCL). https:\/\/developer.nvidia.com\/nccl. 2021. NVIDIA collective communications library (NCCL). https:\/\/developer.nvidia.com\/nccl."},{"key":"e_1_2_1_5_1","unstructured":"2021. NVIDIA HugeCTR. https:\/\/github.com\/NVIDIA\/HugeCTR. 2021. NVIDIA HugeCTR. https:\/\/github.com\/NVIDIA\/HugeCTR."},{"key":"e_1_2_1_6_1","unstructured":"2021. PS-Lite. https:\/\/github.com\/dmlc\/ps-lite. 2021. PS-Lite. https:\/\/github.com\/dmlc\/ps-lite."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.5555\/3026877.3026899"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2151163.2151166"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3289600.3291011"},{"key":"e_1_2_1_10_1","doi-asserted-by":"crossref","unstructured":"Heng-Tze Cheng Levent Koc Jeremiah Harmsen Tal Shaked Tushar Chandra Hrishi Aradhye Glen Anderson Greg Corrado Wei Chai Mustafa Ispir Rohan Anil Zakaria Haque Lichan Hong Vihan Jain Xiaobing Liu and Hemal Shah. 2016. Wide & Deep Learning for Recommender Systems. In DLRS@RecSys. 7--10. Heng-Tze Cheng Levent Koc Jeremiah Harmsen Tal Shaked Tushar Chandra Hrishi Aradhye Glen Anderson Greg Corrado Wei Chai Mustafa Ispir Rohan Anil Zakaria Haque Lichan Hong Vihan Jain Xiaobing Liu and Hemal Shah. 2016. Wide & Deep Learning for Recommender Systems. In DLRS@RecSys. 7--10.","DOI":"10.1145\/2988450.2988454"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3292500.3330925"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/2959100.2959190"},{"key":"e_1_2_1_13_1","volume-title":"Le","author":"Dai Andrew M.","year":"2015","unstructured":"Andrew M. Dai , Christopher Olah , and Quoc V . Le . 2015 . Document Embedding with Paragraph Vectors. CoRR abs\/1507.07998 (2015). Andrew M. Dai, Christopher Olah, and Quoc V. Le. 2015. Document Embedding with Paragraph Vectors. CoRR abs\/1507.07998 (2015)."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.5555\/898758"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10107-014-0846-1"},{"key":"e_1_2_1_16_1","first-page":"12","article-title":"Deep Learning for User Interest and Response Prediction in Online Display Advertising. Data Sci","volume":"5","author":"Gharibshah Zhabiz","year":"2020","unstructured":"Zhabiz Gharibshah , Xingquan Zhu , Arthur Hainline , and Michael Conway . 2020 . Deep Learning for User Interest and Response Prediction in Online Display Advertising. Data Sci . Eng. 5 , 1 (2020), 12 -- 26 . Zhabiz Gharibshah, Xingquan Zhu, Arthur Hainline, and Michael Conway. 2020. Deep Learning for User Interest and Response Prediction in Online Display Advertising. Data Sci. Eng. 5, 1 (2020), 12--26.","journal-title":"Eng."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.5555\/3172077.3172127"},{"key":"e_1_2_1_18_1","volume-title":"Syntax-guided text generation via graph neural network. Sci. China Inf. Sci. 64, 5","author":"Guo Qipeng","year":"2021","unstructured":"Qipeng Guo , Xipeng Qiu , Xiangyang Xue , and Zheng Zhang . 2021. Syntax-guided text generation via graph neural network. Sci. China Inf. Sci. 64, 5 ( 2021 ). Qipeng Guo, Xipeng Qiu, Xiangyang Xue, and Zheng Zhang. 2021. Syntax-guided text generation via graph neural network. Sci. China Inf. Sci. 64, 5 (2021)."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.5555\/3294771.3294869"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1007\/s41019-019-00113-0"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.5555\/2999611.2999748"},{"key":"e_1_2_1_22_1","unstructured":"Weihua Hu Matthias Fey Marinka Zitnik Yuxiao Dong Hongyu Ren Bowen Liu Michele Catasta and Jure Leskovec. 2020. Open Graph Benchmark: Datasets for Machine Learning on Graphs. In NeurIPS. Weihua Hu Matthias Fey Marinka Zitnik Yuxiao Dong Hongyu Ren Bowen Liu Michele Catasta and Jure Leskovec. 2020. Open Graph Benchmark: Datasets for Machine Learning on Graphs. In NeurIPS."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3035918.3035933"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3366424.3383416"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2901318.2901331"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3302424.3303957"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/359545.359563"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.5555\/2685048.2685095"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.14778\/3415478.3415530"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3220023"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.5555\/2969442.2969545"},{"key":"e_1_2_1_32_1","first-page":"3049","article-title":"Asynchronous Decentralized Parallel Stochastic Gradient Descent","volume":"80","author":"Lian Xiangru","year":"2018","unstructured":"Xiangru Lian , Wei Zhang , Ce Zhang , and Ji Liu . 2018 . Asynchronous Decentralized Parallel Stochastic Gradient Descent . In ICML , Vol. 80. 3049 -- 3058 . Xiangru Lian, Wei Zhang, Ce Zhang, and Ji Liu. 2018. Asynchronous Decentralized Parallel Stochastic Gradient Descent. In ICML, Vol. 80. 3049--3058.","journal-title":"ICML"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.14778\/2212351.2212354"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3447548.3467312"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2020.3038109"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3448016.3452773"},{"key":"e_1_2_1_37_1","volume-title":"Lasagne: A Multi-Layer Graph Convolutional Network Frame-work via Node-aware Deep Architecture","author":"Miao Xupeng","year":"2021","unstructured":"Xupeng Miao , Wentao Zhang , Yingxia Shao , Bin Cui , Lei Chen , Ce Zhang , and Jiawei Jiang . 2021 . Lasagne: A Multi-Layer Graph Convolutional Network Frame-work via Node-aware Deep Architecture . IEEE Transactions on Knowledge and Data Engineering ( 2021). Xupeng Miao, Wentao Zhang, Yingxia Shao, Bin Cui, Lei Chen, Ce Zhang, and Jiawei Jiang. 2021. Lasagne: A Multi-Layer Graph Convolutional Network Frame-work via Node-aware Deep Architecture. IEEE Transactions on Knowledge and Data Engineering (2021)."},{"key":"e_1_2_1_38_1","volume-title":"Efficient Estimation of Word Representations in Vector Space. In ICLR Workshop.","author":"Mikolov Tom\u00e1s","year":"2013","unstructured":"Tom\u00e1s Mikolov , Kai Chen , Greg Corrado , and Jeffrey Dean . 2013 . Efficient Estimation of Word Representations in Vector Space. In ICLR Workshop. Tom\u00e1s Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient Estimation of Word Representations in Vector Space. In ICLR Workshop."},{"key":"e_1_2_1_39_1","unstructured":"Maxim Naumov Dheevatsa Mudigere Hao-Jun Michael Shi Jianyu Huang Narayanan Sundaraman Jongsoo Park Xiaodong Wang Udit Gupta Carole-Jean Wu Alisson G. Azzolini Dmytro Dzhulgakov Andrey Mallevich Ilia Cherniavskii Yinghai Lu Raghuraman Krishnamoorthi Ansha Yu Volodymyr Kondratenko Stephanie Pereira Xianjie Chen Wenlin Chen Vijay Rao Bill Jia Liang Xiong and Misha Smelyanskiy. 2019. Deep Learning Recommendation Model for Personalization and Recommendation Systems. CoRR abs\/1906.00091 (2019). Maxim Naumov Dheevatsa Mudigere Hao-Jun Michael Shi Jianyu Huang Narayanan Sundaraman Jongsoo Park Xiaodong Wang Udit Gupta Carole-Jean Wu Alisson G. Azzolini Dmytro Dzhulgakov Andrey Mallevich Ilia Cherniavskii Yinghai Lu Raghuraman Krishnamoorthi Ansha Yu Volodymyr Kondratenko Stephanie Pereira Xianjie Chen Wenlin Chen Vijay Rao Bill Jia Liang Xiong and Misha Smelyanskiy. 2019. Deep Learning Recommendation Model for Personalization and Recommendation Systems. CoRR abs\/1906.00091 (2019)."},{"key":"e_1_2_1_40_1","first-page":"1","article-title":"Dynamic network embedding via incremental skip-gram with negative sampling. Sci","volume":"63","author":"Peng Hao","year":"2020","unstructured":"Hao Peng , Jianxin Li , Hao Yan , Qiran Gong , Senzhang Wang , Lin Liu , Lihong Wang , and Xiang Ren . 2020 . Dynamic network embedding via incremental skip-gram with negative sampling. Sci . China Inf. Sci. 63 , 10 (2020), 1 -- 19 . Hao Peng, Jianxin Li, Hao Yan, Qiran Gong, Senzhang Wang, Lin Liu, Lihong Wang, and Xiang Ren. 2020. Dynamic network embedding via incremental skip-gram with negative sampling. Sci. China Inf. Sci. 63, 10 (2020), 1--19.","journal-title":"China Inf. Sci."},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3341301.3359642"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/2623330.2623732"},{"key":"e_1_2_1_43_1","volume-title":"Horovod: fast and easy distributed deep learning in TensorFlow. CoRR abs\/1802.05799","author":"Sergeev Alexander","year":"2018","unstructured":"Alexander Sergeev and Mike Del Balso . 2018. Horovod: fast and easy distributed deep learning in TensorFlow. CoRR abs\/1802.05799 ( 2018 ). Alexander Sergeev and Mike Del Balso. 2018. Horovod: fast and easy distributed deep learning in TensorFlow. CoRR abs\/1802.05799 (2018)."},{"key":"e_1_2_1_44_1","first-page":"1","article-title":"Deep & Cross Network for Ad Click Predictions","volume":"12","author":"Wang Ruoxi","year":"2017","unstructured":"Ruoxi Wang , Bin Fu , Gang Fu , and Mingliang Wang . 2017 . Deep & Cross Network for Ad Click Predictions . In ADKDD. 12 : 1 -- 12 :7. Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang. 2017. Deep & Cross Network for Ad Click Predictions. In ADKDD. 12:1--12:7.","journal-title":"ADKDD."},{"key":"e_1_2_1_45_1","doi-asserted-by":"crossref","unstructured":"Xiang Wang Xiangnan He Meng Wang Fuli Feng and Tat-Seng Chua. 2019. SIGIR. 165--174. Xiang Wang Xiangnan He Meng Wang Fuli Feng and Tat-Seng Chua. 2019. SIGIR. 165--174.","DOI":"10.1145\/3331184.3331267"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.5555\/3433701.3433728"},{"key":"e_1_2_1_47_1","volume-title":"Strategies and principles of distributed machine learning on big data. Engineering","author":"Xing Eric P","year":"2016","unstructured":"Eric P Xing , Qirong Ho , Pengtao Xie , and Dai Wei . 2016. Strategies and principles of distributed machine learning on big data. Engineering ( 2016 ), 179--195. Eric P Xing, Qirong Ho, Pengtao Xie, and Dai Wei. 2016. Strategies and principles of distributed machine learning on big data. Engineering (2016), 179--195."},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v33i01.33015693"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.14778\/3137628.3137649"},{"key":"e_1_2_1_50_1","doi-asserted-by":"crossref","unstructured":"Junqi Zhang Bing Bai Ye Lin Jian Liang Kun Bai and Fei Wang. 2020. General-Purpose User Embeddings based on Mobile App Usage. In SIGKDD. 2831--2840. Junqi Zhang Bing Bai Ye Lin Jian Liang Kun Bai and Fei Wang. 2020. General-Purpose User Embeddings based on Mobile App Usage. In SIGKDD. 2831--2840.","DOI":"10.1145\/3394486.3403334"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/2766462.2767711"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3299869.3314038"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/3240323.3240367"},{"key":"e_1_2_1_54_1","unstructured":"Weijie Zhao Deping Xie Ronglai Jia Yulei Qian Ruiquan Ding Mingming Sun and Ping Li. 2020. Distributed Hierarchical GPU Parameter Server for Massive Scale Deep Learning Ads Systems. In MLSys. Weijie Zhao Deping Xie Ronglai Jia Yulei Qian Ruiquan Ding Mingming Sun and Ping Li. 2020. Distributed Hierarchical GPU Parameter Server for Massive Scale Deep Learning Ads Systems. In MLSys."},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3219823"}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3489496.3489511","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,17]],"date-time":"2024-09-17T18:34:32Z","timestamp":1726598072000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3489496.3489511"}},"subtitle":["scaling out huge embedding model training via cache-enabled distributed framework"],"short-title":[],"issued":{"date-parts":[[2021,10]]},"references-count":55,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2021,10]]}},"alternative-id":["10.14778\/3489496.3489511"],"URL":"https:\/\/doi.org\/10.14778\/3489496.3489511","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2021,10]]}}}