{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,9,17]],"date-time":"2025-09-17T15:58:36Z","timestamp":1758124716150,"version":"3.32.0"},"reference-count":45,"publisher":"Association for Computing Machinery (ACM)","issue":"12","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2024,8]]},"abstract":"<jats:p>At ByteDance, where we execute over a million Spark jobs and handle 500PB of shuffled data daily, ensuring resource efficiency is paramount for cost savings. However, achieving optimization of resource efficiency in large-scale production environments poses significant challenges. Drawing from our practical experiences, we have identified three key issues critical to addressing resource efficiency in real-world production settings: 1 slow I\/Os leading to excessive CPU and memory idleness, 2 coarse-grained resource control causing wastage, and 3 sub-optimal job configurations resulting in low utilization. To tackle these issues, we propose a resource efficiency governance framework for Spark workloads. Specifically, 1 we devise the multi-mechanism shuffle services, including Enhanced External Shuffle Service (ESS) and Cloud Shuffle Service (CSS), where CSS employs a push-based approach to enhance I\/O efficiency through sequential reading. 2 We modify the Spark configuration parameter protocol, allowing for fine-grained resource control by introducing several new parameters such as milliCores and memoryBurst, as well as supporting operators with additional spill modes. 3 We design a two-stage configuration autotuning method, comprising rule-based and algorithm-based tuning, providing more reliable Spark configuration optimizations. By deploying these techniques on millions of Spark jobs in production over the last two years, we have achieved over 22% CPU utilization increase, 5% memory utilization increase, and 10% shuffle block time ratio decrease, effectively saving millions of CPU cores and petabytes of memory daily.<\/jats:p>","DOI":"10.14778\/3685800.3685804","type":"journal-article","created":{"date-parts":[[2024,11,8]],"date-time":"2024-11-08T17:25:21Z","timestamp":1731086721000},"page":"3759-3771","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":6,"title":["Towards Resource Efficiency: Practical Insights into Large-Scale Spark Workloads at ByteDance"],"prefix":"10.14778","volume":"17","author":[{"given":"Yixin","family":"Wu","sequence":"first","affiliation":[{"name":"ByteDance Inc."}]},{"given":"Xiuqi","family":"Huang","sequence":"additional","affiliation":[{"name":"Shanghai Jiao Tong University"}]},{"given":"Zhongjia","family":"Wei","sequence":"additional","affiliation":[{"name":"ByteDance Inc."}]},{"given":"Hang","family":"Cheng","sequence":"additional","affiliation":[{"name":"ByteDance Inc."}]},{"given":"Chaohui","family":"Xin","sequence":"additional","affiliation":[{"name":"ByteDance Inc."}]},{"given":"Zuzhi","family":"Chen","sequence":"additional","affiliation":[{"name":"ByteDance Inc."}]},{"given":"Binbin","family":"Chen","sequence":"additional","affiliation":[{"name":"ByteDance Inc."}]},{"given":"Yufei","family":"Wu","sequence":"additional","affiliation":[{"name":"ByteDance Inc."}]},{"given":"Hao","family":"Wang","sequence":"additional","affiliation":[{"name":"ByteDance Inc."}]},{"given":"Tieying","family":"Zhang","sequence":"additional","affiliation":[{"name":"ByteDance Inc."}]},{"given":"Rui","family":"Shi","sequence":"additional","affiliation":[{"name":"ByteDance Inc."}]},{"given":"Xiaofeng","family":"Gao","sequence":"additional","affiliation":[{"name":"Shanghai Jiao Tong University"}]},{"given":"Yuming","family":"Liang","sequence":"additional","affiliation":[{"name":"ByteDance Inc."}]},{"given":"Pengwei","family":"Zhao","sequence":"additional","affiliation":[{"name":"ByteDance Inc."}]},{"given":"Guihai","family":"Chen","sequence":"additional","affiliation":[{"name":"Shanghai Jiao Tong University"}]}],"member":"320","published-online":{"date-parts":[[2024,11,8]]},"reference":[{"key":"e_1_2_1_1_1","first-page":"1726","article-title":"Cost-aware Resource Recommendation for DAG-based Big Data Workflows: An Apache Spark Case Study","volume":"16","author":"Aseman-Manzar Mohammad-Mohsen","year":"2022","unstructured":"Mohammad-Mohsen Aseman-Manzar, Soroush Karimian-Aliabadi, Reza Entezari-Maleki, Bernhard Egger, and Ali Movaghar. 2022. Cost-aware Resource Recommendation for DAG-based Big Data Workflows: An Apache Spark Case Study. IEEE Transactions on Services Computing (TSC) 16, 3 (2022), 1726--1737.","journal-title":"IEEE Transactions on Services Computing (TSC)"},{"key":"e_1_2_1_2_1","first-page":"21524","article-title":"BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization","volume":"33","author":"Balandat Maximilian","year":"2020","unstructured":"Maximilian Balandat, Brian Karrer, Daniel Jiang, Samuel Daulton, Ben Letham, Andrew G Wilson, and Eytan Bakshy. 2020. BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization. Advances in Neural Information Processing Systems (NeurIPS) 33 (2020), 21524--21538.","journal-title":"Advances in Neural Information Processing Systems (NeurIPS)"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2890784"},{"key":"e_1_2_1_4_1","first-page":"458","article-title":"d-Simplexed: Adaptive Delaunay Triangulation for Performance Modeling and Prediction on Big Data Analytics","volume":"8","author":"Chen Yuxing","year":"2022","unstructured":"Yuxing Chen, Peter Goetsch, Mohammad Ashraful Hoque, Jiaheng Lu, and Sasu Tarkoma. 2022. d-Simplexed: Adaptive Delaunay Triangulation for Performance Modeling and Prediction on Big Data Analytics. IEEE Trans. Big Data 8, 2 (2022), 458--469.","journal-title":"IEEE Trans. Big Data"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10619-023-07436-y"},{"key":"e_1_2_1_6_1","volume-title":"Cost-Effective Resource Provisioning for Spark Workloads. In ACM International Conference on Information and Knowledge Management (CIKM). ACM, 2477--2480","author":"Chen Yuxing","year":"2019","unstructured":"Yuxing Chen, Jiaheng Lu, Chen Chen, Mohammad Hoque, and Sasu Tarkoma. 2019. Cost-Effective Resource Provisioning for Spark Workloads. In ACM International Conference on Information and Knowledge Management (CIKM). ACM, 2477--2480."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2021.111028"},{"key":"e_1_2_1_8_1","unstructured":"Apache Software Foundation. 2022. SPARK-27495. https:\/\/issues.apache.org\/jira\/browse\/SPARK-27495 SPIP: Support stage level resource configuration and scheduling."},{"key":"e_1_2_1_9_1","unstructured":"Apache Software Foundation. 2024. Apache Celeborn. https:\/\/celeborn.apache.org\/ Celeborn is an intermediate data service for big data compute engines."},{"key":"e_1_2_1_10_1","unstructured":"Apache Software Foundation. 2024. Apache Spark. http:\/\/spark.apache.org Spark is a unified engine for large-scale data analytics."},{"key":"e_1_2_1_11_1","unstructured":"Apache Software Foundation. 2024. Job Scheduling Spark 351 Documentation. https:\/\/spark.apache.org\/docs\/latest\/job-scheduling.html#dynamic-resource-allocation Spark provides a mechanism to dynamically adjust the resources your application occupies based on the workload."},{"key":"e_1_2_1_12_1","unstructured":"Apache Software Foundation. 2024. Tuning Spark 351 Documentation. https:\/\/spark.apache.org\/docs\/latest\/tuning.html A short guide for tuning Spark."},{"key":"e_1_2_1_13_1","volume-title":"International Conference on Extending Database Technology (EDBT). Open Proceedings, 546--549","author":"Gallinucci Enrico","year":"2019","unstructured":"Enrico Gallinucci and Matteo Golfarelli. 2019. SparkTune: Tuning Spark SQL through Query Cost Modeling. In International Conference on Extending Database Technology (EDBT). Open Proceedings, 546--549."},{"volume-title":"HPCC\/DSS\/SmartCity\/DependSys","author":"Gu Rong","key":"e_1_2_1_14_1","unstructured":"Rong Gu, Xu Huang, Haipeng Dai, Xiaoyu Geng, Xiaofei Chen, Yihua Huang, Fu Xiao, and Guihai Chen. 2022. Efficient. Scalable and Robust Data Shuffle Service for Distributed MapReduce Computing on Cloud. In HPCC\/DSS\/SmartCity\/DependSys. IEEE, 337--346."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2016.2587645"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3381027"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2021.3124670"},{"key":"e_1_2_1_18_1","volume-title":"Efficient Data Access Strategies for Hadoop and Spark on HPC Cluster with Heterogeneous Storage. In IEEE International Conference on Big Data (Big Data). IEEE, 223--232","author":"Islam Nusrat Sharmin","year":"2016","unstructured":"Nusrat Sharmin Islam, Md Wasi-ur Rahman, Xiaoyi Lu, and Dhabaleswar K DK Panda. 2016. Efficient Data Access Strategies for Hadoop and Spark on HPC Cluster with Heterogeneous Storage. In IEEE International Conference on Big Data (Big Data). IEEE, 223--232."},{"key":"e_1_2_1_19_1","volume-title":"Robotune: High-Dimensional Configuration Tuning for Cluster-Based Data Analytics. In International Conference on Parallel Processing (ICPP). ACM, 1--10","author":"Khan Md Muhib","year":"2021","unstructured":"Md Muhib Khan and Weikuan Yu. 2021. Robotune: High-Dimensional Configuration Tuning for Cluster-Based Data Analytics. In International Conference on Parallel Processing (ICPP). ACM, 1--10."},{"key":"e_1_2_1_20_1","first-page":"25463","article-title":"Batch Multi-Fidelity Bayesian Optimization with Deep Auto-Regressive Networks","volume":"34","author":"Li Shibo","year":"2021","unstructured":"Shibo Li, Robert Kirby, and Shandian Zhe. 2021. Batch Multi-Fidelity Bayesian Optimization with Deep Auto-Regressive Networks. Advances in Neural Information Processing Systems (NeurIPS) 34 (2021), 25463--25475.","journal-title":"Advances in Neural Information Processing Systems (NeurIPS)"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.14778\/3611540.3611548"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3546868"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE53745.2022.00071"},{"key":"e_1_2_1_24_1","volume-title":"Adaptive Code Learning for Spark Configuration Tuning. In IEEE International Conference on Data Engineering (ICDE). IEEE","author":"Lin Chen","year":"2022","unstructured":"Chen Lin, Junqing Zhuang, Jiadong Feng, Hui Li, Xuanhe Zhou, and Guoliang Li. 2022. Adaptive Code Learning for Spark Configuration Tuning. In IEEE International Conference on Data Engineering (ICDE). IEEE, 1995--2007."},{"key":"e_1_2_1_25_1","volume-title":"Exoshuffle: An Extensible Shuffle Architecture. In ACM International Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication (SIGCOMM). 564--577","author":"Luan Frank Sifei","year":"2023","unstructured":"Frank Sifei Luan, Stephanie Wang, Samyukta Yagati, Sean Kim, Kenneth Lien, Isaac Ong, Tony Hong, Sangbin Cho, Eric Liang, and Ion Stoica. 2023. Exoshuffle: An Extensible Shuffle Architecture. In ACM International Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication (SIGCOMM). 564--577."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDCS.2017.179"},{"key":"e_1_2_1_27_1","volume-title":"Automation & Test in Europe Conference & Exhibition (DATE). IEEE, 76--81","author":"Nikitopoulou Dimitra","year":"2021","unstructured":"Dimitra Nikitopoulou, Dimosthenis Masouros, Sotirios Xydis, and Dimitrios Soudris. 2021. Performance Analysis and Auto-Tuning for Spark In-Memory Analytics. In Design, Automation & Test in Europe Conference & Exhibition (DATE). IEEE, 76--81."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10115-023-02032-z"},{"key":"e_1_2_1_29_1","doi-asserted-by":"crossref","unstructured":"Panagiotis Petridis Anastasios Gounaris and Jordi Torres. 2016. Spark Parameter Tuning via Trial-and-Error. In Advances in Big Data. 226--237.","DOI":"10.1007\/978-3-319-47898-2_24"},{"key":"e_1_2_1_30_1","volume-title":"Why You Should Run TPC-DS: A Workload Analysis. In The International Conference on Very Large Data Bases. ACM, 1138--1149","author":"P\u00f6ss Meikel","year":"2007","unstructured":"Meikel P\u00f6ss, Raghunath Othayoth Nambiar, and David Walrath. 2007. Why You Should Run TPC-DS: A Workload Analysis. In The International Conference on Very Large Data Bases. ACM, 1138--1149."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/TNSM.2020.3034824"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2391229.2391233"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.14778\/3476311.3476362"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.14778\/3415478.3415558"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3580305.3599953"},{"key":"e_1_2_1_36_1","volume-title":"Performance Assurance Model for Applications on SPARK Platform. In Technology Conference on Performance Evaluation and Benchmarking. Springer, 131--146","author":"Singhal Rekha","year":"2017","unstructured":"Rekha Singhal and Praveen Singh. 2017. Performance Assurance Model for Applications on SPARK Platform. In Technology Conference on Performance Evaluation and Benchmarking. Springer, 131--146."},{"volume-title":"IEEE International Conference on Data Engineering (ICDE). IEEE, 396--407","author":"Song Fei","key":"e_1_2_1_37_1","unstructured":"Fei Song, Khaled Zaouk, Chenghao Lyu, Arnab Sinha, Qi Fan, Yanlei Diao, and Prashant J. Shenoy. 2021. Spark-based Cloud Data Analytics using Multi-Objective Optimization. In IEEE International Conference on Data Engineering (ICDE). IEEE, 396--407."},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11227-022-04663-5"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.14778\/3339490.3339503"},{"key":"e_1_2_1_40_1","volume-title":"ACM Symposium on Cloud Computing (SOCC). ACM, 308--323","author":"Xiang Wu","year":"2023","unstructured":"Wu Xiang, Yakun Li, Yuquan Ren, Fan Jiang, Chaohui Xin, Varun Gupta, Chao Xiang, Xinyi Song, Meng Liu, Bing Li, Kaiyang Shao, Chen Xu, Wei Shao, Yuqi Fu, Wilson Wang, Cong Xu, Wei Xu, Caixue Lin, Rui Shi, and Yuming Liang. 2023. G\u00f6del: Unified Large-Scale Resource Management and Scheduling at ByteDance. In ACM Symposium on Cloud Computing (SOCC). ACM, 308--323."},{"key":"e_1_2_1_41_1","volume-title":"LOCAT: Low-Overhead Online Configuration Auto-Tuning of Spark SQL Applications. In ACM SIGMOD International Conference on Management of Data (SIGMOD). ACM, 674--684","author":"Xin Jinhan","year":"2022","unstructured":"Jinhan Xin, Kai Hwang, and Zhibin Yu. 2022. LOCAT: Low-Overhead Online Configuration Auto-Tuning of Spark SQL Applications. In ACM SIGMOD International Conference on Management of Data (SIGMOD). ACM, 674--684."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3190508.3190534"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3448016.3457291"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3127479.3128605"},{"key":"e_1_2_1_45_1","volume-title":"PSACS: Highly-Parallel Shuffle Accelerator on Computational Storage. In IEEE International Conference on Computer Design (ICCD). IEEE, 480--487","author":"Zou Chen","year":"2021","unstructured":"Chen Zou, Hui Zhang, Andrew A Chien, and Yang-Seok Ki. 2021. PSACS: Highly-Parallel Shuffle Accelerator on Computational Storage. In IEEE International Conference on Computer Design (ICCD). IEEE, 480--487."}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3685800.3685804","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,12,31]],"date-time":"2024-12-31T05:25:25Z","timestamp":1735622725000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3685800.3685804"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8]]},"references-count":45,"journal-issue":{"issue":"12","published-print":{"date-parts":[[2024,8]]}},"alternative-id":["10.14778\/3685800.3685804"],"URL":"https:\/\/doi.org\/10.14778\/3685800.3685804","relation":{},"ISSN":["2150-8097"],"issn-type":[{"type":"print","value":"2150-8097"}],"subject":[],"published":{"date-parts":[[2024,8]]},"assertion":[{"value":"2024-11-08","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}