{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,1]],"date-time":"2026-06-01T20:39:14Z","timestamp":1780346354736,"version":"3.54.1"},"reference-count":49,"publisher":"Association for Computing Machinery (ACM)","issue":"12","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2021,7]]},"abstract":"<jats:p>Pervasive needs for data explorations at all scales have populated modern distributed platforms with workloads of different characteristics. The growing complexities and diversities have thereafter imposed distinct challenges to execute them on shared clusters in corporate or public clouds. This paper presents Fangorn, an adaptive execution framework built on an enriched graph model. As the underlying infrastructure for core computation platforms at Alibaba, Fangorn supports various execution modes and caters to heterogeneous workloads. With the capability to orchestrate graph executions with both long-running and requested-on-demand resources at the same time, Fangorn allows exploration of tradeoffs between latency and resource efficiency, for jobs of all scales. By modeling distributed job executions as mutable graphs with pluggable components, Fangorn offers a systematic framework to adjust job executions adaptively, according to data statistics collected during run-time. Fangorn supports an array of different computation engines ranging from relational to deep learning, and is fully deployed on production clusters across Alibaba. It manages tens of millions of distributed jobs daily, with job size scaling from one to half-million.<\/jats:p>","DOI":"10.14778\/3476311.3476376","type":"journal-article","created":{"date-parts":[[2021,10,28]],"date-time":"2021-10-28T22:48:56Z","timestamp":1635461336000},"page":"2972-2985","source":"Crossref","is-referenced-by-count":2,"title":["Fangorn"],"prefix":"10.14778","volume":"14","author":[{"given":"Yingda","family":"Chen","sequence":"first","affiliation":[{"name":"Alibaba Group Inc."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jiamang","family":"Wang","sequence":"additional","affiliation":[{"name":"Alibaba Group Inc."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yifeng","family":"Lu","sequence":"additional","affiliation":[{"name":"Alibaba Group Inc."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ying","family":"Han","sequence":"additional","affiliation":[{"name":"Alibaba Group Inc."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhiqiang","family":"Lv","sequence":"additional","affiliation":[{"name":"Alibaba Group Inc."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xuebin","family":"Min","sequence":"additional","affiliation":[{"name":"Alibaba Group Inc."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hua","family":"Cai","sequence":"additional","affiliation":[{"name":"Alibaba Group Inc."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wei","family":"Zhang","sequence":"additional","affiliation":[{"name":"Alibaba Group Inc."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Haochuan","family":"Fan","sequence":"additional","affiliation":[{"name":"Alibaba Group Inc."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chao","family":"Li","sequence":"additional","affiliation":[{"name":"Alibaba Group Inc."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tao","family":"Guan","sequence":"additional","affiliation":[{"name":"Alibaba Group Inc."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wei","family":"Lin","sequence":"additional","affiliation":[{"name":"Alibaba Group Inc."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yangqing","family":"Jia","sequence":"additional","affiliation":[{"name":"Alibaba Group Inc."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jingren","family":"Zhou","sequence":"additional","affiliation":[{"name":"Alibaba Group Inc."}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2021,10,28]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.5555\/3026877.3026899"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2742797"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/1807128.1807148"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.14778\/2824032.2824066"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.5555\/2685048.2685071"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.14778\/2536222.2536223"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.14778\/2733004.2733020"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.14778\/1454159.1454166"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/TC.2013.15"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/1327452.1327492"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2588555.2595630"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.14778\/3352063.3352132"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/CloudCom.2010.25"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/1272998.1273005"},{"key":"e_1_2_1_15_1","first-page":"9","article-title":"Impala: A Modern, Open-Source SQL Engine for Hadoop","volume":"1","author":"Kornacker Marcel","year":"2015","journal-title":"Cidr"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/1807128.1807140"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2213836.2213840"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.14778\/2850583.2850594"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00778-017-0480-7"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.5555\/2685048.2685095"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/564691.564711"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.5555\/3291168.3291210"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-10424-4_17"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.5555\/3454287.3455008"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/564691.564759"},{"key":"e_1_2_1_26_1","unstructured":"Alibaba Machine Learning Platform for AI. Accessed July 2021. https:\/\/www.alibabacloud.com\/product\/machine-learning.  Alibaba Machine Learning Platform for AI. Accessed July 2021. https:\/\/www.alibabacloud.com\/product\/machine-learning."},{"key":"e_1_2_1_27_1","unstructured":"Alibaba MaxCompute. Accessed July 2021. https:\/\/www.alibabacloud.com\/product\/maxcompute.  Alibaba MaxCompute. Accessed July 2021. https:\/\/www.alibabacloud.com\/product\/maxcompute."},{"key":"e_1_2_1_28_1","unstructured":"Distributed Strategy for Training on TensorFlow. Accessed July 2021. https:\/\/www.tensorflow.org\/guide\/distributed_training\/.  Distributed Strategy for Training on TensorFlow. Accessed July 2021. https:\/\/www.tensorflow.org\/guide\/distributed_training\/."},{"key":"e_1_2_1_29_1","unstructured":"ElasticDL. Accessed July 2021. https:\/\/elasticdl.github.io\/.  ElasticDL. Accessed July 2021. https:\/\/elasticdl.github.io\/."},{"key":"e_1_2_1_30_1","unstructured":"Kubernetes. Accessed July 2021. https:\/\/kubernetes.io\/.  Kubernetes. Accessed July 2021. https:\/\/kubernetes.io\/."},{"key":"e_1_2_1_31_1","unstructured":"NVLink. Accessed July 2021. https:\/\/www.nvidia.com\/en-us\/data-center\/nvlink\/.  NVLink. Accessed July 2021. https:\/\/www.nvidia.com\/en-us\/data-center\/nvlink\/."},{"key":"e_1_2_1_32_1","unstructured":"Spark 3.0. Accessed July 2021. https:\/\/spark.apache.org\/releases\/spark-release-3-0-0.html.  Spark 3.0. Accessed July 2021. https:\/\/spark.apache.org\/releases\/spark-release-3-0-0.html."},{"key":"e_1_2_1_33_1","unstructured":"TorchElastic. Accessed July 2021. https:\/\/github.com\/pytorch\/elastic\/.  TorchElastic. Accessed July 2021. https:\/\/github.com\/pytorch\/elastic\/."},{"key":"e_1_2_1_34_1","unstructured":"TPCx-BB 100TB Benchmark. Accessed July 2021. Official report based on computation platform build on Fangorn. http:\/\/www.tpc.org\/tpcx-bb\/results\/tpcxbb_result_detail5.asp?id=120100202.  TPCx-BB 100TB Benchmark. Accessed July 2021. Official report based on computation platform build on Fangorn. http:\/\/www.tpc.org\/tpcx-bb\/results\/tpcxbb_result_detail5.asp?id=120100202."},{"key":"e_1_2_1_35_1","unstructured":"TPCx-BB 30TB Benchmark. Accessed July 2021. Official report based on computation platform build on Fangorn. http:\/\/www.tpc.org\/tpcx-bb\/results\/tpcxbb_result_detail5.asp?id=120100201.  TPCx-BB 30TB Benchmark. Accessed July 2021. Official report based on computation platform build on Fangorn. http:\/\/www.tpc.org\/tpcx-bb\/results\/tpcxbb_result_detail5.asp?id=120100201."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2742790"},{"key":"e_1_2_1_37_1","volume-title":"Horovod: fast and easy distributed deep learning in TensorFlow. arXiv preprint arXiv:1802.05799","author":"Sergeev Alexander","year":"2018"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2019.00196"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3479162.3479181"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/2523616.2523633"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.5555\/3388242.3388275"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.5555\/3488766.3488796"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.14778\/1687553.1687565"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/1376616.1376720"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.14778\/3192965.3192967"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.5555\/1855741.1855744"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.14778\/2733004.2733012"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2010.5447802"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/3292500.3340404"}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3476311.3476376","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,28]],"date-time":"2022-12-28T11:36:17Z","timestamp":1672227377000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3476311.3476376"}},"subtitle":["adaptive execution framework for heterogeneous workloads on shared clusters"],"short-title":[],"issued":{"date-parts":[[2021,7]]},"references-count":49,"journal-issue":{"issue":"12","published-print":{"date-parts":[[2021,7]]}},"alternative-id":["10.14778\/3476311.3476376"],"URL":"https:\/\/doi.org\/10.14778\/3476311.3476376","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2021,7]]}}}