{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,7,24]],"date-time":"2025-07-24T12:11:58Z","timestamp":1753359118865},"reference-count":62,"publisher":"Association for Computing Machinery (ACM)","issue":"11","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2021,7]]},"abstract":"<jats:p>Easy-to-use programming interfaces paired with cloud-scale processing engines have enabled big data system users to author arbitrarily complex analytical jobs over massive volumes of data. However, as the complexity and scale of analytical jobs increase, they encounter a number of unforeseen problems, hotspots with large intermediate data on temporary storage, longer job recovery time after failures, and worse query optimizer estimates being examples of issues that we are facing at Microsoft.<\/jats:p>\n          <jats:p>To address these issues, we propose Phoebe, an efficient learning-based checkpoint optimizer. Given a set of constraints and an objective function at compile-time, Phoebe is able to determine the decomposition of job plans, and the optimal set of checkpoints to preserve their outputs to durable global storage. Phoebe consists of three machine learning predictors and one optimization module. For each stage of a job, Phoebe makes accurate predictions for: (1) the execution time, (2) the output size, and (3) the start\/end time taking into account the inter-stage dependencies. Using these predictions, we formulate checkpoint optimization as an integer programming problem and propose a scalable heuristic algorithm that meets the latency requirement of the production environment.<\/jats:p>\n          <jats:p>We demonstrate the effectiveness of Phoebe in production workloads, and show that we can free the temporary storage on hotspots by more than 70% and restart failed jobs 68% faster on average with minimum performance impact. Phoebe also illustrates that adding multiple sets of checkpoints is not cost-efficient, which dramatically reduces the complexity of the optimization.<\/jats:p>","DOI":"10.14778\/3476249.3476298","type":"journal-article","created":{"date-parts":[[2021,10,27]],"date-time":"2021-10-27T16:46:23Z","timestamp":1635353183000},"page":"2505-2518","source":"Crossref","is-referenced-by-count":7,"title":["Phoebe"],"prefix":"10.14778","volume":"14","author":[{"given":"Yiwen","family":"Zhu","sequence":"first","affiliation":[{"name":"Microsoft"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Matteo","family":"Interlandi","sequence":"additional","affiliation":[{"name":"Microsoft"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Abhishek","family":"Roy","sequence":"additional","affiliation":[{"name":"Microsoft"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Krishnadhan","family":"Das","sequence":"additional","affiliation":[{"name":"Microsoft"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hiren","family":"Patel","sequence":"additional","affiliation":[{"name":"Microsoft"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Malay","family":"Bag","sequence":"additional","affiliation":[{"name":"Facebook"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hitesh","family":"Sharma","sequence":"additional","affiliation":[{"name":"Google"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alekh","family":"Jindal","sequence":"additional","affiliation":[{"name":"Microsoft"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2021,10,27]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/3292500.3330667"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2012.64"},{"key":"e_1_2_1_3_1","unstructured":"Amazon.com Inc. 2020. Amazon Athena. Retrieved July 4 2020 from hhttps:\/\/aws.amazon.com\/athena\/  Amazon.com Inc. 2020. Amazon Athena. Retrieved July 4 2020 from hhttps:\/\/aws.amazon.com\/athena\/"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/1066157.1066160"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.5555\/944919.944966"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.5555\/1162264"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1010933404324"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2463676.2465282"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.14778\/1454159.1454166"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/BigData.2013.6691580"},{"key":"e_1_2_1_11_1","volume-title":"Azure Machine Learning. Retrieved","author":"Microsoft Corp. 2020.","year":"2020"},{"key":"e_1_2_1_12_1","volume-title":"Retrieved","author":"Microsoft Corp.","year":"2020"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.5555\/3323234.3323250"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.5555\/1134241.1708449"},{"key":"e_1_2_1_15_1","volume-title":"Cloud Data Design, Orchestration, and Management Using Microsoft Azure","author":"Diaz Francesco"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.14778\/3329772.3329780"},{"issue":"9","key":"e_1_2_1_17_1","first-page":"9","article-title":"coin-or\/cbc","volume":"2","author":"Forrest J","year":"2020","journal-title":"Version"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.14778\/3384345.3384349"},{"key":"e_1_2_1_20_1","volume-title":"Exploiting depth and highway connections in convolutional recurrent deep neural networks for speech recognition. cell 50","author":"Hsu Wei-Ning","year":"2016"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.14778\/3192965.3192971"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3357223.3362726"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3183713.3190656"},{"key":"e_1_2_1_24_1","volume-title":"Microlearner: A fine-grained Learning Optimizer for Big Data Workloads at Microsoft. Under Submission","author":"Jindal Alekh","year":"2020"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.5555\/3026877.3026887"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE48307.2020.00132"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.5555\/3294996.3295074"},{"key":"e_1_2_1_28_1","volume-title":"Learned Cardinalities: Estimating Correlated Joins with Deep Learning. In 9th Biennial Conference on Innovative Data Systems Research.","author":"Kipf Andreas","year":"2019"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.14778\/1453856.1453920"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.14778\/3368289.3368299"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3267809.3267814"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1287\/opre.9.3.383"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF01585758"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.14778\/3342263.3342644"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.14778\/3342263.3342646"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.14778\/1687627.1687738"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.5555\/3015812.3016002"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDEW49219.2020.00034"},{"key":"e_1_2_1_39_1","volume-title":"Technologie und Web (BTW) 2019","author":"Neumann Thomas","year":"2013"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3209889.3209890"},{"key":"e_1_2_1_41_1","unstructured":"Laurent Perron and Vincent Furnon. [n.d.]. OR-Tools. Google. https:\/\/developers.google.com\/optimization\/  Laurent Perron and Vincent Furnon. [n.d.]. OR-Tools. Google. https:\/\/developers.google.com\/optimization\/"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3035918.3056100"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2749437"},{"key":"e_1_2_1_44_1","volume-title":"Learning AWS: Design, build, and deploy responsive applications using AWS Cloud components","author":"Sarkar Aurobindo"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3357223.3362716"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/2901318.2901319"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3318464.3380584"},{"key":"e_1_2_1_48_1","volume-title":"Proceeding of 1st International Workshop Social Data Analytics and Management (SoDAM'2016)","author":"Singh Vikram","year":"2016"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.14778\/1687553.1687609"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.5555\/2683754"},{"key":"e_1_2_1_51_1","first-page":"565","article-title":"Analyzing multiple data streams as a single data object","volume":"10","author":"Triou Edward John","year":"2020","journal-title":"US Patent"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/1989323.1989350"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/2523616.2523633"},{"key":"e_1_2_1_54_1","volume-title":"Retrieved","author":"van H\u00f6vell Wenchen Fan Herman","year":"2020"},{"key":"e_1_2_1_55_1","unstructured":"Wikipedia. 2020. Topological Sorting. Retrieved July 4 2020 from https:\/\/en.wikipedia.org\/wiki\/Topological_sorting  Wikipedia. 2020. Topological Sorting. Retrieved July 4 2020 from https:\/\/en.wikipedia.org\/wiki\/Topological_sorting"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.14778\/3291264.3291267"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/2987550.2987576"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2010.5447913"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.14778\/3421424.3421432"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.5555\/2228298.2228301"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.5555\/1863103.1863113"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00778-012-0280-z"}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3476249.3476298","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,28]],"date-time":"2022-12-28T10:07:34Z","timestamp":1672222054000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3476249.3476298"}},"subtitle":["a learning-based checkpoint optimizer"],"short-title":[],"issued":{"date-parts":[[2021,7]]},"references-count":62,"journal-issue":{"issue":"11","published-print":{"date-parts":[[2021,7]]}},"alternative-id":["10.14778\/3476249.3476298"],"URL":"https:\/\/doi.org\/10.14778\/3476249.3476298","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2021,7]]}}}