{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,2,21]],"date-time":"2025-02-21T22:16:28Z","timestamp":1740176188267,"version":"3.37.3"},"reference-count":33,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2019,12,19]],"date-time":"2019-12-19T00:00:00Z","timestamp":1576713600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2019,12,19]],"date-time":"2019-12-19T00:00:00Z","timestamp":1576713600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Data Sci. Eng."],"published-print":{"date-parts":[[2020,3]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Intra-query fault tolerance has increasingly been a concern for online analytical processing, as more and more enterprises migrate data analytical systems from mainframes to commodity computers. Most massive parallel processing (MPP) databases do not support intra-query fault tolerance. They may suffer from prolonged query latency when running on unreliable commodity clusters. While SQL-on-Hadoop systems can utilize the fault tolerance support of low-level frameworks, such as MapReduce and Spark, their cost-effectiveness is not always acceptable. In this paper, we propose a smart intra-query fault tolerance (SIFT) mechanism for MPP databases. SIFT achieves fault tolerance by performing checkpointing, i.e., materializing intermediate results of selected operators. Different from existing approaches, SIFT aims at promoting query success rate within a given time. To achieve its goal, it needs to: (1) minimize query rerunning time after encountering failures and (2) introduce as less checkpointing overhead as possible. To evaluate SIFT in real-world MPP database systems, we implemented it in Greenplum. The experimental results indicate that it can improve success rate of query processing effectively, especially when working with unreliable hardware.<\/jats:p>","DOI":"10.1007\/s41019-019-00114-z","type":"journal-article","created":{"date-parts":[[2019,12,19]],"date-time":"2019-12-19T13:02:19Z","timestamp":1576760539000},"page":"65-79","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":12,"title":["Smart Intra-query Fault Tolerance for Massive Parallel Processing Databases"],"prefix":"10.1007","volume":"5","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6950-1415","authenticated-orcid":false,"given":"Yunhong","family":"Ji","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yunpeng","family":"Chai","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xuan","family":"Zhou","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lipeng","family":"Ren","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yajie","family":"Qin","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2019,12,19]]},"reference":[{"key":"114_CR1","unstructured":"Teradata. https:\/\/www.teradata.com\/"},{"key":"114_CR2","unstructured":"Greenplum. http:\/\/greenplum.org\/"},{"key":"114_CR3","unstructured":"Vertica. https:\/\/www.vertica.com\/"},{"key":"114_CR4","unstructured":"Apache Impala. https:\/\/impala.apache.org\/"},{"key":"114_CR5","unstructured":"Apache HAWQ. http:\/\/hawq.incubator.apache.org\/"},{"key":"114_CR6","doi-asserted-by":"crossref","unstructured":"Salama A, Binnig C, Kraska T, Zamanian E (2015) Cost-based fault-tolerance for parallel data processing. In: Proceedings of the 2015 ACM SIGMOD international conference on management of data, SIGMOD\u201915. ACM, New York, NY, USA, pp 285\u2013297","DOI":"10.1145\/2723372.2749437"},{"key":"114_CR7","unstructured":"Apache Hadoop. http:\/\/hadoop.apache.org\/"},{"key":"114_CR8","unstructured":"Apache Spark. https:\/\/spark.apache.org\/"},{"key":"114_CR9","unstructured":"Cant T, Mahony B, McCarthy J, Vu L (2006) Hierarchical verification environment. In: Proceedings of the 10th Australian workshop on safety critical systems and software\u2014vol 55, SCS\u201905, Darlinghurst, Australia. Australian Computer Society, Inc, Australia, pp 47\u201357"},{"key":"114_CR10","doi-asserted-by":"crossref","unstructured":"Upadhyaya P, Kwon Y, Balazinska M (2011) A latency and fault-tolerance optimizer for online parallel query plans, pp 241\u2013252","DOI":"10.1145\/1989323.1989350"},{"key":"114_CR11","unstructured":"TPC-H Benchmark. http:\/\/www.tpc.org\/tpch\/"},{"key":"114_CR12","doi-asserted-by":"crossref","unstructured":"Raychaudhuri S (2008) Introduction to Monte Carlo simulation. In: Proceedings of the 40th conference on winter simulation, WSC\u201908. Winter simulation conference, pp 91\u2013100","DOI":"10.1109\/WSC.2008.4736059"},{"issue":"2","key":"114_CR13","doi-asserted-by":"publisher","first-page":"185","DOI":"10.1145\/356842.356846","volume":"13","author":"PA Bernstein","year":"1981","unstructured":"Bernstein PA, Goodman N (1981) Concurrency control in distributed database systems. ACM Comput. Surv. 13(2):185\u2013221","journal-title":"ACM Comput. Surv."},{"key":"114_CR14","unstructured":"Zlib. https:\/\/zlib.net\/"},{"key":"114_CR15","unstructured":"Zstd. http:\/\/facebook.github.io\/zstd\/"},{"key":"114_CR16","unstructured":"Bartkowski S et al (2018) High availability and scalability guide for Db2 on Linux, UNIX and Windows. http:\/\/www.redbooks.ibm.com\/redbooks\/pdfs\/sg247363.pdf. Accessed 4 Mar 2012"},{"key":"114_CR17","unstructured":"Bartkowski S et al. (2018) Oracle data guard. https:\/\/docs.oracle.com\/cd\/B19306_01\/server.102\/b14239.pdf. Accessed 4 Mar 2008"},{"key":"114_CR18","unstructured":"Microsoft. High availability solutions (SQL server). https:\/\/technet.microsoft.com\/en-us\/library\/ms190202(v=sql.110).aspx. Accessed 4 Mar 2018"},{"issue":"1","key":"114_CR19","doi-asserted-by":"publisher","first-page":"53","DOI":"10.1007\/BF01277520","volume":"1","author":"H-I Hsiao","year":"1993","unstructured":"Hsiao H-I, Dewitt DJ (1993) A performance study of three high availability data replication strategies. Distrib Parallel Databases 1(1):53\u201379","journal-title":"Distrib Parallel Databases"},{"issue":"1","key":"114_CR20","doi-asserted-by":"publisher","first-page":"107","DOI":"10.1145\/1327452.1327492","volume":"51","author":"J Dean","year":"2008","unstructured":"Dean J, Ghemawat S (2008) Mapreduce: simplified data processing on large clusters. Commun. ACM 51(1):107\u2013113","journal-title":"Commun. ACM"},{"key":"114_CR21","doi-asserted-by":"crossref","unstructured":"Ghemawat S, Gobioff H, Leung S-T (2003) The Google file system. In: Proceedings of the nineteenth ACM symposium on operating systems principles, SOSP\u201903. ACM, New York, NY, USA, pp 29\u201343","DOI":"10.1145\/945445.945450"},{"key":"114_CR22","unstructured":"Condie T, Conway N, Alvaro P, Hellerstein JM, Elmeleegy K, Sears R (2009) Mapreduce online. Technical report UCB\/EECS-2009-136, EECS Department, University of California, Berkeley"},{"key":"114_CR23","doi-asserted-by":"crossref","unstructured":"Yang CM, Yen CY, Tan CC, Madden S (2010) Osprey: implementing mapreduce-style fault tolerance in a shared-nothing distributed database. In: ICDE, pp 657\u2013668, 11","DOI":"10.1109\/ICDE.2010.5447913"},{"key":"114_CR24","unstructured":"Hsiao H-I, DeWitt DJ (1990) Chained declustering: a new availability strategy for multiprocessor database machines. In: Proceedings of the sixth international conference on data engineering. IEEE Computer Society, Washington, DC, USA, pp 456\u2013465"},{"key":"114_CR25","doi-asserted-by":"crossref","unstructured":"Isard M, Budiu M, Yu Y, Birrell A, Fetterly D (2007) Dryad: distributed data-parallel programs from sequential building blocks. In: Proceedings of the 2007 Eurosys conference. Association for Computing Machinery, Inc, Lisbon, Portugal","DOI":"10.1145\/1272996.1273005"},{"issue":"7","key":"114_CR26","doi-asserted-by":"publisher","first-page":"746","DOI":"10.14778\/3192965.3192967","volume":"11","author":"Z Yint","year":"2018","unstructured":"Yint Z, Sun J, Li M, Ekanayake J, Lin H, Friedman M, Blakeley JA, Szyperski C, Devanur NR (2018) Bubble execution: resource-aware reliable analytics at cloud scale. Proc VLDB Endow 11(7):746\u2013758","journal-title":"Proc VLDB Endow"},{"key":"114_CR27","unstructured":"Hwang J-H, Balazinska M, Rasin A, Cetintemel U, Stonebraker M, Zdonik S (2005) High-availability algorithms for distributed stream processing. In: Proceedings of the 21st international conference on data engineering, ICDE\u201905. IEEE Computer Society, Washington, DC, USA, pp 779\u2013790"},{"key":"114_CR28","unstructured":"Flink. https:\/\/ci.apache.org\/projects\/flink\/flink-docs-release-1.4\/internals\/stream_checkpointing.html"},{"key":"114_CR29","first-page":"66","volume-title":"Load management and high availability in the borealis distributed stream processing engine","author":"N Tatbul","year":"2008","unstructured":"Tatbul N, Ahmad Y, \u00c7etintemel U, Hwang J-H, Xing Y, Zdonik S (2008) Load management and high availability in the borealis distributed stream processing engine. Springer, Berlin, pp 66\u201385"},{"key":"114_CR30","doi-asserted-by":"publisher","first-page":"783","DOI":"10.1109\/TR.2017.2712563","volume":"66","author":"H Li","year":"2017","unstructured":"Li H, Wu J, Jiang Z, Li X, Wei X (2017) Minimum backups for stream processing with recovery latency guarantees. IEEE Trans Reliab 66:783\u2013794","journal-title":"IEEE Trans Reliab"},{"issue":"12","key":"114_CR31","doi-asserted-by":"publisher","first-page":"1718","DOI":"10.14778\/3137765.3137777","volume":"10","author":"P Carbone","year":"2017","unstructured":"Carbone P, Ewen S, F\u00f3ra G, Haridi S, Richter S, Tzoumas K (2017) State management in apache flink&reg;: consistent stateful distributed stream processing. Proc VLDB Endow 10(12):1718\u20131729","journal-title":"Proc VLDB Endow"},{"key":"114_CR32","doi-asserted-by":"crossref","unstructured":"Murray DG, McSherry F, Isaacs R, Isard M, Barham P, Abadi M (2013) Naiad: a timely dataflow system. In: Proceedings of the twenty-fourth ACM symposium on operating systems principles, SOSP\u201913. ACM, New York, NY, USA, pp 439\u2013455","DOI":"10.1145\/2517349.2522738"},{"key":"114_CR33","doi-asserted-by":"crossref","unstructured":"Toshniwal A, Taneja S, Shukla A, Ramasamy K, Patel JM, Kulkarni S, Jackson J, Gade K, Fu M, Donham J, Bhagat N, Mittal S, Ryaboy D (2014) Storm@twitter. In: Proceedings of the 2014 ACM SIGMOD international conference on management of data, SIGMOD\u201914. ACM, New York, NY, USA, pp 147\u2013156","DOI":"10.1145\/2588555.2595641"}],"container-title":["Data Science and Engineering"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s41019-019-00114-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1007\/s41019-019-00114-z\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s41019-019-00114-z.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2020,12,18]],"date-time":"2020-12-18T00:45:00Z","timestamp":1608252300000},"score":1,"resource":{"primary":{"URL":"http:\/\/link.springer.com\/10.1007\/s41019-019-00114-z"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,12,19]]},"references-count":33,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2020,3]]}},"alternative-id":["114"],"URL":"https:\/\/doi.org\/10.1007\/s41019-019-00114-z","relation":{},"ISSN":["2364-1185","2364-1541"],"issn-type":[{"type":"print","value":"2364-1185"},{"type":"electronic","value":"2364-1541"}],"subject":[],"published":{"date-parts":[[2019,12,19]]},"assertion":[{"value":"11 June 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"6 December 2019","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"11 December 2019","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"19 December 2019","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}