{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,3]],"date-time":"2026-06-03T11:53:13Z","timestamp":1780487593976,"version":"3.54.1"},"reference-count":48,"publisher":"Association for Computing Machinery (ACM)","issue":"12","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2019,8]]},"abstract":"<jats:p>In 2009 we explored the feasibility of building a hybrid SQL data analysis system that takes the best features from two competing technologies: large-scale data processing systems (such as Google MapReduce and Apache Hadoop) and parallel database management systems (such as Greenplum and Vertica). We built a prototype, HadoopDB, and demonstrated that it can deliver the high SQL query performance and efficiency of parallel database management systems while still providing the scalability, fault tolerance, and flexibility of large-scale data processing systems. Subsequently, HadoopDB grew into a commercial product, Hadapt, whose technology was eventually acquired by Teradata. In this paper, we provide an overview of HadoopDB's original design, and its evolution during the subsequent ten years of research and development effort. We describe how the project innovated both in the research lab, and as a commercial product at Hadapt and Teradata. We then discuss the current vibrant ecosystem of software projects (most of which are open source) that continued HadoopDB's legacy of implementing a systems level integration of large-scale data processing systems and parallel database technology.<\/jats:p>","DOI":"10.14778\/3352063.3352145","type":"journal-article","created":{"date-parts":[[2019,9,18]],"date-time":"2019-09-18T18:36:11Z","timestamp":1568831771000},"page":"2290-2299","source":"Crossref","is-referenced-by-count":4,"title":["Integration of large-scale data processing systems and traditional parallel database technology"],"prefix":"10.14778","volume":"12","author":[{"given":"Azza","family":"Abouzied","sequence":"first","affiliation":[{"name":"New York University, Abu Dhabi"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Daniel J.","family":"Abadi","sequence":"additional","affiliation":[{"name":"University of Maryland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kamil","family":"Bajda-Pawlikowski","sequence":"additional","affiliation":[{"name":"Starburst Data"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Avi","family":"Silberschatz","sequence":"additional","affiliation":[{"name":"Yale University"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2019,8]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"Apache Drill. https:\/\/drill.apache.org.  Apache Drill. https:\/\/drill.apache.org."},{"key":"e_1_2_1_2_1","unstructured":"Apache Hive. https:\/\/hive.apache.org\/.  Apache Hive. https:\/\/hive.apache.org\/."},{"key":"e_1_2_1_3_1","unstructured":"Apache ORC. https:\/\/orc.apache.org.  Apache ORC. https:\/\/orc.apache.org."},{"key":"e_1_2_1_4_1","unstructured":"Apache Parquet. https:\/\/parquet.apache.org.  Apache Parquet. https:\/\/parquet.apache.org."},{"key":"e_1_2_1_5_1","unstructured":"Apache SOLR. https:\/\/lucene.apache.org\/solr\/.  Apache SOLR. https:\/\/lucene.apache.org\/solr\/."},{"key":"e_1_2_1_6_1","unstructured":"Delta Lake. https:\/\/delta.io\/.  Delta Lake. https:\/\/delta.io\/."},{"key":"e_1_2_1_7_1","unstructured":"Hadapt Crunchbase profile. https:\/\/www.crunchbase.com\/organization\/hadapt.  Hadapt Crunchbase profile. https:\/\/www.crunchbase.com\/organization\/hadapt."},{"key":"e_1_2_1_8_1","unstructured":"Introduction to Presto Cost-Based Optimizer. Presto Blog https:\/\/prestosql.io\/blog\/2019\/07\/04\/cbo-introduction.html.  Introduction to Presto Cost-Based Optimizer. Presto Blog https:\/\/prestosql.io\/blog\/2019\/07\/04\/cbo-introduction.html."},{"key":"e_1_2_1_9_1","unstructured":"Presto - Next Chapter. Starburst Blog https:\/\/www.starburstdata.com\/technical-blog\/2017-12-13-presto-next-chapter\/.  Presto - Next Chapter. Starburst Blog https:\/\/www.starburstdata.com\/technical-blog\/2017-12-13-presto-next-chapter\/."},{"key":"e_1_2_1_10_1","unstructured":"Starburst Data. https:\/\/www.starburstdata.com.  Starburst Data. https:\/\/www.starburstdata.com."},{"key":"e_1_2_1_11_1","unstructured":"Teradata Bets Big on Presto for Hadoop SQL. Datanami https:\/\/www.datanami.com\/2015\/06\/08\/teradata-bets-big-on-presto-for-hadoop-sql\/.  Teradata Bets Big on Presto for Hadoop SQL. Datanami https:\/\/www.datanami.com\/2015\/06\/08\/teradata-bets-big-on-presto-for-hadoop-sql\/."},{"key":"e_1_2_1_12_1","first-page":"9336263","article-title":"Data loading systems and methods","author":"Abadi D.","year":"2011","unstructured":"D. Abadi and A. Abouzied . Data loading systems and methods . US Patent 9336263 , 2011 . D. Abadi and A. Abouzied. Data loading systems and methods. US Patent 9336263, 2011.","journal-title":"US Patent"},{"key":"e_1_2_1_13_1","first-page":"8935232","article-title":"Query execution systems and methods","author":"Abadi D.","year":"2011","unstructured":"D. Abadi and K. Bajda-Pawlikowski . Query execution systems and methods . US Patent 8935232 , 2011 . D. Abadi and K. Bajda-Pawlikowski. Query execution systems and methods. US Patent 8935232, 2011.","journal-title":"US Patent"},{"key":"e_1_2_1_14_1","first-page":"9495427","article-title":"Processing of data using a database system in communication with a data processing framework","author":"Abadi D.","year":"2011","unstructured":"D. Abadi , K. Bajda-Pawlikowski , A. Abouzied , and A. Silberschatz . Processing of data using a database system in communication with a data processing framework . US Patent 9495427 , 2011 . D. Abadi, K. Bajda-Pawlikowski, A. Abouzied, and A. Silberschatz. Processing of data using a database system in communication with a data processing framework. US Patent 9495427, 2011.","journal-title":"US Patent"},{"key":"e_1_2_1_15_1","first-page":"9934276","article-title":"Systems and methods for fault tolerant, adaptive execution of arbitrary queries at low latency","author":"Abadi D.","year":"2013","unstructured":"D. Abadi , K. Bajda-Pawlikowski , R. Schlussel , and P. Wickline . Systems and methods for fault tolerant, adaptive execution of arbitrary queries at low latency . US Patent 9934276 , 2013 . D. Abadi, K. Bajda-Pawlikowski, R. Schlussel, and P. Wickline. Systems and methods for fault tolerant, adaptive execution of arbitrary queries at low latency. US Patent 9934276, 2013.","journal-title":"US Patent"},{"key":"e_1_2_1_16_1","first-page":"8886631","article-title":"Query execution systems and methods","author":"Abadi D.","year":"2012","unstructured":"D. Abadi and J. Huang . Query execution systems and methods . US Patent 8886631 , 2012 . D. Abadi and J. Huang. Query execution systems and methods. US Patent 8886631, 2012.","journal-title":"US Patent"},{"key":"e_1_2_1_17_1","first-page":"9471711","article-title":"Schema-less access to stored data","author":"Abadi D.","year":"2013","unstructured":"D. Abadi , D. Tahara , and T. Diamond . Schema-less access to stored data . US Patent 9471711 , 2013 . D. Abadi, D. Tahara, and T. Diamond. Schema-less access to stored data. US Patent 9471711, 2013.","journal-title":"US Patent"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.14778\/1687627.1687731"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2452376.2452377"},{"key":"e_1_2_1_20_1","first-page":"169","volume-title":"VLDB '01","author":"Ailamaki A.","year":"2001","unstructured":"A. Ailamaki , D. J. DeWitt , M. D. Hill , and M. Skounakis . Weaving relations for cache performance . VLDB '01 , pages 169 -- 180 , 2001 . A. Ailamaki, D. J. DeWitt, M. D. Hill, and M. Skounakis. Weaving relations for cache performance. VLDB '01, pages 169--180, 2001."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2742797"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/1989323.1989447"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3183713.3190662"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3299869.3314045"},{"issue":"4","key":"e_1_2_1_25_1","first-page":"28","article-title":"Apache flink<sup>TM<\/sup>: Stream and batch processing in a single engine","volume":"38","author":"Carbone P.","year":"2015","unstructured":"P. Carbone , A. Katsifodimos , S. Ewen , V. Markl , S. Haridi , and K. Tzoumas . Apache flink<sup>TM<\/sup>: Stream and batch processing in a single engine . IEEE Data Eng. Bull. , 38 ( 4 ): 28 -- 38 , 2015 . P. Carbone, A. Katsifodimos, S. Ewen, V. Markl, S. Haridi, and K. Tzoumas. Apache flink<sup>TM<\/sup>: Stream and batch processing in a single engine. IEEE Data Eng. Bull., 38(4):28--38, 2015.","journal-title":"IEEE Data Eng. Bull."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.14778\/1454159.1454166"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2588555.2595636"},{"key":"e_1_2_1_28_1","first-page":"137","volume-title":"OSDI'04","author":"Dean J.","year":"2004","unstructured":"J. Dean and S. Ghemawat . Mapreduce: Simplified data processing on large clusters . OSDI'04 , pages 137 -- 150 , San Francisco, CA , 2004 . J. Dean and S. Ghemawat. Mapreduce: Simplified data processing on large clusters. OSDI'04, pages 137--150, San Francisco, CA, 2004."},{"key":"e_1_2_1_29_1","volume-title":"The Database Column","author":"DeWitt D. J.","year":"2008","unstructured":"D. J. DeWitt and M. Stonebraker . Mapreduce: A major step backwards . The Database Column , 2008 . D. J. DeWitt and M. Stonebraker. Mapreduce: A major step backwards. The Database Column, 2008."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2882903.2882924"},{"key":"e_1_2_1_31_1","volume-title":"Proceedings of the 2017 Conference on Innovative Data Systems Research, CIDR '17","author":"Hellerstein J.","year":"2017","unstructured":"J. Hellerstein , J. Gonzalez , V. Sreekanti , : A data context service . In Proceedings of the 2017 Conference on Innovative Data Systems Research, CIDR '17 , 2017 . J. Hellerstein, J. Gonzalez, V. Sreekanti, et al. Ground: A data context service. In Proceedings of the 2017 Conference on Innovative Data Systems Research, CIDR '17, 2017."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.14778\/2904483.2904486"},{"issue":"21","key":"e_1_2_1_33_1","first-page":"1123","article-title":"Scalable sparql querying of large rdf graphs","volume":"4","author":"Huang J.","year":"2011","unstructured":"J. Huang , D. J. Abadi , and K. Ren . Scalable sparql querying of large rdf graphs . PVLDB , 4 ( 21 ): 1123 -- 1134 , 2011 . J. Huang, D. J. Abadi, and K. Ren. Scalable sparql querying of large rdf graphs. PVLDB, 4(21):1123--1134, 2011.","journal-title":"PVLDB"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2014.6816640"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/1272996.1273005"},{"key":"e_1_2_1_36_1","volume-title":"CIDR","author":"Kornacker M.","year":"2015","unstructured":"M. Kornacker , A. Behm , V. Bittorf , T. Bobrovytsky , C. Ching , A. Choi , J. Erickson , M. Grund , D. Hecht , M. Jacobs , I. Joshi , L. Kuff , D. Kumar , A. Leblang , N. Li , I. Pandis , H. Robinson , D. Rorke , S. Rus , J. Russell , D. Tsirogiannis , S. Wanderman-Milne , and M. Yoder . Impala: A modern, open-source sql engine for hadoop . In CIDR , 2015 . M. Kornacker, A. Behm, V. Bittorf, T. Bobrovytsky, C. Ching, A. Choi, J. Erickson, M. Grund, D. Hecht, M. Jacobs, I. Joshi, L. Kuff, D. Kumar, A. Leblang, N. Li, I. Pandis, H. Robinson, D. Rorke, S. Rus, J. Russell, D. Tsirogiannis, S. Wanderman-Milne, and M. Yoder. Impala: A modern, open-source sql engine for hadoop. In CIDR, 2015."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939856"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.14778\/1920841.1920886"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/1376616.1376726"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/1559845.1559865"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2742790"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2019.00196"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/1629175.1629197"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/2588555.2612183"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.14778\/1687553.1687609"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/2463676.2465288"},{"key":"e_1_2_1_47_1","first-page":"1","volume-title":"OSDI'08","author":"Yu Y.","year":"2008","unstructured":"Y. Yu , M. Isard , D. Fetterly , M. Budiu , U. Erlingsson , P. K. Gunda , and J. Currey . Dryadlinq: A system for general-purpose distributed data-parallel computing using a high-level language . OSDI'08 , pages 1 -- 14 , 2008 . Y. Yu, M. Isard, D. Fetterly, M. Budiu, U. Erlingsson, P. K. Gunda, and J. Currey. Dryadlinq: A system for general-purpose distributed data-parallel computing using a high-level language. OSDI'08, pages 1--14, 2008."},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/2934664"}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3352063.3352145","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,28]],"date-time":"2022-12-28T10:36:18Z","timestamp":1672223778000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3352063.3352145"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,8]]},"references-count":48,"journal-issue":{"issue":"12","published-print":{"date-parts":[[2019,8]]}},"alternative-id":["10.14778\/3352063.3352145"],"URL":"https:\/\/doi.org\/10.14778\/3352063.3352145","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2019,8]]}}}