{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,20]],"date-time":"2026-10-20T03:27:23Z","timestamp":1792466843003,"version":"4.5.0"},"reference-count":26,"publisher":"Association for Computing Machinery (ACM)","issue":"12","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2012,8]]},"abstract":"<jats:p>This tutorial is motivated by the clear need of many organizations, companies, and researchers to deal with big data volumes efficiently. Examples include web analytics applications, scientific applications, and social networks. A popular data processing engine for big data is Hadoop MapReduce. Early versions of Hadoop MapReduce suffered from severe performance problems. Today, this is becoming history. There are many techniques that can be used with Hadoop MapReduce jobs to boost performance by orders of magnitude. In this tutorial we teach such techniques. First, we will briefly familiarize the audience with Hadoop MapReduce and motivate its use for big data processing. Then, we will focus on different data management techniques, going from job optimization to physical data organization like data layouts and indexes. Throughout this tutorial, we will highlight the similarities and differences between Hadoop MapReduce and Parallel DBMS. Furthermore, we will point out unresolved research problems and open issues.<\/jats:p>","DOI":"10.14778\/2367502.2367562","type":"journal-article","created":{"date-parts":[[2014,6,24]],"date-time":"2014-06-24T12:17:57Z","timestamp":1403612277000},"page":"2014-2015","source":"Crossref","is-referenced-by-count":175,"title":["Efficient big data processing in Hadoop MapReduce"],"prefix":"10.14778","volume":"5","author":[{"given":"Jens","family":"Dittrich","sequence":"first","affiliation":[{"name":"Saarland University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jorge-Arnulfo","family":"Quian\u00e9-Ruiz","sequence":"additional","affiliation":[{"name":"Saarland University"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2012,8]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"Hadoop http:\/\/hadoop.apache.org\/mapreduce\/. Hadoop http:\/\/hadoop.apache.org\/mapreduce\/."},{"issue":"2","key":"e_1_2_1_2_1","first-page":"1664","article-title":"Column-Oriented Database Systems","volume":"2","author":"Abadi D.","year":"2009","unstructured":"D. Abadi Column-Oriented Database Systems . PVDLB , 2 ( 2 ): 1664 -- 1665 , 2009 . D. Abadi et al. Column-Oriented Database Systems. PVDLB, 2(2):1664--1665, 2009.","journal-title":"PVDLB"},{"key":"e_1_2_1_3_1","doi-asserted-by":"crossref","first-page":"99","DOI":"10.1145\/1739041.1739056","volume-title":"EDBT","author":"Afrati F. N.","year":"2010","unstructured":"F. N. Afrati and J. D. Ullman . Optimizing Joins in a Map-Reduce Environment . In EDBT , pages 99 -- 110 , 2010 . 10.1145\/1739041.1739056 F. N. Afrati and J. D. Ullman. Optimizing Joins in a Map-Reduce Environment. In EDBT, pages 99--110, 2010. 10.1145\/1739041.1739056"},{"key":"e_1_2_1_4_1","first-page":"137","volume-title":"SOCC","author":"Babu S.","year":"2010","unstructured":"S. Babu . Towards automatic optimization of MapReduce programs . In SOCC , pages 137 -- 142 , 2010 . 10.1145\/1807128.1807150 S. Babu. Towards automatic optimization of MapReduce programs. In SOCC, pages 137--142, 2010. 10.1145\/1807128.1807150"},{"key":"e_1_2_1_5_1","first-page":"975","volume-title":"SIGMOD","author":"Blanas S.","year":"2010","unstructured":"S. Blanas A Comparison of Join Algorithms for Log Processing in MapReduce . In SIGMOD , pages 975 -- 986 , 2010 . 10.1145\/1807167.1807273 S. Blanas et al. A Comparison of Join Algorithms for Log Processing in MapReduce. In SIGMOD, pages 975--986, 2010. 10.1145\/1807167.1807273"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/1629175.1629198"},{"issue":"1","key":"e_1_2_1_7_1","first-page":"519","article-title":"Hadoop++: Making a Yellow Elephant Run Like a Cheetah (Without It Even Noticing)","volume":"3","author":"Dittrich J.","year":"2010","unstructured":"J. Dittrich , J.-A. Quian\u00e9-Ruiz , A. Jindal , Y. Kargin , V. Setty , and J. Schad . Hadoop++: Making a Yellow Elephant Run Like a Cheetah (Without It Even Noticing) . PVLDB , 3 ( 1 ): 519 -- 529 , 2010 . J. Dittrich, J.-A. Quian\u00e9-Ruiz, A. Jindal, Y. Kargin, V. Setty, and J. Schad. Hadoop++: Making a Yellow Elephant Run Like a Cheetah (Without It Even Noticing). PVLDB, 3(1):519--529, 2010.","journal-title":"PVLDB"},{"key":"e_1_2_1_8_1","first-page":"5","article-title":"Only Aggressive Elephants are Fast Elephants","author":"Dittrich J.","year":"2012","unstructured":"J. Dittrich , J.-A. Quian\u00e9-Ruiz , S. Richter , S. Schuh , A. Jindal , and J. Schad . Only Aggressive Elephants are Fast Elephants . PVLDB , 5 , 2012 . J. Dittrich, J.-A. Quian\u00e9-Ruiz, S. Richter, S. Schuh, A. Jindal, and J. Schad. Only Aggressive Elephants are Fast Elephants. PVLDB, 5, 2012.","journal-title":"PVLDB"},{"issue":"7","key":"e_1_2_1_9_1","first-page":"419","article-title":"Column-Oriented Storage Techniques for MapReduce","volume":"4","author":"Floratou A.","year":"2011","unstructured":"A. Floratou Column-Oriented Storage Techniques for MapReduce . PVLDB , 4 ( 7 ): 419 -- 429 , 2011 . A. Floratou et al. Column-Oriented Storage Techniques for MapReduce. PVLDB, 4(7):419--429, 2011.","journal-title":"PVLDB"},{"issue":"2","key":"e_1_2_1_10_1","first-page":"1414","article-title":"Building a HighLevel Dataflow System on Top of MapReduce","volume":"2","author":"Gates A.","year":"2009","unstructured":"A. Gates Building a HighLevel Dataflow System on Top of MapReduce : The Pig Experience. PVLDB , 2 ( 2 ): 1414 -- 1425 , 2009 . A. Gates et al. Building a HighLevel Dataflow System on Top of MapReduce: The Pig Experience. PVLDB, 2(2):1414--1425, 2009.","journal-title":"The Pig Experience. PVLDB"},{"key":"e_1_2_1_11_1","doi-asserted-by":"crossref","first-page":"29","DOI":"10.1145\/945445.945450","volume-title":"SOSP","author":"Ghemawat S.","year":"2003","unstructured":"S. Ghemawat , H. Gobioff , and S.-T. Leung . The Google file system . In SOSP , pages 29 -- 43 , 2003 . 10.1145\/945445.945450 S. Ghemawat, H. Gobioff, and S.-T. Leung. The Google file system. In SOSP, pages 29--43, 2003. 10.1145\/945445.945450"},{"issue":"11","key":"e_1_2_1_12_1","first-page":"1111","article-title":"Profiling, What-if Analysis, and Cost-based Optimization of MapReduce Programs","volume":"4","author":"Herodotou H.","year":"2011","unstructured":"H. Herodotou and S. Babu . Profiling, What-if Analysis, and Cost-based Optimization of MapReduce Programs . PVLDB , 4 ( 11 ): 1111 -- 1122 , 2011 . H. Herodotou and S. Babu. Profiling, What-if Analysis, and Cost-based Optimization of MapReduce Programs. PVLDB, 4(11):1111--1122, 2011.","journal-title":"PVLDB"},{"key":"e_1_2_1_13_1","doi-asserted-by":"crossref","first-page":"59","DOI":"10.1145\/1272996.1273005","volume-title":"EuroSys","author":"Isard M.","year":"2007","unstructured":"M. Isard : Distributed Data-Parallel Programs from Sequential Building Blocks . In EuroSys , pages 59 -- 72 , 2007 . 10.1145\/1272996.1273005 M. Isard et al. Dryad: Distributed Data-Parallel Programs from Sequential Building Blocks. In EuroSys, pages 59--72, 2007. 10.1145\/1272996.1273005"},{"issue":"6","key":"e_1_2_1_14_1","first-page":"385","article-title":"Automatic Optimization for MapReduce Programs","volume":"4","author":"Jahani E.","year":"2011","unstructured":"E. Jahani , M. J. Cafarella , and C. R\u00e9 . Automatic Optimization for MapReduce Programs . PVLDB , 4 ( 6 ): 385 -- 396 , 2011 . E. Jahani, M. J. Cafarella, and C. R\u00e9. Automatic Optimization for MapReduce Programs. PVLDB, 4(6):385--396, 2011.","journal-title":"PVLDB"},{"key":"e_1_2_1_15_1","volume-title":"The Performance of MapReduce: An In-depth Study. PVLDB, 3(1--2):472--483","author":"Jiang D.","year":"2010","unstructured":"D. Jiang The Performance of MapReduce: An In-depth Study. PVLDB, 3(1--2):472--483 , 2010 . D. Jiang et al. The Performance of MapReduce: An In-depth Study. PVLDB, 3(1--2):472--483, 2010."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2038916.2038937"},{"key":"e_1_2_1_17_1","volume-title":"Full-Text Indexing for Optimizing Selection Operations in Large-Scale Data Analytics. MapReduce Workshop, 2011","author":"Lin J.","year":"1996","unstructured":"J. Lin Full-Text Indexing for Optimizing Selection Operations in Large-Scale Data Analytics. MapReduce Workshop, 2011 . 10.1145\/ 1996 092.1996105 J. Lin et al. Full-Text Indexing for Optimizing Selection Operations in Large-Scale Data Analytics. MapReduce Workshop, 2011. 10.1145\/1996092.1996105"},{"key":"e_1_2_1_18_1","first-page":"961","volume-title":"SIGMOD","author":"Lin Y.","year":"2011","unstructured":"Y. Lin : Leveraging Columnar Storage for Scalable Join Processing in the MapReduce Framework . In SIGMOD , pages 961 -- 972 , 2011 . 10.1145\/1989323.1989424 Y. Lin et al. Llama: Leveraging Columnar Storage for Scalable Join Processing in the MapReduce Framework. In SIGMOD, pages 961--972, 2011. 10.1145\/1989323.1989424"},{"key":"e_1_2_1_19_1","first-page":"51","volume-title":"SoCC","author":"Logothetis D.","year":"2010","unstructured":"D. Logothetis Stateful Bulk Processing for Incremental Analytics . In SoCC , pages 51 -- 62 , 2010 . 10.1145\/1807128.1807138 D. Logothetis et al. Stateful Bulk Processing for Incremental Analytics. In SoCC, pages 51--62, 2010. 10.1145\/1807128.1807138"},{"key":"e_1_2_1_20_1","first-page":"949","volume-title":"SIGMOD","author":"Okcan A.","year":"2011","unstructured":"A. Okcan and M. Riedewald . Processing Theta-Joins Using MapReduce . In SIGMOD , pages 949 -- 960 , 2011 . 10.1145\/1989323.1989423 A. Okcan and M. Riedewald. Processing Theta-Joins Using MapReduce. In SIGMOD, pages 949--960, 2011. 10.1145\/1989323.1989423"},{"key":"e_1_2_1_21_1","first-page":"165","volume-title":"SIGMOD","author":"Pavlo A.","year":"2009","unstructured":"A. Pavlo A Comparison of Approaches to Large-Scale Data Analysis . In SIGMOD , pages 165 -- 178 , 2009 . 10.1145\/1559845.1559865 A. Pavlo et al. A Comparison of Approaches to Large-Scale Data Analysis. In SIGMOD, pages 165--178, 2009. 10.1145\/1559845.1559865"},{"key":"e_1_2_1_22_1","first-page":"589","volume-title":"ICDE","author":"Quian\u00e9-Ruiz J.-A.","year":"2011","unstructured":"J.-A. Quian\u00e9-Ruiz , C. Pinkel , J. Schad , and J. Dittrich . RAFTing MapReduce: Fast Recovery on the RAFT . ICDE , pages 589 -- 600 , 2011 . 10.1109\/ICDE.2011.5767877 J.-A. Quian\u00e9-Ruiz, C. Pinkel, J. Schad, and J. Dittrich. RAFTing MapReduce: Fast Recovery on the RAFT. ICDE, pages 589--600, 2011. 10.1109\/ICDE.2011.5767877"},{"key":"e_1_2_1_23_1","first-page":"1013","volume-title":"SIGMOD","author":"Thusoo A.","year":"2010","unstructured":"A. Thusoo Data Warehousing and Analytics Infrastructure at Facebook . In SIGMOD , pages 1013 -- 1020 , 2010 . 10.1145\/1807167.1807278 A. Thusoo et al. Data Warehousing and Analytics Infrastructure at Facebook. In SIGMOD, pages 1013--1020, 2010. 10.1145\/1807167.1807278"},{"key":"e_1_2_1_24_1","first-page":"996","volume-title":"ICDE","author":"Thusoo A.","year":"2010","unstructured":"A. Thusoo Hive -- A Petabyte Scale Data Warehouse Using Hadoop . In ICDE , pages 996 -- 1005 , 2010 . A. Thusoo et al. Hive -- A Petabyte Scale Data Warehouse Using Hadoop. In ICDE, pages 996--1005, 2010."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2038916.2038928"},{"key":"e_1_2_1_26_1","first-page":"29","volume-title":"OSDI","author":"Zaharia M.","year":"2008","unstructured":"M. Zaharia Improving MapReduce Performance in Heterogeneous Environments . In OSDI , pages 29 -- 42 , 2008 . M. Zaharia et al. Improving MapReduce Performance in Heterogeneous Environments. In OSDI, pages 29--42, 2008."}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/2367502.2367562","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,7,14]],"date-time":"2023-07-14T14:43:34Z","timestamp":1689345814000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/2367502.2367562"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2012,8]]},"references-count":26,"journal-issue":{"issue":"12","published-print":{"date-parts":[[2012,8]]}},"alternative-id":["10.14778\/2367502.2367562"],"URL":"https:\/\/doi.org\/10.14778\/2367502.2367562","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2012,8]]}}}