{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,24]],"date-time":"2026-03-24T02:51:20Z","timestamp":1774320680130,"version":"3.50.1"},"reference-count":31,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2023,12,8]],"date-time":"2023-12-08T00:00:00Z","timestamp":1701993600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Manag. Data"],"published-print":{"date-parts":[[2023,12,8]]},"abstract":"<jats:p>Enterprise data lakes often suffer from substantial amounts of duplicate and redundant data, with data volumes ranging from terabytes to petabytes. This leads to both increased storage costs and unnecessarily high maintenance costs for these datasets. In this work, we focus on identifying and reducing redundancy in enterprise data lakes by addressing the problem of \"dataset containment\". To the best of our knowledge, this is one of the first works that addresses table-level containment at a large scale.<\/jats:p>\n          <jats:p>We propose R2D2: a three-step hierarchical pipeline that efficiently identifies almost all instances of containment by progressively reducing the search space in the data lake. It first builds (i) a schema containment graph, followed by (ii) statistical min-max pruning, and finally, (iii) content level pruning. We further propose minimizing the total storage and access costs by optimally identifying redundant datasets that can be deleted (and reconstructed on demand) while respecting latency constraints.<\/jats:p>\n          <jats:p>We implement our system on Azure Databricks clusters using Apache Spark for enterprise data stored in ADLS Gen2, and on AWS clusters for open-source data. In contrast to existing modified baselines that are inaccurate or take several days to run, our pipeline can process an enterprise customer data lake at the TB scale in approximately 5 hours with high accuracy. We present theoretical results as well as extensive empirical validation on both enterprise (scale of TBs) and open-source datasets (scale of MBs - GBs), which showcase the effectiveness of our pipeline.<\/jats:p>","DOI":"10.1145\/3626762","type":"journal-article","created":{"date-parts":[[2023,12,12]],"date-time":"2023-12-12T14:01:21Z","timestamp":1702389681000},"page":"1-25","source":"Crossref","is-referenced-by-count":8,"title":["R2D2: Reducing Redundancy and Duplication in Data Lakes"],"prefix":"10.1145","volume":"1","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2889-7855","authenticated-orcid":false,"given":"Raunak","family":"Shah","sequence":"first","affiliation":[{"name":"University of Illinois, Urbana-Champaign, Champaign, IL, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8690-323X","authenticated-orcid":false,"given":"Koyel","family":"Mukherjee","sequence":"additional","affiliation":[{"name":"Adobe Research, Bangalore, India"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-3904-7953","authenticated-orcid":false,"given":"Atharv","family":"Tyagi","sequence":"additional","affiliation":[{"name":"Adobe Research, Bangalore, India"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1328-5167","authenticated-orcid":false,"given":"Sai Keerthana","family":"Karnam","sequence":"additional","affiliation":[{"name":"Adobe, Bangalore, India"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-4543-846X","authenticated-orcid":false,"given":"Dhruv","family":"Joshi","sequence":"additional","affiliation":[{"name":"Indian Institute of Technology Kharagpur, Kharagpur, India"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5817-5271","authenticated-orcid":false,"given":"Shivam Pravin","family":"Bhosale","sequence":"additional","affiliation":[{"name":"Adobe, Bangalore, India"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-8436-3119","authenticated-orcid":false,"given":"Subrata","family":"Mitra","sequence":"additional","affiliation":[{"name":"Adobe Research, Bangalore, India"}]}],"member":"320","published-online":{"date-parts":[[2023,12,12]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"The Costs of an Unnecessarily Stringent Federal Data Privacy Law. https:\/\/itif.org\/publications\/2019\/08\/05\/costs-unnecessarily-stringent-federal-data-privacy-law\/. [Online","author":"Alan McQuinn Daniel Castro","year":"2023","unstructured":"Daniel Castro Alan McQuinn. 2019. The Costs of an Unnecessarily Stringent Federal Data Privacy Law. https:\/\/itif.org\/publications\/2019\/08\/05\/costs-unnecessarily-stringent-federal-data-privacy-law\/. [Online; accessed 7-July-2023]."},{"key":"e_1_2_1_2_1","volume-title":"https:\/\/spark.apache.org\/. [Online","author":"Spark Apache","year":"2023","unstructured":"Apache Spark. 2023. Apache Spark. https:\/\/spark.apache.org\/. [Online; accessed 13-Sep-2023]."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.14778\/3457390.3457403"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.14778\/2824032.2824035"},{"key":"e_1_2_1_5_1","volume-title":"Dataset Discovery in Data Lakes. CoRR","author":"Bogatu Alex","year":"2020","unstructured":"Alex Bogatu, Alvaro A. A. Fernandes, Norman W. Paton, and Nikolaos Konstantinou. 2020. Dataset Discovery in Data Lakes. CoRR, Vol. abs\/2011.10427 (2020). [arXiv]2011.10427 https:\/\/arxiv.org\/abs\/2011.10427"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3035918.3058740"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.14778\/3115404.3115413"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3183713.3183748"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2019.00040"},{"key":"e_1_2_1_10_1","volume-title":"Efficient Joinable Table Discovery in Data Lakes: A High-Dimensional Similarity-Based Approach. CoRR","author":"Dong Yuyang","year":"2020","unstructured":"Yuyang Dong, Kunihiro Takeoka, Chuan Xiao, and Masafumi Oyamada. 2020. Efficient Joinable Table Discovery in Data Lakes: A High-Dimensional Similarity-Based Approach. CoRR, Vol. abs\/2010.13273 (2020). [arXiv]2010.13273 https:\/\/arxiv.org\/abs\/2010.13273"},{"key":"e_1_2_1_11_1","volume-title":"Proceedings of the 2012 USENIX Conference on Annual Technical Conference (Boston, MA) (USENIX ATC'12). USENIX Association, USA, 26","author":"El-Shimi Ahmed","year":"2012","unstructured":"Ahmed El-Shimi, Ran Kalach, Ankit Kumar, Adi Oltean, Jin Li, and Sudipta Sengupta. 2012. Primary Data Deduplication-Large Scale Study and System Design. In Proceedings of the 2012 USENIX Conference on Annual Technical Conference (Boston, MA) (USENIX ATC'12). USENIX Association, USA, 26."},{"key":"e_1_2_1_12_1","unstructured":"Paul ErdHo s Alfr\u00e9d R\u00e9nyi et al. 1960. On the evolution of random graphs. Publ. math. inst. hung. acad. sci Vol. 5 1 (1960) 17--60."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.14778\/3587136.3587146"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3448016.3457548"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","unstructured":"Javier Flores Sergi Nadal and Oscar Romero. 2020. Scalable Data Discovery Using Profiles. https:\/\/doi.org\/10.48550\/ARXIV.2012.00890","DOI":"10.48550\/ARXIV.2012.00890"},{"key":"e_1_2_1_16_1","volume-title":"The GDPR effect: How data privacy regulation shaped firm performance globally. https:\/\/cepr.org\/voxeu\/columns\/gdpr-effect-how-data-privacy-regulation-shaped-firm-performance-globally. [Online","author":"Carl Benedikt Frey Giorgio Presidente","year":"2023","unstructured":"Giorgio Presidente Carl Benedikt Frey. 2022. The GDPR effect: How data privacy regulation shaped firm performance globally. https:\/\/cepr.org\/voxeu\/columns\/gdpr-effect-how-data-privacy-regulation-shaped-firm-performance-globally. [Online; accessed 7-July-2023]."},{"key":"e_1_2_1_17_1","volume-title":"Do not let ROT Impact your GDPR Compliance and Increase your IT Costs exponentially. https:\/\/www.linkedin.com\/pulse\/do-let-rot-affect-your-gdpr-compliance-increase-costs-robert-healey\/. [Online","author":"Healey Robert","year":"2023","unstructured":"Robert Healey. 2017. Do not let ROT Impact your GDPR Compliance and Increase your IT Costs exponentially. https:\/\/www.linkedin.com\/pulse\/do-let-rot-affect-your-gdpr-compliance-increase-costs-robert-healey\/. [Online; accessed 7-July-2023]."},{"key":"e_1_2_1_18_1","volume-title":"How Much Does GDPR Compliance Cost in 2023? https:\/\/www.itgovernance.eu\/blog\/en\/how-much-does-gdpr-compliance-cost-in-2020. [Online","author":"Irwin Luke","year":"2023","unstructured":"Luke Irwin. 2023. How Much Does GDPR Compliance Cost in 2023? https:\/\/www.itgovernance.eu\/blog\/en\/how-much-does-gdpr-compliance-cost-in-2020. [Online; accessed 7-July-2023]."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3588689"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.14778\/3421424.3421431"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2021.3091101"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.14778\/3192965.3192973"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.14778\/3476311.3476364"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/CloudCom.2013.54"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3017428"},{"key":"e_1_2_1_26_1","unstructured":"Statista. 2023. Storage worldwide- Statista market forecast. (2023). https:\/\/www.statista.com\/outlook\/tmo\/data-center\/storage\/worldwide"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3459637.3482008"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3035918.3058744"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3318464.3389726"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3299869.3300065"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.14778\/2994509.2994534"}],"container-title":["Proceedings of the ACM on Management of Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3626762","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3626762","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,22]],"date-time":"2025-08-22T13:01:39Z","timestamp":1755867699000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3626762"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,12,8]]},"references-count":31,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2023,12,8]]}},"alternative-id":["10.1145\/3626762"],"URL":"https:\/\/doi.org\/10.1145\/3626762","relation":{},"ISSN":["2836-6573"],"issn-type":[{"value":"2836-6573","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,12,8]]}}}