{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,11]],"date-time":"2026-07-11T10:36:03Z","timestamp":1783766163547,"version":"3.55.0"},"reference-count":188,"publisher":"Association for Computing Machinery (ACM)","issue":"3","funder":[{"DOI":"10.13039\/501100012166","name":"National Key R&D Program of China","doi-asserted-by":"crossref","award":["2022YFB4500304"],"award-info":[{"award-number":["2022YFB4500304"]}],"id":[{"id":"10.13039\/501100012166","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62332021, 62461146204"],"award-info":[{"award-number":["62332021, 62461146204"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Comput. Surv."],"published-print":{"date-parts":[[2026,2,28]]},"abstract":"<jats:p>To address the throughput and capacity limitations of single-node centralized deduplication, distributed data deduplication has become a popular technology in big data management to save more storage space, enhance I\/O performance, and improve system scalability. It includes inter-node data assignment from clients to multiple deduplication nodes by a data routing scheme, and independent intra-node redundancy suppression in individual nodes. In this article, we first describe the background of big data deduplication. Then we summarize and classify the state-of-the-art in the key techniques of distributed data deduplication, including data partitioning, chunk fingerprinting, data routing, index lookup, data restoring, garbage collection, the security and reliability of deduplicated data. These help identify and understand the system implementation of the existing distributed data deduplication methods. Moreover, we present some representative industrial products that have successfully applied distributed data deduplication technologies. Finally, we discuss the main challenges and industry trend of distributed data deduplication, and outline the open problems and its future research directions.<\/jats:p>","DOI":"10.1145\/3735508","type":"journal-article","created":{"date-parts":[[2025,5,10]],"date-time":"2025-05-10T04:38:44Z","timestamp":1746851924000},"page":"1-38","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["Distributed Data Deduplication for Big Data: A Survey"],"prefix":"10.1145","volume":"58","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9107-1338","authenticated-orcid":false,"given":"Yinjin","family":"Fu","sequence":"first","affiliation":[{"name":"School of Cyber Science and Technology, Sun Yat-Sen University","place":["Shenzhen, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-7401-7233","authenticated-orcid":false,"given":"Jun","family":"Su","sequence":"additional","affiliation":[{"name":"School of Computer Science and Engineering, Sun Yat-Sen University","place":["Guangzhou, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-4981-7317","authenticated-orcid":false,"given":"Jiahao","family":"Ning","sequence":"additional","affiliation":[{"name":"School of Cyber Science and Technology, Sun Yat-Sen University","place":["Shenzhen, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-1229-4546","authenticated-orcid":false,"given":"Jian","family":"Wu","sequence":"additional","affiliation":[{"name":"School of Cyber Science and Technology, Sun Yat-Sen University","place":["Shenzhen, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5315-3375","authenticated-orcid":false,"given":"Yutong","family":"Lu","sequence":"additional","affiliation":[{"name":"School of Computer Science and Engineering, Sun Yat-Sen University","place":["Guangzhou, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2166-977X","authenticated-orcid":false,"given":"Nong","family":"Xiao","sequence":"additional","affiliation":[{"name":"School of Computer Science and Engineering, Sun Yat-Sen University","place":["Guangzhou, China"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,9,9]]},"reference":[{"key":"e_1_3_1_2_2","unstructured":"2003. EMC Centera Content Addressed Storage System. Retrieved April 7 2025 from https:\/\/m.softchoice.com\/files\/pdf\/brands\/emc\/Centera_Datasheet.pdf"},{"key":"e_1_3_1_3_2","unstructured":"2010. EMC Data Domain Global Deduplication Array. Retrieved April 7 2025 from https:\/\/www.spectra.com\/wp-content\/uploads\/emc-global-dd-array.pdf"},{"key":"e_1_3_1_4_2","unstructured":"2010. SEPATON S2100-ES2 Multifunction Data Protection Appliance. Retrieved April 7 2025 from https:\/\/storagemojo.com\/wp-content\/uploads\/2011\/01\/S2100-ES2-Series-1900-2900-V8.pdf"},{"key":"e_1_3_1_5_2","unstructured":"2016. IBM ProtecTIER Deduplication. Retrieved April 7 2025 from https:\/\/ftpmirror.your.org\/pub\/misc\/ftp.software.ibm.com\/common\/ssi\/ecm\/ts\/en\/tsb03014usen\/systems-system-storage-ts-brochure-tsb03014usen-20170803.pdf"},{"key":"e_1_3_1_6_2","unstructured":"2016. NEC HYDRAstor. Retrieved April 7 2025 from https:\/\/www.necam.com\/docs\/?id=a4864b88-510d-4bba-83a9-722665d99b46"},{"key":"e_1_3_1_7_2","unstructured":"2019. Virtual Tape Library (VTL) Cluster\/Single Instance Repository Cluster Deduplication Gateway. Retrieved April 7 2025 from https:\/\/www.falconstor.com\/wp-content\/uploads\/2020\/06\/VTL-SIR_CLUSTER_10_GW_SPECS.pdf"},{"key":"e_1_3_1_8_2","unstructured":"2023. Commvault Distributed Storage Technical and Architectural Overview Whitepaper. Retrieved April 2 2023 from https:\/\/www.commvault.com\/resources\/commvault-distributed-storage-platform-technical-and-architectural-\/-overview-whitepaper"},{"key":"e_1_3_1_9_2","unstructured":"2024. Enterprise Network Simulation Platform. Retrieved March 10 2024 from https:\/\/support.huawei.com\/enterprise\/en\/nce-data-communication\/ensp-pid-9017384\/software"},{"key":"e_1_3_1_10_2","unstructured":"2024. FSL Home Traces. Retrieved April 7 2025 from http:\/\/tracer.filesystems.org\/traces\/fslhomes\/"},{"key":"e_1_3_1_11_2","unstructured":"2024. GCC. Retrieved April 7 2025 from http:\/\/ftp.tsukuba.wide.ad.jp\/software\/gcc\/releases"},{"key":"e_1_3_1_12_2","unstructured":"2024. Grid\u20195000. Retrieved April 7 2025 from https:\/\/www.grid5000.fr\/"},{"key":"e_1_3_1_13_2","unstructured":"2024. Java Archive. Retrieved April 7 2025 from https:\/\/www.oracle.com\/java\/technologies\/downloads\/archive\/"},{"key":"e_1_3_1_14_2","unstructured":"2024. The Linux Kernel Archives. Retrieved April 7 2025 from http:\/\/www.kernel.org\/"},{"key":"e_1_3_1_15_2","unstructured":"2024. MacOS Traces. Retrieved April 7 2025 from http:\/\/tracer.filesystems.org\/traces\/macos\/"},{"key":"e_1_3_1_16_2","unstructured":"2024. MySQL Product Archives. Retrieved April 7 2025 from https:\/\/downloads.mysql.com\/archives\/community\/"},{"key":"e_1_3_1_17_2","unstructured":"2024. OceanProtect Backup Storage. Retrieved April 7 2025 from https:\/\/e.huawei.com\/en\/products\/storage\/oceanprotect\/x-series"},{"key":"e_1_3_1_18_2","unstructured":"2024. OceanStor Dorado 6.1.3 Technical White Paper. Retrieved April 7 2025 from https:\/\/support.huawei.com\/enterprise\/tr\/doc\/EDOC1100253812\/d5c47b60"},{"key":"e_1_3_1_19_2","unstructured":"2024. OMNeT++ Discrete Event Simulator. Retrieved April 7 2025 from https:\/\/omnetpp.org\/"},{"key":"e_1_3_1_20_2","unstructured":"2024. Open Access VM Images. Retrieved April 7 2025 from http:\/\/mirrors.neusoft.edu.cn"},{"key":"e_1_3_1_21_2","unstructured":"2024. OPNET Network Simulator. Retrieved April 7 2025 from https:\/\/opnetprojects.com\/opnet-network-simulator\/"},{"key":"e_1_3_1_22_2","unstructured":"2024. The Wikimedia Dump Service. Retrieved April 7 2025 from https:\/\/dumps.wikimedia.org\/enwiki\/"},{"key":"e_1_3_1_23_2","unstructured":"2025. Dell PowerScale OneFS Operating System. Retrieved April 20 2025 from https:\/\/www.delltechnologies.com\/asset\/en-us\/products\/storage\/industry-market\/h10719-wp-powerscale-onefs-technical-overview.pdf"},{"key":"e_1_3_1_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/SAI.2016.7556026"},{"key":"e_1_3_1_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/1383422.138344"},{"key":"e_1_3_1_26_2","first-page":"179","volume-title":"Proceedings of the 22nd USENIX Security Symposium (USENIX Security\u201913)","author":"Bellare Mihir","year":"2013","unstructured":"Mihir Bellare, Sriram Keelveedhi, and Thomas Ristenpart. 2013. DupLESS: Server-aided encryption for deduplicated storage. In Proceedings of the 22nd USENIX Security Symposium (USENIX Security\u201913). 179\u2013194."},{"key":"e_1_3_1_27_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-38348-9_18"},{"key":"e_1_3_1_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/RISP.1992.213269"},{"key":"e_1_3_1_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/MASCOT.2009.5366623"},{"key":"e_1_3_1_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/1281192.1281207"},{"key":"e_1_3_1_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/MASCOTS.2006.42"},{"key":"e_1_3_1_32_2","doi-asserted-by":"publisher","DOI":"10.1145\/362686.362692"},{"key":"e_1_3_1_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/1210596.1210599"},{"key":"e_1_3_1_34_2","first-page":"1","volume-title":"Proceedings of the 4th Conference on USENIX Windows Systems Symposium (WSS\u201900)","author":"Bolosky William J.","year":"2000","unstructured":"William J. Bolosky, Scott Corbin, David Goebel, and John R. Douceur. 2000. Single instance storage in windows\u00ae 2000. In Proceedings of the 4th Conference on USENIX Windows Systems Symposium (WSS\u201900). USENIX Association, 1\u201312."},{"key":"e_1_3_1_35_2","doi-asserted-by":"publisher","DOI":"10.5555\/2591272.2591282"},{"key":"e_1_3_1_36_2","first-page":"21","volume-title":"Proceedings of the Compression and Complexity of SEQUENCES","author":"Broder Andrei Z.","year":"1997","unstructured":"Andrei Z. Broder. 1997. On the resemblance and containment of documents. In Proceedings of the Compression and Complexity of SEQUENCES. 21\u201329."},{"key":"e_1_3_1_37_2","doi-asserted-by":"publisher","DOI":"10.5555\/3323298.3323311"},{"key":"e_1_3_1_38_2","first-page":"309","volume-title":"Proceedings of the16th USENIX Conference on File and Storage Technologies (FAST\u201918)","author":"Cao Zhichao","year":"2018","unstructured":"Zhichao Cao, Hao Wen, Fenggang Wu, and David HC Du. 2018. ALACC: Accelerating restore performance of data deduplication systems using adaptive look-ahead window assisted chunk caching. In Proceedings of the16th USENIX Conference on File and Storage Technologies (FAST\u201918). 309\u2013324."},{"key":"e_1_3_1_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCD58817.2023.00050"},{"key":"e_1_3_1_40_2","first-page":"1","volume-title":"Proceedings of the 9th USENIX Conference on File and Stroage Technologies (FAST\u201911)","author":"Chen Feng","year":"2011","unstructured":"Feng Chen, Tian Luo, and Xiaodong Zhang. 2011. CAFTL: A content-aware flash translation layer enhancing the lifespan of flash memory based solid state drives. In Proceedings of the 9th USENIX Conference on File and Stroage Technologies (FAST\u201911). 1\u201314."},{"key":"e_1_3_1_41_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIFS.2015.2470221"},{"key":"e_1_3_1_42_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISCC.2015.7405578"},{"key":"e_1_3_1_43_2","first-page":"291","volume-title":"Proceedings of the14th USENIX Conference on File and Storage Technologies (FAST\u201916)","author":"Chen Zhuan","year":"2016","unstructured":"Zhuan Chen and Kai Shen. 2016. OrderMergeDedup: Efficient, failure-consistent deduplication on flash. In Proceedings of the14th USENIX Conference on File and Storage Technologies (FAST\u201916). 291\u2013299."},{"key":"e_1_3_1_44_2","unstructured":"Gao Cong Wenfei Fan Floris Geerts Xibei Jia and Shuai Ma. 2007. Improving data quality: Consistency and accuracy. In Proceedings of the 33rd International Conference on Very Large Data Bases(VLDB\u201907). VLDB Endowment 315\u2013326. Retrieved from https:\/\/www.research.ed.ac.uk\/en\/publications\/improving-data-quality-consistency-and-accuracy"},{"key":"e_1_3_1_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/844128.844155"},{"key":"e_1_3_1_46_2","doi-asserted-by":"publisher","unstructured":"Girum Dagnaw Wang Hua and Ke Zhou. 2020. SSD assisted caching for restore optimization in distributed deduplication environment. In The 2nd International Conference on High Performance Big Data and Intelligent Systems (HPBD&IS 2020). IEEE 1\u20138. DOI:10.1109\/HPBDIS49115.2020.9130572","DOI":"10.1109\/HPBDIS49115.2020.9130572"},{"key":"e_1_3_1_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/3448016.3457273"},{"key":"e_1_3_1_48_2","doi-asserted-by":"publisher","DOI":"10.1109\/18.771255"},{"key":"e_1_3_1_49_2","volume-title":"Proceedings of the USENIX Conference on USENIX Annual Technical Conference.","author":"Debnath Biplob K.","year":"2010","unstructured":"Biplob K. Debnath, Sudipta Sengupta, and Jin Li. 2010. ChunkStash: Speeding up inline storage deduplication using flash memory. In Proceedings of the USENIX Conference on USENIX Annual Technical Conference.USENIX Association."},{"key":"e_1_3_1_50_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10766-016-0450-5"},{"key":"e_1_3_1_51_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-05051-1_25"},{"key":"e_1_3_1_52_2","first-page":"1","volume-title":"Proceedings of the 9th USENIX Conference on File and Stroage Technologies (FAST\u201911)","author":"Dong Wei","year":"2011","unstructured":"Wei Dong, Fred Douglis, Kai Li, Hugo Patterson, Sazzala Reddy, and Philip Shilane. 2011. Tradeoffs in scalable data routing for deduplication clusters. In Proceedings of the 9th USENIX Conference on File and Stroage Technologies (FAST\u201911). USENIX Association, USA, 1\u201315."},{"key":"e_1_3_1_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/1368436.1368454"},{"key":"e_1_3_1_54_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDCS.2002.1022312"},{"key":"e_1_3_1_55_2","first-page":"1","volume-title":"Proceedings of the 25th Large Installation System Administration Conference.","author":"Douglis Fred","year":"2011","unstructured":"Fred Douglis, Deepti Bhardwaj, Hangwei Qian, and Philip Shilane. 2011. Content-aware load balancing for distributed backup. In Proceedings of the 25th Large Installation System Administration Conference.1\u201318."},{"key":"e_1_3_1_56_2","doi-asserted-by":"publisher","DOI":"10.5555\/3129633.3129637"},{"key":"e_1_3_1_57_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICPP.2014.20"},{"key":"e_1_3_1_58_2","first-page":"197","volume-title":"Proccedings of the 7th Conference on File and Storage Technologies (FAST\u201909)","author":"Dubnicki Cezary","year":"2009","unstructured":"Cezary Dubnicki, Leszek Gryz, Lukasz Heldt, Michal Kaczmarczyk, Wojciech Kilian, Przemyslaw Strzelczak, Jerzy Szczepkowski, Cristian Ungureanu, and Michal Welnicki. 2009. HYDRAstor: A scalable secondary storage. In Proccedings of the 7th Conference on File and Storage Technologies (FAST\u201909). 197\u2013210."},{"key":"e_1_3_1_59_2","doi-asserted-by":"publisher","DOI":"10.13140\/RG.2.2.11032.71687"},{"key":"e_1_3_1_60_2","doi-asserted-by":"publisher","DOI":"10.1145\/2168697.2168707"},{"key":"e_1_3_1_61_2","first-page":"285","volume-title":"Proceedings of the USENIX Annual Technical Conference.","author":"El-Shimi Ahmed","year":"2012","unstructured":"Ahmed El-Shimi, Ran Kalach, Ankit Kumar, Adi Ottean, Jin Li, and Sudipta Sengupta. 2012. Primary data deduplication-large scale study and system design. In Proceedings of the USENIX Annual Technical Conference.285\u2013296."},{"key":"e_1_3_1_62_2","volume-title":"Proceedings of the HPL-2005-30(R.1)","author":"Eshghi Kave","year":"2005","unstructured":"Kave Eshghi and Hsiu Khuern Tang. 2005. A framework for analyzing and improving content-based chunking algorithms. In Proceedings of the HPL-2005-30(R.1). HP Laboratories Palo Alto. Retrieved from www.hpl.hp.com\/techreports\/2005\/HPL-2005-30R1.pdf"},{"key":"e_1_3_1_63_2","doi-asserted-by":"publisher","unstructured":"Bin Fan Dave G. Andersen Michael Kaminsky and Michael D. Mitzenmacher. 2014. Cuckoo filter: Practically better than bloom. ACM International Conference on emerging Networking EXperiments and Technologies Sydney 75\u201388. DOI:10.1145\/2674005.2674994","DOI":"10.1145\/2674005.2674994"},{"key":"e_1_3_1_64_2","doi-asserted-by":"publisher","DOI":"10.1109\/90.851975"},{"key":"e_1_3_1_65_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-981-19-0112-6_1"},{"key":"e_1_3_1_66_2","doi-asserted-by":"publisher","DOI":"10.1145\/2391229.2391246"},{"key":"e_1_3_1_67_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2015.2410781"},{"key":"e_1_3_1_68_2","first-page":"181","volume-title":"Proceedings of the 2014 USENIX Annual Technical Conference (USENIX ATC\u201914)","author":"Fu Min","year":"2014","unstructured":"Min Fu, Dan Feng, Yu Hua, Xubin He, Zuoning Chen, Wen Xia, Fangting Huang, and Qing Liu. 2014. Accelerating restore and garbage collection in deduplication-based backup systems via exploiting historical information. In Proceedings of the 2014 USENIX Annual Technical Conference (USENIX ATC\u201914). 181\u2013192."},{"key":"e_1_3_1_69_2","doi-asserted-by":"publisher","DOI":"10.1109\/TC.2018.2808496"},{"key":"e_1_3_1_70_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-35170-9_18"},{"key":"e_1_3_1_71_2","doi-asserted-by":"publisher","DOI":"10.1109\/CLUSTER.2011.20"},{"key":"e_1_3_1_72_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2021.3123539"},{"key":"e_1_3_1_73_2","doi-asserted-by":"publisher","DOI":"10.1109\/TDSC.2021.3086089"},{"key":"e_1_3_1_74_2","doi-asserted-by":"publisher","DOI":"10.1109\/TCC.2017.2710043"},{"key":"e_1_3_1_75_2","doi-asserted-by":"publisher","DOI":"10.1109\/MM.2023.3237491"},{"key":"e_1_3_1_76_2","doi-asserted-by":"publisher","DOI":"10.1109\/NAS.2017.8026846"},{"key":"e_1_3_1_77_2","first-page":"1","volume-title":"Proceedings of the USENIX Annual Technical Conference","author":"Guo Fanglu","year":"2011","unstructured":"Fanglu Guo and Petros Efstathopoulos. 2011. Building a high-performance deduplication system. In Proceedings of the USENIX Annual Technical Conference. Portland, OR, USA, 1\u201314."},{"key":"e_1_3_1_78_2","doi-asserted-by":"publisher","DOI":"10.1145\/2046707.2046765"},{"key":"e_1_3_1_79_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2010.187"},{"key":"e_1_3_1_80_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2021.3139757"},{"key":"e_1_3_1_81_2","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPS.2004.1303042"},{"key":"e_1_3_1_82_2","doi-asserted-by":"publisher","DOI":"10.1109\/GCRAIT55928.2022.00136"},{"key":"e_1_3_1_83_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSC.2024.3350086"},{"key":"e_1_3_1_84_2","doi-asserted-by":"publisher","DOI":"10.1145\/3620665.3640353"},{"key":"e_1_3_1_85_2","doi-asserted-by":"publisher","DOI":"10.1145\/3605181.3626285"},{"key":"e_1_3_1_86_2","doi-asserted-by":"publisher","DOI":"10.1109\/BigData.2016.7840951"},{"key":"e_1_3_1_87_2","doi-asserted-by":"publisher","DOI":"10.1145\/279310.279321"},{"key":"e_1_3_1_88_2","doi-asserted-by":"publisher","DOI":"10.1145\/2090181.2090187"},{"key":"e_1_3_1_89_2","doi-asserted-by":"publisher","DOI":"10.1145\/2367589.2367600"},{"key":"e_1_3_1_90_2","first-page":"1","volume-title":"Proceedings of the 3rd USENIX Workshop on Hot Topics in Storage and File Systems.","author":"Katiyar Atul","year":"2011","unstructured":"Atul Katiyar and Jon B. Weissman. 2011. ViDeDup: An application-aware framework for video de-duplication. In Proceedings of the 3rd USENIX Workshop on Hot Topics in Storage and File Systems.Irfan Ahmad (Ed.), USENIX Association, 1\u20135."},{"key":"e_1_3_1_91_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11227-017-2210-8"},{"key":"e_1_3_1_92_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2020.3039056"},{"key":"e_1_3_1_93_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-42280-0_2"},{"key":"e_1_3_1_94_2","doi-asserted-by":"publisher","DOI":"10.1145\/3565025"},{"key":"e_1_3_1_95_2","first-page":"33","volume-title":"Proceedings of the 21st USENIX Conference on File and Storage Technologies (FAST\u201923)","author":"Kotlarska Iwona","year":"2023","unstructured":"Iwona Kotlarska, Andrzej Jackowski, Krzysztof Lichota, Michal Welnicki, Cezary Dubnicki, and Konrad Iwanicki. 2023. \\(\\lbrace\\) InftyDedup \\(\\rbrace\\) : Scalable and \\(\\lbrace\\) Cost-Effective \\(\\rbrace\\) cloud tiering with deduplication. In Proceedings of the 21st USENIX Conference on File and Storage Technologies (FAST\u201923). 33\u201348."},{"key":"e_1_3_1_96_2","first-page":"239","volume-title":"Proceedings of the 8th USENIX Conference on File and Storage Technologies.","author":"Kruus Erik","year":"2010","unstructured":"Erik Kruus, Cristian Ungureanu, and Cezary Dubnicki. 2010. Bimodal content defined chunking for backup streams. In Proceedings of the 8th USENIX Conference on File and Storage Technologies.USENIX, 239\u2013252."},{"key":"e_1_3_1_97_2","doi-asserted-by":"publisher","unstructured":"John Kubiatowicz David Bindel Yan Chen Steven Czerwinski Patrick Eaton Dennis Geels Ramakrishna Gummadi Sean Rhea Hakim Weatherspoon Westley Weimer Chris Wells and Ben Zhao. 2000. OceanStore: An architecture for global-scale persistent storage. In Proceedings of the Ninth International Conference on Architectural Support for Programming Languages and Operating Systems. Cambridge Massachusetts USA 190\u2013201. DOI:10.1145\/356989.357007","DOI":"10.1145\/356989.357007"},{"key":"e_1_3_1_98_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.comnet.2010.12.025"},{"key":"e_1_3_1_99_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2107.06499"},{"key":"e_1_3_1_100_2","doi-asserted-by":"publisher","DOI":"10.5555\/3650697.3650712"},{"key":"e_1_3_1_101_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICNP.2011.6089061"},{"key":"e_1_3_1_102_2","doi-asserted-by":"publisher","DOI":"10.1109\/FCCM.2015.43"},{"key":"e_1_3_1_103_2","doi-asserted-by":"publisher","DOI":"10.1109\/TC.2015.2401017"},{"key":"e_1_3_1_104_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2013.284"},{"key":"e_1_3_1_105_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICPADS56603.2022.00096"},{"key":"e_1_3_1_106_2","first-page":"111","volume-title":"Proceedings of the 2015 USENIX Annual Technical Conference (USENIX ATC\u201915)","author":"Li Mingqiang","year":"2015","unstructured":"Mingqiang Li, Chuan Qin, and Patrick PC Lee. 2015. CDStore: Toward reliable, secure, and cost-efficient cloud storage via convergent dispersal. In Proceedings of the 2015 USENIX Annual Technical Conference (USENIX ATC\u201915). 111\u2013124."},{"key":"e_1_3_1_107_2","doi-asserted-by":"publisher","DOI":"10.22214\/ijraset.2020.4087"},{"key":"e_1_3_1_108_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDCS.2019.00102"},{"key":"e_1_3_1_109_2","unstructured":"Xianming Li and Jing Li. 2024. Generative deduplication for socia media data selection. arXiv:2401.05883. Retrieved from https:\/\/arxiv.org\/abs\/2401.05883"},{"key":"e_1_3_1_110_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSESS.2016.7883027"},{"key":"e_1_3_1_111_2","unstructured":"Mark Lillibridge Kave Eshghi and Deepavali Bhagwat. 2013. Improving restore speed for backup systems that use inline chunk-based deduplication. In Proceedings of the 11th USENIX Conference on File and Storage Technologies(FAST\u201913). USENIX Association USA 183\u2013198."},{"key":"e_1_3_1_112_2","first-page":"111","volume-title":"Proccedings of the 7th Conference on File and Storage Technologies (FAST\u201909)","author":"Lillibridge Mark","year":"2009","unstructured":"Mark Lillibridge, Kave Eshghi, Deepavali Bhagwat, Vinay Deolalikar, Greg Trezise, and Peter Camble. 2009. Sparse indexing: Large scale, inline deduplication using sampling and locality. In Proccedings of the 7th Conference on File and Storage Technologies (FAST\u201909). USENIX Association, USA, 111\u2013123."},{"key":"e_1_3_1_113_2","doi-asserted-by":"publisher","DOI":"10.1145\/236711.236713"},{"key":"e_1_3_1_114_2","doi-asserted-by":"publisher","DOI":"10.1145\/1542275.1542327"},{"key":"e_1_3_1_115_2","doi-asserted-by":"publisher","DOI":"10.1109\/SNAPI.2008.11"},{"key":"e_1_3_1_116_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICPADS.2009.103"},{"key":"e_1_3_1_117_2","doi-asserted-by":"publisher","DOI":"10.1145\/2810103.2813623"},{"key":"e_1_3_1_118_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2015.2509060"},{"key":"e_1_3_1_119_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSN53354.2021.00118"},{"key":"e_1_3_1_120_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSN53354.2021.00118"},{"key":"e_1_3_1_121_2","doi-asserted-by":"publisher","DOI":"10.1109\/MASCOTS.2010.37"},{"key":"e_1_3_1_122_2","unstructured":"Cheng Luo Tianle Zhong and Geoffrey Fox. 2023. RTP: Rethinking tensor parallelism with memory deduplication. arXiv:2311.01635. Retrieved from https:\/\/arxiv.org\/abs\/2311.01635"},{"key":"e_1_3_1_123_2","doi-asserted-by":"publisher","DOI":"10.1109\/TCC.2015.2511752"},{"key":"e_1_3_1_124_2","doi-asserted-by":"publisher","DOI":"10.1002\/qub2.99"},{"key":"e_1_3_1_125_2","doi-asserted-by":"publisher","DOI":"10.1145\/2512348"},{"key":"e_1_3_1_126_2","doi-asserted-by":"publisher","DOI":"10.1145\/1534530.1534541"},{"key":"e_1_3_1_127_2","doi-asserted-by":"publisher","DOI":"10.1109\/SC.2012.14"},{"key":"e_1_3_1_128_2","doi-asserted-by":"publisher","DOI":"10.5555\/1960475.1960476"},{"key":"e_1_3_1_129_2","unstructured":"Min Fu Dan Feng Yu Hua Xubin He Zuoning Chen Wen Xia Fangting Huang and Qing Liu. 2014. Accelerating restore and garbage collection in deduplication-based backup systems via exploiting historical information. In 2014 USENIX Annual Technical Conference (ATC) Philadelphia PA USA 181\u2013192."},{"key":"e_1_3_1_130_2","doi-asserted-by":"publisher","DOI":"10.1145\/3578555"},{"key":"e_1_3_1_131_2","first-page":"1","volume-title":"Proceedings of the 20th USENIX Conference on Security","author":"Mulazzani Martin","year":"2011","unstructured":"Martin Mulazzani, Sebastian Schrittwieser, Manuel Leithner, Markus Huber, and Edgar Weippl. 2011. Dark clouds on the horizon: Using cloud storage as attack vector and online slack space. In Proceedings of the 20th USENIX Conference on Security. USENIX, San Francisco, CA, 1\u201311."},{"key":"e_1_3_1_132_2","doi-asserted-by":"publisher","DOI":"10.1145\/502034.502052"},{"key":"e_1_3_1_133_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-45065-5_11"},{"key":"e_1_3_1_134_2","doi-asserted-by":"publisher","DOI":"10.1109\/MASCOTS.2012.32"},{"key":"e_1_3_1_135_2","doi-asserted-by":"publisher","DOI":"10.1145\/3357223.3362731"},{"key":"e_1_3_1_136_2","first-page":"117","volume-title":"Proceedings of the 2023 USENIX Annual Technical Conference (USENIX ATC\u201923)","author":"Oh Myoungwon","year":"2023","unstructured":"Myoungwon Oh, Sungmin Lee, Samuel Just, Young Jin Yu, Duck-Ho Bae, Sage Weil, Sangyeun Cho, and Heon Y Yeom. 2023. \\(\\lbrace\\) TiDedup \\(\\rbrace\\) : A new distributed deduplication architecture for ceph. In Proceedings of the 2023 USENIX Annual Technical Conference (USENIX ATC\u201923). 117\u2013131."},{"key":"e_1_3_1_137_2","first-page":"247","volume-title":"Proceedings of the 20th USENIX Conference on File and Storage Technologies","author":"Park Jisung","year":"2022","unstructured":"Jisung Park, Jeonggyun Kim, Yeseong Kim, Sungjin Lee, and Onur Mutlu. 2022. DeepSketch: A new machine learning-based reference search technique for post-deduplication delta compression. In Proceedings of the 20th USENIX Conference on File and Storage Technologies. USENIX, Santa Clara, CA, 247\u2013264."},{"key":"e_1_3_1_138_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10619-020-07301-2"},{"key":"e_1_3_1_139_2","doi-asserted-by":"publisher","unstructured":"Salvatore Pontarelli Pedro Reviriego and Juan Antonio Maestro. 2016. Improving counting Bloom filter performance with fingerprints. Information Processing Letters 116 4 (2016) 304\u2013309. DOI:10.1016\/j.ipl.2015.11.002","DOI":"10.1016\/j.ipl.2015.11.002"},{"key":"e_1_3_1_140_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jksuci.2020.10.021"},{"key":"e_1_3_1_141_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCD50377.2020.00038"},{"key":"e_1_3_1_142_2","volume-title":"Proceedings of the 1st USENIX Conference on File and Storage Technologies.","author":"Quinlan Sean","year":"2002","unstructured":"Sean Quinlan and Sean Dorward. 2002. Venti: A new approach to archival data storage. In Proceedings of the 1st USENIX Conference on File and Storage Technologies.USENIX Association, USA."},{"key":"e_1_3_1_143_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-024-63242-1"},{"key":"e_1_3_1_144_2","first-page":"957","volume-title":"Proceedings of the 2021 USENIX Annual Technical Conference","author":"Ren Yanjing","year":"2021","unstructured":"Yanjing Ren, Jingwei Li, Zuoru Yang, Patrick P. C. Lee, and Xiaosong Zhang. 2021. Accelerating encrypted deduplication via SGX. In Proceedings of the 2021 USENIX Annual Technical Conference. USENIX Association, 957\u2013971."},{"key":"e_1_3_1_145_2","doi-asserted-by":"publisher","DOI":"10.1109\/LCOMM.2019.2930508"},{"key":"e_1_3_1_146_2","doi-asserted-by":"publisher","DOI":"10.1109\/INFCOM.2012.6195563"},{"key":"e_1_3_1_147_2","doi-asserted-by":"publisher","DOI":"10.1145\/2385603.2385606"},{"key":"e_1_3_1_148_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSC.2017.2748594"},{"key":"e_1_3_1_149_2","doi-asserted-by":"publisher","DOI":"10.1145\/2485732.2485753"},{"key":"e_1_3_1_150_2","doi-asserted-by":"publisher","DOI":"10.1109\/HPCC.and.EUC.2013.122"},{"key":"e_1_3_1_151_2","doi-asserted-by":"publisher","DOI":"10.1109\/TDSC.2023.3237221"},{"key":"e_1_3_1_152_2","volume-title":"Proceedings of the 10th USENIX Conference on File and Storage Technologies.","author":"Srinivasan Kiran","year":"2012","unstructured":"Kiran Srinivasan, Tim Bisson, Garth Goodson, and Kaladhar Voruganti. 2012. IDedup: Latency-aware, inline data deduplication for primary storage. In Proceedings of the 10th USENIX Conference on File and Storage Technologies.USENIX Association, USA."},{"key":"e_1_3_1_153_2","doi-asserted-by":"publisher","DOI":"10.5555\/2591272.2591290"},{"key":"e_1_3_1_154_2","doi-asserted-by":"publisher","DOI":"10.1145\/3183890"},{"key":"e_1_3_1_155_2","doi-asserted-by":"publisher","DOI":"10.1109\/DCC58796.2024.00044"},{"key":"e_1_3_1_156_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2018.2828842"},{"key":"e_1_3_1_157_2","volume-title":"The rsync Algorithm","author":"Trigdell A.","year":"1998","unstructured":"A. Trigdell and P. Mackerras. 1998. The rsync Algorithm. Technical Report. Australian National University. Retrieved from http:\/\/rsync.samba.org"},{"key":"e_1_3_1_158_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41928-018-0193-y"},{"key":"e_1_3_1_159_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSST.2010.5496987"},{"key":"e_1_3_1_160_2","doi-asserted-by":"publisher","unstructured":"Kai-Ting Weng Yun-Shan Hsieh Yen-Ting Chen Yu-Pei Liang Yuan-Hao Chang Po-Chun Huang and Wei-Kuan Shih. 2023. HF-Dedupe: Hierarchical fingerprint scheme for high efficiency data deduplication on flash-based storage systems. In IEEE\/ACM International Conference on Computer Aided Design (ICCAD). 1\u20139. DOI:10.1109\/ICCAD57390.2023.10323732","DOI":"10.1109\/ICCAD57390.2023.10323732"},{"key":"e_1_3_1_161_2","doi-asserted-by":"publisher","DOI":"10.1109\/MASCOT.2008.4770594"},{"key":"e_1_3_1_162_2","doi-asserted-by":"publisher","DOI":"10.1109\/CCGRID.2018.00036"},{"key":"e_1_3_1_163_2","doi-asserted-by":"publisher","DOI":"10.1145\/3677333.3678271"},{"key":"e_1_3_1_164_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2018.2852642"},{"key":"e_1_3_1_165_2","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPS49936.2021.00025"},{"key":"e_1_3_1_166_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.future.2016.02.001"},{"key":"e_1_3_1_167_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2019.2898942"},{"key":"e_1_3_1_168_2","first-page":"325","volume-title":"Proceedings of the 16th USENIX Conference on File and Storage Technologies","author":"Xia Nai","year":"2018","unstructured":"Nai Xia, Chen Tian, Yan Luo, Hang Liu, and Xiaoliang Wang. 2018. UKSM: Swift memory deduplication via hierarchical and adaptive memory region distilling. In Proceedings of the 16th USENIX Conference on File and Storage Technologies. USENIX Association, 325\u2013340."},{"key":"e_1_3_1_169_2","doi-asserted-by":"publisher","DOI":"10.1109\/JPROC.2016.2571298"},{"key":"e_1_3_1_170_2","doi-asserted-by":"publisher","DOI":"10.1109\/NAS.2012.46"},{"key":"e_1_3_1_171_2","doi-asserted-by":"publisher","unstructured":"Wen Xia Lifeng Pu Xiangyu Zou Philip Shilane Shiyi Li Haijun Zhang and Xuan Wang. 2023. The design of fast and lightweight resemblance detection for efficient post-deduplication delta compression. ACM Transactions on Storage 19 3 (2023) 1\u201330. DOI:10.1145\/3584663","DOI":"10.1145\/3584663"},{"key":"e_1_3_1_172_2","first-page":"101","volume-title":"Proceedings of the 2016 USENIX Annual Technical Conference","author":"Xia Wen","year":"2016","unstructured":"Wen Xia, Yukun Zhou, Hong Jiang, Dan Feng, Yu Hua, Yuchong Hu, Qing Liu, and Yucheng Zhang. 2016. FastCDC: A fast and efficient content-defined chunking approach for data deduplication. In Proceedings of the 2016 USENIX Annual Technical Conference. USENIX Association, 101\u2013114."},{"key":"e_1_3_1_173_2","doi-asserted-by":"publisher","DOI":"10.1007\/s12204-015-1591-1"},{"key":"e_1_3_1_174_2","doi-asserted-by":"publisher","DOI":"10.1109\/IWQoS.2015.7404754"},{"key":"e_1_3_1_175_2","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPS.2010.5470468"},{"key":"e_1_3_1_176_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ins.2020.07.005"},{"key":"e_1_3_1_177_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10723-018-9429-3"},{"key":"e_1_3_1_178_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2022.3179992"},{"key":"e_1_3_1_179_2","doi-asserted-by":"publisher","DOI":"10.1109\/INFOCOM.2015.7218510"},{"key":"e_1_3_1_180_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE51399.2021.00164"},{"key":"e_1_3_1_181_2","doi-asserted-by":"publisher","DOI":"10.1109\/TC.2022.3223302"},{"key":"e_1_3_1_182_2","doi-asserted-by":"publisher","DOI":"10.1109\/PADSW.2014.7097802"},{"key":"e_1_3_1_183_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSST.2015.7208297"},{"key":"e_1_3_1_184_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-05054-2_35"},{"key":"e_1_3_1_185_2","first-page":"269","volume-title":"Proceedings of the 6th USENIX Conference on File and Storage Technologies","author":"Zhu Benjamin","year":"2008","unstructured":"Benjamin Zhu, Kai Li, and R. Hugo Patterson. 2008. Avoiding the disk bottleneck in the data domain deduplication file system. In Proceedings of the 6th USENIX Conference on File and Storage Technologies. USENIX Association, San Jose, California, 269\u2013282."},{"key":"e_1_3_1_186_2","doi-asserted-by":"publisher","DOI":"10.1109\/TCC.2018.2808275"},{"key":"e_1_3_1_187_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIT.1977.1055714"},{"key":"e_1_3_1_188_2","unstructured":"Xiangyu Zou Jingsong Yuan Philip Shilane Wen Xia Haijun Zhang and Xuan Wang. 2021. The Dilemma between Deduplication and Locality: Can Both be Achieved? In The 19th USENIX Conference on File and Storage Technologies (FAST). USENIX Association 171\u2013185."},{"key":"e_1_3_1_189_2","doi-asserted-by":"publisher","DOI":"10.1109\/MICRO.2018.00043"}],"container-title":["ACM Computing Surveys"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3735508","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,9,9]],"date-time":"2025-09-09T14:34:31Z","timestamp":1757428471000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3735508"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,9,9]]},"references-count":188,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2026,2,28]]}},"alternative-id":["10.1145\/3735508"],"URL":"https:\/\/doi.org\/10.1145\/3735508","relation":{},"ISSN":["0360-0300","1557-7341"],"issn-type":[{"value":"0360-0300","type":"print"},{"value":"1557-7341","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,9,9]]},"assertion":[{"value":"2024-12-28","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-05-02","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-09-09","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}