{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,5]],"date-time":"2026-06-05T05:31:07Z","timestamp":1780637467906,"version":"3.54.1"},"reference-count":41,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2023,1,11]],"date-time":"2023-01-11T00:00:00Z","timestamp":1673395200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62072214"],"award-info":[{"award-number":["62072214"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100021171","name":"Guangdong Basic and Applied Basic Research Foundation","doi-asserted-by":"crossref","award":["2021B1515120048"],"award-info":[{"award-number":["2021B1515120048"]}],"id":[{"id":"10.13039\/501100021171","id-type":"DOI","asserted-by":"crossref"}]},{"name":"International Cooperation Project of Guangdong Province","award":["2020A0505100040"],"award-info":[{"award-number":["2020A0505100040"]}]},{"name":"Science and Technology Planning Project of Guangzhou","award":["202103000036"],"award-info":[{"award-number":["202103000036"]}]},{"name":"Open Project Program of Wuhan National Laboratory for Optoelectronics","award":["2020WNLOKF006"],"award-info":[{"award-number":["2020WNLOKF006"]}]},{"name":"Industry-University-Research Collaboration Project of Zhuhai","award":["ZH22017001210048PWC"],"award-info":[{"award-number":["ZH22017001210048PWC"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Storage"],"published-print":{"date-parts":[[2023,2,28]]},"abstract":"<jats:p>Inline deduplication removes redundant data in real-time as data is being sent to the storage system. However, it causes data fragmentation: logically consecutive chunks are physically scattered across various containers after data deduplication. Many rewrite algorithms aim to alleviate the performance degradation due to fragmentation by rewriting fragmented duplicate chunks as unique chunks into new containers. Unfortunately, these algorithms determine whether a chunk is fragmented based on a simple pre-set fixed value, ignoring the variance of data characteristics between data segments. Accordingly, when backups are restored, they often fail to select an appropriate set of old containers for rewrite, generating a substantial number of invalid chunks in retrieved containers.<\/jats:p>\n          <jats:p>\n            To address this issue, we propose an inline deduplication approach for storage systems, called\n            <jats:italic>InDe<\/jats:italic>\n            , which uses a greedy algorithm to detect valid container utilization and dynamically adjusts the number of old container references in each segment. InDe fully leverages the distribution of duplicated chunks to improve the restore performance while maintaining high backup performance. We define an effectiveness metric,\n            <jats:italic>valid container referenced counts (VCRC)<\/jats:italic>\n            , to identify appropriate containers for the rewrite. We design a rewrite algorithm\n            <jats:italic>F-greedy<\/jats:italic>\n            that detects valid container utilization to rewrite low-VCRC containers. According to the VCRC distribution of containers, F-greedy dynamically adjusts the number of old container references to only share duplicate chunks with high-utilization containers for each segment, thereby improving the restore speed. To take full advantage of the above features, we further propose another rewrite algorithm called\n            <jats:italic>F-greedy+<\/jats:italic>\n            based on adaptive interval detection of valid container utilization. F-greedy+ makes a more accurate estimation of the valid utilization of old containers by detecting trends of VCRC\u2019s change in two directions and selecting referenced containers in the global scope. We quantitatively evaluate InDe using three real-world backup workloads. The experimental results show that compared with two state-of-the-art algorithms (Capping and SMR), our scheme improves the restore speed by 1.3\u00d7\u20132.4\u00d7 while achieving almost the same backup performance.\n          <\/jats:p>","DOI":"10.1145\/3568426","type":"journal-article","created":{"date-parts":[[2022,11,19]],"date-time":"2022-11-19T10:27:26Z","timestamp":1668853646000},"page":"1-27","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":13,"title":["InDe: An Inline Data Deduplication Approach via Adaptive Detection of Valid Container Utilization"],"prefix":"10.1145","volume":"19","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1991-0293","authenticated-orcid":false,"given":"Lifang","family":"Lin","sequence":"first","affiliation":[{"name":"Department of Computer Science, Jinan University, Guangzhou, Guangdong Province, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1522-8943","authenticated-orcid":false,"given":"Yuhui","family":"Deng","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Jinan University, Guangzhou, Guangdong Province, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1460-322X","authenticated-orcid":false,"given":"Yi","family":"Zhou","sequence":"additional","affiliation":[{"name":"TSYS School of Computer Science, Columbus State University, GA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7438-8149","authenticated-orcid":false,"given":"Yifeng","family":"Zhu","sequence":"additional","affiliation":[{"name":"Department of Electrical and Computer Engineering, University of Maine, Orono, ME, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,1,11]]},"reference":[{"key":"e_1_3_1_2_2","unstructured":"Dell Technologies. 2021. IDC The Business Value of Storage Solutions from Dell Technologies. Retrieved from https:\/\/www.delltechnologies.com\/asset\/zh-cn\/products\/storage\/industry-market\/idc-the-business-value-of-storage-solutions-from-dell-technologies.pdf."},{"key":"e_1_3_1_3_2","unstructured":"FSL. 2021. Traces and Snapshots Public Archive. Retrieved from https:\/\/tracer.filesystems.org\/."},{"key":"e_1_3_1_4_2","unstructured":"R. Bauer. 2018. HDD vs SSD: What Does the Future for Storage Hold? Retrieved from https:\/\/www.backblaze.com\/blog\/hdd-vs-ssd-in-data-centers\/."},{"key":"e_1_3_1_5_2","first-page":"129","volume-title":"Proceedings of the 19th USENIX Conference on File and Storage Technologies (FAST\u201919)","author":"Cao Zhichao","year":"2019","unstructured":"Zhichao Cao, Shiyong Liu, Fenggang Wu, Guohua Wang, Bingzhe Li, and David H. C. Du. 2019. Sliding look-back window assisted data chunk rewriting for improving deduplication restore performance. In Proceedings of the 19th USENIX Conference on File and Storage Technologies (FAST\u201919). 129\u2013142."},{"issue":"2","key":"e_1_3_1_6_2","doi-asserted-by":"crossref","first-page":"558","DOI":"10.1109\/TPDS.2015.2409872","article-title":"TIGER: Thermal-aware file assignment in storage clusters","volume":"27","author":"Chavan Ajit","year":"2015","unstructured":"Ajit Chavan, Mohammed I. Alghamdi, Xunfei Jiang, Xiao Qin, Meikang Qiu, Minghua Jiang, and Jifu Zhang. 2015. TIGER: Thermal-aware file assignment in storage clusters. IEEE Trans. Parallel Distrib. Syst. 27, 2 (2015), 558\u2013573.","journal-title":"IEEE Trans. Parallel Distrib. Syst."},{"issue":"5","key":"e_1_3_1_7_2","first-page":"1103","article-title":"Memory deduplication: An effective approach to improve the memory system","volume":"33","author":"Deng Yuhui","year":"2017","unstructured":"Yuhui Deng, Xinyu Huang, Liangshan Song, Yongtao Zhou, and Frank Z. Wang. 2017. Memory deduplication: An effective approach to improve the memory system. J. Info. Sci. Eng. 33, 5 (2017), 1103\u20131120.","journal-title":"J. Info. Sci. Eng."},{"key":"e_1_3_1_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2015.2410781"},{"key":"e_1_3_1_9_2","first-page":"181","volume-title":"Proceedings of the USENIX Annual Technical Conference (ATC\u201914)","author":"Fu Min","year":"2014","unstructured":"Min Fu, Dan Feng, Yu Hua, Xubin He, Zuoning Chen, Wen Xia, Fangting Huang, and Qing Liu. 2014. Accelerating restore and garbage collection in deduplication-based backup systems via exploiting historical information. In Proceedings of the USENIX Annual Technical Conference (ATC\u201914). 181\u2013192."},{"key":"e_1_3_1_10_2","doi-asserted-by":"publisher","DOI":"10.5555\/2750482.2750507"},{"key":"e_1_3_1_11_2","volume-title":"Proceedings of the USENIX Annual Technical Conference (ATC\u201911)","author":"Guo Fanglu","year":"2011","unstructured":"Fanglu Guo and Petros Efstathopoulos. 2011. Building a high-performance deduplication system. In Proceedings of the USENIX Annual Technical Conference (ATC\u201911)."},{"key":"e_1_3_1_12_2","first-page":"733","volume-title":"Proceedings of the USENIX Annual Technical Conference (ATC\u201917)","author":"Guo Fan","year":"2017","unstructured":"Fan Guo, Yongkun Li, Yinlong Xu, Song Jiang, and John C. S. Lui. 2017. Smartmd: A high performance deduplication engine with mixed pages. In Proceedings of the USENIX Annual Technical Conference (ATC\u201917). 733\u2013744."},{"key":"e_1_3_1_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/2367589.2367600"},{"key":"e_1_3_1_14_2","doi-asserted-by":"crossref","first-page":"959","DOI":"10.1145\/1281192.1281295","volume-title":"Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","author":"Kohavi Ron","year":"2007","unstructured":"Ron Kohavi, Randal M. Henne, and Dan Sommerfield. 2007. Practical guide to controlled experiments on the web: Listen to your customers not to the hippo. In Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 959\u2013967."},{"key":"e_1_3_1_15_2","first-page":"457","volume-title":"Proceedings of the International Conference on Algorithms and Architectures for Parallel Processing","author":"Lai Rongyu","year":"2014","unstructured":"Rongyu Lai, Yu Hua, Dan Feng, Wen Xia, Min Fu, and Yifan Yang. 2014. A near-exact defragmentation scheme to improve restore performance for cloud backup systems. In Proceedings of the International Conference on Algorithms and Architectures for Parallel Processing. Springer, 457\u2013471."},{"key":"e_1_3_1_16_2","doi-asserted-by":"publisher","DOI":"10.5555\/2591272.2591292"},{"key":"e_1_3_1_17_2","volume-title":"Proceedings of the 27th International Conference on Parallel and Distributed Systems (ICPADS\u201921)","author":"Lin Lifang","year":"2021","unstructured":"Lifang Lin, Yuhui Deng, and Yi Zhou. 2021. Improving restore performance of deduplication systems via a greedy rewriting scheme. In Proceedings of the 27th International Conference on Parallel and Distributed Systems (ICPADS\u201921)."},{"key":"e_1_3_1_18_2","first-page":"256","volume-title":"Proceedings of the 12th USENIX Conference on File and Storage Technologies (FAST\u201914)","author":"Lin Xing","year":"2014","unstructured":"Xing Lin, Guanlin Lu, Fred Douglis, Philip Shilane, and Grant Wallace. 2014. Migratory compression: Coarse-grained data reordering to improve compressibility. In Proceedings of the 12th USENIX Conference on File and Storage Technologies (FAST\u201914). 256\u2013273."},{"key":"e_1_3_1_19_2","first-page":"1","volume-title":"Proceedings of the 30th Symposium on Mass Storage Systems and Technologies (MSST\u201914)","author":"Liu Jian","year":"2014","unstructured":"Jian Liu, Yunpeng Chai, Xiao Qin, and Yuan Xiao. 2014. PLC-cache: Endurable SSD cache for deduplication-based primary storage. In Proceedings of the 30th Symposium on Mass Storage Systems and Technologies (MSST\u201914). IEEE, 1\u201312."},{"key":"e_1_3_1_20_2","article-title":"Boafft: Distributed deduplication for big data storage in the cloud","author":"Luo Shengmei","year":"2015","unstructured":"Shengmei Luo, Guangyan Zhang, Chengwen Wu, Samee Khan, and Keqin Li. 2015. Boafft: Distributed deduplication for big data storage in the cloud. IEEE Trans. Cloud Comput. 8, 4 (2015), 1199\u20131211.","journal-title":"IEEE Trans. Cloud Comput."},{"key":"e_1_3_1_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/3078837"},{"issue":"1","key":"e_1_3_1_22_2","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1109\/TCC.2018.2858792","article-title":"Efficient replica migration scheme for distributed cloud storage systems","volume":"9","author":"Mseddi Amina","year":"2018","unstructured":"Amina Mseddi, Mohammad A. Salahuddin, Mohamed Faten Zhani, Halima Elbiaze, and Roch H. Glitho. 2018. Efficient replica migration scheme for distributed cloud storage systems. IEEE Trans. Cloud Comput. 9, 1 (2018), 155\u2013167.","journal-title":"IEEE Trans. Cloud Comput."},{"key":"e_1_3_1_23_2","first-page":"193","volume-title":"Proceedings of the 18th USENIX Conference on File and Storage Technologies (FAST\u201920)","author":"Nachman Aviv","year":"2020","unstructured":"Aviv Nachman, Gala Yadgar, and Sarai Sheinvald. 2020. GoSeed: Generating an optimal seeding plan for deduplicated storage. In Proceedings of the 18th USENIX Conference on File and Storage Technologies (FAST\u201920). 193\u2013207."},{"key":"e_1_3_1_24_2","first-page":"220","volume-title":"Proceedings of the ACM Symposium on Cloud Computing","author":"Ni Fan","year":"2019","unstructured":"Fan Ni and Song Jiang. 2019. RapidCDC: Leveraging duplicate locality to accelerate chunking in CDC-based deduplication systems. In Proceedings of the ACM Symposium on Cloud Computing. 220\u2013232."},{"key":"e_1_3_1_25_2","doi-asserted-by":"publisher","DOI":"10.5555\/2591272.2591290"},{"key":"e_1_3_1_26_2","first-page":"1","volume-title":"Proceedings of the 32nd Symposium on Mass Storage Systems and Technologies (MSST\u201916)","author":"Sun Zhen","year":"2016","unstructured":"Zhen Sun, Geoff Kuenning, Sonam Mandal, Philip Shilane, Vasily Tarasov, Nong Xiao et\u00a0al. 2016. A long-term user-centric analysis of deduplication patterns. In Proceedings of the 32nd Symposium on Mass Storage Systems and Technologies (MSST\u201916). IEEE, 1\u20137."},{"key":"e_1_3_1_27_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2018.2828842"},{"issue":"1","key":"e_1_3_1_28_2","first-page":"214","article-title":"Improving the performance of deduplication-based storage cache via content-driven cache management methods","volume":"32","author":"Tan Yujuan","year":"2020","unstructured":"Yujuan Tan, Congcong Xu, Jing Xie, Zhichao Yan, Hong Jiang, Witawas Srisa-an, Xianzhang Chen, and Duo Liu. 2020. Improving the performance of deduplication-based storage cache via content-driven cache management methods. IEEE Trans. Parallel Distrib. Syst. 32, 1 (2020), 214\u2013228.","journal-title":"IEEE Trans. Parallel Distrib. Syst."},{"key":"e_1_3_1_29_2","first-page":"261","volume-title":"Proceedings of the USENIX Annual Technical Conference (ATC\u201912)","author":"Tarasov Vasily","year":"2012","unstructured":"Vasily Tarasov, Amar Mudrankit, Will Buik, Philip Shilane, Geoff Kuenning, and Erez Zadok. 2012. Generating realistic datasets for deduplication analysis. In Proceedings of the USENIX Annual Technical Conference (ATC\u201912). 261\u2013272."},{"issue":"2","key":"e_1_3_1_30_2","first-page":"169","article-title":"End-to-end delay minimization for scientific workflows in clouds under budget constraint","volume":"3","author":"Wu Chase Qishi","year":"2014","unstructured":"Chase Qishi Wu, Xiangyu Lin, Dantong Yu, Wei Xu, and Li Li. 2014. End-to-end delay minimization for scientific workflows in clouds under budget constraint. IEEE Trans. Cloud Comput. 3, 2 (2014), 169\u2013181.","journal-title":"IEEE Trans. Cloud Comput."},{"issue":"1","key":"e_1_3_1_31_2","first-page":"119","article-title":"Improving restore performance in deduplication systems via a cost-efficient rewriting scheme","volume":"30","author":"Wu Jie","year":"2018","unstructured":"Jie Wu, Yu Hua, Pengfei Zuo, and Yuanyuan Sun. 2018. Improving restore performance in deduplication systems via a cost-efficient rewriting scheme. IEEE Trans. Parallel Distrib. Syst. 30, 1 (2018), 119\u2013132.","journal-title":"IEEE Trans. Parallel Distrib. Syst."},{"key":"e_1_3_1_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2019.2898942"},{"key":"e_1_3_1_33_2","first-page":"325","volume-title":"Proceedings of the 16th USENIX Conference on File and Storage Technologies (FAST\u201918)","author":"Xia Nai","year":"2018","unstructured":"Nai Xia, Chen Tian, Yan Luo, Hang Liu, and Xiaoliang Wang. 2018. UKSM: Swift memory deduplication via hierarchical and adaptive memory region distilling. In Proceedings of the 16th USENIX Conference on File and Storage Technologies (FAST\u201918). 325\u2013340."},{"key":"e_1_3_1_34_2","first-page":"203","volume-title":"Proceedings of the Data Compression Conference","author":"Xia Wen","year":"2014","unstructured":"Wen Xia, Hong Jiang, Dan Feng, and Lei Tian. 2014. Combining deduplication and delta compression to achieve low-overhead data reduction on backup datasets. In Proceedings of the Data Compression Conference. IEEE, 203\u2013212."},{"key":"e_1_3_1_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2020.2984632"},{"key":"e_1_3_1_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/3437261"},{"key":"e_1_3_1_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3459626"},{"key":"e_1_3_1_38_2","first-page":"1337","volume-title":"Proceedings of the IEEE Conference on Computer Communications (INFOCOM\u201915)","author":"Zhang Yucheng","year":"2015","unstructured":"Yucheng Zhang, Hong Jiang, Dan Feng, Wen Xia, Min Fu, Fangting Huang, and Yukun Zhou. 2015. AE: An asymmetric extremum content defined chunking algorithm for fast and bandwidth-efficient data deduplication. In Proceedings of the IEEE Conference on Computer Communications (INFOCOM\u201915). IEEE, 1337\u20131345."},{"key":"e_1_3_1_39_2","first-page":"769","volume-title":"Proceedings of the USENIX Annual Technical Conference (ATC\u201920)","author":"Zhao Nannan","year":"2020","unstructured":"Nannan Zhao, Hadeel Albahar, Subil Abraham, Keren Chen, Vasily Tarasov, Dimitrios Skourtis, Lukas Rupprecht, Ali Anwar, and Ali R Butt. 2020. Duphunter: Flexible high-performance deduplication for docker registries. In Proceedings of the USENIX Annual Technical Conference (ATC\u201920). 769\u2013783."},{"key":"e_1_3_1_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2018.2800763"},{"key":"e_1_3_1_41_2","first-page":"269","volume-title":"Proceedings of the 6th USENIX Conference on File and Storage Technologies (FAST\u201908)","volume":"8","author":"Zhu Benjamin","year":"2008","unstructured":"Benjamin Zhu, Kai Li, and R. Hugo Patterson. 2008. Avoiding the disk bottleneck in the data domain deduplication file system. In Proceedings of the 6th USENIX Conference on File and Storage Technologies (FAST\u201908), Vol. 8. 269\u2013282."},{"key":"e_1_3_1_42_2","first-page":"171","volume-title":"Proceedings of the 19th USENIX Conference on File and Storage Technologies (FAST\u201921)","author":"Zou Xiangyu","year":"2021","unstructured":"Xiangyu Zou, Jingsong Yuan, Philip Shilane, Wen Xia, Haijun Zhang, and Xuan Wang. 2021. The dilemma between deduplication and locality: Can both be achieved? In Proceedings of the 19th USENIX Conference on File and Storage Technologies (FAST\u201921). 171\u2013185."}],"container-title":["ACM Transactions on Storage"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3568426","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3568426","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T17:51:33Z","timestamp":1750182693000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3568426"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,1,11]]},"references-count":41,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2023,2,28]]}},"alternative-id":["10.1145\/3568426"],"URL":"https:\/\/doi.org\/10.1145\/3568426","relation":{},"ISSN":["1553-3077","1553-3093"],"issn-type":[{"value":"1553-3077","type":"print"},{"value":"1553-3093","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,1,11]]},"assertion":[{"value":"2022-01-11","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-07-25","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-01-11","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}