{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,19]],"date-time":"2026-01-19T06:44:52Z","timestamp":1768805092589,"version":"3.49.0"},"reference-count":27,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2016,7,21]],"date-time":"2016-07-21T00:00:00Z","timestamp":1469059200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61572394"],"award-info":[{"award-number":["61572394"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"name":"National Key Research and Development Plan of China","award":["2016YFB1000303"],"award-info":[{"award-number":["2016YFB1000303"]}]},{"name":"Shenzhen fundamental research plan","award":["JCYJ20120615101127404"],"award-info":[{"award-number":["JCYJ20120615101127404"]}]},{"name":"Shenzhen fundamental research plan","award":["JSGG20140519141854753"],"award-info":[{"award-number":["JSGG20140519141854753"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Symmetry"],"abstract":"<jats:p>Deduplication is an efficient data reduction technique, and it is used to mitigate the problem of huge data volume in big data storage systems. Content defined chunking (CDC) is the most widely used algorithm in deduplication systems. The expected chunk size is an important parameter of CDC, and it influences the duplicate elimination ratio (DER) significantly. We collected two realistic datasets to perform an experiment. The experimental results showed that the current approach of setting the expected chunk size to 4 KB or 8 KB empirically cannot optimize DER. Therefore, we present a logistic based mathematical model to reveal the hidden relationship between the expected chunk size and the DER. This model provides a theoretical basis for optimizing DER by setting the expected chunk size reasonably. We used the collected datasets to verify this model. The experimental results showed that the R2 values, which describe the goodness of fit, are above 0.9, validating the correctness of this mathematic model. Based on the DER model, we discussed how to make DER close to the optimum by setting the expected chunk size reasonably.<\/jats:p>","DOI":"10.3390\/sym8070069","type":"journal-article","created":{"date-parts":[[2016,7,21]],"date-time":"2016-07-21T09:48:05Z","timestamp":1469094485000},"page":"69","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["A Logistic Based Mathematical Model to Optimize Duplicate Elimination Ratio in Content Defined Chunking Based Big Data Storage System"],"prefix":"10.3390","volume":"8","author":[{"given":"Longxiang","family":"Wang","sequence":"first","affiliation":[{"name":"The School of Electronic and Information Engineering, Xi\u2019an Jiaotong University, Xi\u2019an 710049, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xiaoshe","family":"Dong","sequence":"additional","affiliation":[{"name":"The School of Electronic and Information Engineering, Xi\u2019an Jiaotong University, Xi\u2019an 710049, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xingjun","family":"Zhang","sequence":"additional","affiliation":[{"name":"The School of Electronic and Information Engineering, Xi\u2019an Jiaotong University, Xi\u2019an 710049, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Fuliang","family":"Guo","sequence":"additional","affiliation":[{"name":"The School of Electronic and Information Engineering, Xi\u2019an Jiaotong University, Xi\u2019an 710049, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yinfeng","family":"Wang","sequence":"additional","affiliation":[{"name":"The Shenzhen Institute of Information Technology, Shenzhen, 518172, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Weifeng","family":"Gong","sequence":"additional","affiliation":[{"name":"State Key Laboratory of High-End Server &amp; Storage Technology, Jinan 250101, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2016,7,21]]},"reference":[{"key":"ref_1","unstructured":"EMC (2012), EMC DATA DOMAIN white papaer. Available online: http:\/\/www.emc.com\/collateral\/software\/white-papers\/h7219-data-domain-data-invul-arch-wp.pdf."},{"key":"ref_2","unstructured":"Dubnicki, C., Gryz, L., Heldt, L., Kaczmarczyk, M., Kilian, W., Strzelczak, P., Szczepkowski, J., Ungureanu, C., and Welnicki, M. (2009, January 24\u201327). Hydrastor: A scalable secondary storage. Proceedings of the 7th USENIX Conference on File and Storage Technologies, San Francisco, CA, USA."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"824","DOI":"10.1109\/TC.2010.263","article-title":"Efficient deduplication techniques for modern backup operation","volume":"60","author":"Min","year":"2011","journal-title":"IEEE Trans. Comput."},{"key":"ref_4","unstructured":"Quinlan, S., and Dorward, S. (2002, January 28\u201330). Venti: A New Approach to Archival Data Storage. Proceedings of the 1st USENIX Conference on File and Storage Technologies, Monterey, CA, USA."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Tsuchiya, Y., and Watanabe, T. (2011, January 23\u201327). DBLK: Deduplication for primary block storage. Proceedings of the IEEE 27th Symposium on Mass Storage Systems and Technologies, Denver, CO, USA.","DOI":"10.1109\/MSST.2011.5937237"},{"key":"ref_6","unstructured":"Srinivasan, K., Bisson, T., Goodson, G., and Voruganti, K. (2012, January 14\u201317). iDedup: Latency-aware, inline data deduplication for primary storage. Proceedings of the 10th USENIX conference on File and Storage Technologies, San Jose, CA, USA."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Muthitacharoen, A., Chen, B., and Mazieres, D. (2001, January 21\u201324). A low-bandwidth network file system. Proceedings of the Eighteenth ACM Symposium on Operating Systems Principles, Banff, AB, Canada.","DOI":"10.1145\/502034.502052"},{"key":"ref_8","unstructured":"You, L.L., and Karamanolis, C. (2004, January 13\u201316). Evaluation of efficient archival storage techniques. Proceedings of the 21st IEEE\/12th NASA Goddard Conference on Mass Storage Systems and Technologies, Greenbelt, MD, USA."},{"key":"ref_9","unstructured":"Roma\u0144ski, \u0141.H.B., Kilian, W., Lichota, K., and Dubnicki, C. (June, January 30). Anchor-driven subchunk deduplication. Proceedings of the 4th Annual International Conference on Systems and Storage, Haifa, Israel."},{"key":"ref_10","unstructured":"Bolosky, W.J., Corbin, S., Goebel, D., and Douceur, J.R. (2000, January 3\u20134). Single instance storage in Windows 2000. Proceedings of the 4th USENIX Windows Systems Symposium, Seattle, WA, USA."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"45","DOI":"10.3233\/ICA-120418","article-title":"A novel approach to data deduplication over the engineering-oriented cloud systems","volume":"20","author":"Sun","year":"2013","journal-title":"Integr. Comput. Aided Eng."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"190","DOI":"10.1145\/356989.357007","article-title":"OceanStore: An architecture for global-scale persistent storage","volume":"35","author":"Kubiatowicz","year":"2000","journal-title":"SIGPLAN Not."},{"key":"ref_13","unstructured":"Rabin, M.O. Available online: http:\/\/www.xmailserver.org\/rabin.pdf."},{"key":"ref_14","unstructured":"Kruus, E., Ungureanu, C., and Dubnicki, C. (2010, January 23\u201326). Bimodal content defined chunking for backup streams. Proceedings of the 8th USENIX Conference on File and Storage Technologies, San Jose, CA, USA."},{"key":"ref_15","unstructured":"Eshghi, K., and Tang, H.K. A framework for analyzing and improving content-based chunking algorithms. 2005. Available online: http:\/\/www.hpl.hp.com\/techreports\/2005\/HPL-2005\u201330R1.pdf."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"424","DOI":"10.1145\/1210596.1210599","article-title":"Improving duplicate elimination in storage systems","volume":"2","author":"Bobbarjung","year":"2006","journal-title":"ACM Trans. Storage"},{"key":"ref_17","unstructured":"Lillibridge, M., Eshghi, K., Bhagwat, D., Deolalikar, V., Trezise, G., and Camble, P. (2009, January 24\u201327). Sparse indexing: Large scale, inline deduplication using sampling and locality. Proceedings of the 7th USENIX Conference on File and Storage Technologies, San Francisco, CA, USA."},{"key":"ref_18","unstructured":"Debnath, B., Sengupta, S., and Li, J. (2010, January 23\u201325). ChunkStash: Speeding up inline storage deduplication using flash memory. Proceedings of the 2010 USENIX Conference on USENIX Annual Technical Conference, Boston, MA, USA."},{"key":"ref_19","unstructured":"Symantec About Deduplication Chunk Size. Available online: https:\/\/sort.symantec.com\/public\/documents\/vis\/7.0\/aix\/productguides\/html\/sf_admin\/ch29s01s01.htm."},{"key":"ref_20","unstructured":"IBM (2016). Determining the Impact of Deduplication on a Tivoli Storage Manager Server Database and Storage Pools. Available online: http:\/\/www-01.ibm.com\/support\/docview.wss?uid=swg21596944."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"399","DOI":"10.1007\/978-3-642-25789-6_55","article-title":"Impact on Chunk Size on Deduplication and Disk Prefetch","volume":"Volume 2","author":"Suzaki","year":"2012","journal-title":"Recent Advances in Computer Science and Information Engineering"},{"key":"ref_22","unstructured":"Linux (2016). The Linux Kernel Archives. Available online: http:\/\/kernel.org\/."},{"key":"ref_23","unstructured":"Wikipedia (2015). svwiki dump. Available online: http:\/\/dumps.wikimedia.org\/svwiki\/20150807\/."},{"key":"ref_24","unstructured":"D.RichardHipp (2016), SQLite homepage. Available online: http:\/\/sqlite.org\/."},{"key":"ref_25","unstructured":"TPCC (2010, 2009\u201302\u201311). TPC BENCHMARK\u2122 C Standard Specification. Available online: http:\/\/www.tpc.org\/tpc_documents_current_versions\/pdf\/tpc-c_v5.11.0.pdf."},{"key":"ref_26","unstructured":"Liu, A. Deduputil homepage. Available online: https:\/\/sourceforge.net\/projects\/deduputil\/."},{"key":"ref_27","unstructured":"Harnik, D., Khaitzin, E., and Sotnikov, D. (2016, January 22\u201325). Estimating Unseen Deduplication-from Theory to Practice. Proceedings of the 14th USENIX Conference on File and Storage Technologies (FAST 16), Santa Clara, CA, USA."}],"container-title":["Symmetry"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2073-8994\/8\/7\/69\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T19:26:42Z","timestamp":1760210802000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2073-8994\/8\/7\/69"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2016,7,21]]},"references-count":27,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2016,7]]}},"alternative-id":["sym8070069"],"URL":"https:\/\/doi.org\/10.3390\/sym8070069","relation":{},"ISSN":["2073-8994"],"issn-type":[{"value":"2073-8994","type":"electronic"}],"subject":[],"published":{"date-parts":[[2016,7,21]]}}}