{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,15]],"date-time":"2026-03-15T09:41:19Z","timestamp":1773567679503,"version":"3.50.1"},"reference-count":47,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2024,9,14]],"date-time":"2024-09-14T00:00:00Z","timestamp":1726272000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100012166","name":"National Key R&D Program of China","doi-asserted-by":"crossref","award":["2021YFF0704001"],"award-info":[{"award-number":["2021YFF0704001"]}],"id":[{"id":"10.13039\/501100012166","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Major Research Plan of the National Natural Science Foundation of China","award":["92373114"],"award-info":[{"award-number":["92373114"]}]},{"DOI":"10.13039\/501100001809","name":"Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62072381"],"award-info":[{"award-number":["62072381"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Natural Science Foundation of Fujian Province of China","award":["2023J06001"],"award-info":[{"award-number":["2023J06001"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Archit. Code Optim."],"published-print":{"date-parts":[[2024,9,30]]},"abstract":"<jats:p>Erasure coding has been demonstrated as a storage-efficient means against failures, yet its tunability remains a challenging issue in data centers, which is prone to induce substantial cross-cluster traffic. In this article, we present<jats:sans-serif>ClusterRT<\/jats:sans-serif>, a cluster-aware redundancy transitioning approach that can dynamically tailor the redundancy degree of erasure coding in data centers.<jats:sans-serif>ClusterRT<\/jats:sans-serif>formulates the data relocation as the maximum flow problem to reduce cross-cluster data transfers. It then designs a parity-coordinated update algorithm, which gathers the parity chunks within the same cluster and leverages encoding dependency to further decrease the cross-cluster update traffic.<jats:sans-serif>ClusterRT<\/jats:sans-serif>finally rotates the parity chunks to balance the cross-cluster transitioning traffic across the data center. Large-scale simulation and Alibaba Cloud ECS experiments show that<jats:sans-serif>ClusterRT<\/jats:sans-serif>reduces 94.0% to 96.2% of transitioning traffic and reduces 70.4% to 88.4% of transitioning time.<\/jats:p>","DOI":"10.1145\/3672077","type":"journal-article","created":{"date-parts":[[2024,6,10]],"date-time":"2024-06-10T11:12:20Z","timestamp":1718017940000},"page":"1-24","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Achieving Tunable Erasure Coding with Cluster-Aware Redundancy Transitioning"],"prefix":"10.1145","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0009-0000-4052-2020","authenticated-orcid":false,"given":"Feng","family":"Zhang","sequence":"first","affiliation":[{"name":"Xiamen University, Xiamen, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7762-3252","authenticated-orcid":false,"given":"Fulin","family":"Nan","sequence":"additional","affiliation":[{"name":"Xiamen University, Xiamen China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-5067-5751","authenticated-orcid":false,"given":"Binbin","family":"Xu","sequence":"additional","affiliation":[{"name":"Alibaba Group, Hangzhou, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2673-5868","authenticated-orcid":false,"given":"Zhirong","family":"Shen","sequence":"additional","affiliation":[{"name":"Xiamen University, Xiamen, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8557-0480","authenticated-orcid":false,"given":"Jiebin","family":"Zhai","sequence":"additional","affiliation":[{"name":"Xiamen University, Xiamen, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2765-4509","authenticated-orcid":false,"given":"Dmitrii","family":"Kalplun","sequence":"additional","affiliation":[{"name":"Saint Petersburg Electrotechnical University \"LETI\", Saint Petersburg Russia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7362-2789","authenticated-orcid":false,"given":"Jiwu","family":"Shu","sequence":"additional","affiliation":[{"name":"Xiamen University, Xiamen China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,9,14]]},"reference":[{"key":"e_1_3_1_2_2","unstructured":"Ceph. 2016. Erasure Coding in Ceph. Retrieved from https:\/\/docs.ceph.com\/en\/latest\/rados\/operations\/erasure-code\/"},{"key":"e_1_3_1_3_2","unstructured":"OpenStack. 2019. Erasure Code Support. Retrieved from https:\/\/docs.openstack.org\/swift\/latest\/overview_erasure_code.html"},{"key":"e_1_3_1_4_2","unstructured":"Apache. 2021. HDFS Erasure Coding. Retrieved from https:\/\/hadoop.apache.org\/docs\/stable\/hadoop-project-dist\/hadoop-hdfs\/HDFSErasureCoding.html"},{"key":"e_1_3_1_5_2","unstructured":"Alibaba Cloud. 2023. Alibaba Cloud Elastic Compute Service. Retrieved from https:\/\/www.alibabacloud.com\/product\/ecs"},{"key":"e_1_3_1_6_2","first-page":"1","volume-title":"2014 USENIX Annual Technical Conference (USENIX ATC \u201914)","author":"Ahmad Faraz","year":"2014","unstructured":"Faraz Ahmad, Srimat T. Chakradhar, Anand Raghunathan, and T. N. Vijaykumar. 2014. ShuffleWatcher: Shuffle-aware scheduling in multi-tenant MapReduce clusters. In 2014 USENIX Annual Technical Conference (USENIX ATC \u201914). 1\u201313."},{"issue":"3","key":"e_1_3_1_7_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3129900","article-title":"Efficient and available in-memory KV-store with hybrid erasure coding and replication","volume":"13","author":"Chen Haibo","year":"2017","unstructured":"Haibo Chen, Heng Zhang, Mingkai Dong, Zhaoguo Wang, Yubin Xia, Haibing Guan, and Binyu Zang. 2017. Efficient and available in-memory KV-store with hybrid erasure coding and replication. ACM Transactions on Storage 13, 3 (2017), 1\u201330.","journal-title":"ACM Transactions on Storage"},{"key":"e_1_3_1_8_2","doi-asserted-by":"crossref","unstructured":"Liangfeng Cheng Yuchong Hu and Patrick P. C. Lee. 2019. Coupling decentralized Key-value stores with erasure coding. In Proceedings of the ACM Symposium on Cloud Computing (SoCC).","DOI":"10.1145\/3357223.3362713"},{"key":"e_1_3_1_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/2534169.2486021"},{"key":"e_1_3_1_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/1807128.1807152"},{"key":"e_1_3_1_11_2","unstructured":"Daniel Ford Fran\u00e7ois Labelle Florentina I. Popovici Murray Stokely Van-Anh Truong Luiz Barroso Carrie Grimes and Sean Quinlan. 2010. Availability in globally distributed storage systems. In 9th USENIX Symposium on Operating Systems Design and Implementation (OSDI\u201910)."},{"key":"e_1_3_1_12_2","doi-asserted-by":"publisher","DOI":"10.1109\/INFOCOM42981.2021.9488813"},{"key":"e_1_3_1_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/1592568.1592576"},{"key":"e_1_3_1_14_2","first-page":"233","volume-title":"19th USENIX Conference on File and Storage Technologies (FAST \u201921)","author":"Hu Yuchong","year":"2021","unstructured":"Yuchong Hu, Liangfeng Cheng, Qiaori Yao, Patrick P. C. Lee, Weichun Wang, and Wei Chen. 2021. Exploiting combined locality for wide-stripe erasure coding in distributed storage. In 19th USENIX Conference on File and Storage Technologies (FAST \u201921). 233\u2013248."},{"key":"e_1_3_1_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3149349"},{"key":"e_1_3_1_16_2","unstructured":"Cheng Huang Huseyin Simitci Yikang Xu Aaron Ogus Brad Calder Parikshit Gopalan Jin Li and Sergey Yekhanin. 2012. Erasure coding in windows azure storage. In 2012 USENIX Annual Technical Conference (USENIX ATC\u201912). 15\u201326."},{"issue":"6","key":"e_1_3_1_17_2","doi-asserted-by":"crossref","first-page":"1704","DOI":"10.1109\/TPDS.2014.2326156","article-title":"Scale-RS: An efficient scaling scheme for RS-coded storage clusters","volume":"26","author":"Huang Jianzhong","year":"2014","unstructured":"Jianzhong Huang, Xianhai Liang, Xiao Qin, Ping Xie, and Changsheng Xie. 2014. Scale-RS: An efficient scaling scheme for RS-coded storage clusters. IEEE Transactions on Parallel and Distributed Systems 26, 6 (2014), 1704\u20131717.","journal-title":"IEEE Transactions on Parallel and Distributed Systems"},{"key":"e_1_3_1_18_2","doi-asserted-by":"publisher","DOI":"10.1002\/net.3230120306"},{"key":"e_1_3_1_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/1416944.1416946"},{"key":"e_1_3_1_20_2","first-page":"413","volume-title":"16th USENIX Symposium on Operating Systems Design and Implementation (OSDI \u201922)","author":"Kadekodi Saurabh","year":"2022","unstructured":"Saurabh Kadekodi, Francisco Maturana, Sanjith Athlur, Arif Merchant, K. V. Rashmi, and Gregory R. Ganger. 2022. Tiger: Disk-adaptive redundancy without placement restrictions. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI \u201922). 413\u2013429."},{"key":"e_1_3_1_21_2","unstructured":"Saurabh Kadekodi K. V. Rashmi and Gregory R. Ganger. 2019. Cluster storage systems gotta have HeART: improving storage efficiency by exploiting disk-reliability heterogeneity. In 17th USENIX Conference on File and Storage Technologies (FAST\u201919). 345\u2013358."},{"key":"e_1_3_1_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/LCOMM.2004.833807"},{"key":"e_1_3_1_23_2","doi-asserted-by":"crossref","unstructured":"Francisco Maturana V. S. Chaitanya Mukka and K. V. Rashmi. 2020. Access-optimal linear MDS convertible codes for all parameters. In IEEE International Symposium on Information Theory (ISIT\u201920) IEEE 577\u2013582.","DOI":"10.1109\/ISIT44484.2020.9173947"},{"key":"e_1_3_1_24_2","first-page":"66","volume-title":"Proceedings of the 11th Innovations in Theoretical Computer Science Conference (ITCS\u201920)","volume":"151","author":"Maturana Francisco","year":"2020","unstructured":"Francisco Maturana and K. V. Rashmi. 2020. Convertible codes: New class of codes for efficient conversion of coded data in distributed storage. In Proceedings of the 11th Innovations in Theoretical Computer Science Conference (ITCS\u201920), Vol. 151. 66."},{"key":"e_1_3_1_25_2","unstructured":"Subramanian Muralidhar Wyatt Lloyd Sabyasachi Roy Cory Hill Ernest Lin Weiwen Liu Satadru Pan Shiva Shankar Viswanath Sivakumar Linpeng Tang and Sanjeev Kumar. 2014. f4: Facebook\u2019s warm BLOB storage system. In 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI\u201914) USENIX Association 383\u2013398."},{"key":"e_1_3_1_26_2","doi-asserted-by":"publisher","DOI":"10.14778\/2536222.2536234"},{"key":"e_1_3_1_27_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIT.2014.2325570"},{"key":"e_1_3_1_28_2","article-title":"Jerasure: A Library in C\/C++ Facilitating Erasure Coding for Storage Applications-Version 1.2","volume":"23","author":"Plank James S.","year":"2008","unstructured":"James S. Plank, Scott Simmerman, and Catherine D. Schuman. 2008. Jerasure: A Library in C\/C++ Facilitating Erasure Coding for Storage Applications-Version 1.2. University of Tennessee, Tech. Rep. CS-08-627 23 (2008).","journal-title":"University of Tennessee, Tech. Rep. CS-08-627"},{"key":"e_1_3_1_29_2","doi-asserted-by":"publisher","DOI":"10.1137\/0108018"},{"key":"e_1_3_1_30_2","doi-asserted-by":"crossref","unstructured":"Zhirong Shen and Patrick P. C. Lee. 2018. Cross-rack-aware updates in erasure-coded data centers. In Proceedings of the 47th International Conference on Parallel Processing. 1\u201310.","DOI":"10.1145\/3225058.3225065"},{"key":"e_1_3_1_31_2","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1109\/IPDPS47924.2020.00015","volume-title":"2020 IEEE International Parallel and Distributed Processing Symposium (IPDPS\u201920)","author":"Shen Zhirong","year":"2020","unstructured":"Zhirong Shen, Jiwu Shu, Zhijie Huang, and Yingxun Fu. 2020. ClusterSR: Cluster-aware scattered repair in erasure-coded storage. In 2020 IEEE International Parallel and Distributed Processing Symposium (IPDPS\u201920). IEEE, 42\u201351."},{"key":"e_1_3_1_32_2","doi-asserted-by":"crossref","unstructured":"Zhirong Shen Jiwu Shu and Patrick P. C. Lee. 2016. Reconsidering single failure recovery in clustered file systems. In 2016 46th Annual IEEE\/IFIP International Conference on Dependable Systems and Networks (DSN\u201916) IEEE 323\u2013334.","DOI":"10.1109\/DSN.2016.37"},{"key":"e_1_3_1_33_2","doi-asserted-by":"crossref","unstructured":"Konstantin Taranov Gustavo Alonso and Torsten Hoefler. 2018. Fast and strongly-consistent per-item resilience in key-value stores. In Proceedings of the Thirteenth European Conference on Computer Systems (EuroSys). 1\u201314.","DOI":"10.1145\/3190508.3190536"},{"key":"e_1_3_1_34_2","doi-asserted-by":"publisher","DOI":"10.5555\/3485"},{"key":"e_1_3_1_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/2822895"},{"key":"e_1_3_1_36_2","doi-asserted-by":"crossref","unstructured":"Zizhong Wang Haixia Wang Airan Shao and Dongsheng Wang. 2020. An adaptive erasure-coded storage scheme with an efficient code-switching algorithm. In Proceedings of the 49th International Conference on Parallel Processing. 1\u201311.","DOI":"10.1145\/3404397.3404420"},{"key":"e_1_3_1_37_2","doi-asserted-by":"publisher","DOI":"10.5555\/646334.687814"},{"key":"e_1_3_1_38_2","doi-asserted-by":"crossref","unstructured":"Chentao Wu Xubin He Jizhong Han Huailiang Tan and Changsheng Xie. 2012. SDM: A stripe-based data migration scheme to improve the scalability of RAID-6. In 2012 IEEE International Conference on Cluster Computing IEEE 284\u2013292.","DOI":"10.1109\/CLUSTER.2012.24"},{"key":"e_1_3_1_39_2","first-page":"1669","volume-title":"IEEE Conference on Computer Communications (IEEE INFOCOM\u201922)","author":"Wu Si","year":"2022","unstructured":"Si Wu, Qingpeng Du, Patrick P. C. Lee, Yongkun Li, and Yinlong Xu. 2022. Optimal data placement for stripe merging in locally repairable codes. In IEEE Conference on Computer Communications (IEEE INFOCOM\u201922). IEEE, 1669\u20131678."},{"key":"e_1_3_1_40_2","doi-asserted-by":"crossref","unstructured":"Si Wu Zhirong Shen and Patrick P. C. Lee. 2020. Enabling I\/O-efficient redundancy transitioning in erasure-coded KV stores via elastic Reed-Solomon codes. In International Symposium on Reliable Distributed Systems (SRDS\u201920) IEEE 246\u2013255.","DOI":"10.1109\/SRDS51746.2020.00032"},{"key":"e_1_3_1_41_2","doi-asserted-by":"crossref","unstructured":"Si Wu Zhirong Shen and Patrick P. C. Lee. 2020. On the optimal repair-scaling trade-off in locally repairable codes. In IEEE INFOCOM 2020-IEEE Conference on Computer Communications IEEE 2155\u20132164.","DOI":"10.1109\/INFOCOM41043.2020.9155417"},{"key":"e_1_3_1_42_2","unstructured":"Mingyuan Xia Mohit Saxena Mario Blaum and David A. Pease. 2015. A tale of two erasure codes in HDFS. In 13th USENIX conference on file and storage technologies (FAST\u201915). 213\u2013226."},{"key":"e_1_3_1_43_2","unstructured":"Erci Xu Mai Zheng Feng Qin Yikang Xu and Jiesheng Wu. 2019. Lessons and actions: What we learned from 10k SSD-Related storage system failures. In 2019 USENIX Annual Technical Conference (USENIX ATC\u201919). 961\u2013976."},{"key":"e_1_3_1_44_2","doi-asserted-by":"crossref","unstructured":"Qiaori Yao Yuchong Hu Liangfeng Cheng Patrick P. C. Lee Dan Feng Weichun Wang and Wei Chen. 2021. Stripemerge: Efficient wide-stripe generation for large-scale erasure-coded storage. In 2021 IEEE 41st International Conference on Distributed Computing Systems (ICDCS\u201921) IEEE 483\u2013493.","DOI":"10.1109\/ICDCS51616.2021.00053"},{"key":"e_1_3_1_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/1227835.1227838"},{"key":"e_1_3_1_46_2","doi-asserted-by":"publisher","DOI":"10.1109\/TC.2009.150"},{"key":"e_1_3_1_47_2","doi-asserted-by":"crossref","unstructured":"Xiaoyang Zhang Yuchong Hu Patrick P. C. Lee and Pan Zhou. 2018. Toward optimal storage scaling via network coding: From theory to practice. In IEEE INFOCOM 2018-IEEE Conference on Computer Communications IEEE 1808\u20131816.","DOI":"10.1109\/INFOCOM.2018.8485961"},{"key":"e_1_3_1_48_2","unstructured":"Weimin Zheng and Guangyan Zhang. 2011. FastScale: Accelerate RAID scaling by minimizing data migration. In 9th USENIX Conference on File and Storage Technologies (FAST\u201911)."}],"container-title":["ACM Transactions on Architecture and Code Optimization"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3672077","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3672077","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:04:29Z","timestamp":1750291469000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3672077"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,9,14]]},"references-count":47,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2024,9,30]]}},"alternative-id":["10.1145\/3672077"],"URL":"https:\/\/doi.org\/10.1145\/3672077","relation":{},"ISSN":["1544-3566","1544-3973"],"issn-type":[{"value":"1544-3566","type":"print"},{"value":"1544-3973","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,9,14]]},"assertion":[{"value":"2023-12-06","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-05-22","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-09-14","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}