{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:14:29Z","timestamp":1750220069678,"version":"3.41.0"},"reference-count":44,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2023,6,19]],"date-time":"2023-06-19T00:00:00Z","timestamp":1687132800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Storage"],"published-print":{"date-parts":[[2023,8,31]]},"abstract":"<jats:p>Data arrangement determines the capacity, resilience, and performance of a distributed storage system. A scalable self-managed system must place its data efficiently not only during stable operation but also after an expansion, planned downscaling, or device failures. In this article, we present Derrick, a data balancing algorithm addressing these needs, which has been developed for HYDRAstor, a highly scalable commercial storage system. Derrick makes its decisions quickly in case of failures but takes additional time to find a nearly optimal data arrangement and a plan for reaching it when the device population changes. Compared to balancing algorithms in two other state-of-the-art systems, Derrick provides better capacity utilization, reduced data movement, and improved performance. Moreover, it can be easily adapted to meet custom placement requirements.<\/jats:p>","DOI":"10.1145\/3594543","type":"journal-article","created":{"date-parts":[[2023,4,28]],"date-time":"2023-04-28T11:58:10Z","timestamp":1682683090000},"page":"1-34","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Derrick: A Three-layer Balancer for Self-managed Continuous Scalability"],"prefix":"10.1145","volume":"19","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4203-3973","authenticated-orcid":false,"given":"Andrzej","family":"Jackowski","sequence":"first","affiliation":[{"name":"9LivesData LLC"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-4433-0252","authenticated-orcid":false,"given":"Leszek","family":"Gryz","sequence":"additional","affiliation":[{"name":"9LivesData LLC"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-0272-4258","authenticated-orcid":false,"given":"Micha\u0142","family":"We\u0142nicki","sequence":"additional","affiliation":[{"name":"9LivesData LLC"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-1844-9218","authenticated-orcid":false,"given":"Cezary","family":"Dubnicki","sequence":"additional","affiliation":[{"name":"9LivesData LLC"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5380-6337","authenticated-orcid":false,"given":"Konrad","family":"Iwanicki","sequence":"additional","affiliation":[{"name":"University of Warsaw"}]}],"member":"320","published-online":{"date-parts":[[2023,6,19]]},"reference":[{"key":"e_1_3_3_2_2","volume-title":"A Data Rebalancing Mechanism for Gluster File System","author":"Aye Kyar Nyo","year":"2014","unstructured":"Kyar Nyo Aye and Thandar Thein. 2014. A Data Rebalancing Mechanism for Gluster File System. Ph. D. Dissertation. MERAL Portal."},{"key":"e_1_3_3_3_2","volume-title":"Proceedings of the 9th USENIX Symposium on Operating Systems Design and Implementation (OSDI 10)","author":"Beaver Doug","year":"2010","unstructured":"Doug Beaver, Sanjeev Kumar, Harry C. Li, Jason Sobel, and Peter Vajgel. 2010. Finding a needle in haystack: Facebook\u2019s photo storage. In Proceedings of the 9th USENIX Symposium on Operating Systems Design and Implementation (OSDI 10)."},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/2043556.2043571"},{"volume-title":"Balancer Plugin","year":"2022","key":"e_1_3_3_5_2","unstructured":"Ceph. 2022. Balancer Plugin. Retrieved April 16, 2023 from https:\/\/docs.ceph.com\/en\/mimic\/mgr\/balancer\/."},{"volume-title":"CHAPTER 3. PLACEMENT GROUPS (PGS)","year":"2022","key":"e_1_3_3_6_2","unstructured":"Ceph. 2022. CHAPTER 3. PLACEMENT GROUPS (PGS). Retrieved April 16, 2023 from https:\/\/access.redhat.com\/documentation\/en-us\/red_hat_ceph_storage\/5\/html\/storage_strategies_guide\/placement_groups_pgs."},{"volume-title":"Placement Groups","year":"2022","key":"e_1_3_3_7_2","unstructured":"Ceph. 2022. Placement Groups. Retrieved April 16, 2023 from https:\/\/docs.ceph.com\/en\/latest\/rados\/operations\/placement-groups\/."},{"volume-title":"V0.94.10 HAMMER","year":"2022","key":"e_1_3_3_8_2","unstructured":"Ceph. 2022. V0.94.10 HAMMER. Retrieved April 16, 2023 from https:\/\/docs.ceph.com\/docs\/master\/releases\/hammer\/."},{"issue":"3","key":"e_1_3_3_9_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2491464","volume":"31","year":"2013","unstructured":"James C. Corbett, Jeffrey Dean, Michael Epstein, Andrew Fikes, Christopher Frost, J. J. Furman, Sanjay Ghemawat, Andrey Gubarev, Christopher Heiser, Peter Hochschild, Wilson Hsieh, Sebastian Kanthak, Eugene Kogan, Hongyi Li, Alexander Lloyd, Sergey Melnik, David Mwaura, David Nagle, Sean Quinlan, Rajesh Rao, Lindsay Rolig, Yasushi Saito, Michal Szymaniak, Christopher Taylor, Ruth Wang, and Dale Woodford. 2013. Spanner: Google\u2019s globally distributed database. ACM Transactions on Computer Systems 31, 3 (2013), 1\u201322.","journal-title":"ACM Transactions on Computer Systems"},{"key":"e_1_3_3_10_2","volume-title":"C1Q 2021 HDD Update","author":"Coughlin Tom","year":"2021","unstructured":"Tom Coughlin. 2021. C1Q 2021 HDD Update. Retrieved April 16, 2023 from https:\/\/forbes.com\/sites\/tomcoughlin\/2021\/05\/04\/c1q-2021-hdd-update\/."},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/24.24571"},{"key":"e_1_3_3_12_2","volume-title":"MASCOTS 2001, Proceedings Ninth International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems","author":"Douceur John R.","year":"2001","unstructured":"John R. Douceur and Roger P. Wattenhofer. 2001. Large-scale simulation of replica placement algorithms for a serverless distributed file system. In MASCOTS 2001, Proceedings Ninth International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems."},{"key":"e_1_3_3_13_2","first-page":"197","volume-title":"FAST","author":"Dubnicki Cezary","year":"2009","unstructured":"Cezary Dubnicki, Leszek Gryz, Lukasz Heldt, Michal Kaczmarczyk, Wojciech Kilian, Przemyslaw Strzelczak, Jerzy Szczepkowski, Cristian Ungureanu, and Michal Welnicki. 2009. HYDRAstor: A scalable secondary storage. In FAST, Vol. 9, 197\u2013210."},{"key":"e_1_3_3_14_2","volume-title":"Administrator\u2019s Guide","author":"Foundation OpenStack","year":"2022","unstructured":"OpenStack Foundation. 2022. Administrator\u2019s Guide. Retrieved April 16, 2023 from https:\/\/docs.openstack.org\/swift\/latest\/admin_guide.html."},{"key":"e_1_3_3_15_2","volume-title":"Increasing Partition Power","author":"Foundation OpenStack","year":"2022","unstructured":"OpenStack Foundation. 2022. Increasing Partition Power. Retrieved April 16, 2023 from https:\/\/specs.openstack.org\/openstack\/swift-specs\/specs\/in_progress\/increasing_partition_power.html."},{"key":"e_1_3_3_16_2","unstructured":"Lukasz Golab Marios Hadjieleftheriou Howard Karloff and Barna Saha. 2013. Distributed data placement via graph partitioning. arXiv:1312.0285. Retrieved May 19 2023 from https:\/\/arxiv.org\/abs\/1312.0285."},{"key":"e_1_3_3_17_2","volume-title":"Building a Consistent Hashing Ring","author":"Holt Greg","year":"2011","unstructured":"Greg Holt. 2011. Building a Consistent Hashing Ring. Retrieved April 16, 2023 from https:\/\/docs.openstack.org\/swift\/latest\/ring_background.html."},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIT.2019.2902835"},{"volume-title":"StorageExperts What\u2019s new with HPE Scalable Object Storage with Scality RING?","year":"2018","key":"e_1_3_3_19_2","unstructured":"HPE. 2018. StorageExperts What\u2019s new with HPE Scalable Object Storage with Scality RING? Retrieved April 16, 2023 from https:\/\/community.hpe.com\/t5\/Around-the-Storage-Block\/What-s-new-with-HPE-Scalable-Object-Storage-with-Scality-RING\/ba-p\/7008183."},{"issue":"5","key":"e_1_3_3_20_2","doi-asserted-by":"crossref","first-page":"951","DOI":"10.1109\/TPDS.2012.196","article-title":"Load rebalancing for distributed file systems in clouds","volume":"24","author":"Hsiao Hung-Chang","year":"2012","unstructured":"Hung-Chang Hsiao, Hsueh-Yi Chung, Haiying Shen, and Yu-Chang Chao. 2012. Load rebalancing for distributed file systems in clouds. IEEE Transactions on Parallel and Distributed Systems 24, 5 (2012), 951\u2013962.","journal-title":"IEEE Transactions on Parallel and Distributed Systems"},{"key":"e_1_3_3_21_2","first-page":"15","volume-title":"Proceedings of the 2012 USENIX Annual Technical Conference.","author":"Huang Cheng","year":"2012","unstructured":"Cheng Huang, Huseyin Simitci, Yikang Xu, Aaron Ogus, Brad Calder, Parikshit Gopalan, Jin Li, and Sergey Yekhanin. 2012. Erasure coding in windows azure storage. In Proceedings of the 2012 USENIX Annual Technical Conference.15\u201326."},{"volume-title":"IBM Cloud Object Storage System\u2122, Storage Pool Expansion Guide","year":"2018","key":"e_1_3_3_22_2","unstructured":"IBM. 2018. IBM Cloud Object Storage System\u2122, Storage Pool Expansion Guide. Retrieved April 16, 2023 from https:\/\/www.ibm.com\/docs\/en\/STXNRM_3.14.1\/coss.doc\/pdfs\/storagePoolExpansion_bookmap.pdf."},{"key":"e_1_3_3_23_2","unstructured":"IDC. 2022. Enterprise Storage Systems Market Share. Retrieved May 1 2022 from https:\/\/idc.com\/promo\/enterprise-storage-systems."},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0166-218X(01)00264-5"},{"key":"e_1_3_3_25_2","first-page":"413","volume-title":"Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation.","author":"Kadekodi Saurabh","year":"2022","unstructured":"Saurabh Kadekodi, Francisco Maturana, Sanjith Athlur, Arif Merchant, K. V. Rashmi, and Gregory R. Ganger. 2022. Tiger: \\(\\lbrace\\) Disk-Adaptive \\(\\rbrace\\) redundancy without placement restrictions. In Proceedings of the 16th USENIX Symposium on Operating Systems Design and Implementation.413\u2013429."},{"key":"e_1_3_3_26_2","first-page":"369","volume-title":"Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation.","author":"Kadekodi Saurabh","year":"2020","unstructured":"Saurabh Kadekodi, Francisco Maturana, Suhas Jayaram Subramanya, Juncheng Yang, K. V. Rashmi, and Gregory R. Ganger. 2020. \\(\\lbrace\\) PACEMAKER \\(\\rbrace\\) : Avoiding \\(\\lbrace\\) HeART \\(\\rbrace\\) attacks in storage clusters with disk-adaptive redundancy. In Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation.369\u2013385."},{"key":"e_1_3_3_27_2","first-page":"345","volume-title":"Proceedings of the 17th USENIX Conference on File and Storage Technologies.","author":"Kadekodi Saurabh","year":"2019","unstructured":"Saurabh Kadekodi, K. V. Rashmi, and Gregory R. Ganger. 2019. Cluster storage systems gotta have HeART: Improving storage efficiency by exploiting disk-reliability heterogeneity. In Proceedings of the 17th USENIX Conference on File and Storage Technologies.345\u2013358."},{"key":"e_1_3_3_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIT.2014.2329872"},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/258533.258660"},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/1773912.1773922"},{"key":"e_1_3_3_31_2","first-page":"1","volume-title":"Proceedings of the 14th EuroSys Conference 2019","author":"Li Huiba","year":"2019","unstructured":"Huiba Li, Yiming Zhang, Dongsheng Li, Zhiming Zhang, Shengyun Liu, Peng Huang, Zheng Qin, Kai Chen, and Yongqiang Xiong. 2019. Ursa: Hybrid block storage for cloud-scale virtual disks. In Proceedings of the 14th EuroSys Conference 2019. 1\u201317."},{"key":"e_1_3_3_32_2","first-page":"192","volume-title":"CCNC","author":"Luby Michael","year":"2006","unstructured":"Michael Luby, Mark Watson, Tiago Gasiba, Thomas Stockhammer, and Wen Xu. 2006. Raptor codes for reliable download delivery in wireless broadcast systems. In CCNC 6 (2006), 192\u2013197."},{"key":"e_1_3_3_33_2","unstructured":"Julia Palmer Jerry Rozeman Chandra Mukhyala and Jeff Vogel. 2021. Magic Quadrant for Distributed File Systems and Object Storage. ID: G00738148. Retrieved May 19 2023 from https:\/\/www.gartner.com\/en\/documents\/4006429."},{"key":"e_1_3_3_34_2","first-page":"217","volume-title":"Proceedings of the 19th USENIX Conference on File and Storage Technologies.","year":"2021","unstructured":"Satadru Pan, Theano Stavrinos, Yunqiao Zhang, Atul Sikaria, Pavel Zakharov, Abhinav Sharma, Shiva Shankar P., Mike Shuey, Richard Wareing, Monika Gangapuram, Guanglei Cao, Christian Preseau, Pratap Singh, Kestutis Patiejunas, JR Tipton, Ethan Katz-Bassett, and Wyatt Lloyd. 2021. Facebook\u2019s tectonic filesystem: Efficiency from exascale. In Proceedings of the 19th USENIX Conference on File and Storage Technologies.217\u2013231."},{"key":"e_1_3_3_35_2","volume-title":"Energy Assisted Magnetic Recording Will Solve the Need for Capacity","author":"Paulsen John","year":"2021","unstructured":"John Paulsen. 2021. Energy Assisted Magnetic Recording Will Solve the Need for Capacity. Retrieved April 16, 2023 from https:\/\/blog.seagate.com\/enterprises\/energy-assisted-magnetic-recording-will-solve-the-need-for-capacity\/."},{"key":"e_1_3_3_36_2","doi-asserted-by":"publisher","DOI":"10.5555\/1594884"},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSST.2010.5496972"},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/2485732.2485737"},{"key":"e_1_3_3_39_2","doi-asserted-by":"publisher","DOI":"10.5555\/2591272.2591290"},{"key":"e_1_3_3_40_2","first-page":"1","volume-title":"Proceedings of the 18th USENIX Conference on File and Storage Technologies.","author":"Wang Li","year":"2020","unstructured":"Li Wang, Yiming Zhang, Jiawei Xu, and Guangtao Xue. 2020. MAPX: Controlled data migration in the expansion of decentralized object-based storage systems. In Proceedings of the 18th USENIX Conference on File and Storage Technologies.1\u201311."},{"key":"e_1_3_3_41_2","first-page":"188","volume-title":"Proceedings of the 2010 IEEE International Conference on Cluster Computing","author":"Wei Qingsong","year":"2010","unstructured":"Qingsong Wei, Bharadwaj Veeravalli, Bozhao Gong, Lingfang Zeng, and Dan Feng. 2010. CDRM: A cost-effective dynamic replication management scheme for cloud storage cluster. In Proceedings of the 2010 IEEE International Conference on Cluster Computing. IEEE, 188\u2013196."},{"key":"e_1_3_3_42_2","first-page":"307","volume-title":"Proceedings of the 7th Symposium on Operating Systems Design and Implementation","author":"Weil Sage A.","year":"2006","unstructured":"Sage A. Weil, Scott A. Brandt, Ethan L. Miller, Darrell D. E. Long, and Carlos Maltzahn. 2006. Ceph: A scalable, high-performance distributed file system. In Proceedings of the 7th Symposium on Operating Systems Design and Implementation. 307\u2013320."},{"key":"e_1_3_3_43_2","doi-asserted-by":"publisher","DOI":"10.1109\/SC.2006.19"},{"key":"e_1_3_3_44_2","first-page":"876","volume-title":"Proceedings of the 2017 IEEE International Parallel and Distributed Processing Symposium.","author":"Xie Wei","year":"2017","unstructured":"Wei Xie and Yong Chen. 2017. Elastic consistent hashing for distributed storage systems. In Proceedings of the 2017 IEEE International Parallel and Distributed Processing Symposium. IEEE, 876\u2013885."},{"key":"e_1_3_3_45_2","doi-asserted-by":"crossref","first-page":"144","DOI":"10.1109\/SRDS.2017.19","volume-title":"Proceedings of the 2017 IEEE 36th Symposium on Reliable Distributed Systems.","author":"Zhang Mi","year":"2017","unstructured":"Mi Zhang, Shujie Han, and Patrick PC Lee. 2017. A simulation analysis of reliability in erasure-coded data centers. In Proceedings of the 2017 IEEE 36th Symposium on Reliable Distributed Systems. IEEE, 144\u2013153."}],"container-title":["ACM Transactions on Storage"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3594543","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3594543","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T18:09:07Z","timestamp":1750183747000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3594543"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,19]]},"references-count":44,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2023,8,31]]}},"alternative-id":["10.1145\/3594543"],"URL":"https:\/\/doi.org\/10.1145\/3594543","relation":{},"ISSN":["1553-3077","1553-3093"],"issn-type":[{"type":"print","value":"1553-3077"},{"type":"electronic","value":"1553-3093"}],"subject":[],"published":{"date-parts":[[2023,6,19]]},"assertion":[{"value":"2022-05-05","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-03-17","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-06-19","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}