{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,24]],"date-time":"2026-01-24T02:58:24Z","timestamp":1769223504991,"version":"3.49.0"},"reference-count":14,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2021,11,9]],"date-time":"2021-11-09T00:00:00Z","timestamp":1636416000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,11,9]],"date-time":"2021-11-09T00:00:00Z","timestamp":1636416000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100012470","name":"CERN","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100012470","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Comput Softw Big Sci"],"published-print":{"date-parts":[[2021,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>CephFS is a network filesystem built upon the Reliable Autonomic Distributed Object Store (RADOS). At CERN we have demonstrated its reliability and elasticity while operating several 100-to-1000TB clusters which provide NFS-like storage to infrastructure applications and services. At the same time, our lab developed EOS to offer high performance 100PB-scale storage for the LHC at extremely low costs while also supporting the complete set of security and functional APIs required by the particle-physics user community. This work seeks to evaluate the performance of CephFS on this cost-optimized hardware when it is combined with EOS to support the missing functionalities. To this end, we have setup a proof-of-concept Ceph Octopus cluster on high-density JBOD servers (840\u00a0TB each) with 100Gig-E networking. The system uses EOS to provide an overlayed namespace and protocol gateways for HTTP(S) and XROOTD, and uses CephFS as an erasure-coded object storage backend. The solution also enables operators to aggregate several CephFS instances and adds features, such as third-party-copy, SciTokens, and high-level user and quota management. Using simple benchmarks we measure the cost\/performance tradeoffs of different erasure-coding layouts, as well as the network overheads of these coding schemes. We demonstrate some relevant limitations of the CephFS metadata server and offer improved tunings which can be generally applicable. To conclude, we reflect on the advantages and drawbacks related to this architecture, such as RADOS-level free space requirements and double-network penalties, and offer ideas for improvements in the future.<\/jats:p>","DOI":"10.1007\/s41781-021-00071-1","type":"journal-article","created":{"date-parts":[[2021,11,9]],"date-time":"2021-11-09T15:06:26Z","timestamp":1636470386000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Evaluating CephFS Performance vs. Cost on High-Density Commodity Disk Servers"],"prefix":"10.1007","volume":"5","author":[{"given":"Andreas J.","family":"Peters","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Daniel C.","family":"van der Ster","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2021,11,9]]},"reference":[{"issue":"1","key":"71_CR1","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s41781-018-0018-8","volume":"3","author":"J Albrecht","year":"2019","unstructured":"Albrecht J et al (2019) A roadmap for HEP software and computing R&D for the 2020s. Comput Softw Big Sci 3(1):1\u201349","journal-title":"Comput Softw Big Sci"},{"key":"71_CR2","unstructured":"Carlson M et al (2014) Software defined storage. Storage Networking Industry Assoc, San Francisco. https:\/\/www.snia.org\/education\/whitepapers"},{"issue":"6","key":"71_CR3","doi-asserted-by":"publisher","first-page":"062051","DOI":"10.1088\/1742-6596\/898\/6\/062051","volume":"898","author":"A Dewhurst","year":"2017","unstructured":"Dewhurst A et al (2017) The deployment of a large scale object store at the RAL Tier-1. J Phys Conf Ser 898(6):062051","journal-title":"J Phys Conf Ser"},{"key":"71_CR4","unstructured":"Weil SA et al (2006) Ceph: a scalable, high-performance distributed file system. In: Proceedings of the 7th symposium on operating systems design and implementation, pp 307\u2013320"},{"issue":"4","key":"71_CR5","doi-asserted-by":"publisher","first-page":"042042","DOI":"10.1088\/1742-6596\/664\/4\/042042","volume":"664","author":"AJ Peters","year":"2015","unstructured":"Peters AJ, Sindrilaru EA, Adde G (2015) EOS as the present and future solution for data storage at CERN. J Phys Conf Ser 664(4):042042","journal-title":"J Phys Conf Ser"},{"key":"71_CR6","unstructured":"QuarkDB\u2014a highly available datastore. https:\/\/github.com\/gbitzes\/QuarkDB. Accessed 1 May 2021"},{"key":"71_CR7","doi-asserted-by":"crossref","unstructured":"Weil SA et al (2007) Rados: a scalable, reliable storage service for petabyte-scale storage clusters. In: Proceedings of the 2nd international workshop on Petascale data storage: held in conjunction with Supercomputing\u201907, pp 35\u201344","DOI":"10.1145\/1374596.1374606"},{"key":"71_CR8","doi-asserted-by":"crossref","unstructured":"Weil SA et al (2006) CRUSH: controlled, scalable, decentralized placement of replicated data. In: SC\u201906: Proceedings of the 2006 ACM\/IEEE conference on supercomputing, pp 31\u201331","DOI":"10.1109\/SC.2006.19"},{"key":"71_CR9","doi-asserted-by":"crossref","unstructured":"Yoo Andy B, Jette Morris A, Grondona Mark (2003) Slurm: simple linux utility for resource management. In: Workshop on job scheduling strategies for parallel processing, pp 44\u201360","DOI":"10.1007\/10968987_3"},{"issue":"3","key":"71_CR10","first-page":"38","volume":"55","author":"O Sefraoui","year":"2012","unstructured":"Sefraoui O, Aissaoui M, Eleuldj M (2012) OpenStack: toward an open-source solution for cloud computing. Int J Comput Appl 55(3):38\u201342","journal-title":"Int J Comput Appl"},{"key":"71_CR11","doi-asserted-by":"crossref","unstructured":"Withers A et al (2018) SciTokens: capability-based secure access to remote scientific data. In: Proceedings of the practice and experience on advanced research computing, pp 1\u20138","DOI":"10.1145\/3219104.3219135"},{"key":"71_CR12","doi-asserted-by":"crossref","unstructured":"Aguado CA et al (2020) CERN\u2019s identity and access management: a journey to open source. In: EPJ Web of Conferences, vol. 245, p 03012","DOI":"10.1051\/epjconf\/202024503012"},{"issue":"4.3","key":"71_CR13","first-page":"348","volume":"1","author":"A Dorigo","year":"2005","unstructured":"Dorigo A et al (2005) XROOTD-A highly scalable architecture for data access. WSEAS Trans Comput 1(4.3):348\u2013353","journal-title":"WSEAS Trans Comput"},{"key":"71_CR14","unstructured":"Github Ceph Pull Request: mds: update defaults for recall configs. https:\/\/github.com\/ceph\/ceph\/pull\/38574. Accessed 1 May 2021"}],"container-title":["Computing and Software for Big Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s41781-021-00071-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s41781-021-00071-1\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s41781-021-00071-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,1,24]],"date-time":"2022-01-24T07:25:09Z","timestamp":1643009109000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s41781-021-00071-1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,11,9]]},"references-count":14,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2021,12]]}},"alternative-id":["71"],"URL":"https:\/\/doi.org\/10.1007\/s41781-021-00071-1","relation":{},"ISSN":["2510-2036","2510-2044"],"issn-type":[{"value":"2510-2036","type":"print"},{"value":"2510-2044","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,11,9]]},"assertion":[{"value":"1 July 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"27 September 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 November 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}],"article-number":"25"}}