{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:25:50Z","timestamp":1750220750566,"version":"3.41.0"},"reference-count":83,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2020,2,29]],"date-time":"2020-02-29T00:00:00Z","timestamp":1582934400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Storage"],"published-print":{"date-parts":[[2020,2,29]]},"abstract":"<jats:p>Oracle File Storage Service (FSS) is an elastic filesystem provided as a managed NFS service. A pipelined Paxos implementation underpins a scalable block store that provides linearizable multipage limited-size transactions. Above the block store, a scalable B-tree holds filesystem metadata and provides linearizable multikey limited-size transactions. Self-validating B-tree nodes and housekeeping operations performed as separate transactions allow each key in a B-tree transaction to require only one page in the underlying block transaction. The filesystem provides snapshots by using versioned key-value pairs. The system is programmed using a nonblocking lock-free programming style. Presentation servers maintain no persistent local state making them scalable and easy to failover. A non-scalable Paxos-replicated hash table holds configuration information required to bootstrap the system. An additional B-tree provides conversational multi-key minitransactions for control-plane information. The system throughput can be predicted by comparing an estimate of the network bandwidth needed for replication to the network bandwidth provided by the hardware. Latency on an unloaded system is about 4\u00a0times higher than a Linux NFS server backed by NVMe, reflecting the cost of replication. FSS has been in production since January 2018 and holds tens of thousands of customer file systems comprising many petabytes of data.<\/jats:p>","DOI":"10.1145\/3377877","type":"journal-article","created":{"date-parts":[[2020,3,5]],"date-time":"2020-03-05T10:20:16Z","timestamp":1583403616000},"page":"1-29","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["Everyone Loves File"],"prefix":"10.1145","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6305-4290","authenticated-orcid":false,"given":"Bradley C.","family":"Kuszmaul","sequence":"first","affiliation":[{"name":"Oracle, MA, Massachusetts"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Matteo","family":"Frigo","sequence":"additional","affiliation":[{"name":"Oracle, MA, Massachusetts"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Justin Mazzola","family":"Paluska","sequence":"additional","affiliation":[{"name":"Oracle, MA, Massachusetts"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alexander (Sasha)","family":"Sandler","sequence":"additional","affiliation":[{"name":"Oracle, MA, Massachusetts"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2020,3,5]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/1629087.1629088"},{"volume-title":"Retrieved","year":"2018","author":"Alibaba","key":"e_1_2_1_2_1"},{"volume-title":"Proceedings of the Pacific Network Operators Group (PacNOG I) Workshop.","year":"2005","author":"Allen Hervey","key":"e_1_2_1_3_1"},{"volume-title":"Retrieved","year":"2018","author":"Amazon","key":"e_1_2_1_4_1"},{"volume-title":"Retrieved","year":"2018","author":"Amazon","key":"e_1_2_1_5_1"},{"volume-title":"Retrieved","year":"2018","author":"Amazon","key":"e_1_2_1_6_1"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/1465482.1465560"},{"key":"e_1_2_1_8_1","unstructured":"Apache Software Foundation. 2009. ZooKeeper Internals. Retrieved from https:\/\/zookeeper.apache.org\/doc\/r3.1.2\/zookeeperInternals.html.  Apache Software Foundation. 2009. ZooKeeper Internals. Retrieved from https:\/\/zookeeper.apache.org\/doc\/r3.1.2\/zookeeperInternals.html."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF00288683"},{"key":"e_1_2_1_10_1","unstructured":"Steve Best and Dave Kleikamp. 2000. JFS layout. IBM Developerworks. Retreived from http:\/\/jfs.sourceforge.net\/project\/pub\/jfslayout.pdf.  Steve Best and Dave Kleikamp. 2000. JFS layout. IBM Developerworks. Retreived from http:\/\/jfs.sourceforge.net\/project\/pub\/jfslayout.pdf."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1006\/jpdc.1996.0107"},{"volume-title":"Proceedings of the 1st USENIX Conference on Hot Topics in Parallelism (HotPar\u201909)","year":"2009","author":"Boehm J.","key":"e_1_2_1_12_1"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.5555\/1972457.1972472"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/321812.321815"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1137\/1.9781611973099.51"},{"volume-title":"Distributed Systems (2 ed.)","author":"Budhiraja Navin","key":"e_1_2_1_16_1"},{"volume-title":"NFS Version 3 Protocol Specification. IETF RFC","year":"1813","author":"Callaghan Brent","key":"e_1_2_1_17_1"},{"volume-title":"Proceedings of the 1st Dutch International Symposium on Linux.","year":"1994","author":"Card R\u00e9my","key":"e_1_2_1_18_1"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/1454456.1454466"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/1281100.1281103"},{"volume-title":"File systems fated for senescence? Nonsense, says science! In Proceedings of the 15th USENIX Conference on File and Storage Technologies (FAST\u201917). 45--58","author":"Conway Alexander","key":"e_1_2_1_21_1"},{"volume-title":"Proceedings of the 10th USENIX Symposium on Operating Systems Design and Implementation (OSDI\u201912)","year":"2012","author":"Corbett James C.","key":"e_1_2_1_22_1"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/5505.5508"},{"volume-title":"Proceedings of the 14th International Conference on Distributed Computing (DISC\u201900)","author":"Detlefs David L.","key":"e_1_2_1_24_1"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00446-002-0079-z"},{"key":"e_1_2_1_26_1","unstructured":"Matthew Dillon. 2008. The Hammer Filesystem. Retrieved from https:\/\/www.dragonflybsd.org\/hammer\/hammer.pdf.  Matthew Dillon. 2008. The Hammer Filesystem. Retrieved from https:\/\/www.dragonflybsd.org\/hammer\/hammer.pdf."},{"volume-title":"Proceedings of the 2006 Linux Symposium. 289--302","year":"2006","author":"Fasheh Mark","key":"e_1_2_1_27_1"},{"key":"e_1_2_1_28_1","unstructured":"Glustre 2005. GlusterFS. Retrieved from http:\/\/www.gluster.org.  Glustre 2005. GlusterFS. Retrieved from http:\/\/www.gluster.org."},{"volume-title":"Retrieved","year":"2012","author":"Google","key":"e_1_2_1_29_1"},{"volume-title":"Retrieved","year":"2018","author":"Google","key":"e_1_2_1_30_1"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/1806907.1806908"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1137\/0117039"},{"volume-title":"Transaction Processing: Concepts and Techniques. Morgan Kaufmann.","year":"1993","author":"Gray Jim","key":"e_1_2_1_33_1"},{"volume-title":"Operating Systems\u2014An Advanced Course","series-title":"Lecture Notes in Computer Science","author":"Gray Jim N.","key":"e_1_2_1_34_1"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/949305.949340"},{"key":"e_1_2_1_36_1","unstructured":"HDFS 2012. Add support for Variable length block. HDFS Ticket. Retrieved from https:\/\/issues.apache.org\/jira\/browse\/HDFS-3689.  HDFS 2012. Add support for Variable length block. HDFS Ticket. Retrieved from https:\/\/issues.apache.org\/jira\/browse\/HDFS-3689."},{"key":"e_1_2_1_37_1","unstructured":"HDFS 2013. HDFS Architecture. Retrieved from http:\/\/hadoop.apache.org\/docs\/current\/hadoop-project-dist\/hadoop-hdfs\/HdfsDesign.html#Large_Data_Sets.  HDFS 2013. HDFS Architecture. Retrieved from http:\/\/hadoop.apache.org\/docs\/current\/hadoop-project-dist\/hadoop-hdfs\/HdfsDesign.html#Large_Data_Sets."},{"key":"e_1_2_1_38_1","first-page":"1","article-title":"Wait-free synchronizatoin","volume":"11","author":"Herlihy Maurice","year":"1991","journal-title":"ACM Trans. Program. Lang. Syst."},{"volume-title":"Proceedings of the 20th Annual International Symposium on Computer Architecture (ISCA\u201993)","author":"Herlihy M.","key":"e_1_2_1_39_1"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/78969.78972"},{"volume-title":"Proceedings of the USENIX","year":"1994","author":"Hitz Dave","key":"e_1_2_1_41_1"},{"key":"e_1_2_1_42_1","unstructured":"Valentin H\u00f6bel. 2016. LizardFS: Software-Defined Storage As It Should Be. Retrieved from https:\/\/www.golem.de\/news\/lizardfs-software-defined-storage-wie-es-sein-soll-1604-119518.html.  Valentin H\u00f6bel. 2016. LizardFS: Software-Defined Storage As It Should Be. Retrieved from https:\/\/www.golem.de\/news\/lizardfs-software-defined-storage-wie-es-sein-soll-1604-119518.html."},{"volume-title":"Data File Handbook.","author":"IBM.","key":"e_1_2_1_43_1"},{"key":"e_1_2_1_44_1","unstructured":"IBM 1983. IBM System\/370 Extended Architecture\u2014Principles of Operation. IBM. Retrieved from https:\/\/archive.org\/details\/bitsavers_ibm370prinrinciplesofOperationMar83_40542805.  IBM 1983. IBM System\/370 Extended Architecture\u2014Principles of Operation. IBM. Retrieved from https:\/\/archive.org\/details\/bitsavers_ibm370prinrinciplesofOperationMar83_40542805."},{"volume-title":"HFS Plus","author":"Apple Inc. 2004.","key":"e_1_2_1_45_1"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/2798729"},{"volume-title":"Ceph: A Linux petabyte-scale distributed file system.","year":"2004","author":"Jones M. Tim","key":"e_1_2_1_48_1"},{"volume-title":"Copy on Write Based File Systems Performance Analysis and Implementation. Master\u2019s thesis. Department of Informatics","author":"Kasampalis Sakis","key":"e_1_2_1_49_1"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/279227.279229"},{"key":"e_1_2_1_51_1","article-title":"Paxos made simple","volume":"32","author":"Lamport Leslie","year":"2001","journal-title":"ACM SIGACT News"},{"volume-title":"Distributed Systems\u2014Architecture and Implementation.","author":"Lampson Butler","key":"e_1_2_1_52_1"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/319628.319663"},{"volume-title":"Proceedings of the The 2nd ACM SIGPLAN Workshop on Transactional Computing.","year":"2007","author":"Lev Yossi","key":"e_1_2_1_54_1"},{"key":"e_1_2_1_55_1","unstructured":"A. J. Lewis. 2002. LVM HOWTO. Retrieved from http:\/\/tldp.org\/HOWTO\/LVM-HOWTO\/.  A. J. Lewis. 2002. LVM HOWTO. Retrieved from http:\/\/tldp.org\/HOWTO\/LVM-HOWTO\/."},{"volume-title":"Single and multi-site recovery facilities","author":"Lindsay Bruce G.","key":"e_1_2_1_56_1"},{"key":"e_1_2_1_58_1","unstructured":"Lustre 2003. The Lustre File System. Retrieved from lustre.org.  Lustre 2003. The Lustre File System. Retrieved from lustre.org."},{"volume-title":"Proceedings of the Linux Symposium.","year":"2007","author":"Mathur Avantika","key":"e_1_2_1_59_1"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/989.990"},{"volume-title":"Microsoft Azure Blob Storage.","author":"Microsoft","key":"e_1_2_1_61_1"},{"key":"e_1_2_1_62_1","unstructured":"Microsoft 2018. Microsoft SMB Protocol and CIFS Protocol Overview. Retrieved from https:\/\/docs.microsoft.com\/en-us\/windows\/desktop\/FileIO\/microsoft-smb-protocol-and-cifs-protocol-overview.  Microsoft 2018. Microsoft SMB Protocol and CIFS Protocol Overview. Retrieved from https:\/\/docs.microsoft.com\/en-us\/windows\/desktop\/FileIO\/microsoft-smb-protocol-and-cifs-protocol-overview."},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/96267.96279"},{"key":"e_1_2_1_64_1","unstructured":"Moose 2018. MooseFS Fact Sheet. Retrieved from https:\/\/moosefs.com\/factsheet\/.  Moose 2018. MooseFS Fact Sheet. Retrieved from https:\/\/moosefs.com\/factsheet\/."},{"volume-title":"Proceedings of the 7th Annual ACM Symposium on Principles of Distributed Computing (PODC\u201988)","year":"1988","author":"Oki Brian","key":"e_1_2_1_65_1"},{"volume-title":"Proceedings of the 2014 USENIX Annual Technical Conference (ATC\u201914)","year":"2014","author":"Ongaro Diego","key":"e_1_2_1_66_1"},{"key":"e_1_2_1_67_1","unstructured":"Diego Ongaro and John Ousterhout. 2014. In Search of an Understandable Consensus Algorithm (Extended Version). Retrieved from https:\/\/raft.github.io\/raft.pdf. Extended version of Reference [66].  Diego Ongaro and John Ousterhout. 2014. In Search of an Understandable Consensus Algorithm (Extended Version). Retrieved from https:\/\/raft.github.io\/raft.pdf. Extended version of Reference [66]."},{"key":"e_1_2_1_68_1","unstructured":"Oracle 2016. Oracle Cloud Infrastructure Block Volumes. Retrieved from https:\/\/cloud.oracle.com\/en_US\/storage\/block-volume\/features.  Oracle 2016. Oracle Cloud Infrastructure Block Volumes. Retrieved from https:\/\/cloud.oracle.com\/en_US\/storage\/block-volume\/features."},{"volume-title":"Black","year":"2001","author":"Ramakrishnan K. K.","key":"e_1_2_1_69_1"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1137\/0108018"},{"key":"e_1_2_1_71_1","unstructured":"Hans T. Reiser. 2006. Reiser4. Retrieved July 6 2006 from https:\/\/web.archive.org\/web\/20060706032252 http:\/\/www.namesys.com:80\/.  Hans T. Reiser. 2006. Reiser4. Retrieved July 6 2006 from https:\/\/web.archive.org\/web\/20060706032252 http:\/\/www.namesys.com:80\/."},{"volume-title":"Proceedings of the USENIX Annual Technical Conference. 145--156","year":"2013","author":"Ren Kai","key":"e_1_2_1_72_1"},{"key":"e_1_2_1_73_1","first-page":"4","article-title":"B-trees, shadowing, and clones","volume":"3","author":"Rodeh Ohad","year":"2008","journal-title":"ACM Trans. Comput. Logic"},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1145\/2501620.2501623"},{"volume-title":"Inside Win2K NTFS, Part 1. ITProToday (22","year":"2000","author":"Russinovich Mark","key":"e_1_2_1_75_1"},{"key":"e_1_2_1_76_1","doi-asserted-by":"crossref","unstructured":"Spencer Shepler Brent Callaghan David Robinson Robert Thurlow Carl Beame Mike Eisler and David Noveck. 2003. Network File System (NFS) version 4 Protocol. IETF RFC 3530. Retrieved from https:\/\/www.ietf.org\/html\/rfc3530.  Spencer Shepler Brent Callaghan David Robinson Robert Thurlow Carl Beame Mike Eisler and David Noveck. 2003. Network File System (NFS) version 4 Protocol. IETF RFC 3530. Retrieved from https:\/\/www.ietf.org\/html\/rfc3530.","DOI":"10.17487\/rfc3530"},{"key":"e_1_2_1_77_1","unstructured":"Chris Siebenmann. 2017. ZFS\u2019s recordsize Holes In Files and Partial Blocks. Retrieved from https:\/\/utcc.utoronto.ca\/cks\/space\/blog\/solaris\/ZFSFilePartialAndHoleStorage.  Chris Siebenmann. 2017. ZFS\u2019s recordsize Holes In Files and Partial Blocks. Retrieved from https:\/\/utcc.utoronto.ca\/cks\/space\/blog\/solaris\/ZFSFilePartialAndHoleStorage."},{"volume-title":"Retrieved","year":"2018","author":"Siebenmann Chris","key":"e_1_2_1_78_1"},{"volume-title":"Retrieved","year":"2009","author":"Stacey Jon","key":"e_1_2_1_79_1"},{"volume-title":"Fast Retransmit and Fast Recovery Algorithms. IETF RFC 2001","year":"1997","author":"Stevens W. Richard","key":"e_1_2_1_80_1"},{"key":"e_1_2_1_81_1","unstructured":"Sun Microsystems. 2006. ZFS On-Disk Specification\u2014draft. Retrieved from http:\/\/www.giis.co.in\/Zfs_ondiskformat.pdf.  Sun Microsystems. 2006. ZFS On-Disk Specification\u2014draft. Retrieved from http:\/\/www.giis.co.in\/Zfs_ondiskformat.pdf."},{"volume-title":"Proceedings of the 1996 USENIX Annual Technical Conference (ATC\u201996)","year":"1996","author":"Sweeny Adam","key":"e_1_2_1_82_1"},{"volume-title":"Proceedings of the 27th ACM Symposium on Parallelism in Algorithms and Architectures (SPAA\u201915)","author":"Xiang Lingxiang","key":"e_1_2_1_83_1"},{"key":"e_1_2_1_84_1","doi-asserted-by":"publisher","DOI":"10.1145\/3032969"},{"volume-title":"Proceedings of the 16th USENIX Conference on File and Storage Technologies (FAST\u201918)","year":"2018","author":"Zhan Yang","key":"e_1_2_1_85_1"}],"container-title":["ACM Transactions on Storage"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3377877","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3377877","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:38:52Z","timestamp":1750199932000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3377877"}},"subtitle":["Oracle File Storage Service"],"short-title":[],"issued":{"date-parts":[[2020,2,29]]},"references-count":83,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2020,2,29]]}},"alternative-id":["10.1145\/3377877"],"URL":"https:\/\/doi.org\/10.1145\/3377877","relation":{},"ISSN":["1553-3077","1553-3093"],"issn-type":[{"type":"print","value":"1553-3077"},{"type":"electronic","value":"1553-3093"}],"subject":[],"published":{"date-parts":[[2020,2,29]]},"assertion":[{"value":"2019-10-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-01-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-03-05","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}