{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,6]],"date-time":"2026-01-06T13:05:06Z","timestamp":1767704706762,"version":"3.41.0"},"reference-count":95,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2020,10,1]],"date-time":"2020-10-01T00:00:00Z","timestamp":1601510400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"the National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["61772300, 61832011"],"award-info":[{"award-number":["61772300, 61832011"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"National Key Research and Development Program of China","award":["2018YFB1003301"],"award-info":[{"award-number":["2018YFB1003301"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Storage"],"published-print":{"date-parts":[[2020,11,30]]},"abstract":"<jats:p>The rapidly increasing data in recent years requires the datacenter infrastructure to store and process data with extremely high throughput and low latency. Fortunately, persistent memory (PM) and RDMA technologies bring new opportunities towards this goal. Both of them are capable of delivering more than 10\u00a0GB\/s of bandwidth and sub-microsecond latency. However, our past experiences and recent studies show that it is non-trivial to build an efficient and distributed storage system with such new hardware. In this article, we design and implement TH-DPMS (&lt;underline&gt;T&lt;\/underline&gt;sing&lt;underline&gt;H&lt;\/underline&gt;ua &lt;underline&gt;D&lt;\/underline&gt;istributed &lt;underline&gt;P&lt;\/underline&gt;ersistent &lt;underline&gt;M&lt;\/underline&gt;emory &lt;underline&gt;S&lt;\/underline&gt;ystem) based on persistent memory and RDMA, which unifies the memory, file system, and key-value interface in a single system. TH-DPMS is designed based on a unified distributed persistent memory abstract, pDSM. pDSM acts as a generic layer to connect the PMs of different storage nodes via high-speed RDMA network and organizes them into a global shared address space. It provides the fundamental functionalities, including global address management, space management, fault tolerance, and crash consistency guarantees. Applications are enabled to access pDSM with a group of flexible and easy-to-use APIs by using either raw read\/write interfaces or the transactional ones with ACID guarantees. Based on pDSM, we implement a distributed file system and a key-value store named pDFS and pDKVS, respectively. Together, they uphold TH-DPMS with high-performance, low-latency, and fault-tolerant data storage. We evaluate TH-DPMS with both micro-benchmarks and real-world memory-intensive workloads. Experimental results show that TH-DPMS is capable of delivering an aggregated bandwidth of 120\u00a0GB\/s with 6 nodes. When processing memory-intensive workloads such as YCSB and Graph500, TH-DPMS improves the performance by one order of magnitude compared to existing systems and keeps consistent high efficiency when the workload size grows to multiple terabytes.<\/jats:p>","DOI":"10.1145\/3412852","type":"journal-article","created":{"date-parts":[[2020,10,2]],"date-time":"2020-10-02T04:14:07Z","timestamp":1601612047000},"page":"1-31","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":17,"title":["TH-DPMS"],"prefix":"10.1145","volume":"16","author":[{"given":"Jiwu","family":"Shu","sequence":"first","affiliation":[{"name":"Tsinghua University, Beijing, China"}]},{"given":"Youmin","family":"Chen","sequence":"additional","affiliation":[{"name":"Tsinghua University, Beijing, China"}]},{"given":"Qing","family":"Wang","sequence":"additional","affiliation":[{"name":"Tsinghua University, Beijing, China"}]},{"given":"Bohong","family":"Zhu","sequence":"additional","affiliation":[{"name":"Tsinghua University, Beijing, China"}]},{"given":"Junru","family":"Li","sequence":"additional","affiliation":[{"name":"Tsinghua University, Beijing, China"}]},{"given":"Youyou","family":"Lu","sequence":"additional","affiliation":[{"name":"Tsinghua University, Beijing, China"}]}],"member":"320","published-online":{"date-parts":[[2020,10]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"Mellanox Technologies. 2019. ConnectX-6 VPI Card. Retrieved from https:\/\/www.mellanox.com\/related-docs\/prod_adapter_cards\/PB_ConnectX-6_VPI_Card.pdf.  Mellanox Technologies. 2019. ConnectX-6 VPI Card. Retrieved from https:\/\/www.mellanox.com\/related-docs\/prod_adapter_cards\/PB_ConnectX-6_VPI_Card.pdf."},{"key":"e_1_2_1_2_1","unstructured":"Intel Corporation. 2019. The Distributed Asynchronous Object Storage. Retrieved from https:\/\/daos-stack.github.io\/.  Intel Corporation. 2019. The Distributed Asynchronous Object Storage. Retrieved from https:\/\/daos-stack.github.io\/."},{"key":"e_1_2_1_3_1","unstructured":"Red Hat. Inc. 2019. GlusterFS. Retrieved from https:\/\/www.gluster.org\/.  Red Hat. Inc. 2019. GlusterFS. Retrieved from https:\/\/www.gluster.org\/."},{"key":"e_1_2_1_4_1","unstructured":"Intel Corporation. 2019. Intel Optane DC Persistent Memory. Retrieved from https:\/\/www.intel.com\/content\/www\/us\/en\/architecture-and-technology\/optane-dc-persistent-memory.html.  Intel Corporation. 2019. Intel Optane DC Persistent Memory. Retrieved from https:\/\/www.intel.com\/content\/www\/us\/en\/architecture-and-technology\/optane-dc-persistent-memory.html."},{"key":"e_1_2_1_5_1","unstructured":"IDC. 2020. The Digital Universe of Opportunities: Rich Data and the Increasing Value of the Internet of Things. Retrieved from https:\/\/www.emc.com\/leadership\/digital-universe\/2014iview\/executive-summary.htm.  IDC. 2020. The Digital Universe of Opportunities: Rich Data and the Increasing Value of the Internet of Things. Retrieved from https:\/\/www.emc.com\/leadership\/digital-universe\/2014iview\/executive-summary.htm."},{"key":"e_1_2_1_6_1","unstructured":"Berkeley Architecture Research. 2020. The Firebox Project. Retrieved from https:\/\/bar.eecs.berkeley.edu\/projects\/firebox.html.  Berkeley Architecture Research. 2020. The Firebox Project. Retrieved from https:\/\/bar.eecs.berkeley.edu\/projects\/firebox.html."},{"key":"e_1_2_1_7_1","unstructured":"Intel Corporation. 2020. Intel Data Direct I\/O Technology. Retrieved from https:\/\/www.intel.com\/content\/www\/us\/en\/io\/data-direct-i-o-technology.html.  Intel Corporation. 2020. Intel Data Direct I\/O Technology. Retrieved from https:\/\/www.intel.com\/content\/www\/us\/en\/io\/data-direct-i-o-technology.html."},{"key":"e_1_2_1_8_1","unstructured":"HP Development Company. 2020. The Machine Project. Retrieved from https:\/\/www.hpl.hp.com\/research\/systems-research\/themachine.  HP Development Company. 2020. The Machine Project. Retrieved from https:\/\/www.hpl.hp.com\/research\/systems-research\/themachine."},{"key":"e_1_2_1_9_1","volume-title":"PMDK: Persistent Memory Development Kit.","author":"Intel Corporation","year":"2020","unstructured":"Intel Corporation . 2020 . PMDK: Persistent Memory Development Kit. Retrieved from https:\/\/github.com\/pmem\/pmdk. Intel Corporation. 2020. PMDK: Persistent Memory Development Kit. Retrieved from https:\/\/github.com\/pmem\/pmdk."},{"key":"e_1_2_1_10_1","unstructured":"Intel Corporation. 2020. pmemkv. Retrieved from https:\/\/github.com\/pmem\/pmemkv\/.  Intel Corporation. 2020. pmemkv. Retrieved from https:\/\/github.com\/pmem\/pmemkv\/."},{"key":"e_1_2_1_11_1","unstructured":"Redis Labs. 2020. Redis. Retrieved from https:\/\/redis.io\/.  Redis Labs. 2020. Redis. Retrieved from https:\/\/redis.io\/."},{"key":"e_1_2_1_12_1","volume-title":"Proceedings of the USENIX Annual Technical Conference (USENIX ATC\u201918)","author":"Aguilera Marcos K.","year":"2018","unstructured":"Marcos K. Aguilera , Nadav Amit , Irina Calciu , Xavier Deguillard , Jayneel Gandhi , Stanko Novakovi\u0107 , Arun Ramanathan , Pratap Subrahmanyam , Lalith Suresh , Kiran Tati , Rajesh Venkatasubramanian , and Michael Wei . 2018 . Remote regions: A simple abstraction for remote memory . In Proceedings of the USENIX Annual Technical Conference (USENIX ATC\u201918) . USENIX Association, 775--787. Retrieved from https:\/\/www.usenix.org\/conference\/atc18\/presentation\/aguilera. Marcos K. Aguilera, Nadav Amit, Irina Calciu, Xavier Deguillard, Jayneel Gandhi, Stanko Novakovi\u0107, Arun Ramanathan, Pratap Subrahmanyam, Lalith Suresh, Kiran Tati, Rajesh Venkatasubramanian, and Michael Wei. 2018. Remote regions: A simple abstraction for remote memory. In Proceedings of the USENIX Annual Technical Conference (USENIX ATC\u201918). USENIX Association, 775--787. Retrieved from https:\/\/www.usenix.org\/conference\/atc18\/presentation\/aguilera."},{"volume-title":"Proceedings of the ACM SIGMOD International Conference on Management of Data. ACM, 707--722","author":"Arulraj Joy","key":"e_1_2_1_13_1","unstructured":"Joy Arulraj , Andrew Pavlo , and Subramanya R. Dulloor . 2015. Let\u2019s talk about storage 8 recovery methods for non-volatile memory database systems . In Proceedings of the ACM SIGMOD International Conference on Management of Data. ACM, 707--722 . Joy Arulraj, Andrew Pavlo, and Subramanya R. Dulloor. 2015. Let\u2019s talk about storage 8 recovery methods for non-volatile memory database systems. In Proceedings of the ACM SIGMOD International Conference on Management of Data. ACM, 707--722."},{"key":"e_1_2_1_14_1","volume-title":"Proceedings of the IEEE International Electron Devices Meeting. IEEE, 587--590","author":"Baek I. G.","year":"2004","unstructured":"I. G. Baek , M. S. Lee , S. Seo , M. J. Lee , D. H. Seo , D.-S. Suh , J. C. Park , S. O. Park , H. S. Kim , I. K. Yoo , et\u00a0al. 2004 . Highly scalable nonvolatile resistive memory using simple binary oxide driven by asymmetric unipolar voltage pulses . In Proceedings of the IEEE International Electron Devices Meeting. IEEE, 587--590 . I. G. Baek, M. S. Lee, S. Seo, M. J. Lee, D. H. Seo, D.-S. Suh, J. C. Park, S. O. Park, H. S. Kim, I. K. Yoo, et\u00a0al. 2004. Highly scalable nonvolatile resistive memory using simple binary oxide driven by asymmetric unipolar voltage pulses. In Proceedings of the IEEE International Electron Devices Meeting. IEEE, 587--590."},{"volume-title":"Proceedings of the 9th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS\u201900)","author":"Berger Emery D.","key":"e_1_2_1_15_1","unstructured":"Emery D. Berger , Kathryn S. McKinley , Robert D. Blumofe , and Paul R. Wilson . 2000. Hoard: A scalable memory allocator for multithreaded applications . In Proceedings of the 9th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS\u201900) . Association for Computing Machinery, New York, NY, 117--128. DOI:https:\/\/doi.org\/10.1145\/378993.379232 10.1145\/378993.379232 Emery D. Berger, Kathryn S. McKinley, Robert D. Blumofe, and Paul R. Wilson. 2000. Hoard: A scalable memory allocator for multithreaded applications. In Proceedings of the 9th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS\u201900). Association for Computing Machinery, New York, NY, 117--128. DOI:https:\/\/doi.org\/10.1145\/378993.379232"},{"key":"e_1_2_1_16_1","unstructured":"Silas Boyd-Wickizer M. Frans Kaashoek Robert Morris and Nickolai Zeldovich. 2014. OpLog: A library for scaling update-heavy data structures. Retrieved from https:\/\/people.csail.mit.edu\/nickolai\/papers\/boyd-wickizer-oplog-tr.pdf.  Silas Boyd-Wickizer M. Frans Kaashoek Robert Morris and Nickolai Zeldovich. 2014. OpLog: A library for scaling update-heavy data structures. Retrieved from https:\/\/people.csail.mit.edu\/nickolai\/papers\/boyd-wickizer-oplog-tr.pdf."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/210126.210127"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.14778\/2752939.2752947"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/TC.2018.2870137"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3302424.3303968"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3373376.3378515"},{"key":"e_1_2_1_22_1","unstructured":"Youmin Chen Youyou Lu Bohong Zhu and Jiwu Shu. 2019. Kernel\/User-level Collaborative Persistent Memory File System with Efficiency and Protection. arxiv:cs.OS\/1908.10740  Youmin Chen Youyou Lu Bohong Zhu and Jiwu Shu. 2019. Kernel\/User-level Collaborative Persistent Memory File System with Efficiency and Protection. arxiv:cs.OS\/1908.10740"},{"key":"e_1_2_1_23_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3204454","article-title":"HiNFS: A persistent memory file system with both buffering and direct-access","volume":"14","author":"Chen Youmin","year":"2018","unstructured":"Youmin Chen , Jiwu Shu , Jiaxin Ou , and Youyou Lu . 2018 . HiNFS: A persistent memory file system with both buffering and direct-access . ACM Trans. Storage 14 , 1 (Apr. 2018). DOI:https:\/\/doi.org\/10.1145\/3204454 10.1145\/3204454 Youmin Chen, Jiwu Shu, Jiaxin Ou, and Youyou Lu. 2018. HiNFS: A persistent memory file system with both buffering and direct-access. ACM Trans. Storage 14, 1 (Apr. 2018). DOI:https:\/\/doi.org\/10.1145\/3204454","journal-title":"ACM Trans. Storage"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/1950365.1950380"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/1629575.1629589"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/1807128.1807152"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3341301.3359637"},{"key":"e_1_2_1_28_1","volume-title":"Proceedings of the 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI\u201914)","author":"Dragojevi\u0107 Aleksandar","year":"2014","unstructured":"Aleksandar Dragojevi\u0107 , Dushyanth Narayanan , Miguel Castro , and Orion Hodson . 2014 . FaRM: Fast remote memory . In Proceedings of the 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI\u201914) . 401--414. Aleksandar Dragojevi\u0107, Dushyanth Narayanan, Miguel Castro, and Orion Hodson. 2014. FaRM: Fast remote memory. In Proceedings of the 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI\u201914). 401--414."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/2815400.2815425"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2592798.2592814"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/2094114.2094126"},{"key":"e_1_2_1_32_1","unstructured":"Sanjay Ghemawat and Jeff Dean. 2011. LevelDB. Retrieved from https:\/\/github.com\/google\/leveldb.  Sanjay Ghemawat and Jeff Dean. 2011. LevelDB. Retrieved from https:\/\/github.com\/google\/leveldb."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3224419"},{"key":"e_1_2_1_34_1","volume-title":"A survey on tiering and caching in high-performance storage systems. arXiv preprint arXiv:1904.11560","author":"Hoseinzadeh Morteza","year":"2019","unstructured":"Morteza Hoseinzadeh . 2019. A survey on tiering and caching in high-performance storage systems. arXiv preprint arXiv:1904.11560 ( 2019 ). Morteza Hoseinzadeh. 2019. A survey on tiering and caching in high-performance storage systems. arXiv preprint arXiv:1904.11560 (2019)."},{"key":"e_1_2_1_35_1","volume-title":"Proceedings of the 16th USENIX Conference on File and Storage Technologies (FAST\u201918)","author":"Hwang Deukyeon","year":"2018","unstructured":"Deukyeon Hwang , Wook-Hee Kim , Youjip Won , and Beomseok Nam . 2018 . Endurable transient inconsistency in byte-addressable persistent B+-tree . In Proceedings of the 16th USENIX Conference on File and Storage Technologies (FAST\u201918) . 187. Deukyeon Hwang, Wook-Hee Kim, Youjip Won, and Beomseok Nam. 2018. Endurable transient inconsistency in byte-addressable persistent B+-tree. In Proceedings of the 16th USENIX Conference on File and Storage Technologies (FAST\u201918). 187."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2629619"},{"volume-title":"Proceedings of the International Conference on Supercomputing (ICS\u201916)","author":"Islam Nusrat Sharmin","key":"e_1_2_1_37_1","unstructured":"Nusrat Sharmin Islam , Md. Wasi-ur Rahman , Xiaoyi Lu , and Dhabaleswar K. Panda . 2016. High performance design for HDFS with byte-addressability of NVM and RDMA . In Proceedings of the International Conference on Supercomputing (ICS\u201916) . Association for Computing Machinery, New York, NY. DOI:https:\/\/doi.org\/10.1145\/2925426.2926290 10.1145\/2925426.2926290 Nusrat Sharmin Islam, Md. Wasi-ur Rahman, Xiaoyi Lu, and Dhabaleswar K. Panda. 2016. High performance design for HDFS with byte-addressability of NVM and RDMA. In Proceedings of the International Conference on Supercomputing (ICS\u201916). Association for Computing Machinery, New York, NY. DOI:https:\/\/doi.org\/10.1145\/2925426.2926290"},{"key":"e_1_2_1_38_1","volume-title":"Zixuan Wang, Yi Xu, Subramanya R. Dulloor, et\u00a0al.","author":"Izraelevitz Joseph","year":"2019","unstructured":"Joseph Izraelevitz , Jian Yang , Lu Zhang , Juno Kim , Xiao Liu , Amirsaman Memaripour , Yun Joon Soh , Zixuan Wang, Yi Xu, Subramanya R. Dulloor, et\u00a0al. 2019 . Basic performance measurements of the Intel Optane DC persistent memory module. arXiv preprint arXiv:1903.05714 (2019). Joseph Izraelevitz, Jian Yang, Lu Zhang, Juno Kim, Xiao Liu, Amirsaman Memaripour, Yun Joon Soh, Zixuan Wang, Yi Xu, Subramanya R. Dulloor, et\u00a0al. 2019. Basic performance measurements of the Intel Optane DC persistent memory module. arXiv preprint arXiv:1903.05714 (2019)."},{"key":"e_1_2_1_39_1","volume-title":"Proceedings of the International Conference on Parallel Processing (ICPP\u201911)","author":"Jose Jithin","year":"2011","unstructured":"Jithin Jose , Hari Subramoni , Miao Luo , Minjia Zhang , Jian Huang , Md. Wasi-ur Rahman , Nusrat S. Islam , Xiangyong Ouyang , Hao Wang , Sayantan Sur , and Dhabaleswar K. Panda . 2011. Memcached design on high performance RDMA capable interconnects . In Proceedings of the International Conference on Parallel Processing (ICPP\u201911) . IEEE Computer Society, 743--752. DOI:https:\/\/doi.org\/10.1109\/ICPP. 2011 .37 10.1109\/ICPP.2011.37 Jithin Jose, Hari Subramoni, Miao Luo, Minjia Zhang, Jian Huang, Md. Wasi-ur Rahman, Nusrat S. Islam, Xiangyong Ouyang, Hao Wang, Sayantan Sur, and Dhabaleswar K. Panda. 2011. Memcached design on high performance RDMA capable interconnects. In Proceedings of the International Conference on Parallel Processing (ICPP\u201911). IEEE Computer Society, 743--752. DOI:https:\/\/doi.org\/10.1109\/ICPP.2011.37"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/2740070.2626299"},{"volume-title":"Proceedings of the USENIX Annual Technical Conference (USENIX ATC\u201916)","author":"Kalia Anuj","key":"e_1_2_1_41_1","unstructured":"Anuj Kalia , Michael Kaminsky , and David G. Andersen . 2016. Design guidelines for high performance RDMA systems . In Proceedings of the USENIX Annual Technical Conference (USENIX ATC\u201916) . 437--450. Anuj Kalia, Michael Kaminsky, and David G. Andersen. 2016. Design guidelines for high performance RDMA systems. In Proceedings of the USENIX Annual Technical Conference (USENIX ATC\u201916). 437--450."},{"volume-title":"Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI\u201916)","author":"Kalia Anuj","key":"e_1_2_1_42_1","unstructured":"Anuj Kalia , Michael Kaminsky , and David G. Andersen . 2016. FaSST: Fast, scalable and simple distributed transactions with two-sided (RDMA) datagram RPCs . In Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI\u201916) . 185--201. Anuj Kalia, Michael Kaminsky, and David G. Andersen. 2016. FaSST: Fast, scalable and simple distributed transactions with two-sided (RDMA) datagram RPCs. In Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI\u201916). 185--201."},{"volume-title":"Proceedings of the 16th USENIX Conference on Networked Systems Design and Implementation (NSDI\u201919)","author":"Kalia Anuj","key":"e_1_2_1_43_1","unstructured":"Anuj Kalia , Michael Kaminsky , and David G. Andersen . 2019. Datacenter RPCs can be general and fast . In Proceedings of the 16th USENIX Conference on Networked Systems Design and Implementation (NSDI\u201919) . USENIX Association, 1--16. Anuj Kalia, Michael Kaminsky, and David G. Andersen. 2019. Datacenter RPCs can be general and fast. In Proceedings of the 16th USENIX Conference on Networked Systems Design and Implementation (NSDI\u201919). USENIX Association, 1--16."},{"key":"e_1_2_1_44_1","volume-title":"Proceedings of the 16th USENIX Conference on File and Storage Technologies. 241","author":"Kannan Sudarsun","year":"2018","unstructured":"Sudarsun Kannan , Andrea C. Arpaci-Dusseau , Remzi H. Arpaci-Dusseau , Yuangang Wang , Jun Xu , and Gopinath Palani . 2018 . Designing a true direct-access file system with DevFS . In Proceedings of the 16th USENIX Conference on File and Storage Technologies. 241 . Sudarsun Kannan, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau, Yuangang Wang, Jun Xu, and Gopinath Palani. 2018. Designing a true direct-access file system with DevFS. In Proceedings of the 16th USENIX Conference on File and Storage Technologies. 241."},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/258533.258660"},{"key":"e_1_2_1_46_1","volume-title":"fast remote persistence. arXiv preprint arXiv:1909.02092","author":"Kashyap Sanidhya","year":"2019","unstructured":"Sanidhya Kashyap , Dai Qin , Steve Byan , Virendra J. Marathe , and Sanketh Nalli . 2019. Correct , fast remote persistence. arXiv preprint arXiv:1909.02092 ( 2019 ). Sanidhya Kashyap, Dai Qin, Steve Byan, Virendra J. Marathe, and Sanketh Nalli. 2019. Correct, fast remote persistence. arXiv preprint arXiv:1909.02092 (2019)."},{"key":"e_1_2_1_47_1","volume-title":"Proceedings of the USENIX Winter Conference","volume":"1994","author":"Keleher Peter J.","year":"1994","unstructured":"Peter J. Keleher , Alan L. Cox , Sandhya Dwarkadas , and Willy Zwaenepoel . 1994 . TreadMarks: Distributed shared memory on standard workstations and operating systems . In Proceedings of the USENIX Winter Conference , Vol. 1994 . 23--36. Peter J. Keleher, Alan L. Cox, Sandhya Dwarkadas, and Willy Zwaenepoel. 1994. TreadMarks: Distributed shared memory on standard workstations and operating systems. In Proceedings of the USENIX Winter Conference, Vol. 1994. 23--36."},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132747.3132770"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/1555754.1555758"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3341301.3359635"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3341302.3342071"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132747.3132756"},{"key":"e_1_2_1_53_1","volume-title":"Proceedings of the International Conference on Parallel Processing 2 88","author":"Li Kai","year":"1988","unstructured":"Kai Li . 1988 . IVY: A shared virtual memory system for parallel computing . In Proceedings of the International Conference on Parallel Processing 2 88 (1988), 94. Kai Li. 1988. IVY: A shared virtual memory system for parallel computing. In Proceedings of the International Conference on Parallel Processing 2 88 (1988), 94."},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/3126908.3126928"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3037697.3037714"},{"key":"e_1_2_1_56_1","volume-title":"Proceedings of the USENIX Conference on Usenix Annual Technical Conference (USENIX ATC\u201917)","author":"Lu Youyou","year":"2017","unstructured":"Youyou Lu , Jiwu Shu , Youmin Chen , and Tao Li . 2017 . Octopus: An RDMA-enabled distributed persistent memory file system . In Proceedings of the USENIX Conference on Usenix Annual Technical Conference (USENIX ATC\u201917) . USENIX Association, 773--785. Youyou Lu, Jiwu Shu, Youmin Chen, and Tao Li. 2017. Octopus: An RDMA-enabled distributed persistent memory file system. In Proceedings of the USENIX Conference on Usenix Annual Technical Conference (USENIX ATC\u201917). USENIX Association, 773--785."},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1109\/MSST.2015.7208274"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/2851504"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCD.2014.6974684"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/3373376.3378511"},{"key":"e_1_2_1_61_1","volume-title":"Proceedings of the USENIX Annual Technical Conference (USENIX ATC\u201913)","author":"Mitchell Christopher","year":"2013","unstructured":"Christopher Mitchell , Yifeng Geng , and Jinyang Li . 2013 . Using one-sided RDMA reads to build a fast, CPU-efficient key-value store . In Proceedings of the USENIX Annual Technical Conference (USENIX ATC\u201913) . 103--114. Christopher Mitchell, Yifeng Geng, and Jinyang Li. 2013. Using one-sided RDMA reads to build a fast, CPU-efficient key-value store. In Proceedings of the USENIX Annual Technical Conference (USENIX ATC\u201913). 103--114."},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.5555\/3323298.3323302"},{"key":"e_1_2_1_63_1","volume-title":"Proceedings of the USENIX Annual Technical Conference (USENIX ATC\u201915)","author":"Nelson Jacob","year":"2015","unstructured":"Jacob Nelson , Brandon Holt , Brandon Myers , Preston Briggs , Luis Ceze , Simon Kahan , and Mark Oskin . 2015 . Latency-tolerant software distributed shared memory . In Proceedings of the USENIX Annual Technical Conference (USENIX ATC\u201915) . USENIX Association, 291--305. Retrieved from https:\/\/www.usenix.org\/conference\/atc15\/technical-session\/presentation\/nelson. Jacob Nelson, Brandon Holt, Brandon Myers, Preston Briggs, Luis Ceze, Simon Kahan, and Mark Oskin. 2015. Latency-tolerant software distributed shared memory. In Proceedings of the USENIX Annual Technical Conference (USENIX ATC\u201915). USENIX Association, 291--305. Retrieved from https:\/\/www.usenix.org\/conference\/atc15\/technical-session\/presentation\/nelson."},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.5555\/2482626.2482663"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/2644865.2541965"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/3319647.3325827"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/3341301.3359631"},{"key":"e_1_2_1_68_1","volume-title":"Proceedings of the USENIX Annual Technical Conference (USENIX ATC\u201914)","author":"Ongaro Diego","year":"2014","unstructured":"Diego Ongaro and John Ousterhout . 2014 . In search of an understandable consensus algorithm . In Proceedings of the USENIX Annual Technical Conference (USENIX ATC\u201914) . USENIX Association, 305--320. Diego Ongaro and John Ousterhout. 2014. In search of an understandable consensus algorithm. In Proceedings of the USENIX Annual Technical Conference (USENIX ATC\u201914). USENIX Association, 305--320."},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/2901318.2901324"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/2882903.2915251"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1145\/2749246.2749267"},{"volume-title":"Proceedings of the 36th International Symposium on Computer Architecture (ISCA\u201909)","author":"Qureshi Moinuddin K.","key":"e_1_2_1_72_1","unstructured":"Moinuddin K. Qureshi , Vijayalakshmi Srinivasan , and Jude A. Rivers . 2009. Scalable high performance main memory system using phase-change memory technology . In Proceedings of the 36th International Symposium on Computer Architecture (ISCA\u201909) . ACM, New York, NY, 24--33. Moinuddin K. Qureshi, Vijayalakshmi Srinivasan, and Jude A. Rivers. 2009. Scalable high performance main memory system using phase-change memory technology. In Proceedings of the 36th International Symposium on Computer Architecture (ISCA\u201909). ACM, New York, NY, 24--33."},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/3299869.3300069"},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1145\/3127479.3128610"},{"key":"e_1_2_1_75_1","volume-title":"Proceedings of the USENIX Annual Technical Conference (USENIX ATC\u201919)","author":"Stuedi Patrick","year":"2019","unstructured":"Patrick Stuedi , Animesh Trivedi , Jonas Pfefferle , Ana Klimovic , Adrian Schuepbach , and Bernard Metzler . 2019 . Unification of temporary storage in the NodeKernel architecture . In Proceedings of the USENIX Annual Technical Conference (USENIX ATC\u201919) . USENIX Association, 767--782. Retrieved from https:\/\/www.usenix.org\/conference\/atc19\/presentation\/stuedi. Patrick Stuedi, Animesh Trivedi, Jonas Pfefferle, Ana Klimovic, Adrian Schuepbach, and Bernard Metzler. 2019. Unification of temporary storage in the NodeKernel architecture. In Proceedings of the USENIX Annual Technical Conference (USENIX ATC\u201919). USENIX Association, 767--782. Retrieved from https:\/\/www.usenix.org\/conference\/atc19\/presentation\/stuedi."},{"key":"e_1_2_1_76_1","volume-title":"Proceedings of the 9th USENIX Conference on File and Storage Technologies (FAST\u201911)","author":"Venkataraman Shivaram","year":"1960","unstructured":"Shivaram Venkataraman , Niraj Tolia , Parthasarathy Ranganathan , and Roy H. Campbell . 2011. Consistent and durable data structures for non-volatile byte-addressable memory . In Proceedings of the 9th USENIX Conference on File and Storage Technologies (FAST\u201911) . USENIX Association, 5--5. Retrieved from http:\/\/dl.acm.org\/citation.cfm?id= 1960 475.1960480. Shivaram Venkataraman, Niraj Tolia, Parthasarathy Ranganathan, and Roy H. Campbell. 2011. Consistent and durable data structures for non-volatile byte-addressable memory. In Proceedings of the 9th USENIX Conference on File and Storage Technologies (FAST\u201911). USENIX Association, 5--5. Retrieved from http:\/\/dl.acm.org\/citation.cfm?id=1960475.1960480."},{"volume-title":"Proceedings of the 9th European Conference on Computer Systems (EuroSys\u201914)","author":"Volos Haris","key":"e_1_2_1_77_1","unstructured":"Haris Volos , Sanketh Nalli , Sankarlingam Panneerselvam , Venkatanathan Varadarajan , Prashant Saxena , and Michael M. Swift . 2014. Aerie: Flexible file-system interfaces to storage-class memory . In Proceedings of the 9th European Conference on Computer Systems (EuroSys\u201914) . ACM, New York, NY. DOI:https:\/\/doi.org\/10.1145\/2592798.2592810 10.1145\/2592798.2592810 Haris Volos, Sanketh Nalli, Sankarlingam Panneerselvam, Venkatanathan Varadarajan, Prashant Saxena, and Michael M. Swift. 2014. Aerie: Flexible file-system interfaces to storage-class memory. In Proceedings of the 9th European Conference on Computer Systems (EuroSys\u201914). ACM, New York, NY. DOI:https:\/\/doi.org\/10.1145\/2592798.2592810"},{"key":"e_1_2_1_78_1","volume-title":"Proceedings of the 16th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS\u201911)","author":"Volos Haris","year":"1950","unstructured":"Haris Volos , Andres Jaan Tack , and Michael M. Swift . 2011. Mnemosyne: Lightweight persistent memory . In Proceedings of the 16th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS\u201911) . ACM, New York, NY, 91--104. DOI:https:\/\/doi.org\/10.1145\/ 1950 365.1950379 10.1145\/1950365.1950379 Haris Volos, Andres Jaan Tack, and Michael M. Swift. 2011. Mnemosyne: Lightweight persistent memory. In Proceedings of the 16th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS\u201911). ACM, New York, NY, 91--104. DOI:https:\/\/doi.org\/10.1145\/1950365.1950379"},{"volume-title":"Deconstructing RDMA-enabled distributed transactions: Hybrid is better! In Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI\u201918). 233--251","author":"Wei Xingda","key":"e_1_2_1_79_1","unstructured":"Xingda Wei , Zhiyuan Dong , Rong Chen , and Haibo Chen . 2018. Deconstructing RDMA-enabled distributed transactions: Hybrid is better! In Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI\u201918). 233--251 . Xingda Wei, Zhiyuan Dong, Rong Chen, and Haibo Chen. 2018. Deconstructing RDMA-enabled distributed transactions: Hybrid is better! In Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI\u201918). 233--251."},{"key":"e_1_2_1_80_1","doi-asserted-by":"publisher","DOI":"10.1145\/2815400.2815419"},{"key":"e_1_2_1_81_1","doi-asserted-by":"publisher","DOI":"10.1145\/1374596.1374606"},{"key":"e_1_2_1_82_1","volume-title":"Proceedings of International Conference for High Performance Computing, Networking, Storage and Analysis (SC\u201911)","author":"Wu Xiaojian","year":"2063","unstructured":"Xiaojian Wu and A. L. Narasimha Reddy . 2011. SCMFS: A file system for storage class memory . In Proceedings of International Conference for High Performance Computing, Networking, Storage and Analysis (SC\u201911) . ACM, New York, NY. DOI:https:\/\/doi.org\/10.1145\/ 2063 384.2063436 10.1145\/2063384.2063436 Xiaojian Wu and A. L. Narasimha Reddy. 2011. SCMFS: A file system for storage class memory. In Proceedings of International Conference for High Performance Computing, Networking, Storage and Analysis (SC\u201911). ACM, New York, NY. DOI:https:\/\/doi.org\/10.1145\/2063384.2063436"},{"key":"e_1_2_1_83_1","volume-title":"Proceedings of the USENIX Annual Technical Conference (USENIX ATC\u201917)","author":"Xia Fei","year":"2017","unstructured":"Fei Xia , Dejun Jiang , Jin Xiong , and Ninghui Sun . 2017 . HiKV: A hybrid index key-value store for DRAM-NVM memory systems . In Proceedings of the USENIX Annual Technical Conference (USENIX ATC\u201917) . 349--362. Fei Xia, Dejun Jiang, Jin Xiong, and Ninghui Sun. 2017. HiKV: A hybrid index key-value store for DRAM-NVM memory systems. In Proceedings of the USENIX Annual Technical Conference (USENIX ATC\u201917). 349--362."},{"key":"e_1_2_1_84_1","volume-title":"Proceedings of the 14th USENIX Conference on File and Storage Technologies (FAST\u201916)","author":"Xu Jian","year":"2016","unstructured":"Jian Xu and Steven Swanson . 2016 . NOVA: A log-structured file system for hybrid volatile\/non-volatile main memories . In Proceedings of the 14th USENIX Conference on File and Storage Technologies (FAST\u201916) . USENIX Association, 323--338. Retrieved from http:\/\/dl.acm.org\/citation.cfm?id=2930583.2930608. Jian Xu and Steven Swanson. 2016. NOVA: A log-structured file system for hybrid volatile\/non-volatile main memories. In Proceedings of the 14th USENIX Conference on File and Storage Technologies (FAST\u201916). USENIX Association, 323--338. Retrieved from http:\/\/dl.acm.org\/citation.cfm?id=2930583.2930608."},{"key":"e_1_2_1_85_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132747.3132761"},{"key":"e_1_2_1_86_1","volume-title":"Proceedings of the 17th USENIX Conference on File and Storage Technologies (FAST\u201919)","author":"Yang Jian","year":"2019","unstructured":"Jian Yang , Joseph Izraelevitz , and Steven Swanson . 2019 . Orion: A distributed file system for non-volatile main memory and RDMA-capable networks . In Proceedings of the 17th USENIX Conference on File and Storage Technologies (FAST\u201919) . USENIX Association, 221--234. Retrieved from https:\/\/www.usenix.org\/conference\/fast19\/presentation\/yang. Jian Yang, Joseph Izraelevitz, and Steven Swanson. 2019. Orion: A distributed file system for non-volatile main memory and RDMA-capable networks. In Proceedings of the 17th USENIX Conference on File and Storage Technologies (FAST\u201919). USENIX Association, 221--234. Retrieved from https:\/\/www.usenix.org\/conference\/fast19\/presentation\/yang."},{"key":"e_1_2_1_87_1","volume-title":"Proceedings of the 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI\u201920)","author":"Yang Jian","year":"2020","unstructured":"Jian Yang , Joseph Izraelevitz , and Steven Swanson . 2020 . FileMR: Rethinking RDMA networking for scalable persistent memory . In Proceedings of the 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI\u201920) . USENIX Association, 111--125. Retrieved from https:\/\/www.usenix.org\/conference\/nsdi20\/presentation\/yang. Jian Yang, Joseph Izraelevitz, and Steven Swanson. 2020. FileMR: Rethinking RDMA networking for scalable persistent memory. In Proceedings of the 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI\u201920). USENIX Association, 111--125. Retrieved from https:\/\/www.usenix.org\/conference\/nsdi20\/presentation\/yang."},{"key":"e_1_2_1_88_1","doi-asserted-by":"publisher","DOI":"10.5555\/3386691.3386708"},{"key":"e_1_2_1_89_1","volume-title":"Proceedings of the 13th USENIX Conference on File and Storage Technologies (FAST\u201915)","author":"Yang Jun","year":"2015","unstructured":"Jun Yang , Qingsong Wei , Cheng Chen , Chundong Wang , Khai Leong Yong , and Bingsheng He . 2015 . NV-tree: Reducing consistency cost for NVM-based single level systems . In Proceedings of the 13th USENIX Conference on File and Storage Technologies (FAST\u201915) . USENIX Association, Berkeley, CA, 167--181. Retrieved from http:\/\/dl.acm.org\/citation.cfm?id=2750482.2750495. Jun Yang, Qingsong Wei, Cheng Chen, Chundong Wang, Khai Leong Yong, and Bingsheng He. 2015. NV-tree: Reducing consistency cost for NVM-based single level systems. In Proceedings of the 13th USENIX Conference on File and Storage Technologies (FAST\u201915). USENIX Association, Berkeley, CA, 167--181. Retrieved from http:\/\/dl.acm.org\/citation.cfm?id=2750482.2750495."},{"key":"e_1_2_1_90_1","volume-title":"Proceedings of the 2nd USENIX Conference on Hot Topics in Cloud Computing (HotCloud\u201910)","author":"Zaharia Matei","year":"2010","unstructured":"Matei Zaharia , Mosharaf Chowdhury , Michael J. Franklin , Scott Shenker , and Ion Stoica . 2010 . Spark: Cluster computing with working sets . In Proceedings of the 2nd USENIX Conference on Hot Topics in Cloud Computing (HotCloud\u201910) . 95. Matei Zaharia, Mosharaf Chowdhury, Michael J. Franklin, Scott Shenker, and Ion Stoica. 2010. Spark: Cluster computing with working sets. In Proceedings of the 2nd USENIX Conference on Hot Topics in Cloud Computing (HotCloud\u201910). 95."},{"key":"e_1_2_1_91_1","doi-asserted-by":"publisher","DOI":"10.23919\/DATE.2017.7927279"},{"key":"e_1_2_1_92_1","doi-asserted-by":"publisher","DOI":"10.1145\/2694344.2694370"},{"key":"e_1_2_1_93_1","volume-title":"Proceedings of the 17th USENIX Conference on File and Storage Technologies (FAST\u201919)","author":"Zheng Shengan","year":"2019","unstructured":"Shengan Zheng , Morteza Hoseinzadeh , and Steven Swanson . 2019 . Ziggurat: A tiered file system for non-volatile main memories and disks . In Proceedings of the 17th USENIX Conference on File and Storage Technologies (FAST\u201919) . 207--219. Shengan Zheng, Morteza Hoseinzadeh, and Steven Swanson. 2019. Ziggurat: A tiered file system for non-volatile main memories and disks. In Proceedings of the 17th USENIX Conference on File and Storage Technologies (FAST\u201919). 207--219."},{"key":"e_1_2_1_94_1","doi-asserted-by":"publisher","DOI":"10.1145\/1555754.1555759"},{"key":"e_1_2_1_95_1","volume-title":"Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI\u201918)","author":"Zuo Pengfei","year":"2018","unstructured":"Pengfei Zuo , Yu Hua , and Jie Wu . 2018 . Write-optimized and high-performance hashing index scheme for persistent memory . In Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI\u201918) . 461--476. Pengfei Zuo, Yu Hua, and Jie Wu. 2018. Write-optimized and high-performance hashing index scheme for persistent memory. In Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI\u201918). 461--476."}],"container-title":["ACM Transactions on Storage"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3412852","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3412852","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:47:02Z","timestamp":1750193222000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3412852"}},"subtitle":["Design and Implementation of an RDMA-enabled Distributed Persistent Memory Storage System"],"short-title":[],"issued":{"date-parts":[[2020,10]]},"references-count":95,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2020,11,30]]}},"alternative-id":["10.1145\/3412852"],"URL":"https:\/\/doi.org\/10.1145\/3412852","relation":{},"ISSN":["1553-3077","1553-3093"],"issn-type":[{"type":"print","value":"1553-3077"},{"type":"electronic","value":"1553-3093"}],"subject":[],"published":{"date-parts":[[2020,10]]},"assertion":[{"value":"2020-02-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-07-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-10-01","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}