{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,10]],"date-time":"2026-03-10T02:36:57Z","timestamp":1773110217082,"version":"3.50.1"},"reference-count":43,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2023,4,25]],"date-time":"2023-04-25T00:00:00Z","timestamp":1682380800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100012166","name":"National Key R&D Program of China","doi-asserted-by":"crossref","award":["2018YFB1003305"],"award-info":[{"award-number":["2018YFB1003305"]}],"id":[{"id":"10.13039\/501100012166","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100001809","name":"NSFC","doi-asserted-by":"crossref","award":["61832020, 61821003"],"award-info":[{"award-number":["61832020, 61821003"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Archit. Code Optim."],"published-print":{"date-parts":[[2023,6,30]]},"abstract":"<jats:p>Disaggregated memory architecture has risen in popularity for large datacenters with the advantage of improved resource utilization, failure isolation, and elasticity. Replicated state machines (RSMs) have been extensively used for reliability and consistency. In traditional RSM protocols, each replica stores replicated data and has the computing power to participate in some part of the protocols. However, traditional RSM protocols fail to work in the disaggregated memory architecture due to asymmetric resources on CPU nodes and memory nodes. This article proposes ECHO, a fast one-sided RDMA-based RSM protocol with lightweight log replication and remote applying, efficient linearizability guarantee, and fast coordinator failure recovery. ECHO enables all operations in the protocol to be efficiently executed using only one-sided RDMA, without the participation of any computing resource in the memory pool. To provide lightweight log replication and remote applying, ECHO couples the replicated log and the state machine to avoid dual-copy and performs remote applying by updating pointers. To enable efficient remote log state management, ECHO leverages a hitchhiked log state updating scheme to eliminate extra network round trips. To provide efficient linearizability guarantee, ECHO performs immediate remote applying after log replication and leverages the local locks at the coordinator to ensure linear consistency. Moreover, ECHO adopts a commit-aware log cache to make data visible immediately after being committed. To achieve fast failure recovery, ECHO leverages a commit point identification scheme to reduce the overhead of log consistency recovery. Experimental results demonstrate that ECHO outperforms the state-of-the-art RSM protocol (namely Sift) in multiple scenarios. For example, ECHO achieves 27%\u201352% higher throughput on typical write-intensive workloads. Moreover, ECHO reduces the consistency recovery time by three orders of magnitude for coordinator failure.<\/jats:p>","DOI":"10.1145\/3587096","type":"journal-article","created":{"date-parts":[[2023,3,10]],"date-time":"2023-03-10T11:46:10Z","timestamp":1678448770000},"page":"1-25","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":6,"title":["Fast One-Sided RDMA-Based State Machine Replication for Disaggregated Memory"],"prefix":"10.1145","volume":"20","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8242-8371","authenticated-orcid":false,"given":"Jingwen","family":"Du","sequence":"first","affiliation":[{"name":"Huazhong University of Science and Technology, Wuhan, Hubei, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2791-4158","authenticated-orcid":false,"given":"Fang","family":"Wang","sequence":"additional","affiliation":[{"name":"Huazhong University of Science and Technology, Wuhan, Hubei, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4674-6006","authenticated-orcid":false,"given":"Dan","family":"Feng","sequence":"additional","affiliation":[{"name":"Huazhong University of Science and Technology, Wuhan, Hubei, China"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-0954-9685","authenticated-orcid":false,"given":"Changchen","family":"Gan","sequence":"additional","affiliation":[{"name":"Huazhong University of Science and Technology, Wuhan, Hubei, China"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-9789-4239","authenticated-orcid":false,"given":"Yuchao","family":"Cao","sequence":"additional","affiliation":[{"name":"Huazhong University of Science and Technology, Wuhan, Hubei, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5867-847X","authenticated-orcid":false,"given":"Xiaomin","family":"Zou","sequence":"additional","affiliation":[{"name":"Huazhong University of Science and Technology, Wuhan, Hubei, China"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-1375-1976","authenticated-orcid":false,"given":"Fan","family":"Li","sequence":"additional","affiliation":[{"name":"Huazhong University of Science and Technology, Wuhan, Hubei, China"}]}],"member":"320","published-online":{"date-parts":[[2023,4,25]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"2019. Gen-Z technology. https:\/\/genzconsortium.org\/. Accessed March 1 2022."},{"key":"e_1_3_2_3_2","unstructured":"2019. Intel Corporation. Intel rack scale design architecture. https:\/\/www.intel.com\/content\/www\/us\/en\/architectureand-technology\/rack-scale-design-overview.html. Accessed March 1 2022."},{"key":"e_1_3_2_4_2","first-page":"775","volume-title":"Proceedings of the 2018 USENIX Annual Technical Conference","author":"Aguilera Marcos K.","year":"2018","unstructured":"Marcos K. Aguilera, Nadav Amit, Irina Calciu, Xavier Deguillard, Jayneel Gandhi, Stanko Novakovic, Arun Ramanathan, Pratap Subrahmanyam, Lalith Suresh, Kiran Tati, Rajesh Venkatasubramanian, and Michael Wei. 2018. Remote regions: A simple abstraction for remote memory. In Proceedings of the 2018 USENIX Annual Technical Conference. 775\u2013787."},{"key":"e_1_3_2_5_2","first-page":"599","volume-title":"Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation","author":"Aguilera Marcos K.","year":"2020","unstructured":"Marcos K. Aguilera, Naama Ben-David, Rachid Guerraoui, Virendra J. Marathe, Athanasios Xygkis, and Igor Zablotchi. 2020. Microsecond consensus for microsecond applications. In Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation. 599\u2013616."},{"key":"e_1_3_2_6_2","doi-asserted-by":"crossref","first-page":"120","DOI":"10.1145\/3317550.3321433","volume-title":"Proceedings of the Workshop on Hot Topics in Operating Systems","author":"Aguilera Marcos K.","year":"2019","unstructured":"Marcos K. Aguilera, Kimberly Keeton, Stanko Novakovic, and Sharad Singhal. 2019. Designing far memory data structures: Think outside the box. In Proceedings of the Workshop on Hot Topics in Operating Systems. 120\u2013126."},{"key":"e_1_3_2_7_2","unstructured":"Mellanox Technologies. 2020. RDMA Aware Networks Programming User Manual v1.7. https:\/\/docs.nvidia.com\/networking\/display\/RDMAAwareProgrammingv17."},{"issue":"1","key":"e_1_3_2_8_2","first-page":"93","article-title":"Strong consistency at scale.","volume":"39","author":"Bezerra Carlos Eduardo Benevides","year":"2016","unstructured":"Carlos Eduardo Benevides Bezerra, Long Hoang Le, and Fernando Pedone. 2016. Strong consistency at scale.IEEE Data Eng. Bull. 39, 1 (2016), 93\u2013103.","journal-title":"IEEE Data Eng. Bull."},{"key":"e_1_3_2_9_2","first-page":"1","volume-title":"Proceedings of the 2015 IEEE 23rd Annual Symposium on High-Performance Interconnects","author":"Birrittella Mark S.","year":"2015","unstructured":"Mark S. Birrittella, Mark Debbage, Ram Huggahalli, James Kunz, Tom Lovett, Todd Rimmer, Keith D. Underwood, and Robert C. Zak. 2015. Intel\u00ae omni-path architecture: Enabling scalable, high performance fabrics. In Proceedings of the 2015 IEEE 23rd Annual Symposium on High-Performance Interconnects. IEEE, 1\u20139."},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.5555\/1298455.1298487"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/1807128.1807152"},{"issue":"3","key":"e_1_3_2_12_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2491245","article-title":"Spanner: Google\u2019s globally distributed database","volume":"31","author":"Corbett James C.","year":"2013","unstructured":"James C. Corbett, Jeffrey Dean, Michael Epstein, Andrew Fikes, Christopher Frost, Jeffrey John Furman, Sanjay Ghemawat, Andrey Gubarev, Christopher Heiser, Peter Hochschild, Wilson Hsieh, Sebastian Kanthak, Eugene Kogan, Hongyi Li, Alexander Lloyd, Sergey Melnik, David Mwaura, David Nagle, Sean Quinlan, Rajesh Rao, Lindsay Rolig, Yasushi Saito, Michal Szymaniak, Christopher Taylor, Ruth Wang, and Dale Woodford. 2013. Spanner: Google\u2019s globally distributed database. ACM Transactions on Computer Systems 31, 3 (2013), 1\u201322.","journal-title":"ACM Transactions on Computer Systems"},{"key":"e_1_3_2_13_2","first-page":"401","volume-title":"Proceedings of the 11th USENIX Symposium on Networked Systems Design and Implementation","author":"Dragojevi\u0107 Aleksandar","year":"2014","unstructured":"Aleksandar Dragojevi\u0107, Dushyanth Narayanan, Miguel Castro, and Orion Hodson. 2014. FaRM: Fast remote memory. In Proceedings of the 11th USENIX Symposium on Networked Systems Design and Implementation. 401\u2013414."},{"key":"e_1_3_2_14_2","first-page":"249","volume-title":"Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation","author":"Gao Peter X.","year":"2016","unstructured":"Peter X. Gao, Akshay Narayan, Sagar Karandikar, Joao Carreira, Sangjin Han, Rachit Agarwal, Sylvia Ratnasamy, and Scott Shenker. 2016. Network requirements for resource disaggregation. In Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation. 249\u2013264."},{"key":"e_1_3_2_15_2","first-page":"649","volume-title":"Proceedings of the 14th USENIX Symposium on Networked Systems Design and Implementation","author":"Gu Juncheng","year":"2017","unstructured":"Juncheng Gu, Youngmoon Lee, Yiwen Zhang, Mosharaf Chowdhury, and Kang G. Shin. 2017. Efficient memory disaggregation with infiniswap. In Proceedings of the 14th USENIX Symposium on Networked Systems Design and Implementation. 649\u2013667."},{"key":"e_1_3_2_16_2","volume-title":"Proceedings of the USENIX Annual Technical Conference","volume":"8","author":"Hunt Patrick","year":"2010","unstructured":"Patrick Hunt, Mahadev Konar, Flavio Paiva Junqueira, and Benjamin Reed. 2010. ZooKeeper: Wait-free coordination for internet-scale systems. In Proceedings of the USENIX Annual Technical Conference. Vol. 8."},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.5555\/3026959.3027000"},{"key":"e_1_3_2_18_2","first-page":"201","volume-title":"Proceedings of the 25th International Conference on Architectural Support for Programming Languages and Operating Systems","author":"Katsarakis Antonios","year":"2020","unstructured":"Antonios Katsarakis, Vasilis Gavrielatos, M. R. Siavash Katebzadeh, Arpit Joshi, Aleksandar Dragojevic, Boris Grot, and Vijay Nagarajan. 2020. Hermes: A fast, fault-tolerant and linearizable replication protocol. In Proceedings of the 25th International Conference on Architectural Support for Programming Languages and Operating Systems. 201\u2013217."},{"key":"e_1_3_2_19_2","doi-asserted-by":"crossref","first-page":"260","DOI":"10.1145\/3359989.3365437","volume-title":"Proceedings of the 15th International Conference on Emerging Networking Experiments and Technologies","author":"Kazhamiaka Mikhail","year":"2019","unstructured":"Mikhail Kazhamiaka, Babar Memon, Chathura Kankanamge, Siddhartha Sahu, Sajjad Rizvi, Bernard Wong, and Khuzaima Daudjee. 2019. Sift: Resource-efficient consensus with RDMA. In Proceedings of the 15th International Conference on Emerging Networking Experiments and Technologies. 260\u2013271."},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/3335772.3335939"},{"key":"e_1_3_2_21_2","doi-asserted-by":"crossref","unstructured":"Leslie Lamport. 2001. Paxos made simple. ACM SIGACT News (Distributed Computing Column) 32 4 (2001) 51\u201358. https:\/\/www.microsoft.com\/en-us\/research\/publication\/paxos-made-simple\/.","DOI":"10.1145\/568425.568433"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.14778\/3565838.3565854"},{"key":"e_1_3_2_23_2","volume-title":"Proceedings of the 21st USENIX Conference on File and Storage Technologies","author":"Li Pengfei","year":"2023","unstructured":"Pengfei Li, Yu Hua, Pengfei Zuo, Zhangyu Chen, and Jiajie Sheng. 2023. ROLEX: A scalable RDMA-oriented learned key-value store for disaggregated memory systems. In Proceedings of the 21st USENIX Conference on File and Storage Technologies."},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/1555815.1555789"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/HPCA.2012.6168955"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.5555\/2643634.2643666"},{"key":"e_1_3_2_27_2","unstructured":"Hewlett Packard. 2021. The Machine: A New Kind of Computer. Retrieved from https:\/\/www.hpl.hp.com\/research\/systems-research\/themachine\/. Accessed April 12 2022."},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/2749246.2749267"},{"key":"e_1_3_2_29_2","first-page":"315","volume-title":"Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation","author":"Ruan Zhenyuan","year":"2020","unstructured":"Zhenyuan Ruan, Malte Schwarzkopf, Marcos K. Aguilera, and Adam Belay. 2020. AIFM: High-performance, application-integrated far memory. In Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation. 315\u2013332."},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/98163.98167"},{"key":"e_1_3_2_31_2","first-page":"69","volume-title":"Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation","author":"Shan Yizhou","year":"2018","unstructured":"Yizhou Shan, Yutong Huang, Yilun Chen, and Yiying Zhang. 2018. LegoOS: A disseminated, distributed OS for hardware resource disaggregation. In Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation. 69\u201387."},{"key":"e_1_3_2_32_2","first-page":"81","volume-title":"Proceedings of the 21st USENIX Conference on File and Storage Technologies","author":"Shen Jiacheng","year":"2023","unstructured":"Jiacheng Shen, Pengfei Zuo, Xuchuan Luo, Tianyi Yang, Yuxin Su, Yangfan Zhou, and Michael R. Lyu. 2023. FUSEE: A fully memory-disaggregated key-value store. In Proceedings of the 21st USENIX Conference on File and Storage Technologies. 81\u201398."},{"key":"e_1_3_2_33_2","first-page":"255","volume-title":"Proceedings of the 16th USENIX Symposium on Networked Systems Design and Implementation","author":"Shrivastav Vishal","year":"2019","unstructured":"Vishal Shrivastav, Asaf Valadarsky, Hitesh Ballani, Paolo Costa, Ki Suh Lee, Han Wang, Rachit Agarwal, and Hakim Weatherspoon. 2019. Shoal: A network architecture for disaggregated racks. In Proceedings of the 16th USENIX Symposium on Networked Systems Design and Implementation. 255\u2013270."},{"key":"e_1_3_2_34_2","first-page":"851","volume-title":"Proceedings of the 2018 USENIX Annual Technical Conference","author":"Taleb Yacine","year":"2018","unstructured":"Yacine Taleb, Ryan Stutsman, Gabriel Antoniu, and Toni Cortes. 2018. Tailwind: Fast and atomic RDMA-based replication. In Proceedings of the 2018 USENIX Annual Technical Conference. 851\u2013863."},{"key":"e_1_3_2_35_2","first-page":"33","volume-title":"Proceedings of the 2020 USENIX Annual Technical Conference","author":"Tsai Shin-Yeh","year":"2020","unstructured":"Shin-Yeh Tsai, Yizhou Shan, and Yiying Zhang. 2020. Disaggregating persistent memory and controlling them remotely: An exploration of passive disaggregated key-value stores. In Proceedings of the 2020 USENIX Annual Technical Conference. 33\u201348."},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00778-013-0343-9"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3127479.3128609"},{"key":"e_1_3_2_38_2","first-page":"261","volume-title":"Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation","author":"Wang Chenxi","year":"2020","unstructured":"Chenxi Wang, Haoran Ma, Shi Liu, Yuanqi Li, Zhenyuan Ruan, Khanh Nguyen, Michael D. Bond, Ravi Netravali, Miryung Kim, and Guoqing Harry Xu. 2020. Semeru: A memory-disaggregated managed runtime. In Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation. 261\u2013280."},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/3514221.3517824"},{"key":"e_1_3_2_40_2","first-page":"523","volume-title":"Proceedings of the 2021 USENIX Annual Technical Conference","author":"Wei Xingda","year":"2021","unstructured":"Xingda Wei, Xiating Xie, Rong Chen, Haibo Chen, and Binyu Zang. 2021. Characterizing and optimizing remote persistent memory with RDMA and NVM. In Proceedings of the 2021 USENIX Annual Technical Conference. 523\u2013536."},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.14778\/3342263.3342639"},{"key":"e_1_3_2_42_2","first-page":"51","volume-title":"Proceedings of the 20th USENIX Conference on File and Storage Technologies","author":"Zhang Ming","year":"2022","unstructured":"Ming Zhang, Yu Hua, Pengfei Zuo, and Lurong Liu. 2022. FORD: Fast One-sided RDMA-based distributed transactions for disaggregated persistent memory. In Proceedings of the 20th USENIX Conference on File and Storage Technologies. 51\u201368."},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.14778\/3467861.3467877"},{"key":"e_1_3_2_44_2","first-page":"15","volume-title":"Proceedings of the 2021 USENIX Annual Technical Conference","author":"Zuo Pengfei","year":"2021","unstructured":"Pengfei Zuo, Jiazhao Sun, Liu Yang, Shuangwu Zhang, and Yu Hua. 2021. One-sided RDMA-conscious extendible hashing for disaggregated memory. In Proceedings of the 2021 USENIX Annual Technical Conference. 15\u201329."}],"container-title":["ACM Transactions on Architecture and Code Optimization"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3587096","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3587096","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T18:08:01Z","timestamp":1750183681000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3587096"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,4,25]]},"references-count":43,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2023,6,30]]}},"alternative-id":["10.1145\/3587096"],"URL":"https:\/\/doi.org\/10.1145\/3587096","relation":{},"ISSN":["1544-3566","1544-3973"],"issn-type":[{"value":"1544-3566","type":"print"},{"value":"1544-3973","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,4,25]]},"assertion":[{"value":"2022-08-03","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-02-21","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-04-25","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}