{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,24]],"date-time":"2026-03-24T15:19:55Z","timestamp":1774365595593,"version":"3.50.1"},"reference-count":56,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2023,6,26]],"date-time":"2023-06-26T00:00:00Z","timestamp":1687737600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGOPS Oper. Syst. Rev."],"published-print":{"date-parts":[[2023,6,26]]},"abstract":"<jats:p>Compute and memory are tightly coupled within each server in traditional datacenters. Large-scale datacenter operators have identified this coupling as a root cause behind fleetwide resource underutilization and increasing Total Cost of Ownership (TCO). With the advent of ultra-fast networks and cache-coherent interfaces, memory disaggregation has emerged as a potential solution, whereby applications can leverage available memory even outside server boundaries.<\/jats:p>\n          <jats:p>This paper summarizes the growing research landscape of memory disaggregation from a software perspective and introduces the challenges toward making it practical under current and future hardware trends. We also reflect on our seven-year journey in the SymbioticLab to build a comprehensive disaggregated memory system over ultra-fast networks. We conclude with some open challenges toward building next-generation memory disaggregation systems leveraging emerging cache-coherent interconnects.<\/jats:p>","DOI":"10.1145\/3606557.3606562","type":"journal-article","created":{"date-parts":[[2023,6,28]],"date-time":"2023-06-28T16:25:16Z","timestamp":1687969516000},"page":"29-37","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":20,"title":["Memory Disaggregation: Advances and Open Challenges"],"prefix":"10.1145","volume":"57","author":[{"given":"Hasan","family":"Al Maruf","sequence":"first","affiliation":[{"name":"University of Michigan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mosharaf","family":"Chowdhury","sequence":"additional","affiliation":[{"name":"University of Michigan"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2023,6,28]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"Alibaba Cluster Trace 2018. https:\/\/github.com\/ alibaba\/clusterdata\/blob\/master\/clustertrace- v2018\/trace_2018.md.  Alibaba Cluster Trace 2018. https:\/\/github.com\/ alibaba\/clusterdata\/blob\/master\/clustertrace- v2018\/trace_2018.md."},{"key":"e_1_2_1_2_1","unstructured":"CCIX. https:\/\/www.ccixconsortium.com\/.  CCIX. https:\/\/www.ccixconsortium.com\/."},{"key":"e_1_2_1_3_1","unstructured":"Compute Express Link (CXL). https: \/\/www.computeexpresslink.org\/.  Compute Express Link (CXL). https: \/\/www.computeexpresslink.org\/."},{"key":"e_1_2_1_4_1","unstructured":"DAMON\n  : Data Access MONitoring Framework for Fun and Memory Management Optimizations. https:\/\/www.linuxplumbersconf.org\/event\/ 7\/contributions\/659\/attachments\/503\/1195\/ damon_ksummit_2020.pdf.  DAMON: Data Access MONitoring Framework for Fun and Memory Management Optimizations. https:\/\/www.linuxplumbersconf.org\/event\/ 7\/contributions\/659\/attachments\/503\/1195\/ damon_ksummit_2020.pdf."},{"key":"e_1_2_1_5_1","unstructured":"Gen-Z. https:\/\/genzconsortium.org\/.  Gen-Z. https:\/\/genzconsortium.org\/."},{"key":"e_1_2_1_6_1","unstructured":"Idle page tracking-based working set estimation. https: \/\/lwn.net\/Articles\/460762\/.  Idle page tracking-based working set estimation. https: \/\/lwn.net\/Articles\/460762\/."},{"key":"e_1_2_1_7_1","unstructured":"NUMA Balancing (AutoNUMA). https: \/\/mirrors.edge.kernel.org\/pub\/linux\/kernel\/ people\/andrea\/autonuma\/autonuma_bench- 20120530.pdf.  NUMA Balancing (AutoNUMA). https: \/\/mirrors.edge.kernel.org\/pub\/linux\/kernel\/ people\/andrea\/autonuma\/autonuma_bench- 20120530.pdf."},{"key":"e_1_2_1_8_1","unstructured":"OpenCAPI. https:\/\/opencapi.org\/.  OpenCAPI. https:\/\/opencapi.org\/."},{"key":"e_1_2_1_9_1","unstructured":"Rack-scale computing at Yahoo! http: \/\/www.intel.com\/content\/dam\/www\/public\/ us\/en\/documents\/presentation\/idf15-yahoorack- scale-computing-presentation.pdf.  Rack-scale computing at Yahoo! http: \/\/www.intel.com\/content\/dam\/www\/public\/ us\/en\/documents\/presentation\/idf15-yahoorack- scale-computing-presentation.pdf."},{"key":"e_1_2_1_10_1","unstructured":"Tencent explores datacenter resource-pooling using Intel rack scale architecture (Intel RSA). http:\/\/www.intel.com\/content\/dam\/www\/public\/ us\/en\/documents\/white-papers\/rsa-tencentpaper. pdf.  Tencent explores datacenter resource-pooling using Intel rack scale architecture (Intel RSA). http:\/\/www.intel.com\/content\/dam\/www\/public\/ us\/en\/documents\/white-papers\/rsa-tencentpaper. pdf."},{"key":"e_1_2_1_11_1","volume-title":"USENIX ATC","author":"Aguilera M. K.","year":"2018","unstructured":"M. K. Aguilera , N. Amit , I. Calciu , X. Deguillard , J. Gandhi , S. Novakovi\u00b4c , A. Ramanathan , P. Subrahmanyam , L. Suresh , K. Tati , R. Venkatasubramanian , and M. Wei . Remote regions: a simple abstraction for remote memory . In USENIX ATC , 2018 . M. K. Aguilera, N. Amit, I. Calciu, X. Deguillard, J. Gandhi, S. Novakovi\u00b4c, A. Ramanathan, P. Subrahmanyam, L. Suresh, K. Tati, R. Venkatasubramanian, and M. Wei. Remote regions: a simple abstraction for remote memory. In USENIX ATC, 2018."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3342195.3387522"},{"key":"e_1_2_1_13_1","volume-title":"HotCarbon","author":"Anderson T.","year":"2022","unstructured":"T. Anderson , A. Belay , M. Chowdhury , A. Cidon , and I. Zhang . Treehouse: A case for carbon-aware datacenter software . In HotCarbon , 2022 . T. Anderson, A. Belay, M. Chowdhury, A. Cidon, and I. Zhang. Treehouse: A case for carbon-aware datacenter software. In HotCarbon, 2022."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3445814.3446713"},{"key":"e_1_2_1_15_1","volume-title":"International Workshop on Virtualization Technology","author":"Chen H.","year":"2008","unstructured":"H. Chen , Y. Luo , X. Wang , B. Zhang , Y. Sun , and Z. Wang . A transparent remote paging model for virtual machines . In International Workshop on Virtualization Technology , 2008 . H. Chen, Y. Luo, X. Wang, B. Zhang, Y. Sun, and Z. Wang. A transparent remote paging model for virtual machines. In International Workshop on Virtualization Technology, 2008."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2408776.2408794"},{"key":"e_1_2_1_17_1","volume-title":"NSDI","author":"Dragojevi\u00b4c A.","year":"2014","unstructured":"A. Dragojevi\u00b4c , D. Narayanan , O. Hodson , and M. Castro . FaRM: Fast remote memory . In NSDI , 2014 . A. Dragojevi\u00b4c, D. Narayanan, O. Hodson, and M. Castro. FaRM: Fast remote memory. In NSDI, 2014."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.5555\/645608.661831"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/224056.224072"},{"key":"e_1_2_1_21_1","volume-title":"USENIX ATC","author":"Gouk D.","year":"2022","unstructured":"D. Gouk , S. Lee , M. Kwon , and M. Jung . Direct access, High-Performance memory disaggregation with DirectCXL . In USENIX ATC , 2022 . D. Gouk, S. Lee, M. Kwon, and M. Jung. Direct access, High-Performance memory disaggregation with DirectCXL. In USENIX ATC, 2022."},{"key":"e_1_2_1_22_1","volume-title":"NSDI","author":"Gu J.","year":"2017","unstructured":"J. Gu , Y. Lee , Y. Zhang , M. Chowdhury , and K. G. Shin . Efficient memory disaggregation with Infiniswap . In NSDI , 2017 . J. Gu, Y. Lee, Y. Zhang, M. Chowdhury, and K. G. Shin. Efficient memory disaggregation with Infiniswap. In NSDI, 2017."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1038\/d41586-018-06610-y"},{"key":"e_1_2_1_24_1","volume-title":"OSDI","author":"Kalia A.","year":"2016","unstructured":"A. Kalia , M. Kaminsky , and D. G. Andersen . FaSST: Fast, scalable and simple distributed transactions with two-sided (RDMA) datagram RPCs . In OSDI , 2016 . A. Kalia, M. Kaminsky, and D. G. Andersen. FaSST: Fast, scalable and simple distributed transactions with two-sided (RDMA) datagram RPCs. In OSDI, 2016."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3297858.3304053"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3477132.3483561"},{"key":"e_1_2_1_27_1","volume-title":"FAST","author":"Lee Y.","year":"2022","unstructured":"Y. Lee , H. A. Maruf , M. Chowdhury , A. Cidon , and K. G. Shin . Hydra : Resilient and highly available remote memory . In FAST , 2022 . Y. Lee, H. A. Maruf, M. Chowdhury, A. Cidon, and K. G. Shin. Hydra : Resilient and highly available remote memory. In FAST, 2022."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3575693.3578835"},{"key":"e_1_2_1_29_1","volume-title":"IEEE International Conference on Cluster Computing","author":"Liang S.","year":"2005","unstructured":"S. Liang , R. Noronha , and D. K. Panda . Swapping to remote memory over InfiniBand: An approach using a high performance network block device . In IEEE International Conference on Cluster Computing , 2005 . S. Liang, R. Noronha, and D. K. Panda. Swapping to remote memory over InfiniBand: An approach using a high performance network block device. In IEEE International Conference on Cluster Computing, 2005."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/1555754.1555789"},{"key":"e_1_2_1_31_1","volume-title":"USENIX ATC","author":"Markatos E. P.","year":"1996","unstructured":"E. P. Markatos and G. Dramitinos . Implementation of a reliable remote memory pager . In USENIX ATC , 1996 . E. P. Markatos and G. Dramitinos. Implementation of a reliable remote memory pager. In USENIX ATC, 1996."},{"key":"e_1_2_1_32_1","volume-title":"USENIX ATC","author":"Maruf H. A.","year":"2020","unstructured":"H. A. Maruf and M. Chowdhury . Effectively prefetching remote memory with Leap . In USENIX ATC , 2020 . H. A. Maruf and M. Chowdhury. Effectively prefetching remote memory with Leap. In USENIX ATC, 2020."},{"key":"e_1_2_1_33_1","volume-title":"ASPLOS","author":"Maruf H. A.","year":"2023","unstructured":"H. A. Maruf , H. Wang , A. Dhanotia , J. Weiner , N. Agarwal , P. Bhattacharya , C. Petersen , M. Chowdhury , S. Kanaujia , and P. Chauhan . TPP: Transparent page placement for CXL-enabled tiered-memory . In ASPLOS , 2023 . H. A. Maruf, H. Wang, A. Dhanotia, J. Weiner, N. Agarwal, P. Bhattacharya, C. Petersen, M. Chowdhury, S. Kanaujia, and P. Chauhan. TPP: Transparent page placement for CXL-enabled tiered-memory. In ASPLOS, 2023."},{"key":"e_1_2_1_34_1","volume-title":"SIGMETRICS","author":"Maruf H. A.","year":"2023","unstructured":"H. A. Maruf , Y. Zhong , H. Wong , M. Chowdhury , A. Cidon , and C. Waldspurger . Memtrade: A disaggregated-memory marketplace for public clouds . In SIGMETRICS , 2023 . H. A. Maruf, Y. Zhong, H. Wong, M. Chowdhury, A. Cidon, and C. Waldspurger. Memtrade: A disaggregated-memory marketplace for public clouds. In SIGMETRICS, 2023."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-45209-6_157"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/2.84877"},{"key":"e_1_2_1_37_1","volume-title":"Yale Environment","author":"Pearce F.","year":"2018","unstructured":"F. Pearce . Energy hogs : Can world's huge data centers be made more efficient ? Yale Environment , 2018 . F. Pearce. Energy hogs: Can world's huge data centers be made more efficient? Yale Environment, 2018."},{"key":"e_1_2_1_38_1","volume-title":"OSDI","author":"Rashmi K.","year":"2016","unstructured":"K. Rashmi , M. Chowdhury , J. Kosaian , I. Stoica , and K. Ramchandran . EC-Cache: Load-balanced, lowlatency cluster caching with online erasure coding . In OSDI , 2016 . K. Rashmi, M. Chowdhury, J. Kosaian, I. Stoica, and K. Ramchandran. EC-Cache: Load-balanced, lowlatency cluster caching with online erasure coding. In OSDI, 2016."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3477132.3483550"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/2391229.2391236"},{"key":"e_1_2_1_41_1","volume-title":"NSDI","author":"Ruan Z.","year":"2023","unstructured":"Z. Ruan , S. J. Park , M. K. Aguilera , A. Belay , and M. Schwarzkopf . Nu: Achieving microsecond-scale resource fungibility with logical processes . In NSDI , 2023 . Z. Ruan, S. J. Park, M. K. Aguilera, A. Belay, and M. Schwarzkopf. Nu: Achieving microsecond-scale resource fungibility with logical processes. In NSDI, 2023."},{"key":"e_1_2_1_42_1","volume-title":"OSDI","author":"Ruan Z.","year":"2020","unstructured":"Z. Ruan , M. Schwarzkopf , M. K. Aguilera , and A. Belay . AIFM: High-performance, application-integrated far memory . In OSDI , 2020 . Z. Ruan, M. Schwarzkopf, M. K. Aguilera, and A. Belay. AIFM: High-performance, application-integrated far memory. In OSDI, 2020."},{"key":"e_1_2_1_43_1","volume-title":"IEEE International Conference on Cluster Computing","author":"Servat H.","year":"2017","unstructured":"H. Servat , A. J. Pe\u00f1a , G. Llort , E. Mercadal , H.-C. Hoppe , and J. Labarta . Automating the application data placement in hybrid memory systems . In IEEE International Conference on Cluster Computing , 2017 . H. Servat, A. J. Pe\u00f1a, G. Llort, E. Mercadal, H.-C. Hoppe, and J. Labarta. Automating the application data placement in hybrid memory systems. In IEEE International Conference on Cluster Computing, 2017."},{"key":"e_1_2_1_44_1","volume-title":"OSDI","author":"Shan Y.","year":"2018","unstructured":"Y. Shan , Y. Huang , Y. Chen , and Y. Zhang . LegoOS: A disseminated, distributed OS for hardware resource disaggregation . In OSDI , 2018 . Y. Shan, Y. Huang, Y. Chen, and Y. Zhang. LegoOS: A disseminated, distributed OS for hardware resource disaggregation. In OSDI, 2018."},{"key":"e_1_2_1_45_1","volume-title":"ATC","author":"Tatar A.","year":"2018","unstructured":"A. Tatar , R. K. Konoth , E. Athanasopoulos , C. Giuffrida , H. Bos , and K. Razavi . Throwhammer: Rowhammer attacks over the network and defenses . In ATC , 2018 . A. Tatar, R. K. Konoth, E. Athanasopoulos, C. Giuffrida, H. Bos, and K. Razavi. Throwhammer: Rowhammer attacks over the network and defenses. In ATC, 2018."},{"key":"e_1_2_1_46_1","volume-title":"USENIX Security","author":"Tsai S.-Y.","year":"2019","unstructured":"S.-Y. Tsai , M. Payer , and Y. Zhang . Pythia: Remote oracles for the masses . In USENIX Security , 2019 . S.-Y. Tsai, M. Payer, and Y. Zhang. Pythia: Remote oracles for the masses. In USENIX Security, 2019."},{"key":"e_1_2_1_47_1","unstructured":"Vladimir Davydov. Idle Memory Tracking. https: \/\/lwn.net\/Articles\/639341\/.  Vladimir Davydov. Idle Memory Tracking. https: \/\/lwn.net\/Articles\/639341\/."},{"key":"e_1_2_1_48_1","volume-title":"NSDI","author":"Vuppalapati M.","year":"2020","unstructured":"M. Vuppalapati , J. Miron , R. Agarwal , D. Truong , A. Motivala , and T. Cruanes . Building an elastic query engine on disaggregated storage . In NSDI , 2020 . M. Vuppalapati, J. Miron, R. Agarwal, D. Truong, A. Motivala, and T. Cruanes. Building an elastic query engine on disaggregated storage. In NSDI, 2020."},{"key":"e_1_2_1_49_1","volume-title":"OSDI","author":"Wang C.","year":"2020","unstructured":"C. Wang , H. Ma , S. Liu , Y. Li , Z. Ruan , K. Nguyen , M. D. Bond , R. Netravali , M. Kim , and G. H. Xu . Semeru: A Memory-Disaggregated managed runtime . In OSDI , 2020 . C.Wang, H. Ma, S. Liu,Y. Li, Z. Ruan, K. Nguyen, M. D. Bond, R. Netravali, M. Kim, and G. H. Xu. Semeru: A Memory-Disaggregated managed runtime. In OSDI, 2020."},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3126908.3126923"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3183713.3196890"},{"key":"e_1_2_1_52_1","volume-title":"NSDI","author":"You J.","year":"2021","unstructured":"J. You , J. Wu , X. Jin , and M. Chowdhury . Ship compute or ship data? why not both ? In NSDI , 2021 . J. You, J. Wu, X. Jin, and M. Chowdhury. Ship compute or ship data? why not both? In NSDI, 2021."},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/3387514.3405857"},{"key":"e_1_2_1_54_1","volume-title":"ACM SIGCOMM KBNets","author":"Zhang Y.","year":"2017","unstructured":"Y. Zhang , J. Gu , Y. Lee , M. Chowdhury , and K. G. Shin . Performance isolation anomalies in RDMA . In ACM SIGCOMM KBNets , 2017 . Y. Zhang, J. Gu, Y. Lee, M. Chowdhury, and K. G. Shin. Performance isolation anomalies in RDMA. In ACM SIGCOMM KBNets, 2017."},{"key":"e_1_2_1_55_1","volume-title":"ACM SIGCOMM","author":"Zhang Y.","year":"2022","unstructured":"Y. Zhang , G. Kumar , N. Dukkipati , X. Wu , P. Jha , M. Chowdhury , and A. Vahdat . Aequitas: Admission control for performance-critical RPCs in datacenters . In ACM SIGCOMM , 2022 . Y. Zhang, G. Kumar, N. Dukkipati, X. Wu, P. Jha, M. Chowdhury, and A. Vahdat. Aequitas: Admission control for performance-critical RPCs in datacenters. In ACM SIGCOMM, 2022."},{"key":"e_1_2_1_56_1","volume-title":"NSDI","author":"Zhang Y.","year":"2022","unstructured":"Y. Zhang , Y. Tan , B. Stephens , and M. Chowdhury . Justitia: Software Multi-Tenancy in hardware Kernel-Bypass networks . In NSDI , 2022 . Y. Zhang, Y. Tan, B. Stephens, and M. Chowdhury. Justitia: Software Multi-Tenancy in hardware Kernel-Bypass networks. In NSDI, 2022."},{"key":"e_1_2_1_57_1","unstructured":"W. Zhao and J. Ning. Project Tioga Pass Rev 0.30 : Facebook Server Intel Motherboard V4.0 Spec. https:\/\/www.opencompute.org\/documents\/ facebook-server-intel-motherboard-v40-spec.  W. Zhao and J. Ning. Project Tioga Pass Rev 0.30 : Facebook Server Intel Motherboard V4.0 Spec. https:\/\/www.opencompute.org\/documents\/ facebook-server-intel-motherboard-v40-spec."}],"container-title":["ACM SIGOPS Operating Systems Review"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3606557.3606562","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3606557.3606562","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T17:48:51Z","timestamp":1750182531000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3606557.3606562"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,26]]},"references-count":56,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2023,6,26]]}},"alternative-id":["10.1145\/3606557.3606562"],"URL":"https:\/\/doi.org\/10.1145\/3606557.3606562","relation":{},"ISSN":["0163-5980"],"issn-type":[{"value":"0163-5980","type":"print"}],"subject":[],"published":{"date-parts":[[2023,6,26]]},"assertion":[{"value":"2023-06-28","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}