{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,28]],"date-time":"2025-08-28T14:10:05Z","timestamp":1756390205287,"version":"3.44.0"},"reference-count":25,"publisher":"Association for Computing Machinery (ACM)","issue":"9","license":[{"start":{"date-parts":[[2025,8,28]],"date-time":"2025-08-28T00:00:00Z","timestamp":1756339200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Commun. ACM"],"published-print":{"date-parts":[[2025,9]]},"abstract":"<jats:p>The unrelenting growth in the memory demands of datacenter applications, combined with DRAM\u2019s volatile prices and ever-increasing costs, has made DRAM a major infrastructure expense. Alternative technologies, such as NVMe SSDs and emerging NVM devices, offer higher capacity than DRAM at a fraction of the cost and power. A promising approach is to transparently offload colder memory to cheaper memory technologies via kernel or hypervisor techniques. The key challenge, however, is to develop a datacenter-scale solution that is robust in dealing with diverse workloads and large performance variance of different offload devices, such as compressed memory, SSD, and NVM. This paper presents TMO, Meta\u2019s transparent memory offloading solution for heterogeneous datacenter environments. TMO introduces a new Linux kernel mechanism that directly measures, in realtime, process stalls due to resource shortages across CPU, memory, and I\/O. Guided by this information and without any prior application knowledge, TMO automatically adjusts how much memory to offload to heterogeneous devices (e.g., compressed memory or SSD) according to the device\u2019s performance characteristics and the application\u2019s sensitivity to memory-access slowdown. To maximize memory savings, TMO targets both anonymous memory and file cache, balancing the swap-in rate of anonymous memory and the reload rate of file pages that were recently evicted from the file cache. Moreover, it identifies offloading opportunities not only from the application containers but also from the sidecar containers that provide infrastructure-level functions. TMO has been in production since 2021, saving 20\u201332% of total memory across millions of servers in our hyperscale datacenter fleet. We have successfully upstreamed TMO into the Linux kernel.<\/jats:p>","DOI":"10.1145\/3746651","type":"journal-article","created":{"date-parts":[[2025,8,27]],"date-time":"2025-08-27T16:42:13Z","timestamp":1756312933000},"page":"102-110","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["TMO: Transparent Memory Offloading in Datacenters"],"prefix":"10.1145","volume":"68","author":[{"given":"Johannes","family":"Weiner","sequence":"first","affiliation":[{"name":"Meta Inc., Menlo Park, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Niket","family":"Agarwal","sequence":"additional","affiliation":[{"name":"Meta Inc., Menlo Park, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dan","family":"Schatzberg","sequence":"additional","affiliation":[{"name":"Meta Inc., Menlo Park, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Leon","family":"Yang","sequence":"additional","affiliation":[{"name":"Meta Inc., Menlo Park, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hao","family":"Wang","sequence":"additional","affiliation":[{"name":"Meta Inc., Menlo Park, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Blaise","family":"Sanouillet","sequence":"additional","affiliation":[{"name":"Meta Inc., Menlo Park, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Bikash","family":"Sharma","sequence":"additional","affiliation":[{"name":"Meta Inc., Menlo Park, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tejun","family":"Heo","sequence":"additional","affiliation":[{"name":"Meta Inc., Menlo Park, USA, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mayank","family":"Jain","sequence":"additional","affiliation":[{"name":"Meta Inc., Menlo Park, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chunqiang","family":"Tang","sequence":"additional","affiliation":[{"name":"Meta Inc., Menlo Park, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dimitrios","family":"Skarlatos","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2025,8,28]]},"reference":[{"key":"e_1_3_1_2_2","doi-asserted-by":"crossref","unstructured":"Agarwal N. and Wenisch T.F. Thermostat: Application-transparent page management for two-tiered main memory. In Proceedings of the 22nd Intern. Conf. on Architectural Support for Programming Languages and Operating Systems\u00a0(2017) \u00a0631\u2013644.","DOI":"10.1145\/3037697.3037706"},{"key":"e_1_3_1_3_2","doi-asserted-by":"crossref","unstructured":"Aguilera M.K. et al. Remote memory in the age of fast networks. In Proceedings of the 2017 Symp. on Cloud Computing 121\u2013127.","DOI":"10.1145\/3127479.3131612"},{"key":"e_1_3_1_4_2","unstructured":"Al Maruf H. and Chowdhury M. Effectively prefetching remote memory with leap. In 2020 USENIX Annual Technical Conf. \u00a0843\u2013857."},{"key":"e_1_3_1_5_2","unstructured":"Berg B. et al. The CacheLib caching engine: Design and Experiences at Scale. In Proceedings of the 14th USENIX Symp. on Operating Systems Design and Implementation 2020."},{"key":"e_1_3_1_6_2","unstructured":"Compute Express Link;\u00a0https:\/\/www.computeexpresslink.org\/."},{"key":"e_1_3_1_7_2","unstructured":"DAMON: Data Access MONitor;\u00a0https:\/\/www.kernel.org\/doc\/html\/v5.17\/vm\/damon\/index.html."},{"key":"e_1_3_1_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/3286588"},{"key":"e_1_3_1_9_2","unstructured":"Gu J. et al. Efficient memory disaggregation with infiniswap. In Proceedings of the\u00a014th USENIX Symp. on Networked Systems Design and Implementation\u00a0(2017) \u00a0649\u2013667."},{"key":"e_1_3_1_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3477132.3483567"},{"key":"e_1_3_1_11_2","doi-asserted-by":"crossref","unstructured":"Kanev S. et al. Profiling a warehouse-scale computer. In Proceedings of the 42nd Annual Intern. Symp. on Computer Architecture (2015) \u00a0158\u2013169.","DOI":"10.1145\/2749469.2750392"},{"key":"e_1_3_1_12_2","doi-asserted-by":"crossref","unstructured":"Kannan S. Ren Y. and Bhattacharjee A. KLOCs: Kernel-level object contexts for heterogeneous memory systems.\u00a0Association for Computing Machinery\u00a0(2021) \u00a065\u201378.","DOI":"10.1145\/3445814.3446745"},{"key":"e_1_3_1_13_2","doi-asserted-by":"crossref","unstructured":"Lagar-Cavilla A. et al. Software-defined far memory in warehouse-scale computers. ASPLOS\u00a0(2019).","DOI":"10.1145\/3297858.3304053"},{"key":"e_1_3_1_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/IEDM.2016.7838026"},{"key":"e_1_3_1_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/1555815.1555789"},{"key":"e_1_3_1_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/3582016.3582063"},{"key":"e_1_3_1_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/3477132.3483568"},{"key":"e_1_3_1_18_2","doi-asserted-by":"crossref","unstructured":"Qureshi M.K. Srinivasan V. and Rivers J.A. Scalable high performance main memory system using phase-change memory technology. In Proceedings of the 36th Ann. Intern. Symp. on Computer Architecture\u00a0(2009) \u00a024\u201333.","DOI":"10.1145\/1555754.1555760"},{"key":"e_1_3_1_19_2","doi-asserted-by":"crossref","unstructured":"Ramos L.E. Gorbatov E. and Bianchini R. Page placement in hybrid memory systems. In Proceedings of the Intern. Conf. on Supercomputing\u00a0(2011) \u00a085\u201395.","DOI":"10.1145\/1995896.1995911"},{"key":"e_1_3_1_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/3477132.3483550"},{"key":"e_1_3_1_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/3314221.3314608"},{"key":"e_1_3_1_22_2","doi-asserted-by":"crossref","unstructured":"Wang Z. et al. Characterizing and modeling non-volatile memory systems. In Proceedings of the\u00a02020 53rd Annual IEEE\/ACM Intern. Symp. on Microarchitecture\u00a0(2020) \u00a0496\u2013508.","DOI":"10.1109\/MICRO50266.2020.00049"},{"key":"e_1_3_1_23_2","doi-asserted-by":"crossref","unstructured":"Yan Z. Lustig D. Nellans D. and Bhattacharjee A. Nimble page management for tiered memory systems. In Proceedings of the 24th Intern. Conf. on Architectural Support for Programming Languages and Operating Systems\u00a0(2019) \u00a0331\u2013345.","DOI":"10.1145\/3297858.3304024"},{"key":"e_1_3_1_24_2","doi-asserted-by":"publisher","DOI":"10.5555\/3386691.3386708"},{"key":"e_1_3_1_25_2","first-page":"207","volume-title":"Proceedings of the\u00a017th USENIX Conf. on File and Storage Technologies","author":"Zheng S.","year":"2019","unstructured":"Zheng, S., Hoseinzadeh, M., and Swanson, S. Ziggurat: A tiered file system for non-volatile main memories and disks. In Proceedings of the\u00a017th USENIX Conf. on File and Storage Technologies, USENIX Association\u00a0(Feb. 2019),\u00a0207\u2013219."},{"key":"e_1_3_1_26_2","unstructured":"zswap;\u00a0https:\/\/www.kernel.org\/doc\/html\/latest\/vm\/zswap.html."}],"container-title":["Communications of the ACM"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3746651","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,28]],"date-time":"2025-08-28T13:56:00Z","timestamp":1756389360000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3746651"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,28]]},"references-count":25,"journal-issue":{"issue":"9","published-print":{"date-parts":[[2025,9]]}},"alternative-id":["10.1145\/3746651"],"URL":"https:\/\/doi.org\/10.1145\/3746651","relation":{},"ISSN":["0001-0782","1557-7317"],"issn-type":[{"type":"print","value":"0001-0782"},{"type":"electronic","value":"1557-7317"}],"subject":[],"published":{"date-parts":[[2025,8,28]]},"assertion":[{"value":"2025-08-28","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}