{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,24]],"date-time":"2026-04-24T09:58:10Z","timestamp":1777024690037,"version":"3.51.4"},"reference-count":64,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2018,11,30]],"date-time":"2018-11-30T00:00:00Z","timestamp":1543536000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Storage"],"published-print":{"date-parts":[[2018,11,30]]},"abstract":"<jats:p>Storage disaggregation separates compute and storage to different nodes to allow for independent resource scaling and, thus, better hardware resource utilization. While disaggregation of hard-drives storage is a common practice, NVMe-SSD (i.e., PCIe-based SSD) disaggregation is considered more challenging. This is because SSDs are significantly faster than hard drives, so the latency overheads (due to both network and CPU processing) as well as the extra compute cycles needed for the offloading stack become much more pronounced.<\/jats:p>\n          <jats:p>In this work, we characterize the overheads of NVMe-SSD disaggregation. We show that NVMe-over-Fabrics (NVMe-oF)\u2014a recently released remote storage protocol specification\u2014reduces the overheads of remote access to a bare minimum, thus greatly increasing the cost-efficiency of Flash disaggregation. Specifically, while recent work showed that SSD storage disaggregation via iSCSI degrades application-level throughput by 20%, we report on negligible performance degradation with NVMe-oF\u2014both when using stress-tests as well as with a more-realistic KV-store workload.<\/jats:p>","DOI":"10.1145\/3239563","type":"journal-article","created":{"date-parts":[[2018,12,4]],"date-time":"2018-12-04T15:32:40Z","timestamp":1543937560000},"page":"1-18","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":27,"title":["Performance Characterization of NVMe-over-Fabrics Storage Disaggregation"],"prefix":"10.1145","volume":"14","author":[{"given":"Zvika","family":"Guz","sequence":"first","affiliation":[{"name":"Samsung Semiconductor, Inc., San Jose, CA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Harry (Huan)","family":"Li","sequence":"additional","affiliation":[{"name":"Samsung Semiconductor, Inc., San Jose, CA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Anahita","family":"Shayesteh","sequence":"additional","affiliation":[{"name":"Samsung Semiconductor, Inc., San Jose, CA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Vijay","family":"Balakrishnan","sequence":"additional","affiliation":[{"name":"Samsung Semiconductor, Inc., San Jose, CA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2018,12,4]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"Amazon. 2008. Amazon Elastic Block Store. Retrieved from https:\/\/aws.amazon.com\/ebs\/.  Amazon. 2008. Amazon Elastic Block Store. Retrieved from https:\/\/aws.amazon.com\/ebs\/."},{"key":"e_1_2_1_2_1","unstructured":"Jens Axboe. 2014. FIO. Retrieved from https:\/\/github.com\/axboe\/fio.  Jens Axboe. 2014. FIO. Retrieved from https:\/\/github.com\/axboe\/fio."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.5555\/2534500"},{"key":"e_1_2_1_4_1","unstructured":"X bit labs. 2016. OCZ Demos 4 TiB 16 TiB Solid-State Drives for Enterprise. https:\/\/web.archive.org\/web\/20130325121004\/http:\/www.xbitlabs.com\/news\/storage\/display\/20120110180208_OCZ_Demos_4TB_16TB_Solid_State_Drives_for_Enterprise.html.  X bit labs. 2016. OCZ Demos 4 TiB 16 TiB Solid-State Drives for Enterprise. https:\/\/web.archive.org\/web\/20130325121004\/http:\/www.xbitlabs.com\/news\/storage\/display\/20120110180208_OCZ_Demos_4TB_16TB_Solid_State_Drives_for_Enterprise.html."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2485732.2485740"},{"key":"e_1_2_1_6_1","unstructured":"Brandon Hoff. 2016. RDMA Interconnects Paving the Way for NVMe over Fabrics Technology. Retrieved from http:\/\/www.roceinitiative.org\/.  Brandon Hoff. 2016. RDMA Interconnects Paving the Way for NVMe over Fabrics Technology. Retrieved from http:\/\/www.roceinitiative.org\/."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/HOTI.2009.23"},{"key":"e_1_2_1_8_1","unstructured":"Chelsio Communications. 2014. Luster over iWARP RDMA at 40Gbps. http:\/\/www.chelsio.com\/wp-content\/uploads\/resources\/Lustre-Over-iWARP-vs-IB-FDR.pdf.  Chelsio Communications. 2014. Luster over iWARP RDMA at 40Gbps. http:\/\/www.chelsio.com\/wp-content\/uploads\/resources\/Lustre-Over-iWARP-vs-IB-FDR.pdf."},{"key":"e_1_2_1_10_1","first-page":"11","article-title":"TPC-C Benchmark Standard Specification","volume":"5","author":"Transaction Processing Performance Council","year":"2010","journal-title":"Revision"},{"key":"e_1_2_1_11_1","volume-title":"Proceedings of the 12th USENIX Conference on File and Storage Technologies (FAST\u201914)","author":"Cully Brendan","year":"2014"},{"key":"e_1_2_1_12_1","unstructured":"Datium. 2018. Open Convergence. Retrieved from http:\/\/www.datrium.com\/open-convergence\/.  Datium. 2018. Open Convergence. Retrieved from http:\/\/www.datrium.com\/open-convergence\/."},{"key":"e_1_2_1_13_1","unstructured":"Excelero. 2017. Excelero NVMesh. Retrieved from https:\/\/www.excelero.com\/product\/nvmesh\/.  Excelero. 2017. Excelero NVMesh. Retrieved from https:\/\/www.excelero.com\/product\/nvmesh\/."},{"key":"e_1_2_1_14_1","unstructured":"Facebook. 2018. RocksDB users. Retrieved from https:\/\/github.com\/facebook\/rocksdb\/blob\/master\/USERS.md.  Facebook. 2018. RocksDB users. Retrieved from https:\/\/github.com\/facebook\/rocksdb\/blob\/master\/USERS.md."},{"key":"e_1_2_1_15_1","unstructured":"Facebook Inc. 2015. RocksDB: A persistent key-value store for fast storage environments. Retrieved from http:\/\/rocksdb.org.  Facebook Inc. 2015. RocksDB: A persistent key-value store for fast storage environments. Retrieved from http:\/\/rocksdb.org."},{"key":"e_1_2_1_16_1","unstructured":"FusionIO. 2013. Fusion-io flash memory as RAM relief.  FusionIO. 2013. Fusion-io flash memory as RAM relief."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2535771.2535778"},{"key":"e_1_2_1_18_1","unstructured":"Kieran Harty. 2016. Don\u2019t Confuse Hyperconvergence With Web-Scale. Retrieved from http:\/\/www.networkcomputing.com\/data-centers\/dont-confuse-hyperconvergence-web-scale\/445839104.  Kieran Harty. 2016. Don\u2019t Confuse Hyperconvergence With Web-Scale. Retrieved from http:\/\/www.networkcomputing.com\/data-centers\/dont-confuse-hyperconvergence-web-scale\/445839104."},{"key":"e_1_2_1_19_1","unstructured":"HGST. 2014. LinkedIn scales to 200 million users with PCIe Flash storage from HGST. Retrieved from https:\/\/www.hgst.com\/sites\/default\/files\/resources\/LinkedIn-Scales-to-200M-Users.pdf.  HGST. 2014. LinkedIn scales to 200 million users with PCIe Flash storage from HGST. Retrieved from https:\/\/www.hgst.com\/sites\/default\/files\/resources\/LinkedIn-Scales-to-200M-Users.pdf."},{"key":"e_1_2_1_20_1","unstructured":"Amber Huffman. 2012. NVM Express Revision 1.1. Retrieved from http:\/\/www.nvmexpress.org\/.  Amber Huffman. 2012. NVM Express Revision 1.1. Retrieved from http:\/\/www.nvmexpress.org\/."},{"key":"e_1_2_1_21_1","unstructured":"IBM Research. 2017. Crail. Retrieved from http:\/\/www.crail.io\/.  IBM Research. 2017. Crail. Retrieved from http:\/\/www.crail.io\/."},{"key":"e_1_2_1_22_1","unstructured":"Facebook Inc. 2015. Open Compute Project. Retrieved from http:\/\/www.opencompute.org\/projects.  Facebook Inc. 2015. Open Compute Project. Retrieved from http:\/\/www.opencompute.org\/projects."},{"key":"e_1_2_1_23_1","unstructured":"Intel. 2016. Intel Xeon Processor E5-2699 v4. Retrieved from https:\/\/ark.intel.com\/products\/91317\/Intel-Xeon-Processor-E5-2699-v4-55M-Cache-2_20-GHz.  Intel. 2016. Intel Xeon Processor E5-2699 v4. Retrieved from https:\/\/ark.intel.com\/products\/91317\/Intel-Xeon-Processor-E5-2699-v4-55M-Cache-2_20-GHz."},{"key":"e_1_2_1_24_1","unstructured":"Intel. 2016. Storage Performance Development Kit. Retrieved from http:\/\/www.spdk.io\/.  Intel. 2016. Storage Performance Development Kit. Retrieved from http:\/\/www.spdk.io\/."},{"key":"e_1_2_1_25_1","unstructured":"Intel. 2017. SPDK NVMe over Fabrics Target. Retrieved from http:\/\/www.spdk.io\/doc\/nvmf.html.  Intel. 2017. SPDK NVMe over Fabrics Target. Retrieved from http:\/\/www.spdk.io\/doc\/nvmf.html."},{"key":"e_1_2_1_26_1","unstructured":"Intel. 2017. SPDK NVMe over Fabrics Target Programming Guide. Retrieved from http:\/\/www.spdk.io\/doc\/nvmf_tgt_pg.html.  Intel. 2017. SPDK NVMe over Fabrics Target Programming Guide. Retrieved from http:\/\/www.spdk.io\/doc\/nvmf_tgt_pg.html."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2749469.2750392"},{"key":"e_1_2_1_28_1","volume-title":"Proceedings of the 8th USENIX Workshop on Hot Topics in Storage and File Systems (HotStorage\u201916)","author":"Kim Hyeong-Jun","year":"2016"},{"key":"e_1_2_1_29_1","unstructured":"John Kim and David Fair. 2016. How ethernet RDMA protocols iWARP and RoCE support NVMe over fabrics (Ethernet Storage Forum). SNIA. Retrieved from https:\/\/www.snia.org\/sites\/default\/files\/ESF\/How_Ethernet_RDMA_Protocols_Support_NVMe_over_Fabrics_Final.pdf.  John Kim and David Fair. 2016. How ethernet RDMA protocols iWARP and RoCE support NVMe over fabrics (Ethernet Storage Forum). SNIA. Retrieved from https:\/\/www.snia.org\/sites\/default\/files\/ESF\/How_Ethernet_RDMA_Protocols_Support_NVMe_over_Fabrics_Final.pdf."},{"key":"e_1_2_1_30_1","volume-title":"Proceedings of the Storage Developer Conference. SNIA.","author":"Kim John F.","year":"2014"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/2901318.2901337"},{"key":"e_1_2_1_32_1","unstructured":"Mike Ko J. Hufferd M. Chadalapaka Uri Elzur H. Shah and P. Thaler. 2003. iSCSI extensions for RDMA specification (version 1.0). Release Specification of the RDMA Consortium (2003).  Mike Ko J. Hufferd M. Chadalapaka Uri Elzur H. Shah and P. Thaler. 2003. iSCSI extensions for RDMA specification (version 1.0). Release Specification of the RDMA Consortium (2003)."},{"key":"e_1_2_1_33_1","unstructured":"Percona Lab. 2008. tpcc-mysql. Retrieved from https:\/\/github.com\/Percona-Lab\/tpcc-mysql.  Percona Lab. 2008. tpcc-mysql. Retrieved from https:\/\/github.com\/Percona-Lab\/tpcc-mysql."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/248209.237157"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/782814.782855"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10723-012-9211-x"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICPP.2013.78"},{"key":"e_1_2_1_38_1","unstructured":"Charlie Manese. 2014. Facebook and open compute designing for efficiency and scale. SC14 Energy Efficient High Performance Computing Working Group.  Charlie Manese. 2014. Facebook and open compute designing for efficiency and scale. SC14 Energy Efficient High Performance Computing Working Group."},{"key":"e_1_2_1_39_1","unstructured":"Mellanox. 2015. Connect X-4 VPI 100Gb. Retrieved from http:\/\/www.mellanox.com\/related-docs\/prod_adapter_cards\/PB_ConnectX-4_VPI_OCP.pdf.  Mellanox. 2015. Connect X-4 VPI 100Gb. Retrieved from http:\/\/www.mellanox.com\/related-docs\/prod_adapter_cards\/PB_ConnectX-4_VPI_OCP.pdf."},{"key":"e_1_2_1_40_1","unstructured":"Mellanox. 2015. SN2700. Retrieved from https:\/\/www.mellanox.com\/related-docs\/prod_eth_switches\/PB_SN2700.pdf.  Mellanox. 2015. SN2700. Retrieved from https:\/\/www.mellanox.com\/related-docs\/prod_eth_switches\/PB_SN2700.pdf."},{"key":"e_1_2_1_41_1","volume-title":"Proceedings of the 11th USENIX Conference on Networked Systems Design and Implementation (NSDI\u201914)","author":"Mickens James","year":"2014"},{"key":"e_1_2_1_42_1","unstructured":"Microsoft. 2016. Improve Performance of a File Server with SMB Direct. Retrieved from https:\/\/technet.microsoft.com\/en-us\/library\/jj134210(v&equals;ws.11).aspx.  Microsoft. 2016. Improve Performance of a File Server with SMB Direct. Retrieved from https:\/\/technet.microsoft.com\/en-us\/library\/jj134210(v&equals;ws.11).aspx."},{"key":"e_1_2_1_43_1","volume-title":"Proceedings of the NVMe All Hands Meeting.","author":"Minturn Dave","year":"2016"},{"key":"e_1_2_1_44_1","volume-title":"Proceedings of the Ethernet Storage Forum. SNIA.","author":"Minturn Dave"},{"key":"e_1_2_1_45_1","unstructured":"MySQL. 1998. MySQL. Retrieved from https:\/\/www.mysql.com\/.  MySQL. 1998. MySQL. Retrieved from https:\/\/www.mysql.com\/."},{"key":"e_1_2_1_46_1","unstructured":"MySQL. 2018. MySQL Customers. Retrieved from https:\/\/www.mysql.com\/customers.  MySQL. 2018. MySQL Customers. Retrieved from https:\/\/www.mysql.com\/customers."},{"key":"e_1_2_1_47_1","unstructured":"Dell Networking. 2015. RDMA over converged ethernet technical brief. http:\/\/pleiades.ucsc.edu\/doc\/dell\/network\/Dell_Networking_RoCE_Configuration.pdf.  Dell Networking. 2015. RDMA over converged ethernet technical brief. http:\/\/pleiades.ucsc.edu\/doc\/dell\/network\/Dell_Networking_RoCE_Configuration.pdf."},{"key":"e_1_2_1_48_1","unstructured":"NVM Express. 2016. NVM Express over Fabric 1.0. Retrieved from http:\/\/www.nvmexpress.org\/.  NVM Express. 2016. NVM Express over Fabric 1.0. Retrieved from http:\/\/www.nvmexpress.org\/."},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/2541940.2541959"},{"key":"e_1_2_1_51_1","volume-title":"Map\/Reduce. In Proceedings of Data @Scale.","author":"Patiejunas Kestutis","year":"2016"},{"key":"e_1_2_1_52_1","unstructured":"Ro Recio P. Culley D. Garcia J. Hilland and B. Metzler. 2005. An RDMA Protocol Specification. Technical Report. IETF Internet-draft draft-ietf-rddp-rdmap-03.txt (work in progress).  Ro Recio P. Culley D. Garcia J. Hilland and B. Metzler. 2005. An RDMA Protocol Specification. Technical Report. IETF Internet-draft draft-ietf-rddp-rdmap-03.txt (work in progress)."},{"key":"e_1_2_1_54_1","unstructured":"Steven Rostedt. 2008. ftrace\u2014Function Tracer. Retrieved from https:\/\/www.kernel.org\/doc\/Documentation\/trace\/ftrace.txt.  Steven Rostedt. 2008. ftrace\u2014Function Tracer. Retrieved from https:\/\/www.kernel.org\/doc\/Documentation\/trace\/ftrace.txt."},{"key":"e_1_2_1_55_1","unstructured":"Brandon Salmon. 2015. Web scale vs. hyperconverged: Understand the differences. Retrieved from http:\/\/www.infoworld.com\/article\/3005572\/enterprise-architecture\/web-scale-vs-hyperconverged-understand-the-differences.html.  Brandon Salmon. 2015. Web scale vs. hyperconverged: Understand the differences. Retrieved from http:\/\/www.infoworld.com\/article\/3005572\/enterprise-architecture\/web-scale-vs-hyperconverged-understand-the-differences.html."},{"key":"e_1_2_1_56_1","unstructured":"Samsung. 2015. PM1725 NVMe PCIe SSD. Retrieved from http:\/\/www.samsung.com\/semiconductor\/global\/file\/insight\/2015\/11\/pm1725-ProdOverview-2015-0.pdf.  Samsung. 2015. PM1725 NVMe PCIe SSD. Retrieved from http:\/\/www.samsung.com\/semiconductor\/global\/file\/insight\/2015\/11\/pm1725-ProdOverview-2015-0.pdf."},{"key":"e_1_2_1_57_1","volume-title":"14th USENIX Conference on File and Storage Technologies (FAST'16)","author":"Schroeder Bianca","year":"2016"},{"key":"e_1_2_1_58_1","unstructured":"Simon Sharwood. 2016. Disaggregated hyper-convergence thinks storage outside the box. Retrieved from https:\/\/www.theregister.co.uk\/2016\/03\/24\/disaggregated_hyper_convergence\/.  Simon Sharwood. 2016. Disaggregated hyper-convergence thinks storage outside the box. Retrieved from https:\/\/www.theregister.co.uk\/2016\/03\/24\/disaggregated_hyper_convergence\/."},{"key":"e_1_2_1_59_1","volume-title":"Proceedings of the USENIX Annual Technical Conference (USENIX-ATC\u201914)","author":"Shin Woong"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCAC.2014.14"},{"key":"e_1_2_1_61_1","unstructured":"T10. 2003. SCSI RDMA protocol-2 (SRP-2). Retrieved from http:\/\/www.t10.org\/ftp\/t10\/drafts\/srp2\/srp2r00a.pdf.  T10. 2003. SCSI RDMA protocol-2 (SRP-2). Retrieved from http:\/\/www.t10.org\/ftp\/t10\/drafts\/srp2\/srp2r00a.pdf."},{"key":"e_1_2_1_62_1","unstructured":"VMWare. 2017. VMware vSAN. Retrieved from http:\/\/www.vmware.com\/products\/virtual-san.html.  VMWare. 2017. VMware vSAN. Retrieved from http:\/\/www.vmware.com\/products\/virtual-san.html."},{"key":"e_1_2_1_63_1","volume-title":"SPDK: Building blocks for scalable, high performance storage applications (Storage Developer Conference). SNIA.","author":"Walker Benjamin","year":"2016"},{"key":"e_1_2_1_64_1","volume-title":"\u201cproblematically fast","author":"Warfield Andrew"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/2757667.2757684"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.5555\/2208461.2208464"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/2619092"}],"container-title":["ACM Transactions on Storage"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3239563","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3239563","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T01:08:20Z","timestamp":1750208900000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3239563"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,11,30]]},"references-count":64,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2018,11,30]]}},"alternative-id":["10.1145\/3239563"],"URL":"https:\/\/doi.org\/10.1145\/3239563","relation":{},"ISSN":["1553-3077","1553-3093"],"issn-type":[{"value":"1553-3077","type":"print"},{"value":"1553-3093","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,11,30]]},"assertion":[{"value":"2018-04-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-07-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-12-04","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}