{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,20]],"date-time":"2026-01-20T10:28:29Z","timestamp":1768904909517,"version":"3.49.0"},"reference-count":27,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2022,1,31]],"date-time":"2022-01-31T00:00:00Z","timestamp":1643587200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000185","name":"Defense Advanced Research Projects Agency","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100000185","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Embed. Comput. Syst."],"published-print":{"date-parts":[[2022,1,31]]},"abstract":"<jats:p>With data-intensive artificial intelligence (AI) and machine learning (ML) applications rapidly surging, modern high-performance embedded systems, with heterogeneous computing resources, critically demand low-latency and high-bandwidth data communication. As such, the newly emerging NVMe (Non-Volatile Memory Express) protocol, with parallel queuing, access prioritization, and optimized I\/O arbitration, starts to be widely adopted as a de facto fast I\/O communication interface. However, effectively leveraging the potential of modern NVMe storage proves to be nontrivial and demands fine-grained control, high processing concurrency, and application-specific optimization. Fortunately, modern FPGA devices, capable of efficient parallel processing and application-specific programmability, readily meet the underlying physical layer requirements of the NVMe protocol, therefore providing unprecedented opportunities to implementing a rich-featured NVMe middleware to benefit modern high-performance embedded computing.<\/jats:p>\n          <jats:p>In this article, we present how to rethink existing accessing mechanisms of NVMe storage and devise innovative hardware-assisted solutions to accelerating NVMe data access performance for the high-performance embedded computing system. Our key idea is to exploit the massively parallel I\/O queuing capability, provided by the NVMe storage system, through leveraging FPGAs\u2019 reconfigurability and native hardware computing power to operate transparently to the main processor. Specifically, our DirectNVM system aims at providing effective hardware constructs for facilitating high-performance and scalable userspace storage applications through (1) hardening all the essential NVMe driver functionalities, therefore avoiding expensive OS syscalls and enabling zero-copy data access from the application, (2) relying on hardware for the I\/O communication control instead of relying on OS-level interrupts that can significantly reduce both total I\/O latency and its variance, and (3) exposing cutting-edge and application-specific weighted-round-robin I\/O traffic scheduling to the userspace.<\/jats:p>\n          <jats:p>\n            To validate our design methodology, we developed a complete DirectNVM system utilizing the Xilinx Zynq MPSoC architecture that incorporates a high-performance application processor (APU) equipped with DDR4 system memory and a hardened configurable PCIe Gen3 block in its programmable logic part. We then measured the storage bandwidth and I\/O latency of both our DirectNVM system and a conventional OS-based system when executing the standard FIO benchmark suite\u00a0[\n            <jats:xref ref-type=\"bibr\">2<\/jats:xref>\n            ]. Specifically, compared against the PetaLinux built-in kernel driver code running on a Zynq MPSoC, our DirectNVM has shown to achieve up to 18.4\u00d7 higher throughput and up to 4.5\u00d7 lower latency. To ensure the fairness of our performance comparison, we also measured our DirectNVM system against the Intel SPDK\u00a0[\n            <jats:xref ref-type=\"bibr\">26<\/jats:xref>\n            ], a highly optimized userspace asynchronous NVMe I\/O framework running on a X86 PC system. Our experiment results have shown that our DirectNVM, even running on a considerably less powerful embedded ARM processor than a full-scale AMD processor, achieved up to 2.2\u00d7 higher throughput and 1.3\u00d7 lower latency. Furthermore, by experimenting with a multi-threading test case, we have demonstrated that our DirectNVM\u2019s weighted-round-robin scheduling can significantly optimize the bandwidth allocation between latency-constraint frontend applications and other backend applications in real-time systems. Finally, we have developed a theoretical framework of performance modeling with classic queuing theory that can quantitatively define the relationship between a system\u2019s I\/O performance and its I\/O implementation.\n          <\/jats:p>","DOI":"10.1145\/3463911","type":"journal-article","created":{"date-parts":[[2022,2,10]],"date-time":"2022-02-10T15:14:22Z","timestamp":1644506062000},"page":"1-24","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":5,"title":["DirectNVM: Hardware-accelerated NVMe SSDs for High-performance Embedded Computing"],"prefix":"10.1145","volume":"21","author":[{"given":"Yu","family":"Zou","sequence":"first","affiliation":[{"name":"University of Central Florida, Orlando, FL, US"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Amro","family":"Awad","sequence":"additional","affiliation":[{"name":"North Carolina State University, Raleigh, NC, US"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mingjie","family":"Lin","sequence":"additional","affiliation":[{"name":"University of Central Florida, Orlando, FL, US"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2022,2,10]]},"reference":[{"key":"e_1_3_1_2_2","unstructured":"Jens Axboe. 2006. Linux Kernel 2.6.18 - Make CFQ the default IO scheduler. Retrieved from https:\/\/git.kernel.org\/cgit\/linux\/kernel\/git\/torvalds\/linux.git\/commit\/?id=b17fd9bceb99610f6dc7998c9a4ed6b71520be2b."},{"key":"e_1_3_1_3_2","unstructured":"Jens Axboe. 2020. Flexible I\/O. Retrieved from https:\/\/github.com\/axboe\/fio."},{"key":"e_1_3_1_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/2485732.2485740"},{"key":"e_1_3_1_5_2","unstructured":"Opsero Electronic Design. 2020. FPGA Drive FMC. Retrieved from https:\/\/opsero.com\/product\/fpga-drive-fmc-dual\/."},{"key":"e_1_3_1_6_2","unstructured":"NVMe Express. 2017. NVMe Specification 1.3. Retrieved from https:\/\/nvmexpress.org\/wp-content\/uploads\/NVM_Express_Revision_1.3.pdf."},{"key":"e_1_3_1_7_2","unstructured":"OpenPOWER Accelerator Work Group. 2020. CAPI Storage Network and Analytics Programming (SNAP) Framework. Retrieved from https:\/\/developer.ibm.com\/linuxonpower\/capi\/snap."},{"key":"e_1_3_1_8_2","first-page":"247","article-title":"Analyzing, modeling, and provisioning QoS for NVME SSDs","author":"Gugnani Shashank","year":"2019","unstructured":"Shashank Gugnani, Xiaoyi Lu, and Dhabaleswar K. D. K. Panda. 2019. Analyzing, modeling, and provisioning QoS for NVME SSDs. In Proceedings of the 11th IEEE\/ACM International Conference on Utility and Cloud Computing. 247\u2013256. DOI:DOI:https:\/\/doi.org\/10.1109\/UCC.2018.00033","journal-title":"I"},{"key":"e_1_3_1_9_2","unstructured":"Jeremy Hsu. 2018. It\u2019s Time to Think Beyond Cloud Computing. Retrieved from https:\/\/www.wired.com\/story\/its-time-to-think-beyond-cloud-computing\/."},{"key":"e_1_3_1_10_2","unstructured":"Intel. 2015. Performance Benchmarking for PCIe and NVMe Enterprise Solid-State Drives. Retrieved from https:\/\/www.intel.com\/content\/dam\/www\/public\/us\/en\/documents\/white-papers\/performance-pcie-nvme-enterprise-ssds-white-paper.pdf."},{"key":"e_1_3_1_11_2","unstructured":"Intel. 2020. Intel Optane Persistent Memory. Retrieved from https:\/\/www.intel.com\/content\/www\/us\/en\/architecture-and-technology\/optane-dc-persistent-memory.html."},{"key":"e_1_3_1_12_2","unstructured":"Intel. 2020. Open Programmable Accelerator Engine. Retrieved from https:\/\/opae.github.io\/latest\/index.html."},{"key":"e_1_3_1_13_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSST.2013.6558444"},{"key":"e_1_3_1_14_2","volume-title":"Proceedings of the 8th USENIX Workshop on Hot Topics in Storage and File Systems (HotStorage\u201916)","author":"Kim Hyeong-Jun","year":"2016","unstructured":"Hyeong-Jun Kim, Young-Sik Lee, and Jin-Soo Kim. 2016. NVMeDirect: A user-space I\/O framework for application-specific optimization on nvme SSDs. In Proceedings of the 8th USENIX Workshop on Hot Topics in Storage and File Systems (HotStorage\u201916)."},{"key":"e_1_3_1_15_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4614-5317-8"},{"key":"e_1_3_1_16_2","volume-title":"Proceedings of the Vault Linux Storage and Filesystems Conference","author":"Moal Damien Le","year":"2017","unstructured":"Damien Le Moal. 2017. I\/O Latency Optimization with polling. In Proceedings of the Vault Linux Storage and Filesystems Conference."},{"key":"e_1_3_1_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/RTSS46320.2019.00016"},{"key":"e_1_3_1_18_2","first-page":"684-700","article-title":"High-performance energy-efficient multicore embedded computing","volume":"23","author":"Munir Arslan","year":"2012","unstructured":"Arslan Munir, Sanjay Ranka, and Ann Gordon-Ross. 2012. High-performance energy-efficient multicore embedded computing. Trans. Parallel Distrib. Syst. 23 (05 2012), 684-700. DOI:DOI:https:\/\/doi.org\/10.1109\/TPDS.2011.214","journal-title":"Trans. Parallel Distrib. Syst."},{"key":"e_1_3_1_19_2","first-page":"379","volume-title":"Proceedings of the USENIX Annual Technical Conference (USENIXATC\u201919)","author":"Ruan Zhenyuan","year":"2019","unstructured":"Zhenyuan Ruan, Tong He, and Jason Cong. 2019. INSIDER: Designing in-storage computing system for emerging high-performance drive. In Proceedings of the USENIX Annual Technical Conference (USENIXATC\u201919). 379\u2013394."},{"key":"e_1_3_1_20_2","unstructured":"Samsung. 2020. Samsung 970 EVO Plus Specification. Retrieved from https:\/\/www.samsung.com\/semiconductor\/minisite\/ssd\/product\/consumer\/970evoplus\/."},{"key":"e_1_3_1_21_2","doi-asserted-by":"publisher","DOI":"10.4218\/etrij.14.0113.0812"},{"key":"e_1_3_1_22_2","volume-title":"Low Overhead & Energy Efficient Storage Path for Next Generation Computer Systems.","author":"Stratikopoulos Athanasios","year":"2019","unstructured":"Athanasios Stratikopoulos. 2019. Low Overhead & Energy Efficient Storage Path for Next Generation Computer Systems.Ph.D. Dissertation. University of Manchester."},{"key":"e_1_3_1_23_2","doi-asserted-by":"publisher","DOI":"10.29007\/rgnt"},{"key":"e_1_3_1_24_2","unstructured":"Xilinx. 2019. DMA\/Bridge Subsystem for PCI Express v4.1. Retrieved from https:\/\/www.xilinx.com\/support\/documentation\/ip_documentation\/xdma\/v4_1\/pg195-pcie-dma.pdf."},{"key":"e_1_3_1_25_2","unstructured":"Xilinx. 2019. PetaLinux Tools Documentation. Retrieved from https:\/\/www.xilinx.com\/support\/documentation\/sw_manuals\/xilinx2019_2\/ug1144-petalinux-tools-reference-guide.pdf."},{"key":"e_1_3_1_26_2","unstructured":"Xilinx. 2019. UltraScale+ Devices Integrated Block for PCI Express v1.3. Retrieved from https:\/\/www.xilinx.com\/support\/documentation\/ip_documentation\/pcie4_uscale_plus\/v1_3\/pg213-pcie4-ultrascale-plus.pdf."},{"key":"e_1_3_1_27_2","doi-asserted-by":"publisher","DOI":"10.1109\/CloudCom.2017.14"},{"key":"e_1_3_1_28_2","unstructured":"Yu Zou. 2020. DirectNVM. Retrieved from https:\/\/github.com\/yu-zou\/DirectNVM."}],"container-title":["ACM Transactions on Embedded Computing Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3463911","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3463911","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T21:25:09Z","timestamp":1750195509000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3463911"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,1,31]]},"references-count":27,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2022,1,31]]}},"alternative-id":["10.1145\/3463911"],"URL":"https:\/\/doi.org\/10.1145\/3463911","relation":{},"ISSN":["1539-9087","1558-3465"],"issn-type":[{"value":"1539-9087","type":"print"},{"value":"1558-3465","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,1,31]]},"assertion":[{"value":"2020-12-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-04-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-02-10","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}