{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,24]],"date-time":"2026-04-24T09:58:02Z","timestamp":1777024682946,"version":"3.51.4"},"reference-count":44,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2021,6,2]],"date-time":"2021-06-02T00:00:00Z","timestamp":1622592000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGOPS Oper. Syst. Rev."],"published-print":{"date-parts":[[2021,6,2]]},"abstract":"<jats:p>Tracing is a popular method for evaluating, investigating, and modeling the performance of today's storage systems. Tracing has become crucial with the increase in complexity of modern storage applications\/systems, that are manipulating an ever-increasing amount of data and are subject to extreme performance requirements. There exists many tracing tools focusing either on the user-level or the kernel-level, however we observe the lack of a unified tracer targeting both levels: this prevents a comprehensive understanding of modern applications' storage performance profiles. In this paper, we present EZIOTracer, a unified I\/O tracer for both (Linux) kernel and user spaces, targeting data intensive applications. EZIOTracer is composed of a userland as well as a kernel space tracer, complemented with a trace analysis framework able to merge the output of the two tracers, and in particular to relate user-level events to kernel-level ones, and vice-versa. On the kernel side, EZIOTracer relies on eBPF to offer safe, low-overhead, low memory footprint, and flexible tracing capabilities. We demonstrate using FIO benchmark the ability of EZIOTracer to track down I\/O performance issues by relating events recorded at both the kernel and user levels. We show that this can be achieved with a relatively low overhead that ranges from 2% to 26% depending on the I\/O intensity.<\/jats:p>","DOI":"10.1145\/3469379.3469391","type":"journal-article","created":{"date-parts":[[2021,6,6]],"date-time":"2021-06-06T12:43:56Z","timestamp":1622983436000},"page":"88-98","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["EZIOTracer"],"prefix":"10.1145","volume":"55","author":[{"given":"Mohammed","family":"Islam Naas","sequence":"first","affiliation":[{"name":"University of Western Brittany, France"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Fran\u00e7ois","family":"Trahay","sequence":"additional","affiliation":[{"name":"T\u00e9l\u00e9com SudParis, Institut Polytechnique de Paris, Paris, France"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alexis","family":"Colin","sequence":"additional","affiliation":[{"name":"T\u00e9l\u00e9com SudParis, Institut Polytechnique de Paris, Paris, France"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Pierre","family":"Olivier","sequence":"additional","affiliation":[{"name":"The University of Manchester, Manchester, United Kingdom"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"St\u00e9phane","family":"Rubini","sequence":"additional","affiliation":[{"name":"University of Western Brittany, France"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Frank","family":"Singhoff","sequence":"additional","affiliation":[{"name":"University of Western Brittany, France"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jalil","family":"Boukhobza","sequence":"additional","affiliation":[{"name":"ENSTA Bretagne, France"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2021,6,6]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"Cloud trace web site. URL https:\/\/cloud.google.com\/trace.  Cloud trace web site. URL https:\/\/cloud.google.com\/trace."},{"key":"e_1_2_1_2_1","unstructured":"Datadoghq web site. URL https:\/\/www.datadoghq.com\/.  Datadoghq web site. URL https:\/\/www.datadoghq.com\/."},{"key":"e_1_2_1_3_1","unstructured":"Jackplay web site. URL https:\/\/github.com\/alfredxiao\/jackplay.  Jackplay web site. URL https:\/\/github.com\/alfredxiao\/jackplay."},{"key":"e_1_2_1_4_1","unstructured":"Service pilot web site. URL https:\/\/www.servicepilot.com.  Service pilot web site. URL https:\/\/www.servicepilot.com."},{"key":"e_1_2_1_5_1","unstructured":"Bcc github repository. URL https:\/\/github.com\/iovisor\/bcc.  Bcc github repository. URL https:\/\/github.com\/iovisor\/bcc."},{"key":"e_1_2_1_6_1","unstructured":"bpftrace github repository. URL https:\/\/github.com\/iovisor\/bprace.  bpftrace github repository. URL https:\/\/github.com\/iovisor\/bprace."},{"key":"e_1_2_1_7_1","unstructured":"iotrace github repository. URL https:\/\/github.com\/Open-CAS\/ standalone-linux-io-tracer.  iotrace github repository. URL https:\/\/github.com\/Open-CAS\/ standalone-linux-io-tracer."},{"key":"e_1_2_1_8_1","unstructured":"Open telemetry web site. URL https:\/\/opentelemetry.io.  Open telemetry web site. URL https:\/\/opentelemetry.io."},{"key":"e_1_2_1_9_1","volume-title":"PROPER - 5thWorkshop on Productivity and Performance -","author":"Aulagnon Charles","year":"2012","unstructured":"Charles Aulagnon , Damien Martin-Guillerez , Fran\u00e7ois Rue , and Fran\u00e7ois Trahay . Runtime function instrumentation with EZTrace . In PROPER - 5thWorkshop on Productivity and Performance - 2012 , August 2012. Charles Aulagnon, Damien Martin-Guillerez, Fran\u00e7ois Rue, and Fran\u00e7ois Trahay. Runtime function instrumentation with EZTrace. In PROPER - 5thWorkshop on Productivity and Performance - 2012, August 2012."},{"key":"e_1_2_1_10_1","volume-title":"Fio-flexible io tester","author":"Axboe Jens","year":"2014","unstructured":"Jens Axboe . Fio-flexible io tester , 2014 . URL https:\/\/fio.readthedocs.io. Jens Axboe. Fio-flexible io tester, 2014. URL https:\/\/fio.readthedocs.io."},{"key":"e_1_2_1_11_1","volume-title":"Blktrace linux manual page","author":"Axboe Jens","year":"2006","unstructured":"Jens Axboe and Alan D Brunelle . Blktrace linux manual page , 2006 . URL https:\/\/man7.org\/linux\/man-pages\/man8\/blktrace.8.html. Jens Axboe and Alan D Brunelle. Blktrace linux manual page, 2006. URL https:\/\/man7.org\/linux\/man-pages\/man8\/blktrace.8.html."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.5555\/3110834"},{"key":"e_1_2_1_13_1","volume-title":"Understanding and improving computational science storage access through continuous characterization. ACM Transactions on Storage (TOS), 7(3):1--26","author":"Carns Philip","year":"2011","unstructured":"Philip Carns , Kevin Harms , William Allcock , Charles Bacon , Samuel Lang , Robert Latham , and Robert Ross . Understanding and improving computational science storage access through continuous characterization. ACM Transactions on Storage (TOS), 7(3):1--26 , 2011 . Philip Carns, Kevin Harms, William Allcock, Charles Bacon, Samuel Lang, Robert Latham, and Robert Ross. Understanding and improving computational science storage access through continuous characterization. ACM Transactions on Storage (TOS), 7(3):1--26, 2011."},{"key":"e_1_2_1_14_1","first-page":"37","volume-title":"Tools for High Performance Computing","author":"Coulomb Kevin","year":"2011","unstructured":"Kevin Coulomb , Augustin Degomme , Mathieu Faverge , and Fran\u00e7ois Trahay . An open-source tool-chain for performance analysis . In Tools for High Performance Computing 2011 , pages 37 -- 48 . Springer , 2012. Kevin Coulomb, Augustin Degomme, Mathieu Faverge, and Fran\u00e7ois Trahay. An open-source tool-chain for performance analysis. In Tools for High Performance Computing 2011, pages 37--48. Springer, 2012."},{"key":"e_1_2_1_15_1","volume-title":"ID-IMAG","author":"de Oliveira Stein Benhur","year":"2010","unstructured":"Benhur de Oliveira Stein , J Chassin de Kergommeaux , and G Mouni\u00e9 . Paj\u00e9 trace file format. Technical report , ID-IMAG , Grenoble, France , 2010 . Benhur de Oliveira Stein, J Chassin de Kergommeaux, and G Mouni\u00e9. Paj\u00e9 trace file format. Technical report, ID-IMAG, Grenoble, France, 2010."},{"key":"e_1_2_1_16_1","first-page":"209","volume-title":"Ottawa Linux Symposium","author":"Desnoyers Mathieu","year":"2006","unstructured":"Mathieu Desnoyers and Michel R Dagenais . The lttng tracer: A low impact performance and behavior monitor for gnu\/linux . In Ottawa Linux Symposium , pages 209 -- 224 , 2006 . Mathieu Desnoyers and Michel R Dagenais. The lttng tracer: A low impact performance and behavior monitor for gnu\/linux. In Ottawa Linux Symposium, pages 209--224, 2006."},{"issue":"1","key":"e_1_2_1_17_1","first-page":"37","article-title":"file system in linux environment: Features and performance analysis","volume":"6","author":"Djordjevic Borislav","year":"2012","unstructured":"Borislav Djordjevic and Valentina Timcenko . Ext4 file system in linux environment: Features and performance analysis . International Journal of Computers , 6 ( 1 ): 37 -- 45 , 2012 . Borislav Djordjevic and Valentina Timcenko. Ext4 file system in linux environment: Features and performance analysis. International Journal of Computers, 6(1):37--45, 2012.","journal-title":"International Journal of Computers"},{"key":"e_1_2_1_18_1","volume-title":"Systemtap tutorial","author":"Eigler Frank Ch","year":"2010","unstructured":"Frank Ch Eigler . Systemtap tutorial , 2010 . URL http:\/\/www.sourceware. org\/systemtap\/tutorial.pdf. Frank Ch Eigler. Systemtap tutorial, 2010. URL http:\/\/www.sourceware. org\/systemtap\/tutorial.pdf."},{"key":"e_1_2_1_19_1","first-page":"481","volume-title":"PARCO","volume":"22","author":"Eschweiler Dominic","year":"2011","unstructured":"Dominic Eschweiler , Michael Wagner , Markus Geimer , Andreas Kn\u00fcpfer , Wolfgang E Nagel , and Felix Wolf . Open trace format 2: The next generation of scalable trace formats and support libraries . In PARCO , volume 22 , pages 481 -- 490 , 2011 . Dominic Eschweiler, Michael Wagner, Markus Geimer, Andreas Kn\u00fcpfer, Wolfgang E Nagel, and Felix Wolf. Open trace format 2: The next generation of scalable trace formats and support libraries. In PARCO, volume 22, pages 481--490, 2011."},{"key":"e_1_2_1_20_1","volume-title":"Sysstat utilities home page. pagespersoorange. fr\/sebastien. godard","author":"Godard Sebastien","year":"2010","unstructured":"Sebastien Godard . Sysstat utilities home page. pagespersoorange. fr\/sebastien. godard , 2010 . Sebastien Godard. Sysstat utilities home page. pagespersoorange. fr\/sebastien. godard, 2010."},{"key":"e_1_2_1_21_1","unstructured":"Brendan Gregg. Perf examples. URL http:\/\/www.brendangregg.com\/ perf.html.  Brendan Gregg. Perf examples. URL http:\/\/www.brendangregg.com\/ perf.html."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.5555\/3384134"},{"key":"e_1_2_1_23_1","volume-title":"Mac OS X","author":"Gregg Brendan","year":"2011","unstructured":"Brendan Gregg and Jim Mauro . DTrace: Dynamic Tracing in Oracle Solaris , Mac OS X , and FreeBSD. Prentice Hall Professional , 2011 . Brendan Gregg and Jim Mauro. DTrace: Dynamic Tracing in Oracle Solaris, Mac OS X, and FreeBSD. Prentice Hall Professional, 2011."},{"key":"e_1_2_1_24_1","unstructured":"Cespedes Juan and Machata Petr. Ltrace man page. URL https:\/\/man7. org\/linux\/man-pages\/man1\/ltrace.1.html.  Cespedes Juan and Machata Petr. Ltrace man page. URL https:\/\/man7. org\/linux\/man-pages\/man1\/ltrace.1.html."},{"key":"e_1_2_1_25_1","volume-title":"Digital me ontology and ethics. arXiv preprint arXiv:2012.14325","author":"Kocarev Ljupco","year":"2020","unstructured":"Ljupco Kocarev and Jasna Koteska . Digital me ontology and ethics. arXiv preprint arXiv:2012.14325 , 2020 . Ljupco Kocarev and Jasna Koteska. Digital me ontology and ethics. arXiv preprint arXiv:2012.14325, 2020."},{"key":"e_1_2_1_26_1","volume-title":"Slicing the io execution with relaytracer. arXiv preprint arXiv:1906.07124","author":"Lee Ganguk","year":"2019","unstructured":"Ganguk Lee , Yeaseul Park , Jeongseob Ahn , and Youngjin Kwon . Slicing the io execution with relaytracer. arXiv preprint arXiv:1906.07124 , 2019 . Ganguk Lee, Yeaseul Park, Jeongseob Ahn, and Youngjin Kwon. Slicing the io execution with relaytracer. arXiv preprint arXiv:1906.07124, 2019."},{"key":"e_1_2_1_27_1","volume-title":"Proceedings of the 1999 Extreme Linux Workshop","author":"Ross Walter B","year":"1999","unstructured":"Walter B Ligon III and Robert B Ross . An overview of the parallel virtual file system . In Proceedings of the 1999 Extreme Linux Workshop , 1999 . Walter B Ligon III and Robert B Ross. An overview of the parallel virtual file system. In Proceedings of the 1999 Extreme Linux Workshop, 1999."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2019.2927481"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3404397.3404440"},{"key":"e_1_2_1_30_1","volume-title":"A block layer introduction part 1: the bio layer","author":"Neil Brown","year":"2017","unstructured":"Brown Neil . A block layer introduction part 1: the bio layer , 2017 . URL https:\/\/lwn.net\/Articles\/736534. Brown Neil. A block layer introduction part 1: the bio layer, 2017. URL https:\/\/lwn.net\/Articles\/736534."},{"key":"e_1_2_1_31_1","volume-title":"Flashmon v2: Monitoring raw nand flash memory i\/o requests on embedded linux. Acm Sigbed Review, 11(1):38--43","author":"Olivier Pierre","year":"2014","unstructured":"Pierre Olivier , Jalil Boukhobza , and Eric Senn . Flashmon v2: Monitoring raw nand flash memory i\/o requests on embedded linux. Acm Sigbed Review, 11(1):38--43 , 2014 . 97 EZIOTracer: Unifying Kernel and User Space I\/O Tracing for Data-Intensive Applications CHEOPS '21, April 26, 2021, Online, United Kingdom Pierre Olivier, Jalil Boukhobza, and Eric Senn. Flashmon v2: Monitoring raw nand flash memory i\/o requests on embedded linux. Acm Sigbed Review, 11(1):38--43, 2014. 97 EZIOTracer: Unifying Kernel and User Space I\/O Tracing for Data-Intensive Applications CHEOPS '21, April 26, 2021, Online, United Kingdom"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.4108\/icst.valuetools.2014.258179"},{"key":"e_1_2_1_33_1","volume-title":"3rd IEEE International Workshop on Real- Time and Distributed Computing in Emerging Applications (REACTION)","author":"Ouarnoughi Hamza","year":"2014","unstructured":"Hamza Ouarnoughi , Jalil Boukhobza , Frank Singhoff , and St\u00e9phane Rubini . A multi-level i\/o tracer for timing and performance storage systems in iaas cloud . In 3rd IEEE International Workshop on Real- Time and Distributed Computing in Emerging Applications (REACTION) , 2014 . Hamza Ouarnoughi, Jalil Boukhobza, Frank Singhoff, and St\u00e9phane Rubini. A multi-level i\/o tracer for timing and performance storage systems in iaas cloud. In 3rd IEEE International Workshop on Real- Time and Distributed Computing in Emerging Applications (REACTION), 2014."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/NOMS.2014.6838328"},{"key":"e_1_2_1_35_1","unstructured":"Steven Rostedt. Ftrace documentation. URL https:\/\/www.kernel.org\/ doc\/Documentation\/trace\/race.txt.  Steven Rostedt. Ftrace documentation. URL https:\/\/www.kernel.org\/ doc\/Documentation\/trace\/race.txt."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-02465-9_7"},{"key":"e_1_2_1_37_1","first-page":"501","volume-title":"European Conference on Parallel Processing","author":"Sch\u00f6ne Robert","year":"2010","unstructured":"Robert Sch\u00f6ne , Ronny Tsch\u00fcter , Thomas Ilsche , and Daniel Hackenberg . The vampirtrace plugin counter interface: introduction and examples . In European Conference on Parallel Processing , pages 501 -- 511 . Springer , 2010 . Robert Sch\u00f6ne, Ronny Tsch\u00fcter, Thomas Ilsche, and Daniel Hackenberg. The vampirtrace plugin counter interface: introduction and examples. In European Conference on Parallel Processing, pages 501--511. Springer, 2010."},{"key":"e_1_2_1_38_1","first-page":"559","volume-title":"ICT and Critical Infrastructure: Proceedings of the 48th Annual Convention of Computer Society of India- Vol II","author":"Shinde Rahul","year":"2014","unstructured":"Rahul Shinde , Vinay Patil , Akshay Bhargava , Atul Phatak , and Amar More . Inline block level data de-duplication technique for ext4 file system. In Suresh Chandra Satapathy, P. S. Avadhani, Siba K. Udgata, and Sadasivuni Lakshminarayana, editors , ICT and Critical Infrastructure: Proceedings of the 48th Annual Convention of Computer Society of India- Vol II , pages 559 -- 566 , Cham , 2014 . Springer International Publishing. Rahul Shinde, Vinay Patil, Akshay Bhargava, Atul Phatak, and Amar More. Inline block level data de-duplication technique for ext4 file system. In Suresh Chandra Satapathy, P. S. Avadhani, Siba K. Udgata, and Sadasivuni Lakshminarayana, editors, ICT and Critical Infrastructure: Proceedings of the 48th Annual Convention of Computer Society of India- Vol II, pages 559--566, Cham, 2014. Springer International Publishing."},{"key":"e_1_2_1_39_1","volume-title":"IEEE\/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), 05","author":"Trahay Fran\u00e7ois","year":"2011","unstructured":"Fran\u00e7ois Trahay , Fran\u00e7ois Rue , Mathieu Faverge , Yutaka Ishikawa , Raymond Namyst , and Jack Dongarra . EZTrace : a generic framework for performance analysis . In IEEE\/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), 05 2011 . Fran\u00e7ois Trahay, Fran\u00e7ois Rue, Mathieu Faverge, Yutaka Ishikawa, Raymond Namyst, and Jack Dongarra. EZTrace: a generic framework for performance analysis. In IEEE\/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), 05 2011."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/1713072.1713080"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.5555\/3312322"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/2757667.2757684"},{"key":"e_1_2_1_43_1","first-page":"379","volume-title":"16th USENIX Symposium on Networked Systems Design and Implementation (NSDI 19)","author":"Yang Bin","year":"2019","unstructured":"Bin Yang , Xu Ji , Xiaosong Ma , Xiyang Wang , Tianyu Zhang , Xiupeng Zhu , Nosayba El-Sayed , Haidong Lan , Yibo Yang , Jidong Zhai , monitoring on a leading supercomputer . In 16th USENIX Symposium on Networked Systems Design and Implementation (NSDI 19) , pages 379 -- 394 , 2019 . Bin Yang, Xu Ji, Xiaosong Ma, Xiyang Wang, Tianyu Zhang, Xiupeng Zhu, Nosayba El-Sayed, Haidong Lan, Yibo Yang, Jidong Zhai, et al. End-to-end i\/o monitoring on a leading supercomputer. In 16th USENIX Symposium on Networked Systems Design and Implementation (NSDI 19), pages 379--394, 2019."},{"key":"e_1_2_1_44_1","first-page":"527","volume-title":"13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18)","author":"Zhou Fang","year":"2018","unstructured":"Fang Zhou , Yifan Gan , Sixiang Ma , and Yang Wang . wperf : generic off-cpu analysis to identify bottleneck waiting events . In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18) , pages 527 -- 543 , 2018 . Fang Zhou, Yifan Gan, Sixiang Ma, and Yang Wang. wperf: generic off-cpu analysis to identify bottleneck waiting events. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), pages 527--543, 2018."}],"container-title":["ACM SIGOPS Operating Systems Review"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3469379.3469391","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3469379.3469391","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T21:28:23Z","timestamp":1750195703000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3469379.3469391"}},"subtitle":["Unifying Kernel and User Space I\/O Tracing for Data-Intensive Applications"],"short-title":[],"issued":{"date-parts":[[2021,6,2]]},"references-count":44,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2021,6,2]]}},"alternative-id":["10.1145\/3469379.3469391"],"URL":"https:\/\/doi.org\/10.1145\/3469379.3469391","relation":{},"ISSN":["0163-5980"],"issn-type":[{"value":"0163-5980","type":"print"}],"subject":[],"published":{"date-parts":[[2021,6,2]]},"assertion":[{"value":"2021-06-06","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}