{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T01:19:59Z","timestamp":1760059199723,"version":"build-2065373602"},"reference-count":47,"publisher":"Association for Computing Machinery (ACM)","issue":"OOPSLA2","funder":[{"DOI":"10.13039\/100017052","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["No. 62272217"],"award-info":[{"award-number":["No. 62272217"]}],"id":[{"id":"10.13039\/100017052","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Program. Lang."],"published-print":{"date-parts":[[2025,10,9]]},"abstract":"<jats:p>Despite extensive in-house testing, bugs often escape to deployed software. Whenever a failure occurs in production software, it is desirable to collect as much execution information as possible so as to help developers reproduce, diagnose and fix the bug. To reconcile the tension between trace capability, runtime overhead, and trace scale, we propose continuous tail tracing for production use. Instead of capturing only crash stacks, we produce the complete sequence of function calls and returns. Importantly, to avoid the overwhelming stress to I\/O, storage, and network transfer caused by the tremendous amount of trace data, we only retain the final segment of trace. To accomplish it, we design a novel trace decoder to support precise tail trace decoding, and an effective path-based instrumentation-site selection algorithm to reduce overhead. We implemented our approach as a tool called TailTracer on top of LLVM, and conducted the evaluations over the SPEC CPU 2017 benchmark suite, the open-source database system, and real-world bugs. The experimental results validate that TailTracer achieves low-overhead tail tracing, while providing more informative trace data than the baseline.<\/jats:p>","DOI":"10.1145\/3763109","type":"journal-article","created":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T08:49:50Z","timestamp":1759999790000},"page":"1647-1670","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["TailTracer: Continuous Tail Tracing for Production Use"],"prefix":"10.1145","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9781-4435","authenticated-orcid":false,"given":"Tianyi","family":"Liu","sequence":"first","affiliation":[{"name":"Nanjing University, Nanjing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-1162-7263","authenticated-orcid":false,"given":"Yi","family":"Li","sequence":"additional","affiliation":[{"name":"Nanjing University, Nanjing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-7412-5293","authenticated-orcid":false,"given":"Yiyu","family":"Zhang","sequence":"additional","affiliation":[{"name":"Nanjing University, Nanjing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-7628-2208","authenticated-orcid":false,"given":"Zhuangda","family":"Wang","sequence":"additional","affiliation":[{"name":"Xiamen University, Xiamen, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4648-3795","authenticated-orcid":false,"given":"Rongxin","family":"Wu","sequence":"additional","affiliation":[{"name":"Xiamen University, Xiamen, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3090-9568","authenticated-orcid":false,"given":"Xuandong","family":"Li","sequence":"additional","affiliation":[{"name":"Nanjing University, Nanjing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7104-9918","authenticated-orcid":false,"given":"Zhiqiang","family":"Zuo","sequence":"additional","affiliation":[{"name":"Nanjing University, Nanjing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2025,10,9]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"2019. Arm Embedded Trace Macrocell Architecture Specification ETMv4.0 to ETMv4.5."},{"key":"e_1_2_1_2_1","volume-title":"Intel Processor Trace.","unstructured":"2019. Intel 64 and IA-32 Architectures Software Developer\u2019s Manual Volume 3(3A, 3B, 3C, & 3D): System Programming Guide. Chapter 35: Intel Processor Trace."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2009.25"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-70592-5_23"},{"key":"e_1_2_1_5_1","unstructured":"OpenTelemetry Authors. 2024. OpenTelemetry eBPF Profiler. https:\/\/github.com\/open-telemetry\/opentelemetry-ebpf-profiler Accessed: 2025-07-20"},{"key":"e_1_2_1_6_1","unstructured":"SQLancer authors. [n. d.]. SQLancer: Automated testing to find logic and performance bugs in database systems. https:\/\/www.sqlancer.com\/"},{"key":"e_1_2_1_7_1","unstructured":"The Perfetto Authors. 2025. Perfetto Documentation. https:\/\/perfetto.dev\/docs\/ Accessed: 2025-07-20"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/183432.183527"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2013.6606582"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/TC.2005.186"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2854038.2854044"},{"key":"e_1_2_1_12_1","unstructured":"Li Chen Ellis Hoag Kyungwoo Lee Julian Mestre and Sergey Pupyrev. 2022. Minimum Coverage Instrumentation. arXiv preprint arXiv:2208.13907."},{"key":"e_1_2_1_13_1","volume-title":"HART: Hardware-Assisted Kernel Module Tracing on Arm. In Computer Security \u2013 ESORICS","author":"Du Yunlan","year":"2020","unstructured":"Yunlan Du, Zhenyu Ning, Jun Xu, Zhilong Wang, Yueh-Hsun Lin, Fengwei Zhang, Xinyu Xing, and Bing Mao. 2020. HART: Hardware-Assisted Kernel Module Tracing on Arm. In Computer Security \u2013 ESORICS 2020, Liqun Chen, Ninghui Li, Kaitai Liang, and Steve Schneider (Eds.). Springer International Publishing, Cham. 316\u2013337. isbn:978-3-030-58951-6"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3640537.3641569"},{"key":"e_1_2_1_15_1","unstructured":"GCC Team. 2025. GCov: a Test Coverage Program. https:\/\/gcc.gnu.org\/onlinedocs\/gcc\/Gcov.html Accessed: 2025-03-23"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/1629575.1629586"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2015.2454513"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.5555\/2337223.2337279"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/PCCC.1999.749423"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/PCCC.1994.504117"},{"key":"e_1_2_1_21_1","volume-title":"Jones and Mary Jean Harrold","author":"James","year":"2005","unstructured":"James A. Jones and Mary Jean Harrold. 2005. Empirical evaluation of the tarantula automatic fault-localization technique. In ASE. 273\u2013282."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF01951942"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/301618.301678"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.5555\/2337223.2337225"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/781131.781148"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/781131.781148"},{"key":"e_1_2_1_27_1","volume-title":"Jordan","author":"Liblit Ben","year":"2005","unstructured":"Ben Liblit, Mayur Naik, Alice X. Zheng, Alex Aiken, and Michael I. Jordan. 2005. Scalable statistical bug isolation. In PLDI. 15\u201326."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.6084\/m9.figshare.29968294.v1"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/HPCA.2017.18"},{"key":"e_1_2_1_30_1","unstructured":"LLVM Team. 2025. SanitizerCoverage Documentation. https:\/\/clang.llvm.org\/docs\/SanitizerCoverage.html Accessed: 2025-03-23"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPS.2008.4536310"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/1189756.1189758"},{"key":"e_1_2_1_33_1","unstructured":"Mozilla. [n. d.]. Mozila Crash Reports. https:\/\/crash-stats.mozilla.org\/"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/CGO.2019.8661201"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/75108.75382"},{"key":"e_1_2_1_36_1","unstructured":"Polar Signals. 2024. Polar Signals Documentation: Overview. https:\/\/www.polarsignals.com\/docs\/overview Accessed: 2025-07-20"},{"key":"e_1_2_1_37_1","unstructured":"Jane Street. 2025. Magic-trace Wiki. https:\/\/github.com\/janestreet\/magic-trace\/wiki Accessed: 2025-07-20"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2016.2521368"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2837614.2837619"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/2610384.2610386"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIFS.2024.3372816"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/587051.587053"},{"key":"e_1_2_1_43_1","volume-title":"The Benefit of Hindsight: Tracing Edge-Cases in Distributed Systems. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23)","author":"Zhang Lei","year":"2023","unstructured":"Lei Zhang, Zhiqiang Xie, Vaastav Anand, Ymir Vigfusson, and Jonathan Mace. 2023. The Benefit of Hindsight: Tracing Edge-Cases in Distributed Systems. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). USENIX Association, Boston, MA. 321\u2013339. isbn:978-1-939133-33-5 https:\/\/www.usenix.org\/conference\/nsdi23\/presentat ion\/zhang-lei"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338961"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2019.2892102"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3453483.3454096"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3544790"}],"container-title":["Proceedings of the ACM on Programming Languages"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3763109","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T17:43:09Z","timestamp":1760031789000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3763109"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,9]]},"references-count":47,"journal-issue":{"issue":"OOPSLA2","published-print":{"date-parts":[[2025,10,9]]}},"alternative-id":["10.1145\/3763109"],"URL":"https:\/\/doi.org\/10.1145\/3763109","relation":{},"ISSN":["2475-1421"],"issn-type":[{"value":"2475-1421","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,10,9]]},"assertion":[{"value":"2025-03-25","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-08-12","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-10-09","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}