{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,16]],"date-time":"2026-07-16T10:10:51Z","timestamp":1784196651598,"version":"3.55.0"},"reference-count":47,"publisher":"Association for Computing Machinery (ACM)","issue":"OOPSLA2","funder":[{"DOI":"10.13039\/100017052","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["No. 62272217"],"award-info":[{"award-number":["No. 62272217"]}],"id":[{"id":"10.13039\/100017052","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Program. Lang."],"published-print":{"date-parts":[[2025,10,9]]},"abstract":"<jats:p>Despite extensive in-house testing, bugs often escape to deployed software. Whenever a failure occurs in production software, it is desirable to collect as much execution information as possible so as to help developers reproduce, diagnose and fix the bug. To reconcile the tension between trace capability, runtime overhead, and trace scale, we propose continuous tail tracing for production use. Instead of capturing only crash stacks, we produce the complete sequence of function calls and returns. Importantly, to avoid the overwhelming stress to I\/O, storage, and network transfer caused by the tremendous amount of trace data, we only retain the final segment of trace. To accomplish it, we design a novel trace decoder to support precise tail trace decoding, and an effective path-based instrumentation-site selection algorithm to reduce overhead. We implemented our approach as a tool called TailTracer on top of LLVM, and conducted the evaluations over the SPEC CPU 2017 benchmark suite, the open-source database system, and real-world bugs. The experimental results validate that TailTracer achieves low-overhead tail tracing, while providing more informative trace data than the baseline.<\/jats:p>","DOI":"10.1145\/3763109","type":"journal-article","created":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T08:49:50Z","timestamp":1759999790000},"page":"1647-1670","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["TailTracer: Continuous Tail Tracing for Production Use"],"prefix":"10.1145","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9781-4435","authenticated-orcid":false,"given":"Tianyi","family":"Liu","sequence":"first","affiliation":[{"name":"Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-1162-7263","authenticated-orcid":false,"given":"Yi","family":"Li","sequence":"additional","affiliation":[{"name":"Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-7412-5293","authenticated-orcid":false,"given":"Yiyu","family":"Zhang","sequence":"additional","affiliation":[{"name":"Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-7628-2208","authenticated-orcid":false,"given":"Zhuangda","family":"Wang","sequence":"additional","affiliation":[{"name":"Xiamen University, Xiamen, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4648-3795","authenticated-orcid":false,"given":"Rongxin","family":"Wu","sequence":"additional","affiliation":[{"name":"Xiamen University, Xiamen, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3090-9568","authenticated-orcid":false,"given":"Xuandong","family":"Li","sequence":"additional","affiliation":[{"name":"Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7104-9918","authenticated-orcid":false,"given":"Zhiqiang","family":"Zuo","sequence":"additional","affiliation":[{"name":"Nanjing University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,10,9]]},"reference":[{"key":"e_1_3_1_2_2","unstructured":"2019. Arm Embedded Trace Macrocell Architecture Specification ETMv4.0 to ETMv4.5."},{"key":"e_1_3_1_3_2","unstructured":"2019. Intel 64 and IA-32 Architectures Software Developer\u2032s Manual Volume 3(3A 3B 3C & 3D): System Programming Guide. Chapter 35: Intel Processor Trace."},{"key":"e_1_3_1_4_2","doi-asserted-by":"publisher","unstructured":"Rui Abreu Peter Zoeteweij and Arjan J.C. van Gemund. 2009. Spectrum-Based Multiple Fault Localization. In 2009 IEEE\/ACM International Conference on Automated Software Engineering. 88\u201399. doi:10.1109\/ASE.2009.25","DOI":"10.1109\/ASE.2009.25"},{"key":"e_1_3_1_5_2","doi-asserted-by":"crossref","unstructured":"Shay Artzi Sunghun Kim and Michael D Ernst. 2008. Recrash: Making software failures reproducible by preserving object states. In ECOOP 2008-Object-Oriented Programming: 22nd European Conference Paphos Cyprus July 7-11 2008 Proceedings 22. Springer 542\u2013565.","DOI":"10.1007\/978-3-540-70592-5_23"},{"key":"e_1_3_1_6_2","unstructured":"OpenTelemetry Authors. 2024. OpenTelemetry eBPF Profiler. https:\/\/github.com\/open-telemetry\/opentelemetry-ebpfprofiler. Accessed: 2025-07-20."},{"key":"e_1_3_1_7_2","unstructured":"SQLancer authors. [n. d.]. SQLancer: Automated testing to find logic and performance bugs in database systems. https:\/\/www.sqlancer.com\/"},{"key":"e_1_3_1_8_2","unstructured":"The Perfetto Authors. 2025. Perfetto Documentation. https:\/\/perfetto.dev\/docs\/. Accessed: 2025-07-20."},{"key":"e_1_3_1_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/183432.183527"},{"key":"e_1_3_1_10_2","doi-asserted-by":"crossref","unstructured":"Jonathan Bell Nikhil Sarda and Gail Kaiser. 2013. Chronicler: Lightweight recording to reproduce field failures. In 2013 35th International Conference on Software Engineering (ICSE). IEEE 362\u2013371.","DOI":"10.1109\/ICSE.2013.6606582"},{"key":"e_1_3_1_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/TC.2005.186"},{"key":"e_1_3_1_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/2854038.2854044"},{"key":"e_1_3_1_13_2","unstructured":"Li Chen Ellis Hoag Kyungwoo Lee Julian Mestre and Sergey Pupyrev. 2022. Minimum Coverage Instrumentation. arXiv preprint arXiv:2208.13907 (2022)."},{"key":"e_1_3_1_14_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58951-6_16"},{"key":"e_1_3_1_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3640537.3641569"},{"key":"e_1_3_1_16_2","unstructured":"GCC Team. 2025. GCov: a Test Coverage Program. https:\/\/gcc.gnu.org\/onlinedocs\/gcc\/Gcov.html Accessed: 2025-03-23."},{"key":"e_1_3_1_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/1629575.1629586"},{"key":"e_1_3_1_18_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2015.2454513"},{"key":"e_1_3_1_19_2","doi-asserted-by":"crossref","unstructured":"Wei Jin and Alessandro Orso. 2012. BugRedux: reproducing field failures for in-house debugging. In Proceedings of the 34th International Conference on Software Engineering (Zurich Switzerland) (ICSE \u203212). IEEE Press 474\u2013484.","DOI":"10.1109\/ICSE.2012.6227168"},{"key":"e_1_3_1_20_2","doi-asserted-by":"publisher","unstructured":"E.E. Johnson. 1999. PDATS II: improved compression of address traces. In 1999 IEEE International Performance Computing and Communications Conference (Cat. No.99CH36305). 72\u201378. doi:10.1109\/PCCC.1999.749423","DOI":"10.1109\/PCCC.1999.749423"},{"key":"e_1_3_1_21_2","doi-asserted-by":"publisher","unstructured":"E.E. Johnson and Jiheng Ha. 1994. PDATS Lossless Address Trace Compression For Reducing File Size And Access Time. In Proceeding of 13th IEEE Annual International Phoenix Conference on Computers and Communications. 213-. doi:10.1109\/PCCC.1994.504117","DOI":"10.1109\/PCCC.1994.504117"},{"key":"e_1_3_1_22_2","doi-asserted-by":"crossref","unstructured":"James A. Jones and Mary Jean Harrold. 2005. Empirical evaluation of the tarantula automatic fault-localization technique. In ASE. 273\u2013282.","DOI":"10.1145\/1101908.1101949"},{"key":"e_1_3_1_23_2","doi-asserted-by":"publisher","DOI":"10.1007\/BF01951942"},{"key":"e_1_3_1_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/301618.301678"},{"key":"e_1_3_1_25_2","doi-asserted-by":"publisher","DOI":"10.5555\/2337223.2337225"},{"key":"e_1_3_1_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/781131.781148"},{"key":"e_1_3_1_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/781131.781148"},{"key":"e_1_3_1_28_2","doi-asserted-by":"crossref","unstructured":"Ben Liblit Mayur Naik Alice X. Zheng Alex Aiken and Michael I. Jordan. 2005. Scalable statistical bug isolation. In PLDI. 15\u201326.","DOI":"10.1145\/1065010.1065014"},{"key":"e_1_3_1_29_2","doi-asserted-by":"publisher","unstructured":"Tianyi Liu Yi Li Zhuangda Wang Yiyu Zhang Rongxin Wu Xuandong Li and Zhiqiang Zuo. 2025. TailTracer: Continuous Tail Tracing for Production Use (Artifact). (8 2025). doi:10.6084\/m9.figshare.29968294.v1","DOI":"10.6084\/m9.figshare.29968294.v1"},{"key":"e_1_3_1_30_2","doi-asserted-by":"publisher","unstructured":"Yutao Liu Peitao Shi Xinran Wang Haibo Chen Binyu Zang and Haibing Guan. 2017. Transparent and Efficient CFI Enforcement with Intel Processor Trace. In 2017 IEEE International Symposium on High Performance Computer Architecture (HPCA). 529\u2013540. doi:10.1109\/HPCA.2017.18","DOI":"10.1109\/HPCA.2017.18"},{"key":"e_1_3_1_31_2","unstructured":"LLVM Team. 2025. SanitizerCoverage Documentation. https:\/\/clang.llvm.org\/docs\/SanitizerCoverage.html Accessed: 2025-03-23."},{"key":"e_1_3_1_32_2","doi-asserted-by":"publisher","unstructured":"Naoya Maruyama and Satoshi Matsuoka. 2008. Model-based fault localization in large-scale computing systems. In 2008 IEEE International Symposium on Parallel and Distributed Processing. 1\u201312. doi:10.1109\/IPDPS.2008.4536310","DOI":"10.1109\/IPDPS.2008.4536310"},{"key":"e_1_3_1_33_2","doi-asserted-by":"publisher","unstructured":"Aleksandar Milenkovic and Milena Milenkovic. 2007. An efficient single-pass trace compression technique utilizing instruction streams. ACM Trans. Model. Comput. Simul. 17 1 (Jan. 2007) 2-es. doi:10.1145\/1189756.1189758","DOI":"10.1145\/1189756.1189758"},{"key":"e_1_3_1_34_2","unstructured":"Mozilla. [n. d.]. Mozila Crash Reports. https:\/\/crash-stats.mozilla.org\/"},{"key":"e_1_3_1_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/CGO.2019.8661201"},{"key":"e_1_3_1_36_2","doi-asserted-by":"crossref","unstructured":"A Dain Samples. 1989. Mache: No-loss trace compaction. In Proceedings of the 1989 ACM SIGMETRICS international conference on Measurement and modeling of computer systems. 89\u201397.","DOI":"10.1145\/75108.75382"},{"key":"e_1_3_1_37_2","unstructured":"Polar Signals. 2024. Polar Signals Documentation: Overview. https:\/\/www.polarsignals.com\/docs\/overview. Accessed: 2025-07-20."},{"key":"e_1_3_1_38_2","unstructured":"Jane Street. 2025. Magic-trace Wiki. https:\/\/github.com\/janestreet\/magic-trace\/wiki. Accessed: 2025-07-20."},{"key":"e_1_3_1_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2016.2521368"},{"key":"e_1_3_1_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/2837614.2837619"},{"key":"e_1_3_1_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/2610384.2610386"},{"key":"e_1_3_1_42_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIFS.2024.3372816"},{"key":"e_1_3_1_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/587051.587053"},{"key":"e_1_3_1_44_2","unstructured":"Lei Zhang Zhiqiang Xie Vaastav Anand Ymir Vigfusson and Jonathan Mace. 2023. The Benefit of Hindsight: Tracing Edge-Cases in Distributed Systems. In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). USENIX Association Boston MA 321\u2013339. https:\/\/www.usenix.org\/conference\/nsdi23\/presentation\/zhang-lei"},{"key":"e_1_3_1_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338961"},{"key":"e_1_3_1_46_2","doi-asserted-by":"publisher","unstructured":"D. Zou J. Liang Y. Xiong M. D. Ernst and L. Zhang. 2019. An Empirical Study of Fault Localization Families and Their Combinations. IEEE Transactions on Software Engineering (2019) 1\u20131. doi:10.1109\/TSE.2019.2892102","DOI":"10.1109\/TSE.2019.2892102"},{"key":"e_1_3_1_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/3453483.3454096"},{"key":"e_1_3_1_48_2","doi-asserted-by":"publisher","DOI":"10.1145\/3544790"}],"container-title":["Proceedings of the ACM on Programming Languages"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3763109","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,16]],"date-time":"2026-07-16T10:02:23Z","timestamp":1784196143000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3763109"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,9]]},"references-count":47,"journal-issue":{"issue":"OOPSLA2","published-print":{"date-parts":[[2025,10,9]]}},"alternative-id":["10.1145\/3763109"],"URL":"https:\/\/doi.org\/10.1145\/3763109","relation":{},"ISSN":["2475-1421"],"issn-type":[{"value":"2475-1421","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,10,9]]},"assertion":[{"value":"2025-03-25","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-08-12","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-10-09","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}