{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,9]],"date-time":"2026-01-09T14:33:39Z","timestamp":1767969219211,"version":"3.49.0"},"reference-count":24,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2020,2,24]],"date-time":"2020-02-24T00:00:00Z","timestamp":1582502400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000001","name":"NSF","doi-asserted-by":"publisher","award":["1452712"],"award-info":[{"award-number":["1452712"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Commun. ACM"],"published-print":{"date-parts":[[2020,2,24]]},"abstract":"<jats:p>Monitoring and troubleshooting distributed systems are notoriously difficult; potential problems are complex, varied, and unpredictable. The monitoring and diagnosis tools commonly used today---logs, counters, and metrics---have two important limitations: what gets recorded is defined a priori, and the information is recorded in a component- or machine-centric way, making it extremely hard to correlate events that cross these boundaries. This paper presents Pivot Tracing, a monitoring framework for distributed systems that addresses both limitations by combining dynamic instrumentation with a novel relational operator: the happened-before join. Pivot Tracing gives users, at runtime, the ability to define arbitrary metrics at one point of the system, while being able to select, filter, and group by events meaningful at other parts of the system, even when crossing component or machine boundaries. Pivot Tracing does not correlate cross-component events using expensive global aggregations, nor does it perform offline analysis. Instead, Pivot Tracing directly correlates events as they happen by piggybacking metadata alongside requests as they execute. This gives Pivot Tracing low runtime overhead---less than 1% for many cross-component monitoring queries.<\/jats:p>","DOI":"10.1145\/3378933","type":"journal-article","created":{"date-parts":[[2020,2,24]],"date-time":"2020-02-24T18:34:54Z","timestamp":1582569294000},"page":"94-102","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["Pivot tracing"],"prefix":"10.1145","volume":"63","author":[{"given":"Jonathan","family":"Mace","sequence":"first","affiliation":[{"name":"Brown University, Providence, RI"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ryan","family":"Roelke","sequence":"additional","affiliation":[{"name":"Brown University, Providence, RI"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Rodrigo","family":"Fonseca","sequence":"additional","affiliation":[{"name":"Brown University, Providence, RI"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2020,2,24]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"http:\/\/htrace.incubator.apache.org\/. [Online","author":"Apache","year":"2015","unstructured":"Apache HTrace. http:\/\/htrace.incubator.apache.org\/. [Online ; accessed March 2015 ]. ( Section 2.3). Apache HTrace. http:\/\/htrace.incubator.apache.org\/. [Online; accessed March 2015]. (Section 2.3)."},{"key":"e_1_2_1_2_1","volume-title":"P. Overview of the workshop of managing large-scale systems via the analysis of system logs and the application of machine learning techniques (SLAML'11)","author":"Bodik","year":"2011","unstructured":"Bodik , P. Overview of the workshop of managing large-scale systems via the analysis of system logs and the application of machine learning techniques (SLAML'11) . SIGOPS Oper. Syst. Rev. 45, 3 ( 2011 ), 20--22. (Section 2.3). Bodik, P. Overview of the workshop of managing large-scale systems via the analysis of system logs and the application of machine learning techniques (SLAML'11). SIGOPS Oper. Syst. Rev. 45, 3 (2011), 20--22. (Section 2.3)."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/1117389.1117401"},{"key":"e_1_2_1_4_1","first-page":"15","volume-title":"USENIX Annual Technical Conference, General Track","author":"Cantrill B.","year":"2004","unstructured":"Cantrill , B. , Shapiro , M.W. , Leventhal , A.H. Dynamic instrumentation of production systems . In USENIX Annual Technical Conference, General Track ( 2004 ), pp. 15 -- 28 . (Sections 1, 2.3, and 4). Cantrill, B., Shapiro, M.W., Leventhal, A.H. Dynamic instrumentation of production systems. In USENIX Annual Technical Conference, General Track (2004), pp. 15--28. (Sections 1, 2.3, and 4)."},{"key":"e_1_2_1_5_1","first-page":"1","article-title":"Java bytecode engineering made simple","volume":"9","author":"Chiba S.","year":"2004","unstructured":"Chiba , S. Javassist : Java bytecode engineering made simple . Java Developer's Journal 9 , 1 ( 2004 ). (Section 4). Chiba, S. Javassist: Java bytecode engineering made simple. Java Developer's Journal 9, 1 (2004). (Section 4).","journal-title":"Java Developer's Journal"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/1807128.1807152"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2382553.2382555"},{"key":"e_1_2_1_8_1","volume-title":"Proceedings of the 4th USENIX Conference on Networked Systems Design & Implementation","author":"Fonseca R.","year":"2007","unstructured":"Fonseca , R. , Porter , G. , Katz , R.H. , Shenker , S. , Stoica , I. X-trace : A pervasive network tracing framework . In Proceedings of the 4th USENIX Conference on Networked Systems Design & Implementation ( Berkeley, CA, USA , 2007 ), NSDI'07, USENIX Association. (Sections 1 and 3). Fonseca, R., Porter, G., Katz, R.H., Shenker, S., Stoica, I. X-trace: A pervasive network tracing framework. In Proceedings of the 4th USENIX Conference on Networked Systems Design & Implementation (Berkeley, CA, USA, 2007), NSDI'07, USENIX Association. (Sections 1 and 3)."},{"key":"e_1_2_1_9_1","volume-title":"https:\/\/issues.apache.org\/jira\/browse\/HBASE-4145. [Online","author":"Provide","year":"2015","unstructured":"HBASE-4145 Provide metrics for HBASE client. https:\/\/issues.apache.org\/jira\/browse\/HBASE-4145. [Online ; accessed 25 February 2015 ]. ( Section 2.3). HBASE-4145 Provide metrics for HBASE client. https:\/\/issues.apache.org\/jira\/browse\/HBASE-4145. [Online; accessed 25 February 2015]. (Section 2.3)."},{"key":"e_1_2_1_10_1","volume-title":"https:\/\/issues.apache.org\/jira\/browse\/HBASE-8370. [Online","author":"Report","year":"2015","unstructured":"HBASE-8370 Report data block cache hit rates apart from aggregate cache hit rates. https:\/\/issues.apache.org\/jira\/browse\/HBASE-8370. [Online ; accessed 25 February 2015 ]. ( Section 2.3). HBASE-8370 Report data block cache hit rates apart from aggregate cache hit rates. https:\/\/issues.apache.org\/jira\/browse\/HBASE-8370. [Online; accessed 25 February 2015]. (Section 2.3)."},{"key":"e_1_2_1_11_1","volume-title":"pseudoSortByDistance when no local node is found. https:\/\/issues.apache.org\/jira\/browse\/HDFS-6268. [Online","author":"Better","year":"2015","unstructured":"HDFS-6268 Better sorting in Network Topology. pseudoSortByDistance when no local node is found. https:\/\/issues.apache.org\/jira\/browse\/HDFS-6268. [Online ; accessed 25 February 2015 ]. ( Sections 1 and 3). HDFS-6268 Better sorting in Network Topology. pseudoSortByDistance when no local node is found. https:\/\/issues.apache.org\/jira\/browse\/HDFS-6268. [Online; accessed 25 February 2015]. (Sections 1 and 3)."},{"key":"e_1_2_1_12_1","first-page":"41","volume-title":"New Frontiers in Information and Software as Services","author":"Huang S.","year":"2010","unstructured":"Huang , S. , Huang , J. , Dai , J. , Xie , T. , Huang , B. The hibench benchmark suite: Characterization of the mapreduce-based data analysis . In New Frontiers in Information and Software as Services ( 2010 ). IEEE, pp. 41 -- 51 . (Section 5). Huang, S., Huang, J., Dai, J., Xie, T., Huang, B. The hibench benchmark suite: Characterization of the mapreduce-based data analysis. In New Frontiers in Information and Software as Services (2010). IEEE, pp. 41--51. (Section 5)."},{"key":"e_1_2_1_13_1","first-page":"327","volume-title":"An Overview of AspectJ. In European Conference on Object-Oriented Programming","author":"Kiczales G.","year":"2001","unstructured":"Kiczales , G. , Hilsdale , E. , Hugunin , J. , Kersten , M. , Palm , J. , Griswold , W.G. An Overview of AspectJ. In European Conference on Object-Oriented Programming ( London, UK , 2001 ). Springer-Verlag, pp. 327 -- 353 . (Section 4). Kiczales, G., Hilsdale, E., Hugunin, J., Kersten, M., Palm, J., Griswold, W.G. An Overview of AspectJ. In European Conference on Object-Oriented Programming (London, UK, 2001). Springer-Verlag, pp. 327--353. (Section 4)."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1007\/BFb0053381"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/359545.359563"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2815400.2815415"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/1142473.1142552"},{"key":"e_1_2_1_18_1","volume-title":"slave, executor, etc. should be collected on a per-task basis. https:\/\/issues.apache.org\/jira\/browse\/MESOS-1949. [Online","author":"All","year":"2015","unstructured":"MESOS-1949 All log messages from master , slave, executor, etc. should be collected on a per-task basis. https:\/\/issues.apache.org\/jira\/browse\/MESOS-1949. [Online ; accessed 25 February 2015 ]. ( Section 2.3). MESOS-1949 All log messages from master, slave, executor, etc. should be collected on a per-task basis. https:\/\/issues.apache.org\/jira\/browse\/MESOS-1949. [Online; accessed 25 February 2015]. (Section 2.3)."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2076450.2076466"},{"key":"e_1_2_1_20_1","volume-title":"2005 Ottawa Linux Symposium","author":"Prasad V.","year":"2005","unstructured":"Prasad , V. , Cohen , W. , Eigler , F.C. , Hunt , M. , Keniston , J. , Chen , B. Locating system problems using dynamic instrumentation . In 2005 Ottawa Linux Symposium ( 2005 ). (Section 2.3). Prasad, V., Cohen, W., Eigler, F.C., Hunt, M., Keniston, J., Chen, B. Locating system problems using dynamic instrumentation. In 2005 Ottawa Linux Symposium (2005). (Section 2.3)."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2012.73"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/MSST.2010.5496972"},{"key":"e_1_2_1_24_1","volume-title":"http:\/\/twitter.github.io\/zipkin\/. [Online","author":"Twitter Zipkin","year":"2015","unstructured":"Twitter Zipkin . http:\/\/twitter.github.io\/zipkin\/. [Online ; accessed March 2015 ]. ( Section 2.3). Twitter Zipkin. http:\/\/twitter.github.io\/zipkin\/. [Online; accessed March 2015]. (Section 2.3)."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2110356.2110360"}],"container-title":["Communications of the ACM"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3378933","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3378933","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3378933","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:41:20Z","timestamp":1750200080000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3378933"}},"subtitle":["dynamic causal monitoring for distributed systems"],"short-title":[],"issued":{"date-parts":[[2020,2,24]]},"references-count":24,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2020,2,24]]}},"alternative-id":["10.1145\/3378933"],"URL":"https:\/\/doi.org\/10.1145\/3378933","relation":{},"ISSN":["0001-0782","1557-7317"],"issn-type":[{"value":"0001-0782","type":"print"},{"value":"1557-7317","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,2,24]]},"assertion":[{"value":"2020-02-24","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}