{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,9]],"date-time":"2026-04-09T04:29:22Z","timestamp":1775708962238,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":42,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,11,1]],"date-time":"2021-11-01T00:00:00Z","timestamp":1635724800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"NSF (National Science Foundation)","doi-asserted-by":"publisher","award":["2016178"],"award-info":[{"award-number":["2016178"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Red Hat, Inc. (Red Hat Collaboratory at Boston University)","award":["55209056"],"award-info":[{"award-number":["55209056"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,11]]},"DOI":"10.1145\/3472883.3487000","type":"proceedings-article","created":{"date-parts":[[2021,10,27]],"date-time":"2021-10-27T10:48:16Z","timestamp":1635331696000},"page":"61-75","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":9,"title":["Automating instrumentation choices for performance problems in distributed applications with VAIF"],"prefix":"10.1145","author":[{"given":"Mert","family":"Toslali","sequence":"first","affiliation":[{"name":"Boston University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Emre","family":"Ates","sequence":"additional","affiliation":[{"name":"Boston University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alex","family":"Ellis","sequence":"additional","affiliation":[{"name":"Tufts University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zhaoqi","family":"Zhang","sequence":"additional","affiliation":[{"name":"Tufts University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Darby","family":"Huye","sequence":"additional","affiliation":[{"name":"Tufts University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lan","family":"Liu","sequence":"additional","affiliation":[{"name":"Tufts University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Samantha","family":"Puterman","sequence":"additional","affiliation":[{"name":"Boston University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ayse K.","family":"Coskun","sequence":"additional","affiliation":[{"name":"Boston University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Raja R.","family":"Sambasivan","sequence":"additional","affiliation":[{"name":"Tufts University"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2021,11]]},"reference":[{"key":"e_1_3_2_2_1_1","unstructured":"2016. Instance stuck resuming from suspend state during load test. https:\/\/bugzilla.redhat.com\/showbug.cgi?id=1425516.  2016. Instance stuck resuming from suspend state during load test. https:\/\/bugzilla.redhat.com\/showbug.cgi?id=1425516."},{"key":"e_1_3_2_2_2_1","unstructured":"2016. nova libvirt driver instance stuck. https:\/\/ask.openstack.org\/en\/question\/91508\/nova-libvirt-driver-instance-stuck-on-spawning\/.  2016. nova libvirt driver instance stuck. https:\/\/ask.openstack.org\/en\/question\/91508\/nova-libvirt-driver-instance-stuck-on-spawning\/."},{"key":"e_1_3_2_2_3_1","unstructured":"2016. Nova list is extremely slow with lots of vms. https:\/\/bugs.launchpad.net\/nova\/+bug\/1160487.  2016. Nova list is extremely slow with lots of vms. https:\/\/bugs.launchpad.net\/nova\/+bug\/1160487."},{"key":"e_1_3_2_2_4_1","unstructured":"2016. OpenStack Identity service is responding slowly. https:\/\/docs.openstack.org\/operations-guide\/ops-maintenance-slow.html.  2016. OpenStack Identity service is responding slowly. https:\/\/docs.openstack.org\/operations-guide\/ops-maintenance-slow.html."},{"key":"e_1_3_2_2_5_1","unstructured":"2016. OVS plugin VIF plugging slow for VMs. https:\/\/ask.openstack.org\/en\/question\/128160\/ovs-plugin-vif-plugging-slow-for-vms-with-multiple-nics\/.  2016. OVS plugin VIF plugging slow for VMs. https:\/\/ask.openstack.org\/en\/question\/128160\/ovs-plugin-vif-plugging-slow-for-vms-with-multiple-nics\/."},{"key":"e_1_3_2_2_6_1","unstructured":"2016. Update on os-vif progress. https:\/\/openstack-dev.openstack.narkive.com\/gLpnyrJl\/nova-neutron-update-on-os-vif-progress-port-binding-negotiation.  2016. Update on os-vif progress. https:\/\/openstack-dev.openstack.narkive.com\/gLpnyrJl\/nova-neutron-update-on-os-vif-progress-port-binding-negotiation."},{"key":"e_1_3_2_2_7_1","unstructured":"V. Anand and J. Mace. 2021. X-Trace DeathStarBench Dataset. https:\/\/gitlab.mpi-sws.org\/cld\/trace-datasets\/deathstarbenchtraces.  V. Anand and J. Mace. 2021. X-Trace DeathStarBench Dataset. https:\/\/gitlab.mpi-sws.org\/cld\/trace-datasets\/deathstarbenchtraces."},{"key":"e_1_3_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/1806799.1806839"},{"key":"e_1_3_2_2_9_1","volume-title":"Proceedings of the ACM Symposium on Cloud Computing","author":"Ates Emre","unstructured":"Emre Ates , Lily Sturmann , Mert Toslali , Orran Krieger , Richard Megginson , Ayse K. Coskun , and Raja R. Sambasivan . 2019. An Automated, Cross-Layer Instrumentation Framework for Diagnosing Performance Problems in Distributed Applications . In Proceedings of the ACM Symposium on Cloud Computing ( Santa Cruz, CA, USA) (SoCC '19). Association for Computing Machinery, New York, NY, USA, 165--170. https:\/\/doi.org\/10.1145\/3357223.3362704 10.1145\/3357223.3362704 Emre Ates, Lily Sturmann, Mert Toslali, Orran Krieger, Richard Megginson, Ayse K. Coskun, and Raja R. Sambasivan. 2019. An Automated, Cross-Layer Instrumentation Framework for Diagnosing Performance Problems in Distributed Applications. In Proceedings of the ACM Symposium on Cloud Computing (Santa Cruz, CA, USA) (SoCC '19). Association for Computing Machinery, New York, NY, USA, 165--170. https:\/\/doi.org\/10.1145\/3357223.3362704"},{"key":"e_1_3_2_2_10_1","volume-title":"ATC '04: Proceedings of the 2004 USENIX Annual Technical Conference.","author":"Cantrill Bryan","unstructured":"Bryan Cantrill , Michael W. Shapiro , and Adam H. Leventhal . 2004. Dynamic instrumentation of production systems . In ATC '04: Proceedings of the 2004 USENIX Annual Technical Conference. Bryan Cantrill, Michael W. Shapiro, and Adam H. Leventhal. 2004. Dynamic instrumentation of production systems. In ATC '04: Proceedings of the 2004 USENIX Annual Technical Conference."},{"key":"e_1_3_2_2_11_1","volume-title":"OSDI' 14: Proceedings of the 11th USENIX conference on Operating Systems Design and Implementation.","author":"Chow Michael","year":"2014","unstructured":"Michael Chow , David Meisner , Jason Flinn , Daniel Peek , and Thomas F Wenisch . 2014 . The mystery machine: end-to-end performance analysis of large-scale internet services . In OSDI' 14: Proceedings of the 11th USENIX conference on Operating Systems Design and Implementation. Michael Chow, David Meisner, Jason Flinn, Daniel Peek, and Thomas F Wenisch. 2014. The mystery machine: end-to-end performance analysis of large-scale internet services. In OSDI' 14: Proceedings of the 11th USENIX conference on Operating Systems Design and Implementation."},{"key":"e_1_3_2_2_12_1","unstructured":"CockroachDB. 2019. https:\/\/www.cockroachlabs.com\/.  CockroachDB. 2019. https:\/\/www.cockroachlabs.com\/."},{"key":"e_1_3_2_2_13_1","volume-title":"ATC '15: Proceedings of the 2015 USENIX Annual Technical Conference.","author":"Ding Rui","year":"2015","unstructured":"Rui Ding , Hucheng Zhou , Jian-Guang Lou , Hongyu Zhang , Qingwei Lin , Qiang Fu , Dongmei Zhang , and Tao Xie . 2015 . Log2: A cost-aware logging mechanism for performance diagnosis . In ATC '15: Proceedings of the 2015 USENIX Annual Technical Conference. Rui Ding, Hucheng Zhou, Jian-Guang Lou, Hongyu Zhang, Qingwei Lin, Qiang Fu, Dongmei Zhang, and Tao Xie. 2015. Log2: A cost-aware logging mechanism for performance diagnosis. In ATC '15: Proceedings of the 2015 USENIX Annual Technical Conference."},{"key":"e_1_3_2_2_14_1","unstructured":"Kiciman Emre and Lakshminarayanan Subramanian. 2005. Root cause localization in large scale systems. In Root cause localization in large scale systems Vol. Proc. 1st Workshop on Hot Topics in Systems Dependability (HotDep).  Kiciman Emre and Lakshminarayanan Subramanian. 2005. Root cause localization in large scale systems. In Root cause localization in large scale systems Vol. Proc. 1st Workshop on Hot Topics in Systems Dependability (HotDep)."},{"key":"e_1_3_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2043556.2043585"},{"key":"e_1_3_2_2_16_1","volume-title":"INM\/WREN '10: Proceedings of the 1st Internet Network Management Workshop\/Workshop on Research on Enterprise Monitoring.","author":"Fonseca Rodrigo","year":"2010","unstructured":"Rodrigo Fonseca , Michael J. Freedman , and George Porter . 2010 . Experiences with tracing causality in networked services . In INM\/WREN '10: Proceedings of the 1st Internet Network Management Workshop\/Workshop on Research on Enterprise Monitoring. Rodrigo Fonseca, Michael J. Freedman, and George Porter. 2010. Experiences with tracing causality in networked services. In INM\/WREN '10: Proceedings of the 1st Internet Network Management Workshop\/Workshop on Research on Enterprise Monitoring."},{"key":"e_1_3_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3297858.3304013"},{"key":"e_1_3_2_2_18_1","unstructured":"Sudhanshu Goswami. 2005. https:\/\/lwn.net\/Articles\/132196\/.  Sudhanshu Goswami. 2005. https:\/\/lwn.net\/Articles\/132196\/."},{"key":"e_1_3_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132747.3132749"},{"key":"e_1_3_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/781131.781148"},{"key":"e_1_3_2_2_21_1","volume-title":"End-to-End Tracing: Adoption and Use Cases. Survey","author":"Mace Jonathan","year":"2017","unstructured":"Jonathan Mace . 2017. End-to-End Tracing: Adoption and Use Cases. Survey . Brown University . https:\/\/cs.brown.edu\/~jcmace\/papers\/mace 2017 survey.pdf. Jonathan Mace. 2017. End-to-End Tracing: Adoption and Use Cases. Survey. Brown University. https:\/\/cs.brown.edu\/~jcmace\/papers\/mace2017survey.pdf."},{"key":"e_1_3_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3190508.3190526"},{"key":"e_1_3_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/2815400.2815415"},{"key":"e_1_3_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.5555\/2170444.2170464"},{"key":"e_1_3_2_2_25_1","unstructured":"Mirantis OSProfiler [n.d.]. OSProfiler. https:\/\/docs.openstack.org\/osprofiler\/latest\/.  Mirantis OSProfiler [n.d.]. OSProfiler. https:\/\/docs.openstack.org\/osprofiler\/latest\/."},{"key":"e_1_3_2_2_26_1","unstructured":"Openstack [n.d.]. OpenStack web site. https:\/\/www.openstack.org.  Openstack [n.d.]. OpenStack web site. https:\/\/www.openstack.org."},{"key":"e_1_3_2_2_27_1","unstructured":"OpenTelemetry website [n.d.]. OpenTelemetry website. http:\/\/opentelemetry.io\/.  OpenTelemetry website [n.d.]. OpenTelemetry website. http:\/\/opentelemetry.io\/."},{"key":"e_1_3_2_2_28_1","volume-title":"Distributed tracing in practice: Instrumenting, analyzing, and debugging microservices","author":"Parker Austin","unstructured":"Austin Parker , Daniel Spoonhower , Jonathan Mace , Ben Sigelman , and Rebecca Isaacs . 2020. Distributed tracing in practice: Instrumenting, analyzing, and debugging microservices . O'Reilly Media . Austin Parker, Daniel Spoonhower, Jonathan Mace, Ben Sigelman, and Rebecca Isaacs. 2020. Distributed tracing in practice: Instrumenting, analyzing, and debugging microservices. O'Reilly Media."},{"key":"e_1_3_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2012.73"},{"key":"e_1_3_2_2_30_1","volume-title":"Proceedings of the 4th USENIX conference on Hot Topics in Cloud Computing. USENIX Association, 21--21","author":"Raja","unstructured":"Raja R. Sambasivan and Gregory R. Ganger. 2012. Automated diagnosis without predictability is a recipe for failure . In Proceedings of the 4th USENIX conference on Hot Topics in Cloud Computing. USENIX Association, 21--21 . Raja R. Sambasivan and Gregory R. Ganger. 2012. Automated diagnosis without predictability is a recipe for failure. In Proceedings of the 4th USENIX conference on Hot Topics in Cloud Computing. USENIX Association, 21--21."},{"key":"e_1_3_2_2_31_1","volume-title":"SoCC '16: Proceedings of the Seventh Symposium on Cloud Computing.","author":"Sambasivan Raja R.","unstructured":"Raja R. Sambasivan , Ilari Shafer , Jonathan Mace , Benjamin H Sigelman , Rodrigo Fonseca , and Gregory R. Ganger . 2016. Principled workflow-centric tracing of distributed systems . In SoCC '16: Proceedings of the Seventh Symposium on Cloud Computing. Raja R. Sambasivan, Ilari Shafer, Jonathan Mace, Benjamin H Sigelman, Rodrigo Fonseca, and Gregory R. Ganger. 2016. Principled workflow-centric tracing of distributed systems. In SoCC '16: Proceedings of the Seventh Symposium on Cloud Computing."},{"key":"e_1_3_2_2_32_1","volume-title":"NSDI'11: Proceedings of the 8th USENIX Conference on Networked Systems Design and Implementation.","author":"Sambasivan Raja R.","unstructured":"Raja R. Sambasivan , Alice X. Zheng , Michael De Rosa , Elie Krevat , Spencer Whitman , Michael Stroucken , William Wang , Lianghong Xu , and Gregory R. Ganger . 2011. Diagnosing performance changes by comparing request flows . In NSDI'11: Proceedings of the 8th USENIX Conference on Networked Systems Design and Implementation. Raja R. Sambasivan, Alice X. Zheng, Michael De Rosa, Elie Krevat, Spencer Whitman, Michael Stroucken, William Wang, Lianghong Xu, and Gregory R. Ganger. 2011. Diagnosing performance changes by comparing request flows. In NSDI'11: Proceedings of the 8th USENIX Conference on Networked Systems Design and Implementation."},{"key":"e_1_3_2_2_33_1","volume-title":"Mastering Distributed Tracing: Analyzing performance in microservices and complex systems","author":"Shkuro Yuri","unstructured":"Yuri Shkuro . 2019. Mastering Distributed Tracing: Analyzing performance in microservices and complex systems . Packt Publishing Ltd . Yuri Shkuro. 2019. Mastering Distributed Tracing: Analyzing performance in microservices and complex systems. Packt Publishing Ltd."},{"key":"e_1_3_2_2_35_1","unstructured":"The Apache Hadoop Distributed File System 2013. The Apache Hadoop Distributed File System. http:\/\/hadoop.apache.org\/hdfs\/.  The Apache Hadoop Distributed File System 2013. The Apache Hadoop Distributed File System. http:\/\/hadoop.apache.org\/hdfs\/."},{"key":"e_1_3_2_2_36_1","first-page":"2","article-title":"Challenges and Solutions for Tracing Storage Systems: A Case Study with Spectrum Scale","volume":"14","author":"Vef Marc-Andr\u00e9","year":"2018","unstructured":"Marc-Andr\u00e9 Vef , Vasily Tarasov , Dean Hildebrand , and Andr\u00e9 Brinkmann . 2018 . Challenges and Solutions for Tracing Storage Systems: A Case Study with Spectrum Scale . ACM Transactions on Storage (TOS) 14 , 2 (May 2018), 18--24. Marc-Andr\u00e9 Vef, Vasily Tarasov, Dean Hildebrand, and Andr\u00e9 Brinkmann. 2018. Challenges and Solutions for Tracing Storage Systems: A Case Study with Spectrum Scale. ACM Transactions on Storage (TOS) 14, 2 (May 2018), 18--24.","journal-title":"ACM Transactions on Storage (TOS)"},{"key":"e_1_3_2_2_37_1","volume-title":"All of Statistics: A Concise Course in Statistical Inference","author":"Wasserman Larry","unstructured":"Larry Wasserman . 2010. All of Statistics: A Concise Course in Statistical Inference . Springer Publishing Company, Inc orporated. Larry Wasserman. 2010. All of Statistics: A Concise Course in Statistical Inference. Springer Publishing Company, Incorporated."},{"key":"e_1_3_2_2_38_1","volume-title":"OSDI '06: Proceedings of the 7th USENIX Symposium on Operating Systems Design and Implementation.","author":"Weil Sage A","year":"2006","unstructured":"Sage A Weil , Scott A Brandt , Ethan L Miller , Darrell D E Long , and Carlos Maltzahn . 2006 . Ceph: a scalable, high-performance distributed file system . In OSDI '06: Proceedings of the 7th USENIX Symposium on Operating Systems Design and Implementation. Sage A Weil, Scott A Brandt, Ethan L Miller, Darrell D E Long, and Carlos Maltzahn. 2006. Ceph: a scalable, high-performance distributed file system. In OSDI '06: Proceedings of the 7th USENIX Symposium on Operating Systems Design and Implementation."},{"key":"e_1_3_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/1736020.1736038"},{"key":"e_1_3_2_2_40_1","volume-title":"OSDI' 12: Proceedings of the 10th conferences on Operating Systems Design & Implementation.","author":"Yuan Ding","year":"2012","unstructured":"Ding Yuan , Soyeon Park , Peng Huang , Yang Liu , Michael M Lee , Xiaoming Tang , Yuanyuan Zhou , and Stefan Savage . 2012 . Be conservative: enhancing failure diagnosis with proactive logging . In OSDI' 12: Proceedings of the 10th conferences on Operating Systems Design & Implementation. Ding Yuan, Soyeon Park, Peng Huang, Yang Liu, Michael M Lee, Xiaoming Tang, Yuanyuan Zhou, and Stefan Savage. 2012. Be conservative: enhancing failure diagnosis with proactive logging. In OSDI' 12: Proceedings of the 10th conferences on Operating Systems Design & Implementation."},{"key":"e_1_3_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/2248487.1950369"},{"key":"e_1_3_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132747.3132778"},{"key":"e_1_3_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/2983990.2984005"}],"event":{"name":"SoCC '21: ACM Symposium on Cloud Computing","location":"Seattle WA USA","acronym":"SoCC '21","sponsor":["SIGMOD ACM Special Interest Group on Management of Data","SIGOPS ACM Special Interest Group on Operating Systems"]},"container-title":["Proceedings of the ACM Symposium on Cloud Computing"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3472883.3487000","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/abs\/10.1145\/3472883.3487000","content-type":"text\/html","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3472883.3487000","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3472883.3487000","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:11:57Z","timestamp":1750191117000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3472883.3487000"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,11]]},"references-count":42,"alternative-id":["10.1145\/3472883.3487000","10.1145\/3472883"],"URL":"https:\/\/doi.org\/10.1145\/3472883.3487000","relation":{},"subject":[],"published":{"date-parts":[[2021,11]]},"assertion":[{"value":"2021-11-01","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}