{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,8]],"date-time":"2026-08-08T17:48:08Z","timestamp":1786211288048,"version":"3.56.0"},"reference-count":57,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2024,12,10]],"date-time":"2024-12-10T00:00:00Z","timestamp":1733788800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100006374","name":"NSF","doi-asserted-by":"publisher","award":["1910568 and 1943456"],"award-info":[{"award-number":["1910568 and 1943456"]}],"id":[{"id":"10.13039\/501100006374","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Meas. Anal. Comput. Syst."],"published-print":{"date-parts":[[2024,12,10]]},"abstract":"<jats:p>\n            Microservice architecture is the computing paradigm of choice for large, service-oriented software catering to real-time requests. Individual programs in such a system perform Remote Procedure Calls (RPCs) to other microservices to accomplish sub-tasks. Microservices are designed to be robust; top-level requests can succeed despite errors returned from RPC sub-tasks, referred to as\n            <jats:bold>\n              <jats:italic toggle=\"yes\">non-fatal errors<\/jats:italic>\n              .\n            <\/jats:bold>\n            Because of this design, the top-level microservices tend to ''live with'' non-fatal errors. Hence, a natural question to ask is ''how prevalent are non-fatal errors and what impact do they have on the exposed latency of top-level requests?''\n          <\/jats:p>\n          <jats:p>In this paper, we present a large-scale study of errors in microservices. We answer the aforementioned question by analyzing 11 Billion RPCs covering 1,900 user-facing endpoints at the Uber serving requests of hundreds of millions of active users. To assess the latency impact of non-fatal errors, we develop a methodology that projects potential latency savings for a given request as if the time spent on failing APIs were eliminated. This estimator allows ranking and bubbling up those APIs that are worthy of further investigations, where the non-fatal errors likely resulted in operational inefficiencies. Finally, we employ our error detection and impact estimation techniques to pinpoint operational inefficiencies, which a) result in a tail latency reduction of a critical endpoint by 30% and b) offer insights into common inefficiency-introducing patterns.<\/jats:p>","DOI":"10.1145\/3700436","type":"journal-article","created":{"date-parts":[[2024,12,13]],"date-time":"2024-12-13T12:12:12Z","timestamp":1734091932000},"page":"1-36","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":12,"title":["The Tale of Errors in Microservices"],"prefix":"10.1145","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0687-5508","authenticated-orcid":false,"given":"I-Ting Angelina","family":"Lee","sequence":"first","affiliation":[{"name":"Washington University in St. Louis, St. Louis, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5517-6308","authenticated-orcid":false,"given":"Zhizhou","family":"Zhang","sequence":"additional","affiliation":[{"name":"Uber Technologies Inc., Sunnyvale, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-4021-3541","authenticated-orcid":false,"given":"Abhishek","family":"Parwal","sequence":"additional","affiliation":[{"name":"Uber Technologies Inc., Sunnyvale, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1021-7644","authenticated-orcid":false,"given":"Milind","family":"Chabbi","sequence":"additional","affiliation":[{"name":"Uber Technologies Inc., Sunnyvale, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,12,13]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"MySQL. https:\/\/www.mysql.com\/. (Accessed on 07\/30\/2023)."},{"key":"e_1_2_1_2_1","unstructured":"Status codes and their use in gRPC . https:\/\/grpc.github.io\/grpc\/core\/md_doc_statuscodes.html."},{"key":"e_1_2_1_3_1","unstructured":"yarpc\/yarpc-go: A message passing platform for go. https:\/\/github.com\/yarpc\/yarpc-go. (Accessed on 07\/30\/2023)."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.5555\/1753228.1753233"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/258915.258924"},{"key":"e_1_2_1_6_1","unstructured":"AppCentrica. Microservices the good bad and the ugly. https:\/\/www.appcentrica.com\/the-rise-of-microservices\/."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3357223.3362704"},{"key":"e_1_2_1_8_1","volume-title":"Jaeger: open source, end-to-end distributed tracing","author":"Authors Jaeger","year":"2022","unstructured":"Jaeger Authors. Jaeger: open source, end-to-end distributed tracing, 2022."},{"key":"e_1_2_1_9_1","volume-title":"Identifying the root causes of wait states in large-scale parallel applications. ACM Trans. Parallel Comput., 3(2), jul","author":"B\u00f6hme David","year":"2016","unstructured":"David B\u00f6hme, Markus Geimer, Lukas Arnold, Felix Voigtlaender, and Felix Wolf. Identifying the root causes of wait states in large-scale parallel applications. ACM Trans. Parallel Comput., 3(2), jul 2016."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/2815400.2815409"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2408776.2408794"},{"key":"e_1_2_1_12_1","volume-title":"Under the hood of uber's experimentation platform. Uber engineering","author":"Deb Anirban","year":"2018","unstructured":"Anirban Deb, Suman Bhattacharya, Jeremey Gu, T Zhou, Eva Feng, and Mandie Liu. Under the hood of uber's experimentation platform. Uber engineering., 2018."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1142\/11673"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3526967"},{"key":"e_1_2_1_15_1","first-page":"1037","volume-title":"2022 USENIX Annual Technical Conference (USENIX ATC 22)","author":"Elhemali Mostafa","year":"2022","unstructured":"Mostafa Elhemali, Niall Gallagher, Bin Tang, Nick Gordon, Hao Huang, Haibo Chen, Joseph Idziorek, Mengtian Wang, Richard Krog, Zongpeng Zhu, et al. Amazon {DynamoDB}: A scalable, predictably performant, and fully managed {NoSQL} database service. In 2022 USENIX Annual Technical Conference (USENIX ATC 22), pages 1037--1048, 2022."},{"key":"e_1_2_1_16_1","volume-title":"Feb","author":"Ensono Paul Durzan","year":"2020","unstructured":"Paul Durzan Ensono. The growth of microservices and why it matters to your business. https:\/\/www. datacenterdynamics.com\/en\/opinions\/growth-microservices-and-why-it-matters-your-business\/, Feb 2020."},{"key":"e_1_2_1_17_1","volume-title":"Fay: Extensible distributed tracing from kernels to clusters. ACM Transactions on Computer Systems (TOCS), 30(4):1--35","author":"Erlingsson \u00dalfar","year":"2012","unstructured":"\u00dalfar Erlingsson, Marcus Peinado, Simon Peter, Mihai Budiu, and Gloria Mainar-Ruiz. Fay: Extensible distributed tracing from kernels to clusters. ACM Transactions on Computer Systems (TOCS), 30(4):1--35, 2012."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.5555\/932295"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3297858.3304013"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3297858.3304004"},{"key":"e_1_2_1_21_1","volume-title":"Uber Engineering Blog","author":"Gluck Adam","year":"2020","unstructured":"Adam Gluck. Introducing domain-oriented microservice architecture. Uber Engineering Blog, 2020."},{"key":"e_1_2_1_22_1","volume-title":"27th Large Installation System Administration Conference. USENIX Association","author":"Gregg Brendan","year":"2013","unstructured":"Brendan Gregg. Blazing performance with flame graphs. In 27th Large Installation System Administration Conference. USENIX Association, 2013."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2007.55"},{"key":"e_1_2_1_24_1","volume-title":"high performance, open-source universal rpc framework","author":"A","year":"2018","unstructured":"A gRPC. high performance, open-source universal rpc framework, 2018."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3417066"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3472883.3486994"},{"key":"e_1_2_1_27_1","first-page":"419","volume-title":"2023 USENIX Annual Technical Conference (USENIX ATC 23)","author":"Huye Darby","year":"2023","unstructured":"Darby Huye, Yuri Shkuro, and Raja R Sambasivan. Lifting the veil on {Meta's} microservice architecture: Analyses of topology and request workflows. In 2023 USENIX Annual Technical Conference (USENIX ATC 23), pages 419--432, 2023."},{"key":"e_1_2_1_28_1","unstructured":"infoq. Controlled Chaos: Taming Organic Federated Growth of Microservices. https:\/\/www.infoq.com\/presentations\/ microservices-federated-growth\/."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132747.3132749"},{"key":"e_1_2_1_30_1","volume-title":"Cassandra: a decentralized structured storage system. ACM SIGOPS operating systems review, 44(2):35--40","author":"Lakshman Avinash","year":"2010","unstructured":"Avinash Lakshman and Prashant Malik. Cassandra: a decentralized structured storage system. ACM SIGOPS operating systems review, 44(2):35--40, 2010."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/IWQOS52092.2021.9521340"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEIP52600.2021.00043"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE5003.2020.00014"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3472883.3487003"},{"key":"e_1_2_1_35_1","volume-title":"Pivot tracing: Dynamic causal monitoring for distributed systems. ACM Transactions on Computer Systems (TOCS), 35(4):1--28","author":"Mace Jonathan","year":"2018","unstructured":"Jonathan Mace, Ryan Roelke, and Rodrigo Fonseca. Pivot tracing: Dynamic causal monitoring for distributed systems. ACM Transactions on Computer Systems (TOCS), 35(4):1--28, 2018."},{"key":"e_1_2_1_36_1","volume-title":"Diagnosing latency in multi-tier black-box services","author":"Ostrowski Krzysztof","year":"2011","unstructured":"Krzysztof Ostrowski, Gideon Mann, and Mark Sandler. Diagnosing latency in multi-tier black-box services. 2011."},{"key":"e_1_2_1_37_1","volume-title":"September","author":"Plantrou Gaspard","year":"2022","unstructured":"Gaspard Plantrou. Microservices are becoming the default application architecture choice -- is it time to jump in? https:\/\/www.scaleway.com\/en\/blog\/microservices-are-becoming-the-default-application-architecture-choice-isit- time-to-jump-in\/, September 2022."},{"key":"e_1_2_1_38_1","first-page":"805","volume-title":"14th USENIX symposium on operating systems design and implementation (OSDI 20)","author":"Qiu Haoran","year":"2020","unstructured":"Haoran Qiu, Subho S Banerjee, Saurabh Jha, Zbigniew T Kalbarczyk, and Ravishankar K Iyer. {FIRM}: An intelligent fine-grained resource management framework for {SLO-Oriented} microservices. In 14th USENIX symposium on operating systems design and implementation (OSDI 20), pages 805--825, 2020."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377813.3381350"},{"key":"e_1_2_1_40_1","unstructured":"Chris Richardson. Service-oriented architecture. https:\/\/microservices.io\/."},{"key":"e_1_2_1_41_1","volume-title":"October","author":"Sasidhara Deepu","year":"2019","unstructured":"Deepu Sasidhara. Microservices the good bad and the ugly. https:\/\/deepu.tech\/microservices-the-good-bad-and-theugly\/, October 2019."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3600006.3613156"},{"key":"e_1_2_1_43_1","volume-title":"Mike Burrows, Pat Stephenson, Manoj Plakal, Donald Beaver, Saul Jaspan, and Chandan Shanbhag. Dapper, a large-scale distributed systems tracing infrastructure.","author":"Sigelman Benjamin H","year":"2010","unstructured":"Benjamin H Sigelman, Luiz Andr\u00e9 Barroso, Mike Burrows, Pat Stephenson, Manoj Plakal, Donald Beaver, Saul Jaspan, and Chandan Shanbhag. Dapper, a large-scale distributed systems tracing infrastructure. 2010."},{"key":"e_1_2_1_44_1","volume-title":"Thrift: Scalable cross-language services implementation. Facebook white paper, 5(8):127","author":"Slee Mark","year":"2007","unstructured":"Mark Slee, Aditya Agarwal, and Marc Kwiatkowski. Thrift: Scalable cross-language services implementation. Facebook white paper, 5(8):127, 2007."},{"key":"e_1_2_1_45_1","volume-title":"Anomaly detection and failure root cause analysis in (micro) service-based cloud applications: A survey. ACM Computing Surveys (CSUR), 55(3):1--39","author":"Soldani Jacopo","year":"2022","unstructured":"Jacopo Soldani and Antonio Brogi. Anomaly detection and failure root cause analysis in (micro) service-based cloud applications: A survey. ACM Computing Surveys (CSUR), 55(3):1--39, 2022."},{"key":"e_1_2_1_46_1","volume-title":"Tania Lorido Botran, and Anshul Gandhi. Gamma: Graph neural network-based multi-bottleneck localization for microservices applications","author":"Somashekar Gagan","year":"2024","unstructured":"Gagan Somashekar, Anurag Dutt, Mainak Adak, Tania Lorido Botran, and Anshul Gandhi. Gamma: Graph neural network-based multi-bottleneck localization for microservices applications. 2024."},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3472883.3487000"},{"key":"e_1_2_1_48_1","volume-title":"Understanding design of microservices architecture at netflix","author":"Varshneya Ketan","year":"2021","unstructured":"Ketan Varshneya. Understanding design of microservices architecture at netflix, 2021."},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1002\/smr.2467"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1109\/NOMS47738.2020.9110353"},{"key":"e_1_2_1_51_1","first-page":"395","volume-title":"16th USENIX Symposium on Networked Systems Design and Implementation (NSDI 19)","author":"Wu Yang","year":"2019","unstructured":"Yang Wu, Ang Chen, and Linh Thi Xuan Phan. Zeno: Diagnosing performance problems with temporal provenance. In 16th USENIX Symposium on Networked Systems Design and Implementation (NSDI 19), pages 395--420, 2019."},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1109\/DCS.1988.12538"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639088"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/3620678.3624787"},{"key":"e_1_2_1_55_1","first-page":"655","volume-title":"2022 USENIX Annual Technical Conference (USENIX ATC 22)","author":"Zhang Zhizhou","year":"2022","unstructured":"Zhizhou Zhang, Murali Krishna Ramanathan, Prithvi Raj, Abhishek Parwal, Timothy Sherwood, and Milind Chabbi. CRISP: Critical path analysis of Large-Scale microservice architectures. In 2022 USENIX Annual Technical Conference (USENIX ATC 22), pages 655--672, Carlsbad, CA, July 2022. USENIX Association."},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2018.2887384"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338961"}],"container-title":["Proceedings of the ACM on Measurement and Analysis of Computing Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3700436","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3700436","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,23]],"date-time":"2025-08-23T00:13:32Z","timestamp":1755908012000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3700436"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,10]]},"references-count":57,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2024,12,10]]}},"alternative-id":["10.1145\/3700436"],"URL":"https:\/\/doi.org\/10.1145\/3700436","relation":{},"ISSN":["2476-1249"],"issn-type":[{"value":"2476-1249","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,12,10]]},"assertion":[{"value":"2024-12-13","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}