{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,7]],"date-time":"2026-03-07T18:38:50Z","timestamp":1772908730184,"version":"3.50.1"},"reference-count":5,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2020,4,30]],"date-time":"2020-04-30T00:00:00Z","timestamp":1588204800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Queue"],"published-print":{"date-parts":[[2020,4,30]]},"abstract":"<jats:p>This article covers the outcomes of research performed in 2019 on how engineers at Google debug production issues, including the types of tools, high-level strategies, and low-level tasks that engineers use in varying combinations to debug effectively. It examines the research approach used to capture data, summarizing the common engineering journeys for production investigations and sharing examples of how experts debug complex distributed systems. Finally, the article extends the Google specifics of this research to provide some practical strategies that you can apply in your organization.<\/jats:p>","DOI":"10.1145\/3400899.3404974","type":"journal-article","created":{"date-parts":[[2020,6,7]],"date-time":"2020-06-07T00:57:45Z","timestamp":1591491465000},"page":"47-66","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["Debugging Incidents in Google\u2019s Distributed Systems"],"prefix":"10.1145","volume":"18","author":[{"given":"Charisma","family":"Chan","sequence":"first","affiliation":[{"name":"Google"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Beth","family":"Cooper","sequence":"additional","affiliation":[{"name":"Google"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2020,6,6]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"2016. Building Secure and Reliable Systems","author":"Beyer B.","unstructured":"Beyer, B., Jones, C., Petoff, J., Murphy, N. R., eds. 2016. Building Secure and Reliable Systems. O'Reilly Media; https:\/\/landing.google.com\/sre\/books\/."},{"key":"e_1_2_1_2_1","volume-title":"2018. The Site Reliability Workbook: Practical Ways to Implement SRE","author":"Beyer B.","unstructured":"Beyer, B., Murphy, N. R., Rensin, D. K., Kawahara, K., Thorne, S., eds. 2018. The Site Reliability Workbook: Practical Ways to Implement SRE. O'Reilly Media; https:\/\/landing.google.com\/sre\/books\/."},{"key":"e_1_2_1_3_1","unstructured":"Google Cloud. 2020. Metric list; https:\/\/cloud.google.com\/monitoring\/api\/metrics."},{"key":"e_1_2_1_4_1","volume-title":"Postmortem culture: learning from failure","author":"Lunney J.","unstructured":"Lunney, J., Lueder, S. 2017. Postmortem culture: learning from failure. O'Reilly Media; https:\/\/landing.google.com\/sre\/sre-book\/chapters\/postmortem-culture\/."},{"key":"e_1_2_1_5_1","volume-title":"Spotlight on Cloud: Reducing the Impact of Service Outages with Generic Mitigations with Jennifer Mace","author":"Mace J.","year":"2034","unstructured":"Mace, J. 2019. Spotlight on Cloud: Reducing the Impact of Service Outages with Generic Mitigations with Jennifer Mace. O'Reilly Media; https:\/\/www.oreilly.com\/library\/view\/spotlight-on-cloud\/0636920347927\/."}],"container-title":["Queue"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3400899.3404974","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3400899.3404974","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:38:43Z","timestamp":1750199923000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3400899.3404974"}},"subtitle":["How experts debug production issues in complex distributed systems"],"short-title":[],"issued":{"date-parts":[[2020,4,30]]},"references-count":5,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2020,4,30]]}},"alternative-id":["10.1145\/3400899.3404974"],"URL":"https:\/\/doi.org\/10.1145\/3400899.3404974","relation":{},"ISSN":["1542-7730","1542-7749"],"issn-type":[{"value":"1542-7730","type":"print"},{"value":"1542-7749","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,4,30]]},"assertion":[{"value":"2020-06-06","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}