{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,6]],"date-time":"2026-03-06T08:48:49Z","timestamp":1772786929703,"version":"3.50.1"},"reference-count":52,"publisher":"Association for Computing Machinery (ACM)","issue":"5","license":[{"start":{"date-parts":[[2017,10,1]],"date-time":"2017-10-01T00:00:00Z","timestamp":1506816000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Queue"],"published-print":{"date-parts":[[2017,10]]},"abstract":"<jats:p>This article presents a call to arms for the distributed systems research community to improve the state of the art in fault tolerance testing. Ordinary users need tools that automate the selection of custom-tailored faults to inject. We conjecture that the process by which superusers select experiments can be effectively modeled in software. The article describes a prototype validating this conjecture, presents early results from the lab and the field, and identifies new research directions that can make this vision a reality.<\/jats:p>","DOI":"10.1145\/3155112.3155114","type":"journal-article","created":{"date-parts":[[2017,10,27]],"date-time":"2017-10-27T12:48:13Z","timestamp":1509108493000},"page":"29-53","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":10,"title":["Abstracting the Geniuses Away from Failure Testing"],"prefix":"10.1145","volume":"15","author":[{"given":"Peter","family":"Alvaro","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Severine","family":"Tymon","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2017,10]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/2987550.2987555"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2723711"},{"key":"e_1_2_1_3_1","unstructured":"Andrus K. 2016. Personal communication.  Andrus K. 2016. Personal communication."},{"key":"e_1_2_1_4_1","unstructured":"Aniszczyk C. 2012. Distributed systems tracing with Zipkin. Twitter Engineering; https:\/\/blog.twitter.com\/2012\/distributed-systems-tracing-with-zipkin.  Aniszczyk C. 2012. Distributed systems tracing with Zipkin. Twitter Engineering; https:\/\/blog.twitter.com\/2012\/distributed-systems-tracing-with-zipkin."},{"key":"e_1_2_1_5_1","unstructured":"Barth D. 2014. Inject failure to make your systems more reliable. DevOps.com; http:\/\/devops.com\/2014\/06\/03\/inject-failure\/.  Barth D. 2014. Inject failure to make your systems more reliable. DevOps.com; http:\/\/devops.com\/2014\/06\/03\/inject-failure\/."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2016.60"},{"key":"e_1_2_1_7_1","unstructured":"Beyer B. Jones C. Petoff J. Murphy N. R. 2016. Site Reliability Engineering. O'Reilly.  Beyer B. Jones C. Petoff J. Murphy N. R. 2016. Site Reliability Engineering. O'Reilly."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2080.357392"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/234533.234549"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/2934872.2934910"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.14778\/2994509.2994530"},{"key":"e_1_2_1_12_1","volume-title":"Proceedings of the 11th Usenix Conference on Operating Systems Design and Implementation: 217-231","author":"Chow M.","year":"2014"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/357775.357777"},{"key":"e_1_2_1_14_1","volume-title":"Proceedings of the 26th International Symposium on Fault-tolerant Computing.","author":"Dawson S.","year":"1996"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3149.214121"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-78800-3_22"},{"key":"e_1_2_1_17_1","unstructured":"Andrus K. Gopalani N. Schmaus B. 2014. FIT: failure injection testing. Netflix Technology Blog; http:\/\/techblog.netflix.com\/2014\/10\/fit-failure-injection-testing.html.  Andrus K. Gopalani N. Schmaus B. 2014. FIT: failure injection testing. Netflix Technology Blog; http:\/\/techblog.netflix.com\/2014\/10\/fit-failure-injection-testing.html."},{"key":"e_1_2_1_19_1","volume-title":"Proceedings of the 8th Usenix Conference on Networked Systems Design and Implementation: 238-252; http:\/\/db.cs.berkeley.edu\/papers\/nsdi11-fate-destini.pdf.","author":"Gunawi H. S.","year":"2011"},{"key":"e_1_2_1_20_1","unstructured":"Holzmann G. 2003. The SPIN Model Checker: Primer and Reference Manual. Addison-Wesley Professional.  Holzmann G. 2003. The SPIN Model Checker: Primer and Reference Manual. Addison-Wesley Professional."},{"key":"e_1_2_1_21_1","unstructured":"Honeycomb. 2016; https:\/\/honeycomb.io\/.  Honeycomb. 2016; https:\/\/honeycomb.io\/."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.14778\/2850583.2850595"},{"key":"e_1_2_1_23_1","unstructured":"Izrailevsky Y. Tseitlin A. 2011. The Netflix Simian Army. Netflix Technology Blog; http:\/\/techblog.netflix.com\/2011\/07\/netflix-simian-army.html.  Izrailevsky Y. Tseitlin A. 2011. The Netflix Simian Army. Netflix Technology Blog; http:\/\/techblog.netflix.com\/2011\/07\/netflix-simian-army.html."},{"key":"e_1_2_1_24_1","unstructured":"Jepsen. 2016. Distributed systems safety research; http:\/\/jepsen.io\/.  Jepsen. 2016. Distributed systems safety research; http:\/\/jepsen.io\/."},{"key":"e_1_2_1_25_1","unstructured":"Jones N. 2016. Personal communication.  Jones N. 2016. Personal communication."},{"key":"e_1_2_1_26_1","unstructured":"Kafka 0.8.0. 2013. Apache; https:\/\/kafka.apache.org\/08\/documentation.html.  Kafka 0.8.0. 2013. Apache; https:\/\/kafka.apache.org\/08\/documentation.html."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/12.364536"},{"key":"e_1_2_1_29_1","first-page":"243","article-title":"Life, death, and the critical transition: finding liveness bugs in systems code","author":"Killian C. E.","year":"2007","journal-title":"Networked System Design and Implementation"},{"key":"e_1_2_1_30_1","unstructured":"Kingsbury K. 2013. Call me maybe: Kafka; http:\/\/aphyr.com\/posts\/293-call-me-maybe-kafka.  Kingsbury K. 2013. Call me maybe: Kafka; http:\/\/aphyr.com\/posts\/293-call-me-maybe-kafka."},{"key":"e_1_2_1_31_1","unstructured":"Kingsbury K. 2016. Personal communication.  Kingsbury K. 2016. Personal communication."},{"key":"e_1_2_1_32_1","unstructured":"Lafeldt M. 2017. The discipline of Chaos Engineering. Gremlin Inc.; https:\/\/blog.gremlininc.com\/the-discipline-of-chaos-engineering-e39d2383c459.  Lafeldt M. 2017. The discipline of Chaos Engineering. Gremlin Inc.; https:\/\/blog.gremlininc.com\/the-discipline-of-chaos-engineering-e39d2383c459."},{"key":"e_1_2_1_33_1","unstructured":"Lampson B. W. 1980. Atomic transactions. In Distributed Systems Architecture and Implementation An Advanced Course: 246-265; https:\/\/link.springer.com\/chapter\/10.1007%2F3-540-10571-9_11.  Lampson B. W. 1980. Atomic transactions. In Distributed Systems Architecture and Implementation An Advanced Course: 246-265; https:\/\/link.springer.com\/chapter\/10.1007%2F3-540-10571-9_11."},{"key":"e_1_2_1_34_1","unstructured":"LightStep. 2016; http:\/\/lightstep.com\/.  LightStep. 2016; http:\/\/lightstep.com\/."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/DSN.2009.5270313"},{"key":"e_1_2_1_36_1","unstructured":"Matloff N. Salzman P. J. 2008. The Art of Debugging with GDB DDD and Eclipse. No Starch Press.  Matloff N. Salzman P. J. 2008. The Art of Debugging with GDB DDD and Eclipse. No Starch Press."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2213836.2213875"},{"key":"e_1_2_1_38_1","unstructured":"Microsoft Azure Documentation. 2016. Introduction to the fault analysis service; https:\/\/azure.microsoft.com\/en-us\/documentation\/articles\/service-fabric-testability-overview\/.  Microsoft Azure Documentation. 2016. Introduction to the fault analysis service; https:\/\/azure.microsoft.com\/en-us\/documentation\/articles\/service-fabric-testability-overview\/."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/1060289.1060297"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.5555\/1855741.1855760"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/1989323.1989459"},{"key":"e_1_2_1_43_1","unstructured":"OpenTracing. 2016; http:\/\/opentracing.io\/.  OpenTracing. 2016; http:\/\/opentracing.io\/."},{"key":"e_1_2_1_44_1","unstructured":"Pasquier T. F. J. -M. Singh J. Eyers D.  M. Bacon J. 2015. CamFlow: managed data-sharing for cloud services; https:\/\/arxiv.org\/pdf\/1506.04391.pdf.  Pasquier T. F. J. -M. Singh J. Eyers D.  M. Bacon J. 2015. CamFlow: managed data-sharing for cloud services; https:\/\/arxiv.org\/pdf\/1506.04391.pdf."},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/50202.50214"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.5555\/3154580.3154593"},{"key":"e_1_2_1_47_1","unstructured":"Reinhold E. 2016. Rewriting Uber engineering: the opportunities microservices provide. Uber Engineering; https:\/\/eng.uber.com\/building-tincup\/.  Reinhold E. 2016. Rewriting Uber engineering: the opportunities microservices provide. Uber Engineering; https:\/\/eng.uber.com\/building-tincup\/."},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/357401.357402"},{"key":"e_1_2_1_49_1","volume-title":"Sun Microsystems. In Proceedings of the","author":"Sandberg R.","year":"1986"},{"key":"e_1_2_1_50_1","volume-title":"Jaeger: Uber's distributed tracing system. Uber Engineering","author":"Shkuro Y.","year":"2017"},{"key":"e_1_2_1_51_1","unstructured":"Sigelman B. H. Barroso L. A. Burrows M. Stephenson P. Plakal M. Beaver D. Jaspan S. Shanbhag C. 2010. Dapper a large-scale distributed systems tracing infrastructure. Technical report. Research at Google; https:\/\/research.google.com\/pubs\/pub36356.html.  Sigelman B. H. Barroso L. A. Burrows M. Stephenson P. Plakal M. Beaver D. Jaspan S. Shanbhag C. 2010. Dapper a large-scale distributed systems tracing infrastructure. Technical report. Research at Google; https:\/\/research.google.com\/pubs\/pub36356.html."},{"key":"e_1_2_1_52_1","unstructured":"Shenoy A. 2016. A deep dive into Simoorg: our open source failure induction framework. Linkedin Engineering; https:\/\/engineering.linkedin.com\/blog\/2016\/03\/deep-dive-Simoorg-open-source-failure-induction-framework.  Shenoy A. 2016. A deep dive into Simoorg: our open source failure induction framework. Linkedin Engineering; https:\/\/engineering.linkedin.com\/blog\/2016\/03\/deep-dive-Simoorg-open-source-failure-induction-framework."},{"key":"e_1_2_1_53_1","volume-title":"Proceedings of the 6th Usenix Symposium on Networked Systems Design and Implementation: 213-228","author":"Yang J.","year":"2009"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.5555\/646704.702012"},{"key":"e_1_2_1_55_1","volume-title":"Proceedings of the 11th Usenix Conference on Operating Systems Design and Implementation: 629-644","author":"Zhao X.","year":"2014"}],"container-title":["Queue"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3155112.3155114","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3155112.3155114","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T03:30:28Z","timestamp":1750217428000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3155112.3155114"}},"subtitle":["Ordinary users need tools that automate the selection of custom-tailored faults to inject."],"short-title":[],"issued":{"date-parts":[[2017,10]]},"references-count":52,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2017,10]]}},"alternative-id":["10.1145\/3155112.3155114"],"URL":"https:\/\/doi.org\/10.1145\/3155112.3155114","relation":{},"ISSN":["1542-7730","1542-7749"],"issn-type":[{"value":"1542-7730","type":"print"},{"value":"1542-7749","type":"electronic"}],"subject":[],"published":{"date-parts":[[2017,10]]},"assertion":[{"value":"2017-10-01","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}