{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,3]],"date-time":"2026-06-03T07:54:04Z","timestamp":1780473244571,"version":"3.54.1"},"publisher-location":"New York, NY, USA","reference-count":50,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,11,1]],"date-time":"2021-11-01T00:00:00Z","timestamp":1635724800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,11]]},"DOI":"10.1145\/3472883.3487005","type":"proceedings-article","created":{"date-parts":[[2021,10,27]],"date-time":"2021-10-27T10:48:16Z","timestamp":1635331696000},"page":"388-402","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":24,"title":["Service-Level Fault Injection Testing"],"prefix":"10.1145","author":[{"given":"Christopher S.","family":"Meiklejohn","sequence":"first","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Andrea","family":"Estrada","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yiwen","family":"Song","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Heather","family":"Miller","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Rohan","family":"Padhye","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2021,11]]},"reference":[{"key":"e_1_3_2_2_1_1","unstructured":"2016. Building Microservices in Python and Flask. https:\/\/codeahoy. com\/2016\/07\/10\/writing-microservices-in-python-using-flask. Accessed: 2021-05-21.  2016. Building Microservices in Python and Flask. https:\/\/codeahoy. com\/2016\/07\/10\/writing-microservices-in-python-using-flask. Accessed: 2021-05-21."},{"key":"e_1_3_2_2_2_1","unstructured":"2018. LinkedOut: A Request-Level Failure Injection Framework. https:\/\/engineering.linkedin.com\/blog\/2018\/05\/linkedout--a-request-level-failure-injection-framework. Accessed: 2021-05-21.  2018. LinkedOut: A Request-Level Failure Injection Framework. https:\/\/engineering.linkedin.com\/blog\/2018\/05\/linkedout--a-request-level-failure-injection-framework. Accessed: 2021-05-21."},{"key":"e_1_3_2_2_3_1","unstructured":"2020. Automating Chaos Attacks at Expedia - Daniel and Nikos. https:\/\/www.youtube.com\/watch?v=xrtbiyfRvb4. Accessed: 2021-05-21.  2020. Automating Chaos Attacks at Expedia - Daniel and Nikos. https:\/\/www.youtube.com\/watch?v=xrtbiyfRvb4. Accessed: 2021-05-21."},{"key":"e_1_3_2_2_4_1","unstructured":"2020. Introducing Domain-Oriented Microservice Architecture. https:\/\/eng.uber.com\/microservice-architecture\/. Accessed: 2021-05-21.  2020. Introducing Domain-Oriented Microservice Architecture. https:\/\/eng.uber.com\/microservice-architecture\/. Accessed: 2021-05-21."},{"key":"e_1_3_2_2_5_1","unstructured":"2020. Rethinking How the Industry Approaches Chaos Engineering. https:\/\/www.infoq.com\/presentations\/rethinking-chaos-engineering. Accessed: 2021-05-21.  2020. Rethinking How the Industry Approaches Chaos Engineering. https:\/\/www.infoq.com\/presentations\/rethinking-chaos-engineering. Accessed: 2021-05-21."},{"key":"e_1_3_2_2_6_1","unstructured":"2021. Amazon EKS | Managed Kubernetes Service. https:\/\/aws.amazon.com\/eks\/. Accessed: 2021-05-21.  2021. Amazon EKS | Managed Kubernetes Service. https:\/\/aws.amazon.com\/eks\/. Accessed: 2021-05-21."},{"key":"e_1_3_2_2_7_1","unstructured":"2021. Audible. https:\/\/www.audible.com. Accessed: 2021-05-21.  2021. Audible. https:\/\/www.audible.com. Accessed: 2021-05-21."},{"key":"e_1_3_2_2_8_1","unstructured":"2021. docker. https:\/\/www.docker.com\/. Accessed: 2021-05-21.  2021. docker. https:\/\/www.docker.com\/. Accessed: 2021-05-21."},{"key":"e_1_3_2_2_9_1","unstructured":"2021. Expedia. https:\/\/www.expedia.com. Accessed: 2021-05-21.  2021. Expedia. https:\/\/www.expedia.com. Accessed: 2021-05-21."},{"key":"e_1_3_2_2_10_1","unstructured":"2021. Filibuster. http:\/\/filibuster.cloud. Accessed: 2021-09-07.  2021. Filibuster. http:\/\/filibuster.cloud. Accessed: 2021-09-07."},{"key":"e_1_3_2_2_11_1","unstructured":"2021. Flask web framework. https:\/\/flask.palletsprojects.com\/en\/2.0.x\/. Accessed: 2021-05-21.  2021. Flask web framework. https:\/\/flask.palletsprojects.com\/en\/2.0.x\/. Accessed: 2021-05-21."},{"key":"e_1_3_2_2_12_1","unstructured":"2021. Gremlin. http:\/\/www.gremlin.com. Accessed: 2021-05-21.  2021. Gremlin. http:\/\/www.gremlin.com. Accessed: 2021-05-21."},{"key":"e_1_3_2_2_13_1","unstructured":"2021. Mailchimp. https:\/\/www.mailchimp.com. Accessed: 2021-05-21.  2021. Mailchimp. https:\/\/www.mailchimp.com. Accessed: 2021-05-21."},{"key":"e_1_3_2_2_14_1","unstructured":"2021. minikube. https:\/\/minikube.sigs.k8s.io\/docs\/. Accessed: 2021-05-21.  2021. minikube. https:\/\/minikube.sigs.k8s.io\/docs\/. Accessed: 2021-05-21."},{"key":"e_1_3_2_2_15_1","unstructured":"2021. Netflix. https:\/\/www.netflix.com. Accessed: 2021-05-21.  2021. Netflix. https:\/\/www.netflix.com. Accessed: 2021-05-21."},{"key":"e_1_3_2_2_16_1","unstructured":"2021. Online Boutique. https:\/\/github.com\/GoogleCloudPlatform\/microservices-demo. Accessed: 2021-05-21.  2021. Online Boutique. https:\/\/github.com\/GoogleCloudPlatform\/microservices-demo. Accessed: 2021-05-21."},{"key":"e_1_3_2_2_17_1","unstructured":"2021. Sock Shop: A Microservices Demo Application. https:\/\/microservices-demo.github.io. Accessed: 2021-05-21.  2021. Sock Shop: A Microservices Demo Application. https:\/\/microservices-demo.github.io. Accessed: 2021-05-21."},{"key":"e_1_3_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2987550.2987555"},{"key":"e_1_3_2_2_19_1","volume-title":"Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data","author":"Alvaro Peter","unstructured":"Peter Alvaro , Joshua Rosen , and Joseph M. Hellerstein . 2015. Lineage-Driven Fault Injection . In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data ( Melbourne, Victoria, Australia) (SIGMOD '15). Association for Computing Machinery, New York, NY, USA, 331--346. https:\/\/doi.org\/10.1145\/2723372.2723711 10.1145\/2723372.2723711 Peter Alvaro, Joshua Rosen, and Joseph M. Hellerstein. 2015. Lineage-Driven Fault Injection. In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data (Melbourne, Victoria, Australia) (SIGMOD '15). Association for Computing Machinery, New York, NY, USA, 331--346. https:\/\/doi.org\/10.1145\/2723372.2723711"},{"key":"e_1_3_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.5555\/1777707.1777710"},{"key":"e_1_3_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2168836.2168865"},{"key":"e_1_3_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/IWESEP.2012.14"},{"key":"e_1_3_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEIP.2019.00012"},{"key":"e_1_3_2_2_24_1","volume-title":"Workshop on Self-Healing, Adaptive and Self-Managed Systems. Citeseer.","author":"Broadwell Pete","year":"2002","unstructured":"Pete Broadwell , Naveen Sastry , and Jonathan Traupman . 2002 . FIG: A prototype tool for online verification of recovery mechanisms . In Workshop on Self-Healing, Adaptive and Self-Managed Systems. Citeseer. Pete Broadwell, Naveen Sastry, and Jonathan Traupman. 2002. FIG: A prototype tool for online verification of recovery mechanisms. In Workshop on Self-Healing, Adaptive and Self-Managed Systems. Citeseer."},{"key":"e_1_3_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/1321631.1321702"},{"key":"e_1_3_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/MSR.2010.5463279"},{"key":"e_1_3_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/2.84874"},{"key":"e_1_3_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/1065010.1065036"},{"key":"e_1_3_2_2_29_1","first-page":"6","article-title":"DART","volume":"40","author":"Godefroid Patrice","year":"2005","unstructured":"Patrice Godefroid , Nils Klarlund , and Koushik Sen . 2005 . DART : Directed Automated Random Testing. SIGPLAN Not. 40 , 6 (June 2005), 213--223. https:\/\/doi.org\/10.1145\/1064978.1065036 10.1145\/1064978.1065036 Patrice Godefroid, Nils Klarlund, and Koushik Sen. 2005. DART: Directed Automated Random Testing. SIGPLAN Not. 40, 6 (June 2005), 213--223. https:\/\/doi.org\/10.1145\/1064978.1065036","journal-title":"Directed Automated Random Testing. SIGPLAN Not."},{"key":"e_1_3_2_2_30_1","volume-title":"Proceedings of the 8th USENIX Conference on Networked Systems Design and Implementation","author":"Gunawi Haryadi S.","year":"2011","unstructured":"Haryadi S. Gunawi , Thanh Do , Pallavi Joshi , Peter Alvaro , Joseph M. Hellerstein , Andrea C. Arpaci-Dusseau , Remzi H. Arpaci-Dusseau , Koushik Sen , and Dhruba Borthakur . 2011 . FATE and DESTINI: A Framework for Cloud Recovery Testing . In Proceedings of the 8th USENIX Conference on Networked Systems Design and Implementation ( Boston, MA) (NSDI'11). USENIX Association, USA, 238--252. Haryadi S. Gunawi, Thanh Do, Pallavi Joshi, Peter Alvaro, Joseph M. Hellerstein, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau, Koushik Sen, and Dhruba Borthakur. 2011. FATE and DESTINI: A Framework for Cloud Recovery Testing. In Proceedings of the 8th USENIX Conference on Networked Systems Design and Implementation (Boston, MA) (NSDI'11). USENIX Association, USA, 238--252."},{"key":"e_1_3_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2005.112"},{"key":"e_1_3_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2629648"},{"key":"e_1_3_2_2_33_1","volume-title":"Gremlin: Systematic Resilience Testing of Microservices. In 2016 IEEE 36th International Conference on Distributed Computing Systems (ICDCS). 57--66","author":"Heorhiadi Victor","year":"2016","unstructured":"Victor Heorhiadi , Shriram Rajagopalan , Hani Jamjoom , Michael K. Reiter , and Vyas Sekar . 2016 . Gremlin: Systematic Resilience Testing of Microservices. In 2016 IEEE 36th International Conference on Distributed Computing Systems (ICDCS). 57--66 . https:\/\/doi.org\/10.1109\/ICDCS.2016.11 10.1109\/ICDCS.2016.11 Victor Heorhiadi, Shriram Rajagopalan, Hani Jamjoom, Michael K. Reiter, and Vyas Sekar. 2016. Gremlin: Systematic Resilience Testing of Microservices. In 2016 IEEE 36th International Conference on Distributed Computing Systems (ICDCS). 57--66. https:\/\/doi.org\/10.1109\/ICDCS.2016.11"},{"key":"e_1_3_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2524211.2524217"},{"key":"e_1_3_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/2048066.2048082"},{"key":"e_1_3_2_2_36_1","volume-title":"Programmable Fault Injection Testbeds for Complex SOA","author":"Juszczyk Lukasz","unstructured":"Lukasz Juszczyk and Schahram Dustdar . 2010. Programmable Fault Injection Testbeds for Complex SOA . In Service-Oriented Computing, Paul P. Maglio, Mathias Weske, Jian Yang, and Marcelo Fantinato (Eds.). Springer Berlin Heidelberg , Berlin, Heidelberg , 411--425. Lukasz Juszczyk and Schahram Dustdar. 2010. Programmable Fault Injection Testbeds for Complex SOA. In Service-Oriented Computing, Paul P. Maglio, Mathias Weske, Jian Yang, and Marcelo Fantinato (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 411--425."},{"key":"e_1_3_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/12.364536"},{"key":"e_1_3_2_2_39_1","volume-title":"Proceedings of the 11th USENIX Conference on Operating Systems Design and Implementation","author":"Leesatapornwongsa Tanakorn","unstructured":"Tanakorn Leesatapornwongsa , Mingzhe Hao , Pallavi Joshi , Jeffrey F. Lukman , and Haryadi S. Gunawi . 2014. SAMC: Semantic-Aware Model Checking for Fast Discovery of Deep Bugs in Cloud Systems . In Proceedings of the 11th USENIX Conference on Operating Systems Design and Implementation ( Broomfield, CO) (OSDI'14). USENIX Association, USA, 399--414. Tanakorn Leesatapornwongsa, Mingzhe Hao, Pallavi Joshi, Jeffrey F. Lukman, and Haryadi S. Gunawi. 2014. SAMC: Semantic-Aware Model Checking for Fast Discovery of Deep Bugs in Cloud Systems. In Proceedings of the 11th USENIX Conference on Operating Systems Design and Implementation (Broomfield, CO) (OSDI'14). USENIX Association, USA, 399--414."},{"key":"e_1_3_2_2_40_1","volume-title":"Proceedings of the Fourteenth EuroSys Conference 2019","author":"Lukman Jeffrey F.","unstructured":"Jeffrey F. Lukman , Huan Ke , Cesar A. Stuardo , Riza O. Suminto , Daniar H. Kurniawan , Dikaimin Simon , Satria Priambada , Chen Tian , Feng Ye , Tanakorn Leesatapornwongsa , Aarti Gupta , Shan Lu , and Haryadi S. Gunawi . 2019. FlyMC: Highly Scalable Testing of Complex Interleavings in Distributed Systems . In Proceedings of the Fourteenth EuroSys Conference 2019 ( Dresden, Germany) (EuroSys '19). Association for Computing Machinery, New York, NY, USA, Article 20, 16 pages. https:\/\/doi.org\/10.1145\/3302424.3303986 10.1145\/3302424.3303986 Jeffrey F. Lukman, Huan Ke, Cesar A. Stuardo, Riza O. Suminto, Daniar H. Kurniawan, Dikaimin Simon, Satria Priambada, Chen Tian, Feng Ye, Tanakorn Leesatapornwongsa, Aarti Gupta, Shan Lu, and Haryadi S. Gunawi. 2019. FlyMC: Highly Scalable Testing of Complex Interleavings in Distributed Systems. In Proceedings of the Fourteenth EuroSys Conference 2019 (Dresden, Germany) (EuroSys '19). Association for Computing Machinery, New York, NY, USA, Article 20, 16 pages. https:\/\/doi.org\/10.1145\/3302424.3303986"},{"key":"e_1_3_2_2_41_1","volume-title":"2009 IEEE\/IFIP International Conference on Dependable Systems Networks. 379--388","author":"Paul","year":"2009","unstructured":"Paul D. Marinescu and George Candea. 2009. LFI: A practical and general library-level fault injector . In 2009 IEEE\/IFIP International Conference on Dependable Systems Networks. 379--388 . https:\/\/doi.org\/10.1109\/DSN. 2009 .5270313 10.1109\/DSN.2009.5270313 Paul D. Marinescu and George Candea. 2009. LFI: A practical and general library-level fault injector. In 2009 IEEE\/IFIP International Conference on Dependable Systems Networks. 379--388. https:\/\/doi.org\/10.1109\/DSN.2009.5270313"},{"key":"e_1_3_2_2_42_1","unstructured":"Friedemann Mattern. 1988. Virtual Time and Global States of Distributed Systems. In PARALLEL AND DISTRIBUTED ALGORITHMS. North-Holland 215--226.  Friedemann Mattern. 1988. Virtual Time and Global States of Distributed Systems. In PARALLEL AND DISTRIBUTED ALGORITHMS. North-Holland 215--226."},{"key":"e_1_3_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/361598.361623"},{"key":"e_1_3_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/MCC.2018.032591616"},{"key":"e_1_3_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2015.59"},{"key":"e_1_3_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/1379022.1375611"},{"key":"e_1_3_2_2_47_1","volume-title":"MODIST: Transparent Model Checking of Unmodified Distributed Systems. In 6th USENIX Symposium on Networked Systems Design and Implementation (NSDI 09)","author":"Yang Junfeng","year":"2009","unstructured":"Junfeng Yang , Tisheng Chen , Ming Wu , Zhilei Xu , Xuezheng Liu , Haoxiang Lin , Mao Yang , Fan Long , Lintao Zhang , and Lidong Zhou . 2009 . MODIST: Transparent Model Checking of Unmodified Distributed Systems. In 6th USENIX Symposium on Networked Systems Design and Implementation (NSDI 09) . USENIX Association, Boston, MA. https:\/\/www.usenix.org\/conference\/nsdi-09\/rnodist-transparent-model-checking-unmodified-distributed-systems Junfeng Yang, Tisheng Chen, Ming Wu, Zhilei Xu, Xuezheng Liu, Haoxiang Lin, Mao Yang, Fan Long, Lintao Zhang, and Lidong Zhou. 2009. MODIST: Transparent Model Checking of Unmodified Distributed Systems. In 6th USENIX Symposium on Networked Systems Design and Implementation (NSDI 09). USENIX Association, Boston, MA. https:\/\/www.usenix.org\/conference\/nsdi-09\/rnodist-transparent-model-checking-unmodified-distributed-systems"},{"key":"e_1_3_2_2_48_1","volume-title":"Xu Zhao, Yongle Zhang, Pranay U. Jain, and Michael Stumm.","author":"Yuan Ding","year":"2014","unstructured":"Ding Yuan , Yu Luo , Xin Zhuang , Guilherme Renna Rodrigues , Xu Zhao, Yongle Zhang, Pranay U. Jain, and Michael Stumm. 2014 . Simple Testing Can Prevent Most Critical Failures: An Analysis of Production Failures in Distributed Data-Intensive Systems. In 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI 14). USENIX Association , Broomfield, CO, 249--265. https:\/\/www.usenix.org\/conference\/osdi14\/technical-sessions\/presentation\/yuan Ding Yuan, Yu Luo, Xin Zhuang, Guilherme Renna Rodrigues, Xu Zhao, Yongle Zhang, Pranay U. Jain, and Michael Stumm. 2014. Simple Testing Can Prevent Most Critical Failures: An Analysis of Production Failures in Distributed Data-Intensive Systems. In 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI 14). USENIX Association, Broomfield, CO, 249--265. https:\/\/www.usenix.org\/conference\/osdi14\/technical-sessions\/presentation\/yuan"},{"key":"e_1_3_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2019.2954871"},{"key":"e_1_3_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2018.2887384"},{"key":"e_1_3_2_2_51_1","volume-title":"Predicting Defects for Eclipse. In Third International Workshop on Predictor Models in Software Engineering (PROMISE'07: ICSE Workshops 2007","author":"Zimmermann Thomas","year":"2007","unstructured":"Thomas Zimmermann , Rahul Premraj , and Andreas Zeller . 2007 . Predicting Defects for Eclipse. In Third International Workshop on Predictor Models in Software Engineering (PROMISE'07: ICSE Workshops 2007 ). 9--9. https:\/\/doi.org\/10.1109\/PROMISE.2007.10 10.1109\/PROMISE.2007.10 Thomas Zimmermann, Rahul Premraj, and Andreas Zeller. 2007. Predicting Defects for Eclipse. In Third International Workshop on Predictor Models in Software Engineering (PROMISE'07: ICSE Workshops 2007). 9--9. https:\/\/doi.org\/10.1109\/PROMISE.2007.10"}],"event":{"name":"SoCC '21: ACM Symposium on Cloud Computing","location":"Seattle WA USA","acronym":"SoCC '21","sponsor":["SIGMOD ACM Special Interest Group on Management of Data","SIGOPS ACM Special Interest Group on Operating Systems"]},"container-title":["Proceedings of the ACM Symposium on Cloud Computing"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3472883.3487005","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3472883.3487005","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:11:57Z","timestamp":1750191117000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3472883.3487005"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,11]]},"references-count":50,"alternative-id":["10.1145\/3472883.3487005","10.1145\/3472883"],"URL":"https:\/\/doi.org\/10.1145\/3472883.3487005","relation":{},"subject":[],"published":{"date-parts":[[2021,11]]},"assertion":[{"value":"2021-11-01","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}