{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,1]],"date-time":"2026-04-01T17:19:12Z","timestamp":1775063952333,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":41,"publisher":"ACM","license":[{"start":{"date-parts":[[2019,7,10]],"date-time":"2019-07-10T00:00:00Z","timestamp":1562716800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2019,7,10]]},"DOI":"10.1145\/3293882.3330570","type":"proceedings-article","created":{"date-parts":[[2019,7,11]],"date-time":"2019-07-11T12:13:31Z","timestamp":1562847211000},"page":"101-111","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":117,"title":["Root causing flaky tests in a large-scale industrial setting"],"prefix":"10.1145","author":[{"given":"Wing","family":"Lam","sequence":"first","affiliation":[{"name":"University of Illinois at Urbana-Champaign, USA"}]},{"given":"Patrice","family":"Godefroid","sequence":"additional","affiliation":[{"name":"Microsoft, USA"}]},{"given":"Suman","family":"Nath","sequence":"additional","affiliation":[{"name":"Microsoft, USA"}]},{"given":"Anirudh","family":"Santhiar","sequence":"additional","affiliation":[{"name":"Microsoft, USA"}]},{"given":"Suresh","family":"Thummalapenta","sequence":"additional","affiliation":[{"name":"Microsoft, USA"}]}],"member":"320","published-online":{"date-parts":[[2019,7,10]]},"reference":[{"key":"e_1_3_2_1_1_1","volume-title":"Event Tracing for Windows (ETW) simplified. https:\/\/bit.ly\/2NgEBzl","year":"2019","unstructured":"2011. Event Tracing for Windows (ETW) simplified. https:\/\/bit.ly\/2NgEBzl . 2019 . Bazel. https:\/\/bazel.build. 2019. Buck. https:\/\/buckbuild.com. 2019. Managed vs. Unmanaged development. https:\/\/bit.ly\/2Or4C3p. 2019. MemorySanitizer. https:\/\/clang.llvm.org\/docs\/MemorySanitizer.html. 2019. MsTest framework. https:\/\/bit.ly\/2NeZmLO. 2019. NUnit framework. https:\/\/nunit.org. 2019. RootFinder . https:\/\/sites.google.com\/view\/root-causing-flaky-tests\/home. 2019. Valgrind. http:\/\/valgrind.org. 2019. xUnit framework. https:\/\/xunit.github.io. 2011. Event Tracing for Windows (ETW) simplified. https:\/\/bit.ly\/2NgEBzl. 2019. Bazel. https:\/\/bazel.build. 2019. Buck. https:\/\/buckbuild.com. 2019. Managed vs. Unmanaged development. https:\/\/bit.ly\/2Or4C3p. 2019. MemorySanitizer. https:\/\/clang.llvm.org\/docs\/MemorySanitizer.html. 2019. MsTest framework. https:\/\/bit.ly\/2NeZmLO. 2019. NUnit framework. https:\/\/nunit.org. 2019. RootFinder. https:\/\/sites.google.com\/view\/root-causing-flaky-tests\/home. 2019. Valgrind. http:\/\/valgrind.org. 2019. xUnit framework. https:\/\/xunit.github.io."},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2009.25"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2009.06.035"},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180164"},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/1646353.1646374"},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2970276.2970347"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.scico.2007.01.015"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.scico.2007.12.001"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1542476.1542490"},{"key":"e_1_3_2_1_10_1","unstructured":"M. Fowler. 2011. Eradicating non-determinism in tests. https:\/\/bit.ly\/2PFHI5B.  M. Fowler. 2011. Eradicating non-determinism in tests. https:\/\/bit.ly\/2PFHI5B."},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"crossref","unstructured":"Z. Gao Y. Liang M. Cohen A. Memon and Z. Wang. 2015. Making system user interactive tests repeatable: When and what should we control?. In ICSE. Florence Italy 55\u201365.   Z. Gao Y. Liang M. Cohen A. Memon and Z. Wang. 2015. Making system user interactive tests repeatable: When and what should we control?. In ICSE. Florence Italy 55\u201365.","DOI":"10.1109\/ICSE.2015.28"},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/263699.263717"},{"key":"e_1_3_2_1_13_1","volume-title":"International SPIN Workshop on Model Checking of Software. Springer, 121\u2013136","author":"Groce A.","unstructured":"A. Groce and W. Visser . 2003. What went wrong: Explaining counterexamples . In International SPIN Workshop on Model Checking of Software. Springer, 121\u2013136 . A. Groce and W. Visser. 2003. What went wrong: Explaining counterexamples. In International SPIN Workshop on Model Checking of Software. Springer, 121\u2013136."},{"key":"e_1_3_2_1_14_1","unstructured":"J. Ha J. Yi P. Dinges J. Manson C. Sadowski and N. Meng. 2013. System to uncover root cause of non-deterministic (flaky) tests. In Google Patent. https: \/\/patents.google.com\/patent\/US9311220  J. Ha J. Yi P. Dinges J. Manson C. Sadowski and N. Meng. 2013. System to uncover root cause of non-deterministic (flaky) tests. In Google Patent. https: \/\/patents.google.com\/patent\/US9311220"},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"crossref","unstructured":"M. Harman and P. O\u2019Hearn. 2018. From start-ups to scale-ups: Opportunities and open problems for static and dynamic program analysis. In SCAM keynote. https:\/\/bit.ly\/2KzBnKQ  M. Harman and P. O\u2019Hearn. 2018. From start-ups to scale-ups: Opportunities and open problems for static and dynamic program analysis. In SCAM keynote. https:\/\/bit.ly\/2KzBnKQ","DOI":"10.1109\/SCAM.2018.00009"},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"crossref","unstructured":"K. Herzig and N. Nagappan. 2015. Empirically detecting false test alarms using association rules. In ICSE. Florence Italy 39\u201348.   K. Herzig and N. Nagappan. 2015. Empirically detecting false test alarms using association rules. In ICSE. Florence Italy 39\u201348.","DOI":"10.1109\/ICSE.2015.133"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2970276.2970358"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2934872.2934882"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"crossref","unstructured":"B. Johnson Y. Song E. Murphy-Hill and R. Bowdidge. 2013. Why don\u2019t software developers use static analysis tools to find bugs?. In ICSE. San Francisco CA USA 672\u2013681.   B. Johnson Y. Song E. Murphy-Hill and R. Bowdidge. 2013. Why don\u2019t software developers use static analysis tools to find bugs?. In ICSE. San Francisco CA USA 672\u2013681.","DOI":"10.1109\/ICSE.2013.6606613"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/1101908.1101949"},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3106237.3106288"},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"crossref","unstructured":"W. Lam R. Oei A. Shi D. Marinov and T. Xie. 2019. iDFlakies: A framework for detecting and partially classifying flaky tests. In ICST. Xi\u2019an China 312\u2013322.  W. Lam R. Oei A. Shi D. Marinov and T. Xie. 2019. iDFlakies: A framework for detecting and partially classifying flaky tests. In ICST. Xi\u2019an China 312\u2013322.","DOI":"10.1109\/ICST.2019.00038"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/1064978.1065014"},{"key":"e_1_3_2_1_24_1","unstructured":"L. Luo S. Nath L. Sivalingam M. Musuvathi and L. Ceze. 2018. Troubleshooting transiently-recurring errors in production systems with blame-proportional logging. In USENIX ATC.   L. Luo S. Nath L. Sivalingam M. Musuvathi and L. Ceze. 2018. Troubleshooting transiently-recurring errors in production systems with blame-proportional logging. In USENIX ATC."},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2635868.2635920"},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEIP.2017.16"},{"key":"e_1_3_2_1_27_1","unstructured":"J. Micco. 2016. Flaky tests at Google and how we mitigate them. https:\/\/testing. googleblog.com\/2016\/05\/flaky-tests-at-google-and-how-we.html  J. Micco. 2016. Flaky tests at Google and how we mitigate them. https:\/\/testing. googleblog.com\/2016\/05\/flaky-tests-at-google-and-how-we.html"},{"key":"e_1_3_2_1_28_1","unstructured":"J. Micco. 2017. The state of continuous integration testing at Google. In ICST Keynote. https:\/\/bit.ly\/2OohAip  J. Micco. 2017. The state of continuous integration testing at Google. In ICST Keynote. https:\/\/bit.ly\/2OohAip"},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"crossref","unstructured":"K. Mu\u015flu B. Soran and J. Wuttke. 2011. Finding bugs by isolating unit tests. In ESEC\/FSE. Szeged Hungary 496\u2013499.  K. Mu\u015flu B. Soran and J. Wuttke. 2011. Finding bugs by isolating unit tests. In ESEC\/FSE. Szeged Hungary 496\u2013499.","DOI":"10.1145\/2025113.2025202"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/1133981.1134018"},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2009.5070538"},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2000791.2000795"},{"key":"e_1_3_2_1_33_1","doi-asserted-by":"crossref","unstructured":"F. Palomba and A. Zaidman. 2017. Does refactoring of test smells induce fixing flaky tests?. In ICSME. Shanghai China 1\u201312.  F. Palomba and A. Zaidman. 2017. Does refactoring of test smells induce fixing flaky tests?. In ICSME. Shanghai China 1\u201312.","DOI":"10.1109\/ICSME.2017.12"},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3275529"},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3062341.3062361"},{"key":"e_1_3_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2889160.2889223"},{"key":"e_1_3_2_1_37_1","doi-asserted-by":"crossref","unstructured":"A. Shi A. Gyori O. Legunsen and D. Marinov. 2016. Detecting assumptions on deterministic implementations of non-deterministic specifications. In ICST. Chicago IL USA 80\u201390.  A. Shi A. Gyori O. Legunsen and D. Marinov. 2016. Detecting assumptions on deterministic implementations of non-deterministic specifications. In ICST. Chicago IL USA 80\u201390.","DOI":"10.1109\/ICST.2016.40"},{"key":"e_1_3_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2017.69"},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2001420.2001458"},{"key":"e_1_3_2_1_40_1","doi-asserted-by":"crossref","unstructured":"S. Thorve C. Sreshtha and N. Meng. 2018. An empirical study of flaky tests in Android apps. In ICSME NIER Track. Madrid Spain 534\u2013538.  S. Thorve C. Sreshtha and N. Meng. 2018. An empirical study of flaky tests in Android apps. In ICSME NIER Track. Madrid Spain 534\u2013538.","DOI":"10.1109\/ICSME.2018.00062"},{"key":"e_1_3_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/TR.2013.2285319"}],"event":{"name":"ISSTA '19: 28th ACM SIGSOFT International Symposium on Software Testing and Analysis","location":"Beijing China","acronym":"ISSTA '19","sponsor":["SIGSOFT ACM Special Interest Group on Software Engineering"]},"container-title":["Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3293882.3330570","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3293882.3330570","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T00:43:50Z","timestamp":1750207430000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3293882.3330570"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,7,10]]},"references-count":41,"alternative-id":["10.1145\/3293882.3330570","10.1145\/3293882"],"URL":"https:\/\/doi.org\/10.1145\/3293882.3330570","relation":{},"subject":[],"published":{"date-parts":[[2019,7,10]]},"assertion":[{"value":"2019-07-10","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}