{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,4]],"date-time":"2026-11-04T18:56:31Z","timestamp":1793818591392,"version":"4.6.1"},"publisher-location":"New York, NY, USA","reference-count":29,"publisher":"ACM","license":[{"start":{"date-parts":[[2023,7,12]],"date-time":"2023-07-12T00:00:00Z","timestamp":1689120000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,7,12]]},"DOI":"10.1145\/3597926.3598083","type":"proceedings-article","created":{"date-parts":[[2023,7,13]],"date-time":"2023-07-13T20:12:53Z","timestamp":1689279173000},"page":"627-638","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":14,"title":["Systematically Producing Test Orders to Detect Order-Dependent Flaky Tests"],"prefix":"10.1145","author":[{"given":"Chengpeng","family":"Li","sequence":"first","affiliation":[{"name":"University of Texas at Austin, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"M. Mahdi","family":"Khosravi","sequence":"additional","affiliation":[{"name":"Middle East Technical University, Turkey"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wing","family":"Lam","sequence":"additional","affiliation":[{"name":"George Mason University, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"August","family":"Shi","sequence":"additional","affiliation":[{"name":"University of Texas at Austin, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,7,13]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"2019. iDFlakies. https:\/\/github.com\/idflakies\/iDFlakies \t\t\t\t  2019. iDFlakies. https:\/\/github.com\/idflakies\/iDFlakies"},{"key":"e_1_3_2_1_2_1","unstructured":"2023. Systematically Producing Test-Orders to Detect Order-Dependent Flaky Tests Dataset. https:\/\/sites.google.com\/view\/systematically-detecting-od \t\t\t\t  2023. Systematically Producing Test-Orders to Detect Order-Dependent Flaky Tests Dataset. https:\/\/sites.google.com\/view\/systematically-detecting-od"},{"key":"e_1_3_2_1_3_1","volume-title":"International Conference on Software Engineering.","author":"Bell Jonathan","year":"2014","unstructured":"Jonathan Bell and Gail Kaiser . 2014 . Unit test virtualization with VMVM . In International Conference on Software Engineering. Jonathan Bell and Gail Kaiser. 2014. Unit test virtualization with VMVM. In International Conference on Software Engineering."},{"key":"e_1_3_2_1_4_1","volume-title":"European Software Engineering Conference and Symposium on the Foundations of Software Engineering.","author":"Bell Jonathan","year":"2015","unstructured":"Jonathan Bell , Gail Kaiser , Eric Melski , and Mohan Dattatreya . 2015 . Efficient dependency detection for safe Java test acceleration . In European Software Engineering Conference and Symposium on the Foundations of Software Engineering. Jonathan Bell, Gail Kaiser, Eric Melski, and Mohan Dattatreya. 2015. Efficient dependency detection for safe Java test acceleration. In European Software Engineering Conference and Symposium on the Foundations of Software Engineering."},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338945"},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2018.00011"},{"key":"e_1_3_2_1_7_1","volume-title":"Golomb and Herbert Taylor","author":"Solomon","year":"1985","unstructured":"Solomon W. Golomb and Herbert Taylor . 1985 . Tuscan squares \u2013 A new family of combinatorial designs. Ars Combinatoria, 20, B ( 1985). Solomon W. Golomb and Herbert Taylor. 1985. Tuscan squares \u2013 A new family of combinatorial designs. Ars Combinatoria, 20, B (1985)."},{"key":"e_1_3_2_1_8_1","volume-title":"International Conference on Software Testing, Verification, and Validation.","author":"Gruber Martin","year":"2021","unstructured":"Martin Gruber , Stephan Lukasczyk , Florian Kroi\u00df , and Gordon Fraser . 2021 . An empirical study of flaky tests in Python . In International Conference on Software Testing, Verification, and Validation. Martin Gruber, Stephan Lukasczyk, Florian Kroi\u00df, and Gordon Fraser. 2021. An empirical study of flaky tests in Python. In International Conference on Software Testing, Verification, and Validation."},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2771783.2771793"},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/SCAM.2018.00009"},{"key":"e_1_3_2_1_11_1","volume-title":"International Symposium on Software Testing and Analysis.","author":"Lam Wing","year":"2019","unstructured":"Wing Lam , Patrice Godefroid , Suman Nath , Anirudh Santhiar , and Suresh Thummalapenta . 2019 . Root causing flaky tests in a large-scale industrial setting . In International Symposium on Software Testing and Analysis. Wing Lam, Patrice Godefroid, Suman Nath, Anirudh Santhiar, and Suresh Thummalapenta. 2019. Root causing flaky tests in a large-scale industrial setting. In International Symposium on Software Testing and Analysis."},{"key":"e_1_3_2_1_12_1","volume-title":"International Conference on Software Engineering.","author":"Lam Wing","year":"2020","unstructured":"Wing Lam , Kivan\u00e7 Mu\u015flu , Hitesh Sajnani , and Suresh Thummalapenta . 2020 . A study on the lifecycle of flaky tests . In International Conference on Software Engineering. Wing Lam, Kivan\u00e7 Mu\u015flu, Hitesh Sajnani, and Suresh Thummalapenta. 2020. A study on the lifecycle of flaky tests. In International Conference on Software Engineering."},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2019.00038"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3395363.3397364"},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE5003.2020.00045"},{"key":"e_1_3_2_1_16_1","volume-title":"European Software Engineering Conference and Symposium on the Foundations of Software Engineering.","author":"Lampel Johannes","year":"2021","unstructured":"Johannes Lampel , Sascha Just , Sven Apel , and Andreas Zeller . 2021 . When life gives you oranges: Detecting and diagnosing intermittent job failures at Mozilla . In European Software Engineering Conference and Symposium on the Foundations of Software Engineering. Johannes Lampel, Sascha Just, Sven Apel, and Andreas Zeller. 2021. When life gives you oranges: Detecting and diagnosing intermittent job failures at Mozilla. In European Software Engineering Conference and Symposium on the Foundations of Software Engineering."},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3533767.3534404"},{"key":"e_1_3_2_1_18_1","volume-title":"International Conference on Software Engineering.","author":"Li Chengpeng","year":"2022","unstructured":"Chengpeng Li , Chenguang Zhu , Wenxi Wang , and August Shi . 2022 . Repairing order-dependent flaky tests via test generation . In International Conference on Software Engineering. Chengpeng Li, Chenguang Zhu, Wenxi Wang, and August Shi. 2022. Repairing order-dependent flaky tests via test generation. In International Conference on Software Engineering."},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2635868.2635920"},{"key":"e_1_3_2_1_20_1","volume-title":"European Software Engineering Conference and Symposium on the Foundations of Software Engineering.","author":"Mu\u015flu Kivan\u00e7","year":"2011","unstructured":"Kivan\u00e7 Mu\u015flu , Bilge Soran , and Jochen Wuttke . 2011 . Finding bugs by isolating unit tests . In European Software Engineering Conference and Symposium on the Foundations of Software Engineering. Kivan\u00e7 Mu\u015flu, Bilge Soran, and Jochen Wuttke. 2011. Finding bugs by isolating unit tests. In European Software Engineering Conference and Symposium on the Foundations of Software Engineering."},{"key":"e_1_3_2_1_21_1","volume-title":"European Software Engineering Conference and Symposium on the Foundations of Software Engineering.","author":"Rahman Md Tajmilur","unstructured":"Md Tajmilur Rahman and Peter C. Rigby . 2018. The impact of failing, flaky, and high failure tests on the number of crash reports associated with Firefox builds . In European Software Engineering Conference and Symposium on the Foundations of Software Engineering. Md Tajmilur Rahman and Peter C. Rigby. 2018. The impact of failing, flaky, and high failure tests on the number of crash reports associated with Firefox builds. In European Software Engineering Conference and Symposium on the Foundations of Software Engineering."},{"key":"e_1_3_2_1_22_1","volume-title":"European Software Engineering Conference and Symposium on the Foundations of Software Engineering.","author":"Ur Rehman Maaz Hafeez","unstructured":"Maaz Hafeez Ur Rehman and Peter C. Rigby . 2021. Quantifying no-fault-found test failures to prioritize inspection of flaky tests at Ericsson . In European Software Engineering Conference and Symposium on the Foundations of Software Engineering. Maaz Hafeez Ur Rehman and Peter C. Rigby. 2021. Quantifying no-fault-found test failures to prioritize inspection of flaky tests at Ericsson. In European Software Engineering Conference and Symposium on the Foundations of Software Engineering."},{"key":"e_1_3_2_1_23_1","volume-title":"An Empirical Analysis of UI-Based Flaky Tests. In International Conference on Software Engineering.","author":"Romano Alan","year":"2021","unstructured":"Alan Romano , Zihe Song , Sampath Grandhi , Wei Yang , and Weihang Wang . 2021 . An Empirical Analysis of UI-Based Flaky Tests. In International Conference on Software Engineering. Alan Romano, Zihe Song, Sampath Grandhi, Wei Yang, and Weihang Wang. 2021. An Empirical Analysis of UI-Based Flaky Tests. In International Conference on Software Engineering."},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338925"},{"key":"e_1_3_2_1_25_1","volume-title":"International Conference on Software Maintenance and Evolution.","author":"Thorve Swapna","year":"2018","unstructured":"Swapna Thorve , Chandani Sreshtha , and Na Meng . 2018 . An empirical study of flaky tests in Android apps . In International Conference on Software Maintenance and Evolution. Swapna Thorve, Chandani Sreshtha, and Na Meng. 2018. An empirical study of flaky tests in Android apps. In International Conference on Software Maintenance and Evolution."},{"key":"e_1_3_2_1_26_1","volume-title":"International Conference on Software Engineering (Tool Demonstrations Track).","author":"Wang Ruixin","year":"2022","unstructured":"Ruixin Wang , Yang Chen , and Wing Lam . 2022 . iPFlakies: A framework for detecting and fixing Python order-dependent flaky tests . In International Conference on Software Engineering (Tool Demonstrations Track). Ruixin Wang, Yang Chen, and Wing Lam. 2022. iPFlakies: A framework for detecting and fixing Python order-dependent flaky tests. In International Conference on Software Engineering (Tool Demonstrations Track)."},{"key":"e_1_3_2_1_27_1","volume-title":"International Conference on Software Engineering.","author":"Wei Anjiang","year":"2022","unstructured":"Anjiang Wei , Pu Yi , Zhengxi Li , Tao Xie , Darko Marinov , and Wing Lam . 2022 . Preempting flaky tests via non-idempotent-outcome tests . In International Conference on Software Engineering. Anjiang Wei, Pu Yi, Zhengxi Li, Tao Xie, Darko Marinov, and Wing Lam. 2022. Preempting flaky tests via non-idempotent-outcome tests. In International Conference on Software Engineering."},{"key":"e_1_3_2_1_28_1","unstructured":"Anjiang Wei Pu Yi Tao Xie Darko Marinov and Wing Lam. 2021. Probabilistic and systematic coverage of consecutive test-method pairs for detecting order-dependent flaky tests. In Tools and Algorithms for the Construction and Analysis of Systems. \t\t\t\t  Anjiang Wei Pu Yi Tao Xie Darko Marinov and Wing Lam. 2021. Probabilistic and systematic coverage of consecutive test-method pairs for detecting order-dependent flaky tests. In Tools and Algorithms for the Construction and Analysis of Systems."},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/2610384.2610404"}],"event":{"name":"ISSTA '23: 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis","location":"Seattle WA USA","acronym":"ISSTA '23","sponsor":["SIGSOFT ACM Special Interest Group on Software Engineering","AITO"]},"container-title":["Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3597926.3598083","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3597926.3598083","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T17:48:42Z","timestamp":1750182522000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3597926.3598083"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,7,12]]},"references-count":29,"alternative-id":["10.1145\/3597926.3598083","10.1145\/3597926"],"URL":"https:\/\/doi.org\/10.1145\/3597926.3598083","relation":{},"subject":[],"published":{"date-parts":[[2023,7,12]]},"assertion":[{"value":"2023-07-13","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}