{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,18]],"date-time":"2026-02-18T22:58:06Z","timestamp":1771455486448,"version":"3.50.1"},"reference-count":66,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2024,12,27]],"date-time":"2024-12-27T00:00:00Z","timestamp":1735257600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100000038","name":"Natural Sciences and Engineering Research Council of Canada","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100000038","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Fonds de recherche du Qu\u00e9bec\u2014Nature et technologies (FRQNT)."}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2025,1,31]]},"abstract":"<jats:p>Unit tests must be readable to help developers understand and evolve production code. Most existing test quality metrics assess test code\u2019s ability to detect bugs. Few metrics focus on test code\u2019s readability. One standard approach to improve readability is the consistent application of conventions. We investigated test convention consistency as a dimension of test quality. We formalized test suite consistency as the extent to which alternatives are used within a code base and introduce two complementary metrics to capture this extent. We elaborated a catalog of over 30 test conventions for the Java language organized in 10 convention classes that group mutual alternatives. We developed tool support to detect occurrences of conventions, compute consistency metrics over a test suite, and view occurrences of conventions in the corresponding code. We applied our tools to study the consistency of the test suites of 20 large open source Java projects. The study validates the design of the test convention classes, provides descriptive statistics on the range of consistency values for 10 different convention classes, and enables us to link observed changes in consistency values to specific events in the change history of our target systems, thus providing evidence of the construct validity of the metrics. We conclude that analyzing test suite consistency via static analysis shows promise as a practical approach to help improve test suite quality.<\/jats:p>","DOI":"10.1145\/3672448","type":"journal-article","created":{"date-parts":[[2024,10,21]],"date-time":"2024-10-21T13:23:29Z","timestamp":1729517009000},"page":"1-39","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Understanding Test Convention Consistency as a Dimension of Test Quality"],"prefix":"10.1145","volume":"34","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0248-1384","authenticated-orcid":false,"given":"Martin P.","family":"Robillard","sequence":"first","affiliation":[{"name":"McGill University, Montr\u00e9al, QC, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0211-7256","authenticated-orcid":false,"given":"Mathieu","family":"Nassif","sequence":"additional","affiliation":[{"name":"McGill University, Montr\u00e9al, QC, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-9017-6197","authenticated-orcid":false,"given":"Muhammad","family":"Sohail","sequence":"additional","affiliation":[{"name":"McGill University, Montr\u00e9al, QC, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,12,27]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/2635868.2635883"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-014-9313-0"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER.2016.105"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSM.2008.4658076"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSR.2009.5069479"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1109\/WETSoM.2017.2"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/2786805.2786838"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.2014.11"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1109\/C-M.1978.218136"},{"key":"e_1_3_2_11_2","first-page":"92","volume-title":"Proceedings of the 2nd International Conference on Extreme Programming and Flexible Processes in Software Engineering","author":"Deursen Arie Van","year":"2001","unstructured":"Arie Van Deursen, Leon Moonen, Alex Bergh, and Gerard Kok. 2001. Refactoring test code. In Proceedings of the 2nd International Conference on Extreme Programming and Flexible Processes in Software Engineering, 92\u201395."},{"key":"e_1_3_2_12_2","unstructured":"Ramesh Fadatare. 2018. JUnit Framework Best Practices. Personal Blog. Retrieved from https:\/\/www.javaguides.net\/2018\/08\/junit-framework-best-practices.html"},{"key":"e_1_3_2_13_2","unstructured":"Viktor Farcic. 2013. Test Driven Development (TDD): Best Practices Using Java Examples. Personal Blog. Retrieved from https:\/\/technologyconversations.com\/2013\/12\/24\/test-driven-development-tdd-best-practices-using-java-examples-2\/"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3379597.3387500"},{"key":"e_1_3_2_15_2","unstructured":"GitHub. 2024. About READMEs. GitHub Documentation Page. Retrieved from https:\/\/docs.github.com\/en\/repositories\/managing-your-repositorys-settings-and-features\/customizing-your-repository\/about-readmes"},{"key":"e_1_3_2_16_2","unstructured":"GitHub. 2024. Setting Guidelines for Repository Contributors. GitHub Documentation Page. Retrieved from https:\/\/docs.github.com\/en\/communities\/setting-up-your-project-for-healthy-contributions\/setting-guidelines-for-repository-contributors"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME46990.2020.00040"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3196321.3196363"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2013.45"},{"key":"e_1_3_2_20_2","unstructured":"Lokesh Gupta. 2020. Unit Testing Best Practices. HowToDoInJava Article. Retrieved from https:\/\/howtodoinjava.com\/best-practices\/unit-testing-best-practices-junit-reference-guide\/"},{"key":"e_1_3_2_21_2","unstructured":"Philipp Hauer. 2022. Modern Best Practices for Testing in Java. Personal Blog. Retrieved from https:\/\/phauer.com\/2019\/modern-best-practices-testing-java\/"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2018.03.010"},{"key":"e_1_3_2_23_2","unstructured":"JavaParser. JavaParser Home Page. Retrieved from https:\/\/javaparser.org\/"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2013.6606613"},{"key":"e_1_3_2_25_2","unstructured":"Petri Kainulainen. 2018. Writing Clean Tests: Naming Matters. Personal Blog. Retrieved from https:\/\/www.petrikainulainen.net\/programming\/testing\/writing-clean-tests-naming-matters\/"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-015-9393-5"},{"key":"e_1_3_2_27_2","unstructured":"Vladimir Khorikov. 2019. You Are Naming Your Tests Wrong! Personal Blog. Retrieved from https:\/\/enterprisecraftsmanship.com\/posts\/you-naming-tests-wrong\/"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEIP.2019.00015"},{"key":"e_1_3_2_29_2","volume-title":"Effective Unit Testing: A Guide for Java Developers","author":"Koskela Lasse","year":"2013","unstructured":"Lasse Koskela. 2013. Effective Unit Testing: A Guide for Java Developers. Manning Publications."},{"key":"e_1_3_2_30_2","unstructured":"Ajitesh Kumar. 2021. 7 Popular Strategies: Unit Test Naming Conventions. DZone Article. Retrieved from https:\/\/dzone.com\/articles\/7-popular-unit-test-naming"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2016.30"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSTW.2009.30"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2018.2884955"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3196321.3196342"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICPC.2019.00040"},{"key":"e_1_3_2_36_2","volume-title":"Clean Code: A Handbook of Agile Software Craftmanship","author":"Martin Robert C.","year":"2009","unstructured":"Robert C. Martin. 2009. Clean Code: A Handbook of Agile Software Craftmanship. Prentice Hall."},{"key":"e_1_3_2_37_2","volume-title":"Xunit Test Patterns: Refactoring Test Code","author":"Meszaros Gerard","year":"2007","unstructured":"Gerard Meszaros. 2007. Xunit Test Patterns: Refactoring Test Code. Addison-Wesley."},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/366246.366248"},{"key":"e_1_3_2_39_2","unstructured":"Nam Ha Minh. 2019. JUnit Tutorial for Beginner with Eclipse. Personal Blog. Retrieved from https:\/\/www.codejava.net\/testing\/junit-tutorial-for-beginner-with-eclipse"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/567793.567795"},{"key":"e_1_3_2_41_2","unstructured":"Oracle. 1999. Code Conventions for the Java Programming Language. Retrieved from https:\/\/www.oracle.com\/java\/technologies\/javase\/codeconventions-contents.html"},{"key":"e_1_3_2_42_2","volume-title":"The Art of Unit Testing","author":"Osherove Roy","year":"2013","unstructured":"Roy Osherove. 2013. The Art of Unit Testing (2nd ed.). Manning Publications.","edition":"2"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME.2018.00040"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3417921"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICPC52881.2021.00039"},{"key":"e_1_3_2_46_2","unstructured":"John Reese. 2022. Unit Testing Best Practices with. NET Core and. NET Standard. Microsoft Documentation Article. Retrieved from https:\/\/docs.microsoft.com\/en-us\/dotnet\/core\/testing\/unit-testing-best-practices"},{"key":"e_1_3_2_47_2","unstructured":"Jon Reid. 2020. Unit Test Naming: The 3 Most Important Parts. Personal Blog. Retrieved from https:\/\/qualitycoding.org\/unit-test-naming\/"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1145\/3649598"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/941566.941569"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","unstructured":"Martin P. Robillard Mathieu Nassif and Muhammad Sohail. 2024. Data from: Understanding test convention consistency as a dimension of test quality. Zenodo. DOI: 10.5281\/zenodo.11267987","DOI":"10.5281\/zenodo.11267987"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1145\/3188720"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1002\/j.1538-7305.1948.tb01338.x"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSM.2011.6080819"},{"key":"e_1_3_2_54_2","unstructured":"Steve Ardalis Smith. 2011. Unit Test Naming Convention. Personal Blog. Retrieved from https:\/\/ardalis.com\/unit-test-naming-convention\/"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME.2018.00010"},{"key":"e_1_3_2_56_2","unstructured":"Stack Exchange Software Engineering Community. 2010. Is It OK to Have Multiple Asserts in a Single Unit Test? Stack Exchange Thread. Retrieved from https:\/\/softwareengineering.stackexchange.com\/questions\/7823"},{"key":"e_1_3_2_57_2","unstructured":"Oleksandr Stefanovskyi. 2019. Unit Test Naming Conventions. Medium Article. Retrieved from https:\/\/medium.com\/@stefanovskyi\/unit-test-naming-conventions-dd9208eadbea"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1109\/IWSM-MENSURA.2011.34"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/3195538.3195541"},{"key":"e_1_3_2_60_2","unstructured":"Stack Overflow User. 2008. Unit Test Naming Best Practices [Closed]. Stack Overflow Thread. Retrieved from https:\/\/stackoverflow.com\/questions\/155436"},{"key":"e_1_3_2_61_2","unstructured":"Stack Overflow User. 2008. What Are Some Popular Naming Conventions for Unit Tests? [Closed]. Stack Overflow Thread. Retrieved from https:\/\/stackoverflow.com\/questions\/96297"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2007.70745"},{"key":"e_1_3_2_63_2","unstructured":"Christian Vasquez. 2018. Introduction to Unit Testing with Java. DEV Community Article. Retrieved from https:\/\/dev.to\/chrisvasqm\/introduction-to-unit-testing-with-java-2544"},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER.2018.8330195"},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2008.47"},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER.2017.7884645"},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-019-09720-x"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3672448","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3672448","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:58:01Z","timestamp":1750294681000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3672448"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,27]]},"references-count":66,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2025,1,31]]}},"alternative-id":["10.1145\/3672448"],"URL":"https:\/\/doi.org\/10.1145\/3672448","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,12,27]]},"assertion":[{"value":"2023-01-11","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-05-22","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-12-27","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}