{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,23]],"date-time":"2026-01-23T07:26:18Z","timestamp":1769153178470,"version":"3.49.0"},"reference-count":106,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2021,2,20]],"date-time":"2021-02-20T00:00:00Z","timestamp":1613779200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,2,20]],"date-time":"2021-02-20T00:00:00Z","timestamp":1613779200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2021,3]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Testing represents a crucial activity to ensure software quality. Recent studies have shown that test-related factors (e.g., code coverage) can be reliable predictors of software code quality, as measured by post-release defects. While these studies provided initial compelling evidence on the relation between tests and post-release defects, they considered different test-related factors separately: as a consequence, there is still a lack of knowledge of whether these factors are still good predictors when considering all together. In this paper, we propose a comprehensive case study on how test-related factors relate to production code quality in <jats:sc>Apache<\/jats:sc> systems. We first investigated how the presence of tests relates to post-release defects; then, we analyzed the role played by the test-related factors previously shown as significantly related to post-release defects. The key findings of the study show that, when controlling for other metrics (e.g., size of the production class), test-related factors have a limited connection to post-release defects.<\/jats:p>","DOI":"10.1007\/s10664-020-09891-y","type":"journal-article","created":{"date-parts":[[2021,2,20]],"date-time":"2021-02-20T17:09:55Z","timestamp":1613840995000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":14,"title":["The Relation of Test-Related Factors to Software Quality: A Case Study on Apache Systems"],"prefix":"10.1007","volume":"26","author":[{"given":"Fabiano","family":"Pecorelli","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Fabio","family":"Palomba","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Andrea","family":"De Lucia","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2021,2,20]]},"reference":[{"key":"9891_CR1","doi-asserted-by":"crossref","unstructured":"Adamopoulos K, Harman M, Hierons R M (2004) How to overcome the equivalent mutant problem and achieve tailored selective mutation using co-evolution. In: Genetic and evolutionary computation conference. Springer, pp 1338\u20131349","DOI":"10.1007\/978-3-540-24855-2_155"},{"issue":"8","key":"9891_CR2","doi-asserted-by":"crossref","first-page":"608","DOI":"10.1109\/TSE.2006.83","volume":"32","author":"JH Andrews","year":"2006","unstructured":"Andrews J H, Briand L C, Labiche Y, Namin A S (2006) Using mutation analysis for assessing and comparing testing coverage criteria. IEEE Trans Softw Eng 32(8):608\u2013624","journal-title":"IEEE Trans Softw Eng"},{"issue":"10","key":"9891_CR3","doi-asserted-by":"crossref","first-page":"751","DOI":"10.1109\/32.544352","volume":"22","author":"VR Basili","year":"1996","unstructured":"Basili V R, Briand L C, Melo W L (1996) A validation of object-oriented design metrics as quality indicators. IEEE Trans Softw Eng 22(10):751\u2013761","journal-title":"IEEE Trans Softw Eng"},{"issue":"4","key":"9891_CR4","doi-asserted-by":"crossref","first-page":"1052","DOI":"10.1007\/s10664-014-9313-0","volume":"20","author":"G Bavota","year":"2015","unstructured":"Bavota G, Qusef A, Oliveto R, De Lucia A, Binkley D (2015) Are test smells really harmful? an empirical study. Empir Softw Eng 20(4):1052\u20131094","journal-title":"Empir Softw Eng"},{"key":"9891_CR5","doi-asserted-by":"crossref","unstructured":"Bowes D, Hall T, Harman M, Jia Y, Sarro F, Wu F (2016) Mutation-aware fault prediction. In: Proceedings of the 25th International Symposium on Software Testing and Analysis. ACM, pp 330\u2013341","DOI":"10.1145\/2931037.2931039"},{"issue":"5","key":"9891_CR6","doi-asserted-by":"crossref","first-page":"72","DOI":"10.1109\/MS.2017.3571562","volume":"34","author":"L Briand","year":"2017","unstructured":"Briand L, Bianculli D, Nejati S, Pastore F, Sabetzadeh M (2017) The case for context-driven software engineering research: Generalizability is overrated. IEEE Softw 34(5):72\u201375","journal-title":"IEEE Softw"},{"key":"9891_CR7","unstructured":"Brown W H, Malveau R C, McCormick H W, Mowbray T J (1998) Antipatterns: refactoring software, architectures, and projects in crisis. Wiley"},{"key":"9891_CR8","doi-asserted-by":"crossref","unstructured":"Budgen D, Brereton P (2006) Performing systematic literature reviews in software engineering. In: Proceedings of the 28th international conference on Software engineering, pp 1051\u20131052","DOI":"10.1145\/1134285.1134500"},{"key":"9891_CR9","doi-asserted-by":"crossref","unstructured":"Cai X, Lyu M R (2007) Software reliability modeling with test coverage: Experimentation and measurement with a fault-tolerant software project. In: The 18th IEEE International Symposium on Software Reliability (ISSRE\u201907). IEEE, pp 17\u201326","DOI":"10.1109\/ISSRE.2007.17"},{"key":"9891_CR10","doi-asserted-by":"crossref","first-page":"14","DOI":"10.1016\/j.jss.2018.05.003","volume":"143","author":"G Catolino","year":"2018","unstructured":"Catolino G, Palomba F, De Lucia A, Ferrucci F, Zaidman A (2018) Enhancing change prediction models using developer-related factors. J Syst Softw 143:14\u201328","journal-title":"J Syst Softw"},{"key":"9891_CR11","doi-asserted-by":"crossref","unstructured":"Catolino G, Di Nucci D, Ferrucci F (2019a) Cross-project just-in-time bug prediction for mobile apps: an empirical assessment. In: 2019 IEEE\/ACM 6th International Conference on Mobile Software Engineering and Systems (MOBILESoft). IEEE, pp 99\u2013110","DOI":"10.1109\/MOBILESoft.2019.00023"},{"key":"9891_CR12","doi-asserted-by":"crossref","unstructured":"Catolino G, Palomba F, Zaidman A, Ferrucci F (2019b) How the experience of development teams relates to assertion density of test classes. In: 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, pp 223\u2013234","DOI":"10.1109\/ICSME.2019.00034"},{"issue":"2","key":"9891_CR13","doi-asserted-by":"crossref","first-page":"165","DOI":"10.1109\/24.963124","volume":"50","author":"M-H Chen","year":"2001","unstructured":"Chen M-H, Lyu M R, Wong W E (2001) Effect of code coverage on software reliability measurement. IEEE Trans Reliab 50(2):165\u2013170","journal-title":"IEEE Trans Reliab"},{"issue":"6","key":"9891_CR14","doi-asserted-by":"crossref","first-page":"476","DOI":"10.1109\/32.295895","volume":"20","author":"SR Chidamber","year":"1994","unstructured":"Chidamber S R, Kemerer C F (1994) A metrics suite for object oriented design. IEEE Trans Softw Eng 20(6):476\u2013493","journal-title":"IEEE Trans Softw Eng"},{"issue":"2","key":"9891_CR15","first-page":"295","volume":"295","author":"WW Chin","year":"1998","unstructured":"Chin W W, et al. (1998) The partial least squares approach to structural equation modeling. Modern Methods Bus Res 295(2):295\u2013336","journal-title":"Modern Methods Bus Res"},{"key":"9891_CR16","doi-asserted-by":"crossref","first-page":"15","DOI":"10.1016\/j.compeleceng.2018.02.043","volume":"67","author":"GR Choudhary","year":"2018","unstructured":"Choudhary G R, Kumar S, Kumar K, Mishra A, Catal C (2018) Empirical analysis of change metrics for software fault prediction. Comput Electr Eng 67:15\u201324","journal-title":"Comput Electr Eng"},{"key":"9891_CR17","unstructured":"Cohen J (1992) Statistical power analysis, vol 1. Sage Publications, Los Angeles"},{"issue":"7","key":"9891_CR18","doi-asserted-by":"crossref","first-page":"641","DOI":"10.1109\/TSE.2016.2616306","volume":"43","author":"DA da Costa","year":"2017","unstructured":"da Costa D A, McIntosh S, Shang W, Kulesza U, Coelho R, Hassan A E (2017) A framework for evaluating the results of the szz approach for identifying bug-introducing changes. IEEE Trans Softw Eng 43(7):641\u2013657","journal-title":"IEEE Trans Softw Eng"},{"key":"9891_CR19","doi-asserted-by":"crossref","unstructured":"D\u2019Ambros M, Lanza M, Robbes R (2009) On the relationship between change coupling and software defects. In: 2009 16th Working Conference on Reverse Engineering. IEEE, pp 135\u2013144","DOI":"10.1109\/WCRE.2009.19"},{"key":"9891_CR20","doi-asserted-by":"crossref","unstructured":"D\u2019Ambros M, Bacchelli A, Lanza M (2010) On the impact of design flaws on software defects. In: 2010 10th International Conference on Quality Software. IEEE, pp 23\u201331","DOI":"10.1109\/QSIC.2010.58"},{"issue":"1","key":"9891_CR21","doi-asserted-by":"crossref","first-page":"47","DOI":"10.1002\/spip.253","volume":"11","author":"L-O Damm","year":"2006","unstructured":"Damm L-O, Lundberg L, Wohlin C (2006) Faults-slip-through\u2013a concept for measuring the efficiency of the test process. Softw Process: Improv Pract 11(1):47\u201359","journal-title":"Softw Process: Improv Pract"},{"issue":"1","key":"9891_CR22","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1109\/TSE.2017.2659747","volume":"44","author":"D Di Nucci","year":"2018","unstructured":"Di Nucci D, Palomba F, De Rosa G, Bavota G, Oliveto R, De Lucia A (2018) A developer centered bug prediction model. IEEE Trans Softw Eng 44(1):5\u201324","journal-title":"IEEE Trans Softw Eng"},{"key":"9891_CR23","unstructured":"Draper N R, Smith H (2014) Applied regression analysis, vol 326. Wiley"},{"key":"9891_CR24","doi-asserted-by":"crossref","unstructured":"Feldt R, Torkar R, Gorschek T, Afzal W (2008) Searching for cognitively diverse tests: Towards universal test diversity metrics. In: 2008 IEEE International Conference on Software Testing Verification and Validation Workshop. IEEE, pp 178\u2013186","DOI":"10.1109\/ICSTW.2008.36"},{"key":"9891_CR25","unstructured":"Feldt R (2014) Do system test cases grow old? In: 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation. IEEE, pp 343\u2013352"},{"key":"9891_CR26","doi-asserted-by":"crossref","unstructured":"Fernandes L, Ribeiro M, Carvalho L, Gheyi R, Mongiovi M, Santos A, Cavalcanti A, Ferrari F, Maldonado J C (2017) Avoiding useless mutants. In: Proceedings of the 16th ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences, pp 187\u2013198","DOI":"10.1145\/3136040.3136053"},{"key":"9891_CR27","doi-asserted-by":"crossref","unstructured":"Fischer M, Pinzger M, Gall H (2003) Populating a release history database from version control and bug tracking systems. In: 2003. ICSM 2003. Proceedings. International Conference on Software Maintenance. IEEE, pp 23\u201332","DOI":"10.1109\/ICSM.2003.1235403"},{"key":"9891_CR28","unstructured":"Fowler M (2018) Refactoring: improving the design of existing code. Addison-Wesley Professional"},{"key":"9891_CR29","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1016\/j.infsof.2018.11.008","volume":"107","author":"E Fregnan","year":"2019","unstructured":"Fregnan E, Baum T, Palomba F, Bacchelli A (2019) A survey on software coupling relations and tools. Inf Softw Technol 107:159\u2013178","journal-title":"Inf Softw Technol"},{"key":"9891_CR30","doi-asserted-by":"crossref","unstructured":"Garousi V, Felderer M, M\u00e4ntyl\u00e4 M V (2016) The need for multivocal literature reviews in software engineering: complementing systematic literature reviews with grey literature. In: Proceedings of the 20th international conference on evaluation and assessment in software engineering. ACM, pp 26","DOI":"10.1145\/2915970.2916008"},{"key":"9891_CR31","doi-asserted-by":"crossref","first-page":"52","DOI":"10.1016\/j.jss.2017.12.013","volume":"138","author":"V Garousi","year":"2018","unstructured":"Garousi V, K\u00fc\u00e7\u00fck B (2018) Smells in software test code: A survey of knowledge in industry and academia. J Syst Softw 138:52\u201381","journal-title":"J Syst Softw"},{"key":"9891_CR32","doi-asserted-by":"crossref","unstructured":"Grano G, Palomba F, Gall H C (2019) Lightweight assessment of test-case effectiveness using source-code-quality indicators. IEEE Transactions on Software Engineering. in press","DOI":"10.1109\/TSE.2019.2903057"},{"key":"9891_CR33","doi-asserted-by":"crossref","unstructured":"Grano G, De Iaco C, Palomba F, Gall H C (2020) Pizza versus pinsa: On the perception and measurability of unit test code quality. to appear","DOI":"10.1109\/ICSME46990.2020.00040"},{"issue":"7","key":"9891_CR34","doi-asserted-by":"crossref","first-page":"653","DOI":"10.1109\/32.859533","volume":"26","author":"TL Graves","year":"2000","unstructured":"Graves T L, Karr A F, Marron J S, Siy H (2000) Predicting fault incidence using software change history. IEEE Trans Softw Eng 26(7):653\u2013661","journal-title":"IEEE Trans Softw Eng"},{"issue":"2","key":"9891_CR35","doi-asserted-by":"crossref","first-page":"1","DOI":"10.18637\/jss.v015.i02","volume":"15","author":"U Halekoh","year":"2006","unstructured":"Halekoh U, H\u00f8jsgaard S, Yan J, et al. (2006) The r package geepack for generalized estimating equations. J Stat Softw 15(2):1\u201311","journal-title":"J Stat Softw"},{"issue":"4","key":"9891_CR36","doi-asserted-by":"crossref","first-page":"33","DOI":"10.1145\/2629648","volume":"23","author":"T Hall","year":"2014","unstructured":"Hall T, Zhang M, Bowes D, Sun Y (2014) Some code smells have a significant but small effect on faults. ACM Trans Softw Eng Methodol (TOSEM) 23 (4):33","journal-title":"ACM Trans Softw Eng Methodol (TOSEM)"},{"key":"9891_CR37","doi-asserted-by":"crossref","unstructured":"Hassan A E (2009) Predicting faults using the complexity of code changes. In: Proceedings of the 31st International Conference on Software Engineering. IEEE Computer Society, pp 78\u201388","DOI":"10.1109\/ICSE.2009.5070510"},{"issue":"4","key":"9891_CR38","doi-asserted-by":"crossref","first-page":"408","DOI":"10.1080\/03637750903310360","volume":"76","author":"AF Hayes","year":"2009","unstructured":"Hayes A F (2009) Beyond baron and kenny: Statistical mediation analysis in the new millennium. Commun Monograph 76(4):408\u2013420","journal-title":"Commun Monograph"},{"issue":"2","key":"9891_CR39","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3305268","volume":"52","author":"C Islam","year":"2019","unstructured":"Islam C, Babar M A, Nepal S (2019) A multi-vocal review of security orchestration. ACM Comput Surv (CSUR) 52(2):1\u201345","journal-title":"ACM Comput Surv (CSUR)"},{"key":"9891_CR40","unstructured":"Johnson B, Song Y, Murphy-Hill E, Bowdidge R (2013) Why don\u2019t software developers use static analysis tools to find bugs? In: 2013 35th International Conference on Software Engineering (ICSE). IEEE, pp 672\u2013681"},{"issue":"6","key":"9891_CR41","doi-asserted-by":"publisher","first-page":"757","DOI":"10.1109\/TSE.2012.70","volume":"39","author":"Y Kamei","year":"2013","unstructured":"Kamei Y, Shihab E, Adams B, Hassan A E, Mockus A, Sinha A, Ubayashi N (2013) A large-scale empirical study of just-in-time quality assurance. IEEE Trans Softw Eng 39(6):757\u2013773. https:\/\/doi.org\/10.1109\/TSE.2012.70","journal-title":"IEEE Trans Softw Eng"},{"key":"9891_CR42","unstructured":"Keele S, et al. (2007) Guidelines for performing systematic literature reviews in software engineering. Technical report, Technical report, Ver. 2.3 EBSE Technical Report. EBSE"},{"issue":"3","key":"9891_CR43","doi-asserted-by":"crossref","first-page":"243","DOI":"10.1007\/s10664-011-9171-y","volume":"17","author":"F Khomh","year":"2012","unstructured":"Khomh F, Di Penta M, Gu\u00e9h\u00e9neuc Y-G, Antoniol G (2012) An exploratory study of the impact of antipatterns on class change-and fault-proneness. Empir Softw Eng 17(3):243\u2013275","journal-title":"Empir Softw Eng"},{"key":"9891_CR44","doi-asserted-by":"crossref","unstructured":"Kim S, Zimmermann T, Whitehead Jr E J, Zeller A (2007) Predicting faults from cached history. In: Proceedings of the 29th international conference on Software Engineering. IEEE Computer Society, pp 489\u2013498","DOI":"10.1109\/ICSE.2007.66"},{"issue":"2","key":"9891_CR45","doi-asserted-by":"publisher","first-page":"181","DOI":"10.1109\/TSE.2007.70773","volume":"34","author":"S Kim","year":"2008","unstructured":"Kim S, Whitehead E J, Zhang Y (2008) Classifying software changes: Clean or buggy?. IEEE Trans Softw Eng 34 (2):181\u2013196. https:\/\/doi.org\/10.1109\/TSE.2007.70773","journal-title":"IEEE Trans Softw Eng"},{"issue":"2004","key":"9891_CR46","first-page":"1","volume":"33","author":"B Kitchenham","year":"2004","unstructured":"Kitchenham B (2004) Procedures for performing systematic reviews. Keele UK Keele Univ 33(2004):1\u201326","journal-title":"Keele UK Keele Univ"},{"key":"9891_CR47","doi-asserted-by":"crossref","unstructured":"Knab P, Pinzger M, Bernstein A (2006) Predicting defect densities in source code files with decision tree learners. In: Proceedings of the 2006 international workshop on Mining software repositories. ACM, pp 119\u2013125","DOI":"10.1145\/1137983.1138012"},{"issue":"4","key":"9891_CR48","doi-asserted-by":"crossref","first-page":"1213","DOI":"10.1109\/TR.2017.2727062","volume":"66","author":"PS Kochhar","year":"2017","unstructured":"Kochhar P S, Lo D, Lawall J, Nagappan N (2017) Code coverage and postrelease defects: A large-scale study on open source projects. IEEE Trans Reliab 66(4):1213\u20131228","journal-title":"IEEE Trans Reliab"},{"issue":"2","key":"9891_CR49","doi-asserted-by":"crossref","first-page":"293","DOI":"10.1109\/TSE.2008.90","volume":"35","author":"AG Koru","year":"2009","unstructured":"Koru A G, Zhang D, El Emam K, Liu H (2009) An investigation into the functional form of the size-defect relationship for software modules. IEEE Trans Softw Eng 35(2):293\u2013304","journal-title":"IEEE Trans Softw Eng"},{"key":"9891_CR50","doi-asserted-by":"crossref","unstructured":"Kudrjavets G, Nagappan N, Ball T (2006) Assessing the relationship between software assertions and faults: An empirical investigation. In: 2006 17th International Symposium on Software Reliability Engineering. IEEE, pp 204\u2013212","DOI":"10.1109\/ISSRE.2006.14"},{"key":"9891_CR51","doi-asserted-by":"crossref","unstructured":"Li N, Praphamontripong U, Offutt J (2009) An experimental comparison of four unit test criteria: Mutation, edge-pair, all-uses and prime path coverage. In: 2009 International Conference on Software Testing, Verification, and Validation Workshops. IEEE, pp 220\u2013229","DOI":"10.1109\/ICSTW.2009.30"},{"key":"9891_CR52","doi-asserted-by":"crossref","unstructured":"Lubsen Z, Zaidman A, Pinzger M (2009) Using association rules to study the co-evolution of production & test code. In: 2009. MSR\u201909. 6th IEEE International Working Conference on Mining Software Repositories. IEEE, pp 151\u2013154","DOI":"10.1109\/MSR.2009.5069493"},{"issue":"1","key":"9891_CR53","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1109\/TSE.2013.44","volume":"40","author":"L Madeyski","year":"2013","unstructured":"Madeyski L, Orzeszyna W, Torkar R, Jozala M (2013) Overcoming the equivalent mutant problem: A systematic literature review and a comparative experiment of second order mutation. IEEE Trans Softw Eng 40(1):23\u201342","journal-title":"IEEE Trans Softw Eng"},{"issue":"5","key":"9891_CR54","doi-asserted-by":"crossref","first-page":"1384","DOI":"10.1007\/s10664-014-9338-4","volume":"20","author":"MV M\u00e4ntyl\u00e4","year":"2015","unstructured":"M\u00e4ntyl\u00e4 M V, Adams B, Khomh F, Engstr\u00f6m E, Petersen K (2015) On rapid releases and software testing: a case study and a semi-systematic literature review. Empir Softw Eng 20(5):1384\u20131425","journal-title":"Empir Softw Eng"},{"key":"9891_CR55","unstructured":"Meszaros G (2007) xunit test patterns: Refactoring test code. Pearson Education"},{"issue":"1","key":"9891_CR56","doi-asserted-by":"crossref","first-page":"20","DOI":"10.1109\/TSE.2009.50","volume":"36","author":"N Moha","year":"2010","unstructured":"Moha N, Gueheneuc Y-G, Duchien L, Le Meur A-F (2010) Decor: A method for the specification and detection of code and design smells. IEEE Trans Softw Eng 36(1):20\u201336","journal-title":"IEEE Trans Softw Eng"},{"key":"9891_CR57","doi-asserted-by":"crossref","unstructured":"Moonen L (2001) Generating robust parsers using island grammars. In: Proceedings of the Eighth Working Conference on Reverse Engineering, WCRE\u201901, Stuttgart, Germany, pp 13","DOI":"10.1109\/WCRE.2001.957806"},{"issue":"1","key":"9891_CR58","doi-asserted-by":"crossref","first-page":"80","DOI":"10.2466\/pms.2003.97.1.80","volume":"97","author":"CM Morrison","year":"2003","unstructured":"Morrison C M (2003) Interpret with caution: multicollinearity in multiple regression of cognitive data. Perceptual Motor Skills 97(1):80\u201382","journal-title":"Perceptual Motor Skills"},{"key":"9891_CR59","doi-asserted-by":"crossref","unstructured":"Nagappan N, Ball T (2005a) Use of relative code churn measures to predict system defect density. In: Proceedings of the 27th international conference on Software engineering. ACM, pp 284\u2013292","DOI":"10.1145\/1062455.1062514"},{"key":"9891_CR60","doi-asserted-by":"crossref","unstructured":"Nagappan N, Williams L, Vouk M, Osborne J (2005b) Early estimation of software quality using in-process testing metrics: a controlled case study. In: ACM SIGSOFT Software Engineering Notes, vol 30, ACM, p 1\u20137","DOI":"10.1145\/1082983.1083304"},{"issue":"3","key":"9891_CR61","doi-asserted-by":"crossref","first-page":"289","DOI":"10.1007\/s10664-008-9062-z","volume":"13","author":"N Nagappan","year":"2008","unstructured":"Nagappan N, Maximilien E M, Bhat T, Williams L (2008) Realizing quality improvement through test driven development: results and experiences of four industrial teams. Empir Softw Eng 13(3):289\u2013302","journal-title":"Empir Softw Eng"},{"key":"9891_CR62","doi-asserted-by":"crossref","unstructured":"Nagappan N, Zeller A, Zimmermann T, Herzig K, Murphy B (2010) Change bursts as defect predictors. In: 2010 IEEE 21st International Symposium on Software Reliability Engineering. IEEE, pp 309\u2013318","DOI":"10.1109\/ISSRE.2010.25"},{"issue":"3","key":"9891_CR63","doi-asserted-by":"crossref","first-page":"370","DOI":"10.2307\/2344614","volume":"135","author":"JA Nelder","year":"1972","unstructured":"Nelder J A, Wedderburn RWM (1972) Generalized linear models. J R Stat Soc Ser A (Gen) 135(3):370\u2013384","journal-title":"J R Stat Soc Ser A (Gen)"},{"issue":"5","key":"9891_CR64","doi-asserted-by":"crossref","first-page":"673","DOI":"10.1007\/s11135-006-9018-6","volume":"41","author":"RM O\u2019brien","year":"2007","unstructured":"O\u2019brien R M (2007) A caution regarding rules of thumb for variance inflation factors. Qual Quant 41(5):673\u2013690","journal-title":"Qual Quant"},{"key":"9891_CR65","unstructured":"Offutt A J, Untch R H (2001) Mutation 2000: Uniting the orthogonal. In: Mutation testing for the new century. Springer, pp 34\u201344"},{"key":"9891_CR66","doi-asserted-by":"crossref","unstructured":"Palomba F, Zaidman A (2017a) Does refactoring of test smells induce fixing flaky tests? In: 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, pp 1\u201312","DOI":"10.1109\/ICSME.2017.12"},{"key":"9891_CR67","doi-asserted-by":"crossref","unstructured":"Palomba F, Zaidman A, Oliveto R, De Lucia A (2017b) An exploratory study on the relationship between changes and refactoring. In: 2017 IEEE\/ACM 25th International Conference on Program Comprehension (ICPC). IEEE, pp 176\u2013185","DOI":"10.1109\/ICPC.2017.38"},{"issue":"2","key":"9891_CR68","doi-asserted-by":"crossref","first-page":"194","DOI":"10.1109\/TSE.2017.2770122","volume":"45","author":"F Palomba","year":"2017","unstructured":"Palomba F, Zanoni M, Fontana F A, De Lucia A, Oliveto R (2017c) Toward a smell-aware bug prediction model. IEEE Trans Softw Eng 45 (2):194\u2013218","journal-title":"IEEE Trans Softw Eng"},{"key":"9891_CR69","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/j.infsof.2018.02.004","volume":"99","author":"F Palomba","year":"2018","unstructured":"Palomba F, Bavota G, Di Penta M, Fasano F, Oliveto R, De Lucia A (2018a) A large-scale empirical study on the lifecycle of code smell co-occurrences. Inf Softw Technol 99:1\u201310","journal-title":"Inf Softw Technol"},{"issue":"3","key":"9891_CR70","doi-asserted-by":"crossref","first-page":"1188","DOI":"10.1007\/s10664-017-9535-z","volume":"23","author":"F Palomba","year":"2018","unstructured":"Palomba F, Bavota G, Di Penta M, Fasano F, Oliveto R, De Lucia A (2018b) On the diffuseness and the impact on maintainability of code smells: a large scale empirical investigation. Empir Softw Eng 23(3):1188\u20131221","journal-title":"Empir Softw Eng"},{"issue":"10","key":"9891_CR71","doi-asserted-by":"crossref","first-page":"977","DOI":"10.1109\/TSE.2017.2752171","volume":"44","author":"F Palomba","year":"2018","unstructured":"Palomba F, Panichella A, Zaidman A, Oliveto R, De Lucia A (2018c) The scent of a smell: An extensive comparison between textual and structural smells. IEEE Trans Softw Eng 44(10):977\u20131000","journal-title":"IEEE Trans Softw Eng"},{"key":"9891_CR72","doi-asserted-by":"crossref","unstructured":"Palomba F, Zaidman A, De Lucia A (2018d) Automatic test smell detection using information retrieval techniques. In: 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, pp 311\u2013322","DOI":"10.1109\/ICSME.2018.00040"},{"key":"9891_CR73","doi-asserted-by":"crossref","unstructured":"Pantiuchina J, Bavota G, Tufano M, Poshyvanyk D (2018) Towards just-in-time refactoring recommenders. In: Proceedings of the 26th Conference on Program Comprehension, pp 312\u2013315","DOI":"10.1145\/3196321.3196365"},{"issue":"4","key":"9891_CR74","doi-asserted-by":"crossref","first-page":"913","DOI":"10.1109\/TR.2014.2338254","volume":"63","author":"RM Parizi","year":"2014","unstructured":"Parizi R M, Lee S P, Dabbagh M (2014) Achievements and challenges in state-of-the-art software traceability between test and code artifacts. IEEE Trans Reliab 63(4):913\u2013926","journal-title":"IEEE Trans Reliab"},{"key":"9891_CR75","doi-asserted-by":"crossref","first-page":"22","DOI":"10.1016\/j.jss.2018.12.001","volume":"150","author":"L Pascarella","year":"2019","unstructured":"Pascarella L, Palomba F, Bacchelli A (2019) Fine-grained just-in-time defect prediction. J Syst Softw 150:22\u201336","journal-title":"J Syst Softw"},{"key":"9891_CR76","doi-asserted-by":"crossref","unstructured":"Pecorelli F (2019a) Test-related factors and post-release defects: an empirical study. In: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp 1235\u20131237","DOI":"10.1145\/3338906.3342500"},{"key":"9891_CR77","doi-asserted-by":"crossref","unstructured":"Pecorelli F, Palomba F, Di Nucci D, De Lucia A (2019b) Comparing heuristic and machine learning approaches for metric-based code smell detection. In: 2019 IEEE\/ACM 27th International Conference on Program Comprehension (ICPC). IEEE, pp 93\u2013104","DOI":"10.1109\/ICPC.2019.00023"},{"key":"9891_CR78","unstructured":"Pecorelli F, Palomba F, De Lucia A, Bacchelli A (2020) The relation of test-related factors to software quality: A case study on apache systems - Online Appendix. https:\/\/figshare.com\/s\/b879d0c891f1423f4935"},{"key":"9891_CR79","unstructured":"Pezz\u00e8 M, Young M (2008) Software testing and analysis: process, principles, and techniques. Wiley"},{"key":"9891_CR80","doi-asserted-by":"crossref","first-page":"139526","DOI":"10.1109\/ACCESS.2019.2943488","volume":"7","author":"A Qusef","year":"2019","unstructured":"Qusef A, Elish M O, Binkley D (2019) An exploratory study of the relationship between software test smells and fault-proneness. IEEE Access 7:139526\u2013139536","journal-title":"IEEE Access"},{"key":"9891_CR81","doi-asserted-by":"publisher","first-page":"147","DOI":"10.1016\/j.jss.2013.10.019","volume":"88","author":"A Qusef","year":"2014","unstructured":"Qusef A, Bavota G, Oliveto R, Lucia A D, Binkley D W (2014) Recovering test-to-code traceability using slicing and textual analysis. J Syst Softw 88:147\u2013168. https:\/\/doi.org\/10.1016\/j.jss.2013.10.019","journal-title":"J Syst Softw"},{"issue":"6","key":"9891_CR82","doi-asserted-by":"crossref","first-page":"835","DOI":"10.1109\/TSE.2012.28","volume":"39","author":"Y Rafique","year":"2013","unstructured":"Rafique Y, Mi\u0161i\u0107 V B (2013) The effects of test-driven development on external quality and productivity: A meta-analysis. IEEE Trans Softw Eng 39(6):835\u2013856","journal-title":"IEEE Trans Softw Eng"},{"key":"9891_CR83","doi-asserted-by":"crossref","unstructured":"Rahman F, Devanbu P (2013) How, and why, process metrics are better. In: 2013 35th International Conference on Software Engineering (ICSE). IEEE, pp 432\u2013441","DOI":"10.1109\/ICSE.2013.6606589"},{"key":"9891_CR84","doi-asserted-by":"crossref","first-page":"164","DOI":"10.1016\/j.infsof.2018.03.009","volume":"99","author":"G Rodr\u00edguez-P\u00e9rez","year":"2018","unstructured":"Rodr\u00edguez-P\u00e9rez G, Robles G, Gonz\u00e1lez-Barahona J M (2018) Reproducibility and credibility in empirical software engineering: A case study based on a systematic literature review of the use of the szz algorithm. Inf Softw Technol 99:164\u2013176","journal-title":"Inf Softw Technol"},{"key":"9891_CR85","unstructured":"Seber GAF, Lee A J (2012) Linear regression analysis. Wiley, vol 329"},{"issue":"3\/4","key":"9891_CR86","doi-asserted-by":"crossref","first-page":"591","DOI":"10.2307\/2333709","volume":"52","author":"SS Shapiro","year":"1965","unstructured":"Shapiro S S, Wilk M B (1965) An analysis of variance test for normality (complete samples). Biometrika 52(3\/4):591\u2013611","journal-title":"Biometrika"},{"key":"9891_CR87","doi-asserted-by":"crossref","unstructured":"Shihab E, Jiang Z M, Ibrahim W M, Adams B, Hassan A E (2010) Understanding the impact of code and process metrics on post-release defects: a case study on the eclipse project. In: Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement, pp 1\u201310","DOI":"10.1145\/1852786.1852792"},{"key":"9891_CR88","unstructured":"\u015aliwerski J, Zimmermann T, Zeller A (2005) When do changes induce fixes? In: ACM sigsoft software engineering notes, vol 30. ACM, pp 1\u20135"},{"issue":"4","key":"9891_CR89","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1109\/MS.2005.111","volume":"22","author":"D Spinellis","year":"2005","unstructured":"Spinellis D (2005) Tool writing: a forgotten art?(software tools). IEEE Softw 22(4):9\u201311","journal-title":"IEEE Softw"},{"key":"9891_CR90","doi-asserted-by":"crossref","unstructured":"Spadini D, Aniche M, Bacchelli A (2018a) Pydriller: Python framework for mining software repositories. In: The 26th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE)","DOI":"10.1145\/3236024.3264598"},{"key":"9891_CR91","doi-asserted-by":"crossref","unstructured":"Spadini D, Palomba F, Zaidman A, Bruntink M, Bacchelli A (2018b) On the relation of test smells to software code quality. In: 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, pp 1\u201312","DOI":"10.1109\/ICSME.2018.00010"},{"issue":"1","key":"9891_CR92","doi-asserted-by":"crossref","first-page":"72","DOI":"10.2307\/1412159","volume":"15","author":"C Spearman","year":"1904","unstructured":"Spearman C (1904) The proof and measurement of association between two things. Amer J Psychol 15(1):72\u2013101","journal-title":"Amer J Psychol"},{"issue":"3","key":"9891_CR93","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1145\/2211616.2211620","volume":"21","author":"J Strecker","year":"2012","unstructured":"Strecker J, Memon A M (2012) Accounting for defect characteristics in evaluations of testing techniques. ACM Trans Softw Eng Methodol (TOSEM) 21(3):17","journal-title":"ACM Trans Softw Eng Methodol (TOSEM)"},{"key":"9891_CR94","doi-asserted-by":"crossref","unstructured":"Tufano M, Palomba F, Bavota G, Di Penta M, Oliveto R, De Lucia A, Poshyvanyk D (2016) An empirical investigation into the nature of test smells. In: Proceedings of the 31st IEEE\/ACM International Conference on Automated Software Engineering, pp 4\u201315","DOI":"10.1145\/2970276.2970340"},{"issue":"4","key":"9891_CR95","doi-asserted-by":"crossref","first-page":"e1838","DOI":"10.1002\/smr.1838","volume":"29","author":"M Tufano","year":"2017","unstructured":"Tufano M, Palomba F, Bavota G, Di Penta M, Oliveto R, De Lucia A, Poshyvanyk D (2017a) There and back again: Can you compile that snapshot?. J Softw Evol Process 29(4):e1838","journal-title":"J Softw Evol Process"},{"issue":"11","key":"9891_CR96","doi-asserted-by":"crossref","first-page":"1063","DOI":"10.1109\/TSE.2017.2653105","volume":"43","author":"M Tufano","year":"2017","unstructured":"Tufano M, Palomba F, Bavota G, Oliveto R, Di Penta M, De Lucia A, Poshyvanyk D (2017b) When and why your code starts to smell bad (and whether the smells go away). IEEE Trans Softw Eng 43(11):1063\u20131088","journal-title":"IEEE Trans Softw Eng"},{"key":"9891_CR97","unstructured":"Van Deursen A, Moonen L, Van Den Bergh A, Kok G (2001) Refactoring test code. In: Proceedings of the 2nd international conference on extreme programming and flexible processes in software engineering (XP2001), pp 92\u201395"},{"key":"9891_CR98","doi-asserted-by":"crossref","unstructured":"Van Rompaey B, Demeyer S (2009) Establishing traceability links between unit test cases and units under test. In: 2009 13th European Conference on Software Maintenance and Reengineering. IEEE, pp 209\u2013218","DOI":"10.1109\/CSMR.2009.39"},{"key":"9891_CR99","doi-asserted-by":"crossref","unstructured":"Vassallo C, Palomba F, Bacchelli A, Gall H C (2018) Continuous code quality: are we (really) doing that? In: Proceedings of the 33rd ACM\/IEEE International Conference on Automated Software Engineering, pp 790\u2013795","DOI":"10.1145\/3238147.3240729"},{"issue":"2","key":"9891_CR100","doi-asserted-by":"crossref","first-page":"1419","DOI":"10.1007\/s10664-019-09750-5","volume":"25","author":"C Vassallo","year":"2020","unstructured":"Vassallo C, Panichella S, Palomba F, Proksch S, Gall H C, Zaidman A (2020) How developers engage with static analysis tools in different contexts. Empir Softw Eng 25(2):1419\u20131457","journal-title":"Empir Softw Eng"},{"key":"9891_CR101","doi-asserted-by":"crossref","unstructured":"Wedyan F, Alrmuny D, Bieman J M (2009) The effectiveness of automated static analysis tools for fault detection and refactoring prediction. In: 2009 International Conference on Software Testing Verification and Validation. IEEE, pp 141\u2013150","DOI":"10.1109\/ICST.2009.21"},{"key":"9891_CR102","doi-asserted-by":"crossref","unstructured":"Wohlin C (2014) Guidelines for snowballing in systematic literature studies and a replication in software engineering. In: Proceedings of the 18th international conference on evaluation and assessment in software engineering, pp 1\u201310","DOI":"10.1145\/2601248.2601268"},{"key":"9891_CR103","doi-asserted-by":"crossref","unstructured":"Zampetti F, Scalabrino S, Oliveto R, Canfora G, Di Penta M (2017) How open source projects use static code analysis tools in continuous integration pipelines. In: 2017 IEEE\/ACM 14th International Conference on Mining Software Repositories (MSR). IEEE, pp 334\u2013344","DOI":"10.1109\/MSR.2017.2"},{"key":"9891_CR104","doi-asserted-by":"crossref","unstructured":"Zazworka N, Shaw M A, Shull F, Seaman C (2011) Investigating the impact of design debt on software quality. In: Proceedings of the 2nd Workshop on Managing Technical Debt. ACM, pp 17\u201323","DOI":"10.1145\/1985362.1985366"},{"key":"9891_CR105","unstructured":"Zimmermann T, Nagappan N, Gall H, Giger E, Murphy B (2009) Cross-project defect prediction: a large scale experiment on data vs. domain vs. process. In: Proceedings of the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering, pp 91\u2013100"},{"key":"9891_CR106","doi-asserted-by":"crossref","unstructured":"Zimmermann T, Premraj R, Zeller A (2007) Predicting defects for eclipse. In: Third International Workshop on Predictor Models in Software Engineering (PROMISE\u201907: ICSE Workshops 2007). IEEE, pp 9\u20139","DOI":"10.1109\/PROMISE.2007.10"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-020-09891-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1007\/s10664-020-09891-y\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-020-09891-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,3,26]],"date-time":"2021-03-26T19:17:37Z","timestamp":1616786257000},"score":1,"resource":{"primary":{"URL":"http:\/\/link.springer.com\/10.1007\/s10664-020-09891-y"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,2,20]]},"references-count":106,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2021,3]]}},"alternative-id":["9891"],"URL":"https:\/\/doi.org\/10.1007\/s10664-020-09891-y","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,2,20]]},"assertion":[{"value":"27 November 2020","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"20 February 2021","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}],"article-number":"18"}}