{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,8,22]],"date-time":"2024-08-22T14:03:35Z","timestamp":1724335415331},"reference-count":39,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2020,3,26]],"date-time":"2020-03-26T00:00:00Z","timestamp":1585180800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,3,26]],"date-time":"2020-03-26T00:00:00Z","timestamp":1585180800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"University of Oulu including Oulu University Hospital"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Software Qual J"],"published-print":{"date-parts":[[2020,6]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Software engineering (SE) experiments suffer from threats to validity that may impact their results. Replication allows researchers building on top of previous experiments\u2019 weaknesses and increasing the reliability of the findings. Illustrating the benefits of replication to increase the reliability of the findings and uncover moderator variables. We replicate an experiment on test-driven development (TDD) and address some of its threats to validity and those of a previous replication. We compare the replications\u2019 results and hypothesize on plausible moderators impacting results. Differences across TDD replications\u2019 results might be due to the operationalization of the response variables, the allocation of subjects to treatments, the allowance to work outside the laboratory, the provision of stubs, or the task. Replications allow examining the robustness of the findings, hypothesizing on plausible moderators influencing results, and strengthening the evidence obtained.<\/jats:p>","DOI":"10.1007\/s11219-020-09512-3","type":"journal-article","created":{"date-parts":[[2020,3,26]],"date-time":"2020-03-26T21:02:48Z","timestamp":1585256568000},"page":"371-395","update-policy":"http:\/\/dx.doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Increasing validity through replication: an illustrative TDD case"],"prefix":"10.1007","volume":"28","author":[{"given":"Adrian","family":"Santos","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sira","family":"Vegas","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Fernando","family":"Uyaguari","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Oscar","family":"Dieste","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Burak","family":"Turhan","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Natalia","family":"Juristo","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2020,3,26]]},"reference":[{"key":"9512_CR1","doi-asserted-by":"crossref","unstructured":"Aniche, M.F., Gerosa, M.A. (2010). Most common mistakes in test-driven development practice: Results from an online survey with developers. In Proceedings of the 2010 Third International Conference on Software Testing, Verification, and Validation Workshops (pp. 469\u2013478). IEEE Computer Society.","DOI":"10.1109\/ICSTW.2010.16"},{"key":"9512_CR2","unstructured":"Beck, K. (2003). Test-driven development: By example. Addison-Wesley Professional."},{"key":"9512_CR3","doi-asserted-by":"crossref","unstructured":"Bezerra, R.M., da Silva, F.Q., Santana, A.M., Magalhaes, C.V., Santos, R.E. (2015). Replication of empirical studies in software engineering: An update of a systematic mapping study. In Proceedings of the 2015 9th International Symposium on Empirical Software Engineering and Measurement (ESEM) (pp. 1\u20134). IEEE.","DOI":"10.1109\/ESEM.2015.7321213"},{"key":"9512_CR4","unstructured":"Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (2011). Introduction to meta-analysis. Wiley."},{"issue":"4","key":"9512_CR5","doi-asserted-by":"publisher","first-page":"207","DOI":"10.1145\/358841.358847","volume":"23","author":"RE Brooks","year":"1980","unstructured":"Brooks, R. E. (1980). Studying programmer behavior experimentally: The problems of proper methodology. Communications of the ACM, 23(4), 207\u2013213.","journal-title":"Communications of the ACM"},{"key":"9512_CR6","unstructured":"Carver, J.C. (2010). Towards reporting guidelines for experimental replications: A proposal. In 1st International Workshop on Replication in Empirical Software Engineering. Citeseer."},{"key":"9512_CR7","unstructured":"Cederqvist, P., Pesch, R., et al. (2002). Version management with CVS. Network Theory Ltd.."},{"key":"9512_CR8","volume-title":"Statistical power analysis for the behavioral sciences","author":"J Cohen","year":"1977","unstructured":"Cohen, J. (1977). Statistical power analysis for the behavioral sciences (revised ed.).","edition":"revised"},{"key":"9512_CR9","unstructured":"Cook, T. D., Campbell, D. T., & Day, A. (1979). Quasi-experimentation: Design & analysis issues for field settings (Vol. 351). Houghton Mifflin Boston."},{"key":"9512_CR10","doi-asserted-by":"crossref","unstructured":"Cumming, G. (2013). Understanding the new statistics: Effect sizes, confidence intervals, and meta-analysis. Routledge.","DOI":"10.4324\/9780203807002"},{"issue":"3","key":"9512_CR11","first-page":"501","volume":"19","author":"FQ Da Silva","year":"2014","unstructured":"Da Silva, F. Q., Suassuna, M., Fran\u00e7a, A. C. C., Grubb, A. M., Gouveia, T. B., Monteiro, C. V., & dos Santos, I. E. (2014). Replication of empirical studies in software engineering research: a systematic mapping study. Empirical Software Engineering, 19(3), 501\u2013557.","journal-title":"Empirical Software Engineering"},{"key":"9512_CR12","doi-asserted-by":"publisher","first-page":"76","DOI":"10.1016\/j.infsof.2015.02.001","volume":"64","author":"CV de Magalh\u00e2es","year":"2015","unstructured":"de Magalh\u00e2es, C. V., da Silva, F. Q., Santos, R. E., & Suassuna, M. (2015). Investigations about replication of empirical studies in software engineering: A systematic mapping study. Information and Software Technology, 64, 76\u2013101.","journal-title":"Information and Software Technology"},{"key":"9512_CR13","unstructured":"Eclipse Foundation, I., IDE Documentation, E. (2016). Eclipse. http:\/\/www.eclipse.org. Accessed 2 March 2020."},{"issue":"3","key":"9512_CR14","doi-asserted-by":"publisher","first-page":"226","DOI":"10.1109\/TSE.2005.37","volume":"31","author":"H Erdogmus","year":"2005","unstructured":"Erdogmus, H., Morisio, M., & Torchiano, M. (2005). On the effectiveness of the test-first approach to programming. IEEE Trans Softw Eng, 31(3), 226\u2013237.","journal-title":"IEEE Trans Softw Eng"},{"key":"9512_CR15","unstructured":"Field, A. (2009). Discovering statistics using SPSS. Sage Publications."},{"key":"9512_CR16","doi-asserted-by":"crossref","unstructured":"Fucci, D., Turhan, B. (2013). A replicated experiment on the effectiveness of test-first development. In 2013 ACM\/IEEE International Symposium on Empirical Software Engineering and Measurement (pp. 103\u2013112). IEEE.","DOI":"10.1109\/ESEM.2013.15"},{"key":"9512_CR17","doi-asserted-by":"crossref","unstructured":"Fucci, D., Turhan, B., Oivo, M. (2015). On the effects of programming and testing skills on external quality and productivity in a test-driven development context. In Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering (p. 25). ACM.","DOI":"10.1145\/2745802.2745826"},{"issue":"5","key":"9512_CR18","doi-asserted-by":"publisher","first-page":"337","DOI":"10.1016\/j.infsof.2003.09.011","volume":"46","author":"B George","year":"2004","unstructured":"George, B., & Williams, L. (2004). A structured experiment of test-driven development. Information and Software Technology, 46(5), 337\u2013342.","journal-title":"Information and Software Technology"},{"issue":"8","key":"9512_CR19","doi-asserted-by":"publisher","first-page":"1033","DOI":"10.1016\/j.infsof.2014.04.004","volume":"56","author":"OS G\u00f3mez","year":"2014","unstructured":"G\u00f3mez, O. S., Juristo, N., & Vegas, S. (2014). Understanding replication of experiments in software engineering: A classification. Information and Software Technology, 56(8), 1033\u20131048.","journal-title":"Information and Software Technology"},{"key":"9512_CR20","unstructured":"Hedges, L. V., & Olkin, I. (2014). Statistical methods for meta-analysis. Academic Press."},{"key":"9512_CR21","unstructured":"ISO\/IEC 25010:2011 (2011). URL https:\/\/www.iso.org\/obp\/ui\/iso:std:iso-iec:25010:ed-1:v1:en. Accessed 2 March 2020."},{"key":"9512_CR22","unstructured":"J\u00f8rgensen, E., Pedersen, A.R. (1998). How to obtain those nasty standard errors from transformed data-and why they should not be used."},{"key":"9512_CR23","doi-asserted-by":"crossref","unstructured":"Juristo, N., Vegas, S. (2009). Using differences among replications of software engineering experiments to gain knowledge. In Proceedings of the 2009 3rd International Symposium on Empirical Software Engineering and Measurement (ESEM) (pp. 356\u2013366). IEEE.","DOI":"10.1109\/ESEM.2009.5314236"},{"issue":"11","key":"9512_CR24","doi-asserted-by":"publisher","first-page":"1073","DOI":"10.1016\/j.infsof.2007.02.015","volume":"49","author":"VB Kampenes","year":"2007","unstructured":"Kampenes, V. B., Dy\u00e2, T., Hannay, J. E., & Sj\u00f8berg, D. I. (2007). A systematic review of effect size in software engineering experiments. Information and Software Technology, 49(11), 1073\u20131086.","journal-title":"Information and Software Technology"},{"issue":"2","key":"9512_CR25","doi-asserted-by":"publisher","first-page":"219","DOI":"10.1007\/s10664-008-9061-0","volume":"13","author":"B Kitchenham","year":"2008","unstructured":"Kitchenham, B. (2008). The role of replications in empirical software engineering-a word of warning. Empirical Software Engineering, 13(2), 219\u2013221.","journal-title":"Empirical Software Engineering"},{"key":"9512_CR26","unstructured":"Martin, C.(2001). Advanced principles, patterns and process of software development."},{"key":"9512_CR27","unstructured":"Massol, V., & Husted, T. (2003). JUnit in action. Manning Publications Co.."},{"issue":"4","key":"9512_CR28","doi-asserted-by":"publisher","first-page":"375","DOI":"10.1016\/j.infsof.2014.01.002","volume":"56","author":"H Munir","year":"2014","unstructured":"Munir, H., Moayyed, M., & Petersen, K. (2014a). Considering rigor and relevance when evaluating test driven development: A systematic review. Information and Software Technology, 56(4), 375\u2013394.","journal-title":"Information and Software Technology"},{"key":"9512_CR29","doi-asserted-by":"crossref","unstructured":"Munir, H., Wnuk, K., Petersen, K., Moayyed, M. (2014b). An experimental evaluation of test driven development vs. test-last development with industry professionals. In Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering (p. 50). ACM.","DOI":"10.1145\/2601248.2601267"},{"issue":"4","key":"9512_CR30","doi-asserted-by":"publisher","first-page":"e1670","DOI":"10.1002\/stvr.1670","volume":"28","author":"J Offutt","year":"2018","unstructured":"Offutt, J. (2018). Why don\u2019t we publish more tdd research papers? Software Testing, Verification and Reliability, 28(4), e1670.","journal-title":"Software Testing, Verification and Reliability"},{"key":"9512_CR31","unstructured":"Oracle, V. (2015). Virtualbox. User Manual-2013."},{"issue":"6","key":"9512_CR32","doi-asserted-by":"publisher","first-page":"835","DOI":"10.1109\/TSE.2012.28","volume":"39","author":"Y Rafique","year":"2013","unstructured":"Rafique, Y., & Mi\u0161i\u0107, V. B. (2013). The effects of test-driven development on external quality and productivity: A meta-analysis. IEEE Transactions on Software Engineering, 39(6), 835\u2013856.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"9512_CR33","doi-asserted-by":"crossref","unstructured":"Shepperd, M. (2016). Replicated results are more trustworthy. In T. Menzies, L. Williams, & T. Zimmermann (Eds.), Perspectives on data science for software engineering, chap. 10 (pp. 289\u2013293). Morgan Kaufmann.","DOI":"10.1016\/B978-0-12-804206-9.00052-0"},{"key":"9512_CR34","doi-asserted-by":"crossref","unstructured":"Shull, F., Basili, V., Carver, J., Maldonado, J.C., Travassos, G.H., Mendon\u00e7a, M., Fabbri, S. (2002). Replicating software engineering experiments: addressing the tacit knowledge problem. In Empirical Software Engineering, 2002. Proceedings. 2002 International Symposium n (pp. 7\u201316). IEEE.","DOI":"10.1109\/ISESE.2002.1166920"},{"issue":"1\u20132","key":"9512_CR35","doi-asserted-by":"publisher","first-page":"111","DOI":"10.1023\/B:EMSE.0000013516.80487.33","volume":"9","author":"F Shull","year":"2004","unstructured":"Shull, F., Mendonc\u00e7a, M. G., Basili, V., Carver, J., Maldonado, J. C., Fabbri, S., Travassos, G. H., & Ferreira, M. C. (2004). Knowledge-sharing issues in experimental software engineering. Empirical Software Engineering, 9(1\u20132), 111\u2013137.","journal-title":"Empirical Software Engineering"},{"issue":"6","key":"9512_CR36","doi-asserted-by":"publisher","first-page":"16","DOI":"10.1109\/MS.2010.152","volume":"27","author":"F Shull","year":"2010","unstructured":"Shull, F., Melnik, G., Turhan, B., Layman, L., Diep, M., & Erdogmus, H. (2010). What do we know about test-driven development? IEEE Software, 27(6), 16\u201319.","journal-title":"IEEE Software"},{"issue":"2","key":"9512_CR37","doi-asserted-by":"publisher","first-page":"120","DOI":"10.1109\/TSE.2015.2467378","volume":"42","author":"S Vegas","year":"2016","unstructured":"Vegas, S., Apa, C., & Juristo, N. (2016). Crossover designs in software engineering experiments: Bene_ts and perils. IEEE Transactions on Software Engineering, 42(2), 120\u2013135.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"9512_CR38","unstructured":"West, B. T., Welch, K. B., & Galecki, A. T. (2014). Linear mixed models: A practical guide using statistical software. CRC Press."},{"key":"9512_CR39","doi-asserted-by":"crossref","unstructured":"Wohlin, C., Runeson, P., H\u00f6st, M., Ohlsson, M. C., Regnell, B., & Wessl\u00e9n, A. (2012). Experimentation in software engineering. Springer Science & Business Media.","DOI":"10.1007\/978-3-642-29044-2"}],"container-title":["Software Quality Journal"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s11219-020-09512-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1007\/s11219-020-09512-3\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s11219-020-09512-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,3,26]],"date-time":"2021-03-26T01:22:24Z","timestamp":1616721744000},"score":1,"resource":{"primary":{"URL":"http:\/\/link.springer.com\/10.1007\/s11219-020-09512-3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,3,26]]},"references-count":39,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2020,6]]}},"alternative-id":["9512"],"URL":"https:\/\/doi.org\/10.1007\/s11219-020-09512-3","relation":{},"ISSN":["0963-9314","1573-1367"],"issn-type":[{"value":"0963-9314","type":"print"},{"value":"1573-1367","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,3,26]]},"assertion":[{"value":"26 March 2020","order":1,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}