{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T05:02:11Z","timestamp":1750309331265,"version":"3.41.0"},"reference-count":68,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2025,1,25]],"date-time":"2025-01-25T00:00:00Z","timestamp":1737763200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2025,2,28]]},"abstract":"<jats:p>Experiments are a commonly used method of research in software engineering (SE). Researchers report their experiments following detailed guidelines. However, researchers do not, in the field of test-driven development (TDD) at least, specify how they operationalized the response variables and, particularly, the measurement process. This article has three aims: (i) identify the response variable operationalization components in TDD experiments that study external quality; (ii) study their influence on the experimental results; (iii) determine if the experiment reports describe the measurement process components that have an impact on the results. We used two-part sequential mixed methods research. The first part of the research adopts a quantitative approach applying a statistical analysis of the impact of the operationalization components on the experimental results. The second part follows with a qualitative approach applying a systematic mapping study (SMS). The test suites, intervention types and measurers have an influence on the measurements and results of the statistical analysis of TDD experiments in SE. The test suites have a major impact on both the measurements and the results of the experiments. The intervention type has less impact on the results than on the measurements. While the measurers have an impact on the measurements, this is not transferred to the experimental results. On the other hand, the results of our SMS confirm that TDD experiments do not usually report either the test suites, the test case generation method, or the details of how external quality was measured. A measurement protocol should be used to ensure that the measurements made by different measurers are similar. It is necessary to report the test cases, the experimental task and the intervention type in order to be able to reproduce the measurements and statistical analyses, as well as to replicate experiments and build dependable families of experiments.<\/jats:p>","DOI":"10.1145\/3688837","type":"journal-article","created":{"date-parts":[[2024,8,22]],"date-time":"2024-08-22T13:33:29Z","timestamp":1724333609000},"page":"1-41","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Relevant Information in TDD Experiment Reporting"],"prefix":"10.1145","volume":"34","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7060-1002","authenticated-orcid":false,"given":"Fernando","family":"Uyaguari","sequence":"first","affiliation":[{"name":"Instituto Superior Tecnol\u00f3gico Wissen, Cuenca, Ecuador"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2135-6113","authenticated-orcid":false,"given":"Silvia T.","family":"Acu\u00f1a","sequence":"additional","affiliation":[{"name":"Escuela Polit\u00e9cnica Superior, Universidad Aut\u00f3noma de Madrid, Madrid, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7938-7564","authenticated-orcid":false,"given":"John W.","family":"Castro","sequence":"additional","affiliation":[{"name":"Departamento de Ingenier\u00eda Inform\u00e1tica y Ciencias de la Computaci\u00f3n, Universidad de Atacama, Copiap\u00f3, Chile"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0679-4361","authenticated-orcid":false,"given":"Davide","family":"Fucci","sequence":"additional","affiliation":[{"name":"Blekinge Institute of Technology, Blekinge, Sweden"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3060-7853","authenticated-orcid":false,"given":"Oscar","family":"Dieste","sequence":"additional","affiliation":[{"name":"Escuela T\u00e9cnica Superior de Ingenieros Inform\u00e1ticos, Universidad Polit\u00e9cnica de Madrid, Madrid, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8535-9386","authenticated-orcid":false,"given":"Sira","family":"Vegas","sequence":"additional","affiliation":[{"name":"Escuela T\u00e9cnica Superior de Ingenieros Inform\u00e1ticos, Universidad Polit\u00e9cnica de Madrid, Madrid, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2025,1,25]]},"reference":[{"key":"e_1_3_2_2_2","first-page":"2","volume-title":"Proceedings of the 5th International Workshop on Enterprise Networking and Computing in Healthcare Industry (IEEE Cat. No. 03EX717)","author":"Abran Alain","year":"2003","unstructured":"Alain Abran, Asma Sellami, and Witold Suryn. 2003. Metrology, measurement and metrics in software engineering. In Proceedings of the 5th International Workshop on Enterprise Networking and Computing in Healthcare Industry (IEEE Cat. No. 03EX717). IEEE, 2\u201311. DOI: 10.1109\/METRIC.2003.1232451"},{"key":"e_1_3_2_3_2","volume-title":"Test-Driven Development: By Example","author":"Beck Kent","year":"2003","unstructured":"Kent Beck. 2003. Test-Driven Development: By Example. Addison-Wesley Professional."},{"key":"e_1_3_2_4_2","doi-asserted-by":"crossref","first-page":"356","DOI":"10.1145\/1159733.1159787","volume-title":"Proceedings of the ACM\/IEEE International Symposium on Empirical Software Engineering (ISESE \u201906)","author":"Bhat Thirumalesh","year":"2006","unstructured":"Thirumalesh Bhat and Nachiappan Nagappan. 2006. Evaluating the efficacy of test-driven development: Industrial case studies. In Proceedings of the ACM\/IEEE International Symposium on Empirical Software Engineering (ISESE \u201906). ACM, New York, NY, 356\u2013363. DOI: 10.1145\/1159733.1159787"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2016.02.004"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0140-6736(86)90837-8"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2006.07.009"},{"key":"e_1_3_2_8_2","unstructured":"SIGIST British Computer Society. 2001. Standard for Software Component Testing. Technical Report Working Draft 3.4. British Computer Society (BCS) Specialist Interest Group in Software Testing (SIGIST). Retrieved from https:\/\/testingfreak.files.wordpress.com\/2008\/12\/component-testing.pdf"},{"key":"e_1_3_2_9_2","volume-title":"Buscar la filosof\u00eda en las ciencias sociales","author":"Bunge Mario","year":"1999","unstructured":"Mario Bunge. 1999. Buscar la filosof\u00eda en las ciencias sociales. Siglo XXI, Buenos Aires, Argentina."},{"key":"e_1_3_2_10_2","doi-asserted-by":"crossref","first-page":"364","DOI":"10.1145\/1159733.1159788","volume-title":"Proceedings of the ACM\/IEEE International Symposium on Empirical Software Engineering (ISESE \u201906)","author":"Canfora Gerardo","year":"2006","unstructured":"Gerardo Canfora, Aniello Cimitile, Felix Garcia, Mario Piattini, and Corrado Aaron Visaggio. 2006. Evaluating advantages of test driven development: A controlled experiment with professionals. In Proceedings of the ACM\/IEEE International Symposium on Empirical Software Engineering (ISESE \u201906). ACM, New York, NY, 364\u2013371. DOI: 10.1145\/1159733.1159788"},{"key":"e_1_3_2_11_2","first-page":"337","volume-title":"Proceedings of the 4th IEEE International Conference on Software Testing, Verification and Validation","author":"Causevic Adnan","year":"2011","unstructured":"Adnan Causevic, Daniel Sundmark, and Sasikumar Punnekkat. 2011. Factors limiting industrial adoption of test driven development: A systematic review. In Proceedings of the 4th IEEE International Conference on Software Testing, Verification and Validation. IEEE, 337\u2013346. DOI: 10.1109\/ICST.2011.19"},{"key":"e_1_3_2_12_2","first-page":"223","volume-title":"Proceedings of the 16th International Conference on Evaluation Assessment in Software Engineering (EASE)","author":"Causevic Adnan","year":"2012","unstructured":"Adnan Causevic, Daniel Sundmark, and Sasikumar Punnekkat. 2012. Test case quality in test driven development: A study design and a pilot experiment. In Proceedings of the 16th International Conference on Evaluation Assessment in Software Engineering (EASE). IET, 223\u2013227. DOI: 10.1049\/ic.2012.0029"},{"key":"e_1_3_2_13_2","volume-title":"Statistical Power Analysis for the Behavioral Sciences","author":"Cohen Jacob","year":"1988","unstructured":"Jacob Cohen. 1988. Statistical Power Analysis for the Behavioral Sciences. Routledge, New York."},{"key":"e_1_3_2_14_2","volume-title":"Research Design: Qualitative, Quantitative, and Mixed Methods Approaches","author":"Creswell John W.","year":"2018","unstructured":"John W. Creswell and David Creswell. 2018. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches (5th ed.). SAGE Publications, London, UK.","edition":"5"},{"key":"e_1_3_2_15_2","first-page":"1144","volume-title":"Proceedings of the IEEE","volume":"68","author":"Curtis Bill","year":"1980","unstructured":"Bill Curtis. 1980. Measurement and experimentation in software engineering. Proceedings of the IEEE 68, 9 (1980), 1144\u20131157. DOI: 10.1109\/PROC.1980.11813"},{"key":"e_1_3_2_16_2","doi-asserted-by":"crossref","first-page":"148","DOI":"10.1145\/1508865.1508921","volume-title":"Proceedings of the 40th ACM Technical Symposium on Computer Science Education (SIGCSE \u201909)","author":"Desai Chetan","year":"2009","unstructured":"Chetan Desai, David S. Janzen, and John Clements. 2009. Implications of integrating test-driven development into CS1\/CS2 curricula. In Proceedings of the 40th ACM Technical Symposium on Computer Science Education (SIGCSE \u201909). ACM, New York, NY, 148\u2013152. DOI: 10.1145\/1508865.1508921"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/3202710.3203163"},{"key":"e_1_3_2_18_2","unstructured":"Oscar Dieste Fernando Uyaguari and Natalia Juristo. 2021. Test cases as a measurement instrument in experimentation. arXiv:2111.05287. Retrieved from https:\/\/arxiv.org\/abs\/2111.05287"},{"key":"e_1_3_2_19_2","first-page":"365","volume-title":"Proceedings of the International Conference on Computational Intelligence and Knowledge Economy (ICCIKE)","author":"Dookhun Avishek Sharma","year":"2019","unstructured":"Avishek Sharma Dookhun and Leckraj Nagowah. 2019. Assessing the effectiveness of test-driven development and behavior-driven development in an industry setting. In Proceedings of the International Conference on Computational Intelligence and Knowledge Economy (ICCIKE). IEEE, 365\u2013370. DOI: 10.1109\/ICCIKE47802.2019.9004328"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2005.37"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1515\/cclm-2018-0421"},{"key":"e_1_3_2_22_2","doi-asserted-by":"crossref","DOI":"10.1201\/b17461","volume-title":"Software Metrics: A Rigorous and Practical Approach","author":"Fenton Norman","year":"2014","unstructured":"Norman Fenton and James Bieman. 2014. Software Metrics: A Rigorous and Practical Approach. CRC Press, Boca Raton, FL."},{"key":"e_1_3_2_23_2","volume-title":"Proceedings of the 10th ACM\/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM \u201916)","author":"Fucci Davide","year":"2016","unstructured":"Davide Fucci, Giuseppe Scanniello, Simone Romano, Martin Shepperd, Boyce Sigweni, Fernando Uyaguari, Burak Turhan, Natalia Juristo, and Markku Oivo. 2016. An external replication on the effects of test-driven development using a multi-site blind analysis approach. In Proceedings of the 10th ACM\/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM \u201916). ACM, New York, NY, Article 3, 10 pages. DOI: 10.1145\/2961111.2962592"},{"key":"e_1_3_2_24_2","first-page":"103","volume-title":"Proceedings of the ACM\/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)","author":"Fucci Davide","year":"2013","unstructured":"Davide Fucci and Burak Turhan. 2013. A replicated experiment on the effectiveness of test-first development. In Proceedings of the ACM\/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). IEEE, 103\u2013112. DOI: 10.1109\/ESEM.2013.15"},{"key":"e_1_3_2_25_2","volume-title":"Proceedings of the 8th ACM\/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM \u201914)","author":"Fucci Davide","year":"2014","unstructured":"Davide Fucci, Burak Turhan, and Markku Oivo. 2014. Impact of process conformance on the effects of test-driven development. In Proceedings of the 8th ACM\/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM \u201914). ACM, New York, NY, Article 10, 10 pages. DOI: 10.1145\/2652524.2652526"},{"key":"e_1_3_2_26_2","volume-title":"Analysis and Quantification of Test-Driven Development Approach","author":"George Body","year":"2002","unstructured":"Body George. 2002. Analysis and Quantification of Test-Driven Development Approach. Master\u2019s Thesis. North Carolina State University, North Carolina State University. Retrieved from http:\/\/www.lib.ncsu.edu\/resolver\/1840.16\/2431"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2003.09.011"},{"key":"e_1_3_2_28_2","first-page":"405","volume-title":"Proceedings of the 10th International Symposium on Software Metrics","author":"Geras Adam","year":"2004","unstructured":"Adam Geras, Michael R. Smith, and James Miller. 2004. A prototype empirical evaluation of test driven development. In Proceedings of the 10th International Symposium on Software Metrics. IEEE, 405\u2013416. DOI: 10.1109\/METRIC.2004.1357925"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1016\/S1443-8461(02)80002-8"},{"key":"e_1_3_2_30_2","first-page":"285","volume-title":"Proceedings of the 1st International Symposium on Empirical Software Engineering and Measurement (ESEM)","author":"Gupta Atul","year":"2007","unstructured":"Atul Gupta and Pankaj Jalote. 2007. An experimental evaluation of the effectiveness and efficiency of the test driven development. In Proceedings of the 1st International Symposium on Empirical Software Engineering and Measurement (ESEM). IEEE, 285\u2013294. DOI: 10.1109\/ESEM.2007.41"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2008.03.007"},{"key":"e_1_3_2_32_2","volume-title":"ISO 5725:1994. Accuracy (Trueness and Precision) of Measurement Methods and Results \u2013 Parts 1 to 6","author":"ISO","year":"1994","unstructured":"ISO. 1994. ISO 5725:1994. Accuracy (Trueness and Precision) of Measurement Methods and Results \u2013 Parts 1 to 6. ISO."},{"key":"e_1_3_2_33_2","volume-title":"ISO\/IEC 9126-1:2001. Information Technology - Software Product Quality - Part 1: Quality Model","author":"ISO\/IEC","year":"2001","unstructured":"ISO\/IEC. 2001. ISO\/IEC 9126-1:2001. Information Technology - Software Product Quality - Part 1: Quality Model. ISO\/IEC."},{"key":"e_1_3_2_34_2","volume-title":"ISO\/IEC 25010:2011. Systems and Software Engineering \u2013 Systems and Software Quality Requirements and Evaluation (SQuaRE) \u2013 System and Software Quality Models","author":"ISO\/IEC","year":"2011","unstructured":"ISO\/IEC. 2011. ISO\/IEC 25010:2011. Systems and Software Engineering \u2013 Systems and Software Quality Requirements and Evaluation (SQuaRE) \u2013 System and Software Quality Models. ISO\/IEC."},{"key":"e_1_3_2_35_2","volume-title":"International Vocabulary of Metrology \u2013 Basic and General Concepts and Associated Terms","author":"Joint Committee for Guides in Metrology","year":"2012","unstructured":"Joint Committee for Guides in Metrology. 2012. International Vocabulary of Metrology \u2013 Basic and General Concepts and Associated Terms. Number JCGM 200:2012 in 3. International Organization of Legal Metrology, Paris, France."},{"key":"e_1_3_2_36_2","first-page":"1","volume-title":"Proceedings of the 4th International Workshop on Conducting Empirical Studies in Industry (CESI \u201916)","author":"Juristo Natalia","year":"2016","unstructured":"Natalia Juristo. 2016. Experiences Conducting Experiments in Industry: The ESEIL FiDiPro Project. In Proceedings of the 4th International Workshop on Conducting Empirical Studies in Industry (CESI \u201916). ACM, New York, NY, 1\u20133. DOI: 10.1145\/2896839.2896846"},{"key":"e_1_3_2_37_2","volume-title":"EBSE Technical Report Version 2.3, EBSE-2007-01","author":"Kitchenham Barbara","year":"2007","unstructured":"Barbara Kitchenham and Stuart Charters. 2007. Procedures for performing systematic literature review in software engineering. In EBSE Technical Report Version 2.3, EBSE-2007-01. Software Eng. Group., UK."},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1109\/32.489070"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/32.950316"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2002.1027796"},{"key":"e_1_3_2_41_2","first-page":"113","volume-title":"Proceedings of the Conference on Software Engineering: Evolution and Emerging Technologies","author":"Madeyski Lech","year":"2005","unstructured":"Lech Madeyski. 2005. Preliminary analysis of the effects of pair programming and test-driven development on the external code quality. In Proceedings of the Conference on Software Engineering: Evolution and Emerging Technologies. IOS Press, NLD, 113\u2013123."},{"key":"e_1_3_2_42_2","volume-title":"Handbook for Calculation of Measurement Uncertainty in Environmental Laboratories","author":"Magnusson Bertil","year":"2003","unstructured":"Bertil Magnusson. 2003. Handbook for Calculation of Measurement Uncertainty in Environmental Laboratories. Nordtest."},{"key":"e_1_3_2_43_2","first-page":"26","volume-title":"Proceedings of 1st International Proficiency Testing Conference","author":"Magnusson Bertil","year":"2007","unstructured":"Bertil Magnusson, H\u00e5vard Hovind, Mikael Krysell, and T Naykki. 2007. Nordest handbook for calculation of measurement uncertainty based on quality control and method validation. In Proceedings of 1st International Proficiency Testing Conference. Springer, 26\u201330."},{"key":"e_1_3_2_44_2","volume-title":"Advanced Principles, Patterns and Process of Software Development","author":"Martin Robert C.","year":"2001","unstructured":"Robert C. Martin. 2001. Advanced Principles, Patterns and Process of Software Development. Prentice Hall, UK."},{"key":"e_1_3_2_45_2","first-page":"131","volume-title":"IEE Proceedings Software","volume":"149","author":"Muller Matthias M","year":"2002","unstructured":"Matthias M Muller and O. Hagner. 2002. Experiment about test-first programming. IEE Proceedings Software 149, 5 (2002), 131\u2013136. DOI: 10.1049\/ip-sen:20020540"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2014.01.002"},{"key":"e_1_3_2_47_2","volume-title":"Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering (EASE \u201914)","author":"Munir Hussan","year":"2014","unstructured":"Hussan Munir, Krzysztof Wnuk, Kai Petersen, and Misagh Moayyed. 2014b. An experimental evaluation of test driven development vs. test-last development with industry professionals. In Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering (EASE \u201914). ACM, New York, NY, Article 50, 10 pages. DOI: 10.1145\/2601248.2601267"},{"key":"e_1_3_2_48_2","doi-asserted-by":"crossref","DOI":"10.1002\/9781119202486","volume-title":"The Art of Software Testing","author":"Myers Glenford J.","year":"2012","unstructured":"Glenford J. Myers, Corey Sandler, and Tom Badgett. 2012. The Art of Software Testing (3rd ed ed.). John Wiley & Sons, Hoboken and NJ.","edition":"3"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-008-9062-z"},{"issue":"6","key":"e_1_3_2_50_2","doi-asserted-by":"crossref","first-page":"603","DOI":"10.1007\/s00769-012-0932-0","article-title":"Software support for the Nordtest method of measurement uncertainty evaluation","volume":"17","author":"N\u00e4ykki Teemu","year":"2012","unstructured":"Teemu N\u00e4ykki, Atte Virtanen, and Ivo Leito. 2012. Software support for the Nordtest method of measurement uncertainty evaluation. Accreditation and Quality Assurance 17, 6 (2012), 603\u2013612.","journal-title":"Accreditation and Quality Assurance"},{"key":"e_1_3_2_51_2","volume-title":"Social Research Methods: Qualitative and Quantitative Approaches","author":"Neuman Lawrence W","year":"2002","unstructured":"Lawrence W Neuman. 2002. Social Research Methods: Qualitative and Quantitative Approaches. Pearson Education Limited, Boston, USA."},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1515\/cclm-2016-0896"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2011.02.002"},{"key":"e_1_3_2_54_2","doi-asserted-by":"crossref","first-page":"83","DOI":"10.1109\/EURCON.2003.1248153","volume-title":"Proceedings of the IEEE Region 8 EUROCON 2003. Computer as a Tool","volume":"2","author":"Pancur Matjaz","year":"2003","unstructured":"Matjaz Pancur, Mojca Ciglaric, M. Trampus, and T. Vidmar. 2003. Towards empirical evaluation of test-driven development in a university environment. In Proceedings of the IEEE Region 8 EUROCON 2003. Computer as a Tool, Vol. 2. IEEE, 83\u201386. DOI: 10.1109\/EURCON.2003.1248153"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2020.3027522"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2012.28"},{"key":"e_1_3_2_57_2","first-page":"465","volume-title":"Proceedings of the 25th Asia-Pacific Software Engineering Conference (APSEC)","author":"Santos Adrian","year":"2018","unstructured":"Adrian Santos, Jaroslav Spisak, Markku Oivo, and Natalia Juristo. 2018. Improving development practices through experimentation: An industrial TDD case. In Proceedings of the 25th Asia-Pacific Software Engineering Conference (APSEC). IEEE Computer Society, 465\u2013473. DOI: 10.1109\/APSEC.2018.00061"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-020-09895-8"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11219-020-09512-3"},{"key":"e_1_3_2_60_2","unstructured":"Dora Elizabeth Jaimes Sequeda. 2015. ?Se producen diferencias considerables en los resultados experimentales en funci\u00f3n del procedimiento de medici\u00f3n? Un estudio comparativo para los constructos calidad y productividad en el marco de un experimento de test-driven development. Retrieved from https:\/\/oa.upm.es\/44268\/"},{"key":"e_1_3_2_61_2","volume-title":"Experimental and Quasi-Experimental Designs for Generalized Causal Inference","author":"Shadish William","year":"2002","unstructured":"William Shadish, Thomas D. Cook, and Donald Thomas Campbell. 2002. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Houghton Mifflin Boston, MA, Boston, MA, USA."},{"key":"e_1_3_2_62_2","doi-asserted-by":"crossref","DOI":"10.4324\/9781315762012","volume-title":"Routledge Handbook of Qualitative Research in Sport and Exercise","author":"Smith Brett","year":"2016","unstructured":"Brett Smith and Andrew C. Sparkes. 2016. Routledge Handbook of Qualitative Research in Sport and Exercise. Routledge, London, UK."},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1016\/S1443-8461(01)80002-2"},{"key":"e_1_3_2_64_2","unstructured":"Matthew Thompson Arpita Tiwari Rongwei Fu Esther Moe and David I. Buckley. 2012. A framework to facilitate the use of systematic reviews and meta-analyses in the design of primary research studies. Retrieved from http:\/\/europepmc.org\/books\/NBK83621"},{"key":"e_1_3_2_65_2","doi-asserted-by":"crossref","DOI":"10.4324\/9780203645727","volume-title":"Mixing Methods in Psychology: The Integration of Qualitative and Quantitative Methods in Theory and Practice","author":"Todd Zazie","year":"2004","unstructured":"Zazie Todd, Nerlich, Brigitte, Suzanne Mckeown, and David D. Clarke. 2004. Mixing Methods in Psychology: The Integration of Qualitative and Quantitative Methods in Theory and Practice. Psychology Press, London, UK."},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-016-9490-0"},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2019.2949811"},{"key":"e_1_3_2_68_2","first-page":"229","volume-title":"Proceedings of the 6th International Conference on Information Technology: New Generations","author":"Vu John V.","year":"2009","unstructured":"John V. Vu, Niklas Frojd, Clay Shenkel-Therolf, and David S. Janzen. 2009. Evaluating test-driven development in an industry-sponsored capstone project. In Proceedings of the 6th International Conference on Information Technology: New Generations. IEEE, 229\u2013234. DOI: 10.1109\/ITNG.2009.11"},{"key":"e_1_3_2_69_2","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-642-29044-2","volume-title":"Experimentation in Software Engineering","author":"Wohlin Claes","year":"2012","unstructured":"Claes Wohlin, Per Runeson, Martin H\u00f6st, Magnus C. Ohlsson, Bj\u00f6rn Regnell, and Anders Wessl\u00e9n. 2012. Experimentation in Software Engineering. Springer, Berlin."}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3688837","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3688837","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:04:10Z","timestamp":1750291450000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3688837"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,1,25]]},"references-count":68,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2025,2,28]]}},"alternative-id":["10.1145\/3688837"],"URL":"https:\/\/doi.org\/10.1145\/3688837","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"type":"print","value":"1049-331X"},{"type":"electronic","value":"1557-7392"}],"subject":[],"published":{"date-parts":[[2025,1,25]]},"assertion":[{"value":"2023-12-05","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-07-12","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-01-25","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}