{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,31]],"date-time":"2025-12-31T00:32:22Z","timestamp":1767141142499,"version":"build-2238731810"},"reference-count":74,"publisher":"Springer Science and Business Media LLC","issue":"6","license":[{"start":{"date-parts":[[2022,12,3]],"date-time":"2022-12-03T00:00:00Z","timestamp":1670025600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,12,3]],"date-time":"2022-12-03T00:00:00Z","timestamp":1670025600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"OPHELIA","award":["RTI2018-101204-B-C22"],"award-info":[{"award-number":["RTI2018-101204-B-C22"]}]},{"name":"EKIPMENT-PLUS","award":["P18-FR-2895"],"award-info":[{"award-number":["P18-FR-2895"]}]},{"name":"MEMENTO","award":["US-1381595"],"award-info":[{"award-number":["US-1381595"]}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Computing"],"published-print":{"date-parts":[[2023,6]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    The need of replicating empirical studies in Computer Science is widely recognized among the research community. It is essential to report the changes of each replication to promote not only the comprehensibility of the evolution of the experimental validity across a family of studies, but also replicability itself. Unfortunately, the lack of proposals for systematic reporting of changes in replications undermines these desirable objectives. The main goal of the work presented in this article is to provide researchers in Computer Science with a systematic tool-supported approach for the specification and reporting of changes in the replications of their empirical studies. Applying\n                    <jats:italic>Design Science Research<\/jats:italic>\n                    , we have developed and validated a composite artifact consisting of (i) a metamodel that formalizes all the relevant concepts related to replications and their changes; (ii) templates and linguistic patterns that facilitate their reporting; and (iii) a proof-of-concept model-based software tool that supports the proposed approach. For its validation, we have carried out a multiple case study that includes 9 families of empirical studies not only from Computer Science, but also from an area as different as Agrobiology , to check the external validity of our approach. The 9 families encompass 23 replication studies and a total of 92 replication changes, for which we have analyzed the suitability of our proposal. The multiple case study revealed some initial limitations of our approach related to threats to experimental validity and context variables. After several improvement iterations on the artifact, all of the 92 replication changes could be properly specified, including also their qualitatively estimated effects on experimental validity and their corresponding visualization. Our proposal for the specification of replication changes seems to fit the needs not only of replications in Computer Science, but also in other research areas. Nevertheless, further research is needed to improve it and disseminate its use among the research community.\n                  <\/jats:p>","DOI":"10.1007\/s00607-022-01133-x","type":"journal-article","created":{"date-parts":[[2022,12,3]],"date-time":"2022-12-03T05:02:41Z","timestamp":1670043761000},"page":"1189-1213","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["A model-based approach for specifying changes in replications of empirical studies in computer Science"],"prefix":"10.1007","volume":"105","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8334-6039","authenticated-orcid":false,"given":"Margarita","family":"Cruz","sequence":"first","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9390-3772","authenticated-orcid":false,"given":"Beatriz","family":"Bern\u00e1rdez","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3630-5511","authenticated-orcid":false,"given":"Amador","family":"Dur\u00e1n","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2470-8287","authenticated-orcid":false,"given":"Cathy","family":"Guevara-Vega","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9827-1834","authenticated-orcid":false,"given":"Antonio","family":"Ruiz-Cort\u00e9s","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2022,12,3]]},"reference":[{"issue":"1","key":"1133_CR1","doi-asserted-by":"publisher","first-page":"241","DOI":"10.1007\/s10664-012-9221-0","volume":"19","author":"\u00d6 Albayrak","year":"2014","unstructured":"Albayrak \u00d6, Carver JC (2014) Investigation of individual factors impacting the effectiveness of requirements inspections: a replicated experiment. Empir Softw Eng 19(1):241\u2013266. https:\/\/doi.org\/10.1007\/s10664-012-9221-0","journal-title":"Empir Softw Eng"},{"key":"1133_CR2","unstructured":"Almqvist JPF (2006) Replication of controlled experiments in empirical software engineering - a survey. Master\u2019s thesis, master\u2019s thesis, Department of Computer Science, Faculty of Science, Lund University, Sweden"},{"key":"1133_CR3","doi-asserted-by":"publisher","unstructured":"Aranda A (2016) Empirical study of the influence of analyst experience and domain knowledge on the effectiveness of requirements education. PhD thesis, Polytechnic University of Madrid, https:\/\/doi.org\/10.20868\/UPM.thesis.40566","DOI":"10.20868\/UPM.thesis.40566"},{"issue":"4","key":"1133_CR4","doi-asserted-by":"publisher","first-page":"1437","DOI":"10.1007\/s10664-015-9391-7","volume":"21","author":"S Assar","year":"2016","unstructured":"Assar S, Borg M, Pfahl D (2016) Using text clustering to predict defect resolution time: a conceptual replication and an evaluation of prediction accuracy. Empir Softw Eng 21(4):1437\u20131475. https:\/\/doi.org\/10.1007\/s10664-015-9391-7","journal-title":"Empir Softw Eng"},{"key":"1133_CR5","unstructured":"Association for Computing Machinery (2020) Artifact review and badging. https:\/\/www.acm.org\/publications\/policies\/artifact-review-and-badging-current"},{"key":"1133_CR6","doi-asserted-by":"publisher","unstructured":"Baldassarre MT, Carver J, Dieste O, et\u00a0al (2014) Replication types: towards a shared taxonomy. In: Proceedings of EASE\u201914, pp 18:1-18:4, https:\/\/doi.org\/10.1145\/2601248.2601299","DOI":"10.1145\/2601248.2601299"},{"key":"1133_CR7","unstructured":"Basili VR, Caldiera G, Rombach HD (1994) Goal question metrics paradigm. Encycl Softw Eng, Wiley, pp 528\u2013532"},{"issue":"4","key":"1133_CR8","doi-asserted-by":"publisher","first-page":"456","DOI":"10.1109\/32.799939","volume":"25","author":"VR Basili","year":"1999","unstructured":"Basili VR, Shull F, Lanubile F (1999) Building knowledge through families of experiments. IEEE Trans Softw Eng 25(4):456\u2013473. https:\/\/doi.org\/10.1109\/32.799939","journal-title":"IEEE Trans Softw Eng"},{"key":"1133_CR9","doi-asserted-by":"publisher","unstructured":"Bern\u00e1rdez B, Dur\u00e1n A, Parejo JA, et\u00a0al (2014) A controlled experiment to evaluate the effects of mindfulness in software engineering. In: Proceedings of ESEM\u201914, pp 17\u201327, https:\/\/doi.org\/10.1145\/2652524.2652539","DOI":"10.1145\/2652524.2652539"},{"issue":"2","key":"1133_CR10","doi-asserted-by":"publisher","first-page":"432","DOI":"10.1109\/TSE.2020.2991699","volume":"48","author":"B Bern\u00e1rdez","year":"2020","unstructured":"Bern\u00e1rdez B, Dur\u00e1n AD, Parejo JA et al (2020) Effects of mindfulness on conceptual modeling performance: a series of experiments. IEEE Trans Software Eng 48(2):432\u2013452. https:\/\/doi.org\/10.1109\/TSE.2020.2991699","journal-title":"IEEE Trans Software Eng"},{"key":"1133_CR11","doi-asserted-by":"publisher","first-page":"153","DOI":"10.1016\/j.jss.2016.06.104","volume":"136","author":"B Bern\u00e1rdez","year":"2018","unstructured":"Bern\u00e1rdez B, Dur\u00e1n A, Parejo JA et al (2018) An experimental replication on the effect of the practice of mindfulness in conceptual modeling performance. J Syst Softw 136:153\u2013172. https:\/\/doi.org\/10.1016\/j.jss.2016.06.104","journal-title":"J Syst Softw"},{"key":"1133_CR12","doi-asserted-by":"publisher","unstructured":"Bezerra RM, da\u00a0Silva FQ, Santana AM, et\u00a0al (2015) Replication of empirical studies in software engineering: An update of a systematic mapping study. In: Proceedings of ESEM\u201915, IEEE, pp 1\u20134, https:\/\/doi.org\/10.1109\/ESEM.2015.7321213","DOI":"10.1109\/ESEM.2015.7321213"},{"key":"1133_CR13","unstructured":"Brooks A, Daly J, Miller J, et\u00a0al (1996) Replication of experimental results in software engineering. Technical Report ISERN-96-10, University of Strathclyde, Glasgow, UK"},{"key":"1133_CR14","doi-asserted-by":"publisher","unstructured":"Brooks A, Roper M, Wood M, et\u00a0al (2008) Replication\u2019s role in software engineering, Springer, pp 365\u2013379. https:\/\/doi.org\/10.1007\/978-1-84800-044-5_14","DOI":"10.1007\/978-1-84800-044-5_14"},{"key":"1133_CR15","unstructured":"Campbell DT, Stanley JC (2015) Experimental and quasi-experimental designs for research. Ravenio books"},{"key":"1133_CR16","unstructured":"Carver JC (2010) Towards reporting guidelines for experimental replications: a proposal. In: Proceedings of the 1st international workshop on replication in empirical software engineering, pp 1\u20134"},{"key":"1133_CR17","unstructured":"Ciolkowski M, Shull F, Biffl S (2002) A family of experiments to investigate the influence of context on the effect of inspection techniques. In: Proceedings of EASE\u201902, pp 48\u201360"},{"key":"1133_CR18","doi-asserted-by":"publisher","first-page":"26,773","DOI":"10.1109\/ACCESS.2019.2952191","volume":"8","author":"M Cruz","year":"2019","unstructured":"Cruz M, Bern\u00e1rdez B, Dur\u00e1n A et al (2019) Replication of studies in empirical software engineering: a systematic mapping study, from 2013 to 2018. IEEE Access 8:26,773-26,791. https:\/\/doi.org\/10.1109\/ACCESS.2019.2952191","journal-title":"IEEE Access"},{"key":"1133_CR19","doi-asserted-by":"publisher","unstructured":"Cruz M, Bern\u00e1rdez B, Dur\u00e1n A et al (2021). Supplemental material: instantiation of the proposed templates in the multiple case study using C\u00c6SAR LaTeX template. https:\/\/doi.org\/10.5281\/zenodo.6631976","DOI":"10.5281\/zenodo.6631976"},{"issue":"3","key":"1133_CR20","doi-asserted-by":"publisher","first-page":"501","DOI":"10.1007\/s10664-012-9227-7","volume":"19","author":"FQ Da Silva","year":"2014","unstructured":"Da Silva FQ, Suassuna M, Fran\u00e7a ACC et al (2014) Replication of empirical studies in software engineering research: a systematic mapping study. Empir Softw Eng 19(3):501\u2013557. https:\/\/doi.org\/10.1007\/s10664-012-9227-7","journal-title":"Empir Softw Eng"},{"key":"1133_CR21","unstructured":"Dur\u00e1n A, Bern\u00e1rdez B, Ruiz-Cort\u00e9s A, et\u00a0al (1999) A requirements elicitation approach based in templates and patterns. In: Proceedings of WER\u201999"},{"key":"1133_CR22","doi-asserted-by":"publisher","unstructured":"Dur\u00e1n A, Corchuelo R, Ruiz-Cort\u00e8s A, et\u00a0al (2002) Supporting requirements verification using XSLT. In: Proceedings of RE\u201902, pp 165\u2013172, https:\/\/doi.org\/10.1109\/ICRE.2002.1048519","DOI":"10.1109\/ICRE.2002.1048519"},{"key":"1133_CR23","doi-asserted-by":"publisher","unstructured":"Fern\u00e1ndez DM, Graziotin D, Wagner S, et\u00a0al (2020) Open science in software engineering, Springer International Publishing, pp 477\u2013501. https:\/\/doi.org\/10.1007\/978-3-030-32489-6_17","DOI":"10.1007\/978-3-030-32489-6_17"},{"issue":"1","key":"1133_CR24","doi-asserted-by":"publisher","first-page":"212","DOI":"10.1007\/s10664-014-9354-4","volume":"21","author":"AM Fern\u00e1ndez-S\u00e1ez","year":"2016","unstructured":"Fern\u00e1ndez-S\u00e1ez AM, Genero M, Caivano D et al (2016) Does the level of detail of UML diagrams affect the maintainability of source code?: a family of experiments. Empir Softw Eng 21(1):212\u2013259. https:\/\/doi.org\/10.1007\/s10664-014-9354-4","journal-title":"Empir Softw Eng"},{"key":"1133_CR25","unstructured":"Gamma E, Helm R, Johnson R, et\u00a0al (1995) Design patterns: elements of reusable object-oriented software. Addison-Wesley"},{"key":"1133_CR26","doi-asserted-by":"publisher","first-page":"554","DOI":"10.1016\/j.foodchem.2015.10.131","volume":"197","author":"A Garc\u00eda","year":"2016","unstructured":"Garc\u00eda A, Rodr\u00edguez-Juan E, Rodr\u00edguez-Guti\u00e9rrez G et al (2016) Extraction of phenolic compounds from virgin olive oil by deep eutectic solvents (DESs). Food Chem 197:554\u2013561. https:\/\/doi.org\/10.1016\/j.foodchem.2015.10.131","journal-title":"Food Chem"},{"key":"1133_CR27","doi-asserted-by":"publisher","unstructured":"G\u00f3mez OS, Juristo N, Vegas S (2010) Replications types in experimental disciplines. In: Proceedings of ESEM\u201910, pp 71\u201375, https:\/\/doi.org\/10.1145\/1852786.1852790","DOI":"10.1145\/1852786.1852790"},{"issue":"8","key":"1133_CR28","doi-asserted-by":"publisher","first-page":"1033","DOI":"10.1016\/j.infsof.2014.04.004","volume":"56","author":"OS G\u00f3mez","year":"2014","unstructured":"G\u00f3mez OS, Juristo N, Vegas S (2014) Understanding replication of experiments in software engineering: a classification. Inf Softw Technol 56(8):1033\u20131048. https:\/\/doi.org\/10.1016\/j.infsof.2014.04.004","journal-title":"Inf Softw Technol"},{"key":"1133_CR29","doi-asserted-by":"crossref","unstructured":"Guevara-Vega C, Bern\u00e1rdez B, Dur\u00e1n A, et\u00a0al (2021) Empirical strategies in software engineering research: A literature survey. In: II International conference on information systems and software technologies (ICI2ST 2021). IEEE Press, Quito, Ecuador","DOI":"10.1109\/ICI2ST51859.2021.00025"},{"key":"1133_CR30","unstructured":"Carvajal de\u00a0la Haza S (2016) Copper extraction by brown mustard (brassica juncea) plants during vegetative growth in an artificially contaminated soil and effect of rhamnolipid application. http:\/\/hdl.handle.net\/11441\/50282, (Final Degree Project). Universidad de Sevilla"},{"issue":"4","key":"1133_CR31","doi-asserted-by":"publisher","first-page":"1866","DOI":"10.1007\/s10664-016-9468-y","volume":"22","author":"S Herbold","year":"2017","unstructured":"Herbold S, Trautsch A, Grabowski J (2017) Global versus local models for cross-project defect prediction. Empir Softw Eng 22(4):1866\u20131902. https:\/\/doi.org\/10.1007\/s10664-016-9468-y","journal-title":"Empir Softw Eng"},{"issue":"2","key":"1133_CR32","doi-asserted-by":"publisher","first-page":"303","DOI":"10.1007\/s10664-013-9266-8","volume":"19","author":"J Itkonen","year":"2014","unstructured":"Itkonen J, M\u00e4ntyl\u00e4 MV (2014) Are test cases needed? replicated comparison between exploratory and test-case-based software testing. Empir Softw Eng 19(2):303\u2013342. https:\/\/doi.org\/10.1007\/s10664-013-9266-8","journal-title":"Empir Softw Eng"},{"key":"1133_CR33","doi-asserted-by":"publisher","unstructured":"Jedlitschka A, Ciolkowski M, Pfahl D (2008) Reporting experiments in software engineering. In: Guide to advanced empirical software engineering. Springer, pp 201\u2013228, https:\/\/doi.org\/10.1007\/978-1-84800-044-5_8","DOI":"10.1007\/978-1-84800-044-5_8"},{"key":"1133_CR34","doi-asserted-by":"publisher","unstructured":"Juristo N, G\u00f3mez OS (2012) Replication of software engineering experiments. In: Empirical software engineering and verification. Springer, pp 60\u201388, https:\/\/doi.org\/10.1007\/978-3-642-25231-0_2","DOI":"10.1007\/978-3-642-25231-0_2"},{"key":"1133_CR35","doi-asserted-by":"publisher","unstructured":"Juristo N, Moreno AM (2013) Basics of software engineering experimentation. Springer Science and Business Media, Berlin. https:\/\/doi.org\/10.1007\/978-1-4757-3304-4","DOI":"10.1007\/978-1-4757-3304-4"},{"key":"1133_CR36","doi-asserted-by":"publisher","unstructured":"Juristo N, Vegas S (2003) Functional testing, structural testing and code reading: what fault type do they each detect? In: Empirical methods and studies in software engineering. Springer, pp 208\u2013232, https:\/\/doi.org\/10.1007\/978-3-540-45143-3_12","DOI":"10.1007\/978-3-540-45143-3_12"},{"key":"1133_CR37","doi-asserted-by":"publisher","unstructured":"Juristo N, Vegas S (2009) Using differences among replications of software engineering experiments to gain knowledge. In: Proceedings of ESEM\u201909, IEEE, pp 356\u2013366, https:\/\/doi.org\/10.1109\/ESEM.2009.5314236","DOI":"10.1109\/ESEM.2009.5314236"},{"issue":"3","key":"1133_CR38","doi-asserted-by":"publisher","first-page":"295","DOI":"10.1007\/s10664-010-9141-9","volume":"16","author":"N Juristo","year":"2011","unstructured":"Juristo N, Vegas S (2011) The role of non-exact replications in software engineering experiments. Empir Softw Eng 16(3):295\u2013324. https:\/\/doi.org\/10.1007\/s10664-010-9141-9","journal-title":"Empir Softw Eng"},{"key":"1133_CR39","doi-asserted-by":"publisher","unstructured":"Juristo N, Vegas S, Solari M, et\u00a0al (2012) Comparing the effectiveness of equivalence partitioning, branch testing and code reading by stepwise abstraction applied by subjects. In: Proceedings of international conference on software testing, verification and validation, pp 330\u2013339, https:\/\/doi.org\/10.1109\/ICST.2012.113","DOI":"10.1109\/ICST.2012.113"},{"issue":"2","key":"1133_CR40","doi-asserted-by":"publisher","first-page":"215","DOI":"10.1016\/j.infsof.2012.07.016","volume":"55","author":"N Juristo","year":"2013","unstructured":"Juristo N, Vegas S, Solari M et al (2013) A process for managing interaction between experimenters to get useful similar replications. Inf Softw Technol 55(2):215\u2013225. https:\/\/doi.org\/10.1016\/j.infsof.2012.07.016","journal-title":"Inf Softw Technol"},{"issue":"2","key":"1133_CR41","doi-asserted-by":"publisher","first-page":"219","DOI":"10.1007\/s10664-008-9061-0","volume":"13","author":"B Kitchenham","year":"2008","unstructured":"Kitchenham B (2008) The role of replications in empirical software engineering a word of warning. Empir Softw Eng 13(2):219\u2013221. https:\/\/doi.org\/10.1007\/s10664-008-9061-0","journal-title":"Empir Softw Eng"},{"key":"1133_CR42","doi-asserted-by":"publisher","first-page":"76","DOI":"10.1016\/j.infsof.2015.02.001","volume":"64","author":"CV de Magalh\u00e3es","year":"2015","unstructured":"de Magalh\u00e3es CV, da Silva FQ, Santos RE et al (2015) Investigations about replication of empirical studies in software engineering: a systematic mapping study. Inf Softw Technol 64:76\u2013101. https:\/\/doi.org\/10.1016\/j.infsof.2015.02.001","journal-title":"Inf Softw Technol"},{"key":"1133_CR43","doi-asserted-by":"publisher","unstructured":"Mendez D, Graziotin D, Wagner S, et\u00a0al (2020) Open science in software engineering. In: Contemporary empirical methods in software engineering. Springer, pp 477\u2013501, https:\/\/doi.org\/10.1007\/978-3-030-32489-6_17","DOI":"10.1007\/978-3-030-32489-6_17"},{"issue":"2","key":"1133_CR44","doi-asserted-by":"publisher","first-page":"693","DOI":"10.1007\/s10664-017-9528-y","volume":"23","author":"M Mondal","year":"2018","unstructured":"Mondal M, Rahman MS, Roy CK et al (2018) Is cloned code really stable? Empir Softw Eng 23(2):693\u2013770. https:\/\/doi.org\/10.1007\/s10664-017-9528-y","journal-title":"Empir Softw Eng"},{"key":"1133_CR45","unstructured":"M\u00e1rquez M (2018) Copper extraction by brown mustard (brassica juncea) plants during vegetative growth in an artificially contaminated soil and effect of rhamnolipid application. https:\/\/hdl.handle.net\/11441\/132481, (Final Degree Project). Universidad de Sevilla"},{"key":"1133_CR46","unstructured":"Navarro D (2018) Learning statistics with R: A tutorial for psychology students and other beginners (version 0.6). https:\/\/learningstatisticswithr.com\/"},{"issue":"3","key":"1133_CR47","doi-asserted-by":"publisher","first-page":"200","DOI":"10.1016\/j.jnutbio.2007.03.002","volume":"19","author":"YM Pacheco","year":"2008","unstructured":"Pacheco YM, L\u00f3pez S, Berm\u00fadez B et al (2008) A meal rich in oleic acid beneficially modulates postprandial sicam-1 and svcam-1 in normotensive and hypertensive hypertriglyceridemic subjects. J Nutr Biochem 19(3):200\u2013205. https:\/\/doi.org\/10.1016\/j.jnutbio.2007.03.002","journal-title":"J Nutr Biochem"},{"key":"1133_CR48","doi-asserted-by":"publisher","first-page":"287","DOI":"10.1016\/j.jss.2016.09.045","volume":"122","author":"JA Parejo","year":"2016","unstructured":"Parejo JA, S\u00e1nchez AB, Segura S et al (2016) Multi-objective test case prioritization in highly configurable systems: a case study. J Syst Softw 122:287\u2013310. https:\/\/doi.org\/10.1016\/j.jss.2016.09.045","journal-title":"J Syst Softw"},{"key":"1133_CR49","doi-asserted-by":"publisher","unstructured":"Plasquy E, Florido MC, Sola-Guirado RR et al (2021) Effects of a harvesting and conservation method for small producers on the quality of the produced olive oil. Agriculture 11. https:\/\/doi.org\/10.3390\/agriculture11050417","DOI":"10.3390\/agriculture11050417"},{"key":"1133_CR50","doi-asserted-by":"publisher","unstructured":"Reimanis D, Izurieta C, Luhr R, et\u00a0al (2014) A replication case study to measure the architectural quality of a commercial system. In: Proceedings of ESEM\u201914, ACM, pp 1\u20138, https:\/\/doi.org\/10.1145\/2652524.2652581","DOI":"10.1145\/2652524.2652581"},{"issue":"4","key":"1133_CR51","doi-asserted-by":"publisher","first-page":"2127","DOI":"10.1007\/s10664-016-9481-1","volume":"22","author":"M Riaz","year":"2017","unstructured":"Riaz M, King J, Slankas J et al (2017) Identifying the implied: findings from three differentiated replications on the use of security requirements templates. Empir Softw Eng 22(4):2127\u20132178. https:\/\/doi.org\/10.1007\/s10664-016-9481-1","journal-title":"Empir Softw Eng"},{"key":"1133_CR52","doi-asserted-by":"publisher","unstructured":"del R\u00edo-Ortega A, Resinas M, Dur\u00e1n A, et\u00a0al (2012) Defining process performance indicators by using templates and patterns. In: Proceedings of BPM\u201912, Springer, pp 223\u2013228, https:\/\/doi.org\/10.1007\/978-3-642-32885-5_18","DOI":"10.1007\/978-3-642-32885-5_18"},{"issue":"2","key":"1133_CR53","doi-asserted-by":"publisher","first-page":"159","DOI":"10.1080\/17517575.2013.867543","volume":"10","author":"A del R\u00edo-Ortega","year":"2016","unstructured":"del R\u00edo-Ortega A, Resinas M, Dur\u00e1n A et al (2016) Using templates and linguistic patterns to define process performance indicators. Enterp Inf Syst 10(2):159\u2013192. https:\/\/doi.org\/10.1080\/17517575.2013.867543","journal-title":"Enterp Inf Syst"},{"key":"1133_CR54","unstructured":"del R\u00edo P (2019) Effect of rhamnolipid JBR\u2013425 on the development of brassica juncea in urban garden soils in Sevilla. https:\/\/hdl.handle.net\/11441\/132478, (Final Degree Project) Universidad de Sevilla"},{"issue":"2","key":"1133_CR55","doi-asserted-by":"publisher","first-page":"131","DOI":"10.1007\/s10664-008-9102-8","volume":"14","author":"P Runeson","year":"2009","unstructured":"Runeson P, H\u00f6st M (2009) Guidelines for conducting and reporting case study research in software engineering. Empir Softw Eng 14(2):131. https:\/\/doi.org\/10.1007\/s10664-008-9102-8","journal-title":"Empir Softw Eng"},{"key":"1133_CR56","doi-asserted-by":"publisher","unstructured":"Runeson P, H\u00f6st M, Rainer A, et\u00a0al (2012) Case study research in software engineering. In: Guidelines and examples. Wiley Online Library, pp 109\u2013126, https:\/\/doi.org\/10.1002\/9781118181034.ch8","DOI":"10.1002\/9781118181034.ch8"},{"key":"1133_CR57","doi-asserted-by":"publisher","unstructured":"S\u00e1nchez AB, Segura S, Ruiz-Cort\u00e9s A (2014) A comparison of test case prioritization criteria for software product lines. In: Proceedings of ICST\u201914, pp 41\u201350, https:\/\/doi.org\/10.1109\/ICST.2014.15","DOI":"10.1109\/ICST.2014.15"},{"issue":"5","key":"1133_CR58","doi-asserted-by":"publisher","first-page":"566","DOI":"10.1109\/TSE.2018.2864633","volume":"46","author":"A Santos","year":"2018","unstructured":"Santos A, G\u00f3mez OS, Juristo N (2018) Analyzing families of experiments in software engineering: a systematic mapping study. IEEE Trans Softw Eng 46(5):566\u2013583. https:\/\/doi.org\/10.1109\/TSE.2018.2864633","journal-title":"IEEE Trans Softw Eng"},{"key":"1133_CR59","doi-asserted-by":"publisher","unstructured":"Santos AR, do Carmo Machado I, de Almeida ES et al (2019) Comparing the influence of using feature-oriented programming and conditional compilation on comprehending feature-oriented software. Empir Softw Eng 24(3):1226\u20131258. https:\/\/doi.org\/10.1007\/s10664-018-9658-x","DOI":"10.1007\/s10664-018-9658-x"},{"key":"1133_CR60","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2022.106870","author":"DA dos Santos","year":"2022","unstructured":"dos Santos DA, de Almeida ES, Ahmed I (2022) Investigating replication challenges through multiple replications of an experiment. Inf Softw Technol. https:\/\/doi.org\/10.1016\/j.infsof.2022.106870","journal-title":"Inf Softw Technol"},{"issue":"6","key":"1133_CR61","doi-asserted-by":"publisher","first-page":"1666","DOI":"10.1007\/s10664-014-9327-7","volume":"20","author":"G Scanniello","year":"2015","unstructured":"Scanniello G, Marcus A, Pascale D (2015) Link analysis algorithms for static concept location: an empirical assessment. Empir Softw Eng 20(6):1666\u20131720. https:\/\/doi.org\/10.1007\/s10664-014-9327-7","journal-title":"Empir Softw Eng"},{"key":"1133_CR62","doi-asserted-by":"publisher","unstructured":"Segura S, Dur\u00e1n A, Troya J, et\u00a0al (2017) A template-based approach to describing metamorphic relations. In: Proceedings of international workshop on metamorphic testing, pp 3\u20139, https:\/\/doi.org\/10.1109\/MET.2017.3","DOI":"10.1109\/MET.2017.3"},{"key":"1133_CR63","doi-asserted-by":"publisher","first-page":"120","DOI":"10.1016\/j.infsof.2018.01.006","volume":"99","author":"M Shepperd","year":"2018","unstructured":"Shepperd M, Ajienka N, Counsell S (2018) The role and value of replication in empirical software engineering results. Inf Softw Technol 99:120\u2013132. https:\/\/doi.org\/10.1016\/j.infsof.2018.01.006","journal-title":"Inf Softw Technol"},{"key":"1133_CR64","doi-asserted-by":"publisher","unstructured":"Shull F, Basili V, Carver J, et\u00a0al (2002) Replicating software engineering experiments: addressing the tacit knowledge problem. In: Proceedings of ISESE\u201902, pp 7\u201316, https:\/\/doi.org\/10.1109\/ISESE.2002.1166920","DOI":"10.1109\/ISESE.2002.1166920"},{"issue":"2","key":"1133_CR65","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1007\/s10664-008-9060-1","volume":"13","author":"F Shull","year":"2008","unstructured":"Shull F, Carver JC, Vegas S et al (2008) The role of replications in empirical software engineering. Empir Softw Eng 13(2):211\u2013218. https:\/\/doi.org\/10.1007\/s10664-008-9060-1","journal-title":"Empir Softw Eng"},{"key":"1133_CR66","doi-asserted-by":"publisher","unstructured":"Solari M (2013) Identifying experimental incidents in software engineering replications. In: Proceedings of ESEM\u201913, IEEE, pp 213\u2013222, https:\/\/doi.org\/10.1109\/ESEM.2013.26","DOI":"10.1109\/ESEM.2013.26"},{"key":"1133_CR67","unstructured":"Solari M, Vegas S (2006) Classifying and analysing replication packages for software engineering experimentation. In: Proceedings of WSESE\u201906 Amsterdam, Paises Bajos, pp 19\u201324"},{"key":"1133_CR68","doi-asserted-by":"publisher","first-page":"64","DOI":"10.1016\/j.infsof.2017.12.016","volume":"97","author":"M Solari","year":"2017","unstructured":"Solari M, Vegas S, Juristo N (2017) Content and structure of laboratory packages for software engineering experiments. Inf Softw Technol 97:64\u201379. https:\/\/doi.org\/10.1016\/j.infsof.2017.12.016","journal-title":"Inf Softw Technol"},{"issue":"9","key":"1133_CR69","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0162874","volume":"11","author":"JR Spence","year":"2016","unstructured":"Spence JR, Stanley DJ (2016) Prediction interval: what to expect when you\u2019re expecting\u2014a replication. PloS One 11(9):e0162,874. https:\/\/doi.org\/10.1371\/journal.pone.0162874","journal-title":"PloS One"},{"key":"1133_CR70","doi-asserted-by":"publisher","first-page":"2844","DOI":"10.1007\/s10664-020-09805-y","volume":"25","author":"S Vegas","year":"2020","unstructured":"Vegas S, Riofr\u00edo P, Marcos E et al (2020) On (mis) perceptions of testing effectiveness: an empirical study. Empir Softw Eng 25:2844\u20132896. https:\/\/doi.org\/10.1007\/s10664-020-09805-y","journal-title":"Empir Softw Eng"},{"issue":"1","key":"1133_CR71","doi-asserted-by":"publisher","first-page":"75","DOI":"10.2307\/25148625","volume":"28","author":"RH Von Alan","year":"2004","unstructured":"Von Alan RH, March ST, Park J et al (2004) Design science in information systems research. MIS Q 28(1):75\u2013105. https:\/\/doi.org\/10.2307\/25148625","journal-title":"MIS Q"},{"key":"1133_CR72","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-662-43839-8","volume-title":"Design science methodology for information systems and software engineering","author":"RJ Wieringa","year":"2014","unstructured":"Wieringa RJ (2014) Design science methodology for information systems and software engineering. Springer, Berlin"},{"key":"1133_CR73","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-29044-2","volume-title":"Experimentation in software engineering: an introduction","author":"C Wohlin","year":"2012","unstructured":"Wohlin C, Runeson P, H\u00f6st M et al (2012) Experimentation in software engineering: an introduction. Springer, Berlin"},{"key":"1133_CR74","series-title":"Design and methods","volume-title":"Case study research","author":"RK Yin","year":"2003","unstructured":"Yin RK et al (2003) Case study research. Design and methods. Thousand Oaks, Sage Publications"}],"updated-by":[{"DOI":"10.1007\/s00607-022-01143-9","type":"correction","label":"Correction","source":"publisher","updated":{"date-parts":[[2023,1,23]],"date-time":"2023-01-23T00:00:00Z","timestamp":1674432000000}}],"container-title":["Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00607-022-01133-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00607-022-01133-x\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00607-022-01133-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,5,11]],"date-time":"2023-05-11T03:31:55Z","timestamp":1683775915000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s00607-022-01133-x"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,12,3]]},"references-count":74,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2023,6]]}},"alternative-id":["1133"],"URL":"https:\/\/doi.org\/10.1007\/s00607-022-01133-x","relation":{},"ISSN":["0010-485X","1436-5057"],"issn-type":[{"value":"0010-485X","type":"print"},{"value":"1436-5057","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,12,3]]},"assertion":[{"value":"30 July 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"15 November 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"3 December 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 January 2023","order":4,"name":"change_date","label":"Change Date","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Correction","order":5,"name":"change_type","label":"Change Type","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"A Correction to this paper has been published:","order":6,"name":"change_details","label":"Change Details","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"https:\/\/doi.org\/10.1007\/s00607-022-01143-9","URL":"https:\/\/doi.org\/10.1007\/s00607-022-01143-9","order":7,"name":"change_details","label":"Change Details","group":{"name":"ArticleHistory","label":"Article History"}}]}}