{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,20]],"date-time":"2026-02-20T23:57:49Z","timestamp":1771631869220,"version":"3.50.1"},"reference-count":26,"publisher":"Springer Science and Business Media LLC","issue":"1-2","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2012,2]]},"DOI":"10.1007\/s10664-011-9181-9","type":"journal-article","created":{"date-parts":[[2011,10,17]],"date-time":"2011-10-17T17:24:26Z","timestamp":1318872266000},"page":"75-89","source":"Crossref","is-referenced-by-count":80,"title":["On the reproducibility of empirical software engineering studies based on data retrieved from development repositories"],"prefix":"10.1007","volume":"17","author":[{"given":"Jes\u00fas M.","family":"Gonz\u00e1lez-Barahona","sequence":"first","affiliation":[]},{"given":"Gregorio","family":"Robles","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2011,10,18]]},"reference":[{"key":"9181_CR1","doi-asserted-by":"crossref","unstructured":"Barr ET, Bird C, Hyatt E, Menzies T, Robles G (2010) On the shoulders of giants. In: FoSER, pp\u00a023\u201328","DOI":"10.1145\/1882362.1882368"},{"issue":"4","key":"9181_CR2","doi-asserted-by":"crossref","first-page":"456","DOI":"10.1109\/32.799939","volume":"25","author":"VR Basili","year":"1999","unstructured":"Basili VR, Shull F, Lanubile F (1999) Building knowledge through families of experiments. IEEE Trans Softw Eng 25(4):456\u2013473","journal-title":"IEEE Trans Softw Eng"},{"key":"9181_CR3","unstructured":"Boetticher G, Menzies T, Ostrand T (2007) PROMISE repository of empirical software engineering data. Department of Computer Science, West Virginia University. http:\/\/promisedata.org\/"},{"key":"9181_CR4","unstructured":"de\u00a0Leeuw J (2001) Reproducible research. The bottom line. Technical report, UC Los Angeles: Department of Statistics, UCLA. http:\/\/escholarship.org\/uc\/item\/9050x4r4"},{"key":"9181_CR5","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1109\/MCSE.2009.15","volume":"11","author":"DL Donoho","year":"2009","unstructured":"Donoho DL, Maleki A, Rahman IU, Shahram M, Stodden V (2009) Reproducible research in computational harmonic analysis. Comput Sci Eng 11:8\u201318","journal-title":"Comput Sci Eng"},{"key":"9181_CR6","first-page":"82","volume-title":"Proceedings of the 2nd international conference on knowledge discovery and data mining, KDD-96, Portland (Oregon, USA)","author":"U Fayyad","year":"1996","unstructured":"Fayyad U, Piatetsky-Shapiro G, Smyth P (1996) Knowledge discovery and data mining: towards a unifying framework. In: Proceedings of the 2nd international conference on knowledge discovery and data mining, KDD-96, Portland (Oregon, USA). AAAI Press, Menlo Park, pp\u00a082\u201388"},{"key":"9181_CR7","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1109\/MCSE.2009.14","volume":"11","author":"S Fomel","year":"2009","unstructured":"Fomel S, Claerbout JF (2009) Guest editors\u2019 introduction: reproducible research. Comput Sci Eng 11:5\u20137","journal-title":"Comput Sci Eng"},{"issue":"1","key":"9181_CR8","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1198\/106186007X178663","volume":"16","author":"R Gentleman","year":"2007","unstructured":"Gentleman R, Lang DT (2007) Statistical analyses and reproducible research. J Comput Graph Stat 16(1):1\u201323","journal-title":"J Comput Graph Stat"},{"key":"9181_CR9","doi-asserted-by":"crossref","unstructured":"Germ\u00e1n DM (2004) Mining CVS repositories, the softChange experience. In: Proceedings of the international workshop on mining software repositories, Edinburghh, UK","DOI":"10.1049\/ic:20040469"},{"key":"9181_CR16","unstructured":"Gomez OS, Juristo N, Vegas S (2010) Replication, reproduction and re-analysis: three ways for verifying experimental findings. In: Proceedings of the 1st international workshop on replication in empirical software engineering research (RESER 2010), Cape Town, South Africa"},{"issue":"3","key":"9181_CR10","doi-asserted-by":"crossref","first-page":"262","DOI":"10.1007\/s10664-008-9100-x","volume":"14","author":"JM Gonz\u00e1lez-Barahona","year":"2009","unstructured":"Gonz\u00e1lez-Barahona JM, Robles G, Michlmayr M, Amor JJ, Germ\u00e1n DM (2009) Macro-level software evolution: a case study of a large software compilation. Empir Software Eng 14(3):262\u2013285","journal-title":"Empir Software Eng"},{"key":"9181_CR11","doi-asserted-by":"crossref","unstructured":"Hayes JH, Dekhtyar A, Sundaram S (2005) Text mining for software engineering: how analyst feedback impacts final results. In: Proceedings of the second international workshop on mining software repositories, St. Louis, USA","DOI":"10.1145\/1083142.1083153"},{"key":"9181_CR12","doi-asserted-by":"crossref","unstructured":"Herraiz I, Izquierdo-Cortazar D, Rivas-Hern\u00e1ndez F (2009) FLOSSMetrics: Free\/Libre\/Open Source Software Metrics. In: CSMR, pp 281\u2013284","DOI":"10.1109\/CSMR.2009.43"},{"key":"9181_CR13","doi-asserted-by":"crossref","unstructured":"Hothorn T, Leisch F (2011) Case studies in reproducibility. Brief Bioinform","DOI":"10.1093\/bib\/bbq084"},{"issue":"3","key":"9181_CR14","first-page":"17","volume":"1","author":"J Howison","year":"2006","unstructured":"Howison J, Conklin M, Crowston K (2006) FLOSSmole: a collaborative repository for FLOSS research data and analyses. IJITWE 1(3):17\u201326","journal-title":"IJITWE"},{"key":"9181_CR17","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1145\/1838687.1838698","volume":"35","author":"CD Knutson","year":"2010","unstructured":"Knutson CD, Krein JL , Prechelt L, Juristo N (2010) Report from the 1st international workshop on replication in empirical software engineering research (RESER 2010). SIGSOFT Softw Eng Notes 35:42\u201344","journal-title":"SIGSOFT Softw Eng Notes"},{"issue":"5","key":"9181_CR18","doi-asserted-by":"crossref","first-page":"833","DOI":"10.1002\/jae.1083","volume":"24","author":"R Koenker","year":"2009","unstructured":"Koenker R, Zeileis A (2009) On reproducible econometric research. J Appl Econ 24(5):833\u2013847","journal-title":"J Appl Econ"},{"key":"9181_CR19","doi-asserted-by":"crossref","first-page":"233","DOI":"10.1016\/j.infsof.2004.08.005","volume":"47","author":"J Miller","year":"2005","unstructured":"Miller J (2005) Replicating software engineering experiments: a poisoned chalice or the holy grail. Inf Softw Technol 47:233\u2013244","journal-title":"Inf Softw Technol"},{"key":"9181_CR20","doi-asserted-by":"crossref","unstructured":"Panjer LD (2007) Predicting eclipse bug lifetimes. In: Proceedings of the fourth international workshop on mining software repositories, MSR \u201907, p\u00a029","DOI":"10.1109\/MSR.2007.25"},{"key":"9181_CR21","doi-asserted-by":"crossref","unstructured":"Robles G (2010) Replicating MSR: A study of the potential replicability of papers published in the mining software repositories proceedings. In: 2010 7th IEEE working conference on mining software repositories (MSR), pp\u00a0171\u2013180","DOI":"10.1109\/MSR.2010.5463348"},{"key":"9181_CR22","unstructured":"Robles G, Germ\u00e1n DM (2010) Beyond replication: an example of the potential benefits of replicability in the mining of software repositories community. In: Proceedings of the 1st international workshop on replication in empirical software engineering sesearch (RESER 2010)"},{"issue":"9","key":"9181_CR23","doi-asserted-by":"crossref","first-page":"1233","DOI":"10.1016\/j.jss.2006.02.048","volume":"79","author":"G Robles","year":"2006","unstructured":"Robles G, Gonz\u00e1lez-Barahona JM, Merelo-Guerv\u00f3s JJ (2006) Beyond source code: the importance of other artifacts in software development (a case study). J Syst Softw 79(9):1233\u20131248","journal-title":"J Syst Softw"},{"issue":"1\u20132","key":"9181_CR24","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1023\/B:EMSE.0000013516.80487.33","volume":"9","author":"F Shull","year":"2004","unstructured":"Shull F, Mendon\u00e7a MG, Basili VR, Carver J, Maldonado JC, Fabbri SCPF, Travassos GH, de Oliveira MCF (2004) Knowledge-sharing issues in experimental software engineering. Empir Software Eng 9(1\u20132):111\u2013137","journal-title":"Empir Software Eng"},{"issue":"2","key":"9181_CR25","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1007\/s10664-008-9060-1","volume":"13","author":"FJ Shull","year":"2008","unstructured":"Shull FJ, Carver JC, Vegas S, Juristo N (2008) The role of replications in empirical software engineering. Empir Software Eng 13(2):211\u2013218","journal-title":"Empir Software Eng"},{"key":"9181_CR15","doi-asserted-by":"crossref","unstructured":"Vandewalle P, Barrenexea G, Jovanovic I, Ridolfi A, Vetterli M (2007) Experiences with reproducible research in various facets of signal processing research. In: Proceedings of the international conference on acoustics, speech and signal processing. ICASSP 2007, vol 4, pp IV-1253\u2013IV-1256","DOI":"10.1109\/ICASSP.2007.367304"},{"key":"9181_CR26","doi-asserted-by":"crossref","unstructured":"Vegas S, Juristo N, Moreno A, Solari M, Letelier P (2006) Analysis of the influence of communication between researchers on experiment replication. In: ISESE \u201906: Proceedings of the 2006 ACM\/IEEE international symposium on empirical software engineering, pp\u00a028\u201337","DOI":"10.1145\/1159733.1159741"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/www.springerlink.com\/index\/pdf\/10.1007\/s10664-011-9181-9","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2019,6,17]],"date-time":"2019-06-17T19:35:12Z","timestamp":1560800112000},"score":1,"resource":{"primary":{"URL":"http:\/\/link.springer.com\/10.1007\/s10664-011-9181-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2011,10,18]]},"references-count":26,"journal-issue":{"issue":"1-2","published-print":{"date-parts":[[2012,2]]}},"alternative-id":["9181"],"URL":"https:\/\/doi.org\/10.1007\/s10664-011-9181-9","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2011,10,18]]}}}