{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,13]],"date-time":"2026-01-13T21:01:16Z","timestamp":1768338076415,"version":"3.49.0"},"publisher-location":"New York, NY, USA","reference-count":48,"publisher":"ACM","license":[{"start":{"date-parts":[[2018,10,11]],"date-time":"2018-10-11T00:00:00Z","timestamp":1539216000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Spanish Ministry of Science and Innovation","award":["TIN2014-60490-P"],"award-info":[{"award-number":["TIN2014-60490-P"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2018,10,11]]},"DOI":"10.1145\/3239235.3239239","type":"proceedings-article","created":{"date-parts":[[2018,10,4]],"date-time":"2018-10-04T12:21:55Z","timestamp":1538655715000},"page":"1-10","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["Comparing techniques for aggregating interrelated replications in software engineering"],"prefix":"10.1145","author":[{"given":"Adrian","family":"Santos","sequence":"first","affiliation":[{"name":"University of Oulu, Finland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Natalia","family":"Juristo","sequence":"additional","affiliation":[{"name":"Universidad Polit\u00e9cnica de Madrid, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2018,10,11]]},"reference":[{"key":"e_1_3_2_1_1_1","first-page":"2011","volume":"25010","year":"2011","unstructured":"2011 . ISO\/IEC 25010 : 2011 . https:\/\/www.iso.Org\/obp\/ui\/#iso:std:iso-iec:25010:ed-1:v1:en 2011. ISO\/IEC 25010:2011. https:\/\/www.iso.Org\/obp\/ui\/#iso:std:iso-iec:25010:ed-1:v1:en","journal-title":"ISO\/IEC"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2012.12.017"},{"key":"e_1_3_2_1_3_1","volume-title":"Individual patient-versus group-level data meta-regressions for the investigation of treatment effect modifiers: ecological bias rears its ugly head. Statistics in medicine 21, 3","author":"Berlin Jesse A","year":"2002","unstructured":"Jesse A Berlin , Jill Santanna , Christopher H Schmid , Lynda A Szczech , and Harold I Feldman . 2002. Individual patient-versus group-level data meta-regressions for the investigation of treatment effect modifiers: ecological bias rears its ugly head. Statistics in medicine 21, 3 ( 2002 ), 371--387. Jesse A Berlin, Jill Santanna, Christopher H Schmid, Lynda A Szczech, and Harold I Feldman. 2002. Individual patient-versus group-level data meta-regressions for the investigation of treatment effect modifiers: ecological bias rears its ugly head. Statistics in medicine 21, 3 (2002), 371--387."},{"key":"e_1_3_2_1_4_1","volume-title":"Julian PT Higgins, and Hannah R Rothstein","author":"Borenstein Michael","year":"2011","unstructured":"Michael Borenstein , Larry V Hedges , Julian PT Higgins, and Hannah R Rothstein . 2011 . Introduction to Meta-Analysis. John Wiley & Sons . Michael Borenstein, Larry V Hedges, Julian PT Higgins, and Hannah R Rothstein. 2011. Introduction to Meta-Analysis. John Wiley & Sons."},{"key":"e_1_3_2_1_5_1","volume-title":"Applied mixed models in medicine","author":"Brown Helen","unstructured":"Helen Brown and Robin Prescott . 2014. Applied mixed models in medicine . John Wiley & Sons . Helen Brown and Robin Prescott. 2014. Applied mixed models in medicine. John Wiley & Sons."},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-013-9248-x"},{"key":"e_1_3_2_1_7_1","unstructured":"Robert Coe. 2002. It's the effect size stupid: What effect size is and why it is important. (2002).  Robert Coe. 2002. It's the effect size stupid: What effect size is and why it is important. (2002)."},{"key":"e_1_3_2_1_8_1","volume-title":"The earth is round (p<. 05). American psychologist 49, 12","author":"Cohen Jacob","year":"1994","unstructured":"Jacob Cohen . 1994. The earth is round (p<. 05). American psychologist 49, 12 ( 1994 ), 997. Jacob Cohen. 1994. The earth is round (p<. 05). American psychologist 49, 12 (1994), 997."},{"key":"e_1_3_2_1_9_1","volume-title":"The relative benefits of meta-analysis conducted with individual participant data versus aggregated data. Psychological methods 14, 2","author":"Cooper Harris","year":"2009","unstructured":"Harris Cooper and Erika A Patall . 2009. The relative benefits of meta-analysis conducted with individual participant data versus aggregated data. Psychological methods 14, 2 ( 2009 ), 165. Harris Cooper and Erika A Patall. 2009. The relative benefits of meta-analysis conducted with individual participant data versus aggregated data. Psychological methods 14, 2 (2009), 165."},{"key":"e_1_3_2_1_10_1","volume-title":"Understanding the new statistics: Effect sizes, confidence intervals, and meta-analysis","author":"Cumming Geoff","unstructured":"Geoff Cumming . 2013. Understanding the new statistics: Effect sizes, confidence intervals, and meta-analysis . Routledge . Geoff Cumming. 2013. Understanding the new statistics: Effect sizes, confidence intervals, and meta-analysis. Routledge."},{"key":"e_1_3_2_1_11_1","volume-title":"Gert Valkenhoef, Orestis Efthimiou, Noemi Hummel, Rolf HH Groenwold, and Johannes B Reitsma.","author":"Debray Thomas","year":"2015","unstructured":"Thomas Debray , Karel GM Moons , Gert Valkenhoef, Orestis Efthimiou, Noemi Hummel, Rolf HH Groenwold, and Johannes B Reitsma. 2015 . Get real in individual participant data (IPD) meta-analysis: a review of the methodology. Research synthesis methods 6, 4 (2015), 293--309. Thomas Debray, Karel GM Moons, Gert Valkenhoef, Orestis Efthimiou, Noemi Hummel, Rolf HH Groenwold, and Johannes B Reitsma. 2015. Get real in individual participant data (IPD) meta-analysis: a review of the methodology. Research synthesis methods 6, 4 (2015), 293--309."},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-016-9471-3"},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2005.08.009"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2005.37"},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1186\/1471-2288-12-78"},{"key":"e_1_3_2_1_16_1","volume-title":"Modeling site effects in the design and analysis of multi-site trials. The American journal of drug and alcohol abuse 37, 5","author":"Feaster Daniel J","year":"2011","unstructured":"Daniel J Feaster , Susan Mikulich-Gilbertson , and Ahnalee M Brincks . 2011. Modeling site effects in the design and analysis of multi-site trials. The American journal of drug and alcohol abuse 37, 5 ( 2011 ), 383--391. Daniel J Feaster, Susan Mikulich-Gilbertson, and Ahnalee M Brincks. 2011. Modeling site effects in the design and analysis of multi-site trials. The American journal of drug and alcohol abuse 37, 5 (2011), 383--391."},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2012.07.043"},{"key":"e_1_3_2_1_18_1","volume-title":"Discovering statistics using IBM SPSS statistics","author":"Field Andy","unstructured":"Andy Field . 2013. Discovering statistics using IBM SPSS statistics . Sage . Andy Field. 2013. Discovering statistics using IBM SPSS statistics. Sage."},{"key":"e_1_3_2_1_19_1","volume-title":"Discovering statistics using R","author":"Field Andy","unstructured":"Andy Field , Jeremy Miles , and Zo\u00eb Field . 2012. Discovering statistics using R . Sage publications. Andy Field, Jeremy Miles, and Zo\u00eb Field. 2012. Discovering statistics using R. Sage publications."},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2010.11.016"},{"key":"e_1_3_2_1_21_1","volume-title":"Meta-analysis of continuous outcome data from individual patients. Statistics in medicine 20, 15","author":"Higgins Julian PT","year":"2001","unstructured":"Julian PT Higgins , Anne Whitehead , Rebecca M Turner , Rumana Z Omar , and Simon G Thompson . 2001. Meta-analysis of continuous outcome data from individual patients. Statistics in medicine 20, 15 ( 2001 ), 2219--2241. Julian PT Higgins, Anne Whitehead, Rebecca M Turner, Rumana Z Omar, and Simon G Thompson. 2001. Meta-analysis of continuous outcome data from individual patients. Statistics in medicine 20, 15 (2001), 2219--2241."},{"key":"e_1_3_2_1_22_1","volume-title":"Multilevel analysis: Techniques and applications","author":"Hox Joop J","unstructured":"Joop J Hox , Mirjam Moerbeek , and Rens van de Schoot . 2010. Multilevel analysis: Techniques and applications . Routledge . Joop J Hox, Mirjam Moerbeek, and Rens van de Schoot. 2010. Multilevel analysis: Techniques and applications. Routledge."},{"key":"e_1_3_2_1_23_1","volume-title":"Basics of software engineering experimentation","author":"Juristo Natalia","unstructured":"Natalia Juristo and Ana M Moreno . 2011. Basics of software engineering experimentation . Springer Science & Business Media . Natalia Juristo and Ana M Moreno. 2011. Basics of software engineering experimentation. Springer Science & Business Media."},{"key":"e_1_3_2_1_24_1","volume-title":"Assessing potential sources of clustering in individually randomised trials. BMC medical research methodology 13, 1","author":"Kahan Brennan C","year":"2013","unstructured":"Brennan C Kahan and Tim P Morris . 2013. Assessing potential sources of clustering in individually randomised trials. BMC medical research methodology 13, 1 ( 2013 ), 58. Brennan C Kahan and Tim P Morris. 2013. Assessing potential sources of clustering in individually randomised trials. BMC medical research methodology 13, 1 (2013), 58."},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2007.02.015"},{"key":"e_1_3_2_1_26_1","first-page":"1","article-title":"Procedures for performing systematic reviews. Keele, UK","volume":"33","author":"Kitchenham Barbara","year":"2004","unstructured":"Barbara Kitchenham . 2004 . Procedures for performing systematic reviews. Keele, UK , Keele University 33 , 2004 (2004), 1 -- 26 . Barbara Kitchenham. 2004. Procedures for performing systematic reviews. Keele, UK, Keele University 33, 2004 (2004), 1--26.","journal-title":"Keele University"},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1093\/oxfordjournals.schbul.a033474"},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0895-4356(01)00414-0"},{"key":"e_1_3_2_1_29_1","volume-title":"Statistical principles for clinical trials (ICH E9): an introductory note on an international guideline. Statistics in medicine 18, 15","author":"Lewis John A","year":"1999","unstructured":"John A Lewis . 1999. Statistical principles for clinical trials (ICH E9): an introductory note on an international guideline. Statistics in medicine 18, 15 ( 1999 ), 1903--1942. John A Lewis. 1999. Statistical principles for clinical trials (ICH E9): an introductory note on an international guideline. Statistics in medicine 18, 15 (1999), 1903--1942."},{"key":"e_1_3_2_1_30_1","volume-title":"Accumulating evidence: Procedures for resolving contradictions among different research studies. Harvard educational review 41, 4","author":"Light Richard","year":"1971","unstructured":"Richard Light and Paul Smith . 1971. Accumulating evidence: Procedures for resolving contradictions among different research studies. Harvard educational review 41, 4 ( 1971 ), 429--471. Richard Light and Paul Smith. 1971. Accumulating evidence: Procedures for resolving contradictions among different research studies. Harvard educational review 41, 4 (1971), 429--471."},{"key":"e_1_3_2_1_31_1","volume-title":"The importance of the normality assumption in large public health data sets. Annual review of public health 23, 1","author":"Lumley Thomas","year":"2002","unstructured":"Thomas Lumley , Paula Diehr , Scott Emerson , and Lu Chen . 2002. The importance of the normality assumption in large public health data sets. Annual review of public health 23, 1 ( 2002 ), 151--169. Thomas Lumley, Paula Diehr, Scott Emerson, and Lu Chen. 2002. The importance of the normality assumption in large public health data sets. Annual review of public health 23, 1 (2002), 151--169."},{"key":"e_1_3_2_1_32_1","unstructured":"Richard McElreath. 2015. Statistical Rethinking. Texts in Statistical Science.  Richard McElreath. 2015. Statistical Rethinking. Texts in Statistical Science."},{"key":"e_1_3_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2010.06.003"},{"key":"e_1_3_2_1_34_1","volume-title":"Likert scales, levels of measurement and the laws of statistics. Advances in health sciences education 15, 5","author":"Norman Geoff","year":"2010","unstructured":"Geoff Norman . 2010. Likert scales, levels of measurement and the laws of statistics. Advances in health sciences education 15, 5 ( 2010 ), 625--632. Geoff Norman. 2010. Likert scales, levels of measurement and the laws of statistics. Advances in health sciences education 15, 5 (2010), 625--632."},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"crossref","unstructured":"Diana B Petitti. 2000. Meta-analysis decision analysis and cost-effectiveness analysis: methods for quantitative synthesis in medicine. Number 31. OUP USA.  Diana B Petitti. 2000. Meta-analysis decision analysis and cost-effectiveness analysis: methods for quantitative synthesis in medicine. Number 31. OUP USA.","DOI":"10.1093\/acprof:oso\/9780195133646.001.0001"},{"key":"e_1_3_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0950-5849(98)00101-3"},{"key":"e_1_3_2_1_37_1","volume-title":"Guidance on the conduct of narrative synthesis in systematic reviews. A product from the ESRC methods programme Version 1","author":"Popay Jennie","year":"2006","unstructured":"Jennie Popay , Helen Roberts , Amanda Sowden , Mark Petticrew , Lisa Arai , Mark Rodgers , Nicky Britten , Katrina Roen , and Steven Duffy . 2006. Guidance on the conduct of narrative synthesis in systematic reviews. A product from the ESRC methods programme Version 1 ( 2006 ), b92. Jennie Popay, Helen Roberts, Amanda Sowden, Mark Petticrew, Lisa Arai, Mark Rodgers, Nicky Britten, Katrina Roen, and Steven Duffy. 2006. Guidance on the conduct of narrative synthesis in systematic reviews. A product from the ESRC methods programme Version 1 (2006), b92."},{"key":"e_1_3_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2009.69"},{"key":"e_1_3_2_1_39_1","volume-title":"Analyzing Families of Experiments in SE: A Systematic Mapping Study. https:\/\/arxiv.org\/abs\/1805.09009. Online","author":"Santos Adrian","year":"2018","unstructured":"Adrian Santos , Omar G\u00f3mez , and Natalia Juristo . 2018. Analyzing Families of Experiments in SE: A Systematic Mapping Study. https:\/\/arxiv.org\/abs\/1805.09009. Online ; accessed 23 May 2018 . Adrian Santos, Omar G\u00f3mez, and Natalia Juristo. 2018. Analyzing Families of Experiments in SE: A Systematic Mapping Study. https:\/\/arxiv.org\/abs\/1805.09009. Online; accessed 23 May 2018."},{"key":"e_1_3_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2003.12.001"},{"key":"e_1_3_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1191\/1740774505cn087oa"},{"key":"e_1_3_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/FOSE.2007.30"},{"key":"e_1_3_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2015.3656"},{"key":"e_1_3_2_1_44_1","volume-title":"To IPD or not to IPD? Advantages and disadvantages of systematic reviews using individual patient data. Evaluation & the health professions 25, 1","author":"Stewart Lesley A","year":"2002","unstructured":"Lesley A Stewart and Jayne F Tierney . 2002. To IPD or not to IPD? Advantages and disadvantages of systematic reviews using individual patient data. Evaluation & the health professions 25, 1 ( 2002 ), 76--97. Lesley A Stewart and Jayne F Tierney. 2002. To IPD or not to IPD? Advantages and disadvantages of systematic reviews using individual patient data. Evaluation & the health professions 25, 1 (2002), 76--97."},{"key":"e_1_3_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-016-9490-0"},{"key":"e_1_3_2_1_46_1","doi-asserted-by":"crossref","unstructured":"Ronald L Wasserstein and Nicole A Lazar. 2016. The ASA's statement on p-values: context process and purpose.  Ronald L Wasserstein and Nicole A Lazar. 2016. The ASA's statement on p-values: context process and purpose.","DOI":"10.1080\/00031305.2016.1154108"},{"key":"e_1_3_2_1_47_1","volume-title":"Meta-analysis of controlled clinical trials","author":"Whitehead Anne","unstructured":"Anne Whitehead . 2002. Meta-analysis of controlled clinical trials . Vol. 7 . John Wiley & Sons . Anne Whitehead. 2002. Meta-analysis of controlled clinical trials. Vol. 7. John Wiley & Sons."},{"key":"e_1_3_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.5555\/2349018"}],"event":{"name":"ESEM '18: ACM \/ IEEE International Symposium on Empirical Software Engineering and Measurement","location":"Oulu Finland","acronym":"ESEM '18","sponsor":["SIGSOFT ACM Special Interest Group on Software Engineering","IEEE CS"]},"container-title":["Proceedings of the 12th ACM\/IEEE International Symposium on Empirical Software Engineering and Measurement"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3239235.3239239","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3239235.3239239","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T00:57:24Z","timestamp":1750208244000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3239235.3239239"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,10,11]]},"references-count":48,"alternative-id":["10.1145\/3239235.3239239","10.1145\/3239235"],"URL":"https:\/\/doi.org\/10.1145\/3239235.3239239","relation":{},"subject":[],"published":{"date-parts":[[2018,10,11]]},"assertion":[{"value":"2018-10-11","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}