{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,28]],"date-time":"2026-02-28T21:14:48Z","timestamp":1772313288389,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":60,"publisher":"ACM","license":[{"start":{"date-parts":[[2019,4,15]],"date-time":"2019-04-15T00:00:00Z","timestamp":1555286400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2019,4,15]]},"DOI":"10.1145\/3319008.3319009","type":"proceedings-article","created":{"date-parts":[[2019,4,10]],"date-time":"2019-04-10T19:07:28Z","timestamp":1554923248000},"page":"134-143","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":13,"title":["Problems with Statistical Practice in Human-Centric Software Engineering Experiments"],"prefix":"10.1145","author":[{"given":"Barbara","family":"Kitchenham","sequence":"first","affiliation":[{"name":"School of Computing and Mathematics, Keele University, Keele, Staffordshire, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lech","family":"Madeyski","sequence":"additional","affiliation":[{"name":"Wroclaw University of Science and Technology, Wroclaw, Poland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Pearl","family":"Brereton","sequence":"additional","affiliation":[{"name":"School of Computing and Mathematics, Keele University, Keele, Staffordshire, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2019,4,15]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2012.27"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2014.09.002"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.799939"},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.2517-6161.1951.tb00067.x"},{"key":"e_1_3_2_1_5_1","volume-title":"Hunter","author":"Box George E.P.","year":"2005","unstructured":"George E.P. Box , J. Stuart Hunter , and William G . Hunter . 2005 . Statistics for Experimenters Design, Innovation and Discovery (second edition ed.). Wiley-InterScience , Hoboken, NJ, USA. George E.P. Box, J. Stuart Hunter, and William G. Hunter. 2005. Statistics for Experimenters Design, Innovation and Discovery (second edition ed.). Wiley-InterScience, Hoboken, NJ, USA."},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1521-4036(200001)42:1<17::AID-BIMJ17>3.0.CO;2-U"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1521-4036(200001)42:1<17::AID-BIMJ17>3.0.CO;2-U"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0378-3758(02)00269-0"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1038\/nrn3475"},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1037\/0033-2909.114.3.494"},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1037\/0033-2909.112.1.155"},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-009-9106-z"},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2011.07.002"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2005.130"},{"key":"e_1_3_2_1_15_1","first-page":"1","article-title":"The impact of an extreme observation in a paired samples design","volume":"14","author":"Derrick Ben","year":"2017","unstructured":"Ben Derrick , A. Broad , D. Toher , and P. White . 2017 . The impact of an extreme observation in a paired samples design . Metodoloki Zvezki -Advances in Methodology and Statistics 14 , 2 (2017), 1 -- 17 . Ben Derrick, A. Broad, D. Toher, and P. White. 2017. The impact of an extreme observation in a paired samples design. Metodoloki Zvezki -Advances in Methodology and Statistics 14, 2 (2017), 1--17.","journal-title":"Metodoloki Zvezki -Advances in Methodology and Statistics"},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2005.08.009"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1186\/1471-2288-12-78"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cct.2009.06.007"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1002\/sim.3561"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2012.07.043"},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-014-9354-4"},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2014.05.014"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2014.05.018"},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2013.05.003"},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1097\/EDE.0b013e31818131e7"},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2015.03.065"},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.22237\/jmasm\/1383278460"},{"key":"e_1_3_2_1_28_1","first-page":"53","article-title":"Testing normality in the multi-group problem: Is it a good practice","volume":"2","author":"Keselman H.J.","year":"2014","unstructured":"H.J. Keselman , Abdul R. Othman , and Rand Wilcox . 2014 . Testing normality in the multi-group problem: Is it a good practice ? Clinical Dermatology 2 , 1 (2014), 53 -- 65 . H.J. Keselman, Abdul R. Othman, and Rand Wilcox. 2014. Testing normality in the multi-group problem: Is it a good practice? Clinical Dermatology 2, 1 (2014), 53--65.","journal-title":"Clinical Dermatology"},{"key":"e_1_3_2_1_29_1","unstructured":"Barbara Kitchenham Lech Madeyski and Pearl Brereton. {n. d.}. Meta-analysis for Families of Experiments in Software Engineering: A Systematic Review and Reproducibility and Validity Assessment. Empirical Software Engineering ({n. d.}). In Review.  Barbara Kitchenham Lech Madeyski and Pearl Brereton. {n. d.}. Meta-analysis for Families of Experiments in Software Engineering: A Systematic Review and Reproducibility and Validity Assessment. Empirical Software Engineering ({n. d.}). In Review."},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-016-9437-5"},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.922713"},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.22237\/jmasm\/1478002140"},{"key":"e_1_3_2_1_33_1","volume-title":"Contributions to probability and statistics. Essays in honor of Harold Hotelling","author":"Levene H.","unstructured":"H. Levene . 1960. Robust tests for equality of variances . In Contributions to probability and statistics. Essays in honor of Harold Hotelling , I. Olkin (Ed.). University Press Stanford , USA , 279--292. H. Levene. 1960. Robust tests for equality of variances. In Contributions to probability and statistics. Essays in honor of Harold Hotelling, I. Olkin (Ed.). University Press Stanford, USA, 279--292."},{"key":"e_1_3_2_1_34_1","volume-title":"Test-Driven Development: An Empirical Evaluation of Agile Practice","author":"Madeyski Lech","unstructured":"Lech Madeyski . 2010. Test-Driven Development: An Empirical Evaluation of Agile Practice . Springer , (Heidelberg, London, New York). Lech Madeyski. 2010. Test-Driven Development: An Empirical Evaluation of Agile Practice. Springer, (Heidelberg, London, New York)."},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.3233\/JIFS-169146"},{"key":"e_1_3_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-017-9574-5"},{"key":"e_1_3_2_1_37_1","volume-title":"Designing Experiments and Analyzing Data A model Comparison Perspective","author":"Maxwell Scott E.","unstructured":"Scott E. Maxwell , Harold D. Delany , and Ken Kelley . 2018. Designing Experiments and Analyzing Data A model Comparison Perspective ( third edition ed.). Routledge , New York, NY, USA . Scott E. Maxwell, Harold D.Delany, and Ken Kelley. 2018. Designing Experiments and Analyzing Data A model Comparison Perspective (third edition ed.). Routledge, New York, NY, USA."},{"key":"e_1_3_2_1_38_1","volume-title":"Beyond ANOVA: Basics of Applied Statistics","author":"Miller Rupert G.","unstructured":"Rupert G. Miller . 1997. Beyond ANOVA: Basics of Applied Statistics . CRC Press , Boca Raton, FL, USA . Rupert G. Miller. 1997. Beyond ANOVA: Basics of Applied Statistics. CRC Press, Boca Raton, FL, USA."},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2015.12.056"},{"key":"e_1_3_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1037\/1082-989X.7.1.105"},{"key":"e_1_3_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1093\/beheco\/arh107"},{"key":"e_1_3_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.3102\/10769986022004389"},{"key":"e_1_3_2_1_43_1","first-page":"53","article-title":"Assessing Normality: Applications in Multi-Group Designs","volume":"9","author":"Othman Abdul R.","year":"2015","unstructured":"Abdul R. Othman , H.J. Keselman , and Rand Wilcox . 2015 . Assessing Normality: Applications in Multi-Group Designs . Malaysian Journal of Mathematical Sciences 9 , 1 (2015), 53 -- 65 . Abdul R. Othman, H.J. Keselman, and Rand Wilcox. 2015. Assessing Normality: Applications in Multi-Group Designs. Malaysian Journal of Mathematical Sciences 9, 1 (2015), 53--65.","journal-title":"Malaysian Journal of Mathematical Sciences"},{"key":"e_1_3_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0950-5849(03)00115-0"},{"key":"e_1_3_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00362-009-0224-x"},{"key":"e_1_3_2_1_46_1","first-page":"21","article-title":"Power comparisons of Shapiro-Wilk, Kolmogorov-Smirnov, Lilliefors and Anderson-Darling tests","volume":"2","author":"Razali Nornadiah Mohd","year":"2011","unstructured":"Nornadiah Mohd Razali and Yap Bee Wah . 2011 . Power comparisons of Shapiro-Wilk, Kolmogorov-Smirnov, Lilliefors and Anderson-Darling tests . Journal of Statistical Modeling and Analytics 2 , 1 (2011), 21 -- 33 . Nornadiah Mohd Razali and Yap Bee Wah. 2011. Power comparisons of Shapiro-Wilk, Kolmogorov-Smirnov, Lilliefors and Anderson-Darling tests. Journal of Statistical Modeling and Analytics 2, 1 (2011), 21--33.","journal-title":"Journal of Statistical Modeling and Analytics"},{"key":"e_1_3_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jcomdis.2015.08.002"},{"key":"e_1_3_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1186\/1471-2288-12-81"},{"key":"e_1_3_2_1_49_1","volume-title":"A sequentially rejective test procedure based on a modified Bonferroni inequality. Biometrika), 77","author":"Rom D.M.","year":"1990","unstructured":"D.M. Rom . 1990. A sequentially rejective test procedure based on a modified Bonferroni inequality. Biometrika), 77 ( 1990 ), 663--666. D.M. Rom. 1990. A sequentially rejective test procedure based on a modified Bonferroni inequality. Biometrika), 77 (1990), 663--666."},{"key":"e_1_3_2_1_50_1","volume-title":"Analyzing Families of Experiments in SE: A Systematic Mapping Study. CoRR abs\/1805.09009","author":"Santos Adrian","year":"2018","unstructured":"Adrian Santos , Omar S. G\u00f3mez , and Natalia Juristo . 2018. Analyzing Families of Experiments in SE: A Systematic Mapping Study. CoRR abs\/1805.09009 ( 2018 ), 1--18. arXiv:1805.09009 http:\/\/arxiv.org\/abs\/1805.09009 Adrian Santos, Omar S. G\u00f3mez, and Natalia Juristo. 2018. Analyzing Families of Experiments in SE: A Systematic Mapping Study. CoRR abs\/1805.09009 (2018), 1--18. arXiv:1805.09009 http:\/\/arxiv.org\/abs\/1805.09009"},{"key":"e_1_3_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/2491912"},{"key":"e_1_3_2_1_52_1","volume-title":"Cross-over Trials in Clinical Research","author":"Senn Stephen","unstructured":"Stephen Senn . 2002. Cross-over Trials in Clinical Research ( 2 nd ed.). John Wiley and Sons, Ltd. , Indianapolis, Indiana, USA. Stephen Senn. 2002. Cross-over Trials in Clinical Research (2nd ed.). John Wiley and Sons, Ltd., Indianapolis, Indiana, USA.","edition":"2"},{"key":"e_1_3_2_1_53_1","volume-title":"Cochran","author":"Snedecor George W.","year":"1980","unstructured":"George W. Snedecor and William G . Cochran . 1980 . Statistical Methods. The Iowa State University Press , Ames, Iowa, USA. George W. Snedecor and William G. Cochran. 1980. Statistical Methods. The Iowa State University Press, Ames, Iowa, USA."},{"key":"e_1_3_2_1_54_1","first-page":"1479","article-title":"A Direct Approach to False Discovery Rates. Journal of the Royal Statistical Society","volume":"64","author":"Storey John D.","year":"2002","unstructured":"John D. Storey . 2002 . A Direct Approach to False Discovery Rates. Journal of the Royal Statistical Society . Series B (Statistical Methodology) , 64 , 3 (2002), 1479 -- 1498 . John D. Storey. 2002. A Direct Approach to False Discovery Rates. Journal of the Royal Statistical Society. Series B (Statistical Methodology), 64, 3 (2002), 1479--498.","journal-title":"Series B (Statistical Methodology)"},{"key":"e_1_3_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2012.06.001"},{"key":"e_1_3_2_1_56_1","volume-title":"Testing for normality","author":"Thode H. C.","unstructured":"H. C. Thode . 2002. Testing for normality . Marcel Dekker, New York, NY , USA. H. C. Thode. 2002. Testing for normality. Marcel Dekker, New York, NY, USA."},{"key":"e_1_3_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2015.2467378"},{"key":"e_1_3_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/29.3-4.350"},{"key":"e_1_3_2_1_59_1","volume-title":"Introduction to Robust Estimation & Hypothesis Testing","author":"Wilcox Rand R.","unstructured":"Rand R. Wilcox . 2012. Introduction to Robust Estimation & Hypothesis Testing ( 3 rd edition ed.). Elsevier, Amsterdam , The Netherlands . Rand R. Wilcox. 2012. Introduction to Robust Estimation & Hypothesis Testing (3rd edition ed.). Elsevier, Amsterdam, The Netherlands.","edition":"3"},{"key":"e_1_3_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1207\/S15328031US0204_03"}],"event":{"name":"EASE '19: Evaluation and Assessment in Software Engineering","location":"Copenhagen Denmark","acronym":"EASE '19","sponsor":["IT University of Copenhagen"]},"container-title":["Proceedings of the Evaluation and Assessment on Software Engineering"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3319008.3319009","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3319008.3319009","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:38:20Z","timestamp":1750199900000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3319008.3319009"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,4,15]]},"references-count":60,"alternative-id":["10.1145\/3319008.3319009","10.1145\/3319008"],"URL":"https:\/\/doi.org\/10.1145\/3319008.3319009","relation":{},"subject":[],"published":{"date-parts":[[2019,4,15]]},"assertion":[{"value":"2019-04-15","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}