{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,25]],"date-time":"2026-07-25T16:37:59Z","timestamp":1784997479794,"version":"3.55.0"},"publisher-location":"New York, NY, USA","reference-count":59,"publisher":"ACM","license":[{"start":{"date-parts":[[2011,5,21]],"date-time":"2011-05-21T00:00:00Z","timestamp":1305936000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2011,5,21]]},"DOI":"10.1145\/1985793.1985795","type":"proceedings-article","created":{"date-parts":[[2011,5,24]],"date-time":"2011-05-24T13:04:30Z","timestamp":1306242270000},"page":"1-10","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":740,"title":["A practical guide for using statistical tests to assess randomized algorithms in software engineering"],"prefix":"10.1145","author":[{"given":"Andrea","family":"Arcuri","sequence":"first","affiliation":[{"name":"Simula Research Laboratory, Lysaker, Norway"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lionel","family":"Briand","sequence":"additional","affiliation":[{"name":"Simula Research Laboratory & University of Oslo, Lysaker, Norway"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2011,5,21]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2008.73"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0950-5849(01)00193-8"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2009.52"},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/SSBSE.2009.16"},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.5555\/1928028.1928036"},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/1831708.1831736"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/CEC.2008.4630793"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0950-5849(01)00194-X"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1068009.1068189"},{"key":"e_1_3_2_1_10_1","volume-title":"Statistical power analysis for the behavioral sciences","author":"Cohen J.","year":"1988","unstructured":"J. Cohen . Statistical power analysis for the behavioral sciences , 1988 . J. Cohen. Statistical power analysis for the behavioral sciences, 1988."},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/314403.314414"},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1037\/0003-066X.37.5.553"},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.1984.5010257"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/SSBSE.2009.21"},{"key":"e_1_3_2_1_15_1","volume-title":"A systematic review of statistical power in software engineering experiments. Information and Software Technology (IST), 48(8):745--755","author":"Dyb\u00e5 T.","year":"2006","unstructured":"T. Dyb\u00e5 , V. Kampenes , and D. Sj\u00f8berg . A systematic review of statistical power in software engineering experiments. Information and Software Technology (IST), 48(8):745--755 , 2006 . T. Dyb\u00e5, V. Kampenes, and D. Sj\u00f8berg. A systematic review of statistical power in software engineering experiments. Information and Software Technology (IST), 48(8):745--755, 2006."},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1214\/09-SS051"},{"key":"e_1_3_2_1_17_1","volume-title":"An Introduction to Probability Theory and Its Applications","author":"Feller W.","year":"1968","unstructured":"W. Feller . An Introduction to Probability Theory and Its Applications , Vol. 1 . Wiley , 3 edition, 1968 . W. Feller. An Introduction to Probability Theory and Its Applications, Vol. 1. Wiley, 3 edition, 1968."},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2009.5070546"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.0030-1299.2004.13046.x"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/SSBSE.2009.25"},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/SSBSE.2009.26"},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/1065010.1065036"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.7326\/0003-4819-130-12-199906150-00008"},{"key":"e_1_3_2_1_24_1","volume-title":"Effect sizes for research: A broad practical approach","author":"Grissom R.","year":"2005","unstructured":"R. Grissom and J. Kim . Effect sizes for research: A broad practical approach . Lawrence Erlbaum , 2005 . R. Grissom and J. Kim. Effect sizes for research: A broad practical approach. Lawrence Erlbaum, 2005."},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2009.71"},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2009.5070541"},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2007.02.015"},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511811692"},{"key":"e_1_3_2_1_30_1","volume-title":"Systematic reviews to support evidence-based medicine: how to review and apply findings of healthcare research","author":"Khan K.","year":"2004","unstructured":"K. Khan , R. Kunz , J. Kleijnen , and G. Antes . Systematic reviews to support evidence-based medicine: how to review and apply findings of healthcare research . RSM Press , 2004 . K. Khan, R. Kunz, J. Kleijnen, and G. Antes. Systematic reviews to support evidence-based medicine: how to review and apply findings of healthcare research. RSM Press, 2004."},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/SSBSE.2009.20"},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2004.837108"},{"key":"e_1_3_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2009.5070521"},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/SSBSE.2009.11"},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2008.09.009"},{"key":"e_1_3_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1007\/b97377"},{"key":"e_1_3_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/SSBSE.2009.24"},{"key":"e_1_3_2_1_38_1","volume-title":"US Dept. Education","author":"Leech N.","year":"2002","unstructured":"N. Leech and A. Onwuegbuzie . A Call for Greater Use of Nonparametric Statistics. Technical report , US Dept. Education , 2002 . N. Leech and A. Onwuegbuzie. A Call for Greater Use of Nonparametric Statistics. Technical report, US Dept. Education, 2002."},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/SSBSE.2009.13"},{"key":"e_1_3_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2009.35"},{"key":"e_1_3_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.v14:2"},{"key":"e_1_3_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2009.5070552"},{"key":"e_1_3_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2006.31"},{"key":"e_1_3_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.5555\/211390"},{"key":"e_1_3_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1093\/beheco\/arh107"},{"key":"e_1_3_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1469-185X.2007.00027.x"},{"key":"e_1_3_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2008.80"},{"key":"e_1_3_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2002.806169"},{"key":"e_1_3_2_1_49_1","volume-title":"What's wrong with Bonferroni adjustments. British Medical Journal, 316:1236--1238","author":"Perneger T.","year":"1998","unstructured":"T. Perneger . What's wrong with Bonferroni adjustments. British Medical Journal, 316:1236--1238 , 1998 . T. Perneger. What's wrong with Bonferroni adjustments. British Medical Journal, 316:1236--1238, 1998."},{"key":"e_1_3_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2010.24"},{"key":"e_1_3_2_1_51_1","volume-title":"R: A Language and Environment for Statistical Computing","author":"Team R Development Core","year":"2008","unstructured":"R Development Core Team . R: A Language and Environment for Statistical Computing . R Foundation for Statistical Computing, Vienna, Austria , 2008 . ISBN 3--900051-07-0. R Development Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria, 2008. ISBN 3--900051-07-0."},{"key":"e_1_3_2_1_52_1","volume-title":"2 edition","author":"Rice J. A.","year":"1994","unstructured":"J. A. Rice . Mathematical Statistics and Data Analysis . Duxbury Press , 2 edition , 1994 . J. A. Rice. Mathematical Statistics and Data Analysis. Duxbury Press, 2 edition, 1994."},{"key":"e_1_3_2_1_53_1","volume-title":"IEEE transactions on Neural Networks, 5(1):96--101","author":"Rudolph G.","year":"1994","unstructured":"G. Rudolph . Convergence analysis of canonical genetic algorithms. IEEE transactions on Neural Networks, 5(1):96--101 , 1994 . G. Rudolph. Convergence analysis of canonical genetic algorithms. IEEE transactions on Neural Networks, 5(1):96--101, 1994."},{"key":"e_1_3_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1093\/beheco\/ark016"},{"key":"e_1_3_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1037\/0033-2909.111.2.352"},{"key":"e_1_3_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1109\/SSBSE.2009.18"},{"key":"e_1_3_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2009.5070526"},{"issue":"2","key":"e_1_3_2_1_58_1","first-page":"101","article-title":"A critique and improvement of the CL common language effect size statistics of McGraw and Wong","volume":"25","author":"Vargha A.","year":"2000","unstructured":"A. Vargha and H. D. Delaney . A critique and improvement of the CL common language effect size statistics of McGraw and Wong . Journal of Educational and Behavioral Statistics , 25 ( 2 ): 101 -- 132 , 2000 . A. Vargha and H. D. Delaney. A critique and improvement of the CL common language effect size statistics of McGraw and Wong. Journal of Educational and Behavioral Statistics, 25(2):101--132, 2000.","journal-title":"Journal of Educational and Behavioral Statistics"},{"key":"e_1_3_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2009.5070536"},{"key":"e_1_3_2_1_60_1","volume-title":"Substantially improving power and accuracy","author":"Wilcox R.","year":"2001","unstructured":"R. Wilcox . Fundamentals of modern statistical methods : Substantially improving power and accuracy . Springer Verlag , 2001 . R. Wilcox. Fundamentals of modern statistical methods: Substantially improving power and accuracy. Springer Verlag, 2001."}],"event":{"name":"ICSE11: International Conference on Software Engineering","location":"Waikiki, Honolulu HI USA","acronym":"ICSE11","sponsor":["SIGSOFT ACM Special Interest Group on Software Engineering","IEEE CS"]},"container-title":["Proceedings of the 33rd International Conference on Software Engineering"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1985793.1985795","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/1985793.1985795","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T11:06:29Z","timestamp":1750244789000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1985793.1985795"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2011,5,21]]},"references-count":59,"alternative-id":["10.1145\/1985793.1985795","10.1145\/1985793"],"URL":"https:\/\/doi.org\/10.1145\/1985793.1985795","relation":{},"subject":[],"published":{"date-parts":[[2011,5,21]]},"assertion":[{"value":"2011-05-21","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}