{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,10]],"date-time":"2026-01-10T03:02:34Z","timestamp":1768014154166,"version":"3.49.0"},"reference-count":64,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2015,1,20]],"date-time":"2015-01-20T00:00:00Z","timestamp":1421712000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGOPS Oper. Syst. Rev."],"published-print":{"date-parts":[[2015,1,20]]},"abstract":"<jats:p>Being able to repeat experiments is considered a hallmark of the scientific method, used to confirm or refute hypotheses and previously obtained results. But this can take many forms, from precise repetition using the original experimental artifacts, to conceptual reproduction of the main experimental idea using new artifacts. Furthermore, the conclusions from previous work can also be corroborated using a different experimental methodology altogether. In order to promote a better understanding and use of such methodologies we propose precise definitions for different terms, and suggest when and why each should be used.<\/jats:p>","DOI":"10.1145\/2723872.2723875","type":"journal-article","created":{"date-parts":[[2015,1,28]],"date-time":"2015-01-28T14:05:51Z","timestamp":1422453951000},"page":"3-11","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":58,"title":["From Repeatability to Reproducibility and Corroboration"],"prefix":"10.1145","volume":"49","author":[{"given":"Dror G.","family":"Feitelson","sequence":"first","affiliation":[{"name":"The Hebrew University of Jerusalem, Jerusalem, Israel"}]}],"member":"320","published-online":{"date-parts":[[2015,1,20]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723872.2723879"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1126\/science.260.5110.895"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/2.982917"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.1987.232881"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.1986.6312975"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.799939"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/1151659.1159916"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1038\/483531a"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1167473.1167488"},{"key":"e_1_2_1_11_1","volume-title":"SIAM","author":"Bornemann F.","year":"2004"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/2.841785"},{"key":"e_1_2_1_14_1","volume-title":"USENIX Tech. Conf.","author":"Clark B.","year":"2004"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/359104.359106"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2451116.2451140"},{"key":"e_1_2_1_17_1","first-page":"23","volume-title":"Intl. Workshop Reproducible Research Methodologies","author":"de Oliveira A. B.","year":"2014"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2479871.2479892"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/1281700.1281710"},{"key":"e_1_2_1_20_1","volume-title":"4th Workshop Evaluation Methods for Machine Learning","author":"Drummond C.","year":"2009"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1037\/0003-066X.63.7.591"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2005.18"},{"key":"e_1_2_1_23_1","unstructured":"D. G. Feitelson \"Experimental computer science: The need for a cultural change\". URL http:\/\/www.cs.huji.ac.il\/\u00bffeit\/papers\/exp05.pdf 2005. D. G. Feitelson \"Experimental computer science: The need for a cultural change\". URL http:\/\/www.cs.huji.ac.il\/\u00bffeit\/papers\/exp05.pdf 2005."},{"key":"e_1_2_1_24_1","doi-asserted-by":"crossref","unstructured":"D. G. Feitelson Workload Modeling for Computer Systems Performance Evaluation. Cambridge University Press 2015. D. G. Feitelson Workload Modeling for Computer Systems Performance Evaluation. Cambridge University Press 2015.","DOI":"10.1017\/CBO9781139939690"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jpdc.2014.06.013"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1155\/2014\/797348"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2095050.2095100"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/1932682.1869536"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/2506164.2506175"},{"key":"e_1_2_1_30_1","volume-title":"John Wiley & Sons","author":"Hellman H.","year":"1998"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pmed.0020124"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/1297797.1297823"},{"key":"e_1_2_1_33_1","volume-title":"Addison Wesley","author":"Jacobson I.","year":"1999"},{"key":"e_1_2_1_34_1","unstructured":"R. Jain The Art of Computer Systems Performance Analysis. John Wiley & Sons 1991. R. Jain The Art of Computer Systems Performance Analysis. John Wiley & Sons 1991."},{"key":"e_1_2_1_35_1","first-page":"3","volume-title":"CHI2013 Workshop on Replication of HCI Research","author":"John B. E.","year":"2013"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-45143-3_12"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2012.113"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1002\/cpe.1677"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.5555\/645385.651507"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1027\/1864-9335\/a000178"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/1964897.1964905"},{"key":"e_1_2_1_42_1","volume-title":"Cambridge University Press","author":"Lilja D. J.","year":"2000"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/1368088.1368115"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/1374780.1374791"},{"key":"e_1_2_1_45_1","first-page":"279","volume-title":"USENIX Ann. Technical Conf.","author":"McVoy L.","year":"1996"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/232790.232791"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/359588.359602"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/2528521.1508275"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/1297797.1297820"},{"key":"e_1_2_1_50_1","unstructured":"J. Sambrook and D. W. Russell Molecular Cloning: A Laboratory Manual. Cold Spring Harbor Laboratory Press 3rd ed. 2001. J. Sambrook and D. W. Russell Molecular Cloning: A Laboratory Manual. Cold Spring Harbor Laboratory Press 3rd ed. 2001."},{"issue":"1","key":"e_1_2_1_51_1","first-page":"91","article-title":"Publication bias: The \"file-drawer\" problem in scientific inference","volume":"14","author":"Scargle J. D.","year":"2000","journal-title":"J. Sci. Explor."},{"key":"e_1_2_1_52_1","unstructured":"S. R. Schach Object-Oriented and Classical Software Engineering. McGraw-Hill 6th ed. 2005. S. R. Schach Object-Oriented and Classical Software Engineering. McGraw-Hill 6th ed. 2005."},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1037\/a0015108"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISESE.2002.1166920"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-008-9060-1"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1177\/1745691613514755"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1109\/MASCOTS.2007.58"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/2038642.2038650"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/1297797.1297822"},{"issue":"7","key":"e_1_2_1_60_1","first-page":"120","article-title":"Are comparative analyses worthwhile?","volume":"29","author":"Wartik S.","year":"1996","journal-title":"Computer"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1037\/0003-066X.61.7.726"},{"key":"e_1_2_1_62_1","doi-asserted-by":"crossref","unstructured":"R. Wilcox Introduction to Robust Estimation & Hypothesis Testing. Academic Press 3rd ed. 2012. R. Wilcox Introduction to Robust Estimation & Hypothesis Testing. Academic Press 3rd ed. 2012.","DOI":"10.1016\/B978-0-12-386983-8.00001-9"},{"key":"e_1_2_1_63_1","doi-asserted-by":"crossref","unstructured":"R. R. Wilcox Fundamentals of Modern Statistical Methods: Substantially Improving Power and Accuracy. Springer 2nd ed. 2010. R. R. Wilcox Fundamentals of Modern Statistical Methods: Substantially Improving Power and Accuracy. Springer 2nd ed. 2010.","DOI":"10.1007\/978-1-4419-5525-8"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/267895.267915"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1109\/TC.2005.184"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1002\/cpe.3240"}],"container-title":["ACM SIGOPS Operating Systems Review"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2723872.2723875","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2723872.2723875","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T06:16:43Z","timestamp":1750227403000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2723872.2723875"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2015,1,20]]},"references-count":64,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2015,1,20]]}},"alternative-id":["10.1145\/2723872.2723875"],"URL":"https:\/\/doi.org\/10.1145\/2723872.2723875","relation":{},"ISSN":["0163-5980"],"issn-type":[{"value":"0163-5980","type":"print"}],"subject":[],"published":{"date-parts":[[2015,1,20]]},"assertion":[{"value":"2015-01-20","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}