{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,13]],"date-time":"2026-06-13T07:21:57Z","timestamp":1781335317056,"version":"3.54.1"},"reference-count":42,"publisher":"Association for Computing Machinery (ACM)","issue":"8","license":[{"start":{"date-parts":[[2020,7,22]],"date-time":"2020-07-22T00:00:00Z","timestamp":1595376000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Commun. ACM"],"published-print":{"date-parts":[[2020,7,22]]},"abstract":"<jats:p>Research replication only works if there is confidence built into the results.<\/jats:p>","DOI":"10.1145\/3360311","type":"journal-article","created":{"date-parts":[[2020,7,22]],"date-time":"2020-07-22T22:14:24Z","timestamp":1595456064000},"page":"70-79","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":133,"title":["Threats of a replication crisis in empirical computer science"],"prefix":"10.1145","volume":"63","author":[{"given":"Andy","family":"Cockburn","sequence":"first","affiliation":[{"name":"University of Cantebury, Christchurch, New Zealand"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pierre","family":"Dragicevic","sequence":"additional","affiliation":[{"name":"Inria, Orsay, France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lonni","family":"Besan\u00e7on","sequence":"additional","affiliation":[{"name":"Link\u00f6ping University, Link\u00f6ping, Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Carl","family":"Gutwin","sequence":"additional","affiliation":[{"name":"University of Saskatchewan, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2020,7,22]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.7717\/peerj.3544"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1038\/533452a"},{"key":"e_1_2_1_3_1","volume-title":"Raise standards for preclinical cancer research. Nature 483, 7391","author":"Begley C. G.","year":"2012","unstructured":"Begley , C. G. , and Ellis , L. M . Raise standards for preclinical cancer research. Nature 483, 7391 ( 2012 ), 531. Begley, C. G., and Ellis, L. M. Raise standards for preclinical cancer research. Nature 483, 7391 (2012), 531."},{"key":"e_1_2_1_4_1","volume-title":"PsyArXiv (July 22","author":"Benjamin D.","year":"2017","unstructured":"Benjamin , D. et al. Redefine statistical significance . PsyArXiv (July 22 , 2017 ). Benjamin, D. et al. Redefine statistical significance. PsyArXiv (July 22, 2017)."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2994031"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1080\/1369118X.2012.678878"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0149144"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173715"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2812803"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0197440"},{"key":"e_1_2_1_11_1","series-title":"Multivariate applications series","volume-title":"Confidence Intervals, and Meta-analysis","author":"Cumming G.","year":"2012","unstructured":"Cumming , G. Understanding the New Statistics: Effect Sizes , Confidence Intervals, and Meta-analysis . Multivariate applications series . Routledge , 2012 . Cumming, G. Understanding the New Statistics: Effect Sizes, Confidence Intervals, and Meta-analysis. Multivariate applications series. Routledge, 2012."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1177\/0956797613504966"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/359015.359016"},{"key":"e_1_2_1_14_1","first-page":"330","article-title":"Fair statistical communication in HCI. Modern Statistical Methods for HCI, J. Robertson and M. Kaptein, eds","volume":"291","author":"Dragicevic P","year":"2016","unstructured":"Dragicevic , P . Fair statistical communication in HCI. Modern Statistical Methods for HCI, J. Robertson and M. Kaptein, eds . Springer International Publishing , 2016 , 291 -- 330 . Dragicevic, P. Fair statistical communication in HCI. Modern Statistical Methods for HCI, J. Robertson and M. Kaptein, eds. Springer International Publishing, 2016, 291--330.","journal-title":"Springer International Publishing"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1177\/0261927X08317947"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1126\/science.1255484"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1002\/sim.4780110705"},{"key":"e_1_2_1_18_1","unstructured":"Gundersen O.E. and Kjensmo S. State of the art: Reproducibility in artificial intelligence. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence the 30th Innovative Applications of Artificial Intelligence and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence. New Orleans LA USA Feb. 2--7 2018) 1644--1651.  Gundersen O.E. and Kjensmo S. State of the art: Reproducibility in artificial intelligence. In Proceedings of the 32 nd AAAI Conference on Artificial Intelligence the 30 th Innovative Applications of Artificial Intelligence and the 8 th AAAI Symposium on Educational Advances in Artificial Intelligence. New Orleans LA USA Feb. 2--7 2018) 1644--1651."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pmed.0020124"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1177\/0956797611430953"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0132382"},{"key":"e_1_2_1_22_1","volume-title":"Proceedings of the 2016 ACM CHI Conference on Human Factors in Computing Systems, 4521--4532","author":"Kay M.","unstructured":"Kay , M. , Nelson , G.L. , and Hekler , E.B . Researcher-centered design of statistics: Why Bayesian statistics better fit the culture and incentives of HCI . In Proceedings of the 2016 ACM CHI Conference on Human Factors in Computing Systems, 4521--4532 . Kay, M., Nelson, G.L., and Hekler, E.B. Researcher-centered design of statistics: Why Bayesian statistics better fit the culture and incentives of HCI. In Proceedings of the 2016 ACM CHI Conference on Human Factors in Computing Systems, 4521--4532."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1207\/s15327957pspr0203_4"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.31219\/osf.io\/f8qey"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2658987"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.3758\/s13423-016-1221-4"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1126\/science.aal3618"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1080\/00031305.2018.1527253"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1093\/pan\/mps022"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1037\/1089-2680.2.2.175"},{"key":"e_1_2_1_31_1","volume-title":"Estimating the reproducibility of psychological science. Science 349, 6251","author":"Open Science Collaboration","year":"2015","unstructured":"Open Science Collaboration and others. Estimating the reproducibility of psychological science. Science 349, 6251 ( 2015 ), aac4716. Open Science Collaboration and others. Estimating the reproducibility of psychological science. Science 349, 6251 (2015), aac4716."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1177\/1745691612465253"},{"key":"e_1_2_1_33_1","volume-title":"instead. Preprint","author":"Perezgonzalez J.D.","year":"2017","unstructured":"Perezgonzalez , J.D. , and Frias-Navarro , D. Retract 0. 005 and propose using JASP , instead. Preprint , 2017 ; https:\/\/psyarxiv.com\/t2fn8. Perezgonzalez, J.D., and Frias-Navarro, D. Retract 0.005 and propose using JASP, instead. Preprint, 2017; https:\/\/psyarxiv.com\/t2fn8."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1001\/jama.292.11.1359"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1037\/0033-2909.86.3.638"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.3758\/BF03212944"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyg.2015.00245"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1177\/0956797611417632"},{"key":"e_1_2_1_39_1","volume-title":"Posterior-hacking: Selective reporting invalidates Bayesian results also. SSRN","author":"Simonsohn U.","year":"2014","unstructured":"Simonsohn , U. Posterior-hacking: Selective reporting invalidates Bayesian results also. SSRN ( 2014 ). Simonsohn, U. Posterior-hacking: Selective reporting invalidates Bayesian results also. SSRN (2014)."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1037\/a0033242"},{"key":"e_1_2_1_41_1","first-page":"285","article-title":"Publication decisions and their possible effects on inferences drawn from tests of significance---or vice versa","volume":"54","author":"Sterling T.D","year":"1959","unstructured":"Sterling , T.D . Publication decisions and their possible effects on inferences drawn from tests of significance---or vice versa . J. American Statistical Assoc. 54 , 285 ( 1959 ), 30--34. Sterling, T.D. Publication decisions and their possible effects on inferences drawn from tests of significance---or vice versa. J. American Statistical Assoc. 54, 285 (1959), 30--34.","journal-title":"J. American Statistical Assoc."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1080\/01973533.2015.1012991"}],"container-title":["Communications of the ACM"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3360311","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3360311","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:44:29Z","timestamp":1750203869000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3360311"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,7,22]]},"references-count":42,"journal-issue":{"issue":"8","published-print":{"date-parts":[[2020,7,22]]}},"alternative-id":["10.1145\/3360311"],"URL":"https:\/\/doi.org\/10.1145\/3360311","relation":{},"ISSN":["0001-0782","1557-7317"],"issn-type":[{"value":"0001-0782","type":"print"},{"value":"1557-7317","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,7,22]]},"assertion":[{"value":"2020-07-22","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}