{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,8]],"date-time":"2025-12-08T22:26:09Z","timestamp":1765232769913,"version":"3.41.0"},"reference-count":40,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2019,8,17]],"date-time":"2019-08-17T00:00:00Z","timestamp":1566000000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Comput. Educ."],"published-print":{"date-parts":[[2019,12,31]]},"abstract":"<jats:p>As the societal demands for application and knowledge in computer science (CS) increase, CS student enrollment keeps growing rapidly around the world. By continuously improving the efficacy of computing education and providing guidelines for learning and teaching practice, computing education research plays a vital role in addressing both educational and societal challenges that emerge from the growth of CS students. Given the significant role of computing education research, it is important to ensure the reliability of studies in this field. The extent to which studies can be replicated in a field is one of the most important standards for reliability. Different fields have paid increasing attention to the replication rates of their studies, but the replication rate of computing education was never systematically studied. To fill this gap, this study investigated the replication rate of computing education between 2009 and 2018. We examined 2,269 published studies from three major conferences and two major journals in computing education, and found that the overall replication rate of computing education was 2.38%. This study demonstrated the need for more replication studies in computing education and discussed how to encourage replication studies through research initiatives and policy making.<\/jats:p>","DOI":"10.1145\/3345328","type":"journal-article","created":{"date-parts":[[2019,8,19]],"date-time":"2019-08-19T19:41:32Z","timestamp":1566243692000},"page":"1-18","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":38,"title":["A Systematic Investigation of Replications in Computing Education Research"],"prefix":"10.1145","volume":"19","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6361-5035","authenticated-orcid":false,"given":"Qiang","family":"Hao","sequence":"first","affiliation":[{"name":"Western Washington University, Bellingham, WA"}]},{"given":"David H.","family":"Smith IV","sequence":"additional","affiliation":[{"name":"Western Washington University, Bellingham, WA"}]},{"given":"Naitra","family":"Iriumi","sequence":"additional","affiliation":[{"name":"Western Washington University, Bellingham, WA"}]},{"given":"Michail","family":"Tsikerdekis","sequence":"additional","affiliation":[{"name":"Western Washington University, Bellingham, WA"}]},{"given":"Amy J.","family":"Ko","sequence":"additional","affiliation":[{"name":"University of Washington, Seattle, WA"}]}],"member":"320","published-online":{"date-parts":[[2019,8,17]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Retrieved","author":"Computing Research Association","year":"2017","unstructured":"Computing Research Association . 2017 . Generation CS: Computer Science Undergraduate Enrollments Surge Since 2006 . Retrieved July 24, 2019 from https:\/\/cra.org\/data\/generation-cs\/. Computing Research Association. 2017. Generation CS: Computer Science Undergraduate Enrollments Surge Since 2006. Retrieved July 24, 2019 from https:\/\/cra.org\/data\/generation-cs\/."},{"doi-asserted-by":"publisher","key":"e_1_2_1_2_1","DOI":"10.1145\/2602484"},{"doi-asserted-by":"publisher","key":"e_1_2_1_3_1","DOI":"10.1145\/2858796.2858799"},{"key":"e_1_2_1_4_1","volume-title":"Retrieved","author":"Google Inc.","year":"2019","unstructured":"Google Inc. and Gallup Inc. 2016. Trends in the State of Computer Science in U.S. K-12 Schools . Retrieved July 24, 2019 from https:\/\/services.google.com\/fh\/files\/misc\/trends-in-the-state-of-computer-science-report.pdf. Google Inc. and Gallup Inc. 2016. Trends in the State of Computer Science in U.S. K-12 Schools. Retrieved July 24, 2019 from https:\/\/services.google.com\/fh\/files\/misc\/trends-in-the-state-of-computer-science-report.pdf."},{"doi-asserted-by":"publisher","key":"e_1_2_1_5_1","DOI":"10.1145\/2668899"},{"volume-title":"The Demon-Haunted World: Science as a Candle in the Dark","author":"Sagan Carl","unstructured":"Carl Sagan . 1999. The Demon-Haunted World: Science as a Candle in the Dark . Ballantine Books , New York, NY . Carl Sagan. 1999. The Demon-Haunted World: Science as a Candle in the Dark. Ballantine Books, New York, NY.","key":"e_1_2_1_6_1"},{"doi-asserted-by":"publisher","key":"e_1_2_1_7_1","DOI":"10.1037\/a0015108"},{"key":"e_1_2_1_8_1","volume-title":"Ellis","author":"Glenn Begley C.","year":"2012","unstructured":"C. Glenn Begley and Lee M . Ellis . 2012 . Drug development: Raise standards for preclinical cancer research. Nature 483, 7391 (2012), 531. C. Glenn Begley and Lee M. Ellis. 2012. Drug development: Raise standards for preclinical cancer research. Nature 483, 7391 (2012), 531."},{"key":"e_1_2_1_9_1","volume-title":"Estimating the reproducibility of psychological science. Science 349, 6251","author":"Collaboration Open Science","year":"2015","unstructured":"Open Science Collaboration . 2015. Estimating the reproducibility of psychological science. Science 349, 6251 ( 2015 ), aac4716. Open Science Collaboration. 2015. Estimating the reproducibility of psychological science. Science 349, 6251 (2015), aac4716."},{"doi-asserted-by":"publisher","key":"e_1_2_1_10_1","DOI":"10.3102\/0013189X14545513"},{"doi-asserted-by":"publisher","key":"e_1_2_1_11_1","DOI":"10.1145\/2661136.2661156"},{"doi-asserted-by":"publisher","key":"e_1_2_1_12_1","DOI":"10.1145\/3230977.3230991"},{"key":"e_1_2_1_13_1","volume-title":"Bad copy. Nature 485, 7398","author":"Yong Ed","year":"2012","unstructured":"Ed Yong . 2012. Bad copy. Nature 485, 7398 ( 2012 ), 298. Ed Yong. 2012. Bad copy. Nature 485, 7398 (2012), 298."},{"unstructured":"What Works Clearinghouse. 2014. Procedures and Standards Handbook (Version 3.0). U.S. Department of Education: Washington DC.  What Works Clearinghouse. 2014. Procedures and Standards Handbook (Version 3.0). U.S. Department of Education: Washington DC.","key":"e_1_2_1_14_1"},{"doi-asserted-by":"publisher","key":"e_1_2_1_15_1","DOI":"10.1007\/s10664-008-9060-1"},{"key":"e_1_2_1_16_1","volume-title":"Retrieved","author":"Association for Computing Machinery.","year":"2019","unstructured":"Association for Computing Machinery. 2019 . Guide to Reviewing Papers . Retrieved July 24, 2019 from https:\/\/chi2019.acm.org\/guide-to-reviewing-papers\/. Association for Computing Machinery. 2019. Guide to Reviewing Papers. Retrieved July 24, 2019 from https:\/\/chi2019.acm.org\/guide-to-reviewing-papers\/."},{"doi-asserted-by":"publisher","key":"e_1_2_1_17_1","DOI":"10.1145\/2999541.2999554"},{"doi-asserted-by":"publisher","key":"e_1_2_1_18_1","DOI":"10.1001\/jama.294.2.218"},{"doi-asserted-by":"publisher","key":"e_1_2_1_19_1","DOI":"10.1037\/h0026141"},{"doi-asserted-by":"publisher","key":"e_1_2_1_20_1","DOI":"10.1177\/1745691612460686"},{"doi-asserted-by":"publisher","key":"e_1_2_1_21_1","DOI":"10.1177\/1745691612460688"},{"doi-asserted-by":"publisher","key":"e_1_2_1_22_1","DOI":"10.1145\/1584322.1584334"},{"doi-asserted-by":"publisher","key":"e_1_2_1_23_1","DOI":"10.1145\/971300.971391"},{"doi-asserted-by":"publisher","key":"e_1_2_1_24_1","DOI":"10.1145\/1113847.1113893"},{"doi-asserted-by":"publisher","key":"e_1_2_1_25_1","DOI":"10.1145\/3230977.3230980"},{"doi-asserted-by":"publisher","key":"e_1_2_1_26_1","DOI":"10.1145\/1404520.1404536"},{"doi-asserted-by":"publisher","key":"e_1_2_1_27_1","DOI":"10.1145\/1839594.1839597"},{"doi-asserted-by":"publisher","key":"e_1_2_1_28_1","DOI":"10.1016\/j.jbusres.2006.12.003"},{"doi-asserted-by":"publisher","key":"e_1_2_1_29_1","DOI":"10.1371\/journal.pmed.0020124"},{"doi-asserted-by":"publisher","key":"e_1_2_1_30_1","DOI":"10.1145\/2534973"},{"doi-asserted-by":"publisher","key":"e_1_2_1_31_1","DOI":"10.5555\/1151869.1151875"},{"doi-asserted-by":"publisher","key":"e_1_2_1_32_1","DOI":"10.5014\/ajot.45.3.214"},{"doi-asserted-by":"publisher","key":"e_1_2_1_33_1","DOI":"10.1016\/j.ijnurstu.2010.06.004"},{"key":"e_1_2_1_34_1","volume-title":"Technical Symposium. 2019. SIGCSE 2019\u2014Call For Participation. Retrieved","author":"SIGCSE","year":"2019","unstructured":"SIGCSE Technical Symposium. 2019. SIGCSE 2019\u2014Call For Participation. Retrieved July 24, 2019 from https:\/\/sigcse2019.sigcse.org\/info\/cfp.html#papers. SIGCSE Technical Symposium. 2019. SIGCSE 2019\u2014Call For Participation. Retrieved July 24, 2019 from https:\/\/sigcse2019.sigcse.org\/info\/cfp.html#papers."},{"doi-asserted-by":"publisher","key":"e_1_2_1_35_1","DOI":"10.1177\/1745691612459058"},{"doi-asserted-by":"publisher","key":"e_1_2_1_36_1","DOI":"10.1177\/1932202X14532177"},{"key":"e_1_2_1_37_1","volume-title":"Retrieved","author":"Association for Computing Machinery.","year":"2018","unstructured":"Association for Computing Machinery. 2018 . Artifact Review and Badging . Retrieved July 24, 2019 from https:\/\/www.acm.org\/publications\/policies\/artifact-review-badging. Association for Computing Machinery. 2018. Artifact Review and Badging. Retrieved July 24, 2019 from https:\/\/www.acm.org\/publications\/policies\/artifact-review-badging."},{"key":"e_1_2_1_38_1","volume-title":"Computing Education Research Conference. 2018","author":"International","year":"2019","unstructured":"International Computing Education Research Conference. 2018 . ICER\u201918\u2014Call for Participation. Retrieved July 24, 2019 from https:\/\/icer.acm.org\/icer-2018\/icer18-call-for-participation\/. International Computing Education Research Conference. 2018. ICER\u201918\u2014Call for Participation. Retrieved July 24, 2019 from https:\/\/icer.acm.org\/icer-2018\/icer18-call-for-participation\/."},{"doi-asserted-by":"publisher","key":"e_1_2_1_39_1","DOI":"10.3102\/0013189X035002013"},{"doi-asserted-by":"publisher","key":"e_1_2_1_40_1","DOI":"10.2466\/pr0.1975.36.2.587"}],"container-title":["ACM Transactions on Computing Education"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3345328","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3345328","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:13:19Z","timestamp":1750201999000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3345328"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,8,17]]},"references-count":40,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2019,12,31]]}},"alternative-id":["10.1145\/3345328"],"URL":"https:\/\/doi.org\/10.1145\/3345328","relation":{},"ISSN":["1946-6226"],"issn-type":[{"type":"electronic","value":"1946-6226"}],"subject":[],"published":{"date-parts":[[2019,8,17]]},"assertion":[{"value":"2018-11-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-07-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-08-17","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}