{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,6]],"date-time":"2026-07-06T11:23:30Z","timestamp":1783337010482,"version":"3.54.6"},"publisher-location":"New York, NY, USA","reference-count":77,"publisher":"ACM","license":[{"start":{"date-parts":[[2022,8,3]],"date-time":"2022-08-03T00:00:00Z","timestamp":1659484800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,8,3]]},"DOI":"10.1145\/3501385.3543971","type":"proceedings-article","created":{"date-parts":[[2022,8,4]],"date-time":"2022-08-04T04:08:58Z","timestamp":1659586138000},"page":"309-322","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":9,"title":["Launching Registered Report Replications in Computer Science Education Research"],"prefix":"10.1145","author":[{"given":"Neil C. C.","family":"Brown","sequence":"first","affiliation":[{"name":"King's College London, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Eva","family":"Marinus","sequence":"additional","affiliation":[{"name":"P\u00e4dagogische Hochschule Schwyz, Switzerland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Aleata","family":"Hubbard Cheuoua","sequence":"additional","affiliation":[{"name":"WestEd, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2022,8,3]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/2999541.2999554"},{"key":"e_1_3_2_1_2_1","volume-title":"Marcel A. L.\u00a0M. van Assen, Elise Crompvoets, How\u00a0Hwee Ong, and Jelte\u00a0M. Wicherts.","author":"Bakker Marjan","year":"2020","unstructured":"Marjan Bakker , Coosje L.\u00a0S. Veldkamp , Olmo\u00a0 R. van\u00a0den Akker , Marcel A. L.\u00a0M. van Assen, Elise Crompvoets, How\u00a0Hwee Ong, and Jelte\u00a0M. Wicherts. 2020 . Recommendations in pre-registrations and internal review board proposals promote formal power analyses but do not increase sample size. PLOS ONE 15, 7 (07 2020), 1\u201315. https:\/\/doi.org\/10.1371\/journal.pone.0236079 10.1371\/journal.pone.0236079 Marjan Bakker, Coosje L.\u00a0S. Veldkamp, Olmo\u00a0R. van\u00a0den Akker, Marcel A. L.\u00a0M. van Assen, Elise Crompvoets, How\u00a0Hwee Ong, and Jelte\u00a0M. Wicherts. 2020. Recommendations in pre-registrations and internal review board proposals promote formal power analyses but do not increase sample size. PLOS ONE 15, 7 (07 2020), 1\u201315. https:\/\/doi.org\/10.1371\/journal.pone.0236079"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1176\/appi.ajp.2018.18070881"},{"key":"e_1_3_2_1_4_1","unstructured":"Richard Bornat. 2014. Camels and humps: a retraction. (2014). http:\/\/eis.sla.mdx.ac.uk\/staffpages\/r_bornat\/papers\/camel_hump_retraction.pdf  Richard Bornat. 2014. Camels and humps: a retraction. (2014). http:\/\/eis.sla.mdx.ac.uk\/staffpages\/r_bornat\/papers\/camel_hump_retraction.pdf"},{"key":"e_1_3_2_1_5_1","volume-title":"Proceedings of the Tenth Conference on Australasian Computing Education -","volume":"78","author":"Bornat Richard","year":"2008","unstructured":"Richard Bornat , Saeed Dehnadi , and Simon. 2008 . Mental Models, Consistency and Programming Aptitude . In Proceedings of the Tenth Conference on Australasian Computing Education - Volume 78 (Wollongong, NSW, Australia) (ACE \u201908). Australian Computer Society, Inc., AUS, 53\u201361. Richard Bornat, Saeed Dehnadi, and Simon. 2008. Mental Models, Consistency and Programming Aptitude. In Proceedings of the Tenth Conference on Australasian Computing Education - Volume 78 (Wollongong, NSW, Australia) (ACE \u201908). Australian Computer Society, Inc., AUS, 53\u201361."},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1098\/rsos.200048"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.urology.2018.11.034"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3230977.3230991"},{"key":"e_1_3_2_1_9_1","volume-title":"Personal Motivations and Systemic Incentives: Scientists on Questionable Research Practices. Science and Engineering Ethics 26, 3 (01","author":"Bruton V.","year":"2020","unstructured":"Samuel\u00a0 V. Bruton , Mary Medlin , Mitch Brown , and Donald\u00a0 F. Sacco . 2020. Personal Motivations and Systemic Incentives: Scientists on Questionable Research Practices. Science and Engineering Ethics 26, 3 (01 Jun 2020 ), 1531\u20131547. https:\/\/doi.org\/10.1007\/s11948-020-00182-9 10.1007\/s11948-020-00182-9 Samuel\u00a0V. Bruton, Mary Medlin, Mitch Brown, and Donald\u00a0F. Sacco. 2020. Personal Motivations and Systemic Incentives: Scientists on Questionable Research Practices. Science and Engineering Ethics 26, 3 (01 Jun 2020), 1531\u20131547. https:\/\/doi.org\/10.1007\/s11948-020-00182-9"},{"key":"e_1_3_2_1_10_1","volume-title":"Assessing Individual Contributions to Software Engineering Projects with Git Logs and User Stories","author":"Buffardi Kevin","unstructured":"Kevin Buffardi . 2020. Assessing Individual Contributions to Software Engineering Projects with Git Logs and User Stories . Association for Computing Machinery , New York, NY, USA , 650\u2013656. https:\/\/doi.org\/10.1145\/3328778.3366948 10.1145\/3328778.3366948 Kevin Buffardi. 2020. Assessing Individual Contributions to Software Engineering Projects with Git Logs and User Stories. Association for Computing Machinery, New York, NY, USA, 650\u2013656. https:\/\/doi.org\/10.1145\/3328778.3366948"},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/1269900.1268845"},{"key":"e_1_3_2_1_12_1","volume-title":"Chambers and Loukia Tzavella","author":"D.","year":"2022","unstructured":"Christopher\u00a0 D. Chambers and Loukia Tzavella . 2022 . The past, present and future of Registered Reports. Nature Human Behaviour 6, 1 (01 Jan 2022), 29\u201342. https:\/\/doi.org\/10.1038\/s41562-021-01193-7 10.1038\/s41562-021-01193-7 Christopher\u00a0D. Chambers and Loukia Tzavella. 2022. The past, present and future of Registered Reports. Nature Human Behaviour 6, 1 (01 Jan 2022), 29\u201342. https:\/\/doi.org\/10.1038\/s41562-021-01193-7"},{"key":"e_1_3_2_1_13_1","volume-title":"Supporting a Culture of Replication: An Examination of Education and Special Education Research Grants Funded by the Institute of Education Sciences. Educational Researcher 47, 9 (Dec","author":"Chhin S.","year":"2018","unstructured":"Christina\u00a0 S. Chhin , Katherine\u00a0 A. Taylor , and Wendy\u00a0 S. Wei . 2018. Supporting a Culture of Replication: An Examination of Education and Special Education Research Grants Funded by the Institute of Education Sciences. Educational Researcher 47, 9 (Dec . 2018 ), 594\u2013605. https:\/\/doi.org\/10.3102\/0013189X18788047 Publisher : American Educational Research Association . 10.3102\/0013189X18788047 Christina\u00a0S. Chhin, Katherine\u00a0A. Taylor, and Wendy\u00a0S. Wei. 2018. Supporting a Culture of Replication: An Examination of Education and Special Education Research Grants Funded by the Institute of Education Sciences. Educational Researcher 47, 9 (Dec. 2018), 594\u2013605. https:\/\/doi.org\/10.3102\/0013189X18788047 Publisher: American Educational Research Association."},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1046\/j.1365-2648.1998.00651.x"},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3360311"},{"key":"e_1_3_2_1_16_1","volume-title":"Does health informatics have a replication crisis?Journal of the American Medical Informatics Association 25, 8 (04","author":"Coiera Enrico","year":"2018","unstructured":"Enrico Coiera , Elske Ammenwerth , Andrew Georgiou , and Farah Magrabi . 2018. Does health informatics have a replication crisis?Journal of the American Medical Informatics Association 25, 8 (04 2018 ), 963\u2013968. https:\/\/doi.org\/10.1093\/jamia\/ocy028 arXiv:https:\/\/academic.oup.com\/jamia\/article-pdf\/25\/8\/963\/34150203\/ocy028.pdf 10.1093\/jamia Enrico Coiera, Elske Ammenwerth, Andrew Georgiou, and Farah Magrabi. 2018. Does health informatics have a replication crisis?Journal of the American Medical Informatics Association 25, 8 (04 2018), 963\u2013968. https:\/\/doi.org\/10.1093\/jamia\/ocy028 arXiv:https:\/\/academic.oup.com\/jamia\/article-pdf\/25\/8\/963\/34150203\/ocy028.pdf"},{"key":"e_1_3_2_1_17_1","volume-title":"Estimating the reproducibility of psychological science. Science 349, 6251","author":"Collaboration Open\u00a0Science","year":"2015","unstructured":"Open\u00a0Science Collaboration . 2015. Estimating the reproducibility of psychological science. Science 349, 6251 ( 2015 ), aac4716. https:\/\/doi.org\/10.1126\/science.aac4716 arXiv:https:\/\/www.science.org\/doi\/pdf\/10.1126\/science.aac4716 10.1126\/science.aac4716 Open\u00a0Science Collaboration. 2015. Estimating the reproducibility of psychological science. Science 349, 6251 (2015), aac4716. https:\/\/doi.org\/10.1126\/science.aac4716 arXiv:https:\/\/www.science.org\/doi\/pdf\/10.1126\/science.aac4716"},{"key":"e_1_3_2_1_18_1","volume-title":"Research Design: Qualitative, Quantitative, and Mixed Methods Approaches","author":"Creswell W.","year":"2008","unstructured":"John\u00a0 W. Creswell . 2008 . Research Design: Qualitative, Quantitative, and Mixed Methods Approaches , 3 rd Edition(3rd edition ed.). SAGE Publications, Inc, Thousand Oaks , Calif . John\u00a0W. Creswell. 2008. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches, 3rd Edition(3rd edition ed.). SAGE Publications, Inc, Thousand Oaks, Calif.","edition":"3"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1126\/science.1216775"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1177\/1477878515606621"},{"key":"e_1_3_2_1_21_1","unstructured":"Saeed Dehnadi and Richard Bornat. 2006. The camel has two humps (working title). (2006). http:\/\/eis.sla.mdx.ac.uk\/research\/PhDArea\/saeed\/paper1.pdf  Saeed Dehnadi and Richard Bornat. 2006. The camel has two humps (working title). (2006). http:\/\/eis.sla.mdx.ac.uk\/research\/PhDArea\/saeed\/paper1.pdf"},{"key":"e_1_3_2_1_22_1","volume-title":"Earp and David Trafimow","author":"D.","year":"2015","unstructured":"Brian\u00a0 D. Earp and David Trafimow . 2015 . Replication, falsification, and the crisis of confidence in social psychology. Frontiers in Psychology 6 (2015). https:\/\/doi.org\/10.3389\/fpsyg.2015.00621 10.3389\/fpsyg.2015.00621 Brian\u00a0D. Earp and David Trafimow. 2015. Replication, falsification, and the crisis of confidence in social psychology. Frontiers in Psychology 6 (2015). https:\/\/doi.org\/10.3389\/fpsyg.2015.00621"},{"key":"e_1_3_2_1_23_1","volume-title":"How Many Scientists Fabricate and Falsify Research? A Systematic Review and Meta-Analysis of Survey Data. PLOS ONE 4, 5 (05","author":"Fanelli Daniele","year":"2009","unstructured":"Daniele Fanelli . 2009. How Many Scientists Fabricate and Falsify Research? A Systematic Review and Meta-Analysis of Survey Data. PLOS ONE 4, 5 (05 2009 ), 1\u201311. https:\/\/doi.org\/10.1371\/journal.pone.0005738 10.1371\/journal.pone.0005738 Daniele Fanelli. 2009. How Many Scientists Fabricate and Falsify Research? A Systematic Review and Meta-Analysis of Survey Data. PLOS ONE 4, 5 (05 2009), 1\u201311. https:\/\/doi.org\/10.1371\/journal.pone.0005738"},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1177\/1948550615612150"},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"crossref","unstructured":"Sabrina Finke Ferenc Kem\u00e9ny Markus Sommer Vesna Krnjic Martin Arendasy Wolfgang Slany and Karin Landerl. 2022. Unravelling the Numerical and Spatial Underpinnings of Computational Thinking: a Pre-Registered Replication Study [in press]. Computer Science Education(2022).  Sabrina Finke Ferenc Kem\u00e9ny Markus Sommer Vesna Krnjic Martin Arendasy Wolfgang Slany and Karin Landerl. 2022. Unravelling the Numerical and Spatial Underpinnings of Computational Thinking: a Pre-Registered Replication Study [in press]. Computer Science Education(2022).","DOI":"10.1080\/08993408.2022.2079864"},{"key":"e_1_3_2_1_26_1","unstructured":"Center for Open\u00a0Science. n.d.. Registered Reports. Retrieved 2022-03-16 from https:\/\/www.cos.io\/initiatives\/registered-reports  Center for Open\u00a0Science. n.d.. Registered Reports. Retrieved 2022-03-16 from https:\/\/www.cos.io\/initiatives\/registered-reports"},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"crossref","unstructured":"Max Fowler David Smith Mohammed Hassan Seth Poulsen Matthew West and Craig Zilles. 2022. Reevaluating the Relationship between Explaining Tracing and Writing Skills in CS1 in a Replication Study [in press]. Computer Science Education(2022).  Max Fowler David Smith Mohammed Hassan Seth Poulsen Matthew West and Craig Zilles. 2022. Reevaluating the Relationship between Explaining Tracing and Writing Skills in CS1 in a Replication Study [in press]. Computer Science Education(2022).","DOI":"10.1080\/08993408.2022.2079866"},{"key":"e_1_3_2_1_28_1","volume-title":"p-Hacking and publication bias interact to distort meta-analytic effect size estimates.Psychological Methods 25, 4","author":"Friese Malte","year":"2020","unstructured":"Malte Friese and Julius Frankenbach . 2020. p-Hacking and publication bias interact to distort meta-analytic effect size estimates.Psychological Methods 25, 4 ( 2020 ), 456\u2013471. https:\/\/doi.org\/10.1037\/met0000246 10.1037\/met0000246 Malte Friese and Julius Frankenbach. 2020. p-Hacking and publication bias interact to distort meta-analytic effect size estimates.Psychological Methods 25, 4 (2020), 456\u2013471. https:\/\/doi.org\/10.1037\/met0000246"},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1511\/2014.111.460"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10147-015-0887-3"},{"key":"e_1_3_2_1_31_1","volume-title":"George and Marc Buyse","author":"L.","year":"2015","unstructured":"Stephen\u00a0 L. George and Marc Buyse . 2015 . Data fraud in clinical trials. Clinical investigation 5, 2 (2015), 161\u2013173. https:\/\/doi.org\/10.4155\/cli.14.116 25729561[pmid]. 10.4155\/cli.14.116 Stephen\u00a0L. George and Marc Buyse. 2015. Data fraud in clinical trials. Clinical investigation 5, 2 (2015), 161\u2013173. https:\/\/doi.org\/10.4155\/cli.14.116 25729561[pmid]."},{"key":"e_1_3_2_1_32_1","volume-title":"Gerko Vink, Ineke Stoop, Jelte\u00a0M. Wicherts, and Lex\u00a0M. Bouter.","author":"Gopalakrishna Gowri","year":"2022","unstructured":"Gowri Gopalakrishna , Gerben ter Riet , Gerko Vink, Ineke Stoop, Jelte\u00a0M. Wicherts, and Lex\u00a0M. Bouter. 2022 . Prevalence of questionable research practices, research misconduct and their potential explanatory factors: A survey among academic researchers in The Netherlands. PLOS ONE 17, 2 (02 2022), 1\u201316. https:\/\/doi.org\/10.1371\/journal.pone.0263023 10.1371\/journal.pone.0263023 Gowri Gopalakrishna, Gerben ter Riet, Gerko Vink, Ineke Stoop, Jelte\u00a0M. Wicherts, and Lex\u00a0M. Bouter. 2022. Prevalence of questionable research practices, research misconduct and their potential explanatory factors: A survey among academic researchers in The Netherlands. PLOS ONE 17, 2 (02 2022), 1\u201316. https:\/\/doi.org\/10.1371\/journal.pone.0263023"},{"key":"e_1_3_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1108\/EBR-12-2016-0155"},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3345328"},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1177\/1609406920976417"},{"key":"e_1_3_2_1_36_1","volume-title":"The Extent and Consequences of P-Hacking in Science. PLOS Biology 13, 3 (03","author":"Head L.","year":"2015","unstructured":"Megan\u00a0 L. Head , Luke Holman , Rob Lanfear , Andrew\u00a0 T. Kahn , and Michael\u00a0 D. Jennions . 2015. The Extent and Consequences of P-Hacking in Science. PLOS Biology 13, 3 (03 2015 ), 1\u201315. https:\/\/doi.org\/10.1371\/journal.pbio.1002106 10.1371\/journal.pbio.1002106 Megan\u00a0L. Head, Luke Holman, Rob Lanfear, Andrew\u00a0T. Kahn, and Michael\u00a0D. Jennions. 2015. The Extent and Consequences of P-Hacking in Science. PLOS Biology 13, 3 (03 2015), 1\u201315. https:\/\/doi.org\/10.1371\/journal.pbio.1002106"},{"key":"e_1_3_2_1_37_1","article-title":"A Systematic Literature Review of Empiricism and Norms of Reporting in Computing Education Research Literature","volume":"22","author":"Heckman Sarah","year":"2021","unstructured":"Sarah Heckman , Jeffrey\u00a0 C. Carver , Mark Sherriff , and Ahmed Al-zubidy. 2021 . A Systematic Literature Review of Empiricism and Norms of Reporting in Computing Education Research Literature . ACM Trans. Comput. Educ. 22 , 1, Article 3 (oct 2021), 46\u00a0pages. https:\/\/doi.org\/10.1145\/3470652 10.1145\/3470652 Sarah Heckman, Jeffrey\u00a0C. Carver, Mark Sherriff, and Ahmed Al-zubidy. 2021. A Systematic Literature Review of Empiricism and Norms of Reporting in Computing Education Research Literature. ACM Trans. Comput. Educ. 22, 1, Article 3 (oct 2021), 46\u00a0pages. https:\/\/doi.org\/10.1145\/3470652","journal-title":"ACM Trans. Comput. Educ."},{"key":"#cr-split#-e_1_3_2_1_38_1.1","doi-asserted-by":"crossref","unstructured":"Emma\u00a0L Henderson. 2022. A guide to preregistration and Registered Reports. https:\/\/doi.org\/10.31222\/osf.io\/x7aqr 10.31222\/osf.io","DOI":"10.31222\/osf.io\/x7aqr"},{"key":"#cr-split#-e_1_3_2_1_38_1.2","doi-asserted-by":"crossref","unstructured":"Emma\u00a0L Henderson. 2022. A guide to preregistration and Registered Reports. https:\/\/doi.org\/10.31222\/osf.io\/x7aqr","DOI":"10.31222\/osf.io\/x7aqr"},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287324.3287420"},{"key":"e_1_3_2_1_40_1","volume-title":"An introduction to the philosophy of methodology","author":"Howell E","unstructured":"Kerry\u00a0 E Howell . 2012. An introduction to the philosophy of methodology . Sage . Kerry\u00a0E Howell. 2012. An introduction to the philosophy of methodology. Sage."},{"key":"e_1_3_2_1_41_1","doi-asserted-by":"crossref","unstructured":"Christopher Hundhausen Phillip Conrad Adam Carter and Olusola Adesope. 2022. Assessing Individual Contributions to Software Engineering Projects: A Replication Study [in press]. Computer Science Education(2022).  Christopher Hundhausen Phillip Conrad Adam Carter and Olusola Adesope. 2022. Assessing Individual Contributions to Software Engineering Projects: A Replication Study [in press]. Computer Science Education(2022).","DOI":"10.1080\/08993408.2022.2071543"},{"key":"e_1_3_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1177\/0956797611430953"},{"key":"e_1_3_2_1_43_1","volume-title":"fast and slow","author":"Kahneman Daniel","unstructured":"Daniel Kahneman . 2011. Thinking , fast and slow . Macmillan . Daniel Kahneman. 2011. Thinking, fast and slow. Macmillan."},{"key":"e_1_3_2_1_44_1","volume-title":"Publication Bias in Psychology: A Diagnosis Based on the Correlation between Effect Size and Sample Size. PLOS ONE 9, 9 (09","author":"K\u00fchberger Anton","year":"2014","unstructured":"Anton K\u00fchberger , Astrid Fritz , and Thomas Scherndl . 2014. Publication Bias in Psychology: A Diagnosis Based on the Correlation between Effect Size and Sample Size. PLOS ONE 9, 9 (09 2014 ), 1\u20138. https:\/\/doi.org\/10.1371\/journal.pone.0105825 10.1371\/journal.pone.0105825 Anton K\u00fchberger, Astrid Fritz, and Thomas Scherndl. 2014. Publication Bias in Psychology: A Diagnosis Based on the Correlation between Effect Size and Sample Size. PLOS ONE 9, 9 (09 2014), 1\u20138. https:\/\/doi.org\/10.1371\/journal.pone.0105825"},{"key":"e_1_3_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/1404520.1404531"},{"key":"e_1_3_2_1_46_1","volume-title":"Rigorous Large-Scale Educational RCTs Are Often Uninformative: Should We Be Concerned?Educational Researcher 48, 3","author":"Lortie-Forgues Hugues","year":"2019","unstructured":"Hugues Lortie-Forgues and Matthew Inglis . 2019. Rigorous Large-Scale Educational RCTs Are Often Uninformative: Should We Be Concerned?Educational Researcher 48, 3 ( 2019 ), 158\u2013166. https:\/\/doi.org\/10.3102\/0013189X19832850 arXiv:https:\/\/doi.org\/10.3102\/0013189X19832850 10.3102\/0013189X19832850 Hugues Lortie-Forgues and Matthew Inglis. 2019. Rigorous Large-Scale Educational RCTs Are Often Uninformative: Should We Be Concerned?Educational Researcher 48, 3 (2019), 158\u2013166. https:\/\/doi.org\/10.3102\/0013189X19832850 arXiv:https:\/\/doi.org\/10.3102\/0013189X19832850"},{"key":"e_1_3_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/1368088.1368115"},{"key":"e_1_3_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1177\/1745691612460688"},{"key":"e_1_3_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1111\/ecoj.12527"},{"key":"e_1_3_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3230977.3230984"},{"key":"e_1_3_2_1_51_1","volume-title":"Retrieved 9th","author":"McCook Alison","year":"2017","unstructured":"Alison McCook . 2017 . \u201c I placed too much faith in underpowered studies:\u201d Nobel Prize winner admits mistakes . Retrieved 9th March, 2022 from https:\/\/retractionwatch.com\/2017\/02\/20\/placed-much-faith-underpowered-studies-nobel-prize-winner-admits-mistakes\/ Alison McCook. 2017. \u201cI placed too much faith in underpowered studies:\u201d Nobel Prize winner admits mistakes. Retrieved 9th March, 2022 from https:\/\/retractionwatch.com\/2017\/02\/20\/placed-much-faith-underpowered-studies-nobel-prize-winner-admits-mistakes\/"},{"key":"e_1_3_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1177\/160940690200100202"},{"key":"e_1_3_2_1_53_1","volume-title":"Newton and Mahallad Miah","author":"M.","year":"2017","unstructured":"Philip\u00a0 M. Newton and Mahallad Miah . 2017 . Evidence-Based Higher Education \u2013 Is the Learning Styles \u2018Myth\u2019 Important?Frontiers in Psychology 8 (2017). https:\/\/doi.org\/10.3389\/fpsyg.2017.00444 10.3389\/fpsyg.2017.00444 Philip\u00a0M. Newton and Mahallad Miah. 2017. Evidence-Based Higher Education \u2013 Is the Learning Styles \u2018Myth\u2019 Important?Frontiers in Psychology 8 (2017). https:\/\/doi.org\/10.3389\/fpsyg.2017.00444"},{"key":"e_1_3_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jclinepi.2006.11.014"},{"key":"e_1_3_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1136\/eb-2015-102054"},{"key":"e_1_3_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1177\/0149206314527133"},{"key":"e_1_3_2_1_57_1","article-title":"Post-positivism: An effective paradigm for social and educational research","volume":"45","author":"Panhwar Abdul\u00a0Hameed","year":"2017","unstructured":"Abdul\u00a0Hameed Panhwar , Sanaullah Ansari , and Asif\u00a0Ali Shah . 2017 . Post-positivism: An effective paradigm for social and educational research . International Research Journal of Arts & Humanities (IRJAH) 45 , 45(2017). Abdul\u00a0Hameed Panhwar, Sanaullah Ansari, and Asif\u00a0Ali Shah. 2017. Post-positivism: An effective paradigm for social and educational research. International Research Journal of Arts & Humanities (IRJAH) 45, 45(2017).","journal-title":"International Research Journal of Arts & Humanities (IRJAH)"},{"key":"e_1_3_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1177\/2378023115625071"},{"key":"e_1_3_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1177\/03063127211005551"},{"key":"e_1_3_2_1_60_1","volume-title":"The logic of scientific discovery","author":"Popper Karl","unstructured":"Karl Popper . 2002. The logic of scientific discovery ( second ed.). Routledge . Karl Popper. 2002. The logic of scientific discovery(second ed.). Routledge."},{"key":"e_1_3_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1111\/phc3.12633"},{"key":"e_1_3_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2016.08.047"},{"key":"e_1_3_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1177\/25152459211007467"},{"key":"e_1_3_2_1_64_1","volume-title":"Nonreplicable publications are cited more than replicable ones. Science Advances 7, 21","author":"Serra-Garcia Marta","year":"2021","unstructured":"Marta Serra-Garcia and Uri Gneezy . 2021. Nonreplicable publications are cited more than replicable ones. Science Advances 7, 21 ( 2021 ), eabd1705. https:\/\/doi.org\/10.1126\/sciadv.abd1705 arXiv:https:\/\/www.science.org\/doi\/pdf\/10.1126\/sciadv.abd1705 10.1126\/sciadv.abd1705 Marta Serra-Garcia and Uri Gneezy. 2021. Nonreplicable publications are cited more than replicable ones. Science Advances 7, 21 (2021), eabd1705. https:\/\/doi.org\/10.1126\/sciadv.abd1705 arXiv:https:\/\/www.science.org\/doi\/pdf\/10.1126\/sciadv.abd1705"},{"key":"e_1_3_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/1584322.1584334"},{"key":"e_1_3_2_1_66_1","volume-title":"Strategies for ensuring trustworthiness in qualitative research projects. Education for information 22, 2","author":"Shenton K","year":"2004","unstructured":"Andrew\u00a0 K Shenton . 2004. Strategies for ensuring trustworthiness in qualitative research projects. Education for information 22, 2 ( 2004 ), 63\u201375. Andrew\u00a0K Shenton. 2004. Strategies for ensuring trustworthiness in qualitative research projects. Education for information 22, 2 (2004), 63\u201375."},{"key":"e_1_3_2_1_67_1","volume-title":"Mine Dogucu, Mark Hong, Michael Luu, Brian Anderson, Aaron Cote, Matthew Ferland, Palak Jain, Tyler LaBonte, Leena Mathur, Ryan Moreno, and Ryan Sakuma.","author":"Shindler Michael","year":"2022","unstructured":"Michael Shindler , Natalia Pinpin , Mia Markovic , Frederick Reiber , Jee\u00a0Hoon Kim , Giles Pierre\u00a0Nunez Carlos , Mine Dogucu, Mark Hong, Michael Luu, Brian Anderson, Aaron Cote, Matthew Ferland, Palak Jain, Tyler LaBonte, Leena Mathur, Ryan Moreno, and Ryan Sakuma. 2022 . Replication of \u201dStudent Misconceptions of Dynamic Programming\u201d [in press]. Computer Science Education( 2022). Michael Shindler, Natalia Pinpin, Mia Markovic, Frederick Reiber, Jee\u00a0Hoon Kim, Giles Pierre\u00a0Nunez Carlos, Mine Dogucu, Mark Hong, Michael Luu, Brian Anderson, Aaron Cote, Matthew Ferland, Palak Jain, Tyler LaBonte, Leena Mathur, Ryan Moreno, and Ryan Sakuma. 2022. Replication of \u201dStudent Misconceptions of Dynamic Programming\u201d [in press]. Computer Science Education(2022)."},{"key":"e_1_3_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/6592.6594"},{"key":"#cr-split#-e_1_3_2_1_69_1.1","doi-asserted-by":"crossref","unstructured":"Poorna Talkad\u00a0Sukumar and Ronald Metoyer. 2019. Replication and transparency of qualitative research from a constructivist perspective. (2019). https:\/\/doi.org\/10.31219\/osf.io\/6efvp 10.31219\/osf.io","DOI":"10.31219\/osf.io\/6efvp"},{"key":"#cr-split#-e_1_3_2_1_69_1.2","doi-asserted-by":"crossref","unstructured":"Poorna Talkad\u00a0Sukumar and Ronald Metoyer. 2019. Replication and transparency of qualitative research from a constructivist perspective. (2019). https:\/\/doi.org\/10.31219\/osf.io\/6efvp","DOI":"10.31219\/osf.io\/6efvp"},{"key":"e_1_3_2_1_70_1","article-title":"Asking Research Questions","volume":"14","author":"Tenenberg Josh","year":"2014","unstructured":"Josh Tenenberg . 2014 . Asking Research Questions : Theoretical Presuppositions. ACM Trans. Comput. Educ. 14 , 3, Article 16 (sep 2014), 8\u00a0pages. https:\/\/doi.org\/10.1145\/2644924 10.1145\/2644924 Josh Tenenberg. 2014. Asking Research Questions: Theoretical Presuppositions. ACM Trans. Comput. Educ. 14, 3, Article 16 (sep 2014), 8\u00a0pages. https:\/\/doi.org\/10.1145\/2644924","journal-title":"Theoretical Presuppositions. ACM Trans. Comput. Educ."},{"key":"e_1_3_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1177\/1077800410383121"},{"key":"e_1_3_2_1_72_1","volume-title":"Is replication relevant for qualitative research?Qualitative Psychology 8, 3","author":"Tuval-Mashiach Rivka","year":"2021","unstructured":"Rivka Tuval-Mashiach . 2021. Is replication relevant for qualitative research?Qualitative Psychology 8, 3 ( 2021 ), 365\u2013377. https:\/\/doi.org\/10.1037\/qup0000217 10.1037\/qup0000217 Rivka Tuval-Mashiach. 2021. Is replication relevant for qualitative research?Qualitative Psychology 8, 3 (2021), 365\u2013377. https:\/\/doi.org\/10.1037\/qup0000217"},{"key":"e_1_3_2_1_73_1","first-page":"1","article-title":"How should educational research respond to the replication \u201ccrisis\u201d in the social sciences? Reflections on the papers in the Special Issue","volume":"0","author":"Wiliam Dylan","year":"2022","unstructured":"Dylan Wiliam . 2022 . How should educational research respond to the replication \u201ccrisis\u201d in the social sciences? Reflections on the papers in the Special Issue . Educational Research and Evaluation 0 , 0 (2022), 1 \u2013 7 . https:\/\/doi.org\/10.1080\/13803611.2021.2022309 arXiv:https:\/\/doi.org\/10.1080\/13803611.2021.2022309 10.1080\/13803611.2021.2022309 Dylan Wiliam. 2022. How should educational research respond to the replication \u201ccrisis\u201d in the social sciences? Reflections on the papers in the Special Issue. Educational Research and Evaluation 0, 0 (2022), 1\u20137. https:\/\/doi.org\/10.1080\/13803611.2021.2022309 arXiv:https:\/\/doi.org\/10.1080\/13803611.2021.2022309","journal-title":"Educational Research and Evaluation"},{"key":"e_1_3_2_1_74_1","volume-title":"Prevalence of Research Misconduct and Questionable Research Practices: A Systematic Review and Meta-Analysis. Science and Engineering Ethics 27, 4 (29","author":"Xie Yu","year":"2021","unstructured":"Yu Xie , Kai Wang , and Yan Kong . 2021. Prevalence of Research Misconduct and Questionable Research Practices: A Systematic Review and Meta-Analysis. Science and Engineering Ethics 27, 4 (29 Jun 2021 ), 41. https:\/\/doi.org\/10.1007\/s11948-021-00314-9 10.1007\/s11948-021-00314-9 Yu Xie, Kai Wang, and Yan Kong. 2021. Prevalence of Research Misconduct and Questionable Research Practices: A Systematic Review and Meta-Analysis. Science and Engineering Ethics 27, 4 (29 Jun 2021), 41. https:\/\/doi.org\/10.1007\/s11948-021-00314-9"},{"key":"e_1_3_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.1145\/3159450.3159528"}],"event":{"name":"ICER 2022: ACM Conference on International Computing Education Research","location":"Lugano and Virtual Event Switzerland","acronym":"ICER 2022","sponsor":["SIGCSE ACM Special Interest Group on Computer Science Education"]},"container-title":["Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3501385.3543971","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3501385.3543971","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:30:20Z","timestamp":1750188620000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3501385.3543971"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,8,3]]},"references-count":77,"alternative-id":["10.1145\/3501385.3543971","10.1145\/3501385"],"URL":"https:\/\/doi.org\/10.1145\/3501385.3543971","relation":{},"subject":[],"published":{"date-parts":[[2022,8,3]]},"assertion":[{"value":"2022-08-03","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}