{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,14]],"date-time":"2026-02-14T05:05:33Z","timestamp":1771045533562,"version":"3.50.1"},"reference-count":74,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2021,10,18]],"date-time":"2021-10-18T00:00:00Z","timestamp":1634515200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"National Science Foundation","award":["1525373, 1525173, and 1525028"],"award-info":[{"award-number":["1525373, 1525173, and 1525028"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Comput. Educ."],"published-print":{"date-parts":[[2022,3,31]]},"abstract":"<jats:p>\n            <jats:bold>Context.<\/jats:bold>\n            Computing Education Research (CER) is critical to help the computing education community and policy makers support the increasing population of students who need to learn computing skills for future careers. For a community to systematically advance knowledge about a topic, the members must be able to understand published work thoroughly enough to perform replications, conduct meta-analyses, and build theories. There is a need to understand whether published research allows the CER community to systematically advance knowledge and build theories.\n          <\/jats:p>\n          <jats:p>\n            <jats:bold>Objectives.<\/jats:bold>\n            The goal of this study is\n            <jats:italic>to characterize the reporting of empiricism in Computing Education Research literature by identifying whether publications include content necessary for researchers to perform replications, meta-analyses, and theory building.<\/jats:italic>\n            We answer three research questions related to this goal: (RQ1) What percentage of papers in CER venues have some form of empirical evaluation? (RQ2) Of the papers that have empirical evaluation, what are the characteristics of the empirical evaluation? (RQ3) Of the papers that have empirical evaluation, do they follow norms (both for inclusion and for labeling of information needed for replication, meta-analysis, and, eventually, theory-building) for reporting empirical work?\n          <\/jats:p>\n          <jats:p>\n            <jats:bold>Methods.<\/jats:bold>\n            We conducted a systematic literature review of the 2014 and 2015 proceedings or issues of five CER venues:\n            <jats:italic>Technical Symposium on Computer Science Education<\/jats:italic>\n            (SIGCSE TS),\n            <jats:italic>International Symposium on Computing Education Research<\/jats:italic>\n            (ICER),\n            <jats:italic>Conference on Innovation and Technology in Computer Science Education<\/jats:italic>\n            (ITiCSE),\n            <jats:italic>ACM Transactions on Computing Education<\/jats:italic>\n            (TOCE), and\n            <jats:italic>Computer Science Education<\/jats:italic>\n            (CSE). We developed and applied the\n            <jats:italic>CER Empiricism Assessment Rubric<\/jats:italic>\n            to the 427 papers accepted and published at these venues over 2014 and 2015. Two people evaluated each paper using the\n            <jats:italic>Base Rubric<\/jats:italic>\n            for characterizing the paper. An individual person applied the other rubrics to characterize the norms of reporting, as appropriate for the paper type. Any discrepancies or questions were discussed between multiple reviewers to resolve.\n          <\/jats:p>\n          <jats:p>\n            <jats:bold>Results.<\/jats:bold>\n            We found that over 80% of papers accepted across all five venues had some form of empirical evaluation. Quantitative evaluation methods were the most frequently reported. Papers most frequently reported results on interventions around pedagogical techniques, curriculum, community, or tools. There was a split in papers that had some type of comparison between an intervention and some other dataset or baseline. Most papers reported related work, following the expectations for doing so in the SIGCSE and CER community. However, many papers were lacking properly reported research objectives, goals, research questions, or hypotheses; description of participants; study design; data collection; and threats to validity. These results align with prior surveys of the CER literature.\n          <\/jats:p>\n          <jats:p>\n            <jats:bold>Conclusions.<\/jats:bold>\n            CER authors are contributing empirical results to the literature; however, not all norms for reporting are met. We encourage authors to provide clear, labeled details about their work so readers can use the study methodologies and results for replications and meta-analyses. As our community grows, our reporting of CER should mature to help establish computing education theory to support the next generation of computing learners.\n          <\/jats:p>\n          <jats:p\/>","DOI":"10.1145\/3470652","type":"journal-article","created":{"date-parts":[[2021,10,19]],"date-time":"2021-10-19T00:57:14Z","timestamp":1634605034000},"page":"1-46","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":23,"title":["A Systematic Literature Review of Empiricism and Norms of Reporting in Computing Education Research Literature"],"prefix":"10.1145","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4351-8611","authenticated-orcid":false,"given":"Sarah","family":"Heckman","sequence":"first","affiliation":[{"name":"North Carolina State University, Campus Box, Raleigh"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7824-9151","authenticated-orcid":false,"given":"Jeffrey C.","family":"Carver","sequence":"additional","affiliation":[{"name":"University of Alabama, Tuscaloosa, AL"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1745-205X","authenticated-orcid":false,"given":"Mark","family":"Sherriff","sequence":"additional","affiliation":[{"name":"University of Virginia, Charlottesville, VA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4305-2153","authenticated-orcid":false,"given":"Ahmed","family":"Al-zubidy","sequence":"additional","affiliation":[{"name":"University of Alabama, Tuscaloosa, AL"}]}],"member":"320","published-online":{"date-parts":[[2021,10,18]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/2632320"},{"key":"e_1_3_3_3_2","volume-title":"Proceedings of the 11th Annual International Conference on International Computing Education Research","year":"2015","unstructured":"2015. Proceedings of the 11th Annual International Conference on International Computing Education Research. Association for Computing Machinery, New York, NY."},{"key":"e_1_3_3_4_2","unstructured":"ACM. 2020. Artifact Review and Badging \u2013 Current. Retrieved from https:\/\/www.acm.org\/publications\/policies\/artifact-review-and-badging-current."},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/2999541.2999554"},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/2839509.2844601"},{"key":"e_1_3_3_7_2","doi-asserted-by":"publisher","DOI":"10.3102\/0013189X035006033"},{"key":"e_1_3_3_9_2","volume-title":"Engaging in the Scholarship of Teaching and Learning","author":"Bishop-Clark Cathy","year":"2012","unstructured":"Cathy Bishop-Clark and Beth Dietz-Uhler. 2012. Engaging in the Scholarship of Teaching and Learning. Stylus Publishing, Sterling, VA."},{"key":"e_1_3_3_10_2","first-page":"56","volume-title":"Computing Education Literature Review and Voices from the Field","author":"Blikstein Paulo","year":"2019","unstructured":"Paulo Blikstein and Sepi Hejazi Moghadam. 2019. Computing Education Literature Review and Voices from the Field. Cambridge University Press, 56\u201378. DOI:DOI:https:\/\/doi.org\/10.1017\/9781108654555.004"},{"key":"e_1_3_3_11_2","first-page":"1","volume-title":"Proceedings of the 1st International Workshop on Replication in Empirical Software Engineering","volume":"1","author":"Carver Jeffrey C.","year":"2010","unstructured":"Jeffrey C. Carver. 2010. Towards reporting guidelines for experimental replications: A proposal. In Proceedings of the 1st International Workshop on Replication in Empirical Software Engineering, Vol. 1. Citeseer, 1\u20134."},{"key":"e_1_3_3_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/1315803.1315806"},{"key":"e_1_3_3_13_2","unstructured":"What Works Clearinghouse. 2020. What Works Clearinghouse Standards Handbook Version 4.1. Retrieved from https:\/\/ies.ed.gov\/ncee\/wwc\/Handbooks."},{"key":"e_1_3_3_14_2","volume-title":"Research Design: Qualitative, Quantitative, and Mixed Methods Approaches (5th ed.)","author":"Creswell John W.","year":"2018","unstructured":"John W. Creswell and J. David Creswell. 2018. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches (5th ed.). SAGE Publications, Inc."},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.5555\/2483716.2483728"},{"key":"e_1_3_3_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/3287324.3287393"},{"key":"e_1_3_3_17_2","doi-asserted-by":"publisher","DOI":"10.5555\/983111"},{"key":"e_1_3_3_18_2","first-page":"40","volume-title":"Computing Education Research Today","author":"Fincher Sally A.","year":"2019","unstructured":"Sally A. Fincher, Josh Tenenberg, Brian Dorn, Christopher Hundhausen, Robert McCartney, and Laurie Murphy. 2019. Computing Education Research Today. Cambridge University Press, 40\u201355. DOI:DOI:https:\/\/doi.org\/10.1017\/9781108654555.003"},{"key":"e_1_3_3_19_2","unstructured":"Institute for Education Science and the National Science Foundation. 2013. Common Guidelines for Education Research and Development. Retrieved from https:\/\/www.nsf.gov\/pubs\/2013\/nsf13126\/nsf13126.pdf."},{"key":"e_1_3_3_20_2","unstructured":"The National Science Foundation and the Institute of Education Sciences. 2018. Companion Guidelines on Replication & Reproducibility in Education Research: A Supplement to the Common Guidelines for Education Research and Development. Retrieved from https:\/\/www.nsf.gov\/pubs\/2019\/nsf19022\/nsf19022.pdf."},{"key":"e_1_3_3_21_2","doi-asserted-by":"publisher","DOI":"10.1109\/EDUCON.2015.7096023"},{"key":"e_1_3_3_22_2","volume-title":"Proceedings of the 1st International Workshop on Replication in Empirical Software Engineering Research (RESER\u201910)","author":"G\u00f3mez Omar S.","year":"2010","unstructured":"Omar S. G\u00f3mez, Natalia Juristo, and Sira Vegas. 2010. Replication, reproduction and re-analysis: Three ways for verifying experimental findings. In Proceedings of the 1st International Workshop on Replication in Empirical Software Engineering Research (RESER\u201910)."},{"key":"e_1_3_3_23_2","first-page":"11","volume-title":"The History of Computing Education Research","author":"Guzdial Mark","year":"2019","unstructured":"Mark Guzdial and Benedict du Boulay. 2019. The History of Computing Education Research. Cambridge University Press, 11\u201339. DOI:DOI:https:\/\/doi.org\/10.1017\/9781108654555.002"},{"key":"e_1_3_3_24_2","first-page":"29","article-title":"Meta-analysis in medical research","volume":"14","author":"Haidich A. B.","year":"2010","unstructured":"A. B. Haidich. 2010. Meta-analysis in medical research. Hippokratia 14 (Suppl 1) (2010), 29\u201337.","journal-title":"Hippokratia"},{"key":"e_1_3_3_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/3345328"},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/3329103.3329104"},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/1124706.1121469"},{"key":"e_1_3_3_28_2","unstructured":"Sarah Heckman Jeffrey C. Carver Mark Sherriff and Ahmed Al-Zubidy. 2021. CER Empiricism Assessment Dataset (2014 & 2015). (June 2021). DOI:DOI:https:\/\/doi.org\/10.6084\/m9.figshare.14816634.v1"},{"key":"e_1_3_3_29_2","volume-title":"Proceedings of the 13th Psychology of Programming Interest Group (PPIG) Workshop","author":"Holmboe Christian","year":"2001","unstructured":"Christian Holmboe, Linda McIver, and Carlisle E. George. 2001. Research agenda for computer science education. In Proceedings of the 13th Psychology of Programming Interest Group (PPIG) Workshop. Retrieved from https:\/\/www.ppig.org\/files\/2001-PPIG-13th-holmboe.pdf."},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/2858796.2858798"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/3287087.3287088"},{"key":"e_1_3_3_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISESE.2005.1541818"},{"key":"e_1_3_3_33_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10639-008-9078-4"},{"key":"e_1_3_3_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/1839594.1839598"},{"key":"e_1_3_3_35_2","volume-title":"Procedures for Performing Systematic Reviews","author":"Kitchenham Barbara","year":"2004","unstructured":"Barbara Kitchenham. 2004. Procedures for Performing Systematic Reviews. Joint Technical Report, Keele University Technical Report TR\/SE-0401, NICTA Technical Report 040011T.1. Retrieved from https:\/\/www.inf.ufsc.br\/aldo.vw\/kitchenham.pdf."},{"key":"e_1_3_3_36_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-007-9053-5"},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/2960310.2960328"},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/3293881.3295779"},{"key":"e_1_3_3_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/3372782.3406279"},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/2632320.2632358"},{"key":"e_1_3_3_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/1839594.1839597"},{"key":"e_1_3_3_42_2","doi-asserted-by":"publisher","DOI":"10.1080\/08993408.2018.1562145"},{"key":"e_1_3_3_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/3364510.3364528"},{"key":"e_1_3_3_44_2","doi-asserted-by":"publisher","DOI":"10.1109\/FIE.2018.8659312"},{"key":"e_1_3_3_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/3372782.3406258"},{"key":"e_1_3_3_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/3159450.3159481"},{"key":"e_1_3_3_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/TE.2018.2864133"},{"key":"e_1_3_3_48_2","doi-asserted-by":"publisher","DOI":"10.1145\/3291279.3339422"},{"key":"e_1_3_3_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/3230977.3230992"},{"key":"e_1_3_3_50_2","doi-asserted-by":"publisher","DOI":"10.1027\/1864-9335\/a000192"},{"key":"e_1_3_3_51_2","volume-title":"Assessing and Responding to the Growth of Computer Science Undergraduate Enrollments","author":"Engineering, National Academies of Sciences,","year":"2018","unstructured":"National Academies of Sciences, Engineering, and Medicine. 2018. Assessing and Responding to the Growth of Computer Science Undergraduate Enrollments. The National Academies Press, Washington, DC. DOI:DOI:https:\/\/doi.org\/10.17226\/24926"},{"key":"e_1_3_3_52_2","doi-asserted-by":"publisher","DOI":"10.1145\/3372782.3406276"},{"key":"e_1_3_3_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/1113847.1113893"},{"key":"e_1_3_3_54_2","doi-asserted-by":"publisher","DOI":"10.1145\/1345443.1345441"},{"key":"e_1_3_3_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/3129166.3129172"},{"key":"e_1_3_3_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/3372782.3406263"},{"key":"e_1_3_3_57_2","doi-asserted-by":"publisher","DOI":"10.5555\/1522330"},{"key":"e_1_3_3_58_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-008-9102-8"},{"key":"e_1_3_3_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/3291279.3339408"},{"key":"e_1_3_3_60_2","doi-asserted-by":"publisher","DOI":"10.1037\/a0015108"},{"key":"e_1_3_3_61_2","doi-asserted-by":"publisher","DOI":"10.1136\/bmj.c332"},{"key":"e_1_3_3_62_2","doi-asserted-by":"publisher","DOI":"10.1145\/1584322.1584334"},{"key":"e_1_3_3_63_2","doi-asserted-by":"publisher","DOI":"10.1080\/08993400701538021"},{"key":"e_1_3_3_64_2","doi-asserted-by":"publisher","DOI":"10.5555\/2449323.2449336"},{"key":"e_1_3_3_65_2","doi-asserted-by":"publisher","DOI":"10.5555\/1862712.1862735"},{"key":"e_1_3_3_66_2","doi-asserted-by":"publisher","DOI":"10.1145\/3373165.3373188"},{"key":"e_1_3_3_67_2","doi-asserted-by":"publisher","DOI":"10.1145\/1404520.1404536"},{"key":"e_1_3_3_68_2","doi-asserted-by":"publisher","DOI":"10.1145\/3341525.3387407"},{"key":"e_1_3_3_69_2","volume-title":"National Advisory Committee on Computing Qualifications","year":"2008","unstructured":"Simon, Judithe Irene Sheard, Angela Carbone, Michael de Raadt, Margaret Hamilton, Raymond Lister, and Errol Thompson. 2008. Eight years of computing education papers at NACCQ. National Advisory Committee on Computing Qualifications (2008). http:\/\/hdl.handle.net\/10453\/12628"},{"key":"e_1_3_3_70_2","doi-asserted-by":"publisher","DOI":"10.1145\/3344429.3372504"},{"key":"e_1_3_3_71_2","doi-asserted-by":"publisher","DOI":"10.1145\/3364510.3364513"},{"key":"e_1_3_3_72_2","doi-asserted-by":"publisher","DOI":"10.1145\/2644924"},{"key":"e_1_3_3_73_2","doi-asserted-by":"publisher","DOI":"10.1145\/971300.971391"},{"key":"e_1_3_3_74_2","doi-asserted-by":"publisher","DOI":"10.1145\/2632320.2632349"},{"key":"e_1_3_3_75_2","doi-asserted-by":"publisher","DOI":"10.1145\/3397568.3397570"},{"key":"e_1_3_3_76_2","article-title":"2019 Taulbee Survey","volume":"32","author":"Zweben Stuart","year":"2020","unstructured":"Stuart Zweben and Betsy Bizot. 2020. 2019 Taulbee Survey. Comput. Res. News 32, 5 (2020).","journal-title":"Comput. Res. News"}],"container-title":["ACM Transactions on Computing Education"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3470652","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3470652","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3470652","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:18:55Z","timestamp":1750191535000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3470652"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,10,18]]},"references-count":74,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2022,3,31]]}},"alternative-id":["10.1145\/3470652"],"URL":"https:\/\/doi.org\/10.1145\/3470652","relation":{},"ISSN":["1946-6226","1946-6226"],"issn-type":[{"value":"1946-6226","type":"print"},{"value":"1946-6226","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,10,18]]},"assertion":[{"value":"2019-09-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-06-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-10-18","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}