{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T15:51:46Z","timestamp":1776095506771,"version":"3.50.1"},"reference-count":99,"publisher":"Association for Computing Machinery (ACM)","issue":"3","funder":[{"DOI":"10.13039\/501100002341","name":"Research Council of Finland","doi-asserted-by":"crossref","award":["356114"],"award-info":[{"award-number":["356114"]}],"id":[{"id":"10.13039\/501100002341","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Comput. Educ."],"published-print":{"date-parts":[[2025,9,30]]},"abstract":"<jats:p>\n            <jats:italic toggle=\"yes\">Motivation<\/jats:italic>\n            : Students learning to program often reach states where they are stuck and can make no forward progress\u2014but this may be outside the classroom where no instructor is available to help. In this situation, an automatically generated next-step hint can help them make forward progress and support their learning. It is important to know what makes a good hint or a bad hint, and how to generate good hints automatically in novice programming tools, for example, using large language models (LLMs).\n          <\/jats:p>\n          <jats:p>\n            <jats:italic toggle=\"yes\">Method and Participants<\/jats:italic>\n            : We recruited 44 Java educators from around the world to participate in an online study. We used a set of real student code states as hint-generation scenarios. Participants used a technique known as comparative judgement to rank a set of candidate next-step Java hints, which were generated by LLMs and by five human experienced educators. Participants ranked the hints without being told how they were generated. The hints were generated with no explicit detail given to the LLMs\/humans on what the target task was. Participants then filled in a survey with follow-up questions. The ranks of the hints were analysed against a set of extracted hint characteristics using a random forest approach.\n          <\/jats:p>\n          <jats:p>\n            <jats:italic toggle=\"yes\">Findings<\/jats:italic>\n            : We found that LLMs had considerable variation in generating high quality next-step hints for programming novices, with GPT-4 outperforming other models tested. When used with a well-designed prompt, GPT-4 outperformed human experts in generating pedagogically valuable hints. A multi-stage prompt was the most effective LLM prompt. According to a fitted random forest model, the two most important factors of a good hint were length (80\u2013160 words being best) and reading level (US grade nine or below being best). Offering alternative approaches to solving the problem was considered bad, and we found no effect of sentiment.\n          <\/jats:p>\n          <jats:p>\n            <jats:italic toggle=\"yes\">Conclusions<\/jats:italic>\n            : Automatic generation of these hints is immediately viable, given that LLMs outperformed humans\u2014even when the students\u2019 task is unknown. Hint length and reading level were more important than several pedagogical features of hints. The fact that it took a group of experts several rounds of experimentation and refinement to design a prompt that achieves this outcome suggests that students on their own are unlikely to be able to produce the same benefit. The prompting task, therefore, should be embedded in an expert-designed tool.\n          <\/jats:p>","DOI":"10.1145\/3737885","type":"journal-article","created":{"date-parts":[[2025,5,29]],"date-time":"2025-05-29T12:42:03Z","timestamp":1748522523000},"page":"1-43","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Howzat? Appealing to Expert Judgement for Evaluating Human and AI Next-Step Hints for Novice Programmers"],"prefix":"10.1145","volume":"25","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6086-2479","authenticated-orcid":false,"given":"Neil C. C.","family":"Brown","sequence":"first","affiliation":[{"name":"King\u2019s College London, London, United Kingdom of Great Britain and Northern Ireland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6999-8767","authenticated-orcid":false,"given":"Pierre","family":"Weill-Tessier","sequence":"additional","affiliation":[{"name":"King\u2019s College London, London, United Kingdom of Great Britain and Northern Ireland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6829-9449","authenticated-orcid":false,"given":"Juho","family":"Leinonen","sequence":"additional","affiliation":[{"name":"Aalto University, Aalto, Finland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5150-9806","authenticated-orcid":false,"given":"Paul","family":"Denny","sequence":"additional","affiliation":[{"name":"The University of Auckland, Auckland, New Zealand"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0544-2003","authenticated-orcid":false,"given":"Michael","family":"K\u00f6lling","sequence":"additional","affiliation":[{"name":"King\u2019s College London, London, United Kingdom of Great Britain and Northern Ireland"}]}],"member":"320","published-online":{"date-parts":[[2025,7,22]]},"reference":[{"key":"e_1_3_4_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/2787622.2787717"},{"key":"e_1_3_4_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377814.3381703"},{"key":"e_1_3_4_4_2","doi-asserted-by":"publisher","DOI":"10.1007\/s40593-015-0089-1"},{"issue":"10","key":"e_1_3_4_5_2","doi-asserted-by":"crossref","first-page":"1340","DOI":"10.1093\/bioinformatics\/btq134","article-title":"Permutation importance: A corrected feature importance measure","volume":"26","author":"Altmann Andr\u00e9","year":"2010","unstructured":"Andr\u00e9 Altmann, Laura Tolo\u015fi, Oliver Sander, and Thomas Lengauer. 2010. Permutation importance: A corrected feature importance measure. Bioinformatics 26, 10 (2010), 1340\u20131347.","journal-title":"Bioinformatics"},{"key":"e_1_3_4_6_2","doi-asserted-by":"publisher","DOI":"10.21061\/ctete-rms.v1.c.1"},{"key":"e_1_3_4_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/3699538.3699556"},{"key":"e_1_3_4_8_2","doi-asserted-by":"crossref","DOI":"10.1201\/9780367816377","volume-title":"Hands-on Machine Learning with R","author":"Boehmke Brad","year":"2019","unstructured":"Brad Boehmke and Brandon M. Greenwell. 2019. Hands-on Machine Learning with R. Chapman and Hall\/CRC."},{"key":"e_1_3_4_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/2994154"},{"key":"e_1_3_4_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3610969.3611117"},{"key":"e_1_3_4_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/2538862.2538924"},{"key":"e_1_3_4_12_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jml.2019.104047"},{"key":"e_1_3_4_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/3328778.3366847"},{"issue":"3","key":"e_1_3_4_14_2","doi-asserted-by":"crossref","first-page":"297","DOI":"10.1080\/17439760.2016.1262613","article-title":"Thematic analysis","volume":"12","author":"Clarke Victoria","year":"2017","unstructured":"Victoria Clarke and Virginia Braun. 2017. Thematic analysis. The Journal of Positive Psychology 12, 3 (2017), 297\u2013298.","journal-title":"The Journal of Positive Psychology"},{"key":"e_1_3_4_15_2","doi-asserted-by":"crossref","first-page":"245","DOI":"10.1145\/365024.365111","volume-title":"Proceedings of the SIGCHI Conference on Human Factors in Computing Systems","author":"Corbett Albert T.","year":"2001","unstructured":"Albert T. Corbett and John R. Anderson. 2001. Locus of feedback control in computer-based tutoring: Impact on learning rate, achievement and attitudes. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 245\u2013252."},{"key":"e_1_3_4_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/3160489.3160492"},{"key":"e_1_3_4_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/3649217.3653595"},{"key":"e_1_3_4_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3626252.3630909"},{"key":"e_1_3_4_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/3649217.3653574"},{"key":"e_1_3_4_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/3624720"},{"key":"e_1_3_4_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445696"},{"key":"e_1_3_4_22_2","doi-asserted-by":"publisher","DOI":"10.1001\/jamaophthalmol.2013.5521"},{"key":"e_1_3_4_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/2361276.2361288"},{"key":"e_1_3_4_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/3544549.3585737"},{"issue":"1","key":"e_1_3_4_25_2","first-page":"5","article-title":"Data driven automatic feedback generation in the iList intelligent tutoring system","volume":"10","author":"Fossati Davide","year":"2015","unstructured":"Davide Fossati, Barbara Di Eugenio, Stellan Ohlsson, Christopher Brown, and Lin Chen. 2015. Data driven automatic feedback generation in the iList intelligent tutoring system. Technology, Instruction, Cognition and Learning 10, 1 (2015), 5\u201326.","journal-title":"Technology, Instruction, Cognition and Learning"},{"key":"e_1_3_4_26_2","first-page":"173","volume-title":"Proceedings of the 7th Australasian Conference on Computing Education (ACE \u201905)","author":"Garner Sandy","year":"2005","unstructured":"Sandy Garner, Patricia Haden, and Anthony Robins. 2005. My program is correct but it doesn\u2019t run: A preliminary investigation of novice programmers\u2019 problems. In Proceedings of the 7th Australasian Conference on Computing Education (ACE \u201905), Vol. 42. Australian Computer Society, Inc., Australia, 173\u2013180."},{"key":"e_1_3_4_27_2","doi-asserted-by":"crossref","first-page":"259","DOI":"10.1007\/978-3-031-64299-9_20","volume-title":"Artificial Intelligence in Education","author":"Ghimire Aashish","year":"2024","unstructured":"Aashish Ghimire and John Edwards. 2024. Coding with AI: How are tools like ChatGPT being used by students in foundational programming courses. In Artificial Intelligence in Education. Andrew M. Olney, Irene-Angelica Chounta, Zitao Liu, Olga C. Santos, and Ig Ibert Bittencourt (Eds.), Springer Nature Switzerland, Cham, 259\u2013267."},{"key":"e_1_3_4_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2820011"},{"key":"e_1_3_4_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/3386527.3406733"},{"key":"e_1_3_4_30_2","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1109\/BLOCKS.2017.8120407","volume-title":"Proceedings of the 2017 IEEE Blocks and Beyond Workshop (B&B)","author":"Gusukuma Luke","year":"2017","unstructured":"Luke Gusukuma, Dennis Kafura, and Austin Cory Bart. 2017. Authoring feedback for novice programmers in a block-based language. In Proceedings of the 2017 IEEE Blocks and Beyond Workshop (B&B), 37\u201340. DOI: 10.1109\/BLOCKS.2017.8120407"},{"key":"e_1_3_4_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/3197091.3205845"},{"key":"e_1_3_4_32_2","doi-asserted-by":"publisher","DOI":"10.1145\/3568813.3600139"},{"key":"e_1_3_4_33_2","volume-title":"Proceedings of the Annual Meeting of the Cognitive Science Society","volume":"35","author":"Hellman Katharina","year":"2013","unstructured":"Katharina Hellman and Matthias Nuckles. 2013. Expert blind spot in pre-service and in-service mathematics teachers: Task design moderates overestimation of novices\u2019 performance. In Proceedings of the Annual Meeting of the Cognitive Science Society, Vol. 35."},{"key":"e_1_3_4_34_2","unstructured":"Luke Hewitt Ashwini Ashokkumar Isaias Ghezae and Robb Willer. 2024. Predicting Results of Social Science Experiments Using Large Language Models. Technical Report. Working Paper. Retrieved from https:\/\/samim.io\/dl\/Predicting%20results%20of%20social%20science%20experiments%20using%20large%20language%20models.pdf"},{"key":"e_1_3_4_35_2","doi-asserted-by":"crossref","first-page":"278","DOI":"10.1109\/ICDAR.1995.598994","volume-title":"Proceedings of the 3rd International Conference on Document Analysis and Recognition (ICDAR \u201995)","author":"Ho Tin Kam","year":"1995","unstructured":"Tin Kam Ho. 1995. Random decision forests. In Proceedings of the 3rd International Conference on Document Analysis and Recognition (ICDAR \u201995), Vol. 1. IEEE Computer Society, 278."},{"key":"e_1_3_4_36_2","doi-asserted-by":"publisher","DOI":"10.1609\/icwsm.v8i1.14550"},{"key":"e_1_3_4_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3202185.3202762"},{"key":"e_1_3_4_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/3571785.3574124"},{"key":"e_1_3_4_39_2","doi-asserted-by":"publisher","DOI":"10.1080\/1743727X.2023.2242273"},{"key":"e_1_3_4_40_2","unstructured":"Ishika Joshi Ritvik Budhiraja Pranav Deepak Tanna Lovenya Jain Mihika Deshpande Arjun Srivastava Srinivas Rallapalli Harshal D. Akolekar Jagat Sesh Challa and Dhruv Kumar. 2023. \u201cWith great power comes great responsibility!\u201d: Student and instructor perspectives on the influence of LLMs on undergraduate engineering education. arXiv:2309.10694. Retrieved from https:\/\/arxiv.org\/abs\/2309.10694"},{"key":"e_1_3_4_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/3544548.3580919"},{"key":"e_1_3_4_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/3593342.3593349"},{"key":"e_1_3_4_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/3231711"},{"key":"e_1_3_4_44_2","first-page":"1","volume-title":"Proceedings of the 2023 IEEE Frontiers in Education Conference (FIE)","author":"Kiesler Natalie","year":"2023","unstructured":"Natalie Kiesler, Dominic Lohr, and Hieke Keuning. 2023. Exploring the potential of large language models to generate formative programming feedback. In Proceedings of the 2023 IEEE Frontiers in Education Conference (FIE), 1\u20135. DOI: 10.1109\/FIE58773.2023.10343457"},{"key":"e_1_3_4_45_2","unstructured":"Natalie Kiesler and Daniel Schiffner. 2023. Large language models in introductory programming education: ChatGPT\u2019s performance and implications for assessments. arXiv:2308.08572. Retrieved from https:\/\/arxiv.org\/abs\/2308.08572"},{"key":"e_1_3_4_46_2","author":"Peter Kincaid","year":"1975","unstructured":"J. Peter Kincaid, Robert P. Fishburne Jr., Richard L. Rogers and Brad S. Chissom. 1975. Derivation of New Readability Formulas (Automated Readability Index, Fog Count And Flesch Reading Ease Formula) For Navy Enlisted Personnel. Institute for Simulation and Training. Retrieved from https:\/\/stars.library.ucf.edu\/istlibrary\/56","journal-title":"Derivation of New Readability Formulas (Automated Readability Index, Fog Count And Flesch Reading Ease Formula) For Navy Enlisted Personnel"},{"issue":"1","key":"e_1_3_4_47_2","first-page":"1","article-title":"PIAAC: A new design for a new era","volume":"5","author":"Kirsch Irwin","year":"2017","unstructured":"Irwin Kirsch and Mary Louise Lennon. 2017. PIAAC: A new design for a new era. Large-Scale Assessments in Education 5, 1 (2017), 1\u201322.","journal-title":"Large-Scale Assessments in Education"},{"issue":"5","key":"e_1_3_4_48_2","doi-asserted-by":"crossref","first-page":"757","DOI":"10.1111\/j.1551-6709.2012.01245.x","article-title":"The knowledge-learning-instruction framework: Bridging the science-practice chasm to enhance robust student learning","volume":"36","author":"Koedinger Kenneth R.","year":"2012","unstructured":"Kenneth R. Koedinger, Albert T. Corbett, and Charles Perfetti. 2012. The knowledge-learning-instruction framework: Bridging the science-practice chasm to enhance robust student learning. Cognitive Science 36, 5 (2012), 757\u2013798.","journal-title":"Cognitive Science"},{"key":"e_1_3_4_49_2","unstructured":"Neil Jones. 2016. \u2018No more marking\u2019: An online tool for comparative judgement. Cambridge English Language Assessment Research Notes 12 63 (2016) 12\u201315. Retrieved from https:\/\/www.cambridgeenglish.org\/images\/301971-research-notes-63.pdf#page=14"},{"key":"e_1_3_4_50_2","doi-asserted-by":"publisher","DOI":"10.1145\/3641554.3701791"},{"key":"e_1_3_4_51_2","doi-asserted-by":"publisher","DOI":"10.1145\/3649217.3653612"},{"key":"e_1_3_4_52_2","doi-asserted-by":"publisher","DOI":"10.1145\/3568813.3600138"},{"key":"e_1_3_4_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/3587102.3588785"},{"key":"e_1_3_4_54_2","doi-asserted-by":"publisher","DOI":"10.1145\/3545945.3569770"},{"key":"e_1_3_4_55_2","doi-asserted-by":"publisher","unstructured":"Mark Liffiton Brad E. Sheese Jaromir Savelka and Paul Denny. 2024. CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes 11 pages. DOI: 10.1145\/3631802.3631830","DOI":"10.1145\/3631802.3631830"},{"issue":"9","key":"e_1_3_4_56_2","doi-asserted-by":"crossref","first-page":"240682","DOI":"10.1098\/rsos.240682","article-title":"Can large language models help predict results from a complex behavioural science study","volume":"11","author":"Lippert Steffen","year":"2024","unstructured":"Steffen Lippert, Anna Dreber, Magnus Johannesson, Warren Tierney, Wilson Cyrus-Lai, Eric Luis Uhlmann, Emotion Expression Collaboration, and Thomas Pfeiffer. 2024. Can large language models help predict results from a complex behavioural science study? Royal Society Open Science 11, 9 (2024), 240682.","journal-title":"Royal Society Open Science"},{"key":"e_1_3_4_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/3649217.3653530"},{"key":"e_1_3_4_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/3545945.3569785"},{"key":"e_1_3_4_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/3610969.3610982"},{"key":"e_1_3_4_60_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-024-10561-6"},{"key":"e_1_3_4_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/3291279.3339420"},{"key":"e_1_3_4_62_2","doi-asserted-by":"publisher","DOI":"10.1145\/3304221.3319759"},{"key":"e_1_3_4_63_2","doi-asserted-by":"publisher","DOI":"10.1109\/TLT.2022.3223577"},{"key":"e_1_3_4_64_2","doi-asserted-by":"publisher","DOI":"10.1145\/3469885"},{"issue":"5","key":"e_1_3_4_65_2","doi-asserted-by":"crossref","first-page":"921","DOI":"10.1080\/03075079.2013.855718","article-title":"An anatomy of feedback: A phenomenographic investigation of undergraduate students\u2019 conceptions of feedback","volume":"40","author":"McLean Angela J.","year":"2015","unstructured":"Angela J. McLean, Carol H. Bond, and Helen D. Nicholson. 2015. An anatomy of feedback: A phenomenographic investigation of undergraduate students\u2019 conceptions of feedback. Studies in Higher Education 40, 5 (2015), 921\u2013932.","journal-title":"Studies in Higher Education"},{"key":"e_1_3_4_66_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639187"},{"key":"e_1_3_4_67_2","first-page":"125","volume-title":"Handbook of Research on Educational Communications and Technology","author":"Narciss Susanne","year":"2008","unstructured":"Susanne Narciss. 2008. Feedback strategies for interactive learning tasks. In Handbook of Research on Educational Communications and Technology. J. Michael Spector, M. David Merrill, Jeroen van Merri\u00ebnboer, and Marcy P. Driscoll (Eds.), Routledge, 125\u2013143."},{"key":"e_1_3_4_68_2","first-page":"644","volume-title":"Proceedings of the 3rd International Conference on Cognitive Science","volume":"644648","author":"Nathan Mitchell J.","year":"2001","unstructured":"Mitchell J. Nathan, Kenneth R. Koedinger, and Martha W. Alibali. 2001. Expert blind spot: When content knowledge eclipses pedagogical content knowledge. In Proceedings of the 3rd International Conference on Cognitive Science, Vol. 644648, 644\u2013648."},{"key":"e_1_3_4_69_2","doi-asserted-by":"publisher","DOI":"10.1145\/3626252.3630960"},{"key":"e_1_3_4_70_2","doi-asserted-by":"publisher","DOI":"10.1145\/3613904.3642706"},{"key":"e_1_3_4_71_2","doi-asserted-by":"publisher","DOI":"10.1145\/3430665.3456344"},{"key":"e_1_3_4_72_2","volume-title":"Joint Proceedings of the Workshops of the 14th International Conference on Educational Data Mining (EDM)","author":"Paa\u00dfen Benjamin","year":"2021","unstructured":"Benjamin Paa\u00dfen, Jessica McBroom, Bryn Jeffries, Irena Koprinska, and Kalina Yacef. 2021. Next steps for next-step hints: Lessons learned from teacher evaluations of automatic programming hints. In Joint Proceedings of the Workshops of the 14th International Conference on Educational Data Mining (EDM)."},{"key":"e_1_3_4_73_2","doi-asserted-by":"publisher","DOI":"10.1145\/3649217.3653608"},{"key":"e_1_3_4_74_2","doi-asserted-by":"publisher","DOI":"10.1145\/3059009.3059058"},{"key":"e_1_3_4_75_2","doi-asserted-by":"publisher","DOI":"10.1080\/0969594X.2012.665354"},{"key":"e_1_3_4_76_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.caeai.2024.100289"},{"key":"e_1_3_4_77_2","doi-asserted-by":"publisher","DOI":"10.1145\/3545945.3569791"},{"key":"e_1_3_4_78_2","doi-asserted-by":"publisher","DOI":"10.1145\/3230977.3230981"},{"key":"e_1_3_4_79_2","doi-asserted-by":"publisher","DOI":"10.1145\/3105726.3106169"},{"key":"e_1_3_4_80_2","doi-asserted-by":"publisher","DOI":"10.1145\/3617367"},{"key":"e_1_3_4_81_2","doi-asserted-by":"publisher","DOI":"10.1145\/3632620.3671116"},{"key":"e_1_3_4_82_2","doi-asserted-by":"publisher","DOI":"10.1145\/3430895.3460159"},{"key":"e_1_3_4_83_2","doi-asserted-by":"crossref","first-page":"246","DOI":"10.1007\/978-3-030-52240-7_45","volume-title":"Artificial Intelligence in Education","author":"Price Thomas W.","year":"2020","unstructured":"Thomas W. Price, Samiha Marwan, Michael Winters, and Joseph Jay Williams. 2020. An evaluation of data-driven programming hints in a classroom setting. In Artificial Intelligence in Education. Ig Ibert Bittencourt, Mutlu Cukurova, Kasia Muldner, Rose Luckin, and Eva Mill\u00e1n (Eds.), Springer International Publishing, Cham, 246\u2013251."},{"key":"e_1_3_4_84_2","doi-asserted-by":"publisher","DOI":"10.1145\/3561833.3561843"},{"issue":"2","key":"e_1_3_4_85_2","doi-asserted-by":"crossref","first-page":"244","DOI":"10.1016\/j.jesp.2005.04.005","article-title":"Productivity is not enough: A comparison of interactive and nominal brainstorming groups on idea generation and selection","volume":"42","author":"Rietzschel Eric F.","year":"2006","unstructured":"Eric F. Rietzschel, Bernard A. Nijstad, and Wolfgang Stroebe. 2006. Productivity is not enough: A comparison of interactive and nominal brainstorming groups on idea generation and selection. Journal of Experimental Social Psychology 42, 2 (2006), 244\u2013251.","journal-title":"Journal of Experimental Social Psychology"},{"key":"e_1_3_4_86_2","volume-title":"Automated Data-Driven Hint Generation for Learning Programming","author":"Rivers Kelly","year":"2017","unstructured":"Kelly Rivers. 2017. Automated Data-Driven Hint Generation for Learning Programming. Ph.D. Dissertation. Carnegie Mellon University."},{"key":"e_1_3_4_87_2","doi-asserted-by":"publisher","DOI":"10.1145\/3636243.3636259"},{"key":"e_1_3_4_88_2","doi-asserted-by":"publisher","DOI":"10.1080\/0969594X.2019.1602027"},{"key":"e_1_3_4_89_2","doi-asserted-by":"crossref","unstructured":"Andreas Scholl and Natalie Kiesler. 2024. How novice programmers use and experience ChatGPT when solving programming exercises in an introductory course. arXiv:2407.20792. Retrieved from https:\/\/arxiv.org\/abs\/2407.20792","DOI":"10.1109\/FIE61694.2024.10893442"},{"key":"e_1_3_4_90_2","doi-asserted-by":"publisher","DOI":"10.1145\/3626252.3630880"},{"key":"e_1_3_4_91_2","doi-asserted-by":"publisher","DOI":"10.1145\/3636243.3636249"},{"key":"e_1_3_4_92_2","doi-asserted-by":"publisher","DOI":"10.3102\/0034654307313795"},{"key":"e_1_3_4_93_2","doi-asserted-by":"publisher","DOI":"10.1145\/3287324.3287394"},{"key":"e_1_3_4_94_2","doi-asserted-by":"publisher","DOI":"10.1145\/3027063.3053187"},{"key":"e_1_3_4_95_2","doi-asserted-by":"publisher","DOI":"10.1145\/3589004"},{"key":"e_1_3_4_96_2","doi-asserted-by":"publisher","DOI":"10.1145\/3408877.3432538"},{"key":"e_1_3_4_97_2","doi-asserted-by":"publisher","DOI":"10.18637\/jss.v077.i01"},{"key":"e_1_3_4_98_2","doi-asserted-by":"publisher","DOI":"10.1145\/3613905.3650937"},{"key":"e_1_3_4_99_2","doi-asserted-by":"publisher","DOI":"10.1145\/3639474.3640076"},{"key":"e_1_3_4_100_2","doi-asserted-by":"publisher","DOI":"10.1145\/3544548.3581388"}],"container-title":["ACM Transactions on Computing Education"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3737885","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,22]],"date-time":"2025-07-22T23:24:40Z","timestamp":1753226680000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3737885"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,22]]},"references-count":99,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2025,9,30]]}},"alternative-id":["10.1145\/3737885"],"URL":"https:\/\/doi.org\/10.1145\/3737885","relation":{},"ISSN":["1946-6226"],"issn-type":[{"value":"1946-6226","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,7,22]]},"assertion":[{"value":"2024-11-26","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-05-04","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-07-22","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}