{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,24]],"date-time":"2026-04-24T17:48:23Z","timestamp":1777052903977,"version":"3.51.4"},"publisher-location":"New York, NY, USA","reference-count":44,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,3,3]],"date-time":"2021-03-03T00:00:00Z","timestamp":1614729600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["1915257"],"award-info":[{"award-number":["1915257"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,3,3]]},"DOI":"10.1145\/3408877.3432539","type":"proceedings-article","created":{"date-parts":[[2021,3,5]],"date-time":"2021-03-05T18:46:01Z","timestamp":1614969961000},"page":"1163-1169","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":35,"title":["Autograding \"Explain in Plain English\" questions using NLP"],"prefix":"10.1145","author":[{"given":"Max","family":"Fowler","sequence":"first","affiliation":[{"name":"University of Illinois, Urbana, IL, USA"}]},{"given":"Binglin","family":"Chen","sequence":"additional","affiliation":[{"name":"University of Illinois, Urbana, IL, USA"}]},{"given":"Sushmita","family":"Azad","sequence":"additional","affiliation":[{"name":"University of Illinois, Urbana, IL, USA"}]},{"given":"Matthew","family":"West","sequence":"additional","affiliation":[{"name":"University of Illinois, Urbana, IL, USA"}]},{"given":"Craig","family":"Zilles","sequence":"additional","affiliation":[{"name":"University of Illinois, Champaign, IL, USA"}]}],"member":"320","published-online":{"date-parts":[[2021,3,5]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/199688.199694"},{"key":"e_1_3_2_1_2_1","volume-title":"Strategies for Deploying Unreliable AI Graders in High-Transparency High-Stakes Exams. In International Conference on Artificial Intelligence in Education. Springer, 16--28","author":"Azad Sushmita","year":"2020","unstructured":"Sushmita Azad , Binglin Chen , Maxwell Fowler , Matthew West , and Craig Zilles . 2020 . Strategies for Deploying Unreliable AI Graders in High-Transparency High-Stakes Exams. In International Conference on Artificial Intelligence in Education. Springer, 16--28 . Sushmita Azad, Binglin Chen, Maxwell Fowler, Matthew West, and Craig Zilles. 2020. Strategies for Deploying Unreliable AI Graders in High-Transparency High-Stakes Exams. In International Conference on Artificial Intelligence in Education. Springer, 16--28."},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1007\/s40593-014-0026-8"},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3328778.3366879"},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"crossref","unstructured":"Binglin Chen Matthew West and Craig Zilles. 2018. How Much Randomization is Needed to Deter Collaborative Cheating on Asynchronous Exams?. In Learning at Scale .  Binglin Chen Matthew West and Craig Zilles. 2018. How Much Randomization is Needed to Deter Collaborative Cheating on Asynchronous Exams?. In Learning at Scale .","DOI":"10.1145\/3231644.3231664"},{"key":"e_1_3_2_1_6_1","volume-title":"Patterns and Pedagogy. In The Proceedings of the Thirtieth SIGCSE Technical Symposium on Computer Science Education (SIGCSE '99)","author":"Michael","unstructured":"Michael J. Clancy and Marcia C. Linn. 1999 . Patterns and Pedagogy. In The Proceedings of the Thirtieth SIGCSE Technical Symposium on Computer Science Education (SIGCSE '99) . ACM, New York, NY, USA, 37--42. Michael J. Clancy and Marcia C. Linn. 1999. Patterns and Pedagogy. In The Proceedings of the Thirtieth SIGCSE Technical Symposium on Computer Science Education (SIGCSE '99). ACM, New York, NY, USA, 37--42."},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2538862.2538911"},{"key":"e_1_3_2_1_8_1","volume-title":"Proceedings of the Thirteenth Australasian Computing Education Conference -","volume":"114","author":"Corney Malcolm","year":"2011","unstructured":"Malcolm Corney , Raymond Lister , and Donna Teague . 2011 . Early Relational Reasoning and the Novice Programmer: Swapping As the \"Hello World\" of Relational Reasoning . In Proceedings of the Thirteenth Australasian Computing Education Conference - Volume 114 (ACE '11). 95--104. Malcolm Corney, Raymond Lister, and Donna Teague. 2011. Early Relational Reasoning and the Novice Programmer: Swapping As the \"Hello World\" of Relational Reasoning. In Proceedings of the Thirteenth Australasian Computing Education Conference - Volume 114 (ACE '11). 95--104."},{"key":"e_1_3_2_1_9_1","volume-title":"Proceedings of the 2nd joint conference on lexical and computational semantics, M. Diab, T. Baldwin, and M. Baroni (Eds.). 1--12","author":"Dzikovska M. O.","year":"2013","unstructured":"M. O. Dzikovska 2013 . SemEval-2013 task 7: The joint student response analysis and eighth recognizing textual entailment challenge . In Proceedings of the 2nd joint conference on lexical and computational semantics, M. Diab, T. Baldwin, and M. Baroni (Eds.). 1--12 . M. O. Dzikovska et almbox. 2013. SemEval-2013 task 7: The joint student response analysis and eighth recognizing textual entailment challenge. In Proceedings of the 2nd joint conference on lexical and computational semantics, M. Diab, T. Baldwin, and M. Baroni (Eds.). 1--12."},{"key":"e_1_3_2_1_10_1","unstructured":"Hewlett Foundation. 2012. Automated student assessment prize: Phase two -- short answer scoring Kaggle Competition.  Hewlett Foundation. 2012. Automated student assessment prize: Phase two -- short answer scoring Kaggle Competition."},{"key":"e_1_3_2_1_11_1","volume-title":"Advances in Artificial Intelligence - IBERAMIA 2018 , , Guillermo R","author":"Galhardi Lucas Busatta","unstructured":"Lucas Busatta Galhardi and Jacques Du\u00edlio Brancher . 2018. Machine Learning Approach for Automatic Short Answer Grading: A Systematic Review . In Advances in Artificial Intelligence - IBERAMIA 2018 , , Guillermo R . Simari, Eduardo Ferm\u00e9, Flabio Guti\u00e9rrez Segura, and Jos\u00e9 Antonio Rodr\u00edguez Melquiades (Eds.). Springer International Publishing , Cham , 380--391. Lucas Busatta Galhardi and Jacques Du\u00edlio Brancher. 2018. Machine Learning Approach for Automatic Short Answer Grading: A Systematic Review. In Advances in Artificial Intelligence - IBERAMIA 2018 , , Guillermo R. Simari, Eduardo Ferm\u00e9, Flabio Guti\u00e9rrez Segura, and Jos\u00e9 Antonio Rodr\u00edguez Melquiades (Eds.). Springer International Publishing, Cham, 380--391."},{"key":"e_1_3_2_1_12_1","volume-title":"Steve Croker, Peter CH Cheng, Gary Jones, Iain Oliver, and Julian M Pine.","author":"Gobet Fernand","year":"2001","unstructured":"Fernand Gobet , Peter CR Lane , Steve Croker, Peter CH Cheng, Gary Jones, Iain Oliver, and Julian M Pine. 2001 . Chunking mechanisms in human learning. Trends in cognitive sciences , Vol. 5 , 6 (2001), 236--243. Fernand Gobet, Peter CR Lane, Steve Croker, Peter CH Cheng, Gary Jones, Iain Oliver, and Julian M Pine. 2001. Chunking mechanisms in human learning. Trends in cognitive sciences , Vol. 5, 6 (2001), 236--243."},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-14118-9_59"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1177\/0092055X13490752"},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3408877.3432442"},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v33i01.33019662"},{"key":"e_1_3_2_1_17_1","volume-title":"Kate Sanders, Otto Sepp\"al\"a, Beth Simon, and Lynda Thomas.","author":"Lister Raymond","year":"2004","unstructured":"Raymond Lister , Elizabeth S Adams , Sue Fitzgerald , William Fone , John Hamer , Morten Lindholm , Robert McCartney , Jan Erik Mostr\u00f6m , Kate Sanders, Otto Sepp\"al\"a, Beth Simon, and Lynda Thomas. 2004 . A multi-national study of reading and tracing skills in novice programmers. ACM SIGCSE Bulletin , Vol. 36 , 4 (2004), 119--150. Raymond Lister, Elizabeth S Adams, Sue Fitzgerald, William Fone, John Hamer, Morten Lindholm, Robert McCartney, Jan Erik Mostr\u00f6m, Kate Sanders, Otto Sepp\"al\"a, Beth Simon, and Lynda Thomas. 2004. A multi-national study of reading and tracing skills in novice programmers. ACM SIGCSE Bulletin , Vol. 36, 4 (2004), 119--150."},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/1562877.1562930"},{"key":"e_1_3_2_1_19_1","volume-title":"Gale Yan Huang, and Zitao Liu","author":"Liu Tiaoqiao","year":"2019","unstructured":"Tiaoqiao Liu , Wenbiao Ding , Zhiwei Wang , Jiliang Tang , Gale Yan Huang, and Zitao Liu . 2019 . Automatic Short Answer Grading via Multiway Attention Networks . arXiv:1909.10166 [cs] (2019). http:\/\/arxiv.org\/abs\/1909.10166 Tiaoqiao Liu, Wenbiao Ding, Zhiwei Wang, Jiliang Tang, Gale Yan Huang, and Zitao Liu. 2019. Automatic Short Answer Grading via Multiway Attention Networks . arXiv:1909.10166 [cs] (2019). http:\/\/arxiv.org\/abs\/1909.10166"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/1404520.1404531"},{"key":"e_1_3_2_1_21_1","volume-title":"Vector Based Techniques for Short Answer Grading. In FLAIRS Conference .","author":"Magooda Ahmed","unstructured":"Ahmed Magooda , Mohamed A. Zahran , Mohsen Rashwan , Hazem M. Raafat , and Magda B. Fayek . 2016 . Vector Based Techniques for Short Answer Grading. In FLAIRS Conference . Ahmed Magooda, Mohamed A. Zahran, Mohsen Rashwan, Hazem M. Raafat, and Magda B. Fayek. 2016. Vector Based Techniques for Short Answer Grading. In FLAIRS Conference ."},{"key":"e_1_3_2_1_22_1","unstructured":"Alvarado Mantecon and Jesus Gerardo. 2019. Towards the Automatic Classification of Student Answers to Open-ended Questions. Thesis. Universit\u00e9 d'Ottawa \/ University of Ottawa. https:\/\/doi.org\/10.20381\/ruor-23341  Alvarado Mantecon and Jesus Gerardo. 2019. Towards the Automatic Classification of Student Answers to Open-ended Questions. Thesis. Universit\u00e9 d'Ottawa \/ University of Ottawa. https:\/\/doi.org\/10.20381\/ruor-23341"},{"key":"e_1_3_2_1_23_1","volume-title":"Schemas in problem solving","author":"Marshall Sandra P","unstructured":"Sandra P Marshall . 1995. Schemas in problem solving . Cambridge University Press . Sandra P Marshall. 1995. Schemas in problem solving .Cambridge University Press."},{"key":"e_1_3_2_1_24_1","volume-title":"mbox","author":"McCracken Michael","year":"2001","unstructured":"Michael McCracken mbox . 2001 . A Multi-national, Multi-institutional Study of Assessment of Programming Skills of First-year CS Students. In Working Group Reports from ITiCSE on Innovation and Technology in Computer Science Education (ITiCSE-WGR '01). ACM, New York, NY, USA, 125--180. Michael McCracken et almbox. 2001. A Multi-national, Multi-institutional Study of Assessment of Programming Skills of First-year CS Students. In Working Group Reports from ITiCSE on Innovation and Technology in Computer Science Education (ITiCSE-WGR '01). ACM, New York, NY, USA, 125--180."},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1016\/0010-0285(81)90012-8"},{"key":"e_1_3_2_1_26_1","volume-title":"Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies . 752--762","author":"Mohler M.","unstructured":"M. Mohler , R. Bunescu , and R. Mihalcea . 2011. Learning to grade short answer questions using semantic similarity measures and dependency graph alignments . In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies . 752--762 . M. Mohler, R. Bunescu, and R. Mihalcea. 2011. Learning to grade short answer questions using semantic similarity measures and dependency graph alignments. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies . 752--762."},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2157136.2157249"},{"key":"e_1_3_2_1_28_1","volume-title":"Amadi","author":"Ndukwe Ifeanyi G.","year":"2019","unstructured":"Ifeanyi G. Ndukwe , Ben K. Daniel , and Chukwudi E . Amadi . 2019 . A Machine Learning Grading System Using Chatbots. In Artificial Intelligence in Education (Lecture Notes in Computer Science ). Springer International Publishing , Cham, 365--368. Ifeanyi G. Ndukwe, Ben K. Daniel, and Chukwudi E. Amadi. 2019. A Machine Learning Grading System Using Chatbots. In Artificial Intelligence in Education (Lecture Notes in Computer Science ). Springer International Publishing, Cham, 365--368."},{"key":"e_1_3_2_1_29_1","volume-title":"Timing matters: the impact of immediate and delayed feedback on artificial language learning. Frontiers in human neuroscience","author":"Opitz Bertram","year":"2011","unstructured":"Bertram Opitz , Nicola K Ferdinand , and Axel Mecklinger . 2011. Timing matters: the impact of immediate and delayed feedback on artificial language learning. Frontiers in human neuroscience , Vol. 5 ( 2011 ), 8. Bertram Opitz, Nicola K Ferdinand, and Axel Mecklinger. 2011. Timing matters: the impact of immediate and delayed feedback on artificial language learning. Frontiers in human neuroscience , Vol. 5 (2011), 8."},{"key":"e_1_3_2_1_30_1","volume-title":"Learning from Worked-Out Examples: The Effects of Example Variability and Elicited Self-Explanations. Contemporary educational psychology","author":"Renkl Alexander","year":"1998","unstructured":"Alexander Renkl , Robin Stark , Hans Gruber , and Heinz Mandl . 1998. Learning from Worked-Out Examples: The Effects of Example Variability and Elicited Self-Explanations. Contemporary educational psychology , Vol. 23 (01 1998 ), 90--108. https:\/\/doi.org\/10.1006\/ceps.1997.0959 Alexander Renkl, Robin Stark, Hans Gruber, and Heinz Mandl. 1998. Learning from Worked-Out Examples: The Effects of Example Variability and Elicited Self-Explanations. Contemporary educational psychology , Vol. 23 (01 1998), 90--108. https:\/\/doi.org\/10.1006\/ceps.1997.0959"},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1207\/s15516709cog1303_3"},{"key":"e_1_3_2_1_32_1","volume-title":"Artificial Intelligence in Education (Lecture Notes in Computer Science )","author":"Saha Swarnadeep","unstructured":"Swarnadeep Saha , Tejas I. Dhamecha , Smit Marvaniya , Renuka Sindhgatta , and Bikram Sengupta . 2018. Sentence Level or Token Level Features for Automatic Short Answer Grading?: Use Both . In Artificial Intelligence in Education (Lecture Notes in Computer Science ) . Springer International Publishing , Cham , 503--517. Swarnadeep Saha, Tejas I. Dhamecha, Smit Marvaniya, Renuka Sindhgatta, and Bikram Sengupta. 2018. Sentence Level or Token Level Features for Automatic Short Answer Grading?: Use Both. In Artificial Intelligence in Education (Lecture Notes in Computer Science ). Springer International Publishing, Cham, 503--517."},{"key":"e_1_3_2_1_33_1","volume-title":"Tejas Indulal Dhamecha, and Nirmal Mukhi","author":"Sung Chul","year":"2019","unstructured":"Chul Sung , Tejas Indulal Dhamecha, and Nirmal Mukhi . 2019 . Improving Short Answer Grading Using Transformer-Based Pre-training. In Artificial Intelligence in Education . Vol. 11625 . Springer International Publishing , Cham, 469--481. Chul Sung, Tejas Indulal Dhamecha, and Nirmal Mukhi. 2019. Improving Short Answer Grading Using Transformer-Based Pre-training. In Artificial Intelligence in Education . Vol. 11625. Springer International Publishing, Cham, 469--481."},{"key":"e_1_3_2_1_34_1","volume-title":"Automatic Short Answer Grading and Feedback Using Text Mining Methods . CoRR","author":"Suzen Neslihan","year":"2019","unstructured":"Neslihan Suzen , Alexander Gorban , Jeremy Levesley , and Evgeny Mirkes . 2019. Automatic Short Answer Grading and Feedback Using Text Mining Methods . CoRR ( 2019 ). http:\/\/arxiv.org\/abs\/1807.10543 arXiv: 1807.10543. Neslihan Suzen, Alexander Gorban, Jeremy Levesley, and Evgeny Mirkes. 2019. Automatic Short Answer Grading and Feedback Using Text Mining Methods . CoRR (2019). http:\/\/arxiv.org\/abs\/1807.10543 arXiv: 1807.10543."},{"key":"e_1_3_2_1_35_1","volume-title":"Psychology of learning and motivation .","author":"Sweller John","unstructured":"John Sweller . 2011. Cognitive Load Theory . In Psychology of learning and motivation . Vol. 55 . Elsevier , 37--76. John Sweller. 2011. Cognitive Load Theory . In Psychology of learning and motivation . Vol. 55. Elsevier, 37--76."},{"key":"e_1_3_2_1_36_1","volume-title":"Jeroen JG Van Merrienboer, and Fred GWC Paas","author":"Sweller John","year":"1998","unstructured":"John Sweller , Jeroen JG Van Merrienboer, and Fred GWC Paas . 1998 . Cognitive architecture and instructional design. Educational psychology review , Vol. 10 , 3 (1998), 251--296. John Sweller, Jeroen JG Van Merrienboer, and Fred GWC Paas. 1998. Cognitive architecture and instructional design. Educational psychology review , Vol. 10, 3 (1998), 251--296."},{"key":"e_1_3_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/1584322.1584336"},{"key":"e_1_3_2_1_38_1","volume-title":"et almbox","author":"Viera Anthony J","year":"2005","unstructured":"Anthony J Viera , Joanne M Garrett , et almbox . 2005 . Understanding interobserver agreement: the kappa statistic. Fam med , Vol. 37 , 5 (2005), 360--363. Anthony J Viera, Joanne M Garrett, et almbox. 2005. Understanding interobserver agreement: the kappa statistic. Fam med , Vol. 37, 5 (2005), 360--363."},{"key":"e_1_3_2_1_39_1","volume-title":"Eighth Australasian Computing Education Conference (ACE2006)","author":"Whalley Jacqueline","year":"2006","unstructured":"Jacqueline Whalley , Raymond Lister , Errol Thompson , Tony Clear , Phil Robbins , P K Ajith Kumar , and Christine Prasad . 2006 . An Australasian study of Reading and Comprehension Skills in Novice Programmers, using the Bloom and SOLO Taxonomies . Eighth Australasian Computing Education Conference (ACE2006) (2006). Jacqueline Whalley, Raymond Lister, Errol Thompson, Tony Clear, Phil Robbins, P K Ajith Kumar, and Christine Prasad. 2006. An Australasian study of Reading and Comprehension Skills in Novice Programmers, using the Bloom and SOLO Taxonomies . Eighth Australasian Computing Education Conference (ACE2006) (2006)."},{"key":"e_1_3_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0020-7373(85)80041-9"},{"key":"e_1_3_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/234867.234872"},{"key":"e_1_3_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1080\/08993408.2019.1565235"},{"key":"e_1_3_2_1_43_1","volume-title":"Artificial Intelligence in Education (Lecture Notes in Computer Science )","author":"Yang Xi","unstructured":"Xi Yang , Yuwei Huang , Fuzhen Zhuang , Lishan Zhang , and Shengquan Yu. 2018. Automatic Chinese Short Answer Grading with Deep Autoencoder . In Artificial Intelligence in Education (Lecture Notes in Computer Science ) . Springer International Publishing , Cham , 399--404. Xi Yang, Yuwei Huang, Fuzhen Zhuang, Lishan Zhang, and Shengquan Yu. 2018. Automatic Chinese Short Answer Grading with Deep Autoencoder. In Artificial Intelligence in Education (Lecture Notes in Computer Science ). Springer International Publishing, Cham, 399--404."},{"key":"e_1_3_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.5220\/0007753304140420"}],"event":{"name":"SIGCSE '21: The 52nd ACM Technical Symposium on Computer Science Education","location":"Virtual Event USA","acronym":"SIGCSE '21","sponsor":["SIGCSE ACM Special Interest Group on Computer Science Education"]},"container-title":["Proceedings of the 52nd ACM Technical Symposium on Computer Science Education"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3408877.3432539","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3408877.3432539","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3408877.3432539","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T21:24:26Z","timestamp":1750195466000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3408877.3432539"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,3,3]]},"references-count":44,"alternative-id":["10.1145\/3408877.3432539","10.1145\/3408877"],"URL":"https:\/\/doi.org\/10.1145\/3408877.3432539","relation":{},"subject":[],"published":{"date-parts":[[2021,3,3]]},"assertion":[{"value":"2021-03-05","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}