{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,4]],"date-time":"2026-03-04T02:47:17Z","timestamp":1772592437103,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":58,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,5,6]],"date-time":"2021-05-06T00:00:00Z","timestamp":1620259200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100004705","name":"Capital One Financial Corporation","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100004705","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["1564041"],"award-info":[{"award-number":["1564041"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,5,6]]},"DOI":"10.1145\/3411764.3445424","type":"proceedings-article","created":{"date-parts":[[2021,5,8]],"date-time":"2021-05-08T02:46:03Z","timestamp":1620441963000},"page":"1-15","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":48,"title":["Attitudes Surrounding an Imperfect AI Autograder"],"prefix":"10.1145","author":[{"given":"Silas","family":"Hsu","sequence":"first","affiliation":[{"name":"Department of Computer Science, University of Illinois at Urbana-Champaign, United States"}]},{"given":"Tiffany Wenting","family":"Li","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Illinois at Urbana-Champaign, United States"}]},{"given":"Zhilin","family":"Zhang","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Illinois at Urbana-Champaign, United States"}]},{"given":"Max","family":"Fowler","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Illinois at Urbana-Champaign, United States"}]},{"given":"Craig","family":"Zilles","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Illinois at Urbana-Champaign, United States"}]},{"given":"Karrie","family":"Karahalios","sequence":"additional","affiliation":[{"name":"Department of Computer Science, University of Illinois at Urbana-Champaign, United States"}]}],"member":"320","published-online":{"date-parts":[[2021,5,7]]},"reference":[{"key":"e_1_3_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.2147\/JMDH.S104807"},{"key":"e_1_3_2_2_2_1","volume-title":"Automated essay scoring with e-rater\u00ae v. 2.0. ETS Research Report Series 2004, 2","author":"Attali Yigal","year":"2004","unstructured":"Yigal Attali and Jill Burstein . 2004. Automated essay scoring with e-rater\u00ae v. 2.0. ETS Research Report Series 2004, 2 ( 2004 ), i\u201321. Yigal Attali and Jill Burstein. 2004. Automated essay scoring with e-rater\u00ae v. 2.0. ETS Research Report Series 2004, 2 (2004), i\u201321."},{"key":"e_1_3_2_2_3_1","volume-title":"Effect of immediate feedback and revision on psychometric properties of open-ended GRE\u00ae subject test items. ETS Research Report Series 2008, 1","author":"Attali Yigal","year":"2008","unstructured":"Yigal Attali and Don Powers . 2008. Effect of immediate feedback and revision on psychometric properties of open-ended GRE\u00ae subject test items. ETS Research Report Series 2008, 1 ( 2008 ), i\u201323. Yigal Attali and Don Powers. 2008. Effect of immediate feedback and revision on psychometric properties of open-ended GRE\u00ae subject test items. ETS Research Report Series 2008, 1 (2008), i\u201323."},{"key":"e_1_3_2_2_4_1","volume-title":"AUTOMATED SCORING OF SHORT-ANSWER OPEN-ENDED GRE\u00ae SUBJECT TEST ITEMS. ETS Research Report Series 2008, 1","author":"Attali Yigal","year":"2008","unstructured":"Yigal Attali , Don Powers , Marshall Freedman , Marissa Harrison , and Susan Obetz . 2008. AUTOMATED SCORING OF SHORT-ANSWER OPEN-ENDED GRE\u00ae SUBJECT TEST ITEMS. ETS Research Report Series 2008, 1 ( 2008 ), i\u201322. Yigal Attali, Don Powers, Marshall Freedman, Marissa Harrison, and Susan Obetz. 2008. AUTOMATED SCORING OF SHORT-ANSWER OPEN-ENDED GRE\u00ae SUBJECT TEST ITEMS. ETS Research Report Series 2008, 1 (2008), i\u201322."},{"key":"e_1_3_2_2_5_1","volume-title":"International Conference on Artificial Intelligence in Education","author":"Azad Sushmita","unstructured":"Sushmita Azad , Binglin Chen , Maxwell Fowler , Matthew West , and Craig Zilles . 2020. Strategies for Deploying Unreliable AI Graders in High-Transparency High-Stakes Exams . In International Conference on Artificial Intelligence in Education . Springer, Springer International Publishing , Cham, 16\u201328. Sushmita Azad, Binglin Chen, Maxwell Fowler, Matthew West, and Craig Zilles. 2020. Strategies for Deploying Unreliable AI Graders in High-Transparency High-Stakes Exams. In International Conference on Artificial Intelligence in Education. Springer, Springer International Publishing, Cham, 16\u201328."},{"key":"e_1_3_2_2_6_1","unstructured":"BBC. 2020. A-levels and GCSEs: How did the exam algorithm work?https:\/\/www.bbc.com\/news\/explainers-53807730. Accessed 2020-09-16.  BBC. 2020. A-levels and GCSEs: How did the exam algorithm work?https:\/\/www.bbc.com\/news\/explainers-53807730. Accessed 2020-09-16."},{"key":"e_1_3_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.2517-6161.1995.tb02031.x"},{"key":"e_1_3_2_2_8_1","volume-title":"Considering Fairness and Validity in Evaluating Automated Scoring. In Annual meeting of the National Council on Measurement in Education","author":"Bridgeman Brent","year":"2009","unstructured":"Brent Bridgeman , Catherine Trapani , and Yigal Attali . 2009 . Considering Fairness and Validity in Evaluating Automated Scoring. In Annual meeting of the National Council on Measurement in Education ( San Diego, CA, USA). NCME, Mt. Royal, NJ, 1\u201318. Brent Bridgeman, Catherine Trapani, and Yigal Attali. 2009. Considering Fairness and Validity in Evaluating Automated Scoring. In Annual meeting of the National Council on Measurement in Education (San Diego, CA, USA). NCME, Mt. Royal, NJ, 1\u201318."},{"key":"e_1_3_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1080\/08957347.2012.635502"},{"key":"e_1_3_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1007\/s40593-014-0026-8"},{"key":"e_1_3_2_2_11_1","volume-title":"Handbook of Automated Essay Evaluation: Current Applications and New Directions, Mark\u00a0D Shermis and Jill Burstein (Eds.)","author":"Burstein Jill","unstructured":"Jill Burstein , Joel Tetreault , and Nitin Madnani . 2013. The e-rater automated essay scoring system . In Handbook of Automated Essay Evaluation: Current Applications and New Directions, Mark\u00a0D Shermis and Jill Burstein (Eds.) . Routledge , New York, NY, USA , Chapter\u00a04, 55\u201367. Jill Burstein, Joel Tetreault, and Nitin Madnani. 2013. The e-rater automated essay scoring system. In Handbook of Automated Essay Evaluation: Current Applications and New Directions, Mark\u00a0D Shermis and Jill Burstein (Eds.). Routledge, New York, NY, USA, Chapter\u00a04, 55\u201367."},{"key":"e_1_3_2_2_12_1","unstructured":"Monica Chin. 2020. These students figured out their tests were graded by AI \u2014 and the easy way to cheat. https:\/\/www.theverge.com\/2020\/9\/2\/21419012\/edgenuity-online-class-ai-grading-keyword-mashing-students-school-cheating-algorithm-glitch. Accessed 2020-09-13.  Monica Chin. 2020. These students figured out their tests were graded by AI \u2014 and the easy way to cheat. https:\/\/www.theverge.com\/2020\/9\/2\/21419012\/edgenuity-online-class-ai-grading-keyword-mashing-students-school-cheating-algorithm-glitch. Accessed 2020-09-13."},{"key":"e_1_3_2_2_13_1","first-page":"89","article-title":"Student perceptions of project essay grade (PEG) software","volume":"14","author":"Curran J","year":"2013","unstructured":"Michael\u00a0 J Curran , Peter Draus , George Maruschock , and T Maier . 2013 . Student perceptions of project essay grade (PEG) software . Issues in Information Systems 14 , 1 (2013), 89 \u2013 98 . Michael\u00a0J Curran, Peter Draus, George Maruschock, and T Maier. 2013. Student perceptions of project essay grade (PEG) software. Issues in Information Systems 14, 1 (2013), 89\u201398.","journal-title":"Issues in Information Systems"},{"key":"e_1_3_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.asw.2012.10.002"},{"key":"e_1_3_2_2_15_1","volume-title":"Introduction: The discipline and practice of qualitative research.In Strategies of qualitative inquiry(3 ed.). Vol.\u00a02","author":"Denzin K","year":"2008","unstructured":"Norman\u00a0 K Denzin and Yvonna\u00a0 S Lincoln . 2008 . Introduction: The discipline and practice of qualitative research.In Strategies of qualitative inquiry(3 ed.). Vol.\u00a02 . Sage Publications, Inc, Thousand Oaks, CA, USA , Chapter\u00a0 1, 1\u2013\u201343. Norman\u00a0K Denzin and Yvonna\u00a0S Lincoln. 2008. Introduction: The discipline and practice of qualitative research.In Strategies of qualitative inquiry(3 ed.). Vol.\u00a02. Sage Publications, Inc, Thousand Oaks, CA, USA, Chapter\u00a01, 1\u2013\u201343."},{"key":"e_1_3_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025659"},{"key":"e_1_3_2_2_17_1","first-page":"114","article-title":"Algorithm aversion: People erroneously avoid algorithms after seeing them err.Journal of Experimental Psychology","volume":"144","author":"Dietvorst J","year":"2015","unstructured":"Berkeley\u00a0 J Dietvorst , Joseph\u00a0 P Simmons , and Cade Massey . 2015 . Algorithm aversion: People erroneously avoid algorithms after seeing them err.Journal of Experimental Psychology : General 144 , 1 (2015), 114 . Berkeley\u00a0J Dietvorst, Joseph\u00a0P Simmons, and Cade Massey. 2015. Algorithm aversion: People erroneously avoid algorithms after seeing them err.Journal of Experimental Psychology: General 144, 1 (2015), 114.","journal-title":"General"},{"key":"e_1_3_2_2_18_1","volume-title":"Proceedings of the ACL 2010 System Demonstrations. Association for Computational Linguistics","author":"Dzikovska O","year":"2010","unstructured":"Myroslava\u00a0 O Dzikovska , Johanna\u00a0 D Moore , Natalie Steinhauser , Gwendolyn Campbell , Elaine Farrow , and Charles\u00a0 B Callaway . 2010 . Beetle II: a system for tutoring and computational linguistics experimentation . In Proceedings of the ACL 2010 System Demonstrations. Association for Computational Linguistics , Stroudsburg, PA, USA, 13\u201318. Myroslava\u00a0O Dzikovska, Johanna\u00a0D Moore, Natalie Steinhauser, Gwendolyn Campbell, Elaine Farrow, and Charles\u00a0B Callaway. 2010. Beetle II: a system for tutoring and computational linguistics experimentation. In Proceedings of the ACL 2010 System Demonstrations. Association for Computational Linguistics, Stroudsburg, PA, USA, 13\u201318."},{"key":"e_1_3_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1016\/S1071-5819(03)00038-7"},{"key":"e_1_3_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858494"},{"key":"e_1_3_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300724"},{"key":"e_1_3_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1177\/160940690600500107"},{"key":"e_1_3_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/364914.364937"},{"key":"e_1_3_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3408877.3432539"},{"key":"e_1_3_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/230538.230561"},{"key":"e_1_3_2_2_26_1","volume-title":"Analyzing Qualitative Data. Vol.\u00a02. SAGE","author":"Gibbs R","unstructured":"Graham\u00a0 R Gibbs . 2007. Thematic coding and categorizing . In Analyzing Qualitative Data. Vol.\u00a02. SAGE , Thousand Oaks, CA, USA , Chapter\u00a0 4, 38\u201356. Graham\u00a0R Gibbs. 2007. Thematic coding and categorizing. In Analyzing Qualitative Data. Vol.\u00a02. SAGE, Thousand Oaks, CA, USA, Chapter\u00a04, 38\u201356."},{"key":"e_1_3_2_2_27_1","first-page":"44","article-title":"Utility in a fallible tool: A multi-site case study of automated writing evaluation","volume":"8","author":"Grimes Douglas","year":"2010","unstructured":"Douglas Grimes and Mark Warschauer . 2010 . Utility in a fallible tool: A multi-site case study of automated writing evaluation . The Journal of Technology, Learning and Assessment 8 , 6 (2010), 44 . Douglas Grimes and Mark Warschauer. 2010. Utility in a fallible tool: A multi-site case study of automated writing evaluation. The Journal of Technology, Learning and Assessment 8, 6 (2010), 44.","journal-title":"The Journal of Technology, Learning and Assessment"},{"key":"e_1_3_2_2_28_1","volume-title":"How many interviews are enough? An experiment with data saturation and variability. Field methods 18, 1","author":"Guest Greg","year":"2006","unstructured":"Greg Guest , Arwen Bunce , and Laura Johnson . 2006. How many interviews are enough? An experiment with data saturation and variability. Field methods 18, 1 ( 2006 ), 59\u201382. Greg Guest, Arwen Bunce, and Laura Johnson. 2006. How many interviews are enough? An experiment with data saturation and variability. Field methods 18, 1 (2006), 59\u201382."},{"key":"e_1_3_2_2_29_1","first-page":"229","article-title":"Student perceptions of the web-based homework program WeBWorK in moderate enrollment college algebra classes","volume":"24","author":"Hauk Shandy","year":"2005","unstructured":"Shandy Hauk and Angelo Segalla . 2005 . Student perceptions of the web-based homework program WeBWorK in moderate enrollment college algebra classes . Journal of Computers in Mathematics and Science Teaching 24 , 3(2005), 229 \u2013 253 . Shandy Hauk and Angelo Segalla. 2005. Student perceptions of the web-based homework program WeBWorK in moderate enrollment college algebra classes. Journal of Computers in Mathematics and Science Teaching 24, 3(2005), 229\u2013253.","journal-title":"Journal of Computers in Mathematics and Science Teaching"},{"key":"e_1_3_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1207\/s15516709cog1001_3"},{"key":"e_1_3_2_2_31_1","volume-title":"Cindy Matthews, and Ronald Lamb.","author":"Kintsch Eileen","year":"2000","unstructured":"Eileen Kintsch , Dave Steinhart , Gerry Stahl , LSA Research\u00a0Group LSA Research\u00a0Group , Cindy Matthews, and Ronald Lamb. 2000 . Developing summarization skills through the use of LSA-based feedback. Interactive learning environments 8, 2 (2000), 87\u2013109. Eileen Kintsch, Dave Steinhart, Gerry Stahl, LSA Research\u00a0Group LSA Research\u00a0Group, Cindy Matthews, and Ronald Lamb. 2000. Developing summarization skills through the use of LSA-based feedback. Interactive learning environments 8, 2 (2000), 87\u2013109."},{"key":"e_1_3_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858402"},{"key":"e_1_3_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300641"},{"key":"e_1_3_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1025779619903"},{"key":"e_1_3_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1518\/hfes.46.1.50_30392"},{"key":"e_1_3_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1177\/2053951718756684"},{"key":"e_1_3_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1007\/s13347-017-0279-x"},{"key":"e_1_3_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/1041624.1041673"},{"key":"e_1_3_2_2_39_1","volume-title":"Grounded theory in management research","author":"Locke D","unstructured":"Karen\u00a0 D Locke . 2000. Grounded theory in management research . Sage , Thousand Oaks, CA, USA . Karen\u00a0D Locke. 2000. Grounded theory in management research. Sage, Thousand Oaks, CA, USA."},{"key":"e_1_3_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/1404520.1404531"},{"key":"e_1_3_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0148-2963(01)00276-4"},{"key":"e_1_3_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/2157136.2157249"},{"key":"e_1_3_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cedpsych.2018.07.002"},{"key":"e_1_3_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1518\/001872097778543886"},{"key":"e_1_3_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2016.12.076"},{"key":"e_1_3_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1111\/medu.13504"},{"key":"e_1_3_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.asw.2013.04.001"},{"key":"e_1_3_2_2_48_1","volume-title":"Handbook of automated essay evaluation: Current applications and new directions","author":"Shermis D","unstructured":"Mark\u00a0 D Shermis and Jill Burstein . 2013. Handbook of automated essay evaluation: Current applications and new directions . Routledge , New York, NY . Mark\u00a0D Shermis and Jill Burstein. 2013. Handbook of automated essay evaluation: Current applications and new directions. Routledge, New York, NY."},{"key":"e_1_3_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICICT.2013.6732782"},{"key":"e_1_3_2_2_50_1","unstructured":"Tovia Smith. 2018. More States Opting to \u2018Robo-Grade\u2019 Student Essays By Computer. https:\/\/www.npr.org\/2018\/06\/30\/624373367\/more-states-opting-to-robo-grade-student-essays-by-computer. Accessed 2020-08-17.  Tovia Smith. 2018. More States Opting to \u2018Robo-Grade\u2019 Student Essays By Computer. https:\/\/www.npr.org\/2018\/06\/30\/624373367\/more-states-opting-to-robo-grade-student-essays-by-computer. Accessed 2020-08-17."},{"key":"e_1_3_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.asw.2013.11.007"},{"key":"e_1_3_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/2335356.2335362"},{"key":"e_1_3_2_2_53_1","doi-asserted-by":"publisher","DOI":"10.28945\/331"},{"key":"e_1_3_2_2_54_1","volume-title":"ICCE. Citeseer","author":"Wang Hao-Chuan","unstructured":"Hao-Chuan Wang , Chun-Yen Chang , Tsai-Yen Li , 2005. Automated Scoring for Creative Problem Solving Ability with Ideation-Explanation Modeling .. In ICCE. Citeseer , ICCE , Singapore, Singapore , 524\u2013531. Hao-Chuan Wang, Chun-Yen Chang, Tsai-Yen Li, 2005. Automated Scoring for Creative Problem Solving Ability with Ideation-Explanation Modeling.. In ICCE. Citeseer, ICCE, Singapore, Singapore, 524\u2013531."},{"key":"e_1_3_2_2_55_1","volume-title":"A framework for evaluation and use of automated scoring. Educational measurement: issues and practice 31, 1","author":"Williamson M","year":"2012","unstructured":"David\u00a0 M Williamson , Xiaoming Xi , and F\u00a0Jay Breyer . 2012. A framework for evaluation and use of automated scoring. Educational measurement: issues and practice 31, 1 ( 2012 ), 2\u201313. David\u00a0M Williamson, Xiaoming Xi, and F\u00a0Jay Breyer. 2012. A framework for evaluation and use of automated scoring. Educational measurement: issues and practice 31, 1 (2012), 2\u201313."},{"key":"e_1_3_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.compedu.2016.05.004"},{"key":"e_1_3_2_2_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174230"},{"key":"e_1_3_2_2_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300509"}],"event":{"name":"CHI '21: CHI Conference on Human Factors in Computing Systems","location":"Yokohama Japan","acronym":"CHI '21","sponsor":["SIGCHI ACM Special Interest Group on Computer-Human Interaction"]},"container-title":["Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3411764.3445424","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3411764.3445424","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3411764.3445424","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T21:28:35Z","timestamp":1750195715000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3411764.3445424"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,5,6]]},"references-count":58,"alternative-id":["10.1145\/3411764.3445424","10.1145\/3411764"],"URL":"https:\/\/doi.org\/10.1145\/3411764.3445424","relation":{},"subject":[],"published":{"date-parts":[[2021,5,6]]},"assertion":[{"value":"2021-05-07","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}