{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,25]],"date-time":"2026-02-25T17:16:02Z","timestamp":1772039762564,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":60,"publisher":"ACM","license":[{"start":{"date-parts":[[2022,1,8]],"date-time":"2022-01-08T00:00:00Z","timestamp":1641600000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Center for Design and New Media, Indraprastha Institute of Information Technology"},{"name":"Infosys Centre for Artificial Intelligence, Indraprastha institute of Information Technology"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,1,8]]},"DOI":"10.1145\/3493700.3493765","type":"proceedings-article","created":{"date-parts":[[2022,1,7]],"date-time":"2022-01-07T23:54:21Z","timestamp":1641599661000},"page":"90-99","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":5,"title":["Evaluation Toolkit For Robustness Testing Of Automatic Essay Scoring Systems"],"prefix":"10.1145","author":[{"given":"Anubha","family":"Kabra","sequence":"first","affiliation":[{"name":"Adobe, IN"}]},{"given":"Mehar","family":"Bhatia","sequence":"additional","affiliation":[{"name":"IIIT-Delhi, IN"}]},{"given":"Yaman Kumar","family":"Singla","sequence":"additional","affiliation":[{"name":"Adobe Media Data Science Research, India and IIIT-Delhi; State University of New York at Buffalo, USA"}]},{"given":"Junyi","family":"Jessy Li","sequence":"additional","affiliation":[{"name":"University of Texas at Austin, United States of America, US"}]},{"given":"Rajiv","family":"Ratn Shah","sequence":"additional","affiliation":[{"name":"IIIT Delhi, IN"}]}],"member":"320","published-online":{"date-parts":[[2022,1,8]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"ASAP-AES. 2012. The Hewlett Foundation: Automated Essay Scoring Develop an automated scoring algorithm for student-written essays.https:\/\/www.kaggle.com\/c\/asap-aes\/."},{"key":"e_1_3_2_1_2_1","volume-title":"Automated essay scoring with e-rater\u00ae v. 2.0. ETS Research Report Series 2004, 2","author":"Attali Yigal","year":"2004","unstructured":"Yigal Attali and Jill Burstein. 2004. Automated essay scoring with e-rater\u00ae v. 2.0. ETS Research Report Series 2004, 2 (2004), i\u201321."},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1162\/coli.2008.34.1.1"},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"crossref","unstructured":"Isaac\u00a0I Bejar Robert\u00a0J Mislevy and Mo Zhang. 2017. Automated scoring with validity in mind. The wiley handbook of cognition and assessment: Frameworks Methodologies and applications(2017) 226\u2013246.","DOI":"10.1002\/9781118956588.ch10"},{"key":"e_1_3_2_1_5_1","volume-title":"How artificial intelligence will impact K-12 teachers. Retrieved May 12(2020)","author":"Bryant J","year":"2020","unstructured":"J Bryant, C Heitz, S Sanghvi, and D Wagle. 2020. How artificial intelligence will impact K-12 teachers. Retrieved May 12(2020), 2020."},{"key":"e_1_3_2_1_6_1","volume-title":"Automated essay evaluation: The Criterion online writing service. Ai magazine 25, 3","author":"Burstein Jill","year":"2004","unstructured":"Jill Burstein, Martin Chodorow, and Claudia Leacock. 2004. Automated essay evaluation: The Criterion online writing service. Ai magazine 25, 3 (2004), 27\u201327."},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2018.8462562"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1177\/0265532210364406"},{"key":"e_1_3_2_1_9_1","volume-title":"Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805(2018).","author":"Devlin Jacob","year":"2018","unstructured":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805(2018)."},{"key":"e_1_3_2_1_10_1","unstructured":"The Indian\u00a0Education Diary. 2020. The Open University Of China Awarded UNESCO Prize For Its Use Of AI To Empower Rural Learners. https:\/\/indiaeducationdiary.in\/the-open-university-of-china-awarded-unesco-prize-for-its-use-of-ai-to-empower-rural-learners\/."},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.coling-main.76"},{"key":"e_1_3_2_1_12_1","unstructured":"Afrizal Doewes and Mykola Pechenizkiy. 2021. On the Limitations of Human-Computer Agreement in Automated Essay Scoring. In EDM."},{"key":"e_1_3_2_1_13_1","unstructured":"Edx EASE. 2013. EASE (Enhanced AI Scoring Engine) is a library that allows for machine learning based classification of textual content. This is useful for tasks such as scoring student essays.https:\/\/github.com\/edx\/ease."},{"key":"e_1_3_2_1_14_1","unstructured":"ETS. [n.d.]. Automated Scoring. What it is and why it\u2019s a big deal. https:\/\/news.ets.org\/stories\/automated-scoring\/."},{"key":"e_1_3_2_1_15_1","unstructured":"Todd Feathers. 2019. Flawed Algorithms Are Grading Millions of Students\u2019 Essays. https:\/\/www.vice.com\/en\/article\/pa7dj9\/flawed-algorithms-are-grading-millions-of-students-essays."},{"key":"e_1_3_2_1_16_1","unstructured":"Peter\u00a0W Foltz Lynn\u00a0A Streeter Karen\u00a0E Lochbaum and Thomas\u00a0K Landauer. 2013. Implementation and applications of the Intelligent Essay Assessor. Handbook of automated essay evaluation(2013) 68\u201388."},{"key":"e_1_3_2_1_17_1","unstructured":"Peter Greene. 2018. Automated essay scoring remains an empty dream. Retrieved from Forbes: https:\/\/www. forbes. com\/sites\/petergreene\/2018\/07\/02\/automated-essay-scoring-remains-an-empty-dream(2018)."},{"key":"e_1_3_2_1_18_1","volume-title":"Algorithms on Strings, Trees and Sequences","author":"Gusfield Dan","unstructured":"Dan Gusfield. 1997. Algorithms on Strings, Trees and Sequences. Cambridge University Press, Cambridge, UK."},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1111\/emip.12036"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.604"},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"crossref","unstructured":"Zixuan Ke Winston Carlile Nishant Gurrapadi and Vincent Ng. 2018. Learning to Give Feedback: Modeling Attributes Affecting Argument Persuasiveness in Student Essays.. In IJCAI. 4130\u20134136.","DOI":"10.24963\/ijcai.2018\/574"},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"crossref","unstructured":"Zixuan Ke and Vincent Ng. 2019. Automated Essay Scoring: A Survey of the State of the Art.. In IJCAI Vol.\u00a019. 6300\u20136308.","DOI":"10.24963\/ijcai.2019\/879"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.697"},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v33i01.33019662"},{"key":"e_1_3_2_1_25_1","unstructured":"Jiawei Liu Yang Xu and Yaguang Zhu. 2019. Automated essay scoring based on two-stage learning. arXiv preprint arXiv:1901.07744(2019)."},{"key":"e_1_3_2_1_26_1","volume-title":"National Council on Measurement in Education Conference (NCME)","author":"Lochbaum E","year":"2013","unstructured":"Karen\u00a0E Lochbaum, Mark Rosenstein, Peter Foltz, Marcia\u00a0A Derr, 2013. Detection of gaming in automated scoring of essays with the IEA. In National Council on Measurement in Education Conference (NCME), San Francisco, CA."},{"key":"e_1_3_2_1_27_1","volume-title":"Proceedings of the 27th International Conference on Computational Linguistics. 1099\u20131109","author":"Madnani Nitin","year":"2018","unstructured":"Nitin Madnani and Aoife Cahill. 2018. Automated scoring: Beyond natural language processing. In Proceedings of the 27th International Conference on Computational Linguistics. 1099\u20131109."},{"key":"e_1_3_2_1_28_1","volume-title":"Managing the data base environment","author":"Martin James","unstructured":"James Martin. 1983. Managing the data base environment. Prentice Hall PTR."},{"key":"e_1_3_2_1_29_1","volume-title":"Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC","author":"Mathias Sandeep","year":"2018","unstructured":"Sandeep Mathias and Pushpak Bhattacharyya. 2018. ASAP++: Enriching the ASAP Automated Essay Grading Dataset with Essay Attribute Scores. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). European Language Resources Association (ELRA), Miyazaki, Japan. https:\/\/www.aclweb.org\/anthology\/L18-1187"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N18-1097"},{"key":"e_1_3_2_1_31_1","unstructured":"Patrick O\u2019Donnell. 2020. Computers are now grading essays on Ohio\u2019s state tests. https:\/\/www.cleveland.com\/metro\/2018\/03\/computers_are_now_grading_essays_on_ohios_state_tests_your_ch.html."},{"key":"e_1_3_2_1_32_1","unstructured":"Swapnil Parekh Yaman\u00a0Kumar Singla Changyou Chen Junyi\u00a0Jessy Li and Rajiv\u00a0Ratn Shah. 2020. My Teacher Thinks The World Is Flat! Interpreting Automatic Essay Scoring Mechanism. arXiv preprint arXiv:2012.13872(2020)."},{"key":"e_1_3_2_1_33_1","unstructured":"Rajaswa Patil Yaman\u00a0Kumar Singla Rajiv\u00a0Ratn Shah Mika Hama and Roger Zimmermann. 2020. Towards Modelling Coherence in Spoken Discourse. arXiv preprint arXiv:2101.00056(2020)."},{"key":"e_1_3_2_1_34_1","unstructured":"PEG. 2017. The Engine Driving Automated Essay Scoring. https:\/\/utahcompose.com\/sites\/default\/files\/peg-Info-report.pdf."},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.asw.2014.05.001"},{"key":"e_1_3_2_1_36_1","unstructured":"Les Perelman Louis Sobel Milo Beckman and Damien Jiang. 2014. Basic Automatic B.S. Essay Language Generator (BABEL). https:\/\/babel-generator.herokuapp.com\/."},{"key":"e_1_3_2_1_37_1","unstructured":"Les Perelman Louis Sobel Milo Beckman and Damien Jiang. 2014. Basic Automatic B.S. Essay Language Generator (BABEL) by Les Perelman Ph.D.http:\/\/lesperelman.com\/writing-assessment-robo-grading\/babel-generator\/."},{"key":"e_1_3_2_1_38_1","volume-title":"Stumping E-Rater: Challenging the validity of automated essay scoring. ETS Research Report Series 2001, 1","author":"Powers E","year":"2001","unstructured":"Donald\u00a0E Powers, Jill\u00a0C Burstein, Martin Chodorow, Mary\u00a0E Fowles, and Karen Kukich. 2001. Stumping E-Rater: Challenging the validity of automated essay scoring. ETS Research Report Series 2001, 1 (2001), i\u201344."},{"key":"e_1_3_2_1_39_1","first-page":"52","article-title":"Why can\u2019t it mark this one?: A qualitative analysis of student writing rejected by an automated essay scoring system","volume":"53","author":"Reinertsen Nathanael","year":"2018","unstructured":"Nathanael Reinertsen 2018. Why can\u2019t it mark this one?: A qualitative analysis of student writing rejected by an automated essay scoring system. English in Australia 53, 1 (2018), 52.","journal-title":"English in Australia"},{"key":"e_1_3_2_1_40_1","volume-title":"The intellimetric automated essay scoring engine-a review and an application to chinese essay scoring","author":"Schultz T","unstructured":"Matthew\u00a0T Schultz. 2013. The intellimetric automated essay scoring engine-a review and an application to chinese essay scoring. New York: Routledge."},{"key":"e_1_3_2_1_41_1","unstructured":"Jui Shah Yaman\u00a0Kumar Singla Changyou Chen and Rajiv\u00a0Ratn Shah. 2021. What all do audio transformer models hear? probing acoustic representations for language delivery and its structure. arXiv preprint arXiv:2101.00387(2021)."},{"key":"e_1_3_2_1_42_1","volume-title":"Feature Enhanced Capsule Networks for Robust Automatic Essay Scoring. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 365\u2013380","author":"Sharma Arushi","year":"2021","unstructured":"Arushi Sharma, Anubha Kabra, and Rajiv Kapoor. 2021. Feature Enhanced Capsule Networks for Robust Automatic Essay Scoring. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 365\u2013380."},{"key":"e_1_3_2_1_43_1","unstructured":"Mark\u00a0D Shermis and Ben Hamner. 2012. Contrasting state-of-the-art automated scoring of essays: Analysis. In Annual national council on measurement in education meeting. 14\u201316."},{"key":"e_1_3_2_1_44_1","doi-asserted-by":"crossref","unstructured":"Yaman\u00a0Kumar Singla Avykat Gupta Shaurya Bagga Changyou Chen Balaji Krishnamurthy and Rajiv\u00a0Ratn Shah. 2021. Speaker-Conditioned Hierarchical Modeling for Automated Speech Scoring. arXiv preprint arXiv:2109.00928(2021).","DOI":"10.1145\/3459637.3482395"},{"key":"e_1_3_2_1_45_1","unstructured":"Yaman\u00a0Kumar Singla Swapnil Parekh Somesh Singh Junyi\u00a0Jessy Li Rajiv\u00a0Ratn Shah and Changyou Chen. 2021. AES Systems Are Both Overstable And Oversensitive: Explaining Why And Proposing Defenses. arXiv preprint arXiv:2109.11728(2021)."},{"key":"e_1_3_2_1_46_1","unstructured":"Tovia Smith. 2018. More states opting to\u2019robo-grade\u2019student essays by computer. Retrieved from NPR: https:\/\/www. npr. org\/2018\/06\/30\/624373367\/more-states-opting-to-robo-grade-student-essays-by-computer(2018)."},{"key":"e_1_3_2_1_47_1","doi-asserted-by":"crossref","unstructured":"Makesh\u00a0Narsimhan Sreedhar Kun Ni and Siva Reddy. 2020. Learning Improvised Chatbots from Adversarial Modifications of Natural Language Feedback. arXiv preprint arXiv:2010.07261(2020).","DOI":"10.18653\/v1\/2020.findings-emnlp.221"},{"key":"e_1_3_2_1_48_1","volume-title":"Twenty-Second International FLAIRS Conference.","author":"Sukkarieh Jana\u00a0Zuheir","year":"2009","unstructured":"Jana\u00a0Zuheir Sukkarieh and John Blackmore. 2009. C-rater: Automatic content scoring for short constructed responses. In Twenty-Second International FLAIRS Conference."},{"key":"e_1_3_2_1_49_1","unstructured":"Christian Szegedy Wojciech Zaremba Ilya Sutskever Joan Bruna Dumitru Erhan Ian Goodfellow and Rob Fergus. 2013. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199(2013)."},{"key":"e_1_3_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D16-1193"},{"key":"e_1_3_2_1_51_1","volume-title":"Automated writing evaluation in an EFL setting: Lessons from China.JALT CALL Journal 13, 2","author":"Tang Jinlan","year":"2017","unstructured":"Jinlan Tang and Changhua\u00a0Sun Rich. 2017. Automated writing evaluation in an EFL setting: Lessons from China.JALT CALL Journal 13, 2 (2017), 117\u2013146."},{"key":"e_1_3_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v32i1.12045"},{"key":"e_1_3_2_1_53_1","doi-asserted-by":"crossref","unstructured":"Eric Wallace Shi Feng Nikhil Kandpal Matt Gardner and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing NLP. arXiv preprint arXiv:1908.07125(2019).","DOI":"10.18653\/v1\/D19-1221"},{"key":"e_1_3_2_1_54_1","volume-title":"2018 AAAI Spring Symposium Series.","author":"West-Smith Patti","year":"2018","unstructured":"Patti West-Smith, Stephanie Butler, and Elijah Mayfield. 2018. Trustworthy Automated Essay Scoring without Explicit Construct Validity. In 2018 AAAI Spring Symposium Series."},{"key":"e_1_3_2_1_55_1","unstructured":"Peng Xu Hamidreza Saghir Jin\u00a0Sung Kang Teng Long Avishek\u00a0Joey Bose Yanshuai Cao and Jackie Chi\u00a0Kit Cheung. 2019. A cross-domain transferable neural coherence model. arXiv preprint arXiv:1905.11912(2019)."},{"key":"e_1_3_2_1_56_1","volume-title":"Handbook of automated scoring: Theory into practice","author":"Yan Duanli","unstructured":"Duanli Yan, Andr\u00e9\u00a0A Rupp, and Peter\u00a0W Foltz. 2020. Handbook of automated scoring: Theory into practice. CRC Press."},{"key":"e_1_3_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/3194452.3194474"},{"key":"e_1_3_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N18-3008"},{"key":"e_1_3_2_1_59_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3374217","article-title":"Adversarial attacks on deep-learning models in natural language processing: A survey","volume":"11","author":"Zhang Wei\u00a0Emma","year":"2020","unstructured":"Wei\u00a0Emma Zhang, Quan\u00a0Z Sheng, Ahoud Alhazmi, and Chenliang Li. 2020. Adversarial attacks on deep-learning models in natural language processing: A survey. ACM Transactions on Intelligent Systems and Technology (TIST) 11, 3(2020), 1\u201341.","journal-title":"ACM Transactions on Intelligent Systems and Technology (TIST)"},{"key":"e_1_3_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/3051457.3053982"}],"event":{"name":"CODS-COMAD 2022: 5th Joint International Conference on Data Science & Management of Data (9th ACM IKDD CODS and 27th COMAD)","location":"Bangalore India","acronym":"CODS-COMAD 2022","sponsor":["SIGGRAPH ACM Special Interest Group on Computer Graphics and Interactive Techniques"]},"container-title":["Proceedings of the 5th Joint International Conference on Data Science & Management of Data (9th ACM IKDD CODS and 27th COMAD)"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3493700.3493765","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3493700.3493765","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:30:44Z","timestamp":1750188644000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3493700.3493765"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,1,8]]},"references-count":60,"alternative-id":["10.1145\/3493700.3493765","10.1145\/3493700"],"URL":"https:\/\/doi.org\/10.1145\/3493700.3493765","relation":{},"subject":[],"published":{"date-parts":[[2022,1,8]]},"assertion":[{"value":"2022-01-08","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}