{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,15]],"date-time":"2025-11-15T10:24:55Z","timestamp":1763202295287},"reference-count":50,"publisher":"MIT Press - Journals","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Transactions of the Association for Computational Linguistics"],"published-print":{"date-parts":[[2019,11]]},"abstract":"<jats:p> Until now, grammatical error correction (GEC) has been primarily evaluated on text written by non-native English speakers, with a focus on student essays. This paper enables GEC development on text written by native speakers by providing a new data set and metric. We present a multiple-reference test corpus for GEC that includes 4,000 sentences in two new domains ( formal and informal writing by native English speakers) and 2,000 sentences from a diverse set of non-native student writing. We also collect human judgments of several GEC systems on this new test set and perform a meta-evaluation, assessing how reliable automatic metrics are across these domains. We find that commonly used GEC metrics have inconsistent performance across domains, and therefore we propose a new ensemble metric that is robust on all three domains of text. <\/jats:p>","DOI":"10.1162\/tacl_a_00282","type":"journal-article","created":{"date-parts":[[2019,9,11]],"date-time":"2019-09-11T19:59:46Z","timestamp":1568231986000},"page":"551-566","source":"Crossref","is-referenced-by-count":9,"title":["Enabling Robust Grammatical Error Correction in New Domains: Data Sets, Metrics, and Analyses"],"prefix":"10.1162","volume":"7","author":[{"given":"Courtney","family":"Napoles","sequence":"first","affiliation":[{"name":"Grammarly."}]},{"given":"Maria","family":"N\u0103dejde","sequence":"additional","affiliation":[{"name":"Grammarly"}]},{"given":"Joel","family":"Tetreault","sequence":"additional","affiliation":[{"name":"Grammarly"}]}],"member":"281","reference":[{"key":"bib1","doi-asserted-by":"crossref","first-page":"99","DOI":"10.18653\/v1\/W17-4710","volume-title":"Proceedings of the Second Conference on Machine Translation","author":"Barone Miceli","year":"2017"},{"key":"bib2","doi-asserted-by":"crossref","first-page":"489","DOI":"10.18653\/v1\/W17-4755","volume-title":"Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers","author":"Bojar Ond\u0159ej","year":"2017"},{"key":"bib3","first-page":"199","volume-title":"Proceedings of the First Conference on Machine Translation","author":"Bojar Ond\u0159ej","year":"2016"},{"key":"bib4","volume-title":"Proceedings of the 14th Workshop on Innovative Use of NLP for Building Educational Applications","author":"Bryant Christopher","year":"2019"},{"key":"bib5","doi-asserted-by":"crossref","first-page":"793","DOI":"10.18653\/v1\/P17-1074","volume-title":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Bryant Christopher","year":"2017"},{"key":"bib6","first-page":"697","volume-title":"Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)","author":"Bryant Christopher","year":"2015"},{"key":"bib7","doi-asserted-by":"crossref","first-page":"136","DOI":"10.3115\/1626355.1626373","volume-title":"Proceedings of the Second Workshop on Statistical Machine Translation","author":"Callison-Burch Chris","year":"2007"},{"key":"bib8","first-page":"70","volume-title":"Proceedings of the Third Workshop on Statistical Machine Translation","author":"Callison-Burch Chris","year":"2008"},{"key":"bib9","first-page":"5755","volume-title":"Proceedings of The Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18)","author":"Chollampatt Shamil","year":"2018"},{"key":"bib10","doi-asserted-by":"crossref","first-page":"2528","DOI":"10.18653\/v1\/D18-1274","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Chollampatt Shamil","year":"2018"},{"key":"bib11","first-page":"2730","volume-title":"Proceedings of the 27th International Conference on Computational Linguistics","author":"Chollampatt Shamil","year":"2018"},{"key":"bib12","doi-asserted-by":"crossref","first-page":"1372","DOI":"10.18653\/v1\/P18-1127","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Choshen Leshem","year":"2018"},{"key":"bib13","first-page":"124","volume-title":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)","author":"Choshen Leshem","year":"2018"},{"key":"bib14","first-page":"568","volume-title":"Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Dahlmeier Daniel","year":"2012"},{"key":"bib15","first-page":"22","volume-title":"Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications","author":"Dahlmeier Daniel","year":"2013"},{"key":"bib16","doi-asserted-by":"crossref","first-page":"53","DOI":"10.18653\/v1\/W16-0506","volume-title":"Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications","author":"Daudaravicius Vidas","year":"2016"},{"key":"bib17","first-page":"578","volume-title":"Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Felice Mariano","year":"2015"},{"key":"bib18","doi-asserted-by":"crossref","first-page":"105","DOI":"10.18653\/v1\/W16-0511","volume-title":"Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications","author":"Flickinger Dan","year":"2016"},{"key":"bib19","first-page":"1243","volume-title":"Proceedings of the 34th International Conference on Machine Learning","volume":"70","author":"Gehring Jonas","year":"2017"},{"key":"bib20","doi-asserted-by":"crossref","first-page":"172","DOI":"10.3115\/v1\/D14-1020","volume-title":"Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Graham Yvette","year":"2014"},{"key":"bib21","first-page":"1183","volume-title":"Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Graham Yvette","year":"2015"},{"key":"bib22","first-page":"1","volume-title":"Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Graham Yvette","year":"2016"},{"key":"bib23","first-page":"478","volume-title":"International Conference on Natural Language Processing","author":"Grundkiewicz Roman","year":"2014"},{"key":"bib24","doi-asserted-by":"crossref","first-page":"461","DOI":"10.18653\/v1\/D15-1052","volume-title":"Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing","author":"Grundkiewicz Roman","year":"2015"},{"key":"bib25","doi-asserted-by":"crossref","first-page":"174","DOI":"10.3115\/v1\/P14-2029","volume-title":"Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)","author":"Heilman Michael","year":"2014"},{"key":"bib26","doi-asserted-by":"crossref","first-page":"1546","DOI":"10.18653\/v1\/D16-1161","volume-title":"Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing","author":"Junczys-Dowmunt Marcin","year":"2016"},{"key":"bib27","doi-asserted-by":"crossref","first-page":"116","DOI":"10.18653\/v1\/P18-4020","volume-title":"Proceedings of ACL 2018, System Demonstrations","author":"Junczys-Dowmunt Marcin","year":"2018"},{"key":"bib28","first-page":"177","volume-title":"Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Papers)","author":"Klein Guillaume","year":"2018"},{"key":"bib29","first-page":"177","volume-title":"Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions","author":"Koehn Philipp","year":"2007"},{"key":"bib30","doi-asserted-by":"crossref","first-page":"102","DOI":"10.3115\/1654650.1654666","volume-title":"Proceedings on the Workshop on Statistical Machine Translation","author":"Koehn Philipp","year":"2006"},{"key":"bib31","first-page":"682","volume-title":"Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers","author":"Ma Qingsong","year":"2018"},{"key":"bib32","first-page":"598","volume-title":"Proceedings of the Second Conference on Machine Translation","author":"Ma Qingsong","year":"2017"},{"key":"bib33","first-page":"45","volume-title":"Proceedings of the Eighth Workshop on Statistical Machine Translation","author":"Mach\u00e1\u010dek Matou\u0161","year":"2013"},{"key":"bib34","first-page":"588","volume-title":"Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)","author":"Napoles Courtney","year":"2015"},{"key":"bib35","author":"Napoles Courtney","year":"2016","journal-title":"CoRR"},{"key":"bib36","doi-asserted-by":"crossref","first-page":"2109","DOI":"10.18653\/v1\/D16-1228","volume-title":"Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing","author":"Napoles Courtney","year":"2016"},{"key":"bib37","first-page":"229","volume-title":"Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers","author":"Napoles Courtney","year":"2017"},{"key":"bib38","first-page":"1","volume-title":"Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task","author":"Ng Hwee Tou","year":"2014"},{"key":"bib39","first-page":"1","volume-title":"Proceedings of the Seventeenth Conference on Computational Natural Language Learning: Shared Task","author":"Ng Hwee Tou","year":"2013"},{"key":"bib40","first-page":"72","volume-title":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)","author":"Novikova Jekaterina","year":"2018"},{"key":"bib41","first-page":"311","volume-title":"Proceedings of 40th Annual Meeting of the Association for Computational Linguistics","author":"Papineni Kishore","year":"2002"},{"key":"bib42","first-page":"2825","volume":"12","author":"Pedregosa Fabian","year":"2011","journal-title":"Journal of Machine Learning Research"},{"key":"bib43","doi-asserted-by":"crossref","first-page":"612","DOI":"10.18653\/v1\/W17-4770","volume-title":"Proceedings of the Second Conference on Machine Translation","author":"Popovi\u0107 Maja","year":"2017"},{"key":"bib44","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00091"},{"key":"bib45","doi-asserted-by":"crossref","first-page":"208","DOI":"10.18653\/v1\/P18-1020","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Sakaguchi Keisuke","year":"2018"},{"key":"bib46","doi-asserted-by":"crossref","first-page":"389","DOI":"10.18653\/v1\/W17-4739","volume-title":"Proceedings of the Second Conference on Machine Translation","author":"Sennrich Rico","year":"2017"},{"key":"bib47","doi-asserted-by":"crossref","first-page":"414","DOI":"10.3115\/v1\/W14-3354","volume-title":"Proceedings of the Ninth Workshop on Statistical Machine Translation","author":"Stanojevic Milos","year":"2014"},{"key":"bib48","first-page":"5998","volume-title":"Advances in Neural Information Processing Systems 30","author":"Vaswani Ashish","year":"2017"},{"key":"bib49","first-page":"180","volume-title":"Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies","author":"Yannakoudakis Helen","year":"2011"},{"key":"bib50","doi-asserted-by":"crossref","first-page":"417","DOI":"10.18653\/v1\/W15-3053","volume-title":"Proceedings of the Tenth Workshop on Statistical Machine Translation","author":"Hui Yu","year":"2015"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/tacl_a_00282","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,3,12]],"date-time":"2021-03-12T21:39:27Z","timestamp":1615585167000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/43519"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,11]]},"references-count":50,"alternative-id":["10.1162\/tacl_a_00282"],"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00282","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,11]]}}}