{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,6]],"date-time":"2026-07-06T18:13:59Z","timestamp":1783361639006,"version":"3.54.6"},"reference-count":62,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2026,7,6]],"date-time":"2026-07-06T00:00:00Z","timestamp":1783296000000},"content-version":"vor","delay-in-days":186,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2026,7,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Automatic evaluation in grammatical error correction (GEC) is crucial for selecting the best-performing systems. Currently, reference-based metrics are a popular choice, which basically measure the similarity between hypothesis and reference sentences. However, similarity measures based on embeddings, such as BERTScore, are often ineffective, since many words in the source sentences remain unchanged in both the hypothesis and the reference. This study focuses on edits specifically designed for GEC, i.e., ERRANT, and computes similarity measured over the edits from the source sentence. To this end, we propose edit vector, a representation for an edit, and introduce a new metric, UOT-ERRANT, which transports these edit vectors from hypothesis to reference using unbalanced optimal transport. Experiments with SEEDA meta-evaluation show that UOT-ERRANT improves evaluation performance, particularly in the +Fluency domain where many edits occur. Moreover, our method is highly interpretable because the transport plan can be interpreted as a soft edit alignment, making UOT-ERRANT a useful metric for both system ranking and analyzing GEC systems. Our code is available from https:\/\/github.com\/gotutiyan\/uot-errant.<\/jats:p>","DOI":"10.1162\/tacl.a.747","type":"journal-article","created":{"date-parts":[[2026,7,6]],"date-time":"2026-07-06T17:14:53Z","timestamp":1783358093000},"page":"1692-1710","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":0,"title":["Grammatical Error Correction Evaluation by Optimally Transporting Edit Representation"],"prefix":"10.1162","volume":"14","author":[{"given":"Takumi","family":"Goto","sequence":"first","affiliation":[{"name":"Nara Institute of Science and Technology, Japan, goto.takumi.gv7@is.naist.jp"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yusuke","family":"Sakai","sequence":"additional","affiliation":[{"name":"Nara Institute of Science and Technology, Japan, sakai.yusuke.sr9@is.naist.jp"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Taro","family":"Watanabe","sequence":"additional","affiliation":[{"name":"Nara Institute of Science and Technology, Japan, taro@is.naist.jp"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2026,7,1]]},"reference":[{"key":"2026070613144911700_bib1","doi-asserted-by":"crossref","first-page":"3966","DOI":"10.18653\/v1\/2023.acl-long.219","article-title":"Unbalanced optimal transport for unbalanced word alignment","volume-title":"Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Arase","year":"2023"},{"key":"2026070613144911700_bib2","doi-asserted-by":"crossref","first-page":"18208","DOI":"10.1109\/CVPR52688.2022.01767","article-title":"Blended diffusion for text-driven editing of natural images","volume-title":"2022 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","author":"Avrahami","year":"2022"},{"key":"2026070613144911700_bib3","first-page":"1","article-title":"Findings of the 2013 Workshop on Statistical Machine Translation","volume-title":"Proceedings of the Eighth Work-shop on Statistical Machine Translation","author":"Bojar","year":"2013"},{"key":"2026070613144911700_bib4","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"Brown","year":"2020","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2026070613144911700_bib5","doi-asserted-by":"crossref","first-page":"52","DOI":"10.18653\/v1\/W19-4406","article-title":"The BEA-2019 shared task on grammatical error correction","volume-title":"Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications","author":"Bryant","year":"2019"},{"key":"2026070613144911700_bib6","doi-asserted-by":"crossref","first-page":"793","DOI":"10.18653\/v1\/P17-1074","article-title":"Automatic annotation and evaluation of error types for grammatical error correction","volume-title":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Bryant","year":"2017"},{"key":"2026070613144911700_bib7","first-page":"697","article-title":"How far are we from fully automatic high quality grammatical error correction?","volume-title":"Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)","author":"Bryant","year":"2015"},{"key":"2026070613144911700_bib8","volume-title":"Scaling algorithms for unbalanced transport problems","author":"Chizat","year":"2017"},{"key":"2026070613144911700_bib9","doi-asserted-by":"crossref","first-page":"632","DOI":"10.18653\/v1\/P18-1059","article-title":"Inherent biases in reference-based evaluation for grammatical error correction","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Choshen","year":"2018"},{"key":"2026070613144911700_bib10","doi-asserted-by":"crossref","first-page":"494","DOI":"10.18653\/v1\/W19-5356","article-title":"WMDO: Fluency-based word mover\u2019s distance for machine translation evaluation","volume-title":"Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)","author":"Chow","year":"2019"},{"key":"2026070613144911700_bib11","article-title":"ELEC-TRA: pre-training text encoders as discriminators rather than generators","volume-title":"8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26\u201330, 2020","author":"Clark","year":"2020"},{"key":"2026070613144911700_bib12","doi-asserted-by":"crossref","first-page":"10450","DOI":"10.18653\/v1\/2021.emnlp-main.817","article-title":"Automatic text evaluation through the lens of Wasserstein barycenters","volume-title":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing","author":"Colombo","year":"2021"},{"key":"2026070613144911700_bib13","first-page":"568","article-title":"Better evaluation for grammatical error correction","volume-title":"Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Dahlmeier","year":"2012"},{"key":"2026070613144911700_bib14","first-page":"4171","article-title":"BERT: Pre-training of deep bidirectional transformers for language understanding","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Devlin","year":"2019"},{"key":"2026070613144911700_bib15","first-page":"825","article-title":"Automatic extraction of learner errors in ESL sentences using linguistically enhanced alignments","volume-title":"Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers","author":"Felice","year":"2016"},{"key":"2026070613144911700_bib16","doi-asserted-by":"crossref","first-page":"878","DOI":"10.18653\/v1\/2022.acl-long.62","article-title":"Language-agnostic BERT sentence embedding","volume-title":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Feng","year":"2022"},{"key":"2026070613144911700_bib17","doi-asserted-by":"crossref","first-page":"8467","DOI":"10.18653\/v1\/2020.emnlp-main.680","article-title":"Grammatical error correction in low error density domains: A new benchmark and analyses","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Flachs","year":"2020"},{"issue":"78","key":"2026070613144911700_bib18","first-page":"1","article-title":"POT: Python optimal transport","volume":"22","author":"Flamary","year":"2021","journal-title":"Journal of Machine Learning Research"},{"key":"2026070613144911700_bib19","volume-title":"POT python optimal transport (version 0.9.5)","author":"Flamary","year":"2024"},{"key":"2026070613144911700_bib20","article-title":"Learning with a wasserstein loss","volume-title":"Advances in Neural Information Processing Systems","author":"Frogner","year":"2015"},{"key":"2026070613144911700_bib21","doi-asserted-by":"crossref","first-page":"6891","DOI":"10.18653\/v1\/2022.emnlp-main.463","article-title":"Revisiting grammatical error correction evaluation and beyond","volume-title":"Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing","author":"Gong","year":"2022"},{"key":"2026070613144911700_bib22","doi-asserted-by":"crossref","first-page":"524","DOI":"10.18653\/v1\/2025.acl-demo.50","article-title":"gec-metrics: A unified library for grammatical error correction evaluation","volume-title":"Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)","author":"Goto","year":"2025"},{"key":"2026070613144911700_bib23","doi-asserted-by":"crossref","first-page":"1165","DOI":"10.18653\/v1\/2025.acl-short.92","article-title":"Rethinking evaluation metrics for grammatical error correction: Why use a different evaluation process than human?","volume-title":"Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)","author":"Goto","year":"2025"},{"key":"2026070613144911700_bib24","doi-asserted-by":"crossref","first-page":"2085","DOI":"10.18653\/v1\/2020.coling-main.188","article-title":"Taking the correction difficulty into account in grammatical error correction evaluation","volume-title":"Proceedings of the 28th International Conference on Computational Linguistics","author":"Gotou","year":"2020"},{"key":"2026070613144911700_bib25","doi-asserted-by":"crossref","first-page":"461","DOI":"10.18653\/v1\/D15-1052","article-title":"Human evaluation of grammatical error correction systems","volume-title":"Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing","author":"Grundkiewicz","year":"2015"},{"key":"2026070613144911700_bib26","article-title":"Trueskill\u2122: A bayesian skill rating system","volume-title":"Advances in Neural Information Processing Systems","author":"Herbrich","year":"2006"},{"key":"2026070613144911700_bib27","doi-asserted-by":"crossref","first-page":"3009","DOI":"10.18653\/v1\/2021.emnlp-main.239","article-title":"Is this the end of the gold standard? A straightforward reference-less grammatical error correction metric","volume-title":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing","author":"Islam","year":"2021"},{"issue":"1","key":"2026070613144911700_bib28","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1287\/mnsc.5.1.1","article-title":"On the translocation of masses","volume":"5","author":"Kantorovich","year":"1958","journal-title":"Manage. Sci."},{"key":"2026070613144911700_bib29","doi-asserted-by":"crossref","first-page":"7831","DOI":"10.63317\/3yoruwn5jgvk","article-title":"GPT-3.5 for grammatical error correction","volume-title":"Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)","author":"Katinskaia","year":"2024"},{"key":"2026070613144911700_bib30","first-page":"199","article-title":"Re-evaluating automatic metrics for image captioning","volume-title":"Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers","author":"Kilickaya","year":"2017"},{"key":"2026070613144911700_bib31","first-page":"68","article-title":"Large language models are state-of-the-art evaluator for grammatical error correction","volume-title":"Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024)","author":"Kobayashi","year":"2024"},{"key":"2026070613144911700_bib32","doi-asserted-by":"crossref","first-page":"837","DOI":"10.1162\/tacl_a_00676","article-title":"Revisiting meta-evaluation for grammatical error correction","volume":"12","author":"Kobayashi","year":"2024","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2026070613144911700_bib33","doi-asserted-by":"crossref","first-page":"303","DOI":"10.18653\/v1\/2024.inlg-main.25","article-title":"n-gram F-score for evaluating grammatical error correction","volume-title":"Proceedings of the 17th International Natural Language Generation Conference","author":"Koyama","year":"2024"},{"key":"2026070613144911700_bib34","first-page":"209","article-title":"Word-level translation quality estimation based on optimal transport","volume-title":"Proceedings of the 16th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track)","author":"Kuroda","year":"2024"},{"key":"2026070613144911700_bib35","first-page":"957","article-title":"From word embeddings to document distances","volume-title":"Proceedings of the 32nd International Conference on Machine Learning","author":"Kusner","year":"2015"},{"key":"2026070613144911700_bib36","doi-asserted-by":"crossref","first-page":"4328","DOI":"10.18653\/v1\/2021.findings-emnlp.367","article-title":"FastCorrect 2: Fast error correction on multiple candidates for automatic speech recognition","volume-title":"Findings of the Association for Computational Linguistics: EMNLP 2021","author":"Leng","year":"2021"},{"key":"2026070613144911700_bib37","volume-title":"RoBERTa: A robustly optimized bert pretraining approach","author":"Liu","year":"2019"},{"issue":"86","key":"2026070613144911700_bib38","first-page":"2579","article-title":"Visualizing data using t-sne","volume":"9","author":"van der Maaten","year":"2008","journal-title":"Journal of Machine Learning Research"},{"key":"2026070613144911700_bib39","first-page":"3578","article-title":"IMPARA: Impact-based metric for GEC using parallel data","volume-title":"Proceedings of the 29th International Conference on Computational Linguistics","author":"Maeda","year":"2022"},{"key":"2026070613144911700_bib40","article-title":"Efficient estimation of word representations in vector space","volume-title":"ArXiv preprint arXiv:1301.3781","author":"Mikolov","year":"2013"},{"key":"2026070613144911700_bib41","doi-asserted-by":"crossref","first-page":"551","DOI":"10.1162\/tacl_a_00282","article-title":"Enabling robust grammatical error correction in new domains: Data sets, metrics, and analyses","volume":"7","author":"Napoles","year":"2019","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2026070613144911700_bib42","first-page":"588","article-title":"Ground truth for grammatical error correction metrics","volume-title":"Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)","author":"Napoles","year":"2015"},{"key":"2026070613144911700_bib43","article-title":"GLEU without tuning","volume-title":"ArXiv preprint arXiv:1605.02592","author":"Napoles","year":"2016"},{"key":"2026070613144911700_bib44","doi-asserted-by":"crossref","first-page":"1","DOI":"10.3115\/v1\/W14-1701","article-title":"The CoNLL-2014 shared task on grammatical error correction","volume-title":"Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task","author":"Ng","year":"2014"},{"key":"2026070613144911700_bib45","doi-asserted-by":"crossref","first-page":"163","DOI":"10.18653\/v1\/2020.bea-1.16","article-title":"GECToR \u2013 grammatical error correction: Tag, not rewrite","volume-title":"Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications","author":"Omelianchuk","year":"2020"},{"key":"2026070613144911700_bib46","first-page":"17","article-title":"Pillars of grammatical error correction: Comprehensive inspection of contemporary approaches in the era of large language models","volume-title":"Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024)","author":"Omelianchuk","year":"2024"},{"key":"2026070613144911700_bib47","article-title":"GPT-4 technical report","volume-title":"ArXiv preprint arXiv:2303.08774","author":"OpenAI","year":"2024"},{"key":"2026070613144911700_bib48","doi-asserted-by":"crossref","first-page":"460","DOI":"10.1109\/ICCV.2009.5459199","article-title":"Fast and robust earth mover\u2019s distances","volume-title":"2009 IEEE 12th International Conference on Computer Vision","author":"Pele","year":"2009"},{"issue":"8","key":"2026070613144911700_bib49","first-page":"9","article-title":"Language models are unsupervised multitask learners","volume":"1","author":"Radford","year":"2019","journal-title":"OpenAI blog"},{"key":"2026070613144911700_bib50","first-page":"702","article-title":"A simple recipe for multilingual grammatical error correction","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)","author":"Rothe","year":"2021"},{"key":"2026070613144911700_bib51","doi-asserted-by":"crossref","first-page":"169","DOI":"10.1162\/tacl_a_00091","article-title":"Reassessing the goals of grammatical error correction: Fluency instead of grammaticality","volume":"4","author":"Sakaguchi","year":"2016","journal-title":"Transactions of the Association for Computational Linguistics"},{"issue":"3","key":"2026070613144911700_bib52","doi-asserted-by":"crossref","first-page":"A1443","DOI":"10.1137\/16M1106018","article-title":"Stabilized sparse scaling algorithms for entropy regularized transport problems","volume":"41","author":"Schmitzer","year":"2019","journal-title":"SIAM Journal on Scientific Computing"},{"key":"2026070613144911700_bib53","first-page":"571","article-title":"The BEA 2024 shared task on the multilingual lexical simplification pipeline","volume-title":"Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024)","author":"Shardlow","year":"2024"},{"issue":"2","key":"2026070613144911700_bib54","doi-asserted-by":"crossref","first-page":"876","DOI":"10.1214\/aoms\/1177703591","article-title":"A relationship between arbitrary positive matrices and doubly stochastic matrices","volume":"35","author":"Sinkhorn","year":"1964","journal-title":"The annals of mathematical statistics"},{"key":"2026070613144911700_bib55","doi-asserted-by":"crossref","first-page":"5609","DOI":"10.18653\/v1\/2020.acl-main.496","article-title":"Rationalizing text matching: Learning sparse alignments via optimal transport","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Swanson","year":"2020"},{"key":"2026070613144911700_bib56","doi-asserted-by":"crossref","first-page":"3842","DOI":"10.18653\/v1\/2022.acl-long.266","article-title":"Ensembling and knowledge distilling of large sequence taggers for grammatical error correction","volume-title":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Tarnavskyi","year":"2022"},{"key":"2026070613144911700_bib57","volume-title":"Qwen3 technical report","author":"Yang","year":"2025"},{"key":"2026070613144911700_bib58","doi-asserted-by":"crossref","first-page":"6174","DOI":"10.18653\/v1\/2023.emnlp-main.378","article-title":"CLEME: Debiasing multi-reference evaluation for grammatical error correction","volume-title":"Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing","author":"Ye","year":"2023"},{"key":"2026070613144911700_bib59","doi-asserted-by":"crossref","first-page":"6516","DOI":"10.18653\/v1\/2020.coling-main.573","article-title":"SOME: Reference-less sub-metrics optimized for manual evaluations of grammatical error correction","volume-title":"Proceedings of the 28th International Conference on Computational Linguistics","author":"Yoshimura","year":"2020"},{"key":"2026070613144911700_bib60","article-title":"Hierarchical optimal transport for document representation","volume-title":"Advances in Neural Information Processing Systems","author":"Yurochkin","year":"2019"},{"key":"2026070613144911700_bib61","article-title":"BERTScore: Evaluating text generation with BERT","volume-title":"8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26\u201330, 2020","author":"Zhang","year":"2020"},{"key":"2026070613144911700_bib62","doi-asserted-by":"crossref","first-page":"563","DOI":"10.18653\/v1\/D19-1053","article-title":"MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Zhao","year":"2019"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/TACL.a.747\/2612393\/tacl.a.747.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/TACL.a.747\/2612393\/tacl.a.747.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,6]],"date-time":"2026-07-06T17:15:07Z","timestamp":1783358107000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/TACL.a.747\/137437\/Grammatical-Error-Correction-Evaluation-by"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026]]},"references-count":62,"URL":"https:\/\/doi.org\/10.1162\/tacl.a.747","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2026]]},"published":{"date-parts":[[2026]]}}}