{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,24]],"date-time":"2026-03-24T03:31:33Z","timestamp":1774323093613,"version":"3.50.1"},"reference-count":39,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2023,3,10]],"date-time":"2023-03-10T00:00:00Z","timestamp":1678406400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Aid for Scientific Research from the Japan Society for the Promotion of Science","award":["JP19K12099 and JP19KK0286"],"award-info":[{"award-number":["JP19K12099 and JP19KK0286"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Asian Low-Resour. Lang. Inf. Process."],"published-print":{"date-parts":[[2023,3,31]]},"abstract":"<jats:p>\n            In recent studies, pre-trained models and pseudo data have been key factors in improving the performance of the English\n            <jats:bold>grammatical error correction (GEC)<\/jats:bold>\n            task. However, few studies have examined the role of pre-trained models and pseudo data in the Chinese GEC task. Therefore, we develop Chinese GEC models based on three pre-trained models: Chinese BERT, Chinese T5, and Chinese BART, and then incorporate these models with pseudo data to determine the best configuration for the Chinese GEC task. On the\n            <jats:bold>natural language processing and Chinese computing (NLPCC)<\/jats:bold>\n            2018 GEC shared task test set, all our single models outperform the ensemble models developed by the top team of the shared task. Chinese BART achieves an F score of 37.15, which is a state-of-the-art result. We then combine our Chinese GEC models with three kinds of pseudo data: Lang-8 (MaskGEC), Wiki (MaskGEC), and Wiki (Backtranslation). We find that most models can benefit from pseudo data, and BART+Lang-8 (MaskGEC) is the ideal setting in terms of accuracy and training efficiency. The experimental results demonstrate the effectiveness of the pre-trained models and pseudo data on the Chinese GEC task and provide an easily reproducible and adaptable baseline for future works. Finally, we annotate the error types of the development data; the results show that word-level errors dominate all error types, and word selection errors must be addressed even when using pre-trained models and pseudo data. Our codes are available at\n            <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"url\" xlink:href=\"https:\/\/github.com\/wang136906578\/BERT-encoder-ChineseGEC\">https:\/\/github.com\/wang136906578\/BERT-encoder-ChineseGEC<\/jats:ext-link>\n            .\n          <\/jats:p>","DOI":"10.1145\/3570209","type":"journal-article","created":{"date-parts":[[2022,11,2]],"date-time":"2022-11-02T13:02:51Z","timestamp":1667394171000},"page":"1-12","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":6,"title":["Chinese Grammatical Error Correction Using Pre-trained Models and Pseudo Data"],"prefix":"10.1145","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7169-0796","authenticated-orcid":false,"given":"Hongfei","family":"Wang","sequence":"first","affiliation":[{"name":"Tokyo Metropolitan University, Hinoshi, Tokyo, Japan"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7452-9861","authenticated-orcid":false,"given":"Michiki","family":"Kurosawa","sequence":"additional","affiliation":[{"name":"Tokyo Metropolitan University, Hinoshi, Tokyo, Japan"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9028-3598","authenticated-orcid":false,"given":"Satoru","family":"Katsumata","sequence":"additional","affiliation":[{"name":"Tokyo Metropolitan University, Hinoshi, Tokyo, Japan"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6210-3716","authenticated-orcid":false,"given":"Masato","family":"Mita","sequence":"additional","affiliation":[{"name":"CyberAgent, Inc., Shibuya-ku, Tokyo, Japan"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1166-1739","authenticated-orcid":false,"given":"Mamoru","family":"Komachi","sequence":"additional","affiliation":[{"name":"Tokyo Metropolitan University, Hinoshi, Tokyo, Japan"}]}],"member":"320","published-online":{"date-parts":[[2023,3,10]]},"reference":[{"key":"e_1_3_3_2_2","first-page":"49","volume-title":"NLPTEA","author":"Cao Yongchang","year":"2020","unstructured":"Yongchang Cao, Liang He, Robert Ridley, and Xinyu Dai. 2020. Integrating BERT and score-based feature gates for Chinese grammatical error diagnosis. In NLPTEA. 49\u201356."},{"key":"e_1_3_3_3_2","first-page":"7162","volume-title":"EMNLP","author":"Chen Mengyun","year":"2020","unstructured":"Mengyun Chen, Tao Ge, Xingxing Zhang, Furu Wei, and Ming Zhou. 2020. Improving the efficiency of grammatical error correction with erroneous span detection and correction. In EMNLP. 7162\u20137169."},{"key":"e_1_3_3_4_2","first-page":"5755","volume-title":"AAAI","author":"Chollampatt Shamil","year":"2018","unstructured":"Shamil Chollampatt and Hwee Tou Ng. 2018. A multilayer convolutional encoder-decoder neural network for grammatical error correction. In AAAI. 5755\u20135762."},{"key":"e_1_3_3_5_2","volume-title":"Findings of EMNLP","author":"Cui Yiming","year":"2020","unstructured":"Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. 2020. Revisiting pre-trained models for Chinese natural language processing. In Findings of EMNLP."},{"key":"e_1_3_3_6_2","first-page":"4171","volume-title":"NAACL-HLT","author":"Devlin Jacob","year":"2019","unstructured":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT. 4171\u20134186."},{"key":"e_1_3_3_7_2","first-page":"67","volume-title":"NLPTEA","author":"Fang Meiyuan","year":"2020","unstructured":"Meiyuan Fang, Kai Fu, Jiping Wang, Yang Liu, Jin Huang, and Yitao Duan. 2020. A hybrid system for NLPTEA-2020 CGED shared task. In NLPTEA. 67\u201377."},{"key":"e_1_3_3_8_2","first-page":"341","volume-title":"NLPCC","author":"Fu Kai","year":"2018","unstructured":"Kai Fu, Jin Huang, and Yitao Duan. 2018. Youdao\u2019s winning solution to the NLPCC-2018 task 2 challenge: A neural machine translation approach to Chinese grammatical error correction. In NLPCC. 341\u2013350."},{"key":"e_1_3_3_9_2","first-page":"2191","volume-title":"COLING","author":"Hinson Charles","year":"2020","unstructured":"Charles Hinson, Hen-Hsen Huang, and Hsin-Hsi Chen. 2020. Heterogeneous recycle generation for Chinese grammatical error correction. In COLING. 2191\u20132201."},{"key":"e_1_3_3_10_2","first-page":"595","volume-title":"NAACL-HLT","author":"Junczys-Dowmunt Marcin","year":"2018","unstructured":"Marcin Junczys-Dowmunt, Roman Grundkiewicz, Shubha Guha, and Kenneth Heafield. 2018. Approaching neural grammatical error correction as a low-resource machine translation task. In NAACL-HLT. 595\u2013606."},{"key":"e_1_3_3_11_2","first-page":"207","volume-title":"BEA","author":"Kaneko Masahiro","year":"2019","unstructured":"Masahiro Kaneko, Kengo Hotate, Satoru Katsumata, and Mamoru Komachi. 2019. TMU transformer system using BERT for re-ranking at BEA 2019 grammatical error correction on restricted track. In BEA. 207\u2013212."},{"key":"e_1_3_3_12_2","first-page":"4248","volume-title":"ACL","author":"Kaneko Masahiro","year":"2020","unstructured":"Masahiro Kaneko, Masato Mita, Shun Kiyono, Jun Suzuki, and Kentaro Inui. 2020. Encoder-decoder models can benefit from pre-trained masked language models in grammatical error correction. In ACL. 4248\u20134254."},{"key":"e_1_3_3_13_2","first-page":"139","volume-title":"BEA","author":"Kantor Yoav","year":"2019","unstructured":"Yoav Kantor, Yoav Katz, Leshem Choshen, Edo Cohen-Karlik, Naftali Liberman, Assaf Toledo, Amir Menczel, and Noam Slonim. 2019. Learning to combine grammatical error corrections. In BEA. 139\u2013148."},{"key":"e_1_3_3_14_2","first-page":"827","volume-title":"AACL-IJCNLP","author":"Katsumata Satoru","year":"2020","unstructured":"Satoru Katsumata and Mamoru Komachi. 2020. Stronger baselines for grammatical error correction using a pretrained encoder-decoder model. In AACL-IJCNLP. 827\u2013832."},{"key":"e_1_3_3_15_2","first-page":"1236","volume-title":"EMNLP-IJCNLP","author":"Kiyono Shun","year":"2019","unstructured":"Shun Kiyono, Jun Suzuki, Masato Mita, Tomoya Mizumoto, and Kentaro Inui. 2019. An empirical study of incorporating pseudo data into grammatical error correction. In EMNLP-IJCNLP. 1236\u20131242."},{"key":"e_1_3_3_16_2","first-page":"7871","volume-title":"ACL","author":"Lewis Mike","year":"2020","unstructured":"Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In ACL. 7871\u20137880."},{"key":"e_1_3_3_17_2","first-page":"57","volume-title":"NLPTEA","author":"Liang Deng","year":"2020","unstructured":"Deng Liang, Chen Zheng, Lei Guo, Xin Cui, Xiuzhang Xiong, Hengqiao Rong, and Jinpeng Dong. 2020. BERT enhanced neural machine translation and sequence tagging model for Chinese grammatical error diagnosis. In NLPTEA. 57\u201366."},{"key":"e_1_3_3_18_2","first-page":"13","article-title":"RoBERTa: A robustly optimized BERT pretraining approach","volume":"1907","author":"Liu Y.","year":"2019","unstructured":"Y. Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, M. Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. ArXiv abs\/1907.11692 (2019), 13 pages.","journal-title":"ArXiv"},{"key":"e_1_3_3_19_2","first-page":"147","volume-title":"IJCNLP","author":"Mizumoto Tomoya","year":"2011","unstructured":"Tomoya Mizumoto, Mamoru Komachi, Masaaki Nagata, and Yuji Matsumoto. 2011. Mining revision log of language learning SNS for automated Japanese error correction of second language learners. In IJCNLP. 147\u2013155."},{"key":"e_1_3_3_20_2","first-page":"1837","volume-title":"ACL","author":"Nagata Ryo","year":"2016","unstructured":"Ryo Nagata and Keisuke Sakaguchi. 2016. Phrase structure annotation and parsing for learner English. In ACL. 1837\u20131847."},{"key":"e_1_3_3_21_2","first-page":"346","volume-title":"W-NUT","author":"N\u00e1plava Jakub","year":"2019","unstructured":"Jakub N\u00e1plava and Milan Straka. 2019. Grammatical error correction in low-resource scenarios. In W-NUT. 346\u2013356."},{"key":"e_1_3_3_22_2","first-page":"1","volume-title":"CoNLL","author":"Ng Hwee Tou","year":"2014","unstructured":"Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. 2014. The CoNLL-2014 shared task on grammatical error correction. In CoNLL. 1\u201314."},{"key":"e_1_3_3_23_2","first-page":"1","volume-title":"CoNLL","author":"Ng Hwee Tou","year":"2013","unstructured":"Hwee Tou Ng, Siew Mei Wu, Yuanbin Wu, Christian Hadiwinoto, and Joel Tetreault. 2013. The CoNLL-2013 shared task on grammatical error correction. In CoNLL. 1\u201312."},{"key":"e_1_3_3_24_2","first-page":"163","volume-title":"BEA","author":"Omelianchuk Kostiantyn","year":"2020","unstructured":"Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem Chernodub, and Oleksandr Skurzhanskyi. 2020. GECToR \u2013 grammatical error correction: Tag, not rewrite. In BEA. 163\u2013170."},{"key":"e_1_3_3_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2019.2940607"},{"issue":"140","key":"e_1_3_3_26_2","first-page":"1","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","volume":"21","author":"Raffel Colin","year":"2020","unstructured":"Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR 21, 140 (2020), 1\u201367.","journal-title":"JMLR"},{"key":"e_1_3_3_27_2","first-page":"401","volume-title":"NLPCC","author":"Ren Hongkai","year":"2018","unstructured":"Hongkai Ren, Liner Yang, and Endong Xun. 2018. A sequence to sequence learning for Chinese grammatical error correction. In NLPCC. 401\u2013410."},{"key":"e_1_3_3_28_2","first-page":"86","volume-title":"ACL","author":"Sennrich Rico","year":"2016","unstructured":"Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In ACL. 86\u201396."},{"key":"e_1_3_3_29_2","first-page":"9","article-title":"CPT: A pre-trained unbalanced transformer for both Chinese language understanding and generation","volume":"2109","author":"Shao Yunfan","year":"2021","unstructured":"Yunfan Shao, Zhichao Geng, Yitao Liu, Junqi Dai, Fei Yang, Li Zhe, Hujun Bao, and Xipeng Qiu. 2021. CPT: A pre-trained unbalanced transformer for both Chinese language understanding and generation. arXiv abs\/2109.05729 (2021), 9 pages.","journal-title":"arXiv"},{"key":"e_1_3_3_30_2","volume-title":"T5 PEGASUS - ZhuiyiAI","author":"Su Jianlin","year":"2021","unstructured":"Jianlin Su. 2021. T5 PEGASUS - ZhuiyiAI. Technical Report. https:\/\/github.com\/ZhuiyiTechnology\/t5-pegasus."},{"key":"e_1_3_3_31_2","first-page":"5937","volume-title":"ACL-IJCNLP","author":"Sun Xin","year":"2021","unstructured":"Xin Sun, Tao Ge, Furu Wei, and Houfeng Wang. 2021. Instantaneous grammatical error correction with shallow aggressive decoding. In ACL-IJCNLP. 5937\u20135947."},{"key":"e_1_3_3_32_2","first-page":"15","volume-title":"ICLR","author":"Wang Benyou","year":"2020","unstructured":"Benyou Wang, Donghao Zhao, Christina Lioma, Qiuchi Li, Peng Zhang, and Jakob Grue Simonsen. 2020. Encoding word order in complex embeddings. In ICLR. 15 pages."},{"key":"e_1_3_3_33_2","article-title":"YACLC: A Chinese learner corpus with multidimensional annotation","volume":"2112","author":"Wang Yingying","year":"2021","unstructured":"Yingying Wang, Cunliang Kong, Liner Yang, Yijun Wang, Xiaorong Lu, Renfen Hu, Shan He, Zhenghao Liu, Yuxiang Chen, Erhong Yang, and Maosong Sun. 2021. YACLC: A Chinese learner corpus with multidimensional annotation. ArXiv abs\/2112.15043 (2021).","journal-title":"ArXiv"},{"key":"e_1_3_3_34_2","first-page":"78","volume-title":"BEA","author":"Wang Yi","year":"2020","unstructured":"Yi Wang, Ruibin Yuan, Yan\u2018gen Luo, Yufang Qin, NianYong Zhu, Peng Cheng, and Lihuan Wang. 2020. Chinese grammatical error correction based on hybrid models with data augmentation. In BEA. 78\u201386."},{"key":"e_1_3_3_35_2","first-page":"619","volume-title":"NAACL-HLT","author":"Xie Ziang","year":"2018","unstructured":"Ziang Xie, Guillaume Genthial, Stanley Xie, Andrew Ng, and Dan Jurafsky. 2018. Noising and denoising natural language: Diverse backtranslation for grammar correction. In NAACL-HLT. 619\u2013628."},{"key":"e_1_3_3_36_2","first-page":"13","volume-title":"Findings of ACL","author":"Xu Heng-Da","year":"2021","unstructured":"Heng-Da Xu, Zhongli Li, Qingyu Zhou, Chao Li, Zizhen Wang, Yunbo Cao, Heyan Huang, and Xian-Ling Mao. 2021. Read, listen, and see: Leveraging multimodal information helps Chinese spell checking. In Findings of ACL. 13 pages."},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.aiopen.2021.06.001"},{"key":"e_1_3_3_38_2","first-page":"12","volume-title":"ICML","author":"Zhang Jingqing","year":"2020","unstructured":"Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2020. PEGASUS: Pre-training with extracted gap-sentences for abstractive summarization. In ICML. 12 pages."},{"key":"e_1_3_3_39_2","first-page":"439","volume-title":"NLPCC","author":"Zhao Yuanyuan","year":"2018","unstructured":"Yuanyuan Zhao, Nan Jiang, Weiwei Sun, and Xiaojun Wan. 2018. Overview of the NLPCC 2018 shared task: Grammatical error correction. In NLPCC. 439\u2013445."},{"key":"e_1_3_3_40_2","first-page":"1226","volume-title":"AAAI","author":"Zhao Zewei","year":"2020","unstructured":"Zewei Zhao and Houfeng Wang. 2020. MaskGEC: Improving neural grammatical error correction via dynamic masking. In AAAI. 1226\u20131233."}],"container-title":["ACM Transactions on Asian and Low-Resource Language Information Processing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3570209","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3570209","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T17:49:37Z","timestamp":1750182577000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3570209"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,3,10]]},"references-count":39,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2023,3,31]]}},"alternative-id":["10.1145\/3570209"],"URL":"https:\/\/doi.org\/10.1145\/3570209","relation":{},"ISSN":["2375-4699","2375-4702"],"issn-type":[{"value":"2375-4699","type":"print"},{"value":"2375-4702","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,3,10]]},"assertion":[{"value":"2021-07-25","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-10-11","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-03-10","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}