{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,21]],"date-time":"2026-04-21T14:50:57Z","timestamp":1776783057026,"version":"3.51.2"},"reference-count":69,"publisher":"Association for Computing Machinery (ACM)","issue":"8","license":[{"start":{"date-parts":[[2024,11,22]],"date-time":"2024-11-22T00:00:00Z","timestamp":1732233600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2024,11,30]]},"abstract":"<jats:p>With the wide application of machine translation, the testing of Machine Translation Systems (MTSs) has attracted much attention. Recent works apply Metamorphic Testing (MT) to address the oracle problem in MTS testing. Existing MT methods for MTS generally follow the workflow of input transformation and output relation comparison, which generates a follow-up input sentence by mutating the source input and compares the source and follow-up output translations to detect translation errors, respectively. These methods use various input transformations to generate the test case pairs and have successfully triggered numerous translation errors. However, they have limitations in performing fine-grained and rigorous output relation comparison and thus may report many false alarms and miss many true errors. In this article, we propose a word closure-based output comparison method to address the limitations of the existing MTS MT methods. We first propose word closure as a new comparison unit, where each closure includes a group of correlated input and output words in the test case pair. Word closures suggest the linkages between the appropriate fragment in the source output translation and its counterpart in the follow-up output for comparison. Next, we compare the semantics on the level of word closure to identify the translation errors. In this way, we perform a fine-grained and rigorous semantic comparison for the outputs and thus realize more effective violation identification. We evaluate our method with the test cases generated by five existing input transformations and the translation outputs from three popular MTSs. Results show that our method significantly outperforms the existing works in violation identification by improving the precision and recall and achieving an average increase of 29.9% in F1 score. It also helps to increase the F1 score of translation error localization by 35.9%.<\/jats:p>","DOI":"10.1145\/3675396","type":"journal-article","created":{"date-parts":[[2024,7,1]],"date-time":"2024-07-01T16:54:10Z","timestamp":1719852850000},"page":"1-46","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":6,"title":["Word Closure-Based Metamorphic Testing for Machine Translation"],"prefix":"10.1145","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2828-7165","authenticated-orcid":false,"given":"Xiaoyuan","family":"Xie","sequence":"first","affiliation":[{"name":"School of Computer Science, Wuhan University, Wuhan, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5761-5290","authenticated-orcid":false,"given":"Shuo","family":"Jin","sequence":"additional","affiliation":[{"name":"School of Computer Science, Wuhan University, Wuhan, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1220-8728","authenticated-orcid":false,"given":"Songqiang","family":"Chen","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3508-7172","authenticated-orcid":false,"given":"Shing-Chi","family":"Cheung","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,11,22]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"2023. Bing Microsoft Translator. Retrieved from https:\/\/www.bing.com\/translator"},{"key":"e_1_3_2_3_2","unstructured":"2023. Dataset Replication Package and Supplementary Material for This Paper. Retrieved from https:\/\/github.com\/imjinshuo\/Word-Closure-Based-MT"},{"key":"e_1_3_2_4_2","unstructured":"2023. Google Translate. Retrieved from https:\/\/translate.google.com\/"},{"key":"e_1_3_2_5_2","unstructured":"2023. Youdao Translate. Retrieved from https:\/\/fanyi.youdao.com\/"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2021.3136169"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2014.2372785"},{"key":"e_1_3_2_8_2","unstructured":"Terena Bell. 2021. Google Translate Causes Vaccine Mishap. Retrieved from https:\/\/multilingual.com\/google-translate-causes-vaccine-mishap\/"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/3490488"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3440755"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE51524.2021.9678670"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3468569"},{"key":"e_1_3_2_13_2","volume-title":"Metamorphic Testing: A New Approach for Generating Next Test Cases. Technical Report HKUST-CS98-01","author":"Chen Tsong Yueh","year":"1998","unstructured":"Tsong Yueh Chen, S. C. Cheung, and Siu-Ming Yiu. 1998. Metamorphic Testing: A New Approach for Generating Next Test Cases. Technical Report HKUST-CS98-01, Department of Computer Science, Hong Kong University of Science and Technology, Hong Kong."},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3143561"},{"key":"e_1_3_2_15_2","unstructured":"Steve Clayton. 2013. Translation Tech Powers Automatic Subtitles for Everyday Life. Retrieved from https:\/\/blogs.microsoft.com\/ai\/translation-tech-powers-automatic-subtitles-for-everyday-life\/"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1177\/001316446002000104"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1609\/AAAI.V36I10.21299"},{"key":"e_1_3_2_18_2","unstructured":"Gareth Davies. 2017. Palestinian Man is Arrested by Police After Posting \u2018Good Morning\u2019 in Arabic on Facebook Which was Wrongly Translated as \u2018Attack Them\u2019. Retrieved from https:\/\/www.dailymail.co.uk\/news\/article-5005489\/Good-morning-Facebook-post-leads-arrest-Palestinian.html"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/N19-1423"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.eacl-main.181"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/D19-1006"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1007\/S11704-021-0277-0"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1016\/J.NEUROIMAGE.2007.01.015"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.5555\/972791.972799"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/W18-6317"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409756"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380339"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00047"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE51524.2021.9678715"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE52982.2021.00033"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1016\/J.INFSOF.2022.106966"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE51524.2021.9678807"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.3115\/1654650.1654666"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/P18-2023"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/2021.FINDINGS-ACL.86"},{"key":"e_1_3_2_36_2","first-page":"74","volume-title":"Text Summarization Branches Out","author":"Lin Chin-Yew","year":"2004","unstructured":"Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out. Chin-Yew Lin (Ed.), Association for Computational Linguistics, Barcelona, Spain, 74\u201381."},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1162\/TACL_A_00343"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","unstructured":"Zixi Liu Yang Feng and Zhenyu Chen. 2021. DialTest: Automated Testing for Recurrent-Neural-Network-Driven Dialogue Systems. In Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA \u201921). Cristian Cadar and Xiangyu Zhang (Eds.) ACM New York NY 115\u2013126. DOI: 10.1145\/3460319.3464829","DOI":"10.1145\/3460319.3464829"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3556929"},{"issue":"1","key":"e_1_3_2_40_2","first-page":"3","article-title":"Automatically Building a Stopword List for an Information Retrieval System","volume":"3","author":"Lo Rachel Tsz-Wai","year":"2005","unstructured":"Rachel Tsz-Wai Lo, Ben He, and Iadh Ounis. 2005. Automatically Building a Stopword List for an Information Retrieval System. J. Digit. Inf. Manag. 3, 1 (2005), 3\u20138.","journal-title":"J. Digit. Inf. Manag"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.24963\/IJCAI.2020\/64"},{"key":"e_1_3_2_42_2","unstructured":"Fiona Macdonald. 2015. The Greatest Mistranslations Ever. Retrieved from https:\/\/www.bbc.com\/culture\/article\/20150202-the-greatest-mistranslations-ever"},{"key":"e_1_3_2_43_2","first-page":"1","volume-title":"Proceedings of the 1st International Conference on Learning Representations (ICLR \u201913)","author":"Mikolov Tom\u00e1s","year":"2013","unstructured":"Tom\u00e1s Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient Estimation of Word Representations in Vector Space. In Proceedings of the 1st International Conference on Learning Representations (ICLR \u201913), 1\u201312."},{"key":"e_1_3_2_44_2","first-page":"746","volume-title":"Proceedings of the Human Language Technologies: Conference of the North American Chapter of the Association of Computational Linguistics","author":"Mikolov Tom\u00e1s","year":"2013","unstructured":"Tom\u00e1s Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013b. Linguistic Regularities in Continuous Space Word Representations. In Proceedings of the Human Language Technologies: Conference of the North American Chapter of the Association of Computational Linguistics. Association for Computational Linguistics, 746\u2013751."},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/219717.219748"},{"key":"e_1_3_2_46_2","first-page":"167","volume-title":"Proceedings of the 19th International Conference on Software Engineering & Knowledge Engineering (SEKE \u201907)","author":"Murphy Chris","year":"2007","unstructured":"Chris Murphy, Gail E. Kaiser, and Marta Arias. 2007. An Approach to Software Testing of Machine Learning Applications. In Proceedings of the 19th International Conference on Software Engineering & Knowledge Engineering (SEKE \u201907). Knowledge Systems Institute Graduate School, 167."},{"key":"e_1_3_2_47_2","unstructured":"Arika Okrent. 2016. 9 Little Translation Mistakes That Caused Big Problems. Retrieved from https:\/\/www.mentalfloss.com\/article\/48795\/9-little-translation-mistakes-caused-big-problems"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.3115\/1073083.1073135"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/3193977.3193980"},{"key":"e_1_3_2_50_2","unstructured":"The Copenhagen Post. 2012. Police Admit Using Google Translation in Terror Investigation was Mistake. Retrieved from https:\/\/cphpost.dk\/2012-12-12\/general\/police-admit-using-google-translation-in-terror-investigation-was-mistake\/"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/2020.ACL-MAIN.442"},{"key":"e_1_3_2_52_2","unstructured":"Stephen Shankland. 2013. Google Translate Now Serves 200 Million People Daily. Retrieved from https:\/\/www.cnet.com\/tech\/services-and-software\/google-translate-now-serves-200-million-people-daily"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3556953"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-73289-1_31"},{"key":"e_1_3_2_55_2","first-page":"127","volume-title":"Proceedings of the Australasian Language Technology Workshop (ALTA \u201905)","author":"Somers Harold L.","year":"2005","unstructured":"Harold L. Somers. 2005. Round-trip Translation: What Is It Good For?. In Proceedings of the Australasian Language Technology Workshop (ALTA \u201905). Australasian Language Technology Association, 127\u2013133."},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380420"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510206"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1093\/oso\/9780195142242.003.0008"},{"key":"e_1_3_2_59_2","first-page":"2214","volume-title":"Proceedings of the 8th International Conference on Language Resources and Evaluation (LREC\u201912)European Language Resources Association","author":"Tiedemann J\u00f6rg","year":"2012","unstructured":"J\u00f6rg Tiedemann. 2012. Parallel Data, Tools and Interfaces in OPUS. In Proceedings of the 8th International Conference on Language Resources and Evaluation (LREC\u201912). Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Mehmet Ugur Dogan, Bente Maegaard, Joseph Mariani, Jan Odijk, and Stelios Piperidis (Eds.), European Language Resources Association, 2214\u20132218."},{"key":"e_1_3_2_60_2","first-page":"137","article-title":"Investigating the Use of Google Translate in\u201d Terms and Conditions\u201d in an Airline's Official Website: Errors and Implications","volume":"49","author":"Vidhayasai Tya","year":"2015","unstructured":"Tya Vidhayasai, Sonthida Keyuravong, and Thanis Bunsom. 2015. Investigating the Use of Google Translate in\u201d Terms and Conditions\u201d in an Airline's Official Website: Errors and Implications. PASAA J Lang Teach Learn Thai 49, 137\u2013169.","journal-title":"PASAA J Lang Teach Learn Thai"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00200"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1109\/DSN-INDUSTRY.2019.00007"},{"key":"e_1_3_2_63_2","unstructured":"Wikipedia. 2023a. Lexical Analysis. Retrieved from https:\/\/en.wikipedia.org\/wiki\/Lexical_analysis"},{"key":"e_1_3_2_64_2","unstructured":"Wikipedia. 2023b. Phrase. Retrieved from https:\/\/en.wikipedia.org\/wiki\/Phrase"},{"key":"e_1_3_2_65_2","unstructured":"Wikipedia. 2023c. Verb Phrase. Retrieved from https:\/\/en.wikipedia.org\/wiki\/Verb_phrase"},{"key":"e_1_3_2_66_2","unstructured":"WMT. 2018. News-Commentary. Retrieved from http:\/\/data.statmt.org\/wmt18\/translation-task\/"},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","DOI":"10.1007\/S10515-023-00380-2"},{"key":"e_1_3_2_68_2","doi-asserted-by":"publisher","unstructured":"Boxi Yu Yiyan Hu Qiuyang Mang Wenhan Hu and Pinjia He. 2023. Automated Testing and Improvement of Named Entity Recognition Systems. arXiv:2308.07937. 10.48550\/ARXIV.2308.07937","DOI":"10.48550\/ARXIV.2308.07937"},{"key":"e_1_3_2_69_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASWEC.2018.00021"},{"key":"e_1_3_2_70_2","first-page":"434","volume-title":"Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL \u201913)","volume":"1","author":"Zhu Muhua","year":"2013","unstructured":"Muhua Zhu, Yue Zhang, Wenliang Chen, Min Zhang, and Jingbo Zhu. 2013. Fast and Accurate Shift-Reduce Constituent Parsing. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL \u201913), Vol. 1, Long Papers. Association for Computer Linguistics, 434\u2013443."}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3675396","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3675396","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:04:23Z","timestamp":1750291463000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3675396"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,11,22]]},"references-count":69,"journal-issue":{"issue":"8","published-print":{"date-parts":[[2024,11,30]]}},"alternative-id":["10.1145\/3675396"],"URL":"https:\/\/doi.org\/10.1145\/3675396","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,11,22]]},"assertion":[{"value":"2024-01-21","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-06-11","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-11-22","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}