{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,10]],"date-time":"2026-03-10T15:01:30Z","timestamp":1773154890686,"version":"3.50.1"},"reference-count":79,"publisher":"Association for Computing Machinery (ACM)","issue":"5","license":[{"start":{"date-parts":[[2024,6,4]],"date-time":"2024-06-04T00:00:00Z","timestamp":1717459200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2024,6,30]]},"abstract":"<jats:p>Machine translation systems have been widely adopted in our daily life, making life easier and more convenient. Unfortunately, erroneous translations may result in severe consequences, such as financial losses. This requires to improve the accuracy and the reliability of machine translation systems. However, it is challenging to test machine translation systems because of the complexity and intractability of the underlying neural models. To tackle these challenges, we propose a novel metamorphic testing approach by syntactic tree pruning (STP) to validate machine translation systems. Our key insight is that a pruned sentence should have similar crucial semantics compared with the original sentence. Specifically, STP (1) proposes a core semantics-preserving pruning strategy by basic sentence structures and dependency relations on the level of syntactic tree representation, (2) generates source sentence pairs based on the metamorphic relation, and (3) reports suspicious issues whose translations break the consistency property by a bag-of-words model. We further evaluate STP on two state-of-the-art machine translation systems (i.e., Google Translate and Bing Microsoft Translator) with 1,200 source sentences as inputs. The results show that STP accurately finds 5,073 unique erroneous translations in Google Translate and 5,100 unique erroneous translations in Bing Microsoft Translator (400% more than state-of-the-art techniques), with 64.5% and 65.4% precision, respectively. The reported erroneous translations vary in types and more than 90% of them are not found by state-of-the-art techniques. There are 9,393 erroneous translations unique to STP, which is 711.9% more than state-of-the-art techniques. Moreover, STP is quite effective in detecting translation errors for the original sentences with a recall reaching 74.0%, improving state-of-the-art techniques by 55.1% on average.<\/jats:p>","DOI":"10.1145\/3640329","type":"journal-article","created":{"date-parts":[[2024,1,10]],"date-time":"2024-01-10T12:26:16Z","timestamp":1704889576000},"page":"1-39","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":7,"title":["Machine Translation Testing via Syntactic Tree Pruning"],"prefix":"10.1145","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2495-3805","authenticated-orcid":false,"given":"Quanjun","family":"Zhang","sequence":"first","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5017-8016","authenticated-orcid":false,"given":"Juan","family":"Zhai","sequence":"additional","affiliation":[{"name":"Manning College of Information &amp; Computer Sciences, University of Massachusetts Amherst, Amherst, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9930-7111","authenticated-orcid":false,"given":"Chunrong","family":"Fang","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4930-9637","authenticated-orcid":false,"given":"Jiawei","family":"Liu","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9236-8264","authenticated-orcid":false,"given":"Weisong","family":"Sun","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-3007-488X","authenticated-orcid":false,"given":"Haichuan","family":"Hu","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-1693-4166","authenticated-orcid":false,"given":"Qingyu","family":"Wang","sequence":"additional","affiliation":[{"name":"State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,6,4]]},"reference":[{"key":"e_1_3_3_2_2","unstructured":"BBC. 2022. The British Broadcasting Corporation (BBC) News Homepage. Retrieved from https:\/\/www.bbc.com\/(accessed August 2022)."},{"key":"e_1_3_3_3_2","first-page":"1","volume-title":"Proceedings of the 6th International Conference on Learning Representations (ICLR\u201918)","author":"Belinkov Yonatan","year":"2018","unstructured":"Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation. In Proceedings of the 6th International Conference on Learning Representations (ICLR\u201918). 1\u201313."},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/3490488"},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-tutorials.8"},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE51524.2021.9678670"},{"key":"e_1_3_3_7_2","unstructured":"Tsong Y. Chen Shing C. Cheung and Shiu Ming Yiu. 2020. Metamorphic testing: A new approach for generating next test cases. Retrieved from https:\/\/arXiv:2002.12543"},{"key":"e_1_3_3_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/3143561"},{"key":"e_1_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1425"},{"key":"e_1_3_3_10_2","doi-asserted-by":"publisher","DOI":"10.1515\/9783110218329"},{"key":"e_1_3_3_11_2","unstructured":"CNN. 2022. The Cable News Network (CNN) News Homepage. Retrieved from https:\/\/edition.cnn.com\/(accessed August 2022)."},{"key":"e_1_3_3_12_2","unstructured":"China Daily. 2022. China Daily News Homepage. Retrieved from https:\/\/www.chinadaily.com.cn\/"},{"key":"e_1_3_3_13_2","unstructured":"IBM Cloud Docs. 2016. Machine Translation Tips. Retrieved from https:\/\/cloud.ibm.com\/docs\/GlobalizationPipeline?topic=GlobalizationPipeline-globalizationpipeline_tips&locale=en(accessed August 2022)."},{"key":"e_1_3_3_14_2","unstructured":"Yinpeng Dong Qi-An Fu Xiao Yang Tianyu Pang Hang Su Zihao Xiao and Jun Zhu. 2019. Benchmarking adversarial robustness. Retrieved from https:\/\/arXiv:1912.11852"},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338954"},{"key":"e_1_3_3_16_2","first-page":"653","volume-title":"Proceedings of the 27th International Conference on Computational Linguistics (COLING\u201918)","author":"Ebrahimi Javid","year":"2018","unstructured":"Javid Ebrahimi, Daniel Lowd, and Dejing Dou. 2018. On adversarial examples for character-level neural machine translation. In Proceedings of the 27th International Conference on Computational Linguistics (COLING\u201918). 653\u2013663."},{"key":"e_1_3_3_17_2","unstructured":"Google. 2022. Google Translate. Retrieved from https:\/\/translate.google.com(accessed August 2022)."},{"key":"e_1_3_3_18_2","unstructured":"Stanford NLP Group. 2022. CoreNLP. Retrieved from https:\/\/stanfordnlp.github.io\/CoreNLP(accessed August 2022)."},{"key":"e_1_3_3_19_2","first-page":"863","volume-title":"Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE\u201920)","author":"Gupta Shashij","year":"2020","unstructured":"Shashij Gupta, Pinjia He, Clara Meister, and Zhendong Su. 2020. Machine translation testing via pathological invariance. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE\u201920). 863\u2013875."},{"key":"e_1_3_3_20_2","unstructured":"Hany Hassan Anthony Aue Chang Chen Vishal Chowdhary Jonathan Clark Christian Federmann Xuedong Huang Marcin Junczys-Dowmunt William Lewis Mu Li et\u00a0al. 2018. Achieving human parity on automatic chinese to english news translation. Retrieved from https:\/\/arXiv:1803.05567"},{"key":"e_1_3_3_21_2","unstructured":"Pinjia He. 2022. Machine Translation Testing Toolkit. Retrieved from https:\/\/github.com\/RobustNLP\/TestTranslation(accessed August 2022)."},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380339"},{"key":"e_1_3_3_23_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00047"},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/3533767.3534394"},{"key":"e_1_3_3_25_2","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9781139165785"},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE51524.2021.9678715"},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D17-1215"},{"key":"e_1_3_3_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00108"},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/2594291.2594334"},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/3586027"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i05.6347"},{"key":"e_1_3_3_32_2","doi-asserted-by":"publisher","DOI":"10.1145\/2737924.2737986"},{"key":"e_1_3_3_33_2","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Lin Ji","year":"2018","unstructured":"Ji Lin, Chuang Gan, and Song Han. 2018. Defensive quantization: When efficiency meets robustness. In Proceedings of the International Conference on Learning Representations."},{"key":"e_1_3_3_34_2","first-page":"129","volume-title":"Proceedings of the 37th IEEE\/ACM International Conference on Software Engineering (ICSE\u201915)","volume":"2","author":"Lindvall Mikael","year":"2015","unstructured":"Mikael Lindvall, Dharmalingam Ganesan, Ragnar \u00c1rdal, and Robert E. Wiegand. 2015. Metamorphic model-based testing applied on NASA DAT\u2013an experience report. In Proceedings of the 37th IEEE\/ACM International Conference on Software Engineering (ICSE\u201915), Vol. 2. 129\u2013138."},{"key":"e_1_3_3_35_2","unstructured":"Lingua. 2022. The 20 Most Spoken Languages in the World in 2022. Retrieved from https:\/\/lingua.edu\/the-20-most-spoken-languages-in-the-world-in-2022\/(accessed August 2022)."},{"key":"e_1_3_3_36_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.227"},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511810213"},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3236082"},{"key":"e_1_3_3_39_2","doi-asserted-by":"publisher","DOI":"10.1515\/text.1.1988.8.3.243"},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P14-5010"},{"key":"e_1_3_3_41_2","unstructured":"Microsoft. 2022. Bing Microsoft Translator. Retrieved from https:\/\/www.bing.com\/translator(accessed August 2022)."},{"key":"e_1_3_3_42_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-1176"},{"key":"e_1_3_3_43_2","first-page":"867","volume-title":"Proceedings of the 20th International Conference on Software Engineering and Knowledge Engineering (SEKE\u201908)","author":"Murphy Christian","year":"2008","unstructured":"Christian Murphy, Gail E. Kaiser, Lifeng Hu, and Leon Wu. 2008. Properties of machine learning applications for use in metamorphic testing. In Proceedings of the 20th International Conference on Software Engineering and Knowledge Engineering (SEKE\u201908). 867\u2013872."},{"key":"e_1_3_3_44_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1333"},{"key":"e_1_3_3_45_2","first-page":"3956","volume-title":"Proceedings of the 35th International Conference on Machine Learning (ICML\u201918)","author":"Ott Myle","year":"2018","unstructured":"Myle Ott, Michael Auli, David Grangier, and Marc\u2019Aurelio Ranzato. 2018. Analyzing uncertainty in neural machine translation. In Proceedings of the 35th International Conference on Machine Learning (ICML\u201918). 3956\u20133965."},{"key":"e_1_3_3_46_2","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2016.41"},{"key":"e_1_3_3_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/3193977.3193980"},{"key":"e_1_3_3_48_2","volume-title":"A Comprehensive Grammar of the English Language","author":"Quirk Randolph","year":"2010","unstructured":"Randolph Quirk. 2010. A Comprehensive Grammar of the English Language. Pearson Education India."},{"key":"e_1_3_3_49_2","unstructured":"Reuters. 2022. Reuters News Homepage. Retrieved from https:\/\/www.reuters.com\/(accessed August 2022)."},{"key":"e_1_3_3_50_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2016.2532875"},{"key":"e_1_3_3_51_2","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3556953"},{"key":"e_1_3_3_52_2","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3556964"},{"key":"e_1_3_3_53_2","unstructured":"Punardeep Sikka Manmeet Singh Allen Pink and Vijay Mago. 2020. A survey on text simplification. Retrieved from https:\/\/arXiv:2008.08612"},{"key":"e_1_3_3_54_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASWEC.2018.00021"},{"key":"e_1_3_3_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380420"},{"key":"e_1_3_3_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510206"},{"key":"e_1_3_3_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180220"},{"key":"e_1_3_3_58_2","unstructured":"Barak Turovsky. 2016. Ten Years of Google Translate. Retrieved from https:\/\/blog.google\/products\/translate\/ten-years-of-google-translate\/"},{"key":"e_1_3_3_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510062"},{"key":"e_1_3_3_60_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00126"},{"key":"e_1_3_3_61_2","doi-asserted-by":"publisher","DOI":"10.1109\/DSN-Industry.2019.00007"},{"key":"e_1_3_3_62_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2010.2045169"},{"key":"e_1_3_3_63_2","unstructured":"Yonghui Wu Mike Schuster Zhifeng Chen Quoc V. Le Mohammad Norouzi Wolfgang Macherey Maxim Krikun Yuan Cao Qin Gao Klaus Macherey et\u00a0al. 2016. Google\u2019s neural machine translation system: Bridging the gap between human and machine translation. Retrieved from https:\/\/arXiv:1609.08144"},{"key":"e_1_3_3_64_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00935"},{"key":"e_1_3_3_65_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00706"},{"key":"e_1_3_3_66_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2010.11.920"},{"key":"e_1_3_3_67_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.451"},{"key":"e_1_3_3_68_2","unstructured":"Youdao. 2022. Youdao Translator. Retrieved from http:\/\/www.youdao.com(accessed August 2022)."},{"key":"e_1_3_3_69_2","doi-asserted-by":"publisher","DOI":"10.1145\/3533767.3534389"},{"key":"e_1_3_3_70_2","doi-asserted-by":"crossref","unstructured":"Fuyuan Zhang Sankalan Pal Chowdhury and Maria Christakis. 2019. DeepSearch: Simple and effective blackbox fuzzing of deep neural networks. Retrieved from https:\/\/arXiv:1910.06296","DOI":"10.1145\/3368089.3409750"},{"key":"e_1_3_3_71_2","doi-asserted-by":"publisher","DOI":"10.1145\/2642937.2642994"},{"key":"e_1_3_3_72_2","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238187"},{"key":"e_1_3_3_73_2","unstructured":"Quanjun Zhang and Haichuan Hu. 2023. STP Reproduction Artifacts. Retrieved from https:\/\/github.com\/iSEngLab\/STP(accessed December 2023)."},{"key":"e_1_3_3_74_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.acl-long.153"},{"key":"e_1_3_3_75_2","doi-asserted-by":"publisher","DOI":"10.1145\/3213846.3213866"},{"key":"e_1_3_3_76_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1244"},{"key":"e_1_3_3_77_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-Companion.2019.00131"},{"key":"e_1_3_3_78_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2015.2478001"},{"key":"e_1_3_3_79_2","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.437"},{"key":"e_1_3_3_80_2","first-page":"434","volume-title":"Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL\u201913)","author":"Zhu Muhua","year":"2013","unstructured":"Muhua Zhu, Yue Zhang, Wenliang Chen, Min Zhang, and Jingbo Zhu. 2013. Fast and accurate shift-reduce constituent parsing. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL\u201913). 434\u2013443."}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3640329","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3640329","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T22:50:26Z","timestamp":1750287026000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3640329"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6,4]]},"references-count":79,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2024,6,30]]}},"alternative-id":["10.1145\/3640329"],"URL":"https:\/\/doi.org\/10.1145\/3640329","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,6,4]]},"assertion":[{"value":"2022-08-31","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-12-31","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-06-04","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}