{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T14:51:21Z","timestamp":1787064681539,"version":"build-2736575974"},"reference-count":98,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2022,4,1]],"date-time":"2022-04-01T00:00:00Z","timestamp":1648771200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"National Key Research and Development Program of China","award":["2019YFE0198100"],"award-info":[{"award-number":["2019YFE0198100"]}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["61932021, 61872339, 61472405, and 62002125"],"award-info":[{"award-number":["61932021, 61872339, 61472405, and 62002125"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Hong Kong ITF","award":["MHP\/055\/19"],"award-info":[{"award-number":["MHP\/055\/19"]}]},{"name":"Hong Kong RGC\/RIF","award":["R5034-18"],"award-info":[{"award-number":["R5034-18"]}]},{"name":"MSRA Collaborative Research Grant"},{"name":"Huawei PhD Fellowship"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2022,4,30]]},"abstract":"<jats:p>Machine translation has wide applications in daily life. In mission-critical applications such as translating official documents, incorrect translation can have unpleasant or sometimes catastrophic consequences. This motivates recent research on the testing methodologies for machine translation systems. Existing methodologies mostly rely on metamorphic relations designed at the textual level (e.g., Levenshtein distance) or syntactic level (e.g., distance between grammar structures) to determine the correctness of translation results. However, these metamorphic relations do not consider whether the original and the translated sentences have the same meaning (i.e., semantic similarity). To address this problem, in this article we propose SemMT, an automatic testing approach for machine translation systems based on semantic similarity checking. SemMT applies round-trip translation and measures the semantic similarity between the original and the translated sentences. Our insight is that the semantics concerning logical relations and quantifiers in sentences can be captured by regular expressions (or deterministic finite automata) where efficient semantic equivalence\/similarity checking algorithms can be applied. Leveraging the insight, we propose three semantic similarity metrics and implement them in SemMT. We compared SemMT with related state-of-the-art testing techniques, demonstrating the effectiveness of mistranslation detection. The experiment results show that SemMT outperforms existing metrics, achieving an increase of 34.2% and 15.4% on accuracy and F-score, respectively. We also study the possibility of further enhancing the performance by combining various metrics. Finally, we discuss a solution to locate the suspicious trip in round-trip translation, which provides hints for bug diagnosis.<\/jats:p>","DOI":"10.1145\/3490488","type":"journal-article","created":{"date-parts":[[2022,4,1]],"date-time":"2022-04-01T05:53:52Z","timestamp":1648792432000},"page":"1-36","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":38,"title":["SemMT: A Semantic-Based Testing Approach for Machine Translation Systems"],"prefix":"10.1145","volume":"31","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4892-6294","authenticated-orcid":false,"given":"Jialun","family":"Cao","sequence":"first","affiliation":[{"name":"The Hong Kong University of Science and Technology, Hong Kong, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5947-4030","authenticated-orcid":false,"given":"Meiziniu","family":"Li","sequence":"additional","affiliation":[{"name":"The Hong Kong University of Science and Technology, Hong Kong, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0991-4231","authenticated-orcid":false,"given":"Yeting","family":"Li","sequence":"additional","affiliation":[{"name":"University of Chinese Academy of Sciences, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5588-9618","authenticated-orcid":false,"given":"Ming","family":"Wen","sequence":"additional","affiliation":[{"name":"Huazhong University of Science and Technology, Wuhan, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3508-7172","authenticated-orcid":false,"given":"Shing-Chi","family":"Cheung","sequence":"additional","affiliation":[{"name":"The Hong Kong University of Science and Technology, Hong Kong, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2659-4148","authenticated-orcid":false,"given":"Haiming","family":"Chen","sequence":"additional","affiliation":[{"name":"University of Chinese Academy of Sciences, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2022,4]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"StatMT.org. 2013. ACL 2013 Eighth Workshop on Statistical Machine Translation. Retrieved January 25 2022 from http:\/\/www.statmt.org\/wmt13\/."},{"key":"e_1_3_2_3_2","unstructured":"GitHub. 2021. SemMT Retrieved January 25 2022 from https:\/\/github.com\/ArabelaTso\/SemMT."},{"key":"e_1_3_2_4_2","unstructured":"Khurshid Ahmad and Heather Fulford. 1992. Semantic Relations and Their Use in Elaborating Terminology . Computing Sciences Report CS-92-07. University of Surry Guildford England."},{"issue":"1","key":"e_1_3_2_5_2","article-title":"The efficacy of round-trip translation for MT evaluation","volume":"14","author":"Aiken Milam","year":"2010","unstructured":"Milam Aiken and Mina Park. 2010. The efficacy of round-trip translation for MT evaluation. Translation Journal 14, 1 (2010), 1\u201310.","journal-title":"Translation Journal"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511621017"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.5555\/1182635.1164206"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-demos.20"},{"key":"e_1_3_2_9_2","volume-title":"Quantification in Natural Languages","author":"Bach Elke","year":"2013","unstructured":"Elke Bach, Eloise Jelinek, Angelika Kratzer, and Barbara B. H. Partee. 2013. Quantification in Natural Languages. Vol. 54. Springer Science & Business Media."},{"key":"e_1_3_2_10_2","doi-asserted-by":"crossref","first-page":"241","DOI":"10.1007\/978-94-009-2727-8_10","volume-title":"Philosophy, Language, and Artificial Intelligence","author":"Barwise Jon","year":"1981","unstructured":"Jon Barwise and Robin Cooper. 1981. Generalized quantifiers and natural language. In Philosophy, Language, and Artificial Intelligence. Springer, 241\u2013301."},{"key":"e_1_3_2_11_2","volume-title":"Proceedings of the 6th International Conference on Learning Representations (ICLR\u201918)","author":"Belinkov Yonatan","year":"2018","unstructured":"Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation. In Proceedings of the 6th International Conference on Learning Representations (ICLR\u201918). https:\/\/openreview.net\/forum?id=BJ8vJebC-."},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/1841909.1841911"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/1735886.1735890"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1086\/268494"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/S17-2001"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/2931037.2931073"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/D14-1082"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3385412.3385988"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-1152"},{"key":"e_1_3_2_20_2","unstructured":"Tsong Yueh Chen S. C. Cheung and Siu-Ming Yiu. 1998. Metamorphic Testing: A New Approach for Generating Next Test Cases . Technical Report HKUST-CS98-01. Department of Computer Science Hong Kong University of Science and Technology Hong Kong."},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/3143561"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.173"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0019-9958(58)90082-2"},{"key":"e_1_3_2_24_2","first-page":"63","volume-title":"Proceedings of MT Summit IX","author":"Coughlin Deborah","year":"2003","unstructured":"Deborah Coughlin. 2003. Correlating automated and human assessments of machine translation quality. In Proceedings of MT Summit IX. 63\u201370."},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.tcs.2013.05.040"},{"key":"e_1_3_2_26_2","article-title":"Diarrheal diseases","author":"Dadonaite Bernadeta","year":"2018","unstructured":"Bernadeta Dadonaite and Hannah Ritchie. 2018. Diarrheal diseases. Our World in Data. Retrieved January 25, 2022 from https:\/\/ourworldindata.org\/diarrheal-diseases.","journal-title":"Our World in Data."},{"key":"e_1_3_2_27_2","unstructured":"Gareth Davies. 2017. Palestinian man is arrested by police after posting \u2018good morning\u2019 in Arabic on Facebook which was wrongly translated as \u2018attack them.\u2019 Daily Mail . Retrieved January 25 2022 from https:\/\/www.dailymail.co.uk\/news\/article-5005489\/Good-morning-Facebook-post-leads-arrest-Palestinian.html."},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00048"},{"key":"e_1_3_2_29_2","first-page":"4171","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT\u201919)","author":"Devlin Jacob","year":"2019","unstructured":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT\u201919). 4171\u20134186."},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/D14-1148"},{"key":"e_1_3_2_31_2","volume-title":"Proceedings of the 27th International Conference on Computational Linguistics","author":"Ebrahimi Javid","year":"2018","unstructured":"Javid Ebrahimi, Daniel Lowd, and Dejing Dou. 2018. On adversarial examples for character-level neural machine translation. In Proceedings of the 27th International Conference on Computational Linguistics."},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W19-5401"},{"key":"e_1_3_2_33_2","volume-title":"Database Systems: The Complete Book","author":"Garcia-Molina Hector","year":"2008","unstructured":"Hector Garcia-Molina. 2008. Database Systems: The Complete Book. Pearson Education India."},{"key":"e_1_3_2_34_2","volume-title":"Proceedings of the 3rd International Conference on Learning Representations (ICLR\u201915)","author":"Goodfellow Ian J.","year":"2015","unstructured":"Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and harnessing adversarial examples. In Proceedings of the 3rd International Conference on Learning Representations (ICLR\u201915)."},{"key":"e_1_3_2_35_2","article-title":"Few, a Few\u2013What\u2019s the Difference? Retrieved January 25, 2022 from","unstructured":"Grammarly. n.d. Few, a Few\u2013What\u2019s the Difference? Retrieved January 25, 2022 from https:\/\/www.grammarly.com\/blog\/few-a-few\/.","journal-title":"https:\/\/www.grammarly.com\/blog\/few-a-few\/."},{"key":"e_1_3_2_36_2","first-page":"5133","volume-title":"Proceedings of the 32nd AAAI Conference on Artificial Intelligence (AAAI\u201918), the 30th Conference on Innovative Applications of Artificial Intelligence (IAAI\u201918), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI\u201918)","author":"Gu Jiatao","year":"2018","unstructured":"Jiatao Gu, Yong Wang, Kyunghyun Cho, and Victor O. K. Li. 2018. Search engine guided neural machine translation. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence (AAAI\u201918), the 30th Conference on Innovative Applications of Artificial Intelligence (IAAI\u201918), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI\u201918). 5133\u20135140."},{"key":"e_1_3_2_37_2","first-page":"863","volume-title":"Proceedings of the ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE\u201920)","author":"Gupta Shashij","year":"2020","unstructured":"Shashij Gupta, Pinjia He, Clara Meister, and Zhendong Su. 2020. Machine translation testing via pathological invariance. In Proceedings of the ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE\u201920). 863\u2013875."},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-srw.35"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/HPCA.2018.00059"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380339"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1007\/BF01064028"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.5555\/1454320"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1093\/ijpor\/15.1.83"},{"key":"e_1_3_2_44_2","article-title":"Q: What is the difference between \u201cfew\u201d and \u201ca few\u201d? Editage Insights. Retrieved January 25, 2022 from","author":"Insights Editage","year":"2019","unstructured":"Editage Insights. 2019. Q: What is the difference between \u201cfew\u201d and \u201ca few\u201d? Editage Insights. Retrieved January 25, 2022 from https:\/\/www.editage.com\/insights\/few-and-a-few-what-is-the-difference.","journal-title":"https:\/\/www.editage.com\/insights\/few-and-a-few-what-is-the-difference."},{"key":"e_1_3_2_45_2","first-page":"79","volume-title":"Proceedings of the MT Summit","volume":"5","author":"Koehn Philipp","year":"2005","unstructured":"Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of the MT Summit, Vol. 5. 79\u201386."},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.3115\/1654650.1654666"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W19-5438"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1016\/0010-0277(93)90042-T"},{"key":"e_1_3_2_49_2","first-page":"826","volume-title":"Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Kushman Nate","year":"2013","unstructured":"Nate Kushman and Regina Barzilay. 2013. Using semantic unification to generate regular expressions from natural language. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 826\u2013836."},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1023\/A:1005638918877"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1512"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W19-5358"},{"key":"e_1_3_2_53_2","first-page":"53","article-title":"Quantification as a major module of natural language semantics","volume":"8","author":"L\u00f6bner Sebastian","year":"1987","unstructured":"Sebastian L\u00f6bner. 1987. Quantification as a major module of natural language semantics. Studies in Discourse Representation Theory and the Theory of Generalized Quantifiers 8 (1987), 53.","journal-title":"Studies in Discourse Representation Theory and the Theory of Generalized Quantifiers"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D16-1197"},{"key":"e_1_3_2_55_2","article-title":"Housing Study","author":"Ltd Fordham Research","unstructured":"Fordham Research Ltd. 2005. Housing Study. Retrieved January 25, 2022 from https:\/\/www.ipswich.gov.uk\/sites\/default\/files\/Housing_Needs_Study.pdf.","journal-title":"https:\/\/www.ipswich.gov.uk\/sites\/default\/files\/Housing_Needs_Study.pdf."},{"key":"e_1_3_2_56_2","unstructured":"Fiona Macdonald. 2015. The greatest mistranslations ever. BBC . Retrieved January 25 2022 from https:\/\/www.bbc.com\/culture\/article\/20150202-the-greatest-mistranslations-ever."},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1269"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.448"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1080\/713659792"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.4314\/sajpem.v25i2.31440"},{"key":"e_1_3_2_61_2","first-page":"91","volume-title":"Proceedings of the 22nd Annual Annual Conference of the European Association for Machine Translation (EAMT\u201920)","author":"Moon Jihyung","year":"2020","unstructured":"Jihyung Moon, Hyunchang Cho, and Eunjeong L. Park. 2020. Revisiting round-trip translation for quality estimation. In Proceedings of the 22nd Annual Annual Conference of the European Association for Machine Translation (EAMT\u201920). 91\u2013104."},{"key":"e_1_3_2_62_2","unstructured":"Arika Okrent. 2016. 9 little translation mistakes that caused big problems. Mental Floss . Retrieved January 25 2022 from http:\/\/mentalfloss.com\/article\/48795\/9-little-translation-mistakes-caused-big-problems."},{"key":"e_1_3_2_63_2","unstructured":"Thuy Ong. 2017. Facebook apologizes after wrong translation sees Palestinian man arrested for posting \u2018good morning.\u2019 The Verge . Retrieved January 25 2022 from https:\/\/www.theverge.com\/us-world\/2017\/10\/24\/16533496\/facebook-apology-wrong-translation-palestinian-arrested-post-good-morning."},{"key":"e_1_3_2_64_2","first-page":"311","volume-title":"Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL\u201902)","author":"Papineni Kishore","year":"2002","unstructured":"Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL\u201902). 311\u2013318."},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-1677"},{"key":"e_1_3_2_66_2","volume-title":"Mathematical Methods in Linguistics","author":"Partee Barbara B. H.","year":"1990","unstructured":"Barbara B. H. Partee, A. G. ter Meulen, and R. Wall. 1990. Mathematical Methods in Linguistics. Vol. 30. Springer Science & Business Media."},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","DOI":"10.1145\/3193977.3193980"},{"key":"e_1_3_2_68_2","volume-title":"Quantifiers in Language and Logic","author":"Westerstahl Dag","year":"2006","unstructured":"Stanley Peters and Dag Westerstahl. 2006. Quantifiers in Language and Logic. Oxford University Press."},{"key":"e_1_3_2_69_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-2019"},{"key":"e_1_3_2_70_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W15-3049"},{"key":"e_1_3_2_71_2","unstructured":"Christopher Potts. 2007. Logic for Linguists. Course for LSA Institute."},{"key":"e_1_3_2_72_2","volume-title":"Policy Brief: The Impact of COVID-19 on Children","author":"Programm United Nations World Food","year":"2020","unstructured":"United Nations World Food Programm. 2020. Policy Brief: The Impact of COVID-19 on Children. Technical Report. United Nations World Food Programm. https:\/\/www.wfp.org\/publications\/policy-brief-impact-covid-19-children."},{"key":"e_1_3_2_73_2","doi-asserted-by":"publisher","DOI":"10.3115\/1611533.1611541"},{"key":"e_1_3_2_74_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-1410"},{"key":"e_1_3_2_75_2","first-page":"287","volume-title":"Proceedings of the 14th International Conference on Machine Learning (ICML\u201997)","author":"Ristad Eric Sven","year":"1997","unstructured":"Eric Sven Ristad and Peter N. Yianilos. 1997. Learning string edit distance. In Proceedings of the 14th International Conference on Machine Learning (ICML\u201997). 287\u2013295."},{"key":"e_1_3_2_76_2","doi-asserted-by":"publisher","DOI":"10.1002\/j.1538-7305.1948.tb01338.x"},{"key":"e_1_3_2_77_2","doi-asserted-by":"publisher","DOI":"10.2307\/2184738"},{"key":"e_1_3_2_78_2","doi-asserted-by":"crossref","unstructured":"Tomohiro Shigenobu. 2007. Evaluation and usability of back translation for intercultural communication. In Usability and Internationalization: Global and Local User Interfaces . Lecture Notes in Computer Science Vol. 4560. Springer 259\u2013265.","DOI":"10.1007\/978-3-540-73289-1_31"},{"key":"e_1_3_2_79_2","volume-title":"Proceedings of the Association for Machine Translation in the Americas","volume":"200","author":"Snover Matthew","year":"2006","unstructured":"Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of the Association for Machine Translation in the Americas, Vol. 200."},{"key":"e_1_3_2_80_2","first-page":"157","article-title":"Vagueness in quantity: Two case studies from a linguistic perspective","author":"Solt Stephanie","year":"2011","unstructured":"Stephanie Solt. 2011. Vagueness in quantity: Two case studies from a linguistic perspective. In Understanding Vagueness: Logical, Philosophical and Linguistic Perspectives, P. Cintula, C. G. Fermueller, and L. Godo (Eds.). Studies in Logic. College Publications, 157\u2013174.","journal-title":"Understanding Vagueness: Logical, Philosophical and Linguistic Perspectives, P. Cintula, C. G. Fermueller, and L. Godo (Eds.). Studies in Logic."},{"key":"e_1_3_2_81_2","first-page":"127","volume-title":"Proceedings of the Australasian Language Technology Workshop (ALTA\u201905)","author":"Somers Harold L.","year":"2005","unstructured":"Harold L. Somers. 2005. Round-trip translation: What is it good for? In Proceedings of the Australasian Language Technology Workshop (ALTA\u201905). 127\u2013133."},{"key":"e_1_3_2_82_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-1029"},{"issue":"10","key":"e_1_3_2_83_2","first-page":"1951","article-title":"Concepts and semantic relations in information science","volume":"61","author":"Stock Wolfgang G.","year":"2010","unstructured":"Wolfgang G. Stock. 2010. Concepts and semantic relations in information science. Journal of the Association for Information Science and Technology 61, 10 (2010), 1951\u20131969.","journal-title":"Journal of the Association for Information Science and Technology"},{"key":"e_1_3_2_84_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380420"},{"key":"e_1_3_2_85_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-28749-2"},{"key":"e_1_3_2_86_2","first-page":"2214","volume-title":"Proceedings of the Eighth International Conference on Language Resources and Evaluation, LREC 2012, Istanbul, Turkey, May 23\u201325, 2012","author":"Tiedemann J\u00f6rg","year":"2012","unstructured":"J\u00f6rg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and Evaluation, LREC 2012, Istanbul, Turkey, May 23\u201325, 2012, Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Mehmet Ugur Dogan, Bente Maegaard, Joseph Mariani, Jan Odijk, and Stelios Piperidis (Eds.). European Language Resources Association, 2214\u20132218."},{"key":"e_1_3_2_87_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.americasnlp-1.29"},{"key":"e_1_3_2_88_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2010.15"},{"key":"e_1_3_2_89_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1116"},{"key":"e_1_3_2_90_2","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3236072"},{"key":"e_1_3_2_91_2","doi-asserted-by":"publisher","DOI":"10.1109\/DSN-Industry.2019.00007"},{"key":"e_1_3_2_92_2","doi-asserted-by":"publisher","DOI":"10.1002\/acp.2350080506"},{"key":"e_1_3_2_93_2","doi-asserted-by":"publisher","DOI":"10.1145\/1180875.1180955"},{"key":"e_1_3_2_94_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N18-1120"},{"key":"e_1_3_2_95_2","unstructured":"Zhengli Zhao Dheeru Dua and Sameer Singh. 2018. Generating natural adversarial examples. In Proceedings of the 6th International Conference on Learning Representations (ICLR\u201918) . https:\/\/openreview.net\/forum?id=H1BLjgZCb."},{"key":"e_1_3_2_96_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1189"},{"key":"e_1_3_2_97_2","series-title":"Proceedings of the Workshops of the 32nd AAAI Conference on Artificial Intelligence","first-page":"791","volume":"18","author":"Zhong Zexuan","year":"2018","unstructured":"Zexuan Zhong, Jiaqi Guo, Wei Yang, Tao Xie, Jian-Guang Lou, Ting Liu, and Dongmei Zhang. 2018. Generating regular expressions from natural language specifications: Are we there yet? In Proceedings of the Workshops of the 32nd AAAI Conference on Artificial Intelligence(AAAI Workshops, Vol. WS-18). 791\u2013794."},{"key":"e_1_3_2_98_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2015.2478001"},{"key":"e_1_3_2_99_2","first-page":"96","volume-title":"Proceedings of the 25th Australasian Software Engineering Conference (ASWEC\u201918)","author":"Zhou Zhi Quan","year":"2018","unstructured":"Zhi Quan Zhou and Liqun Sun. 2018. Metamorphic testing for machine translations: MT4MT. In Proceedings of the 25th Australasian Software Engineering Conference (ASWEC\u201918). IEEE, Los Alamitos, CA, 96\u2013100."}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3490488","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3490488","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T15:03:00Z","timestamp":1750172580000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3490488"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,4]]},"references-count":98,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2022,4,30]]}},"alternative-id":["10.1145\/3490488"],"URL":"https:\/\/doi.org\/10.1145\/3490488","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,4]]},"assertion":[{"value":"2020-12-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-09-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-04-01","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}