{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,17]],"date-time":"2026-03-17T00:53:26Z","timestamp":1773708806541,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":91,"publisher":"ACM","license":[{"start":{"date-parts":[[2020,11,8]],"date-time":"2020-11-08T00:00:00Z","timestamp":1604793600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2020,11,8]]},"DOI":"10.1145\/3368089.3409756","type":"proceedings-article","created":{"date-parts":[[2020,11,8]],"date-time":"2020-11-08T06:03:47Z","timestamp":1604815427000},"page":"863-875","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":33,"title":["Machine translation testing via pathological invariance"],"prefix":"10.1145","author":[{"given":"Shashij","family":"Gupta","sequence":"first","affiliation":[{"name":"IIT Bombay, India"}]},{"given":"Pinjia","family":"He","sequence":"additional","affiliation":[{"name":"ETH Zurich, Switzerland"}]},{"given":"Clara","family":"Meister","sequence":"additional","affiliation":[{"name":"ETH Zurich, Switzerland"}]},{"given":"Zhendong","family":"Su","sequence":"additional","affiliation":[{"name":"ETH Zurich, Switzerland"}]}],"member":"320","published-online":{"date-parts":[[2020,11,8]]},"reference":[{"key":"e_1_3_2_2_1_1","unstructured":"[n.d.]. fairseq: A Fast Extensible Toolkit for Sequence Modeling. https:\/\/github. com\/\/pytorch\/fairseq  [n.d.]. fairseq: A Fast Extensible Toolkit for Sequence Modeling. https:\/\/github. com\/\/pytorch\/fairseq"},{"key":"e_1_3_2_2_2_1","unstructured":"[n.d.]. Google Translate. https:\/\/translate.google.com  [n.d.]. Google Translate. https:\/\/translate.google.com"},{"key":"e_1_3_2_2_3_1","unstructured":"[n.d.]. Thesaurus. https:\/\/www.thesaurus.com\/  [n.d.]. Thesaurus. https:\/\/www.thesaurus.com\/"},{"key":"e_1_3_2_2_4_1","unstructured":"[n.d.]. WordsAPI. https:\/\/www.wordsapi.com\/  [n.d.]. WordsAPI. https:\/\/www.wordsapi.com\/"},{"key":"e_1_3_2_2_5_1","unstructured":"2018. 15 000 Eggs Delivered to Norwegian Olympic Team After Google Translate Error. https:\/\/www.nbcwashington.com\/news\/national-international\/ googletranslate-fail-norway-olympic-team-gets-15k-eggs-delivered\/2034392\/  2018. 15 000 Eggs Delivered to Norwegian Olympic Team After Google Translate Error. https:\/\/www.nbcwashington.com\/news\/national-international\/ googletranslate-fail-norway-olympic-team-gets-15k-eggs-delivered\/2034392\/"},{"key":"e_1_3_2_2_6_1","unstructured":"2018. Greedy Britle Opaque and Shallow: The Downsides to Deep Learning. https:\/\/www.wired.com\/story\/greedy-brittle-opaque-and-shallow-thedownsides-to-deep-learning\/  2018. Greedy Britle Opaque and Shallow: The Downsides to Deep Learning. https:\/\/www.wired.com\/story\/greedy-brittle-opaque-and-shallow-thedownsides-to-deep-learning\/"},{"key":"e_1_3_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1316"},{"key":"e_1_3_2_2_8_1","volume-title":"Proceedings of the 35th International Conference on Machine Learning (ICML).","author":"Athalye Anish","year":"2018","unstructured":"Anish Athalye , Nicholas Carlini , and David Wagner . 2018 . Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples . In Proceedings of the 35th International Conference on Machine Learning (ICML). Anish Athalye, Nicholas Carlini, and David Wagner. 2018. Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples. In Proceedings of the 35th International Conference on Machine Learning (ICML)."},{"key":"e_1_3_2_2_9_1","volume-title":"Proceedings of the 6th International Conference on Learning Representations (ICLR).","author":"Belinkov Yonatan","year":"2018","unstructured":"Yonatan Belinkov and Yonatan Bisk . 2018 . Synthetic and Natural Noise Both Break Neural Machine Translation . In Proceedings of the 6th International Conference on Learning Representations (ICLR). Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and Natural Noise Both Break Neural Machine Translation. In Proceedings of the 6th International Conference on Learning Representations (ICLR)."},{"key":"e_1_3_2_2_10_1","volume-title":"Steven and Ewan Klein","author":"Bird Edward Loper","year":"2009","unstructured":"Edward Loper Bird , Steven and Ewan Klein . 2009 . Natural Language Processing with Python. O'Reilly Media Inc . Edward Loper Bird, Steven and Ewan Klein. 2009. Natural Language Processing with Python. O'Reilly Media Inc."},{"key":"e_1_3_2_2_11_1","first-page":"272 18","volume-title":"Findings of the 2018 Conference on Machine Translation (WMT18). In Proceedings of the Third Conference on Machine Translation","volume":"2","author":"Bojar Ond","year":"2018","unstructured":"Ond rej Bojar , Christian Federmann , Mark Fishel , Yvete Graham , Barry Haddow , Mathias Huck , Philipp Koehn , and Christof Monz . 2018 . Findings of the 2018 Conference on Machine Translation (WMT18). In Proceedings of the Third Conference on Machine Translation , Volume 2 : Shared Task Papers. Association for Computational Linguistics, Belgium, Brussels , 272 - 307 . http:\/\/www.aclweb. org\/anthology\/W 18 - 6401 Ond rej Bojar, Christian Federmann, Mark Fishel, Yvete Graham, Barry Haddow, Mathias Huck, Philipp Koehn, and Christof Monz. 2018. Findings of the 2018 Conference on Machine Translation (WMT18). In Proceedings of the Third Conference on Machine Translation, Volume 2 : Shared Task Papers. Association for Computational Linguistics, Belgium, Brussels, 272-307. http:\/\/www.aclweb. org\/anthology\/W18-6401"},{"key":"e_1_3_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.5555\/3241094.3241135"},{"key":"e_1_3_2_2_13_1","volume-title":"Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al.","author":"Cer Daniel","year":"2018","unstructured":"Daniel Cer , Yinfei Yang , Sheng-yi Kong, Nan Hua , Nicole Limtiaco , Rhomni St John , Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018 . Universal sentence encoder. arXiv preprint arXiv: 1803. 11175 ( 2018 ). Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder. arXiv preprint arXiv: 1803. 11175 ( 2018 )."},{"key":"e_1_3_2_2_14_1","volume-title":"Exploring the Robustness of NMT Systems to Nonsensical Inputs. arXiv preprint arXiv","author":"Chaturvedi Akshay","year":"1908","unstructured":"Akshay Chaturvedi , Abijith KP, and Utpal Garain . 2019. Exploring the Robustness of NMT Systems to Nonsensical Inputs. arXiv preprint arXiv : 1908 . 01165 ( 2019 ). Akshay Chaturvedi, Abijith KP, and Utpal Garain. 2019. Exploring the Robustness of NMT Systems to Nonsensical Inputs. arXiv preprint arXiv: 1908. 01165 ( 2019 )."},{"key":"e_1_3_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3143561"},{"key":"e_1_3_2_2_17_1","doi-asserted-by":"crossref","unstructured":"N. Chomsky. 1957. Syntactic Structures. Mouton The Hague.  N. Chomsky. 1957. Syntactic Structures. Mouton The Hague.","DOI":"10.1515\/9783112316009"},{"key":"e_1_3_2_2_18_1","volume-title":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL).","author":"Chu Chenhui","year":"2017","unstructured":"Chenhui Chu , Raj Dabre , and Sadao Kurohashi . 2017 . An empirical comparison of simple domain adaptation methods for neural machine translation . In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL). Chenhui Chu, Raj Dabre, and Sadao Kurohashi. 2017. An empirical comparison of simple domain adaptation methods for neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL)."},{"key":"e_1_3_2_2_19_1","unstructured":"Gareth Davies. 2017. Palestinian man is arrested by police after posting 'Good morning' in Arabic on Facebook which was wrongly translated as 'attack them'. https:\/\/www.dailymail.co.uk\/news\/article-5005489\/ Good-morningFacebook-post-leads-arrest-Palestinian.html  Gareth Davies. 2017. Palestinian man is arrested by police after posting 'Good morning' in Arabic on Facebook which was wrongly translated as 'attack them'. https:\/\/www.dailymail.co.uk\/news\/article-5005489\/ Good-morningFacebook-post-leads-arrest-Palestinian.html"},{"key":"e_1_3_2_2_20_1","volume-title":"Bert: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv","author":"Devlin Jacob","year":"2018","unstructured":"Jacob Devlin , Ming-Wei Chang , Kenton Lee , and Kristina Toutanova . 2018 . Bert: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv : 1810. 04805 ( 2018 ). Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv: 1810. 04805 ( 2018 )."},{"key":"e_1_3_2_2_21_1","volume-title":"Benchmarking Adversarial Robustness. arXiv preprint arXiv","author":"Dong Yinpeng","year":"1912","unstructured":"Yinpeng Dong , Qi-An Fu , Xiao Yang , Tianyu Pang , Hang Su , Zihao Xiao , and Jun Zhu . 2019. Benchmarking Adversarial Robustness. arXiv preprint arXiv : 1912 . 11852 ( 2019 ). Yinpeng Dong, Qi-An Fu, Xiao Yang, Tianyu Pang, Hang Su, Zihao Xiao, and Jun Zhu. 2019. Benchmarking Adversarial Robustness. arXiv preprint arXiv: 1912. 11852 ( 2019 )."},{"key":"e_1_3_2_2_22_1","volume-title":"SirenAtack: Generating Adversarial Audio for End-to-End Acoustic Systems. arXiv preprint arXiv","author":"Du Tianyu","year":"1901","unstructured":"Tianyu Du , Shouling Ji , Jinfeng Li , Qinchen Gu , Ting Wang , and Raheem Beyah . 2019. SirenAtack: Generating Adversarial Audio for End-to-End Acoustic Systems. arXiv preprint arXiv : 1901 . 07846 ( 2019 ). Tianyu Du, Shouling Ji, Jinfeng Li, Qinchen Gu, Ting Wang, and Raheem Beyah. 2019. SirenAtack: Generating Adversarial Audio for End-to-End Acoustic Systems. arXiv preprint arXiv: 1901. 07846 ( 2019 )."},{"key":"e_1_3_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338954"},{"key":"e_1_3_2_2_24_1","volume-title":"Proceedings of the 27th International Conference on Computational Linguistics (COLING).","author":"Ebrahimi Javid","year":"2018","unstructured":"Javid Ebrahimi , Daniel Lowd , and Dejing Dou . 2018 . On Adversarial Examples for Character-Level Neural Machine Translation . In Proceedings of the 27th International Conference on Computational Linguistics (COLING). Javid Ebrahimi, Daniel Lowd, and Dejing Dou. 2018. On Adversarial Examples for Character-Level Neural Machine Translation. In Proceedings of the 27th International Conference on Computational Linguistics (COLING)."},{"key":"e_1_3_2_2_25_1","unstructured":"Hugging Face. [n.d.]. Transformers: State-of-the-art Natural Language Processing for TensorFlow 2.0 and PyTorch. https:\/\/github.com\/huggingface\/ transformers  Hugging Face. [n.d.]. Transformers: State-of-the-art Natural Language Processing for TensorFlow 2.0 and PyTorch. https:\/\/github.com\/huggingface\/ transformers"},{"key":"e_1_3_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3293882.3330566"},{"key":"e_1_3_2_2_27_1","volume-title":"Proceedings of the 3rd International Conference on Learning Representations (ICLR).","author":"Goodfellow Ian J","year":"2015","unstructured":"Ian J Goodfellow , Jonathon Shlens , and Christian Szegedy . 2015 . Explaining and Harnessing Adversarial Examples . Proceedings of the 3rd International Conference on Learning Representations (ICLR). Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and Harnessing Adversarial Examples. Proceedings of the 3rd International Conference on Learning Representations (ICLR)."},{"key":"e_1_3_2_2_28_1","unstructured":"Stanford NLP Group. [n.d.]. Stanford CoreNLP-Natural language software. https:\/\/stanfordnlp.github.io\/CoreNLP\/  Stanford NLP Group. [n.d.]. Stanford CoreNLP-Natural language software. https:\/\/stanfordnlp.github.io\/CoreNLP\/"},{"key":"e_1_3_2_2_29_1","unstructured":"Hany Hassan Anthony Aue Chang Chen Vishal Chowdhary Jonathan Clark Christian Federmann Xuedong Huang Marcin Junczys-Dowmunt William Lewis Mu Li etal 2018. Achieving Human Parity on Automatic Chinese to English News Translation. arXiv preprint arXiv: 1803. 05567 ( 2018 ).  Hany Hassan Anthony Aue Chang Chen Vishal Chowdhary Jonathan Clark Christian Federmann Xuedong Huang Marcin Junczys-Dowmunt William Lewis Mu Li et al. 2018. Achieving Human Parity on Automatic Chinese to English News Translation. arXiv preprint arXiv: 1803. 05567 ( 2018 )."},{"key":"e_1_3_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380339"},{"key":"e_1_3_2_2_31_1","volume-title":"Towards Structured Evaluation of Deep Neural Network Supervisors. In 2019 IEEE International Conference On Artificial Intelligence Testing (AITest).","author":"Henriksson J.","unstructured":"J. Henriksson , C. Berger , M. Borg , L. Tornberg , C. Englund , S. R. Sathyamoorthy , and S. Ursing . 2019 . Towards Structured Evaluation of Deep Neural Network Supervisors. In 2019 IEEE International Conference On Artificial Intelligence Testing (AITest). J. Henriksson, C. Berger, M. Borg, L. Tornberg, C. Englund, S. R. Sathyamoorthy, and S. Ursing. 2019. Towards Structured Evaluation of Deep Neural Network Supervisors. In 2019 IEEE International Conference On Artificial Intelligence Testing (AITest)."},{"key":"e_1_3_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N18-1170"},{"key":"e_1_3_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N18-1170"},{"key":"e_1_3_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D17-1215"},{"key":"e_1_3_2_2_35_1","volume-title":"Adversarial Logit Pairing. arXiv preprint arXiv","author":"Kannan Harini","year":"1803","unstructured":"Harini Kannan , Alexey Kurakin , and Ian Goodfellow . 2018. Adversarial Logit Pairing. arXiv preprint arXiv : 1803 . 06373 ( 2018 ). Harini Kannan, Alexey Kurakin, and Ian Goodfellow. 2018. Adversarial Logit Pairing. arXiv preprint arXiv: 1803. 06373 ( 2018 )."},{"key":"e_1_3_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00108"},{"key":"e_1_3_2_2_37_1","unstructured":"Fred. Lambert. 2016. Understanding the fatal Tesla accident on Autopilot and the NHTSA probe. https:\/\/electrek.co\/ 2016 \/07\/01\/understanding-fatal-teslaaccident-autopilot-nhtsa-probe\/  Fred. Lambert. 2016. Understanding the fatal Tesla accident on Autopilot and the NHTSA probe. https:\/\/electrek.co\/ 2016 \/07\/01\/understanding-fatal-teslaaccident-autopilot-nhtsa-probe\/"},{"key":"e_1_3_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/2666356.2594334"},{"key":"e_1_3_2_2_39_1","unstructured":"Sam Levin. 2018. Tesla fatal crash: 'autopilot' mode sped up car before driver killed report finds. https:\/\/www.theguardian.com\/technology\/2018\/jun\/ 07\/tesla-fatal-crash-silicon-valley-autopilot-mode-report  Sam Levin. 2018. Tesla fatal crash: 'autopilot' mode sped up car before driver killed report finds. https:\/\/www.theguardian.com\/technology\/2018\/jun\/ 07\/tesla-fatal-crash-silicon-valley-autopilot-mode-report"},{"key":"e_1_3_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.14722\/ndss.2019.23138"},{"key":"e_1_3_2_2_41_1","volume-title":"Many-Core Compiler Fuzzing. In ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI).","author":"Lidbury Christopher","unstructured":"Christopher Lidbury , Andrei Lascu , Nathan Chong , and Alastair F. Donaldson . 2015 . Many-Core Compiler Fuzzing. In ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI). Christopher Lidbury, Andrei Lascu, Nathan Chong, and Alastair F. Donaldson. 2015. Many-Core Compiler Fuzzing. In ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI)."},{"key":"e_1_3_2_2_42_1","volume-title":"Proceedings of the 7th International Conference on Learning Representations (ICLR).","author":"Lin Ji","year":"2019","unstructured":"Ji Lin , Chuang Gan , and Song Han . 2019 . Defensive Quantization: When Eficiency Meets Robustness . In Proceedings of the 7th International Conference on Learning Representations (ICLR). Ji Lin, Chuang Gan, and Song Han. 2019. Defensive Quantization: When Eficiency Meets Robustness. In Proceedings of the 7th International Conference on Learning Representations (ICLR)."},{"key":"e_1_3_2_2_43_1","volume-title":"Proceedings of the 37th International Conference on Software Engineering (ICSE).","author":"Lindvall Mikael","unstructured":"Mikael Lindvall , Dharmalingam Ganesan , Ragnar \u00c1rdal , and Robert E. Wiegand . 2015. Metamorphic Model-based Testing Applied on NASA DAT-an experience report . In Proceedings of the 37th International Conference on Software Engineering (ICSE). Mikael Lindvall, Dharmalingam Ganesan, Ragnar \u00c1rdal, and Robert E. Wiegand. 2015. Metamorphic Model-based Testing Applied on NASA DAT-an experience report. In Proceedings of the 37th International Conference on Software Engineering (ICSE)."},{"key":"e_1_3_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238202"},{"key":"e_1_3_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3236082"},{"key":"e_1_3_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.14722\/ndss.2019.23415"},{"key":"e_1_3_2_2_47_1","unstructured":"Fiona Macdonald. 2015. The Greatest Mistranslations Ever. http:\/\/www.bbc. com\/culture\/story\/20150202-the-greatest-mistranslations-ever  Fiona Macdonald. 2015. The Greatest Mistranslations Ever. http:\/\/www.bbc. com\/culture\/story\/20150202-the-greatest-mistranslations-ever"},{"key":"e_1_3_2_2_48_1","volume-title":"Proceedings of the 6th International Conference on Learning Representations (ICLR).","author":"Madry Aleksander","year":"2018","unstructured":"Aleksander Madry , Aleksandar Makelov , Ludwig Schmidt , Dimitris Tsipras , and Adrian Vladu . 2018 . Towards Deep Learning Models Resistant to Adversarial Atacks . In Proceedings of the 6th International Conference on Learning Representations (ICLR). Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2018. Towards Deep Learning Models Resistant to Adversarial Atacks. In Proceedings of the 6th International Conference on Learning Representations (ICLR)."},{"key":"e_1_3_2_2_49_1","volume-title":"The Stanford CoreNLP Natural Language Processing Toolkit","author":"Manning Christopher D.","unstructured":"Christopher D. Manning , Mihai Surdeanu , John Bauer , Jenny Finkel , Steven J. Bethard , and David McClosky . 2014. The Stanford CoreNLP Natural Language Processing Toolkit . In Association for Computational Linguistics (ACL) System Demonstrations . Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP Natural Language Processing Toolkit. In Association for Computational Linguistics (ACL) System Demonstrations."},{"key":"e_1_3_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N19-1314"},{"key":"e_1_3_2_2_51_1","unstructured":"Tomas Mikolov Kai Chen Greg Corrado and Jefrey Dean. 2013. Eficient Estimation of Word Representations in Vector Space. arXiv e-prints ( 2013 ).  Tomas Mikolov Kai Chen Greg Corrado and Jefrey Dean. 2013. Eficient Estimation of Word Representations in Vector Space. arXiv e-prints ( 2013 )."},{"key":"e_1_3_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-1176"},{"key":"e_1_3_2_2_53_1","volume-title":"Proceedings of the 20th International Conference on Software Engineering and Knowledge Engineering (SEKE).","author":"Murphy Christian","year":"2008","unstructured":"Christian Murphy , Gail E. Kaiser , Lifeng Hu , and Leon Wu . 2008 . Properties of Machine Learning Applications for Use in Metamorphic Testing . In Proceedings of the 20th International Conference on Software Engineering and Knowledge Engineering (SEKE). Christian Murphy, Gail E. Kaiser, Lifeng Hu, and Leon Wu. 2008. Properties of Machine Learning Applications for Use in Metamorphic Testing. In Proceedings of the 20th International Conference on Software Engineering and Knowledge Engineering (SEKE)."},{"key":"e_1_3_2_2_54_1","unstructured":"Arika Okrent. 2016. 9 Litle Translation Mistakes That Caused Big Problems. http:\/\/mentalfloss.com\/article\/48795\/9-little-translation-mistakescaused-big-problems  Arika Okrent. 2016. 9 Litle Translation Mistakes That Caused Big Problems. http:\/\/mentalfloss.com\/article\/48795\/9-little-translation-mistakescaused-big-problems"},{"key":"e_1_3_2_2_55_1","unstructured":"Thuy Ong. 2017. Facebook apologizes after wrong translation sees Palestinian man arrested for posting 'good morning'. https:\/\/www.theverge.com\/usworld\/2017\/10\/24\/16533496\/facebook-apology-wrong-translation-palestinianarrested-post-good-morning  Thuy Ong. 2017. Facebook apologizes after wrong translation sees Palestinian man arrested for posting 'good morning'. https:\/\/www.theverge.com\/usworld\/2017\/10\/24\/16533496\/facebook-apology-wrong-translation-palestinianarrested-post-good-morning"},{"key":"e_1_3_2_2_56_1","volume-title":"Analyzing Uncertainty in Neural Machine Translation. arXiv:cs.CL\/","author":"Ot Myle","year":"1803","unstructured":"Myle Ot , Michael Auli , David Grangier , and Marc'Aurelio Ranzato . 2018. Analyzing Uncertainty in Neural Machine Translation. arXiv:cs.CL\/ 1803 .00047 Myle Ot, Michael Auli, David Grangier, and Marc'Aurelio Ranzato. 2018. Analyzing Uncertainty in Neural Machine Translation. arXiv:cs.CL\/ 1803.00047"},{"key":"e_1_3_2_2_57_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2016.41"},{"key":"e_1_3_2_2_58_1","volume-title":"Proceedings of the 40th Annual Meeting on Association for Computational Linguistics (ACL).","author":"Papineni Kishore","year":"2002","unstructured":"Kishore Papineni , Salim Roukos , Todd Ward , and Wei-Jing Zhu . 2002 . BLEU: A Method for Automatic Evaluation of Machine Translation . In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics (ACL). Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics (ACL)."},{"key":"e_1_3_2_2_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132747.3132785"},{"key":"e_1_3_2_2_60_1","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/D14-1162"},{"key":"e_1_3_2_2_61_1","doi-asserted-by":"crossref","unstructured":"Mathew E. Peters Mark Neumann Mohit Iyyer Mat Gardner Christopher Clark Kenton Lee and Luke Zetlemoyer. 2018. Deep contextualized word representations. arXiv e-prints ( 2018 ).  Mathew E. Peters Mark Neumann Mohit Iyyer Mat Gardner Christopher Clark Kenton Lee and Luke Zetlemoyer. 2018. Deep contextualized word representations. arXiv e-prints ( 2018 ).","DOI":"10.18653\/v1\/N18-1202"},{"key":"e_1_3_2_2_62_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00107"},{"key":"e_1_3_2_2_63_1","volume-title":"Proc. of the 57th Annual Meeting of the Association for Computational Linguistics (ACL).","author":"Pruthi Danish","unstructured":"Danish Pruthi , Bhuwan Dhingra , and Zachary C. Lipton . 2019. Combating Adversarial Misspellings with Robust Word Recognition . In Proc. of the 57th Annual Meeting of the Association for Computational Linguistics (ACL). Danish Pruthi, Bhuwan Dhingra, and Zachary C. Lipton. 2019. Combating Adversarial Misspellings with Robust Word Recognition. In Proc. of the 57th Annual Meeting of the Association for Computational Linguistics (ACL)."},{"key":"e_1_3_2_2_64_1","unstructured":"Alec Radford. 2018. Improving Language Understanding by Generative PreTraining.  Alec Radford. 2018. Improving Language Understanding by Generative PreTraining."},{"key":"e_1_3_2_2_65_1","unstructured":"RobustNLP. 2020. A toolkit for testing machine translation. https:\/\/github.com\/ RobustNLP\/TestTranslation  RobustNLP. 2020. A toolkit for testing machine translation. https:\/\/github.com\/ RobustNLP\/TestTranslation"},{"key":"e_1_3_2_2_66_1","unstructured":"Benny Royston. 2018. Israel Eurovision winner Neta called 'a real cow'by Prime Minister in auto-translate fail. https:\/\/metro.co.uk\/ 2018 \/05\/13\/israeleurovision-winner-netta-called-a-real-cow-by-prime-minister-in-autotranslate-fail-7541925\/  Benny Royston. 2018. Israel Eurovision winner Neta called 'a real cow'by Prime Minister in auto-translate fail. https:\/\/metro.co.uk\/ 2018 \/05\/13\/israeleurovision-winner-netta-called-a-real-cow-by-prime-minister-in-autotranslate-fail-7541925\/"},{"key":"e_1_3_2_2_67_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2016.2532875"},{"key":"e_1_3_2_2_68_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P16-1009"},{"key":"e_1_3_2_2_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380420"},{"key":"e_1_3_2_2_70_1","volume-title":"Proceedings of the 34th Conference on Neural Information Processing Systems (NeurIPS).","author":"Tao Guanhong","year":"2018","unstructured":"Guanhong Tao , Shiqing Ma , Yingqi Liu , and Xiangyu Zhang . 2018 . Atacks Meet Interpretability: Atribute-steered Detection of Adversarial Samples . In Proceedings of the 34th Conference on Neural Information Processing Systems (NeurIPS). Guanhong Tao, Shiqing Ma, Yingqi Liu, and Xiangyu Zhang. 2018. Atacks Meet Interpretability: Atribute-steered Detection of Adversarial Samples. In Proceedings of the 34th Conference on Neural Information Processing Systems (NeurIPS)."},{"key":"e_1_3_2_2_71_1","volume":"30","author":"Taylor Wilson L.","year":"1953","unstructured":"Wilson L. Taylor . 1953 . \u201dCloze Procedure\u201d : A New Tool for Measuring Readability. Journalism Bulletin 30 , 4 ( 1953 ), 415-433. Wilson L. Taylor. 1953. \u201dCloze Procedure\u201d : A New Tool for Measuring Readability. Journalism Bulletin 30, 4 ( 1953 ), 415-433.","journal-title":"Journalism Bulletin"},{"key":"e_1_3_2_2_72_1","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180220"},{"key":"e_1_3_2_2_73_1","unstructured":"Barak Turovsky. 2016. Ten years of Google Translate. https:\/\/blog.google\/ products\/translate\/ten-years-of-google-translate\/  Barak Turovsky. 2016. Ten years of Google Translate. https:\/\/blog.google\/ products\/translate\/ten-years-of-google-translate\/"},{"key":"e_1_3_2_2_74_1","volume-title":"Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS).","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani , Noam Shazeer , Niki Parmar , Jakob Uszkoreit , Llion Jones , Aidan N. Gomez , and Illia Kaiser , Lukasz abd Polosukhin . 2017 . Atention is All you Need . In Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS). Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, and Illia Kaiser, Lukasz abd Polosukhin. 2017. Atention is All you Need. In Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS)."},{"key":"e_1_3_2_2_75_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00126"},{"key":"e_1_3_2_2_76_1","doi-asserted-by":"publisher","DOI":"10.1109\/DSN-Industry.2019.00007"},{"key":"e_1_3_2_2_77_1","unstructured":"Yonghui Wu Mike Schuster Zhifeng Chen Quoc V Le Mohammad Norouzi Wolfgang Macherey Maxim Krikun Yuan Cao Qin Gao Klaus Macherey etal 2016. Google's Neural Machine Translation System: Bridging the Gap Between Human and Machine Translation. arXiv preprint arXiv:1609.08144 ( 2016 ).  Yonghui Wu Mike Schuster Zhifeng Chen Quoc V Le Mohammad Norouzi Wolfgang Macherey Maxim Krikun Yuan Cao Qin Gao Klaus Macherey et al. 2016. Google's Neural Machine Translation System: Bridging the Gap Between Human and Machine Translation. arXiv preprint arXiv:1609.08144 ( 2016 )."},{"key":"e_1_3_2_2_78_1","volume-title":"Christian Murphy, Gail Kaiser, Baowen Xu, and Tsong Yueh Chen.","author":"Xie Xiaoyuan","year":"2011","unstructured":"Xiaoyuan Xie , Joshua WK Ho , Christian Murphy, Gail Kaiser, Baowen Xu, and Tsong Yueh Chen. 2011 . Testing and Validating Machine Learning Classifiers by Metamorphic Testing. Journal of Systems and Software (JSS) 84 ( 2011 ). Issue 4. Xiaoyuan Xie, Joshua WK Ho, Christian Murphy, Gail Kaiser, Baowen Xu, and Tsong Yueh Chen. 2011. Testing and Validating Machine Learning Classifiers by Metamorphic Testing. Journal of Systems and Software (JSS) 84 ( 2011 ). Issue 4."},{"key":"e_1_3_2_2_79_1","doi-asserted-by":"publisher","DOI":"10.1145\/3293882.3330579"},{"key":"e_1_3_2_2_80_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00935"},{"key":"e_1_3_2_2_81_1","doi-asserted-by":"publisher","DOI":"10.14722\/ndss.2018.23198"},{"key":"e_1_3_2_2_82_1","volume-title":"Proceedings of the 2019 IEEE Conference on Computer Vision and Patern Recognition (CVPR).","author":"Yang Dawei","year":"2019","unstructured":"Dawei Yang , Chaowei Xiao , Bo Li , Jia Deng , and Mingyan Liu . 2019 . Realistic Adversarial Examples in 3D Meshes . In Proceedings of the 2019 IEEE Conference on Computer Vision and Patern Recognition (CVPR). Dawei Yang, Chaowei Xiao, Bo Li, Jia Deng, and Mingyan Liu. 2019. Realistic Adversarial Examples in 3D Meshes. In Proceedings of the 2019 IEEE Conference on Computer Vision and Patern Recognition (CVPR)."},{"key":"e_1_3_2_2_83_1","volume-title":"Sankalan Pal Chowdhury, and Maria Christakis","author":"Zhang Fuyuan","year":"2019","unstructured":"Fuyuan Zhang , Sankalan Pal Chowdhury, and Maria Christakis . 2019 . DeepSearch: Simple and Efective Blackbox Fuzzing of Deep Neural Networks . arXiv preprint arXiv: 1910. 06296 ( 2019 ). Fuyuan Zhang, Sankalan Pal Chowdhury, and Maria Christakis. 2019. DeepSearch: Simple and Efective Blackbox Fuzzing of Deep Neural Networks. arXiv preprint arXiv: 1910. 06296 ( 2019 )."},{"key":"e_1_3_2_2_84_1","doi-asserted-by":"publisher","DOI":"10.1145\/2642937.2642994"},{"key":"e_1_3_2_2_85_1","volume-title":"Machine Learning Testing: Survey, Landscapes and Horizons","author":"Zhang Jie M.","year":"2019","unstructured":"Jie M. Zhang , Mark Harman , Lei Ma , and Yang Liu . 2019. Machine Learning Testing: Survey, Landscapes and Horizons . IEEE Transactions on Software Engineering ( 2019 ). Jie M. Zhang, Mark Harman, Lei Ma, and Yang Liu. 2019. Machine Learning Testing: Survey, Landscapes and Horizons. IEEE Transactions on Software Engineering ( 2019 )."},{"key":"e_1_3_2_2_86_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238187"},{"key":"e_1_3_2_2_87_1","volume-title":"Proceedings of the 6th International Conference on Learning Representations (ICLR).","author":"Zhao Zhengli","year":"2018","unstructured":"Zhengli Zhao , Dheeru Dua , and Sameer Singh . 2018 . Generating natural adversarial examples . In Proceedings of the 6th International Conference on Learning Representations (ICLR). Zhengli Zhao, Dheeru Dua, and Sameer Singh. 2018. Generating natural adversarial examples. In Proceedings of the 6th International Conference on Learning Representations (ICLR)."},{"key":"e_1_3_2_2_88_1","unstructured":"Wujie Zheng Wenyu Wang Dian Liu Changrong Zhang Qinsong Zeng Yuetang Deng Wei Yang Pinjia He and Tao Xie. 2018. Testing Untestable Neural Machine Translation: An Industrial Case. arXiv preprint arXiv:1807. 02340 ( 2018 ).  Wujie Zheng Wenyu Wang Dian Liu Changrong Zhang Qinsong Zeng Yuetang Deng Wei Yang Pinjia He and Tao Xie. 2018. Testing Untestable Neural Machine Translation: An Industrial Case. arXiv preprint arXiv:1807. 02340 ( 2018 )."},{"key":"e_1_3_2_2_89_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-Companion.2019.00131"},{"key":"e_1_3_2_2_90_1","volume-title":"Metamorphic Testing for Software Quality Assessment: A Study of Search Engines","author":"Zhou Zhi Quan","year":"2016","unstructured":"Zhi Quan Zhou , Shaowen Xiang , and Tsong Yueh Chen . 2016. Metamorphic Testing for Software Quality Assessment: A Study of Search Engines . IEEE Transactions on Software Engineering (TSE) 42 ( 2016 ). Issue 3. Zhi Quan Zhou, Shaowen Xiang, and Tsong Yueh Chen. 2016. Metamorphic Testing for Software Quality Assessment: A Study of Search Engines. IEEE Transactions on Software Engineering (TSE) 42 ( 2016 ). Issue 3."},{"key":"e_1_3_2_2_91_1","volume-title":"Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1 : Long Papers). Association for Computational Linguistics.","author":"Zhu Muhua","year":"2013","unstructured":"Muhua Zhu , Yue Zhang , Wenliang Chen , Min Zhang , and Jingbo Zhu . 2013 . Fast and Accurate Shift-Reduce Constituent Parsing . In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1 : Long Papers). Association for Computational Linguistics. Muhua Zhu, Yue Zhang, Wenliang Chen, Min Zhang, and Jingbo Zhu. 2013. Fast and Accurate Shift-Reduce Constituent Parsing. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1 : Long Papers). Association for Computational Linguistics."},{"key":"e_1_3_2_2_92_1","unstructured":"Chris. Ziegler. 2016. A Google self-driving car caused a crash for the first time. https:\/\/www.theverge.com\/ 2016 \/2\/29\/11134344\/google-self-driving-carcrash-report  Chris. Ziegler. 2016. A Google self-driving car caused a crash for the first time. https:\/\/www.theverge.com\/ 2016 \/2\/29\/11134344\/google-self-driving-carcrash-report"}],"event":{"name":"ESEC\/FSE '20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering","location":"Virtual Event USA","acronym":"ESEC\/FSE '20","sponsor":["SIGSOFT ACM Special Interest Group on Software Engineering"]},"container-title":["Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3368089.3409756","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3368089.3409756","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:01:58Z","timestamp":1750197718000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3368089.3409756"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,11,8]]},"references-count":91,"alternative-id":["10.1145\/3368089.3409756","10.1145\/3368089"],"URL":"https:\/\/doi.org\/10.1145\/3368089.3409756","relation":{},"subject":[],"published":{"date-parts":[[2020,11,8]]},"assertion":[{"value":"2020-11-08","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}