{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,22]],"date-time":"2026-04-22T02:19:51Z","timestamp":1776824391862,"version":"3.51.2"},"reference-count":92,"publisher":"Association for Computing Machinery (ACM)","issue":"FSE","license":[{"start":{"date-parts":[[2024,7,12]],"date-time":"2024-07-12T00:00:00Z","timestamp":1720742400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Softw. Eng."],"published-print":{"date-parts":[[2024,7,12]]},"abstract":"<jats:p>\n                    The emergence of the \u2018code naturalness\u2019 concept, which suggests that software code shares statistical properties with natural language, paves the way for deep neural networks (DNNs) in software engineering (SE). However, DNNs can be vulnerable to certain human imperceptible variations in the input, known as adversarial examples (AEs), which could lead to adverse model performance. Numerous attack strategies have been proposed to generate AEs in the context of computer vision and natural language processing, but the same is less true for source code of programming languages in SE. One of the challenges is derived from various constraints including syntactic, semantics and minimal modification ratio. These constraints, however, are subjective and can be conflicting with the purpose of fooling DNNs. This paper develops a multi-objective adversarial attack method (dubbed\n                    <jats:monospace>MOAA<\/jats:monospace>\n                    ), a tailored NSGA-II, a powerful evolutionary multi-objective (EMO) algorithm, integrated with\n                    <jats:monospace>CodeT5<\/jats:monospace>\n                    to generate high-quality AEs based on contextual information of the original code snippet. Experiments on 5 source code tasks with 10 datasets of 6 different programming languages show that our approach can generate a diverse set of high-quality AEs with promising transferability. In addition, using our AEs, for the first time, we provide insights into the internal behavior of pre-trained models.\n                  <\/jats:p>","DOI":"10.1145\/3660808","type":"journal-article","created":{"date-parts":[[2024,7,12]],"date-time":"2024-07-12T10:22:09Z","timestamp":1720779729000},"page":"2285-2308","source":"Crossref","is-referenced-by-count":15,"title":["Evolutionary Multi-objective Optimization for Contextual Adversarial Example Generation"],"prefix":"10.1145","volume":"1","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5972-5290","authenticated-orcid":false,"given":"Shasha","family":"Zhou","sequence":"first","affiliation":[{"name":"University of Electronic Science and Technology of China, Chengdu, China"},{"name":"University of Exeter, Exeter, United Kingdom"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2829-8673","authenticated-orcid":false,"given":"Mingyu","family":"Huang","sequence":"additional","affiliation":[{"name":"University of Electronic Science and Technology of China, Chengdu, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8991-7592","authenticated-orcid":false,"given":"Yanan","family":"Sun","sequence":"additional","affiliation":[{"name":"Sichuan University, Chengdu, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7200-4244","authenticated-orcid":false,"given":"Ke","family":"Li","sequence":"additional","affiliation":[{"name":"University of Exeter, Exeter, United Kingdom"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,7,12]]},"reference":[{"key":"e_1_3_1_2_2","doi-asserted-by":"crossref","unstructured":"Naveed Akhtar and Ajmal S. Mian. 2018. Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey. IEEE Access 6 (2018) 14410-14430.","DOI":"10.1109\/ACCESS.2018.2807385"},{"issue":"4","key":"e_1_3_1_3_2","first-page":"81:1","article-title":"A Survey of Machine Learning for Big Code and Naturalness","volume":"51","author":"Allamanis Miltiadis","year":"2018","unstructured":"Miltiadis Allamanis, Earl T. Barr, Premkumar T. Devanbu, and Charles Sutton. 2018. A Survey of Machine Learning for Big Code and Naturalness. ACM Comput. Surv. 51, 4 (2018), 81:1-81:37.","journal-title":"ACM Comput. Surv."},{"key":"e_1_3_1_4_2","doi-asserted-by":"crossref","unstructured":"Bander Alsulami Edwin Dauber Richard E. Harang Spiros Mancoridis and Rachel Greenstadt. 2017. Source Code Authorship Attribution Using Long Short-Term Memory Based Networks. In ESORICS\u201917: Proc. of the 22nd European Symposium on Research in Computer Security Vol. 10492. 65-82.","DOI":"10.1007\/978-3-319-66402-6_6"},{"key":"e_1_3_1_5_2","doi-asserted-by":"publisher","DOI":"10.1162\/EVCO_a_00009"},{"key":"e_1_3_1_6_2","article-title":"Code Generation Tools (Almost) for Free? A Study of Few-Shot, Pre-Trained Language Models on Code","author":"Barei\u00df Patrick","year":"2022","unstructured":"Patrick Barei\u00df, Beatriz Souza, Marcelo d\u2019Amorim, and Michael Pradel. 2022. Code Generation Tools (Almost) for Free? A Study of Few-Shot, Pre-Trained Language Models on Code. CoRR (2022).","journal-title":"CoRR"},{"key":"e_1_3_1_7_2","unstructured":"Pavol Bielik and Martin T. Vechev. 2020. Adversarial Robustness for Code. In ICML\u201920: Proc. of the 37th International Conference on Machine Learning Vol. 119. 896-907."},{"key":"e_1_3_1_8_2","doi-asserted-by":"crossref","unstructured":"Lutz B\u00fcch and Artur Andrzejak. 2019. Learning-Based Recursive Aggregation of Abstract Syntax Trees for Code Clone Detection. In SANER\u201919: Proc. of the 26th IEEE International Conference on Software Analysis Evolution and Reengineering. 95-104.","DOI":"10.1109\/SANER.2019.8668039"},{"issue":"9","key":"e_1_3_1_9_2","doi-asserted-by":"crossref","first-page":"3280","DOI":"10.1109\/TSE.2021.3087402","article-title":"Deep Learning Based Vulnerability Detection: Are We There Yet? IEEE Trans","volume":"48","author":"Chakraborty Saikat","year":"2022","unstructured":"Saikat Chakraborty, Rahul Krishna, Yangruibo Ding, and Baishakhi Ray. 2022. Deep Learning Based Vulnerability Detection: Are We There Yet? IEEE Trans. Software Eng. 48, 9 (2022), 3280-3296.","journal-title":"Software Eng."},{"key":"e_1_3_1_10_2","first-page":"764","volume-title":"SMC\u201921: Proc. of the IEEE International Conference on Systems, Man, and Cybernetics","author":"Chen !Renzhi","year":"2021","unstructured":"!Renzhi Chen and Ke Li 2021 Knee Point Identification Based on the Geometric Characteristic In SMC\u201921: Proc. of the IEEE International Conference on Systems, Man, and Cybernetics. IEEE, 764\u2013769."},{"key":"e_1_3_1_11_2","doi-asserted-by":"crossref","unstructured":"Xilun Chen Ahmed Hassan Awadallah Hany Hassan Wei Wang and Claire Cardie 2019 Multi-Source Cross-Lingual Model Transfer: Learning What to Share In ACL\u201919: Proc. of the 57th Conference of the Association for Computational Linguistics ACL 2019 Florence Italy July 28- August 2 2019 Volume 1: Long Papers 3098\u20133112.","DOI":"10.18653\/v1\/P19-1299"},{"key":"e_1_3_1_12_2","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2016.2519378"},{"key":"e_1_3_1_13_2","doi-asserted-by":"crossref","unstructured":"Xin Cheng Shen Gao Lemao Liu Dongyan Zhao and Rui Yan. 2022. Neural Machine Translation with Contrastive Translation Memories. In EMNLP\u201922: Proc. of the Conference on Empirical Methods in Natural Language Processing. 3591-3601.","DOI":"10.18653\/v1\/2022.emnlp-main.235"},{"key":"e_1_3_1_14_2","doi-asserted-by":"crossref","unstructured":"Kalyanmoy Deb Samir Agrawal Amrit Pratap and T. Meyarivan. 2000. A Fast Elitist Non-dominated Sorting Genetic Algorithm for Multi-objective Optimisation: NSGA-II. In PPSN\u201900: Proc. of the Parallel Problem Solving from Nature Vol. 1917. 849-858.","DOI":"10.1007\/3-540-45356-3_83"},{"key":"e_1_3_1_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2013.2281535"},{"key":"e_1_3_1_16_2","doi-asserted-by":"crossref","unstructured":"Jacob Devlin Ming-Wei Chang Kenton Lee and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL-HLT\u201919: Proc. of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 4171-4186.","DOI":"10.18653\/v1\/N19-1423"},{"key":"e_1_3_1_17_2","doi-asserted-by":"crossref","unstructured":"Xiaohu Du Ming Wen Zichao Wei Shangwen Wang and Hai Jin. 2023. An Extensive Study on Adversarial Attack against Pre-trained Models of Code. In ESEC\/FSE\u201923: Proc. of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 489-501.","DOI":"10.1145\/3611643.3616356"},{"key":"e_1_3_1_18_2","unstructured":"Javid Ebrahimi Daniel Lowd and Dejing Dou 2018 On Adversarial Examples for Character-Level Neural Machine Translation. COLING\u201918: Proc. of the 27th International Conference on Computational Linguistics. 653\u2013663."},{"key":"e_1_3_1_19_2","doi-asserted-by":"crossref","unstructured":"Javid Ebrahimi Anyi Rao Daniel Lowd and Dejing Dou. 2018. HotFlip: White-Box Adversarial Examples for Text Classification. In ACL\u201918: Proc. of the 56th Annual Meeting of the Association for Computational Linguistics. 31-36.","DOI":"10.18653\/v1\/P18-2006"},{"key":"e_1_3_1_20_2","doi-asserted-by":"crossref","unstructured":"Zhangyin Feng Daya Guo Duyu Tang Nan Duan Xiaocheng Feng Ming Gong Linjun Shou Bing Qin Ting Liu Daxin Jiang and Ming Zhou. 2020. CodeBERT: A Pre-Trained Model for Programming and Natural Languages. In EMNLP\u201920: Findings of the Association for Computational Linguistics. 1536-1547.","DOI":"10.18653\/v1\/2020.findings-emnlp.139"},{"key":"e_1_3_1_21_2","unstructured":"Patrick Fernandes Miltiadis Allamanis and Marc Brockschmidt. 2019. Structured Neural Summarization. In ICLR\u201919: Proc. of the 7th International Conference on Learning Representations."},{"key":"e_1_3_1_22_2","doi-asserted-by":"crossref","unstructured":"Siddhant Garg and Goutham Ramakrishnan. 2020. BAE: BERT-based Adversarial Examples for Text Classification. In EMNLP\u201920: Proc. of the 2020 Conference on Empirical Methods in Natural Language Processing. 6174-6181.","DOI":"10.18653\/v1\/2020.emnlp-main.498"},{"key":"e_1_3_1_23_2","unstructured":"Ian J. Goodfellow Jonathon Shlens and Christian Szegedy. 2015. Explaining and Harnessing Adversarial Examples. In ICLR\u201915: Proc. of the 3rd International Conference on Learning Representations."},{"key":"e_1_3_1_24_2","doi-asserted-by":"crossref","unstructured":"Jian Gu Pasquale Salza and Harald C. Gall. 2022. Assemble Foundation Models for Automatic Code Summarization. In SANER\u201922: Proc. of the IEEE International Conference on Software Analysis Evolution and Reengineering. 935-946.","DOI":"10.1109\/SANER53432.2022.00112"},{"key":"e_1_3_1_25_2","unstructured":"Daya Guo Shuo Ren Shuai Lu Zhangyin Feng Duyu Tang Shujie Liu Long Zhou Nan Duan Alexey Svyatkovskiy Shengyu Fu Michele Tufano Shao Kun Deng Colin B. Clement Dawn Drain Neel Sundaresan Jian Yin Daxin Jiang and Ming Zhou. 2021. GraphCodeBERT: Pre-training Code Representations with Data Flow. In ICLR\u201921: Proc. of the 9th International Conference on Learning Representations."},{"key":"e_1_3_1_26_2","doi-asserted-by":"crossref","unstructured":"W Keith Hastings. 1970. Monte Carlo sampling methods using Markov chains and their applications. (1970).","DOI":"10.2307\/2334940"},{"key":"e_1_3_1_27_2","unstructured":"Jingxuan He Luca Beurer-Kellner and Martin T. Vechev. 2022. On Distribution Shift in Learning-based Bug Detectors. In ICML\u201922: Proc. of the International Conference on Machine Learning Vol. 162. 8559-8580."},{"key":"e_1_3_1_28_2","doi-asserted-by":"crossref","unstructured":"Jordan Henkel Goutham Ramakrishnan Zi Wang Aws Albarghouthi Somesh Jha and Thomas W. Reps. 2022. Semantic Robustness of Models of Source Code. In SANER\u201922: Proc. of the IEEE International Conference on Software Analysis Evolution and Reengineering. 526-537.","DOI":"10.1109\/SANER53432.2022.00070"},{"key":"e_1_3_1_29_2","doi-asserted-by":"crossref","unstructured":"Abram Hindle Earl T. Barr Zhendong Su Mark Gabel and Premkumar T. Devanbu. 2012. On the naturalness of software. In ICSE\u201912: Proc. of the 34th International Conference on Software Engineering. 837-847.","DOI":"10.1109\/ICSE.2012.6227135"},{"key":"e_1_3_1_30_2","doi-asserted-by":"crossref","unstructured":"Hossein Hosseini and Radha Poovendran. 2018. Semantic Adversarial Examples. In CVPR\u201918: Proc. of the 2018 IEEE Conference on Computer Vision and Pattern Recognition Workshops. 1614-1619.","DOI":"10.1109\/CVPRW.2018.00212"},{"key":"e_1_3_1_31_2","doi-asserted-by":"crossref","unstructured":"Yu-Lun Hsieh Minhao Cheng Da-Cheng Juan Wei Wei Wen-Lian Hsu and Cho-Jui Hsieh. 2019. On the Robustness of Self-Attentive Models. In ACL\u201919: Proc. of Conference of the Association for Computational Linguistics. 1520-1529.","DOI":"10.18653\/v1\/P19-1147"},{"key":"e_1_3_1_32_2","article-title":"A Survey of Decomposition-Based Evolutionary Multi-Objective Optimization: Part II - A Data Science Perspective","author":"Huang Mingyu","year":"2024","unstructured":"Mingyu Huang and Ke Li. 2024. A Survey of Decomposition-Based Evolutionary Multi-Objective Optimization: Part II - A Data Science Perspective. CoRR (2024).","journal-title":"CoRR"},{"key":"e_1_3_1_33_2","doi-asserted-by":"crossref","unstructured":"Robin Jia and Percy Liang. 2017. Adversarial Examples for Evaluating Reading Comprehension Systems. In EMNLP\u201917: Proc. of the 2017 Conference on Empirical Methods in Natural Language Processing. 2021-2031.","DOI":"10.18653\/v1\/D17-1215"},{"key":"e_1_3_1_34_2","doi-asserted-by":"crossref","unstructured":"Di Jin Zhijing Jin Joey Tianyi Zhou and Peter Szolovits. 2020. Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment. In AAAI\u201920: Proc. of the 34th AAAI Conference on Artificial Intelligence. 8018-8025.","DOI":"10.1609\/aaai.v34i05.6311"},{"key":"e_1_3_1_35_2","doi-asserted-by":"publisher","DOI":"10.1093\/comjnl\/27.2.97"},{"key":"e_1_3_1_36_2","doi-asserted-by":"crossref","unstructured":"Mike Lewis Yinhan Liu Naman Goyal Marjan Ghazvininejad Abdelrahman Mohamed Omer Levy Veselin Stoyanov and Luke Zettlemoyer. 2020. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation Translation and Comprehension. In ACL\u201920: Proc. of the Association for Computational Linguistics. 7871-7880.","DOI":"10.18653\/v1\/2020.acl-main.703"},{"key":"e_1_3_1_37_2","article-title":"A Survey of Decomposition-Based Evolutionary Multi-Objective Optimization: Part I-Past and Future","author":"Li Ke","year":"2024","unstructured":"Ke Li. 2024. A Survey of Decomposition-Based Evolutionary Multi-Objective Optimization: Part I-Past and Future. CoRR (2024).","journal-title":"CoRR"},{"key":"e_1_3_1_38_2","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2014.2373386"},{"key":"e_1_3_1_39_2","doi-asserted-by":"crossref","unstructured":"Zhen Li Qian (Guenevere) Chen Chen Chen Yayi Zou and Shouhuai Xu. 2022. RoPGen: Towards Robust Code Authorship Attribution via Automatic Coding Style Transformation. In ICSE\u201922: Proc. of the 44th IEEE\/ACM 44th International Conference on Software Engineering. 1906-1918.","DOI":"10.1145\/3510003.3510181"},{"key":"e_1_3_1_40_2","doi-asserted-by":"crossref","unstructured":"Muyang Liu Ke Li and Tao Chen. 2020. DeepSQLi: deep semantic learning for testing SQL injection. In ISSTA\u201920: Proc. of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis. 286-297.","DOI":"10.1145\/3395363.3397375"},{"key":"e_1_3_1_41_2","unstructured":"Shuai Lu Daya Guo Shuo Ren Junjie Huang Alexey Svyatkovskiy Ambrosio Blanco Colin B. Clement Dawn Drain Daxin Jiang Duyu Tang Ge Li Lidong Zhou Linjun Shou Long Zhou Michele Tufano Ming Gong Ming Zhou Nan Duan Neel Sundaresan Shao Kun Deng Shengyu Fu and Shujie Liu. 2021. CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation. In NeurIPS\u201921: Proc. of the Neural Information Processing Systems Track on Datasets and Benchmarks."},{"key":"e_1_3_1_42_2","unstructured":"Changze Lv Jianhan Xu and Xiaoqing Zheng. 2023. Spiking Convolutional Neural Networks for Text Classification. In ICLR\u201923: Proc. of the Eleventh International Conference on Learning Representations."},{"key":"e_1_3_1_43_2","unstructured":"Aleksander Madry Aleksandar Makelov Ludwig Schmidt Dimitris Tsipras and Adrian Vladu. 2018. Towards Deep Learning Models Resistant to Adversarial Attacks. In ICLR\u201918: Proc. of the 6th International Conference on Learning Representations."},{"key":"e_1_3_1_44_2","doi-asserted-by":"crossref","unstructured":"Rishabh Maheshwary Saket Maheshwary and Vikram Pudi. 2021. Generating Natural Language Attacks in a Hard Label Black Box Setting. In AAAI\u201921: Proc. of the 35th AAAI Conference on Artificial Intelligence. 13525-13533.","DOI":"10.1609\/aaai.v35i15.17595"},{"key":"e_1_3_1_45_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2021.3105556"},{"key":"e_1_3_1_46_2","doi-asserted-by":"crossref","unstructured":"Nicholas Metropolis Arianna W. Rosenbluth Marshall N. Rosenbluth Augusta H. Teller and Edwards Teller. 1953. Equation of State Calculations by Fast Computing Machines. (1953) 1087-1092.","DOI":"10.1063\/1.1699114"},{"key":"e_1_3_1_47_2","doi-asserted-by":"crossref","unstructured":"John X. Morris Eli Lifland Jack Lanchantin Yangfeng Ji and Yanjun Qi. 2020. Reevaluating Adversarial Examples in Natural Language. In EMNLP\u201920: Findings of the Association for Computational Linguistics. 3829-3839.","DOI":"10.18653\/v1\/2020.findings-emnlp.341"},{"key":"e_1_3_1_48_2","unstructured":"Daniel Naber et al. 2003. A rule-based style and grammar checker. (2003)."},{"key":"e_1_3_1_49_2","doi-asserted-by":"crossref","unstructured":"Phuong T. Nguyen Claudio Di Sipio Juri Di Rocco Massimiliano Di Penta and Davide Di Ruscio. 2021. Adversarial Attacks to API Recommender Systems: Time to Wake Up and Smell the Coffee f. In ASE\u201921: Proc. of the 36th IEEE\/ACM International Conference on Automated Software Engineering. 253-265.","DOI":"10.1109\/ASE51524.2021.9678946"},{"key":"e_1_3_1_50_2","first-page":"1081","volume-title":"SMC\u201920: Proc. of the IEEE International Conference on Systems, Man, and Cybernetics","author":"Nie Haifeng","year":"2020","unstructured":"Haifeng Nie, Huiru Gao, and Ke Li 2020 Knee Point Identification Based on Voronoi Diagram In SMC\u201920: Proc. of the IEEE International Conference on Systems, Man, and Cybernetics. IEEE, 1081\u20131086."},{"key":"e_1_3_1_51_2","doi-asserted-by":"crossref","unstructured":"Nicolas Papernot Patrick D. McDaniel Ananthram Swami and Richard E. Harang. 2016. Crafting adversarial input sequences for recurrent neural networks. In MILCOM\u201916: Proc. of the 2016 IEEE Military Communications Conference. 49-54.","DOI":"10.1109\/MILCOM.2016.7795300"},{"key":"e_1_3_1_52_2","doi-asserted-by":"crossref","unstructured":"Kishore Papineni Salim Roukos Todd Ward and Wei-Jing Zhu. 2002. Bleu: a Method for Automatic Evaluation of Machine Translation. In ACL\u201902: Proc. of the 40th Annual Meeting of the Association for Computational Linguistics. 311-318.","DOI":"10.3115\/1073083.1073135"},{"key":"e_1_3_1_53_2","doi-asserted-by":"crossref","unstructured":"Maryam Vahdat Pour Zhuo Li Lei Ma and Hadi Hemmati. 2021. A Search-Based Testing Framework for Deep Neural Networks of Source Code Embedding. In ICST\u201921: Proc. of the 14th IEEE Conference on Software Testing Verification and Validation. 36-46.","DOI":"10.1109\/ICST49551.2021.00016"},{"key":"e_1_3_1_54_2","doi-asserted-by":"crossref","unstructured":"Md. Rafiqul Islam Rabin Nghi D. Q. Bui Ke Wang Yijun Yu Lingxiao Jiang and Mohammad Amin Alipour. 2021. On the generalizability of Neural Program Models with respect to semantic-preserving program transformations. Inf. Softw. Technol. 135 (2021) 106552.","DOI":"10.1016\/j.infsof.2021.106552"},{"key":"e_1_3_1_55_2","article-title":"Robust Speech Recognition via Large-Scale Weak Supervision","author":"Radford Alec","year":"2022","unstructured":"Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022. Robust Speech Recognition via Large-Scale Weak Supervision. CoRR (2022).","journal-title":"CoRR"},{"key":"e_1_3_1_56_2","unstructured":"Colin Raffel Noam Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. J. Mach. Learn. Res. 21 (2020) 140:1-140:67."},{"key":"e_1_3_1_57_2","doi-asserted-by":"crossref","unstructured":"Shuhuai Ren Yihe Deng Kun He and Wanxiang Che 2019 Generating Natural Language Adversarial Examples through Probability Weighted Word Saliency In ACL\u201919: Proc. of the 57th Conference of the Association for Computational Linguistics 1085\u20131097.","DOI":"10.18653\/v1\/P19-1103"},{"key":"e_1_3_1_58_2","article-title":"CodeBLEU: a Method for Automatic Evaluation of Code Synthesis","author":"Ren Shuo","year":"2020","unstructured":"Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. CodeBLEU: a Method for Automatic Evaluation of Code Synthesis. CoRR (2020).","journal-title":"CoRR"},{"key":"e_1_3_1_59_2","doi-asserted-by":"crossref","unstructured":"Marco Tulio Ribeiro Sameer Singh and Carlos Guestrin 2016 \"Why Should I Trust You?\": Explaining the Predictions of Any Classifier In KDD\u201916: Proc. of the 22nd Conference on Knowledge Discovery and Data Mining 1135\u20131144.","DOI":"10.1145\/2939672.2939778"},{"key":"e_1_3_1_60_2","unstructured":"Baptiste Rozi\u00e8re Jie Zhang Fran\u00e7ois Charton Mark Harman Gabriel Synnaeve and Guillaume Lample. 2022. Leveraging Automated Unit Tests for Unsupervised Code Translation. In ICLR\u201922: Proc. of the 10th International Conference on Learning Representations."},{"key":"e_1_3_1_61_2","article-title":"TransFool: An Adversarial Attack against Neural Machine Translation Models","author":"Sadrizadeh Sahar","year":"2023","unstructured":"Sahar Sadrizadeh, Ljiljana Dolamic, and Pascal Frossard. 2023. TransFool: An Adversarial Attack against Neural Machine Translation Models. CoRR (2023).","journal-title":"CoRR"},{"key":"e_1_3_1_62_2","unstructured":"Roei Schuster Congzheng Song Eran Tromer and Vitaly Shmatikov. 2021. You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completion. In USENIX\u201921: Proc. of the 30th USENIX Security Symposium. 1559-1575."},{"key":"e_1_3_1_63_2","unstructured":"Shashank Srikant Sijia Liu Tamara Mitrovska Shiyu Chang Quanfu Fan Gaoyuan Zhang and Una-May O\u2019Reilly. 2021. Generating Adversarial Computer Programs using Optimized Obfuscations. In ICLR\u201921: Proc. of the 9th International Conference on Learning Representations."},{"key":"e_1_3_1_64_2","unstructured":"Marc Szafraniec Baptiste Rozi\u00e8re Hugh Leather Patrick Labatut Fran\u00e7ois Charton and Gabriel Synnaeve. 2023. Code Translation with Compiler Representations. In ICLR\u201923: Proc. of the 11th International Conference on Learning Representations."},{"key":"e_1_3_1_65_2","unstructured":"Christian Szegedy Wojciech Zaremba Ilya Sutskever Joan Bruna Dumitru Erhan Ian J. Goodfellow and Rob Fergus. 2014. Intriguing properties of neural networks. In ICLR\u201914: Proc. of the 2nd International Conference on Learning Representations."},{"key":"e_1_3_1_66_2","doi-asserted-by":"crossref","unstructured":"Yun Tang Hongyu Gong Ning Dong Changhan Wang Wei-Ning Hsu Jiatao Gu Alexei Baevski Xian Li Abdelrahman Mohamed Michael Auli and Juan Miguel Pino. 2022. Unified Speech-Text Pre-training for Speech Translation and Recognition. In ACL\u201922: Proc. of the 60th Annual Meeting of the Association for Computational Linguistics. 1488-1499.","DOI":"10.18653\/v1\/2022.acl-long.105"},{"key":"e_1_3_1_67_2","article-title":"Adversarial Attacks on Neural Models of Code via Code Difference Reduction","author":"Tian Zhao","year":"2023","unstructured":"Zhao Tian, Junjie Chen, and Zhi Jin. 2023. Adversarial Attacks on Neural Models of Code via Code Difference Reduction. CoRR (2023).","journal-title":"CoRR"},{"key":"e_1_3_1_68_2","unstructured":"Hsiang-Sheng Tsai Heng-Jui Chang Wen-Chin Huang Zili Huang Kushal Lakhotia Shu-Wen Yang Shuyan Dong Andy T. Liu Cheng-I Lai Jiatong Shi Xuankai Chang Phil Hall Hsuan-Jui Chen Shang-Wen Li Shinji Watanabe Abdelrahman Mohamed and Hung-yi Lee. 2022. SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilities. In ACL\u201922: Proc. of the 60th Annual Meeting of the Association for Computational Linguistics. 8479-8492."},{"key":"e_1_3_1_69_2","unstructured":"Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N. Gomez Lukasz Kaiser and Illia Polosukhin. 2017. Attention is All you Need. In NeurIPS\u201917: Advances in Neural Information Processing Systems. 5998-6008."},{"key":"e_1_3_1_70_2","doi-asserted-by":"crossref","unstructured":"Yao Wan Wei Zhao Hongyu Zhang Yulei Sui Guandong Xu and Hai Jin. 2022. What Do They Capture? - A Structural Analysis of Pre-Trained Language Models for Source Code. In ICSE\u201922: Proc. of the 44th IEEE\/ACM 44th International Conference on Software Engineering. 2377-2388.","DOI":"10.1145\/3510003.3510050"},{"key":"e_1_3_1_71_2","doi-asserted-by":"crossref","unstructured":"Song Wang Taiyue Liu and Lin Tan. 2016. Automatically learning semantic features for defect prediction. In ICSE\u201916: Proc. of the 38th International Conference on Software Engineering. 297-308.","DOI":"10.1145\/2884781.2884804"},{"key":"e_1_3_1_72_2","doi-asserted-by":"crossref","unstructured":"Wenhan Wang Ge Li Bo Ma Xin Xia and Zhi Jin 2020 Detecting Code Clones with Graph Neural Network and Flow-Augmented Abstract Syntax Tree In SANER\u201920: Proc. of the 27th Conference on Software Analysis Evolution and Reengineering 261\u2013271.","DOI":"10.1109\/SANER48275.2020.9054857"},{"key":"e_1_3_1_73_2","doi-asserted-by":"crossref","unstructured":"Xuezhi Wang Haohan Wang and Diyi Yang. 2022. Measure and Improve Robustness in NLP Models: A Survey. In NAACL\u201922: Proc. of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 4569-4586.","DOI":"10.18653\/v1\/2022.naacl-main.339"},{"key":"e_1_3_1_74_2","doi-asserted-by":"crossref","unstructured":"Yue Wang Weishi Wang Shafiq R. Joty and Steven C. H. Hoi. 2021. CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation. In EMNLP\u201921: Proc. of the 2021 Conference on Empirical Methods in Natural Language Processing. 8696-8708.","DOI":"10.18653\/v1\/2021.emnlp-main.685"},{"key":"e_1_3_1_75_2","unstructured":"Phoenix Williams and Ke Li. 2023. CamoPatch: An Evolutionary Strategy for Generating Camoflauged Adversarial Patches. In NeurIPS\u201923: Proc. in Neural Information Processing Systems."},{"key":"e_1_3_1_76_2","doi-asserted-by":"crossref","unstructured":"Phoenix Neale Williams and Ke Li. 2023. Black-Box Sparse Adversarial Attack via Multi-Objective Optimisation CVPR Proceedings. In CVPR\u201923: Proc. of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 12291-12301.","DOI":"10.1109\/CVPR52729.2023.01183"},{"key":"e_1_3_1_77_2","doi-asserted-by":"crossref","unstructured":"Phoenix Neale Williams Ke Li and Geyong Min 2023 Sparse Adversarial Attack via Bi-objective Optimization In EMO\u201923: Proc. of the Evolutionary Multi-Criterion Optimization - 12th International Conference. Springer 118\u2013133. Vol. 13970.","DOI":"10.1007\/978-3-031-27250-9_9"},{"issue":"10","key":"e_1_3_1_78_2","article-title":"A Survey on Deep Learning for Software Engineering","volume":"54","author":"Yang Yanming","year":"2022","unstructured":"Yanming Yang, Xin Xia, David Lo, and John Grundy. 2022. A Survey on Deep Learning for Software Engineering. ACM Comput. Surv. 54, 10s (2022).","journal-title":"ACM Comput. Surv."},{"key":"e_1_3_1_79_2","doi-asserted-by":"crossref","unstructured":"Zhou Yang Jieke Shi Junda He and David Lo. 2022. Natural Attack for Pre-trained Models of Code. In ICSE\u201922: Proc. of the 44th IEEE\/ACM 44th International Conference on Software Engineering. 1482-1493.","DOI":"10.1145\/3510003.3510146"},{"key":"e_1_3_1_80_2","first-page":"162:1","article-title":"Adversarial examples for models of code","volume":"4","author":"Yefet Noam","year":"2020","unstructured":"Noam Yefet, Uri Alon, and Eran Yahav. 2020. Adversarial examples for models of code. Proc. ACM Program. Lang. 4, OOPSLA (2020), 162:1-162:30.","journal-title":"Proc. ACM Program. Lang."},{"key":"e_1_3_1_81_2","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2018.2886017"},{"key":"e_1_3_1_82_2","doi-asserted-by":"crossref","unstructured":"Zhengran Zeng Hanzhuo Tan Haotian Zhang Jing Li Yuqun Zhang and Lingming Zhang. 2022. An extensive study on pre-trained models for program understanding and generation. In ISSTA\u201922: Proc. of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis. 39-51.","DOI":"10.1145\/3533767.3534390"},{"issue":"3","key":"e_1_3_1_83_2","first-page":"50:1","article-title":"Towards Robustness of Deep Program Processing Models - Detection, Estimation, and Enhancement","volume":"31","author":"Zhang Huangzhao","year":"2022","unstructured":"Huangzhao Zhang, Zhiyi Fu, Ge Li, Lei Ma, Zhehao Zhao, Hua\u2019an Yang, Yizhe Sun, Yang Liu, and Zhi Jin. 2022. Towards Robustness of Deep Program Processing Models - Detection, Estimation, and Enhancement. ACM Trans. Softw. Eng. Methodol. 31, 3 (2022), 50:1-50:40.","journal-title":"ACM Trans. Softw. Eng. Methodol."},{"key":"e_1_3_1_84_2","doi-asserted-by":"crossref","unstructured":"Huangzhao Zhang Zhuo Li Ge Li Lei Ma Yang Liu and Zhi Jin. 2020. Generating Adversarial Examples for Holding Robustness of Source Code Processing Models. In AAAI\u201920: Proc. of the Conference on Artificial Intelligence. 1169-1176.","DOI":"10.1609\/aaai.v34i01.5469"},{"key":"e_1_3_1_85_2","doi-asserted-by":"publisher","DOI":"10.1109\/TEVC.2007.892759"},{"key":"e_1_3_1_86_2","unstructured":"Xiang Zhang Junbo Jake Zhao and Yann LeCun. 2015. Character-level Convolutional Networks for Text Classification. In NIPS\u201915: Advances in Neural Information Processing Systems. 649-657."},{"key":"e_1_3_1_87_2","unstructured":"Zhengli Zhao Dheeru Dua and Sameer Singh. 2018. Generating Natural Adversarial Examples. In ICLR\u201918: Proc. of the 6th International Conference on Learning Representations."},{"key":"e_1_3_1_88_2","doi-asserted-by":"crossref","unstructured":"Shasha Zhou Ke Li and Geyong Min. 2022. Adversarial example generation via genetic algorithm: a preliminary result. In GECCO\u201922: Proc. of the Genetic and Evolutionary Computation Conference.","DOI":"10.1145\/3520304.3528981"},{"key":"e_1_3_1_89_2","first-page":"341","volume-title":"PPSN\u201922: Proc. of the Parallel Problem Solving from Nature","author":"Zhou Shasha","year":"2022","unstructured":"Shasha Zhou, Ke Li, and Geyong Min 2022 Attention-Based Genetic Algorithm for Adversarial Attack in Natural Language Processing In PPSN\u201922: Proc. of the Parallel Problem Solving from Nature. Springer, 341\u2013355. Vol. 13398."},{"issue":"4","key":"e_1_3_1_90_2","first-page":"60:1","article-title":"Adversarial Robustness of Deep Code Comment Generation","volume":"31","author":"Zhou Yu","year":"2022","unstructured":"Yu Zhou, Xiaoqing Zhang, Juanjuan Shen, Tingting Han, Taolue Chen, and Harald C. Gall. 2022. Adversarial Robustness of Deep Code Comment Generation. ACM Trans. Softw. Eng. Methodol. 31, 4 (2022), 60:1-60:30.","journal-title":"ACM Trans. Softw. Eng. Methodol."},{"key":"e_1_3_1_91_2","doi-asserted-by":"crossref","unstructured":"Eckart Zitzler and Simon K\u00fcnzli. 2004. Indicator-Based Selection in Multiobjective Search. In PPSN\u201904: Proc. of the Parallel Problem Solving from Nature Vol. 3242. 832-842.","DOI":"10.1007\/978-3-540-30217-9_84"},{"key":"e_1_3_1_92_2","article-title":"A Reinforced Generation of Adversarial Samples for Neural Machine Translation","author":"Zou Wei","year":"2019","unstructured":"Wei Zou, Shujian Huang, Jun Xie, Xinyu Dai, and Jiajun Chen. 2019. A Reinforced Generation of Adversarial Samples for Neural Machine Translation. CoRR (2019).","journal-title":"CoRR"},{"key":"e_1_3_1_93_2","unstructured":"Daniel Z\u00fcgner Tobias Kirschstein Michele Catasta Jure Leskovec and Stephan G\u00fcnnemann. 2021. Language-Agnostic Representation Learning of Source Code from Structure and Context. In ICLR\u201921: Proc. of the 9th International Conference on Learning Representations."}],"container-title":["Proceedings of the ACM on Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3660808","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3660808","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,2,4]],"date-time":"2026-02-04T07:59:59Z","timestamp":1770191999000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3660808"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,12]]},"references-count":92,"journal-issue":{"issue":"FSE","published-print":{"date-parts":[[2024,7,12]]}},"alternative-id":["10.1145\/3660808"],"URL":"https:\/\/doi.org\/10.1145\/3660808","relation":{},"ISSN":["2994-970X"],"issn-type":[{"value":"2994-970X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,7,12]]}}}