{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,4]],"date-time":"2026-04-04T18:21:11Z","timestamp":1775326871126,"version":"3.50.1"},"reference-count":50,"publisher":"Association for Computing Machinery (ACM)","issue":"1","funder":[{"name":"Luxembourg National Research Fund","award":["BRIDGES\/2022\/IS\/17437536"],"award-info":[{"award-number":["BRIDGES\/2022\/IS\/17437536"]}]},{"name":"European Union\u2019s Horizon Research and Innovation Programme, as part of the project LAZARUS","award":["101070303"],"award-info":[{"award-number":["101070303"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2026,1,31]]},"abstract":"<jats:p>Robustness testing is essential for evaluating deep learning models, particularly under unforeseen circumstances. Adversarial test generation, a fundamental approach in robustness testing, is prevalent in computer vision and natural language processing, and it has gained considerable attention in code tasks recently. The Variable Renaming-Based Adversarial Test Generation (VRTG), which deceives models by altering variable names, is a key focus. VRTG involves substitution construction and variable name searching, but its systematic design remains a challenge due to the empirical nature of these components. This article introduces the first benchmark to examine the impact of various substitutions and search algorithms on VRTG effectiveness, exploring improvements for existing VRTGs. Our benchmark includes three substitution construction types, six substitution position rank ways and seven search algorithms. Analysis of four code understanding tasks and three pre-trained code models using our benchmark reveals that combining RNNS and Genetic Algorithm with code-based substitution is more effective for VRTG construction. Notably, this method outperforms the advanced black-box variable renaming test generation technique, ALERT, by up to 22.57%.<\/jats:p>","DOI":"10.1145\/3723353","type":"journal-article","created":{"date-parts":[[2025,3,14]],"date-time":"2025-03-14T11:30:16Z","timestamp":1741951816000},"page":"1-28","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["Variable Renaming-Based Adversarial Test Generation for Code Model: Benchmark and Enhancement"],"prefix":"10.1145","volume":"35","author":[{"ORCID":"https:\/\/orcid.org\/0009-0004-7187-7296","authenticated-orcid":false,"given":"Jin","family":"Wen","sequence":"first","affiliation":[{"name":"University of Luxembourg, Esch-sur-Alzette, Luxembourg"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8251-1669","authenticated-orcid":false,"given":"Qiang","family":"Hu","sequence":"additional","affiliation":[{"name":"Tianjin University, Tianjin, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5535-2420","authenticated-orcid":false,"given":"Yuejun","family":"Guo","sequence":"additional","affiliation":[{"name":"Luxembourg Institute of Science and Technology, Esch-sur-Alzette, Luxembourg"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8312-1358","authenticated-orcid":false,"given":"Maxime","family":"Cordy","sequence":"additional","affiliation":[{"name":"University of Luxembourg, Esch-sur-Alzette, Luxembourg"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1045-4861","authenticated-orcid":false,"given":"Yves Le","family":"Traon","sequence":"additional","affiliation":[{"name":"University of Luxembourg, Esch-sur-Alzette, Luxembourg"}]}],"member":"320","published-online":{"date-parts":[[2025,12,11]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/3212695"},{"key":"e_1_3_2_3_2","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1007\/978-3-319-66402-6_6","volume-title":"Computer Security \u2013 ESORICS 2017","author":"Alsulami Bander","year":"2017","unstructured":"Bander Alsulami, Edwin Dauber, Richard Harang, Spiros Mancoridis, and Rachel Greenstadt. 2017. Source code authorship attribution using long short-term memory based networks. In Computer Security \u2013 ESORICS 2017. Simon N. Foley, Dieter Gollmann, and Einar Snekkenes (Eds.), Springer International Publishing, Cham, 65\u201382."},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58592-1_29"},{"key":"e_1_3_2_5_2","unstructured":"AtCoder Inc. 2017. Problem statement: B - shift only. Accessed December 30 2024 from https:\/\/atcoder.jp\/contests\/abc081\/tasks\/abc081_b"},{"key":"e_1_3_2_6_2","unstructured":"AtCoder Inc. 2018. Problem statement: C - *3 or \/2. Accessed December 30 2024 from https:\/\/atcoder.jp\/contests\/abc100\/tasks\/abc100_c"},{"key":"e_1_3_2_7_2","first-page":"780","volume-title":"Proceedings of the 38th International Conference on Machine LearningProceedings of Machine Learning Research","volume":"139","author":"Berabi Berkay","year":"2021","unstructured":"Berkay Berabi, Jingxuan He, Veselin Raychev, and Martin Vechev. 2021. TFix: learning to fix coding errors with a text-to-text transformer. In Proceedings of the 38th International Conference on Machine Learning. Marina Meila and Tong Zhang (Eds.), Proceedings of Machine Learning Research, Vol. 139, PMLR, 780\u2013791. Retrieved from https:\/\/proceedings.mlr.press\/v139\/berabi21a.html"},{"key":"e_1_3_2_8_2","volume-title":"Proceedings of the European Conference on Computer Vision","author":"Nitin Bhagoji Arjun","year":"2018","unstructured":"Arjun Nitin Bhagoji, Warren He, Bo Li, and Dawn Song. 2018. Practical black-box attacks on deep neural networks using efficient query mechanisms. In Proceedings of the European Conference on Computer Vision. Springer Science+Business Media, Germany."},{"key":"e_1_3_2_9_2","first-page":"12","volume-title":"Proceedings of the 37th International Conference on Machine Learning (ICML\u201920)","volume":"84","author":"Bielik Pavol","year":"2020","unstructured":"Pavol Bielik and Martin Vechev. 2020. Adversarial robustness for code. In Proceedings of the 37th International Conference on Machine Learning (ICML\u201920). JMLR.org, Article 84, 12 pages."},{"key":"e_1_3_2_10_2","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Brendel Wieland","year":"2018","unstructured":"Wieland Brendel, Jonas Rauber, and Matthias Bethge. 2018. Decision-based adversarial attacks: Reliable attacks against black-box machine learning models. In Proceedings of the International Conference on Learning Representations. OpenReview.net."},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2017.49"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510163"},{"key":"e_1_3_2_13_2","first-page":"2196","volume-title":"Proceedings of the International Conference on Machine Learning","volume":"119","author":"Croce Francesco","year":"2020","unstructured":"Francesco Croce and Matthias Hein. 2020. Minimally distorted adversarial examples with a fast adaptive boundary attack. In Proceedings of the International Conference on Machine Learning. Hal Daum\u00e9 III and Aarti Singh (Eds.), Proceedings of Machine Learning Research, Vol. 119, PMLR, 2196\u20132205. Retrieved from https:\/\/proceedings.mlr.press\/v119\/croce20a.html"},{"key":"e_1_3_2_14_2","unstructured":"Zeming Dong Qiang Hu Xiaofei Xie Maxime Cordy Mike Papadakis and Jianjun Zhao. 2024. Importance guided data augmentation for neural-based code understanding. arXiv:2402.15769. Retrieved from http:\/\/arxiv.org\/abs\/2402.15769"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-2006"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.findings-emnlp.139"},{"key":"e_1_3_2_17_2","unstructured":"GitHub Inc. 2021. GitHub Copilot Your AI pair programmer. Accessed December 30 2024 from https:\/\/github.com\/features\/copilot"},{"key":"e_1_3_2_18_2","unstructured":"Ian J. Goodfellow Jonathon Shlens and Christian Szegedy. 2015. Explaining and harnessing adversarial examples. arXiv:1412.6572. Retrieved from https:\/\/arxiv.org\/abs\/1412.6572"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180167"},{"key":"e_1_3_2_20_2","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Guo Daya","year":"2021","unstructured":"Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, et al. 2021. GraphCodeBERT: Pre-training code representations with data flow. In Proceedings of the International Conference on Learning Representations. OpenReview.net."},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/3511598"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-NIER58687.2023.00007"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/3543507.3583348"},{"key":"e_1_3_2_24_2","unstructured":"huggingface. 2016. Hugging Face \u2013 The AI community building the future. Accessed December 30 2024 from https:\/\/huggingface.coAccessed"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1609\/AAAI.V37I12.26739"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i05.6311"},{"key":"e_1_3_2_27_2","volume-title":"Proceedings of the International Conference on Machine Learning (ICML)","author":"Koh Pang Wei","year":"2021","unstructured":"Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, and Irena Gao. 2021. WILDS: A benchmark of in-the-wild distribution shifts. In Proceedings of the International Conference on Machine Learning (ICML)."},{"key":"e_1_3_2_28_2","volume-title":"ICLR Workshop","author":"Kurakin Alexey","year":"2017","unstructured":"Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. 2017. Adversarial examples in the physical world. In ICLR Workshop."},{"key":"e_1_3_2_29_2","volume-title":"Proceedings of the 6th International Conference on Learning Representations (ICLR\u201918)","author":"Madry Aleksander","year":"2018","unstructured":"Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2018. Towards deep learning models resistant to adversarial attacks. In Proceedings of the 6th International Conference on Learning Representations (ICLR\u201918). OpenReview.net. Retrieved from https:\/\/openreview.net\/forum?id=rJzIBfZAb"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2022\/775"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.nlp4prog-1.5"},{"key":"e_1_3_2_32_2","unstructured":"Ruchir Puri David S. Kung Geert Janssen Wei Zhang Giacomo Domeniconi Vladimir Zolotov Julian Dolby Jie Chen Mihir Choudhury Lindsey Decker et al. 2021. CodeNet: A large-scale AI for code dataset for learning a diversity of coding tasks. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks. Retrieved from https:\/\/datasets-benchmarks-proceedings.neurips.cc\/paper\/2021\/hash\/a5bfc9e07964f8dddeb95fc584cd965d-Abstract-round2.html"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1103"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1007\/S10664-020-09881-0"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510548.3519373"},{"key":"e_1_3_2_36_2","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Srikant Shashank","year":"2021","unstructured":"Shashank Srikant, Sijia Liu, Tamara Mitrovska, Shiyu Chang, Quanfu Fan, Gaoyuan Zhang, and Una-May O\u2019Reilly. 2021. Generating adversarial computer programs using optimized obfuscations. In Proceedings of the International Conference on Learning Representations. Retrieved from https:\/\/openreview.net\/forum?id=PH5PH9ZO_4"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME.2014.77"},{"key":"e_1_3_2_38_2","volume-title":"Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020 (NeurIPS \u201920)","author":"Tram\u00e8r Florian","year":"2020","unstructured":"Florian Tram\u00e8r, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. 2020. On adaptive attacks to adversarial example defenses. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020 (NeurIPS \u201920). Hugo Larochelle, Marc\u2019Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (Eds.). Retrieved from https:\/\/proceedings.neurips.cc\/paper\/2020\/hash\/11f38f8ecd71867b42433548d1078e38-Abstract.html"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/AITEST58265.2023.00014"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/3639474.3640069"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.685"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.4236\/jsea.2024.1711044"},{"key":"e_1_3_2_43_2","unstructured":"Jin Wen. 2023. Black-box variable renaming attack for code model: Benchmark and enhancement. Retrieved December 30 2024 from https:\/\/sites.google.com\/view\/variable-attack-for-code-model"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/2970276.2970326"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510146"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/3428230"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.findings-emnlp.81"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i01.5469"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/2023.FINDINGS-EMNLP.649"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2019.2962027"},{"key":"e_1_3_2_51_2","first-page":"11","volume-title":"Proceedings of the 33rd International Conference on Neural Information Processing Systems","volume":"915","author":"Zhou Yaqin","year":"2019","unstructured":"Yaqin Zhou, Shangqing Liu, Jingkai Siow, Xiaoning Du, and Yang Liu. 2019. Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks. In Proceedings of the 33rd International Conference on Neural Information Processing Systems. Curran Associates Inc., Red Hook, NY, Article 915, 11 pages."}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3723353","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,12,11]],"date-time":"2025-12-11T15:56:52Z","timestamp":1765468612000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3723353"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,12,11]]},"references-count":50,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2026,1,31]]}},"alternative-id":["10.1145\/3723353"],"URL":"https:\/\/doi.org\/10.1145\/3723353","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,12,11]]},"assertion":[{"value":"2023-12-20","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-03-05","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-12-11","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}