{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,3]],"date-time":"2025-10-03T17:52:46Z","timestamp":1759513966003,"version":"3.41.0"},"reference-count":130,"publisher":"Association for Computing Machinery (ACM)","issue":"6","license":[{"start":{"date-parts":[[2023,9,29]],"date-time":"2023-09-29T00:00:00Z","timestamp":1695945600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2023,11,30]]},"abstract":"<jats:p>AI models of code have made significant progress over the past few years. However, many models are actually not learning task-relevant source code features. Instead, they often fit non-relevant but correlated data, leading to a lack of robustness and generalizability, and limiting the subsequent practical use of such models. In this work, we focus on improving the model quality through<jats:italic>signal awareness<\/jats:italic>, i.e., learning the relevant signals in the input for making predictions. We do so by leveraging the heterogeneity of code samples in terms of their signal-to-noise content. We perform an end-to-end exploration of model signal awareness, comprising: (i) uncovering the reliance of AI models of code on task-irrelevant signals, via prediction-preserving input minimization; (ii) improving models\u2019 signal awareness by incorporating the notion of code complexity during model training, via curriculum learning; (iii) improving models\u2019 signal awareness by generating simplified signal-preserving programs and augmenting them to the training dataset; and (iv) presenting a novel interpretation of the model learning behavior from the perspective of the dataset, using its code complexity distribution. We propose a new metric to measure model signal awareness, Signal-aware Recall, which captures how much of the model\u2019s performance is attributable to task-relevant signal learning. Using a software vulnerability detection use-case, our model probing approach uncovers a significant lack of signal awareness in the models, across three different neural network architectures and three datasets. Signal-aware Recall is observed to be in the sub-50s for models with traditional Recall in the high 90s, suggesting that the models are presumably picking up a lot of noise or dataset nuances while learning their logic. With our code-complexity-aware model learning enhancement techniques, we are able to assist the models toward more task-relevant learning, recording up-to 4.8\u00d7 improvement in model signal awareness. Finally, we employ our model learning introspection approach to uncover the aspects of source code where the model is facing difficulty, and we analyze how our learning enhancement techniques alleviate it.<\/jats:p>","DOI":"10.1145\/3597202","type":"journal-article","created":{"date-parts":[[2023,5,16]],"date-time":"2023-05-16T12:05:33Z","timestamp":1684238733000},"page":"1-40","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Incorporating Signal Awareness in Source Code Modeling: An Application to Vulnerability Detection"],"prefix":"10.1145","volume":"32","author":[{"ORCID":"https:\/\/orcid.org\/0009-0005-5094-5779","authenticated-orcid":false,"given":"Sahil","family":"Suneja","sequence":"first","affiliation":[{"name":"IBM Research T.J. Watson, NY"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4063-6460","authenticated-orcid":false,"given":"Yufan","family":"Zhuang","sequence":"additional","affiliation":[{"name":"University of California, San Diego"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6794-3199","authenticated-orcid":false,"given":"Yunhui","family":"Zheng","sequence":"additional","affiliation":[{"name":"IBM Research T.J. Watson, NY"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4915-0304","authenticated-orcid":false,"given":"Jim","family":"Laredo","sequence":"additional","affiliation":[{"name":"IBM Research T.J. Watson, NY"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-5006-8817","authenticated-orcid":false,"given":"Alessandro","family":"Morari","sequence":"additional","affiliation":[{"name":"IBM Research T.J. Watson, NY"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8113-1210","authenticated-orcid":false,"given":"Udayan","family":"Khurana","sequence":"additional","affiliation":[{"name":"IBM Research T.J. Watson, NY"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2023,9,29]]},"reference":[{"key":"e_1_3_4_2_2","volume-title":"Proceedings of the NeurIPS","author":"Adebayo Julius","year":"2018","unstructured":"Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. 2018. Sanity checks for saliency maps. In Proceedings of the NeurIPS."},{"key":"e_1_3_4_3_2","volume-title":"Proceedings of the ACL","author":"Ahmad Wasi","year":"2021","unstructured":"Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. Unified pre-training for program understanding and generation. In Proceedings of the ACL."},{"key":"e_1_3_4_4_2","volume-title":"Proceedings of the ACM SPLASH Onward!","author":"Allamanis M.","year":"2019","unstructured":"M. Allamanis. 2019. The adverse effects of code duplication in machine learning models of code. In Proceedings of the ACM SPLASH Onward!"},{"key":"e_1_3_4_5_2","volume-title":"Proceedings of the FSE","author":"Allamanis M.","year":"2015","unstructured":"M. Allamanis, E. Barr, C. Bird, and C. Sutton. 2015. Suggesting accurate method and class names. In Proceedings of the FSE."},{"key":"e_1_3_4_6_2","unstructured":"M. Allamanis M. Brockschmidt and M. Khademi. 2018. Learning to represent programs with graphs. InProceedings of the ICLR."},{"key":"e_1_3_4_7_2","volume-title":"Proceedings of the ICML","author":"Allamanis M.","year":"2016","unstructured":"M. Allamanis, H. Peng, and C. Sutton. 2016. A convolutional attention network for extreme summarization of source code. In Proceedings of the ICML."},{"key":"e_1_3_4_8_2","article-title":"AI in software engineering at facebook","author":"Bader Johannes","year":"2021","unstructured":"Johannes Bader, Sonia Seohyun Kim, Frank Sifei Luan, Satish Chandra, and Erik Meijer. 2021. AI in software engineering at facebook. IEEE Softw. 38, 4 (2021), 52\u201361.","journal-title":"IEEE Softw."},{"key":"e_1_3_4_9_2","doi-asserted-by":"crossref","unstructured":"Abdul Ali Bangash Hareem Sahar Abram Hindle and Karim Ali. 2020. On the time-based conclusion stability of cross-project defect prediction models. Empirical Software Engineering 25 6 (2020) 5047\u20135083.","DOI":"10.1007\/s10664-020-09878-9"},{"key":"e_1_3_4_10_2","unstructured":"Rohan Bavishi Michael Pradel and Koushik Sen. 2018. Context2Name: A Deep Learning-Based Approach to Infer Natural Variable Names from Usage Contexts. Retrieved from https:\/\/arXiv:1809.05193."},{"key":"e_1_3_4_11_2","volume-title":"Proceedings of the ICML","author":"Bengio Y.","year":"2009","unstructured":"Y. Bengio, J. Louradour, R. Collobert, and J. Weston. 2009. Curriculum learning. In Proceedings of the ICML."},{"key":"e_1_3_4_12_2","doi-asserted-by":"crossref","unstructured":"Amel Bennaceur and Karl Meinke. 2018. Machine learning for software analysis: models methods and applicationsLecture Notes in Computer Science.","DOI":"10.1007\/978-3-319-96562-8_1"},{"key":"e_1_3_4_13_2","volume-title":"Proceedings of the ICML","author":"Bielik Pavol","year":"2020","unstructured":"Pavol Bielik and Martin T. Vechev. 2020. Adversarial robustness for code. In Proceedings of the ICML."},{"key":"e_1_3_4_14_2","volume-title":"Proceedings of the FSE","author":"Brown David Bingham","year":"2017","unstructured":"David Bingham Brown, Michael Vaughn, Ben Liblit, and Thomas Reps. 2017. The care and feeding of wild-caught mutants. In Proceedings of the FSE."},{"key":"e_1_3_4_15_2","volume-title":"Proceedings of the NeurIPS","author":"Brown T.","year":"2020","unstructured":"T. Brown, B. Mann, N. Ryder et\u00a0al. 2020. Language models are few-shot learners. In Proceedings of the NeurIPS."},{"key":"e_1_3_4_16_2","unstructured":"Alfredo Canziani Adam Paszke and Eugenio Culurciello. 2016. An Analysis of Deep Neural Network Models for Practical Applications. Retrieved from https\/\/arXiv:1605.07678."},{"key":"e_1_3_4_17_2","volume-title":"Proceedings of the S&P","author":"Carlini Nicholas","year":"2017","unstructured":"Nicholas Carlini and David Wagner. 2017. Towards evaluating the robustness of neural networks. In Proceedings of the S&P."},{"key":"e_1_3_4_18_2","volume-title":"Proceedings of the ACM FSE","author":"Chakraborty J.","year":"2021","unstructured":"J. Chakraborty, S. Majumder, and T. Menzies. 2021. Bias in machine learning software: Why? how? what to do? In Proceedings of the ACM FSE."},{"key":"e_1_3_4_19_2","unstructured":"Saikat Chakraborty Rahul Krishna Yangruibo Ding and Baishakhi Ray. 2020. Deep learning based vulnerability detection: Are we there yet? Retrieved from https:\/\/arXiv:2009.07235."},{"key":"e_1_3_4_20_2","doi-asserted-by":"crossref","unstructured":"N. Chawla K. Bowyer L. Hall and W. Kegelmeyer. 2002. SMOTE: Synthetic minority over-sampling technique. Journal of Artificial Intelligence Research 16 (2002) 321\u2013357.","DOI":"10.1613\/jair.953"},{"key":"e_1_3_4_21_2","article-title":"Evaluating large language models trained on code","author":"Chen Mark","year":"2021","unstructured":"Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman et\u00a0al. 2021. Evaluating large language models trained on code. Retrieved from https:\/\/arXiv:2107.03374.","journal-title":"Retrieved from https:\/\/arXiv:2107.03374"},{"key":"e_1_3_4_22_2","unstructured":"Tsong Yueh Chen S. C. Cheung and Siu-Ming Yiu. 2020. Metamorphic testing: A new approach for generating next test cases. Retrieved from https:\/\/arXiv:2002.12543."},{"key":"e_1_3_4_23_2","unstructured":"Junyoung Chung Caglar Gulcehre KyungHyun Cho and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. Retrieved from https:\/\/arXiv:1412.3555."},{"key":"e_1_3_4_24_2","volume-title":"Proceedings of the MSR","author":"Ciniselli Matteo","year":"2021","unstructured":"Matteo Ciniselli, Nathan Cooper, Luca Pascarella, Denys Poshyvanyk, Massimiliano Di Penta, and Gabriele Bavota. 2021. An empirical study on the usage of bert models for code completion. In Proceedings of the MSR."},{"key":"e_1_3_4_25_2","volume-title":"Proceedings of the ICSE-SEIP","author":"Cito J\u00fcrgen","year":"2022","unstructured":"J\u00fcrgen Cito, Isil Dillig, Vijayaraghavan Murali, and Satish Chandra. 2022. Counterfactual explanations for models of code. In Proceedings of the ICSE-SEIP."},{"key":"e_1_3_4_26_2","doi-asserted-by":"crossref","first-page":"233","DOI":"10.1007\/978-3-642-33826-7_16","volume-title":"Proceedings of the International Conference on Software Engineering and Formal Methods","author":"Cuoq Pascal","year":"2012","unstructured":"Pascal Cuoq, Florent Kirchner, Nikolai Kosmatov, Virgile Prevosto, Julien Signoles, and Boris Yakobowski. 2012. Frama-c. In Proceedings of the International Conference on Software Engineering and Formal Methods. Springer, 233\u2013247."},{"key":"e_1_3_4_27_2","article-title":"Automatic feature learning for predicting vulnerable software components","author":"Dam Hoa Khanh","year":"2021","unstructured":"Hoa Khanh Dam, Truyen Tran, Trang Pham, Shien Wee Ng, John Grundy, and Aditya Ghose. 2021. Automatic feature learning for predicting vulnerable software components. In Proceedings of the TSE.","journal-title":"Proceedings of the TSE"},{"key":"e_1_3_4_28_2","unstructured":"Arun Das and Paul Rad. 2020. Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey. Retrieved from https:\/\/arXiv:2006.11371."},{"key":"e_1_3_4_29_2","unstructured":"Jacob Devlin Ming-Wei Chang Kenton Lee and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. Retrieved from https:\/\/arXiv:1810.04805."},{"key":"e_1_3_4_30_2","volume-title":"Proceedings of the IEEE S&P","author":"Dolan-Gavitt Brendan","year":"2016","unstructured":"Brendan Dolan-Gavitt, Patrick Hulin, Engin Kirda et\u00a0al. 2016. LAVA: Large-scale automated vulnerability addition. In Proceedings of the IEEE S&P."},{"key":"e_1_3_4_31_2","unstructured":"Melissa Elliott. 2021. Risk Assessment of GitHub Copilot. Retrieved from https:\/\/gist.github.com\/0xabad1dea\/be18e11beb2e12433d93475d72016902."},{"key":"e_1_3_4_32_2","volume-title":"Proceedings of the PLMR","author":"Engstrom Logan","year":"2019","unstructured":"Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. 2019. Exploring the landscape of spatial robustness. In Proceedings of the PLMR."},{"key":"e_1_3_4_33_2","unstructured":"Facebook. 2015. Infer Static Analyzer. Retrieved from https:\/\/fbinfer.com\/."},{"key":"e_1_3_4_34_2","doi-asserted-by":"crossref","unstructured":"Zhangyin Feng Daya Guo Duyu Tang Nan Duan Xiaocheng Feng Ming Gong Linjun Shou Bing Qin Ting Liu Daxin Jiang et\u00a0al. 2020. Codebert: A pre-trained model for programming and natural languages. Retrieved from https:\/\/arXiv:2002.08155.","DOI":"10.18653\/v1\/2020.findings-emnlp.139"},{"key":"e_1_3_4_35_2","unstructured":"Aayush Garg Renzo Degiovanni Matthieu Jimenez Maxime Cordy Mike Papadakis and Yves Le Traon. 2020. Learning To predict vulnerabilities from vulnerability-fixes: A machine translation approach. Retrieved from https:\/\/arXiv:2012.11701."},{"key":"e_1_3_4_36_2","volume-title":"Proceedings of the ICML","author":"Gilmer Justin","year":"2017","unstructured":"Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. 2017. Neural message passing for quantum chemistry. In Proceedings of the ICML."},{"key":"e_1_3_4_37_2","unstructured":"GitHub Copolit. 2021. Your AI pair programmer. Retrieved from https:\/\/copilot.github.com\/."},{"key":"e_1_3_4_38_2","volume-title":"Proceedings of the ICST","author":"Golagha Mojdeh","year":"2020","unstructured":"Mojdeh Golagha, Alexander Pretschner, and Lionel C. Briand. 2020. Can we predict the quality of spectrum-based fault localization? In Proceedings of the ICST."},{"key":"e_1_3_4_39_2","volume-title":"Proceedings of the ICLR","author":"Goodfellow I.","year":"2015","unstructured":"I. Goodfellow, J. Shlens, and C. Szegedy. 2015. Explaining and harnessing adversarial examples. In Proceedings of the ICLR."},{"key":"e_1_3_4_40_2","unstructured":"Ian J. Goodfellow Jonathon Shlens and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. Retrieved from https:\/\/arXiv:1412.6572."},{"key":"e_1_3_4_41_2","volume-title":"Proceedings of the ICML","author":"Graves Alex","year":"2017","unstructured":"Alex Graves, Marc G. Bellemare, Jacob Menick, Remi Munos, and K. Kavukcuoglu. 2017. Automated curriculum learning for neural networks. In Proceedings of the ICML."},{"key":"e_1_3_4_42_2","volume-title":"Proceedings of the ASE","author":"Gros David","year":"2020","unstructured":"David Gros, Hariharan Sezhiyan, Prem Devanbu, and Zhou Yu. 2020. Code to Comment \u201cTranslation\u201d: Data, metrics, baselining & Evaluation. In Proceedings of the ASE."},{"key":"e_1_3_4_43_2","volume-title":"Proceedings of the CCS","author":"Guo Wenbo","year":"2018","unstructured":"Wenbo Guo, Dongliang Mu, Jun Xu, Purui Su, Gang Wang, and Xinyu Xing. 2018. Lemna: Explaining deep learning based security applications. In Proceedings of the CCS."},{"key":"e_1_3_4_44_2","volume-title":"Proceedings of the ASE","author":"Gupta N.","year":"2005","unstructured":"N. Gupta, H. He, X. Zhang, and R. Gupta. 2005. Locating faulty code using failure-inducing chops. In Proceedings of the ASE."},{"key":"e_1_3_4_45_2","volume-title":"Proceedings of the ICML","author":"Hacohen G.","year":"2019","unstructured":"G. Hacohen and D. Weinshall. 2019. On The power of curriculum learning in training deep networks. In Proceedings of the ICML."},{"key":"e_1_3_4_46_2","unstructured":"Murics H. Halstead. 1977. Elements of Software Science (Operating and programming systems series) . Elsevier Science Inc."},{"key":"e_1_3_4_47_2","volume-title":"Proceedings of the ESORICS","author":"Hassanshahi Behnaz","year":"2015","unstructured":"Behnaz Hassanshahi, Yaoqi Jia, Roland H. C. Yap, Prateek Saxena, and Zhenkai Liang. 2015. Web-to-application injection attacks on android: Characterization and detection. In Proceedings of the ESORICS."},{"key":"e_1_3_4_48_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_3_4_49_2","volume-title":"Proceedings of the ICSE","author":"He Pinjia","year":"2020","unstructured":"Pinjia He, Clara Meister, and Zhendong Su. 2020. Structure-invariant testing for machine translation. In Proceedings of the ICSE."},{"key":"e_1_3_4_50_2","volume-title":"Proceedings of the PPoPP","author":"Hestness Joel","year":"2019","unstructured":"Joel Hestness, Newsha Ardalani, and Gregory F. Diamos. 2019. Beyond human-level accuracy: Computational challenges in deep learning. In Proceedings of the PPoPP."},{"key":"e_1_3_4_51_2","doi-asserted-by":"crossref","unstructured":"Sepp Hochreiter and J\u00fcrgen Schmidhuber. 1997. Long short-term memory. Neural Comput 9 8 (1997) 1735\u20131780.","DOI":"10.1162\/neco.1997.9.8.1735"},{"key":"e_1_3_4_52_2","volume-title":"Proceedings of the ACL","author":"Iyer S.","year":"2016","unstructured":"S. Iyer, I. Konstas, A. Cheung, and L. Zettlemoyer. 2016. Summarizing source code using a neural attention model. In Proceedings of the ACL."},{"key":"e_1_3_4_53_2","volume-title":"Proceedings of the NAACL","author":"Jain Sarthak","year":"2019","unstructured":"Sarthak Jain and Byron C. Wallace. 2019. Attention is not explanation. In Proceedings of the NAACL."},{"key":"e_1_3_4_54_2","volume-title":"Proceedings of the FSE","author":"Jimenez M.","year":"2019","unstructured":"M. Jimenez, R. Rwemalika, M. Papadakis, F. Sarro, Y. Le Traon, and M. Harman. 2019. The importance of accounting for real-world labelling when predicting software vulnerabilities. In Proceedings of the FSE."},{"key":"e_1_3_4_55_2","doi-asserted-by":"crossref","DOI":"10.1038\/s41524-019-0248-2","article-title":"Reliable and explainable machine-learning methods for accelerated material discovery","author":"Kailkhura Bhavya","year":"2019","unstructured":"Bhavya Kailkhura, Brian Gallagher, Sookyung Kim, Anna Hiszpanski, and T. Yong-Jin Han. 2019. Reliable and explainable machine-learning methods for accelerated material discovery. NPJ Comput. Mater. 5, 1 (2019), 108.","journal-title":"NPJ Comput. Mater."},{"key":"e_1_3_4_56_2","volume-title":"Proceedings of the ICML","author":"Kanade Aditya","year":"2020","unstructured":"Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. 2020. Learning and evaluating contextual embedding of source code. In Proceedings of the ICML."},{"key":"e_1_3_4_57_2","volume-title":"Proceedings of the AAAI","author":"Kerber Randy","year":"1992","unstructured":"Randy Kerber. 1992. Chimerge: Discretization of numeric attributes. In Proceedings of the AAAI."},{"key":"e_1_3_4_58_2","volume-title":"Proceedings of the ICSE","author":"Kim Seohyun","year":"2021","unstructured":"Seohyun Kim, Jinman Zhao, Yuchi Tian, and Satish Chandra. 2021. Code prediction by feeding trees to transformers. In Proceedings of the ICSE."},{"key":"e_1_3_4_59_2","unstructured":"Diederik P. Kingma and Jimmy Ba. 2017. Adam: A Method for Stochastic Optimization. REtrieved from https:\/\/arXiv:1412.6980."},{"key":"e_1_3_4_60_2","unstructured":"Thomas N. Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. Retrieved from https:\/\/arXiv:1609.02907."},{"key":"e_1_3_4_61_2","volume-title":"Proceedings of the MAPL","author":"Koc Ugur","year":"2017","unstructured":"Ugur Koc, Parsa Saadatpanah, Jeffrey S. Foster, and Adam A. Porter. 2017. Learning a classifier for false positive error reports emitted by static code analysis tools. In Proceedings of the MAPL."},{"key":"e_1_3_4_62_2","article-title":"Imagenet classification with deep convolutional neural networks","author":"Krizhevsky Alex","year":"2012","unstructured":"Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Proceedings of the NeurIPS.","journal-title":"Proceedings of the NeurIPS"},{"key":"e_1_3_4_63_2","doi-asserted-by":"crossref","unstructured":"Alexey Kurakin Ian Goodfellow and Samy Bengio. 2017. Adversarial examples in the physical world. Retrieved from https:\/\/arXiv:1607.02533.","DOI":"10.1201\/9781351251389-8"},{"key":"e_1_3_4_64_2","volume-title":"Proceedings of the ICLR","author":"Laidlaw Cassidy","year":"2020","unstructured":"Cassidy Laidlaw, Sahil Singla, and Soheil Feizi. 2020. Perceptual adversarial robustness: Defense against unseen threat models. In Proceedings of the ICLR."},{"key":"e_1_3_4_65_2","volume-title":"Proceedings of the ICPC","author":"LeClair Alexander","year":"2020","unstructured":"Alexander LeClair, Sakib Haque, Lingfei Wu, and Collin McMillan. 2020. Improved code summarization via a graph neural network. In Proceedings of the ICPC."},{"key":"e_1_3_4_66_2","volume-title":"Proceedings of the ACL","author":"Lewis M.","year":"2020","unstructured":"M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the ACL."},{"key":"e_1_3_4_67_2","article-title":"Code completion with neural attention and pointer networks","author":"Li J.","year":"2018","unstructured":"J. Li, Y. Wang, M. Lyu, and I. King. 2018. Code completion with neural attention and pointer networks. In Proceedings of the IJCAI.","journal-title":"Proceedings of the IJCAI"},{"key":"e_1_3_4_68_2","volume-title":"Proceedings of the ICLR","author":"Li Y.","year":"2016","unstructured":"Y. Li, D. Tarlow, M. Brockschmidt, and R. Zemel. 2016. Gated graph sequence networks. In Proceedings of the ICLR."},{"key":"e_1_3_4_69_2","unstructured":"Zhen Li Deqing Zou Shouhuai Xu Hai Jin Yawei Zhu and Zhaoxuan Chen. 2018. SySeVR: A Framework for Using Deep Learning to Detect Software Vulnerabilities. Retrieved from https:\/\/arXiv:1807.06756."},{"key":"e_1_3_4_70_2","volume-title":"Proceedings of the NDSS","author":"Li Z.","year":"2018","unstructured":"Z. Li, D. Zou, S. Xu, X. Ou, H. Jin, S. Wang, Z. Deng, and Y. Zhong. 2018. VulDeePecker: A Deep learning-based system for vulnerability detection. In Proceedings of the NDSS."},{"key":"e_1_3_4_71_2","volume-title":"Proceedings of the ASE","author":"Liu Fang","year":"2020","unstructured":"Fang Liu, Ge Li, Yunfei Zhao, and Zhi Jin. 2020. Multi-task learning based pre-trained language model for code completion. In Proceedings of the ASE."},{"key":"e_1_3_4_72_2","volume-title":"Proceedings of the ICSE","author":"Liu Kui","year":"2019","unstructured":"Kui Liu, Dongsun Kim, Tegawend\u00e9 F. Bissyand\u00e9, Tae-young Kim, Kisub Kim, Anil Koyuncu, Suntae Kim, and Yves Le Traon. 2019. Learning to spot and refactor inconsistent method names. In Proceedings of the ICSE."},{"key":"e_1_3_4_73_2","unstructured":"Pengfei Liu Xipeng Qiu and Xuanjing Huang. 2016. Recurrent neural network for text classification with multi-task learning. Retrieved from https:\/\/arXiv:1605.05101."},{"key":"e_1_3_4_74_2","unstructured":"Lizard. 2012. A simple code complexity analyser. Retrieved from https:\/\/github.com\/terryyin\/lizard."},{"key":"e_1_3_4_75_2","volume-title":"Proceedings of the FSE","author":"Lou Yiling","year":"2021","unstructured":"Yiling Lou, Qihao Zhu, Jinhao Dong, Xia Li, Zeyu Sun, Dan Hao, Lu Zhang, and Lingming Zhang. 2021. Boosting coverage-based fault localization via graph-based representation learning. In Proceedings of the FSE."},{"key":"e_1_3_4_76_2","volume-title":"Proceedings of the NIPS","author":"Lundberg S.","year":"2017","unstructured":"S. Lundberg and S. Lee. 2017. A unified approach to interpreting model predictions. In Proceedings of the NIPS."},{"key":"e_1_3_4_77_2","volume-title":"Proceedings of the ICLR","author":"Madry A.","year":"2018","unstructured":"A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. 2018. Towards deep learning models resistant to adversarial attacks. In Proceedings of the ICLR."},{"key":"e_1_3_4_78_2","volume-title":"Proceedings of the MVA","author":"Meshgi Kourosh","year":"2015","unstructured":"Kourosh Meshgi and Shin Ishii. 2015. Expanding histogram of colors with gridding to improve tracking accuracy. In Proceedings of the MVA."},{"key":"e_1_3_4_79_2","volume-title":"Proceedings of the ICST","author":"Mirshokraie S.","year":"2013","unstructured":"S. Mirshokraie, A. Mesbah, and K. Pattabiraman. 2013. In Proceedings of the ICST."},{"key":"e_1_3_4_80_2","volume-title":"Proceedings of the ICSE","author":"Misherghi Ghassan","year":"2006","unstructured":"Ghassan Misherghi and Zhendong Su. 2006. HDD: Hierarchical delta debugging. In Proceedings of the ICSE."},{"key":"e_1_3_4_81_2","volume-title":"Proceedings of the CVPR","author":"Moosavi-Dezfooli Seyed-Mohsen","year":"2016","unstructured":"Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. 2016. DeepFool: A simple and accurate method to fool deep neural networks. In Proceedings of the CVPR."},{"key":"e_1_3_4_82_2","volume-title":"Proceedings of the ICSE","author":"Nguyen Phuong T.","year":"2019","unstructured":"Phuong T. Nguyen, Juri Di Rocco, Davide Di Ruscio, Lina Ochoa, Thomas Degueule, and Massimiliano Di Penta. 2019. FOCUS: A recommender system for mining API function calls and usage patterns. In Proceedings of the ICSE."},{"key":"e_1_3_4_83_2","unstructured":"NIST. 2017. Juliet Test Suite for C\/C++ Version 1.3. Retrieved from https:\/\/samate.nist.gov\/SRD\/testsuite.php."},{"key":"e_1_3_4_84_2","volume-title":"Proceedings of the ACL","author":"Panthaplackel Sheena","year":"2020","unstructured":"Sheena Panthaplackel, Pengyu Nie, Milos Gligoric, Junyi Jessy Li, and Raymond J. Mooney. 2020. Learning to update natural language comments based on code changes. In Proceedings of the ACL, Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (Eds.)."},{"key":"e_1_3_4_85_2","doi-asserted-by":"crossref","unstructured":"J. Patra and M. Pradel. 2021. Semantic bug seeding: A learning-based approach for creating realistic bugs. InProceedings of the FSE.","DOI":"10.1145\/3468264.3468623"},{"key":"e_1_3_4_86_2","volume-title":"Proceedings of the S&P","author":"Pearce Hammond","year":"2022","unstructured":"Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2022. Asleep at the keyboard? Assessing the Security of GitHub copilot\u2019s code contributions. In Proceedings of the S&P."},{"key":"e_1_3_4_87_2","doi-asserted-by":"crossref","unstructured":"M. Pradel and K. Sen. 2018. A learning approach to name-based bug detection. InProceedings of the OOPSLA.","DOI":"10.1145\/3276517"},{"key":"e_1_3_4_88_2","volume-title":"Proceedings of the FSE","author":"Rabin M. R. I.","year":"2021","unstructured":"M. R. I. Rabin, V. J. Hellendoorn, and M. A. Alipour. 2021. Understanding neural code intelligence through program simplification. In Proceedings of the FSE."},{"key":"e_1_3_4_89_2","volume-title":"Proceedings of the PLDI","author":"Regehr John","year":"2012","unstructured":"John Regehr, Yang Chen, Pascal Cuoq, Eric Eide, Chucky Ellison, and Xuejun Yang. 2012. Test-case reduction for C compiler bugs. In Proceedings of the PLDI."},{"key":"e_1_3_4_90_2","unstructured":"Shaoqing Ren Kaiming He Ross Girshick and Jian Sun. 2015. Faster R-CNN: Towards real-time object detection with region proposal networks. Retrieved from https:\/\/arXiv:1506.01497."},{"key":"e_1_3_4_91_2","volume-title":"Proceedings of the KDD","author":"Ribeiro M. T.","year":"2016","unstructured":"M. T. Ribeiro, S. Singh, and C. Guestrin. 2016. \u201cWhy should I trust you?\u201d Explaining the predictions of any classifier. In Proceedings of the KDD."},{"key":"e_1_3_4_92_2","volume-title":"Proceedings of the FSE","author":"Roy S.","year":"2018","unstructured":"S. Roy, A. Pandey, B. Dolan-Gavitt, and Y. Hu. 2018. Bug synthesis: Challenging bug-finding tools with deep faults. In Proceedings of the FSE."},{"key":"e_1_3_4_93_2","volume-title":"Proceedings of the ICMLA","author":"Russell R.","year":"2018","unstructured":"R. Russell, L. Kim, L. Hamilton, et\u00a0al. 2018. Automated vulnerability detection in source code using deep representation learning. In Proceedings of the ICMLA."},{"key":"e_1_3_4_94_2","doi-asserted-by":"crossref","unstructured":"Hasim Sak Andrew W. Senior and Fran\u00e7oise Beaufays. 2014. Long short-term memory recurrent neural network architectures for large-scale acoustic modeling. In INTERSPEECH .","DOI":"10.21437\/Interspeech.2014-80"},{"key":"e_1_3_4_95_2","doi-asserted-by":"crossref","unstructured":"R. Selvaraju M. Cogswell A. Das R. Vedantam D. Parikh and D. Batra. 2017. Grad-cam: Visual explanations via gradient-based localization. InProceedings of the ICCV.","DOI":"10.1109\/ICCV.2017.74"},{"key":"e_1_3_4_96_2","unstructured":"C. Sestili W. Snavely and N. VanHoudnos. 2018. Towards security defect prediction with AI. Retrieved from https:\/\/abs\/1808.09897."},{"key":"e_1_3_4_97_2","volume-title":"Proceedings of the ASE","author":"Shar L. K.","year":"2012","unstructured":"L. K. Shar and H. B. K. Tan. 2012. Predicting common web application vulnerabilities from input validation and sanitization code patterns. In Proceedings of the ASE."},{"key":"e_1_3_4_98_2","volume-title":"Proceedings of the CCS","author":"Sharif Mahmood","year":"2016","unstructured":"Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K. Reiter. 2016. Accessorize to a crime: Real and stealthy attacks on state-of-the-art face recognition. In Proceedings of the CCS."},{"key":"e_1_3_4_99_2","doi-asserted-by":"crossref","DOI":"10.1186\/s40537-019-0197-0","article-title":"A survey on image data augmentation for deep learning","author":"Shorten C.","year":"2019","unstructured":"C. Shorten and T. M. Khoshgoftaar. 2019. A survey on image data augmentation for deep learning. J. Big Data (2019).","journal-title":"J. Big Data"},{"key":"e_1_3_4_100_2","volume-title":"Proceedings of the ICML","author":"Shrikumar A.","year":"2017","unstructured":"A. Shrikumar, P. Greenside, and A. Kundaje. 2017. Learning important features through propagating activation differences. In Proceedings of the ICML."},{"key":"e_1_3_4_101_2","unstructured":"Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. Retrieved from https:\/\/arXiv:1409.1556."},{"key":"e_1_3_4_102_2","volume-title":"Proceedings of the ICSE","author":"Sun Chengnian","year":"2018","unstructured":"Chengnian Sun, Yuanbo Li, Qirun Zhang, Tianxiao Gu, and Zhendong Su. 2018. Perses: Syntax-guided program reduction. In Proceedings of the ICSE."},{"key":"e_1_3_4_103_2","unstructured":"M. Sundararajan A. Taly and Q. Yan. 2017. Axiomatic attribution for deep nets Proceedings of the ICML."},{"key":"e_1_3_4_104_2","unstructured":"Sahil Suneja Yunhui Zheng Yufan Zhuang Jim Laredo and Alessandro Morari. 2020. Learning to map source code to software vulnerability using code-as-a-graph. Retrieved from https:\/\/abs\/2006.08614."},{"key":"e_1_3_4_105_2","doi-asserted-by":"crossref","unstructured":"S. Suneja Y. Zheng Y. Zhuang J. Laredo and A. Morari. 2021. Probing model signal-awareness via prediction-preserving input minimization Proceedings of the FSE.","DOI":"10.1145\/3468264.3468545"},{"key":"e_1_3_4_106_2","volume-title":"Proceedings of the SoCC","author":"Suneja Sahil","year":"2021","unstructured":"Sahil Suneja, Yunhui Zheng, Yufan Zhuang, Jim A. Laredo, and Alessandro Morari. 2021. Towards reliable ai for source code understanding. In Proceedings of the SoCC."},{"key":"e_1_3_4_107_2","unstructured":"Christian Szegedy Wojciech Zaremba Ilya Sutskever Joan Bruna Dumitru Erhan Ian Goodfellow and Rob Fergus. 2013. Intriguing properties of neural networks. Retrieved from https:\/\/arXiv:1312.6199."},{"key":"e_1_3_4_108_2","unstructured":"SNYK TEAM. 2020. Accelerating our developer-first vision with DeepCode. Retrieved from https:\/\/snyk.io\/blog\/accelerating-developer-first-vision-with-deepcode\/."},{"key":"e_1_3_4_109_2","article-title":"Security slicing for auditing common injection vulnerabilities","author":"Thom\u00e9 Julian","year":"2018","unstructured":"Julian Thom\u00e9, Lwin Khin Shar, Domenico Bianculli, and Lionel C. Briand. 2018. Security slicing for auditing common injection vulnerabilities. J. Syst. Softw. 137 (2018), 766\u2013783.","journal-title":"J. Syst. Softw."},{"key":"e_1_3_4_110_2","volume-title":"Proceedings of the CCS","author":"Tripp Omer","year":"2014","unstructured":"Omer Tripp, Salvatore Guarnieri, Marco Pistoia, and Aleksandr Aravkin. 2014. ALETHEIA: Improving the usability of static security analysis. In Proceedings of the CCS."},{"key":"e_1_3_4_111_2","volume-title":"Proceedings of the ICSE","author":"Tufano M.","year":"2019","unstructured":"M. Tufano, J. Pantiuchina, C. Watson, G. Bavota, and D. Poshyvanyk. 2019. On learning meaningful code changes via neural machine translation. In Proceedings of the ICSE."},{"key":"e_1_3_4_112_2","doi-asserted-by":"crossref","unstructured":"Michele Tufano Cody Watson Gabriele Bavota Massimiliano Di Penta Martin White and Denys Poshyvanyk. 2019. An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translation. ACM Transactions on Software Engineering and Methodology (TOSEM) 28 4 (2019) 1\u201329.","DOI":"10.1145\/3340544"},{"key":"e_1_3_4_113_2","volume-title":"Proceedings of the ICSME","author":"Tufano M.","year":"2019","unstructured":"M. Tufano, C. Watson, G. Bavota, M. Di Penta, M. White, and D. Poshyvanyk. 2019. Learning how to mutate source code from bug-fixes. In Proceedings of the ICSME."},{"key":"e_1_3_4_114_2","article-title":"Attention is all you need","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the NeurIPS.","journal-title":"Proceedings of the NeurIPS"},{"key":"e_1_3_4_115_2","volume-title":"Proceedings of the ICLR","author":"Veli\u010dkovi\u0107 Petar","year":"2018","unstructured":"Petar Veli\u010dkovi\u0107, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Li\u00f2, and Yoshua Bengio. 2018. Graph attention networks. In Proceedings of the ICLR."},{"issue":"14","key":"e_1_3_4_116_2","article-title":"Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties","volume":"120","author":"Xie Tian","year":"2018","unstructured":"Tian Xie and Jeffrey C. Grossman. 2018. Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties. Phys. Rev. Lett. 120, 14 (2018).","journal-title":"Phys. Rev. Lett."},{"key":"e_1_3_4_117_2","volume-title":"Proceedings of the IEEE SP","author":"Yamaguchi Fabian","year":"2014","unstructured":"Fabian Yamaguchi, Nico Golde, Daniel Arp, and Konrad Rieck. 2014. Modeling and discovering vulnerabilities with code property graphs. In Proceedings of the IEEE SP."},{"key":"e_1_3_4_118_2","unstructured":"Yanming Yang Xin Xia David Lo and John C. Grundy. 2020. A survey on deep learning for software engineering. Retrieved from https:\/\/arXiv:2011.14597."},{"key":"e_1_3_4_119_2","article-title":"Adversarial examples for models of code","author":"Yefet Noam","year":"2020","unstructured":"Noam Yefet, Uri Alon, and Eran Yahav. 2020. Adversarial examples for models of code. OOPSLA (2020).","journal-title":"OOPSLA"},{"key":"e_1_3_4_120_2","unstructured":"Rex Ying Dylan Bourgeois Jiaxuan You Marinka Zitnik and Jure Leskovec. 2019. GNNExplainer: Generating Explanations for Graph Neural Networks. Retrieved from https:\/\/arXiv:1903.03894."},{"key":"e_1_3_4_121_2","doi-asserted-by":"crossref","unstructured":"L. Yu W. Zhang J. Wang and Y. Yu. 2017. Seqgan: Sequence generative adversarial nets with policy gradient. InProceedings of the AAAI.","DOI":"10.1609\/aaai.v31i1.10804"},{"key":"e_1_3_4_122_2","volume-title":"Proceedings of the ICSM","author":"Y\u00fcksel Ulas","year":"2013","unstructured":"Ulas Y\u00fcksel and Hasan S\u00f6zer. 2013. Automated classification of static code analysis alerts: A case study. In Proceedings of the ICSM."},{"key":"e_1_3_4_123_2","volume-title":"Proceedings of the FSE","author":"Zeller Andreas","year":"2002","unstructured":"Andreas Zeller. 2002. Isolating cause-effect chains from computer programs. In Proceedings of the FSE."},{"key":"e_1_3_4_124_2","doi-asserted-by":"crossref","unstructured":"Andreas Zeller and Ralf Hildebrandt. 2002. Simplifying and isolating failure-inducing input. IEEE Transactions on Software Engineering 28 2 (2002) 183\u2013200.","DOI":"10.1109\/32.988498"},{"key":"e_1_3_4_125_2","volume-title":"Proceedings of the ICLR","author":"Zhang H.","year":"2019","unstructured":"H. Zhang, H. Chen, C. Xiao, et\u00a0al. 2019. Towards stable and efficient training of verifiably robust neural networks. In Proceedings of the ICLR."},{"key":"e_1_3_4_126_2","unstructured":"Muhan Zhang and Yixin Chen. 2018. Link prediction based on graph neural networks. Retrieved from https:\/\/arXiv:1802.09691."},{"key":"e_1_3_4_127_2","volume-title":"Proceedings of the FSE","author":"Zhang Zhaowei","year":"2022","unstructured":"Zhaowei Zhang, Hongyu Zhang, Beijun Shen, and Xiaodong Gu. 2022. Diet code is healthy: Simplifying programs for pre-trained models of code. In Proceedings of the FSE."},{"key":"e_1_3_4_128_2","doi-asserted-by":"crossref","unstructured":"Y. Zheng S. Pujar B. Lewis L. Buratti et\u00a0al. 2021. D2A: A dataset built for ai-based vulnerability detection methods using differential analysis. InProceedings of the ICSE-SEIP.","DOI":"10.1109\/ICSE-SEIP52600.2021.00020"},{"key":"e_1_3_4_129_2","volume-title":"Proceedings of the CVPR","author":"Zhou B.","year":"2016","unstructured":"B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. 2016. Learning deep features for discriminative localization. In Proceedings of the CVPR."},{"key":"e_1_3_4_130_2","volume-title":"Proceedings of the NeurIPS","author":"Zhou Y.","year":"2019","unstructured":"Y. Zhou, S. Liu, J. Siow, X. Du, and Y. Liu. 2019. Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks. In Proceedings of the NeurIPS."},{"key":"e_1_3_4_131_2","article-title":"Metamorphic testing of driverless cars","author":"Zhou Zhi Quan","year":"2019","unstructured":"Zhi Quan Zhou and Liqun Sun. 2019. Metamorphic testing of driverless cars. Commun. ACM 62, 3 (2019), 61\u201367.","journal-title":"Commun. ACM"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3597202","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3597202","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T17:49:06Z","timestamp":1750182546000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3597202"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,29]]},"references-count":130,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2023,11,30]]}},"alternative-id":["10.1145\/3597202"],"URL":"https:\/\/doi.org\/10.1145\/3597202","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"type":"print","value":"1049-331X"},{"type":"electronic","value":"1557-7392"}],"subject":[],"published":{"date-parts":[[2023,9,29]]},"assertion":[{"value":"2022-04-07","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-04-19","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-09-29","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}