{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,2]],"date-time":"2025-11-02T09:37:02Z","timestamp":1762076222404,"version":"build-2065373602"},"reference-count":31,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2022,11,29]],"date-time":"2022-11-29T00:00:00Z","timestamp":1669680000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>State-of-the-art neural networks build an internal model of the training data, tailored to a given classification task. The study of such a model is of interest, and therefore, research on explainable artificial intelligence (XAI) aims at investigating if, in the internal states of a network, it is possible to identify rules that associate data to their corresponding classification. This work moves toward XAI research on neural networks trained in the classification of source code snippets, in the specific domain of cybersecurity. In this context, typically, textual instances have firstly to be encoded with non-invertible transformation into numerical vectors to feed the models, and this limits the applicability of known XAI methods based on the differentiation of neural signals with respect to real valued instances. In this work, we start from the known TCAV method, designed to study the human understandable concepts that emerge in the internal layers of a neural network, and we adapt it to transformers architectures trained in solving source code classification problems. We first determine domain-specific concepts (e.g., the presence of given patterns in the source code), and for each concept, we train support vector classifiers to separate points in the vector activation spaces that represent input instances with the concept from those without the concept. Then, we study if the presence (or the absence) of such concepts affects the decision process of the neural network. Finally, we discuss about how our approach contributes to general XAI goals and we suggest specific applications in the source code analysis field.<\/jats:p>","DOI":"10.3390\/a15120449","type":"journal-article","created":{"date-parts":[[2022,11,29]],"date-time":"2022-11-29T02:09:58Z","timestamp":1669687798000},"page":"449","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Do Neural Transformers Learn Human-Defined Concepts? An Extensive Study in Source Code Processing Domain"],"prefix":"10.3390","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9582-5045","authenticated-orcid":false,"given":"Claudio","family":"Ferretti","sequence":"first","affiliation":[{"name":"Department of Informatics, Systems and Communication, University of Milano-Bicocca, 20126 Milano, Italy"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7250-8987","authenticated-orcid":false,"given":"Martina","family":"Saletta","sequence":"additional","affiliation":[{"name":"Department of Informatics, Systems and Communication, University of Milano-Bicocca, 20126 Milano, Italy"}]}],"member":"1968","published-online":{"date-parts":[[2022,11,29]]},"reference":[{"key":"ref_1","unstructured":"Kanade, A., Maniatis, P., Balakrishnan, G., and Shi, K. (2020, January 13\u201318). Learning and evaluating contextual embedding of source code. Proceedings of the 37th International Conference on Machine Learning, ICML 2020, Virtual."},{"key":"ref_2","unstructured":"Kim, B., Wattenberg, M., Gilmer, J., Cai, C.J., Wexler, J., Vi\u00e9gas, F.B., and Sayres, R. (2018, January 10\u201315). Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV). Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholm, Sweden."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Saletta, M., and Ferretti, C. (2022, January 9\u201313). Towards the Evolutionary Assessment of Neural Transformers Trained on Source Code. Proceedings of the GECCO \u201922: Genetic and Evolutionary Computation Conference, Companion Volume, Boston, MA, USA.","DOI":"10.1145\/3520304.3534044"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Gosain, A., and Sharma, G. (2015). Static analysis: A survey of techniques and tools. Intelligent Computing and Applications, Springer.","DOI":"10.1007\/978-81-322-2268-2_59"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3212695","article-title":"A Survey of Machine Learning for Big Code and Naturalness","volume":"51","author":"Allamanis","year":"2018","journal-title":"ACM Comput. Surv."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3383458","article-title":"Deep Learning for Source Code Modeling and Generation: Models, Applications, and Challenges","volume":"53","author":"Le","year":"2020","journal-title":"ACM Comput. Surv."},{"key":"ref_7","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017, January 4\u20139). Attention is All you Need. Proceedings of the Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems (NIPS), Long Beach, CA, USA."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Ahmad, W., Chakraborty, S., Ray, B., and Chang, K.W. (2021, January 6\u201311). Unified Pre-training for Program Understanding and Generation. Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online.","DOI":"10.18653\/v1\/2021.naacl-main.211"},{"key":"ref_9","unstructured":"Devlin, J., Chang, M., Lee, K., and Toutanova, K. (2019, January 2\u20137). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, Minneapolis, MN, USA."},{"key":"ref_10","unstructured":"Koza, J.R. (1989, January 20\u201325). Hierarchical Genetic Algorithms Operating on Populations of Computer Programs. Proceedings of the 11th International Joint Conference on Artificial Intelligence, Detroit, MI, USA."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"349","DOI":"10.1109\/4235.942529","article-title":"Grammatical evolution","volume":"5","author":"Ryan","year":"2001","journal-title":"IEEE Trans. Evol. Comput."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Sobania, D., and Rothlauf, F. (2020, January 15\u201317). Challenges of Program Synthesis with Grammatical Evolution. Proceedings of the Genetic Programming\u201423rd European Conference (EuroGP), Seville, Spain.","DOI":"10.1007\/978-3-030-44094-7_14"},{"key":"ref_13","unstructured":"Auger, A., and St\u00fctzle, T. (2019, January 13\u201317). On domain knowledge and novelty to improve program synthesis performance with grammatical evolution. Proceedings of the Genetic and Evolutionary Computation Conference, Prague, Czech Republic."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"O\u2019Neill, M., Nicolau, M., and Agapitos, A. (2014, January 6\u201311). Experiments in program synthesis with grammatical evolution: A focus on Integer Sorting. Proceedings of the IEEE Congress on Evolutionary Computation, Beijing, China.","DOI":"10.1109\/CEC.2014.6900578"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Louren\u00e7o, N., Assun\u00e7\u00e3o, F., Pereira, F.B., Costa, E., and Machado, P. (2018). Structured grammatical evolution: A dynamic approach. Handbook of Grammatical Evolution, Springer.","DOI":"10.1007\/978-3-319-78717-6_6"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"251","DOI":"10.1007\/s10710-015-9262-4","article-title":"Unveiling the properties of structured grammatical evolution","volume":"17","author":"Pereira","year":"2016","journal-title":"Genet. Program. Evol. Mach."},{"key":"ref_17","unstructured":"Doshi-Velez, F., and Kim, B. (2017). Towards a rigorous science of interpretable machine learning. arXiv."},{"key":"ref_18","first-page":"1","article-title":"Visualizing higher-layer features of a deep network","volume":"1341","author":"Erhan","year":"2009","journal-title":"Univ. Montr."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"e7","DOI":"10.23915\/distill.00007","article-title":"Feature visualization","volume":"2","author":"Olah","year":"2017","journal-title":"Distill"},{"key":"ref_20","unstructured":"Nguyen, A.M., Dosovitskiy, A., Yosinski, J., Brox, T., and Clune, J. (2016, January 5\u201310). Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. Proceedings of the Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems, Barcelona, Spain."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Zeiler, M.D., and Fergus, R. (2014, January 6\u201312). Visualizing and Understanding Convolutional Networks. Proceedings of the 13th European Conference of Computer Vision ECCV, Zurich, Switzerland.","DOI":"10.1007\/978-3-319-10590-1_53"},{"key":"ref_22","unstructured":"Shrikumar, A., Greenside, P., and Kundaje, A. (2017, January 6\u201311). Learning Important Features Through Propagating Activation Differences. Proceedings of the 34th International Conference on Machine Learning, ICML, Sydney, Australia."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Z\u00fcgner, D., Akbarnejad, A., and G\u00fcnnemann, S. (2018, January 19\u201323). Adversarial Attacks on Neural Networks for Graph Data. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, London, UK.","DOI":"10.1145\/3219819.3220078"},{"key":"ref_24","unstructured":"Le, Q.V., Ranzato, M., Monga, R., Devin, M., Corrado, G., Chen, K., Dean, J., and Ng, A.Y. (July, January 26). Building high-level features using large scale unsupervised learning. Proceedings of the 29th International Conference on Machine Learning, ICML, Edinburgh, UK."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"McGrath, T., Kapishnikov, A., Toma\u0161ev, N., Pearce, A., Hassabis, D., Kim, B., Paquet, U., and Kramnik, V. (2021). Acquisition of Chess Knowledge in AlphaZero. arXiv.","DOI":"10.1073\/pnas.2206625119"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Boser, B.E., Guyon, I., and Vapnik, V. (1992, January 27\u201329). A Training Algorithm for Optimal Margin Classifiers. Proceedings of the 5th Annual ACM Conference on Computational Learning Theory, COLT, Pittsburgh, PA, USA.","DOI":"10.1145\/130385.130401"},{"key":"ref_27","first-page":"407007","article-title":"Brain-score: Which artificial neural network for object recognition is most brain-like?","volume":"2020","author":"Schrimpf","year":"2020","journal-title":"BioRxiv"},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3505244","article-title":"Transformers in Vision: A Survey","volume":"54","author":"Khan","year":"2021","journal-title":"ACM Comput. Surv."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"88","DOI":"10.1109\/MC.2012.345","article-title":"The Juliet C\/C++ and Java Test Suite","volume":"45","author":"Boland","year":"2012","journal-title":"Comput. IEEE Comput."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Black, P.E. (2018). Juliet 1.3 Test Suite: Changes From 1.2, Technical Note.","DOI":"10.6028\/NIST.TN.1995"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"308","DOI":"10.1109\/TSE.1976.233837","article-title":"A Complexity Measure","volume":"2","author":"McCabe","year":"1976","journal-title":"IEEE Trans. Softw. Eng."}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/15\/12\/449\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T01:28:52Z","timestamp":1760146132000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/15\/12\/449"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,11,29]]},"references-count":31,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2022,12]]}},"alternative-id":["a15120449"],"URL":"https:\/\/doi.org\/10.3390\/a15120449","relation":{},"ISSN":["1999-4893"],"issn-type":[{"type":"electronic","value":"1999-4893"}],"subject":[],"published":{"date-parts":[[2022,11,29]]}}}