{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,3]],"date-time":"2026-03-03T12:09:52Z","timestamp":1772539792539,"version":"3.50.1"},"reference-count":45,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2024,12,20]],"date-time":"2024-12-20T00:00:00Z","timestamp":1734652800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>Distinguishing whether some code portions were implemented by humans or generated by a tool based on artificial intelligence has become hard. However, such a classification would be important as it could point developers towards some further validation for the produced code. Additionally, it holds significant importance in security, legal contexts, and educational settings, where upholding academic integrity is of utmost importance. We present EX-CODE, a novel and explainable model that leverages the probability of the occurrence of some tokens, within a code snippet, estimated according to a language model, to distinguish human-written from AI-generated code. EX-CODE has been evaluated on a heterogeneous real-world dataset and stands out for its ability to provide human-understandable explanations of its outcomes. It achieves this by uncovering the features that for a snippet of code make it classified as human-written code (or AI-generated code).<\/jats:p>","DOI":"10.3390\/info15120819","type":"journal-article","created":{"date-parts":[[2024,12,20]],"date-time":"2024-12-20T07:42:18Z","timestamp":1734680538000},"page":"819","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["EX-CODE: A Robust and Explainable Model to Detect AI-Generated Code"],"prefix":"10.3390","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1165-853X","authenticated-orcid":false,"given":"Luana","family":"Bulla","sequence":"first","affiliation":[{"name":"Dipartimento di Matematica e Informatica, University of Catania, 95125 Catania, Italy"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9575-8054","authenticated-orcid":false,"given":"Alessandro","family":"Midolo","sequence":"additional","affiliation":[{"name":"Dipartimento di Matematica e Informatica, University of Catania, 95125 Catania, Italy"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0528-5490","authenticated-orcid":false,"given":"Misael","family":"Mongiov\u00ec","sequence":"additional","affiliation":[{"name":"Dipartimento di Matematica e Informatica, University of Catania, 95125 Catania, Italy"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7169-659X","authenticated-orcid":false,"given":"Emiliano","family":"Tramontana","sequence":"additional","affiliation":[{"name":"Dipartimento di Matematica e Informatica, University of Catania, 95125 Catania, Italy"}]}],"member":"1968","published-online":{"date-parts":[[2024,12,20]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., and Zhang, J.M. (2023, January 14\u201320). Large Language Models for Software Engineering: Survey and Open Problems. Proceedings of the IEEE\/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE), Melbourne, Australia.","DOI":"10.1109\/ICSE-FoSE59343.2023.00008"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Pearce, H., Ahmad, B., Tan, B., Dolan-Gavitt, B., and Karri, R. (2022, January 22\u201326). Asleep at the keyboard? Assessing the security of github copilot\u2019s code contributions. Proceedings of the Symposium on Security and Privacy (SP), San Francisco, CA, USA.","DOI":"10.1109\/SP46214.2022.9833571"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1561\/3300000041","article-title":"Identifying and Mitigating the Security Risks of Generative AI","volume":"6","author":"Barrett","year":"2023","journal-title":"Found. Trends Priv. Secur."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., and Chung, W. (2023, January 1\u20134). A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity. Proceedings of the International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), Nusa Dua, Bali, Indonesia.","DOI":"10.18653\/v1\/2023.ijcnlp-main.45"},{"key":"ref_5","unstructured":"Tian, E., and Cui, A. (2024). GPTZero: Towards detection of AI-generated text using zero-shot and supervised methods. GPTZero, Available online: https:\/\/gptzero.me."},{"key":"ref_6","unstructured":"Mitchell, E., Lee, Y., Khazatsky, A., Manning, C.D., and Finn, C. (2023, January 23\u201329). DetectGPT: Zero-shot machine-generated text detection using probability curvature. Proceedings of the International Conference on Machine Learning (ICML), Honolulu, HI, USA."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Su, J., Zhuo, T.Y., Wang, D., and Nakov, P. (2023). Detectllm: Leveraging log rank information for zero-shot detection of machine-generated text. arXiv.","DOI":"10.18653\/v1\/2023.findings-emnlp.827"},{"key":"ref_8","unstructured":"Zhan, H., He, X., Xu, Q., Wu, Y., and Stenetorp, P. (2023). G3detector: General gpt-generated text detector. arXiv."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Pan, W.H., Chok, M.J., Wong, J.L.S., Shin, Y.X., Poon, Y.S., Yang, Z., Chong, C.Y., Lo, D., and Lim, M.K. (2024). Assessing AI Detectors in Identifying AI-Generated Code: Implications for Education. arXiv.","DOI":"10.1145\/3639474.3640068"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"112059","DOI":"10.1016\/j.jss.2024.112059","article-title":"GPTSniffer: A CodeBERT-based classifier to detect source code written by ChatGPT","volume":"214","author":"Nguyen","year":"2024","journal-title":"J. Syst. Softw."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"100230","DOI":"10.1016\/j.dajour.2023.100230","article-title":"A systematic review of Explainable Artificial Intelligence models and applications: Recent developments and future trends","volume":"7","author":"Saranya","year":"2023","journal-title":"Decis. Anal. J."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"45","DOI":"10.1007\/s12559-023-10179-8","article-title":"Interpreting black-box models: A review on explainable artificial intelligence","volume":"16","author":"Hassija","year":"2024","journal-title":"Cogn. Comput."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"e1391","DOI":"10.1002\/widm.1391","article-title":"A historical perspective of explainable Artificial Intelligence","volume":"11","author":"Confalonieri","year":"2021","journal-title":"WIREs Data Min. Knowl. Discov."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"e1424","DOI":"10.1002\/widm.1424","article-title":"Explainable artificial intelligence: An analytical review","volume":"11","author":"Angelov","year":"2021","journal-title":"WIREs Data Min. Knowl. Discov."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"101603","DOI":"10.1109\/ACCESS.2024.3431437","article-title":"Explainable artificial intelligence for autonomous driving: A comprehensive overview and field guide for future research directions","volume":"12","author":"Atakishiyev","year":"2024","journal-title":"IEEE Access"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"3503","DOI":"10.1007\/s10462-021-10088-y","article-title":"Explainable artificial intelligence: A comprehensive review","volume":"55","author":"Minh","year":"2022","journal-title":"Artif. Intell. Rev."},{"key":"ref_17","unstructured":"Shi, Y., Zhang, H., Wan, C., and Gu, X. (2024). Between Lines of Code: Unraveling the Distinct Patterns of Machine and Human Programmers. arXiv."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., and Jiang, D. (2020). Codebert: A pre-trained model for programming and natural languages. arXiv.","DOI":"10.18653\/v1\/2020.findings-emnlp.139"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Lee, T., Hong, S., Ahn, J., Hong, I., Lee, H., Yun, S., Shin, J., and Kim, G. (2023). Who wrote this code? watermarking for code generation. arXiv.","DOI":"10.18653\/v1\/2024.acl-long.268"},{"key":"ref_20","unstructured":"Yang, X., Zhang, K., Chen, H., Petzold, L., Wang, W.Y., and Cheng, W. (2023). Zero-Shot Detection of Machine-Generated Codes. arXiv."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Sun, Z., Du, X., Song, F., and Li, L. (2023, January 3\u20139). CodeMark: Imperceptible Watermarking for Code Datasets against Neural Code Completion Models. Proceedings of the ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE), San Francisco, CA, USA.","DOI":"10.1145\/3611643.3616297"},{"key":"ref_22","unstructured":"Li, B., Zhang, M., Zhang, P., Sun, J., and Wang, X. (2024). Resilient Watermarking for LLM-Generated Codes. arXiv."},{"key":"ref_23","unstructured":"Bao, G., Zhao, Y., Teng, Z., Yang, L., and Zhang, Y. (2023). Fast-detectgpt: Efficient zero-shot detection of machine-generated text via conditional probability curvature. arXiv."},{"key":"ref_24","unstructured":"Tian, Y., Chen, H., Wang, X., Bai, Z., Zhang, Q., Li, R., Xu, C., and Wang, Y. (2023). Multiscale positive-unlabeled detection of ai-generated texts. arXiv."},{"key":"ref_25","unstructured":"Chen, Y., Kang, H., Zhai, V., Li, L., Singh, R., and Ramakrishnan, B. (2023). Gpt-sentinel: Distinguishing human and chatgpt generated content. arXiv."},{"key":"ref_26","unstructured":"Zeng, C., Tang, S., Yang, X., Chen, Y., Sun, Y., Li, Y., Chen, H., Cheng, W., and Xu, D. (2024). DALD: Improving Logits-based Detector without Logits from Black-box LLMs. arXiv."},{"key":"ref_27","unstructured":"Yang, X., Cheng, W., Wu, Y., Petzold, L., Wang, W.Y., and Chen, H. (2023). DNA-GPT: Divergent N-Gram Analysis for Training-Free Detection of GPT-Generated Text. arXiv."},{"key":"ref_28","unstructured":"Wang, J., Liu, S., Xie, X., and Li, Y. (2023). Evaluating AIGC detectors on code content. arXiv."},{"key":"ref_29","first-page":"27730","article-title":"Training language models to follow instructions with human feedback","volume":"35","author":"Ouyang","year":"2022","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_30","unstructured":"Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., and Anadkat, S. (2023). Gpt-4 technical report. arXiv."},{"key":"ref_31","unstructured":"Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv."},{"key":"ref_32","unstructured":"Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., and Bhosale, S. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv."},{"key":"ref_33","unstructured":"Mistral (2024, November 04). Frontier AI in Your Hands. Available online: https:\/\/mistral.ai."},{"key":"ref_34","unstructured":"OpenAI (2024, November 04). GPT 3.5 Turbo. Available online: https:\/\/openai.com\/blog\/gpt-3-5-turbo-fine-tuning-and-api-updates."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Liesenfeld, A., Lopez, A., and Dingemanse, M. (2023, January 19\u201321). Opening up ChatGPT: Tracking openness, transparency, and accountability in instruction-tuned text generators. Proceedings of the International Conference on Conversational User Interfaces, Eindhoven, The Netherlands.","DOI":"10.1145\/3571884.3604316"},{"key":"ref_36","unstructured":"Husain, H., Wu, H.H., Gazit, T., Allamanis, M., and Brockschmidt, M. (2019). CodeSearchNet challenge: Evaluating the state of semantic code search. arXiv."},{"key":"ref_37","unstructured":"Gokul, Y., Ramalingam, M., Chemmalar, S.G., Supriya, Y., Gautam, S., Praveen, K.R.M., Deepti, R.G., Rutvij, H.J., Prabadevi, B., and Weizheng, W. (2023). Generative Pre-trained Transformer: A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directions. arXiv."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Bauer, J., Siegmund, J., Peitek, N., Hofmeister, J.C., and Apel, S. (2019, January 25). Indentation: Simply a Matter of Style or Support for Program Comprehension?. Proceedings of the IEEE\/ACM International Conference on Program Comprehension (ICPC), Montreal, QC, Canada.","DOI":"10.1109\/ICPC.2019.00033"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Alsuhaibani, R.S., Newman, C.D., Decker, M.J., Collard, M.L., and Maletic, J.I. (2021, January 22\u201330). On the Naming of Methods: A Survey of Professional Developers. Proceedings of the International Conference on Software Engineering (ICSE), Madrid, Spain.","DOI":"10.1109\/ICSE43902.2021.00061"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Steidl, D., Hummel, B., and Juergens, E. (2013, January 20\u201321). Quality analysis of source code comments. Proceedings of the International Conference on Program Comprehension (ICPC), San Francisco, CA, USA.","DOI":"10.1109\/ICPC.2013.6613836"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"230","DOI":"10.1016\/j.infsof.2006.10.017","article-title":"Semantic clustering: Identifying topics in source code","volume":"49","author":"Kuhn","year":"2007","journal-title":"Inf. Softw. Technol."},{"key":"ref_42","unstructured":"Smith, N., Van Bruggen, D., and Tomassetti, F. (2017). Javaparser: Visited, Leanpub."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Hosmer, D.W., Lemeshow, S., and Sturdivant, R.X. (2013). Applied Logistic Regression, John Wiley & Sons.","DOI":"10.1002\/9781118548387"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"366","DOI":"10.1016\/j.eswa.2006.09.004","article-title":"Comparing performances of logistic regression, classification and regression tree, and neural networks for predicting coronary artery disease","volume":"34","author":"Kurt","year":"2008","journal-title":"Expert Syst. Appl."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"1087","DOI":"10.1016\/j.jclinepi.2006.01.014","article-title":"Review: A gentle introduction to imputation of missing values","volume":"59","author":"Donders","year":"2006","journal-title":"J. Clin. Epidemiol."}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/15\/12\/819\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T16:56:12Z","timestamp":1760115372000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/15\/12\/819"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,20]]},"references-count":45,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2024,12]]}},"alternative-id":["info15120819"],"URL":"https:\/\/doi.org\/10.3390\/info15120819","relation":{},"ISSN":["2078-2489"],"issn-type":[{"value":"2078-2489","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,12,20]]}}}