{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,23]],"date-time":"2026-08-23T11:21:14Z","timestamp":1787484074507,"version":"build-2736575974"},"reference-count":69,"publisher":"Association for Computing Machinery (ACM)","issue":"FSE","license":[{"start":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:00:00Z","timestamp":1750291200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"funder":[{"DOI":"10.13039\/100010669","name":"H2020 LEIT Information and Communication Technologies","doi-asserted-by":"publisher","award":["952647 (AssureMOSS)"],"award-info":[{"award-number":["952647 (AssureMOSS)"]}],"id":[{"id":"10.13039\/100010669","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100018698","name":"HORIZON EUROPE Civil security for society","doi-asserted-by":"publisher","award":["101120393 (Sec4AI4Sec)"],"award-info":[{"award-number":["101120393 (Sec4AI4Sec)"]}],"id":[{"id":"10.13039\/100018698","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100003246","name":"Dutch Research Council","doi-asserted-by":"crossref","award":["NWA.1215.18.006 (Theseus), KIC1.VE01.20.004 (HEWSTI)"],"award-info":[{"award-number":["NWA.1215.18.006 (Theseus), KIC1.VE01.20.004 (HEWSTI)"]}],"id":[{"id":"10.13039\/501100003246","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Softw. Eng."],"published-print":{"date-parts":[[2025,6,19]]},"abstract":"<jats:p>Vulnerability datasets used for ML testing implicitly contain retrospective information. When tested on the field, one can only use the labels available at the time of training and testing (e.g. seen and assumed negatives). As vulnerabilities are discovered across calendar time, labels change and past performance is not necessarily aligned with future performance. Past works only considered the slices of the whole history (e.g. DiverseVUl) or individual differences between releases (e.g. Jimenez et al. ESEC\/FSE 2019). Such approaches are either too optimistic in training (e.g. the whole history) or too conservative (e.g. consecutive releases). We propose a method to restructure a dataset into a series of datasets in which both training and testing labels change to account for the knowledge available at the time. If the model is actually learning, it should improve its performance over time as more data becomes available and data becomes more stable, an effect that can be checked with the Mann-Kendall test. We validate our methodology for vulnerability detection with 4 time-based datasets (3 projects from BigVul dataset + Vuldeepecker\u2019s NVD) and 5 ML models (Code2Vec, CodeBERT, LineVul, ReGVD, and Vuldeepecker). In contrast to the intuitive expectation (more retrospective information, better performance), the trend results show that performance changes inconsistently across the years, showing that most models are not learning.<\/jats:p>","DOI":"10.1145\/3715731","type":"journal-article","created":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T11:16:02Z","timestamp":1750331762000},"page":"335-357","source":"Crossref","is-referenced-by-count":0,"title":["Today\u2019s Cat Is Tomorrow\u2019s Dog: Accounting for Time-Based Changes in the Labels of ML Vulnerability Detection Approaches"],"prefix":"10.1145","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6682-4243","authenticated-orcid":false,"given":"Ranindya","family":"Paramitha","sequence":"first","affiliation":[{"name":"University of Trento, Trento, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-0641-7260","authenticated-orcid":false,"given":"Yuan","family":"Feng","sequence":"additional","affiliation":[{"name":"University of Trento, Trento, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1091-8486","authenticated-orcid":false,"given":"Fabio","family":"Massacci","sequence":"additional","affiliation":[{"name":"University of Trento, Trento, Italy"},{"name":"Vrije Universiteit Amsterdam, Amsterdam, Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,6,19]]},"reference":[{"key":"e_1_3_1_2_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.naacl-main.211"},{"key":"e_1_3_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290353"},{"key":"e_1_3_1_4_1","doi-asserted-by":"publisher","unstructured":"Daniel Arp Erwin Quiring Feargus Pendlebury Alexander Warnecke Fabio Pierazzi Christian Wressnegger Lorenzo Cavallaro and Konrad Rieck. 2022. Dos and don\u2019ts of machine learning in computer security. In 31st USENIX Security Symposium (USENIX Security 22). 3971\u20133988. doi:10.48550\/arXiv.2010.09470.","DOI":"10.48550\/arXiv.2010.09470"},{"key":"e_1_3_1_5_1","doi-asserted-by":"publisher","unstructured":"Guru Bhandari Amara Naseer and Leon Moonen. 2021. CVEfixes: automated collection of vulnerabilities and their fixes from open-source software. In Proceedings of the 17th International Conference on Predictive Models and Data Analytics in Software Engineering. 30\u201339. doi:10.1145\/3475960.3475985.","DOI":"10.1145\/3475960.3475985"},{"key":"e_1_3_1_6_1","doi-asserted-by":"publisher","unstructured":"Saikat Chakraborty Rahul Krishna Yangruibo Ding and Baishakhi Ray. 2021. Deep learning based vulnerability detection: Are we there yet. IEEE Transactions on Software Engineering. doi:10.1109\/TSE.2021.3087402.","DOI":"10.1109\/TSE.2021.3087402"},{"key":"e_1_3_1_7_1","doi-asserted-by":"publisher","unstructured":"Haipeng Chen Rui Liu Noseong Park and VS Subrahmanian. 2019. Using twitter to predict when vulnerabilities will be exploited. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data Mining. 3143\u20133152. doi:10.1145\/3292500.3330742.","DOI":"10.1145\/3292500.3330742"},{"key":"e_1_3_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3607199.3607242"},{"key":"e_1_3_1_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00022"},{"key":"e_1_3_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/1321631.1321702"},{"key":"e_1_3_1_11_1","doi-asserted-by":"publisher","unstructured":"Nelson Tavares De Sousa and Wilhelm Hasselbring. 2021. JavaBERT: Training a Transformer-Based Model for the Java Programming Language. In 2021 36th IEEE\/ACM International Conference on Automated Software Engineering Workshops (ASEW). 90\u201395. doi:10.1109\/ASEW52652.2021.00028.","DOI":"10.1109\/ASEW52652.2021.00028"},{"key":"e_1_3_1_12_1","unstructured":"Geanderson E dos Santos and Eduardo Figueiredo. 2020. Commit Classification using Natural Language Processing: Experiments over Labeled Datasets. In CIbSE. 110\u2013123."},{"key":"e_1_3_1_13_1","doi-asserted-by":"publisher","unstructured":"Jiahao Fan Yi Li Shaohua Wang and Tien N Nguyen. 2020. A C\/C++ code vulnerability dataset with code changes and CVE summaries. In Proceedings of the 17th International Conference on Mining Software Repositories. 508\u2013512. doi:10.1145\/3379597.3387501.","DOI":"10.1145\/3379597.3387501"},{"key":"e_1_3_1_14_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.findings-emnlp.139"},{"key":"e_1_3_1_15_1","doi-asserted-by":"publisher","unstructured":"Michael Fu and Chakkrit Tantithamthavorn. 2022. Linevul: A transformer-based line-level vulnerability prediction. In Proceedings of the 19th International Conference on Mining Software Repositories. 608\u2013620. doi:10.1145\/3524842.3528452.","DOI":"10.1145\/3524842.3528452"},{"key":"e_1_3_1_16_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2021.106566"},{"key":"e_1_3_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/IJCNN55064.2022.9892280"},{"key":"e_1_3_1_18_1","doi-asserted-by":"publisher","unstructured":"Jinchang Hu Lyuye Zhang Chengwei Liu Sen Yang Song Huang and Yang Liu. 2024. Empirical Analysis of Vulnerabilities Life Cycle in Golang Ecosystem. In Proceedings of the IEEE\/ACM 46th International Conference on Software Engineering. 1\u201313. doi:10.1145\/3597503.3639230.","DOI":"10.1145\/3597503.3639230"},{"key":"e_1_3_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/SCAM.2018.00014"},{"key":"e_1_3_1_20_1","doi-asserted-by":"publisher","unstructured":"Matthieu Jimenez Mike Papadakis and Yves Le Traon. 2016. Vulnerability Prediction Models: A Case Study on the Linux Kernel. In 2016 IEEE 16th International Working Conference on Source Code Analysis and Manipulation (SCAM). 1\u201310. doi:10.1109\/SCAM.2016.15.","DOI":"10.1109\/SCAM.2016.15"},{"key":"e_1_3_1_21_1","doi-asserted-by":"publisher","unstructured":"Matthieu Jimenez Renaud Rwemalika Mike Papadakis Federica Sarro Yves Le Traon and Mark Harman. 2019. The importance of accounting for real-world labelling when predicting software vulnerabilities. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 695\u2013705. doi:10.1145\/3338906.3338941.","DOI":"10.1145\/3338906.3338941"},{"key":"e_1_3_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2009.9"},{"key":"e_1_3_1_23_1","unstructured":"Johnb110. 2022. VDPython. https:\/\/github.com\/johnb110\/VDPython."},{"key":"e_1_3_1_24_1","doi-asserted-by":"publisher","unstructured":"Ren\u00e9 Just Darioush Jalali and Michael D. Ernst. 2014. Defects4J: A Database of existing faults to enable controlled testing studies for Java programs. In ISSTA 2014 Proceedings of the 2014 International Symposium on Software Testing and Analysis. San Jose CA USA 437\u2013440. doi:10.1145\/2610384.2628055 Tool demo.","DOI":"10.1145\/2610384.2628055"},{"key":"e_1_3_1_25_1","unstructured":"Christian K\u00e4stner. 2022. From Models to Systems: Rethinking the Role of Software Engineering for ML. https:\/\/www.youtube.com\/watch?v=_m-m90S_4Gg."},{"key":"e_1_3_1_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE51524.2021.9678622"},{"key":"e_1_3_1_27_1","doi-asserted-by":"publisher","unstructured":"Stanislav Levin and Amiram Yehudai. 2017. Boosting automatic commit classification into maintenance activities by utilizing source code changes. In Proceedings of the 13th International Conference on Predictive Models and Data Analytics in Software Engineering. 97\u2013106. doi:10.1145\/3127005.3127016.","DOI":"10.1145\/3127005.3127016"},{"key":"e_1_3_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639218"},{"key":"e_1_3_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2019.2930578"},{"key":"e_1_3_1_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/tdsc.2021.3051525"},{"key":"e_1_3_1_31_1","doi-asserted-by":"publisher","unstructured":"Zhen Li Deqing Zou Shouhuai Xu Xinyu Ou Hai Jin Sujuan Wang Zhijun Deng and Yuyi Zhong. 2018. VulDeePecker: A Deep Learning-Based System for Vulnerability Detection. In Proceedings 2018 Network and Distributed System Security Symposium. doi:10.14722\/ndss.2018.23158 arXiv:1801.01681 [cs].","DOI":"10.14722\/ndss.2018.23158"},{"key":"e_1_3_1_32_1","unstructured":"Zhen Li Deqing Zou Shouhuai Xu Xinyu Ou Hai Jin Sujuan Wang Zhijun Deng and Yuyi Zhong. 2022. VulDeePecker. https:\/\/github.com\/CGCL-codes\/VulDeePecker."},{"key":"e_1_3_1_33_1","doi-asserted-by":"publisher","unstructured":"Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy Mike Lewis Luke Zettlemoyer and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. doi:10.48550\/arXiv.1907.11692 arXiv:1907.11692 [cs.CL].","DOI":"10.48550\/arXiv.1907.11692"},{"key":"e_1_3_1_34_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2020.106368"},{"key":"e_1_3_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3559534"},{"key":"e_1_3_1_36_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2406.09701"},{"key":"e_1_3_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/MSEC.2022.3176058"},{"key":"e_1_3_1_38_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-19125-1_15"},{"key":"e_1_3_1_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/MSR.2015.71"},{"key":"e_1_3_1_40_1","unstructured":"Yisroel Mirsky George Macon Michael Brown Carter Yagemann Matthew Pruett Evan Downing Sukarno Mertoguno and Wenke Lee. 2023. VulChecker: Graph-based Vulnerability Localization in Source Code. In 31st USENIX Security Symposium Security 2022. https:\/\/dl.acm.org\/doi\/abs\/10.5555\/3620237.3620604."},{"key":"e_1_3_1_41_1","doi-asserted-by":"publisher","unstructured":"Van-Anh Nguyen Dai Quoc Nguyen Van Nguyen Trung Le Quan Hung Tran and Dinh Phung. 2022. ReGVD: Revisiting graph neural networks for vulnerability detection. In Proceedings of the ACM\/IEEE 44th International Conference on Software Engineering: Companion Proceedings. 178\u2013182. doi:10.1145\/3510454.3516865.","DOI":"10.1145\/3510454.3516865"},{"key":"e_1_3_1_42_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-015-9408-2"},{"key":"e_1_3_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3611643.3616358"},{"key":"e_1_3_1_44_1","doi-asserted-by":"publisher","unstructured":"Georgios Nikitopoulos Konstantina Dritsa Panos Louridas and Dimitris Mitropoulos. 2021. CrossVul: a cross-language vulnerability dataset with commit data. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1565\u20131569. doi:10.1145\/3468264.3473122.","DOI":"10.1145\/3468264.3473122"},{"key":"e_1_3_1_45_1","unstructured":"NIST. 2024a. National Vulnerability Database. https:\/\/nvd.nist.gov\/."},{"key":"e_1_3_1_46_1","unstructured":"NIST. 2024b. National Vulnerability Database: Vulnerability API. https:\/\/nvd.nist.gov\/developers\/vulnerabilities."},{"key":"e_1_3_1_47_1","unstructured":"NIST. 2024c. Software Assurance Reference Dataset. https:\/\/samate.nist.gov\/SRD\/index.php."},{"key":"e_1_3_1_48_1","doi-asserted-by":"publisher","DOI":"10.6028\/NIST.SP.500-297"},{"key":"e_1_3_1_49_1","doi-asserted-by":"publisher","DOI":"10.5555\/1267336.1267343"},{"key":"e_1_3_1_50_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.engappai.2022.104773"},{"key":"e_1_3_1_51_1","doi-asserted-by":"publisher","unstructured":"Shengyi Pan Lingfeng Bao Xin Xia David Lo and Shanping Li. 2023. Fine-grained Commit-level Vulnerability Type Prediction by CWE Tree Structure. In 2023 IEEE\/ACM 45th International Conference on Software Engineering (ICSE). 957\u2013969. doi:10.1109\/ICSE48619.2023.00088.","DOI":"10.1109\/ICSE48619.2023.00088"},{"key":"e_1_3_1_52_1","doi-asserted-by":"publisher","unstructured":"Ranindya Paramitha Yuan Feng and Fabio Massacci. 2025. Replication package on Zenodo Part 1 (NVD Vuldeepecker Dataset). https:\/\/doi.org\/10.5281\/zenodo.8207883 10.5281\/zenodo.8207883.","DOI":"10.5281\/zenodo.8207883"},{"key":"e_1_3_1_53_1","doi-asserted-by":"publisher","unstructured":"Ranindya Paramitha Yuan Feng and Fabio Massacci. 2025. Replication package on Zenodo Part 2 (LINUX Dataset). https:\/\/doi.org\/10.5281\/zenodo.10960662 10.5281\/zenodo.10960662.","DOI":"10.5281\/zenodo.10960662"},{"key":"e_1_3_1_54_1","doi-asserted-by":"publisher","unstructured":"Ranindya Paramitha Yuan Feng and Fabio Massacci. 2025. Replication package on Zenodo Part 3 (OPENSSL Dataset). https:\/\/doi.org\/10.5281\/zenodo.10966117 10.5281\/zenodo.10966117.","DOI":"10.5281\/zenodo.10966117"},{"key":"e_1_3_1_55_1","doi-asserted-by":"publisher","unstructured":"Ranindya Paramitha Yuan Feng and Fabio Massacci. 2025. Replication package on Zenodo Part 4 (POPPLER Dataset). https:\/\/doi.org\/10.5281\/zenodo.14713143 10.5281\/zenodo.14713143.","DOI":"10.5281\/zenodo.14713143"},{"key":"e_1_3_1_56_1","doi-asserted-by":"publisher","unstructured":"Henning Perl Sergej Dechand Matthew Smith Daniel Arp Fabian Yamaguchi Konrad Rieck Sascha Fahl and Yasemin Acar. 2015. Vccfinder: Finding potential vulnerabilities in open-source projects to assist code audits. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security. 426\u2013437. doi:10.1145\/2810103.2813604.","DOI":"10.1145\/2810103.2813604"},{"key":"e_1_3_1_57_1","doi-asserted-by":"publisher","unstructured":"Serena E. Ponta Henrik Plate Antonino Sabetta Michele Bezzi and C\u00e9dric Dangremont. 2019. A Manually-Curated Dataset of Fixes to Vulnerabilities of Open-Source Software. In Proceedings of the 16th International Conference on Mining Software Repositories. doi:10.1109\/MSR.2019.00064.","DOI":"10.1109\/MSR.2019.00064"},{"key":"e_1_3_1_58_1","doi-asserted-by":"publisher","unstructured":"Niklas Risse and Marcel B\u00f6hme. 2024. Uncovering the Limits of Machine Learning for Automatic Vulnerability Detection. In USENIX Security Symposium 2024. 19. doi:10.5555\/3698900.3699138.","DOI":"10.5555\/3698900.3699138"},{"key":"e_1_3_1_59_1","doi-asserted-by":"publisher","DOI":"10.1080\/00273171.2012.738184"},{"key":"e_1_3_1_60_1","doi-asserted-by":"publisher","DOI":"10.1109\/CCWC60891.2024.10427574"},{"key":"e_1_3_1_61_1","unstructured":"Nasir Safdari. 2018. Multi-Class-Text-Classification\u2014Random-Forest. Github. https:\/\/github.com\/nxs5899\/Multi-Class-Text-Classification----Random-Forest."},{"key":"e_1_3_1_62_1","unstructured":"J. SayyadShirabad and T.J. Menzies. 2005. The PROMISE Repository of Software Engineering Databases. School of Information Technology and Engineering University of Ottawa Canada. http:\/\/promise.site.uottawa.ca\/SERepository."},{"key":"e_1_3_1_63_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2014.2340398"},{"key":"e_1_3_1_64_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2013.11"},{"key":"e_1_3_1_65_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2010.81"},{"key":"e_1_3_1_66_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00188"},{"key":"e_1_3_1_67_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00075"},{"key":"e_1_3_1_68_1","doi-asserted-by":"publisher","unstructured":"Bin Yuan Yifan Lu Yilin Fang Yueming Wu Deqing Zou Zhen Li Zhi Li and Hai Jin. 2023. Enhancing Deep Learning-based Vulnerability Detection by Building Behavior Graph Model. In 2023 IEEE\/ACM 45th International Conference on Software Engineering (ICSE). 2262\u20132274. doi:10.1109\/ICSE48619.2023.00190.","DOI":"10.1109\/ICSE48619.2023.00190"},{"key":"e_1_3_1_69_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEIP52600.2021.00020"},{"key":"e_1_3_1_70_1","doi-asserted-by":"publisher","unstructured":"Yaqin Zhou Shangqing Liu Jingkai Siow Xiaoning Du and Yang Liu. 2019. Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks. In Advances in Neural Information Processing Systems. 10197\u201310207. doi:10.5555\/3454287.3455202.","DOI":"10.5555\/3454287.3455202"}],"container-title":["Proceedings of the ACM on Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3715731","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,23]],"date-time":"2026-08-23T11:05:27Z","timestamp":1787483127000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3715731"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,19]]},"references-count":69,"journal-issue":{"issue":"FSE","published-print":{"date-parts":[[2025,6,19]]}},"alternative-id":["10.1145\/3715731"],"URL":"https:\/\/doi.org\/10.1145\/3715731","relation":{},"ISSN":["2994-970X"],"issn-type":[{"value":"2994-970X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,19]]}}}