{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,14]],"date-time":"2026-01-14T19:01:28Z","timestamp":1768417288514,"version":"3.49.0"},"publisher-location":"Cham","reference-count":28,"publisher":"Springer International Publishing","isbn-type":[{"value":"9783031093562","type":"print"},{"value":"9783031093579","type":"electronic"}],"license":[{"start":{"date-parts":[[2022,1,1]],"date-time":"2022-01-01T00:00:00Z","timestamp":1640995200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,6,30]],"date-time":"2022-06-30T00:00:00Z","timestamp":1656547200000},"content-version":"vor","delay-in-days":180,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Software security is a critical consideration for software development companies that want to provide their customers with high-quality and dependable software. The automated detection of software vulnerabilities is a critical aspect in software security. Vulnerability prediction is a mechanism that enables the detection and mitigation of software vulnerabilities early enough in the development cycle. Recently the scientific community has dedicated a lot of effort on the design of Deep learning models based on text mining techniques. Initially, Bag-of-Words was the most promising method but recently more complex models have been proposed focusing on the sequences of instructions in the source code. Recent research endeavors have started utilizing word embedding vectors, which are widely used in text classification tasks like semantic analysis, for representing the words (i.e., code instructions) in vector format. These vectors could be trained either jointly with the other layers of the neural network, or they can be pre-trained using popular algorithms like word2vec and fast-text. In this paper, we empirically examine whether the utilization of word embedding vectors that are pre-trained separately from the vulnerability predictor could lead to more accurate vulnerability prediction models. For the purposes of the present study, a popular vulnerability dataset maintained by NIST was utilized. The results of the analysis suggest that pre-training the embedding vectors separately from the neural network leads to better vulnerability predictors with respect to their effectiveness and performance.<\/jats:p>","DOI":"10.1007\/978-3-031-09357-9_3","type":"book-chapter","created":{"date-parts":[[2022,6,29]],"date-time":"2022-06-29T19:02:49Z","timestamp":1656529369000},"page":"23-37","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["An Empirical Evaluation of\u00a0the\u00a0Usefulness of\u00a0Word Embedding Techniques in\u00a0Deep Learning-Based Vulnerability Prediction"],"prefix":"10.1007","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5118-2508","authenticated-orcid":false,"given":"Ilias","family":"Kalouptsoglou","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3251-8723","authenticated-orcid":false,"given":"Miltiadis","family":"Siavvas","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6912-3493","authenticated-orcid":false,"given":"Dionysios","family":"Kehagias","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5381-8418","authenticated-orcid":false,"given":"Alexandros","family":"Chatzigeorgiou","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5764-7302","authenticated-orcid":false,"given":"Apostolos","family":"Ampatzoglou","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2022,6,30]]},"reference":[{"key":"3_CR1","series-title":"Communications in Computer and Information Science","doi-asserted-by":"publisher","first-page":"142","DOI":"10.1007\/978-3-319-95189-8_13","volume-title":"Security in Computer and Information Sciences","author":"M Siavvas","year":"2018","unstructured":"Siavvas, M., Gelenbe, E., Kehagias, D., Tzovaras, D.: Static analysis-based approaches for\u00a0secure software development. In: Gelenbe, E., et al. (eds.) Euro-CYBERSEC 2018. CCIS, vol. 821, pp. 142\u2013157. Springer, Cham (2018). https:\/\/doi.org\/10.1007\/978-3-319-95189-8_13"},{"key":"3_CR2","doi-asserted-by":"crossref","unstructured":"Walden, J., Stuckman, J., Scandariato, R.: Predicting vulnerable components: software metrics vs text mining. In: 2014 IEEE 25th International Symposium on Software Reliability Engineering. IEEE (2014)","DOI":"10.1109\/ISSRE.2014.32"},{"key":"3_CR3","doi-asserted-by":"crossref","unstructured":"Chakraborty, S., Krishna, R., Ding, Y., Ray, B.: Deep learning based vulnerability detection: are we there yet. IEEE Trans. Softw. Eng. (2021)","DOI":"10.1109\/TSE.2021.3087402"},{"key":"3_CR4","unstructured":"Zhou, Y., Liu, S., Siow, J., Du, X., Liu, Y.: Devign: effective vulnerability identification by learning comprehensive program semantics via graph neural networks. arXiv preprint arXiv:1909.03496 (2019)"},{"key":"3_CR5","doi-asserted-by":"crossref","unstructured":"Li, Z., et al.: Vuldeepecker: a deep learning-based system for vulnerability detection. arXiv preprint arXiv:1801.01681 (2018)","DOI":"10.14722\/ndss.2018.23158"},{"key":"3_CR6","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2021.106576","volume":"136","author":"S Cao","year":"2021","unstructured":"Cao, S., Sun, X., Bo, L., Wei, Y., Li, B.: BGNN4VD: constructing bidirectional graph neural-network for vulnerability detection. Inf. Softw. Technol. 136, 106576 (2021)","journal-title":"Inf. Softw. Technol."},{"key":"3_CR7","doi-asserted-by":"crossref","unstructured":"Pang, Y., Xue, X., Wang, H.: Predicting vulnerable software components through deep neural network. In: Proceedings of the 2017 International Conference on Deep Learning Technologies, pp. 6\u201310 (2017)","DOI":"10.1145\/3094243.3094245"},{"key":"3_CR8","doi-asserted-by":"crossref","unstructured":"Dam, H.K., Tran, T., Pham, T.T.M., Ng, S.W., Grundy, J., Ghose, A.: Automatic feature learning for predicting vulnerable software components. IEEE Trans. Softw. Eng. 47, 67\u201385 (2018)","DOI":"10.1109\/TSE.2018.2881961"},{"key":"3_CR9","doi-asserted-by":"crossref","unstructured":"Scandariato, R., Walden, J., Hovsepyan, A., Joosen, W.: Predicting vulnerable software components via text mining. IEEE Trans. Softw. Eng. 40, 993\u20131006 (2014)","DOI":"10.1109\/TSE.2014.2340398"},{"key":"3_CR10","doi-asserted-by":"crossref","unstructured":"Hovsepyan, A., Scandariato, R., Joosen, W., Walden, J.: Software vulnerability prediction using text analysis techniques. In: Proceedings of the 4th International Workshop on Security Measurements and Metrics, pp. 7\u201310 (2012)","DOI":"10.1145\/2372225.2372230"},{"key":"3_CR11","doi-asserted-by":"crossref","unstructured":"Medhat, W., Hassan, A., Korashy, H.: Sentiment analysis algorithms and applications: a survey. Ain Shams Eng. J. 5(4), 1093\u20131113 (2014)","DOI":"10.1016\/j.asej.2014.04.011"},{"key":"3_CR12","doi-asserted-by":"crossref","unstructured":"Russell, R., et al.: Automated vulnerability detection in source code using deep representation learning. In: 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), pp. 757\u2013762. IEEE (2018)","DOI":"10.1109\/ICMLA.2018.00120"},{"key":"3_CR13","unstructured":"Turian, J., Ratinov, L., Bengio, Y.: Word representations: a simple and general method for semi-supervised learning. In: Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pp. 384\u2013394 (2010)"},{"key":"3_CR14","unstructured":"Rong, X.: word2vec parameter learning explained. arXiv (2014)"},{"key":"3_CR15","doi-asserted-by":"crossref","unstructured":"Pennington, J., Socher, R., Manning, C.D.: Glove: global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532\u20131543 (2014)","DOI":"10.3115\/v1\/D14-1162"},{"key":"3_CR16","doi-asserted-by":"crossref","unstructured":"Joulin, A., Grave, E., Bojanowski, P., Mikolov, T.: Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759 (2016)","DOI":"10.18653\/v1\/E17-2068"},{"key":"3_CR17","unstructured":"Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013)"},{"key":"3_CR18","doi-asserted-by":"publisher","first-page":"216","DOI":"10.1016\/j.ins.2018.09.001","volume":"471","author":"RA Stein","year":"2019","unstructured":"Stein, R.A., Jaques, P.A., Valiati, J.F.: An analysis of hierarchical text classification using word embeddings. Inf. Sci. 471, 216\u2013232 (2019)","journal-title":"Inf. Sci."},{"key":"3_CR19","doi-asserted-by":"crossref","unstructured":"Ma, Y., Peng, H., Cambria, E.: Targeted aspect-based sentiment analysis via embedding commonsense knowledge into an attentive LSTM. In: Thirty-Second AAAI Conference on Artificial Intelligence (2018)","DOI":"10.1609\/aaai.v32i1.12048"},{"key":"3_CR20","doi-asserted-by":"crossref","unstructured":"Fang, Y., Liu, Y., Huang, C., Liu, L.: Fastembed: Predicting vulnerability exploitation possibility based on ensemble machine learning algorithm. Plos One 15, e0228439 (2020)","DOI":"10.1371\/journal.pone.0228439"},{"key":"3_CR21","doi-asserted-by":"crossref","unstructured":"Goldberg, Y.: Neural network methods for natural language processing. Synth. Lect. Hum. Lang. Technol. 10(1), 1\u2013309 (2017)","DOI":"10.2200\/S00762ED1V01Y201703HLT037"},{"key":"3_CR22","unstructured":"Van der Maaten, L., Hinton, G.: Visualizing data using t-SNE. J. Mach. Learn. Res. 9(11) (2008)"},{"key":"3_CR23","doi-asserted-by":"crossref","unstructured":"Sundermeyer, M., Oparin, I., Gauvain, J.L., Freiberg, B., Schl\u00fcter, R., Ney, H.: Comparison of feedforward and recurrent neural network language models. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (2013)","DOI":"10.1109\/ICASSP.2013.6639310"},{"key":"3_CR24","doi-asserted-by":"publisher","unstructured":"Filus, K., Siavvas, M., Doma\u0144ska, J., Gelenbe, E.: The random neural network as a bonding model for software vulnerability prediction. In: Calzarossa, M.C., Gelenbe, E., Grochla, K., Lent, R., Czach\u00f3rski, T. (eds.) MASCOTS 2020. LNCS, vol. 12527, pp. 102\u2013116. Springer, Cham (2021). https:\/\/doi.org\/10.1007\/978-3-030-68110-4_7","DOI":"10.1007\/978-3-030-68110-4_7"},{"key":"3_CR25","doi-asserted-by":"crossref","unstructured":"Sundermeyer, M., Schl\u00fcter, R., Ney, H.: LSTM neural networks for language modeling. In: Thirteenth Annual Conference of the ISCA (2012)","DOI":"10.21437\/Interspeech.2012-65"},{"key":"3_CR26","unstructured":"Chung, J., Gulcehre, C., Cho, K., Bengio, Y.: Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv (2014)"},{"issue":"11","key":"3_CR27","doi-asserted-by":"publisher","first-page":"2673","DOI":"10.1109\/78.650093","volume":"45","author":"M Schuster","year":"1997","unstructured":"Schuster, M., Paliwal, K.K.: Bidirectional recurrent neural networks. IEEE Trans. Signal Process. 45(11), 2673\u20132681 (1997)","journal-title":"IEEE Trans. Signal Process."},{"key":"3_CR28","doi-asserted-by":"crossref","unstructured":"Hochreiter, S.: The vanishing gradient problem during learning recurrent neural nets and problem solutions. Int. J. Uncertainty 6, 107\u2013116 (1998)","DOI":"10.1142\/S0218488598000094"}],"container-title":["Communications in Computer and Information Science","Security in Computer and Information Sciences"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-031-09357-9_3","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,6,29]],"date-time":"2022-06-29T19:12:17Z","timestamp":1656529937000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/978-3-031-09357-9_3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022]]},"ISBN":["9783031093562","9783031093579"],"references-count":28,"URL":"https:\/\/doi.org\/10.1007\/978-3-031-09357-9_3","relation":{},"ISSN":["1865-0929","1865-0937"],"issn-type":[{"value":"1865-0929","type":"print"},{"value":"1865-0937","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022]]},"assertion":[{"value":"30 June 2022","order":1,"name":"first_online","label":"First Online","group":{"name":"ChapterHistory","label":"Chapter History"}},{"value":"EuroCybersec","order":1,"name":"conference_acronym","label":"Conference Acronym","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"International ISCIS Security Workshop","order":2,"name":"conference_name","label":"Conference Name","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Nice","order":3,"name":"conference_city","label":"Conference City","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"France","order":4,"name":"conference_country","label":"Conference Country","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"2021","order":5,"name":"conference_year","label":"Conference Year","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"25 October 2021","order":7,"name":"conference_start_date","label":"Conference Start Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"26 October 2021","order":8,"name":"conference_end_date","label":"Conference End Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"2","order":9,"name":"conference_number","label":"Conference Number","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"iscissecurityworkshop2021","order":10,"name":"conference_id","label":"Conference ID","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"https:\/\/iotac.eu\/eurocybersec2021-workshop\/","order":11,"name":"conference_url","label":"Conference URL","group":{"name":"ConferenceInfo","label":"Conference Information"}}]}}