{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,22]],"date-time":"2026-06-22T17:28:35Z","timestamp":1782149315462,"version":"3.54.5"},"reference-count":51,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2023,6,28]],"date-time":"2023-06-28T00:00:00Z","timestamp":1687910400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,6,28]],"date-time":"2023-06-28T00:00:00Z","timestamp":1687910400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001777","name":"The University of Wollongong","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100001777","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Autom Softw Eng"],"published-print":{"date-parts":[[2023,11]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>The recent advent of data protection laws and regulations has emerged to protect privacy and personal information of individuals. As the cases of privacy breaches and vulnerabilities are rapidly increasing, people are aware and more concerned about their privacy. These bring a significant attention to software development teams to address privacy concerns in developing software applications. As today\u2019s software development adopts an agile, issue-driven approach, issues in an issue tracking system become a centralised pool that gathers new requirements, requests for modification and all the tasks of the software project. Hence, establishing an alignment between those issues and privacy requirements is an important step in developing privacy-aware software systems. This alignment also facilitates privacy compliance checking which may be required as an underlying part of regulations for organisations. However, manually establishing those alignments is labour intensive and time consuming. In this paper, we explore a wide range of machine learning and natural language processing techniques which can automatically classify privacy requirements in issue reports. We employ six popular techniques namely Bag-of-Words (BoW), N-gram Inverse Document Frequency (N-gram IDF), Term Frequency-Inverse Document Frequency (TF-IDF), Word2Vec, Convolutional Neural Network (CNN) and Bidirectional Encoder Representations from Transformers (BERT) to perform the classification on privacy-related issue reports in Google Chrome and Moodle projects. The evaluation showed that BoW, N-gram IDF, TF-IDF and Word2Vec techniques are suitable for classifying privacy requirements in those issue reports. In addition, N-gram IDF is the best performer in both projects.<\/jats:p>","DOI":"10.1007\/s10515-023-00387-9","type":"journal-article","created":{"date-parts":[[2023,6,28]],"date-time":"2023-06-28T11:02:20Z","timestamp":1687950140000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["An empirical study of automated privacy requirements classification in issue reports"],"prefix":"10.1007","volume":"30","author":[{"given":"Pattaraporn","family":"Sangaroonsilp","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Morakot","family":"Choetkiertikul","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hoa Khanh","family":"Dam","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Aditya","family":"Ghose","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2023,6,28]]},"reference":[{"key":"387_CR1","doi-asserted-by":"crossref","unstructured":"Almhana, R., Mkaouer, W., Kessentini, M., Ouni, A.: Recommending relevant classes for bug reports using multi-objective search. In: Proceedings of the 31st IEEE\/ACM International Conference on Automated Software Engineering (ASE), pp. 286\u2013295 (2016)","DOI":"10.1145\/2970276.2970344"},{"issue":"3","key":"387_CR2","doi-asserted-by":"publisher","first-page":"169","DOI":"10.1007\/s00766-003-0183-z","volume":"9","author":"AI Ant\u00f3n","year":"2004","unstructured":"Ant\u00f3n, A.I., Earp, J.B.: A requirements taxonomy for reducing Web site privacy vulnerabilities. Requir. Eng. 9(3), 169\u2013185 (2004)","journal-title":"Requir. Eng."},{"issue":"3","key":"387_CR3","doi-asserted-by":"publisher","first-page":"219","DOI":"10.1002\/stvr.1486","volume":"24","author":"A Arcuri","year":"2014","unstructured":"Arcuri, A., Briand, L.: A Hitchhiker\u2019s guide to statistical tests for assessing randomized algorithms in software engineering. Softw. Test. Verif. Reliab. 24(3), 219\u2013250 (2014)","journal-title":"Softw. Test. Verif. Reliab."},{"key":"387_CR4","unstructured":"Asia-Pacific Economic Cooperation (APEC): APEC Privacy Framework (2015). Technical report, APEC Secretariat (2017)"},{"key":"387_CR5","doi-asserted-by":"crossref","unstructured":"Ayala-Rivera, V., Pasquale, L.: The grace period has ended: an approach to operationalize GDPR requirements. In: Proceedings - 2018 IEEE 26th International Requirements Engineering Conference, RE 2018, pp. 136\u2013146 (2018)","DOI":"10.1109\/RE.2018.00023"},{"key":"387_CR6","unstructured":"BBC: British Airways fined \u00a320m over data breach. https:\/\/www.bbc.com\/news\/technology-54568784 (2020)"},{"key":"387_CR7","first-page":"993","volume":"3","author":"DM Blei","year":"2003","unstructured":"Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent Dirichlet allocation. J. Mach. Learn. Res. 3, 993\u20131022 (2003)","journal-title":"J. Mach. Learn. Res."},{"key":"387_CR8","doi-asserted-by":"publisher","unstructured":"Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching Word Vectors with Subword Information. Trans. Assoc. Comput. Linguist. 5, 135\u2013146 (2017). https:\/\/doi.org\/10.1162\/tacl_a_00051arXiv:1607.04606","DOI":"10.1162\/tacl_a_00051"},{"issue":"1","key":"387_CR9","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1023\/A:1010933404324","volume":"45","author":"L Breiman","year":"2001","unstructured":"Breiman, L.: Random forests. Mach. Learn. 45(1), 5\u201332 (2001)","journal-title":"Mach. Learn."},{"key":"387_CR10","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2021.106743","volume":"142","author":"H Cho","year":"2022","unstructured":"Cho, H., Lee, S., Kang, S.: Classifying issue reports according to feature descriptions in a user manual based on a deep learning model. Inf. Softw. Technol. 142, 106743 (2022). https:\/\/doi.org\/10.1016\/j.infsof.2021.106743","journal-title":"Inf. Softw. Technol."},{"key":"387_CR11","doi-asserted-by":"crossref","unstructured":"Choetkiertikul, M., Dam, H.K., Tran, T., Pham, T., Ghose, A.: Poster: Predicting components for issue reports using deep learning with information retrieval. In: Proceedings - International Conference on Software Engineering, 244\u2013245 (2018)","DOI":"10.1145\/3183440.3194952"},{"key":"387_CR12","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-020-09898-5","author":"M Choetkiertikul","year":"2021","unstructured":"Choetkiertikul, M., Dam, H.K., Tran, T., Pham, T., Ragkhitwetsagul, C., Ghose, A.: Automatically recommending components for issue reports using deep learning. Empir. Softw. Eng. (2021). https:\/\/doi.org\/10.1007\/s10664-020-09898-5","journal-title":"Empir. Softw. Eng."},{"key":"387_CR13","unstructured":"CNET: British Airways faces \\$230M GDPR fine for 2018 data breach (2019). https:\/\/cnet.co\/2EHboiu"},{"key":"387_CR14","unstructured":"Data Privacy Manager: 5 biggest GDPR fines so far [2020] - Data Privacy Manager. https:\/\/dataprivacymanager.net\/5-biggest-gdpr-fines-so-far-2020\/ (2020). Accessed 24 March 2020"},{"key":"387_CR15","unstructured":"Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding (2018). https:\/\/arxiv.org\/pdf\/1810.04805.pdf"},{"key":"387_CR16","unstructured":"European Commission: GDPR IN NUMBERS 2019 #HAPPYBIRTHDAYGDPR. Technical report (2019)"},{"key":"387_CR17","doi-asserted-by":"publisher","unstructured":"Fan, Q., Yu, Y., Yin, G., Wang, T., Wang, H.: Where is the road for issue reports classification based on text mining?. In: International Symposium on Empirical Software Engineering and Measurement, vol. 2017-November, pp. 121\u2013130 (2017). https:\/\/doi.org\/10.1109\/ESEM.2017.19","DOI":"10.1109\/ESEM.2017.19"},{"key":"387_CR18","unstructured":"Google: word2vec. https:\/\/code.google.com\/archive\/p\/word2vec\/ (2013). Accessed 16 February 2023"},{"key":"387_CR19","unstructured":"G\u00fcrses, S., Troncoso, C., Diaz, C.: Engineering privacy by design. In: Computers, Privacy & Data Protection, pp. 1\u201325 (2011)"},{"issue":"2\u20133","key":"387_CR20","doi-asserted-by":"publisher","first-page":"146","DOI":"10.1080\/00437956.1954.11659520","volume":"10","author":"ZS Harris","year":"1954","unstructured":"Harris, Z.S.: Distributional structure. WORD 10(2\u20133), 146\u2013162 (1954)","journal-title":"WORD"},{"key":"387_CR21","unstructured":"Huggingface: Tokenizers (2020). https:\/\/github.com\/huggingface\/tokenizers Accessed 10 January 2021"},{"key":"387_CR22","unstructured":"Huggingface: Transformers (2020). https:\/\/huggingface.co\/transformers\/"},{"key":"387_CR23","unstructured":"ISO\/IEC: International Standard ISO\/IEC 29100. Technical report, ISO\/IEC (2011)"},{"key":"387_CR24","doi-asserted-by":"crossref","unstructured":"Kowsari, K., Meimandi, K.J., Heidarysafa, M., Mendu, S., Barnes, L., Brown, D.: Text classification algorithms: a survey. In: Information, vol. 10, pp. 1\u201368 (2019). https:\/\/www.mdpi.com\/2078-2489\/10\/4\/150","DOI":"10.3390\/info10040150"},{"key":"387_CR25","doi-asserted-by":"publisher","unstructured":"Lee, S.R., Heo, M.J., Lee, C.G., Kim, M., Jeong, G.: Applying deep learning based automatic bug triager to industrial projects. In: Proceedings of the ACM SIGSOFT Symposium on the Foundations of Software Engineering, vol. Part F1301, pp. 926\u2013931 (2017). https:\/\/doi.org\/10.1145\/3106237.3117776","DOI":"10.1145\/3106237.3117776"},{"issue":"2","key":"387_CR26","doi-asserted-by":"publisher","first-page":"361","DOI":"10.1037\/0033-2909.111.2.361","volume":"111","author":"KO McGraw","year":"1992","unstructured":"McGraw, K.O., Wong, S.P.: A common language effect size statistic. Psychol. Bull. 111(2), 361\u2013365 (1992)","journal-title":"Psychol. Bull."},{"key":"387_CR27","unstructured":"Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. In: 1st International Conference on Learning Representations, ICLR 2013 - Workshop Track Proceedings (2013)"},{"issue":"2","key":"387_CR28","doi-asserted-by":"publisher","first-page":"621","DOI":"10.1016\/j.eswa.2012.07.059","volume":"40","author":"R Moraes","year":"2013","unstructured":"Moraes, R., Valiati, J.F., Gaviao Neto, W.P.: Document-level sentiment classification: an empirical comparison between SVM and ANN. Expert Syst. Appl. 40(2), 621\u2013633 (2013)","journal-title":"Expert Syst. Appl."},{"key":"387_CR29","doi-asserted-by":"crossref","unstructured":"M\u00fcller, N.M., Kowatsch, D., Debus, P., Mirdita, D., B\u00f6ttinger, K.: On GDPR compliance of companies\u2019 privacy policies. In: Lecture Notes in Computer Science (including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) vol. 11697, pp. 151\u2013159 (2019)","DOI":"10.1007\/978-3-030-27947-9_13"},{"key":"387_CR30","unstructured":"National Legislative Assembly: The Thailand personal data protection act. Technical report (2019). https:\/\/thainetizen.org\/wp-content\/uploads\/2019\/11\/thailand-personal-data-protection-act-2019-en.pdf"},{"key":"387_CR31","unstructured":"Office Journal of the European Union: General Data Protection Regulation (GDPR). Technical report. https:\/\/eur-lex.europa.eu\/legal-content\/EN\/TXT\/PDF\/?uri=CELEX:32016R0679 (2016)"},{"issue":"4","key":"387_CR32","doi-asserted-by":"publisher","first-page":"279","DOI":"10.1007\/s11334-017-0294-1","volume":"13","author":"N Pandey","year":"2017","unstructured":"Pandey, N., Sanyal, D.K., Hudait, A., Sen, A.: Automated classification of software issue reports using machine learning techniques: an empirical study. Innov. Syst. Softw. Eng. 13(4), 279\u2013297 (2017). https:\/\/doi.org\/10.1007\/s11334-017-0294-1","journal-title":"Innov. Syst. Softw. Eng."},{"key":"387_CR33","doi-asserted-by":"publisher","unstructured":"Pennington, J., Socher, R., Manning, C.D.: GloVe: Global vectors for word representation. In: EMNLP 2014 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference, pp. 1532\u20131543 (2014). https:\/\/doi.org\/10.3115\/v1\/d14-1162 . https:\/\/nlp.stanford.edu\/projects\/glove\/","DOI":"10.3115\/v1\/d14-1162"},{"key":"387_CR34","doi-asserted-by":"publisher","unstructured":"Peters, M.E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., Zettlemoyer, L.: Deep contextualized word representations. In: NAACL HLT 2018 - 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference, vol. 1, pp. 2227\u20132237 (2018). https:\/\/doi.org\/10.18653\/v1\/n18-1202. https:\/\/arxiv.org\/abs\/1802.05365v2","DOI":"10.18653\/v1\/n18-1202"},{"key":"387_CR35","unstructured":"Privacy Affairs: GDPR Fines List: find all GDPR fines and detailed statistics. https:\/\/www.privacyaffairs.com\/gdpr-fines\/ (2020)"},{"key":"387_CR36","unstructured":"Reina, Y.: Text preprocessing for BERT (2020). https:\/\/www.kaggle.com\/c\/google-quest-challenge\/discussion\/127881"},{"key":"387_CR37","unstructured":"Reuters: ICO fines Marriott 18.4 million pounds for failing to secure customer data. https:\/\/www.reuters.com\/article\/us-marriott-intnl-ico-idUSKBN27F1LH (2020)"},{"key":"387_CR38","unstructured":"Sangaroonsilp, P., Choetkiertikul, M., Dam, H.K., Ghose, A.: Replication package for automated classification of privacy requirements in issue reports (2022). http:\/\/bit.ly\/AutomaticPRrecommendation"},{"key":"387_CR39","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2023.107162","author":"P Sangaroonsilp","year":"2023","unstructured":"Sangaroonsilp, P., Dam, H.K., Choetkiertikul, M., Ragkhitwetsagul, C., Ghose, A.: A taxonomy for mining and classifying privacy requirements in issue reports. Inf. Softw. Technol. (2023). https:\/\/doi.org\/10.1016\/j.infsof.2023.107162","journal-title":"Inf. Softw. Technol."},{"key":"387_CR40","volume-title":"Deep Learning: A Comprehensive Overview on Techniques, Taxonomy","author":"IH Sarker","year":"2021","unstructured":"Sarker, I.H.: Deep Learning: A Comprehensive Overview on Techniques, Taxonomy. Springer, Applications and Research Directions (2021)"},{"key":"387_CR41","doi-asserted-by":"crossref","unstructured":"Shirakawa, M., Hara, T., Nishio, S.: N-gram IDF: a global term weighting scheme based on information distance. In: Proceedings of the 24th International Conference on World Wide Web, pp. 960\u2013970 (2015)","DOI":"10.1145\/2736277.2741628"},{"key":"387_CR42","first-page":"1929","volume":"15","author":"N Srivastava","year":"2014","unstructured":"Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15, 1929\u20131958 (2014)","journal-title":"J. Mach. Learn. Res."},{"key":"387_CR43","unstructured":"State of California Department of Justice: California Consumer Privacy Act (CCPA) https:\/\/oag.ca.gov\/privacy\/ccpa (2018)"},{"key":"387_CR44","unstructured":"Swinhoe, D.: The 14 biggest data breaches of the 21st century. https:\/\/www.csoonline.com\/article\/2130877\/the-biggest-data-breaches-of-the-21st-century.html (2020). Accessed 29 March 2020"},{"key":"387_CR45","unstructured":"TensorFlow: The Sequential Model (2020). https:\/\/www.tensorflow.org\/guide\/keras\/sequential_model"},{"key":"387_CR46","doi-asserted-by":"crossref","unstructured":"Tirilly, P., Claveau, V., Gros, P.: Language modeling for bag-of-visual words image categorization. In: Proceedings of the International Conference on Content-based Image and Video Retrieval, pp. 249\u2013258 (2008)","DOI":"10.1145\/1386352.1386388"},{"key":"387_CR47","doi-asserted-by":"crossref","unstructured":"Torre, D., Abualhaija, S., Sabetzadeh, M., Briand, L., Baetens, K., Goes, P., Forastier, S.: An AI-assisted Approach for Checking the Completeness of Privacy Policies Against GDPR. In 2020 IEEE 28th International Requirements Engineering Conference (RE), 136\u2013146 (2020)","DOI":"10.1109\/RE48521.2020.00025"},{"key":"387_CR48","unstructured":"United Nations Conference on Trade and Development (UNCTAD): Data protection and privacy legislation worldwide (2020). https:\/\/unctad.org\/page\/data-protection-and-privacy-legislation-worldwide Accessed 19 May 2021"},{"issue":"2","key":"387_CR49","first-page":"101","volume":"25","author":"A Vargha","year":"2000","unstructured":"Vargha, A., Delaney, H.D.: A critique and improvement of the CL common language effect size statistics of McGraw and Wong. J. Educ. Behav. Stat. 25(2), 101\u2013132 (2000)","journal-title":"J. Educ. Behav. Stat."},{"key":"387_CR50","unstructured":"Wild, C.: The Wilcoxon Rank-Sum Test. Technical report, Department of Statistics, University of Auckland (1997)"},{"key":"387_CR51","doi-asserted-by":"crossref","unstructured":"Ye, X., Fang, F., Wu, J., Bunescu, R., Liu, C.: Bug report classification using LSTM architecture for more accurate software defect locating. In: Proceedings - 17th IEEE International Conference on Machine Learning and Applications, ICMLA 2018, pp. 1438\u20131445 (2019)","DOI":"10.1109\/ICMLA.2018.00234"}],"container-title":["Automated Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10515-023-00387-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10515-023-00387-9\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10515-023-00387-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,10,25]],"date-time":"2023-10-25T07:07:48Z","timestamp":1698217668000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10515-023-00387-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,28]]},"references-count":51,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2023,11]]}},"alternative-id":["387"],"URL":"https:\/\/doi.org\/10.1007\/s10515-023-00387-9","relation":{},"ISSN":["0928-8910","1573-7535"],"issn-type":[{"value":"0928-8910","type":"print"},{"value":"1573-7535","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,6,28]]},"assertion":[{"value":"1 July 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"4 June 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"28 June 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"20"}}