{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,9]],"date-time":"2026-07-09T04:23:55Z","timestamp":1783571035715,"version":"3.55.0"},"reference-count":46,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2023,6,3]],"date-time":"2023-06-03T00:00:00Z","timestamp":1685750400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100005802","name":"Innovative Solution for Optimizing User Productivity","doi-asserted-by":"publisher","award":["366\/390042\/27.09.2021"],"award-info":[{"award-number":["366\/390042\/27.09.2021"]}],"id":[{"id":"10.13039\/501100005802","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>Email classification systems are essential for handling and organizing the massive flow of communication, especially in a business context. Although many solutions exist, the lack of standardized classification categories limits their applicability. Furthermore, the lack of Romanian language business-oriented public datasets makes the development of such solutions difficult. To this end, we introduce a versatile automated email classification system based on a novel public dataset of 1447 manually annotated Romanian business-oriented emails. Our corpus is annotated with 5 token-related labels, as well as 5 sequence-related classes. We establish a strong baseline using pre-trained Transformer models for token classification and multi-task classification, achieving an F1-score of 0.752 and 0.764, respectively. We publicly release our code together with the dataset of labeled emails.<\/jats:p>","DOI":"10.3390\/info14060321","type":"journal-article","created":{"date-parts":[[2023,6,5]],"date-time":"2023-06-05T02:18:29Z","timestamp":1685931509000},"page":"321","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Multi-Task Romanian Email Classification in a Business Context"],"prefix":"10.3390","volume":"14","author":[{"given":"Alexandru","family":"Dima","sequence":"first","affiliation":[{"name":"Computer Science & Engineering Department, University Politehnica of Bucharest, 313 Splaiul Independentei, 060042 Bucharest, Romania"},{"name":"Research Technology, 19D Soseaua Virtutii, 060782 Bucharest, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0380-6814","authenticated-orcid":false,"given":"Stefan","family":"Ruseti","sequence":"additional","affiliation":[{"name":"Computer Science & Engineering Department, University Politehnica of Bucharest, 313 Splaiul Independentei, 060042 Bucharest, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Denis","family":"Iorga","sequence":"additional","affiliation":[{"name":"Research Technology, 19D Soseaua Virtutii, 060782 Bucharest, Romania"},{"name":"Interdisciplinary School of Doctoral Studies, University of Bucharest, 4-12 Bulevardul Regina Elisabeta, 030018 Bucharest, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cosmin Karl","family":"Banica","sequence":"additional","affiliation":[{"name":"Research Technology, 19D Soseaua Virtutii, 060782 Bucharest, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4815-9227","authenticated-orcid":false,"given":"Mihai","family":"Dascalu","sequence":"additional","affiliation":[{"name":"Computer Science & Engineering Department, University Politehnica of Bucharest, 313 Splaiul Independentei, 060042 Bucharest, Romania"},{"name":"Research Technology, 19D Soseaua Virtutii, 060782 Bucharest, Romania"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2023,6,3]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Klimt, B., and Yang, Y. (2004, January 20\u201324). The enron corpus: A new dataset for email classification research. Proceedings of the Machine Learning: ECML 2004: 15th European Conference on Machine Learning, Pisa, Italy.","DOI":"10.1007\/978-3-540-30115-8_22"},{"key":"ref_2","unstructured":"Srirag, D. (2023, May 03). Text Classification on Emails. Available online: https:\/\/www.kaggle.com\/datasets\/dipankarsrirag\/topic-modelling-on-emails."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Jabbari, S., Allison, B., Guthrie, D., and Guthrie, L. (2006, January 17\u201318). Towards the Orwellian nightmare: Separation of business and personal emails. Proceedings of the COLING\/ACL 2006 Main Conference Poster Sessions, Sydney, Australia.","DOI":"10.3115\/1273073.1273126"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Alkhereyf, S., and Rambow, O. (2017, January 3). Work hard, play hard: Email classification on the Avocado and Enron corpora. Proceedings of the TextGraphs-11: The Workshop on Graph-Based Methods for Natural Language Processing, Vancouver, BC, Canada.","DOI":"10.18653\/v1\/W17-2408"},{"key":"ref_5","unstructured":"Oard, D., Webber, W., Kirsch, D., and Golitsynskiy, S. (2015). Avocado Research Email Collection, Linguistic Data Consortium."},{"key":"ref_6","unstructured":"Mason, J. (2023, May 03). The Apache SpamAssassin Public Corpus. Available online: https:\/\/spamassassin.apache.org\/old\/publiccorpus\/."},{"key":"ref_7","unstructured":"Dua, D., and Graff, C. (2023, May 02). UCI Machine Learning Repository. Available online: http:\/\/archive.ics.uci.edu\/ml."},{"key":"ref_8","unstructured":"Metsis, V., Androutsopoulos, I., and Paliouras, G. (2006, January 27\u201328). Spam filtering with naive bayes-which naive bayes?. Proceedings of the CEAS, Mountain View, CA, USA."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Cormack, G.V., and Lynam, T.R. (2005, January 15\u201318). TREC 2005 Spam Track Overview. Proceedings of the TREC, Gaithersburg, MD, USA.","DOI":"10.6028\/NIST.SP.500-266.spam-overview"},{"key":"ref_10","unstructured":"Nazario, J. (2023, May 12). Phishing Corpus. Available online: http:\/\/monkey.org\/~jose\/wiki\/doku.php."},{"key":"ref_11","unstructured":"Radev, D. (2023, May 12). CLAIR Collection of Fraud Email. Available online: http:\/\/aclweb.org\/aclwiki."},{"key":"ref_12","first-page":"8","article-title":"Email classification using artificial neural network","volume":"2","author":"Alghoul","year":"2018","journal-title":"Int. J. Acad. Eng. Res."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"56","DOI":"10.1016\/j.jnca.2018.12.002","article-title":"Design of multi-view based email classification for IoT systems via semi-supervised learning","volume":"128","author":"Li","year":"2019","journal-title":"J. Netw. Comput. Appl."},{"key":"ref_14","unstructured":"Sharaff, A., and Gupta, H. (2018, January 20\u201321). Extra-tree classifier with metaheuristics approach for email classification. Proceedings of the Advances in Computer Communication and Computational Sciences: Proceedings of IC4S 2018, Bangkok, Thailand."},{"key":"ref_15","first-page":"1","article-title":"Semantic graph neural network: A conversion from spam email classification to graph classification","volume":"2022","author":"Pan","year":"2022","journal-title":"Sci. Program."},{"key":"ref_16","first-page":"993","article-title":"Latent dirichlet allocation","volume":"3","author":"Blei","year":"2003","journal-title":"J. Mach. Learn. Res."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Saleh, A.J., Karim, A., Shanmugam, B., Azam, S., Kannoorpatti, K., Jonkman, M., and Boer, F.D. (2019). An intelligent spam detection model based on artificial immune system. Information, 10.","DOI":"10.3390\/info10060209"},{"key":"ref_18","unstructured":"Forrest, S., Perelson, A.S., Allen, L., and Cherukuri, R. (1994, January 16\u201318). Self-nonself discrimination in a computer. Proceedings of the 1994 IEEE Computer Society Symposium on Research in Security and Privacy, Oakland, CA, USA."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Yasin, A., and Abuhasan, A. (2016). An intelligent classification model for phishing email detection. arXiv.","DOI":"10.5121\/ijnsa.2016.8405"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Niu, W., Zhang, X., Yang, G., Ma, Z., and Zhuo, Z. (2017, January 12\u201315). Phishing emails detection using CS-SVM. Proceedings of the 2017 IEEE International Symposium on Parallel and Distributed Processing with Applications and 2017 IEEE International Conference on Ubiquitous Computing and Communications (ISPA\/IUCC), Guangzhou, China.","DOI":"10.1109\/ISPA\/IUCC.2017.00160"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Egozi, G., and Verma, R. (2018, January 17\u201320). Phishing email detection using robust nlp techniques. Proceedings of the 2018 IEEE International Conference on Data Mining Workshops (ICDMW), Singapore.","DOI":"10.1109\/ICDMW.2018.00009"},{"key":"ref_22","unstructured":"Harikrishnan, N., Vinayakumar, R., and Soman, K. (2018, January 21). A machine learning approach towards phishing email detection. Proceedings of the Anti-Phishing Pilot at ACM International Workshop on Security and Privacy Analytics (IWSPA AP), Tempe, AZ, USA."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"56329","DOI":"10.1109\/ACCESS.2019.2913705","article-title":"Phishing email detection using improved RCNN model with multilevel vectors and attention mechanism","volume":"7","author":"Fang","year":"2019","journal-title":"IEEE Access"},{"key":"ref_24","unstructured":"Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"102414","DOI":"10.1016\/j.cose.2021.102414","article-title":"Applying machine learning and natural language processing to detect phishing email","volume":"110","author":"Alhogail","year":"2021","journal-title":"Comput. Secur."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Baccouche, A., Ahmed, S., Sierra-Sosa, D., and Elmaghraby, A. (2020). Malicious text identification: Deep learning from public comments and emails. Information, 11.","DOI":"10.3390\/info11060312"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput."},{"key":"ref_28","unstructured":"Harsha Kadam, S., and Paniskaki, K. (2020). Text Analysis for Email Multi Label Classification. [Master\u2019s Thesis, University of Gothenburg]."},{"key":"ref_29","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, \u0141., and Polosukhin, I. (2017, January 4\u20139). Attention is all you need. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"19","DOI":"10.4018\/IJWLTT.2020040102","article-title":"ML-EC2: An algorithm for multi-label email classification using clustering","volume":"15","author":"Sharaff","year":"2020","journal-title":"Int. J. Web-Based Learn. Teach. Technol. (IJWLTT)"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Jlailaty, D., Grigori, D., and Belhajjame, K. (2017, January 25\u201330). Business process instances discovery from email logs. Proceedings of the 2017 IEEE International Conference on Services Computing (SCC), Honolulu, HI, USA.","DOI":"10.1109\/SCC.2017.12"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1007\/s10994-006-6226-1","article-title":"Extremely randomized trees","volume":"63","author":"Geurts","year":"2006","journal-title":"Mach. Learn."},{"key":"ref_33","unstructured":"Alkhereyf, S., and Rambow, O. (2020, January 11\u201316). Email classification incorporating social networks and thread structure. Proceedings of the Twelfth Language Resources and Evaluation Conference, Marseille, France."},{"key":"ref_34","unstructured":"Hamilton, W., Ying, Z., and Leskovec, J. (2017, January 4\u20139). Inductive representation learning on large graphs. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"1155","DOI":"10.2298\/CSIS220212034S","article-title":"Effective Methods for Email Classification: Is it a Business or Personal Email?","volume":"19","author":"Graovac","year":"2022","journal-title":"Comput. Sci. Inf. Syst."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"378","DOI":"10.1037\/h0031619","article-title":"Measuring nominal scale agreement among many raters","volume":"76","author":"Fleiss","year":"1971","journal-title":"Psychol. Bull."},{"key":"ref_37","unstructured":"Fleiss, J.L., Levin, B., and Paik, M.C. (2013). Statistical Methods for Rates and Proportions, John Wiley & Sons."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Krippendorff, K. (2018). Content Analysis: An Introduction to Its Methodology, Sage Publications.","DOI":"10.4135\/9781071878781"},{"key":"ref_39","first-page":"25","article-title":"Computing Krippendorff\u2019s Alpha-Reliability","volume":"1","author":"Krippendorff","year":"2011","journal-title":"Computing"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"159","DOI":"10.2307\/2529310","article-title":"The measurement of observer agreement for categorical data","volume":"33","author":"Landis","year":"1977","journal-title":"Biometrics"},{"key":"ref_41","unstructured":"Brandsen, A., Verberne, S., Lambers, K., Wansleeben, M., Calzolari, N., B\u00e9chet, F., Blache, P., Choukri, K., Cieri, C., and Declerck, T. (2020, January 11\u201316). Creating a dataset for named entity recognition in the archaeology domain. Proceedings of the Conference Proceedings LREC 2020, Marseille, France."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Sechidis, K., Tsoumakas, G., and Vlahavas, I. (2011, January 5\u20139). On the stratification of multi-label data. Proceedings of the Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2011, Athens, Greece.","DOI":"10.1007\/978-3-642-23808-6_10"},{"key":"ref_43","unstructured":"Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Masala, M., Ruseti, S., and Dascalu, M. (2020, January 8\u201313). Robert\u2013a romanian bert model. Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain.","DOI":"10.18653\/v1\/2020.coling-main.581"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzm\u00e1n, F., Grave, E., Ott, M., Zettlemoyer, L., and Stoyanov, V. (2019). Unsupervised cross-lingual representation learning at scale. arXiv.","DOI":"10.18653\/v1\/2020.acl-main.747"},{"key":"ref_46","unstructured":"Sagen, M. (2021). Large-Context Question Answering with Cross-Lingual Transfer. [Master\u2019s Thesis, Uppsala University, Department of Information Technology]."}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/14\/6\/321\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T19:47:45Z","timestamp":1760125665000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/14\/6\/321"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,3]]},"references-count":46,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2023,6]]}},"alternative-id":["info14060321"],"URL":"https:\/\/doi.org\/10.3390\/info14060321","relation":{},"ISSN":["2078-2489"],"issn-type":[{"value":"2078-2489","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,6,3]]}}}