{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,3]],"date-time":"2026-02-03T18:35:54Z","timestamp":1770143754093,"version":"3.49.0"},"reference-count":28,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2024,1,21]],"date-time":"2024-01-21T00:00:00Z","timestamp":1705795200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>Analysing message streams in a dynamic environment is challenging. Various methods and metrics are used to evaluate message classification solutions, but often fail to realistically simulate the actual environment. As a result, the evaluation can produce overly optimistic results, rendering current solution evaluations inadequate for real-world environments. This paper proposes a framework based on the simulation of real-world message streams to evaluate classification solutions. The framework consists of four modules: message stream simulation, processing, classification and evaluation. The simulation module uses techniques and queueing theory to replicate a real-world message stream. The processing module refines the input messages for optimal classification. The classification module categorises the generated message stream using existing solutions. The evaluation module evaluates the performance of the classification solutions by measuring accuracy, precision and recall. The framework can model different behaviours from different sources, such as different spammers with different attack strategies, press media or social network sources. Each profile generates a message stream that is combined into the main stream for greater realism. A spam detection case study is developed that demonstrates the implementation of the proposed framework and identifies latency and message body obfuscation as critical classification quality parameters.<\/jats:p>","DOI":"10.3390\/a17010047","type":"journal-article","created":{"date-parts":[[2024,1,22]],"date-time":"2024-01-22T06:49:31Z","timestamp":1705906171000},"page":"47","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Framework Based on Simulation of Real-World Message Streams to Evaluate Classification Solutions"],"prefix":"10.3390","volume":"17","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8298-3439","authenticated-orcid":false,"given":"Wenny","family":"Hojas-Mazo","sequence":"first","affiliation":[{"name":"Departamento de Inteligencia Artificial e Infraestructura de Sistemas Inform\u00e1ticos, Facultad de Ingenier\u00eda Inform\u00e1tica, Universidad Tecnol\u00f3gica de La Habana, Jos\u00e9 Antonio Echeverr\u00eda, Calle 114 #11901, entre 119 y 127, CUJAE, Marianao, La Habana 19390, Cuba"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2516-4728","authenticated-orcid":false,"given":"Francisco","family":"Maci\u00e1-P\u00e9rez","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Technology, University of Alicante, 03690 Alicante, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9007-6054","authenticated-orcid":false,"given":"Jos\u00e9 Vicente","family":"Bern\u00e1 Mart\u00ednez","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Technology, University of Alicante, 03690 Alicante, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7613-3382","authenticated-orcid":false,"given":"Mailyn","family":"Moreno-Espino","sequence":"additional","affiliation":[{"name":"Centro de Investigaci\u00f3n en Computaci\u00f3n, Instituto Polit\u00e9cnico Nacional, Ciudad de M\u00e9xico 07738, Mexico"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3597-4836","authenticated-orcid":false,"given":"Iren","family":"Lorenzo Fonseca","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Technology, University of Alicante, 03690 Alicante, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9553-8123","authenticated-orcid":false,"given":"Juan","family":"Pav\u00f3n","sequence":"additional","affiliation":[{"name":"Instituto de Tecnolog\u00eda del Conocimiento, Universidad Complutense de Madrid, 28040 Madrid, Spain"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2024,1,21]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Bularca, M., Nechita, F., Sargu, L., Motoi, G., Otovescu, A., and Coman, C. (2022). Looking for the Sustainability Messages of European Universities\u2019 Social Media Communication during the COVID-19 Pandemic. Sustainability, 14.","DOI":"10.3390\/su14031554"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"100423","DOI":"10.1016\/j.infoandorg.2022.100423","article-title":"Aligning adoption messages with audiences? priorities: A mixed-methods study of the diffusion of enterprise architecture among the US state governments","volume":"32","author":"Bui","year":"2022","journal-title":"Inf. Organ."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Hemker, S., Herrando, C., and Constantinides, E. (2021). The Transformation of Data Marketing: How an Ethical Lens on Consumer Data Collection Shapes the Future of Marketing. Sustainability, 13.","DOI":"10.3390\/su132011208"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"247","DOI":"10.15611\/aoe.2022.1.11","article-title":"The impact of social media peer communication on customer behaviour\u2014Evidence from Romania","volume":"1","author":"Anastasiei","year":"2022","journal-title":"Argum. Oecon."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"100356","DOI":"10.1016\/j.bdr.2022.100356","article-title":"Data Stream Classification Based on Extreme Learning Machine: Review","volume":"30","author":"Zheng","year":"2022","journal-title":"Big Data Res."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1145","DOI":"10.1007\/s10462-022-10195-4","article-title":"A review of spam email detection: Analysis of spammer strategies and the dataset shift problem","volume":"56","author":"Fidalgo","year":"2023","journal-title":"Artif. Intell. Rev."},{"key":"ref_7","first-page":"459","article-title":"The impact of latency on online classification learning with concept drift","volume":"Volume 6291","author":"Bi","year":"2010","journal-title":"Proceedings of the Knowledge Science, Engineering and Management 2010 (KSEM 2010)"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Souza, V., Pinho, T., and Batista, G. (2018, January 22\u201325). Evaluating Stream Classifiers with Delayed Labels Information. Proceedings of the 7th Brazilian Conference on Intelligent Systems (BRACIS), Sao Paulo, Brazil.","DOI":"10.1109\/BRACIS.2018.00077"},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"317","DOI":"10.1016\/j.patcog.2018.07.023","article-title":"Wild Patterns: Ten Years after the Rise of Adversarial Machine Learning","volume":"84","author":"Biggio","year":"2018","journal-title":"Pattern Recogn."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Dada, E., Bassi, J., Chiroma, H., Abdulhamid, S., Adetunmbi, A., and Ajibuwa, O. (2019). Machine learning for email spam filtering: Review, approaches and open research problems. Heliyon, 5.","DOI":"10.1016\/j.heliyon.2019.e01802"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"12487","DOI":"10.1016\/j.eswa.2012.04.064","article-title":"SDAI: An integral evaluation methodology for content-based spam filtering mode","volume":"39","year":"2012","journal-title":"Expert Syst. Appl."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Nelson, B., Biggio, B., and Laskov, P. (2011, January 21). Understanding the Risk Factors of Learning in Adversarial Environments. Proceedings of the 4th ACM Workshop on Security and Artificial Intelligence; AISec \u201911, Chicago, IL, USA.","DOI":"10.1145\/2046684.2046698"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Blockeel, H., Kersting, K., Nijssen, S., and \u017delezn\u00fd, F. (2013, January 23\u201327). Evasion Attacks against Machine Learning at Test Time. Proceedings of the Machine Learning and Knowledge Discovery in Databases, Prague, Czech Republic.","DOI":"10.1007\/978-3-642-40988-2"},{"key":"ref_14","unstructured":"(2024, January 16). Jubatus: Distributed Online Machine Learning Framework. Available online: http:\/\/jubat.us\/en\/."},{"key":"ref_15","first-page":"1601","article-title":"MOA: Massive Online Analysis","volume":"11","author":"Bifet","year":"2010","journal-title":"J. Mach. Learn. Res."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Clever, L., Pohl, J., Bossek, J., Kerschke, P., and Trautmann, H. (2022). Process-Oriented Stream Classification Pipeline: A Literature Review. Appl. Sci., 12.","DOI":"10.3390\/app12189094"},{"key":"ref_17","unstructured":"Gartner IT Glossary (2022, September 05). Frameworks. Available online: https:\/\/www.gartner.com\/en\/information-technology\/glossary\/framework."},{"key":"ref_18","unstructured":"Apache Software Foundation (2021). Apache Spark\u2014Unified Analytics Engine for Big Data, Apache Software Foundation."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1109\/MMUL.2022.3206398","article-title":"Distributed Architecture for an Elderly Accompaniment Service Based on IoT Devices, AI, and Cloud Services","volume":"30","author":"Fonseca","year":"2023","journal-title":"IEEE MultiMedia"},{"key":"ref_20","unstructured":"Hiller, F., and Lieberman, G. (2010). Introduction to Operations Research; Raghothaman Srinivasan, McGraw-Hill Science. Chapter Simulation."},{"key":"ref_21","unstructured":"Hiller, F., and Lieberman, G. (2010). Introduction to Operations Research; Raghothaman Srinivasan, McGraw-Hill Science. Chapter Queueing Theory."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Garc\u00eda, S., Luengo, J., and Herrera, F. (2015). Data Preprocessing in Data Mining, Springer. [1st ed.].","DOI":"10.1007\/978-3-319-10247-4"},{"key":"ref_23","unstructured":"Pyle, D. (1999). Data Preparation for Data Mining, Morgan Kaufmann Publishers Inc.. [1st ed.]."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1016\/j.neucom.2017.01.078","article-title":"A survey on data preprocessing for data stream mining: Current status and future directions","volume":"239","author":"Krawczyk","year":"2017","journal-title":"Neurocomputing"},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"191","DOI":"10.1007\/s10844-008-0053-8","article-title":"An adaptive personalized news dissemination system","volume":"32","author":"Katakis","year":"2009","journal-title":"J. Intell. Inf. Syst."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"5019","DOI":"10.1007\/s10462-020-09814-9","article-title":"Applicability of machine learning in spam and phishing email filtering: Review and approaches","volume":"53","author":"Gangavarapu","year":"2020","journal-title":"Artif. Intell. Rev."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Ali, S., Islam, N., Rauf, A., Din, I., Guizani, M., and Rodrigues, J. (2018). Privacy and Security Issues in Online Social Networks. Future Internet, 10.","DOI":"10.3390\/fi10120114"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Yang, H., Liu, Q., Zhou, S., and Luo, Y. (2019). A Spam Filtering Method Based on Multi-Modal Fusion. Appl. Sci., 9.","DOI":"10.3390\/app9061152"}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/17\/1\/47\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T13:46:34Z","timestamp":1760103994000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/17\/1\/47"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,1,21]]},"references-count":28,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,1]]}},"alternative-id":["a17010047"],"URL":"https:\/\/doi.org\/10.3390\/a17010047","relation":{},"ISSN":["1999-4893"],"issn-type":[{"value":"1999-4893","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,1,21]]}}}