{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,16]],"date-time":"2025-12-16T12:48:17Z","timestamp":1765889297530,"version":"build-2065373602"},"reference-count":43,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2024,4,7]],"date-time":"2024-04-07T00:00:00Z","timestamp":1712448000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Norte Portugal Regional Operational Programme (NORTE 2020)","award":["NORTE-01-0145-FEDER-000044","UIDB\/05757\/2020","UIDP\/05757\/2020","LA\/P\/0007\/2020"],"award-info":[{"award-number":["NORTE-01-0145-FEDER-000044","UIDB\/05757\/2020","UIDP\/05757\/2020","LA\/P\/0007\/2020"]}]},{"name":"PORTUGAL 2020 Partnership Agreement","award":["NORTE-01-0145-FEDER-000044","UIDB\/05757\/2020","UIDP\/05757\/2020","LA\/P\/0007\/2020"],"award-info":[{"award-number":["NORTE-01-0145-FEDER-000044","UIDB\/05757\/2020","UIDP\/05757\/2020","LA\/P\/0007\/2020"]}]},{"name":"European Regional Development Fund (ERDF)","award":["NORTE-01-0145-FEDER-000044","UIDB\/05757\/2020","UIDP\/05757\/2020","LA\/P\/0007\/2020"],"award-info":[{"award-number":["NORTE-01-0145-FEDER-000044","UIDB\/05757\/2020","UIDP\/05757\/2020","LA\/P\/0007\/2020"]}]},{"name":"national funds through FCT\/MCTES (PIDDAC)","award":["NORTE-01-0145-FEDER-000044","UIDB\/05757\/2020","UIDP\/05757\/2020","LA\/P\/0007\/2020"],"award-info":[{"award-number":["NORTE-01-0145-FEDER-000044","UIDB\/05757\/2020","UIDP\/05757\/2020","LA\/P\/0007\/2020"]}]},{"name":"CeDRI","award":["NORTE-01-0145-FEDER-000044","UIDB\/05757\/2020","UIDP\/05757\/2020","LA\/P\/0007\/2020"],"award-info":[{"award-number":["NORTE-01-0145-FEDER-000044","UIDB\/05757\/2020","UIDP\/05757\/2020","LA\/P\/0007\/2020"]}]},{"name":"SusTEC","award":["NORTE-01-0145-FEDER-000044","UIDB\/05757\/2020","UIDP\/05757\/2020","LA\/P\/0007\/2020"],"award-info":[{"award-number":["NORTE-01-0145-FEDER-000044","UIDB\/05757\/2020","UIDP\/05757\/2020","LA\/P\/0007\/2020"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Systems"],"abstract":"<jats:p>The rapid evolution of technology has fostered an exponential rise in the number of individuals and devices interconnected via the Internet. This interconnectedness has prompted companies to expand their computing and communication infrastructures significantly to accommodate the escalating demands. However, this proliferation of connectivity has also opened new avenues for cyber threats, emphasizing the critical need for Intrusion Detection Systems (IDSs) to adapt and operate efficiently in this evolving landscape. In response, companies are increasingly seeking IDSs characterized by horizontal, modular, and elastic attributes, capable of dynamically scaling with the fluctuating volume of network data flows deemed essential for effective monitoring and threat detection. Yet, the task extends beyond mere data capture and storage; robust IDSs must integrate sophisticated components for data analysis and anomaly detection, ideally functioning in real-time or near real-time. While Machine Learning (ML) techniques present promising avenues for detecting and mitigating malicious activities, their efficacy hinges on the availability of high-quality training datasets, which in turn poses a significant challenge. This paper proposes a comprehensive solution in the form of an architecture and reference implementation for (near) real-time capture, storage, and analysis of network data within a 1 Gbps network environment. Performance benchmarks provided offer valuable insights for prototype optimization, demonstrating the capability of the proposed IDS architecture to meet objectives even under realistic operational scenarios.<\/jats:p>","DOI":"10.3390\/systems12040126","type":"journal-article","created":{"date-parts":[[2024,4,8]],"date-time":"2024-04-08T03:11:33Z","timestamp":1712545893000},"page":"126","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Parameterization and Performance Analysis of a Scalable, near Real-Time Packet Capturing Platform"],"prefix":"10.3390","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4997-4757","authenticated-orcid":false,"given":"Rafael","family":"Oliveira","sequence":"first","affiliation":[{"name":"Research Centre in Digitalization and Intelligent Robotics (CeDRI), Instituto Polit\u00e9cnico de Bragan\u00e7a, Campus de Santa Apol\u00f3nia, 5300-253 Bragan\u00e7a, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4873-2705","authenticated-orcid":false,"given":"Tiago","family":"Pedrosa","sequence":"additional","affiliation":[{"name":"Research Centre in Digitalization and Intelligent Robotics (CeDRI), Instituto Polit\u00e9cnico de Bragan\u00e7a, Campus de Santa Apol\u00f3nia, 5300-253 Bragan\u00e7a, Portugal"},{"name":"Laborat\u00f3rio Associado Para a Sustentabilidade e Tecnologia em Regi\u00f5es de Montanha (SuSTEC), Instituto Polit\u00e9cnico de Bragan\u00e7a, Campus de Santa Apol\u00f3nia, 5300-253 Bragan\u00e7a, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1344-8264","authenticated-orcid":false,"given":"Jos\u00e9","family":"Rufino","sequence":"additional","affiliation":[{"name":"Research Centre in Digitalization and Intelligent Robotics (CeDRI), Instituto Polit\u00e9cnico de Bragan\u00e7a, Campus de Santa Apol\u00f3nia, 5300-253 Bragan\u00e7a, Portugal"},{"name":"Laborat\u00f3rio Associado Para a Sustentabilidade e Tecnologia em Regi\u00f5es de Montanha (SuSTEC), Instituto Polit\u00e9cnico de Bragan\u00e7a, Campus de Santa Apol\u00f3nia, 5300-253 Bragan\u00e7a, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9170-5078","authenticated-orcid":false,"given":"Rui Pedro","family":"Lopes","sequence":"additional","affiliation":[{"name":"Research Centre in Digitalization and Intelligent Robotics (CeDRI), Instituto Polit\u00e9cnico de Bragan\u00e7a, Campus de Santa Apol\u00f3nia, 5300-253 Bragan\u00e7a, Portugal"},{"name":"Laborat\u00f3rio Associado Para a Sustentabilidade e Tecnologia em Regi\u00f5es de Montanha (SuSTEC), Instituto Polit\u00e9cnico de Bragan\u00e7a, Campus de Santa Apol\u00f3nia, 5300-253 Bragan\u00e7a, Portugal"}]}],"member":"1968","published-online":{"date-parts":[[2024,4,7]]},"reference":[{"key":"ref_1","unstructured":"World Economic Forum (2024, February 01). The Global Risks Report 2019. Available online: http:\/\/www3.weforum.org\/docs\/WEF_Global_Risks_Report_2019.pdf."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1125","DOI":"10.1007\/s10207-023-00682-2","article-title":"A systematic literature review for network intrusion detection system (IDS)","volume":"22","author":"Abdulganiyu","year":"2023","journal-title":"Int. J. Inf. Secur."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Oliveira, R., Almeida, J., Pra\u00e7a, I., Lopes, R., and Pedrosa, T. (2021, January 19\u201321). A scalable, real-time packet capturing solution. Proceedings of the International Conference on Optimization, Learning Algorithms and Applications, Bragan\u00e7a, Portugal.","DOI":"10.1007\/978-3-030-91885-9_46"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Venkatesan, N.J., Kim, E., and Shin, D.R. (2016, January 6\u20138). PoN: Open source solution for real-time data analysis. Proceedings of the 2016 Third International Conference on Digital Information Processing, Data Mining, and Wireless Communications (DIPDMWC), Moscow, Russia.","DOI":"10.1109\/DIPDMWC.2016.7529409"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Wahal, M., Choudhury, T., and Arora, M. (2018, January 11\u201312). Intrusion Detection System in Python. Proceedings of the 8th International Conference on Cloud Computing, Data Science & Engineering (Confluence), Noida, India.","DOI":"10.1109\/CONFLUENCE.2018.8442909"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"629","DOI":"10.1016\/j.ins.2019.10.018","article-title":"A fully scalable big data framework for Botnet detection based on network traffic analysis","volume":"512","author":"Mousavi","year":"2020","journal-title":"Inf. Sci."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"490","DOI":"10.1109\/TLA.2021.9447699","article-title":"Performance Analysis of Packet Sniffing Techniques Applied to Network Monitoring","volume":"19","year":"2021","journal-title":"IEEE Lat. Am. Trans."},{"key":"ref_8","unstructured":"Chappell, L., and Combs, G. (2017). Wireshark 101: Essential Skills for Network Analysis, Laura Chappell University. [2nd ed.]."},{"key":"ref_9","unstructured":"Alleyne, N. (2020). Learning by Practicing: Mastering TShark Network Forensics: Moving from Zero to Hero, n3Security Inc."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Shen, Y., Zhang, Q.f., Ping, L.d., Wang, Y.f., and Li, W.j. (2012, January 25\u201327). A Multi-tunnel VPN Concurrent System for New Generation Network Based on User Space. Proceedings of the 2012 IEEE 11th International Conference on Trust, Security and Privacy in Computing and Communications, Liverpool, UK.","DOI":"10.1109\/TrustCom.2012.41"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Moon, J.H., and Shine, Y.T. (2020, January 19\u201322). A study of distributed SDN controller based on apache kafka. Proceedings of the Proceedings-2020 IEEE International Conference on Big Data and Smart Computing, BigComp, Busan, Republic of Korea.","DOI":"10.1109\/BigComp48618.2020.0-101"},{"key":"ref_12","unstructured":"Kafka Team (2021, September 20). Documentation. Available online: https:\/\/kafka.apache.org\/documentation."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Surekha, D., Swamy, G., and Venkatramaphanikumar, S. (2016, January 25\u201327). Real time streaming data storage and processing using storm and analytics with Hive. Proceedings of the 2016 International Conference on Advanced Communication Control and Computing Technologies, ICACCCT, Ramanathapuram, India.","DOI":"10.1109\/ICACCCT.2016.7831712"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Kumar, S., and Goel, E. (2018, January 20\u201321). Changing the world of Autonomous Vehicles using Cloud and Big Data. Proceedings of the International Conference on Inventive Communication and Computational Technologies, ICICCT, Coimbatore, India.","DOI":"10.1109\/ICICCT.2018.8473347"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Toshniwal, A., Rathore, K.S., Dubey, A., Dhasal, P., and Maheshwari, R. (2020, January 13\u201315). Media Streaming in Cloud with Special Reference to Amazon Web Services: A Comprehensive Review. Proceedings of the 2020 4th International Conference on Intelligent Computing and Control Systems (ICICCS), Madurai, India.","DOI":"10.1109\/ICICCS48265.2020.9121097"},{"key":"ref_16","unstructured":"Dhruba Borthakur (2021, September 20). HDFS Architecture Guide. Available online: https:\/\/hadoop.apache.org\/docs\/r1.2.1\/hdfs_design.html."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Ghemawat, S., Gobioff, H., and Leung, S.T. (2003, January 19\u201322). The Google File System. Proceedings of the 19th ACM Symposium on Operating Systems Principles, Bolton Landing, NY, USA.","DOI":"10.1145\/945445.945450"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Rao, B.P., and Rao, N.N. (2017, January 23\u201324). HDFS memory usage analysis. Proceedings of the International Conference on Inventive Computing and Informatics, ICICI, Coimbatore, India.","DOI":"10.1109\/ICICI.2017.8365298"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Tian, Y., and Yu, X. (2021, January 22\u201324). Trustworthiness study of HDFS data storage based on trustworthiness metrics and KMS encryption. Proceedings of the 2021 IEEE International Conference on Power Electronics, Computer Applications, ICPECA, Shenyang, China.","DOI":"10.1109\/ICPECA51329.2021.9362537"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Madhu, K.S., Reddy, B.C., Damarukanadhan, C.H., Polireddy, M., and Ravinder, N. (2021, January 20\u201322). Real Time Sentimental Analysis on Twitter. Proceedings of the 6th International Conference on Inventive Computation Technologies, ICICT, Coimbatore, India.","DOI":"10.1109\/ICICT50816.2021.9358772"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Mishra, S., Shukla, P.K., and Agarwal, R. (2020, January 28\u201329). Location wise opinion mining of real time twitter data using hadoop to reduce cyber crimes. Proceedings of the 2nd International Conference on Data, Engineering and Applications, IDEA, Bhopal, India.","DOI":"10.1109\/IDEA49133.2020.9170700"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Aziz, K., Zaidouni, D., and Bellafkih, M. (2018, January 26\u201327). Real-time data analysis using Spark and Hadoop. Proceedings of the 2018 International Conference on Optimization and Applications, ICOA, Mohammedia, Morocco.","DOI":"10.1109\/ICOA.2018.8370593"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Kamal, R., Shah, M.A., Hanif, A., and Ahmad, J. (2017, January 7\u20138). Real-time opinion mining of Twitter data using spring XD and Hadoop. Proceedings of the ICAC 2017-2017 23rd IEEE International Conference on Automation and Computing: Addressing Global Challenges through Automation and Computing, Huddersfield, UK.","DOI":"10.23919\/IConAC.2017.8082091"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Tsai, J., Chang, T.Y., Fang, Y.H., and Chang, E.S. (2018, January 19\u201321). A Real-Time Traffic Flow Prediction System for National Freeways Based on the Spark Streaming Technique. Proceedings of the 2018 IEEE International Conference on Consumer Electronics-Taiwan, ICCE-TW 2018, Taichung, Taiwan.","DOI":"10.1109\/ICCE-China.2018.8448998"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Wu, C.F., Hsu, T.C., Yang, H., and Chung, Y.C. (2017, January 13\u201317). File placement mechanisms for improving write throughputs of cloud storage services based on Ceph and HDFS. Proceedings of the 2017 IEEE International Conference on Applied System Innovation: Applied System Innovation for Modern Technology, ICASI, Sapporo, Japan.","DOI":"10.1109\/ICASI.2017.7988272"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Kirby, A., Henson, B., Thomas, J., Armstrong, M., and Galloway, M. (2019, January 8\u201310). Storage and file structure of a bioinformatics cloud architecture. Proceedings of the Proceedings-2019 3rd IEEE International Conference on Cloud and Fog Computing Technologies and Applications, Cloud Summit, Washington, DC, USA.","DOI":"10.1109\/CloudSummit47114.2019.00024"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Yang, C.T., Lien, W.H., Shen, Y.C., and Leu, F.Y. (2015, January 24\u201327). Implementation of a Software-Defined Storage Service with Heterogeneous Storage Technologies. Proceedings of the IEEE 29th International Conference on Advanced Information Networking and Applications Workshops, WAINA 2015, Gwangju, Republic of Korea.","DOI":"10.1109\/WAINA.2015.50"},{"key":"ref_28","unstructured":"Zaharia, M., Chowdhury, M., Franklin, M.J., Shenker, S., and Stoica, I. (2010, January 22\u201325). Spark: Cluster Computing with Working Sets. Proceedings of the 2nd USENIX Conference on Hot Topics in Cloud Computing, Boston MA, USA."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Benlachmi, Y., and Hasnaoui, M.L. (2020, January 27\u201328). Big data and spark: Comparison with hadoop. Proceedings of the World Conference on Smart Trends in Systems, Security and Sustainability, WS4, London, UK.","DOI":"10.1109\/WorldS450073.2020.9210353"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Verma, A., Mansuri, A.H., and Jain, N. (2016, January 18\u201319). Big data management processing with Hadoop MapReduce and spark technology: A comparison. Proceedings of the 2016 Symposium on Colossal Data Analysis and Networking (CDAN), Indore, India.","DOI":"10.1109\/CDAN.2016.7570891"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Mishra, S., and Hota, C. (2019, January 13\u201315). A REST Framework on IoT Streams using Apache Spark for Smart Cities. Proceedings of the 2019 IEEE 16th India Council International Conference (INDICON).","DOI":"10.1109\/INDICON47234.2019.9029012"},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Saraswathi, A., Mummoorthy, A., Raman G.R., A., and Porkodi, K. (2019, January 18\u201319). Real-Time Traffic Monitoring System Using Spark. Proceedings of the 2019 International Conference on Emerging Trends in Science and Engineering (ICESE), Hyderabad, India.","DOI":"10.1109\/ICESE46178.2019.9194613"},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Drohobytskiy, Y., Brevus, V., and Skorenkyy, Y. (2020, January 21\u201325). Spark structured streaming: Customizing kafka stream processing. Proceedings of the 2020 IEEE 3rd International Conference on Data Stream Mining and Processing, DSMP, Lviv, Ukraine.","DOI":"10.1109\/DSMP47368.2020.9204304"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Chen, Z., Chen, N., and Gong, J. (2015, January 20\u201324). Design and implementation of the real-time GIS data model and Sensor Web service platform for environmental big data management with the Apache Storm. Proceedings of the 2015 Fourth International Conference on Agro-Geoinformatics (Agro-geoinformatics), Istanbul, Turkey.","DOI":"10.1109\/Agro-Geoinformatics.2015.7248139"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Karimov, J., Rabl, T., Katsifodimos, A., Samarev, R., Heiskanen, H., and Markl, V. (2018, January 16\u201319). Benchmarking distributed stream data processing systems. Proceedings of the IEEE 34th International Conference on Data Engineering, ICDE 2018, Paris, France.","DOI":"10.1109\/ICDE.2018.00169"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Karakaya, Z., Yazici, A., and Alayyoub, M. (2017, January 6\u20137). A Comparison of Stream Processing Frameworks. Proceedings of the 2017 International Conference on Computer and Applications (ICCA), Doha, Qatar.","DOI":"10.1109\/COMAPP.2017.8079733"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Tun, M.T., Nyaung, D.E., and Phyu, M.P. (2019, January 6\u20137). Performance Evaluation of Intrusion Detection Streaming Transactions Using Apache Kafka and Spark Streaming. Proceedings of the 2019 International Conference on Advanced Information Technologies, ICAIT, Yangon, Myanmar.","DOI":"10.1109\/AITC.2019.8920960"},{"key":"ref_38","unstructured":"Dean, J., and Ghemawat, S. (2004, January 6\u20138). MapReduce: Simplified Data Processing on Large Clusters. Proceedings of the OSDI\u201904: Sixth Symposium on Operating System Design and Implementation, San Francisco, CA, USA."},{"key":"ref_39","first-page":"1","article-title":"Big data and hadoop-A technological survey","volume":"Volume 2018","author":"Manwal","year":"2017","journal-title":"Proceedings of the 2017 International Conference on Emerging Trends in Computing and Communication Technologies, ICETCCT"},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Chen, F., Liu, J., and Zhu, Y. (2017, January 25-30). A Real-Time Scheduling Strategy Based on Processing Framework of Hadoop. Proceedings of the 2017 IEEE International Congress on Big Data (BigData Congress), Honolulu, HI, USA.","DOI":"10.1109\/BigDataCongress.2017.48"},{"key":"ref_41","unstructured":"Dugan, J., Elliott, S., Mah, B.A., Poskanzer, J., and Prabhu, K. (2021, September 20). iPerf3 Network Benchmark. Available online: https:\/\/iperf.fr\/."},{"key":"ref_42","unstructured":"Stahn, M. (2021, September 20). Pypacker\u2014The Fastest and Simplest Packet Manipulation Lib for Python. Available online: https:\/\/gitlab.com\/mike01\/pypacker."},{"key":"ref_43","unstructured":"Biondi, P. (2021, September 20). Scapy\u2014Packet Crafting for Python2 and Python3. Available online: https:\/\/scapy.net\/."}],"container-title":["Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2079-8954\/12\/4\/126\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T14:24:26Z","timestamp":1760106266000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2079-8954\/12\/4\/126"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,7]]},"references-count":43,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2024,4]]}},"alternative-id":["systems12040126"],"URL":"https:\/\/doi.org\/10.3390\/systems12040126","relation":{},"ISSN":["2079-8954"],"issn-type":[{"type":"electronic","value":"2079-8954"}],"subject":[],"published":{"date-parts":[[2024,4,7]]}}}