{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,1]],"date-time":"2026-04-01T18:17:00Z","timestamp":1775067420772,"version":"3.50.1"},"reference-count":58,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2019,10,5]],"date-time":"2019-10-05T00:00:00Z","timestamp":1570233600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2019,10,5]],"date-time":"2019-10-05T00:00:00Z","timestamp":1570233600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Big Data"],"published-print":{"date-parts":[[2019,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>With the growing use of information technology in all life domains, hacking has become more negatively effective than ever before. Also with developing technologies, attacks numbers are growing exponentially every few months and become more sophisticated so that traditional IDS becomes inefficient detecting them. This paper proposes a solution to detect not only new threats with higher detection rate and lower false positive than already used IDS, but also it could detect collective and contextual security attacks. We achieve those results by using Networking Chatbot, a deep recurrent neural network: Long Short Term Memory (LSTM) on top of Apache Spark Framework that has an input of flow traffic and traffic aggregation and the output is a language of two words, normal or abnormal. We propose merging the concepts of language processing, contextual analysis, distributed deep learning, big data, anomaly detection of flow analysis. We propose a model that describes the network abstract normal behavior from a sequence of millions of packets within their context and analyzes them in near real-time to detect point, collective and contextual anomalies. Experiments are done on MAWI dataset, and it shows better detection rate not only than signature IDS, but also better than traditional anomaly IDS. The experiment shows lower false positive, higher detection rate and better point anomalies detection. As for prove of contextual and collective anomalies detection, we discuss our claim and the reason behind our hypothesis. But the experiment is done on random small subsets of the dataset because of hardware limitations, so we share experiment and our future vision thoughts as we wish that full prove will be done in future by other interested researchers who have better hardware infrastructure than ours.<\/jats:p>","DOI":"10.1186\/s40537-019-0248-6","type":"journal-article","created":{"date-parts":[[2019,10,5]],"date-time":"2019-10-05T08:02:44Z","timestamp":1570262564000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":40,"title":["Big data analysis and distributed deep learning for next-generation intrusion detection system optimization"],"prefix":"10.1186","volume":"6","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-9474-9204","authenticated-orcid":false,"given":"Khloud","family":"Al Jallad","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mohamad","family":"Aljnidi","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mohammad Said","family":"Desouki","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2019,10,5]]},"reference":[{"issue":"10","key":"248_CR1","first-page":"6132","volume":"2","author":"MC Raja","year":"2014","unstructured":"Raja MC, Rabbani MA. Big data analytics security issues in data driven information system. Int J Innov Res Comput Commun Eng. 2014;2(10):6132\u20135.","journal-title":"Int J Innov Res Comput Commun Eng"},{"issue":"3","key":"248_CR2","first-page":"69","volume":"4","author":"M Bijone","year":"2016","unstructured":"Bijone M. A survey on secure network: intrusion detection & prevention approaches. Am J Inf Syst. 2016;4(3):69\u201388.","journal-title":"Am J Inf Syst"},{"key":"248_CR3","unstructured":"Alaidaros H, Mahmuddin M, Mazari AA. An overview of flow-based and packet-based intrusion detection performance in high speed networks. 2017."},{"key":"248_CR4","unstructured":"Djordjevic V. Anomaly detection. 2018. http:\/\/thingsolver.com\/anomaly-detection\/ . Accessed 2018."},{"issue":"1","key":"248_CR5","doi-asserted-by":"publisher","first-page":"48","DOI":"10.12691\/jcn-4-1-5","volume":"4","author":"L Wang","year":"2017","unstructured":"Wang L. Big Data in intrusion detection systems and intrusion prevention systems. J Comput Netw. 2017;4(1):48\u201355.","journal-title":"J Comput Netw"},{"key":"248_CR6","doi-asserted-by":"publisher","first-page":"4","DOI":"10.1016\/j.ins.2011.08.011","volume":"231","author":"J Song","year":"2013","unstructured":"Song J, Takakura H, Okabe Y, Nakao K. Toward a more practical unsupervised anomaly detection system. Inf Sci. 2013;231:4\u201314.","journal-title":"Inf Sci"},{"key":"248_CR7","doi-asserted-by":"crossref","unstructured":"Zhao G, Song J, Song J. Analysis about performance of multiclass SVM applying in IDS. In: International conference on information business and education technology (ICIBET 2013). Atlantis Press. 2013.","DOI":"10.2991\/icibet.2013.46"},{"key":"248_CR8","doi-asserted-by":"publisher","first-page":"325","DOI":"10.1007\/978-3-319-12643-2_40","volume-title":"Neural Information Processing","author":"Shohei Araki","year":"2014","unstructured":"Araki S, Yamaguchi Y, Shimada H, Takakura H. Unknown attack detection by multistage One-Class SVM focusing on communication interval. In: Neural information processing. 2014. p. 325\u201332."},{"key":"248_CR9","doi-asserted-by":"crossref","unstructured":"Enache AC, Patriciu VV. Intrusions detection based on support vector machine optimized with swarm intelligence. In: 9th international symposium on applied computational intelligence and informatics (SACI). 2014.","DOI":"10.1109\/SACI.2014.6840052"},{"issue":"2","key":"248_CR10","doi-asserted-by":"publisher","first-page":"1822","DOI":"10.1016\/j.eswa.2011.08.068","volume":"39","author":"CA Catania","year":"2012","unstructured":"Catania CA, Bromberg F, Garino CG. An autonomous labeling approach to support vector machines algorithms for network traffic anomaly detection. Expert Syst Appl. 2012;39(2):1822\u20139.","journal-title":"Expert Syst Appl"},{"issue":"6","key":"248_CR11","doi-asserted-by":"publisher","first-page":"7698","DOI":"10.1016\/j.eswa.2010.12.141","volume":"38","author":"Y Yi","year":"2011","unstructured":"Yi Y, Wu J, Xu W. Incremental SVM based on reserved set for network intrusion detection. Expert Syst Appl. 2011;38(6):7698\u2013707.","journal-title":"Expert Syst Appl"},{"key":"248_CR12","unstructured":"Atefi K, Yahya S, Dak AY, Atefi A. A hybrid intrusion detection system based on different machine learning algorithms. In: 4th international conference on computing and informatics, Sarawak, Malaysia. 2013."},{"issue":"7\u20138","key":"248_CR13","doi-asserted-by":"publisher","first-page":"1671","DOI":"10.1007\/s00521-013-1370-6","volume":"24","author":"I Ahmad","year":"2014","unstructured":"Ahmad I, Hussain M, Alghamdi A, Alelaiwi A. Enhancing SVM performance in intrusion detection using optimal feature subset selection based on genetic principal components. Neural Comput Appl. 2014;24(7\u20138):1671\u201382.","journal-title":"Neural Comput Appl"},{"key":"248_CR14","unstructured":"Sung AH, Mukkamala S. Identifying important features for intrusion detection using support vector machines and neural networks. In: Symposium on applications and the internet. 2003."},{"key":"248_CR15","doi-asserted-by":"publisher","first-page":"178","DOI":"10.1016\/j.asoc.2014.01.028","volume":"18","author":"F Kuang","year":"2014","unstructured":"Kuang F, Xu W, Zhang S. A novel hybrid KPCA and SVM with GA model for intrusion detection. Appl Soft Comput. 2014;18:178\u201384.","journal-title":"Appl Soft Comput"},{"key":"248_CR16","doi-asserted-by":"publisher","first-page":"231","DOI":"10.1016\/j.cose.2014.06.006","volume":"45","author":"R Chitrakar","year":"2014","unstructured":"Chitrakar R, Huang C. Selection of candidate support vectors in incremental SVM for network intrusion detection. Comput Secur. 2014;45:231\u201341.","journal-title":"Comput Secur"},{"key":"248_CR17","doi-asserted-by":"crossref","unstructured":"Ahn S-H, Kim N-U, Chung T-M. Big data analysis system concept for detecting unknown attacks. In: ICACT. 2014. p. 269\u201372.","DOI":"10.1109\/ICACT.2014.6778962"},{"issue":"12","key":"248_CR18","first-page":"229","volume":"4","author":"MGK Jangla","year":"2015","unstructured":"Jangla MGK, Deepa M, Amne A. Development of an intrusion detection system based on Big data for detecting unknown attacks. Int J Adv Res Comput Commun Eng. 2015;4(12):229\u201332.","journal-title":"Int J Adv Res Comput Commun Eng"},{"issue":"9150965","key":"248_CR19","first-page":"17","volume":"2017","author":"M Iturbe","year":"2017","unstructured":"Iturbe M, Garitano I, Zurutuza U, Uribeetxeberria R. Towards large-scale, heterogeneous anomaly detection systems in industrial networks: a survey of current trends. Secur Commun Netw. 2017;2017(9150965):17.","journal-title":"Secur Commun Netw"},{"key":"248_CR20","doi-asserted-by":"crossref","unstructured":"Yen T-F, Oprea A, Onarlioglu K. Beehive: large-scale log analysis for detecting suspicious activity in enterprise networks. In: Proceedings of the 29th annual computer security applications conference, ACSAC 2013. 2013.","DOI":"10.1145\/2523649.2523670"},{"key":"248_CR21","doi-asserted-by":"crossref","unstructured":"Bumgardner VKC, Marek VW. Scalable hybrid stream and hadoop network analysis system. In: Proceedings of the 5th ACM\/SPEC international conference on performance engineering, ICPE 2014, Dublin. 2014.","DOI":"10.1145\/2568088.2568103"},{"key":"248_CR22","doi-asserted-by":"crossref","unstructured":"Camacho J, Macia-Fernandez G, Diaz-Verdejo J, Garcia-Teodoro P. Tackling the big data 4 vs for anomaly detection. In: Proceedings of the IEEE conference on computer communications workshops INFOCOM WKSHPS 2014. 2014. p. 500\u20135.","DOI":"10.1109\/INFCOMW.2014.6849282"},{"key":"248_CR23","doi-asserted-by":"crossref","unstructured":"Dromard J, Roudire G, Owezarski P. Unsupervised network anomaly detection in real-time on big data. In: New trends in databases and information systems, Berlin, Germany. 2015.","DOI":"10.1007\/978-3-319-23201-0_22"},{"issue":"3","key":"248_CR24","first-page":"93","volume":"1","author":"P Giura","year":"2012","unstructured":"Giura P, Wang W. Using large scale distributed computing to unveil advanced persistent threats. Science. 2012;1(3):93\u2013105.","journal-title":"Science"},{"key":"248_CR25","doi-asserted-by":"publisher","first-page":"824","DOI":"10.1016\/j.procs.2016.07.238","volume":"93","author":"GP Gupta","year":"2016","unstructured":"Gupta GP, Kulariya M. A framework for fast and efficient cyber security network intrusion detection using apache spark. Procedia Comput Sci. 2016;93:824\u201331.","journal-title":"Procedia Comput Sci"},{"key":"248_CR26","doi-asserted-by":"crossref","unstructured":"Gon\u00e7alves D, Bota J, Correia M. Big data analytics for detecting host misbehavior in large logs. In: Proceedings of the 14th IEEE international conference on trust. 2015.","DOI":"10.1109\/Trustcom.2015.380"},{"key":"248_CR27","doi-asserted-by":"crossref","unstructured":"Fontugne R, Mazel J, Fukuda K. Hashdoop: a MapReduce framework for network anomaly detection. In: Proceedings of the 2014 IEEE conference on computer communications workshops, INFOCOM WKSHPS 2014. 2014.","DOI":"10.1109\/INFCOMW.2014.6849281"},{"key":"248_CR28","unstructured":"Marchal S, Jiang X, State R, Engel T. A big data architecture for large scale security monitoring. In: Proceedings of the 3rd IEEE international congress on big data, Alaska, USA."},{"key":"248_CR29","doi-asserted-by":"crossref","unstructured":"Tazaki H, Okada K, Sekiya Y, Kadobayashi Y. MATATABI: multi-layer threat analysis platform with Hadoop. In: Proceedings of the 3rd international workshop on building analysis datasets and gathering experience returns for security. 2014.","DOI":"10.1109\/BADGERS.2014.12"},{"issue":"9","key":"248_CR30","doi-asserted-by":"publisher","first-page":"3489","DOI":"10.1007\/s11227-015-1615-5","volume":"72","author":"MM Rathore","year":"2016","unstructured":"Rathore MM, Ahmad A, Paul A. Real time intrusiondetection system for ultra-high-speed big data environments. J Supercomput. 2016;72(9):3489\u2013510.","journal-title":"J Supercomput"},{"key":"248_CR31","doi-asserted-by":"crossref","unstructured":"Ratner AS, Kelly P. Anomalies in network traffic. In: Proceedings of the 11th IEEE international conference on intelligence and security informatics, IEEE ISI 2013. 2013.","DOI":"10.1109\/ISI.2013.6578820"},{"key":"248_CR32","doi-asserted-by":"crossref","unstructured":"Therdphapiyanak J, Piromsopa K. An analysis of suitable parameters for efficiently applying K-means clustering to large TCPdump data set using Hadoop framework. In: Proceedings 2013 10th international conference on electrical engineering\/electronics, computer, telecommunications and information technology, ECTI-CON 2013. 2013.","DOI":"10.1109\/ECTICon.2013.6559650"},{"key":"248_CR33","first-page":"175","volume-title":"Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering","author":"Geng Tian","year":"2015","unstructured":"Tian G, Wang Z, Yin X. TADOOP: mining network traffic anomalies with Hadoop. In: Security and privacy in communication networks. 2015."},{"key":"248_CR34","doi-asserted-by":"crossref","unstructured":"Wang Z, Yang J, Zhang H, Li C, Zhang S, Wang H. Towards online anomaly detection by combining multiple detection methods and storm. In: Proceedings of the 2016 IEEE\/IFIP network operations and management symposium. 2016.","DOI":"10.1109\/NOMS.2016.7502903"},{"key":"248_CR35","doi-asserted-by":"crossref","unstructured":"Xu W, Huang L, Fox A, Patterson D, Jordan MI. Detecting large-scale system problems by mining consolelogs. In: Proceedings of the 22nd ACM SIGOPS symposium on operating systems principles, SOSP\u201909. 2009.","DOI":"10.1145\/1629575.1629587"},{"issue":"4","key":"248_CR36","doi-asserted-by":"publisher","first-page":"231","DOI":"10.1007\/s10207-012-0163-8","volume":"11","author":"D Hadziosmanovi\u00b4c","year":"2012","unstructured":"Hadziosmanovi\u00b4c D, Bolzoni D, Hartel PH. A log mining approach for process monitoring in SCADA. Int J Inf Secur. 2012;11(4):231\u201351.","journal-title":"Int J Inf Secur"},{"issue":"6","key":"248_CR37","doi-asserted-by":"publisher","first-page":"39","DOI":"10.1109\/MIC.2013.84","volume":"17","author":"DE Difallah","year":"2013","unstructured":"Difallah DE, Cudre-Mauroux P, McKenna SA. Scalable anomaly detection for smart city infrastructure networks. IEEE Internet Comput. 2013;17(6):39\u201347.","journal-title":"IEEE Internet Comput"},{"key":"248_CR38","doi-asserted-by":"publisher","first-page":"417","DOI":"10.1016\/B978-0-12-805394-2.00017-9","volume-title":"Big data: principles and paradigms","author":"S Wallace","year":"2016","unstructured":"Wallace S, Zhao X, Nguyen D, Lu K-T. Chapter 17 - Big data analytics on a smart grid: mining PMU data for event and anomaly detection. In: Buyya R, Calheiros RN, Dastjerdi AV, editors. Big data: principles and paradigms. San Mateo, CA: Morgan Kaufmann; 2016. p. 417\u201329."},{"key":"248_CR39","doi-asserted-by":"crossref","unstructured":"Kiss I, Genge B, Haller P, Sebestyen G. Data clustering-based anomaly detection in industrial control systems. In: Proceedings of the 2014 10th IEEE international conference on intelligent computer communication and processing, ICCP 2014, Romania. 2014.","DOI":"10.1109\/ICCP.2014.6937009"},{"key":"248_CR40","doi-asserted-by":"crossref","unstructured":"Hurst W, Merabti M, Fergus P. Big data analysis techniques for cyber-threat detection in critical infrastructures. In: Proceedings of the 28th IEEE international conference on advanced information networking and applications workshops, IEEE WAINA 2014, 2014.","DOI":"10.1109\/WAINA.2014.141"},{"key":"248_CR41","doi-asserted-by":"crossref","unstructured":"Cho K. MAWI Working Group Traffic Archive. WIDE Project. 2011. http:\/\/mawi.wide.ad.jp\/mawi\/ . Accessed 17 Aug 2018.","DOI":"10.32722\/eb.v17i1.1146"},{"key":"248_CR42","unstructured":"WIDE Project. 1988\u20132018. http:\/\/www.wide.ad.jp\/ . Accessed 17 Aug 2018."},{"key":"248_CR43","unstructured":"\u201cnecoma-project (Nippon-European Cyberdefense-Oriented Multilayer threat Analysis),\u201d International Collaborative R&D Promotion Project of the Ministry of Internal Affairs and Communication, Japan, and by the European Union Seventh Framework Programme. 2013. http:\/\/www.necoma-project.eu\/ . Accessed 17 Aug 2018."},{"key":"248_CR44","volume-title":"Measuring and implementing internet backbone security: current challenges, upcoming deployment, and future trends","author":"M W\u00e4hlisch","year":"2015","unstructured":"W\u00e4hlisch M. Measuring and implementing internet backbone security: current challenges, upcoming deployment, and future trends. Berlin: Freie Universit\u00e4t Berlin; 2015."},{"key":"248_CR45","unstructured":"MAWIlab. http:\/\/www.fukuda-lab.org\/mawilab\/ . Accessed 17 Aug 2018."},{"key":"248_CR46","doi-asserted-by":"crossref","unstructured":"Mazel J, Fontugne R, Fukuda K. A taxonomy of anomalies in backbone network traffic. In: 5th international workshop on TRaffic analysis and characterization (TRAC). 2014.","DOI":"10.1109\/IWCMC.2014.6906328"},{"key":"248_CR47","unstructured":"Mawilab Documentation. http:\/\/www.fukuda-lab.org\/mawilab\/documentation.html#admd . Accessed 2018."},{"key":"248_CR48","unstructured":"Agurim. http:\/\/mawi.wide.ad.jp\/~agurim\/dataset\/ . Accessed 17 Aug 2018."},{"key":"248_CR49","doi-asserted-by":"crossref","unstructured":"Cho K. Recursive Lattice search: hierarchical heavy hitters revisited. In: ACM IMC 2017. 2017.","DOI":"10.1145\/3131365.3131377"},{"key":"248_CR50","unstructured":"Kato M, Cho K, Honda M, Tokuda H. Monitoring the dynamics of network traffic by recursive multi-dimensional aggregation. In: OSDI2012 MAD workshop, Hollywood, CA. 2012."},{"key":"248_CR51","unstructured":"Understanding-LSTMs. 2015. http:\/\/colah.github.io\/posts\/2015-08-Understanding-LSTMs\/ . Accessed 17 Aug 2018."},{"issue":"8","key":"248_CR52","doi-asserted-by":"publisher","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","volume":"9","author":"S Hochreiter","year":"1997","unstructured":"Hochreiter S, Schmidhuber J. Long short-term memory. Neural Comput. 1997;9(8):1735\u201380.","journal-title":"Neural Comput"},{"key":"248_CR53","unstructured":"Understanding-lstm-and-its-diagrams, medium. https:\/\/medium.com\/mlreview\/understanding-lstm-and-its-diagrams-37e2f46f1714 . Accessed 2019."},{"key":"248_CR54","unstructured":"Colab. https:\/\/colab.research.google.com\/ . Accessed 2018."},{"key":"248_CR55","unstructured":"Keras. https:\/\/keras.io\/ . Accessed 2018."},{"key":"248_CR56","unstructured":"Spark. http:\/\/spark.apache.org\/ . Accessed 2018."},{"key":"248_CR57","unstructured":"Noel C, Osindero S. Dogwild! distributed Hogwild for CPU & GPU. In: Distributed machine learning and matrix computations, NIPS 2014 workshop. 2014."},{"key":"248_CR58","unstructured":"Elephas. https:\/\/github.com\/maxpumperla\/elephas . Accessed 2018."}],"container-title":["Journal of Big Data"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s40537-019-0248-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/article\/10.1186\/s40537-019-0248-6\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1186\/s40537-019-0248-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,10,1]],"date-time":"2022-10-01T01:45:02Z","timestamp":1664588702000},"score":1,"resource":{"primary":{"URL":"https:\/\/journalofbigdata.springeropen.com\/articles\/10.1186\/s40537-019-0248-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,10,5]]},"references-count":58,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2019,12]]}},"alternative-id":["248"],"URL":"https:\/\/doi.org\/10.1186\/s40537-019-0248-6","relation":{},"ISSN":["2196-1115"],"issn-type":[{"value":"2196-1115","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,10,5]]},"assertion":[{"value":"10 April 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 September 2019","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"5 October 2019","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors Ethics approval and consent to participate.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"The authors consent for publication.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare that they have no competing interests.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"88"}}