{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,25]],"date-time":"2025-11-25T05:03:09Z","timestamp":1764046989043,"version":"3.37.3"},"reference-count":56,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2021,5,20]],"date-time":"2021-05-20T00:00:00Z","timestamp":1621468800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,5,20]],"date-time":"2021-05-20T00:00:00Z","timestamp":1621468800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Big Data"],"published-print":{"date-parts":[[2021,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Class rarity is a frequent challenge in cybersecurity. Rarity occurs when the positive (attack) class only has a small number of instances for machine learning classifiers to train upon, thus making it difficult for the classifiers to discriminate and learn from the positive class. To investigate rarity, we examine three individual web attacks in big data from the CSE-CIC-IDS2018 dataset: \u201cBrute Force-Web\u201d, \u201cBrute Force-XSS\u201d, and \u201cSQL Injection\u201d. These three individual web attacks are also severely imbalanced, and so we evaluate whether random undersampling (RUS) treatments can improve the classification performance for these three individual web attacks. The following eight different levels of RUS ratios are evaluated: no sampling, 999:1, 99:1, 95:5, 9:1, 3:1, 65:35, and 1:1. For measuring classification performance, Area Under the Receiver Operating Characteristic Curve (AUC) metrics are obtained for the following seven different classifiers: Random Forest (RF), CatBoost (CB), LightGBM (LGB), XGBoost (XGB), Decision Tree (DT), Naive Bayes (NB), and Logistic Regression (LR) (with the first four learners being ensemble learners and for comparison, the last three being single learners). We find that applying random undersampling does improve overall classification performance with the AUC metric in a statistically significant manner. Ensemble learners achieve the top AUC scores after massive undersampling is applied, but the ensemble learners break down and have poor performance (worse than NB and DT) when no sampling is applied to our unique and harsh experimental conditions of severe class imbalance and rarity.<\/jats:p>","DOI":"10.1186\/s40537-021-00462-6","type":"journal-article","created":{"date-parts":[[2021,5,20]],"date-time":"2021-05-20T18:04:36Z","timestamp":1621533876000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["Investigating rarity in web attacks with ensemble learners"],"prefix":"10.1186","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5526-1094","authenticated-orcid":false,"given":"Richard","family":"Zuech","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"John","family":"Hancock","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Taghi M.","family":"Khoshgoftaar","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2021,5,20]]},"reference":[{"key":"462_CR1","unstructured":"Young J. US ecommerce sales grow 14.9% in 2019. https:\/\/www.digitalcommerce360.com\/article\/us-ecommerce-sales\/. Accessed 28 Nov 2020."},{"key":"462_CR2","doi-asserted-by":"crossref","unstructured":"Leevy JL, Hancock J, Zuech R, Khoshgoftaar TM. Detecting cybersecurity attacks using different network features with lightgbm and xgboost learners. In: 2020 IEEE second international conference on cognitive machine intelligence (CogMI). IEEE; 2020, pp. 190\u20137.","DOI":"10.1109\/CogMI50398.2020.00032"},{"key":"462_CR3","doi-asserted-by":"crossref","unstructured":"Wald R, Villanustre F, Khoshgoftaar TM, Zuech R, Robinson J, Muharemagic E. Using feature selection and classification to build effective and efficient firewalls. In: Proceedings of the 2014 IEEE 15th international conference on information reuse and integration (IEEE IRI 2014). IEEE; 2014, pp. 850\u20134.","DOI":"10.1109\/IRI.2014.7051979"},{"issue":"01","key":"462_CR4","doi-asserted-by":"publisher","first-page":"1650001","DOI":"10.1142\/S0218539316500017","volume":"23","author":"MM Najafabadi","year":"2016","unstructured":"Najafabadi MM, Khoshgoftaar TM, Seliya N. Evaluating feature selection methods for network intrusion detection with kyoto data. Int J Reliabil Qual Saf Eng. 2016;23(01):1650001.","journal-title":"Int J Reliabil Qual Saf Eng"},{"key":"462_CR5","unstructured":"Amit I, Matherly J, Hewlett W, Xu Z, Meshi Y, Weinberger Y. Machine learning in cyber-security-problems, challenges and data sets. arXiv preprint arXiv:1812.07858; 2018."},{"issue":"3","key":"462_CR6","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1109\/MSP.2011.67","volume":"9","author":"R Langner","year":"2011","unstructured":"Langner R. Stuxnet: dissecting a cyberwarfare weapon. IEEE Secur Privacy. 2011;9(3):49\u201351.","journal-title":"IEEE Secur Privacy"},{"key":"462_CR7","doi-asserted-by":"crossref","unstructured":"Bauder RA, Khoshgoftaar TM, Hasanin T. An empirical study on class rarity in big data. In: 2018 17th IEEE international conference on machine learning and applications (ICMLA). IEEE; 2018, pp. 785\u201390.","DOI":"10.1109\/ICMLA.2018.00125"},{"issue":"1","key":"462_CR8","doi-asserted-by":"publisher","first-page":"141","DOI":"10.3233\/IDA-184415","volume":"24","author":"RA Bauder","year":"2020","unstructured":"Bauder RA, Khoshgoftaar TM. A study on rare fraud predictions with big medicare claims fraud data. Intell Data Anal. 2020;24(1):141\u201361.","journal-title":"Intell Data Anal"},{"key":"462_CR9","doi-asserted-by":"crossref","unstructured":"Sharafaldin I, Lashkari AH, Ghorbani AA. Toward generating a new intrusion detection dataset and intrusion traffic characterization. In: ICISSP; 2018, pp. 108\u201316 .","DOI":"10.5220\/0006639801080116"},{"key":"462_CR10","unstructured":"CICIDS2017 Dataset. https:\/\/www.unb.ca\/cic\/datasets\/ids-2017.html. Accessed 28 Aug 2020."},{"key":"462_CR11","unstructured":"CSE-CIC-IDS2018 Dataset. https:\/\/www.unb.ca\/cic\/datasets\/ids-2018.html. Accessed 28 Aug 2020."},{"key":"462_CR12","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s40537-019-0278-0","volume":"7","author":"JL Leevy","year":"2020","unstructured":"Leevy JL, Khoshgoftaar TM. A survey and analysis of intrusion detection models based on cse-cic-ids2018 big data. J Big Data. 2020;7:1\u20139.","journal-title":"J Big Data"},{"issue":"1","key":"462_CR13","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s40537-018-0151-6","volume":"5","author":"JL Leevy","year":"2018","unstructured":"Leevy JL, Khoshgoftaar TM, Bauder RA, Seliya N. A survey on addressing high-class imbalance in big data. J Big Data. 2018;5(1):1\u201330.","journal-title":"J Big Data"},{"key":"462_CR14","first-page":"194","volume":"2","author":"RC Soltysik","year":"2013","unstructured":"Soltysik RC, Yarnold PR. Megaoda large sample and big data time trials: separating the chaff. Optimal Data Anal. 2013;2:194\u20137.","journal-title":"Optimal Data Anal"},{"issue":"2","key":"462_CR15","doi-asserted-by":"publisher","first-page":"423","DOI":"10.2308\/acch-51068","volume":"29","author":"M Cao","year":"2015","unstructured":"Cao M, Chychyla R, Stewart T. Big data analytics in financial statement audits. Account Horizons. 2015;29(2):423\u20139.","journal-title":"Account Horizons"},{"key":"462_CR16","unstructured":"Damn Vulnerable Web App GitHub website. https:\/\/github.com\/digininja\/DVWA. Accessed 30 Jan 2021."},{"key":"462_CR17","unstructured":"Selenium framework website. https:\/\/www.selenium.dev\/. Accessed 30 Jan 2021."},{"key":"462_CR18","doi-asserted-by":"publisher","first-page":"170","DOI":"10.1016\/j.infsof.2014.07.010","volume":"58","author":"I Hydara","year":"2015","unstructured":"Hydara I, Sultan ABM, Zulzalil H, Admodisastro N. Current state of research on cross-site scripting (xss)\u2014a systematic literature review. Inform Softw Technol. 2015;58:170\u201386.","journal-title":"Inform Softw Technol"},{"key":"462_CR19","unstructured":"Halfond WG, Viegas J, Orso A. et al. A classification of sql-injection attacks and countermeasures. In: Proceedings of the IEEE international symposium on secure software engineering. IEEE; 2006, vol. 1, pp. 13\u20135."},{"issue":"4","key":"462_CR20","first-page":"1","volume":"9","author":"RB Basnet","year":"2019","unstructured":"Basnet RB, Shash R, Johnson C, Walgren L, Doleck T. Towards detecting and classifying network intrusion traffic using deep learning frameworks. J Internet Serv Inf Secur. 2019;9(4):1\u201317.","journal-title":"J Internet Serv Inf Secur"},{"key":"462_CR21","doi-asserted-by":"publisher","first-page":"3571","DOI":"10.1007\/s11227-020-03410-y","volume":"77","author":"R Atefinia","year":"2020","unstructured":"Atefinia R, Ahmadi M. Network intrusion detection using multi-architectural modular deep neural network. J Supercomput. 2020;77:3571\u201393.","journal-title":"J Supercomput"},{"key":"462_CR22","doi-asserted-by":"publisher","first-page":"101851","DOI":"10.1016\/j.cose.2020.101851","volume":"95","author":"X Li","year":"2020","unstructured":"Li X, Chen W, Zhang Q, Wu L. Building auto-encoder intrusion detection system based on random forest feature selection. Comput Secur. 2020;95:101851.","journal-title":"Comput Secur"},{"key":"462_CR23","first-page":"102564","volume":"54","author":"L D\u2019hooge","year":"2020","unstructured":"D\u2019hooge L, Wauters T, Volckaert B, De Turck F. Inter-dataset generalization strength of supervised machine learning methods for intrusion detection. J Inform Secur Appl. 2020;54:102564.","journal-title":"J Inform Secur Appl"},{"key":"462_CR24","doi-asserted-by":"publisher","first-page":"107315","DOI":"10.1016\/j.comnet.2020.107315","volume":"177","author":"H Zhang","year":"2020","unstructured":"Zhang H, Huang L, Wu CQ, Li Z. An effective convolutional neural network based on smote and Gaussian mixture model for intrusion detection in imbalanced dataset. Comput Netw. 2020;177:107315.","journal-title":"Comput Netw"},{"key":"462_CR25","doi-asserted-by":"publisher","first-page":"40","DOI":"10.1214\/09-SS054","volume":"4","author":"S Arlot","year":"2010","unstructured":"Arlot S, Celisse A, et al. A survey of cross-validation procedures for model selection. Stat Surv. 2010;4:40\u201379.","journal-title":"Stat Surv"},{"issue":"1","key":"462_CR26","doi-asserted-by":"publisher","first-page":"49","DOI":"10.1145\/1882471.1882479","volume":"12","author":"G Forman","year":"2010","unstructured":"Forman G, Scholz M. Apples-to-apples in cross-validation studies: pitfalls in classifier performance measurement. Acm Sigkdd Explorations Newsletter. 2010;12(1):49\u201357.","journal-title":"Acm Sigkdd Explorations Newsletter"},{"key":"462_CR27","unstructured":"Kohavi R. et al. A study of cross-validation and bootstrap for accuracy estimation and model selection. In: Ijcai, 1995; 14, 1137\u201345 . Montreal, Canada."},{"key":"462_CR28","unstructured":"Scikit-learn website. https:\/\/scikit-learn.org\/stable\/. Accessed 30 Jan 2021."},{"issue":"6","key":"462_CR29","doi-asserted-by":"publisher","first-page":"275","DOI":"10.1002\/cem.873","volume":"18","author":"AJ Myles","year":"2004","unstructured":"Myles AJ, Feudale RN, Liu Y, Woody NA, Brown SD. An introduction to decision tree modeling. J Chemometr J Chemometr Soc. 2004;18(6):275\u201385.","journal-title":"J Chemometr J Chemometr Soc"},{"issue":"1","key":"462_CR30","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1023\/B:AMAI.0000018580.96245.c6","volume":"41","author":"LE Raileanu","year":"2004","unstructured":"Raileanu LE, Stoffel K. Theoretical comparison between the gini index and information gain criteria. Ann Math Artif Intell. 2004;41(1):77\u201393.","journal-title":"Ann Math Artif Intell"},{"issue":"1","key":"462_CR31","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1023\/A:1010933404324","volume":"45","author":"L Breiman","year":"2001","unstructured":"Breiman L. Random forests. Mach Learn. 2001;45(1):5\u201332.","journal-title":"Mach Learn"},{"issue":"2","key":"462_CR32","first-page":"123","volume":"24","author":"L Breiman","year":"1996","unstructured":"Breiman L. Bagging predictors. Mach Learn. 1996;24(2):123\u201340.","journal-title":"Mach Learn"},{"key":"462_CR33","unstructured":"CatBoost home page. https:\/\/catboost.ai\/. Accessed 28 Aug 2020."},{"key":"462_CR34","unstructured":"Prokhorenkova L, Gusev G, Vorobev A, Dorogush AV, Gulin A. Catboost: unbiased boosting with categorical features. In: Advances in neural information processing systems; 2018, pp. 6638\u201348."},{"key":"462_CR35","unstructured":"LightGBM GitHub website. https:\/\/github.com\/microsoft\/LightGBM. Accessed 28 Aug 2020."},{"key":"462_CR36","doi-asserted-by":"publisher","first-page":"21","DOI":"10.3389\/fnbot.2013.00021","volume":"7","author":"A Natekin","year":"2013","unstructured":"Natekin A, Knoll A. Gradient boosting machines, a tutorial. Front Neurorob. 2013;7:21.","journal-title":"Front Neurorob"},{"key":"462_CR37","unstructured":"Ke G, Meng Q, Finley T, Wang T, Chen W, Ma W, Ye Q, Liu T-Y. Lightgbm: a highly efficient gradient boosting decision tree. In: Advances in neural information processing systems; 2017, pp. 3146\u201354."},{"key":"462_CR38","doi-asserted-by":"crossref","unstructured":"Chen T, Guestrin C. Xgboost: A scalable tree boosting system. In: Proceedings of the 22nd Acm Sigkdd international conference on knowledge discovery and data mining; 2016, pp. 785\u201394.","DOI":"10.1145\/2939672.2939785"},{"key":"462_CR39","unstructured":"Guo C, Berkhahn F. Entity embeddings of categorical variables. arXiv preprint arXiv:1604.06737; 2016."},{"key":"462_CR40","unstructured":"Naive Bayes scikit-learn documentation. https:\/\/scikit-learn.org\/stable\/modules\/naive_bayes.html. Accessed 28 Aug 2020."},{"key":"462_CR41","volume-title":"Bayes theory","author":"JA Hartigan","year":"2012","unstructured":"Hartigan JA. Bayes theory. Berlin\/Heidelberg: Springer; 2012."},{"key":"462_CR42","unstructured":"sklearn.linear\\_model.LogisticRegression scikit-learn documentation. https:\/\/scikit-learn.org\/stable\/modules\/generated\/sklearn.linear_model.LogisticRegression.html. Accessed 28 Aug 2020."},{"key":"462_CR43","volume-title":"Introduction to linear regression analysis","author":"DC Montgomery","year":"2012","unstructured":"Montgomery DC, Peck EA, Vining GG. Introduction to linear regression analysis, vol. 821. Hoboken: Wiley; 2012."},{"issue":"1","key":"462_CR44","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1002\/isaf.1460","volume":"27","author":"S Lahmiri","year":"2020","unstructured":"Lahmiri S, Bekiros S, Giakoumelou A, Bezzina F. Performance assessment of ensemble learning systems in financial data classification. Intell Syst Account Fin Manage. 2020;27(1):3\u20139.","journal-title":"Intell Syst Account Fin Manage"},{"key":"462_CR45","unstructured":"Kaggle competitions website. https:\/\/www.kaggle.com\/competitions. Accessed 30 Jan 2021."},{"issue":"7","key":"462_CR46","doi-asserted-by":"publisher","first-page":"1145","DOI":"10.1016\/S0031-3203(96)00142-2","volume":"30","author":"AP Bradley","year":"1997","unstructured":"Bradley AP. The use of the area under the roc curve in the evaluation of machine learning algorithms. Pattern Recogn. 1997;30(7):1145\u201359.","journal-title":"Pattern Recogn"},{"issue":"6","key":"462_CR47","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/cc3000","volume":"8","author":"V Bewick","year":"2004","unstructured":"Bewick V, Cheek L, Ball J. Statistics review 13: receiver operating characteristic curves. Crit Care. 2004;8(6):1\u20135.","journal-title":"Crit Care"},{"issue":"7","key":"462_CR48","doi-asserted-by":"publisher","first-page":"928","DOI":"10.1161\/CIRCULATIONAHA.106.672402","volume":"115","author":"NR Cook","year":"2007","unstructured":"Cook NR. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation. 2007;115(7):928\u201335.","journal-title":"Circulation"},{"issue":"1","key":"462_CR49","doi-asserted-by":"publisher","first-page":"107","DOI":"10.1186\/s40537-019-0274-4","volume":"6","author":"T Hasanin","year":"2019","unstructured":"Hasanin T, Khoshgoftaar TM, Leevy JL, Bauder RA. Severely imbalanced big data challenges: investigating data sampling approaches. J Big Data. 2019;6(1):107.","journal-title":"J Big Data"},{"issue":"1","key":"462_CR50","doi-asserted-by":"publisher","first-page":"9","DOI":"10.1007\/s13755-018-0051-3","volume":"6","author":"RA Bauder","year":"2018","unstructured":"Bauder RA, Khoshgoftaar TM. The effects of varying class distribution on learner behavior for medicare fraud detection with imbalanced big data. Health Inform Sci Syst. 2018;6(1):9.","journal-title":"Health Inform Sci Syst"},{"issue":"1","key":"462_CR51","doi-asserted-by":"publisher","first-page":"67","DOI":"10.1186\/s40537-019-0230-3","volume":"6","author":"CL Calvert","year":"2019","unstructured":"Calvert CL, Khoshgoftaar TM. Impact of class distribution on the detection of slow http dos attacks using big data. J Big Data. 2019;6(1):67.","journal-title":"J Big Data"},{"key":"462_CR52","doi-asserted-by":"crossref","unstructured":"Hasanin T, Khoshgoftaar TM, Bauder RA. Impact of data sampling with severely imbalanced big data. In: Reuse in intelligent systems. 2020, p. 1.","DOI":"10.1201\/9781003034971-1"},{"issue":"1","key":"462_CR53","doi-asserted-by":"publisher","first-page":"21","DOI":"10.1186\/s40537-019-0181-8","volume":"6","author":"M Herland","year":"2019","unstructured":"Herland M, Bauder RA, Khoshgoftaar TM. The effects of class rarity on the evaluation of supervised healthcare fraud detection models. J Big Data. 2019;6(1):21.","journal-title":"J Big Data"},{"key":"462_CR54","volume-title":"Experimental designs using ANOVA","author":"BG Tabachnick","year":"2007","unstructured":"Tabachnick BG, Fidell LS. Experimental designs using ANOVA. Belmont: Thomson\/Brooks\/Cole; 2007."},{"key":"462_CR55","doi-asserted-by":"publisher","first-page":"99","DOI":"10.2307\/3001913","volume":"5","author":"JW Tukey","year":"1949","unstructured":"Tukey JW. Comparing individual means in the analysis of variance. Biometrics. 1949;5:99\u2013114.","journal-title":"Biometrics"},{"issue":"1","key":"462_CR56","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s40537-020-00301-0","volume":"7","author":"T Hasanin","year":"2020","unstructured":"Hasanin T, Khoshgoftaar TM, Leevy JL, Bauder RA. Investigating class rarity in big data. J Big Data. 2020;7(1):1\u201317.","journal-title":"J Big Data"}],"container-title":["Journal of Big Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s40537-021-00462-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s40537-021-00462-6\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s40537-021-00462-6.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,5,20]],"date-time":"2021-05-20T18:06:54Z","timestamp":1621534014000},"score":1,"resource":{"primary":{"URL":"https:\/\/journalofbigdata.springeropen.com\/articles\/10.1186\/s40537-021-00462-6"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,5,20]]},"references-count":56,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2021,12]]}},"alternative-id":["462"],"URL":"https:\/\/doi.org\/10.1186\/s40537-021-00462-6","relation":{},"ISSN":["2196-1115"],"issn-type":[{"type":"electronic","value":"2196-1115"}],"subject":[],"published":{"date-parts":[[2021,5,20]]},"assertion":[{"value":"13 March 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"3 May 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"20 May 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare that they have no competing interests.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"71"}}