{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,8]],"date-time":"2026-05-08T22:14:13Z","timestamp":1778278453674,"version":"3.51.4"},"reference-count":34,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2021,4,21]],"date-time":"2021-04-21T00:00:00Z","timestamp":1618963200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["61672392, 62072342 and 71790612"],"award-info":[{"award-number":["61672392, 62072342 and 71790612"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Knowl. Discov. Data"],"published-print":{"date-parts":[[2021,6,30]]},"abstract":"<jats:p>Logs that record system abnormal states (anomaly logs) can be regarded as outliers, and the k-Nearest Neighbor (kNN) algorithm has relatively high accuracy in outlier detection methods. Therefore, we use the kNN algorithm to detect anomalies in the log data. However, there are some problems when using the kNN algorithm to detect anomalies, three of which are: excessive vector dimension leads to inefficient kNN algorithm, unlabeled log data cannot support the kNN algorithm, and the imbalance of the number of log data distorts the classification decision of kNN algorithm. In order to solve these three problems, we propose an efficient log anomaly detection method based on an improved kNN algorithm with an automatically labeled sample set. This method first proposes a log parsing method based on N-gram and frequent pattern mining (FPM) method, which reduces the dimension of the log vector converted with Term frequency.Inverse Document Frequency (TF-IDF) technology. Then we use clustering and self-training method to get labeled log data sample set from historical logs automatically. Finally, we improve the kNN algorithm using average weighting technology, which improves the accuracy of the kNN algorithm on unbalanced samples. The method in this article is validated on six log datasets with different types.<\/jats:p>","DOI":"10.1145\/3441448","type":"journal-article","created":{"date-parts":[[2021,4,21]],"date-time":"2021-04-21T15:42:54Z","timestamp":1619019774000},"page":"1-22","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":77,"title":["An Improved KNN-Based Efficient Log Anomaly Detection Method with Automatically Labeled Samples"],"prefix":"10.1145","volume":"15","author":[{"given":"Shi","family":"Ying","sequence":"first","affiliation":[{"name":"Wuhan University, HuBei, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Bingming","family":"Wang","sequence":"additional","affiliation":[{"name":"Wuhan University, HuBei, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Lu","family":"Wang","sequence":"additional","affiliation":[{"name":"Xidian University, XiAn, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Qingshan","family":"Li","sequence":"additional","affiliation":[{"name":"Xidian University, XiAn, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yishi","family":"Zhao","sequence":"additional","affiliation":[{"name":"China University of Geosciences, Wuhan, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jianga","family":"Shang","sequence":"additional","affiliation":[{"name":"China University of Geosciences, Wuhan, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hao","family":"Huang","sequence":"additional","affiliation":[{"name":"Wuhan University, HuBei, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Guoli","family":"Cheng","sequence":"additional","affiliation":[{"name":"Wuhan University, HuBei, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zhe","family":"Yang","sequence":"additional","affiliation":[{"name":"Wuhan University, HuBei, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jiangyi","family":"Geng","sequence":"additional","affiliation":[{"name":"Wuhan University, HuBei, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2021,4,21]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/2889160.2889231"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.2017.43"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDCS.2018.00105"},{"key":"e_1_2_1_4_1","volume-title":"Proceedings of the 2nd International Conference on Knowledge Discovery and Data Mining (KDD\u201996)","author":"Ester Martin","year":"1996","unstructured":"Martin Ester , Hans-Peter Kriegel , J\u00f6rg Sander , and Xiaowei Xu . 1996 . A density-based algorithm for discovering clusters in large spatial databases with noise . In Proceedings of the 2nd International Conference on Knowledge Discovery and Data Mining (KDD\u201996) . Evangelos Simoudis, Jiawei Han, and Usama M. Fayyad (Eds.). AAAI Press, 226\u2013231. Retrieved from http:\/\/www.aaai.org\/Library\/KDD\/ 1996\/kdd96-037.php. Martin Ester, Hans-Peter Kriegel, J\u00f6rg Sander, and Xiaowei Xu. 1996. A density-based algorithm for discovering clusters in large spatial databases with noise. In Proceedings of the 2nd International Conference on Knowledge Discovery and Data Mining (KDD\u201996). Evangelos Simoudis, Jiawei Han, and Usama M. Fayyad (Eds.). AAAI Press, 226\u2013231. Retrieved from http:\/\/www.aaai.org\/Library\/KDD\/1996\/kdd96-037.php."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2009.60"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.datak.2009.04.004"},{"key":"e_1_2_1_7_1","volume-title":"Proceedings of the 27th IEEE International Symposium on Software Reliability Engineering (ISSRE\u201916)","author":"He Shilin","year":"2016","unstructured":"Shilin He , Jieming Zhu , Pinjia He , and Michael R. Lyu . 2016. Experience report: System log analysis for anomaly detection . In Proceedings of the 27th IEEE International Symposium on Software Reliability Engineering (ISSRE\u201916) . 207\u2013218. DOI:https:\/\/doi.org\/10.1109\/ISSRE. 2016 .21 10.1109\/ISSRE.2016.21 Shilin He, Jieming Zhu, Pinjia He, and Michael R. Lyu. 2016. Experience report: System log analysis for anomaly detection. In Proceedings of the 27th IEEE International Symposium on Software Reliability Engineering (ISSRE\u201916). 207\u2013218. DOI:https:\/\/doi.org\/10.1109\/ISSRE.2016.21"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1147\/sj.413.0475"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICAC.2005.42"},{"key":"e_1_2_1_10_1","volume-title":"Proceedings of the 2006 International Conference on Dependable Systems and Networks (DSN\u201906)","author":"Liang Yinglung","year":"2006","unstructured":"Yinglung Liang , Yanyong Zhang , Anand Sivasubramaniam , Morris Jette , and Ramendra K. Sahoo . 2006. BlueGene\/L failure analysis and prediction models . In Proceedings of the 2006 International Conference on Dependable Systems and Networks (DSN\u201906) . 425\u2013434. DOI:https:\/\/doi.org\/10.1109\/DSN. 2006 .18 10.1109\/DSN.2006.18 Yinglung Liang, Yanyong Zhang, Anand Sivasubramaniam, Morris Jette, and Ramendra K. Sahoo. 2006. BlueGene\/L failure analysis and prediction models. In Proceedings of the 2006 International Conference on Dependable Systems and Networks (DSN\u201906). 425\u2013434. DOI:https:\/\/doi.org\/10.1109\/DSN.2006.18"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2889160.2889232"},{"key":"e_1_2_1_12_1","volume-title":"Proceedings of the 29th International Joint Conference on Artificial Intelligence (IJCAI\u201920)","author":"Liu Fanzhen","year":"2020","unstructured":"Fanzhen Liu , Shan Xue , Jia Wu , Chuan Zhou , Wenbin Hu , C\u00e9cile Paris , Surya Nepal , Jian Yang , and Philip S. Yu . 2020. Deep learning for community detection: Progress, challenges and opportunities . In Proceedings of the 29th International Joint Conference on Artificial Intelligence (IJCAI\u201920) . Christian Bessiere (Ed.). 4981\u20134987. DOI:https:\/\/doi.org\/10.24963\/ijcai. 2020 \/693 10.24963\/ijcai.2020 Fanzhen Liu, Shan Xue, Jia Wu, Chuan Zhou, Wenbin Hu, C\u00e9cile Paris, Surya Nepal, Jian Yang, and Philip S. Yu. 2020. Deep learning for community detection: Progress, challenges and opportunities. In Proceedings of the 29th International Joint Conference on Artificial Intelligence (IJCAI\u201920). Christian Bessiere (Ed.). 4981\u20134987. DOI:https:\/\/doi.org\/10.24963\/ijcai.2020\/693"},{"key":"e_1_2_1_13_1","volume-title":"Kai Ming Ting, and Zhi-Hua Zhou","author":"Liu Fei Tony","year":"2012","unstructured":"Fei Tony Liu , Kai Ming Ting, and Zhi-Hua Zhou . 2012 . Isolation-based anomaly detection. ACM Transactions on Knowledge Discovery from Data 6, 1 (2012), 3:1\u20133:39. DOI:https:\/\/doi.org\/10.1145\/2133360.2133363 10.1145\/2133360.2133363 Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. 2012. Isolation-based anomaly detection. ACM Transactions on Knowledge Discovery from Data 6, 1 (2012), 3:1\u20133:39. DOI:https:\/\/doi.org\/10.1145\/2133360.2133363"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/1557019.1557083"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/1740390.1740411"},{"key":"e_1_2_1_16_1","volume-title":"Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 1255\u20131264","author":"Makanju Adetokunbo","unstructured":"Adetokunbo Makanju , A. Nur Zincir-Heywood , and Evangelos E. Milios . 2009. Clustering event logs using iterative partitioning . In Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 1255\u20131264 . DOI:https:\/\/doi.org\/10.1145\/1557019.1557154 10.1145\/1557019.1557154 Adetokunbo Makanju, A. Nur Zincir-Heywood, and Evangelos E. Milios. 2009. Clustering event logs using iterative partitioning. In Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 1255\u20131264. DOI:https:\/\/doi.org\/10.1145\/1557019.1557154"},{"key":"e_1_2_1_17_1","volume-title":"Proceedings of the 7th International Working Conference on Mining Software Repositories. 114\u2013117","author":"Nagappan Meiyappan","year":"2010","unstructured":"Meiyappan Nagappan and Mladen A. Vouk . 2010. Abstracting log lines to log event types for mining software system logs . In Proceedings of the 7th International Working Conference on Mining Software Repositories. 114\u2013117 . DOI:https:\/\/doi.org\/10.1109\/MSR. 2010 .5463281 10.1109\/MSR.2010.5463281 Meiyappan Nagappan and Mladen A. Vouk. 2010. Abstracting log lines to log event types for mining software system logs. In Proceedings of the 7th International Working Conference on Mining Software Repositories. 114\u2013117. DOI:https:\/\/doi.org\/10.1109\/MSR.2010.5463281"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/DSNW.2011.5958800"},{"key":"e_1_2_1_19_1","volume-title":"Proceedings of the 12th IFIP\/IEEE International Symposium on Integrated Network Management. 377\u2013384","author":"Reidemeister Thomas","year":"2011","unstructured":"Thomas Reidemeister , Miao Jiang , and Paul A. S. Ward . 2011. Mining unstructured log files for recurrent fault diagnosis . In Proceedings of the 12th IFIP\/IEEE International Symposium on Integrated Network Management. 377\u2013384 . DOI:https:\/\/doi.org\/10.1109\/INM. 2011 .5990536 10.1109\/INM.2011.5990536 Thomas Reidemeister, Miao Jiang, and Paul A. S. Ward. 2011. Mining unstructured log files for recurrent fault diagnosis. In Proceedings of the 12th IFIP\/IEEE International Symposium on Integrated Network Management. 377\u2013384. DOI:https:\/\/doi.org\/10.1109\/INM.2011.5990536"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.5555\/1111682.1111732"},{"key":"e_1_2_1_21_1","volume-title":"Proceedings of the 8th IEEE International Symposium on Cluster Computing and the Grid (CCGrid\u201908)","author":"Stearley Jon","year":"2008","unstructured":"Jon Stearley and Adam J. Oliner . 2008. Bad words: Finding faults in spirit\u2019s syslogs . In Proceedings of the 8th IEEE International Symposium on Cluster Computing and the Grid (CCGrid\u201908) . 765\u2013770. DOI:https:\/\/doi.org\/10.1109\/CCGRID. 2008 .107 10.1109\/CCGRID.2008.107 Jon Stearley and Adam J. Oliner. 2008. Bad words: Finding faults in spirit\u2019s syslogs. In Proceedings of the 8th IEEE International Symposium on Cluster Computing and the Grid (CCGrid\u201908). 765\u2013770. DOI:https:\/\/doi.org\/10.1109\/CCGRID.2008.107"},{"key":"e_1_2_1_22_1","volume-title":"Proceedings of the 1st USENIX Workshop on the Analysis of System Logs (WASL\u201908)","author":"Tan Jiaqi","year":"2008","unstructured":"Jiaqi Tan , Xinghao Pan , Soila Kavulya , Rajeev Gandhi , and Priya Narasimhan . 2008 . SALSA: Analyzing logs as state machines . In Proceedings of the 1st USENIX Workshop on the Analysis of System Logs (WASL\u201908) . Retrieved from http:\/\/www.usenix.org\/events\/wasl\/tech\/full_papers\/tan\/tan.pdf. Jiaqi Tan, Xinghao Pan, Soila Kavulya, Rajeev Gandhi, and Priya Narasimhan. 2008. SALSA: Analyzing logs as state machines. In Proceedings of the 1st USENIX Workshop on the Analysis of System Logs (WASL\u201908). Retrieved from http:\/\/www.usenix.org\/events\/wasl\/tech\/full_papers\/tan\/tan.pdf."},{"key":"e_1_2_1_23_1","volume-title":"Proceedings of the Workshop on Hot Topics in Cloud Computing (HotCloud\u201909)","author":"Tan Jiaqi","year":"2009","unstructured":"Jiaqi Tan , Xinghao Pan , Soila Kavulya , Rajeev Gandhi , and Priya Narasimhan . 2009 . Mochi: Visual log-analysis based tools for debugging hadoop . In Proceedings of the Workshop on Hot Topics in Cloud Computing (HotCloud\u201909) . Retrieved from https:\/\/www.usenix.org\/conference\/hotcloud-09\/mochi-visual-log-analysis-based-tools-debugging-hadoop. Jiaqi Tan, Xinghao Pan, Soila Kavulya, Rajeev Gandhi, and Priya Narasimhan. 2009. Mochi: Visual log-analysis based tools for debugging hadoop. In Proceedings of the Workshop on Hot Topics in Cloud Computing (HotCloud\u201909). Retrieved from https:\/\/www.usenix.org\/conference\/hotcloud-09\/mochi-visual-log-analysis-based-tools-debugging-hadoop."},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2063576.2063690"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/IPOM.2003.1251233"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/NOMS.2008.4575281"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/CNSM.2015.7367331"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCYB.2014.2327246"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2013.2297923"},{"key":"e_1_2_1_30_1","volume-title":"Proceedings of the 22nd ACM Symposium on Operating Systems Principles (SOSP\u201909)","author":"Xu Wei","unstructured":"Wei Xu , Ling Huang , Armando Fox , David A. Patterson , and Michael I. Jordan . 2009. Detecting large-scale system problems by mining console logs . In Proceedings of the 22nd ACM Symposium on Operating Systems Principles (SOSP\u201909) . 117\u2013132. DOI:https:\/\/doi.org\/10.1145\/1629575.1629587 10.1145\/1629575.1629587 Wei Xu, Ling Huang, Armando Fox, David A. Patterson, and Michael I. Jordan. 2009. Detecting large-scale system problems by mining console logs. In Proceedings of the 22nd ACM Symposium on Operating Systems Principles (SOSP\u201909). 117\u2013132. DOI:https:\/\/doi.org\/10.1145\/1629575.1629587"},{"key":"e_1_2_1_31_1","volume-title":"Proceedings of the 9th IEEE International Conference on Data Mining. 588\u2013597","author":"Xu Wei","year":"2009","unstructured":"Wei Xu , Ling Huang , Armando Fox , David A. Patterson , and Michael I. Jordan . 2009. Online system problem detection by mining patterns of console logs . In Proceedings of the 9th IEEE International Conference on Data Mining. 588\u2013597 . DOI:https:\/\/doi.org\/10.1109\/ICDM. 2009 .19 10.1109\/ICDM.2009.19 Wei Xu, Ling Huang, Armando Fox, David A. Patterson, and Michael I. Jordan. 2009. Online system problem detection by mining patterns of console logs. In Proceedings of the 9th IEEE International Conference on Data Mining. 588\u2013597. DOI:https:\/\/doi.org\/10.1109\/ICDM.2009.19"},{"key":"e_1_2_1_32_1","volume-title":"Proceedings of the Workshop on Managing Systems via Log Analysis and Machine Learning Techniques (SLAML\u201910)","author":"Xu Wei","unstructured":"Wei Xu , Ling Huang , and Michael I. Jordan . 2010. Experience mining Google\u2019s production console logs . In Proceedings of the Workshop on Managing Systems via Log Analysis and Machine Learning Techniques (SLAML\u201910) . Retrieved from https:\/\/www.usenix.org\/conference\/slaml10\/experience-mining-googles-production-console-logs. Wei Xu, Ling Huang, and Michael I. Jordan. 2010. Experience mining Google\u2019s production console logs. In Proceedings of the Workshop on Managing Systems via Log Analysis and Machine Learning Techniques (SLAML\u201910). Retrieved from https:\/\/www.usenix.org\/conference\/slaml10\/experience-mining-googles-production-console-logs."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2484313.2484342"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPS.2008.4536397"}],"container-title":["ACM Transactions on Knowledge Discovery from Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3441448","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3441448","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:03:05Z","timestamp":1750197785000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3441448"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,4,21]]},"references-count":34,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2021,6,30]]}},"alternative-id":["10.1145\/3441448"],"URL":"https:\/\/doi.org\/10.1145\/3441448","relation":{},"ISSN":["1556-4681","1556-472X"],"issn-type":[{"value":"1556-4681","type":"print"},{"value":"1556-472X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,4,21]]},"assertion":[{"value":"2019-11-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-12-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-04-21","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}