{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T04:38:05Z","timestamp":1787027885013,"version":"build-2736575974"},"reference-count":39,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2021,6,23]],"date-time":"2021-06-23T00:00:00Z","timestamp":1624406400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"The project has been supported by the European Union, co-financed by the European Social Fund.","award":["EFOP-3.6.3-VEKOP-16-2017-00002"],"award-info":[{"award-number":["EFOP-3.6.3-VEKOP-16-2017-00002"]}]},{"name":"Grants of &quot;Application Domain Specific Highly Reliable IT Solutions&quot; project that has been implemented with the support provided from the National Research, Development and Innovation Fund of Hungary, financed under the Thematic Excellence Programme","award":["TKP2020-NKA-06 (National Challenges Subprogramme)"],"award-info":[{"award-number":["TKP2020-NKA-06 (National Challenges Subprogramme)"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computers"],"abstract":"<jats:p>Presently, almost every computer software produces many log messages based on events and activities during the usage of the software. These files contain valuable runtime information that can be used in a variety of applications such as anomaly detection, error prediction, template mining, and so on. Usually, the generated log messages are raw, which means they have an unstructured format. This indicates that these messages have to be parsed before data mining models can be applied. After parsing, template miners can be applied on the data to retrieve the events occurring in the log file. These events are made from two parts, the template, which is the fixed part and is the same for all instances of the same event type, and the parameter part, which varies for all the instances. To decrease the size of the log messages, we use the mined templates to build a dictionary for the events, and only store the dictionary, the event ID, and the parameter list. We use six template miners to acquire the templates namely IPLoM, LenMa, LogMine, Spell, Drain, and MoLFI. In this paper, we evaluate the compression capacity of our dictionary method with the use of these algorithms. Since parameters could be sensitive information, we also encrypt the files after compression and measure the changes in file size. We also examine the speed of the log miner algorithms. Based on our experiments, LenMa has the best compression rate with an average of 67.4%; however, because of its high runtime, we would suggest the combination of our dictionary method with IPLoM and FFX, since it is the fastest of all methods, and it has a 57.7% compression rate.<\/jats:p>","DOI":"10.3390\/computers10070083","type":"journal-article","created":{"date-parts":[[2021,6,23]],"date-time":"2021-06-23T11:28:41Z","timestamp":1624447721000},"page":"83","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["The Use of Template Miners and Encryption in Log Message Compression"],"prefix":"10.3390","volume":"10","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1462-5629","authenticated-orcid":false,"given":"P\u00e9ter","family":"Marjai","sequence":"first","affiliation":[{"name":"Department of Information Systems, ELTE E\u00f6tv\u00f6s Lor\u00e1nd University, 1117 Budapest, Hungary"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0884-4297","authenticated-orcid":false,"given":"P\u00e9ter","family":"Lehotay-K\u00e9ry","sequence":"additional","affiliation":[{"name":"Department of Information Systems, ELTE E\u00f6tv\u00f6s Lor\u00e1nd University, 1117 Budapest, Hungary"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8174-6194","authenticated-orcid":false,"given":"Attila","family":"Kiss","sequence":"additional","affiliation":[{"name":"Department of Information Systems, ELTE E\u00f6tv\u00f6s Lor\u00e1nd University, 1117 Budapest, Hungary"},{"name":"Department of Informatics, J. Selye University, 94501 Kom\u00e1rno, Slovakia"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2021,6,23]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"449","DOI":"10.1007\/978-3-662-46578-3_53","article-title":"Anomaly Detection from Log Files Using Data Mining Techniques","volume":"Volume 339","author":"Breier","year":"2015","journal-title":"Lecture Notes in Electrical Engineering"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Zimek, A., and Schubert, E. (2017). Outlier Detection. Encyclopedia of Database Systems, Springer.","DOI":"10.1007\/978-1-4899-7993-3_80719-1"},{"key":"ref_3","unstructured":"Nagaraj, K., Killian, C., and Neville, J. tructured Comparative Analysis of Systems Logs to Diagnose Performance Problems. Proceedings of the 9th USENIX Symposium on Networked Systems Design and Implementation (NSDI 12)."},{"key":"ref_4","unstructured":"Sun, Y., Li, H., Councill, I.G., Huang, J., Lee, W.C., and Giles, C.L. Personalized Ranking for Digital Libraries Based on Log Analysis. Proceedings of the 10th ACM Workshop on Web Information and Data Management."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Reidemeister, T., Munawar, M.A., Jiang, M., and Ward, P.A.S. Diagnosis of Recurrent Faults Using Log Files. Proceedings of the 2009 Conference of the Center for Advanced Studies on Collaborative Research.","DOI":"10.1145\/1723028.1723031"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Basin, D., Caronni, G., Ereth, S., Harvan, M., Klaedtke, F., and Mantel, H. (2014). Scalable Offline Monitoring. Lecture Notes in Computer Science, Springer International Publishing.","DOI":"10.1007\/978-3-319-11164-3_4"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"189","DOI":"10.1016\/j.dss.2004.11.012","article-title":"An e-customer behavior model with online analytical mining for internet marketing planning","volume":"41","author":"Kwan","year":"2005","journal-title":"Decis. Support Syst."},{"key":"ref_8","unstructured":"He, S., He, P., Chen, Z., Yang, T., Su, Y., and Lyu, M.R. (2021, June 06). A Survey on Automated Log Analysis for Reliability Engineering. Available online: https:\/\/arxiv.org\/abs\/2009.07237."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"345","DOI":"10.1109\/TKDE.2007.44","article-title":"K-Means+ ID3: A novel method for supervised anomaly detection by cascading K-Means clustering and ID3 decision tree learning methods","volume":"19","author":"Gaddam","year":"2007","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1991","DOI":"10.1109\/ACCESS.2018.2886457","article-title":"DeepAnT: A deep learning approach for unsupervised anomaly detection in time series","volume":"7","author":"Munir","year":"2018","journal-title":"IEEE Access"},{"key":"ref_11","unstructured":"Akcay, S., Atapour-Abarghouei, A., and Breckon, T.P. (2018). GANomaly: Semi-supervised Anomaly Detection via Adversarial Training. Asian Conference on Computer Vision, Springer."},{"key":"ref_12","unstructured":"He, S., Zhu, J., He, P., and Lyu, M.R. (2020). Loghub: A large collection of system log datasets towards automated log analytics. arXiv."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"3043","DOI":"10.1007\/s10664-020-09822-x","article-title":"A study of the performance of general compressors on log files","volume":"25","author":"Yao","year":"2020","journal-title":"Empir. Softw. Eng."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Gazdag, A., Buttyan, L., and Szalay, Z. (2017, January 21\u201323). Efficient lossless compression of CAN traffic logs. Proceedings of the 2017 25th International Conference on Software, Telecommunications and Computer Networks (SoftCOM), Split, Croatia.","DOI":"10.23919\/SOFTCOM.2017.8115527"},{"key":"ref_15","unstructured":"Hassan, A.E., Martin, D.J., Flora, P., Mansfield, P., and Dietz, D. (2008, January 10\u201318). An industrial case study of customizing operational profiles using log compression. Proceedings of the 30th International Conference on Software Engineering, New York, NY, USA."},{"key":"ref_16","unstructured":"Fei, P., Li, Z., Wang, Z., Yu, X., Li, D., and Jee, K. (2021). SEAL: Storage-efficient Causality Analysis on Enterprise Logs with Query-friendly Compression. 30th USENIX Security Symposium (USENIX Security 21), USENIX Association."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Liu, J., Zhu, J., He, S., He, P., Zheng, Z., and Lyu, M.R. (2019, January 11). Logzip: Extracting Hidden Structures via Iterative Clustering for LogCompression. Proceedings of the 34th IEEE\/ACM International Conference on Automated Software Engineering (ASE), San Diego, CA, USA.","DOI":"10.1109\/ASE.2019.00085"},{"key":"ref_18","unstructured":"Yao, K., Sayagh, M., Shang, W., and Hassan, A.E. (2021). Improving State-of-the-art Compression Techniques for Log Management Tools. IEEE Trans. Softw. Eng., 1."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Ryabko, B. (2019, January 7\u201312). Time-universal data compression and prediction. Proceedings of the 2019 IEEE International Symposium on Information Theory, Paris, France.","DOI":"10.1109\/ISIT.2019.8849224"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Vestergaard, R., Zhang, Q., and Lucani, D.E. (2019, January 9\u201313). Lossless compression of time series data with generalized deduplication. Proceedings of the IEEE Global Communications Conference (GLOBECOM), Waikoloa, HI, USA.","DOI":"10.1109\/GLOBECOM38437.2019.9013957"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Doblander, C., Khatayee, A., and Jacobsen, H.A. (2018, January 10\u201314). Predict: Predictive dictionary maintenance for message compression in publish\/subscribe. Proceedings of the 19th International Middleware Conference, Rennes, France.","DOI":"10.1145\/3274808.3274822"},{"key":"ref_22","unstructured":"Bateni, M., Chen, L., Esfandiari, H., Fu, T., Mirrokni, V., and Rostamizadeh, A. (2019, January 9\u201315). Categorical feature compression via submodular optimization. Proceedings of the International Conference on Machine Learning, Long Beach, CA, USA."},{"key":"ref_23","unstructured":"Hoogeboom, E., Peters, W., Berg, R.V., and Welling, M. (2019). Integer discrete flows and lossless compression. arXiv."},{"key":"ref_24","unstructured":"Wei, J., Zhang, G., Wang, Y., Liu, Z., Zhu, Z., Chen, J., Sun, T., and Zhou, Q. (2021, January 23\u201325). On the Feasibility of Parser-based Log Compression in Large-Scale Cloud Systems. Proceedings of the 19th USENIX Conference on File and Storage Technologies (FAST 21), Berkeley, CA, USA."},{"key":"ref_25","unstructured":"Ji, C., Chang, L.P., Pan, R., Wu, C., Gao, C., Shi, L., Kuo, T.W., and Xue, C.J. (2021, January 23\u201325). Pattern-Guided file compression with user-Experience enhancement for log-Structured file system on mobile devices. Proceedings of the 19th USENIX Conference on File and Storage Technologies (FAST 21), Berkeley, CA, USA."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"He, P., Zhu, J., Zheng, Z., and Lyu, M.R. (2017, January 25\u201330). Drain: An online log parsing approach with fixed depth tree. Proceedings of the IEEE International Conference on Web Services (ICWS), Honolulu, HI, USA.","DOI":"10.1109\/ICWS.2017.13"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Li, L., Man, Y., and Chen, M. (2018). A Method of Large\u2014Scale Log Pattern Mining. Lecture Notes in Computer Science, Springer International Publishing.","DOI":"10.1007\/978-3-319-74521-3_9"},{"key":"ref_28","unstructured":"Shima, K. (2016). Length matters: Clustering system log messages using length of words. arXiv."},{"key":"ref_29","unstructured":"Daemen, J., and Rijmen, V. (2021, June 06). AES Proposal: Rijndael; Gaithersburg, MD, USA. Available online: https:\/\/www.cs.miami.edu\/home\/burt\/learning\/Csc688.012\/rijndael\/rijndael_doc_V2.pdf."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Schneier, B. (1994). Description of a new variable-length key, 64-bit block cipher (Blowfish). Fast Software Encryption, Springer.","DOI":"10.1007\/3-540-58108-1_24"},{"key":"ref_31","unstructured":"He, P., Zhu, J., He, S., Li, J., and Lyu, M.R. An Evaluation Study on Log Parsing and Its Use in Log Mining. Proceedings of the 46th Annual IEEE\/IFIP International Conference on Dependable Systems and Networks (DSN)."},{"key":"ref_32","first-page":"1921","article-title":"A Lightweight Algorithm for Message Type Extraction in System Application Logs","volume":"24","author":"Makanju","year":"2012","journal-title":"IEEE Trans. Softw. Eng."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Du, M., and Li, F. (2016, January 12\u201315). Spell: Streaming parsing of system event logs. Proceedings of the IEEE 16th International Conference on Data Mining (ICDM), Barcelona, Spain.","DOI":"10.1109\/ICDM.2016.0103"},{"key":"ref_34","unstructured":"Messaoudi, S., Panichella, A., Bianculli, D., Briand, L., and Sasnauskas, R. (June, January 27). A search-based approach for accurate identification of log message formats. Proceedings of the IEEE\/ACM 26th International Conference on Program Comprehension (ICPC), Gothenburg, Sweden."},{"key":"ref_35","unstructured":"Sivanandam, S.N., and Deepa, S.N. (2008). Introduction to Genetic Algorithms, Springer. [1st ed.]."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Nyberg, K. (1996). Generalized Feistel networks. ASIACRYPT 1996 -Lecture Notes in Computer Science, Springer.","DOI":"10.1007\/BFb0034838"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Bellare, M., Ristenpart, T., Rogaway, P., and Stegers, T. (2009). Format-preserving encryption. International Workshop on Selected Areas in Cryptography, Springer.","DOI":"10.1007\/978-3-642-05445-7_19"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Liskov, M., Rivest, R.L., and Wagner, D. (2002). Tweakable block ciphers. Annual International Cryptology Conference, Springer.","DOI":"10.1007\/3-540-45708-9_3"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Zhu, J., He, S., Liu, J., He, P., Xie, Q., Zheng, Z., and Lyu, M.R. (2019, January 25\u201331). Tools and benchmarks for automated log parsing. Proceedings of the IEEE\/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), Montreal, QC, Canada.","DOI":"10.1109\/ICSE-SEIP.2019.00021"}],"container-title":["Computers"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2073-431X\/10\/7\/83\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T06:22:06Z","timestamp":1760163726000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2073-431X\/10\/7\/83"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,6,23]]},"references-count":39,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2021,7]]}},"alternative-id":["computers10070083"],"URL":"https:\/\/doi.org\/10.3390\/computers10070083","relation":{},"ISSN":["2073-431X"],"issn-type":[{"value":"2073-431X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,6,23]]}}}