{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,29]],"date-time":"2026-05-29T14:28:44Z","timestamp":1780064924514,"version":"3.54.0"},"reference-count":67,"publisher":"Association for Computing Machinery (ACM)","issue":"ISSTA","funder":[{"name":"2023 Alibaba Innovation Research (AIR) program","award":["CRS54WHZ11230017 24230427"],"award-info":[{"award-number":["CRS54WHZ11230017 24230427"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Softw. Eng."],"published-print":{"date-parts":[[2025,6,22]]},"abstract":"<jats:p>Logs generated by large-scale software systems contain a huge amount of useful information. As the first step of automated log analysis, log parsing has been extensively studied. General log parsing techniques focus on identifying static templates from raw logs, but overlook the more important semantics implied in dynamic log parameters. With the popularity of Artificial Intelligence for IT Operations (AIOps), traditional log parsing methods no longer meet the requirements of various downstream tasks. Researchers are now exploring the next generation of log parsing techniques, i.e., semantic log parsing, to identify both log templates and semantics in log parameters. However, the absence of semantic annotations in existing datasets hinders the training and evaluation of semantic log parsers, thereby stalling the progress of semantic log parsing.<\/jats:p>\n          <jats:p>To fill this gap and advance the field of semantic log parsing, we construct LogBase, the first semantic log parsing benchmark dataset. LogBase consists of logs from 130 popular open-source projects, containing 85,300 semantically annotated log templates, surpassing existing datasets in both log source diversity and template richness.  \nTo build Logbase, we develop the framework GenLog for constructing semantic log parsing datasets. GenLog mines log template-parameter-context triplets from popular open-source repositories on GitHub, and uses chain-of-thought (CoT) techniques with large language models (LLMs) to generate high-quality logs. Meanwhile, GenLog employs human feedback to improve the quality of the generated data and ensure its reliability. GenLog is highly automated and cost-effective, enabling researchers to easily and efficiently construct semantic log parsing datasets.  \nFurthermore, we also design a set of comprehensive evaluation metrics for LogBase, including general log parser metrics and the metrics specifically for semantic log parsers and LLM-based parsers.<\/jats:p>\n          <jats:p>With LogBase, we extensively evaluate 15 existing log parsers, revealing their true performance in complex scenarios. We believe that this work provides researchers with valuable data, reliable tools, and insightful findings to support and guide the future research of semantic log parsing.<\/jats:p>","DOI":"10.1145\/3728969","type":"journal-article","created":{"date-parts":[[2025,6,22]],"date-time":"2025-06-22T10:52:56Z","timestamp":1750589576000},"page":"2091-2112","source":"Crossref","is-referenced-by-count":5,"title":["LogBase: A Large-Scale Benchmark for Semantic Log Parsing"],"prefix":"10.1145","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6773-598X","authenticated-orcid":false,"given":"Chenbo","family":"Zhang","sequence":"first","affiliation":[{"name":"Fudan University, Shanghai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-0688-576X","authenticated-orcid":false,"given":"Wenying","family":"Xu","sequence":"additional","affiliation":[{"name":"Fudan University, Shanghai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-6149-2651","authenticated-orcid":false,"given":"Jinbu","family":"Liu","sequence":"additional","affiliation":[{"name":"Alibaba Group, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9532-5219","authenticated-orcid":false,"given":"Lu","family":"Zhang","sequence":"additional","affiliation":[{"name":"Fudan University, Shanghai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-3383-9409","authenticated-orcid":false,"given":"Guiyang","family":"Liu","sequence":"additional","affiliation":[{"name":"Alibaba Group, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2313-7635","authenticated-orcid":false,"given":"Jihong","family":"Guan","sequence":"additional","affiliation":[{"name":"Tongji University, Shanghai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7759-800X","authenticated-orcid":false,"given":"Qi","family":"Zhou","sequence":"additional","affiliation":[{"name":"Alibaba Group, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1949-2768","authenticated-orcid":false,"given":"Shuigeng","family":"Zhou","sequence":"additional","affiliation":[{"name":"Fudan University, Shanghai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,6,22]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Diogo Almeida, Janko Altenschmidt, Sam Altman, and Shyamal Anadkat.","author":"Achiam Josh","year":"2023","unstructured":"Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, and Shyamal Anadkat. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774."},{"key":"e_1_2_1_2_1","doi-asserted-by":"crossref","unstructured":"Toufique Ahmed Premkumar Devanbu Christoph Treude and Michael Pradel. 2024. Can LLMs Replace Manual Annotation of Software Engineering Artifacts? arXiv preprint arXiv:2408.05534.","DOI":"10.1109\/MSR66628.2025.00086"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00031"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.58496\/MJCSC\/2023\/002"},{"key":"e_1_2_1_5_1","volume-title":"Michael Greenberg, Abhinav Jangda, and Arjun Guha.","author":"Cassano Federico","year":"2023","unstructured":"Federico Cassano, John Gouwar, Francesca Lucchetti, Claire Schlesinger, Carolyn Jane Anderson, Michael Greenberg, Abhinav Jangda, and Arjun Guha. 2023. Knowledge Transfer from High-Resource to Low-Resource Programming Languages for Code LLMs. arXiv preprint arXiv:2308.09895."},{"key":"e_1_2_1_6_1","first-page":"879","article-title":"Logram: Efficient log parsing using n n-gram dictionaries","volume":"48","author":"Dai Hetong","year":"2020","unstructured":"Hetong Dai, Heng Li, Che-Shao Chen, Weiyi Shang, and Tse-Hsun Chen. 2020. Logram: Efficient log parsing using n n-gram dictionaries. IEEE Transactions on Software Engineering, 48, 3 (2020), 879\u2013892.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3208040.3208051"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2016.0103"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3133956.3134015"},{"key":"e_1_2_1_10_1","volume-title":"Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155.","author":"Feng Zhangyin","year":"2020","unstructured":"Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, and Daxin Jiang. 2020. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155."},{"key":"e_1_2_1_11_1","unstructured":"Qiang Fu Jian-Guang Lou Yi Wang and Jiang Li. 2009. Execution anomaly detection in distributed systems through unstructured log analysis. In 2009 ninth IEEE international conference on data mining. 149\u2013158."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3558947"},{"key":"e_1_2_1_13_1","unstructured":"Aaron Grattafiori Abhimanyu Dubey Abhinav Jauhri Abhinav Pandey Abhishek Kadian Ahmad Al-Dahle Aiesha Letman Akhil Mathur Alan Schelten and Alex Vaughan. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783."},{"key":"e_1_2_1_14_1","volume-title":"Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural networks, 18, 5-6","author":"Graves Alex","year":"2005","unstructured":"Alex Graves and J\u00fcrgen Schmidhuber. 2005. Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural networks, 18, 5-6 (2005), 602\u2013610."},{"key":"e_1_2_1_15_1","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence. 38","author":"Guo Hongcheng","year":"2024","unstructured":"Hongcheng Guo, Jian Yang, Jiaheng Liu, Jiaqi Bai, Boyang Wang, Zhoujun Li, Tieqiao Zheng, Bo Zhang, Junran Peng, and Qi Tian. 2024. Logformer: A pre-train and tuning pipeline for log anomaly detection. In Proceedings of the AAAI Conference on Artificial Intelligence. 38, 135\u2013143."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2983323.2983358"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICWS.2017.13"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3236083"},{"key":"e_1_2_1_19_1","unstructured":"Junjie Huang Zhihan Jiang Zhuangbin Chen and Michael R Lyu. 2024. ULog: Unsupervised Log Parsing with Large Language Models through Log Contrastive Units. arXiv preprint arXiv:2406.07174."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00082"},{"key":"e_1_2_1_21_1","volume-title":"Proceedings of the ACM on Software Engineering, 1, FSE","author":"Jiang Zhihan","year":"2024","unstructured":"Zhihan Jiang, Jinyang Liu, Zhuangbin Chen, Yichen Li, Junjie Huang, Yintong Huo, Pinjia He, Jiazhen Gu, and Michael R Lyu. 2024. Lilac: Log parsing using llms with adaptive parsing cache. Proceedings of the ACM on Software Engineering, 1, FSE (2024), 137\u2013160."},{"key":"e_1_2_1_22_1","volume-title":"Lyu","author":"Jiang Zhihan","year":"2024","unstructured":"Zhihan Jiang, Jinyang Liu, Junjie Huang, Yichen Li, Yintong Huo, Jiazhen Gu, Zhuangbin Chen, Jieming Zhu, and Michael R. Lyu. 2024. A Large-Scale Evaluation for Log Parsing Techniques: How Far Are We? arxiv:2308.10828."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/QSIC.2008.50"},{"key":"e_1_2_1_24_1","volume-title":"2019 IEEE\/ACM 16th International Conference on Mining Software Repositories (MSR). 529\u2013533","author":"Kamp Marius","year":"2019","unstructured":"Marius Kamp, Patrick Kreutzer, and Michael Philippsen. 2019. SeSaMe: A data set of semantically similar Java methods. In 2019 IEEE\/ACM 16th International Conference on Mining Software Repositories (MSR). 529\u2013533."},{"key":"e_1_2_1_25_1","unstructured":"Kamal Raj Kanakarajan. 2019. Bert-NER. https:\/\/github.com\/kamalkraj\/BERT-NER"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00194"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510101"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510155"},{"key":"e_1_2_1_29_1","volume-title":"2023 IEEE\/ACM 45th International Conference on Software Engineering (ICSE). 2438\u20132449","author":"Le Van-Hoang","year":"2023","unstructured":"Van-Hoang Le and Hongyu Zhang. 2023. Log parsing with prompt-based few-shot learning. In 2023 IEEE\/ACM 45th International Conference on Software Engineering (ICSE). 2438\u20132449."},{"key":"e_1_2_1_30_1","volume-title":"Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, and Jenny Chim.","author":"Li Raymond","year":"2023","unstructured":"Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, and Jenny Chim. 2023. Starcoder: may the source be with you!. arXiv preprint arXiv:2305.06161."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE56229.2023.00089"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00078"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2889160.2889232"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3485447.3511993"},{"key":"e_1_2_1_35_1","volume-title":"2010 USENIX Annual Technical Conference (USENIX ATC 10)","author":"Lou Jian-Guang","year":"2010","unstructured":"Jian-Guang Lou, Qiang Fu, Shenqi Yang, Ye Xu, and Jiang Li. 2010. Mining invariants from console logs for system problem detection. In 2010 USENIX Annual Technical Conference (USENIX ATC 10)."},{"key":"e_1_2_1_36_1","doi-asserted-by":"crossref","unstructured":"Qinyu Luo Yining Ye Shihao Liang Zhong Zhang Yujia Qin Yaxi Lu Yesai Wu Xin Cong Yankai Lin and Yingli Zhang. 2024. RepoAgent: An LLM-Powered Open-Source Framework for Repository-level Code Documentation Generation. arXiv preprint arXiv:2402.16667.","DOI":"10.18653\/v1\/2024.emnlp-demo.46"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/1557019.1557154"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3196321.3196340"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/SCC.2013.73"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/MSR.2010.5463281"},{"key":"e_1_2_1_41_1","volume-title":"Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474.","author":"Nijkamp Erik","year":"2022","unstructured":"Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474."},{"key":"e_1_2_1_42_1","volume-title":"Proceedings of the 44th International Conference on Software Engineering. 1469\u20131481","author":"Patra Jibesh","year":"2022","unstructured":"Jibesh Patra and Michael Pradel. 2022. Nalin: learning from runtime behavior to find name-value inconsistencies in jupyter notebooks. In Proceedings of the 44th International Conference on Software Engineering. 1469\u20131481."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/D14-1162"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICPC58990.2023.00023"},{"key":"e_1_2_1_45_1","unstructured":"Keiichi Shima. 2016. Length matters: Clustering system log messages using length of words. arXiv preprint arXiv:1611.03213."},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/2063576.2063690"},{"key":"e_1_2_1_47_1","volume-title":"Pier Giuseppe Sessa, and Cassidy Hardin et al","author":"Team Gemma","year":"2024","unstructured":"Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, and Cassidy Hardin et al.. 2024. Gemma 2: Improving Open Language Models at a Practical Size. arXiv preprint arXiv:2408.00118."},{"key":"e_1_2_1_48_1","unstructured":"Federico Tomassetti N Smith C Maximilien and S Kirsch. 2021. JavaParser."},{"key":"e_1_2_1_49_1","volume-title":"Code Less","author":"Tsai Yun-Da","unstructured":"Yun-Da Tsai, Mingjie Liu, and Haoxing Ren. 2024. Code Less, Align More: Efficient LLM Fine-tuning for Code Generation with Data Pruning. arXiv preprint arXiv:2407.05040."},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1109\/IPOM.2003.1251233"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1109\/CNSM.2015.7367331"},{"key":"e_1_2_1_52_1","volume-title":"\u0141 ukasz Kaiser, and Illia Polosukhin","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, \u0141 ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30 (2017)."},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549176"},{"key":"e_1_2_1_54_1","volume-title":"Chi, Quoc V Le, and Denny Zhou","author":"Wei Jason","year":"2022","unstructured":"Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35 (2022), 24824\u201324837."},{"key":"e_1_2_1_55_1","unstructured":"Yeming Wen Pengcheng Yin Kensen Shi Henryk Michalewski Swarat Chaudhuri and Alex Polozov. 2024. Grounding Data Science Code Generation with Input-Output Specifications. arXiv preprint arXiv:2402.08073."},{"key":"e_1_2_1_56_1","unstructured":"Jason Wu Eldon Schoop Alan Leung Titus Barik Jeffrey P Bigham and Jeffrey Nichols. 2024. UICoder: Finetuning Large Language Models to Generate User Interface Code through Automated Feedback. arXiv preprint arXiv:2406.07739."},{"key":"e_1_2_1_57_1","unstructured":"Yifan Wu Bingxu Chai Siyu Yu Ying Li Pinjia He Wei Jiang and Jianguo Li. 2024. LogPTR: Variable-Aware Log Parsing with Pointer Network. arXiv preprint arXiv:2401.05986."},{"key":"e_1_2_1_58_1","unstructured":"Yifan Wu Siyu Yu and Ying Li. 2024. Log Parsing with Self-Generated In-Context Learning and Self-Correction. arXiv preprint arXiv:2406.03376."},{"key":"e_1_2_1_59_1","unstructured":"Yi Xiao Van-Hoang Le and Hongyu Zhang. 2024. Stronger Faster and Cheaper Log Parsing with LLMs. arXiv preprint arXiv:2406.06156."},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639155"},{"key":"e_1_2_1_61_1","article-title":"Brain: Log parsing with bidirectional parallel tree","author":"Yu Siyu","year":"2023","unstructured":"Siyu Yu, Pinjia He, Ningjiang Chen, and Yifan Wu. 2023. Brain: Log parsing with bidirectional parallel tree. IEEE Transactions on Services Computing.","journal-title":"IEEE Transactions on Services Computing."},{"key":"e_1_2_1_62_1","unstructured":"Zhiqiang Yuan Yiling Lou Mingwei Liu Shiji Ding Kaixin Wang Yixuan Chen and Xin Peng. 2023. No more manual tests? evaluating and improving chatgpt for unit test generation. arXiv preprint arXiv:2305.04207."},{"key":"e_1_2_1_63_1","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1007\/s10515-022-00370-w","article-title":"Semi-supervised and unsupervised anomaly detection by mining numerical workflow relations from system logs","volume":"30","author":"Zhang Bo","year":"2023","unstructured":"Bo Zhang, Hongyu Zhang, Van-Hoang Le, Pablo Moscato, and Aozhong Zhang. 2023. Semi-supervised and unsupervised anomaly detection by mining numerical workflow relations from system logs. Automated Software Engineering, 30, 1 (2023), 4.","journal-title":"Automated Software Engineering"},{"key":"e_1_2_1_64_1","volume-title":"2020 International Symposium on Reliable Distributed Systems (SRDS). 195\u2013204","author":"Zhang Bo","year":"2020","unstructured":"Bo Zhang, Hongyu Zhang, Pablo Moscato, and Aozhong Zhang. 2020. Anomaly detection via mining numerical workflow relations from logs. In 2020 International Symposium on Reliable Distributed Systems (SRDS). 195\u2013204."},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274465"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338931"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE59848.2023.00071"}],"container-title":["Proceedings of the ACM on Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3728969","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,16]],"date-time":"2025-07-16T16:47:23Z","timestamp":1752684443000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3728969"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,22]]},"references-count":67,"journal-issue":{"issue":"ISSTA","published-print":{"date-parts":[[2025,6,22]]}},"alternative-id":["10.1145\/3728969"],"URL":"https:\/\/doi.org\/10.1145\/3728969","relation":{},"ISSN":["2994-970X"],"issn-type":[{"value":"2994-970X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,22]]}}}