{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,12]],"date-time":"2026-09-12T01:56:55Z","timestamp":1789178215325,"version":"build-2803163510"},"reference-count":90,"publisher":"Association for Computing Machinery (ACM)","issue":"5","license":[{"start":{"date-parts":[[2024,6,3]],"date-time":"2024-06-03T00:00:00Z","timestamp":1717372800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62322208, 62232001"],"award-info":[{"award-number":["62322208, 62232001"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"CCF Young Elite Scientists Sponsorship Program"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2024,6,30]]},"abstract":"<jats:p>With the rapid development of deep learning (DL), the recent trend of log-based anomaly detection focuses on extracting semantic information from log events (i.e., templates of log messages) and designing more advanced DL models for anomaly detection. Indeed, the effectiveness of log-based anomaly detection can be improved, but these DL-based techniques further suffer from the limitations of more heavy dependency on training data (such as data quality or data labels) and higher costs in time and resources due to the complexity and scale of DL models, which hinder their practical use. On the contrary, the techniques based on traditional machine learning or data mining algorithms are less dependent on training data and more efficient but produce worse effectiveness than DL-based techniques, which is mainly caused by the problem of unseen log events (some log events in incoming log messages are unseen in training data) confirmed by our motivating study. Intuitively, if we can improve the effectiveness of traditional techniques to be comparable with advanced DL-based techniques, then log-based anomaly detection can be more practical. Indeed, an existing study in the other area (i.e., linking questions posted on Stack Overflow) has pointed out that traditional techniques with some optimizations can indeed achieve comparable effectiveness with the state-of-the-art DL-based technique, indicating the feasibility of enhancing traditional log-based anomaly detection techniques to some degree.<\/jats:p>\n          <jats:p>\n            Inspired by the idea of \u201ctry-with-simpler,\u201d we conducted the first empirical study to explore the potential of improving traditional techniques for more practical log-based anomaly detection. In this work, we optimized the traditional unsupervised PCA (Principal Component Analysis) technique by incorporating a lightweight semantic-based log representation in it, called\n            <jats:italic>SemPCA<\/jats:italic>\n            , and conducted an extensive study to investigate the potential of\n            <jats:italic>SemPCA<\/jats:italic>\n            for more practical log-based anomaly detection. By comparing seven log-based anomaly detection techniques (including four DL-based techniques, two traditional techniques, and\n            <jats:italic>SemPCA<\/jats:italic>\n            ) on both public and industrial datasets, our results show that\n            <jats:italic>SemPCA<\/jats:italic>\n            achieves comparable effectiveness as advanced supervised\/semi-supervised DL-based techniques while being much more stable under insufficient training data and more efficient, demonstrating that the traditional technique can still excel after small but useful adaptation.\n          <\/jats:p>","DOI":"10.1145\/3644386","type":"journal-article","created":{"date-parts":[[2024,2,7]],"date-time":"2024-02-07T12:00:01Z","timestamp":1707307201000},"page":"1-27","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":24,"title":["Try with Simpler - An Evaluation of Improved Principal Component Analysis in Log-based Anomaly Detection"],"prefix":"10.1145","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4475-0925","authenticated-orcid":false,"given":"Lin","family":"Yang","sequence":"first","affiliation":[{"name":"College of Intelligence and Computing, Tianjin University, Tianjin, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3056-9962","authenticated-orcid":false,"given":"Junjie","family":"Chen","sequence":"additional","affiliation":[{"name":"College of Intelligence and Computing, Tianjin University, Tianjin, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-4441-7971","authenticated-orcid":false,"given":"Shutao","family":"Gao","sequence":"additional","affiliation":[{"name":"Tianjin International Engineering Institute, the School of Future Technology, Tianjin University, Tianjin, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-5318-8473","authenticated-orcid":false,"given":"Zhihao","family":"Gong","sequence":"additional","affiliation":[{"name":"College of Intelligence and Computing, Tianjin University, Tianjin, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3063-9425","authenticated-orcid":false,"given":"Hongyu","family":"Zhang","sequence":"additional","affiliation":[{"name":"Chongqing University, Chongqing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-1862-4199","authenticated-orcid":false,"given":"Yue","family":"Kang","sequence":"additional","affiliation":[{"name":"College of Intelligence and Computing, Tianjin University, Tianjin, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-9227-2351","authenticated-orcid":false,"given":"Huaan","family":"Li","sequence":"additional","affiliation":[{"name":"College of Intelligence and Computing, Tianjin University, Tianjin, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,6,3]]},"reference":[{"key":"e_1_3_3_2_2","unstructured":"Fabian Pedregosa Gael Varoquaux Alexandre Gramfort and Vincent Michel. 2013. ECML PKDD Workshop: Languages for Data Mining and Machine Learning 108\u2013122. Retrieved from https:\/\/scikit-learn.org\/stable\/index.html"},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.1002\/wics.101"},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/2889160.2889231"},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10462-021-09975-1"},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00051"},{"key":"e_1_3_3_7_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-662-46578-3_53"},{"issue":"1","key":"e_1_3_3_8_2","first-page":"9","article-title":"Improving stability of PCA-based network anomaly detection by means of kernel-PCA","volume":"16","author":"Callegari Christian","year":"2018","unstructured":"Christian Callegari, Lisa Donatini, Stefano Giordano, and Michele Pagano. 2018. Improving stability of PCA-based network anomaly detection by means of kernel-PCA. Int. J. Comput. Sci. Eng. 16, 1 (2018), 9\u201316.","journal-title":"Int. J. Comput. Sci. Eng."},{"key":"e_1_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEIP.2019.00020"},{"key":"e_1_3_3_10_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00042"},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.5555\/1078026.1078407"},{"key":"e_1_3_3_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/3569932"},{"key":"e_1_3_3_13_2","article-title":"Experience report: Deep learning-based system log analysis for anomaly detection","volume":"2107","author":"Chen Zhuangbin","year":"2021","unstructured":"Zhuangbin Chen, Jinyang Liu, Wenwei Gu, Yuxin Su, and Michael R. Lyu. 2021. Experience report: Deep learning-based system log analysis for anomaly detection. CoRR abs\/2107.05908 (2021).","journal-title":"CoRR"},{"key":"e_1_3_3_14_2","article-title":"Empirical evaluation of gated recurrent neural networks on sequence modeling","volume":"1412","author":"Chung Junyoung","year":"2014","unstructured":"Junyoung Chung, \u00c7aglar G\u00fcl\u00e7ehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. CoRR abs\/1412.3555 (2014).","journal-title":"CoRR"},{"key":"e_1_3_3_15_2","article-title":"Logram: Efficient log parsing using n-gram dictionaries","volume":"2001","author":"Dai Hetong","year":"2020","unstructured":"Hetong Dai, Heng Li, Weiyi Shang, Tse-Hsun Chen, and Che-Shao Chen. 2020. Logram: Efficient log parsing using n-gram dictionaries. CoRR abs\/2001.03038 (2020).","journal-title":"CoRR"},{"key":"e_1_3_3_16_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDCS.2018.00105"},{"key":"e_1_3_3_17_2","article-title":"Avoiding overfitting: A survey on regularization methods for convolutional neural networks","volume":"2201","author":"Santos Claudio Filipi Gon\u00e7alves dos","year":"2022","unstructured":"Claudio Filipi Gon\u00e7alves dos Santos and Jo\u00e3o Paulo Papa. 2022. Avoiding overfitting: A survey on regularization methods for convolutional neural networks. CoRR abs\/2201.03299 (2022).","journal-title":"CoRR"},{"key":"e_1_3_3_18_2","first-page":"1285","volume-title":"CCS","author":"Du Min","year":"2017","unstructured":"Min Du, Feifei Li, Guineng Zheng, and Vivek Srikumar. 2017. DeepLog: Anomaly detection and diagnosis from system logs through deep learning. In CCS. ACM, 1285\u20131298."},{"key":"e_1_3_3_19_2","volume-title":"ACC","author":"Dunia Ricardo","year":"1997","unstructured":"Ricardo Dunia and S. Joe Qin. 1997. Multi-dimensional fault diagnosis using a subspace approach. In ACC. Citeseer."},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER.2018.8330265"},{"key":"e_1_3_3_21_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSRS53853.2021.9660694"},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/3106237.3106256"},{"key":"e_1_3_3_23_2","series-title":"JMLR Workshop and Conference Proceedings","first-page":"1833","volume-title":"ICML","volume":"37","author":"G\u00f6rnitz Nico","year":"2015","unstructured":"Nico G\u00f6rnitz, Mikio L. Braun, and Marius Kloft. 2015. Hidden Markov anomaly detection. In ICML(JMLR Workshop and Conference Proceedings, Vol. 37). JMLR.org, 1833\u20131842."},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.2307\/2346439"},{"key":"e_1_3_3_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICWS.2017.13"},{"issue":"6","key":"e_1_3_3_26_2","first-page":"130:1\u2013130:37","article-title":"A survey on automated log analysis for reliability engineering","volume":"54","author":"He Shilin","year":"2021","unstructured":"Shilin He, Pinjia He, Zhuangbin Chen, Tianyi Yang, Yuxin Su, and Michael R. Lyu. 2021. A survey on automated log analysis for reliability engineering. ACM Comput. Surv. 54, 6 (2021), 130:1\u2013130:37.","journal-title":"ACM Comput. Surv."},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3236083"},{"key":"e_1_3_3_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3558963"},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.2016.21"},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.1007\/s007999900025"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME.2017.51"},{"key":"e_1_3_3_32_2","doi-asserted-by":"publisher","DOI":"10.1080\/00401706.1979.10489779"},{"key":"e_1_3_3_33_2","article-title":"FastText.zip: Compressing text classification models","author":"Joulin Armand","year":"2016","unstructured":"Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, H\u00e9rve J\u00e9gou, and Tomas Mikolov. 2016. FastText.zip: Compressing text classification models. arXiv preprint arXiv:1612.03651 (2016).","journal-title":"arXiv preprint arXiv:1612.03651"},{"key":"e_1_3_3_34_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/E17-2068"},{"key":"e_1_3_3_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE51524.2021.9678773"},{"key":"e_1_3_3_36_2","article-title":"Log-based anomaly detection with deep learning: How far are we?","volume":"2202","author":"Le Van-Hoang","year":"2022","unstructured":"Van-Hoang Le and Hongyu Zhang. 2022. Log-based anomaly detection with deep learning: How far are we? CoRR abs\/2202.04301 (2022).","journal-title":"CoRR"},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.1976.225946"},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.1109\/IWQOS52092.2021.9521340"},{"key":"e_1_3_3_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/2889160.2889232"},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00085"},{"key":"e_1_3_3_41_2","volume-title":"USENIX Annual Technical Conference","author":"Lou Jian-Guang","year":"2010","unstructured":"Jian-Guang Lou, Qiang Fu, Shengqi Yang, Ye Xu, and Jiang Li. 2010. Mining invariants from console logs for system problem detection. In USENIX Annual Technical Conference. USENIX Association."},{"key":"e_1_3_3_42_2","doi-asserted-by":"publisher","DOI":"10.1109\/DASC\/PiCom\/DataCom\/CyberSciTec.2018.00037"},{"key":"e_1_3_3_43_2","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511809071"},{"key":"e_1_3_3_44_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDMW.2017.12"},{"key":"e_1_3_3_45_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCCN49398.2020.9209707"},{"key":"e_1_3_3_46_2","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2019\/658"},{"key":"e_1_3_3_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSR.2010.5463281"},{"key":"e_1_3_3_48_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P16-2074"},{"key":"e_1_3_3_49_2","doi-asserted-by":"publisher","DOI":"10.1109\/DSN.2007.103"},{"key":"e_1_3_3_50_2","article-title":"Robust and transferable anomaly detection in log data using pre-trained language models","volume":"2102","author":"Ott Harold","year":"2021","unstructured":"Harold Ott, Jasmin Bogatinovski, Alexander Acker, Sasho Nedelkoski, and Odej Kao. 2021. Robust and transferable anomaly detection in log data using pre-trained language models. CoRR abs\/2102.11570 (2021).","journal-title":"CoRR"},{"key":"e_1_3_3_51_2","article-title":"Robust PCA for anomaly detection in cyber networks","volume":"1801","author":"Paffenroth Randy C.","year":"2018","unstructured":"Randy C. Paffenroth, Kathleen Kay, and Les Servi. 2018. Robust PCA for anomaly detection in cyber networks. CoRR abs\/1801.01571 (2018).","journal-title":"CoRR"},{"key":"e_1_3_3_52_2","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/D14-1162"},{"key":"e_1_3_3_53_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2015.2421279"},{"key":"e_1_3_3_54_2","doi-asserted-by":"publisher","DOI":"10.1145\/3234150"},{"key":"e_1_3_3_55_2","article-title":"Evaluation: From precision, recall and F-measure to ROC, informedness, markedness and correlation","volume":"2010","author":"Powers David M. W.","year":"2020","unstructured":"David M. W. Powers. 2020. Evaluation: From precision, recall and F-measure to ROC, informedness, markedness and correlation. CoRR abs\/2010.16061 (2020).","journal-title":"CoRR"},{"key":"e_1_3_3_56_2","article-title":"The use and interpretation of principal component analysis in applied research","author":"Rao C. Radhakrishna","year":"1964","unstructured":"C. Radhakrishna Rao. 1964. The use and interpretation of principal component analysis in applied research. Sankhy\u0101: Indian J. Stat., Series A 26 (1964), 329\u2013358.","journal-title":"Sankhy\u0101: Indian J. Stat., Series A"},{"key":"e_1_3_3_57_2","volume-title":"Pattern Recognition and Neural Networks","author":"Ripley Brian D.","year":"2007","unstructured":"Brian D. Ripley. 2007. Pattern Recognition and Neural Networks. Cambridge University Press."},{"key":"e_1_3_3_58_2","doi-asserted-by":"publisher","DOI":"10.1016\/0306-4573(88)90021-0"},{"key":"e_1_3_3_59_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2019.2912200"},{"key":"e_1_3_3_60_2","volume-title":"A Novel Anomaly Detection Scheme Based on Principal Component Classifier","author":"Shyu Mei-Ling","year":"2003","unstructured":"Mei-Ling Shyu, Shu-Ching Chen, Kanoksri Sarinnapakorn, and LiWu Chang. 2003. A Novel Anomaly Detection Scheme Based on Principal Component Classifier. Technical Report. Miami University Coral Gables Florida Department of Electrical and Computer Engineering."},{"key":"e_1_3_3_61_2","doi-asserted-by":"publisher","DOI":"10.1109\/CCGRID.2008.107"},{"key":"e_1_3_3_62_2","doi-asserted-by":"publisher","DOI":"10.1109\/TASLP.2015.2400218"},{"key":"e_1_3_3_63_2","first-page":"273","volume-title":"Middleware","author":"Tak Byungchul","year":"2021","unstructured":"Byungchul Tak and Wook-Shin Han. 2021. Lognroll: Discovering accurate log templates by iterative filtering. In Middleware. ACM, 273\u2013285."},{"key":"e_1_3_3_64_2","doi-asserted-by":"crossref","unstructured":"Naoya Takeishi. 2019. Shapley values of reconstruction errors of PCA for explaining anomaly detection. ICDM Workshops IEEE 793\u2013798.","DOI":"10.1109\/ICDMW.2019.00117"},{"key":"e_1_3_3_65_2","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3236054"},{"key":"e_1_3_3_66_2","doi-asserted-by":"publisher","DOI":"10.1109\/IPOM.2003.1251233"},{"key":"e_1_3_3_67_2","first-page":"1","volume-title":"CNSM","author":"Vaarandi Risto","year":"2015","unstructured":"Risto Vaarandi and Mauno Pihelgas. 2015. LogCluster\u2014A data clustering and pattern mining algorithm for event logs. In CNSM. IEEE Computer Society, 1\u20137."},{"key":"e_1_3_3_68_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICACCI.2017.8125846"},{"key":"e_1_3_3_69_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICWS49710.2020.00026"},{"key":"e_1_3_3_70_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2022.107119"},{"key":"e_1_3_3_71_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE52982.2021.00017"},{"key":"e_1_3_3_72_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00046"},{"key":"e_1_3_3_73_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-34637-9_5"},{"key":"e_1_3_3_74_2","article-title":"LogGD: Detecting anomalies from system logs by graph neural networks","volume":"2209","author":"Xie Yongzheng","year":"2022","unstructured":"Yongzheng Xie, Hongyu Zhang, and Muhammad Ali Babar. 2022. LogGD: Detecting anomalies from system logs by graph neural networks. CoRR abs\/2209.07869 (2022).","journal-title":"CoRR"},{"key":"e_1_3_3_75_2","first-page":"37","volume-title":"ICML","author":"Xu Wei","year":"2010","unstructured":"Wei Xu, Ling Huang, Armando Fox, David A. Patterson, and Michael I. Jordan. 2010. Detecting large-scale system problems by mining console logs. In ICML. Omnipress, 37\u201346."},{"key":"e_1_3_3_76_2","doi-asserted-by":"publisher","DOI":"10.1145\/1081870.1081927"},{"key":"e_1_3_3_77_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00130"},{"key":"e_1_3_3_78_2","volume-title":"Anomaly Detection as a Service: Challenges, Advances, and Opportunities","author":"Yao Danfeng (Daphne)","year":"2017","unstructured":"Danfeng (Daphne) Yao, Xiaokui Shu, Long Cheng, and Salvatore J. Stolfo. 2017. Anomaly Detection as a Service: Challenges, Advances, and Opportunities. Morgan & Claypool Publishers."},{"key":"e_1_3_3_79_2","doi-asserted-by":"publisher","DOI":"10.1109\/SRDS51746.2020.00027"},{"key":"e_1_3_3_80_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510180"},{"key":"e_1_3_3_81_2","doi-asserted-by":"publisher","DOI":"10.1145\/3534678.3539155"},{"key":"e_1_3_3_82_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338931"},{"key":"e_1_3_3_83_2","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPS.2008.4536397"},{"key":"e_1_3_3_84_2","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3556920"},{"key":"e_1_3_3_85_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3468543"},{"key":"e_1_3_3_86_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3473933"},{"key":"e_1_3_3_87_2","article-title":"Trine: Syslog anomaly detection with three transformer encoders in one generative adversarial network","author":"Zhao Zhenfei","year":"2021","unstructured":"Zhenfei Zhao, Weina Niu, Xiaosong Zhang, Runzi Zhang, Zhenqi Yu, and Cheng Huang. 2021. Trine: Syslog anomaly detection with three transformer encoders in one generative adversarial network. Appl. Intell. 52, 8 (2022), 8810\u20138819.","journal-title":"Appl. Intell."},{"key":"e_1_3_3_88_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIFS.2022.3201379"},{"key":"e_1_3_3_89_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338961"},{"key":"e_1_3_3_90_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2015.60"},{"key":"e_1_3_3_91_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEIP.2019.00021"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3644386","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3644386","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T22:50:48Z","timestamp":1750287048000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3644386"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6,3]]},"references-count":90,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2024,6,30]]}},"alternative-id":["10.1145\/3644386"],"URL":"https:\/\/doi.org\/10.1145\/3644386","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,6,3]]},"assertion":[{"value":"2023-01-02","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-01-15","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-06-03","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}