{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,18]],"date-time":"2026-07-18T02:43:27Z","timestamp":1784342607386,"version":"3.55.0"},"reference-count":102,"publisher":"Association for Computing Machinery (ACM)","issue":"8","license":[{"start":{"date-parts":[[2024,11,30]],"date-time":"2024-11-30T00:00:00Z","timestamp":1732924800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Australian Research Council\u2019s Discovery Early Career Researcher","award":["DE200100941, DE210101091"],"award-info":[{"award-number":["DE200100941, DE210101091"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2024,11,30]]},"abstract":"<jats:p>In recent years, large pre-trained Language Models of Code (CodeLMs) have shown promising results on various software engineering tasks. One such task is automatic code update recommendation, which transforms outdated code snippets into their approved and revised counterparts. Although many CodeLM-based approaches have been proposed, claiming high accuracy, their effectiveness and reliability on real-world code update tasks remain questionable. In this article, we present the first extensive evaluation of state-of-the-art CodeLMs for automatically recommending code updates. We assess their performance on two diverse datasets of paired updated methods, considering factors such as temporal evolution, project specificity, method size, and update complexity. Our results reveal that while CodeLMs exhibit higher performance in settings that ignore temporal information, they struggle in more realistic time-wise scenarios and generalize poorly to new projects. Furthermore, CodeLM performance decreases significantly for larger methods and more complex updates. Furthermore, we observe that many CodeLM-generated \u201cupdates\u201d are actually null, especially in time-wise settings, and meaningful edits remain challenging. Our findings highlight the significant gap between the perceived and actual effectiveness of CodeLMs for real-world code update recommendation and emphasize the need for more research on improving their practicality, robustness, and generalizability.<\/jats:p>","DOI":"10.1145\/3678167","type":"journal-article","created":{"date-parts":[[2024,7,16]],"date-time":"2024-07-16T13:37:01Z","timestamp":1721137021000},"page":"1-27","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":14,"title":["Automatically Recommend Code Updates: Are We There Yet?"],"prefix":"10.1145","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5593-5917","authenticated-orcid":false,"given":"Yue","family":"Liu","sequence":"first","affiliation":[{"name":"Monash University, Melbourne, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5516-9984","authenticated-orcid":false,"given":"Chakkrit","family":"Tantithamthavorn","sequence":"additional","affiliation":[{"name":"Monash University, Melbourne, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7548-5100","authenticated-orcid":false,"given":"Yonghui","family":"Liu","sequence":"additional","affiliation":[{"name":"Monash University, Melbourne, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6328-8839","authenticated-orcid":false,"given":"Patanamon","family":"Thongtanunam","sequence":"additional","affiliation":[{"name":"The University of Melbourne, Melbourne, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2990-1614","authenticated-orcid":false,"given":"Li","family":"Li","sequence":"additional","affiliation":[{"name":"Beihang University, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,12,3]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"Google bigquery. 2021. Retrieved from https:\/\/console.cloud.google.com\/marketplace\/details\/github\/github-repos"},{"key":"e_1_3_2_3_2","unstructured":"Gerrit code review. 2022. Retrieved from https:\/\/www.gerritcodereview.com\/"},{"key":"e_1_3_2_4_2","unstructured":"Kotlin 2022. Retrieved from https:\/\/kotlinlang.org\/"},{"key":"e_1_3_2_5_2","unstructured":"Android\u2019s gerrit code review repositories. 2024. Retrieved from https:\/\/android-review.googlesource.com\/"},{"key":"e_1_3_2_6_2","unstructured":"Google\u2019s gerrit code review repositories. 2024. Retrieved from https:\/\/gerrit-review.googlesource.com\/"},{"key":"e_1_3_2_7_2","unstructured":"Ovirt\u2019s gerrit code review repositories. 2024. Retrieved from https:\/\/gerrit.ovirt.org\/"},{"key":"e_1_3_2_8_2","first-page":"1050","volume-title":"Proceedings of the 2018 IEEE\/ACM 40th International Conference on Software Engineering (ICSE)","author":"Agrawal A.","year":"2018","unstructured":"A. Agrawal and T. Menzies. 2018. Is \u201cbetter data\u201d better than \u201cbetter data miners\u201d? In Proceedings of the 2018 IEEE\/ACM 40th International Conference on Software Engineering (ICSE). IEEE, 1050\u20131061."},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-020-09808-9"},{"key":"e_1_3_2_10_2","doi-asserted-by":"crossref","unstructured":"S. Alagarsamy C. Tantithamthavorn and A. Aleti. 2023. A3test: Assertion-augmented automated test case generation. arXiv:2302.10352. Retrieved from https:\/\/doi.org\/10.48550\/arXiv.2302.10352","DOI":"10.2139\/ssrn.4724885"},{"key":"e_1_3_2_11_2","doi-asserted-by":"crossref","unstructured":"S. Alagarsamy C. Tantithamthavorn C. Arora and A. Aleti. 2024. Enhancing large language models for text-to-testcase generation. arXiv:2402.11910. Retrieved from https:\/\/arxiv.org\/abs\/2402.11910","DOI":"10.2139\/ssrn.4732705"},{"key":"e_1_3_2_12_2","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"Brown T.","year":"2020","unstructured":"T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, and A. Askell. 2020. Language models are few-shot learners, Advances in Neural Information Processing Systems. 33 (2020), 1877\u20131901.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180240"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380327"},{"key":"e_1_3_2_15_2","unstructured":"M. Chen J. Tworek H. Jun Q. Yuan H. P. d. O. Pinto J. Kaplan H. Edwards Y. Burda N. Joseph G. Brockman. 2021. Evaluating large language models trained on code. arXiv:2107.03374. Retrieved from https:\/\/arxiv.org\/abs\/2107.03374"},{"key":"e_1_3_2_16_2","volume-title":"PMD Applied","author":"Copeland T.","year":"2005","unstructured":"T. Copeland. 2005. PMD Applied (vol. 10). Centennial Books, San Francisco, CA, USA."},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2015.131"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSR.2019.00017"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2018.2881961"},{"key":"e_1_3_2_20_2","unstructured":"A. Fan B. Gokkaya M. Harman M. Lyubarskiy S. Sengupta S. Yoo and J. M. Zhang. 2023. Large language models for software engineering: Survey and open problems. arXiv: 2310.03533. Retrieved from https:\/\/arxiv.org\/abs\/2310.03533"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.findings-emnlp.139"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/3632746"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2023.3305244"},{"key":"e_1_3_2_24_2","doi-asserted-by":"crossref","unstructured":"M. Fu J. Pasuksmit and C. Tantithamthavorn. 2024. AI for devsecops: A landscape and future opportunities. arXiv:2404.04839. Retrieved from https:\/\/arxiv.org\/abs\/2404.04839","DOI":"10.1145\/3712190"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/3524842.3528452"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-023-10346-3"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549098"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/APSEC60848.2023.00085"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2016.04.017"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2015.91"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSR.2017.18"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-019-09727-4"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10710-017-9314-z"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180167"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.acl-long.499"},{"key":"e_1_3_2_36_2","first-page":"531","article-title":"Polisis: Automated analysis and presentation of privacy policies using deep learning","author":"Harkous H.","year":"2018","unstructured":"H. Harkous, K. Fawaz, R. Lebret, F. Schaub, K. G. Shin, and K. Aberer. 2018. Polisis: Automated analysis and presentation of privacy policies using deep learning. In Proceedings of the 27th USENIX Security Symposium (USENIX Security 18), 531\u2013548.","journal-title":"Proceedings of the 27th USENIX Security Symposium (USENIX Security 18)"},{"key":"e_1_3_2_37_2","first-page":"370","volume-title":"Proceedings of the ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE)","author":"Hong Y.","year":"2024","unstructured":"Y. Hong, C. Tantithamthavorn, J. Pasuksmit, P. Thongtanunam, A. Friedman, X. Zhao, and A. Krasikov. 2024. Practitioners\u2019 challenges and perceptions of ci build failure predictions at atlassian. In Proceedings of the ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE), 370\u2013381."},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549119"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER53432.2022.00121"},{"key":"e_1_3_2_40_2","unstructured":"X. Hou Y. Zhao Y. Liu Z. Yang K. Wang L. Li X. Luo D. Lo J. Grundy and H. Wang. 2023. Large language models for software engineering: A systematic literature review. arXiv: 2308.10620. Retrieved from https:\/\/arxiv.org\/abs\/2308.10620"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2019.2920771"},{"key":"e_1_3_2_42_2","unstructured":"H. Husain H.-H. Wu T. Gazit M. Allamanis and M. Brockschmidt. 2019. Codesearchnet challenge: Evaluating the state of semantic code search. arXiv: 1909.09436. Retrieved from https:\/\/arxiv.org\/abs\/1909.09436"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510172"},{"key":"e_1_3_2_44_2","unstructured":"J. Jiarpakdee C. Tantithamthavorn and A. E. Hassan. 2018. The impact of correlated metrics on defect models. arXiv:1801.10271. Retrieved from https:\/\/arxiv.org\/abs\/1801.10271"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSREW.2016.30"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSME.2018.00018"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2020.2982385"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338941"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2016.2597849"},{"key":"e_1_3_2_50_2","first-page":"4171","volume-title":"Proceedings of NAACL-HLT","author":"Kenton J. D. M.-W. C.","year":"2019","unstructured":"J. D. M.-W. C. Kenton and L. K. Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, 4171\u20134186."},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00026"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSR.2007.26"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/2884781.2884840"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1038\/nature14539"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597207"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549081"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416591"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/3379597.3387503"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/3560815"},{"issue":"5","key":"e_1_3_2_60_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3643674","article-title":"Refining chatgpt-generated code: Characterizing and mitigating code quality issues.","volume":"33","author":"Liu Y.","year":"2023","unstructured":"Y. Liu, T. Le-Cong, R. Widyasari, C. Tantithamthavorn, L. Li, X.-B. D. Le, and D. Lo. 2023. Refining chatgpt-generated code: Characterizing and mitigating code quality issues. ACM Transactions on Software Engineering and Methodology 33, 5 (2024), 1\u201326.","journal-title":"ACM Transactions on Software Engineering and Methodology"},{"issue":"8","key":"e_1_3_2_61_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3544968","article-title":"Deep learning for android malware defenses: a systematic literature review.","volume":"55","author":"Liu Y.","year":"2022","unstructured":"Y. Liu, C. Tantithamthavorn, L. Li, and Y. Liu. 2022. Deep learning for android malware defenses: a systematic literature review. ACM Computing Surveys 55, 8 (2022), 1\u201336.","journal-title":"ACM Computing Surveys"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE55969.2022.00026"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1145\/3649594"},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597926.3598080"},{"key":"e_1_3_2_65_2","unstructured":"S. Lu D. Guo S. Ren J. Huang A. Svyatkovskiy A. Blanco C. Clement D. Drain D. Jiang and D. Tang. 2021. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv:2102.04664. Retrieved from https:\/\/arxiv.org\/abs\/2102.04664"},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.1145\/3439726"},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00180"},{"key":"e_1_3_2_68_2","first-page":"311","volume-title":"Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics","author":"Papineni K.","year":"2002","unstructured":"K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, 311\u2013318."},{"key":"e_1_3_2_69_2","first-page":"729","article-title":"{TESSERACT}: Eliminating experimental bias in malware classification across space and time","author":"Pendlebury F.","year":"2019","unstructured":"F. Pendlebury, F. Pierazzi, R. Jordaney, J. Kinder, and L. Cavallaro. 2019. {TESSERACT}: Eliminating experimental bias in malware classification across space and time. In 28th USENIX Security Symposium (USENIX Security 19), 729\u2013746.","journal-title":"28th USENIX Security Symposium (USENIX Security 19)"},{"key":"e_1_3_2_70_2","unstructured":"C. Pornprasit and C. Tantithamthavorn. 2024. Gpt-3.5 for code review automation: How do few-shot learning prompt design and model fine-tuning impact their performance? arXiv: 2402.00905. Retrieved from https:\/\/arxiv.org\/abs\/2402.00905"},{"key":"e_1_3_2_71_2","doi-asserted-by":"publisher","DOI":"10.1109\/SANER56733.2023.00036"},{"key":"e_1_3_2_72_2","doi-asserted-by":"publisher","DOI":"10.1109\/MSR52588.2021.00049"},{"key":"e_1_3_2_73_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2022.3144348"},{"key":"e_1_3_2_74_2","doi-asserted-by":"publisher","DOI":"10.5555\/3455716.3455856"},{"key":"e_1_3_2_75_2","unstructured":"S. Ren D. Guo S. Lu L. Zhou S. Liu D. Tang N. Sundaresan M. Zhou A. Blanco and S. Ma. 2020. Codebleu: A method for automatic evaluation of code synthesis. arXiv: 2009.10297. Retrieved from https:\/\/arxiv.org\/abs\/2009.10297"},{"key":"e_1_3_2_76_2","doi-asserted-by":"publisher","DOI":"10.1145\/2491411.2491444"},{"key":"e_1_3_2_77_2","unstructured":"X. She Y. Liu Y. Zhao Y. He L. Li C. Tantithamthavorn Z. Qin and H. Wang. 2023. Pitfalls in language models for code intelligence: A taxonomy and survey. arXiv: 2310.17903. Retrieved from https:\/\/arxiv.org\/abs\/2310.17903."},{"key":"e_1_3_2_78_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510060"},{"key":"e_1_3_2_79_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549145"},{"key":"e_1_3_2_80_2","doi-asserted-by":"publisher","DOI":"10.1145\/3183519.3183547"},{"key":"e_1_3_2_81_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2018.2876537"},{"key":"e_1_3_2_82_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2015.93"},{"key":"e_1_3_2_83_2","doi-asserted-by":"publisher","DOI":"10.1145\/2884781.2884857"},{"key":"e_1_3_2_84_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2016.2584050"},{"key":"e_1_3_2_85_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510067"},{"key":"e_1_3_2_86_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00027"},{"key":"e_1_3_2_87_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00021"},{"key":"e_1_3_2_88_2","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3240732"},{"key":"e_1_3_2_89_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510621"},{"key":"e_1_3_2_90_2","first-page":"1","article-title":"Attention is all you need.","volume":"30","author":"Vaswani A.","year":"2017","unstructured":"A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, \u0141 Kaiser, and I. Polosukhin. 2017. Attention is all you need. Advances in Neural Information Processing Systems 30 (2017), 1\u201311.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_91_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510062"},{"key":"e_1_3_2_92_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.emnlp-main.68"},{"key":"e_1_3_2_93_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.685"},{"key":"e_1_3_2_94_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380429"},{"key":"e_1_3_2_95_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2020.3023177"},{"key":"e_1_3_2_96_2","doi-asserted-by":"crossref","unstructured":"T. Wolf L. Debut V. Sanh J. Chaumond C. Delangue A. Moi P. Cistac T. Rault R. Louf and M. Funtowicz. 2019. Huggingface\u2019s transformers: State-of-the-art natural language processing. arXiv:1910.03771. Retrieved from https:\/\/arxiv.org\/abs\/1910.03771","DOI":"10.18653\/v1\/2020.emnlp-demos.6"},{"key":"e_1_3_2_97_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380357"},{"key":"e_1_3_2_98_2","doi-asserted-by":"publisher","DOI":"10.1145\/3505243"},{"key":"e_1_3_2_99_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00075"},{"key":"e_1_3_2_100_2","doi-asserted-by":"publisher","DOI":"10.1109\/TST.2016.7399288"},{"key":"e_1_3_2_101_2","doi-asserted-by":"publisher","DOI":"10.1145\/3533767.3534390"},{"key":"e_1_3_2_102_2","doi-asserted-by":"publisher","DOI":"10.1145\/3372297.3417291"},{"key":"e_1_3_2_103_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510128"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3678167","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3678167","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T22:54:08Z","timestamp":1750287248000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3678167"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,11,30]]},"references-count":102,"journal-issue":{"issue":"8","published-print":{"date-parts":[[2024,11,30]]}},"alternative-id":["10.1145\/3678167"],"URL":"https:\/\/doi.org\/10.1145\/3678167","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,11,30]]},"assertion":[{"value":"2022-08-23","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-07-03","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-12-03","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}