{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,2]],"date-time":"2025-08-02T19:01:33Z","timestamp":1754161293104,"version":"3.41.2"},"publisher-location":"New York, NY, USA","reference-count":27,"publisher":"ACM","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2025,6,23]]},"DOI":"10.1145\/3696630.3728520","type":"proceedings-article","created":{"date-parts":[[2025,7,28]],"date-time":"2025-07-28T19:08:09Z","timestamp":1753729689000},"page":"666-670","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Measuring What Matters: An Aggregate Metric for Assessing Enterprise Code Summaries"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0009-0001-5244-0695","authenticated-orcid":false,"given":"Ashita","family":"Saxena","sequence":"first","affiliation":[{"name":"IBM Research, Bangalore, India"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-6096-1045","authenticated-orcid":false,"given":"Palanivel","family":"Kodeswaran","sequence":"additional","affiliation":[{"name":"IBM Research, Bangalore, India"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3791-3842","authenticated-orcid":false,"given":"Sayandeep","family":"Sen","sequence":"additional","affiliation":[{"name":"IBM Research, Bangalore, India"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3718-4849","authenticated-orcid":false,"given":"Srikanth","family":"Tamilselvam","sequence":"additional","affiliation":[{"name":"IBM Research, Bangalore, India"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2025,7,28]]},"reference":[{"key":"e_1_3_2_1_1_1","volume-title":"CodeMirage: Hallucinations in Code Generated by Large Language Models. arXiv preprint arXiv:2408.08333","author":"Agarwal Vibhor","year":"2024","unstructured":"Vibhor Agarwal, Yulong Pei, Salwa Alamir, and Xiaomo Liu. 2024. CodeMirage: Hallucinations in Code Generated by Large Language Models. arXiv preprint arXiv:2408.08333 (2024)."},{"key":"e_1_3_2_1_2_1","volume-title":"Md Muktadir Mazumder, and Salman Ahsan","author":"Akib Md Ahnaf","year":"2024","unstructured":"Md Ahnaf Akib, Md Muktadir Mazumder, and Salman Ahsan. 2024. Analysis on LLMs Performance for Code Summarization. arXiv preprint arXiv:2412.17094 (2024)."},{"key":"e_1_3_2_1_3_1","volume-title":"Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and\/or summarization. 65\u201372","author":"Banerjee Satanjeev","year":"2005","unstructured":"Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and\/or summarization. 65\u201372."},{"key":"e_1_3_2_1_4_1","volume-title":"Analyzing the performance of large language models on code summarization. arXiv preprint arXiv:2404.08018","author":"Haldar Rajarshi","year":"2024","unstructured":"Rajarshi Haldar and Julia Hockenmaier. 2024. Analyzing the performance of large language models on code summarization. arXiv preprint arXiv:2404.08018 (2024)."},{"key":"e_1_3_2_1_5_1","volume-title":"Kazi Sajeed Mehrab, Md Mahim Anjum Haque, Tahmid Hasan, Wasi Uddin Ahmad, Anindya Iqbal, and Rifat Shahriyar.","author":"Hasan Masum","year":"2021","unstructured":"Masum Hasan, Tanveer Muttaqueen, Abdullah Al Ishtiaq, Kazi Sajeed Mehrab, Md Mahim Anjum Haque, Tahmid Hasan, Wasi Uddin Ahmad, Anindya Iqbal, and Rifat Shahriyar. 2021. Codesc: A large code-description parallel dataset. arXiv preprint arXiv:2105.14220 (2021)."},{"key":"e_1_3_2_1_6_1","volume-title":"Codesearchnet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.09436","author":"Husain Hamel","year":"2019","unstructured":"Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. Codesearchnet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.09436 (2019)."},{"key":"e_1_3_2_1_7_1","volume-title":"A Survey on Large Language Models for Code Generation. arXiv preprint arXiv:2406.00515","author":"Jiang Juyong","year":"2024","unstructured":"Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024. A Survey on Large Language Models for Code Generation. arXiv preprint arXiv:2406.00515 (2024)."},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1007\/s10489-024-06197-0","article-title":"Leveraging meta-data of code for adapting prompt tuning for code summarization","volume":"55","author":"Jiang Zhihua","year":"2025","unstructured":"Zhihua Jiang, Di Wang, and Dongning Rao. 2025. Leveraging meta-data of code for adapting prompt tuning for code summarization. Applied Intelligence 55, 2 (2025), 211.","journal-title":"Applied Intelligence"},{"key":"e_1_3_2_1_9_1","volume-title":"Identifying inaccurate descriptions in llm-generated code comments via test execution. arXiv preprint arXiv:2406.14836","author":"Kang Sungmin","year":"2024","unstructured":"Sungmin Kang, Louis Milliken, and Shin Yoo. 2024. Identifying inaccurate descriptions in llm-generated code comments via test execution. arXiv preprint arXiv:2406.14836 (2024)."},{"key":"e_1_3_2_1_10_1","volume-title":"Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74\u201381.","author":"Lin Chin-Yew","year":"2004","unstructured":"Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74\u201381."},{"key":"e_1_3_2_1_11_1","volume-title":"Exploring and evaluating hallucinations in llm-powered code generation. arXiv preprint arXiv:2404.00971","author":"Liu Fang","year":"2024","unstructured":"Fang Liu, Yang Liu, Lin Shi, Houkun Huang, Ruifeng Wang, Zhen Yang, Li Zhang, Zhongqi Li, and Yuchi Ma. 2024. Exploring and evaluating hallucinations in llm-powered code generation. arXiv preprint arXiv:2404.00971 (2024)."},{"key":"e_1_3_2_1_12_1","volume-title":"Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664","author":"Lu Shuai","year":"2021","unstructured":"Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. 2021. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664 (2021)."},{"key":"e_1_3_2_1_13_1","volume-title":"ETF: An Entity Tracing Framework for Hallucination Detection in Code Summaries. arXiv preprint arXiv:2410.14748","author":"Maharaj Kishan","year":"2024","unstructured":"Kishan Maharaj, Vitobha Munigala, Srikanth G Tamilselvam, Prince Kumar, Sayandeep Sen, Palani Kodeswaran, Abhijit Mishra, and Pushpak Bhattacharyya. 2024. ETF: An Entity Tracing Framework for Hallucination Detection in Code Summaries. arXiv preprint arXiv:2410.14748 (2024)."},{"key":"e_1_3_2_1_14_1","volume-title":"Proceedings of the IEEE\/ACM 46th International Conference on Software Engineering. 1\u201313","author":"Mastropaolo Antonio","year":"2024","unstructured":"Antonio Mastropaolo, Matteo Ciniselli, Massimiliano Di Penta, and Gabriele Bavota. 2024. Evaluating Code Summarization Techniques: A New Metric and an Empirical Characterization. In Proceedings of the IEEE\/ACM 46th International Conference on Software Engineering. 1\u201313."},{"key":"e_1_3_2_1_15_1","volume-title":"Proceedings of the 40th annual meeting of the Association for Computational Linguistics. 311\u2013318","author":"Papineni Kishore","year":"2002","unstructured":"Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics. 311\u2013318."},{"key":"e_1_3_2_1_16_1","volume-title":"Proceedings of the tenth workshop on statistical machine translation. 392\u2013395","author":"Popovi\u0107 Maja","year":"2015","unstructured":"Maja Popovi\u0107. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the tenth workshop on statistical machine translation. 392\u2013395."},{"key":"e_1_3_2_1_17_1","volume-title":"Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084","author":"Reimers Nils","year":"2019","unstructured":"Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084 (2019)."},{"key":"e_1_3_2_1_18_1","volume-title":"Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1105\u20131116","author":"Roy Devjeet","year":"2021","unstructured":"Devjeet Roy, Sarah Fakhoury, and Venera Arnaoudova. 2021. Reassessing automatic evaluation metrics for code summarization tasks. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1105\u20131116."},{"key":"e_1_3_2_1_19_1","volume-title":"Evaluating Code Summarization with Improved Correlation with Human Assessment. In 2021 IEEE 21st International Conference on Software Quality, Reliability and Security (QRS). IEEE, 990\u20131001","author":"Shen Juanjuan","year":"2021","unstructured":"Juanjuan Shen, Yu Zhou, Yongchao Wang, Xiang Chen, Tingting Han, and Taolue Chen. 2021. Evaluating Code Summarization with Improved Correlation with Human Assessment. In 2021 IEEE 21st International Conference on Software Quality, Reliability and Security (QRS). IEEE, 990\u20131001."},{"key":"e_1_3_2_1_20_1","volume-title":"Michael Pradel","author":"Spiess Claudio","year":"2024","unstructured":"Claudio Spiess, David Gros, Kunal Suresh Pai, Michael Pradel, Md Rafiqul Islam Rabin, Amin Alipour, Susmit Jha, Prem Devanbu, and Toufique Ahmed. 2024. Calibration and correctness of language models for code. arXiv preprint arXiv:2402.02047 (2024)."},{"key":"e_1_3_2_1_21_1","volume-title":"We have a package for you! a comprehensive analysis of package hallucinations by code generating llms. arXiv preprint arXiv:2406.10279","author":"Spracklen Joseph","year":"2024","unstructured":"Joseph Spracklen, Raveen Wijewickrama, AHM Sakib, Anindya Maiti, Bimal Viswanath, and Murtuza Jadliwala. 2024. We have a package for you! a comprehensive analysis of package hallucinations by code generating llms. arXiv preprint arXiv:2406.10279 (2024)."},{"key":"e_1_3_2_1_22_1","volume-title":"CodeHalu: Code Hallucinations in LLMs Driven by Execution-based Verification. arXiv preprint arXiv:2405.00253","author":"Tian Yuchen","year":"2024","unstructured":"Yuchen Tian, Weixiang Yan, Qian Yang, Qian Chen, Wen Wang, Ziyang Luo, and Lei Ma. 2024. CodeHalu: Code Hallucinations in LLMs Driven by Execution-based Verification. arXiv preprint arXiv:2405.00253 (2024)."},{"key":"e_1_3_2_1_23_1","volume-title":"Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. arXiv preprint arXiv:2109.00859","author":"Wang Yue","year":"2021","unstructured":"Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. 2021. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. arXiv preprint arXiv:2109.00859 (2021)."},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"crossref","first-page":"471","DOI":"10.3390\/sym14030471","article-title":"A survey of automatic source code summarization","volume":"14","author":"Zhang Chunyan","year":"2022","unstructured":"Chunyan Zhang, Junchao Wang, Qinglei Zhou, Ting Xu, Ke Tang, Hairen Gui, and Fudong Liu. 2022. A survey of automatic source code summarization. Symmetry 14, 3 (2022), 471.","journal-title":"Symmetry"},{"key":"e_1_3_2_1_25_1","volume-title":"Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675","author":"Zhang Tianyi","year":"2019","unstructured":"Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 (2019)."},{"key":"e_1_3_2_1_26_1","first-page":"46595","article-title":"Judging llm-as-a-judge with mt-bench and chatbot arena","volume":"36","author":"Zheng Lianmin","year":"2023","unstructured":"Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems 36 (2023), 46595\u201346623.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_1_27_1","volume-title":"The 41st international ACM SIGIR conference on research & development in information retrieval. 1097\u20131100","author":"Zhu Yaoming","year":"2018","unstructured":"Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texygen: A benchmarking platform for text generation models. In The 41st international ACM SIGIR conference on research & development in information retrieval. 1097\u20131100."}],"event":{"sponsor":["SIGSOFT ACM Special Interest Group on Software Engineering"],"acronym":"FSE Companion '25","name":"FSE Companion '25: 33rd ACM International Conference on the Foundations of Software Engineering","location":"Clarion Hotel Trondheim Trondheim Norway"},"container-title":["Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3696630.3728520","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,28]],"date-time":"2025-07-28T19:13:11Z","timestamp":1753729991000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3696630.3728520"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,23]]},"references-count":27,"alternative-id":["10.1145\/3696630.3728520","10.1145\/3696630"],"URL":"https:\/\/doi.org\/10.1145\/3696630.3728520","relation":{},"subject":[],"published":{"date-parts":[[2025,6,23]]},"assertion":[{"value":"2025-07-28","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}