{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,12]],"date-time":"2026-03-12T15:47:44Z","timestamp":1773330464948,"version":"3.50.1"},"reference-count":57,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2025,1,18]],"date-time":"2025-01-18T00:00:00Z","timestamp":1737158400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["No. 62372005 and No. 62402482"],"award-info":[{"award-number":["No. 62372005 and No. 62402482"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2025,2,28]]},"abstract":"<jats:p>Commit messages are important for developers to understand the content and the reason for code changes. However, poor and even empty commit messages widely exist. To improve the quality of commit messages and development efficiency, many commit message generation methods have been proposed. Nevertheless, previous methods mainly focus on a brief generation problem, where both the input code change and the output commit messages are restricted to short. This may initiate a debate on the performance of these methods in practice. In this article, we attempt to remove the restrictions and move the needle forward to a holistic commit message generation problem. In particular, we conduct experiments to evaluate the performance of existing commit message generation methods in holistic commit message generation. In the experiments, we choose seven state-of-the-art commit generation methods and focus on two important scenarios in commit message generation (i.e., the within-project scenario and the cross-project scenario). To conduct our experiments, we publish a holistic commit message dataset HORDA with test data manually labeled. In our evaluations, we find that in generating holistic commit messages, the IR-based method has a better performance than non-pre-trained generation-based methods in the within-project scenario, contradicting previous research findings. Further, while the pre-trained generation-based methods are better than non-pre-trained generation-based methods, they are still constrained by the limitations of generation models.<\/jats:p>","DOI":"10.1145\/3695996","type":"journal-article","created":{"date-parts":[[2024,9,16]],"date-time":"2024-09-16T15:20:13Z","timestamp":1726500013000},"page":"1-28","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Is It Hard to Generate Holistic Commit Message?"],"prefix":"10.1145","volume":"34","author":[{"ORCID":"https:\/\/orcid.org\/0009-0000-5208-4750","authenticated-orcid":false,"given":"Guoqing","family":"Wang","sequence":"first","affiliation":[{"name":"Peking University, Beijing, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9990-9120","authenticated-orcid":false,"given":"Zeyu","family":"Sun","sequence":"additional","affiliation":[{"name":"Chinese Academy of Sciences, Beijing, China"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-0416-6896","authenticated-orcid":false,"given":"Jinhao","family":"Dong","sequence":"additional","affiliation":[{"name":"Peking University, Beijing, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6358-1568","authenticated-orcid":false,"given":"Yuxia","family":"Zhang","sequence":"additional","affiliation":[{"name":"Beijing Institute of Technology, Beijing, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8676-4017","authenticated-orcid":false,"given":"Mingxuan","family":"Zhu","sequence":"additional","affiliation":[{"name":"Peking University, Beijing, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1697-3451","authenticated-orcid":false,"given":"Qingyuan","family":"Liang","sequence":"additional","affiliation":[{"name":"Peking University, Beijing, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8295-303X","authenticated-orcid":false,"given":"Dan","family":"Hao","sequence":"additional","affiliation":[{"name":"Peking University, Beijing, China"}]}],"member":"320","published-online":{"date-parts":[[2025,1,18]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"GitHub. 2022. Retrieved from https:\/\/git-scm.com\/"},{"key":"e_1_3_2_3_2","unstructured":"GitHub. 2022. Retrieved from https:\/\/github.com\/"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","unstructured":"HORDA. 2023. Retrieved from 10.5281\/zenodo.10428056","DOI":"10.5281\/zenodo.10428056"},{"key":"e_1_3_2_5_2","first-page":"4","volume-title":"Proceedings of 5th International Conference on Natural Language Processing (ICON-2007)","author":"Ananthakrishnan R.","year":"2007","unstructured":"R. Ananthakrishnan, Pushpak Bhattacharyya, M. Sasikumar, and Ritesh M. Shah. 2007. Some issues in automatic evaluation of english-hindi mt: more blues for bleu. In Proceedings of 5th International Conference on Natural Language Processing (ICON-2007), 4\u20136."},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2002.1041053"},{"key":"e_1_3_2_7_2","first-page":"65","volume-title":"Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and\/or Summarization","author":"Banerjee Satanjeev","year":"2005","unstructured":"Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and\/or Summarization, 65\u201372."},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/1858996.1859005"},{"key":"e_1_3_2_9_2","first-page":"249","volume-title":"Proceedings of the 11th Conference of the European Chapter of the Association for Computational Linguistics","author":"Callison-Burch Chris","year":"2006","unstructured":"Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006. Re-evaluating the role of BLEU in machine translation research. In Proceedings of the 11th Conference of the European Chapter of the Association for Computational Linguistics, 249\u2013256."},{"key":"e_1_3_2_10_2","unstructured":"Kyunghyun Cho Bart Van Merri\u00ebnboer Caglar Gulcehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv:1406.1078. Retrieved from https:\/\/arxiv.org\/abs\/1406.1078"},{"key":"e_1_3_2_11_2","unstructured":"Junyoung Chung Caglar Gulcehre KyungHyun Cho and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv:1412.3555. Retrieved from https:\/\/arxiv.org\/abs\/1412.3555"},{"key":"e_1_3_2_12_2","first-page":"275","volume-title":"Proceedings of the IEEE 14th International Working Conference on Source Code Analysis and Manipulation","author":"Cort\u00e9s-Coy Luis Fernando","year":"2014","unstructured":"Luis Fernando Cort\u00e9s-Coy, Mario Linares-V\u00e1squez, Jairo Aponte, and Denys Poshyvanyk. 2014. On automatically generating commit messages via summarization of source code changes. In Proceedings of the IEEE 14th International Working Conference on Source Code Analysis and Manipulation. IEEE, 275\u2013284."},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510069"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.5555\/2486788.2486844"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE56229.2023.00078"},{"key":"e_1_3_2_16_2","first-page":"788","volume-title":"Proceedings of the IEEE 47th Annual Computers, Software, and Applications Conference (COMPSAC)","author":"Farag\u00f3 David","year":"2023","unstructured":"David Farag\u00f3, Michael F\u00e4rber, and Christian Petrov. 2023. A full-fledged commit message quality checker based on machine learning. In Proceedings of the IEEE 47th Annual Computers, Software, and Applications Conference (COMPSAC). IEEE, 788\u2013799."},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE56229.2023.00109"},{"key":"e_1_3_2_18_2","unstructured":"GitHub. 2023. GitHub Copilot X. Retrieved from https:\/\/github.com\/github\/copilot-x"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/IJCNN.2005.1555942"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.5555\/3294771.3294869"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597926.3598096"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380361"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1997.9.8.1735"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549119"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11390-020-0496-0"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.5555\/3155562.3155583"},{"key":"e_1_3_2_27_2","doi-asserted-by":"crossref","unstructured":"Tae-Hwan Jung. 2021. Commitbert: Commit message generation using pre-trained programming language model. arXiv:2105.14242. Retrieved from https:\/\/arxiv.org\/abs\/2105.14242","DOI":"10.18653\/v1\/2021.nlp4prog-1.3"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298932"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.21236\/ADA461156"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jcm.2016.02.012"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00076"},{"key":"e_1_3_2_32_2","doi-asserted-by":"crossref","unstructured":"Bo Lin Shangwen Wang Zhongxin Liu Yepang Liu Xin Xia and Xiaoguang Mao. 2023. CCT5: A code-change-oriented pre-trained model. arXiv:2305.10785. Retrieved from https:\/\/arxiv.org\/abs\/2305.10785","DOI":"10.1145\/3611643.3616339"},{"key":"e_1_3_2_33_2","volume-title":"Proceedings of the NTCIR Workshop","author":"Lin Chin-Yew","year":"2004","unstructured":"Chin-Yew Lin and F. J. Och. 2004. Looking for a few good metrics: ROUGE and its evaluation. In Proceedings of the NTCIR Workshop."},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.3115\/1218955.1219032"},{"key":"e_1_3_2_35_2","first-page":"299","volume-title":"Proceedings of the IEEE\/ACM 16th International Conference on Mining Software Repositories (MSR)","author":"Liu Qin","year":"2019","unstructured":"Qin Liu, Zihe Liu, Hongming Zhu, Hongfei Fan, Bowen Du, and Yu Qian. 2019. Generating commit messages from diffs using pointer-generator network. In Proceedings of the IEEE\/ACM 16th International Conference on Mining Software Repositories (MSR). IEEE, 299\u2013309."},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2020.3038681"},{"key":"e_1_3_2_37_2","first-page":"1","volume-title":"Proceedings of the 2023 IEEE\/ACM 45th International Conference on Software Engineering","author":"Liu Zhongxin","year":"2023","unstructured":"Zhongxin Liu, Zhijie Tang, Xin Xia, and Xiaohu Yang. 2023. CCRep: Learning code change representations via pre-trained code model and query back. In Proceedings of the 2023 IEEE\/ACM 45th International Conference on Software Engineering. 1\u201313."},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238190"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511809071"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/337180.337209"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSM.2000.883028"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2021.05.039"},{"key":"e_1_3_2_43_2","unstructured":"OpenAI. 2021. ChatGPT: A large-scale chatbot model. OpenAI Blog. Retrieved from https:\/\/www.openai.com\/blog\/chatgpt"},{"key":"e_1_3_2_44_2","first-page":"311","volume-title":"Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics","author":"Papineni Kishore","year":"2002","unstructured":"Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, 311\u2013318."},{"key":"e_1_3_2_45_2","first-page":"167","volume-title":"Proceedings of the 4th International Conference on Advances in Computing and Communications","author":"Roopak S.","year":"2014","unstructured":"S. Roopak and Tony Thomas. 2014. A novel phishing page detection mechanism using html source code comparison and cosine similarity. In Proceedings of the 4th International Conference on Advances in Computing and Communications. IEEE, 167\u2013170."},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1109\/78.650093"},{"key":"e_1_3_2_47_2","first-page":"5520","volume-title":"Proceedings of the Conference on Empirical Methods in Natural Language Processing","author":"Shi Ensheng","year":"2022","unstructured":"Ensheng Shi, Yanlin Wang, Wei Tao, Lun Du, Hongyu Zhang, Shi Han, Dongmei Zhang, and Hongbin Sun. 2022. RACE: Retrieval-augmented Commit Message Generation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. ACL, 5520\u20135530. Retrieved from https:\/\/aclanthology.org\/2022.emnlp-main.372"},{"key":"e_1_3_2_48_2","article-title":"Sequence to sequence learning with neural networks","author":"Sutskever Ilya","year":"2014","unstructured":"Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Proceedings of the 27th International Conference on Neural Information Processing Systems.","journal-title":"Proceedings of the 27th International Conference on Neural Information Processing Systems"},{"key":"e_1_3_2_49_2","first-page":"126","volume-title":"Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME)","author":"Tao Wei","year":"2021","unstructured":"Wei Tao, Yanlin Wang, Ensheng Shi, Lun Du, Shi Han, Hongyu Zhang, Dongmei Zhang, and Wenqiang Zhang. 2021. On the evaluation of commit message generation models: an experimental study. In Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 126\u2013136."},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510205"},{"key":"e_1_3_2_51_2","doi-asserted-by":"crossref","unstructured":"Rosalia Tufano Simone Masiero Antonio Mastropaolo Luca Pascarella Denys Poshyvanyk and Gabriele Bavota. 2022. Using pre-trained models to boost code review automation. arXiv:2201.06850. Retrieved from https:\/\/arxiv.org\/abs\/2201.06850","DOI":"10.1145\/3510003.3510621"},{"key":"e_1_3_2_52_2","article-title":"Pointer networks","author":"Vinyals Oriol","year":"2015","unstructured":"Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems.","journal-title":"Proceedings of the 28th International Conference on Neural Information Processing Systems"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298935"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1145\/3464689"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4612-4380-9_16"},{"key":"e_1_3_2_56_2","unstructured":"Yonghui Wu Mike Schuster Zhifeng Chen Quoc V. Le Mohammad Norouzi Wolfgang Macherey Maxim Krikun Yuan Cao Qin Gao Klaus Macherey Jeff Klingner Apurva Shah Melvin Johnson Xiaobing Liu \u0141ukasz Kaiser Stephan Gouws Yoshikiyo Kato Taku Kudo Hideto Kazawa Keith Stevens George Kurian Nishant Patil Wei Wang Cliff Young Jason Smith Jason Riesa Alex Rudnick Oriol Vinyals Greg Corrado Macduff Hughes and Jeffrey Dean. 2016. Google\u2019s neural machine translation system: Bridging the gap between human and machine translation. arXiv:1609.08144."},{"key":"e_1_3_2_57_2","volume-title":"Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI)","author":"Xu Shengbin","year":"2019","unstructured":"Shengbin Xu, Yuan Yao, Feng Xu, Tianxiao Gu, Hanghang Tong, and Jian Lu. 2019. Commit message generation for source code changes. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI)."},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2024.3364675"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3695996","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3695996","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:04:30Z","timestamp":1750291470000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3695996"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,1,18]]},"references-count":57,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2025,2,28]]}},"alternative-id":["10.1145\/3695996"],"URL":"https:\/\/doi.org\/10.1145\/3695996","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,1,18]]},"assertion":[{"value":"2024-02-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-08-10","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-01-18","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}