{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,1]],"date-time":"2026-07-01T04:26:38Z","timestamp":1782879998775,"version":"3.54.5"},"reference-count":69,"publisher":"Association for Computing Machinery (ACM)","issue":"FSE","license":[{"start":{"date-parts":[[2024,7,12]],"date-time":"2024-07-12T00:00:00Z","timestamp":1720742400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Starry Night Science Fund of Zhejiang University Shanghai Institute for Advanced Study","award":["No. SN-ZJU-SIAS-001"],"award-info":[{"award-number":["No. SN-ZJU-SIAS-001"]}]},{"DOI":"10.13039\/501100012166","name":"National Key Research and Development Program of China","doi-asserted-by":"publisher","award":["No. 2021YFB2701102"],"award-info":[{"award-number":["No. 2021YFB2701102"]}],"id":[{"id":"10.13039\/501100012166","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Shanghai Sailing Program","award":["23YF1446900"],"award-info":[{"award-number":["23YF1446900"]}]},{"DOI":"10.13039\/501100001809","name":"National Science Foundation of China","doi-asserted-by":"crossref","award":["No. 62202341, No.62372398, No.72342025, and U20A20173"],"award-info":[{"award-number":["No. 62202341, No.62372398, No.72342025, and U20A20173"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100012226","name":"Fundamental Research Funds for the Central Universities","doi-asserted-by":"publisher","award":["No. 226-2022-00064"],"award-info":[{"award-number":["No. 226-2022-00064"]}],"id":[{"id":"10.13039\/501100012226","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100007834","name":"Ningbo Natural Science Foundation","doi-asserted-by":"crossref","award":["No. 2023J292"],"award-info":[{"award-number":["No. 2023J292"]}],"id":[{"id":"10.13039\/100007834","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Softw. Eng."],"published-print":{"date-parts":[[2024,7,12]]},"abstract":"<jats:p>\n                    Nowadays, more and more developers resort to Stack Overflow for solutions (e.g., code snippets) when they encounter technical problems. Although domain experts provide huge amounts of valuable solutions in Stack Overflow, these code snippets are often difficult to reuse directly. Developers have to digest the information within relevant posts and make necessary modifications, and the whole solution-seeking process can be time-consuming and tedious. To facilitate the reuse of Stack Overflow code snippets, Terragni et al. first explored transforming a code snippet in Stack Overflow into a well-formed method API (\n                    <jats:bold>A<\/jats:bold>\n                    pplication\n                    <jats:bold>P<\/jats:bold>\n                    rogram\n                    <jats:bold>I<\/jats:bold>\n                    nterface) by using a rule-based approach, named APIzator. The reported performance of their approach is promising, however, after our in-depth analysis of their experiment results, we find that (1) 92.5% of APIs generated by APIzator are pointless and thus are difficult to use in practice. This is because the method name generated by APIzator (extracting\n                    <jats:monospace>verb + object<\/jats:monospace>\n                    ) can rarely represent the method\u2019s functionality, which can hardly be claimed as meaningful\/reusable APIs. (2) The authors manually summarized a number of rules to identify parameter variables and return statements for Java methods. These hand-crafted rules are extremely complex and sophisticated, and the manual rule design process is labor-intensive and error-prone. Moreover, since these rules are designed for Java, they can hardly be extended to other programming languages.\n                  <\/jats:p>\n                  <jats:p>\n                    Inspired by the great potential of Large Language Models (LLMs) for solving complex coding tasks, in this paper, we propose a novel approach, named\n                    <jats:sc>Code<\/jats:sc>\n                    2API, to automatically perform APIzation for Stack Overflow code snippets.\n                    <jats:sc>Code<\/jats:sc>\n                    2API does not require additional model training or any manual crafting rules and can be easily deployed on personal computers without relying on other external tools. Specifically,\n                    <jats:sc>Code<\/jats:sc>\n                    2API guides the LLMs through well-designed prompts to generate well-formed APIs for given code snippets. To elicit knowledge and logical reasoning from LLMs, we used\n                    <jats:bold>c<\/jats:bold>\n                    hain-\n                    <jats:bold>o<\/jats:bold>\n                    f-\n                    <jats:bold>t<\/jats:bold>\n                    hought (CoT) reasoning and few-shot in-context learning, which can help the LLMs fully understand the APIzation task and solve it step by step in a manner similar to a developer. Our evaluations show that\n                    <jats:sc>Code<\/jats:sc>\n                    2API achieves a remarkable accuracy in identifying method parameters (65%) and return statements (66%) equivalent to human-generated ones, surpassing the current state-of-the-art approach, APIzator, by 15.0% and 16.5% respectively. Moreover, compared with APIzator, our user study demonstrates that\n                    <jats:sc>Code<\/jats:sc>\n                    2API exhibits superior performance in generating meaningful method names, even surpassing the human-level performance, and developers are more willing to use APIs generated by our approach, highlighting the applicability of our tool in practice. Finally, we successfully extend our framework to the Python dataset, achieving a comparable performance with Java, which verifies the generalizability of our tool.\n                  <\/jats:p>","DOI":"10.1145\/3660811","type":"journal-article","created":{"date-parts":[[2024,7,12]],"date-time":"2024-07-12T10:22:09Z","timestamp":1720779729000},"page":"2355-2377","source":"Crossref","is-referenced-by-count":4,"title":["Are Human Rules Necessary? Generating Reusable APIs with CoT Reasoning and In-Context Learning"],"prefix":"10.1145","volume":"1","author":[{"ORCID":"https:\/\/orcid.org\/0009-0003-8200-1997","authenticated-orcid":false,"given":"Yubo","family":"Mai","sequence":"first","affiliation":[{"name":"Zhejiang University, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3030-9917","authenticated-orcid":false,"given":"Zhipeng","family":"Gao","sequence":"additional","affiliation":[{"name":"Shanghai Institute for Advanced Study of Zhejiang University, Shanghai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0093-3292","authenticated-orcid":false,"given":"Xing","family":"Hu","sequence":"additional","affiliation":[{"name":"Zhejiang University, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1846-0921","authenticated-orcid":false,"given":"Lingfeng","family":"Bao","sequence":"additional","affiliation":[{"name":"Zhejiang University, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-8163-689X","authenticated-orcid":false,"given":"Yu","family":"Liu","sequence":"additional","affiliation":[{"name":"Zhejiang University, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8799-6020","authenticated-orcid":false,"given":"JianLing","family":"Sun","sequence":"additional","affiliation":[{"name":"Zhejiang University, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,7,12]]},"reference":[{"key":"e_1_3_1_2_2","article-title":"Code generation tools (almost) for free? a study of few-shot, pre-trained language models on code","author":"Barei\u00df Patrick","year":"2022","unstructured":"Patrick Barei\u00df, Beatriz Souza, Marcelo d\u2019Amorim, and Michael Pradel. 2022. Code generation tools (almost) for free? a study of few-shot, pre-trained language models on code. arXiv preprint arXiv:2206.01335 (2022).","journal-title":"arXiv preprint arXiv:2206.01335"},{"key":"e_1_3_1_3_2","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"Brown Tom","year":"2020","unstructured":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877\u20131901.","journal-title":"Advances in neural information processing systems"},{"key":"e_1_3_1_4_2","first-page":"1273","article-title":"Automated query reformulation for efficient search based on query logs from stack overflow","author":"Cao Kaibo","year":"2021","unstructured":"Kaibo Cao, Chunyang Chen, Sebastian Baltes, Christoph Treude, and Xiang Chen. 2021. Automated query reformulation for efficient search based on query logs from stack overflow. In 2021 IEEE\/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 1273\u20131285.","journal-title":"2021 IEEE\/ACM 43rd International Conference on Software Engineering (ICSE)"},{"key":"e_1_3_1_5_2","article-title":"Evaluating large language models trained on code","author":"Chen Mark","year":"2021","unstructured":"Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021).","journal-title":"arXiv preprint arXiv:2107.03374"},{"key":"e_1_3_1_6_2","article-title":"Palm: Scaling language modeling with pathways","author":"Chowdhery Aakanksha","year":"2022","unstructured":"Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 (2022).","journal-title":"arXiv preprint arXiv:2204.02311"},{"key":"e_1_3_1_7_2","article-title":"PyMT5: multi-mode translation of natural language and Python code with transformers","author":"Clement Colin B","year":"2020","unstructured":"Colin B Clement, Dawn Drain, Jonathan Timcheck, Alexey Svyatkovskiy, and Neel Sundaresan. 2020. PyMT5: multi-mode translation of natural language and Python code with transformers. arXiv preprint arXiv:2010.03150 (2020).","journal-title":"arXiv preprint arXiv:2010.03150"},{"key":"e_1_3_1_8_2","unstructured":"Code2API. 2023. Our chrome extension tutorial. https:\/\/youtu.be\/AHOPEWDEkCE"},{"key":"e_1_3_1_9_2","doi-asserted-by":"publisher","unstructured":"Code2API. 2023. Our replicate package. https:\/\/doi.org\/10.6084\/m9.figshare.24219856.v1 10.6084\/m9.figshare.24219856.v1","DOI":"10.6084\/m9.figshare.24219856.v1"},{"key":"e_1_3_1_10_2","doi-asserted-by":"publisher","DOI":"10.1037\/h0026256"},{"key":"e_1_3_1_11_2","article-title":"Bert: Pre-training of deep bidirectional transformers for language understanding","author":"Devlin Jacob","year":"2018","unstructured":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).","journal-title":"arXiv preprint arXiv:1810.04805"},{"key":"e_1_3_1_12_2","article-title":"A survey for in-context learning","author":"Dong Qingxiu","year":"2022","unstructured":"Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey for in-context learning. arXiv preprint arXiv:2301.00234 (2022).","journal-title":"arXiv preprint arXiv:2301.00234"},{"key":"e_1_3_1_13_2","article-title":"Self-collaboration Code Generation via ChatGPT","author":"Dong Yihong","year":"2023","unstructured":"Yihong Dong, Xue Jiang, Zhi Jin, and Ge Li. 2023. Self-collaboration Code Generation via ChatGPT. arXiv preprint arXiv:2304.07590 (2023).","journal-title":"arXiv preprint arXiv:2304.07590"},{"key":"e_1_3_1_14_2","article-title":"Prompting Is All Your Need: Automated Android Bug Replay with Large Language Models","author":"Feng Sidong","year":"2023","unstructured":"Sidong Feng and Chunyang Chen. 2023. Prompting Is All Your Need: Automated Android Bug Replay with Large Language Models. arXiv preprint arXiv:2306.01987 (2023).","journal-title":"arXiv preprint arXiv:2306.01987"},{"key":"e_1_3_1_15_2","article-title":"Codebert: A pre-trained model for programming and natural languages","author":"Feng Zhangyin","year":"2020","unstructured":"Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. 2020. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155 (2020).","journal-title":"arXiv preprint arXiv:2002.08155"},{"key":"e_1_3_1_16_2","article-title":"Incoder: A generative model for code infilling and synthesis","author":"Fried Daniel","year":"2022","unstructured":"Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. 2022. Incoder: A generative model for code infilling and synthesis. arXiv preprint arXiv:2204.05999 (2022).","journal-title":"arXiv preprint arXiv:2204.05999"},{"key":"e_1_3_1_17_2","article-title":"Making pre-trained language models better few-shot learners","author":"Gao Tianyu","year":"2020","unstructured":"Tianyu Gao, Adam Fisch, and Danqi Chen. 2020. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.15723 (2020).","journal-title":"arXiv preprint arXiv:2012.15723"},{"key":"e_1_3_1_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3401026"},{"key":"e_1_3_1_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/3412845"},{"key":"e_1_3_1_20_2","first-page":"1525","article-title":"Code2que: A tool for improving question titles from mined code snippets in stack overflow","author":"Gao Zhipeng","year":"2021","unstructured":"Zhipeng Gao, Xin Xia, David Lo, John Grundy, and Yuan-Fang Li. 2021. Code2que: A tool for improving question titles from mined code snippets in stack overflow. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations ofSoftware Engineering. 1525\u20131529.","journal-title":"Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations ofSoftware Engineering"},{"key":"e_1_3_1_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/3550150"},{"key":"e_1_3_1_22_2","first-page":"631","article-title":"Deep API learning","author":"Gu Xiaodong","year":"2016","unstructured":"Xiaodong Gu, Hongyu Zhang, Dongmei Zhang, and Sunghun Kim. 2016. Deep API learning. In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations ofSoftware Engineering. 631\u2013642.","journal-title":"Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations ofSoftware Engineering"},{"key":"e_1_3_1_23_2","article-title":"How good are gpt models at machine translation? a comprehensive evaluation","author":"Hendy Amr","year":"2023","unstructured":"Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. How good are gpt models at machine translation? a comprehensive evaluation. arXiv preprint arXiv:2302.09210 (2023).","journal-title":"arXiv preprint arXiv:2302.09210"},{"key":"e_1_3_1_24_2","doi-asserted-by":"crossref","first-page":"293","DOI":"10.1145\/3238147.3238191","article-title":"API method recommendation without worrying about the task-API knowledge gap","author":"Huang Qiao","year":"2018","unstructured":"Qiao Huang, Xin Xia, Zhenchang Xing, David Lo, and Xinyu Wang. 2018. API method recommendation without worrying about the task-API knowledge gap. In Proceedings of the 33rd ACM\/IEEE International Conference on Automated Software Engineering. 293\u2013304.","journal-title":"Proceedings of the 33rd ACM\/IEEE International Conference on Automated Software Engineering"},{"key":"e_1_3_1_25_2","article-title":"A Chain of AI-based Solutions for Resolving FQNs and Fixing Syntax Errors in Partial Code","author":"Huang Qing","year":"2023","unstructured":"Qing Huang, Jiahui Zhu, Zhenchang Xing, Huan Jin, Changjing Wang, and Xiwei Xu. 2023. A Chain of AI-based Solutions for Resolving FQNs and Fixing Syntax Errors in Partial Code. arXiv preprint arXiv:2306.11981 (2023).","journal-title":"arXiv preprint arXiv:2306.11981"},{"key":"e_1_3_1_26_2","article-title":"SelfEvolve: A Code Evolution Framework via Large Language Models","author":"Jiang Shuyang","year":"2023","unstructured":"Shuyang Jiang, Yuhao Wang, and Yu Wang. 2023. SelfEvolve: A Code Evolution Framework via Large Language Models. arXiv preprint arXiv:2306.02907 (2023).","journal-title":"arXiv preprint arXiv:2306.02907"},{"key":"e_1_3_1_27_2","first-page":"22199","article-title":"Large language models are zero-shot reasoners","volume":"35","author":"Kojima Takeshi","year":"2022","unstructured":"Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems 35 (2022), 22199\u201322213.","journal-title":"Advances in neural information processing systems"},{"key":"e_1_3_1_28_2","doi-asserted-by":"publisher","DOI":"10.2307\/2529310"},{"key":"e_1_3_1_29_2","article-title":"CodeIE: Large Code Generation Models are Better Few-Shot Information Extractors","author":"Li Peng","year":"2023","unstructured":"Peng Li, Tianxiang Sun, Qiong Tang, Hang Yan, Yuanbin Wu, Xuanjing Huang, and Xipeng Qiu. 2023. CodeIE: Large Code Generation Models are Better Few-Shot Information Extractors. arXiv preprint arXiv:2305.05711 (2023).","journal-title":"arXiv preprint arXiv:2305.05711"},{"key":"e_1_3_1_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/3560815"},{"key":"e_1_3_1_31_2","article-title":"Is prompt all you need? no. A comprehensive and broader view of instruction learning","author":"Lou Renze","year":"2023","unstructured":"Renze Lou, Kai Zhang, and Wenpeng Yin. 2023. Is prompt all you need? no. A comprehensive and broader view of instruction learning. arXiv preprint arXiv:2303.10475 (2023).","journal-title":"arXiv preprint arXiv:2303.10475"},{"key":"e_1_3_1_32_2","article-title":"Codexglue: A machine learning benchmark dataset for code understanding and generation","author":"Lu Shuai","year":"2021","unstructured":"Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. 2021. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664 (2021).","journal-title":"arXiv preprint arXiv:2102.04664"},{"key":"e_1_3_1_33_2","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1109\/ICSM.2012.6405249","article-title":"What makes a good code example?: A study of programming Q&A in StackOverflow","author":"Nasehi Seyed Mehdi","year":"2012","unstructured":"Seyed Mehdi Nasehi, Jonathan Sillito, Frank Maurer, and Chris Burns. 2012. What makes a good code example?: A study of programming Q&A in StackOverflow. In 2012 28th IEEE International Conference on Software Maintenance (ICSM). IEEE, 25\u201334.","journal-title":"2012 28th IEEE International Conference on Software Maintenance (ICSM)"},{"key":"e_1_3_1_34_2","doi-asserted-by":"crossref","first-page":"1050","DOI":"10.1109\/ICSE.2019.00109","volume-title":"2019 IEEE\/ACM 41st International Conference on Software Engineering (ICSE)","author":"Nguyen Phuong T","year":"2019","unstructured":"Phuong T Nguyen, Juri Di Rocco, Davide Di Ruscio, Lina Ochoa, Thomas Degueule, and Massimiliano Di Penta. 2019. Focus: A recommender system for mining api function calls and usage patterns. In 2019 IEEE\/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 1050\u20131060."},{"key":"e_1_3_1_35_2","unstructured":"OpenAI. 2022. Introducing ChatGPT. OpenAI Blog (November 2022)."},{"key":"e_1_3_1_36_2","first-page":"27730","article-title":"Training language models to follow instructions with human feedback","volume":"35","author":"Ouyang Long","year":"2022","unstructured":"Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advancesin NeuralInformation Processing Systems 35 (2022), 27730\u201327744.","journal-title":"Advancesin NeuralInformation Processing Systems"},{"key":"e_1_3_1_37_2","article-title":"Gorilla: Large language model connected with massive apis","author":"Patil Shishir G","year":"2023","unstructured":"Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. 2023. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15334 (2023).","journal-title":"arXiv preprint arXiv:2305.15334"},{"key":"e_1_3_1_38_2","unstructured":"Rishov Paul Md Mohib Hossain Mohammed Latif Siddiq Masum Hasan Anindya Iqbal and Joanna CS Santos. [n. d.]. Enhancing Automated Program Repair through Fine-tuning and Prompt Engineering. ([n. d.])."},{"key":"e_1_3_1_39_2","doi-asserted-by":"crossref","first-page":"102","DOI":"10.1145\/2597073.2597077","article-title":"Mining stackoverflow to turn the ide into a self-confident programming prompter","author":"Ponzanelli Luca","year":"2014","unstructured":"Luca Ponzanelli, Gabriele Bavota, Massimiliano Di Penta, Rocco Oliveto, and Michele Lanza. 2014. Mining stackoverflow to turn the ide into a self-confident programming prompter. In Proceedings of the 11th working conference on mining software repositories. 102\u2013111.","journal-title":"Proceedings of the 11th working conference on mining software repositories"},{"issue":"8","key":"e_1_3_1_40_2","first-page":"9","article-title":"Language models are unsupervised multitask learners","volume":"1","author":"Radford Alec","year":"2019","unstructured":"Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9.","journal-title":"OpenAI blog"},{"key":"e_1_3_1_41_2","article-title":"Code llama: Open foundation models for code","author":"Roziere Baptiste","year":"2023","unstructured":"Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jeremy Rapin, et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 (2023).","journal-title":"arXiv preprint arXiv:2308.12950"},{"key":"e_1_3_1_42_2","article-title":"PostFinder: Mining Stack Overflow posts to support software developers","volume":"127","author":"Rubei Riccardo","year":"2020","unstructured":"Riccardo Rubei, Claudio Di Sipio, Phuong T Nguyen, Juri Di Rocco, and Davide Di Ruscio. 2020. PostFinder: Mining Stack Overflow posts to support software developers. Information and Software Technology 127 (2020), 106367.","journal-title":"Information and Software Technology"},{"key":"e_1_3_1_43_2","article-title":"Statistics: 2.3 The Mann-Whitney U Test","volume":"15","author":"Shier Rosie","year":"2004","unstructured":"Rosie Shier. 2004. Statistics: 2.3 The Mann-Whitney U Test. Mathematics Learning Support Centre. Last accessed 15 (2004), 2013.","journal-title":"Mathematics Learning Support Centre. Last accessed"},{"key":"e_1_3_1_44_2","article-title":"Exploring the Effectiveness of Large Language Models in Generating Unit Tests","author":"Siddiq Mohammed Latif","year":"2023","unstructured":"Mohammed Latif Siddiq, Joanna Santos, Ridwanul Hasan Tanvir, Noshin Ulfat, Fahmid Al Rifat, and Vinicius Carvalho Lopes. 2023. Exploring the Effectiveness of Large Language Models in Generating Unit Tests. arXivpreprint arXiv:2305.00418 (2023).","journal-title":"arXivpreprint arXiv:2305.00418"},{"key":"e_1_3_1_45_2","first-page":"316","volume-title":"2023 IEEE\/ACM 31st International Conference on Program Comprehension (ICPC)","author":"Su Yanqi","year":"2023","unstructured":"Yanqi Su, Zheming Han, Zhipeng Gao, Zhenchang Xing, Qinghua Lu, and Xiwei Xu. 2023. Still confusing for bugcomponent triaging? deep feature learning and ensemble setting to rescue. In 2023 IEEE\/ACM 31st International Conference on Program Comprehension (ICPC). IEEE, 316\u2013327."},{"key":"e_1_3_1_46_2","doi-asserted-by":"crossref","first-page":"85","DOI":"10.1109\/MSR.2013.6624012","volume-title":"2013 10th Working Conference on Mining Software Repositories (MSR)","author":"Subramanian Siddharth","year":"2013","unstructured":"Siddharth Subramanian and Reid Holmes. 2013. Making sense of online code snippets. In 2013 10th Working Conference on Mining Software Repositories (MSR). IEEE, 85\u201388."},{"key":"e_1_3_1_47_2","doi-asserted-by":"crossref","first-page":"643","DOI":"10.1145\/2568225.2568313","article-title":"Live API documentation","author":"Subramanian Siddharth","year":"2014","unstructured":"Siddharth Subramanian, Laura Inozemtseva, and Reid Holmes. 2014. Live API documentation. In Proceedings of the 36th international conference on software engineering. 643\u2013652.","journal-title":"Proceedings of the 36th international conference on software engineering"},{"key":"e_1_3_1_48_2","article-title":"Copilot for Xcode: Exploring AI-Assisted Programming by Prompting Cloud-based Large Language Models","author":"Tan Chee Wei","year":"2023","unstructured":"Chee Wei Tan, Shangxin Guo, Man Fai Wong, and Ching Nam Hang. 2023. Copilot for Xcode: Exploring AI-Assisted Programming by Prompting Cloud-based Large Language Models. arXiv preprint arXiv:2307.14349 (2023).","journal-title":"arXiv preprint arXiv:2307.14349"},{"key":"e_1_3_1_49_2","doi-asserted-by":"crossref","first-page":"145","DOI":"10.1145\/1294261.1294276","article-title":"\/* iComment: Bugs or bad comments?*","author":"Tan Lin","year":"2007","unstructured":"Lin Tan, Ding Yuan, Gopal Krishna, and Yuanyuan Zhou. 2007. \/* iComment: Bugs or bad comments?*. In Proceedings oftwenty-first ACM SIGOPS symposium on Operating systems principles. 145\u2013158.","journal-title":"Proceedings oftwenty-first ACM SIGOPS symposium on Operating systems principles"},{"key":"e_1_3_1_50_2","doi-asserted-by":"publisher","DOI":"10.1145\/1985793.1985796"},{"key":"e_1_3_1_51_2","first-page":"260","volume-title":"2012 IEEE Fifth International Conference on Software Testing, Verification and Validation","author":"Tan Shin Hwei","year":"2012","unstructured":"Shin Hwei Tan, Darko Marinov, Lin Tan, and Gary T Leavens. 2012. @ tcomment: Testing javadoc comments to detect comment-code inconsistencies. In 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation. IEEE, 260\u2013269."},{"key":"e_1_3_1_52_2","doi-asserted-by":"crossref","first-page":"118","DOI":"10.1145\/2931037.2931058","article-title":"CSNIPPEX: automated synthesis of compilable code snippets from Q&A sites","author":"Terragni Valerio","year":"2016","unstructured":"Valerio Terragni, Yepang Liu, and Shing-Chi Cheung. 2016. CSNIPPEX: automated synthesis of compilable code snippets from Q&A sites. In Proceedings of the 25th international symposium on software testing and analysis. 118\u2013129.","journal-title":"Proceedings of the 25th international symposium on software testing and analysis"},{"key":"e_1_3_1_53_2","doi-asserted-by":"crossref","first-page":"542","DOI":"10.1109\/ASE51524.2021.9678576","volume-title":"2021 36th IEEE\/ACM International Conference on Automated Software Engineering (ASE)","author":"Terragni Valerio","year":"2021","unstructured":"Valerio Terragni and Pasquale Salza. 2021. APIzation: Generating reusable APIs from StackOverflow code snippets. In 2021 36th IEEE\/ACM International Conference on Automated Software Engineering (ASE). IEEE, 542\u2013554."},{"key":"e_1_3_1_54_2","doi-asserted-by":"crossref","first-page":"481","DOI":"10.1145\/3564625.3567985","article-title":"Transformer-based language models for software vulnerability detection","author":"Thapa Chandra","year":"2022","unstructured":"Chandra Thapa, Seung Ick Jang, Muhammad Ejaz Ahmed, Seyit Camtepe, Josef Pieprzyk, and Surya Nepal. 2022. Transformer-based language models for software vulnerability detection. In Proceedings of the 38th Annual Computer Security Applications Conference. 481\u2013496.","journal-title":"Proceedings of the 38th Annual Computer Security Applications Conference"},{"key":"e_1_3_1_55_2","article-title":"Llama: Open and efficient foundation language models","author":"Touvron Hugo","year":"2023","unstructured":"Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth\u00e9e Lacroix, Baptiste Rozi\u00e8re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023).","journal-title":"arXiv preprint arXiv:2302.13971"},{"key":"e_1_3_1_56_2","article-title":"Software Testing with Large Language Model: Survey, Landscape, and Vision","author":"Wang Junjie","year":"2023","unstructured":"Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. 2023. Software Testing with Large Language Model: Survey, Landscape, and Vision. arXiv preprint arXiv:2307.07221 (2023).","journal-title":"arXiv preprint arXiv:2307.07221"},{"key":"e_1_3_1_57_2","article-title":"Measuring and Mitigating Constraint Violations of In-Context Learning for Utterance-to-API Semantic Parsing","author":"Wang Shufan","year":"2023","unstructured":"Shufan Wang, Sebastien Jean, Sailik Sengupta, James Gung, Nikolaos Pappas, and Yi Zhang. 2023. Measuring and Mitigating Constraint Violations of In-Context Learning for Utterance-to-API Semantic Parsing. arXiv preprint arXiv:2305.15338 (2023).","journal-title":"arXiv preprint arXiv:2305.15338"},{"key":"e_1_3_1_58_2","article-title":"Self-instruct: Aligning language model with self generated instructions","author":"Wang Yizhong","year":"2022","unstructured":"Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560 (2022).","journal-title":"arXiv preprint arXiv:2212.10560"},{"key":"e_1_3_1_59_2","article-title":"Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation","author":"Wang Yue","year":"2021","unstructured":"Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. 2021. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. arXiv preprint arXiv:2109.00859 (2021).","journal-title":"arXiv preprint arXiv:2109.00859"},{"key":"e_1_3_1_60_2","first-page":"24824","article-title":"Chain-of-thought prompting elicits reasoning in large language models","volume":"35","author":"Wei Jason","year":"2022","unstructured":"Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems 35 (2022), 24824\u201324837.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_61_2","first-page":"376","article-title":"Clear: contrastive learning for api recommendation","author":"Wei Moshi","year":"2022","unstructured":"Moshi Wei, Nima Shiri Harzevili, Yuchao Huang, Junjie Wang, and Song Wang. 2022. Clear: contrastive learning for api recommendation. In Proceedings of the 44th International Conference on Software Engineering. 376\u2013387.","journal-title":"Proceedings of the 44th International Conference on Software Engineering"},{"key":"e_1_3_1_62_2","article-title":"ChatUniTest: a ChatGPT-based automated unit test generation tool","author":"Xie Zhuokui","year":"2023","unstructured":"Zhuokui Xie, Yinghao Chen, Chen Zhi, Shuiguang Deng, and Jianwei Yin. 2023. ChatUniTest: a ChatGPT-based automated unit test generation tool. arXiv preprint arXiv:2305.04764 (2023).","journal-title":"arXiv preprint arXiv:2305.04764"},{"key":"e_1_3_1_63_2","first-page":"1876","volume-title":"2023 38th IEEE\/ACM International Conference on Automated Software Engineering (ASE)","author":"Xue Zhipeng","year":"2023","unstructured":"Zhipeng Xue, Zhipeng Gao, Xing Hu, and Shanping Li. 2023. ACWRecommender: A Tool for Validating Actionable Warnings with Weak Supervision. In 2023 38th IEEE\/ACM International Conference on Automated Software Engineering (ASE). IEEE, 1876\u20131880."},{"key":"e_1_3_1_64_2","first-page":"1887","volume-title":"2023 38th IEEE\/ACM International Conference on Automated Software Engineering (ASE)","author":"Yan Dapeng","year":"2023","unstructured":"Dapeng Yan, Zhipeng Gao, and Zhiming Liu. 2023. A Closer Look at Different Difficulty Levels Code Generation Abilities of ChatGPT. In 2023 38th IEEE\/ACM International Conference on Automated Software Engineering (ASE). IEEE, 1887\u20131898."},{"key":"e_1_3_1_65_2","doi-asserted-by":"crossref","first-page":"391","DOI":"10.1145\/2901739.2901767","article-title":"From query to usable code: an analysis of stack overflow code snippets","author":"Yang Di","year":"2016","unstructured":"Di Yang, Aftab Hussain, and Cristina Videira Lopes. 2016. From query to usable code: an analysis of stack overflow code snippets. In Proceedings of the 13th International Conference on Mining Software Repositories. 391\u2013402.","journal-title":"Proceedings of the 13th International Conference on Mining Software Repositories"},{"key":"e_1_3_1_66_2","article-title":"Exploring the limits of chatgpt for query or aspect-based text summarization","author":"Yang Xianjun","year":"2023","unstructured":"Xianjun Yang, Yan Li, Xinlu Zhang, Haifeng Chen, and Wei Cheng. 2023. Exploring the limits of chatgpt for query or aspect-based text summarization. arXiv preprint arXiv:2302.08081 (2023).","journal-title":"arXiv preprint arXiv:2302.08081"},{"key":"e_1_3_1_67_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2019.2906315"},{"key":"e_1_3_1_68_2","article-title":"ToolCoder: Teach Code Generation Models to use APIs with search tools","author":"Zhang Kechi","year":"2023","unstructured":"Kechi Zhang, Ge Li, Jia Li, Zhuo Li, and Zhi Jin. 2023. ToolCoder: Teach Code Generation Models to use APIs with search tools. arXiv preprint arXiv:2305.04032 (2023).","journal-title":"arXiv preprint arXiv:2305.04032"},{"key":"e_1_3_1_69_2","article-title":"A survey of large language models","author":"Zhao Wayne Xin","year":"2023","unstructured":"Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223 (2023).","journal-title":"arXiv preprint arXiv:2303.18223"},{"key":"e_1_3_1_70_2","article-title":"Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x","author":"Zheng Qinkai","year":"2023","unstructured":"Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, et al. 2023. Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x. arXiv preprint arXiv:2303.17568 (2023).","journal-title":"arXiv preprint arXiv:2303.17568"}],"container-title":["Proceedings of the ACM on Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3660811","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3660811","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,2,4]],"date-time":"2026-02-04T08:02:53Z","timestamp":1770192173000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3660811"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,12]]},"references-count":69,"journal-issue":{"issue":"FSE","published-print":{"date-parts":[[2024,7,12]]}},"alternative-id":["10.1145\/3660811"],"URL":"https:\/\/doi.org\/10.1145\/3660811","relation":{},"ISSN":["2994-970X"],"issn-type":[{"value":"2994-970X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,7,12]]}}}