{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,17]],"date-time":"2026-03-17T10:53:46Z","timestamp":1773744826482,"version":"3.50.1"},"reference-count":76,"publisher":"Association for Computing Machinery (ACM)","issue":"8","license":[{"start":{"date-parts":[[2024,11,30]],"date-time":"2024-11-30T00:00:00Z","timestamp":1732924800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Nature Science Foundation of China","doi-asserted-by":"crossref","award":["62262031"],"award-info":[{"award-number":["62262031"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100012456","name":"National Social Science Foundation","doi-asserted-by":"crossref","award":["20 & ZD068"],"award-info":[{"award-number":["20 & ZD068"]}],"id":[{"id":"10.13039\/501100012456","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100004479","name":"Natural Science Foundation of Jiangxi Province","doi-asserted-by":"crossref","award":["20224ACB202007"],"award-info":[{"award-number":["20224ACB202007"]}],"id":[{"id":"10.13039\/501100004479","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100009102","name":"Jiangxi Provincial Department of Education","doi-asserted-by":"crossref","award":["GJJ2200303, GJJ2200304"],"award-info":[{"award-number":["GJJ2200303, GJJ2200304"]}],"id":[{"id":"10.13039\/501100009102","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2024,11,30]]},"abstract":"<jats:p>APIs have intricate relations that can be described in text and represented as knowledge graphs to aid software engineering tasks. Existing relation extraction methods have limitations, such as limited API text corpus, and are affected by the characteristics of the input text. To address these limitations, we propose utilizing large language models (LLMs) (e.g., GPT-3.5) as a neural knowledge base for API relation inference. This approach leverages the entire Web used to pre-train LLMs as a knowledge base and is insensitive to the context and complexity of input texts. To ensure accurate inference, we design an AI chain consisting of three AI modules: API Fully Qualified Name (FQN) Parser, API Knowledge Extractor, and API Relation Decider. The accuracy of the API FQN Parser and API Relation Decider is 0.81 and 0.83, respectively. Using the generative capacity of the LLM and our approach\u2019s inference capability, we achieve an average F1 value of 0.76 under the three datasets, significantly higher than the state-of-the-art method\u2019s average F1 value of 0.40. Compared to the original CoT and modularized CoT methods, our AI chain design has improved the performance of API relation inference by 71% and 49%, respectively. Meanwhile, the prompt ensembling strategy enhances the performance of our approach by 32%. The API relations inferred by our method can be further organized into structured forms to provide support for other software engineering tasks.<\/jats:p>","DOI":"10.1145\/3680469","type":"journal-article","created":{"date-parts":[[2024,7,23]],"date-time":"2024-07-23T15:46:18Z","timestamp":1721749578000},"page":"1-34","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":8,"title":["Let\u2019s Discover More API Relations: A Large Language Model-Based AI Chain for Unsupervised API Relation Inference"],"prefix":"10.1145","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8877-4267","authenticated-orcid":false,"given":"Qing","family":"Huang","sequence":"first","affiliation":[{"name":"Jiangxi Normal University, School of Computer Information Engineering, Nanchang, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-6047-1554","authenticated-orcid":false,"given":"Yanbang","family":"Sun","sequence":"additional","affiliation":[{"name":"Jiangxi Normal University, School of Computer Information Engineering, Nanchang, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7663-1421","authenticated-orcid":false,"given":"Zhenchang","family":"Xing","sequence":"additional","affiliation":[{"name":"CSIRO\u2019s Data61, Canberra, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6557-6559","authenticated-orcid":false,"given":"Yuanlong","family":"Cao","sequence":"additional","affiliation":[{"name":"Jiangxi Normal University, School of Computer Information Engineering, Nanchang, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2700-7478","authenticated-orcid":false,"given":"Jieshan","family":"Chen","sequence":"additional","affiliation":[{"name":"CSIRO\u2019s Data61, Canberra, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2540-973X","authenticated-orcid":false,"given":"Xiwei","family":"Xu","sequence":"additional","affiliation":[{"name":"CSIRO\u2019s Data61, Sydney, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-6129-1654","authenticated-orcid":false,"given":"Huan","family":"Jin","sequence":"additional","affiliation":[{"name":"Jiangxi University of Technology, Nanchang, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-8718-1716","authenticated-orcid":false,"given":"Jiaxing","family":"Lu","sequence":"additional","affiliation":[{"name":"Jiangxi Normal University, School of Computer Information Engineering, Nanchang, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,12,3]]},"reference":[{"issue":"3","key":"e_1_3_2_2_2","first-page":"1540","article-title":"1+1>2: Programming know-what and know-how knowledge fusion, semantic enrichment and coherent application","volume":"16","author":"Huang Qing","year":"2023","unstructured":"Qing Huang, Zhiqiang Yuan, Zhenchang Xing, Zhengkang Zuo, Changjing Wang, and Xin Xia. 2023. 1+1>2: Programming know-what and know-how knowledge fusion, semantic enrichment and coherent application. IEEE Transactions on Services Computing 16, 3 (2023), 1540\u20131554.","journal-title":"IEEE Transactions on Services Computing"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2023.3252259"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2023.3346954"},{"key":"e_1_3_2_5_2","first-page":"293","volume-title":"Proceedings of the 33rd IEEE\/ACM International Conference on Automated Software Engineering (ASE)","author":"Huang Qiao","year":"2018","unstructured":"Qiao Huang, Xin Xia, Zhenchang Xing, D. Lo, and Xinyu Wang. 2018. API method recommendation without worrying about the task-api knowledge gap. Proceedings of the 33rd IEEE\/ACM International Conference on Automated Software Engineering (ASE), 293\u2013304."},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2022.3197063"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/3611643.3616305"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE56229.2023.00143"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE56229.2023.00159"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00179"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-021-09968-2"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/3630009"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416551"},{"key":"e_1_3_2_14_2","first-page":"183","volume-title":"Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME)","author":"Li Hongwei","year":"2018","unstructured":"Hongwei Li, Sirui Li, Jiamou Sun, Zhenchang Xing, Xin Peng, Mingwei Liu, and Xuejiao Zhao. 2018. Improving api caveats accessibility by mining api caveats knowledge graph. In Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME), 183\u2013193."},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3473112"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11432-019-2745-5"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00032"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416628"},{"key":"e_1_3_2_19_2","first-page":"448","volume-title":"Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME)","author":"Sun Jiamou","year":"2020","unstructured":"Jiamou Sun, Zhenchang Xing, Xin Peng, Xiwei Xu, and Liming Zhu. 2020. Task-oriented api usage examples prompting powered by programming task knowledge graph. In Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME), 448\u2013459."},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238208"},{"issue":"2","key":"e_1_3_2_21_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3617174","article-title":"Fqn inference in partial code by prompt-tuned language model of code","volume":"33","author":"Huang Qing","year":"2023","unstructured":"Qing Huang, Zhiqiang Yuan, Zhenchang Xing, Xin Peng, Xiwei Xu, and Qinghua Lu. 2023. Fqn inference in partial code by prompt-tuned language model of code. ACM Transactions on Software Engineering and Methodology 33, 2 (2023), 1\u201332.","journal-title":"ACM Transactions on Software Engineering and Methodology"},{"key":"e_1_3_2_22_2","unstructured":"OpenAI. 2023. OpenAi GPT-3.5 Model. Retrieved February 2023 from https:\/\/platform.openai.com\/docs\/models\/gpt-3-5"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.acl-short.24"},{"key":"e_1_3_2_24_2","unstructured":"Rishi Bommasani Drew A Hudson Ehsan Adeli Russ Altman Simran Arora Sydney von Arx Michael S Bernstein Jeannette Bohg Antoine Bosselut Emma Brunskill Erik Brynjolfsson Shyamal Buch Dallas Card Rodrigo Castellon Niladri Chatterji Annie Chen Kathleen Creel Jared Quincy Davis Dora Demszky Chris Donahue Moussa Doumbouya Esin Durmus Stefano Ermon John Etchemendy Kawin Ethayarajh Li Fei-Fei Chelsea Finn Trevor Gale Lauren Gillespie Karan Goel Noah Goodman Shelby Grossman Neel Guha Tatsunori Hashimoto Peter Henderson John Hewitt Daniel E. Ho Jenny Hong Kyle Hsu Jing Huang Thomas Icard Saahil Jain Dan Jurafsky Pratyusha Kalluri Siddharth Karamcheti Geoff Keeling Fereshte Khani Omar Khattab Pang Wei Koh Mark Krass Ranjay Krishna Rohith Kuditipudi Ananya Kumar Faisal Ladhak Mina Lee Tony Lee Jure Leskovec Isabelle Levent Xiang Lisa Li Xuechen Li Tengyu Ma Ali Malik Christopher D. Manning Suvir Mirchandani Eric Mitchell Zanele Munyikwa Suraj Nair Avanika Narayan Deepak Narayanan Ben Newman Allen Nie Juan Carlos Niebles Hamed Nilforoshan Julian Nyarko Giray Ogut Laurel Orr Isabel Papadimitriou Joon Sung Park Chris Piech Eva Portelance Christopher Potts Aditi Raghunathan Rob Reich Hongyu Ren Frieda Rong Yusuf Roohani Camilo Ruiz Jack Ryan Christopher R\u00e9 Dorsa Sadigh Shiori Sagawa Keshav Santhanam Andy Shih Krishnan Srinivasan Alex Tamkin Rohan Taori Armin W. Thomas Florian Tram\u00e8r Rose E. Wang and William Wang. 2021. On the opportunities and risks of foundation models. arXiv:2108.07258. Retrieved from https:\/\/arxiv.org\/abs\/2108.07258"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549113"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3517582"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-Companion58688.2023.00013"},{"key":"e_1_3_2_28_2","first-page":"100","volume-title":"Proceedings of the Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures","author":"Liu Jiachang","year":"2022","unstructured":"Jiachang Liu, Dinghan Shen, Yizhe Zhang, William B. Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for gpt-3? In Proceedings of the Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, 100\u2013114."},{"key":"e_1_3_2_29_2","volume-title":"Proceedings of the 11th International Conference on Learning Representations","author":"Arora Simran","year":"2023","unstructured":"Simran Arora, Avanika Narayan, Mayee F Chen, Laurel Orr, Neel Guha, Kush Bhatia, Ines Chami, and Christopher Re. 2023. Ask me anything: A simple strategy for prompting language models. In Proceedings of the 11th International Conference on Learning Representations."},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/3571730"},{"key":"e_1_3_2_31_2","first-page":"675","volume-title":"Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics","volume":"1","author":"Bang Yejin","year":"2023","unstructured":"Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, Vol. 1, Long Papers, 675\u2013718."},{"key":"e_1_3_2_32_2","first-page":"389","volume-title":"Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME)","author":"Ye Deheng","year":"2016","unstructured":"Deheng Ye, Zhenchang Xing, Chee Yong Foo, Jing Li, and Nachiket Kapre. 2016. Learning to extract api mentions from informal natural language discussions. In Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME), 389\u2013399."},{"key":"e_1_3_2_33_2","first-page":"90","volume-title":"Proceedings of the IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER)","volume":"1","author":"Ye Deheng","year":"2016","unstructured":"Deheng Ye, Zhenchang Xing, Chee Yong Foo, Zi Qun Ang, Jing Li, and Nachiket Kapre. 2016. Software-specific named entity recognition in software engineering social content. In Proceedings of the IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER), Vol. 1, 90\u2013101."},{"key":"e_1_3_2_34_2","unstructured":"Qing Huang Dianshu Liao Zhenchang Xing Zhiqiang Yuan Qinghua Lu Xiwei Xu and Jiaxing Lu. 2022. Se factual knowledge in frozen giant code model: A study on fqn and its retrieval. arXiv:2212.08221. Retrieved from https:\/\/arxiv.org\/abs\/2212.08221"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.emnlp-main.759"},{"key":"e_1_3_2_36_2","unstructured":"Saurav Kadavath Tom Conerly Amanda Askell T. J. Henighan Dawn Drain Ethan Perez Nicholas Schiefer Zachary Dodds Nova DasSarma Eli Tran-Johnson Scott Johnston Sheer El-Showk Andy Jones Nelson Elhage Tristan Hume Anna Chen Yuntao Bai Sam Bowman Stanislav Fort Deep Ganguli Danny Hernandez Josh Jacobson John Kernion Shauna Kravec Liane Lovitt Kamal Ndousse Catherine Olsson Sam Ringer Dario Amodei Tom B. Brown Jack Clark Nicholas Joseph Benjamin Mann Sam McCandlish Christopher Olah and Jared Kaplan. 2022. Language models (mostly) know what they know. arXiv:2207.05221. Retrieved from https:\/\/arxiv.org\/abs\/2207.05221"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338963"},{"key":"e_1_3_2_38_2","doi-asserted-by":"crossref","DOI":"10.1007\/978-94-017-1404-4","author":"Singh Ravindra Pal","year":"1996","unstructured":"Ravindra Pal Singh and Naurang Singh Mangat. 1996. Elements of Survey Sampling. Kluwer Academic Publishers.","journal-title":"Elements of Survey Sampling"},{"key":"e_1_3_2_39_2","doi-asserted-by":"crossref","first-page":"5190","DOI":"10.18653\/v1\/2023.findings-emnlp.345","volume-title":"Proceedings of the Findings of the Association for Computational Linguistics (EMNLP \u201923)","author":"Chen Jiefeng","year":"2023","unstructured":"Jiefeng Chen, Jinsung Yoon, Sayna Ebrahimi, Sercan Arik, Tomas Pfister, and Somesh Jha. 2023. Adaptation with self-evaluation to improve selective prediction in llms. In Proceedings of the Findings of the Association for Computational Linguistics (EMNLP \u201923), 5190\u20135213."},{"key":"e_1_3_2_40_2","first-page":"6556","volume-title":"Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","volume":"1","author":"Fu Jinlan","year":"2024","unstructured":"Jinlan Fu, See Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2024. Gptscore: Evaluate as you desire. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol. 1, Long Papers, 6556\u20136576."},{"key":"e_1_3_2_41_2","first-page":"832","volume-title":"Proceedings of the 35th International Conference on Software Engineering (ICSE)","author":"Rigby Peter C.","year":"2024","unstructured":"Peter C. Rigby and Martin P. Robillard. 2024. Discovering essential code elements in informal documentation. In Proceedings of the 35th International Conference on Software Engineering (ICSE), 832\u2013841, 2013."},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.5555\/2337223.2337230"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338971"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.2991\/ijndc.k.190911.002"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i05.6281"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/3447548.3467438"},{"key":"e_1_3_2_47_2","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"Brown Tom","year":"2020","unstructured":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. Advances in Neural Information Processing Systems 33 (2020), 1877\u20131901.","journal-title":"Advances in Neural Information Processing Systems"},{"issue":"140","key":"e_1_3_2_48_2","first-page":"1","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","volume":"21","author":"Raffel Colin","year":"2020","unstructured":"Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research 21, 140 (2020), 1\u201367.","journal-title":"The Journal of Machine Learning Research"},{"issue":"8","key":"e_1_3_2_49_2","first-page":"9","article-title":"Language models are unsupervised multitask learners","volume":"1","author":"Radford Alec","year":"2019","unstructured":"Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9.","journal-title":"OpenAI Blog"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v37i4.25642"},{"key":"e_1_3_2_51_2","unstructured":"Chunqiu Steven Xia Yuxiang Wei and Lingming Zhang. 2022. Practical program repair in the era of large pre-trained language models. arXiv:2210.14179. Retrieved from https:\/\/arxiv.org\/abs\/2210.14179"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00194"},{"key":"e_1_3_2_53_2","volume-title":"Proceedings of the 11th International Conference on Learning Representations","author":"Chen Bei","year":"2023","unstructured":"Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2023. Codet: Code generation with generated tests. In Proceedings of the 11th International Conference on Learning Representations."},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00181"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00119"},{"key":"e_1_3_2_56_2","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing","volume":"1","author":"Gao Tianyu","year":"2021","unstructured":"Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Vol. 1, Long Papers."},{"key":"e_1_3_2_57_2","doi-asserted-by":"crossref","first-page":"589","DOI":"10.18653\/v1\/2022.findings-acl.50","volume-title":"Proceedings of the Findings of the Association for Computational Linguistics (ACL \u201922)","author":"Mishra Swaroop","year":"2022","unstructured":"Swaroop Mishra, Daniel Khashabi, Chitta Baral, Yejin Choi, and Hannaneh Hajishirzi. 2022. Reframing instructional prompts to gptk\u2019s language. In Proceedings of the Findings of the Association for Computational Linguistics (ACL \u201922), 589\u2013612."},{"key":"e_1_3_2_58_2","volume-title":"Proceedings of the 11th International Conference on Learning Representations","author":"Wang Xuezhi","year":"2023","unstructured":"Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In Proceedings of the 11th International Conference on Learning Representations."},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.naacl-main.191"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.acl-demo.9"},{"key":"e_1_3_2_61_2","doi-asserted-by":"crossref","first-page":"390","DOI":"10.18653\/v1\/2021.emnlp-main.32","volume-title":"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing","author":"Schick Timo","year":"2021","unstructured":"Timo Schick and Hinrich Sch\u00fctze. 2021. Few-shot text generation with natural language instructions. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 390\u2013402."},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.407"},{"key":"e_1_3_2_63_2","first-page":"2627","volume-title":"Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Le Scao Teven","year":"2021","unstructured":"Teven Le Scao and Alexander M. Rush. 2021. How many data points is a prompt worth? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2627\u20132636."},{"key":"e_1_3_2_64_2","first-page":"24824","article-title":"Chain-of-thought prompting elicits reasoning in large language models","volume":"35","author":"Wei Jason","year":"2022","unstructured":"Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V. Le, Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems 35 (2022), 24824\u201324837.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","DOI":"10.1145\/3490099.3511105"},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3502030"},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3501819"},{"key":"e_1_3_2_68_2","first-page":"604","volume-title":"Proceedings of the IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)","author":"Li Xiangwei","year":"2023","unstructured":"Xiangwei Li, Xiaoning Ren, Yinxing Xue, Zhenchang Xing, and Jiamou Sun. 2023. Prediction of vulnerability characteristics based on vulnerability description and prompt learning. In Proceedings of the IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), 604\u2013615."},{"key":"e_1_3_2_69_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.aiopen.2022.11.003"},{"key":"e_1_3_2_70_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.243"},{"key":"e_1_3_2_71_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.acl-long.353"},{"key":"e_1_3_2_72_2","first-page":"61","volume-title":"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics","volume":"2","author":"Liu Xiao","year":"2022","unstructured":"Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2022. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, Vol. 2, Short Papers, 61\u201368."},{"key":"e_1_3_2_73_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.eacl-main.20"},{"key":"e_1_3_2_74_2","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3556912"},{"key":"e_1_3_2_75_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.eacl-main.153"},{"key":"e_1_3_2_76_2","unstructured":"Chenguang Wang Xiao Liu and Dawn Xiaodong Song. 2020. Language models are open knowledge graphs. arXiv:2010.11967. Retrieved from https:\/\/arxiv.org\/abs\/2010.11967"},{"key":"e_1_3_2_77_2","first-page":"4171","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","volume":"1","author":"Devlin Jacob","year":"2019","unstructured":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol. 1, Long and Short Papers, 4171\u20134186."}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3680469","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3680469","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:58:26Z","timestamp":1750294706000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3680469"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,11,30]]},"references-count":76,"journal-issue":{"issue":"8","published-print":{"date-parts":[[2024,11,30]]}},"alternative-id":["10.1145\/3680469"],"URL":"https:\/\/doi.org\/10.1145\/3680469","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,11,30]]},"assertion":[{"value":"2023-11-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-06-26","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-12-03","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}