{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,24]],"date-time":"2025-12-24T08:53:41Z","timestamp":1766566421059,"version":"3.48.0"},"publisher-location":"New York, NY, USA","reference-count":51,"publisher":"ACM","funder":[{"name":"TU Wien Bibliothek for financial support through its Open Access Funding Programme."}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2025,6,17]]},"DOI":"10.1145\/3756681.3756997","type":"proceedings-article","created":{"date-parts":[[2025,12,24]],"date-time":"2025-12-24T08:30:04Z","timestamp":1766565004000},"page":"1045-1055","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Large Language Models for API Classification: An Explorative Study"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1113-7873","authenticated-orcid":false,"given":"Gabriel","family":"Morais","sequence":"first","affiliation":[{"name":"D\u00e9partement de math\u00e9matiques, informatique et g\u00e9nie, Universit\u00e9 du Qu\u00e9bec \u00e0 Rimouski (UQAR), L\u00e9vis, Qu\u00e9bec, Canada"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-4396-9195","authenticated-orcid":false,"given":"Edwin","family":"Lemelin","sequence":"additional","affiliation":[{"name":"D\u00e9partement de math\u00e9matiques, informatique et g\u00e9nie, Universit\u00e9 du Qu\u00e9bec \u00e0 Rimouski (UQAR), L\u00e9vis, Canada and Universit\u00e9 Laval, Qu\u00e9bec, Qu\u00e9bec, Canada"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5327-1758","authenticated-orcid":false,"given":"Mehdi","family":"Adda","sequence":"additional","affiliation":[{"name":"D\u00e9partement de math\u00e9matiques, informatique et g\u00e9nie, Universit\u00e9 du Qu\u00e9bec \u00e0 Rimouski (UQAR), L\u00e9vis, Qu\u00e9bec, Canada"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8259-2297","authenticated-orcid":false,"given":"Dominik","family":"Bork","sequence":"additional","affiliation":[{"name":"Business Informatics Group, TU Wien, Vienna, Austria and Universit\u00e9 du Qu\u00e9bec \u00e0 Rimouski (UQAR), L\u00e9vis, Qu\u00e9bec, Canada"}]}],"member":"320","published-online":{"date-parts":[[2025,12,24]]},"reference":[{"key":"e_1_3_3_2_2_2","doi-asserted-by":"crossref","unstructured":"Ramakrishna Bairi Atharv Sonwane Aditya Kanade Arun Iyer Suresh Parthasarathy Sriram Rajamani B Ashok and Shashank Shet. 2024. Codeplan: Repository-level coding using llms and planning. Proceedings of the ACM on Software Engineering 1 FSE (2024) 675\u2013698.","DOI":"10.1145\/3643757"},{"key":"e_1_3_3_2_3_2","volume-title":"Proceedings of the 34th International Conference on Neural Information Processing Systems","author":"Brown Tom\u00a0B.","year":"2020","unstructured":"Tom\u00a0B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel\u00a0M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems. Curran Associates Inc., Red Hook, NY, USA, Article 159."},{"key":"e_1_3_3_2_4_2","doi-asserted-by":"crossref","unstructured":"Yupeng Chang Xu Wang Jindong Wang Yuan Wu Linyi Yang Kaijie Zhu Hao Chen Xiaoyuan Yi Cunxiang Wang Yidong Wang et\u00a0al. 2024. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology 15 3 (2024) 1\u201345.","DOI":"10.1145\/3641289"},{"key":"e_1_3_3_2_5_2","unstructured":"Lingjiao Chen Matei Zaharia and James Zou. 2023. How is ChatGPT\u2019s behavior changing over time? arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2307.09009 (2023)."},{"key":"e_1_3_3_2_6_2","unstructured":"Wei-Lin Chiang Lianmin Zheng Ying Sheng Anastasios\u00a0Nikolas Angelopoulos Tianle Li Dacheng Li Hao Zhang Banghua Zhu Michael Jordan Joseph\u00a0E. Gonzalez and Ion Stoica. 2024. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. arxiv:https:\/\/arXiv.org\/abs\/2403.04132\u00a0[cs.AI]"},{"key":"e_1_3_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/3643991.3644903"},{"key":"e_1_3_3_2_8_2","doi-asserted-by":"crossref","unstructured":"Leonardo da Rocha\u00a0Araujo Guillermo Rodr\u00edguez Santiago Vidal Claudia Marcos and Rodrigo\u00a0Pereira dos Santos. 2021. Empirical Analysis on OpenAPI Topic Exploration and Discovery to Support the Developer Community. Computing and Informatics 40 6 (2021) 1345\u20131369.","DOI":"10.31577\/cai_2021_6_1345"},{"key":"e_1_3_3_2_9_2","unstructured":"Joost\u00a0CF de Winter. 2023. Can ChatGPT pass high school exams on English language comprehension? International Journal of Artificial Intelligence in Education (2023) 1\u201316."},{"key":"e_1_3_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3661167.3661226"},{"key":"e_1_3_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-FoSE59343.2023.00008"},{"key":"e_1_3_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3608134"},{"key":"e_1_3_3_2_13_2","doi-asserted-by":"crossref","unstructured":"Tony Gorschek Per Garre Stig Larsson and Claes Wohlin. 2006. A model for technology transfer in practice. IEEE software 23 6 (2006) 88\u201395.","DOI":"10.1109\/MS.2006.147"},{"key":"e_1_3_3_2_14_2","unstructured":"Trevor Hastie and Robert Tibshirani. 1997. Classification by pairwise coupling. Advances in neural information processing systems 10 (1997)."},{"key":"e_1_3_3_2_15_2","unstructured":"Xinyi Hou Yanjie Zhao Yue Liu Zhou Yang Kailong Wang Li Li Xiapu Luo David Lo John Grundy and Haoyu Wang. 2023. Large language models for software engineering: A systematic literature review. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2308.10620 (2023)."},{"key":"e_1_3_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639585"},{"key":"e_1_3_3_2_17_2","unstructured":"OpenAPI Initiative. 2021. OpenAPI Specification v3.1.0. https:\/\/spec.openapis.org\/oas\/latest.html."},{"key":"e_1_3_3_2_18_2","doi-asserted-by":"crossref","unstructured":"Katikapalli\u00a0Subramanyam Kalyan. 2023. A survey of GPT-3 family large language models including ChatGPT and GPT-4. Natural Language Processing Journal (2023) 100048.","DOI":"10.1016\/j.nlp.2023.100048"},{"key":"e_1_3_3_2_19_2","unstructured":"Takeshi Kojima Shixiang\u00a0Shane Gu Machel Reid Yutaka Matsuo and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems 35 (2022) 22199\u201322213."},{"key":"e_1_3_3_2_20_2","volume-title":"Content analysis: An introduction to its methodology","author":"Krippendorff Klaus","year":"2018","unstructured":"Klaus Krippendorff. 2018. Content analysis: An introduction to its methodology. Sage publications."},{"key":"e_1_3_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/3661167.3661210"},{"key":"e_1_3_3_2_22_2","volume-title":"The Fin-Tech API Descriptions Dataset (FinTechAPIs)","author":"Lemelin Edwin","year":"2024","unstructured":"Edwin Lemelin, Gabriel Morais, and Oumeima Nuigues. 2024. The Fin-Tech API Descriptions Dataset (FinTechAPIs). https:\/\/github.com\/UQAR-TUW\/FinTechAPIsDataset"},{"key":"e_1_3_3_2_23_2","unstructured":"James Lewis and Martin Fowler. 2014. Microservices: a definition of this new architectural term. MartinFowler. com 25 14-26 (2014) 12."},{"key":"e_1_3_3_2_24_2","unstructured":"Jiawei Liu Chunqiu\u00a0Steven Xia Yuyao Wang and Lingming Zhang. 2024. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems 36 (2024)."},{"key":"e_1_3_3_2_25_2","doi-asserted-by":"crossref","unstructured":"Nelson\u00a0F Liu Kevin Lin John Hewitt Ashwin Paranjape Michele Bevilacqua Fabio Petroni and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics 12 (2024) 157\u2013173.","DOI":"10.1162\/tacl_a_00638"},{"key":"e_1_3_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/3643916.3644403"},{"key":"e_1_3_3_2_27_2","doi-asserted-by":"crossref","unstructured":"Giacomo Marzi Marco Balzano and Davide Marchiori. 2024. K-Alpha Calculator\u2013Krippendorff\u2019s Alpha Calculator: A user-friendly tool for computing Krippendorff\u2019s Alpha inter-rater reliability coefficient. MethodsX 12 (2024) 102545.","DOI":"10.1016\/j.mex.2023.102545"},{"key":"e_1_3_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/3652620.3687799"},{"key":"e_1_3_3_2_29_2","volume-title":"EDOC 2024: 28th International Conference on Enterprise Design, Operations, and Computing \u2013 Forum Paper","author":"Morais Gabriel","year":"2024","unstructured":"Gabriel Morais, Edwin Lemelin, Mehdi Adda, and Dominik Bork. 2024. Enhancing API Labelling with BERT and GPT: An Exploratory Study. In EDOC 2024: 28th International Conference on Enterprise Design, Operations, and Computing \u2013 Forum Paper. Springer."},{"key":"e_1_3_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639187"},{"key":"e_1_3_3_2_31_2","unstructured":"Peter Norvig and Stuart Russell. 2021. Artificial intelligence: a modern approach. Pearson Harlow 1 (2021) 1239\u20131269."},{"key":"e_1_3_3_2_32_2","doi-asserted-by":"crossref","unstructured":"Joshua Ofoeda Richard Boateng and John Effah. 2019. Application programming interface (API) research: A review of the past to inform the future. International Journal of Enterprise Information Systems (IJEIS) 15 3 (2019) 76\u201395.","DOI":"10.4018\/IJEIS.2019070105"},{"key":"e_1_3_3_2_33_2","unstructured":"Shuyin Ouyang Jie\u00a0M Zhang Mark Harman and Meng Wang. 2024. An empirical study of the non-determinism of chatgpt in code generation. ACM Transactions on Software Engineering and Methodology (2024)."},{"key":"e_1_3_3_2_34_2","volume-title":"Manual of political economy: a critical and variorum edition","author":"Pareto Vilfredo","year":"2014","unstructured":"Vilfredo Pareto. 2014. Manual of political economy: a critical and variorum edition. OUP Oxford."},{"key":"e_1_3_3_2_35_2","unstructured":"Mohaimenul Azam\u00a0Khan Raiaan Md\u00a0Saddam\u00a0Hossain Mukta Kaniz Fatema Nur\u00a0Mohammad Fahad Sadman Sakib Most Marufatul\u00a0Jannat Mim Jubaer Ahmad Mohammed\u00a0Eunus Ali and Sami Azam. 2024. A review on large Language Models: Architectures applications taxonomies open issues and challenges. IEEE Access (2024)."},{"key":"e_1_3_3_2_36_2","doi-asserted-by":"crossref","unstructured":"Navid Sahebjamnia S\u00a0Ali Torabi and S\u00a0Afshin Mansouri. 2015. Integrated business continuity and disaster recovery planning: Towards organizational resilience. European journal of operational research 242 1 (2015) 261\u2013273.","DOI":"10.1016\/j.ejor.2014.09.055"},{"key":"e_1_3_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3639476.3639764"},{"key":"e_1_3_3_2_38_2","doi-asserted-by":"crossref","unstructured":"Glorin Sebastian. 2023. Privacy and data protection in ChatGPT and other AI Chatbots: strategies for securing user information. International Journal of Security and Privacy in Pervasive Computing (IJSPPC) 15 1 (2023) 1\u201314.","DOI":"10.4018\/IJSPPC.325475"},{"key":"e_1_3_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/3643991.3644932"},{"key":"e_1_3_3_2_40_2","doi-asserted-by":"crossref","unstructured":"Marina Sokolova and Guy Lapalme. 2009. A systematic analysis of performance measures for classification tasks. Information processing & management 45 4 (2009) 427\u2013437.","DOI":"10.1016\/j.ipm.2009.03.002"},{"key":"e_1_3_3_2_41_2","doi-asserted-by":"crossref","unstructured":"Arun\u00a0James Thirunavukarasu Darren Shu\u00a0Jeng Ting Kabilan Elangovan Laura Gutierrez Ting\u00a0Fang Tan and Daniel Shu\u00a0Wei Ting. 2023. Large language models in medicine. Nature medicine 29 8 (2023) 1930\u20131940.","DOI":"10.1038\/s41591-023-02448-8"},{"key":"e_1_3_3_2_42_2","unstructured":"Miles Turpin Julian Michael Ethan Perez and Samuel Bowman. 2024. Language models don\u2019t always say what they think: unfaithful explanations in chain-of-thought prompting. Advances in Neural Information Processing Systems 36 (2024)."},{"key":"e_1_3_3_2_43_2","doi-asserted-by":"crossref","unstructured":"Yao Wan Zhangqian Bi Yang He Jianguo Zhang Hongyu Zhang Yulei Sui Guandong Xu Hai Jin and Philip Yu. 2024. Deep Learning for Code Intelligence: Survey Benchmark and Toolkit. Comput. Surveys (2024).","DOI":"10.1145\/3664597"},{"key":"e_1_3_3_2_44_2","doi-asserted-by":"crossref","unstructured":"Li Wang Xi Chen XiangWen Deng Hao Wen MingKe You WeiZhi Liu Qi Li and Jian Li. 2024. Prompt engineering in consistency and reliability with the evidence-based guideline for LLMs. npj Digital Medicine 7 1 (2024) 41.","DOI":"10.1038\/s41746-024-01029-4"},{"key":"e_1_3_3_2_45_2","doi-asserted-by":"crossref","unstructured":"Muhammad Waseem Peng Liang and Mojtaba Shahin. 2020. A systematic mapping study on microservices architecture in devops. Journal of Systems and Software 170 (2020) 110798.","DOI":"10.1016\/j.jss.2020.110798"},{"key":"e_1_3_3_2_46_2","unstructured":"Jason Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Fei Xia Ed Chi Quoc\u00a0V Le Denny Zhou et\u00a0al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 (2022) 24824\u201324837."},{"key":"e_1_3_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.5555\/2700546"},{"key":"e_1_3_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-29044-2"},{"key":"e_1_3_3_2_49_2","doi-asserted-by":"crossref","unstructured":"Jingfeng Yang Hongye Jin Ruixiang Tang Xiaotian Han Qizhang Feng Haoming Jiang Shaochen Zhong Bing Yin and Xia Hu. 2024. Harnessing the power of llms in practice: A survey on chatgpt and beyond. ACM Transactions on Knowledge Discovery from Data 18 6 (2024) 1\u201332.","DOI":"10.1145\/3649506"},{"key":"e_1_3_3_2_50_2","doi-asserted-by":"crossref","unstructured":"Yanming Yang Xin Xia David Lo and John Grundy. 2022. A survey on deep learning for software engineering. ACM Computing Surveys (CSUR) 54 10s (2022) 1\u201373.","DOI":"10.1145\/3505243"},{"key":"e_1_3_3_2_51_2","doi-asserted-by":"crossref","unstructured":"Shangbo Yun Shuhuai Lin Xiaodong Gu and Beijun Shen. 2024. Project-specific code summarization with in-context learning. Journal of Systems and Software (2024) 112149.","DOI":"10.1016\/j.jss.2024.112149"},{"key":"e_1_3_3_2_52_2","doi-asserted-by":"crossref","unstructured":"Xin Zhou Shanshan Li Lingli Cao He Zhang Zijia Jia Chenxing Zhong Zhihao Shan and Muhammad\u00a0Ali Babar. 2023. Revisiting the practices and pains of microservice architecture in reality: An industrial inquiry. Journal of Systems and Software 195 (2023) 111521.","DOI":"10.1016\/j.jss.2022.111521"}],"event":{"name":"EASE '25: Evaluation and Assessment in Software Engineering","location":"Istanbul Turkiye","acronym":"EASE '25"},"container-title":["Proceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3756681.3756997","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,12,24]],"date-time":"2025-12-24T08:39:13Z","timestamp":1766565553000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3756681.3756997"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,17]]},"references-count":51,"alternative-id":["10.1145\/3756681.3756997","10.1145\/3756681"],"URL":"https:\/\/doi.org\/10.1145\/3756681.3756997","relation":{},"subject":[],"published":{"date-parts":[[2025,6,17]]},"assertion":[{"value":"2025-12-24","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}