{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,6]],"date-time":"2026-06-06T19:07:26Z","timestamp":1780772846952,"version":"3.54.1"},"publisher-location":"New York, NY, USA","reference-count":21,"publisher":"ACM","license":[{"start":{"date-parts":[[2024,8,24]],"date-time":"2024-08-24T00:00:00Z","timestamp":1724457600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,8,25]]},"DOI":"10.1145\/3637528.3671592","type":"proceedings-article","created":{"date-parts":[[2024,8,25]],"date-time":"2024-08-25T04:54:55Z","timestamp":1724561695000},"page":"5680-5689","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":5,"title":["Measuring an LLM's Proficiency at using APIs: A Query Generation Strategy"],"prefix":"10.1145","author":[{"given":"Ying","family":"Sheng","sequence":"first","affiliation":[{"name":"Google Research, Mountain View, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sudeep","family":"Gandhe","sequence":"additional","affiliation":[{"name":"Google Research, Mountain View, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bhargav","family":"Kanagal","sequence":"additional","affiliation":[{"name":"Google Research, Mountain View, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nick","family":"Edmonds","sequence":"additional","affiliation":[{"name":"Google Research, Mountain View, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zachary","family":"Fisher","sequence":"additional","affiliation":[{"name":"Google Research, Mountain View, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sandeep","family":"Tata","sequence":"additional","affiliation":[{"name":"Google Research, Mountain View, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Aarush","family":"Selvan","sequence":"additional","affiliation":[{"name":"Google, Mountain View, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,8,24]]},"reference":[{"key":"e_1_3_2_2_1_1","unstructured":"Rohan Anil Andrew M. Dai Orhan Firat Melvin Johnson Dmitry Lepikhin Alexandre Passos Siamak Shakeri Emanuel Taropa Paige Bailey Zhifeng Chen Eric Chu Jonathan H. Clark Laurent El Shafey Yanping Huang Kathy Meier-Hellstern Gaurav Mishra Erica Moreira Mark Omernick Kevin Robinson Sebastian Ruder Yi Tay Kefan Xiao Yuanzhong Xu Yujing Zhang Gustavo Hernandez Abrego Junwhan Ahn Jacob Austin Paul Barham Jan Botha James Bradbury Siddhartha Brahma Kevin Brooks Michele Catasta Yong Cheng Colin Cherry Christopher A. Choquette-Choo Aakanksha Chowdhery Cl\u00e9ment Crepy Shachi Dave Mostafa Dehghani Sunipa Dev Jacob Devlin Mark D\u00edaz Nan Du Ethan Dyer Vlad Feinberg Fangxiaoyu Feng Vlad Fienber Markus Freitag Xavier Garcia Sebastian Gehrmann Lucas Gonzalez Guy Gur-Ari Steven Hand Hadi Hashemi Le Hou Joshua Howland Andrea Hu Jeffrey Hui Jeremy Hurwitz Michael Isard Abe Ittycheriah Matthew Jagielski Wenhao Jia Kathleen Kenealy Maxim Krikun Sneha Kudugunta Chang Lan Katherine Lee Benjamin Lee Eric Li Music Li Wei Li YaGuang Li Jian Li Hyeontaek Lim Hanzhao Lin Zhongtao Liu Frederick Liu Marcello Maggioni Aroma Mahendru Joshua Maynez Vedant Misra Maysam Moussalem Zachary Nado John Nham Eric Ni Andrew Nystrom Alicia Parrish Marie Pellat Martin Polacek Alex Polozov Reiner Pope Siyuan Qiao Emily Reif Bryan Richter Parker Riley Alex Castro Ros Aurko Roy Brennan Saeta Rajkumar Samuel Renee Shelby Ambrose Slone Daniel Smilkov David R. So Daniel Sohn Simon Tokumine Dasha Valter Vijay Vasudevan Kiran Vodrahalli Xuezhi Wang Pidong Wang Zirui Wang Tao Wang John Wieting Yuhuai Wu Kelvin Xu Yunhan Xu Linting Xue Pengcheng Yin Jiahui Yu Qiao Zhang Steven Zheng Ce Zheng Weikang Zhou Denny Zhou Slav Petrov and Yonghui Wu. 2023. PaLM 2 Technical Report. arXiv:cs.CL\/2305.10403"},{"key":"e_1_3_2_2_2_1","unstructured":"Bard [n.d.]. Google Bard. http:\/\/bard.google.com. Accessed: 2024-01-19."},{"key":"e_1_3_2_2_3_1","unstructured":"bard-ext [n.d.]. Bard can now connect to your Google apps and services. https: \/\/blog.google\/products\/bard\/google-bard-new-features-update-sept-2023\/. Accessed: 2024-01-19."},{"key":"e_1_3_2_2_4_1","unstructured":"Bing [n.d.]. Reinventing search with a new AI-powered Microsoft Bing and Edge your copilot for the web. https:\/\/blogs.microsoft.com\/blog\/2023\/02\/07\/ reinventing-search-with-a-new-ai-powered-microsoft-bing-and-edge-your-copilot-for-the-web\/. Accessed: 2024-01-22."},{"key":"e_1_3_2_2_5_1","unstructured":"Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell Sandhini Agarwal Ariel Herbert-Voss Gretchen Krueger Tom Henighan Rewon Child Aditya Ramesh Daniel M. Ziegler Jeffrey Wu Clemens Winter Christopher Hesse Mark Chen Eric Sigler Mateusz Litwin Scott Gray Benjamin Chess Jack Clark Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever and Dario Amodei. 2020. Language Models are Few-Shot Learners. arXiv:cs.CL\/2005.14165"},{"key":"e_1_3_2_2_6_1","unstructured":"ChatGPT [n.d.]. ChatGPT. http:\/\/chat.openai.com. Accessed: 2024-01-19."},{"key":"e_1_3_2_2_7_1","unstructured":"functioncalling [n.d.]. Function Calling. https:\/\/platform.openai.com\/docs\/ guides\/function-calling. Accessed: 2024-01--29."},{"key":"e_1_3_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00276"},{"key":"e_1_3_2_2_9_1","volume-title":"Daniel Fried, Sida Wang, and Tao Yu.","author":"Lai Yuhang","year":"2022","unstructured":"Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettle-moyer, Scott Wen tau Yih, Daniel Fried, Sida Wang, and Tao Yu. 2022. DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation. arXiv:cs.SE\/2211.11501"},{"key":"e_1_3_2_2_10_1","unstructured":"Minghao Li Yingxiu Zhao Bowen Yu Feifan Song Hangyu Li Haiyang Yu Zhoujun Li Fei Huang and Yongbin Li. 2023. API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs. arXiv:cs.CL\/2304.08244"},{"key":"e_1_3_2_2_11_1","volume-title":"Song-Chun Zhu, and Jianfeng Gao.","author":"Lu Pan","year":"2023","unstructured":"Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. 2023. Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models. arXiv:cs.CL\/2304.09842"},{"key":"e_1_3_2_2_12_1","unstructured":"Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright Pamela Mishkin Chong Zhang Sandhini Agarwal Katarina Slama Alex Ray John Schulman Jacob Hilton Fraser Kelton Luke Miller Maddie Simens Amanda Askell Peter Welinder Paul Christiano Jan Leike and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. arXiv:cs.CL\/2203.02155"},{"key":"e_1_3_2_2_13_1","volume-title":"TALM: Tool Augmented Language Models. arXiv:cs.CL\/2205.12255","author":"Parisi Aaron","year":"2022","unstructured":"Aaron Parisi, Yao Zhao, and Noah Fiedel. 2022. TALM: Tool Augmented Language Models. arXiv:cs.CL\/2205.12255"},{"key":"e_1_3_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.naacl-main.168"},{"key":"e_1_3_2_2_15_1","volume-title":"Gonzalez","author":"Patil Shishir G.","year":"2023","unstructured":"Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. 2023. Gorilla: Large Language Model Connected with Massive APIs. arXiv:cs.CL\/2305.15334"},{"key":"e_1_3_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1"},{"key":"e_1_3_2_2_17_1","unstructured":"Yujia Qin Shihao Liang Yining Ye Kunlun Zhu Lan Yan Yaxi Lu Yankai Lin Xin Cong Xiangru Tang Bill Qian Sihan Zhao Lauren Hong Runchu Tian Ruobing Xie Jie Zhou Mark Gerstein Dahai Li Zhiyuan Liu and Maosong Sun. 2023. ToolLLM: Facilitating Large Language Models to Master 16000 Real-world APIs. arXiv:cs.AI\/2307.16789"},{"key":"e_1_3_2_2_18_1","volume-title":"Toolformer: Language Models Can Teach Themselves to Use Tools. arXiv:cs.CL\/2302.04761","author":"Schick Timo","year":"2023","unstructured":"Timo Schick, Jane Dwivedi-Yu, Roberto Dess\u00ec, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language Models Can Teach Themselves to Use Tools. arXiv:cs.CL\/2302.04761"},{"key":"e_1_3_2_2_19_1","unstructured":"SGE [n.d.]. Supercharging Search with generative AI. https:\/\/blog.google\/ products\/search\/generative-ai-search\/. Accessed: 2024-01-22."},{"key":"e_1_3_2_2_20_1","unstructured":"Gemini Team Rohan Anil Sebastian Borgeaud Yonghui Wu Jean-Baptiste Alayrac Jiahui Yu Radu Soricut Johan Schalkwyk Andrew M Dai Anja Hauth et al. 2023. Gemini: a family of highly capable multimodal models."},{"key":"e_1_3_2_2_21_1","unstructured":"Shunyu Yao Jeffrey Zhao Dian Yu Nan Du Izhak Shafran Karthik Narasimhan and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. arXiv:csCL\/2210.03629."}],"event":{"name":"KDD '24: The 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining","location":"Barcelona Spain","acronym":"KDD '24","sponsor":["SIGMOD ACM Special Interest Group on Management of Data","SIGKDD ACM Special Interest Group on Knowledge Discovery in Data"]},"container-title":["Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3637528.3671592","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3637528.3671592","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:04:20Z","timestamp":1750291460000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3637528.3671592"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,24]]},"references-count":21,"alternative-id":["10.1145\/3637528.3671592","10.1145\/3637528"],"URL":"https:\/\/doi.org\/10.1145\/3637528.3671592","relation":{},"subject":[],"published":{"date-parts":[[2024,8,24]]},"assertion":[{"value":"2024-08-24","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}