{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,4]],"date-time":"2026-05-04T05:47:20Z","timestamp":1777873640739,"version":"3.51.4"},"publisher-location":"New York, NY, USA","reference-count":50,"publisher":"ACM","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2025,8,3]]},"DOI":"10.1145\/3711896.3736884","type":"proceedings-article","created":{"date-parts":[[2025,8,1]],"date-time":"2025-08-01T13:32:14Z","timestamp":1754055134000},"page":"932-943","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Cognify: Supercharging Gen-AI Workflows With Hierarchical Autotuning"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0009-0004-8827-2664","authenticated-orcid":false,"given":"Zijian","family":"He","sequence":"first","affiliation":[{"name":"University of California, San Diego, San Diego, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-6763-0108","authenticated-orcid":false,"given":"Reyna","family":"Abhyankar","sequence":"additional","affiliation":[{"name":"University of California, San Diego, La Jolla, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-5919-813X","authenticated-orcid":false,"given":"Vikranth","family":"Srivatsa","sequence":"additional","affiliation":[{"name":"University of California, San Diego, San Diego, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-6263-7802","authenticated-orcid":false,"given":"Yiying","family":"Zhang","sequence":"additional","affiliation":[{"name":"University of California, San Diego, San Diego, CA, USA and GenseeAI Inc., San Diego, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2025,8,3]]},"reference":[{"key":"e_1_3_2_2_1_1","unstructured":"Fireworks AI. 2024. API Reference - Fireworks AI. https:\/\/docs.fireworks.ai\/api-reference\/introduction. Accessed: 2025-02-10."},{"key":"e_1_3_2_2_2_1","unstructured":"BIG bench authors. 2023. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research(2023). https:\/\/openreview.net\/forum?id=uyTL5Bvosj"},{"key":"e_1_3_2_2_3_1","volume-title":"Proceedings of the 25th International Conference on Neural Information Processing Systems (NeurIPS). 2546-2554","author":"Bergstra James","year":"2011","unstructured":"James Bergstra, R\u00e9mi Bardenet, Yoshua Bengio, and Bal\u00e1zs K\u00e9gl. 2011. Algorithms for Hyper-Parameter Optimization. In Proceedings of the 25th International Conference on Neural Information Processing Systems (NeurIPS). 2546-2554."},{"key":"e_1_3_2_2_4_1","unstructured":"Ilan Bigio James Hills Shyamal Anadkat Charu Jaiswal Colin Jarvis and Katia Gil Guzman. [n.d.]. Swarm by OpenAI. https:\/\/github.com\/openai\/swarm."},{"key":"e_1_3_2_2_5_1","unstructured":"Lingjiao Chen Matei Zaharia and James Zou. 2024. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. Transactions on Machine Learning Research(2024)."},{"key":"e_1_3_2_2_6_1","unstructured":"Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde de Oliveira Pinto Jared Kaplan Harri Edwards Yuri Burda Nicholas Joseph Greg Brockman Alex Ray Raul Puri Gretchen Krueger Michael Petrov Heidy Khlaaf Girish Sastry Pamela Mishkin Brooke Chan Scott Gray Nick Ryder Mikhail Pavlov Alethea Power Lukasz Kaiser Mohammad Bavarian Clemens Winter Philippe Tillet Felipe Petroski Such Dave Cummings Matthias Plappert Fotios Chantzis Elizabeth Barnes Ariel Herbert-Voss William Hebgen Guss Alex Nichol Alex Paino Nikolas Tezak Jie Tang Igor Babuschkin Suchir Balaji Shantanu Jain William Saunders Christopher Hesse Andrew N. Carr Jan Leike Josh Achiam Vedant Misra Evan Morikawa Alec Radford Matthew Knight Miles Brundage Mira Murati Katie Mayer Peter Welinder Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv: 2107.03374(2021)."},{"key":"e_1_3_2_2_7_1","doi-asserted-by":"crossref","unstructured":"Ching-An Cheng Allen Nie and Adith Swaminathan. 2024. Trace is the Next AutoDiff: Generative Optimization with Rich Feedback Execution Traces and LLMs. In Advances in Neural Information Processing Systems. https:\/\/neurips.cc\/virtual\/2024\/poster\/93431","DOI":"10.52202\/079017-2287"},{"key":"e_1_3_2_2_8_1","unstructured":"Cognition AI. [n.d.]. Devin AI: Autonomous AI Software Engineer. https:\/\/devin.ai\/."},{"key":"e_1_3_2_2_9_1","unstructured":"Coze. [n.d.]. Coze: Intelligent Communication Solutions. https:\/\/www.coze.com\/."},{"key":"e_1_3_2_2_10_1","unstructured":"CrewAI. [n.d.]. CrewAI: The Leading Multi-Agent Platform. https:\/\/www.crewai.com\/. Website."},{"key":"e_1_3_2_2_11_1","unstructured":"Databricks. [n.d.]. Compound AI Systems: Integrating Multiple AI Models and Tools. https:\/\/www.databricks.com\/glossary\/compound-ai-systems."},{"key":"e_1_3_2_2_12_1","unstructured":"Dify. [n.d.]. Dify: Empowering AI Applications. https:\/\/dify.ai\/."},{"key":"e_1_3_2_2_13_1","volume-title":"Proceedings of the 35th International Conference on Machine Learning (ICML). 1437-1446","author":"Falkner Stefan","year":"2018","unstructured":"Stefan Falkner, Aaron Klein, and Frank Hutter. 2018. BOHB: Robust and Efficient Hyperparameter Optimization at Scale. In Proceedings of the 35th International Conference on Machine Learning (ICML). 1437-1446. arXiv:1807.01774"},{"key":"e_1_3_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2308.15363"},{"key":"e_1_3_2_2_15_1","unstructured":"Google Cloud. [n.d.]. Vertex AI Search: Build Google-quality search applications. https:\/\/cloud.google.com\/enterprise-search. Accessed: 2025-01-20."},{"key":"e_1_3_2_2_16_1","unstructured":"Wenlong Huang Pieter Abbeel Deepak Pathak and Igor Mordatch. 2022. Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents. arXiv preprint arXiv:2201.07207(2022)."},{"key":"e_1_3_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-05318-5"},{"key":"e_1_3_2_2_18_1","first-page":"2016","article-title":"Neural Architecture Search with Bayesian Optimisation and Optimal Transport","author":"Kandasamy Kirthevasan","year":"2018","unstructured":"Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnabas Poczos, and Eric P. Xing. 2018. Neural Architecture Search with Bayesian Optimisation and Optimal Transport. In Advances in Neural Information Processing Systems (NeurIPS). 2016-2025. arXiv:1802.07191","journal-title":"Advances in Neural Information Processing Systems (NeurIPS)."},{"key":"e_1_3_2_2_19_1","volume-title":"David Hall, Percy Liang, Christopher Potts, and Matei Zaharia.","author":"Khattab Omar","year":"2022","unstructured":"Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. 2022. Demonstrate-Search-Predict: Composing Retrieval and Language Models for Knowledge-Intensive NLP. arXiv preprint arXiv:2212.14024(2022)."},{"key":"e_1_3_2_2_20_1","volume-title":"The Twelfth International Conference on Learning Representations.","author":"Khattab Omar","year":"2024","unstructured":"Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. 2024. DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines. The Twelfth International Conference on Learning Representations."},{"key":"e_1_3_2_2_21_1","volume-title":"Decomposed Prompting: A Modular Approach for Solving Complex Tasks. In The Eleventh International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=_nGgzQjzaRy","author":"Khot Tushar","year":"2023","unstructured":"Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. 2023. Decomposed Prompting: A Modular Approach for Solving Complex Tasks. In The Eleventh International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=_nGgzQjzaRy"},{"key":"e_1_3_2_2_22_1","unstructured":"Sandip Kulkarni and Alexandra Savelieva. 2025. The Path to a Golden Dataset or How to Evaluate Your RAG. https:\/\/medium.com\/data-science-at-microsoft\/the-path-to-a-golden-dataset-or-how-to-evaluate-your-rag-045e23d1f13f Accessed: 2025-02-10."},{"key":"e_1_3_2_2_23_1","unstructured":"LangChain. [n.d.]. LangChain: Building Applications with LLMs through Composability. https:\/\/github.com\/langchain-ai\/langchain."},{"key":"e_1_3_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1007\/3-540-49430-8_2"},{"key":"e_1_3_2_2_25_1","first-page":"9459","volume-title":"Lin(Eds.)","volume":"33","author":"Lewis Patrick","year":"2020","unstructured":"Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K\u00fcttler, Mike Lewis, Wen-tau Yih, Tim Rockt\u00e4schel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin(Eds.), Vol. 33. Curran Associates, Inc., 9459-9474. https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2020\/file\/6b493230205f780e1bc26945df7481e5-Paper.pdf"},{"key":"e_1_3_2_2_26_1","unstructured":"LlamaIndex. [n.d.]. LlamaIndex: Connecting LLMs with your Data. https:\/\/www.llamaindex.ai\/."},{"key":"e_1_3_2_2_27_1","unstructured":"Meta. 2024. Introducing Llama 3.1: Our most capable models to date. https:\/\/ai.meta.com\/blog\/meta-llama-3-1\/."},{"key":"e_1_3_2_2_28_1","unstructured":"Andrew Ng. [n.d.]. What's Next for AI Agentic Workflows ft. Andrew Ng of AI Fund. https:\/\/www.youtube.com\/watch?v=sal78ACtGTc&t=740s."},{"key":"e_1_3_2_2_29_1","unstructured":"Isaac Ong Amjad Almahairi Vincent Wu Wei-Lin Chiang Tianhao Wu Joseph E. Gonzalez M Waleed Kadous and Ion Stoica. 2024. RouteLLM: Learning to Route LLMs with Preference Data. arXiv:2406.18665 [cs.LG] https:\/\/arxiv.org\/abs\/2406.18665"},{"key":"e_1_3_2_2_30_1","unstructured":"OpenAI. 2024a. GPT-4o mini: Advancing Cost-Efficient Intelligence. https:\/\/openai.com\/index\/gpt-4o-mini-advancing-cost-efficient-intelligence\/ Accessed: 2025-02-10."},{"key":"e_1_3_2_2_31_1","unstructured":"OpenAI. 2024b. openai-python. https:\/\/github.com\/openai\/openai-python. Accessed: 2025-02-10."},{"key":"e_1_3_2_2_32_1","doi-asserted-by":"crossref","unstructured":"Krista Opsahl-Ong Michael J Ryan Josh Purtell David Broman Christopher Potts Matei Zaharia and Omar Khattab. 2024. Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs. arXiv:2406.11695 [cs.CL] https:\/\/arxiv.org\/abs\/2406.11695","DOI":"10.18653\/v1\/2024.emnlp-main.525"},{"key":"e_1_3_2_2_33_1","unstructured":"Hieu Pham Melody Y. Guan Barret Zoph Quoc V. Le and Jeff Dean. 2018. Efficient Neural Architecture Search via Parameter Sharing. arXiv:1802.03268 [cs.LG] https:\/\/arxiv.org\/abs\/1802.03268"},{"key":"e_1_3_2_2_34_1","volume-title":"Learning representations by back-propagating errors. nature","author":"Rumelhart David E","year":"1986","unstructured":"David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. 1986. Learning representations by back-propagating errors. nature, Vol. 323, 6088 (1986), 533-536."},{"key":"e_1_3_2_2_35_1","unstructured":"Salesforce. [n.d.]. Agentforce: Create Powerful AI Agents. https:\/\/www.salesforce.com\/agentforce\/. Accessed: 2025-01-20."},{"key":"e_1_3_2_2_36_1","volume-title":"Han Jin, Yuhang Yao, Salman Avestimehr, and Chaoyang He.","author":"Stripelis Dimitris","year":"2024","unstructured":"Dimitris Stripelis, Zijian Hu, Jipeng Zhang, Zhaozhuo Xu, Alay Dilipbhai Shah, Han Jin, Yuhang Yao, Salman Avestimehr, and Chaoyang He. 2024. TensorOpera Router: A Multi-Model Router for Efficient LLM Inference. arXiv preprint arXiv:2408.12320(2024)."},{"key":"e_1_3_2_2_37_1","unstructured":"Vellum AI. [n.d.]. Vellum: Streamlining Large Language Model Operations. https:\/\/www.vellum.ai\/."},{"key":"e_1_3_2_2_38_1","volume-title":"Proceedings of the 36th International Conference on Neural Information Processing Systems","author":"Wei Jason","year":"2024","unstructured":"Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2024. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems. New Orleans, LA."},{"key":"e_1_3_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4615-3618-5_2"},{"key":"e_1_3_2_2_40_1","volume-title":"The Twelfth International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=Bb4VGOWELI","author":"Yang Chengrun","year":"2024","unstructured":"Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. 2024a. Large Language Models as Optimizers. In The Twelfth International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=Bb4VGOWELI"},{"key":"e_1_3_2_2_41_1","doi-asserted-by":"crossref","unstructured":"Hongyang Yang Boyu Zhang Neng Wang Cheng Guo Xiaoli Zhang Likun Lin Junlin Wang Tianyu Zhou Mao Guan Runjia Zhang et al. 2024b. FinRobot: An Open-Source AI Agent Platform for Financial Applications using Large Language Models. arXiv preprint arXiv:2405.14767(2024).","DOI":"10.2139\/ssrn.4841493"},{"key":"e_1_3_2_2_42_1","doi-asserted-by":"crossref","unstructured":"Zhilin Yang Peng Qi Saizheng Zhang Yoshua Bengio William W Cohen Ruslan Salakhutdinov and Christopher D Manning. 2018. HotpotQA: A dataset for diverse explainable multi-hop question answering. arXiv preprint arXiv:1809.09600(2018).","DOI":"10.18653\/v1\/D18-1259"},{"key":"e_1_3_2_2_43_1","doi-asserted-by":"crossref","unstructured":"Zhiyu Yang Zihan Zhou Shuo Wang Xin Cong Xu Han Yukun Yan Zhenghao Liu Zhixing Tan Pengyuan Liu Dong Yu Zhiyuan Liu Xiaodong Shi and Maosong Sun. 2024c. MatPlotAgent: Method and Evaluation for LLM-Based Agentic Scientific Data Visualization. arXiv:2402.11453 [cs.CL]","DOI":"10.18653\/v1\/2024.findings-acl.701"},{"key":"e_1_3_2_2_44_1","volume-title":"Thirty-seventh Conference on Neural Information Processing Systems","author":"Yao Shunyu","year":"2023","unstructured":"Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023a. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. In Thirty-seventh Conference on Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine(Eds.), Vol. 36. New Orleans, Louisiana. https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2023\/file\/271db9922b8d1f4dd7aaef84ed5ac703-Paper-Conference.pdf"},{"key":"e_1_3_2_2_45_1","volume-title":"ReAct: Synergizing Reasoning and Acting in Language Models. In The Eleventh International Conference on Learning Representations. Kigali, Rwanda. https:\/\/openreview.net\/forum?id=WE_vluYUL-X","author":"Yao Shunyu","year":"2023","unstructured":"Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023b. ReAct: Synergizing Reasoning and Acting in Language Models. In The Eleventh International Conference on Learning Representations. Kigali, Rwanda. https:\/\/openreview.net\/forum?id=WE_vluYUL-X"},{"key":"e_1_3_2_2_46_1","unstructured":"Mert Yuksekgonul Federico Bianchi Joseph Boen Sheng Liu Zhi Huang Carlos Guestrin and James Zou. 2024. TextGrad: Automatic ''Differentiation'' via Text. (2024). arXiv:2406.07496"},{"key":"e_1_3_2_2_47_1","volume-title":"Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track","author":"Zheng Lianmin","year":"2023","unstructured":"Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track. New Orleans, Louisiana. https:\/\/openreview.net\/forum?id=uccHPGDlao"},{"key":"e_1_3_2_2_48_1","unstructured":"Wangchunshu Zhou Yixin Ou Shengwei Ding Long Li Jialong Wu Tiannan Wang Jiamin Chen Shuai Wang Xiaohua Xu Ningyu Zhang et al. 2024. Symbolic learning enables self-evolving agents. arXiv preprint arXiv:2406.18532(2024)."},{"key":"e_1_3_2_2_49_1","volume-title":"Proceedings of the 41st International Conference on Machine Learning(Proceedings of Machine Learning Research","volume":"62767","author":"Zhuge Mingchen","year":"2024","unstructured":"Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and J\u00fcrgen Schmidhuber. 2024. GPTSwarm: Language Agents as Optimizable Graphs. In Proceedings of the 41st International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol. 235), Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp(Eds.). PMLR, 62743-62767. https:\/\/proceedings.mlr.press\/v235\/zhuge24a.html"},{"key":"e_1_3_2_2_50_1","volume-title":"Proceedings of International Conference on Learning Representations.","author":"Zoph Barrett","year":"2017","unstructured":"Barrett Zoph and Quoc Le. 2017. Neural Architecture Search With Reinforcement Learning. In Proceedings of International Conference on Learning Representations."}],"event":{"name":"KDD '25: The 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining","location":"Toronto ON Canada","acronym":"KDD '25","sponsor":["SIGKDD ACM Special Interest Group on Knowledge Discovery in Data","SIGMOD ACM Special Interest Group on Management of Data"]},"container-title":["Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3711896.3736884","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,30]],"date-time":"2026-04-30T18:08:13Z","timestamp":1777572493000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3711896.3736884"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,8,3]]},"references-count":50,"alternative-id":["10.1145\/3711896.3736884","10.1145\/3711896"],"URL":"https:\/\/doi.org\/10.1145\/3711896.3736884","relation":{},"subject":[],"published":{"date-parts":[[2025,8,3]]},"assertion":[{"value":"2025-08-03","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}