{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,22]],"date-time":"2026-05-22T04:08:31Z","timestamp":1779422911548,"version":"3.53.1"},"publisher-location":"New York, NY, USA","reference-count":30,"publisher":"ACM","license":[{"start":{"date-parts":[[2026,5,26]],"date-time":"2026-05-26T00:00:00Z","timestamp":1779753600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2026,5,26]]},"DOI":"10.1145\/3786335.3813156","type":"proceedings-article","created":{"date-parts":[[2026,5,22]],"date-time":"2026-05-22T03:16:22Z","timestamp":1779419782000},"page":"689-701","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Benchmarking Agents in Insurance Underwriting Environments"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0009-0006-7011-1656","authenticated-orcid":false,"given":"Amanda","family":"Dsouza","sequence":"first","affiliation":[{"name":"Snorkel AI, Redwood City, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-3618-0906","authenticated-orcid":false,"given":"Ramya","family":"Ramakrishnan","sequence":"additional","affiliation":[{"name":"Snorkel AI, Redwood City, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0368-6859","authenticated-orcid":false,"given":"Charles","family":"Dickens","sequence":"additional","affiliation":[{"name":"Snorkel AI, Redwood City, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-6790-0925","authenticated-orcid":false,"given":"Bhavishya","family":"Pohani","sequence":"additional","affiliation":[{"name":"Snorkel AI, Redwood City, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2973-6762","authenticated-orcid":false,"given":"Christopher M","family":"Glaze","sequence":"additional","affiliation":[{"name":"Snorkel AI, Redwood City, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,5,26]]},"reference":[{"key":"e_1_3_3_2_2_2","unstructured":"Victor Barres Honghua Dong Soham Ray Xujie Si and Karthik Narasimhan. 2025. \u03c42-Bench: Evaluating Conversational Agents in a Dual-Control Environment. arxiv:https:\/\/arXiv.org\/abs\/2506.07982\u00a0[cs.AI] https:\/\/arxiv.org\/abs\/2506.07982"},{"key":"e_1_3_3_2_3_2","unstructured":"Mert Cemri Melissa\u00a0Z. Pan Shuyi Yang Lakshya\u00a0A. Agrawal Bhavya Chopra Rishabh Tiwari Kurt Keutzer Aditya Parameswaran Dan Klein Kannan Ramchandran Matei Zaharia Joseph\u00a0E. Gonzalez and Ion Stoica. 2025. Why Do Multi-Agent LLM Systems Fail? arxiv:https:\/\/arXiv.org\/abs\/2503.13657\u00a0[cs.AI] https:\/\/arxiv.org\/abs\/2503.13657"},{"key":"e_1_3_3_2_4_2","volume-title":"The Thirteenth International Conference on Learning Representations","author":"Chan Jun\u00a0Shern","year":"2025","unstructured":"Jun\u00a0Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Aleksander Madry, and Lilian Weng. 2025. MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering. In The Thirteenth International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=6s5uXNWGIh"},{"key":"e_1_3_3_2_5_2","unstructured":"De Chezelles Thibault Le\u00a0Sellier Sahar\u00a0Omidi Shayegan Lawrence\u00a0Keunho Jang Xing\u00a0Han L\u00f9 Ori Yoran Dehan Kong Frank\u00a0F Xu Siva Reddy Quentin Cappart et\u00a0al. 2024. The browsergym ecosystem for web agent research. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2412.05467 (2024)."},{"key":"e_1_3_3_2_6_2","unstructured":"Dylan Hutson Daniel Vennemeyer Aneesh Deshmukh Justin Zhan and Tianyu Jiang. 2025. GuessingGame: Measuring the Informativeness of Open-Ended Questions in Large Language Models. arxiv:https:\/\/arXiv.org\/abs\/2509.19593\u00a0[cs.CL] https:\/\/arxiv.org\/abs\/2509.19593"},{"key":"e_1_3_3_2_7_2","volume-title":"The Twelfth International Conference on Learning Representations","author":"Jimenez Carlos\u00a0E","year":"2024","unstructured":"Carlos\u00a0E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik\u00a0R Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-world Github Issues?. In The Twelfth International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=VTF8yNQM66"},{"key":"e_1_3_3_2_8_2","unstructured":"Sayash Kapoor Benedikt Stroebl Peter Kirgis Nitya Nadgir Zachary\u00a0S Siegel Boyi Wei Tianci Xue Ziru Chen Felix Chen Saiteja Utpala Franck Ndzomga Dheeraj Oruganty Sophie Luskin Kangheng Liu Botao Yu Amit Arora Dongyoon Hahm Harsh Trivedi Huan Sun Juyong Lee Tengjun Jin Yifan Mai Yifei Zhou Yuxuan Zhu Rishi Bommasani Daniel Kang Dawn Song Peter Henderson Yu Su Percy Liang and Arvind Narayanan. 2025. Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation. arxiv:https:\/\/arXiv.org\/abs\/2510.11977\u00a0[cs.AI] https:\/\/arxiv.org\/abs\/2510.11977"},{"key":"e_1_3_3_2_9_2","unstructured":"Sayash Kapoor Benedikt Stroebl Zachary\u00a0S Siegel Nitya Nadgir and Arvind Narayanan. 2025. AI Agents That Matter. Transactions on Machine Learning Research (2025). https:\/\/openreview.net\/forum?id=Zy4uFzMviZ"},{"key":"e_1_3_3_2_10_2","unstructured":"LangChain Labs. 2024. LangGraph. https:\/\/www.langchain.com\/langgraph. Framework for building LLM-powered agent workflows."},{"key":"e_1_3_3_2_11_2","unstructured":"Xiao Liu Hao Yu Hanchen Zhang Yifan Xu Xuanyu Lei Hanyu Lai Yu Gu Hangliang Ding Kaiwen Men Kejuan Yang et\u00a0al. 2023. Agentbench: Evaluating llms as agents. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2308.03688 (2023)."},{"key":"e_1_3_3_2_12_2","unstructured":"Davide Mazzaccara Alberto Testoni and Raffaella Bernardi. 2024. Learning to Ask Informative Questions: Enhancing LLMs with Preference Optimization and Expected Information Gain. arxiv:https:\/\/arXiv.org\/abs\/2406.17453\u00a0[cs.CL] https:\/\/arxiv.org\/abs\/2406.17453"},{"key":"e_1_3_3_2_13_2","unstructured":"Model Context Protocol Project. 2024. Model Context Protocol (MCP). https:\/\/modelcontextprotocol.io\/. Open-standard protocol for connecting large-language models to external tools and data sources."},{"key":"e_1_3_3_2_14_2","unstructured":"Atsunori Moteki Shoichi Masui Fan Yang Yueqi Song Yonatan Bisk Graham Neubig Ikuo Kusajima Yasuto Watanabe Hiroyuki Ishida Jun Takahashi and Shan Jiang. 2025. FieldWorkArena: Agentic AI Benchmark for Real Field Work Tasks. arxiv:https:\/\/arXiv.org\/abs\/2505.19662\u00a0[cs.AI] https:\/\/arxiv.org\/abs\/2505.19662"},{"key":"e_1_3_3_2_15_2","volume-title":"Forty-second International Conference on Machine Learning","author":"Patil Shishir\u00a0G","year":"2025","unstructured":"Shishir\u00a0G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph\u00a0E. Gonzalez. 2025. The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models. In Forty-second International Conference on Machine Learning. https:\/\/openreview.net\/forum?id=2GmDdhBdDk"},{"key":"e_1_3_3_2_16_2","doi-asserted-by":"crossref","unstructured":"Tejal Patwardhan Rachel Dias Elizabeth Proehl Grace Kim Michele Wang Olivia Watkins Sim\u00f3n\u00a0Posada Fishman Marwan Aljubeh Phoebe Thacker Laurance Fauconnet Natalie\u00a0S. Kim Patrick Chao Samuel Miserendino Gildas Chabot David Li Michael Sharman Alexandra Barr Amelia Glaese and Jerry Tworek. 2025. GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks. arxiv:https:\/\/arXiv.org\/abs\/2510.04374\u00a0[cs.LG] https:\/\/arxiv.org\/abs\/2510.04374","DOI":"10.70777\/si.v2i4.17197"},{"key":"e_1_3_3_2_17_2","volume-title":"Forty-second International Conference on Machine Learning","author":"Starace Giulio","year":"2025","unstructured":"Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun\u00a0Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. 2025. PaperBench: Evaluating AI\u2019s Ability to Replicate AI Research. In Forty-second International Conference on Machine Learning. https:\/\/openreview.net\/forum?id=xF5PuTLPbn"},{"key":"e_1_3_3_2_18_2","unstructured":"The Terminal-Bench Team. 2025. Terminal-Bench: A Benchmark for AI Agents in Terminal Environments. https:\/\/github.com\/laude-institute\/terminal-bench"},{"key":"e_1_3_3_2_19_2","unstructured":"Weixuan Wang Dongge Han Daniel\u00a0Madrigal Diaz Jin Xu Victor R\u00fchle and Saravan Rajmohan. 2025. OdysseyBench: Evaluating LLM Agents on Long-Horizon Complex Office Application Workflows. arxiv:https:\/\/arXiv.org\/abs\/2508.09124\u00a0[cs.CL] https:\/\/arxiv.org\/abs\/2508.09124"},{"key":"e_1_3_3_2_20_2","unstructured":"Frank\u00a0F Xu Yufan Song Boxuan Li Yuxuan Tang Kritanjali Jain Mengxue Bao Zora\u00a0Z Wang Xuhui Zhou Zhitong Guo Murong Cao et\u00a0al. 2024. Theagentcompany: benchmarking llm agents on consequential real world tasks. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2412.14161 (2024)."},{"key":"e_1_3_3_2_21_2","unstructured":"Ran Xu Yuchen Zhuang Yishan Zhong Yue Yu Zifeng Wang Xiangru Tang Hang Wu May\u00a0D. Wang Peifeng Ruan Donghan Yang Tao Wang Guanghua Xiao Xin Liu Carl Yang Yang Xie and Wenqi Shi. 2025. MedAgentGym: A Scalable Agentic Training Environment for Code-Centric Reasoning in Biomedical Data Science. arxiv:https:\/\/arXiv.org\/abs\/2506.04405\u00a0[cs.CL] https:\/\/arxiv.org\/abs\/2506.04405"},{"key":"e_1_3_3_2_22_2","unstructured":"Shunyu Yao Noah Shinn Pedram Razavi and Karthik Narasimhan. 2024. \u03c4 -bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arxiv:https:\/\/arXiv.org\/abs\/2406.12045\u00a0[cs.AI] https:\/\/arxiv.org\/abs\/2406.12045"},{"key":"e_1_3_3_2_23_2","volume-title":"The Thirteenth International Conference on Learning Representations","author":"Yao Shunyu","year":"2025","unstructured":"Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik\u00a0R Narasimhan. 2025. {$\\tau$}-bench: A Benchmark for \\underline{T}ool-\\underline{A}gent-\\underline{U}ser Interaction in Real-World Domains. In The Thirteenth International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=roNSXZpUDN"},{"key":"e_1_3_3_2_24_2","unstructured":"Shunyu Yao Jeffrey Zhao Dian Yu Nan Du Izhak Shafran Karthik Narasimhan and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. arxiv:https:\/\/arXiv.org\/abs\/2210.03629\u00a0[cs.CL] https:\/\/arxiv.org\/abs\/2210.03629"},{"key":"e_1_3_3_2_25_2","unstructured":"Asaf Yehudai Lilach Eden Alan Li Guy Uziel Yilun Zhao Roy Bar-Haim Arman Cohan and Michal Shmueli-Scheuer. 2025. Survey on Evaluation of LLM-based Agents. arxiv:https:\/\/arXiv.org\/abs\/2503.16416\u00a0[cs.AI] https:\/\/arxiv.org\/abs\/2503.16416"},{"key":"e_1_3_3_2_26_2","unstructured":"Zhaojian Yu Yilun Zhao Arman Cohan and Xiao-Ping Zhang. 2024. Humaneval pro and mbpp pro: Evaluating large language models on self-invoking code generation. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2412.21199 (2024)."},{"key":"e_1_3_3_2_27_2","unstructured":"Guibin Zhang Hejia Geng Xiaohang Yu Zhenfei Yin Zaibin Zhang Zelin Tan Heng Zhou Zhongzhi Li Xiangyuan Xue Yijiang Li Yifan Zhou Yang Chen Chen Zhang Yutao Fan Zihu Wang Songtao Huang Yue Liao Hongru Wang Mengyue Yang Heng Ji Michael Littman Jun Wang Shuicheng Yan Philip Torr and Lei Bai. 2025. The Landscape of Agentic Reinforcement Learning for LLMs: A Survey. arxiv:https:\/\/arXiv.org\/abs\/2509.02547\u00a0[cs.AI] https:\/\/arxiv.org\/abs\/2509.02547"},{"key":"e_1_3_3_2_28_2","unstructured":"Yizhe Zhang Jiarui Lu and Navdeep Jaitly. 2024. Probing the Multi-turn Planning Capabilities of LLMs via 20 Question Games. arxiv:https:\/\/arXiv.org\/abs\/2310.01468\u00a0[cs.CL] https:\/\/arxiv.org\/abs\/2310.01468"},{"key":"e_1_3_3_2_29_2","unstructured":"Shuyan Zhou Frank\u00a0F Xu Hao Zhu Xuhui Zhou Robert Lo Abishek Sridhar Xianyi Cheng Tianyue Ou Yonatan Bisk Daniel Fried et\u00a0al. 2023. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2307.13854 (2023)."},{"key":"e_1_3_3_2_30_2","unstructured":"Yuxuan Zhu Tengjun Jin Yada Pruksachatkun Andy Zhang Shu Liu Sasha Cui Sayash Kapoor Shayne Longpre Kevin Meng Rebecca Weiss Fazl Barez Rahul Gupta Jwala Dhamala Jacob Merizian Mario Giulianelli Harry Coppock Cozmin Ududec Jasjeet Sekhon Jacob Steinhardt Antony Kellermann Sarah Schwettmann Matei Zaharia Ion Stoica Percy Liang and Daniel Kang. 2025. Establishing Best Practices for Building Rigorous Agentic Benchmarks. arxiv:https:\/\/arXiv.org\/abs\/2507.02825\u00a0[cs.AI] https:\/\/arxiv.org\/abs\/2507.02825"},{"key":"e_1_3_3_2_31_2","unstructured":"Terry\u00a0Yue Zhuo Minh\u00a0Chien Vu Jenny Chim Han Hu Wenhao Yu Ratnadira Widyasari Imam Nur\u00a0Bani Yusuf Haolan Zhan Junda He Indraneil Paul et\u00a0al. 2024. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:https:\/\/arXiv.org\/abs\/2406.15877 (2024)."}],"event":{"name":"CAIS '26: ACM Conference on AI and Agentic Systems","location":"San Jose CA USA","acronym":"CAIS '26"},"container-title":["Proceedings of the ACM Conference on AI and Agentic Systems"],"original-title":[],"deposited":{"date-parts":[[2026,5,22]],"date-time":"2026-05-22T03:36:07Z","timestamp":1779420967000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3786335.3813156"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,5,26]]},"references-count":30,"alternative-id":["10.1145\/3786335.3813156","10.1145\/3786335"],"URL":"https:\/\/doi.org\/10.1145\/3786335.3813156","relation":{},"subject":[],"published":{"date-parts":[[2026,5,26]]},"assertion":[{"value":"2026-05-26","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}