{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,7]],"date-time":"2026-08-07T16:40:05Z","timestamp":1786120805871,"version":"build-2736575974"},"publisher-location":"New York, NY, USA","reference-count":40,"publisher":"ACM","license":[{"start":{"date-parts":[[2026,8,8]],"date-time":"2026-08-08T00:00:00Z","timestamp":1786147200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2026,8,9]]},"DOI":"10.1145\/3770855.3818476","type":"proceedings-article","created":{"date-parts":[[2026,8,6]],"date-time":"2026-08-06T17:15:43Z","timestamp":1786036543000},"page":"7555-7565","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["SAGE: Scalable AI Governance & Evaluation"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0009-0001-6096-478X","authenticated-orcid":false,"given":"Benjamin H.","family":"Le","sequence":"first","affiliation":[{"name":"LinkedIn Corporation, Mountain View, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9882-8783","authenticated-orcid":false,"given":"Xueying","family":"Lu","sequence":"additional","affiliation":[{"name":"LinkedIn Corporation, Mountain View, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-5423-9844","authenticated-orcid":false,"given":"Nicholas","family":"Stern","sequence":"additional","affiliation":[{"name":"LinkedIn Corporation, Mountain View, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-2771-0528","authenticated-orcid":false,"given":"Wenqiong","family":"Liu","sequence":"additional","affiliation":[{"name":"LinkedIn Corporation, Mountain View, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-2859-8043","authenticated-orcid":false,"given":"Igor","family":"Lapchuk","sequence":"additional","affiliation":[{"name":"LinkedIn Corporation, Mountain View, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-5422-9723","authenticated-orcid":false,"given":"Xiang","family":"Li","sequence":"additional","affiliation":[{"name":"LinkedIn Corporation, Mountain View, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-7494-2899","authenticated-orcid":false,"given":"Baofen","family":"Zheng","sequence":"additional","affiliation":[{"name":"LinkedIn Corporation, Mountain View, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-5039-5560","authenticated-orcid":false,"given":"Kevin","family":"Rosenberg","sequence":"additional","affiliation":[{"name":"LinkedIn Corporation, Mountain View, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-1679-2230","authenticated-orcid":false,"given":"Jiewen","family":"Huang","sequence":"additional","affiliation":[{"name":"LinkedIn Corporation, Mountain View, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4331-7887","authenticated-orcid":false,"given":"Zhe","family":"Zhang","sequence":"additional","affiliation":[{"name":"LinkedIn Corporation, Mountain View, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-8700-7321","authenticated-orcid":false,"given":"Abraham","family":"Cabangbang","sequence":"additional","affiliation":[{"name":"LinkedIn Corporation, Mountain View, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-9805-4191","authenticated-orcid":false,"given":"Satej Milind","family":"Wagle","sequence":"additional","affiliation":[{"name":"LinkedIn Corporation, Mountain View, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-5258-6523","authenticated-orcid":false,"given":"Jianqiang","family":"Shen","sequence":"additional","affiliation":[{"name":"LinkedIn Corporation, Mountain View, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-9652-1539","authenticated-orcid":false,"given":"Raghavan","family":"Muthuregunathan","sequence":"additional","affiliation":[{"name":"LinkedIn Corporation, Mountain View, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-5318-2518","authenticated-orcid":false,"given":"Abhinav","family":"Gupta","sequence":"additional","affiliation":[{"name":"LinkedIn Corporation, Mountain View, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-1266-8123","authenticated-orcid":false,"given":"Mathew","family":"Teoh","sequence":"additional","affiliation":[{"name":"LinkedIn Corporation, Mountain View, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-8343-7813","authenticated-orcid":false,"given":"Andrew J. N.","family":"Kirk","sequence":"additional","affiliation":[{"name":"LinkedIn Corporation, Mountain View, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-3720-5458","authenticated-orcid":false,"given":"Thomas","family":"Kwan","sequence":"additional","affiliation":[{"name":"LinkedIn Corporation, Mountain View, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-5751-9389","authenticated-orcid":false,"given":"Jingwei","family":"Wu","sequence":"additional","affiliation":[{"name":"LinkedIn Corporation, Mountain View, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-6510-1222","authenticated-orcid":false,"given":"Wenjing","family":"Zhang","sequence":"additional","affiliation":[{"name":"LinkedIn Corporation, Mountain View, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,8,8]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"Amazon. 2026. Amazon Mechanical Turk Pricing. https:\/\/requester.mturk.com\/pricing. Accessed: 2026-02-06."},{"key":"e_1_3_2_1_2_1","unstructured":"Rajat Arora Ye Tao Jianqiang Shen Ping Liu Muchen Wu Qianqi Shen Benjamin Le Fedor Borisyuk Jingwei Wu and Wenjing Zhang. 2026. High Fidelity Textual User Representation over Heterogeneous Sources via Reinforcement Learning. Preprint arXiv:2602.07333."},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2212.08073"},{"key":"e_1_3_2_1_4_1","unstructured":"Fedor Borisyuk Sriram Vasudevan Muchen Wu Guoyao Li Benjamin H. Le Shaobo Zhang Kay Qianqi Shen Yuchin Juan Kayhan Behdin Liming Dong Kaixu Yang Shusen Jing Ravi Teja Pothamsetty Rajat Arora Sophie Yanying Sheng Vitaly Abdrashitov Yang Zhao Lin Su Xiaoqing Wang Chujie Zheng Sarang Metkar Rupesh Gupta Igor Lapchuk David N. Racca Madhumitha Mohan Yanbo Li Haojun Li Saloni Gandhi Xueying Lu Chetan Bhole Ali Hooshmand Xin Yang Raghavan Muthuregunathan Jiajun Zhang Mathew Teoh Adam Coler Abhinav Gupta Xiaojing Ma Sundara Raman Ramachandran Morteza Ramezani Yubo Wang Lijuan Zhang Richard Li Jian Sheng Chanh Nguyen Yen-Chi Chen Chuanrui Zhu Claire Zhang Jiahao Xu Deepti Kulkarni Qing Lan Arvind Subramaniam Ata Fatahibaarzi Steven Shimizu Yanning Chen Zhipeng Wang Ran He Zhengze Zhou Qingquan Song Yun Dai Caleb Johnson Ping Liu Shaghayegh Gharghabi Gokulraj Mohanasundaram Juan Pablo Bottaro Santhosh Sachindran Qi Guo Yunxiang Ren Chengming Jiang Di Mo Luke Simon Jianqiang Shen Jingwei Wu and Wenjing Zhang. 2026. Semantic Search At LinkedIn. Preprint."},{"key":"e_1_3_2_1_5_1","volume-title":"Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM). 320-336","author":"Braun Bertil","year":"2025","unstructured":"Bertil Braun and Martin Forell. 2025. (Towards) Scalable Reliable Automated Evaluation with Large Language Models. In Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM). 320-336."},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2309.15217"},{"key":"e_1_3_2_1_7_1","volume-title":"Criteria-Based LLM Relevance Judgments. arXiv preprint arXiv:2507.09488","author":"Farzi Naghmeh","year":"2025","unstructured":"Naghmeh Farzi and Laura Dietz. 2025. Criteria-Based LLM Relevance Judgments. arXiv preprint arXiv:2507.09488 (2025)."},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-1107"},{"key":"e_1_3_2_1_9_1","unstructured":"Jiawei Gu Xuhui Jiang Zhichao Shi Hexiang Tan Xuehao Zhai Chengjin Xu Wei Li Yinghan Shen Shengjie Ma Honghao Liu et al. 2024. A survey on llm-as-a-judge. The Innovation (2024)."},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3726302.3731938"},{"key":"e_1_3_2_1_11_1","volume-title":"Distilling the Knowledge in a Neural Network. arXiv preprint arXiv:1503.02531","author":"Hinton Geoffrey","year":"2015","unstructured":"Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the Knowledge in a Neural Network. arXiv preprint arXiv:1503.02531 (2015)."},{"key":"e_1_3_2_1_12_1","volume-title":"Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685","author":"Hu Edward J","year":"2021","unstructured":"Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021)."},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3701716.3715265"},{"key":"e_1_3_2_1_14_1","volume-title":"V32, 1","author":"Hunter David R","year":"2004","unstructured":"David R Hunter. 2004. MM algorithms for generalized Bradley-Terry models. The annals of statistics, V32, 1 (2004), 384-406."},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/582415.582418"},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3018661.3018699"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3705328.3748116"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3109859.3109921"},{"key":"e_1_3_2_1_19_1","volume-title":"Proceedings of the Twelfth International Conference on Learning Representations (ICLR).","author":"Kim Seungone","year":"2024","unstructured":"Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. 2024. Prometheus: Inducing Fine-grained Evaluation Capability in Language Models. In Proceedings of the Twelfth International Conference on Learning Representations (ICLR)."},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.2307\/2529310"},{"key":"e_1_3_2_1_21_1","volume-title":"Curse of knowledge: When complex evaluation context benefits yet biases llm judges. arXiv preprint arXiv:2509.03419","author":"Li Weiyuan","year":"2025","unstructured":"Weiyuan Li, Xintao Wang, Siyu Yuan, Rui Xu, Jiangjie Chen, Qingqing Dong, Yanghua Xiao, and Deqing Yang. 2025. Curse of knowledge: When complex evaluation context benefits yet biases llm judges. arXiv preprint arXiv:2509.03419 (2025)."},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.emnlp-main.153"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3711896.3737273"},{"key":"e_1_3_2_1_24_1","unstructured":"Navid Mehrdad Hrushikesh Mohapatra Mossaab Bagdouri Prijith Chandran Alessandro Magnani Xunfan Cai Ajit Puthenputhussery Sachin Yadav Tony Lee ChengXiang Zhai and Ciya Liao. 2024. Large Language Models for Relevance Judgment in Product Search. arXiv:2406.00247 [cs.IR] https:\/\/arxiv.org\/abs\/2406.00247"},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/SC41405.2020.00024"},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.442"},{"key":"e_1_3_2_1_27_1","volume-title":"A Practical Guide for Evaluating LLMs and LLM-Reliant Systems. arXiv preprint arXiv:2506.13023","author":"Rudd Ethan M.","year":"2025","unstructured":"Ethan M. Rudd, Christopher Andrews, and Philip Tully. 2025. A Practical Guide for Evaluating LLMs and LLM-Reliant Systems. arXiv preprint arXiv:2506.13023 (2025)."},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3690624.3709413"},{"key":"e_1_3_2_1_29_1","volume-title":"Large Language Models Can Accurately Predict Searcher Preferences. arXiv preprint arXiv:2309.10621","author":"Thomas Paul","year":"2023","unstructured":"Paul Thomas, Seth Spielman, Nick Craswell, and Bhaskar Mitra. 2023. Large Language Models Can Accurately Predict Searcher Preferences. arXiv preprint arXiv:2309.10621 (2023)."},{"key":"e_1_3_2_1_30_1","volume-title":"Knowledge Distillation for Efficient and Effective Relevance Search on E-commerce. In SIGIR Workshop on eCommerce.","author":"Vo Nguyen","year":"2024","unstructured":"Nguyen Vo, Hongwei Shang, Zhen Yang, Juexin Lin, Seyed Danial Mohseni Taheri, and Changsung Kang. 2024. Knowledge Distillation for Efficient and Effective Relevance Search on E-commerce. In SIGIR Workshop on eCommerce."},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2305.17926"},{"key":"e_1_3_2_1_32_1","volume-title":"Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le.","author":"Wei Jason","year":"2021","unstructured":"Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652 (2021)."},{"key":"e_1_3_2_1_33_1","volume-title":"Denny Zhou, et al.","author":"Wei Jason","year":"2022","unstructured":"Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al., 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, V35 (2022), 24824-24837."},{"key":"e_1_3_2_1_34_1","volume-title":"AtC: Aggregate-then-Calibrate for Human-centered Assessment. In The Fourteenth International Conference on Learning Representations.","author":"Xie Zejun","unstructured":"Zejun Xie, Xintong Li, Guang Wang, and Desheng Zhang. [n.d.]. AtC: Aggregate-then-Calibrate for Human-centered Assessment. In The Fourteenth International Conference on Learning Representations."},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3711896.3737193"},{"key":"e_1_3_2_1_36_1","volume-title":"Flask: Fine-grained language model evaluation based on alignment skill sets. arXiv preprint arXiv:2307.10928","author":"Ye Seonghyeon","year":"2023","unstructured":"Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thorne, Juho Kim, and Minjoon Seo. 2023. Flask: Fine-grained language model evaluation based on alignment skill sets. arXiv preprint arXiv:2307.10928 (2023)."},{"key":"e_1_3_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939677"},{"key":"e_1_3_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3640457.3688075"},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2306.05685"},{"key":"e_1_3_2_1_40_1","volume-title":"LIMA: Less Is More for Alignment. Advances in Neural Information Processing Systems, V36","author":"Zhou Chunting","year":"2023","unstructured":"Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023. LIMA: Less Is More for Alignment. Advances in Neural Information Processing Systems, V36 (2023)."}],"event":{"name":"KDD '26: The 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining","location":"Jeju Island Republic of Korea","sponsor":["SIGMOD ACM Special Interest Group on Management of Data","SIGKDD ACM Special Interest Group on Knowledge Discovery in Data"]},"container-title":["Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2"],"original-title":[],"deposited":{"date-parts":[[2026,8,7]],"date-time":"2026-08-07T01:11:11Z","timestamp":1786065071000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3770855.3818476"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,8]]},"references-count":40,"alternative-id":["10.1145\/3770855.3818476","10.1145\/3770855"],"URL":"https:\/\/doi.org\/10.1145\/3770855.3818476","relation":{},"subject":[],"published":{"date-parts":[[2026,8,8]]},"assertion":[{"value":"2026-08-08","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}