{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,25]],"date-time":"2026-03-25T00:45:58Z","timestamp":1774399558111,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":13,"publisher":"ACM","funder":[{"DOI":"10.13039\/501100006374","name":"National Science Foundation","doi-asserted-by":"publisher","award":["1846017"],"award-info":[{"award-number":["1846017"]}],"id":[{"id":"10.13039\/501100006374","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2025,7,13]]},"DOI":"10.1145\/3726302.3730317","type":"proceedings-article","created":{"date-parts":[[2025,7,14]],"date-time":"2025-07-14T01:38:52Z","timestamp":1752457132000},"page":"3214-3222","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":5,"title":["Does UMBRELA work on other LLMs?"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0009-0000-3297-8888","authenticated-orcid":false,"given":"Naghmeh","family":"Farzi","sequence":"first","affiliation":[{"name":"University of New Hampshire, Durham, NH, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1624-3907","authenticated-orcid":false,"given":"Laura","family":"Dietz","sequence":"additional","affiliation":[{"name":"University of New Hampshire, Durham, NH, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2025,7,13]]},"reference":[{"key":"e_1_3_2_1_1_1","volume-title":"Clarke and Laura Dietz. Llm-based relevance assessment still can't replace human relevance assessment","author":"Charles L.","year":"2024","unstructured":"Charles L. A. Clarke and Laura Dietz. Llm-based relevance assessment still can't replace human relevance assessment, 2024."},{"key":"e_1_3_2_1_2_1","volume-title":"Overview of the trec 2020 deep learning track. arXiv preprint arXiv:2102.07662","author":"Craswell Nick","year":"2021","unstructured":"Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. Overview of the trec 2020 deep learning track. arXiv preprint arXiv:2102.07662, 2021."},{"key":"e_1_3_2_1_3_1","volume-title":"Overview of the trec 2019 deep learning track. arXiv preprint arXiv:2003.07820","author":"Craswell Nick","year":"2020","unstructured":"Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M Voorhees. Overview of the trec 2019 deep learning track. arXiv preprint arXiv:2003.07820, 2020."},{"key":"e_1_3_2_1_4_1","unstructured":"DeepSeek-AI Aixin Liu Bei Feng Bing Xue Bing-Li Wang Bochao Wu Chengda Lu Chenggang Zhao Chengqi Deng Chenyu Zhang Chong Ruan Damai Dai Daya Guo Dejian Yang Deli Chen Dong-Li Ji Erhang Li Fangyun Lin Fucong Dai Fuli Luo Guangbo Hao Guanting Chen Guowei Li H. Zhang Han Bao Hanwei Xu Haocheng Wang Haowei Zhang Honghui Ding Huajian Xin Huazuo Gao Hui Li Hui Qu J. L. Cai Jian Liang Jianzhong Guo Jiaqi Ni Jiashi Li Jiawei Wang Jin Chen Jingchang Chen Jingyang Yuan Junjie Qiu Junlong Li Jun-Mei Song Kai Dong Kai Hu Kaige Gao Kang Guan Kexin Huang Kuai Yu Lean Wang Lecong Zhang Lei Xu Leyi Xia Liang Zhao Litong Wang Liyue Zhang Meng Li Miaojun Wang Mingchuan Zhang Minghua Zhang Minghui Tang Mingming Li Ning Tian Panpan Huang Peiyi Wang Peng Zhang Qiancheng Wang Qihao Zhu Qinyu Chen Qiushi Du R. J. Chen R. L. Jin Ruiqi Ge Ruisong Zhang Ruizhe Pan Runji Wang Runxin Xu Ruoyu Zhang Ruyi Chen S. S. Li Shanghao Lu Shangyan Zhou Shanhuang Chen Shao-Ping Wu Shengfeng Ye Shirong Ma Shiyu Wang Shuang Zhou Shuiping Yu Shunfeng Zhou Shuting Pan T. Wang Tao Yun Tian Pei Tianyu Sun W. L. Xiao Wangding Zeng Wanjia Zhao Wei An Wen Liu Wenfeng Liang Wenjun Gao Wen-Xuan Yu Wentao Zhang X. Q. Li Xiangyu Jin Xianzu Wang Xiaoling Bi Xiaodong Liu Xiaohan Wang Xi-Cheng Shen Xiaokang Chen Xiaokang Zhang Xiaosha Chen Xiaotao Nie Xiaowen Sun Xiaoxiang Wang Xin Cheng Xin Liu Xin Xie Xingchao Liu Xingkai Yu Xinnan Song Xinxia Shan Xinyi Zhou Xinyu Yang Xinyuan Li Xuecheng Su Xuheng Lin Y. K. Li Y. Q. Wang Y. X. Wei Y. X. Zhu Yang Zhang Yanhong Xu Yanping Huang Yao Li Yao Zhao Yaofeng Sun Yao Li Yaohui Wang Yi Yu Yi Zheng Yichao Zhang Yifan Shi Yi Xiong Ying He Ying Tang Yishi Piao Yisong Wang Yixuan Tan Yi-Bing Ma Yiyuan Liu Yongqiang Guo Yu Wu Yuan Ou Yuchen Zhu Yuduan Wang Yue Gong Yuheng Zou Yujia He Yukun Zha Yunfan Xiong Yunxiang Ma Yuting Yan Yu-Wei Luo Yu mei You Yuxuan Liu Yuyang Zhou Z. F. Wu Zehui Ren Zehui Ren Zhangli Sha Zhe Fu Zhean Xu Zhen Huang Zhen Zhang Zhenda Xie Zhen guo Zhang Zhewen Hao Zhibin Gou Zhicheng Ma Zhigang Yan Zhihong Shao Zhipeng Xu Zhiyu Wu Zhongyu Zhang Zhuoshu Li Zihui Gu Zijia Zhu Zijun Liu Zi-An Li Ziwei Xie Ziyang Song Ziyi Gao and Zizheng Pan. Deepseek-v3 technical report. ArXiv abs\/2412.19437 2024."},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3578337.3605136"},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3664190.3672511"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/30.1-2.81"},{"key":"e_1_3_2_1_8_1","volume-title":"Gpt-4o system card","author":"AI.","year":"2024","unstructured":"OpenAI. Gpt-4o system card, 2024. arXiv:2410.21276."},{"key":"e_1_3_2_1_9_1","volume-title":"August","author":"Rahmani Hossein A.","year":"2024","unstructured":"Hossein A. Rahmani, Emine Yilmaz, Nick Craswell, Bhaskar Mitra, Paul Thomas, Charles L. A. Clarke, Mohammad Aliannejadi, Clemencia Siro, and Guglielmo Faggioli. LLMJudge: LLMs for Relevance Judgments, August 2024. arXiv:2408.08896."},{"key":"e_1_3_2_1_10_1","volume-title":"Is chatgpt good at search? investigating large language models as re-ranking agent. ArXiv, abs\/2304.09542","author":"Sun Weiwei","year":"2023","unstructured":"Weiwei Sun, Lingyong Yan, Xinyu Ma, Pengjie Ren, Dawei Yin, and Zhaochun Ren. Is chatgpt good at search? investigating large language models as re-ranking agent. ArXiv, abs\/2304.09542, 2023."},{"key":"e_1_3_2_1_11_1","volume-title":"Llm-assisted relevance assessments: When should we ask llms for help? arXiv preprint arXiv:2411.06877","author":"Takehi Rikiya","year":"2024","unstructured":"Rikiya Takehi, Ellen M Voorhees, Tetsuya Sakai, and Ian Soboroff. Llm-assisted relevance assessments: When should we ask llms for help? arXiv preprint arXiv:2411.06877, 2024."},{"key":"e_1_3_2_1_12_1","volume-title":"Annual International ACM SIGIR Conference on Research and Development in Information Retrieval","author":"Thomas Paul","year":"2023","unstructured":"Paul Thomas, Seth Spielman, Nick Craswell, and Bhaskar Mitra. Large language models can accurately predict searcher preferences. In Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 2023."},{"key":"e_1_3_2_1_13_1","volume-title":"Umbrela: Umbrela is the (open-source reproduction of the) bing relevance assessor","author":"Upadhyay Shivani","year":"2024","unstructured":"Shivani Upadhyay, Ronak Pradeep, Nandan Thakur, Nick Craswell, and Jimmy Lin. Umbrela: Umbrela is the (open-source reproduction of the) bing relevance assessor, 2024. graphy"}],"event":{"name":"SIGIR '25: The 48th International ACM SIGIR Conference on Research and Development in Information Retrieval","location":"Padua Italy","acronym":"SIGIR '25","sponsor":["SIGIR ACM Special Interest Group on Information Retrieval"]},"container-title":["Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3726302.3730317","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,22]],"date-time":"2025-08-22T12:11:17Z","timestamp":1755864677000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3726302.3730317"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,13]]},"references-count":13,"alternative-id":["10.1145\/3726302.3730317","10.1145\/3726302"],"URL":"https:\/\/doi.org\/10.1145\/3726302.3730317","relation":{},"subject":[],"published":{"date-parts":[[2025,7,13]]},"assertion":[{"value":"2025-07-13","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}