{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,15]],"date-time":"2026-07-15T18:07:32Z","timestamp":1784138852494,"version":"3.55.0"},"publisher-location":"New York, NY, USA","reference-count":46,"publisher":"ACM","license":[{"start":{"date-parts":[[2026,7,19]],"date-time":"2026-07-19T00:00:00Z","timestamp":1784419200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2026,7,20]]},"DOI":"10.1145\/3805712.3809580","type":"proceedings-article","created":{"date-parts":[[2026,7,10]],"date-time":"2026-07-10T14:28:19Z","timestamp":1783693699000},"page":"324-335","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Learning to Rank with Multi-Criteria LLM-Judge Annotations"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0009-0000-3297-8888","authenticated-orcid":false,"given":"Naghmeh","family":"Farzi","sequence":"first","affiliation":[{"name":"University of New Hampshire, Durham, NH, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1624-3907","authenticated-orcid":false,"given":"Laura","family":"Dietz","sequence":"additional","affiliation":[{"name":"University of New Hampshire, Durham, NH, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,7,19]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"Zahra Abbasiantaeb Chuan Meng Leif Azzopardi and Mohammad Aliannejadi. 2024. Can We Use Large Language Models to Fill Relevance Judgment Holes?. In Joint Proceedings of the 1st Workshop on Evaluation Methodologies Testbeds and Community for Information Access Research (EMTCIR 2024) and the 1st Workshop on User Modelling in Conversational Information Retrieval (UM-CIR 2024) co-located with SIGIR-AP 2024 (CEUR Workshop Proceedings Vol. 3854). CEUR-WS.org. https:\/\/ceur-ws.org\/Vol-3854\/emtcir-2.pdf"},{"key":"e_1_3_2_1_2_1","volume-title":"Clarke","author":"Arabzadeh Negar","year":"2024","unstructured":"Negar Arabzadeh and Charles L. A. Clarke. 2024. A Comparison of Methods for Evaluating Generative IR. arXiv:2404.04044 [cs.IR] https:\/\/arxiv.org\/abs\/2404.04044"},{"key":"e_1_3_2_1_3_1","first-page":"23","article-title":"From ranknet to lambdarank to lambdamart: An overview","volume":"11","author":"Burges Christopher JC","year":"2010","unstructured":"Christopher JC Burges. 2010. From ranknet to lambdarank to lambdamart: An overview. Learning, Vol. 11, 23-581 (2010), 81.","journal-title":"Learning"},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/1273496.1273513"},{"key":"e_1_3_2_1_5_1","volume-title":"Efficient algorithms for ranking with SVMs. Information retrieval","author":"Chapelle Olivier","year":"2010","unstructured":"Olivier Chapelle and S Sathiya Keerthi. 2010. Efficient algorithms for ranking with SVMs. Information retrieval, Vol. 13, 3 (2010), 201-215."},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.20736\/0002002105"},{"key":"e_1_3_2_1_7_1","volume-title":"Overview of the TREC 2020 deep learning track. arXiv preprint arXiv:2102","author":"Craswell Nick","year":"2021","unstructured":"Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. 2021. Overview of the TREC 2020 deep learning track. arXiv preprint arXiv:2102.07662 (2021)."},{"key":"e_1_3_2_1_8_1","volume-title":"Overview of the TREC 2021 deep learning track. arXiv:2507","author":"Craswell Nick","year":"2025","unstructured":"Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Jimmy Lin. 2025a. Overview of the TREC 2021 deep learning track. arXiv:2507.08191 [cs.IR] https:\/\/arxiv.org\/abs\/2507.08191"},{"key":"e_1_3_2_1_9_1","volume-title":"Overview of the TREC 2019 deep learning track. arXiv preprint arXiv:2003","author":"Craswell Nick","year":"2020","unstructured":"Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M Voorhees. 2020. Overview of the TREC 2019 deep learning track. arXiv preprint arXiv:2003.07820 (2020)."},{"key":"e_1_3_2_1_10_1","volume-title":"Overview of the TREC 2023 deep learning track. arXiv:2507","author":"Craswell Nick","year":"2025","unstructured":"Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Hossein A. Rahmani, Daniel Campos, Jimmy Lin, Ellen M. Voorhees, and Ian Soboroff. 2025b. Overview of the TREC 2023 deep learning track. arXiv:2507.08890 [cs.IR] https:\/\/arxiv.org\/abs\/2507.08890"},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3773966.3779397"},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3331184.3331257"},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-032-21300-6_20"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"crossref","unstructured":"Laura Dietz Bryan Li Eugene Yang Dawn Lawrie William Walden and James Mayfield. 2026 b. Insider Knowledge: How Much Can RAG Systems Gain from Evaluation Secrets? arXiv:2601.13227 [cs.IR] https:\/\/arxiv.org\/abs\/2601.13227","DOI":"10.1007\/978-3-032-21289-4_37"},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3731120.3744588"},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.findings-emnlp.950"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3578337.3605136"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3664190.3672511"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3731120.3744591"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3726302.3730317"},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3767695.3769508"},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/1150402.1150429"},{"key":"e_1_3_2_1_23_1","unstructured":"Carlos Lassance Ronak Pradeep and Jimmy Lin. 2023. Naverloo @ TREC Deep Learning and Neuclir 2023: As Easy as Zero One Two Three - Cascading Dual Encoders Mono Duo and Listo for Ad-Hoc Retrieval. https:\/\/openreview.net\/forum?id=2S3jl2JkQU"},{"key":"e_1_3_2_1_24_1","first-page":"225","volume-title":"Foundations and Trends\u00ae in Information Retrieval","volume":"3","author":"Tie-Yan","year":"2009","unstructured":"Tie-Yan Liu et al., 2009. Learning to rank for information retrieval. Foundations and Trends\u00ae in Information Retrieval, Vol. 3, 3 (2009), 225-331."},{"key":"e_1_3_2_1_25_1","unstructured":"Xueguang Ma Xinyu Zhang Ronak Pradeep and Jimmy Lin. 2023. Zero-Shot Listwise Document Reranking with a Large Language Model. arXiv:2305.02156 [cs.IR] https:\/\/arxiv.org\/abs\/2305.02156"},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3539618.3592032"},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3626772.3657846"},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-006-9019-z"},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3726302.3730119"},{"key":"e_1_3_2_1_30_1","unstructured":"Ronak Pradeep Sahel Sharifymoghaddam and Jimmy Lin. 2023a. RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Models. arXiv:2309.15088 [cs.IR] https:\/\/arxiv.org\/abs\/2309.15088"},{"key":"e_1_3_2_1_31_1","unstructured":"Ronak Pradeep Sahel Sharifymoghaddam and Jimmy Lin. 2023b. RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze! arXiv:2312.02724 [cs.IR] https:\/\/arxiv.org\/abs\/2312.02724"},{"key":"e_1_3_2_1_32_1","volume-title":"Initial Nugget Evaluation Results for the TREC 2024 RAG Track with the AutoNuggetizer Framework. arXiv:2411","author":"Pradeep Ronak","year":"2024","unstructured":"Ronak Pradeep, Nandan Thakur, Shivani Upadhyay, Daniel Campos, Nick Craswell, and Jimmy Lin. 2024. Initial Nugget Evaluation Results for the TREC 2024 RAG Track with the AutoNuggetizer Framework. arXiv:2411.09607 [cs.IR] https:\/\/arxiv.org\/abs\/2411.09607"},{"key":"e_1_3_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.findings-naacl.97"},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3626772.3657942"},{"key":"e_1_3_2_1_35_1","volume-title":"Proceedings of the First Workshop on Large Language Models for Evaluation in Information Retrieval (LLM4Eval 2024) co-located with SIGIR 2024 (CEUR Workshop Proceedings","volume":"3","author":"Rahmani Hossein A.","year":"2024","unstructured":"Hossein A. Rahmani, Emine Yilmaz, Nick Craswell, Bhaskar Mitra, Paul Thomas, Charles L. A. Clarke, Mohammad Aliannejadi, Clemencia Siro, and Guglielmo Faggioli. 2024b. LLMJudge: LLMs for Relevance Judgments. In Proceedings of the First Workshop on Large Language Models for Evaluation in Information Retrieval (LLM4Eval 2024) co-located with SIGIR 2024 (CEUR Workshop Proceedings, Vol. 3752). CEUR-WS.org, 1-3. https:\/\/ceur-ws.org\/Vol-3752\/paper8.pdf"},{"key":"e_1_3_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-88708-6_15"},{"key":"e_1_3_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.emnlp-main.249"},{"key":"e_1_3_2_1_38_1","volume-title":"Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agent. ArXiv","author":"Sun Weiwei","year":"2023","unstructured":"Weiwei Sun, Lingyong Yan, Xinyu Ma, Pengjie Ren, Dawei Yin, and Zhaochun Ren. 2023. Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agent. ArXiv, Vol. abs\/2304.09542 (2023). https:\/\/api.semanticscholar.org\/CorpusID:258212638"},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2487575.2487630"},{"key":"e_1_3_2_1_40_1","volume-title":"Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. https:\/\/api.semanticscholar.org\/CorpusID:262054847","author":"Thomas Paul","year":"2023","unstructured":"Paul Thomas, Seth Spielman, Nick Craswell, and Bhaskar Mitra. 2023. Large language models can accurately predict searcher preferences. In Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. https:\/\/api.semanticscholar.org\/CorpusID:262054847"},{"key":"e_1_3_2_1_41_1","volume-title":"Hoa Trang Dang, and Jimmy Lin","author":"Upadhyay Shivani","year":"2024","unstructured":"Shivani Upadhyay, Ronak Pradeep, Nandan Thakur, Daniel Campos, Nick Craswell, Ian Soboroff, Hoa Trang Dang, and Jimmy Lin. 2024a. A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial Look. arXiv:2411.08275 [cs.IR] https:\/\/arxiv.org\/abs\/2411.08275"},{"key":"e_1_3_2_1_42_1","volume-title":"UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor. arXiv:2406.06519 [cs.IR] https:\/\/arxiv.org\/abs\/2406.06519","author":"Upadhyay Shivani","year":"2024","unstructured":"Shivani Upadhyay, Ronak Pradeep, Nandan Thakur, Nick Craswell, and Jimmy Lin. 2024b. UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor. arXiv:2406.06519 [cs.IR] https:\/\/arxiv.org\/abs\/2406.06519"},{"key":"e_1_3_2_1_43_1","volume-title":"Yu Hou, Dawn Lawrie, James Mayfield, and Eugene Yang.","author":"Walden William","year":"2025","unstructured":"William Walden, Marc Mason, Orion Weller, Laura Dietz, John Conroy, Neil Molino, Hannah Recknor, Bryan Li, Gabrielle Kaili-May Liu, Yu Hou, Dawn Lawrie, James Mayfield, and Eugene Yang. 2025. Auto-ARGUE: LLM-Based Report Generation Evaluation. arXiv:2509.26184 [cs.IR] https:\/\/arxiv.org\/abs\/2509.26184"},{"key":"e_1_3_2_1_44_1","unstructured":"Yidong Wang Yunze Song Tingyuan Zhu Xuanwang Zhang Zhuohao Yu Hao Chen Chiyu Song Qiufeng Wang Cunxiang Wang Zhen Wu Xinyu Dai Yue Zhang Wei Ye and Shikun Zhang. 2025. TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them. arXiv:2509.21117 [cs.AI] https:\/\/arxiv.org\/abs\/2509.21117"},{"key":"e_1_3_2_1_45_1","unstructured":"Lulu Yu Keping Bi Jiafeng Guo Shihao Liu Shuaiqiang Wang Dawei Yin and Xueqi Cheng. 2025. Can LLM Annotations Replace User Clicks for Learning to Rank? arXiv:2511.06635 [cs.IR] https:\/\/arxiv.org\/abs\/2511.06635"},{"key":"e_1_3_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3626772.3657813"}],"event":{"name":"SIGIR '26: The 49th International ACM SIGIR Conference on Research and Development in Information Retrieval","location":"Melbourne VIC Australia","sponsor":["SIGIR ACM Special Interest Group on Information Retrieval"]},"container-title":["Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval"],"original-title":[],"deposited":{"date-parts":[[2026,7,15]],"date-time":"2026-07-15T17:26:54Z","timestamp":1784136414000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3805712.3809580"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,19]]},"references-count":46,"alternative-id":["10.1145\/3805712.3809580","10.1145\/3805712"],"URL":"https:\/\/doi.org\/10.1145\/3805712.3809580","relation":{},"subject":[],"published":{"date-parts":[[2026,7,19]]},"assertion":[{"value":"2026-07-19","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}