{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,15]],"date-time":"2026-07-15T18:04:48Z","timestamp":1784138688482,"version":"3.55.0"},"publisher-location":"New York, NY, USA","reference-count":48,"publisher":"ACM","license":[{"start":{"date-parts":[[2026,7,19]],"date-time":"2026-07-19T00:00:00Z","timestamp":1784419200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2026,7,20]]},"DOI":"10.1145\/3805712.3809713","type":"proceedings-article","created":{"date-parts":[[2026,7,15]],"date-time":"2026-07-15T17:06:26Z","timestamp":1784135186000},"page":"415-425","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Topic-Specific Classifiers are Better Relevance Judges than Prompted LLMs"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5707-3751","authenticated-orcid":false,"given":"Lukas","family":"Gienapp","sequence":"first","affiliation":[{"name":"University of Kassel, Kassel, Germany, hessian.AI, Darmstadt, Germany, and ScaDS.AI, Leipzig, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2451-0665","authenticated-orcid":false,"given":"Martin","family":"Potthast","sequence":"additional","affiliation":[{"name":"University of Kassel, Kassel, Germany, hessian.AI, Darmstadt, Germany, and ScaDS.AI, Leipzig, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5970-880X","authenticated-orcid":false,"given":"Andrew","family":"Yates","sequence":"additional","affiliation":[{"name":"HLTCOE, Johns Hopkins University, Baltimore, MD, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9578-7157","authenticated-orcid":false,"given":"Harrisen","family":"Scells","sequence":"additional","affiliation":[{"name":"University of T\u00fcbingen, T\u00fcbingen, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0051-1535","authenticated-orcid":false,"given":"Eugene","family":"Yang","sequence":"additional","affiliation":[{"name":"HLTCOE, Johns Hopkins University, Baltimore, MD, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,7,19]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1108\/eb050097"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3269206.3271766"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2911473"},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3726302.3730308"},{"key":"e_1_3_2_1_5_1","unstructured":"Ellen M. Voorhees Ian Soboroff and Jimmy Lin. 2022. Can old TREC collections reliably evaluate modern neural retrieval models? CoRR abs\/2201.11086. https:\/\/arxiv.org\/abs\/2201.11086."},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-88708-6_13"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3731120.3744605"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3726302.3730159"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","unstructured":"Hossein A. Rahmani Clemencia Siro Mohammad Aliannejadi Nick Craswell Charles L. A. Clarke Guglielmo Faggioli Bhaskar Mitra Paul Thomas and Emine Yilmaz. 2025. Judging the judges: A collection of llm-generated relevance judgements. CoRR abs\/2502.13908. https:\/\/doi.org\/10.48550\/arXiv.2502.13908.","DOI":"10.48550\/arXiv.2502.13908"},{"key":"e_1_3_2_1_10_1","unstructured":"Zahra Abbasiantaeb Chuan Meng Leif Azzopardi and Mohammad Aliannejadi. 2024. Can we use large language models to fill relevance judgment holes? In Joint Proceedings of the 1st Workshop on Evaluation Methodologies Testbeds and Community for Information Access Research (EMTCIR 2024) and the 1st Workshop on User Modelling in Conversational Information Retrieval (UM-CIR 2024) co-located with the 2nd International ACM SIGIR Conference on Information Retrieval in the Asia Pacific (SIGIR-AP 2024) Tokyo Japan December 12 2024 (CEUR Workshop Proceedings). CEUR-WS.org. https:\/\/ceur-ws.org\/Vol-3854\/emtcir-2.pdf."},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","unstructured":"Shivani Upadhyay Ronak Pradeep Nandan Thakur Nick Craswell and Jimmy Lin. 2024. UMBRELA: umbrela is the (open-source reproduction of the) bing relevance assessor. CoRR abs\/2406.06519. https:\/\/doi.org\/10.48550\/arXiv.2406.06519.","DOI":"10.48550\/arXiv.2406.06519"},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.54195\/IRRJ.19625"},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3731120.3744588"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.52202\/079017-2197"},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","unstructured":"Koki Wataoka Tsubasa Takahashi and Ryokan Ri. 2024. Self-preference bias in llm-as-a-judge. CoRR abs\/2410.21819. https:\/\/doi.org\/10.48550\/arXiv.2410.21819.","DOI":"10.48550\/arXiv.2410.21819"},{"key":"e_1_3_2_1_16_1","volume-title":"The Thirteenth International Conference on Learning Representations, ICLR 2025","author":"Ye Jiayi","year":"2025","unstructured":"Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, Nitesh V. Chawla, and Xiangliang Zhang. 2025. Justice or prejudice? quantifying biases in llm-as-ajudge. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. https:\/\/openreview.net\/forum?id=3GTtZFiajM."},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3637528.3671458"},{"key":"e_1_3_2_1_18_1","volume-title":"Proceedings of the Thirteenth Text REtrieval Conference, TREC 2004","author":"Voorhees Ellen M.","year":"2004","unstructured":"Ellen M. Voorhees. 2004.Overviewof the TREC 2004 robust track. In Proceedings of the Thirteenth Text REtrieval Conference, TREC 2004, Gaithersburg, Maryland, USA, November 16-19, 2004 (NIST Special Publication). National Institute of Standards and Technology (NIST). http:\/\/trec.nist.gov\/pubs\/trec13\/papers\/ROBUST.OVERVIEW.pdf."},{"key":"e_1_3_2_1_19_1","volume-title":"Proceedings of the Twenty-Eighth Text REtrieval Conference, TREC 2019","author":"Craswell Nick","year":"2019","unstructured":"Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. 2019.Overview of the TREC 2019 deep learning track. In Proceedings of the Twenty-Eighth Text REtrieval Conference, TREC 2019, Gaithersburg, Maryland, November 13-15, 2019 (NIST Special Publication). National Institute of Standards and Technology (NIST). https:\/\/trec.nist.gov\/pubs\/trec28\/papers\/OVERVIEW.DL.pdf."},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.6028\/NIST.SP.1266.deep-overview"},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1002\/ASI.4630270302"},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4471-2099-5_23"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4471-2099-5_1"},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","unstructured":"Chuan Meng Jiqun Liu Mohammad Aliannejadi Fengran Mo Jeff Dalton and Maarten de Rijke. 2026. Re-rankers as relevance judges. (2026). https:\/\/doi.org\/10.48550\/arXiv.2601.04455.","DOI":"10.48550\/arXiv.2601.04455"},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/775047.775067"},{"key":"e_1_3_2_1_26_1","volume-title":"The TREC 2002 filtering track report. In Proc. of TREC (NIST Special Publication).","volume":"251","author":"Stephen","unstructured":"Stephen E. Robertson and Ian Soboroff. 2002. The TREC 2002 filtering track report. In Proc. of TREC (NIST Special Publication). Vol. 500-251. NIST. http:\/\/trec.nist.gov\/pubs\/trec11\/papers\/OVER.FILTERING.pdf."},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/383952.383961"},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/1277741.1277806"},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/1148170.1148219"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3726302.3729916"},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/2022.EMNLP-MAIN.414"},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2600428.2609601"},{"key":"e_1_3_2_1_33_1","volume-title":"TREC 2016 total recall track overview. In Proceedings of The Twenty-Fifth Text REtrieval Conference, TREC 2016","volume":"321","author":"Grossman Maura R.","year":"2016","unstructured":"Maura R. Grossman, Gordon V. Cormack, and Adam Roegiest. 2016. TREC 2016 total recall track overview. In Proceedings of The Twenty-Fifth Text REtrieval Conference, TREC 2016, Gaithersburg, Maryland, USA, November 15-18, 2016 NIST Special Publication). Vol. 500-321. NIST. http:\/\/trec.nist.gov\/pubs\/trec25\/papers\/Overview-TR.pdf."},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/2022.EMNLP-MAIN.414"},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","unstructured":"Ronak Pradeep Sahel Sharifymoghaddam and Jimmy Lin. 2023. Rankzephyr: effective and robust zero-shot listwise reranking is a breeze! CoRR abs\/2312.02724. https:\/\/doi.org\/10.48550\/arXiv.2312.02724.","DOI":"10.48550\/arXiv.2312.02724"},{"key":"e_1_3_2_1_36_1","doi-asserted-by":"publisher","unstructured":"Ronak Pradeep Sahel Sharifymoghaddam and Jimmy Lin. 2023. Rankvicuna: zero-shot listwise document reranking with open-source large language models. CoRR abs\/2309.15088. https:\/\/doi.org\/10.48550\/arXiv.2309.15088.","DOI":"10.48550\/arXiv.2309.15088"},{"key":"e_1_3_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3701716.3715536"},{"key":"e_1_3_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3477495.3531863"},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3701716.3715311"},{"key":"e_1_3_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3626772.3657942"},{"key":"e_1_3_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/1277741.1277755"},{"key":"e_1_3_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/2020.FINDINGS-EMNLP.63"},{"key":"e_1_3_2_1_43_1","volume-title":"The Tenth International Conference on Learning Representations, ICLR 2022","author":"Hu Edward J.","year":"2022","unstructured":"Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net. https:\/\/openreview.net\/forum?id=nZeVKeeFYf9."},{"key":"e_1_3_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.18653\/V1\/D19-1410"},{"key":"e_1_3_2_1_45_1","unstructured":"Tri Nguyen Mir Rosenberg Xia Song Jianfeng Gao Saurabh Tiwary Rangan Majumder and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In Proceedings of the Workshop on Cognitive Computation: Integrating neural and symbolic approaches 2016 co-located with the 30th Annual Conference on Neural Information Processing Systems (NIPS 2016) Barcelona Spain December 9 2016 (CEUR Workshop Proceedings). CEURWS.org. https:\/\/ceur-ws.org\/Vol-1773\/CoCoNIPS_2016_paper9.pdf."},{"key":"e_1_3_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1162\/coli.07-034-R2"},{"key":"e_1_3_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3578337.3605136"},{"key":"e_1_3_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390435"}],"event":{"name":"SIGIR '26: The 49th International ACM SIGIR Conference on Research and Development in Information Retrieval","location":"Melbourne VIC Australia","sponsor":["SIGIR ACM Special Interest Group on Information Retrieval"]},"container-title":["Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval"],"original-title":[],"deposited":{"date-parts":[[2026,7,15]],"date-time":"2026-07-15T17:08:28Z","timestamp":1784135308000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3805712.3809713"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,19]]},"references-count":48,"alternative-id":["10.1145\/3805712.3809713","10.1145\/3805712"],"URL":"https:\/\/doi.org\/10.1145\/3805712.3809713","relation":{},"subject":[],"published":{"date-parts":[[2026,7,19]]},"assertion":[{"value":"2026-07-19","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}