{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,25]],"date-time":"2026-03-25T01:02:42Z","timestamp":1774400562088,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":33,"publisher":"ACM","license":[{"start":{"date-parts":[[2024,12,8]],"date-time":"2024-12-08T00:00:00Z","timestamp":1733616000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100006374","name":"Australian Research Council","doi-asserted-by":"publisher","award":["DP190101113"],"award-info":[{"award-number":["DP190101113"]}],"id":[{"id":"10.13039\/501100006374","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,12,8]]},"DOI":"10.1145\/3673791.3698431","type":"proceedings-article","created":{"date-parts":[[2024,12,8]],"date-time":"2024-12-08T06:24:16Z","timestamp":1733639056000},"page":"32-41","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":29,"title":["LLMs can be Fooled into Labelling a Document as Relevant: best caf\u00e9 near me; this paper is perfectly relevant"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0008-8650","authenticated-orcid":false,"given":"Marwah","family":"Alaofi","sequence":"first","affiliation":[{"name":"RMIT University, Melbourne, Australia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2425-3136","authenticated-orcid":false,"given":"Paul","family":"Thomas","sequence":"additional","affiliation":[{"name":"Microsoft, Adelaide, Australia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9094-0810","authenticated-orcid":false,"given":"Falk","family":"Scholer","sequence":"additional","affiliation":[{"name":"RMIT University, Melbourne, Australia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0487-9609","authenticated-orcid":false,"given":"Mark","family":"Sanderson","sequence":"additional","affiliation":[{"name":"RMIT University, Melbourne, Australia"}]}],"member":"320","published-online":{"date-parts":[[2024,12,8]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"Zahra Abbasiantaeb Chuan Meng Leif Azzopardi and Mohammad Aliannejadi. 2024. Can We Use Large Language Models to Fill Relevance Judgment Holes? arxiv: 2405.05600 [cs.IR] https:\/\/arxiv.org\/abs\/2405.05600"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390447"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/1099554.1099733"},{"key":"e_1_3_2_1_4_1","volume-title":"A coefficient of agreement for nominal scales. Educational and psychological measurement","author":"Cohen Jacob","year":"1960","unstructured":"Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, Vol. 20, 1 (1960), 37--46."},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/290941.291009"},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.6028\/NIST.SP.500-335.deep-overview"},{"key":"e_1_3_2_1_7_1","volume-title":"Proceedings of the Thirty-First Text REtrieval Conference, TREC 2022","author":"Craswell Nick","year":"2022","unstructured":"Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, Jimmy Lin, Ellen M. Voorhees, and Ian Soboroff. 2022. Overview of the TREC 2022 Deep Learning Track. In Proceedings of the Thirty-First Text REtrieval Conference, TREC 2022, online, November 15--19, 2022 (NIST Special Publication, Vol. 500--338), Ian Soboroff and Angela Ellis (Eds.). National Institute of Standards and Technology (NIST). https:\/\/trec.nist.gov\/pubs\/trec31\/papers\/Overview_deep.pdf"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3077136.3080729"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3578337.3605136"},{"key":"e_1_3_2_1_10_1","volume-title":"Trec-6 ad-hoc retrieval. NIST SPECIAL PUBLICATION SP","author":"Franz Martin","year":"1998","unstructured":"Martin Franz and Salim Roukos. 1998. Trec-6 ad-hoc retrieval. NIST SPECIAL PUBLICATION SP (1998), 511--516."},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3336191.3371857"},{"key":"e_1_3_2_1_12_1","volume-title":"OHSUMED: An Interactive Retrieval Evaluation and New Large Test Collection for Research. In SIGIR '94, Bruce W","author":"Hersh William","year":"1994","unstructured":"William Hersh, Chris Buckley, T. J. Leone, and David Hickam. 1994. OHSUMED: An Interactive Retrieval Evaluation and New Large Test Collection for Research. In SIGIR '94, Bruce W. Croft and C. J. van Rijsbergen (Eds.). Springer London, London, 192--201."},{"key":"e_1_3_2_1_13_1","volume-title":"Scaling Laws for Neural Language Models. CoRR","author":"Kaplan Jared","year":"2020","unstructured":"Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling Laws for Neural Language Models. CoRR, Vol. abs\/2001.08361 (2020). arXiv:2001.08361 https:\/\/arxiv.org\/abs\/2001.08361"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401075"},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/1458082.1458160"},{"key":"e_1_3_2_1_16_1","unstructured":"Klaus Krippendorff. 2011. Computing Krippendorff's alpha-reliability. (2011)."},{"key":"e_1_3_2_1_17_1","volume-title":"Mary Lois Marckworth, Laura M Bell, and John Bissell Carroll.","author":"Ku\u010dera Henry","year":"1967","unstructured":"Henry Ku\u010dera, Winthrop Francis, William Freeman Twaddell, Mary Lois Marckworth, Laura M Bell, and John Bissell Carroll. 1967. Computational analysis of present-day American English. International Journal of American Linguistics (1967)."},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3269206.3271764"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3404835.3463238"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3539618.3592032"},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3409256.3409829"},{"key":"e_1_3_2_1_22_1","unstructured":"Tri Nguyen Mir Rosenberg Xia Song Jianfeng Gao Saurabh Tiwary Rangan Majumder and Li Deng. 2016. MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. In Proceedings of the Workshop on Cognitive Computation: Integrating neural and symbolic approaches 2016 co-located with the 30th Annual Conference on Neural Information Processing Systems (NIPS 2016) Barcelona Spain December 9 2016 (CEUR Workshop Proceedings Vol. 1773) Tarek Richard Besold Antoine Bordes Artur S. d'Avila Garcez and Greg Wayne (Eds.). CEUR-WS.org. https:\/\/ceur-ws.org\/Vol-1773\/CoCoNIPS_2016_paper9.pdf"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.findings-emnlp.63"},{"key":"e_1_3_2_1_24_1","volume-title":"Passage Re-ranking with BERT. CoRR","author":"Nogueira Rodrigo Frassetto","year":"2019","unstructured":"Rodrigo Frassetto Nogueira and Kyunghyun Cho. 2019. Passage Re-ranking with BERT. CoRR, Vol. abs\/1901.04085 (2019). arXiv:1901.04085 http:\/\/arxiv.org\/abs\/1901.04085"},{"key":"e_1_3_2_1_25_1","volume-title":"Document Expansion by Query Prediction. CoRR","author":"Nogueira Rodrigo Frassetto","year":"2019","unstructured":"Rodrigo Frassetto Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. 2019. Document Expansion by Query Prediction. CoRR, Vol. abs\/1904.08375 (2019). arXiv:1904.08375 http:\/\/arxiv.org\/abs\/1904.08375"},{"key":"e_1_3_2_1_26_1","volume-title":"Australasian Document Computing Symposium. https:\/\/api.semanticscholar.org\/CorpusID:14426189","author":"Sanderson Mark","year":"2010","unstructured":"Mark Sanderson, Falk Scholer, and Andrew Turpin. 2010. Relatively relevant: Assessor shift in document judgements. In Australasian Document Computing Symposium. https:\/\/api.semanticscholar.org\/CorpusID:14426189"},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2009916.2010057"},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3498366.3505815"},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3626772.3657707"},{"key":"e_1_3_2_1_30_1","unstructured":"Shivani Upadhyay Ehsan Kamalloo and Jimmy Lin. 2024. LLMs Can Patch Up Missing Relevance Judgments in Evaluation. arxiv: 2405.04727 [cs.IR] https:\/\/arxiv.org\/abs\/2405.04727"},{"key":"e_1_3_2_1_31_1","volume-title":"UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor. arxiv: 2406.06519 [cs.IR] https:\/\/arxiv.org\/abs\/2406.06519","author":"Upadhyay Shivani","year":"2024","unstructured":"Shivani Upadhyay, Ronak Pradeep, Nandan Thakur, Nick Craswell, and Jimmy Lin. 2024. UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor. arxiv: 2406.06519 [cs.IR] https:\/\/arxiv.org\/abs\/2406.06519"},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.6028\/NIST.SP.500--261"},{"key":"e_1_3_2_1_33_1","unstructured":"Lee Xiong Chenyan Xiong Ye Li Kwok-Fung Tang Jialin Liu Paul Bennett Junaid Ahmed and Arnold Overwijk. 2021. Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval. https:\/\/openreview.net\/forum?id=zeFrfgyZln"}],"event":{"name":"SIGIR-AP 2024: Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region","location":"Tokyo Japan","acronym":"SIGIR-AP 2024","sponsor":["SIGIR ACM Special Interest Group on Information Retrieval"]},"container-title":["Proceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3673791.3698431","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3673791.3698431","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,22]],"date-time":"2025-08-22T16:23:00Z","timestamp":1755879780000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3673791.3698431"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,8]]},"references-count":33,"alternative-id":["10.1145\/3673791.3698431","10.1145\/3673791"],"URL":"https:\/\/doi.org\/10.1145\/3673791.3698431","relation":{},"subject":[],"published":{"date-parts":[[2024,12,8]]},"assertion":[{"value":"2024-12-08","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}