{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,14]],"date-time":"2026-10-14T14:13:28Z","timestamp":1791987208971,"version":"4.4.0"},"publisher-location":"New York, NY, USA","reference-count":57,"publisher":"ACM","license":[{"start":{"date-parts":[[2024,7,10]],"date-time":"2024-07-10T00:00:00Z","timestamp":1720569600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,7,10]]},"DOI":"10.1145\/3626772.3657707","type":"proceedings-article","created":{"date-parts":[[2024,7,11]],"date-time":"2024-07-11T12:40:05Z","timestamp":1720701605000},"page":"1930-1940","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":138,"title":["Large Language Models can Accurately Predict Searcher Preferences"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2425-3136","authenticated-orcid":false,"given":"Paul","family":"Thomas","sequence":"first","affiliation":[{"name":"Microsoft, Adelaide, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5089-7632","authenticated-orcid":false,"given":"Seth","family":"Spielman","sequence":"additional","affiliation":[{"name":"Microsoft, Boulder, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9351-8137","authenticated-orcid":false,"given":"Nick","family":"Craswell","sequence":"additional","affiliation":[{"name":"Microsoft, Seattle, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5270-5550","authenticated-orcid":false,"given":"Bhaskar","family":"Mitra","sequence":"additional","affiliation":[{"name":"Microsoft Research, Montreal, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,7,11]]},"reference":[{"key":"e_1_3_2_1_1_1","volume-title":"Proceedings of the Third International Workshop on Recent Trends in News Information Retrieval. 9--14","author":"Agarwal Aashish","year":"2019","unstructured":"Aashish Agarwal, Ankita Mandal, Matthias Schaffeld, Fangzheng Ji, Jhiao Zhan, Yiqi Sun, and Ahmet Aker. 2019. Good, neutral or bad news classification. In Proceedings of the Third International Workshop on Recent Trends in News Information Retrieval. 9--14."},{"key":"e_1_3_2_1_2_1","volume-title":"Maria Korobeynikovo, and Fabrizio Gilardi.","author":"Alizadeh Meysam","year":"2023","unstructured":"Meysam Alizadeh, Ma\u00ebl Kubli, Zeynab Samei, Shirin Dehghani, Juan Diego Bermeo, Maria Korobeynikovo, and Fabrizio Gilardi. 2023. Open-source large language models outperform crowd workers and approach ChatGPT in text-annotation tasks. arXiv:2307.02179 [cs.CL]"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390447"},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3442188.3445922"},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.485"},{"key":"e_1_3_2_1_6_1","volume-title":"Proceedings of the ACM International Conference on Information and Knowledge Management. 85--94","author":"Bolotova Valeria","year":"2020","unstructured":"Valeria Bolotova, Vladislav Blinov, Yukun Zheng,WBruce Croft, Falk Scholer, and Mark Sanderson. 2020. Do people and neural nets pay attention to the samewords: studying eye-tracking data for non-factoid QA evaluation. In Proceedings of the ACM International Conference on Information and Knowledge Management. 85--94."},{"key":"e_1_3_2_1_7_1","volume-title":"Man is to computer programmer as woman is to homemaker? Debiasingword embeddings. Advances in neural information processing systems 29","author":"Bolukbasi Tolga","year":"2016","unstructured":"Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? Debiasingword embeddings. Advances in neural information processing systems 29 (2016)."},{"key":"e_1_3_2_1_8_1","unstructured":"Rishi Bommasani Drew A Hudson Ehsan Adeli Russ Altman Simran Arora Sydney von Arx Michael S Bernstein Jeannette Bohg Antoine Bosselut Emma Brunskill et al. 2021. On the opportunities and risks of foundation models. arXiv:2108.07258 [cs.LG]"},{"key":"e_1_3_2_1_9_1","volume-title":"ACM Sigir forum","author":"Broder Andrei","unstructured":"Andrei Broder. 2002. A taxonomy of web search. In ACM Sigir forum, Vol. 36. ACM New York, NY, USA, 3--10."},{"key":"e_1_3_2_1_10_1","volume-title":"Semantics derived automatically from language corpora contain human-like biases. Science 356, 6334","author":"Caliskan Aylin","year":"2017","unstructured":"Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science 356, 6334 (2017), 183--186."},{"key":"e_1_3_2_1_11_1","volume-title":"Proceedings of the European Conference on Information Retrieval. 16--27","author":"Carterette Ben","year":"2008","unstructured":"Ben Carterette, Paul N Bennett, David Maxwell Chickering, and Susan T Dumais. 2008. Here or there: Preference judgments for relevance. In Proceedings of the European Conference on Information Retrieval. 16--27."},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/1189702.1189703"},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.acl-long.870"},{"key":"e_1_3_2_1_14_1","volume-title":"Mizen","author":"Alec Chrystal K.","year":"2001","unstructured":"K. Alec Chrystal and Paul D. Mizen. 2001. Goodhart's law: Its origins, meaning and implications for monetary policy. Prepared for the Festschrift in honour of Charles Goodhart."},{"key":"e_1_3_2_1_15_1","unstructured":"Charles LAClarke Gianluca Demartini Laura Dietz Guglielmo Faggioli Matthias Hagen Claudia Hauff Noriko Kando Evangelos Kanoulas Martin Potthast Ian Soboroff Benno Stein and Henning Wachsmuth. 2023. HMC: A spectrum of human--machine-collaborative relevance judgment frameworks. In Frontiers of Information Access Experimentation for Research and Education Christine Bauer Ben Carterette Nicola Ferro and Norbert Fuhr (Eds.). Vol. 13. Leibniz-Zentrum f\u00fcr Informatik. Issue 1."},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/MIC.2012.95"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/290941.291009"},{"key":"e_1_3_2_1_18_1","volume-title":"Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval.","author":"Damessie Tadele T.","unstructured":"Tadele T. Damessie, Taho P. Nghiem, Falk Scholer, and J. Shane Culpepper. 2017. Gauging the quality of relevance assessments using inter-rater agreement. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval."},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3531146.3533234"},{"key":"e_1_3_2_1_20_1","volume-title":"Ways of knowing in HCI, Judith S","author":"Dumais Susan","unstructured":"Susan Dumais, Robin Jeffries, , Daniel M. Russell, Diane Tang, and Jaime Teevan. 2014. Understanding user behavior through log data and analysis. In Ways of knowing in HCI, Judith S. Olson andWendy A. Kellogg (Eds.). Springer, New York, 349--372."},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"crossref","unstructured":"Guglielmo Faggioli Laura Dietz Charles Clarke Gianluca Demartini Matthias Hagen Claudia Hauff Noriko Kando Evangelos Kanoulas Martin Potthast Benno Stein and Henning Wachsmuth. 2023. Perspectives on large language models for relevance judgment. arXiv:2304.09161 [cs.IR]","DOI":"10.1145\/3578337.3605136"},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"crossref","unstructured":"Fabrizio Gilardi Meysam Alizadeh and Ma\u00ebl Kubli. 2023. ChatGPT outperforms crowd-workers for text-annotation tasks. arXiv:2303.15056 [cs.CL]","DOI":"10.1073\/pnas.2305016120"},{"key":"e_1_3_2_1_23_1","volume-title":"Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 609--614","author":"Gonen Hila","year":"2019","unstructured":"Hila Gonen and Yoav Goldberg. 2019. Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 609--614."},{"key":"e_1_3_2_1_24_1","volume-title":"Papers in Monetary Economics.","author":"Goodhart Charles A E","unstructured":"Charles A E Goodhart. 1975. Problems of monetary management: The UK experience. In Papers in Monetary Economics. Vol. 1. Reserve Bank of Australia."},{"key":"e_1_3_2_1_25_1","volume-title":"Downloaded","author":"General Guidelines Google LLC.","year":"2023","unstructured":"Google LLC. 2022. General Guidelines. https:\/\/guidelines.raterhub.com\/ searchqualityevaluatorguidelines.pdf, Downloaded 29 July 2023.."},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4471-2099-5_20"},{"key":"e_1_3_2_1_27_1","unstructured":"Geoffrey Hinton Oriol Vinyals and Jeff Dean. 2015. Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation LearningWorkshop. http:\/\/arxiv.org\/abs\/1503.02531"},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401224"},{"key":"e_1_3_2_1_29_1","volume-title":"Accountability: Power, ethos and technologies of managing","author":"Hoskin Keith","unstructured":"Keith Hoskin. 1996. The ?awful' idea of accountability: Inscribing people into the measurement of objects. In Accountability: Power, ethos and technologies of managing, R Munro and J Mouritsen (Eds.). International Thompson Business Press, London."},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"crossref","unstructured":"Oana Inel Tim Draws and Lora Aroyo. 2023. Collect measure repeat: Reliability factors for responsible AI data collection. arXiv:2308.12885 [cs.LG]","DOI":"10.1609\/hcomp.v11i1.27547"},{"key":"e_1_3_2_1_31_1","unstructured":"Andrej Karpathy. 2023. State of GPT. Seminar at Microsoft Build. https: \/\/build.microsoft.com\/en-US\/sessions\/db3f4859-cd30--4445-a0cd-553c3304f8e2."},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-99739-7_18"},{"key":"e_1_3_2_1_33_1","first-page":"22199","article-title":"Large language models are zero-shot reasoners","volume":"35","author":"Kojima Takeshi","year":"2022","unstructured":"Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems, Vol. 35. 22199--22213.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_1_34_1","volume-title":"Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda.","author":"Liang Percy","year":"2022","unstructured":"Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, YuhuaiWu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, ChristopherD. Manning, Christopher R\u00e9, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, JueWang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. 2022. Holistic evaluation of language models. arXiv:2211.09110 [cs.CL]"},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1561\/1500000016"},{"key":"e_1_3_2_1_36_1","volume-title":"Ruochen Xu, and Chenguang Zhu.","author":"Liu Yang","year":"2023","unstructured":"Yang Liu, Dan Iter, Yichong xu amd ShuohangWang, Ruochen Xu, and Chenguang Zhu. 2023. G-EVAL: NLG evaluation using GPT-4 with better human alignment. arXiv:2303.16634v3 [cs.CL]"},{"key":"e_1_3_2_1_37_1","volume-title":"Algorithms of oppression","author":"Noble Safiya Umoja","unstructured":"Safiya Umoja Noble. 2018. Algorithms of oppression. New York University Press."},{"key":"e_1_3_2_1_38_1","unstructured":"OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL]"},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2022.3148714"},{"key":"e_1_3_2_1_40_1","unstructured":"David Patterson Joseph Gonzalez Quoc Le Chen Liang Lluis-Miquel Munguia Daniel Rothchild David So Maud Texier and Jeff Dean. 2021. Carbon emissions and large neural network training. (2021). arXiv:2104.10350 [cs.LG]"},{"key":"e_1_3_2_1_41_1","volume-title":"Chenguang Zhu, and Michael Zeng.","author":"Pryzant Reid","year":"2023","unstructured":"Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. 2023. Automatic prompt optimization with ?gradient descent\" and beam search. arXiv:2305.03495"},{"key":"e_1_3_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1561\/1500000009"},{"key":"e_1_3_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1353\/lib.0.0000"},{"key":"e_1_3_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/2484028.2484090"},{"key":"e_1_3_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/2447976.2447990"},{"key":"e_1_3_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3498366.3505815"},{"key":"e_1_3_2_1_47_1","volume-title":"Thomas and David Uminsky","author":"Rachel","year":"2022","unstructured":"Rachel L. Thomas and David Uminsky. 2022. Reliance on metrics is a fundamental challenge for AI. Patterns 3, 5 (2022)."},{"key":"e_1_3_2_1_48_1","unstructured":"Petter T\u00f6rnberg. 2023. ChatGPT-4 outperforms experts and crowdworkers in annotating politicalTwitter messages with zero-shot learning. arXiv:2304.06588 [cs.CL]"},{"key":"e_1_3_2_1_49_1","volume-title":"Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval. 315--323","year":"1998","unstructured":"EllenMVoorhees. 1998. Variations in relevance judgments and the measurement of retrieval effectiveness. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval. 315--323."},{"key":"e_1_3_2_1_50_1","volume-title":"Proceedings of the Text REtrieval Conference.","year":"2004","unstructured":"EllenMVoorhees. 2004. Overview of the TREC 2004 Robust Retrieval Track. In Proceedings of the Text REtrieval Conference."},{"key":"e_1_3_2_1_51_1","volume-title":"DavidWadden","author":"Wang Yizhong","year":"2023","unstructured":"Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, DavidWadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. 2023. How far can camels go? Exploring the state of instruction tuning on open resources. arXiv:2306.04751 [cs.CL]"},{"key":"e_1_3_2_1_52_1","article-title":"A similarity measure for indefinite rankings","volume":"28","author":"Moffat Alistair","year":"2010","unstructured":"WilliamWebber, Alistair Moffat, and Justin Zobel. 2010. A similarity measure for indefinite rankings. ACM Transactions on Information Systems 28, 4, Article 20 (Nov. 2010).","journal-title":"ACM Transactions on Information Systems"},{"key":"e_1_3_2_1_53_1","volume-title":"Chi, Quoc Le, and Denny Zhou","author":"Wei Jason","year":"2022","unstructured":"Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. arXiv:2201.11903 [cs.CL]"},{"key":"e_1_3_2_1_54_1","first-page":"795","article-title":"Sustainable AI: Environmental implications, challenges and opportunities","volume":"4","author":"Raghavendra Ramya","year":"2022","unstructured":"Carole-JeanWu, Ramya Raghavendra, Udit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng, Gloria Chang, Fiona Aga, Jinshi Huang, Charles Bai, et al. 2022. Sustainable AI: Environmental implications, challenges and opportunities. Proceedings of Machine Learning and Systems 4 (2022), 795--813.","journal-title":"Proceedings of Machine Learning and Systems"},{"key":"e_1_3_2_1_55_1","unstructured":"Chengrun Yang Xuezhi Wang Yifeng Lu Hanxiao Liu Quoc V. Le Denny Zhou and Xinyun Chen. 2023. Large language models as optimisers. arXiv:2309.03409 [cs.LG]"},{"key":"e_1_3_2_1_56_1","volume-title":"Gonzalez","author":"Zhang Tianjun","year":"2022","unstructured":"Tianjun Zhang, Xuezhi Wang, Denny Zhou, Dale Schuurmans, and Joseph E. Gonzalez. 2022. TEMPERA: Test-time prompt editing via reinforcement learning. arXiv:2211.11890 [cs.CL]"},{"key":"e_1_3_2_1_57_1","volume-title":"Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba.","author":"Zhou Yongchao","year":"2022","unstructured":"Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022. Large language models are human-level prompt engineers. arXiv:2211.01910 [cs.LG]"}],"event":{"name":"SIGIR 2024: The 47th International ACM SIGIR Conference on Research and Development in Information Retrieval","location":"Washington DC USA","acronym":"SIGIR 2024","sponsor":["SIGIR ACM Special Interest Group on Information Retrieval"]},"container-title":["Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3626772.3657707","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3626772.3657707","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,22]],"date-time":"2025-08-22T05:38:12Z","timestamp":1755841092000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3626772.3657707"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,10]]},"references-count":57,"alternative-id":["10.1145\/3626772.3657707","10.1145\/3626772"],"URL":"https:\/\/doi.org\/10.1145\/3626772.3657707","relation":{},"subject":[],"published":{"date-parts":[[2024,7,10]]},"assertion":[{"value":"2024-07-11","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}