{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,3]],"date-time":"2026-11-03T13:33:48Z","timestamp":1793712828798,"version":"4.6.1"},"reference-count":65,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2020,12,31]],"date-time":"2020-12-31T00:00:00Z","timestamp":1609372800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2021,4,30]]},"abstract":"<jats:p>We examine the \u201cgoodness\u201d of ranked retrieval evaluation measures in terms of how well they align with users\u2019 Search Engine Result Page (SERP) preferences for web search. The SERP preferences cover 1,127 topic-SERP-SERP triplets extracted from the NTCIR-9 INTENT task, reflecting the views of 15 different assessors. Each assessor made two SERP preference judgements for each triplet: one in terms of relevance and the other in terms of diversity. For each evaluation measure, we compute the Agreement Rate (AR) of each triplet: the proportion of assessors that agree with the measure\u2019s SERP preference. We then compare the mean ARs of the measures as well as those of best\/median\/worst assessors using Tukey HSD tests. Our first experiment compares traditional ranked retrieval measures based on the SERP relevance preferences: we find that normalised Discounted Cumulative Gain (nDCG) and intentwise Rank-biased Utility (iRBU) perform best in that they are the only measures that are statistically indistinguishable from our best assessor; nDCG also statistically significantly outperforms our median assessor. Our second experiment utilises 119,646 document preferences that we collected for a subset of the above topic-SERP-SERP triplets (containing 894 triplets) to compare preference-based evaluation measures as well as traditional ones. Again, we evaluate them based on the SERP relevance preferences. The results suggest that measures such as wpref5 are the most promising among the preference-based measures considered, although they underperform the best traditional measures such as nDCG on average. Our third experiment compares diversified search measures based on the SERP diversity preferences as well as the SERP relevance preferences, and it shows that D\u266f-measures are clearly the most reliable: in particular, D\u266f-nDCG and D\u266f-RBP statistically significantly outperform the median assessor and all intent-aware measures; they also outperform the recently proposed RBU on average. Also, in terms of agreement with SERP diversity preferences, D\u266f-nDCG statistically significantly outperforms RBU. Hence, if IR researchers want to use evaluation measures that align well with users\u2019 SERP preferences, then we recommend nDCG and iRBU for traditional search, and D\u266f-measures such as D\u266f-nDCG for diversified search. As for document preference-based measures that we have examined, we do not have a strong reason to recommended them over traditional measures like nDCG, since they align slightly less well with users\u2019 SERP preferences despite their quadratic assessment cost.<\/jats:p>","DOI":"10.1145\/3431813","type":"journal-article","created":{"date-parts":[[2020,12,31]],"date-time":"2020-12-31T23:08:59Z","timestamp":1609456139000},"page":"1-35","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":23,"title":["Retrieval Evaluation Measures that Agree with Users\u2019 SERP Preferences"],"prefix":"10.1145","volume":"39","author":[{"given":"Tetsuya","family":"Sakai","sequence":"first","affiliation":[{"name":"Waseda University, Okubo, Shinjuku, Tokyo, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhaohao","family":"Zeng","sequence":"additional","affiliation":[{"name":"Waseda University, Okubo, Shinjuku, Tokyo, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2020,12,31]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/1498759.1498824"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/1498759.1498766"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390347"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3291992.3291996"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2484028.2484081"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3210024"},{"key":"e_1_2_1_7_1","volume-title":"Proceedings of ADCS","author":"Bailey Peter","year":"2007","unstructured":"Peter Bailey , Paul Thomas , and David Hawking . 2007 . Does brandname influence perceived search result quality? Yahoo!, Google, and WebKumara . In Proceedings of ADCS 2007. Peter Bailey, Paul Thomas, and David Hawking. 2007. Does brandname influence perceived search result quality? Yahoo!, Google, and WebKumara. In Proceedings of ADCS 2007."},{"key":"e_1_2_1_8_1","volume-title":"Proceedings of ACM SIGIR","author":"Bashir Maryam","year":"2013","unstructured":"Maryam Bashir , Jesse Anderton , Jie Wu , Peter B. Golbus , Virgil Pavlu , and Javed A. Aslam . 2013. A document rating system for preference judgements . In Proceedings of ACM SIGIR 2013 . 909--912. Maryam Bashir, Jesse Anderton, Jie Wu, Peter B. Golbus, Virgil Pavlu, and Javed A. Aslam. 2013. A document rating system for preference judgements. In Proceedings of ACM SIGIR 2013. 909--912."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/792550.792552"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/1102351.1102363"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2009916.2010037"},{"key":"e_1_2_1_12_1","volume-title":"Proceedings of ACM SIGIR","author":"Carterette Ben","year":"2008","unstructured":"Ben Carterette and Paul N. Bennett . 2008. Evaluation measures for preference judgments . In Proceedings of ACM SIGIR 2008 . 685--686. Ben Carterette and Paul N. Bennett. 2008. Evaluation measures for preference judgments. In Proceedings of ACM SIGIR 2008. 685--686."},{"key":"e_1_2_1_13_1","volume-title":"Proceedings of ECIR 2008 (LNCS 4956)","author":"Carterette Ben","unstructured":"Ben Carterette , Paul N. Bennett , David Maxwell Chickering , and Susan T. Dumais . 2008. Here or there: Preference judgments for relevance . In Proceedings of ECIR 2008 (LNCS 4956) . 16--27. Ben Carterette, Paul N. Bennett, David Maxwell Chickering, and Susan T. Dumais. 2008. Here or there: Preference judgments for relevance. In Proceedings of ECIR 2008 (LNCS 4956). 16--27."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/2348283.2348398"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2484028.2484094"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.5555\/2070786.2070817"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/1645953.1646033"},{"key":"e_1_2_1_18_1","volume-title":"Proceedings of ACM SIGIR","author":"Chuklin Aleksandr","year":"2013","unstructured":"Aleksandr Chuklin , Ke Zhou , Anne Schuth , Floor Sietsma , and Maarten de Rijke . 2013 . Evaluating intuitiveness of vertica-aware click models . In Proceedings of ACM SIGIR 2013. 1075--1078. Aleksandr Chuklin, Ke Zhou, Anne Schuth, Floor Sietsma, and Maarten de Rijke. 2013. Evaluating intuitiveness of vertica-aware click models. In Proceedings of ACM SIGIR 2013. 1075--1078."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/1935826.1935847"},{"key":"e_1_2_1_20_1","volume-title":"Overview of the TREC 2012 web track. In Proceedings of TREC 2012.","author":"Clarke Charles L. A.","unstructured":"Charles L. A. Clarke , Nick Craswell , and Ellen M. Voorhees . 2013 . Overview of the TREC 2012 web track. In Proceedings of TREC 2012. Charles L. A. Clarke, Nick Craswell, and Ellen M. Voorhees. 2013. Overview of the TREC 2012 web track. In Proceedings of TREC 2012."},{"key":"e_1_2_1_21_1","volume-title":"Proceedings of ACM SIGIR","author":"Clarke Charles L. A.","year":"2009","unstructured":"Charles L. A. Clarke , Maheedhar Kolla , Gordon V. Cormack , Olga Vechtomova , Azin Ashkan , Stefan B\u00fcttcher , and Ian MacKinnon . 2009 . Novelty and diversity in information retrieval evaluation . In Proceedings of ACM SIGIR 2008. 659--666. Charles L. A. Clarke, Maheedhar Kolla, Gordon V. Cormack, Olga Vechtomova, Azin Ashkan, Stefan B\u00fcttcher, and Ian MacKinnon. 2009. Novelty and diversity in information retrieval evaluation. In Proceedings of ACM SIGIR 2008. 659--666."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-04417-5_17"},{"key":"e_1_2_1_23_1","volume-title":"Overview of the TREC 2009 web track. In Proceedings of TREC 2009.","author":"Clarke Charles L. A.","year":"2009","unstructured":"Charles L. A. Clarke , Nick Craswell , and Ian Soboroff . 2009 . Overview of the TREC 2009 web track. In Proceedings of TREC 2009. Charles L. A. Clarke, Nick Craswell, and Ian Soboroff. 2009. Overview of the TREC 2009 web track. In Proceedings of TREC 2009."},{"key":"e_1_2_1_24_1","volume-title":"Proceedings of ACM SIGIR","author":"Beer Jan De","year":"2006","unstructured":"Jan De Beer and Marie-Fancine Moens . 2006 . Rpref: A generalization of Bpref towards graded relevance judgements . In Proceedings of ACM SIGIR 2006. 637--638. Jan De Beer and Marie-Fancine Moens. 2006. Rpref: A generalization of Bpref towards graded relevance judgements. In Proceedings of ACM SIGIR 2006. 637--638."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-019-09356-x"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1016\/0306-4573(91)90046-O"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-012-9218-8"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2566486.2568015"},{"key":"e_1_2_1_29_1","unstructured":"Nadine Hoechstoetter and Dirk Lewandowski. 2015. What Users See\u2014Structures in Search Engine Results Pages. Retrieved from https:\/\/arxiv:cs.IR\/1511.05802.  Nadine Hoechstoetter and Dirk Lewandowski. 2015. What Users See\u2014Structures in Search Engine Results Pages. Retrieved from https:\/\/arxiv:cs.IR\/1511.05802."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1561\/9781680831627"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3121050.3121095"},{"key":"e_1_2_1_32_1","volume-title":"Proceedings of ECIR 2017 (LNCS 10193)","author":"Hui Kai","year":"2017","unstructured":"Kai Hui and Klaus Berberich . 2017 . Transitivity, time consumption, and quality of preference judgements in crowdsourcing . In Proceedings of ECIR 2017 (LNCS 10193) . 239--251. Kai Hui and Klaus Berberich. 2017. Transitivity, time consumption, and quality of preference judgements in crowdsourcing. In Proceedings of ECIR 2017 (LNCS 10193). 239--251."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/582415.582418"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2505515.2505716"},{"key":"e_1_2_1_35_1","volume-title":"Content Analysis: An Introduction to Its Methodology","author":"Krippendorff Klaus","year":"2018","unstructured":"Klaus Krippendorff . 2018 . Content Analysis: An Introduction to Its Methodology ( 4 th ed.). SAGE Publications . Klaus Krippendorff. 2018. Content Analysis: An Introduction to Its Methodology (4th ed.). SAGE Publications.","edition":"4"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2911507"},{"key":"e_1_2_1_37_1","volume-title":"Incorporating user expectations and behavior into the measurement of search effectiveness. ACM TOIS 35, 3","author":"Moffat Alistair","year":"2017","unstructured":"Alistair Moffat , Peter Bailey , Falk Scholer , and Paul Thomas . 2017. Incorporating user expectations and behavior into the measurement of search effectiveness. ACM TOIS 35, 3 ( 2017 ). Alistair Moffat, Peter Bailey, Falk Scholer, and Paul Thomas. 2017. Incorporating user expectations and behavior into the measurement of search effectiveness. ACM TOIS 35, 3 (2017)."},{"key":"e_1_2_1_38_1","volume-title":"Rank-biased precision for measurement of retrieval effectiveness. ACM TOIS 27, 1","author":"Moffat Alistair","year":"2008","unstructured":"Alistair Moffat and Justin Zobel . 2008. Rank-biased precision for measurement of retrieval effectiveness. ACM TOIS 27, 1 ( 2008 ). Alistair Moffat and Justin Zobel. 2008. Rank-biased precision for measurement of retrieval effectiveness. ACM TOIS 27, 1 (2008)."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/1935826.1935850"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390453"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-31871-2_22"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/1277741.1277756"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/2187836.2187904"},{"key":"e_1_2_1_44_1","doi-asserted-by":"crossref","unstructured":"Tetsuya Sakai. 2014. Metrics statistics tests. In PROMISE Winter School 2013: Bridging between Information Retrieval and Databases (LNCS 8173). 116--163.  Tetsuya Sakai. 2014. Metrics statistics tests. In PROMISE Winter School 2013: Bridging between Information Retrieval and Databases (LNCS 8173). 116--163.","DOI":"10.1007\/978-3-642-54798-0_6"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-015-9273-z"},{"key":"e_1_2_1_46_1","volume-title":"Effect Sizes, and Statistical Power","author":"Sakai Tetsuya","unstructured":"Tetsuya Sakai . 2018. Laboratory Experiments in Information Retrieval: Sample Sizes , Effect Sizes, and Statistical Power . Springer . Tetsuya Sakai. 2018. Laboratory Experiments in Information Retrieval: Sample Sizes, Effect Sizes, and Statistical Power. Springer."},{"key":"e_1_2_1_47_1","volume-title":"Information Retrieval Evaluation in a Changing World: Lessons Learned from 20 Years of CLEF","author":"Sakai Tetsuya","unstructured":"Tetsuya Sakai . 2019. How to run an evaluation task: With a primary focus on ad hoc information retrieval . In Information Retrieval Evaluation in a Changing World: Lessons Learned from 20 Years of CLEF , Nicola Ferro and Carol Peters (Eds.). Springer , 71--102. Tetsuya Sakai. 2019. How to run an evaluation task: With a primary focus on ad hoc information retrieval. In Information Retrieval Evaluation in a Changing World: Lessons Learned from 20 Years of CLEF, Nicola Ferro and Carol Peters (Eds.). Springer, 71--102."},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/2484028.2484031"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-008-9059-7"},{"key":"e_1_2_1_50_1","volume-title":"Proceedings of EVIA","author":"Sakai Tetsuya","year":"2008","unstructured":"Tetsuya Sakai and Stephen Robertson . 2008 . Modelling a user population for designing information retrieval metrics . In Proceedings of EVIA 2008. 30--41. Tetsuya Sakai and Stephen Robertson. 2008. Modelling a user population for designing information retrieval metrics. In Proceedings of EVIA 2008. 30--41."},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/2009916.2010055"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-012-9208-x"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/3331184.3331215"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401115"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835449.1835542"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1561\/1500000040"},{"key":"e_1_2_1_57_1","volume-title":"Proceedings of ACM SIGIR","author":"Mark","year":"2012","unstructured":"Mark D. Smucker and Charlies L. A. Clarke. 2012. Time-based calibration of effectiveness measures . In Proceedings of ACM SIGIR 2012 . 95--104. Mark D. Smucker and Charlies L. A. Clarke. 2012. Time-based calibration of effectiveness measures. In Proceedings of ACM SIGIR 2012. 95--104."},{"key":"e_1_2_1_58_1","volume-title":"Proceedings of NTCIR-9. 82--105","author":"Song Ruihua","year":"2011","unstructured":"Ruihua Song , Min Zhang , Tetsuya Sakai , Makoto P. Kato , Yiqun Liu , Miho Sugimoto , Qinglei Wang , and Naoki Orii . 2011 . Overview of the NTCIR-9 INTENT task . In Proceedings of NTCIR-9. 82--105 . Ruihua Song, Min Zhang, Tetsuya Sakai, Makoto P. Kato, Yiqun Liu, Miho Sugimoto, Qinglei Wang, and Naoki Orii. 2011. Overview of the NTCIR-9 INTENT task. In Proceedings of NTCIR-9. 82--105."},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/1148170.1148176"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/1571941.1572138"},{"key":"e_1_2_1_61_1","volume-title":"Proceedings of ACM SIGIR","author":"Ellen","year":"2002","unstructured":"Ellen M. Voorhees and Chris Buckley. 2002. The effect of topic set size on retrieval experiment error . In Proceedings of ACM SIGIR 2002 . 316--323. Ellen M. Voorhees and Chris Buckley. 2002. The effect of topic set size on retrieval experiment error. In Proceedings of ACM SIGIR 2002. 316--323."},{"key":"e_1_2_1_62_1","first-page":"156","article-title":"Search result diversity evaluation based on intent hierarchies","volume":"30","author":"Wang Xiaojie","year":"2018","unstructured":"Xiaojie Wang , Ji-Rong Wen , Zhicheng Dou , Tetsuya Sakai , and Rui Zhang . 2018 . Search result diversity evaluation based on intent hierarchies . IEEE TKDE 30 (2018), 156 -- 169 . Issue 1. Xiaojie Wang, Ji-Rong Wen, Zhicheng Dou, Tetsuya Sakai, and Rui Zhang. 2018. Search result diversity evaluation based on intent hierarchies. IEEE TKDE 30 (2018), 156--169. Issue 1.","journal-title":"IEEE TKDE"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/3336191.3371799"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/860435.860440"},{"key":"e_1_2_1_65_1","volume-title":"Proceedings of ACM CIKM","author":"Zhou Ke","year":"2013","unstructured":"Ke Zhou , Mounia Lalmas , Tetsuya Sakai , Ronan Cummins , and Joemon M. Jose . 2013. On the reliability and intuitiveness of aggregated search metrics . In Proceedings of ACM CIKM 2013 . 689--698. Ke Zhou, Mounia Lalmas, Tetsuya Sakai, Ronan Cummins, and Joemon M. Jose. 2013. On the reliability and intuitiveness of aggregated search metrics. In Proceedings of ACM CIKM 2013. 689--698."}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3431813","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3431813","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T21:24:46Z","timestamp":1750195486000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3431813"}},"subtitle":["Traditional, Preference-based, and Diversity Measures"],"short-title":[],"issued":{"date-parts":[[2020,12,31]]},"references-count":65,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2021,4,30]]}},"alternative-id":["10.1145\/3431813"],"URL":"https:\/\/doi.org\/10.1145\/3431813","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"value":"1046-8188","type":"print"},{"value":"1558-2868","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,12,31]]},"assertion":[{"value":"2020-06-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-10-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-12-31","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}