{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,11]],"date-time":"2026-07-11T02:28:19Z","timestamp":1783736899349,"version":"3.55.0"},"reference-count":44,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2021,5,5]],"date-time":"2021-05-05T00:00:00Z","timestamp":1620172800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2021,7,26]]},"abstract":"<jats:p>\n            Assessors make preference judgments faster and more consistently than graded judgments. Preference judgments can also recognize distinctions between items that appear equivalent under graded judgments. Unfortunately, preference judgments can require more than linear effort to fully order a pool of items, and evaluation measures for preference judgments are not as well established as those for graded judgments, such as NDCG. In this article, we explore the assessment process for partial preference judgments, with the aim of identifying and ordering the top items in the pool, rather than fully ordering the entire pool. To measure the performance of a ranker, we compare its output to this preferred ordering by applying a rank similarity measure. We demonstrate the practical feasibility of this approach by crowdsourcing partial preferences for the TREC 2019 Conversational Assistance Track, replacing NDCG with a new measure named\n            <jats:italic>compatibility<\/jats:italic>\n            . This new measure has its most striking impact when comparing modern neural rankers, where it is able to recognize significant improvements in quality that would otherwise be missed by NDCG.\n          <\/jats:p>","DOI":"10.1145\/3451161","type":"journal-article","created":{"date-parts":[[2021,5,6]],"date-time":"2021-05-06T04:14:52Z","timestamp":1620274492000},"page":"1-21","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":22,"title":["Assessing Top- Preferences"],"prefix":"10.1145","volume":"39","author":[{"given":"Charles L. A.","family":"Clarke","sequence":"first","affiliation":[{"name":"University of Waterloo, Waterloo, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alexandra","family":"Vtyurina","sequence":"additional","affiliation":[{"name":"University of Waterloo, Waterloo, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mark D.","family":"Smucker","sequence":"additional","affiliation":[{"name":"University of Waterloo, Waterloo, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2021,5,5]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Proceedings of the 28th ACM International Conference on Information and Knowledge Management. 1833\u20131842","author":"Abualsaud Mustafa"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/1480506.1480508"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390447"},{"key":"e_1_2_1_4_1","volume-title":"Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval. 909\u2013912","author":"Bashir Maryam"},{"key":"e_1_2_1_6_1","volume-title":"Proceedings of the SIGIR Workshop on Beyond Binary Relevance: Preferences, Diversity, and Set-level Judgments.","author":"Carterette Ben","year":"2008"},{"key":"e_1_2_1_7_1","volume-title":"Proceedings of the 31st International ACM SIGIR Conference on Research and Development in Information Retrieval. 685\u2013686","author":"Carterette Ben"},{"key":"e_1_2_1_8_1","volume-title":"David Maxwell Chickering, and Susan T. Dumais","author":"Carterette Ben","year":"2008"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2348283.2348398"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/2484028.2484094"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/1645953.1646033"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/2433396.2433420"},{"key":"e_1_2_1_13_1","volume-title":"Proceedings of the 28th Text REtrieval Conference.","author":"Clarke Charles L. A.","year":"2019"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3340531.3411915"},{"key":"e_1_2_1_15_1","volume-title":"Proceedings of the ACM SIGIR International Conference on the Theory of Information Retrieval.","author":"Clarke Charles L. A."},{"key":"e_1_2_1_16_1","volume-title":"Proceedings of the 28th Text REtrieval Conference.","author":"Dalton Jeffrey","year":"2019"},{"key":"e_1_2_1_17_1","volume-title":"Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics.","author":"Devlin Jacob","year":"2019"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1016\/0306-4573(91)90046-O"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/1008992.1009079"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2661829.2661913"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-56608-5_58"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-56608-5_19"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/582415.582418"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3240323.3240364"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2505515.2505716"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2484028.2484168"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300742"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2505515.2505685"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/2093346.2093356"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3002172"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/2348283.2348384"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/1935826.1935850"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(199012)41:8<590::AID-ASI5>3.0.CO;2-T"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/1148170.1148261"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401115"},{"key":"e_1_2_1_36_1","volume-title":"What Is It Really? Morgan & Claypool","author":"Saracevic Tefko"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/290941.291017"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/1852102.1852106"},{"key":"e_1_2_1_39_1","volume-title":"Proceedings of the 22nd International Conference on Neural Information Processing Systems. 2098\u20132106","author":"Xia Fen","year":"2009"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401146"},{"key":"e_1_2_1_41_1","volume-title":"Proceedings of the 28th Text REtrieval Conference.","author":"Yang Jheng-Hong","year":"2019"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3331184.3331340"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3291992.3291995"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.5555\/208098.208105"},{"key":"e_1_2_1_45_1","volume-title":"Proceedings of the SIGIR Workshop on Crowdsourcing for Search Evaluation.","author":"Zhu Dongqing","year":"2010"}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3451161","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3451161","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T17:49:25Z","timestamp":1750268965000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3451161"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,5,5]]},"references-count":44,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2021,7,26]]}},"alternative-id":["10.1145\/3451161"],"URL":"https:\/\/doi.org\/10.1145\/3451161","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"value":"1046-8188","type":"print"},{"value":"1558-2868","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,5,5]]},"assertion":[{"value":"2020-07-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-02-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-05-05","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}