{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,15]],"date-time":"2026-07-15T05:21:25Z","timestamp":1784092885654,"version":"3.55.0"},"publisher-location":"New York, NY, USA","reference-count":44,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,4,19]],"date-time":"2021-04-19T00:00:00Z","timestamp":1618790400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,4,19]]},"DOI":"10.1145\/3442381.3450080","type":"proceedings-article","created":{"date-parts":[[2021,6,3]],"date-time":"2021-06-03T19:02:07Z","timestamp":1622746927000},"page":"1065-1075","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":37,"title":["Estimation of Fair Ranking Metrics with Incomplete Judgments"],"prefix":"10.1145","author":[{"given":"\u00d6mer","family":"K\u0131rnap","sequence":"first","affiliation":[{"name":"University College London, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fernando","family":"Diaz","sequence":"additional","affiliation":[{"name":"Microsoft Research, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Asia","family":"Biega","sequence":"additional","affiliation":[{"name":"Max Planck Institute for Security and Privacy, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael","family":"Ekstrand","sequence":"additional","affiliation":[{"name":"Boise State University, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ben","family":"Carterette","sequence":"additional","affiliation":[{"name":"Spotify Research, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Emine","family":"Yilmaz","sequence":"additional","affiliation":[{"name":"University College London, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2021,6,3]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/1148170.1148263"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3292500.3330745"},{"key":"e_1_3_2_1_3_1","volume-title":"Overview of the TREC 2019 Fair Ranking Track. In The Twenty-Eighth Text REtrieval Conference (TREC 2019) Proceedings.","author":"Biega J.","year":"2019","unstructured":"Asia\u00a0 J. Biega , Fernando Diaz , Michael\u00a0 D. Ekstrand , and Sebastian Kohlmeier . 2019 . Overview of the TREC 2019 Fair Ranking Track. In The Twenty-Eighth Text REtrieval Conference (TREC 2019) Proceedings. Asia\u00a0J. Biega, Fernando Diaz, Michael\u00a0D. Ekstrand, and Sebastian Kohlmeier. 2019. Overview of the TREC 2019 Fair Ranking Track. In The Twenty-Eighth Text REtrieval Conference (TREC 2019) Proceedings."},{"key":"e_1_3_2_1_4_1","unstructured":"Asia\u00a0J Biega Fernando Diaz Michael\u00a0D Ekstrand and Sebastian Kohlmeier. 2020. Overview of the trec 2019 fair ranking track. arXiv preprint arXiv:2003.11650(2020).  Asia\u00a0J Biega Fernando Diaz Michael\u00a0D Ekstrand and Sebastian Kohlmeier. 2020. Overview of the trec 2019 fair ranking track. arXiv preprint arXiv:2003.11650(2020)."},{"key":"e_1_3_2_1_5_1","volume-title":"The 41st international acm sigir conference on research & development in information retrieval. 405\u2013414.","author":"Biega J","unstructured":"Asia\u00a0 J Biega , Krishna\u00a0 P Gummadi , and Gerhard Weikum . 2018. Equity of attention: Amortizing individual fairness in rankings . In The 41st international acm sigir conference on research & development in information retrieval. 405\u2013414. Asia\u00a0J Biega, Krishna\u00a0P Gummadi, and Gerhard Weikum. 2018. Equity of attention: Amortizing individual fairness in rankings. In The 41st international acm sigir conference on research & development in information retrieval. 405\u2013414."},{"key":"e_1_3_2_1_6_1","volume-title":"Sampling with unequal probabilities. Vol.\u00a015","author":"Brewer RW","unstructured":"Ken\u00a0 RW Brewer and Muhammad Hanif . 2013. Sampling with unequal probabilities. Vol.\u00a015 . Springer Science & Business Media . Ken\u00a0RW Brewer and Muhammad Hanif. 2013. Sampling with unequal probabilities. Vol.\u00a015. Springer Science & Business Media."},{"key":"e_1_3_2_1_7_1","volume-title":"Retrieval Evaluation with Incomplete Information(SIGIR \u201904)","author":"Buckley Chris","unstructured":"Chris Buckley and Ellen\u00a0 M. Voorhees . 2004. Retrieval Evaluation with Incomplete Information(SIGIR \u201904) . Association for Computing Machinery , New York, NY, USA , 25\u201332. Chris Buckley and Ellen\u00a0M. Voorhees. 2004. Retrieval Evaluation with Incomplete Information(SIGIR \u201904). Association for Computing Machinery, New York, NY, USA, 25\u201332."},{"key":"e_1_3_2_1_8_1","volume-title":"Multisided Fairness for Recommendation. (July","author":"Burke Robin","year":"2017","unstructured":"Robin Burke . 2017. Multisided Fairness for Recommendation. (July 2017 ). arxiv:1707.00093\u00a0[cs.CY] Robin Burke. 2017. Multisided Fairness for Recommendation. (July 2017). arxiv:1707.00093\u00a0[cs.CY]"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1148170.1148219"},{"key":"e_1_3_2_1_10_1","unstructured":"L\u00a0Elisa Celis Damian Straszak and Nisheeth\u00a0K Vishnoi. 2017. Ranking with fairness constraints. arXiv preprint arXiv:1704.06840(2017).  L\u00a0Elisa Celis Damian Straszak and Nisheeth\u00a0K Vishnoi. 2017. Ranking with fairness constraints. arXiv preprint arXiv:1704.06840(2017)."},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/1645953.1646033"},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"crossref","unstructured":"Aleksandr Chuklin Ilya Markov and Maarten\u00a0de Rijke. 2015. Click models for web search. Synthesis lectures on information concepts retrieval and services 7 3(2015) 1\u2013115.  Aleksandr Chuklin Ilya Markov and Maarten\u00a0de Rijke. 2015. Click models for web search. Synthesis lectures on information concepts retrieval and services 7 3(2015) 1\u2013115.","DOI":"10.2200\/S00654ED1V01Y201507ICR043"},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390446"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/1341531.1341545"},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"crossref","unstructured":"Fernando Diaz Bhaskar Mitra Michael\u00a0D Ekstrand Asia\u00a0J Biega and Ben Carterette. 2020. Evaluating Stochastic Rankings with Expected Exposure. arXiv preprint arXiv:2004.13157(2020).  Fernando Diaz Bhaskar Mitra Michael\u00a0D Ekstrand Asia\u00a0J Biega and Ben Carterette. 2020. Evaluating Stochastic Rankings with Expected Exposure. arXiv preprint arXiv:2004.13157(2020).","DOI":"10.1145\/3340531.3411962"},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2090236.2090255"},{"key":"e_1_3_2_1_17_1","volume-title":"Ekstrand and Daniel Kluver","author":"D.","year":"2021","unstructured":"Michael\u00a0 D. Ekstrand and Daniel Kluver . 2021 . Exploring Author Gender in Book Rating and Recommendation. User Modeling and User-Adapted Interaction( 2021). https:\/\/doi.org\/10.1007\/s11257-020-09284-2 10.1007\/s11257-020-09284-2 Michael\u00a0D. Ekstrand and Daniel Kluver. 2021. Exploring Author Gender in Book Rating and Recommendation. User Modeling and User-Adapted Interaction(2021). https:\/\/doi.org\/10.1007\/s11257-020-09284-2"},{"key":"e_1_3_2_1_18_1","unstructured":"Moritz Hardt Eric Price and Nati Srebro. 2016. Equality of opportunity in supervised learning. In Advances in neural information processing systems. 3315\u20133323.  Moritz Hardt Eric Price and Nati Srebro. 2016. Equality of opportunity in supervised learning. In Advances in neural information processing systems. 3315\u20133323."},{"key":"e_1_3_2_1_19_1","volume-title":"Vol.\u00a080), Jennifer Dy and Andreas Krause (Eds.). PMLR","author":"Hashimoto Tatsunori","year":"1929","unstructured":"Tatsunori Hashimoto , Megha Srivastava , Hongseok Namkoong , and Percy Liang . 2018. Fairness Without Demographics in Repeated Loss Minimization(Proceedings of Machine Learning Research , Vol.\u00a080), Jennifer Dy and Andreas Krause (Eds.). PMLR , Stockholmsm\u00e4ssan, Stockholm Sweden , 1929 \u20131938. http:\/\/proceedings.mlr.press\/v80\/hashimoto18a.html Tatsunori Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang. 2018. Fairness Without Demographics in Repeated Loss Minimization(Proceedings of Machine Learning Research, Vol.\u00a080), Jennifer Dy and Andreas Krause (Eds.). PMLR, Stockholmsm\u00e4ssan, Stockholm Sweden, 1929\u20131938. http:\/\/proceedings.mlr.press\/v80\/hashimoto18a.html"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300830"},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2007.289"},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/582415.582418"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"crossref","unstructured":"Nathan Kallus Xiaojie Mao and Angela Zhou. 2019. Assessing algorithmic fairness with unobserved protected class using data combination. arXiv preprint arXiv:1906.00285(2019).  Nathan Kallus Xiaojie Mao and Angela Zhou. 2019. Assessing algorithmic fairness with unobserved protected class using data combination. arXiv preprint arXiv:1906.00285(2019).","DOI":"10.1145\/3351095.3373154"},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3308558.3313443"},{"key":"e_1_3_2_1_25_1","volume-title":"Chi","author":"Lahoti Preethi","year":"2020","unstructured":"Preethi Lahoti , Alex Beutel , Jilin Chen , Kang Lee , Flavien Prost , Nithum Thain , Xuezhi Wang , and Ed Chi . 2020 . Fairness without Demographics through Adversarially Reweighted Learning. In Advances in Neural Information Processing Systems, H.\u00a0Larochelle, M.\u00a0Ranzato, R.\u00a0Hadsell, M.\u00a0F. Balcan, and H.\u00a0Lin (Eds.), Vol.\u00a033. Curran Associates, Inc ., 728\u2013740. Preethi Lahoti, Alex Beutel, Jilin Chen, Kang Lee, Flavien Prost, Nithum Thain, Xuezhi Wang, and Ed Chi. 2020. Fairness without Demographics through Adversarially Reweighted Learning. In Advances in Neural Information Processing Systems, H.\u00a0Larochelle, M.\u00a0Ranzato, R.\u00a0Hadsell, M.\u00a0F. Balcan, and H.\u00a0Lin (Eds.), Vol.\u00a033. Curran Associates, Inc., 728\u2013740."},{"key":"e_1_3_2_1_26_1","volume-title":"Prediction-Based Decisions and Fairness: A Catalogue of Choices, Assumptions, and Definitions. (Nov","author":"Mitchell Shira","year":"2018","unstructured":"Shira Mitchell , Eric Potash , Solon Barocas , Alexander D\u2019Amour , and Kristian Lum . 2018. Prediction-Based Decisions and Fairness: A Catalogue of Choices, Assumptions, and Definitions. (Nov . 2018 ). arxiv:1811.07867\u00a0[stat.AP] Shira Mitchell, Eric Potash, Solon Barocas, Alexander D\u2019Amour, and Kristian Lum. 2018. Prediction-Based Decisions and Fairness: A Catalogue of Choices, Assumptions, and Definitions. (Nov. 2018). arxiv:1811.07867\u00a0[stat.AP]"},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/1416950.1416952"},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"crossref","unstructured":"Marco Morik Ashudeep Singh Jessica Hong and Thorsten Joachims. 2020. Controlling Fairness and Bias in Dynamic Learning-to-Rank. arXiv preprint arXiv:2005.14713(2020).  Marco Morik Ashudeep Singh Jessica Hong and Thorsten Joachims. 2020. Controlling Fairness and Bias in Dynamic Learning-to-Rank. arXiv preprint arXiv:2005.14713(2020).","DOI":"10.1145\/3397271.3401100"},{"key":"e_1_3_2_1_29_1","volume-title":"A practical sampling strategy for efficient retrieval evaluation","author":"Pavlu V","year":"2007","unstructured":"V Pavlu and J Aslam . 2007. A practical sampling strategy for efficient retrieval evaluation . College of Computer and Information Science, Northeastern University ( 2007 ). V Pavlu and J Aslam. 2007. A practical sampling strategy for efficient retrieval evaluation. College of Computer and Information Science, Northeastern University (2007)."},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1137\/1.9781611972795.50"},{"key":"e_1_3_2_1_31_1","first-page":"2","article-title":"FACTS-IR: Fairness, Accountability, Confidentiality, Transparency, and Safety in Information Retrieval","volume":"53","author":"Roegiest Adam","year":"2019","unstructured":"Adam Roegiest , Aldo Lipani , Alex Beutel , Alexandra Olteanu , Ana Lucic , Ana-Andreea Stoica , Anubrata Das , Asia Biega , Bart Voorn , Claudia Hauff , Damiano Spina , David Lewis , Douglas\u00a0 W Oard , Emine Yilmaz , Faegheh Hasibi , Gabriella Kazai , Graham McDonald , Hinda Haned , Iadh Ounis , Ilse van\u00a0der Linden , Jean Garcia-Gathright , Joris Baan , Kamuela\u00a0 N Lau , Krisztian Balog , Maarten de Rijke , Mahmoud Sayed , Maria Panteli , Mark Sanderson , Matthew Lease , Michael\u00a0 D Ekstrand , Preethi Lahoti , and Toshihiro Kamishima . 2019 . FACTS-IR: Fairness, Accountability, Confidentiality, Transparency, and Safety in Information Retrieval . SIGIR Forum 53 , 2 (Dec. 2019), 20\u201343. Adam Roegiest, Aldo Lipani, Alex Beutel, Alexandra Olteanu, Ana Lucic, Ana-Andreea Stoica, Anubrata Das, Asia Biega, Bart Voorn, Claudia Hauff, Damiano Spina, David Lewis, Douglas\u00a0W Oard, Emine Yilmaz, Faegheh Hasibi, Gabriella Kazai, Graham McDonald, Hinda Haned, Iadh Ounis, Ilse van\u00a0der Linden, Jean Garcia-Gathright, Joris Baan, Kamuela\u00a0N Lau, Krisztian Balog, Maarten de Rijke, Mahmoud Sayed, Maria Panteli, Mark Sanderson, Matthew Lease, Michael\u00a0D Ekstrand, Preethi Lahoti, and Toshihiro Kamishima. 2019. FACTS-IR: Fairness, Accountability, Confidentiality, Transparency, and Safety in Information Retrieval. SIGIR Forum 53, 2 (Dec. 2019), 20\u201343.","journal-title":"SIGIR Forum"},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2019.102149"},{"key":"e_1_3_2_1_33_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3241064","article-title":"Reproduce. generalize. extend. on information retrieval evaluation without relevance judgments","volume":"10","author":"Roitero Kevin","year":"2018","unstructured":"Kevin Roitero , Marco Passon , Giuseppe Serra , and Stefano Mizzaro . 2018 . Reproduce. generalize. extend. on information retrieval evaluation without relevance judgments . Journal of Data and Information Quality (JDIQ) 10 , 3 (2018), 1 \u2013 32 . Kevin Roitero, Marco Passon, Giuseppe Serra, and Stefano Mizzaro. 2018. Reproduce. generalize. extend. on information retrieval evaluation without relevance judgments. Journal of Data and Information Quality (JDIQ) 10, 3 (2018), 1\u201332.","journal-title":"Journal of Data and Information Quality (JDIQ)"},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3220088"},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.2517-6161.1958.tb00304.x"},{"key":"e_1_3_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2043932.2043987"},{"key":"e_1_3_2_1_37_1","unstructured":"S.\u00a0K. Thompson. 2002. Sampling. Wiley-Interscience.  S.\u00a0K. Thompson. 2002. Sampling. Wiley-Interscience."},{"key":"e_1_3_2_1_38_1","volume-title":"Wile-Interscience","author":"Thompson K.","year":"2002","unstructured":"Steven\u00a0 K. Thompson . 2002. Sampling. Wile-Interscience , second edition, 2002 . Steven\u00a0K. Thompson. 2002. Sampling. Wile-Interscience, second edition, 2002."},{"key":"e_1_3_2_1_39_1","volume-title":"Evaluation of Cross-Language Information Retrieval Systems","author":"Voorhees M","unstructured":"Ellen\u00a0 M Voorhees . 2001. The Philosophy of Information Retrieval Evaluation . In Evaluation of Cross-Language Information Retrieval Systems , Carol Peters, Martin Braschler, Julio Gonzalo, and Michael Kluck (Eds.). Springer Berlin Heidelberg , 355\u2013370. http:\/\/link.springer.com\/chapter\/10.1007\/3-540-45691-0_34 Ellen\u00a0M Voorhees. 2001. The Philosophy of Information Retrieval Evaluation. In Evaluation of Cross-Language Information Retrieval Systems, Carol Peters, Martin Braschler, Julio Gonzalo, and Michael Kluck (Eds.). Springer Berlin Heidelberg, 355\u2013370. http:\/\/link.springer.com\/chapter\/10.1007\/3-540-45691-0_34"},{"key":"e_1_3_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3240323.3240369"},{"key":"e_1_3_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3085504.3085526"},{"key":"e_1_3_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/1183614.1183633"},{"key":"e_1_3_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390437"},{"key":"e_1_3_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132847.3132938"}],"event":{"name":"WWW '21: The Web Conference 2021","location":"Ljubljana Slovenia","acronym":"WWW '21","sponsor":["SIGWEB ACM Special Interest Group on Hypertext, Hypermedia, and Web"]},"container-title":["Proceedings of the Web Conference 2021"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3442381.3450080","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3442381.3450080","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T21:24:27Z","timestamp":1750195467000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3442381.3450080"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,4,19]]},"references-count":44,"alternative-id":["10.1145\/3442381.3450080","10.1145\/3442381"],"URL":"https:\/\/doi.org\/10.1145\/3442381.3450080","relation":{},"subject":[],"published":{"date-parts":[[2021,4,19]]},"assertion":[{"value":"2021-06-03","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}