{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,15]],"date-time":"2026-07-15T05:22:24Z","timestamp":1784092944894,"version":"3.55.0"},"publisher-location":"New York, NY, USA","reference-count":49,"publisher":"ACM","license":[{"start":{"date-parts":[[2020,10,19]],"date-time":"2020-10-19T00:00:00Z","timestamp":1603065600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2020,10,19]]},"DOI":"10.1145\/3340531.3411962","type":"proceedings-article","created":{"date-parts":[[2020,10,19]],"date-time":"2020-10-19T05:31:04Z","timestamp":1603085464000},"page":"275-284","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":135,"title":["Evaluating Stochastic Rankings with Expected Exposure"],"prefix":"10.1145","author":[{"given":"Fernando","family":"Diaz","sequence":"first","affiliation":[{"name":"Microsoft, Montr\u00e9al, PQ, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bhaskar","family":"Mitra","sequence":"additional","affiliation":[{"name":"Microsoft, Montr\u00e9al, PQ, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael D.","family":"Ekstrand","sequence":"additional","affiliation":[{"name":"Boise State University, Boise, ID, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Asia J.","family":"Biega","sequence":"additional","affiliation":[{"name":"Microsoft, Montr\u00e9al, PQ, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ben","family":"Carterette","sequence":"additional","affiliation":[{"name":"Spotify, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2020,10,19]]},"reference":[{"key":"e_1_3_2_2_1_1","volume-title":"Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432","author":"Bengio Yoshua","year":"2013","unstructured":"Yoshua Bengio , Nicholas L\u00e9onard , and Aaron Courville . 2013. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432 ( 2013 ). Yoshua Bengio, Nicholas L\u00e9onard, and Aaron Courville. 2013. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432 (2013)."},{"key":"e_1_3_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3292500.3330745"},{"key":"e_1_3_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3210063"},{"key":"e_1_3_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3336191.3371844"},{"key":"e_1_3_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/1102351.1102363"},{"key":"e_1_3_2_2_6_1","volume-title":"Multisided Fairness for Recommendation. (July","author":"Burke Robin","year":"2017","unstructured":"Robin Burke . 2017. Multisided Fairness for Recommendation. (July 2017 ). arxiv: cs.CY\/1707.00093 Robin Burke. 2017. Multisided Fairness for Recommendation. (July 2017). arxiv: cs.CY\/1707.00093"},{"key":"e_1_3_2_2_7_1","volume-title":"45th International Colloquium on Automata, Languages, and Programming, ICALP 2018","author":"Celis L. Elisa","year":"2018","unstructured":"L. Elisa Celis , Damian Straszak , and Nisheeth K. Vishnoi . 2018. Ranking with Fairness Constraints. In 45th International Colloquium on Automata, Languages, and Programming, ICALP 2018 , July 9 --13 , 2018 , Prague, Czech Republic. 28:1--28:15. L. Elisa Celis, Damian Straszak, and Nisheeth K. Vishnoi. 2018. Ranking with Fairness Constraints. In 45th International Colloquium on Automata, Languages, and Programming, ICALP 2018, July 9--13, 2018, Prague, Czech Republic. 28:1--28:15."},{"key":"e_1_3_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3240323.3240370"},{"key":"e_1_3_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1645953"},{"key":"e_1_3_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1007\/11776420_44"},{"key":"e_1_3_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF00535487"},{"key":"e_1_3_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/2505515.2505717"},{"key":"e_1_3_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2090236.2090255"},{"key":"e_1_3_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3340531.3412778"},{"key":"e_1_3_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2019.102138"},{"key":"e_1_3_2_2_16_1","first-page":"4","article-title":"The MovieLens Datasets: History and Context","volume":"5","author":"Maxwell Harper F","year":"2015","unstructured":"F Maxwell Harper and Joseph A Konstan . 2015 . The MovieLens Datasets: History and Context . ACM Transactions on Interactive Intelligent Systems , Vol. 5 , 4 (Dec. 2015), 19:1--19:19. F Maxwell Harper and Joseph A Konstan. 2015. The MovieLens Datasets: History and Context. ACM Transactions on Interactive Intelligent Systems, Vol. 5, 4 (Dec. 2015), 19:1--19:19.","journal-title":"ACM Transactions on Interactive Intelligent Systems"},{"key":"e_1_3_2_2_17_1","volume-title":"Proceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research), Jennifer Dy and Andreas Krause (Eds.)","volume":"80","author":"Hashimoto Tatsunori","year":"2018","unstructured":"Tatsunori Hashimoto , Megha Srivastava , Hongseok Namkoong , and Percy Liang . 2018 . Fairness Without Demographics in Repeated Loss Minimization . In Proceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research), Jennifer Dy and Andreas Krause (Eds.) , Vol. 80 . 1929--1938. Tatsunori Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang. 2018. Fairness Without Demographics in Repeated Loss Minimization. In Proceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research), Jennifer Dy and Andreas Krause (Eds.), Vol. 80. 1929--1938."},{"key":"e_1_3_2_2_18_1","volume-title":"Fast and Reliable Online Learning to Rank for Information Retrieval. phd","author":"Hofmann K.","unstructured":"K. Hofmann . 2013. Fast and Reliable Online Learning to Rank for Information Retrieval. phd . University of Amsterdam. K. Hofmann. 2013. Fast and Reliable Online Learning to Rank for Information Retrieval. phd. University of Amsterdam."},{"key":"e_1_3_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3018661.3018699"},{"key":"e_1_3_2_2_20_1","unstructured":"Matthew Joseph Michael Kearns Jamie H Morgenstern and Aaron Roth. 2016. Fairness in Learning: Classic and Contextual Bandits. In Advances in Neural Information Processing Systems 29 D D Lee M Sugiyama U V Luxburg I Guyon and R Garnett (Eds.). 325--333.  Matthew Joseph Michael Kearns Jamie H Morgenstern and Aaron Roth. 2016. Fairness in Learning: Classic and Contextual Bandits. In Advances in Neural Information Processing Systems 29 D D Lee M Sugiyama U V Luxburg I Guyon and R Garnett (Eds.). 325--333."},{"key":"e_1_3_2_2_21_1","volume-title":"Proceedings of the 34th International Conference on Machine Learning (Proceedings of Machine Learning Research), Doina Precup and Yee Whye Teh (Eds.)","volume":"70","author":"Kearns Michael","year":"2017","unstructured":"Michael Kearns , Aaron Roth , and Zhiwei Steven Wu . 2017 . Meritocratic Fairness for Cross-Population Selection . In Proceedings of the 34th International Conference on Machine Learning (Proceedings of Machine Learning Research), Doina Precup and Yee Whye Teh (Eds.) , Vol. 70 . 1828--1836. Michael Kearns, Aaron Roth, and Zhiwei Steven Wu. 2017. Meritocratic Fairness for Cross-Population Selection. In Proceedings of the 34th International Conference on Machine Learning (Proceedings of Machine Learning Research), Doina Precup and Yee Whye Teh (Eds.), Vol. 70. 1828--1836."},{"key":"e_1_3_2_2_22_1","unstructured":"R. Duncan Luce. 1959. Individual Choice Behavior.  R. Duncan Luce. 1959. Individual Choice Behavior."},{"key":"e_1_3_2_2_23_1","unstructured":"Chris J. Maddison Andriy Mnih and Yee Whye Teh. 2017. The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables.  Chris J. Maddison Andriy Mnih and Yee Whye Teh. 2017. The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables."},{"key":"e_1_3_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3041021.3054197"},{"key":"e_1_3_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/1416950.1416952"},{"key":"e_1_3_2_2_26_1","unstructured":"Sandeep Pandey Sourashis Roy Christopher Olston Junghoo Cho and Soumen Chakrabarti. 2005. Shuffling a Stacked Deck: The Case for Partially Randomized Ranking of Search Engine Results. In VLDB. 781--792.  Sandeep Pandey Sourashis Roy Christopher Olston Junghoo Cho and Soumen Chakrabarti. 2005. Shuffling a Stacked Deck: The Case for Partially Randomized Ranking of Search Engine Results. In VLDB. 781--792."},{"key":"e_1_3_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/1864708.1864726"},{"key":"e_1_3_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.2307\/2346567"},{"key":"e_1_3_2_2_29_1","volume-title":"abs\/1306.2597","author":"Qin Tao","year":"2013","unstructured":"Tao Qin and Tie-Yan Liu . 2013. Introducing LETOR 4.0 Datasets . Co RR , Vol . abs\/1306.2597 ( 2013 ). Tao Qin and Tie-Yan Liu. 2013. Introducing LETOR 4.0 Datasets. CoRR, Vol. abs\/1306.2597 (2013)."},{"key":"e_1_3_2_2_30_1","volume-title":"A general approximation framework for direct optimization of information retrieval measures. Information retrieval","author":"Qin Tao","year":"2010","unstructured":"Tao Qin , Tie-Yan Liu , and Hang Li. 2010. A general approximation framework for direct optimization of information retrieval measures. Information retrieval , Vol. 13 , 4 ( 2010 ), 375--397. Tao Qin, Tie-Yan Liu, and Hang Li. 2010. A general approximation framework for direct optimization of information retrieval measures. Information retrieval, Vol. 13, 4 (2010), 375--397."},{"key":"e_1_3_2_2_31_1","volume-title":"Proceedings of the 21st National Conference on Artificial Intelligence -","volume":"2","author":"Radlinski Filip","year":"2006","unstructured":"Filip Radlinski and Thorsten Joachims . 2006 . Minimally Invasive Randomization for Collecting Unbiased Preferences from Clickthrough Logs . In Proceedings of the 21st National Conference on Artificial Intelligence - Volume 2 (AAAI'06). 1406--1412. Filip Radlinski and Thorsten Joachims. 2006. Minimally Invasive Randomization for Collecting Unbiased Preferences from Clickthrough Logs. In Proceedings of the 21st National Conference on Artificial Intelligence - Volume 2 (AAAI'06). 1406--1412."},{"key":"e_1_3_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/1281192.1281254"},{"key":"e_1_3_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390156.1390255"},{"key":"e_1_3_2_2_34_1","volume-title":"Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence (UAI '09)","author":"Rendle Steffen","year":"2009","unstructured":"Steffen Rendle , Christoph Freudenthaler , Zeno Gantner , and Lars Schmidt-Thieme . 2009 . BPR: Bayesian Personalized Ranking from Implicit Feedback . In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence (UAI '09) . 452--461. Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2009. BPR: Bayesian Personalized Ranking from Implicit Feedback. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence (UAI '09). 452--461."},{"key":"e_1_3_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/2009916.2010055"},{"key":"e_1_3_2_2_36_1","volume-title":"Search Result Diversification. Foundations and Trends in Information Retrieval","author":"Santos Rodrygo L. T.","unstructured":"Rodrygo L. T. Santos , Craig Macdonald , and Iadh Ounis . 2015. Search Result Diversification. Foundations and Trends in Information Retrieval , Vol. 9 . Now Publishers Inc . Rodrygo L. T. Santos, Craig Macdonald, and Iadh Ounis. 2015. Search Result Diversification. Foundations and Trends in Information Retrieval, Vol. 9. Now Publishers Inc."},{"key":"e_1_3_2_2_37_1","volume-title":"Quantifying the Impact of User Attentionon Fair Group Representation in Ranked Lists. In Companion Proceedings of The 2019 World Wide Web Conference on - WWW '19. 553--562","author":"Sapiezynski Piotr","year":"2019","unstructured":"Piotr Sapiezynski , Wesley Zeng , Ronald E Robertson , Alan Mislove , and Christo Wilson . 2019 . Quantifying the Impact of User Attentionon Fair Group Representation in Ranked Lists. In Companion Proceedings of The 2019 World Wide Web Conference on - WWW '19. 553--562 . Piotr Sapiezynski, Wesley Zeng, Ronald E Robertson, Alan Mislove, and Christo Wilson. 2019. Quantifying the Impact of User Attentionon Fair Group Representation in Ranked Lists. In Companion Proceedings of The 2019 World Wide Web Conference on - WWW '19. 553--562."},{"key":"e_1_3_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3159652.3159700"},{"key":"e_1_3_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3220088"},{"key":"e_1_3_2_2_40_1","unstructured":"Ashudeep Singh and Thorsten Joachims. 2019. Policy Learning for Fairness in Ranking. In Advances in Neural Information Processing Systems.  Ashudeep Singh and Thorsten Joachims. 2019. Policy Learning for Fairness in Ranking. In Advances in Neural Information Processing Systems."},{"key":"e_1_3_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3220046"},{"key":"e_1_3_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3292500.3330793"},{"key":"e_1_3_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2911537"},{"key":"e_1_3_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/1645953.1646266"},{"key":"e_1_3_2_2_45_1","volume-title":"Fair Learning-to-Rank from Implicit Feedback. arXiv preprint arXiv:1911.08054","author":"Yadav Himank","year":"2019","unstructured":"Himank Yadav , Zhengxiao Du , and Thorsten Joachims . 2019. Fair Learning-to-Rank from Implicit Feedback. arXiv preprint arXiv:1911.08054 ( 2019 ). Himank Yadav, Zhengxiao Du, and Thorsten Joachims. 2019. Fair Learning-to-Rank from Implicit Feedback. arXiv preprint arXiv:1911.08054 (2019)."},{"key":"e_1_3_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3085504.3085526"},{"key":"e_1_3_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132847.3132938"},{"key":"e_1_3_2_2_48_1","volume-title":"Reducing Disparate Exposure in Ranking: A Learning To Rank Approach. In WWW '20: The Web Conference 2020","author":"Zehlike Meike","year":"2020","unstructured":"Meike Zehlike and Carlos Castillo . 2020 . Reducing Disparate Exposure in Ranking: A Learning To Rank Approach. In WWW '20: The Web Conference 2020 , Taipei, Taiwan, April 20--24 , 2020, Yennun Huang, Irwin King, Tie-Yan Liu, and Maarten van Steen (Eds.). 2849--2855. Meike Zehlike and Carlos Castillo. 2020. Reducing Disparate Exposure in Ranking: A Learning To Rank Approach. In WWW '20: The Web Conference 2020, Taipei, Taiwan, April 20--24, 2020, Yennun Huang, Irwin King, Tie-Yan Liu, and Maarten van Steen (Eds.). 2849--2855."},{"key":"e_1_3_2_2_49_1","volume-title":"SIGIR '09: Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval. 99--106","author":"Zhu Jianhan","unstructured":"Jianhan Zhu , Jun Wang , Ingemar J. Cox , and Michael J. Taylor . 2009. Risky business: modeling and exploiting uncertainty in information retrieval . In SIGIR '09: Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval. 99--106 . Jianhan Zhu, Jun Wang, Ingemar J. Cox, and Michael J. Taylor. 2009. Risky business: modeling and exploiting uncertainty in information retrieval. In SIGIR '09: Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval. 99--106."}],"event":{"name":"CIKM '20: The 29th ACM International Conference on Information and Knowledge Management","location":"Virtual Event Ireland","acronym":"CIKM '20","sponsor":["SIGWEB ACM Special Interest Group on Hypertext, Hypermedia, and Web","SIGIR ACM Special Interest Group on Information Retrieval"]},"container-title":["Proceedings of the 29th ACM International Conference on Information & Knowledge Management"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3340531.3411962","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3340531.3411962","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:01:22Z","timestamp":1750197682000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3340531.3411962"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,10,19]]},"references-count":49,"alternative-id":["10.1145\/3340531.3411962","10.1145\/3340531"],"URL":"https:\/\/doi.org\/10.1145\/3340531.3411962","relation":{},"subject":[],"published":{"date-parts":[[2020,10,19]]},"assertion":[{"value":"2020-10-19","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}