{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,16]],"date-time":"2026-07-16T05:18:53Z","timestamp":1784179133087,"version":"3.55.0"},"reference-count":63,"publisher":"Association for Computing Machinery (ACM)","issue":"6","license":[{"start":{"date-parts":[[2022,12,7]],"date-time":"2022-12-07T00:00:00Z","timestamp":1670371200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"NSF","award":["1934464, 1916505, and 1922658"],"award-info":[{"award-number":["1934464, 1916505, and 1922658"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Comput. Surv."],"published-print":{"date-parts":[[2023,7,31]]},"abstract":"<jats:p>In the past few years, there has been much work on incorporating fairness requirements into algorithmic rankers, with contributions coming from the data management, algorithms, information retrieval, and recommender systems communities. In this survey, we give a systematic overview of this work, offering a broad perspective that connects formalizations and algorithmic approaches across subfields. An important contribution of our work is in developing a common narrative around the value frameworks that motivate specific fairness-enhancing interventions in ranking. This allows us to unify the presentation of mitigation objectives and of algorithmic techniques to help meet those objectives or identify trade-offs.<\/jats:p>\n          <jats:p\/>\n          <jats:p>In the first part of this survey, we describe four classification frameworks for fairness-enhancing interventions, along which we relate the technical methods surveyed in this article, discuss evaluation datasets, and present technical work on fairness in score-based ranking. In the second part of this survey, we present methods that incorporate fairness in supervised learning, and also give representative examples of recent work on fairness in recommendation and matchmaking systems. We also discuss evaluation frameworks for fair score-based ranking and fair learning-to-rank, and draw a set of recommendations for the evaluation of fair ranking methods.<\/jats:p>","DOI":"10.1145\/3533380","type":"journal-article","created":{"date-parts":[[2022,5,11]],"date-time":"2022-05-11T11:59:28Z","timestamp":1652270368000},"page":"1-41","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":55,"title":["Fairness in Ranking, Part II: Learning-to-Rank and Recommender Systems"],"prefix":"10.1145","volume":"55","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8738-2993","authenticated-orcid":false,"given":"Meike","family":"Zehlike","sequence":"first","affiliation":[{"name":"Humboldt University of Berlin, Max Planck Institute for Software Systems, and Zalando Research, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1617-5986","authenticated-orcid":false,"given":"Ke","family":"Yang","sequence":"additional","affiliation":[{"name":"New York University, NY, and University of Massachusetts, Amherst, MA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1587-0450","authenticated-orcid":false,"given":"Julia","family":"Stoyanovich","sequence":"additional","affiliation":[{"name":"New York University, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2022,12,7]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"AirBnB. (????). AirBnB. Retrieved from https:\/\/insideairbnb.com."},{"key":"e_1_3_2_3_2","unstructured":"Julia Angwin Jeff Larson Surya Mattu and Lauren Kirchner. 2016. Machine bias. ProPublica. Retrieved from https:\/\/www.propublica.org\/article\/machine-bias-risk-assessments-in-criminal-sentencing."},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/3209581"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/3292500.3330745"},{"key":"e_1_3_2_6_2","volume-title":"Proceedings of the 28th Text REtrieval Conference","author":"Biega Asia J.","year":"2019","unstructured":"Asia J. Biega, Fernando Diaz, Michael D. Ekstrand, and Sebastian Kohlmeier. 2019. Overview of the TREC 2019 fair ranking track. In Proceedings of the 28th Text REtrieval Conference."},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3210063"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/3077136.3080830"},{"issue":"23","key":"e_1_3_2_9_2","first-page":"81","article-title":"From ranknet to lambdarank to lambdamart: An overview","volume":"11","author":"Burges Christopher J. C.","year":"2010","unstructured":"Christopher J. C. Burges. 2010. From ranknet to lambdarank to lambdamart: An overview. Learning 11, 23\u2013581 (2010), 81.","journal-title":"Learning"},{"key":"e_1_3_2_10_2","article-title":"Multisided fairness for recommendation","author":"Burke Robin","year":"2017","unstructured":"Robin Burke. 2017. Multisided fairness for recommendation. arXiv:1707.00093. Retrieved from https:\/\/arxiv.org\/abs\/1707.00093.","journal-title":"arXiv:1707.00093"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/1273496.1273513"},{"key":"e_1_3_2_12_2","volume-title":"Proceedings of the 45th International Colloquium on Automata, Languages, and Programming","author":"Celis L. Elisa","year":"2018","unstructured":"L. Elisa Celis, Damian Straszak, and Nisheeth K. Vishnoi. 2018. Ranking with fairness constraints. In Proceedings of the 45th International Colloquium on Automata, Languages, and Programming. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik."},{"key":"e_1_3_2_13_2","article-title":"Bias and debias in recommender system: A survey and future directions","author":"Chen Jiawei","year":"2020","unstructured":"Jiawei Chen, Hande Dong, Xiang Wang, Fuli Feng, Meng Wang, and Xiangnan He. 2020. Bias and debias in recommender system: A survey and future directions. arXiv:2010.03240. Retrieved from https:\/\/arxiv.org\/abs\/2010.03240.","journal-title":"arXiv:2010.03240"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4899-7637-6_4"},{"key":"e_1_3_2_15_2","first-page":"1","article-title":"A flexible framework for evaluating user and item fairness in recommender systems","author":"Deldjoo Yashar","year":"2021","unstructured":"Yashar Deldjoo, Vito Walter Anelli, Hamed Zamani, Alejandro Bellogin, and Tommaso Di Noia. 2021. A flexible framework for evaluating user and item fairness in recommender systems. User Modeling and User-Adapted Interaction 31, 3 (2021), 1\u201355.","journal-title":"User Modeling and User-Adapted Interaction"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/2090236.2090255"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/3331184.3331380"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3383313.3411545"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/3437963.3441824"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/3292500.3330691"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/2827872"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/1864708.1864736"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/582415.582418"},{"key":"e_1_3_2_24_2","volume-title":"ACM SIGIR Forum","author":"Joachims Thorsten","year":"2017","unstructured":"Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, and Geri Gay. 2017. Accurately interpreting clickthrough data as implicit feedback. ACM SIGIR Forum, Vol. 51. Acm New York, NY, 4\u201311."},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/956750.956823"},{"key":"e_1_3_2_26_2","first-page":"187","volume-title":"Proceedings of the Conference on Fairness, Accountability and Transparency","author":"Kamishima Toshihiro","year":"2018","unstructured":"Toshihiro Kamishima, Shotaro Akaho, Hideki Asoh, and Jun Sakuma. 2018. Recommendation independence. In Proceedings of the Conference on Fairness, Accountability and Transparency. 187\u2013201."},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2019.00121"},{"key":"e_1_3_2_28_2","article-title":"Operationalizing individual fairness with pairwise fair representations","author":"Lahoti Preethi","year":"2019","unstructured":"Preethi Lahoti, Krishna P. Gummadi, and Gerhard Weikum. 2019. Operationalizing individual fairness with pairwise fair representations. arXiv:1907.01439. Retrieved from https:\/\/arxiv.org\/abs\/1907.01439.","journal-title":"arXiv:1907.01439"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/3184558.3186949"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.2200\/S00607ED2V01Y201410HLT026"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/3442381.3449866"},{"key":"e_1_3_2_32_2","unstructured":"Yunqi Li Yingqiang Ge and Yongfeng Zhang. 2021. Tutorial on fairness of machine learning in recommender systems. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval ."},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/3298689.3347016"},{"key":"e_1_3_2_34_2","first-page":"188","article-title":"Evaluation in information retrieval","volume":"1","author":"Manning Christopher D.","year":"2008","unstructured":"Christopher D. Manning, Prabhakar Raghavan, and Hinrich Sch\u00fctze. 2008. Evaluation in information retrieval. Introduction to Information Retrieval 1 (2008), 188\u2013210.","journal-title":"Introduction to Information Retrieval"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/3269206.3272027"},{"key":"e_1_3_2_36_2","unstructured":"ProPublica. (????). Correctional Offender Management Profiling for Alternative Sanctions. Retrieved from https:\/\/github.com\/propublica\/compas-analysis."},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3289600.3291002"},{"key":"e_1_3_2_38_2","unstructured":"UCI Machine Learning Repository. (????). German Credit. Retrieved from https:\/\/archive.ics.uci.edu\/ml\/machine-learning-databases\/statlog\/german\/."},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/1242572.1242643"},{"key":"e_1_3_2_40_2","unstructured":"SAT. (????). SAT. Retrieved from https:\/\/www.qsleap.com\/sat\/resources\/sat-2014-percentiles."},{"key":"e_1_3_2_41_2","first-page":"197","article-title":"Equality of what?","volume":"1","author":"Sen Amartya","year":"1980","unstructured":"Amartya Sen. 1980. Equality of what?The Tanner Lecture on Human Values 1 (1980), 197\u2013220.","journal-title":"The Tanner Lecture on Human Values"},{"key":"e_1_3_2_42_2","unstructured":"Law School Admission Council Research Report Series. (????). LSAC National Longitudinal Bar Passage Study. Retrieved from https:\/\/github.com\/MilkaLichtblau\/DELTR-Experiments\/tree\/master\/data\/LawStudents."},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3220088"},{"key":"e_1_3_2_44_2","article-title":"Policy learning for fairness in ranking","author":"Singh Ashudeep","year":"2019","unstructured":"Ashudeep Singh and Thorsten Joachims. 2019. Policy learning for fairness in ranking. arXiv:1902.04056. Retrieved from https:\/\/arxiv.org\/abs\/1902.04056.","journal-title":"arXiv:1902.04056"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/3314183.3323845"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/3450613.3456835"},{"key":"e_1_3_2_47_2","unstructured":"StackExchange. (????). StackExchange. Retrieved from https:\/\/stackexchange.com\/."},{"key":"e_1_3_2_48_2","unstructured":"Engineering students. (????). Engineering Students. Retrieved from https:\/\/github.com\/MilkaLichtblau\/DELTR-Experiments\/tree\/master\/data\/EngineeringStudents."},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/3292500.3330793"},{"key":"e_1_3_2_50_2","article-title":"Ricci vs. DeStefano (Nos. 07-1428 and 08-328), 530 F. 3d 87, Reversed and Remanded","author":"States Supreme Court of the United","year":"2009","unstructured":"Supreme Court of the United States. 2009. Ricci vs. DeStefano (Nos. 07-1428 and 08-328), 530 F. 3d 87, Reversed and Remanded. Retrieved from https:\/\/www.law.cornell.edu\/supct\/html\/07-1428.ZO.html.","journal-title":"Retrieved from https:\/\/www.law.cornell.edu\/supct\/html\/07-1428.ZO.html"},{"key":"e_1_3_2_51_2","unstructured":"TREC. (????). W3C Experts. Retrieved from https:\/\/github.com\/MilkaLichtblau\/DELTR-Experiments\/tree\/master\/data\/TREC."},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1145\/3437963.3441732"},{"key":"e_1_3_2_53_2","volume-title":"LSAC National Longitudinal Bar Passage Study","author":"Wightman Linda F.","year":"1998","unstructured":"Linda F. Wightman and Henry Ramsey. 1998. LSAC National Longitudinal Bar Passage Study. Law School Admission Council."},{"key":"e_1_3_2_54_2","unstructured":"Xing. (????). XING. Retrieved from https:\/\/github.com\/MilkaLichtblau\/xing_dataset."},{"key":"e_1_3_2_55_2","unstructured":"Yahoo. (????). The Yahoo Webscope Program. Retrieved from https:\/\/webscope.sandbox.yahoo.com\/."},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/3085504.3085526"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/3183713.3193568"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/3132847.3132938"},{"key":"e_1_3_2_59_2","article-title":"Reducing disparate exposure in ranking: A learning to rank approach","author":"Zehlike Meike","year":"2018","unstructured":"Meike Zehlike and Carlos Castillo. 2018. Reducing disparate exposure in ranking: A learning to rank approach. arXiv:1805.08716. Retrieved from https:\/\/arxiv.org\/abs\/1805.08716.","journal-title":"arXiv:1805.08716"},{"key":"e_1_3_2_60_2","first-page":"1","article-title":"Matching code and law: Achieving algorithmic fairness with optimal transport","author":"Zehlike Meike","year":"2017","unstructured":"Meike Zehlike, Philipp Hacker, and Emil Wiedemann. 2017. Matching code and law: Achieving algorithmic fairness with optimal transport. Data Mining and Knowledge Discovery 34, 1 (2017), 1\u201338.","journal-title":"Data Mining and Knowledge Discovery"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2021.102707"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1145\/3366424.3383534"},{"key":"e_1_3_2_63_2","unstructured":"Yi Zhang. (????). Yow News Recommendation. Retrieved from https:\/\/www.younow.com."},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","DOI":"10.1145\/3269206.3271795"}],"container-title":["ACM Computing Surveys"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3533380","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3533380","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3533380","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:00:17Z","timestamp":1750186817000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3533380"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,12,7]]},"references-count":63,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2023,7,31]]}},"alternative-id":["10.1145\/3533380"],"URL":"https:\/\/doi.org\/10.1145\/3533380","relation":{},"ISSN":["0360-0300","1557-7341"],"issn-type":[{"value":"0360-0300","type":"print"},{"value":"1557-7341","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,12,7]]},"assertion":[{"value":"2021-06-08","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-04-23","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-12-07","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}