{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,11]],"date-time":"2026-06-11T09:15:46Z","timestamp":1781169346923,"version":"3.54.1"},"reference-count":86,"publisher":"Association for Computing Machinery (ACM)","issue":"6","license":[{"start":{"date-parts":[[2022,12,7]],"date-time":"2022-12-07T00:00:00Z","timestamp":1670371200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"NSF","award":["1934464, 1916505, and 1922658"],"award-info":[{"award-number":["1934464, 1916505, and 1922658"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Comput. Surv."],"published-print":{"date-parts":[[2023,7,31]]},"abstract":"<jats:p>In the past few years, there has been much work on incorporating fairness requirements into algorithmic rankers, with contributions coming from the data management, algorithms, information retrieval, and recommender systems communities. In this survey, we give a systematic overview of this work, offering a broad perspective that connects formalizations and algorithmic approaches across sub-fields. An important contribution of our work is in developing a common narrative around the value frameworks that motivate specific fairness-enhancing interventions in ranking. This allows us to unify the presentation of mitigation objectives and of algorithmic techniques to help meet those objectives or identify trade-offs.<\/jats:p>\n          <jats:p>In this first part of this survey, we describe four classification frameworks for fairness-enhancing interventions, along which we relate the technical methods surveyed in this article, discuss evaluation datasets, and present technical work on fairness in score-based ranking. In the second part of this survey, we present methods that incorporate fairness in supervised learning, and also give representative examples of recent work on fairness in recommendation and matchmaking systems. We also discuss evaluation frameworks for fair score-based ranking and fair learning-to-rank, and draw a set of recommendations for the evaluation of fair ranking methods.<\/jats:p>","DOI":"10.1145\/3533379","type":"journal-article","created":{"date-parts":[[2022,5,11]],"date-time":"2022-05-11T11:59:28Z","timestamp":1652270368000},"page":"1-36","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":59,"title":["Fairness in Ranking, Part I: Score-Based Ranking"],"prefix":"10.1145","volume":"55","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8738-2993","authenticated-orcid":false,"given":"Meike","family":"Zehlike","sequence":"first","affiliation":[{"name":"Humboldt University of Berlin, Max Planck Institute for Software Systems, and Zalando Research, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1617-5986","authenticated-orcid":false,"given":"Ke","family":"Yang","sequence":"additional","affiliation":[{"name":"New York University, NY, and University of Massachusetts, Amherst, MA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1587-0450","authenticated-orcid":false,"given":"Julia","family":"Stoyanovich","sequence":"additional","affiliation":[{"name":"New York University, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2022,12,7]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"AirBnB. AirBnB. (????). https:\/\/insideairbnb.com."},{"key":"e_1_3_2_3_2","unstructured":"Julia Angwin Jeff Larson Surya Mattu and Lauren Kirchner. 2016. Machine bias. ProPublica. https:\/\/www.propublica.org\/article\/machine-bias-risk-assessments-in-criminal-sentencing."},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1111\/ecoj.12531"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/3299869.3300079"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.14778\/3415478.3415566"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/1399589.1399596"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/3209581"},{"key":"e_1_3_2_9_2","first-page":"2","article-title":"Fairness in machine learning","volume":"1","author":"Barocas Solon","year":"2017","unstructured":"Solon Barocas, Moritz Hardt, and Arvind Narayanan. 2017. Fairness in machine learning. Nips Tutorial 1 (2017), 2.","journal-title":"Nips Tutorial"},{"key":"e_1_3_2_10_2","unstructured":"Surender Baswana P. P. Chakrabarti Yashodhan Kanoria Utkarsh Patange and Sharat Chandran. 2019. Joint seat allocation 2018: An algorithmic perspective. arxiv:1904.06698. Retrieved from http:\/\/arxiv.org\/abs\/1904.06698."},{"key":"e_1_3_2_11_2","unstructured":"Rachel K. E. Bellamy Kuntal Dey Michael Hind Samuel C. Hoffman Stephanie Houde Kalapriya Kannan Pranay Lohia Jacquelyn Martino Sameep Mehta Aleksandra Mojsilovic Seema Nagar Karthikeyan Natesan Ramamurthy John T. Richards Diptikalyan Saha Prasanna Sattigeri Moninder Singh Kush R. Varshney and Yunfeng Zhang. 2018. AI fairness 360: An extensible toolkit for detecting understanding and mitigating unwanted algorithmic bias. arxiv:1810.01943. Retrieved from http:\/\/arxiv.org\/abs\/1810.01943."},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3210063"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/3077136.3080830"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.2307\/2334029"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3308774.3308783"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372858"},{"key":"e_1_3_2_17_2","volume-title":"Proceedings of the 45th International Colloquium on Automata, Languages, and Programming","author":"Celis L. Elisa","year":"2018","unstructured":"L. Elisa Celis, Damian Straszak, and Nisheeth K. Vishnoi. 2018. Ranking with fairness constraints. In Proceedings of the 45th International Colloquium on Automata, Languages, and Programming. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik."},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3376898"},{"key":"e_1_3_2_19_2","first-page":"S44\u2013S50","article-title":"The medical expenditure panel survey: A national information resource to support healthcare cost research and inform policy and practice","author":"Cohen Joel W.","year":"2009","unstructured":"Joel W. Cohen, Steven B. Cohen, and Jessica S. Banthin. 2009. The medical expenditure panel survey: A national information resource to support healthcare cost research and inform policy and practice. Medical Care (2009), S44\u2013S50.","journal-title":"Medical Care"},{"key":"e_1_3_2_20_2","unstructured":"MIT Collective Learning Group. Pantheon. (????). Retrieved from https:\/\/github.com\/DataResponsibly\/Datasets."},{"issue":"3","key":"e_1_3_2_21_2","volume":"82","author":"Collins Brian","year":"2007","unstructured":"Brian Collins. 2007. NYU Law Review 82, 3 (2007), 870.","journal-title":"NYU Law Review"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.4324\/9780203900055"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/3306618.3314236"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.2307\/1229039"},{"key":"e_1_3_2_25_2","unstructured":"CS Rankings. CSRankings: Computer Science Rankings. (????). Retrieved from https:\/\/csrankings.org."},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/11805.001.0001"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1089\/big.2016.0054"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/2090236.2090255"},{"issue":"3","key":"e_1_3_2_29_2","first-page":"185","article-title":"What is equality? part 1: Equality of welfare","volume":"10","author":"Dworkin Ronald","year":"1981","unstructured":"Ronald Dworkin. 1981. What is equality? part 1: Equality of welfare. Philosophy and Public Affairs 10, 3 (1981), 185\u2013246. Retrieved from http:\/\/www.jstor.org\/stable\/2264894.","journal-title":"Philosophy and Public Affairs"},{"key":"e_1_3_2_30_2","article-title":"The optimum choice of the instant for stopping a Markov process","volume":"4","author":"Dynkin E. B.","year":"1963","unstructured":"E. B. Dynkin. 1963. The optimum choice of the instant for stopping a Markov process. Sov. Math. Dokl. 4 (1963), 627\u2013629.","journal-title":"Sov. Math. Dokl."},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/3298689.3346964"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1214\/ss\/1177012493"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1093\/acprof:oso\/9780199812141.001.0001"},{"key":"e_1_3_2_34_2","unstructured":"Forbes. Forbes Richest Americans. (????). Retrieved from https:\/\/github.com\/DataResponsibly\/Datasets."},{"key":"e_1_3_2_35_2","article-title":"On the (im) possibility of fairness","author":"Friedler Sorelle A.","year":"2016","unstructured":"Sorelle A. Friedler, Carlos Scheidegger, and Suresh Venkatasubramanian. 2016. On the (im) possibility of fairness. arXiv:1609.07236. Retrieved from https:\/\/arxiv.org\/abs\/1609.07236.","journal-title":"arXiv:1609.07236"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/230538.230561"},{"key":"e_1_3_2_37_2","doi-asserted-by":"crossref","unstructured":"Vincent Grari Boris Ruf Sylvain Lamprier and Marcin Detyniecki. 2019. Fairness-Aware Neural R\u00e9yni Minimization for Continuous Features. arxiv:1911.04929. Retrieved from https:\/\/arxiv.org\/abs\/1911.04929.","DOI":"10.24963\/ijcai.2020\/313"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2945386"},{"key":"e_1_3_2_39_2","first-page":"3315","volume-title":"Proceedings of the Advances in Neural Information Processing Systems","author":"Hardt Moritz","year":"2016","unstructured":"Moritz Hardt, Eric Price, and Nati Srebro. 2016. Equality of opportunity in supervised learning. In Proceedings of the Advances in Neural Information Processing Systems. 3315\u20133323."},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287584"},{"key":"e_1_3_2_41_2","volume-title":"Proceedings of the ACM Conference on Fairness, Accountability, and Transparency","author":"Khan Falaah Arif","year":"2021","unstructured":"Falaah Arif Khan, Eleni Manis, and Julia Stoyanovich. 2021. Translation tutorial: Fairness and friends. In Proceedings of the ACM Conference on Fairness, Accountability, and Transparency."},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.4230\/LIPIcs.ITCS.2018.33"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/1772690.1772749"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2019.00121"},{"key":"e_1_3_2_45_2","unstructured":"M. Lichman. 2013. UCI Machine Learning Repository. (2013)."},{"issue":"1","key":"e_1_3_2_46_2","first-page":"39","article-title":"Dynamic programming and decision theory","volume":"10","author":"Lindley D. V.","year":"1961","unstructured":"D. V. Lindley. 1961. Dynamic programming and decision theory. Journal of the Royal Statistical Society 10, 1 (March1961), 39\u201351.","journal-title":"Journal of the Royal Statistical Society"},{"key":"e_1_3_2_47_2","article-title":"Multiple, compound and intersectional discrimination: Bringing the experiences of the most marginalized to the fore","author":"Makkonen Timo","year":"2002","unstructured":"Timo Makkonen. 2002. Multiple, compound and intersectional discrimination: Bringing the experiences of the most marginalized to the fore. Institute for Human Rights, \u00c5bo Akademi University (2002).","journal-title":"Institute for Human Rights, \u00c5bo Akademi University"},{"key":"e_1_3_2_48_2","first-page":"4382","volume-title":"Proceedings of the 36th International Conference on Machine Learning (Proceedings of Machine Learning Research)","volume":"97","author":"Mary Jeremie","year":"2019","unstructured":"Jeremie Mary, Cl\u00e9ment Calauz\u00e8nes, and Noureddine El Karoui. 2019. Fairness-aware learning for continuous attributes and treatments. In Proceedings of the 36th International Conference on Machine Learning (Proceedings of Machine Learning Research), Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.), Vol. 97. PMLR, 4382\u20134391. Retrieved from https:\/\/proceedings.mlr.press\/v97\/mary19a.html."},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/3457607"},{"key":"e_1_3_2_50_2","unstructured":"NASA. Astronauts. (????). Retrieved from https:\/\/github.com\/DataResponsibly\/Datasets."},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.2307\/j.ctt1pwt9w5"},{"key":"e_1_3_2_52_2","unstructured":"Department of Health and Human Services. Medical Expenditure Panel Survey. (????). Retrieved from https:\/\/meps.ahrq.gov\/mepsweb\/."},{"key":"e_1_3_2_53_2","unstructured":"Bureau of Transportation Statistics. National Summary of U.S. Flights. (????). Retrieved from https:\/\/www.transtats.bts.gov."},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1111\/j.1083-6101.2007.00351.x"},{"key":"e_1_3_2_55_2","unstructured":"Dana Pessach and Erez Shmueli. 2020. Algorithmic fairness. arXiv:2001.09784. Retrieved from https:\/\/arxiv.org\/abs\/2001.09784."},{"key":"e_1_3_2_56_2","unstructured":"ProPublica. Correctional Offender Management Profiling for Alternative Sanctions. (????). Retrieved from https:\/\/github.com\/propublica\/compas-analysis."},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.4159\/9780674042605"},{"key":"e_1_3_2_58_2","unstructured":"Richard V. Reeves and Dimitrios Halikias. 2017. Race gaps in SAT scores highlight inequality and hinder upward mobility. (2017). Retrieved from https:\/\/www.brookings.edu\/research\/race-gaps-in-sat-scores-highlight-inequality-and-hinder-upward-mobility."},{"key":"e_1_3_2_59_2","unstructured":"UCI Machine Learning Repository. German Credit. (????). Retrieved from https:\/\/archive.ics.uci.edu\/ml\/machine-learning-databases\/statlog\/german\/."},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1007\/s003550100123"},{"key":"e_1_3_2_61_2","unstructured":"SAT. SAT. (????). Retrieved from https:\/\/www.qsleap.com\/sat\/resources\/sat-2014-percentiles."},{"key":"e_1_3_2_62_2","unstructured":"Semantic Scholar. Semantic Scholar Open Research Corpus. (????). Retrieved from https:\/\/api.semanticscholar.org\/corpus\/."},{"key":"e_1_3_2_63_2","unstructured":"Law School Admission Council Research Report Series. LSAC National Longitudinal Bar Passage Study. (????). Retrieved from https:\/\/github.com\/MilkaLichtblau\/DELTR-Experiments\/tree\/master\/data\/LawStudents."},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11199-008-9501-8"},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3220088"},{"key":"e_1_3_2_66_2","article-title":"Policy learning for fairness in ranking","author":"Singh Ashudeep","year":"2019","unstructured":"Ashudeep Singh and Thorsten Joachims. 2019. Policy learning for fairness in ranking. arXiv:1902.04056. Retrieved from https:\/\/arxiv.org\/abs\/1902.04056.","journal-title":"arXiv:1902.04056"},{"key":"e_1_3_2_67_2","doi-asserted-by":"crossref","unstructured":"Charles Spearman. 1961. The proof and measurement of association between two things. (1961).","DOI":"10.1037\/11491-005"},{"key":"e_1_3_2_68_2","unstructured":"StackExchange. StackExchange. (????). Retrieved from https:\/\/stackexchange.com\/."},{"key":"e_1_3_2_69_2","doi-asserted-by":"publisher","DOI":"10.5441\/002\/edbt.2018.22"},{"key":"e_1_3_2_70_2","unstructured":"Engineering students. Engineering Students. (????). Retrieved from https:\/\/github.com\/MilkaLichtblau\/DELTR-Experiments\/tree\/master\/data\/EngineeringStudents."},{"key":"e_1_3_2_71_2","unstructured":"Supreme Court of the United States. 2009. Ricci v. DeStefano (Nos. 07-1428 and 08-328) 530 F. 3d 87 Reversed and Remanded. Retrieved from https:\/\/www.law.cornell.edu\/supct\/html\/07-1428.ZO.html. (2009)."},{"key":"e_1_3_2_72_2","unstructured":"Indian Institute Of Technology. IIT-JEE. (????). Retrieved from https:\/\/indiankanoon.org\/doc\/1955304\/."},{"key":"e_1_3_2_73_2","unstructured":"The College Board. 2014. SAT Percentile Ranks. (2014)."},{"key":"e_1_3_2_74_2","unstructured":"TREC. W3C Experts. (????). Retrieved from https:\/\/github.com\/MilkaLichtblau\/DELTR-Experiments\/tree\/master\/data\/TREC."},{"key":"e_1_3_2_75_2","volume-title":"LSAC National Longitudinal Bar Passage Study","author":"Wightman Linda F.","year":"1998","unstructured":"Linda F. Wightman and Henry Ramsey. 1998. LSAC National Longitudinal Bar Passage Study. Law School Admission Council."},{"key":"e_1_3_2_76_2","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3220087"},{"key":"e_1_3_2_77_2","unstructured":"Xing. XING. (????). Retrieved from https:\/\/github.com\/MilkaLichtblau\/xing_dataset."},{"key":"e_1_3_2_78_2","unstructured":"Yahoo. The Yahoo Webscope Program. (????). Retrieved from https:\/\/webscope.sandbox.yahoo.com\/."},{"key":"e_1_3_2_79_2","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2019\/836"},{"key":"e_1_3_2_80_2","doi-asserted-by":"publisher","DOI":"10.4230\/LIPIcs.FORC.2021.7"},{"key":"e_1_3_2_81_2","doi-asserted-by":"publisher","DOI":"10.1145\/3085504.3085526"},{"key":"e_1_3_2_82_2","doi-asserted-by":"publisher","DOI":"10.1145\/3132847.3132938"},{"key":"e_1_3_2_83_2","article-title":"Reducing disparate exposure in ranking: A learning to rank approach","author":"Zehlike Meike","year":"2018","unstructured":"Meike Zehlike and Carlos Castillo. 2018. Reducing disparate exposure in ranking: A learning to rank approach. arXiv:1805.08716. Retrieved from https:\/\/arxiv.org\/abs\/1805.08716.","journal-title":"arXiv:1805.08716"},{"key":"e_1_3_2_84_2","first-page":"1","article-title":"Matching code and law: Achieving algorithmic fairness with optimal transport","author":"Zehlike Meike","year":"2017","unstructured":"Meike Zehlike, Philipp Hacker, and Emil Wiedemann. 2017. Matching code and law: Achieving algorithmic fairness with optimal transport. Data Mining and Knowledge Discovery 34, 1 (2017), 1\u201338.","journal-title":"Data Mining and Knowledge Discovery"},{"key":"e_1_3_2_85_2","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Zemel Richard S.","year":"2013","unstructured":"Richard S. Zemel, Yu Wu, Kevin Swersky, Toniann Pitassi, and Cynthia Dwork. 2013. Learning fair representations. In Proceedings of the International Conference on Machine Learning. http:\/\/jmlr.org\/proceedings\/papers\/v28\/zemel13.html."},{"key":"e_1_3_2_86_2","unstructured":"Yi Zhang. Yow News Recommendation. (????). Retrieved from https:\/\/www.younow.com."},{"key":"e_1_3_2_87_2","doi-asserted-by":"publisher","DOI":"10.5555\/1145256"}],"container-title":["ACM Computing Surveys"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3533379","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3533379","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3533379","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:00:17Z","timestamp":1750186817000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3533379"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,12,7]]},"references-count":86,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2023,7,31]]}},"alternative-id":["10.1145\/3533379"],"URL":"https:\/\/doi.org\/10.1145\/3533379","relation":{},"ISSN":["0360-0300","1557-7341"],"issn-type":[{"value":"0360-0300","type":"print"},{"value":"1557-7341","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,12,7]]},"assertion":[{"value":"2021-06-08","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-04-23","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-12-07","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}