{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,14]],"date-time":"2026-04-14T18:02:18Z","timestamp":1776189738710,"version":"3.50.1"},"reference-count":55,"publisher":"Association for Computing Machinery (ACM)","issue":"1-2","license":[{"start":{"date-parts":[[2025,1,17]],"date-time":"2025-01-17T00:00:00Z","timestamp":1737072000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"crossref","award":["530081187"],"award-info":[{"award-number":["530081187"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Trans. Soc. Comput."],"published-print":{"date-parts":[[2025,6,30]]},"abstract":"<jats:p>In recent years, several measures have been developed for evaluating group fairness of rankings. Given that these measures were developed with different application contexts and ranking algorithms in mind, it is not straightforward which measure to choose for a given scenario. Previous work has already laid out some categorizations of measures and explored relationships between these; however, to the best of our knowledge, there has not yet been a thorough mathematical analysis of practically grounded properties of individual measures. In this article, we therefore apply an axiomatic approach to perform a comprehensive analysis of existing group fairness measures that have been developed in the context of fair ranking. To this end, we propose a set of 14 properties for group fairness measures that consider different ranking settings. These properties specifically provide information about how fairness scores of ranked outputs can be interpreted and contextualized. For a given use case, one can then identify which properties are of interest and select a measure based on whether it satisfies these properties. We further apply our properties on 12 existing group fairness measures, and through both empirical and theoretical results demonstrate that most of these measures only satisfy a small subset of the proposed properties. These findings highlight limitations of existing measures, and provide insights into how to evaluate and interpret different fairness measures in practical deployment. Overall, our work can assist practitioners in selecting appropriate group fairness measures for a specific application, and also aid researchers in designing and evaluating such measures.<\/jats:p>","DOI":"10.1145\/3674883","type":"journal-article","created":{"date-parts":[[2024,8,27]],"date-time":"2024-08-27T10:13:57Z","timestamp":1724753637000},"page":"1-45","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":6,"title":["Properties of Group Fairness Measures for Rankings"],"prefix":"10.1145","volume":"8","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3091-5095","authenticated-orcid":false,"given":"Tobias","family":"Schumacher","sequence":"first","affiliation":[{"name":"University of Mannheim, Germany and RWTH Aachen University, Mannheim, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8265-2410","authenticated-orcid":false,"given":"Marlene","family":"Lutz","sequence":"additional","affiliation":[{"name":"University of Mannheim, Mannheim, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8957-7364","authenticated-orcid":false,"given":"Sandipan","family":"Sikdar","sequence":"additional","affiliation":[{"name":"Leibniz Universitat Hannover, Hanover, Germany"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5485-5720","authenticated-orcid":false,"given":"Markus","family":"Strohmaier","sequence":"additional","affiliation":[{"name":"University of Mannheim, Mannheim, Germany, GESIS - Leibniz Institute for the Social Sciences, Cologne, Germany and Complexity Science Hub, Vienna, Austria"}]}],"member":"320","published-online":{"date-parts":[[2025,1,17]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-008-9066-8"},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/2484028.2484081"},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-019-09355-y"},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-020-09374-0"},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3210024"},{"key":"e_1_3_3_7_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2022.103115"},{"key":"e_1_3_3_8_2","first-page":"254","volume-title":"Ethics of Data and Analytics","author":"Angwin Julia","year":"2016","unstructured":"Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2016. Machine bias. In Ethics of Data and Analytics. Auerbach Publications, New York, NY, USA, 254\u2013264."},{"key":"e_1_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/3209581"},{"key":"e_1_3_3_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3539618.3591933"},{"key":"e_1_3_3_11_2","volume-title":"Fairness and Machine Learning: Limitations and Opportunities","author":"Barocas Solon","year":"2023","unstructured":"Solon Barocas, Moritz Hardt, and Arvind Narayanan. 2023. Fairness and Machine Learning: Limitations and Opportunities. MIT Press, Cambridge, MA, USA."},{"key":"e_1_3_3_12_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.aacl-main.61"},{"key":"e_1_3_3_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/3292500.3330745"},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3210063"},{"key":"e_1_3_3_15_2","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Bower Amanda","year":"2021","unstructured":"Amanda Bower, Hamid Eftekhari, Mikhail Yurochkin, and Yuekai Sun. 2021. Individually fair rankings. In Proceedings of the International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=71zCSP_HuBN"},{"key":"e_1_3_3_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/3308774.3308783"},{"key":"e_1_3_3_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372858"},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","unstructured":"L. Elisa Celis Damian Straszak and Nisheeth K. Vishnoi. 2018. Ranking with fairness constraints. In 45th International Colloquium on Automata Languages and Programming (ICALP\u201918) Ioannis Chatzigiannakis Christos Kalkamanis D\u00e1niel Marx and Donald Sannella (Eds.). Leibniz International Proceedings in Informatics Vol. 107. Schloss Dagstuhl\u2013Leibniz-Zentrum fuer Informatik Dagstuhl Germany Article 28 15 pages. DOI:10.4230\/LIPIcs.ICALP.2018.28","DOI":"10.4230\/LIPIcs.ICALP.2018.28"},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","DOI":"10.1089\/big.2016.0047"},{"key":"e_1_3_3_20_2","first-page":"296","volume-title":"Ethics of Data and Analytics","author":"Dastin Jeffrey","year":"2018","unstructured":"Jeffrey Dastin. 2018. Amazon scraps secret AI recruiting tool that showed bias against women. In Ethics of Data and Analytics. Auerbach Publications, New York, NY, USA, 296\u2013299."},{"key":"e_1_3_3_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/3340531.3411962"},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468507.3468515"},{"key":"e_1_3_3_23_2","doi-asserted-by":"publisher","DOI":"10.6028\/NIST.SP.500-335.fair-overview"},{"key":"e_1_3_3_24_2","volume-title":"The Stanford Encyclopedia of Philosophy (Fall 2023 ed.)","author":"Elford Gideon","year":"2023","unstructured":"Gideon Elford. 2023. Equality of opportunity. In The Stanford Encyclopedia of Philosophy (Fall 2023 ed.), Edward N. Zalta and Uri Nodelman (Eds.). Metaphysics Research Lab, Stanford University, Stanford, CA, USA."},{"key":"e_1_3_3_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/3539597.3570459"},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/3433949"},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/230538.230561"},{"key":"e_1_3_3_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/3292500.3330691"},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/3404835.3462850"},{"key":"e_1_3_3_30_2","unstructured":"Sruthi Gorantla Amit Deshpande and Anand Louis. 2021. On the problem of underranking in group-fair ranking. In Proceedings of the 38th International Conference on Machine Learning Vol. 139. 3777\u20133787. https:\/\/proceedings.mlr.press\/v139\/gorantla21a.html"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998327"},{"key":"e_1_3_3_32_2","doi-asserted-by":"publisher","DOI":"10.1561\/1500000051"},{"key":"e_1_3_3_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/582415.582418"},{"key":"e_1_3_3_34_2","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/30.1-2.81"},{"key":"e_1_3_3_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/3461702.3462588"},{"key":"e_1_3_3_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401100"},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i04.5970"},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/3460231.3474234"},{"key":"e_1_3_3_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/3459637.3482099"},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1007\/s00778-021-00697-y"},{"key":"e_1_3_3_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/3477495.3532018"},{"key":"e_1_3_3_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/3631943"},{"key":"e_1_3_3_43_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2006.07.020"},{"key":"e_1_3_3_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/3308560.3317595"},{"key":"e_1_3_3_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/2808194.2809449"},{"key":"e_1_3_3_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3220088"},{"key":"e_1_3_3_47_2","volume-title":"Advances in Neural Information Processing Systems","author":"Singh Ashudeep","year":"2019","unstructured":"Ashudeep Singh and Thorsten Joachims. 2019. Policy learning for fairness in ranking. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alch\u00e9-Buc, E. Fox, and R. Garnett (Eds.). Vol. 32. Curran Associates, 1\u201311.https:\/\/proceedings.neurips.cc\/paper\/2019\/file\/9e82757e9a1c12cb710ad680db11f6f1-Paper.pdf"},{"key":"e_1_3_3_48_2","unstructured":"Yining Wang Liwei Wang Yuanzhi Li Di He and Tie-Yan Liu. 2013. A theoretical analysis of NDCG type ranking measures. In Proceedings of the 26th Annual Conference on Learning Theory. 25\u201354. https:\/\/proceedings.mlr.press\/v30\/Wang13.html"},{"key":"e_1_3_3_49_2","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v35i1.2504"},{"key":"e_1_3_3_50_2","volume-title":"LSAC National Longitudinal Bar Passage Study. Technical Report. LSAC Research Report Series.","author":"Wightman Linda F.","year":"1998","unstructured":"Linda F. Wightman. 1998. LSAC National Longitudinal Bar Passage Study. Technical Report. LSAC Research Report Series. Law School Admission Council, Newtown, PA, USA."},{"key":"e_1_3_3_51_2","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3220087"},{"key":"e_1_3_3_52_2","doi-asserted-by":"publisher","unstructured":"Ke Yang Joshua R. Loftus and Julia Stoyanovich. 2021. Causal intersectionality and fair ranking. In 2nd Symposium on Foundations of Responsible Computing (FORC\u201921) Katrina Ligett and Swati Gupta (Eds.). Leibniz International Proceedings in Informatics Vol. 192. Schloss Dagstuhl\u2013Leibniz-Zentrum f\u00fcr Informatik Dagstuhl Germany Article 7 20 pages. DOI:10.4230\/LIPIcs.FORC.2021.7","DOI":"10.4230\/LIPIcs.FORC.2021.7"},{"key":"e_1_3_3_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/3085504.3085526"},{"key":"e_1_3_3_54_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10618-019-00658-8"},{"key":"e_1_3_3_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/3533379"},{"key":"e_1_3_3_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/3533380"}],"container-title":["ACM Transactions on Social Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3674883","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3674883","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:05:56Z","timestamp":1750291556000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3674883"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,1,17]]},"references-count":55,"journal-issue":{"issue":"1-2","published-print":{"date-parts":[[2025,6,30]]}},"alternative-id":["10.1145\/3674883"],"URL":"https:\/\/doi.org\/10.1145\/3674883","relation":{},"ISSN":["2469-7818","2469-7826"],"issn-type":[{"value":"2469-7818","type":"print"},{"value":"2469-7826","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,1,17]]},"assertion":[{"value":"2023-04-29","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-05-30","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-01-17","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}