{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,25]],"date-time":"2026-03-25T01:00:03Z","timestamp":1774400403404,"version":"3.50.1"},"reference-count":68,"publisher":"Association for Computing Machinery (ACM)","issue":"3","funder":[{"name":"Algorithms, Data, and Democracy project"},{"DOI":"10.13039\/501100000923","name":"Australian Research Council","doi-asserted-by":"publisher","award":["DP190101113"],"award-info":[{"award-number":["DP190101113"]}],"id":[{"id":"10.13039\/501100000923","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Recomm. Syst."],"published-print":{"date-parts":[[2026,9,30]]},"abstract":"<jats:p>\n                    Recommender Systems (RSs) aim at providing relevant items to users, with a recent emphasis on improving recommendation fairness. Quantifying fairness of the recommended items can be done with two types of evaluation measures: measures that are purely based on item exposure (\n                    <jats:italic toggle=\"yes\">exposure-based<\/jats:italic>\n                    ) and measures that account for both item exposure and item relevance (\n                    <jats:italic toggle=\"yes\">relevance-aware<\/jats:italic>\n                    ). While exposure-based measures have been thoroughly analysed, relevance-aware measures have not been examined in such detail yet. We gather all existing relevance-aware individual item fairness measures for RSs and study their theoretical properties. We find that all measures suffer from one or more limitations, which may cause issues in their computation, interpretability, or expressiveness. To address this, we correct the affected measures or explain why a limitation is unresolvable. Further, we empirically investigate the extent of the limitations on the measures and compare the original measures to our reformulations under common and extreme evaluation scenarios across real-world and synthetic data. Our experiments show that our reformulated measures successfully resolve the issues in the original measures. We conclude by providing practical guidelines on how to select measures for a range of use cases.\n                  <\/jats:p>\n                  <jats:p\/>","DOI":"10.1145\/3765624","type":"journal-article","created":{"date-parts":[[2025,9,4]],"date-time":"2025-09-04T11:10:56Z","timestamp":1756984256000},"page":"1-45","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Relevance-aware Individual Item Fairness Measures for Recommender Systems: Limitations and Usage Guidelines"],"prefix":"10.1145","volume":"4","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1233-7690","authenticated-orcid":false,"given":"Theresia Veronika","family":"Rampisela","sequence":"first","affiliation":[{"name":"Computer Science, University of Copenhagen","place":["Kobenhavn, Denmark"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7001-4817","authenticated-orcid":false,"given":"Maria","family":"Maistro","sequence":"additional","affiliation":[{"name":"Computer Science, University of Copenhagen","place":["Kobenhavn, Denmark"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2203-4928","authenticated-orcid":false,"given":"Tuukka","family":"Ruotsalo","sequence":"additional","affiliation":[{"name":"Computer Science, University of Copenhagen","place":["Kobenhavn, Denmark"]},{"name":"LUT University","place":["Lappeenranta, Finland"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9094-0810","authenticated-orcid":false,"given":"Falk","family":"Scholer","sequence":"additional","affiliation":[{"name":"RMIT University","place":["Melbourne, Australia"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2600-2701","authenticated-orcid":false,"given":"Christina","family":"Lioma","sequence":"additional","affiliation":[{"name":"University of Copenhagen","place":["Kobenhavn, Denmark"]}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2026,3,5]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-07012-9_39"},{"key":"e_1_3_3_3_2","doi-asserted-by":"publisher","DOI":"10.1016\/J.IPM.2022.103115"},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3210063"},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/3523227.3546775"},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/3297662.3365798"},{"key":"e_1_3_3_7_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2025.103560"},{"key":"e_1_3_3_8_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-020-09371-3"},{"key":"e_1_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/2043932.2044016"},{"key":"e_1_3_3_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3219894"},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-023-09364-z"},{"key":"e_1_3_3_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/963770.963776"},{"key":"e_1_3_3_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/3340531.3411962"},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390392"},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/2090236.2090255"},{"key":"e_1_3_3_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/3613455"},{"key":"e_1_3_3_17_2","doi-asserted-by":"publisher","DOI":"10.1561\/1500000079"},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","DOI":"10.1002\/ASI.24648"},{"key":"e_1_3_3_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/3447548.3467349"},{"key":"e_1_3_3_20_2","volume-title":"Variabilit\u00e0 e Mutabilit\u00e0","author":"Gini Corrado","year":"1912","unstructured":"Corrado Gini. 1912. Variabilit\u00e0 e Mutabilit\u00e0. C. Cuppini, Bologna. Contributo allo Studio delle Distribuzioni e delle Relazioni Statistiche."},{"key":"e_1_3_3_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/2827872"},{"key":"e_1_3_3_22_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-023-09366-x"},{"key":"e_1_3_3_23_2","unstructured":"Hellinger distance. 2025. Encyclopedia of Mathematics. Retrieved 10 January 2025 from https:\/\/encyclopediaofmath.org\/wiki\/Hellinger_distance"},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/3583780.3615011"},{"key":"e_1_3_3_25_2","unstructured":"Rajendra K. Jain Dah-Ming W. Chiu William R. Hawe and others. 1998. A Quantitative Measure Of Fairness And Discrimination For Resource Allocation In Shared Computer Systems. DEC Research Report TR-301 September 1984. arXiv:9809099. Retrieved from http:\/\/arxiv.org\/abs\/cs\/9809099"},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/582415.582418"},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/3460231.3474248"},{"key":"e_1_3_3_28_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.patter.2022.100568"},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/258525.258587"},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/3610302"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/3178876.3186150"},{"key":"e_1_3_3_32_2","doi-asserted-by":"publisher","DOI":"10.1145\/3485447.3512104"},{"key":"e_1_3_3_33_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.98"},{"key":"e_1_3_3_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3340631.3394860"},{"key":"e_1_3_3_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/3383313.3418479"},{"key":"e_1_3_3_36_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-45068-6_1"},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/1416950.1416952"},{"key":"e_1_3_3_38_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-1018"},{"key":"e_1_3_3_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/3404835.3462830"},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/3459637.3482099"},{"key":"e_1_3_3_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/3366423.3380196"},{"key":"e_1_3_3_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/3531146.3533238"},{"key":"e_1_3_3_43_2","doi-asserted-by":"publisher","DOI":"10.1007\/S00778-021-00697-Y\/TABLES\/3"},{"key":"e_1_3_3_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/3477495.3532018"},{"key":"e_1_3_3_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/3631943"},{"key":"e_1_3_3_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/3626772.3657832"},{"key":"e_1_3_3_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/3696410.3714589"},{"key":"e_1_3_3_48_2","first-page":"452","volume-title":"Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence.","author":"Rendle Steffen","year":"2009","unstructured":"Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2009. BPR: Bayesian personalized ranking from implicit feedback. In Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence.AUAI Press, Arlington, Virginia, USA, 452\u2013461."},{"key":"e_1_3_3_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/3534678.3539353"},{"key":"e_1_3_3_50_2","doi-asserted-by":"publisher","DOI":"10.1145\/3674883"},{"key":"e_1_3_3_51_2","doi-asserted-by":"publisher","DOI":"10.1002\/j.1538-7305.1948.tb00917.x"},{"key":"e_1_3_3_52_2","volume-title":"Proceedings of the Advances in Neural Information Processing Systems.","volume":"32","author":"Singh Ashudeep","year":"2019","unstructured":"Ashudeep Singh and Thorsten Joachims. 2019. Policy learning for fairness in ranking. In Proceedings of the Advances in Neural Information Processing Systems.H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alch\u00e9-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32, Curran Associates, Inc., Red Hook, NY, USA. Retrieved from https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2019\/file\/9e82757e9a1c12cb710ad680db11f6f1-Paper.pdf"},{"key":"e_1_3_3_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/3543507.3583204"},{"key":"e_1_3_3_54_2","doi-asserted-by":"publisher","DOI":"10.1145\/1835804.1835835"},{"key":"e_1_3_3_55_2","volume-title":"Proceedings of the 37th International Conference on Neural Information Processing Systems","author":"Tsepenekas Leonidas","year":"2023","unstructured":"Leonidas Tsepenekas, Ivan Brugere, Freddy Lecue, and Daniele Maggazeni. 2023. Comparing apples to oranges: Learning similarity functions for data produced by different distributions. In Proceedings of the 37th International Conference on Neural Information Processing Systems (New Orleans, LA, USA). Curran Associates Inc., Red Hook, NY, USA, Article 1215, 16 pages."},{"key":"e_1_3_3_56_2","doi-asserted-by":"publisher","DOI":"10.1037\/h0033967"},{"key":"e_1_3_3_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/3531146.3533079"},{"key":"e_1_3_3_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/3547333"},{"key":"e_1_3_3_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/3589334.3645518"},{"key":"e_1_3_3_60_2","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390456"},{"key":"e_1_3_3_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/3477495.3532007"},{"key":"e_1_3_3_62_2","doi-asserted-by":"publisher","DOI":"10.1145\/3604558"},{"key":"e_1_3_3_63_2","doi-asserted-by":"publisher","DOI":"10.1145\/3543507.3583296"},{"key":"e_1_3_3_64_2","doi-asserted-by":"publisher","DOI":"10.1145\/3583780.3614877"},{"key":"e_1_3_3_65_2","doi-asserted-by":"publisher","DOI":"10.1145\/3539597.3570474"},{"key":"e_1_3_3_66_2","first-page":"11480","article-title":"Tenrec: A large-scale multipurpose benchmark dataset for recommender systems","volume":"35","author":"Yuan Guanghu","year":"2022","unstructured":"Guanghu Yuan, Fajie Yuan, Yudong Li, Beibei Kong, Shujie Li, Lei Chen, Min Yang, Chenyun Yu, Bo Hu, Zang Li, Yu Xu, and Xiaohu Qie. 2022. Tenrec: A large-scale multipurpose benchmark dataset for recommender systems. Advances in Neural Information Processing Systems 35(2022), 11480\u201311493. https:\/\/papers.nips.cc\/paper_files\/paper\/2022\/hash\/4ad4fc1528374422dd7a69dea9e72948-Abstract-Datasets_and_Benchmarks.html","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_3_67_2","doi-asserted-by":"publisher","DOI":"10.1145\/3533380"},{"key":"e_1_3_3_68_2","doi-asserted-by":"publisher","DOI":"10.1145\/3459637.3482016"},{"key":"e_1_3_3_69_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2020.3007617"}],"container-title":["ACM Transactions on Recommender Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3765624","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,3,15]],"date-time":"2026-03-15T09:28:04Z","timestamp":1773566884000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3765624"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,3,5]]},"references-count":68,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2026,9,30]]}},"alternative-id":["10.1145\/3765624"],"URL":"https:\/\/doi.org\/10.1145\/3765624","relation":{},"ISSN":["2770-6699"],"issn-type":[{"value":"2770-6699","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,3,5]]},"assertion":[{"value":"2025-02-03","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-07-03","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-03-05","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}