{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,11]],"date-time":"2026-04-11T04:29:48Z","timestamp":1775881788776,"version":"3.50.1"},"reference-count":67,"publisher":"Association for Computing Machinery (ACM)","issue":"6","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Intell. Syst. Technol."],"published-print":{"date-parts":[[2025,12,31]]},"abstract":"<jats:p>\n                    This work takes a critical stance on previous studies concerning fairness evaluation in Large-Language Model (LLM)-based recommender systems, which have primarily assessed consumer fairness by comparing recommendation lists generated with and without sensitive user attributes. Such approaches implicitly treat discrepancies in recommended items as biases, overlooking whether these changes might stem from genuine personalization aligned with true preferences of users. Moreover, these earlier studies typically address single sensitive attributes in isolation, neglecting the complex interplay of intersectional identities. In response to these shortcomings, we introduce\n                    <jats:italic toggle=\"yes\">CFaiRLLM<\/jats:italic>\n                    , an enhanced evaluation framework that not only incorporates\n                    <jats:italic toggle=\"yes\">true preference alignment<\/jats:italic>\n                    but also rigorously examines\n                    <jats:italic toggle=\"yes\">intersectional fairness<\/jats:italic>\n                    by considering overlapping sensitive attributes. Additionally, CFaiRLLM introduces diverse user profile sampling strategies\u2014\n                    <jats:italic toggle=\"yes\">random<\/jats:italic>\n                    ,\n                    <jats:italic toggle=\"yes\">top-rated<\/jats:italic>\n                    , and\n                    <jats:italic toggle=\"yes\">recency-focused<\/jats:italic>\n                    \u2014to better understand the impact of profile generation fed to LLMs in light of inherent token limitations in these systems. Given that fairness depends on accurately understanding users\u2019 tastes and preferences, these strategies provide a more realistic assessment of fairness within RecLLMs.\n                  <\/jats:p>\n                  <jats:p>\n                    To validate the efficacy of CFaiRLLM, we conducted extensive experiments using\n                    <jats:monospace>MovieLens<\/jats:monospace>\n                    and\n                    <jats:monospace>LastFM<\/jats:monospace>\n                    datasets, applying various sampling strategies and sensitive attribute configurations. The evaluation metrics include both item similarity measures and true preference alignment considering both hit and ranking (Jaccard Similarity and PRAG), thereby conducting a multi-faceted analysis of recommendation fairness. The results demonstrated that true preference alignment offers a more personalized and fair assessment compared to similarity-based measures, revealing significant disparities when sensitive and intersectional attributes are incorporated. Notably, our study finds that intersectional attributes amplify fairness gaps more prominently, especially in less structured domains such as music recommendations in LastFM. These findings suggest that future fairness evaluations in RecLLMs should incorporate true preference alignment to ensure equitable and genuinely personalized recommendations.\n                  <\/jats:p>","DOI":"10.1145\/3725853","type":"journal-article","created":{"date-parts":[[2025,3,25]],"date-time":"2025-03-25T14:27:37Z","timestamp":1742912857000},"page":"1-31","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":21,"title":["CFaiRLLM: Consumer Fairness Evaluation in Large-Language Model Recommender System"],"prefix":"10.1145","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6767-358X","authenticated-orcid":false,"given":"Yashar","family":"Deldjoo","sequence":"first","affiliation":[{"name":"SisInf Lab, Department of Electrical Engineering and Information Technology, Politecnico di Bari, Bari, Italy"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0939-5462","authenticated-orcid":false,"given":"Tommaso Di","family":"Noia","sequence":"additional","affiliation":[{"name":"Politecnico di Bari, Bari, Italy"}]}],"member":"320","published-online":{"date-parts":[[2025,11,24]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2022.103115"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.emnlp-main.446"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/3657631"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/3109859.3109893"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-021-09294-8"},{"key":"e_1_3_2_7_2","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"Brown Tom","year":"2020","unstructured":"Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, Vol. 33, 1877\u20131901.","journal-title":"Advances in Neural Information Processing Systems, Vol"},{"key":"e_1_3_2_8_2","first-page":"202","volume-title":"Conference on Fairness, Accountability and Transparency","author":"Burke Robin","year":"2018","unstructured":"Robin Burke, Nasim Sonboli, and Aldo Ordonez-Gauger. 2018. Balanced neighborhoods for multi-sided fairness in recommendation. In Conference on Fairness, Accountability and Transparency. PMLR, 202\u2013214."},{"key":"e_1_3_2_9_2","unstructured":"Abhijnan Chakraborty Aniko Hannak Asia J. Biega and Krishna P. Gummadi. 2017. Fair sharing for sharing economy platforms. In Proceedings of Fairness Accountability and Transparency in Recommender Systems-Workshop on Responsible Recommendation."},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3641289"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2021.115112"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4899-7637-6_4"},{"key":"e_1_3_2_13_2","article-title":"Fairness of ChatGPT and the Role of explainable-guided prompts","author":"Deldjoo Yashar","year":"2023","unstructured":"Yashar Deldjoo. 2023. Fairness of ChatGPT and the Role of explainable-guided prompts. In COLLM@ECML-PKDD\u201923.","journal-title":"COLLM@ECML-PKDD\u201923"},{"key":"e_1_3_2_14_2","volume-title":"ACM Transactions on Recommender Systems","author":"Deldjoo Yashar","year":"2024","unstructured":"Yashar Deldjoo. 2024. Understanding biases in ChatGPT-based recommender systems: Provider fairness, temporal stability, and recency. ACM Transactions on Recommender Systems (2024)."},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-020-09285-1"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401046"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/3439729"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3637528.3671474"},{"key":"e_1_3_2_19_2","doi-asserted-by":"crossref","unstructured":"Yashar Deldjoo Zhankui He Julian McAuley Anton Korikov Scott Sanner Arnau Ramisa Rene Vidal Maheswaran Sathiamoorthy Atoosa Kasrizadeh Silvia Milano et al. 2024. Recommendation with generative models. arXiv:2409.15173. Retrieved from https:\/\/arxiv.org\/abs\/2409.15173","DOI":"10.1145\/3701551.3703485"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-023-09364-z"},{"key":"e_1_3_2_21_2","unstructured":"Yashar Deldjoo and Fatemeh Nazary. 2024. A Normative framework for benchmarking consumer fairness in large language model recommender system. In ROEGen@RecSys\u201924."},{"key":"e_1_3_2_22_2","first-page":"1","volume-title":"Italian Information Retrieval Workshop","author":"Deldjoo Yashar","year":"2018","unstructured":"Yashar Deldjoo, Markus Schedl, Paolo Cremonesi, and Gabirella Pasi. 2018. Content-based multimedia recommendation systems: Definition and application domains. In Italian Information Retrieval Workshop, 1\u20134."},{"key":"e_1_3_2_23_2","unstructured":"Dario Di Palma Giovanni Maria Biancofiore Vito Walter Anelli Fedelucio Narducci Tommaso Di Noia and Eugenio Di Sciascio. 2023. Evaluating ChatGPT as a recommender system: A rigorous approach. arXiv:2309.03613. Retrieved from https:\/\/arxiv.org\/abs\/2309.03613"},{"key":"e_1_3_2_24_2","article-title":"Two-sided fairness in rankings via Lorenz dominance","volume":"34","author":"Do Virginie","year":"2021","unstructured":"Virginie Do, Sam Corbett-Davies, Jamal Atif, and Nicolas Usunier. 2021. Two-sided fairness in rankings via Lorenz dominance. In Advances in Neural Information Processing Systems, Vol. 34.","journal-title":"Advances in Neural Information Processing Systems, Vol"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2021.3113975"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.frl.2023.103662"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/3298689.3346964"},{"key":"e_1_3_2_28_2","unstructured":"Golnoosh Farnadi Pigi Kouki Spencer K. Thompson Sriram Srinivasan and Lise Getoor. 2018. A fairness-aware hybrid recommender system. arXiv:1809.09030. Retrieved from https:\/\/arxiv.org\/abs\/1809.09030"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/3437963.3441824"},{"key":"e_1_3_2_30_2","first-page":"299","volume-title":"Proceedings of the 16th ACM Conference on Recommender Systems","author":"Geng Shijie","unstructured":"Shijie Geng, Shuchang Liu, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. 2022. Recommendation as language processing (RLP): A unified pretrain, personalized prompt & predict paradigm (p5). In Proceedings of the 16th ACM Conference on Recommender Systems, 299\u2013315."},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/3404835.3463235"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1145\/3474085.3475706"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/3583780.3614949"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3534678.3539381"},{"key":"e_1_3_2_35_2","unstructured":"Mingyu Jin Qinkai Yu Chong Zhang Dong Shu Suiyuan Zhu Mengnan Du Yongfeng Zhang and Yanda Meng. 2024. Health-LLM: Personalized retrieval-augmented disease prediction model. arXiv:2402.00746. Retrieved from https:\/\/arxiv.org\/abs\/2402.00746"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.clsr.2024.106053"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3442381.3450080"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2014.07.024"},{"key":"e_1_3_2_39_2","unstructured":"Lei Li Yongfeng Zhang Dugang Liu and Li Chen. 2023. Large language models for generative recommendation: A survey and visionary discussions. arXiv:2309.01157. Retrieved from https:\/\/arxiv.org\/abs\/2309.01157"},{"key":"e_1_3_2_40_2","unstructured":"Xinyi Li Yongfeng Zhang and Edward C. Malthouse. 2023. A preliminary study of ChatGPT on news recommendation: Personalization provider fairness fake news. arXiv:2306.10702. Retrieved from https:\/\/arxiv.org\/abs\/2306.10702"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/3442381.3449866"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/3539618.3594250"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/3404835.3462943"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1108\/K-05-2018-0216"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-47426-3_13"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/3477495.3531959"},{"key":"e_1_3_2_47_2","first-page":"382","volume-title":"European Conference on Artificial Intelligence","author":"Nazary Fatemeh","year":"2023","unstructured":"Fatemeh Nazary, Yashar Deldjoo, and Tommaso Di Noia. 2023. ChatGPT-HealthPrompt. Harnessing the power of XAI in prompt-based healthcare decision support using ChatGPT. In European Conference on Artificial Intelligence. Springer, 382\u2013397."},{"key":"e_1_3_2_48_2","doi-asserted-by":"crossref","unstructured":"Fatemeh Nazary Yashar Deldjoo and Tommaso di Noia. 2025. Poison-RAG: Adversarial data poisoning attacks on retrieval-augmented generation in recommender systems. arXiv:2501.11759. Retrieved from https:\/\/arxiv.org\/abs\/2501.11759","DOI":"10.1007\/978-3-031-88717-8_18"},{"key":"e_1_3_2_49_2","unstructured":"Fatemeh Nazary Yashar Deldjoo Tommaso Di Noia and Eugenio di Sciascio. 2024. XAI4LLM. Let machine learning models and LLMs collaborate for enhanced in-context learning in healthcare. arXiv:2405.06270. Retrieved from https:\/\/arxiv.org\/abs\/2405.06270"},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1145\/3366423.3380196"},{"key":"e_1_3_2_51_2","article-title":"The unfairness of active users and popularity bias in point-of-interest recommendation. In","author":"Rahmani Hossein A.","year":"2022","unstructured":"Hossein A. Rahmani, Yashar Deldjoo, Ali Tourani, and Mohammadmehdi Naghiaei. 2022. The unfairness of active users and popularity bias in point-of-interest recommendation. In Bias@ECIR\u201922.","journal-title":"Bias@ECIR\u201922"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1145\/3604915.3608845"},{"key":"e_1_3_2_53_2","unstructured":"Dougal Shakespeare Lorenzo Porcaro Emilia G\u00f3mez and Carlos Castillo. 2020. Exploring artist gender bias in music recommendation. arXiv:2009.01715. Retrieved from https:\/\/arxiv.org\/abs\/2009.01715"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2022.103139"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/3461702.3462602"},{"key":"e_1_3_2_56_2","doi-asserted-by":"crossref","unstructured":"Tu Vu Brian Lester Noah Constant Rami Al-Rfou and Daniel Cer. 2021. Spot: Better frozen model adaptation through soft prompt transfer. arXiv:2110.07904. Retrieved from https:\/\/arxiv.org\/abs\/2110.07904","DOI":"10.18653\/v1\/2022.acl-long.346"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/3336191.3371855"},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/3356994.3365497"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v35i5.16573"},{"key":"e_1_3_2_60_2","unstructured":"Shijie Wu Ozan Irsoy Steven Lu Vadim Dabravolski Mark Dredze Sebastian Gehrmann Prabhanjan Kambadur David Rosenberg and Gideon Mann. 2023. Bloomberggpt: A large language model for finance. arXiv:2303.17564. Retrieved from https:\/\/arxiv.org\/abs\/2303.17564"},{"key":"e_1_3_2_61_2","doi-asserted-by":"crossref","unstructured":"Yao Wu Jian Cao Guandong Xu and Yudong Tan. 2021. TFROM: A two-sided fairness-aware recommendation model for both customers and providers. arXiv:2104.09024. Retrieved from https:\/\/arxiv.org\/abs\/2104.09024","DOI":"10.1145\/3404835.3462882"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jnca.2020.102579"},{"key":"e_1_3_2_63_2","unstructured":"Lanling Xu Junjie Zhang Bingqian Li Jinpeng Wang Mingchen Cai Wayne Xin Zhao and Ji-Rong Wen. 2024. Prompting large language models for recommender systems: A comprehensive framework and empirical analysis. arXiv:2401.04997. Retrieved from https:\/\/arxiv.org\/abs\/2401.04997"},{"key":"e_1_3_2_64_2","unstructured":"Shuyuan Xu Wenyue Hua and Yongfeng Zhang. 2023. OpenP5: Benchmarking foundation models for recommendation. arXiv:2306.11134. Retrieved from https:\/\/arxiv.org\/abs\/2306.11134"},{"key":"e_1_3_2_65_2","first-page":"4067","article-title":"Fairness with overlapping groups; a probabilistic perspective","volume":"33","author":"Yang Forest","year":"2020","unstructured":"Forest Yang, Mouhamadou Cisse, and Sanmi Koyejo. 2020. Fairness with overlapping groups; a probabilistic perspective. In Advances in Neural Information Processing Systems, Vol. 33, 4067\u20134078.","journal-title":"Advances in Neural Information Processing Systems, Vol"},{"key":"e_1_3_2_66_2","first-page":"993","volume-title":"Proceedings of the 17th ACM Conference on Recommender Systems","author":"Zhang Jizhi","unstructured":"Jizhi Zhang, Keqin Bao, Yang Zhang, Wenjie Wang, Fuli Feng, and Xiangnan He. 2023. Is ChatGPT fair for recommendation? evaluating fairness in large language model recommendation. In Proceedings of the 17th ACM Conference on Recommender Systems, 993\u2013999."},{"key":"e_1_3_2_67_2","unstructured":"Lemei Zhang Peng Liu Yashar Deldjoo Yong Zheng and Jon Atle Gulla. 2024. Understanding language modeling paradigm adaptations in recommender systems: Lessons learned and open challenges. In Proceedings of the 27th European Conference on Artificial Intelligence (ECAI \u201924)."},{"key":"e_1_3_2_68_2","doi-asserted-by":"publisher","DOI":"10.1145\/3404835.3462948"}],"container-title":["ACM Transactions on Intelligent Systems and Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3725853","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,11,24]],"date-time":"2025-11-24T15:12:32Z","timestamp":1763997152000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3725853"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,11,24]]},"references-count":67,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2025,12,31]]}},"alternative-id":["10.1145\/3725853"],"URL":"https:\/\/doi.org\/10.1145\/3725853","relation":{},"ISSN":["2157-6904","2157-6912"],"issn-type":[{"value":"2157-6904","type":"print"},{"value":"2157-6912","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,11,24]]},"assertion":[{"value":"2024-03-08","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-01-15","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-11-24","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}