{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,21]],"date-time":"2026-03-21T08:12:56Z","timestamp":1774080776772,"version":"3.50.1"},"reference-count":47,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2024,7,29]],"date-time":"2024-07-29T00:00:00Z","timestamp":1722211200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-sa\/4.0\/"}],"funder":[{"name":"Departmental Strategic Plan (PSD) of the University of Udine, Interdepartmental Project on Artificial Intelligence"},{"name":"SERICS","award":["PE00000014"],"award-info":[{"award-number":["PE00000014"]}]},{"name":"EU - NGEU"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Intell. Syst. Technol."],"published-print":{"date-parts":[[2024,8,31]]},"abstract":"<jats:p>Popularity bias is the tendency of recommender systems to further suggest popular items while disregarding niche ones, hence giving no chance for items with low popularity to emerge. Although the literature is rich in debiasing techniques, it still lacks quality measures that effectively enable their analyses and comparisons.<\/jats:p>\n          <jats:p>\n            In this article, we first introduce a formal, data-driven, and parameter-free strategy for classifying items into low, medium, and high popularity categories. Then we introduce\n            <jats:italic>Balanced Quality Score (BQS)<\/jats:italic>\n            , a quality measure that rewards the debiasing techniques that successfully push a recommender system to suggest niche items, without losing points in its predictive capability in terms of global accuracy.\n          <\/jats:p>\n          <jats:p>We conduct tests of BQS on three distinct baseline collaborative filtering frameworks: one based on history-embedding and two on user\/item-embedding modeling. These evaluations are performed on multiple benchmark datasets and against various state-of-the-art competitors, demonstrating the effectiveness of BQS.<\/jats:p>","DOI":"10.1145\/3650043","type":"journal-article","created":{"date-parts":[[2024,3,1]],"date-time":"2024-03-01T12:09:30Z","timestamp":1709294970000},"page":"1-27","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":9,"title":["Balanced Quality Score: Measuring Popularity Debiasing in Recommendation"],"prefix":"10.1145","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4670-8157","authenticated-orcid":false,"given":"Erica","family":"Coppolillo","sequence":"first","affiliation":[{"name":"University of Calabria, Rende, Italy and ICAR-CNR, Rende, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9641-8916","authenticated-orcid":false,"given":"Marco","family":"Minici","sequence":"additional","affiliation":[{"name":"University of Pisa, Pisa, Italy and ICAR-CNR, Rende, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3978-9291","authenticated-orcid":false,"given":"Ettore","family":"Ritacco","sequence":"additional","affiliation":[{"name":"University of Udine, Udine, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0173-0131","authenticated-orcid":false,"given":"Luciano","family":"Caroprese","sequence":"additional","affiliation":[{"name":"University of Chieti-Pescara, Chieti, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2922-0835","authenticated-orcid":false,"given":"Francesco","family":"Pisani","sequence":"additional","affiliation":[{"name":"ICAR-CNR, Cosenza, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9672-3833","authenticated-orcid":false,"given":"Giuseppe","family":"Manco","sequence":"additional","affiliation":[{"name":"ICAR-CNR, Cosenza, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2024,7,29]]},"reference":[{"key":"e_1_3_3_2_2","doi-asserted-by":"publisher","unstructured":"Himan Abdollahpouri Robin Burke and Bamshad Mobasher. 2017. Controlling popularity bias in learning-to-rank recommendation. InProceedings of the ACM Recommender Systems Conference (RecSys \u201917). Association for Computing Machinery New York NY. DOI:10.1145\/3109859.3109912","DOI":"10.1145\/3109859.3109912"},{"key":"e_1_3_3_3_2","first-page":"413","volume-title":"Proceedings of the International Florida Artificial Intelligence Research Society Conference (FLAIRS \u201919)","author":"Abdollahpouri Himan","year":"2019","unstructured":"Himan Abdollahpouri, Robin Burke, and Bamshad Mobasher. 2019. Managing popularity bias in recommender systems with personalized re-ranking. In Proceedings of the International Florida Artificial Intelligence Research Society Conference (FLAIRS \u201919). 413\u2013418."},{"key":"e_1_3_3_4_2","volume-title":"Proceedings of the Workshop on Recommendation in Multi-stakeholder Environments (CEUR Workshop Proceedings \u201919)","volume":"2440","author":"Abdollahpouri Himan","year":"2019","unstructured":"Himan Abdollahpouri, Masoud Mansoury, Robin Burke, and Bamshad Mobasher. 2019. The unfairness of popularity bias in recommendation. In Proceedings of the Workshop on Recommendation in Multi-stakeholder Environments (CEUR Workshop Proceedings \u201919), Vol. 2440."},{"key":"e_1_3_3_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/3383313.3418487"},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2011.15"},{"key":"e_1_3_3_7_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-29659-3"},{"key":"e_1_3_3_8_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2020.102387"},{"key":"e_1_3_3_9_2","volume-title":"Proceedings of the EDBT Workshop on BigVis 2020: Big Data Visual Exploration and Analytics (EDBT\/ICDT Workshops)","author":"Borges Rodrigo","year":"2020","unstructured":"Rodrigo Borges and Kostas Stefanidis. 2020. On measuring popularity bias in collaborative filtering data. In Proceedings of the EDBT Workshop on BigVis 2020: Big Data Visual Exploration and Analytics (EDBT\/ICDT Workshops)."},{"key":"e_1_3_3_10_2","first-page":"657","volume-title":"Proceedings of the International Florida Artificial Intelligence Research Society Conference (FLAIRS \u201917)","author":"Channamsetty Sushma","year":"2017","unstructured":"Sushma Channamsetty and Michael D. Ekstrand. 2017. Recommender response to diversity and popularity bias in user profiles. In Proceedings of the International Florida Artificial Intelligence Research Society Conference (FLAIRS \u201917). 657\u2013660."},{"issue":"3","key":"e_1_3_3_11_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3564284","article-title":"Bias and debias in recommender system: A survey and future directions","volume":"41","author":"Chen Jiawei","year":"2023","unstructured":"Jiawei Chen, Hande Dong, Xiang Wang, Fuli Feng, Meng Wang, and Xiangnan He. 2023. Bias and debias in recommender system: A survey and future directions. ACM Trans. Inf. Syst. 41, 3 (2023), 1\u201339.","journal-title":"ACM Trans. Inf. Syst."},{"key":"e_1_3_3_12_2","doi-asserted-by":"publisher","unstructured":"Zhihong Chen Jiawei Wu Chenliang Li Jingxu Chen Rong Xiao and Binqiang Zhao. 2022. Co-training disentangled domain adaptation network for leveraging popularity bias in recommenders. Association for Computing Machinery New York NY. DOI:10.1145\/3477495.3531952","DOI":"10.1145\/3477495.3531952"},{"key":"e_1_3_3_13_2","doi-asserted-by":"publisher","DOI":"10.1137\/070710111"},{"key":"e_1_3_3_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3477495.3532002"},{"key":"e_1_3_3_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3209991"},{"key":"e_1_3_3_16_2","first-page":"172","volume-title":"Proceedings of the Conference on Fairness, Accountability, and Transparency (PMLR \u201918)","author":"Ekstrand Michael D.","year":"2018","unstructured":"Michael D. Ekstrand, Mucun Tian, Ion Madrazo Azpiazu, Jennifer D. Ekstrand, Oghenemaro Anuyah, David McNeill, and Maria Soledad Pera. 2018. All the cool kids, how do they fit in?: Popularity and demographic biases in recommender evaluation and effectiveness. In Proceedings of the Conference on Fairness, Accountability, and Transparency (PMLR \u201918). 172\u2013186."},{"key":"e_1_3_3_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/230538.230561"},{"key":"e_1_3_3_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3038912.3052569"},{"key":"e_1_3_3_19_2","volume-title":"Understanding Research Methods and Statistics: An Integrated Introduction for Psychology","author":"Heiman Gary W.","year":"2001","unstructured":"Gary W. Heiman. 2001. Understanding Research Methods and Statistics: An Integrated Introduction for Psychology. Houghton Mifflin Company."},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/3604915.3608839"},{"key":"e_1_3_3_21_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-015-9165-3"},{"key":"e_1_3_3_22_2","volume-title":"Proceedings of the ACM Conference on Recommender Systems (RecSys \u201914)","author":"Kamishima Toshihiro","year":"2014","unstructured":"Toshihiro Kamishima, Shotaro Akaho, and Hideki Asoh. 2014. Correcting popularity bias by enhancing recommendation neutrality. In Proceedings of the ACM Conference on Recommender Systems (RecSys \u201914)."},{"key":"e_1_3_3_23_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-45442-5_5"},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.1145\/3097983.3098077"},{"key":"e_1_3_3_25_2","doi-asserted-by":"publisher","DOI":"10.1145\/2872427.2883090"},{"key":"e_1_3_3_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/3178876.3186150"},{"key":"e_1_3_3_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/3292500.3330707"},{"key":"e_1_3_3_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/3340531.3412152"},{"key":"e_1_3_3_29_2","doi-asserted-by":"publisher","DOI":"10.1080\/00107510500052444"},{"key":"e_1_3_3_30_2","doi-asserted-by":"publisher","DOI":"10.1137\/s003614450342480"},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/2168752.2168771"},{"key":"e_1_3_3_32_2","first-page":"452","volume-title":"Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI \u201909)","author":"Rendle S.","year":"2009","unstructured":"S. Rendle, C. Freudenthaler, Z. Gantner, and L. Schmidt-Thieme. 2009. BPR: Bayesian personalized ranking from implicit feedback. In Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI \u201909). 452\u2013461."},{"key":"e_1_3_3_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/3289600.3291007"},{"key":"e_1_3_3_34_2","first-page":"1670","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Schnabel Tobias","year":"2016","unstructured":"Tobias Schnabel, Adith Swaminathan, Ashudeep Singh, Navin Chandak, and Thorsten Joachims. 2016. Recommendations as treatments: Debiasing learning and evaluation. In Proceedings of the International Conference on Machine Learning. PMLR, 1670\u20131679."},{"key":"e_1_3_3_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/2740908.2742726"},{"key":"e_1_3_3_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/3336191.3371831"},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3240323.3240372"},{"key":"e_1_3_3_38_2","volume-title":"Proceedings of the Workshop on Recommendation in Multi-stakeholder Environments (CEUR Workshop Proceedings \u201919)","volume":"2440","author":"Tsintzou Virginia","year":"2019","unstructured":"Virginia Tsintzou, Evaggelia Pitoura, and Panayiotis Tsaparas. 2019. Bias disparity in recommendation systems. In Proceedings of the Workshop on Recommendation in Multi-stakeholder Environments (CEUR Workshop Proceedings \u201919), Vol. 2440."},{"key":"e_1_3_3_39_2","article-title":"Learning to rank by optimizing ndcg measure","volume":"22","author":"Valizadegan Hamed","year":"2009","unstructured":"Hamed Valizadegan, Rong Jin, Ruofei Zhang, and Jianchang Mao. 2009. Learning to rank by optimizing ndcg measure. In Advances in Neural Information Processing Systems, Vol. 22.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_3_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/3447548.3467249"},{"key":"e_1_3_3_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/3331184.3331267"},{"key":"e_1_3_3_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/3447548.3467289"},{"key":"e_1_3_3_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/3477495.3531907"},{"key":"e_1_3_3_44_2","doi-asserted-by":"publisher","DOI":"10.14778\/2311906.2311916"},{"key":"e_1_3_3_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/3477495.3531937"},{"key":"e_1_3_3_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/3442381.3449788"},{"key":"e_1_3_3_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/3269206.3271795"},{"key":"e_1_3_3_48_2","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401177"}],"container-title":["ACM Transactions on Intelligent Systems and Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3650043","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3650043","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:03:43Z","timestamp":1750291423000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3650043"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,29]]},"references-count":47,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2024,8,31]]}},"alternative-id":["10.1145\/3650043"],"URL":"https:\/\/doi.org\/10.1145\/3650043","relation":{},"ISSN":["2157-6904","2157-6912"],"issn-type":[{"value":"2157-6904","type":"print"},{"value":"2157-6912","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,7,29]]},"assertion":[{"value":"2023-06-30","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-02-14","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-07-29","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}