{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,2]],"date-time":"2025-08-02T14:33:26Z","timestamp":1754145206119,"version":"3.41.2"},"reference-count":53,"publisher":"Association for Computing Machinery (ACM)","issue":"ISSTA","funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62472310,62322208,62202025"],"award-info":[{"award-number":["62472310,62322208,62202025"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Beijing Natural Science Foundation","award":["L241050"],"award-info":[{"award-number":["L241050"]}]},{"name":"Young Elite Scientist Sponsorship Program by CAST","award":["YESS20230566"],"award-info":[{"award-number":["YESS20230566"]}]},{"name":"CCF-Huawei Populus Grove Fund","award":["CCF-HuaweiFM2024005"],"award-info":[{"award-number":["CCF-HuaweiFM2024005"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Softw. Eng."],"published-print":{"date-parts":[[2025,6,22]]},"abstract":"<jats:p>Recommender systems play an increasingly important role in modern society, powering digital platforms that suggest a wide array of content, from news and music to job listings, and influencing many aspects of daily life. To improve personalization, these systems often use demographic information. However, ensuring fairness in recommendation quality across demographic groups is challenging, especially since recommender systems are susceptible to the \"rich get richer'' Matthew effect due to user feedback loops. With the adoption of deep learning algorithms, uncovering fairness issues has become even more complex. Researchers have started to explore methods for identifying the most disadvantaged user groups using optimization algorithms. Despite this, suboptimal disadvantaged groups remain underexplored, which leaves the risk of bias amplification due to the Matthew effect unaddressed. In this paper, we argue for the necessity of identifying both the most disadvantaged and suboptimal disadvantaged groups. We introduce FairAS, an adaptive sampling based approach, to achieve this goal. Through evaluations on four deep recommender systems and six datasets, FairAS demonstrates an average improvement of 19.2% in identifying the most disadvantaged groups over the state-of-the-art fairness testing approach (FairRec), while reducing testing time by 43.07%. Additionally, the extra suboptimal disadvantaged groups identified by FairAS help improve system fairness, achieving an average improvement of 70.27% over FairRec across all subjects.<\/jats:p>","DOI":"10.1145\/3728948","type":"journal-article","created":{"date-parts":[[2025,6,22]],"date-time":"2025-06-22T10:52:56Z","timestamp":1750589576000},"page":"1607-1629","source":"Crossref","is-referenced-by-count":0,"title":["No Bias Left Behind: Fairness Testing for Deep Recommender Systems Targeting General Disadvantaged Groups"],"prefix":"10.1145","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0009-0006-0165-8746","authenticated-orcid":false,"given":"Zhuo","family":"Wu","sequence":"first","affiliation":[{"name":"Tianjin University, Tianjin, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6173-8170","authenticated-orcid":false,"given":"Zan","family":"Wang","sequence":"additional","affiliation":[{"name":"Tianjin University, Tianjin, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5028-1064","authenticated-orcid":false,"given":"Chuan","family":"Luo","sequence":"additional","affiliation":[{"name":"Beihang University, Beijing, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3728-9541","authenticated-orcid":false,"given":"Xiaoning","family":"Du","sequence":"additional","affiliation":[{"name":"Monash University, Melbourne, Australia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-3056-9962","authenticated-orcid":false,"given":"Junjie","family":"Chen","sequence":"additional","affiliation":[{"name":"Tianjin University, Tianjin, China"}]}],"member":"320","published-online":{"date-parts":[[2025,6,22]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"Accessed: 2023. BlackFriday. https:\/\/www.kaggle.com\/datasets\/sdolezel\/black-friday"},{"key":"e_1_2_1_2_1","unstructured":"Accessed: 2023. Homepage. https:\/\/github.com\/anonyProjects\/FairAS"},{"key":"e_1_2_1_3_1","unstructured":"Accessed: 2023. Pandas. https:\/\/pandas.pydata.org\/"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3450613.3456821"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3264590"},{"key":"e_1_2_1_6_1","volume-title":"Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1114\u20131126","author":"Baranov Eduard","year":"2020","unstructured":"Eduard Baranov, Axel Legay, and Kuldeep S Meel. 2020. Baital: an adaptive weighted sampling approach for improved t-wise coverage. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1114\u20131126."},{"key":"e_1_2_1_7_1","doi-asserted-by":"crossref","unstructured":"\u00d2scar Celma Herrada. 2009. Music recommendation and discovery in the long tail. Universitat Pompeu Fabra.","DOI":"10.1007\/978-3-642-13287-2"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3459637.3481915"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3394112"},{"key":"e_1_2_1_10_1","volume-title":"REASONER: An Explainable Recommendation Dataset with Multi-aspect Real User Labeled Ground Truths Towards more Measurable Explainable Recommendation. CoRR, abs\/2303.00168","author":"Chen Xu","year":"2023","unstructured":"Xu Chen, Jingsen Zhang, Lei Wang, Quanyu Dai, Zhenhua Dong, Ruiming Tang, Rui Zhang, Li Chen, and Ji-Rong Wen. 2023. REASONER: An Explainable Recommendation Dataset with Multi-aspect Real User Labeled Ground Truths Towards more Measurable Explainable Recommendation. CoRR, abs\/2303.00168 (2023)."},{"key":"e_1_2_1_11_1","volume-title":"Fairness Testing: A Comprehensive Survey and Analysis of Trends. ACM Transactions on Software Engineering and Methodology.","author":"Chen Zhenpeng","year":"2023","unstructured":"Zhenpeng Chen, Jie M Zhang, Max Hort, Mark Harman, and Federica Sarro. 2023. Fairness Testing: A Comprehensive Survey and Analysis of Trends. ACM Transactions on Software Engineering and Methodology."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/2988450.2988454"},{"key":"e_1_2_1_13_1","volume-title":"Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 1541\u20131553","author":"Dasu Vishnu Asutosh","year":"2024","unstructured":"Vishnu Asutosh Dasu, Ashish Kumar, Saeid Tizpaz-Niari, and Gang Tan. 2024. NeuFair: Neural Network Fairness Repair with Dropout. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 1541\u20131553."},{"key":"e_1_2_1_14_1","volume-title":"Conference on fairness, accountability and transparency. 172\u2013186","author":"Ekstrand Michael D","year":"2018","unstructured":"Michael D Ekstrand, Mucun Tian, Ion Madrazo Azpiazu, Jennifer D Ekstrand, Oghenemaro Anuyah, David McNeill, and Maria Soledad Pera. 2018. All the cool kids, how do they fit in?: Popularity and demographic biases in recommender evaluation and effectiveness. In Conference on fairness, accountability and transparency. 172\u2013186."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3106237.3106277"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3511808.3557220"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3404835.3463235"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00042"},{"key":"e_1_2_1_19_1","doi-asserted-by":"crossref","unstructured":"Huizhong Guo Jinfeng Li Jingyi Wang Xiangyu Liu Dongxia Wang Zehong Hu Rong Zhang and Hui Xue. 2023. FairRec: Fairness Testing for Deep Recommender Systems. In ISSTA. ACM 310\u2013321.","DOI":"10.1145\/3597926.3598058"},{"key":"e_1_2_1_20_1","unstructured":"Huifeng Guo Ruiming Tang Yunming Ye Zhenguo Li and Xiuqiang He. 2017. DeepFM: a factorization-machine based neural network for CTR prediction. arXiv preprint arXiv:1703.04247."},{"key":"e_1_2_1_21_1","volume-title":"The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis), 5, 4","author":"Maxwell Harper F","year":"2015","unstructured":"F Maxwell Harper and Joseph A Konstan. 2015. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis), 5, 4 (2015), 1\u201319."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3038912.3052569"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/582415.582418"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSMCB.2010.2103055"},{"key":"e_1_2_1_25_1","doi-asserted-by":"crossref","unstructured":"Yunqi Li Hanxiong Chen Zuohui Fu Yingqiang Ge and Yongfeng Zhang. 2021. User-oriented Fairness in Recommendation. In WWW. ACM \/ IW3C2 624\u2013632.","DOI":"10.1145\/3442381.3449866"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3404835.3462966"},{"key":"e_1_2_1_27_1","first-page":"519","article-title":"AUC: a statistically consistent and more discriminating measure than accuracy","volume":"3","author":"Ling Charles X","year":"2003","unstructured":"Charles X Ling, Jin Huang, and Harry Zhang. 2003. AUC: a statistically consistent and more discriminating measure than accuracy. In Ijcai. 3, 519\u2013524.","journal-title":"Ijcai."},{"key":"e_1_2_1_28_1","doi-asserted-by":"crossref","unstructured":"Bin Liu Ruiming Tang Yingzhi Chen Jinkai Yu Huifeng Guo and Yuzhou Zhang. 2019. Feature Generation by Convolutional Neural Network for Click-Through Rate Prediction. In WWW. ACM 1119\u20131129.","DOI":"10.1145\/3308558.3313497"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3468622"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2931037.2931054"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3041021.3054197"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00136"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.5555\/2540128.2540517"},{"key":"e_1_2_1_34_1","volume-title":"ICML (Proceedings of Machine Learning Research","volume":"9659","author":"Si Nian","year":"2021","unstructured":"Nian Si, Karthyek Murthy, Jose H. Blanchet, and Viet Anh Nguyen. 2021. Testing Group Fairness via Optimal Transport Projections. In ICML (Proceedings of Machine Learning Research, Vol. 139). PMLR, 9649\u20139659."},{"key":"e_1_2_1_35_1","volume-title":"Practical bayesian optimization of machine learning algorithms. Advances in neural information processing systems, 25","author":"Snoek Jasper","year":"2012","unstructured":"Jasper Snoek, Hugo Larochelle, and Ryan P Adams. 2012. Practical bayesian optimization of machine learning algorithms. Advances in neural information processing systems, 25 (2012)."},{"volume-title":"Sampling","author":"Thompson Steven K","key":"e_1_2_1_36_1","unstructured":"Steven K Thompson. 2012. Sampling, Third Edition. John Wiley & Sons, Inc.."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510202"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238165"},{"key":"e_1_2_1_39_1","first-page":"101","article-title":"A critique and improvement of the CL common language effect size statistics of McGraw and Wong","volume":"25","author":"Vargha Andr\u00e1s","year":"2000","unstructured":"Andr\u00e1s Vargha and Harold D Delaney. 2000. A critique and improvement of the CL common language effect size statistics of McGraw and Wong. Journal of Educational and Behavioral Statistics, 25, 2 (2000), 101\u2013132.","journal-title":"Journal of Educational and Behavioral Statistics"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1017\/S1351324901002789"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3336191.3371855"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3447548.3467249"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409761"},{"volume-title":"Breakthroughs in Statistics: Methodology and Distribution","author":"Wilcoxon Frank","key":"e_1_2_1_44_1","unstructured":"Frank Wilcoxon. 1992. Individual comparisons by ranking methods. In Breakthroughs in Statistics: Methodology and Distribution. Springer, 196\u2013202."},{"volume-title":"Fairness-aware News Recommendation with Decomposed Adversarial Learning","author":"Wu Chuhan","key":"e_1_2_1_45_1","unstructured":"Chuhan Wu, Fangzhao Wu, Xiting Wang, Yongfeng Huang, and Xing Xie. 2021. Fairness-aware News Recommendation with Decomposed Adversarial Learning. In AAAI. AAAI Press, 4462\u20134469."},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3442381.3450015"},{"key":"e_1_2_1_47_1","volume-title":"Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 210\u2013222","author":"Yang Junjie","year":"2024","unstructured":"Junjie Yang, Jiajun Jiang, Zeyu Sun, and Junjie Chen. 2024. A large-scale empirical study on improving the fairness of image classification models. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 210\u2013222."},{"volume-title":"Regression Fuzzing for Deep Learning Systems","author":"You Hanmo","key":"e_1_2_1_48_1","unstructured":"Hanmo You, Zan Wang, Junjie Chen, Shuang Liu, and Shuochuan Li. 2023. Regression Fuzzing for Deep Learning Systems. In ICSE. IEEE, 82\u201394."},{"key":"e_1_2_1_49_1","doi-asserted-by":"crossref","unstructured":"Mengdi Zhang Jun Sun Jingyi Wang and Bing Sun. 2022. TestSGD: Interpretable Testing of Neural Networks Against Subtle Group Discrimination. ACM Transactions on Software Engineering and Methodology.","DOI":"10.1145\/3591869"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380331"},{"key":"e_1_2_1_51_1","volume-title":"Deep learning based recommender system: A survey and new perspectives. ACM computing surveys (CSUR), 52, 1","author":"Zhang Shuai","year":"2019","unstructured":"Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay. 2019. Deep learning based recommender system: A survey and new perspectives. ACM computing surveys (CSUR), 52, 1 (2019), 1\u201338."},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510123"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/1060745.1060754"}],"container-title":["Proceedings of the ACM on Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3728948","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,16]],"date-time":"2025-07-16T16:49:53Z","timestamp":1752684593000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3728948"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,22]]},"references-count":53,"journal-issue":{"issue":"ISSTA","published-print":{"date-parts":[[2025,6,22]]}},"alternative-id":["10.1145\/3728948"],"URL":"https:\/\/doi.org\/10.1145\/3728948","relation":{},"ISSN":["2994-970X"],"issn-type":[{"type":"electronic","value":"2994-970X"}],"subject":[],"published":{"date-parts":[[2025,6,22]]}}}