{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,21]],"date-time":"2026-02-21T19:35:25Z","timestamp":1771702525099,"version":"3.50.1"},"reference-count":50,"publisher":"Association for Computing Machinery (ACM)","issue":"6","license":[{"start":{"date-parts":[[2023,9,30]],"date-time":"2023-09-30T00:00:00Z","timestamp":1696032000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001459","name":"Ministry of Education, Singapore","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100001459","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Academic Research Fund Tier 3","award":["MOET32020-0004"],"award-info":[{"award-number":["MOET32020-0004"]}]},{"name":"Key R&D Program of Zhejiang","award":["2022C01018"],"award-info":[{"award-number":["2022C01018"]}]},{"name":"NSFC Program","award":["62102359"],"award-info":[{"award-number":["62102359"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2023,11,30]]},"abstract":"<jats:p>\n            Discrimination has been shown in many machine learning applications, which calls for sufficient fairness testing before their deployment in ethic-relevant domains. One widely concerning type of discrimination, testing against\n            <jats:italic>group discrimination, mostly hidden<\/jats:italic>\n            , is much less studied, compared with identifying\n            <jats:italic>individual discrimination<\/jats:italic>\n            . In this work, we propose\n            <jats:sc>TestSGD<\/jats:sc>\n            , an interpretable testing approach that systematically identifies and measures hidden (which we call\n            <jats:italic>\u201csubtle\u201d) group discrimination<\/jats:italic>\n            of a neural network characterized by\n            <jats:italic>conditions over combinations of the sensitive attributes<\/jats:italic>\n            . Specifically, given a neural network,\n            <jats:sc>TestSGD<\/jats:sc>\n            first automatically generates an interpretable rule set that categorizes the input space into two groups. Alongside,\n            <jats:sc>TestSGD<\/jats:sc>\n            also provides an estimated group discrimination score based on sampling the input space to measure the degree of the identified subtle group discrimination, which is guaranteed to be accurate up to an error bound. We evaluate\n            <jats:sc>TestSGD<\/jats:sc>\n            on multiple neural network models trained on popular datasets including both structured data and text data. The experiment results show that\n            <jats:sc>TestSGD<\/jats:sc>\n            is effective and efficient in identifying and measuring such subtle group discrimination that has never been revealed before. Furthermore, we show that the testing results of\n            <jats:sc>TestSGD<\/jats:sc>\n            can be used to mitigate such discrimination through retraining with negligible accuracy drop.\n          <\/jats:p>","DOI":"10.1145\/3591869","type":"journal-article","created":{"date-parts":[[2023,4,10]],"date-time":"2023-04-10T13:15:53Z","timestamp":1681132553000},"page":"1-24","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":10,"title":["<scp>TestSGD<\/scp>\n            : Interpretable Testing of Neural Networks against Subtle Group Discrimination"],"prefix":"10.1145","volume":"32","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-3239-4804","authenticated-orcid":false,"given":"Mengdi","family":"Zhang","sequence":"first","affiliation":[{"name":"Singapore Management University, Singapore"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3545-1392","authenticated-orcid":false,"given":"Jun","family":"Sun","sequence":"additional","affiliation":[{"name":"Singapore Management University, Singapore"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7113-7635","authenticated-orcid":false,"given":"Jingyi","family":"Wang","sequence":"additional","affiliation":[{"name":"Zhejiang University, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3322-5420","authenticated-orcid":false,"given":"Bing","family":"Sun","sequence":"additional","affiliation":[{"name":"Singapore Management University, Singapore"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2023,9,30]]},"reference":[{"key":"e_1_3_2_2_2","first-page":"265","volume-title":"Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI\u201916)","author":"Abadi Mart\u00edn","year":"2016","unstructured":"Mart\u00edn Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard et\u00a0al. 2016. Tensorflow: A system for large-scale machine learning. In Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI\u201916). 265\u2013283."},{"key":"e_1_3_2_3_2","article-title":"Iterative orthogonal feature projection for diagnosing bias in black-box models","author":"Adebayo Julius","year":"2016","unstructured":"Julius Adebayo and Lalana Kagal. 2016. Iterative orthogonal feature projection for diagnosing bias in black-box models. arXiv preprint arXiv:1611.04967 (2016).","journal-title":"arXiv preprint arXiv:1611.04967"},{"key":"e_1_3_2_4_2","article-title":"Automated test generation to detect individual discrimination in AI models","author":"Agarwal Aniya","year":"2018","unstructured":"Aniya Agarwal, Pranay Lohia, Seema Nagar, Kuntal Dey, and Diptikalyan Saha. 2018. Automated test generation to detect individual discrimination in AI models. arXiv preprint arXiv:1809.03260 (2018).","journal-title":"arXiv preprint arXiv:1809.03260"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3264590"},{"key":"e_1_3_2_6_2","article-title":"Machine bias: There\u2019s software used across the country to predict future criminals. And it\u2019s biased against blacks.","author":"Angwin Julia","year":"2016","unstructured":"Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2016. Machine bias: There\u2019s software used across the country to predict future criminals. And it\u2019s biased against blacks. ProPublica. Retrieved from https:\/\/www.propublica.org\/article\/machine-bias-risk-assessments-in-criminal-sentencing.","journal-title":"ProPublica"},{"key":"e_1_3_2_7_2","unstructured":"Lisa C. Anthony and Mei Liu. 2003. Analysis of differential prediction of law school performance by racial\/ethnic subgroups based on the 1996\u20131998 entering law school classes. LSAC research report series. (2003)."},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409704"},{"key":"e_1_3_2_9_2","article-title":"Racial disparity in natural language processing: A case study of social media African-American English","author":"Blodgett Su Lin","year":"2017","unstructured":"Su Lin Blodgett and Brendan O\u2019Connor. 2017. Racial disparity in natural language processing: A case study of social media African-American English. arXiv preprint arXiv:1707.00061 (2017).","journal-title":"arXiv preprint arXiv:1707.00061"},{"key":"e_1_3_2_10_2","article-title":"End to end learning for self-driving cars","author":"Bojarski Mariusz","year":"2016","unstructured":"Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang et\u00a0al. 2016. End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316 (2016).","journal-title":"arXiv preprint arXiv:1604.07316"},{"key":"e_1_3_2_11_2","unstructured":"Tolga Bolukbasi Kai-Wei Chang James Y. Zou Venkatesh Saligrama and Adam Tauman Kalai. 2016. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. (2016)."},{"key":"e_1_3_2_12_2","first-page":"77","volume-title":"Proceedings of the Conference on Fairness, Accountability and Transparency","author":"Buolamwini Joy","year":"2018","unstructured":"Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Proceedings of the Conference on Fairness, Accountability and Transparency. PMLR, 77\u201391."},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10618-010-0190-x"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3278721.3278729"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1145\/2090236.2090255"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1145\/2783258.2783311"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/3106237.3106277"},{"key":"e_1_3_2_18_2","article-title":"Justicia: A stochastic SAT approach to formally verify fairness","author":"Ghosh Bishwamittra","year":"2020","unstructured":"Bishwamittra Ghosh, Debabrota Basu, and Kuldeep S. Meel. 2020. Justicia: A stochastic SAT approach to formally verify fairness. arXiv preprint arXiv:2009.06516 (2020).","journal-title":"arXiv preprint arXiv:2009.06516"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1001\/jamainternmed.2018.3763"},{"key":"e_1_3_2_20_2","article-title":"A research agenda: Dynamic models to defend against correlated attacks","author":"Goodfellow Ian","year":"2019","unstructured":"Ian Goodfellow. 2019. A research agenda: Dynamic models to defend against correlated attacks. arXiv preprint arXiv:1903.06293 (2019).","journal-title":"arXiv preprint arXiv:1903.06293"},{"key":"e_1_3_2_21_2","article-title":"Equality of opportunity in supervised learning","volume":"29","author":"Hardt Moritz","year":"2016","unstructured":"Moritz Hardt, Eric Price, and Nati Srebro. 2016. Equality of opportunity in supervised learning. Adv. Neural Inf. Process. Syst. 29 (2016).","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"e_1_3_2_22_2","unstructured":"Hans Hofmann. 1994. German credit dataset. Retrieved from https:\/\/archive.ics.uci.edu\/ml\/datasets\/statlog+(german+credit+data)."},{"key":"e_1_3_2_23_2","unstructured":"Marianne Huchard Christian K\u00e4stner and Gordon Fraser. 2018. In Proceedings of the 33rd ACM\/IEEE International Conference on Automated Software Engineering (ASE\u201918) . ACM Press."},{"key":"e_1_3_2_24_2","article-title":"Fairness in learning: Classic and contextual bandits","volume":"29","author":"Joseph Matthew","year":"2016","unstructured":"Matthew Joseph, Michael Kearns, Jamie H. Morgenstern, and Aaron Roth. 2016. Fairness in learning: Classic and contextual bandits. Adv. Neural Inf. Process. Syst. 29 (2016).","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"e_1_3_2_25_2","first-page":"2564","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Kearns Michael","year":"2018","unstructured":"Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. 2018. Preventing fairness gerrymandering: Auditing and learning for subgroup fairness. In Proceedings of the International Conference on Machine Learning. PMLR, 2564\u20132572."},{"key":"e_1_3_2_26_2","article-title":"Inherent trade-offs in the fair determination of risk scores","author":"Kleinberg Jon","year":"2016","unstructured":"Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. 2016. Inherent trade-offs in the fair determination of risk scores. arXiv preprint arXiv:1609.05807 (2016).","journal-title":"arXiv preprint arXiv:1609.05807"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0933-3657(01)00077-X"},{"key":"e_1_3_2_28_2","article-title":"Counterfactual fairness","author":"Kusner Matt J.","year":"2017","unstructured":"Matt J. Kusner, Joshua R. Loftus, Chris Russell, and Ricardo Silva. 2017. Counterfactual fairness. arXiv preprint arXiv:1703.06856 (2017).","journal-title":"arXiv preprint arXiv:1703.06856"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939874"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2020\/64"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.5555\/2002472.2002491"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.dss.2014.03.001"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/3132747.3132785"},{"key":"e_1_3_2_34_2","unstructured":"Michael Redmond. 2009. Communities and Crime dataset. Retrieved from http:\/\/archive.ics.uci.edu\/ml\/\/datasets\/Communities+and+Crime)."},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1023\/A:1022607331053"},{"key":"e_1_3_2_36_2","unstructured":"Barry Becker and Ronny Kohavi. 1996. Data mining and visualization. Retrieved from https:\/\/archive.ics.uci.edu\/ml\/datasets\/adult."},{"key":"e_1_3_2_37_2","article-title":"Learning certified individually fair representations","author":"Ruoss Anian","year":"2020","unstructured":"Anian Ruoss, Mislav Balunovi\u0107, Marc Fischer, and Martin Vechev. 2020. Learning certified individually fair representations. arXiv preprint arXiv:2002.10312 (2020).","journal-title":"arXiv preprint arXiv:2002.10312"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/3299869.3319901"},{"key":"e_1_3_2_39_2","doi-asserted-by":"crossref","unstructured":"Motoki Sato Jun Suzuki Hiroyuki Shindo and Yuji Matsumoto. 2018. Interpretable adversarial perturbation in input embedding space for text. (2018).","DOI":"10.24963\/ijcai.2018\/601"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1708.08559"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1109\/EuroSP.2017.29"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238165"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/3194770.3194776"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4612-0919-5_18"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177730197"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1610.08914"},{"key":"e_1_3_2_47_2","first-page":"3921","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Yang Hongyu","year":"2017","unstructured":"Hongyu Yang, Cynthia Rudin, and Margo Seltzer. 2017. Scalable Bayesian rule lists. In Proceedings of the International Conference on Machine Learning. PMLR, 3921\u20133930."},{"key":"e_1_3_2_48_2","first-page":"962","volume-title":"Artificial Intelligence and Statistics","author":"Zafar Muhammad Bilal","year":"2017","unstructured":"Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rogriguez, and Krishna P. Gummadi. 2017. Fairness constraints: Mechanisms for fair classification. In Artificial Intelligence and Statistics. PMLR, 962\u2013970."},{"key":"e_1_3_2_49_2","unstructured":"Mengdi Zhang. 2022. GitHub repository for the subtle discrimination testing project. Retrieved from https:\/\/github.com\/zhangmengling\/subtle_discrimination_testing.git."},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2107.08176"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2021.3101478"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3591869","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3591869","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:37:45Z","timestamp":1750178265000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3591869"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,30]]},"references-count":50,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2023,11,30]]}},"alternative-id":["10.1145\/3591869"],"URL":"https:\/\/doi.org\/10.1145\/3591869","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,9,30]]},"assertion":[{"value":"2022-08-29","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-03-09","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-09-30","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}