{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,13]],"date-time":"2026-11-13T13:58:27Z","timestamp":1794578307807,"version":"4.9.1"},"reference-count":319,"publisher":"Association for Computing Machinery (ACM)","issue":"5","license":[{"start":{"date-parts":[[2024,6,4]],"date-time":"2024-06-04T00:00:00Z","timestamp":1717459200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"ERC Advanced","award":["741278"],"award-info":[{"award-number":["741278"]}]},{"name":"EPIC: Evolutionary Program Improvement Collaborators"},{"name":"Research Council of Norway through the secureIT project","award":["IKTPLUSS #288787"],"award-info":[{"award-number":["IKTPLUSS #288787"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2024,6,30]]},"abstract":"<jats:p>Unfair behaviors of Machine Learning (ML) software have garnered increasing attention and concern among software engineers. To tackle this issue, extensive research has been dedicated to conducting fairness testing of ML software, and this article offers a comprehensive survey of existing studies in this field. We collect 100 papers and organize them based on the testing workflow (i.e., how to test) and testing components (i.e., what to test). Furthermore, we analyze the research focus, trends, and promising directions in the realm of fairness testing. We also identify widely adopted datasets and open-source tools for fairness testing.<\/jats:p>","DOI":"10.1145\/3652155","type":"journal-article","created":{"date-parts":[[2024,3,11]],"date-time":"2024-03-11T08:18:30Z","timestamp":1710145110000},"page":"1-59","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":84,"title":["Fairness Testing: A Comprehensive Survey and Analysis of Trends"],"prefix":"10.1145","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4765-1893","authenticated-orcid":false,"given":"Zhenpeng","family":"Chen","sequence":"first","affiliation":[{"name":"University College London, London, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0481-7264","authenticated-orcid":false,"given":"Jie M.","family":"Zhang","sequence":"additional","affiliation":[{"name":"King's College London, London, United Kingdom of"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8684-5909","authenticated-orcid":false,"given":"Max","family":"Hort","sequence":"additional","affiliation":[{"name":"Simula Research Laboratory, Oslo, Norway"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5864-4488","authenticated-orcid":false,"given":"Mark","family":"Harman","sequence":"additional","affiliation":[{"name":"University College London, London, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9146-442X","authenticated-orcid":false,"given":"Federica","family":"Sarro","sequence":"additional","affiliation":[{"name":"University College London, London, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,6,4]]},"reference":[{"key":"e_1_3_3_2_2","unstructured":"Data.world. 1977. The US Executions dataset. Retrieved from https:\/\/data.world\/markmarkoh\/executions-since-1977"},{"key":"e_1_3_3_3_2","unstructured":"UCI Machine Learning Repository. 1994. The Diabetes dataset. Retrieved from https:\/\/archive.ics.uci.edu\/ml\/datasets\/diabetes"},{"key":"e_1_3_3_4_2","unstructured":"UCI Machine Learning Repository. 1994. The German Credit dataset. Retrieved from https:\/\/archive.ics.uci.edu\/ml\/datasets\/Statlog+%28German+Credit+Data%29"},{"key":"e_1_3_3_5_2","unstructured":"Kaggle. 1998. The Law School dataset. Retrieved from https:\/\/www.kaggle.com\/datasets\/danofer\/law-school-admissions-bar-passage"},{"key":"e_1_3_3_6_2","unstructured":"UCI Machine Learning Repository. 2000. The Census-Income (KDD) dataset. Retrieved from https:\/\/archive.ics.uci.edu\/dataset\/117\/census+income+kdd"},{"key":"e_1_3_3_7_2","unstructured":"Kaggle. 2000. The Fraud Detection dataset. Retrieved from https:\/\/www.kaggle.com\/competitions\/frauddetection\/data"},{"key":"e_1_3_3_8_2","unstructured":"UCI Machine Learning Repository. 2001. The Heart Health dataset. Retrieved from https:\/\/archive.ics.uci.edu\/ml\/datasets\/Heart+Disease"},{"key":"e_1_3_3_9_2","unstructured":"Gary B. Huang Manu Ramesh Tamara Berg and Erik Learned-Miller. 2007. LFW. Retrieved from http:\/\/vis-www.cs.umass.edu\/lfw\/"},{"key":"e_1_3_3_10_2","first-page":"1","article-title":"IEEE standard classification for software anomalies","unstructured":"IEEE. 2010. IEEE standard classification for software anomalies. IEEE Std 1044-2009 (Revision of IEEE Std 1044-1993). 1\u201323.","journal-title":"IEEE Std 1044-2009 (Revision of IEEE Std 1044-1993)"},{"key":"e_1_3_3_11_2","unstructured":"UCI Machine Learning Repository. 2011. The Communities and Crime dataset. Retrieved from http:\/\/archive.ics.uci.edu\/ml\/datasets\/Communities%20and%20Crime%20Unnormalized"},{"key":"e_1_3_3_12_2","unstructured":"IMDb.com Inc. 2011. The IMDB dataset. Retrieved from https:\/\/www.imdb.com\/interfaces\/"},{"key":"e_1_3_3_13_2","unstructured":"Andrew L. Maas Raymond E. Daly Peter T. Pham Dan Huang Andrew Y. Ng and Christopher Potts. 2011. The Large Movie Review dataset. Retrieved from https:\/\/ai.stanford.edu\/amaas\/data\/sentiment\/"},{"key":"e_1_3_3_14_2","unstructured":"UCI Machine Learning Repository. 2014. The Bank dataset. Retrieved from https:\/\/archive.ics.uci.edu\/ml\/datasets\/Bank+Marketing"},{"key":"e_1_3_3_15_2","unstructured":"Minnesota Population Center. 2014. The Dutch Census of 2001 dataset. Retrieved from https:\/\/microdata.worldbank.org\/index.php\/catalog\/2102"},{"key":"e_1_3_3_16_2","unstructured":"UCI Machine Learning Repository. 2014. The Student Performance dataset. Retrieved from https:\/\/archive.ics.uci.edu\/ml\/datasets\/Student+Performance"},{"key":"e_1_3_3_17_2","unstructured":"Ziwei Liu Ping Luo Xiaogang Wang and Xiaoou Tang. 2015. The CelebA dataset. Retrieved from https:\/\/mmlab.ie.cuhk.edu.hk\/projects\/CelebA.html"},{"key":"e_1_3_3_18_2","unstructured":"ForeverData.org. 2015. The Heritage Health dataset. Retrieved from https:\/\/foreverdata.org\/1015\/index.html"},{"key":"e_1_3_3_19_2","unstructured":"Oscar Celma. 2015. The LFM360K dataset. Retrieved from https:\/\/www.upf.edu\/web\/mtg\/lastfm360k"},{"key":"e_1_3_3_20_2","unstructured":"Agency for Healthcare Research and Quality. 2015. The Mep15 dataset. Retrieved from https:\/\/meps.ahrq.gov\/mepsweb\/data_stats\/download_data_files_detail.jsp?cboPufNumber=HC-181"},{"key":"e_1_3_3_21_2","unstructured":"GroupLens Research. 2015. The MovieLens dataset. Retrieved from https:\/\/movielens.org\/"},{"key":"e_1_3_3_22_2","unstructured":"O. M. Parkhi A. Vedaldi A. Zisserman. 2015. The VGGFace dataset. Retrieved from https:\/\/www.robots.ox.ac.uk\/vgg\/data\/vgg_face\/#publi"},{"key":"e_1_3_3_23_2","unstructured":"Rafi Letzter. 2016. Amazon just showed us that \u201cunbiased\u201d algorithms can be inadvertently racist. Retrieved from https:\/\/www.businessinsider.com\/how-algorithms-can-be-racist-2016-4?r=US&IR=T"},{"key":"e_1_3_3_24_2","unstructured":"Jeff Larson Surya Mattu Lauren Kirchner and Julia Angwin. 2016. The Compas dataset. Retrieved from https:\/\/github.com\/propublica\/compas-analysis"},{"key":"e_1_3_3_25_2","unstructured":"UCI Machine Learning Repository. 2016. The Default Credit dataset. Retrieved from https:\/\/archive.ics.uci.edu\/ml\/datasets\/default+of+credit+card+clients"},{"key":"e_1_3_3_26_2","unstructured":"Julia Angwin Jeff Larson Surya Mattu and Lauren Kirchner. 2016. Machine bias. Retrieved from https:\/\/www.propublica.org\/article\/machine-bias-risk-assessments-in-criminal-sentencing"},{"key":"e_1_3_3_27_2","unstructured":"Agency for Healthcare Research and Quality. 2016. The Mep16 dataset. Retrieved from https:\/\/meps.ahrq.gov\/mepsweb\/data_stats\/download_data_files_detail.jsp?cboPufNumber=HC-192"},{"key":"e_1_3_3_28_2","unstructured":"UCI Machine Learning Repository. 2017. The Adult Census Income dataset. Retrieved from https:\/\/archive.ics.uci.edu\/ml\/datasets\/adult"},{"key":"e_1_3_3_29_2","unstructured":"Jieyu Zhao Tianlu Wang Mark Yatskar Vicente Ordonez Kai-Wei Chang. 2017. The COCO-gender dataset. Retrieved from https:\/\/github.com\/uclanlp\/reducingbias"},{"key":"e_1_3_3_30_2","unstructured":"Florian Tramer Vaggelis Atlidakis Roxana Geambasu Daniel Hsu Jean-Pierre Hubaux Mathias Humbert Ari Juels and Huang Lin. 2017. FairTest Retrieved from https:\/\/github.com\/columbia\/fairtest"},{"key":"e_1_3_3_31_2","unstructured":"Kaggle. 2017. The Home Credit dataset. Retrieved from https:\/\/www.kaggle.com\/c\/home-credit-default-risk"},{"key":"e_1_3_3_32_2","unstructured":"Richard Socher Alex Perelygin Jean Wu Jason Chuang Christopher Manning Andrew Ng and Christopher Potts. 2017. The SST dataset. Retrieved from http:\/\/nlpprogress.com\/english\/sentiment_analysis.html"},{"key":"e_1_3_3_33_2","unstructured":"Shan Li Weihong Deng and JunPing Du. 2017. The RAF-DB dataset. Retrieved from http:\/\/whdeng.cn\/RAF\/model1.html"},{"key":"e_1_3_3_34_2","unstructured":"Sainyam Galhotra Yuriy Brun and Alexandra Meliou. 2017. Themis. Retrieved from https:\/\/github.com\/LASER-UMASS\/Themis"},{"key":"e_1_3_3_35_2","unstructured":"Kaggle. 2017. The Titanic dataset. Retrieved from https:\/\/www.kaggle.com\/c\/titanic\/data"},{"key":"e_1_3_3_36_2","unstructured":"Conversation-AI. 2017. The Wikipedia comment dataset. Retrieved from https:\/\/github.com\/conversationai\/unintended-ml-bias-analysis"},{"key":"e_1_3_3_37_2","unstructured":"Sakshi Udeshi Pryanshu Arora and Sudipta Chattopadhyay. 2018. Aequitas. Retrieved from https:\/\/github.com\/sakshiudeshi\/Aequitas"},{"key":"e_1_3_3_38_2","unstructured":"IBM Research. 2018. AIF360. Retrieved from https:\/\/aif360.readthedocs.io\/en\/stable\/index.html"},{"key":"e_1_3_3_39_2","unstructured":"Svetlana Kiritchenko and Saif M. Mohammad. 2018. The EEC dataset. Retrieved from https:\/\/competitions.codalab.org\/competitions\/17751"},{"key":"e_1_3_3_40_2","unstructured":"Joy Buolamwini and Timnit Gebru. 2018. The PBB dataset. Retrieved from http:\/\/gendershades.org\/overview.html"},{"key":"e_1_3_3_41_2","unstructured":"Jieyu Zhao Tianlu Wang Mark Yatskar Vicente Ordonez Kai-Wei Chang. 2018. The WinoBias dataset. Retrieved from https:\/\/paperswithcode.com\/dataset\/winobias"},{"key":"e_1_3_3_42_2","unstructured":"Rachel Rudinger Jason Naradowsky Brian Leonard and Benjamin Van Durme. 2018. The Winogender dataset. Retrieved from https:\/\/github.com\/rudinger\/winogender-schemas"},{"key":"e_1_3_3_43_2","unstructured":"Kaggle. 2019. The BlackFriday dataset. Retrieved from https:\/\/www.kaggle.com\/datasets\/sdolezel\/black-friday"},{"key":"e_1_3_3_44_2","unstructured":"Mei Wang Weihong Deng Jiani Hu Xunqiang Tao and Yaohai Huang. 2019. The BUPT-Transferface dataset. Retrieved from http:\/\/www.whdeng.cn\/RFW\/Trainingdataste.html"},{"key":"e_1_3_3_45_2","unstructured":"\u00c1ngel Alexander Cabrera Will Epperson Fred Hohman Minsuk Kahng Jamie Morgenstern and Duen Horng Chau. 2019. FairVis. Retrieved from https:\/\/github.com\/poloclub\/FairVis"},{"key":"e_1_3_3_46_2","unstructured":"Jane Coaston. 2019. The intersectionality wars. Retrieved from https:\/\/www.vox.com\/the-highlight\/2019\/5\/20\/18542843\/intersectionality-conservatism-law-race-gender-discrimination"},{"key":"e_1_3_3_47_2","unstructured":"Kaggle. 2019. The Jigsaw comment dataset. Retrieved from https:\/\/www.kaggle.com\/c\/jigsaw-toxic-comment-classification-challenge"},{"key":"e_1_3_3_48_2","unstructured":"Ninareh Mehrabi Thamme Gowda Fred Morstatter Nanyun Peng and Aram Galstyan. 2019. NERGenderBias. Retrieved from https:\/\/github.com\/Ninarehm\/NERGenderBias"},{"key":"e_1_3_3_49_2","unstructured":"Emily Sheng Kai-Wei Chang Premkumar Natarajan and Nanyun Peng. 2019. The nlg-bias dataset. Retrieved from https:\/\/github.com\/ewsheng\/nlg-bias"},{"key":"e_1_3_3_50_2","unstructured":"Bruno Castro da Silva. 2019. The UFRGS Entrance Exam and GPA Data. Retrieved from https:\/\/dataverse.harvard.edu\/dataset.xhtml?persistentId=doi:10.7910\/DVN\/O35FW8"},{"key":"e_1_3_3_51_2","unstructured":"Peixin Zhang Jingyi Wang Jun Sun Guoliang Dong Xinyu Wang Xingen Wang Ting Dai and Jinsong Dong. 2020. ADF. Retrieved from https:\/\/github.com\/pxzhang94\/ADF"},{"key":"e_1_3_3_52_2","unstructured":"Marco Tulio Ribeiro Tongshuang Wu Carlos Guestrin and Sameer Singh. 2020. CheckList. Retrieved from https:\/\/github.com\/marcotcr\/checklist"},{"key":"e_1_3_3_53_2","unstructured":"Jesse Vig Sebastian Gehrmann Yonatan Belinkov Sharon Qian Daniel Nevo Simas Sakenis Jason Huang Yaron Singer and Stuart Shieber. 2020. CMA. Retrieved from https:\/\/github.com\/sebastianGehrmann\/CausalMediationAnalysis"},{"key":"e_1_3_3_54_2","unstructured":"Yuchi Tian Ziyuan Zhong Vicente Ordonez Gail E. Kaiser and Baishakhi Ray. 2020. DeepInspect. Retrieved from https:\/\/github.com\/ARiSE-Lab\/DeepInspect"},{"key":"e_1_3_3_55_2","unstructured":"Haochen Liu Jamell Dacon Wenqi Fan Hui Liu Zitao Liu and Jiliang Tang. 2020. DialogueFairness. Retrieved from https:\/\/github.com\/zgahhblhc\/DialogueFairness"},{"key":"e_1_3_3_56_2","unstructured":"Arnab Sharma and Heike Wehrheim. 2020. fairCheck. Retrieved from https:\/\/github.com\/arnabsharma91\/fairCheck"},{"key":"e_1_3_3_57_2","unstructured":"Joymallya Chakraborty Suvodeep Majumder Zhe Yu and Tim Menzies. 2020. Fairway. Retrieved from https:\/\/github.com\/joymallyac\/Fairway"},{"key":"e_1_3_3_58_2","doi-asserted-by":"crossref","unstructured":"Sriram Vasudevan and Krishnaram Kenthapadi. 2020. LiFT. Retrieved from https:\/\/github.com\/linkedin\/LiFT","DOI":"10.1145\/3340531.3412705"},{"key":"e_1_3_3_59_2","unstructured":"Pingchuan Ma Shuai Wang and Jin Liu. 2020. MT-NLP. Retrieved from https:\/\/github.com\/pckennethma\/MT-NLP"},{"key":"e_1_3_3_60_2","unstructured":"Marta R. Costa-jussa. 2020. WinoST. Retrieved from https:\/\/zenodo.org\/record\/4139080#.YtGEc-zMJAc"},{"key":"e_1_3_3_61_2","unstructured":"Muhammad Hilmi Asyrofi Zhou Yang Imam Nur Bani Yusuf Hong Jin Kang Ferdian Thung and David Lo. 2021. BiasFinder. Retrieved from https:\/\/github.com\/soarsmu\/BiasFinder"},{"key":"e_1_3_3_62_2","unstructured":"Zhou Yang Muhammad Hilmi Asyrofi and David Lo. 2021. BiasRV. Retrieved from https:\/\/github.com\/soarsmu\/BiasRV"},{"key":"e_1_3_3_63_2","unstructured":"Lingfeng Zhang Yueling Zhang and Min Zhang. 2021. EIDIG. Retrieved from https:\/\/github.com\/LingfengZhang98\/EIDIG"},{"key":"e_1_3_3_64_2","unstructured":"Ming Fan Wenying Wei Wuxia Jin Zijiang Yang and Ting Liu. 2021. ExpGA. Retrieved from https:\/\/github.com\/waving7799\/ExpGA"},{"key":"e_1_3_3_65_2","doi-asserted-by":"crossref","unstructured":"Joymallya Chakraborty Suvodeep Majumder and Tim Menzies. 2021. Fair-SMOTE. Retrieved from https:\/\/github.com\/joymallyac\/Fair-SMOTE","DOI":"10.1145\/3524491.3527305"},{"key":"e_1_3_3_66_2","unstructured":"Max Hort Jie M. Zhang Federica Sarro and Mark Harman. 2021. Fairea. Retrieved from https:\/\/github.com\/maxhort\/Fairea"},{"key":"e_1_3_3_67_2","unstructured":"Kimmo Karkkainen and Jungseock Joo. 2021. The FairFace dataset. Retrieved from https:\/\/github.com\/dchen236\/FairFace"},{"key":"e_1_3_3_68_2","unstructured":"Kaggle. 2021. The Sentiment140 dataset. Retrieved from https:\/\/www.kaggle.com\/datasets\/kazanova\/sentiment140"},{"key":"e_1_3_3_69_2","unstructured":"Sai Sathiesh Rajan Sakshi Udeshi and Sudipta Chattopadhyay. 2022. AequeVox. Retrieved from https:\/\/github.com\/sparkssss\/AequeVox"},{"key":"e_1_3_3_70_2","unstructured":"Sai Sathiesh Rajan Sakshi Udeshi and Sudipta Chattopadhyay. 2022. The AequeVox dataset. Retrieved from https:\/\/zenodo.org\/record\/5897347"},{"key":"e_1_3_3_71_2","unstructured":"UCI Machine Learning Repository. 2022. The Arrhythmia dataset. Retrieved from https:\/\/archive.ics.uci.edu\/dataset\/5\/arrhythmia"},{"key":"e_1_3_3_72_2","unstructured":"Ezekiel Soremekun Sakshi Udeshi and Sudipta Chattopadhyay. 2022. ASTRAEA. Retrieved from https:\/\/github.com\/sakshiudeshi\/Astraea"},{"key":"e_1_3_3_73_2","unstructured":"Angelina Wang and Olga Russakovsky. 2022. \\(BiasAmp_{\\rightarrow }\\) . Retrieved from https:\/\/github.com\/princetonvisualai\/directional-bias-amp"},{"key":"e_1_3_3_74_2","unstructured":"DBLP computer science bibliography. 2022. DBLP. Retrieved from https:\/\/dblp.org"},{"key":"e_1_3_3_75_2","unstructured":"Xuanqi Gao Juan Zhai Shiqing Ma Chao Shen Yufei Chen and Qian Wang. 2022. FairNeuron. Retrieved from https:\/\/github.com\/Antimony5292\/FairNeuron"},{"key":"e_1_3_3_76_2","unstructured":"Jiang Zhang Jiang Zhang Sergey Mechtaev and Abhik Roychoudhury. 2022. FairRepair. Retrieved from https:\/\/github.com\/fairrepair\/fair-repair"},{"key":"e_1_3_3_77_2","unstructured":"Eric Michael Smith Melissa Hall Melanie Kambadur Eleonora Presani and Adina Williams. 2022. HOLISTICBIAS. Retrieved from https:\/\/github.com\/facebookresearch\/ResponsibleNLP\/tree\/main\/holistic_bias"},{"key":"e_1_3_3_78_2","unstructured":"Minghua Ma Zhao Tian Max Hort Federica Sarro Hongyu Zhang Qingwei Lin and Dongmei Zhang. 2022. I&D. Retrieved from https:\/\/anonymous.4open.science\/r\/fairness-095F\/README.md"},{"key":"e_1_3_3_79_2","unstructured":"Yanhui Li Linghan Meng Lin Chen Li Yu Di Wu Yuming Zhou and Baowen Xu. 2022. LTDD. Retrieved from https:\/\/github.com\/fairnesstest\/LTDD"},{"key":"e_1_3_3_80_2","unstructured":"Zhenpeng Chen Jie M. Zhang Federica Sarro and Mark Harman. 2022. MAAT. Retrieved from https:\/\/github.com\/chenzhenpeng18\/FSE22-MAAT"},{"key":"e_1_3_3_81_2","unstructured":"Arnab Sharma Caglar Demir Axel-Cyrille Ngonga Ngomo and Heike Wehrheim. 2022. MLCheck. Retrieved from https:\/\/github.com\/anonymseal\/MLCheck"},{"key":"e_1_3_3_82_2","doi-asserted-by":"crossref","unstructured":"Haibin Zheng Zhiqing Chen Tianyu Du Xuhong Zhang Yao Cheng Shouling Ji Jingyi Wang Yue Yu and Jinyin Chen. 2022. NeuronFair. Retrieved from https:\/\/github.com\/haibinzheng\/NeuronFair","DOI":"10.1145\/3510003.3510123"},{"key":"e_1_3_3_83_2","unstructured":"Saeid Tizpaz-Niari Ashish Kumar Gang Tan and Ashutosh Trivedi. 2022. Parfait-ML. Retrieved from https:\/\/github.com\/Tizpaz\/Parfait-ML"},{"key":"e_1_3_3_84_2","unstructured":"Angelina Wang Alexander Liu Ryan Zhang Anat Kleiman Leslie Kim Dora Zhao Iroha Shirai Arvind Narayanan and Olga Russakovsky. 2022. REVISE. Retrieved from https:\/\/github.com\/princetonvisualai\/revise-tool"},{"key":"e_1_3_3_85_2","unstructured":"Guanhong Tao Weisong Sun Tingxu Han Chunrong Fang and Xiangyu Zhang. 2022. RULER. Retrieved from https:\/\/github.com\/wssun\/RULER"},{"key":"e_1_3_3_86_2","unstructured":"Anjana Perera Aldeida Aleti Chakkrit Tantithamthavorn Jirayus Jiarpakdee Burak Turhan Lisa Kuhn and Katie Walker. 2022. SBFT. Retrieved from https:\/\/github.com\/search-based-fairness-testing\/sbft"},{"key":"e_1_3_3_87_2","unstructured":"Shuo Yang Ping Luo Chen Change Loy and Xiaoou Tang. 2022. WIDER FACE. Retrieved from http:\/\/shuoyang1213.me\/WIDERFACE\/"},{"key":"e_1_3_3_88_2","unstructured":"Marcel R. Ackermann. 2023. 6 million publications. Retrieved from https:\/\/blog.dblp.org\/2022\/02\/22\/6-million-publications\/"},{"key":"e_1_3_3_89_2","unstructured":"Yuxuan Wan Wenxuan Wang Pinjia He Jiazhen Gu Haonan Bai and Michael Lyu. 2023. BiasAsker. Retrieved from https:\/\/github.com\/yxwan123\/BiasAsker"},{"key":"e_1_3_3_90_2","unstructured":"Verya Monjezi Ashutosh Trivedi Gang Tan and Saeid Tizpaz-Niari. 2023. DICE. Retrieved from https:\/\/github.com\/armanunix\/Fairness-testing"},{"key":"e_1_3_3_91_2","unstructured":"Usman Gohar Sumon Biswas and Hridesh Rajan. 2023. FairEnsembles. Retrieved from https:\/\/github.com\/UsmanGohar\/FairEnsemble"},{"key":"e_1_3_3_92_2","unstructured":"Hilde Weerts Miroslav Dud\u00edk Richard Edgar Adrin Jalali Roman Lutz and Michael Madaio. 2023. Fairlearn. Retrieved from https:\/\/fairlearn.org\/"},{"key":"e_1_3_3_93_2","unstructured":"Kewen Peng Joymallya Chakraborty and Tim Menzies. 2023. FairMask. Retrieved from https:\/\/github.com\/anonymous12138\/biasmitigation"},{"key":"e_1_3_3_94_2","unstructured":"OpenAI. 2023. How should AI systems behave and who should decide? Retrieved from https:\/\/openai.com\/blog\/how-should-ai-systems-behave"},{"key":"e_1_3_3_95_2","unstructured":"Mengdi Zhang Jun Sun Jingyi Wang and Bing Sun. 2023. TestSGD Retrieved from https:\/\/github.com\/zhangmengling\/subtle_discrimination_testing"},{"key":"e_1_3_3_96_2","doi-asserted-by":"crossref","unstructured":"Jwala Dhamala Tony Sun Varun Kumar Satyapriya Krishna Yada Pruksachatkun Kai-Wei Chang and Rahul Gupta. 2021. The BOLD dataset. Retrieved from https:\/\/github.com\/jwaladhamala\/BOLD-Bias-in-open-ended-language-generation","DOI":"10.1145\/3442188.3445924"},{"key":"e_1_3_3_97_2","first-page":"265","volume-title":"Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI\u201916)","author":"Abadi Mart\u00edn","year":"2016","unstructured":"Mart\u00edn Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek Gordon Murray, Benoit Steiner, Paul A. Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. TensorFlow: A system for large-scale machine learning. In Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI\u201916). 265\u2013283."},{"key":"e_1_3_3_98_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338937"},{"key":"e_1_3_3_99_2","first-page":"1","volume-title":"Proceedings of the 29th IEEE International Requirements Engineering Conference (RE\u201921)","author":"Ahmad Khlood","year":"2021","unstructured":"Khlood Ahmad, Muneera Bano, Mohamed Abdelrazek, Chetan Arora, and John C. Grundy. 2021. What\u2019s up with requirements engineering for artificial intelligence systems?. In Proceedings of the 29th IEEE International Requirements Engineering Conference (RE\u201921). 1\u201312."},{"key":"e_1_3_3_100_2","first-page":"80:1\u201380:30","article-title":"FairSquare: Probabilistic verification of program fairness","author":"Albarghouthi Aws","year":"2017","unstructured":"Aws Albarghouthi, Loris D\u2019Antoni, Samuel Drews, and Aditya V. Nori. 2017. FairSquare: Probabilistic verification of program fairness. Proc. ACM Program. Lang.OOPSLA (2017), 80:1\u201380:30.","journal-title":"Proc. ACM Program. Lang."},{"key":"e_1_3_3_101_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSA-C52384.2021.00022"},{"key":"e_1_3_3_102_2","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3264590"},{"key":"e_1_3_3_103_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2021.3136169"},{"key":"e_1_3_3_104_2","doi-asserted-by":"publisher","DOI":"10.1145\/3442188.3445884"},{"key":"e_1_3_3_105_2","doi-asserted-by":"publisher","DOI":"10.1145\/3194770.3194778"},{"key":"e_1_3_3_106_2","doi-asserted-by":"publisher","DOI":"10.1145\/3560905.3568433"},{"key":"e_1_3_3_107_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58523-5_32"},{"key":"e_1_3_3_108_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368308.3415382"},{"key":"e_1_3_3_109_2","doi-asserted-by":"publisher","DOI":"10.1109\/RE57278.2023.00046"},{"key":"e_1_3_3_110_2","first-page":"671","article-title":"Big data\u2019s disparate impact","volume":"104","author":"Barocas Solon","year":"2016","unstructured":"Solon Barocas and Andrew D. Selbst. 2016. Big data\u2019s disparate impact. Calif. Law Rev. 104 (2016), 671.","journal-title":"Calif. Law Rev."},{"key":"e_1_3_3_111_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2014.2372785"},{"key":"e_1_3_3_112_2","doi-asserted-by":"publisher","DOI":"10.1145\/3360544"},{"key":"e_1_3_3_113_2","article-title":"AI fairness 360: An extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias","volume":"1810","author":"Bellamy Rachel K. E.","year":"2018","unstructured":"Rachel K. E. Bellamy, Kuntal Dey, Michael Hind, Samuel C. Hoffman, Stephanie Houde, Kalapriya Kannan, Pranay Lohia, Jacquelyn Martino, Sameep Mehta, Aleksandra Mojsilovic, Seema Nagar, Karthikeyan Natesan Ramamurthy, John T. Richards, Diptikalyan Saha, Prasanna Sattigeri, Moninder Singh, Kush R. Varshney, and Yunfeng Zhang. 2018. AI fairness 360: An extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias. CoRR abs\/1810.01943 (2018).","journal-title":"CoRR"},{"key":"e_1_3_3_114_2","doi-asserted-by":"publisher","DOI":"10.1177\/0049124118782533"},{"key":"e_1_3_3_115_2","doi-asserted-by":"publisher","DOI":"10.1145\/3292500.3330745"},{"key":"e_1_3_3_116_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409704"},{"key":"e_1_3_3_117_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3468536"},{"key":"e_1_3_3_118_2","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372845"},{"key":"e_1_3_3_119_2","article-title":"Simon says: Evaluating and mitigating bias in pruned neural networks with knowledge distillation","volume":"2106","author":"Blakeney Cody","year":"2021","unstructured":"Cody Blakeney, Nathaniel Huish, Yan Yan, and Ziliang Zong. 2021. Simon says: Evaluating and mitigating bias in pruned neural networks with knowledge distillation. CoRR abs\/2106.07849 (2021).","journal-title":"CoRR"},{"key":"e_1_3_3_120_2","doi-asserted-by":"publisher","DOI":"10.1093\/oxrep\/grab020"},{"key":"e_1_3_3_121_2","volume-title":"Proceedings of Machine Learning and Systems (MLSys\u201919)","author":"Breck Eric","year":"2019","unstructured":"Eric Breck, Neoklis Polyzotis, Sudip Roy, Steven Whang, and Martin Zinkevich. 2019. Data validation for machine learning. In Proceedings of Machine Learning and Systems (MLSys\u201919)."},{"key":"e_1_3_3_122_2","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3264838"},{"key":"e_1_3_3_123_2","doi-asserted-by":"publisher","DOI":"10.1109\/VAST47406.2019.8986948"},{"key":"e_1_3_3_124_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-022-07939-1"},{"key":"e_1_3_3_125_2","doi-asserted-by":"publisher","DOI":"10.1613\/jair.1.13197"},{"key":"e_1_3_3_126_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3468537"},{"key":"e_1_3_3_127_2","doi-asserted-by":"publisher","DOI":"10.1145\/3524491.3527305"},{"key":"e_1_3_3_128_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409697"},{"key":"e_1_3_3_129_2","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3418932"},{"key":"e_1_3_3_130_2","article-title":"Software engineering for fairness: A case study with hyperparameter optimization","volume":"1905","author":"Chakraborty Joymallya","year":"2019","unstructured":"Joymallya Chakraborty, Tianpei Xia, Fahmid M. Fahid, and Tim Menzies. 2019. Software engineering for fairness: A case study with hyperparameter optimization. CoRR abs\/1905.05786 (2019).","journal-title":"CoRR"},{"key":"e_1_3_3_131_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-662-46681-0_25"},{"key":"e_1_3_3_132_2","doi-asserted-by":"publisher","DOI":"10.5555\/3272525.3272526"},{"issue":"1","key":"e_1_3_3_133_2","first-page":"4:1\u20134:36","article-title":"A survey of compiler testing","volume":"53","author":"Chen Junjie","year":"2020","unstructured":"Junjie Chen, Jibesh Patra, Michael Pradel, Yingfei Xiong, Hongyu Zhang, Dan Hao, and Lu Zhang. 2020. A survey of compiler testing. Comput. Surv. 53, 1 (2020), 4:1\u20134:36.","journal-title":"Comput. Surv."},{"key":"e_1_3_3_134_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.01471"},{"key":"e_1_3_3_135_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409759"},{"key":"e_1_3_3_136_2","doi-asserted-by":"publisher","DOI":"10.1145\/3178876.3186157"},{"key":"e_1_3_3_137_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00068"},{"key":"e_1_3_3_138_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549093"},{"key":"e_1_3_3_139_2","doi-asserted-by":"publisher","DOI":"10.1145\/3583561"},{"key":"e_1_3_3_140_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597503.3639083"},{"key":"e_1_3_3_141_2","article-title":"A survey of model compression and acceleration for deep neural networks","volume":"1710","author":"Cheng Yu","year":"2017","unstructured":"Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. 2017. A survey of model compression and acceleration for deep neural networks. CoRR abs\/1710.09282 (2017).","journal-title":"CoRR"},{"key":"e_1_3_3_142_2","doi-asserted-by":"publisher","DOI":"10.1089\/big.2016.0047"},{"key":"e_1_3_3_143_2","doi-asserted-by":"publisher","DOI":"10.1002\/j.2333-8504.1966.tb00529.x"},{"key":"e_1_3_3_144_2","doi-asserted-by":"publisher","DOI":"10.1111\/j.1745-3984.1968.tb00613.x"},{"key":"e_1_3_3_145_2","doi-asserted-by":"publisher","DOI":"10.1145\/3097983.3098095"},{"key":"e_1_3_3_146_2","first-page":"23","volume-title":"Feminist Legal Theories","author":"Crenshaw Kimberl\u00e9","year":"2013","unstructured":"Kimberl\u00e9 Crenshaw. 2013. Demarginalizing the intersection of race and sex: A black feminist critique of antidiscrimination doctrine, feminist theory and antiracist politics. In Feminist Legal Theories. Routledge, 23\u201351."},{"key":"e_1_3_3_147_2","doi-asserted-by":"publisher","DOI":"10.1109\/WACV51458.2022.00393"},{"key":"e_1_3_3_148_2","doi-asserted-by":"publisher","DOI":"10.5555\/1792734.1792766"},{"key":"e_1_3_3_149_2","volume-title":"Proceedings of the CVPR Workshop on Fairness Accountability Transparency and Ethics in Computer Vision","author":"Denton Emily","year":"2019","unstructured":"Emily Denton, Ben Hutchinson, Margaret Mitchell, and Timnit Gebru. 2019. Detecting bias with generative counterfactual face attribute augmentation. In Proceedings of the CVPR Workshop on Fairness Accountability Transparency and Ethics in Computer Vision."},{"key":"e_1_3_3_150_2","volume-title":"Proceedings of the CVPR Workshop on Fairness Accountability Transparency and Ethics in Computer Vision","author":"Denton Emily","year":"2019","unstructured":"Emily Denton, Ben Hutchinson, Margaret Mitchell, Timnit Gebru, and Andrew Zaldivar. 2019. Image counterfactual sensitivity analysis for detecting unintended bias. In Proceedings of the CVPR Workshop on Fairness Accountability Transparency and Ethics in Computer Vision."},{"key":"e_1_3_3_151_2","doi-asserted-by":"publisher","DOI":"10.1145\/3442188.3445924"},{"key":"e_1_3_3_152_2","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173986"},{"key":"e_1_3_3_153_2","doi-asserted-by":"publisher","DOI":"10.1145\/3394486.3403199"},{"key":"e_1_3_3_154_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.1984.5010257"},{"key":"e_1_3_3_155_2","doi-asserted-by":"publisher","DOI":"10.1145\/2090236.2090255"},{"key":"e_1_3_3_156_2","article-title":"Algorithmic fairness datasets: The story so far","volume":"2202","author":"Fabris Alessandro","year":"2022","unstructured":"Alessandro Fabris, Stefano Messina, Gianmaria Silvello, and Gian Antonio Susto. 2022. Algorithmic fairness datasets: The story so far. CoRR abs\/2202.01711 (2022).","journal-title":"CoRR"},{"key":"e_1_3_3_157_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510137"},{"key":"e_1_3_3_158_2","doi-asserted-by":"publisher","DOI":"10.1145\/2783258.2783311"},{"key":"e_1_3_3_159_2","doi-asserted-by":"publisher","DOI":"10.1137\/130929886"},{"key":"e_1_3_3_160_2","first-page":"115","volume-title":"Proceedings of the 16th IEEE International Requirements Engineering Conference (RE\u201908)","author":"Finkelstein Anthony","year":"2008","unstructured":"Anthony Finkelstein, Mark Harman, S. Afshin Mansouri, Jian Ren, and Yuanyuan Zhang. 2008. \u201cFairness Analysis\u201d in requirements assignments. In Proceedings of the 16th IEEE International Requirements Engineering Conference (RE\u201908). 115\u2013124."},{"key":"e_1_3_3_161_2","doi-asserted-by":"publisher","DOI":"10.1145\/3106237.3106277"},{"key":"e_1_3_3_162_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510087"},{"key":"e_1_3_3_163_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2015.11.003"},{"key":"e_1_3_3_164_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-020-09386-w"},{"key":"e_1_3_3_165_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v35i9.16925"},{"key":"e_1_3_3_166_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v36i9.21187"},{"key":"e_1_3_3_167_2","doi-asserted-by":"publisher","DOI":"10.1109\/DSAA.2018.00018"},{"key":"e_1_3_3_168_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00133"},{"key":"e_1_3_3_169_2","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2023\/742"},{"key":"e_1_3_3_170_2","doi-asserted-by":"publisher","unstructured":"Eric Goldman. 2020. An Introduction to the California Consumer Privacy Act (CCPA) (July 1 2020). Santa Clara Univ. Legal Studies Research Paper Available at SSRN: https:\/\/ssrn.com\/abstract=3211013 or 10.2139\/ssrn.3211013","DOI":"10.2139\/ssrn.3211013"},{"key":"e_1_3_3_171_2","first-page":"2672","volume-title":"Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS\u201914)","author":"Goodfellow Ian J.","year":"2014","unstructured":"Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS\u201914). 2672\u20132680."},{"key":"e_1_3_3_172_2","first-page":"2","volume-title":"NIPS Symposium on Machine Learning and the Law","author":"Grgic-Hlaca Nina","year":"2016","unstructured":"Nina Grgic-Hlaca, Muhammad Bilal Zafar, Krishna P. Gummadi, and Adrian Weller. 2016. The case for process fairness in learning: Feature selection for fair decision making. In NIPS Symposium on Machine Learning and the Law, Vol. 1. 2."},{"key":"e_1_3_3_173_2","doi-asserted-by":"publisher","DOI":"10.5555\/3153803"},{"key":"e_1_3_3_174_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597926.3598058"},{"key":"e_1_3_3_175_2","article-title":"Error parity fairness: Testing for group fairness in regression tasks","volume":"2208","author":"Gursoy Furkan","year":"2022","unstructured":"Furkan Gursoy and Ioannis A. Kakadiaris. 2022. Error parity fairness: Testing for group fairness in regression tasks. CoRR abs\/2208.08279 (2022).","journal-title":"CoRR"},{"key":"e_1_3_3_176_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-13448-7_10"},{"key":"e_1_3_3_177_2","first-page":"3315","volume-title":"Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS\u201916)","author":"Hardt Moritz","year":"2016","unstructured":"Moritz Hardt, Eric Price, and Nati Srebro. 2016. Equality of opportunity in supervised learning. In Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS\u201916). 3315\u20133323."},{"key":"e_1_3_3_178_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2015.7102580"},{"key":"e_1_3_3_179_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0950-5849(01)00189-6"},{"issue":"4","key":"e_1_3_3_180_2","first-page":"811","article-title":"Measuring algorithmic fairness","volume":"106","author":"Hellman Deborah","year":"2020","unstructured":"Deborah Hellman. 2020. Measuring algorithmic fairness. Virginia Law Rev. 106, 4 (2020), 811\u2013866.","journal-title":"Virginia Law Rev."},{"key":"e_1_3_3_181_2","article-title":"Characterising bias in compressed models","volume":"2010","author":"Hooker Sara","year":"2020","unstructured":"Sara Hooker, Nyalleng Moorosi, Gregory Clark, Samy Bengio, and Emily Denton. 2020. Characterising bias in compressed models. CoRR abs\/2010.03058 (2020).","journal-title":"CoRR"},{"key":"e_1_3_3_182_2","doi-asserted-by":"publisher","DOI":"10.1145\/3631326"},{"key":"e_1_3_3_183_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE51524.2021.9678568"},{"key":"e_1_3_3_184_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3468565"},{"key":"e_1_3_3_185_2","doi-asserted-by":"publisher","DOI":"10.1109\/34.1000242"},{"key":"e_1_3_3_186_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.findings-emnlp.7"},{"key":"e_1_3_3_187_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3180235"},{"key":"e_1_3_3_188_2","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287600"},{"key":"e_1_3_3_189_2","doi-asserted-by":"publisher","DOI":"10.1145\/2372251.2372257"},{"key":"e_1_3_3_190_2","doi-asserted-by":"publisher","DOI":"10.1145\/1595696.1595715"},{"key":"e_1_3_3_191_2","first-page":"749","volume-title":"Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence (UAI\u201920)","author":"John Philips George","year":"2020","unstructured":"Philips George John, Deepak Vijaykeerthy, and Diptikalyan Saha. 2020. Verifying individual fairness in machine learning models. In Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence (UAI\u201920). 749\u2013758."},{"key":"e_1_3_3_192_2","article-title":"Gender Slopes: Counterfactual fairness for computer vision models by attribute manipulation","volume":"2005","author":"Joo Jungseock","year":"2020","unstructured":"Jungseock Joo and Kimmo K\u00e4rkk\u00e4inen. 2020. Gender Slopes: Counterfactual fairness for computer vision models by attribute manipulation. CoRR abs\/2005.10430 (2020).","journal-title":"CoRR"},{"key":"e_1_3_3_193_2","doi-asserted-by":"publisher","DOI":"10.1109\/WACV48630.2021.00159"},{"key":"e_1_3_3_194_2","volume-title":"Proceedings of the 6th International Conference on Learning Representations (ICLR\u201918)","author":"Karras Tero","year":"2018","unstructured":"Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. 2018. Progressive growing of GANs for improved quality, stability, and variation. In Proceedings of the 6th International Conference on Learning Representations (ICLR\u201918)."},{"issue":"6","key":"e_1_3_3_195_2","article-title":"A comparative study of white box, black box and grey box testing techniques","volume":"3","author":"Khan Mohd Ehmer","year":"2012","unstructured":"Mohd Ehmer Khan and Farmeena Khan. 2012. A comparative study of white box, black box and grey box testing techniques. Int. J. Advanc. Comput. Sci. Applic. 3, 6 (2012).","journal-title":"Int. J. Advanc. Comput. Sci. Applic."},{"key":"e_1_3_3_196_2","first-page":"139","volume-title":"The University of Chicago Legal Forum","author":"Kimberly Crenshaw","year":"1989","unstructured":"Crenshaw Kimberly. 1989. Demarginalizing the intersection of race and sex: A black feminist critique of anti-discrimination doctrine, feminist theory and anti-racist politics. In The University of Chicago Legal Forum, Vol. 140. 139."},{"key":"e_1_3_3_197_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/S18-2005"},{"key":"e_1_3_3_198_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-21251-2_7"},{"key":"e_1_3_3_199_2","first-page":"43:1\u201343:23","volume-title":"Proceedings of the 8th Innovations in Theoretical Computer Science Conference (ITCS\u201917)","author":"Kleinberg Jon M.","year":"2017","unstructured":"Jon M. Kleinberg, Sendhil Mullainathan, and Manish Raghavan. 2017. Inherent trade-offs in the fair determination of risk scores. In Proceedings of the 8th Innovations in Theoretical Computer Science Conference (ITCS\u201917). 43:1\u201343:23."},{"key":"e_1_3_3_200_2","doi-asserted-by":"publisher","DOI":"10.1145\/3461702.3462588"},{"key":"e_1_3_3_201_2","doi-asserted-by":"publisher","DOI":"10.1145\/3308558.3313443"},{"key":"e_1_3_3_202_2","doi-asserted-by":"publisher","DOI":"10.5555\/2500930"},{"key":"e_1_3_3_203_2","first-page":"4066","volume-title":"Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS\u201917)","author":"Kusner Matt J.","year":"2017","unstructured":"Matt J. Kusner, Joshua R. Loftus, Chris Russell, and Ricardo Silva. 2017. Counterfactual fairness. In Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS\u201917). 4066\u20134076."},{"key":"e_1_3_3_204_2","doi-asserted-by":"publisher","DOI":"10.1145\/3183440.3194970"},{"key":"e_1_3_3_205_2","doi-asserted-by":"publisher","DOI":"10.1145\/1276958.1277175"},{"key":"e_1_3_3_206_2","first-page":"5967","volume-title":"Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS\u201917)","author":"Lample Guillaume","year":"2017","unstructured":"Guillaume Lample, Neil Zeghidour, Nicolas Usunier, Antoine Bordes, Ludovic Denoyer, and Marc\u2019Aurelio Ranzato. 2017. Fader networks: Manipulating images by sliding attributes. In Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS\u201917). 5967\u20135976."},{"key":"e_1_3_3_207_2","article-title":"Do artifacts have politics?","author":"Langdon Winner","year":"1986","unstructured":"Winner Langdon. 1986. Do artifacts have politics? Whale Reactor (1986).","journal-title":"Whale Reactor"},{"key":"e_1_3_3_208_2","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445261"},{"key":"e_1_3_3_209_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.198.4318.722"},{"key":"e_1_3_3_210_2","article-title":"A survey on fairness in large language models","volume":"2308","author":"Li Yingji","year":"2023","unstructured":"Yingji Li, Mengnan Du, Rui Song, Xin Wang, and Ying Wang. 2023. A survey on fairness in large language models. CoRR abs\/2308.10149 (2023).","journal-title":"CoRR"},{"key":"e_1_3_3_211_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510091"},{"key":"e_1_3_3_212_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.01470"},{"key":"e_1_3_3_213_2","doi-asserted-by":"publisher","DOI":"10.1145\/3490388"},{"key":"e_1_3_3_214_2","article-title":"FairGRAPE: Fairness-aware GRAdient pruning method for face attribute classification","author":"Lin Xiaofeng","year":"2022","unstructured":"Xiaofeng Lin, Seungbae Kim, and Jungseock Joo. 2022. FairGRAPE: Fairness-aware GRAdient pruning method for face attribute classification. In Proceedings of the European Conference on Computer Vision (ECCV\u201922).","journal-title":"Proceedings of the European Conference on Computer Vision (ECCV\u201922)"},{"key":"e_1_3_3_215_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.coling-main.390"},{"key":"e_1_3_3_216_2","article-title":"Software engineering for responsible AI: An empirical study and operationalised patterns","volume":"2111","author":"Lu Qinghua","year":"2021","unstructured":"Qinghua Lu, Liming Zhu, Xiwei Xu, Jon Whittle, David Douglas, and Conrad Sanderson. 2021. Software engineering for responsible AI: An empirical study and operationalised patterns. CoRR abs\/2111.09478 (2021).","journal-title":"CoRR"},{"key":"e_1_3_3_217_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510457.3513063"},{"key":"e_1_3_3_218_2","first-page":"4765","volume-title":"Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS\u201917)","author":"Lundberg Scott M.","year":"2017","unstructured":"Scott M. Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. In Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS\u201917). 4765\u20134774."},{"key":"e_1_3_3_219_2","article-title":"Enhanced fairness testing via generating effective initial individual discriminatory instances","volume":"2209","author":"Ma Minghua","year":"2022","unstructured":"Minghua Ma, Zhao Tian, Max Hort, Federica Sarro, Hongyu Zhang, Qingwei Lin, and Dongmei Zhang. 2022. Enhanced fairness testing via generating effective initial individual discriminatory instances. CoRR abs\/2209.08321 (2022).","journal-title":"CoRR"},{"key":"e_1_3_3_220_2","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2020\/64"},{"key":"e_1_3_3_221_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICPR56361.2022.9956151"},{"key":"e_1_3_3_222_2","article-title":"Finding trends in software research","author":"Mathew George","year":"2018","unstructured":"George Mathew, Amritanshu Agrawal, and Tim Menzies. 2018. Finding trends in software research. IEEE Trans. Softw. Eng. (2018).","journal-title":"IEEE Trans. Softw. Eng."},{"key":"e_1_3_3_223_2","doi-asserted-by":"publisher","DOI":"10.1145\/3372923.3404804"},{"key":"e_1_3_3_224_2","doi-asserted-by":"publisher","DOI":"10.1145\/3372923.3404804"},{"key":"e_1_3_3_225_2","doi-asserted-by":"publisher","DOI":"10.1145\/3457607"},{"key":"e_1_3_3_226_2","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-statistics-042720-125902"},{"key":"e_1_3_3_227_2","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287574"},{"key":"e_1_3_3_228_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00136"},{"key":"e_1_3_3_229_2","unstructured":"Rebecca Moussa and Federica Sarro. 2022. Do not take it for granted: Comparing open-source libraries for software development effort estimation. Retrieved from https:\/\/arxiv.org\/abs\/2207.01705"},{"key":"e_1_3_3_230_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPRW.2019.00282"},{"key":"e_1_3_3_231_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510209"},{"key":"e_1_3_3_232_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2018.2870895"},{"key":"e_1_3_3_233_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i04.5970"},{"key":"e_1_3_3_234_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00078"},{"key":"e_1_3_3_235_2","doi-asserted-by":"publisher","DOI":"10.2307\/j.ctt1pwt9w5.11"},{"key":"e_1_3_3_236_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.gebnlp-1.17"},{"key":"e_1_3_3_237_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.naacl-main.188"},{"key":"e_1_3_3_238_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSTW55395.2022.00030"},{"key":"e_1_3_3_239_2","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511803161"},{"key":"e_1_3_3_240_2","doi-asserted-by":"publisher","DOI":"10.5555\/1953048.2078195"},{"key":"e_1_3_3_241_2","doi-asserted-by":"publisher","DOI":"10.1145\/1401890.1401959"},{"key":"e_1_3_3_242_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2022.3220713"},{"issue":"79","key":"e_1_3_3_243_2","article-title":"Search-based fairness testing for regression-based machine learning systems","volume":"27","author":"Perera Anjana","year":"2022","unstructured":"Anjana Perera, Aldeida Aleti, Chakkrit Tantithamthavorn, Jirayus Jiarpakdee, Burak Turhan, Lisa Kuhn, and Katie Walker. 2022. Search-based fairness testing for regression-based machine learning systems. Empir. Softw. Eng. 27, 79 (2022).","journal-title":"Empir. Softw. Eng."},{"key":"e_1_3_3_244_2","doi-asserted-by":"publisher","DOI":"10.1145\/3494672"},{"key":"e_1_3_3_245_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00107"},{"key":"e_1_3_3_246_2","article-title":"Fairness in rankings and recommendations: An overview","author":"Pitoura Evaggelia","year":"2021","unstructured":"Evaggelia Pitoura, Kostas Stefanidis, and Georgia Koutrika. 2021. Fairness in rankings and recommendations: An overview. VLDB J. (2021).","journal-title":"VLDB J."},{"key":"e_1_3_3_247_2","article-title":"Beneath the tip of the iceberg: Current challenges and new directions in sentiment analysis research","author":"Poria Soujanya","year":"2020","unstructured":"Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, and Rada Mihalcea. 2020. Beneath the tip of the iceberg: Current challenges and new directions in sentiment analysis research. IEEE Trans. Affect. Comput. (2020).","journal-title":"IEEE Trans. Affect. Comput."},{"key":"e_1_3_3_248_2","doi-asserted-by":"publisher","DOI":"10.1145\/3524846.3527337"},{"key":"e_1_3_3_249_2","first-page":"30211","article-title":"Are my deep learning systems fair? An empirical study of fixed-seed training","volume":"34","author":"Qian Shangshu","year":"2021","unstructured":"Shangshu Qian, Viet Hung Pham, Thibaud Lutellier, Zeou Hu, Jungwon Kim, Lin Tan, Yaoliang Yu, Jiahao Chen, and Sameena Shah. 2021. Are my deep learning systems fair? An empirical study of fixed-seed training. Adv. Neural Inf. Process. Syst. 34 (2021), 30211\u201330227.","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"e_1_3_3_250_2","article-title":"A survey on datasets for fairness-aware machine learning","volume":"2110","author":"Quy Tai Le","year":"2021","unstructured":"Tai Le Quy, Arjun Roy, Vasileios Iosifidis, and Eirini Ntoutsi. 2021. A survey on datasets for fairness-aware machine learning. CoRR abs\/2110.00530 (2021).","journal-title":"CoRR"},{"key":"e_1_3_3_251_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-99429-7_14"},{"key":"e_1_3_3_252_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.442"},{"key":"e_1_3_3_253_2","doi-asserted-by":"publisher","DOI":"10.18293\/SEKE2022-064"},{"key":"e_1_3_3_254_2","doi-asserted-by":"publisher","DOI":"10.1109\/RE57278.2023.00010"},{"key":"e_1_3_3_255_2","first-page":"738","volume-title":"Proceedings of the 20th IEEE International Conference on Machine Learning and Applications (ICMLA\u201921)","author":"Sharma Arnab","year":"2021","unstructured":"Arnab Sharma, Caglar Demir, Axel-Cyrille Ngonga Ngomo, and Heike Wehrheim. 2021. MLCHECK\u2013Property-driven testing of machine learning classifiers. In Proceedings of the 20th IEEE International Conference on Machine Learning and Applications (ICMLA\u201921). 738\u2013745."},{"key":"e_1_3_3_256_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-64881-7_16"},{"key":"e_1_3_3_257_2","volume-title":"Proceedings of the NeurIPS Workshop on Dataset Curation and Security","author":"Sharma Shanya","year":"2020","unstructured":"Shanya Sharma, Manan Dey, and Koustuv Sinha. 2020. Evaluating gender bias in natural language inference. In Proceedings of the NeurIPS Workshop on Dataset Curation and Security."},{"key":"e_1_3_3_258_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-1339"},{"key":"e_1_3_3_259_2","first-page":"9649","volume-title":"Proceedings of the 38th International Conference on Machine Learning (ICML\u201921)","author":"Si Nian","year":"2021","unstructured":"Nian Si, Karthyek Murthy, Jose H. Blanchet, and Viet Anh Nguyen. 2021. Testing group fairness via optimal transport projections. In Proceedings of the 38th International Conference on Machine Learning (ICML\u201921). 9649\u20139659."},{"key":"e_1_3_3_260_2","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3220088"},{"key":"e_1_3_3_261_2","doi-asserted-by":"publisher","DOI":"10.1038\/d41586-022-01202-3"},{"key":"e_1_3_3_262_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.emnlp-main.625"},{"key":"e_1_3_3_263_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2022.3141758"},{"key":"e_1_3_3_264_2","article-title":"Software fairness: An analysis and survey","volume":"2205","author":"Soremekun Ezekiel O.","year":"2022","unstructured":"Ezekiel O. Soremekun, Mike Papadakis, Maxime Cordy, and Yves Le Traon. 2022. Software fairness: An analysis and survey. CoRR abs\/2205.08809 (2022).","journal-title":"CoRR"},{"key":"e_1_3_3_265_2","volume-title":"Proceedings of Workshop on Applied Affect Recognition at the 26th International Conference on Pattern Recognition (ICPR\u201922)","author":"Stoychev Samuil","year":"2022","unstructured":"Samuil Stoychev and Hatice Gunes. 2022. The effect of model compression on fairness in facial expression recognition. In Proceedings of Workshop on Applied Affect Recognition at the 26th International Conference on Pattern Recognition (ICPR\u201922)."},{"key":"e_1_3_3_266_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-90870-6_5"},{"key":"e_1_3_3_267_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510080"},{"key":"e_1_3_3_268_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1159"},{"key":"e_1_3_3_269_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380420"},{"key":"e_1_3_3_270_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510206"},{"key":"e_1_3_3_271_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549169"},{"key":"e_1_3_3_272_2","doi-asserted-by":"publisher","DOI":"10.1145\/3442188.3445927"},{"key":"e_1_3_3_273_2","doi-asserted-by":"crossref","first-page":"502","DOI":"10.1145\/2020408.2020488","volume-title":"Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD\u201911)","author":"Thanh Binh Luong","year":"2011","unstructured":"Binh Luong Thanh, Salvatore Ruggieri, and Franco Turini. 2011. k-NN as an implementation of situation testing for discrimination discovery and prevention. In Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD\u201911). 502\u2013510."},{"key":"e_1_3_3_274_2","first-page":"1122","volume-title":"Proceedings of the 42nd International Conference on Software Engineering (ICSE\u201920)","author":"Tian Yuchi","year":"2020","unstructured":"Yuchi Tian, Ziyuan Zhong, Vicente Ordonez, Gail E. Kaiser, and Baishakhi Ray. 2020. Testing DNN image classifiers for confusion & bias errors. In Proceedings of the 42nd International Conference on Software Engineering (ICSE\u201920). 1122\u20131134."},{"key":"e_1_3_3_275_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510202"},{"key":"e_1_3_3_276_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2011.5995347"},{"key":"e_1_3_3_277_2","doi-asserted-by":"publisher","DOI":"10.1109\/EuroSP.2017.29"},{"key":"e_1_3_3_278_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2021.3139961"},{"key":"e_1_3_3_279_2","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238165"},{"key":"e_1_3_3_280_2","doi-asserted-by":"publisher","DOI":"10.1145\/3428253"},{"key":"e_1_3_3_281_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.2019.00046"},{"key":"e_1_3_3_282_2","doi-asserted-by":"publisher","DOI":"10.1145\/3340531.3412705"},{"key":"e_1_3_3_283_2","doi-asserted-by":"publisher","DOI":"10.1145\/3194770.3194776"},{"key":"e_1_3_3_284_2","volume-title":"Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS\u201920)","author":"Vig Jesse","year":"2020","unstructured":"Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart M. Shieber. 2020. Investigating gender bias in language models using causal mediation analysis. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS\u201920)."},{"key":"e_1_3_3_285_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-71050-9"},{"issue":"3152676","key":"e_1_3_3_286_2","first-page":"10","article-title":"The EU general data protection regulation (GDPR)","volume":"10","author":"Voigt Paul","year":"2017","unstructured":"Paul Voigt and Axel Von dem Bussche. 2017. The EU general data protection regulation (GDPR). A Practical Guide, 1st Ed., Cham: Springer International Publishing 10, 3152676 (2017), 10\u20135555.","journal-title":"A Practical Guide, 1st Ed., Cham: Springer International Publishing"},{"key":"e_1_3_3_287_2","doi-asserted-by":"publisher","DOI":"10.1145\/3611643.3616310"},{"key":"e_1_3_3_288_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58580-8_43"},{"key":"e_1_3_3_289_2","first-page":"10882","volume-title":"Proceedings of the 38th International Conference on Machine Learning (ICML\u201921)","author":"Wang Angelina","year":"2021","unstructured":"Angelina Wang and Olga Russakovsky. 2021. Directional bias amplification. In Proceedings of the 38th International Conference on Machine Learning (ICML\u201921). 10882\u201310893."},{"key":"e_1_3_3_290_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510165"},{"key":"e_1_3_3_291_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.acl-long.184"},{"key":"e_1_3_3_292_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00541"},{"key":"e_1_3_3_293_2","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409761"},{"key":"e_1_3_3_294_2","first-page":"8780","volume-title":"Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS\u201919)","author":"Wick Michael L.","year":"2019","unstructured":"Michael L. Wick, Swetasudha Panda, and Jean-Baptiste Tristan. 2019. Unlocking fairness: A trade-off revisited. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS\u201919). 8780\u20138789."},{"key":"e_1_3_3_295_2","doi-asserted-by":"publisher","DOI":"10.1145\/3597926.3598099"},{"key":"e_1_3_3_296_2","doi-asserted-by":"publisher","DOI":"10.1109\/TrustCom50675.2020.00029"},{"key":"e_1_3_3_297_2","article-title":"Can model compression improve NLP fairness","volume":"2201","author":"Xu Guangxuan","year":"2022","unstructured":"Guangxuan Xu and Qingyuan Hu. 2022. Can model compression improve NLP fairness. CoRR abs\/2201.08542 (2022).","journal-title":"CoRR"},{"key":"e_1_3_3_298_2","doi-asserted-by":"publisher","DOI":"10.1145\/3085504.3085526"},{"key":"e_1_3_3_299_2","doi-asserted-by":"publisher","DOI":"10.1145\/3514094.3534153"},{"key":"e_1_3_3_300_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3473117"},{"key":"e_1_3_3_301_2","doi-asserted-by":"publisher","DOI":"10.1145\/3524491.3527306"},{"key":"e_1_3_3_302_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11432-014-5241-2"},{"key":"e_1_3_3_303_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2018.2809496"},{"key":"e_1_3_3_304_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00129"},{"key":"e_1_3_3_305_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2019.2962027"},{"key":"e_1_3_3_306_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11390-018-1864-x"},{"key":"e_1_3_3_307_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-39931-7_9"},{"key":"e_1_3_3_308_2","first-page":"2718","volume-title":"Proceedings of the 25th International Joint Conference on Artificial Intelligence (IJCAI\u201916)","author":"Zhang Lu","year":"2016","unstructured":"Lu Zhang, Yongkai Wu, and Xintao Wu. 2016. Situation testing-based discrimination discovery: A causal inference approach. In Proceedings of the 25th International Joint Conference on Artificial Intelligence (IJCAI\u201916). 2718\u20132724."},{"key":"e_1_3_3_309_2","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2017\/549"},{"key":"e_1_3_3_310_2","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2018.2872988"},{"key":"e_1_3_3_311_2","doi-asserted-by":"publisher","DOI":"10.1145\/3460319.3464820"},{"key":"e_1_3_3_312_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549103"},{"key":"e_1_3_3_313_2","doi-asserted-by":"publisher","DOI":"10.1145\/3591869"},{"key":"e_1_3_3_314_2","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380331"},{"key":"e_1_3_3_315_2","article-title":"Fairness testing of deep image classification with adequacy metrics","volume":"2111","author":"Zhang Peixin","year":"2021","unstructured":"Peixin Zhang, Jingyi Wang, Jun Sun, and Xinyu Wang. 2021. Fairness testing of deep image classification with adequacy metrics. CoRR abs\/2111.08856 (2021).","journal-title":"CoRR"},{"key":"e_1_3_3_316_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2021.3101478"},{"key":"e_1_3_3_317_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D17-1323"},{"key":"e_1_3_3_318_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-21251-2_3"},{"key":"e_1_3_3_319_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510123"},{"key":"e_1_3_3_320_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.244"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3652155","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3652155","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T20:03:14Z","timestamp":1750276994000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3652155"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6,4]]},"references-count":319,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2024,6,30]]}},"alternative-id":["10.1145\/3652155"],"URL":"https:\/\/doi.org\/10.1145\/3652155","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,6,4]]},"assertion":[{"value":"2023-07-19","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-03-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-06-04","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}