{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,14]],"date-time":"2026-04-14T00:40:35Z","timestamp":1776127235228,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":69,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,8,18]],"date-time":"2021-08-18T00:00:00Z","timestamp":1629244800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100000781","name":"European Research Council","doi-asserted-by":"publisher","award":["741278"],"award-info":[{"award-number":["741278"]}],"id":[{"id":"10.13039\/501100000781","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,8,20]]},"DOI":"10.1145\/3468264.3468565","type":"proceedings-article","created":{"date-parts":[[2021,8,19]],"date-time":"2021-08-19T01:40:03Z","timestamp":1629337203000},"page":"994-1006","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":64,"title":["Fairea: a model behaviour mutation approach to benchmarking bias mitigation methods"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-8684-5909","authenticated-orcid":false,"given":"Max","family":"Hort","sequence":"first","affiliation":[{"name":"University College London, UK"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0481-7264","authenticated-orcid":false,"given":"Jie M.","family":"Zhang","sequence":"additional","affiliation":[{"name":"University College London, UK"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9146-442X","authenticated-orcid":false,"given":"Federica","family":"Sarro","sequence":"additional","affiliation":[{"name":"University College London, UK"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5864-4488","authenticated-orcid":false,"given":"Mark","family":"Harman","sequence":"additional","affiliation":[{"name":"University College London, UK"}]}],"member":"320","published-online":{"date-parts":[[2021,8,18]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338937"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3264590"},{"key":"e_1_3_2_1_3_1","unstructured":"Julia Angwin Jeff Larson Surya Mattu and Lauren Kirchner. 2016. Machine bias. ProPublica. See https:\/\/www. propublica. org\/article\/machine-bias-risk-assessments-in-criminal-sentencing.  Julia Angwin Jeff Larson Surya Mattu and Lauren Kirchner. 2016. Machine bias. ProPublica. See https:\/\/www. propublica. org\/article\/machine-bias-risk-assessments-in-criminal-sentencing."},{"key":"e_1_3_2_1_4_1","unstructured":"Rachel KE Bellamy Kuntal Dey Michael Hind Samuel C Hoffman Stephanie Houde Kalapriya Kannan Pranay Lohia Jacquelyn Martino Sameep Mehta and Aleksandra Mojsilovic. 2018. AI Fairness 360: An extensible toolkit for detecting understanding and mitigating unwanted algorithmic bias. arXiv preprint arXiv:1810.01943.  Rachel KE Bellamy Kuntal Dey Michael Hind Samuel C Hoffman Stephanie Houde Kalapriya Kannan Pranay Lohia Jacquelyn Martino Sameep Mehta and Aleksandra Mojsilovic. 2018. AI Fairness 360: An extensible toolkit for detecting understanding and mitigating unwanted algorithmic bias. arXiv preprint arXiv:1810.01943."},{"key":"e_1_3_2_1_5_1","volume-title":"FAT-ML Workshop.","author":"Berk R.","unstructured":"R. Berk , H. Heidari , S. Jabbari , M. Joseph , M. Kearns , J. Morgenstern , S. Neel , and A. Roth . 2017. A Convex Framework for Fair Regression . FAT-ML Workshop. R. Berk, H. Heidari, S. Jabbari, M. Joseph, M. Kearns, J. Morgenstern, S. Neel, and A. Roth. 2017. A Convex Framework for Fair Regression. FAT-ML Workshop."},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1177\/0049124118782533"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1126\/science.187.4175.398"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409704"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"crossref","unstructured":"Sumon Biswas and Hridesh Rajan. 2021. Fair Preprocessing: Towards Understanding Compositional Fairness of Data Transformers in Machine Learning Pipeline. In The ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE) (to appear).  Sumon Biswas and Hridesh Rajan. 2021. Fair Preprocessing: Towards Understanding Compositional Fairness of Data Transformers in Machine Learning Pipeline. In The ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE) (to appear).","DOI":"10.1145\/3468264.3468536"},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3264838"},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1093\/iwc\/iwv046"},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDMW.2009.83"},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2013.114"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10618-010-0190-x"},{"key":"e_1_3_2_1_15_1","volume-title":"Karthikeyan Natesan Ramamurthy, and Kush R Varshney","author":"Calmon Flavio","year":"2017","unstructured":"Flavio Calmon , Dennis Wei , Bhanukiran Vinzamuri , Karthikeyan Natesan Ramamurthy, and Kush R Varshney . 2017 . Optimized pre-processing for discrimination prevention. In Advances in Neural Information Processing Systems . 3992\u20134001. Flavio Calmon, Dennis Wei, Bhanukiran Vinzamuri, Karthikeyan Natesan Ramamurthy, and Kush R Varshney. 2017. Optimized pre-processing for discrimination prevention. In Advances in Neural Information Processing Systems. 3992\u20134001."},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287586"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409697"},{"key":"e_1_3_2_1_18_1","unstructured":"Joymallya Chakraborty Tianpei Xia Fahmid M. Fahid and Tim Menzies. 2019. Software Engineering for Fairness: A Case Study with Hyperparameter Optimization. arxiv:1905.05786.  Joymallya Chakraborty Tianpei Xia Fahmid M. Fahid and Tim Menzies. 2019. Software Engineering for Fairness: A Case Study with Hyperparameter Optimization. arxiv:1905.05786."},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287594"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1089\/big.2016.0047"},{"key":"e_1_3_2_1_21_1","unstructured":"Sam Corbett-Davies and Sharad Goel. 2018. The measure and mismeasure of fairness: A critical review of fair machine learning. arXiv preprint arXiv:1808.00023.  Sam Corbett-Davies and Sharad Goel. 2018. The measure and mismeasure of fairness: A critical review of fair machine learning. arXiv preprint arXiv:1808.00023."},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3097983.3098095"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/2090236.2090255"},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2783258.2783311"},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00766-009-0075-y"},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287589"},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3106237.3106277"},{"key":"e_1_3_2_1_28_1","unstructured":"Moritz Hardt Eric Price and Nati Srebro. 2016. Equality of opportunity in supervised learning. In Advances in neural information processing systems. 3315\u20133323.  Moritz Hardt Eric Price and Nati Srebro. 2016. Equality of opportunity in supervised learning. In Advances in neural information processing systems. 3315\u20133323."},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372831"},{"key":"e_1_3_2_1_30_1","unstructured":"Dr. Hans Hofmann. [n.d.]. Statlog (german credit data) data set. http:\/\/ archive.ics.uci.edu\/ml\/datasets\/statlog+(german+credit+data).  Dr. Hans Hofmann. [n.d.]. Statlog (german credit data) data set. http:\/\/ archive.ics.uci.edu\/ml\/datasets\/statlog+(german+credit+data)."},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/RE.2019.00050"},{"key":"e_1_3_2_1_32_1","unstructured":"Max Hort Jie Zhang Federica Sarro and Mark Harman. [n.d.]. On-line Appendix to the paper Fairea: A Model Behaviour Mutation Approach to Benchmarking Bias Mitigation Methods. https:\/\/github.com\/maxhort\/Fairea\/  Max Hort Jie Zhang Federica Sarro and Mark Harman. [n.d.]. On-line Appendix to the paper Fairea: A Model Behaviour Mutation Approach to Benchmarking Bias Mitigation Methods. https:\/\/github.com\/maxhort\/Fairea\/"},{"key":"e_1_3_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST46399.2020.00018"},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2010.62"},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/IC4.2009.4909197"},{"key":"e_1_3_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10115-011-0463-8"},{"key":"e_1_3_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2010.50"},{"key":"e_1_3_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2012.45"},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ins.2017.09.064"},{"key":"e_1_3_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-33486-3_3"},{"key":"e_1_3_2_1_41_1","volume-title":"Proceedings of Machine Learning Research","volume":"80","author":"Kearns Michael","year":"2018","unstructured":"Michael Kearns , Seth Neel , Aaron Roth , and Zhiwei Steven Wu . 2018 . Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness . Proceedings of Machine Learning Research , Vol. 80 . PMLR, Stockholmsm\u00e4ssan, Stockholm Sweden. 2564\u20132572. http:\/\/proceedings.mlr.press\/v80\/kearns18a.html Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. 2018. Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness. Proceedings of Machine Learning Research, Vol. 80. PMLR, Stockholmsm\u00e4ssan, Stockholm Sweden. 2564\u20132572. http:\/\/proceedings.mlr.press\/v80\/kearns18a.html"},{"key":"e_1_3_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287592"},{"key":"e_1_3_2_1_43_1","unstructured":"Jon Kleinberg Sendhil Mullainathan and Manish Raghavan. 2016. Inherent trade-offs in the fair determination of risk scores. arXiv preprint arXiv:1609.05807.  Jon Kleinberg Sendhil Mullainathan and Manish Raghavan. 2016. Inherent trade-offs in the fair determination of risk scores. arXiv preprint arXiv:1609.05807."},{"key":"e_1_3_2_1_44_1","unstructured":"Ron Kohav. [n.d.]. Adult data set. http:\/\/archive.ics.uci.edu\/ml\/ datasets\/adult.  Ron Kohav. [n.d.]. Adult data set. http:\/\/archive.ics.uci.edu\/ml\/ datasets\/adult."},{"key":"e_1_3_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377812.3382146"},{"key":"e_1_3_2_1_46_1","unstructured":"Ninareh Mehrabi Fred Morstatter Nripsuta Saxena Kristina Lerman and Aram Galstyan. 2019. A survey on bias and fairness in machine learning. arXiv preprint arXiv:1908.09635.  Ninareh Mehrabi Fred Morstatter Nripsuta Saxena Kristina Lerman and Aram Galstyan. 2019. A survey on bias and fairness in machine learning. arXiv preprint arXiv:1908.09635."},{"key":"e_1_3_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/2390231.2390245"},{"key":"e_1_3_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1111\/1475-3995.00375"},{"key":"e_1_3_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1016\/bs.adcom.2018.03.015"},{"key":"e_1_3_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/1401890.1401959"},{"key":"e_1_3_2_1_51_1","unstructured":"Geoff Pleiss Manish Raghavan Felix Wu Jon Kleinberg and Kilian Q Weinberger. 2017. On fairness and calibration. In Advances in Neural Information Processing Systems. 5680\u20135689.  Geoff Pleiss Manish Raghavan Felix Wu Jon Kleinberg and Kilian Q Weinberger. 2017. On fairness and calibration. In Advances in Neural Information Processing Systems. 5680\u20135689."},{"key":"e_1_3_2_1_52_1","unstructured":"propublica. [n.d.]. data for the propublica story \u2018machine bias\u2019. https: \/\/github.com\/propublica\/compas-analysis\/.  propublica. [n.d.]. data for the propublica story \u2018machine bias\u2019. https: \/\/github.com\/propublica\/compas-analysis\/."},{"key":"e_1_3_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1017\/S0269888913000039"},{"key":"e_1_3_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2016.2532875"},{"key":"e_1_3_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3220046"},{"key":"e_1_3_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380420"},{"key":"e_1_3_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1109\/EuroSP.2017.29"},{"key":"e_1_3_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/3238147.3238165"},{"key":"e_1_3_2_1_59_1","unstructured":"Michael Wick and Jean-Baptiste Tristan. 2019. Unlocking Fairness: a Trade-off Revisited. In Advances in Neural Information Processing Systems. 8783\u20138792.  Michael Wick and Jean-Baptiste Tristan. 2019. Unlocking Fairness: a Trade-off Revisited. In Advances in Neural Information Processing Systems. 8783\u20138792."},{"key":"e_1_3_2_1_60_1","volume-title":"Manuel Gomez Rogriguez, and Krishna P Gummadi","author":"Zafar Muhammad Bilal","year":"2017","unstructured":"Muhammad Bilal Zafar , Isabel Valera , Manuel Gomez Rogriguez, and Krishna P Gummadi . 2017 . Fairness constraints: Mechanisms for fair classification. In Artificial Intelligence and Statistics . 962\u2013970. Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rogriguez, and Krishna P Gummadi. 2017. Fairness constraints: Mechanisms for fair classification. In Artificial Intelligence and Statistics. 962\u2013970."},{"key":"e_1_3_2_1_61_1","volume-title":"International Conference on Machine Learning. 325\u2013333","author":"Zemel Rich","year":"2013","unstructured":"Rich Zemel , Yu Wu , Kevin Swersky , Toni Pitassi , and Cynthia Dwork . 2013 . Learning fair representations . In International Conference on Machine Learning. 325\u2013333 . Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. 2013. Learning fair representations. In International Conference on Machine Learning. 325\u2013333."},{"key":"e_1_3_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/3278721.3278779"},{"key":"e_1_3_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/2642937.2642994"},{"key":"e_1_3_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00129"},{"key":"e_1_3_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/2931037.2931038"},{"key":"e_1_3_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1109\/tse.2019.2962027"},{"key":"e_1_3_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380331"},{"key":"e_1_3_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1521"},{"key":"e_1_3_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2011.72"}],"event":{"name":"ESEC\/FSE '21: 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering","location":"Athens Greece","acronym":"ESEC\/FSE '21","sponsor":["SIGSOFT ACM Special Interest Group on Software Engineering"]},"container-title":["Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3468264.3468565","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3468264.3468565","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T21:24:50Z","timestamp":1750195490000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3468264.3468565"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,8,18]]},"references-count":69,"alternative-id":["10.1145\/3468264.3468565","10.1145\/3468264"],"URL":"https:\/\/doi.org\/10.1145\/3468264.3468565","relation":{},"subject":[],"published":{"date-parts":[[2021,8,18]]},"assertion":[{"value":"2021-08-18","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}