{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,10]],"date-time":"2026-07-10T02:52:40Z","timestamp":1783651960968,"version":"3.55.0"},"publisher-location":"New York, NY, USA","reference-count":47,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,5,6]],"date-time":"2021-05-06T00:00:00Z","timestamp":1620259200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,5,6]]},"DOI":"10.1145\/3411764.3445261","type":"proceedings-article","created":{"date-parts":[[2021,5,8]],"date-time":"2021-05-08T04:26:17Z","timestamp":1620447977000},"page":"1-13","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":106,"title":["The Landscape and Gaps in Open Source Fairness Toolkits"],"prefix":"10.1145","author":[{"given":"Michelle Seng Ah","family":"Lee","sequence":"first","affiliation":[{"name":"Department of Computer Science & Technology, University of Cambridge, Cambridge, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jat","family":"Singh","sequence":"additional","affiliation":[{"name":"Department of Computer Science & Technology, University of Cambridge, Cambridge, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2021,5,7]]},"reference":[{"key":"e_1_3_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/SP.2017.52"},{"key":"e_1_3_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702509"},{"key":"e_1_3_2_2_3_1","doi-asserted-by":"crossref","unstructured":"Aaron Bangor Philip\u00a0T Kortum and James\u00a0T Miller. 2008. An empirical evaluation of the system usability scale. Intl. Journal of Human\u2013Computer Interaction 24 6(2008) 574\u2013594.  Aaron Bangor Philip\u00a0T Kortum and James\u00a0T Miller. 2008. An empirical evaluation of the system usability scale. Intl. Journal of Human\u2013Computer Interaction 24 6(2008) 574\u2013594.","DOI":"10.1080\/10447310802205776"},{"key":"e_1_3_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1147\/JRD.2019.2942287"},{"key":"e_1_3_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372860"},{"key":"e_1_3_2_2_6_1","unstructured":"Flavio Calmon Dennis Wei Bhanukiran Vinzamuri Karthikeyan\u00a0Natesan Ramamurthy and Kush\u00a0R Varshney. 2017. Optimized pre-processing for discrimination prevention. In Advances in Neural Information Processing Systems. 3992\u20134001.  Flavio Calmon Dennis Wei Bhanukiran Vinzamuri Karthikeyan\u00a0Natesan Ramamurthy and Kush\u00a0R Varshney. 2017. Optimized pre-processing for discrimination prevention. In Advances in Neural Information Processing Systems. 3992\u20134001."},{"key":"e_1_3_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3172944.3172950"},{"key":"e_1_3_2_2_8_1","unstructured":"Aisling\u00a0Ni Chonaire and Jannna\u00a0Ter Meer. 2020. The perception of fairness of algorithms and proxy information in financial services: A report for the Centre for Data Ethics and Innovation. The Behavioural Insights Team(2020).  Aisling\u00a0Ni Chonaire and Jannna\u00a0Ter Meer. 2020. The perception of fairness of algorithms and proxy information in financial services: A report for the Centre for Data Ethics and Innovation. The Behavioural Insights Team(2020)."},{"key":"e_1_3_2_2_9_1","unstructured":"Kate Crawford. 2017. Artificial intelligence with very real biases. The Wall Street Journal. Retrieved from https:\/\/www. wsj. com\/articles\/artificial-intelligencewith-very-real-biases-1508252717(2017).  Kate Crawford. 2017. Artificial intelligence with very real biases. The Wall Street Journal. Retrieved from https:\/\/www. wsj. com\/articles\/artificial-intelligencewith-very-real-biases-1508252717(2017)."},{"key":"e_1_3_2_2_10_1","volume-title":"Amazon scraps secret AI recruiting tool that showed bias against women","author":"Dastin Jeffrey","year":"2018","unstructured":"Jeffrey Dastin . 2018. Amazon scraps secret AI recruiting tool that showed bias against women . Reuters , October 2018 . Jeffrey Dastin. 2018. Amazon scraps secret AI recruiting tool that showed bias against women. Reuters, October 2018."},{"key":"e_1_3_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2090236.2090255"},{"key":"e_1_3_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/2783258.2783311"},{"key":"e_1_3_2_2_13_1","volume-title":"A computer program used for bail and sentencing decisions was labeled biased against blacks. It\u2019s actually not that clear. The Washington Post","author":"Feller Avi","year":"2016","unstructured":"Avi Feller , Emma Pierson , Sam Corbett-Davies , and Sharad Goel . 2016. A computer program used for bail and sentencing decisions was labeled biased against blacks. It\u2019s actually not that clear. The Washington Post ( 2016 ). Avi Feller, Emma Pierson, Sam Corbett-Davies, and Sharad Goel. 2016. A computer program used for bail and sentencing decisions was labeled biased against blacks. It\u2019s actually not that clear. The Washington Post (2016)."},{"key":"e_1_3_2_2_14_1","unstructured":"Moritz Hardt Eric Price and Nati Srebro. 2016. Equality of opportunity in supervised learning. In Advances in neural information processing systems. 3315\u20133323.  Moritz Hardt Eric Price and Nati Srebro. 2016. Equality of opportunity in supervised learning. In Advances in neural information processing systems. 3315\u20133323."},{"key":"e_1_3_2_2_15_1","volume-title":"Denmark introduces mandatory legislation for AI and data ethics","author":"Hauch Rasmus","year":"2021","unstructured":"Rasmus Hauch . 2020. Denmark introduces mandatory legislation for AI and data ethics . 2021 .ai (2020). https:\/\/2021.ai\/denmark-introduces-mandatory-legislation-ai-data-ethics\/ Rasmus Hauch. 2020. Denmark introduces mandatory legislation for AI and data ethics. 2021.ai (2020). https:\/\/2021.ai\/denmark-introduces-mandatory-legislation-ai-data-ethics\/"},{"key":"e_1_3_2_2_16_1","volume-title":"Comment: Snowball versus respondent-driven sampling. Sociological methodology 41, 1","author":"Heckathorn D","year":"2011","unstructured":"Douglas\u00a0 D Heckathorn . 2011 . Comment: Snowball versus respondent-driven sampling. Sociological methodology 41, 1 (2011), 355\u2013366. Douglas\u00a0D Heckathorn. 2011. Comment: Snowball versus respondent-driven sampling. Sociological methodology 41, 1 (2011), 355\u2013366."},{"key":"e_1_3_2_2_17_1","doi-asserted-by":"crossref","unstructured":"Hoda Heidari Michele Loi Krishna\u00a0P Gummadi and Andreas Krause. 2018. A moral framework for understanding of fair ml through economic models of equality of opportunity. arXiv preprint arXiv:1809.03400(2018).  Hoda Heidari Michele Loi Krishna\u00a0P Gummadi and Andreas Krause. 2018. A moral framework for understanding of fair ml through economic models of equality of opportunity. arXiv preprint arXiv:1809.03400(2018).","DOI":"10.1145\/3287560.3287584"},{"key":"e_1_3_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300830"},{"key":"e_1_3_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287600"},{"key":"e_1_3_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-019-0088-2"},{"key":"e_1_3_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2012.45"},{"key":"e_1_3_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-33486-3_3"},{"key":"e_1_3_2_2_23_1","volume-title":"International Conference on Machine Learning. PMLR, 2564\u20132572","author":"Kearns Michael","year":"2018","unstructured":"Michael Kearns , Seth Neel , Aaron Roth , and Zhiwei\u00a0Steven Wu . 2018 . Preventing fairness gerrymandering: Auditing and learning for subgroup fairness . In International Conference on Machine Learning. PMLR, 2564\u20132572 . Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei\u00a0Steven Wu. 2018. Preventing fairness gerrymandering: Auditing and learning for subgroup fairness. In International Conference on Machine Learning. PMLR, 2564\u20132572."},{"key":"e_1_3_2_2_24_1","unstructured":"Jon Kleinberg Sendhil Mullainathan and Manish Raghavan. 2016. Inherent trade-offs in the fair determination of risk scores. arXiv preprint arXiv:1609.05807(2016).  Jon Kleinberg Sendhil Mullainathan and Manish Raghavan. 2016. Inherent trade-offs in the fair determination of risk scores. arXiv preprint arXiv:1609.05807(2016)."},{"key":"e_1_3_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2678025.2701399"},{"key":"e_1_3_2_2_26_1","volume-title":"Article arXiv:1703.06856 (March","author":"Kusner J.","year":"2017","unstructured":"Matt\u00a0 J. Kusner , Joshua\u00a0 R. Loftus , Chris Russell , and Ricardo Silva . 2017. Counterfactual Fairness . arXiv e-prints , Article arXiv:1703.06856 (March 2017 ), arXiv:1703.06856\u00a0pages. arxiv:1703.06856\u00a0[stat.ML] Matt\u00a0J. Kusner, Joshua\u00a0R. Loftus, Chris Russell, and Ricardo Silva. 2017. Counterfactual Fairness. arXiv e-prints, Article arXiv:1703.06856 (March 2017), arXiv:1703.06856\u00a0pages. arxiv:1703.06856\u00a0[stat.ML]"},{"key":"e_1_3_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998230"},{"key":"e_1_3_2_2_28_1","volume-title":"From fairness metrics to key ethics indicators (KEIs): a context-aware approach to algorithmic ethics in an unequal society. Available on SSRN","author":"Seng\u00a0Ah Lee Michelle","year":"2020","unstructured":"Michelle Seng\u00a0Ah Lee , Luciano Floridi , and Jatinder Singh . 2020. From fairness metrics to key ethics indicators (KEIs): a context-aware approach to algorithmic ethics in an unequal society. Available on SSRN ( 2020 ). https:\/\/papers.ssrn.com\/sol3\/papers.cfm?abstract_id=3679975 Michelle Seng\u00a0Ah Lee, Luciano Floridi, and Jatinder Singh. 2020. From fairness metrics to key ethics indicators (KEIs): a context-aware approach to algorithmic ethics in an unequal society. Available on SSRN (2020). https:\/\/papers.ssrn.com\/sol3\/papers.cfm?abstract_id=3679975"},{"key":"e_1_3_2_2_29_1","unstructured":"Microsoft and contributors. 2019. Fairlearn. https:\/\/fairlearn.github.io\/  Microsoft and contributors. 2019. Fairlearn. https:\/\/fairlearn.github.io\/"},{"key":"e_1_3_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2896587"},{"key":"e_1_3_2_2_31_1","unstructured":"Arvind Narayanan. 2018. Tutorial: 21 Definitions of Fairness and their Politics. YouTube. https:\/\/www.youtube.com\/watch?v=jIXIuYdnyyk  Arvind Narayanan. 2018. Tutorial: 21 Definitions of Fairness and their Politics. YouTube. https:\/\/www.youtube.com\/watch?v=jIXIuYdnyyk"},{"key":"e_1_3_2_2_32_1","unstructured":"Geoff Pleiss Manish Raghavan Felix Wu Jon Kleinberg and Kilian\u00a0Q Weinberger. 2017. On fairness and calibration. In Advances in Neural Information Processing Systems. 5680\u20135689.  Geoff Pleiss Manish Raghavan Felix Wu Jon Kleinberg and Kilian\u00a0Q Weinberger. 2017. On fairness and calibration. In Advances in Neural Information Processing Systems. 5680\u20135689."},{"key":"e_1_3_2_2_33_1","volume-title":"Aequitas: A bias and fairness audit toolkit. arXiv preprint arXiv:1811.05577(2018).","author":"Saleiro Pedro","year":"2018","unstructured":"Pedro Saleiro , Benedict Kuester , Loren Hinkson , Jesse London , Abby Stevens , Ari Anisfeld , Kit\u00a0 T Rodolfa , and Rayid Ghani . 2018 . Aequitas: A bias and fairness audit toolkit. arXiv preprint arXiv:1811.05577(2018). Pedro Saleiro, Benedict Kuester, Loren Hinkson, Jesse London, Abby Stevens, Ari Anisfeld, Kit\u00a0T Rodolfa, and Rayid Ghani. 2018. Aequitas: A bias and fairness audit toolkit. arXiv preprint arXiv:1811.05577(2018)."},{"key":"e_1_3_2_2_34_1","unstructured":"Kacper Sokol Raul Santos-Rodriguez and Peter Flach. 2019. FAT Forensics: A Python toolbox for algorithmic fairness accountability and transparency. arXiv preprint arXiv:1909.05167(2019).  Kacper Sokol Raul Santos-Rodriguez and Peter Flach. 2019. FAT Forensics: A Python toolbox for algorithmic fairness accountability and transparency. arXiv preprint arXiv:1909.05167(2019)."},{"key":"e_1_3_2_2_35_1","unstructured":"Aaron Springer Jean Garcia-Gathright and Henriette Cramer. 2018. Assessing and Addressing Algorithmic Bias-But Before We Get There.... In AAAI Spring Symposia.  Aaron Springer Jean Garcia-Gathright and Henriette Cramer. 2018. Assessing and Addressing Algorithmic Bias-But Before We Get There.... In AAAI Spring Symposia."},{"key":"e_1_3_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3292500.3330664"},{"key":"e_1_3_2_2_37_1","volume-title":"AI Barometer Report","author":"Taylor Roger","year":"2020","unstructured":"Roger Taylor . 2020. AI Barometer Report . Centre for Data Ethics and Innovation( 2020 ). https:\/\/assets.publishing.service.gov.uk\/government\/uploads\/system\/uploads\/attachment_data\/file\/894170\/CDEI_AI_Barometer.pdf Roger Taylor. 2020. AI Barometer Report. Centre for Data Ethics and Innovation(2020). https:\/\/assets.publishing.service.gov.uk\/government\/uploads\/system\/uploads\/attachment_data\/file\/894170\/CDEI_AI_Barometer.pdf"},{"key":"e_1_3_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174014"},{"key":"e_1_3_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3194770.3194776"},{"key":"e_1_3_2_2_40_1","volume-title":"Apple card investigated after gender discrimination complaints. The New York Times","author":"Vigdor Neil","year":"2019","unstructured":"Neil Vigdor . 2019. Apple card investigated after gender discrimination complaints. The New York Times ( 2019 ). Neil Vigdor. 2019. Apple card investigated after gender discrimination complaints. The New York Times (2019)."},{"key":"e_1_3_2_2_41_1","unstructured":"Matthijs Vincent and ManyOthers. 2019. scikit-fairness. https:\/\/github.com\/koaning\/scikit-fairness  Matthijs Vincent and ManyOthers. 2019. scikit-fairness. https:\/\/github.com\/koaning\/scikit-fairness"},{"key":"e_1_3_2_2_42_1","volume-title":"Why fairness cannot be automated: Bridging the gap between EU non-discrimination law and AI. Available at SSRN","author":"Wachter Sandra","year":"2020","unstructured":"Sandra Wachter , Brent Mittelstadt , and Chris Russell . 2020. Why fairness cannot be automated: Bridging the gap between EU non-discrimination law and AI. Available at SSRN ( 2020 ). Sandra Wachter, Brent Mittelstadt, and Chris Russell. 2020. Why fairness cannot be automated: Bridging the gap between EU non-discrimination law and AI. Available at SSRN (2020)."},{"key":"e_1_3_2_2_43_1","volume-title":"The growth of open source software in organizations. Publication Report","author":"Walli Stephen","year":"2005","unstructured":"Stephen Walli , Dave Gynn , and Bruno\u00a0von Rotz . 2005. The growth of open source software in organizations. Publication Report . Optaros Inc( 2005 ). Stephen Walli, Dave Gynn, and Bruno\u00a0von Rotz. 2005. The growth of open source software in organizations. Publication Report. Optaros Inc(2005)."},{"key":"e_1_3_2_2_44_1","volume-title":"The what-if tool: Interactive probing of machine learning models","author":"Wexler James","year":"2019","unstructured":"James Wexler , Mahima Pushkarna , Tolga Bolukbasi , Martin Wattenberg , Fernanda Vi\u00e9gas , and Jimbo Wilson . 2019. The what-if tool: Interactive probing of machine learning models . IEEE transactions on visualization and computer graphics 26, 1( 2019 ), 56\u201365. James Wexler, Mahima Pushkarna, Tolga Bolukbasi, Martin Wattenberg, Fernanda Vi\u00e9gas, and Jimbo Wilson. 2019. The what-if tool: Interactive probing of machine learning models. IEEE transactions on visualization and computer graphics 26, 1(2019), 56\u201365."},{"key":"e_1_3_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174230"},{"key":"e_1_3_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3278721.3278779"},{"key":"e_1_3_2_2_47_1","first-page":"255","article-title":"What makes APIs difficult to use","volume":"8","author":"Zibran Minhaz","year":"2008","unstructured":"Minhaz Zibran . 2008 . What makes APIs difficult to use . International Journal of Computer Science and Network Security (IJCSNS) 8 , 4(2008), 255 \u2013 261 . Minhaz Zibran. 2008. What makes APIs difficult to use. International Journal of Computer Science and Network Security (IJCSNS) 8, 4(2008), 255\u2013261.","journal-title":"International Journal of Computer Science and Network Security (IJCSNS)"}],"event":{"name":"CHI '21: CHI Conference on Human Factors in Computing Systems","location":"Yokohama Japan","acronym":"CHI '21","sponsor":["SIGCHI ACM Special Interest Group on Computer-Human Interaction"]},"container-title":["Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3411764.3445261","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3411764.3445261","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T21:28:43Z","timestamp":1750195723000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3411764.3445261"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,5,6]]},"references-count":47,"alternative-id":["10.1145\/3411764.3445261","10.1145\/3411764"],"URL":"https:\/\/doi.org\/10.1145\/3411764.3445261","relation":{},"subject":[],"published":{"date-parts":[[2021,5,6]]},"assertion":[{"value":"2021-05-07","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}