{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T20:55:24Z","timestamp":1776113724292,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":52,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,5,6]],"date-time":"2021-05-06T00:00:00Z","timestamp":1620259200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,5,6]]},"DOI":"10.1145\/3411764.3445604","type":"proceedings-article","created":{"date-parts":[[2021,5,8]],"date-time":"2021-05-08T03:50:37Z","timestamp":1620445837000},"page":"1-13","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":47,"title":["Towards Fairness in Practice: A Practitioner-Oriented Rubric for Evaluating Fair ML Toolkits"],"prefix":"10.1145","author":[{"given":"Brianna","family":"Richardson","sequence":"first","affiliation":[{"name":"Spotify, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jean","family":"Garcia-Gathright","sequence":"additional","affiliation":[{"name":"Spotify, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Samuel F.","family":"Way","sequence":"additional","affiliation":[{"name":"Spotify, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jennifer","family":"Thom","sequence":"additional","affiliation":[{"name":"Spotify, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Henriette","family":"Cramer","sequence":"additional","affiliation":[{"name":"Spotify, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2021,5,7]]},"reference":[{"key":"e_1_3_2_2_1_1","unstructured":"2020. TensorFlow Extended (TFX) | ML Production Pipelines. https:\/\/www.tensorflow.org\/tfx  2020. TensorFlow Extended (TFX) | ML Production Pipelines. https:\/\/www.tensorflow.org\/tfx"},{"key":"e_1_3_2_2_2_1","unstructured":"2020. Tensorflow\u2019s Fairness Evaluation and Visualization Toolkit. https:\/\/github.com\/tensorflow\/fairness-indicators  2020. Tensorflow\u2019s Fairness Evaluation and Visualization Toolkit. https:\/\/github.com\/tensorflow\/fairness-indicators"},{"key":"e_1_3_2_2_3_1","unstructured":"ACM. 2020. ACM FAccT. https:\/\/facctconference.org\/  ACM. 2020. ACM FAccT. https:\/\/facctconference.org\/"},{"key":"e_1_3_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338937"},{"key":"e_1_3_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1080\/13648470.2013.876355"},{"key":"e_1_3_2_2_6_1","unstructured":"Vijay Arya Rachel K\u00a0E Bellamy Yu Chen Amit Dhurandhar Michael Hind Samuel\u00a0C Hoffman Stephanie Houde Q\u00a0Vera Liao Ronny Luss Aleksandra Mojsilovi\u0107 Sami Mourad Pablo Pedemonte Ramya Raghavendra John Richards Prasanna Sattigeri Karthikeyan Shanmugam Moninder Singh Kush\u00a0R Varshney Dennis Wei and Yunfeng Zhang. [n.d.]. One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques. Technical Report. arxiv:1909.03012v2http:\/\/aix360.  Vijay Arya Rachel K\u00a0E Bellamy Yu Chen Amit Dhurandhar Michael Hind Samuel\u00a0C Hoffman Stephanie Houde Q\u00a0Vera Liao Ronny Luss Aleksandra Mojsilovi\u0107 Sami Mourad Pablo Pedemonte Ramya Raghavendra John Richards Prasanna Sattigeri Karthikeyan Shanmugam Moninder Singh Kush\u00a0R Varshney Dennis Wei and Yunfeng Zhang. [n.d.]. One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques. Technical Report. arxiv:1909.03012v2http:\/\/aix360."},{"key":"e_1_3_2_2_7_1","volume-title":"A Guide for Business Users and Data Scientists","author":"Baer Tobias","unstructured":"Tobias Baer . 2019. Understand, Manage, and Prevent Algorithmic Bias : A Guide for Business Users and Data Scientists . Apress . Tobias Baer. 2019. Understand, Manage, and Prevent Algorithmic Bias: A Guide for Business Users and Data Scientists. Apress."},{"key":"e_1_3_2_2_8_1","unstructured":"Solon Barocas Moritz Hardt and Arvind Narayanan. 2019. Fairness and machine learning. fairmlbook.org. https:\/\/fairmlbook.org\/index.html  Solon Barocas Moritz Hardt and Arvind Narayanan. 2019. Fairness and machine learning. fairmlbook.org. https:\/\/fairmlbook.org\/index.html"},{"key":"e_1_3_2_2_9_1","volume-title":"Nips (oct","author":"Bellamy Rachel","year":"2018","unstructured":"Rachel K.\u00a0E. Bellamy , Kuntal Dey , Michael Hind , Samuel\u00a0 C. Hoffman , Stephanie Houde , Kalapriya Kannan , Pranay Lohia , Jacquelyn Martino , Sameep Mehta , Aleksandra Mojsilovic , Seema Nagar , Karthikeyan\u00a0Natesan Ramamurthy , John Richards , Diptikalyan Saha , Prasanna Sattigeri , Moninder Singh , Kush\u00a0 R. Varshney , and Yunfeng Zhang . 2018. AI Fairness 360: An Extensible Toolkit for Detecting, Understanding, and Mitigating Unwanted Algorithmic Bias. Advances in Neural Information Processing Systems 2017-Decem , Nips (oct 2018 ), 5681\u20135690. arxiv:1810.01943http:\/\/arxiv.org\/abs\/1810.01943 Rachel K.\u00a0E. Bellamy, Kuntal Dey, Michael Hind, Samuel\u00a0C. Hoffman, Stephanie Houde, Kalapriya Kannan, Pranay Lohia, Jacquelyn Martino, Sameep Mehta, Aleksandra Mojsilovic, Seema Nagar, Karthikeyan\u00a0Natesan Ramamurthy, John Richards, Diptikalyan Saha, Prasanna Sattigeri, Moninder Singh, Kush\u00a0R. Varshney, and Yunfeng Zhang. 2018. AI Fairness 360: An Extensible Toolkit for Detecting, Understanding, and Mitigating Unwanted Algorithmic Bias. Advances in Neural Information Processing Systems 2017-Decem, Nips (oct 2018), 5681\u20135690. arxiv:1810.01943http:\/\/arxiv.org\/abs\/1810.01943"},{"key":"e_1_3_2_2_10_1","volume-title":"Proceedings of the Sixty-second Annual Meeting of the American Society for Information Science.","author":"Budzik Jay","year":"1999","unstructured":"Jay Budzik and Kristian Hammond . 1999 . Watson: Anticipating and Contextualizing Information Needs . In Proceedings of the Sixty-second Annual Meeting of the American Society for Information Science. Jay Budzik and Kristian Hammond. 1999. Watson: Anticipating and Contextualizing Information Needs. In Proceedings of the Sixty-second Annual Meeting of the American Society for Information Science."},{"key":"e_1_3_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.5555\/3295222.3295256"},{"key":"e_1_3_2_2_12_1","unstructured":"Civil Comments. 2019. Jigsaw Unintended Bias in Toxicity Classification. https:\/\/www.kaggle.com\/c\/jigsaw-unintended-bias-in-toxicity-classification  Civil Comments. 2019. Jigsaw Unintended Bias in Toxicity Classification. https:\/\/www.kaggle.com\/c\/jigsaw-unintended-bias-in-toxicity-classification"},{"key":"e_1_3_2_2_13_1","volume-title":"The Measure and Mismeasure of Fairness: A Critical Review of Fair Machine Learning. (jul","author":"Corbett-Davies Sam","year":"2018","unstructured":"Sam Corbett-Davies and Sharad Goel . 2018. The Measure and Mismeasure of Fairness: A Critical Review of Fair Machine Learning. (jul 2018 ). arxiv:1808.00023http:\/\/arxiv.org\/abs\/1808.00023 Sam Corbett-Davies and Sharad Goel. 2018. The Measure and Mismeasure of Fairness: A Critical Review of Fair Machine Learning. (jul 2018). arxiv:1808.00023http:\/\/arxiv.org\/abs\/1808.00023"},{"key":"e_1_3_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290607.3299057"},{"key":"e_1_3_2_2_15_1","unstructured":"Kate Crawford. 2017. The Trouble with Bias. https:\/\/www.youtube.com\/watch?v=fMym_BKWQzk  Kate Crawford. 2017. The Trouble with Bias. https:\/\/www.youtube.com\/watch?v=fMym_BKWQzk"},{"key":"e_1_3_2_2_16_1","first-page":"139","article-title":"Demarginalizing the Intersection of Race and Sex: A Black Feminist Critique of Antidiscrimination Doctrine, Feminist Theory and Antiracist Politics","volume":"1989","author":"Crenshaw Kimberle","year":"1989","unstructured":"Kimberle Crenshaw . 1989 . Demarginalizing the Intersection of Race and Sex: A Black Feminist Critique of Antidiscrimination Doctrine, Feminist Theory and Antiracist Politics . University of Chicago Legal Forum 1989 , 1 (1989), 139 \u2013 167 . http:\/\/chicagounbound.uchicago.edu\/uclfhttp:\/\/chicagounbound.uchicago.edu\/uclf\/vol1989\/iss1\/8 Kimberle Crenshaw. 1989. Demarginalizing the Intersection of Race and Sex: A Black Feminist Critique of Antidiscrimination Doctrine, Feminist Theory and Antiracist Politics. University of Chicago Legal Forum 1989, 1 (1989), 139\u2013167. http:\/\/chicagounbound.uchicago.edu\/uclfhttp:\/\/chicagounbound.uchicago.edu\/uclf\/vol1989\/iss1\/8","journal-title":"University of Chicago Legal Forum"},{"key":"e_1_3_2_2_17_1","unstructured":"Matt Day. 2016. How LinkedIn\u2019s search engine may reflect a gender bias. https:\/\/www.seattletimes.com\/business\/microsoft\/how-linkedins-search-engine-may-reflect-a-bias\/  Matt Day. 2016. How LinkedIn\u2019s search engine may reflect a gender bias. https:\/\/www.seattletimes.com\/business\/microsoft\/how-linkedins-search-engine-may-reflect-a-bias\/"},{"key":"e_1_3_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3301275.3302310"},{"key":"e_1_3_2_2_19_1","unstructured":"Dheeru Dua and Casey Graff. 2017. UCI Machine Learning Repository. http:\/\/archive.ics.uci.edu\/ml  Dheeru Dua and Casey Graff. 2017. UCI Machine Learning Repository. http:\/\/archive.ics.uci.edu\/ml"},{"key":"e_1_3_2_2_20_1","volume-title":"A comparative study of fairness-enhancing interventions in machine learning. FAT* 2019 - Proceedings of the 2019 Conference on Fairness, Accountability, and Transparency (feb","author":"Friedler A.","year":"2018","unstructured":"Sorelle\u00a0 A. Friedler , Carlos Scheidegger , Suresh Venkatasubramanian , Sonam Choudhary , Evan\u00a0 P. Hamilton , and Derek Roth . 2018. A comparative study of fairness-enhancing interventions in machine learning. FAT* 2019 - Proceedings of the 2019 Conference on Fairness, Accountability, and Transparency (feb 2018 ), 329\u2013338. arxiv:1802.04422http:\/\/arxiv.org\/abs\/1802.04422 Sorelle\u00a0A. Friedler, Carlos Scheidegger, Suresh Venkatasubramanian, Sonam Choudhary, Evan\u00a0P. Hamilton, and Derek Roth. 2018. A comparative study of fairness-enhancing interventions in machine learning. FAT* 2019 - Proceedings of the 2019 Conference on Fairness, Accountability, and Transparency (feb 2018), 329\u2013338. arxiv:1802.04422http:\/\/arxiv.org\/abs\/1802.04422"},{"key":"e_1_3_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1186\/s41044-016-0014-0"},{"key":"e_1_3_2_2_22_1","volume-title":"Proceedings ofthe AAAI2018 Spring Symposium: Designing the User Experience ofArtificial Intelligence. arxiv:1809","author":"Garcia-Gathright Jean","year":"2018","unstructured":"Jean Garcia-Gathright , Aaron Springer , and Henriette Cramer . 2018 . Assessing and Addressing Algorithmic Bias - But Before We Get There . In Proceedings ofthe AAAI2018 Spring Symposium: Designing the User Experience ofArtificial Intelligence. arxiv:1809 .03332http:\/\/arxiv.org\/abs\/1809.03332 Jean Garcia-Gathright, Aaron Springer, and Henriette Cramer. 2018. Assessing and Addressing Algorithmic Bias - But Before We Get There. In Proceedings ofthe AAAI2018 Spring Symposium: Designing the User Experience ofArtificial Intelligence. arxiv:1809.03332http:\/\/arxiv.org\/abs\/1809.03332"},{"key":"e_1_3_2_2_23_1","unstructured":"Google. 2020. Google Colaboratory.  Google. 2020. Google Colaboratory."},{"key":"e_1_3_2_2_24_1","unstructured":"Google. 2020. What-If Tool. https:\/\/pair-code.github.io\/what-if-tool\/get-started\/  Google. 2020. What-If Tool. https:\/\/pair-code.github.io\/what-if-tool\/get-started\/"},{"key":"e_1_3_2_2_25_1","unstructured":"Moritz Hardt Eric Price and Nati Srebro. 2016. Equality of opportunity in supervised learning. In Advances in neural information processing systems. 3315\u20133323.  Moritz Hardt Eric Price and Nati Srebro. 2016. Equality of opportunity in supervised learning. In Advances in neural information processing systems. 3315\u20133323."},{"key":"e_1_3_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1080\/1369118X.2019.1573912"},{"key":"e_1_3_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300830"},{"key":"e_1_3_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-019-0088-2"},{"key":"e_1_3_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1126\/science.aaa8415"},{"key":"e_1_3_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10115-011-0463-8"},{"key":"e_1_3_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376219"},{"key":"e_1_3_2_2_32_1","unstructured":"Ronny Kohavi and Barry Becker. 1996. Adult Data Set. http:\/\/archive.ics.uci.edu\/ml\/datasets\/Adult  Ronny Kohavi and Barry Becker. 1996. Adult Data Set. http:\/\/archive.ics.uci.edu\/ml\/datasets\/Adult"},{"key":"e_1_3_2_2_33_1","volume-title":"Misleading Black Box Explanations. AIES 2020 - Proceedings of the AAAI\/ACM Conference on AI, Ethics, and Society (nov 2019","author":"Lakkaraju Himabindu","year":"2019","unstructured":"Himabindu Lakkaraju and Osbert Bastani . 2019 . \u201d How do I fool you?\u201d: Manipulating User Trust via Misleading Black Box Explanations. AIES 2020 - Proceedings of the AAAI\/ACM Conference on AI, Ethics, and Society (nov 2019 ), 79\u201385. arxiv:1911.06473http:\/\/arxiv.org\/abs\/1911.06473 Himabindu Lakkaraju and Osbert Bastani. 2019. \u201dHow do I fool you?\u201d: Manipulating User Trust via Misleading Black Box Explanations. AIES 2020 - Proceedings of the AAAI\/ACM Conference on AI, Ethics, and Society (nov 2019), 79\u201385. arxiv:1911.06473http:\/\/arxiv.org\/abs\/1911.06473"},{"key":"e_1_3_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.31219\/osf.io\/uvjqh"},{"key":"e_1_3_2_2_36_1","volume-title":"IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings 2019-May (dec 2018","author":"Lohia K.","year":"2018","unstructured":"Pranay\u00a0 K. Lohia , Karthikeyan\u00a0Natesan Ramamurthy , Manish Bhide , Diptikalyan Saha , Kush\u00a0 R. Varshney , and Ruchir Puri . 2018 . Bias Mitigation Post-processing for Individual and Group Fairness. ICASSP , IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings 2019-May (dec 2018 ), 2847\u20132851. arxiv:1812.06135http:\/\/arxiv.org\/abs\/1812.06135 Pranay\u00a0K. Lohia, Karthikeyan\u00a0Natesan Ramamurthy, Manish Bhide, Diptikalyan Saha, Kush\u00a0R. Varshney, and Ruchir Puri. 2018. Bias Mitigation Post-processing for Individual and Group Fairness. ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings 2019-May (dec 2018), 2847\u20132851. arxiv:1812.06135http:\/\/arxiv.org\/abs\/1812.06135"},{"key":"e_1_3_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376445"},{"key":"e_1_3_2_2_38_1","unstructured":"Ninareh Mehrabi Fred Morstatter Nripsuta Saxena Kristina Lerman and Aram Galstyan. 2019. A Survey on Bias and Fairness in Machine Learning. (2019). arxiv:1908.09635v2  Ninareh Mehrabi Fred Morstatter Nripsuta Saxena Kristina Lerman and Aram Galstyan. 2019. A Survey on Bias and Fairness in Machine Learning. (2019). arxiv:1908.09635v2"},{"key":"e_1_3_2_2_39_1","unstructured":"Microsoft. 2020. Fairlearn. https:\/\/fairlearn.github.io\/  Microsoft. 2020. Fairlearn. https:\/\/fairlearn.github.io\/"},{"key":"e_1_3_2_2_40_1","unstructured":"Arvind Narayanan. 2018. 21 fairness definitions and their politics. https:\/\/fairmlbook.org\/tutorial2.html https:\/\/www.youtube.com\/embed\/jIXIuYdnyyk  Arvind Narayanan. 2018. 21 fairness definitions and their politics. https:\/\/fairmlbook.org\/tutorial2.html https:\/\/www.youtube.com\/embed\/jIXIuYdnyyk"},{"key":"e_1_3_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.obhdp.2020.03.008"},{"key":"e_1_3_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.2307\/j.ctt1pwt9w5"},{"key":"e_1_3_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.3389\/fdata.2019.00013"},{"key":"e_1_3_2_2_44_1","unstructured":"Qualtrics. 2020. Qualtrics. https:\/\/www.qualtrics.com  Qualtrics. 2020. Qualtrics. https:\/\/www.qualtrics.com"},{"key":"e_1_3_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2019\/456"},{"key":"e_1_3_2_2_47_1","volume-title":"Aequitas: A Bias and Fairness Audit Toolkit. (nov","author":"Saleiro Pedro","year":"2018","unstructured":"Pedro Saleiro , Benedict Kuester , Loren Hinkson , Jesse London , Abby Stevens , Ari Anisfeld , Kit\u00a0 T. Rodolfa , and Rayid Ghani . 2018 . Aequitas: A Bias and Fairness Audit Toolkit. (nov 2018). arxiv:1811.05577http:\/\/arxiv.org\/abs\/1811.05577 Pedro Saleiro, Benedict Kuester, Loren Hinkson, Jesse London, Abby Stevens, Ari Anisfeld, Kit\u00a0T. Rodolfa, and Rayid Ghani. 2018. Aequitas: A Bias and Fairness Audit Toolkit. (nov 2018). arxiv:1811.05577http:\/\/arxiv.org\/abs\/1811.05577"},{"key":"e_1_3_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/icalt.2006.1652397"},{"key":"e_1_3_2_2_49_1","unstructured":"Sriram Vasudevan Cyrus DiCiccio and Kinjal Basu. 2020. Addressing bias in large-scale AI applications: The LinkedIn Fairness Toolkit. https:\/\/engineering.linkedin.com\/blog\/2020\/lift-addressing-bias-in-large-scale-ai-applications  Sriram Vasudevan Cyrus DiCiccio and Kinjal Basu. 2020. Addressing bias in large-scale AI applications: The LinkedIn Fairness Toolkit. https:\/\/engineering.linkedin.com\/blog\/2020\/lift-addressing-bias-in-large-scale-ai-applications"},{"key":"e_1_3_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1177\/2053951717743530"},{"key":"e_1_3_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174014"},{"key":"e_1_3_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3194770.3194776"},{"key":"e_1_3_2_2_53_1","volume-title":"Predictive Inequity in Object Detection. (feb","author":"Wilson Benjamin","year":"2019","unstructured":"Benjamin Wilson , Judy Hoffman , and Jamie Morgenstern . 2019. Predictive Inequity in Object Detection. (feb 2019 ). arxiv:1902.11097http:\/\/arxiv.org\/abs\/1902.11097 Benjamin Wilson, Judy Hoffman, and Jamie Morgenstern. 2019. Predictive Inequity in Object Detection. (feb 2019). arxiv:1902.11097http:\/\/arxiv.org\/abs\/1902.11097"},{"key":"e_1_3_2_2_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132847.3132938"}],"event":{"name":"CHI '21: CHI Conference on Human Factors in Computing Systems","location":"Yokohama Japan","acronym":"CHI '21","sponsor":["SIGCHI ACM Special Interest Group on Computer-Human Interaction"]},"container-title":["Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3411764.3445604","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3411764.3445604","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T21:28:38Z","timestamp":1750195718000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3411764.3445604"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,5,6]]},"references-count":52,"alternative-id":["10.1145\/3411764.3445604","10.1145\/3411764"],"URL":"https:\/\/doi.org\/10.1145\/3411764.3445604","relation":{},"subject":[],"published":{"date-parts":[[2021,5,6]]},"assertion":[{"value":"2021-05-07","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}