{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,29]],"date-time":"2026-07-29T19:07:07Z","timestamp":1785352027411,"version":"3.55.0"},"publisher-location":"New York, NY, USA","reference-count":113,"publisher":"ACM","license":[{"start":{"date-parts":[[2019,5,2]],"date-time":"2019-05-02T00:00:00Z","timestamp":1556755200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2019,5,2]]},"DOI":"10.1145\/3290605.3300830","type":"proceedings-article","created":{"date-parts":[[2019,4,29]],"date-time":"2019-04-29T17:36:58Z","timestamp":1556559418000},"page":"1-16","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":694,"title":["Improving Fairness in Machine Learning Systems"],"prefix":"10.1145","author":[{"given":"Kenneth","family":"Holstein","sequence":"first","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jennifer","family":"Wortman Vaughan","sequence":"additional","affiliation":[{"name":"Microsoft Research, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"suffix":"III","given":"Hal","family":"Daum\u00e9","sequence":"additional","affiliation":[{"name":"Microsoft Research & University of Maryland, New York City, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Miro","family":"Dudik","sequence":"additional","affiliation":[{"name":"Microsoft Research, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hanna","family":"Wallach","sequence":"additional","affiliation":[{"name":"Microsoft Research, New York City, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2019,5,2]]},"reference":[{"key":"e_1_3_2_2_1_1","volume-title":"ACM Conference on Fairness, Accountability, and Transparency (ACM FAT*). https:\/\/fatconference.org\/ Accessed: 2018-0615","author":"ACM.","year":"2018","unstructured":"ACM. 2018 . ACM Conference on Fairness, Accountability, and Transparency (ACM FAT*). https:\/\/fatconference.org\/ Accessed: 2018-0615 . ACM. 2018. ACM Conference on Fairness, Accountability, and Transparency (ACM FAT*). https:\/\/fatconference.org\/ Accessed: 2018-0615."},{"key":"e_1_3_2_2_2_1","unstructured":"ACM. 2018. FAT\/ML. https:\/\/www.fatml.org.Accessed: 2018-06--15.  ACM. 2018. FAT\/ML. https:\/\/www.fatml.org.Accessed: 2018-06--15."},{"key":"e_1_3_2_2_3_1","volume-title":"Proceedings of the Thirty-fifth International Conference on Machine Learning (ICML","author":"Agarwal Alekh","year":"2018","unstructured":"Alekh Agarwal , Alina Beygelzimer , Miroslav Dud\u00edk , John Langford , and Hanna Wallach . 2018 . A reductions approach to fair classification . In Proceedings of the Thirty-fifth International Conference on Machine Learning (ICML 2018). Alekh Agarwal, Alina Beygelzimer, Miroslav Dud\u00edk, John Langford, and Hanna Wallach. 2018. A reductions approach to fair classification. In Proceedings of the Thirty-fifth International Conference on Machine Learning (ICML 2018)."},{"key":"e_1_3_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173860"},{"key":"e_1_3_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702509"},{"key":"e_1_3_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N18-1021"},{"key":"e_1_3_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3264590"},{"key":"e_1_3_2_2_8_1","volume-title":"Machine bias: There's software used across the country to predict future criminals, and it's biased against blacks. ProPublica","author":"Angwin Julia","year":"2016","unstructured":"Julia Angwin , Jeff Larson , Surya Mattu , and Lauren Kirchner . 2016. Machine bias: There's software used across the country to predict future criminals, and it's biased against blacks. ProPublica ( 2016 ). Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2016. Machine bias: There's software used across the country to predict future criminals, and it's biased against blacks. ProPublica (2016)."},{"key":"e_1_3_2_2_9_1","first-page":"2","article-title":"Beat the machine: Challenging workers to find the unknown unknowns","volume":"11","author":"Attenberg Josh","year":"2011","unstructured":"Josh Attenberg , Panagiotis G Ipeirotis , and Foster J Provost . 2011 . Beat the machine: Challenging workers to find the unknown unknowns . Human Computation 11 , 11 (2011), 2 -- 7 . Josh Attenberg, Panagiotis G Ipeirotis, and Foster J Provost. 2011. Beat the machine: Challenging workers to find the unknown unknowns. Human Computation 11, 11 (2011), 2--7.","journal-title":"Human Computation"},{"key":"e_1_3_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11257-007-9045-6"},{"key":"e_1_3_2_2_11_1","first-page":"671","article-title":"Big data's disparate impact","volume":"104","author":"Barocas Solon","year":"2016","unstructured":"Solon Barocas and Andrew D Selbst . 2016 . Big data's disparate impact . Cal. L. Rev. 104 (2016), 671 . Solon Barocas and Andrew D Selbst. 2016. Big data's disparate impact. Cal. L. Rev. 104 (2016), 671.","journal-title":"Cal. L. Rev."},{"key":"e_1_3_2_2_12_1","volume-title":"Google searches expose racial bias, says study of names. BBC News (Feb","author":"BBC.","year":"2013","unstructured":"BBC. 2013. Google searches expose racial bias, says study of names. BBC News (Feb 2013 ). https:\/\/www.bbc.com\/news\/technology21322183. Accessed : 2018-09-03. BBC. 2013. Google searches expose racial bias, says study of names. BBC News (Feb 2013). https:\/\/www.bbc.com\/news\/technology21322183. Accessed: 2018-09-03."},{"key":"e_1_3_2_2_13_1","doi-asserted-by":"crossref","unstructured":"Emily Bender and Batya Friedman. 2018. Data statements for NLP: Toward mitigating system bias and enabling better science. OpenReview Preprint.  Emily Bender and Batya Friedman. 2018. Data statements for NLP: Toward mitigating system bias and enabling better science. OpenReview Preprint.","DOI":"10.1162\/tacl_a_00041"},{"key":"e_1_3_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1177\/0049124118782533"},{"key":"e_1_3_2_2_15_1","first-page":"149","article-title":"Fairness in machine learning: Lessons from political philosophy","volume":"81","author":"Binns Reuben","year":"2018","unstructured":"Reuben Binns . 2018 . Fairness in machine learning: Lessons from political philosophy . Proceedings of Machine Learning Research 81 , 149 -- 159 . Reuben Binns. 2018. Fairness in machine learning: Lessons from political philosophy. Proceedings of Machine Learning Research 81, 149--159.","journal-title":"Proceedings of Machine Learning Research"},{"key":"e_1_3_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173951"},{"key":"e_1_3_2_2_17_1","volume-title":"Advances in Neural Information Processing Systems (NeurIPS","author":"Bolukbasi Tolga","year":"2016","unstructured":"Tolga Bolukbasi , Kai-Wei Chang , James Y Zou , Venkatesh Saligrama , and Adam T Kalai . 2016. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings . In Advances in Neural Information Processing Systems (NeurIPS 2016 ). 4349--4357. Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In Advances in Neural Information Processing Systems (NeurIPS 2016). 4349--4357."},{"key":"e_1_3_2_2_18_1","volume-title":"Proceedings of the 2016 International Joint Conference on Artificial Intelligence (IJCAI","author":"Bosch Nigel","year":"2016","unstructured":"Nigel Bosch , Sidney K D'Mello , Ryan S Baker , Jaclyn Ocumpaugh , Valerie Shute , Matthew Ventura , Lubin Wang , and Weinan Zhao . 2016 . Detecting student emotions in computer-enabled classrooms . In Proceedings of the 2016 International Joint Conference on Artificial Intelligence (IJCAI 2016). 4125--4129. Nigel Bosch, Sidney K D'Mello, Ryan S Baker, Jaclyn Ocumpaugh, Valerie Shute, Matthew Ventura, Lubin Wang, and Weinan Zhao. 2016. Detecting student emotions in computer-enabled classrooms. In Proceedings of the 2016 International Joint Conference on Artificial Intelligence (IJCAI 2016). 4125--4129."},{"key":"e_1_3_2_2_19_1","volume-title":"Test-driven machine learning","author":"Bozonier Justin","unstructured":"Justin Bozonier . 2015. Test-driven machine learning . Packt Publishing Ltd . Justin Bozonier. 2015. Test-driven machine learning. Packt Publishing Ltd."},{"key":"e_1_3_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1080\/1369118X.2016.1154086"},{"key":"e_1_3_2_2_21_1","volume-title":"Proceedings of the 2018 Conference on Fairness, Accountability and Transparency (FAT*","author":"Buolamwini Joy","year":"2018","unstructured":"Joy Buolamwini and Timnit Gebru . 2018 . Gender Shades: Intersectional accuracy disparities in commercial gender classification . In Proceedings of the 2018 Conference on Fairness, Accountability and Transparency (FAT* 2018). 77--91. Joy Buolamwini and Timnit Gebru. 2018. Gender Shades: Intersectional accuracy disparities in commercial gender classification. In Proceedings of the 2018 Conference on Fairness, Accountability and Transparency (FAT* 2018). 77--91."},{"key":"e_1_3_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3026044"},{"key":"e_1_3_2_2_23_1","volume-title":"Why is my classifier discriminatory? Advances in Neural Information Processing Systems (NeurIPS","author":"Chen Irene","year":"2018","unstructured":"Irene Chen , Fredrik D Johansson , and David Sontag . 2018. Why is my classifier discriminatory? Advances in Neural Information Processing Systems (NeurIPS 2018 ). Irene Chen, Fredrik D Johansson, and David Sontag. 2018. Why is my classifier discriminatory? Advances in Neural Information Processing Systems (NeurIPS 2018)."},{"key":"e_1_3_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3172944.3172950"},{"key":"e_1_3_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1089\/big.2016.0047"},{"key":"e_1_3_2_2_26_1","volume-title":"ACM Conference on Fairness, Accountability, and Transparency (FAT*","author":"Chouldechova Alexandra","year":"2018","unstructured":"Alexandra Chouldechova , Diana Benavides-Prado , Oleksandr Fialko , and Rhema Vaithianathan . 2018 . A case study of algorithm-assisted decision making in child maltreatment hotline screening decisions . In ACM Conference on Fairness, Accountability, and Transparency (FAT* 2018). 134--148. Alexandra Chouldechova, Diana Benavides-Prado, Oleksandr Fialko, and Rhema Vaithianathan. 2018. A case study of algorithm-assisted decision making in child maltreatment hotline screening decisions. In ACM Conference on Fairness, Accountability, and Transparency (FAT* 2018). 134--148."},{"key":"e_1_3_2_2_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290607.3299057"},{"key":"e_1_3_2_2_28_1","unstructured":"Kate Crawford. 2017. Artificial intelligence with very real biases. http:\/\/www.wsj.com\/articles\/artificial-intelligencewith-veryreal-biases-1508252717. Accessed: 2018-06--15.  Kate Crawford. 2017. Artificial intelligence with very real biases. http:\/\/www.wsj.com\/articles\/artificial-intelligencewith-veryreal-biases-1508252717. Accessed: 2018-06--15."},{"key":"e_1_3_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173986"},{"key":"e_1_3_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025739"},{"key":"e_1_3_2_2_31_1","volume-title":"Aequitas: Bias and fairness audit toolkit","author":"DSSG.","year":"2018","unstructured":"DSSG. 2018 . Aequitas: Bias and fairness audit toolkit . http:\/\/aequitas.dssg.io. Accessed: 2018-08--29. DSSG. 2018. Aequitas: Bias and fairness audit toolkit. http:\/\/aequitas.dssg.io. Accessed: 2018-08--29."},{"key":"e_1_3_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2090236.2090255"},{"key":"e_1_3_2_2_33_1","unstructured":"Cynthia Dwork and Christina Ilvento. 2018. Fairness under composition. CoRR arXiv:1806.06122.  Cynthia Dwork and Christina Ilvento. 2018. Fairness under composition. CoRR arXiv:1806.06122."},{"key":"e_1_3_2_2_34_1","volume-title":"Dermatologistlevel classification of skin cancer with deep neural networks. Nature 542, 7639","author":"Esteva Andre","year":"2017","unstructured":"Andre Esteva , Brett Kuprel , Roberto A Novoa , Justin Ko , Susan M Swetter , Helen M Blau , and Sebastian Thrun . 2017. Dermatologistlevel classification of skin cancer with deep neural networks. Nature 542, 7639 ( 2017 ), 115. Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and Sebastian Thrun. 2017. Dermatologistlevel classification of skin cancer with deep neural networks. Nature 542, 7639 (2017), 115."},{"key":"e_1_3_2_2_35_1","volume-title":"Fairness in precision medicine","author":"Ferryman Kadija","year":"2018","unstructured":"Kadija Ferryman and Mikaela Pitcan . 2018. Fairness in precision medicine . Data & Society ( 2018 ). Kadija Ferryman and Mikaela Pitcan. 2018. Fairness in precision medicine. Data & Society (2018)."},{"key":"e_1_3_2_2_36_1","volume-title":"Algorithmic accountability","author":"Web Foundation World Wide","year":"2017","unstructured":"World Wide Web Foundation . 2017. Algorithmic accountability . World Wide Web Foundation ( 2017 ). World Wide Web Foundation. 2017. Algorithmic accountability. World Wide Web Foundation (2017)."},{"key":"e_1_3_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/230538.230561"},{"key":"e_1_3_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3106237.3106277"},{"key":"e_1_3_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025930"},{"key":"e_1_3_2_2_40_1","volume-title":"Hanna Wallach, Hal Daum\u00e9 III, and Kate Crawford.","author":"Gebru Timnit","year":"2018","unstructured":"Timnit Gebru , Jamie Morgenstern , Briana Vecchione , Jennifer Wortman Vaughan , Hanna Wallach, Hal Daum\u00e9 III, and Kate Crawford. 2018 . Datasheets for datasets. CoRR arXiv:1803.09010. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daum\u00e9 III, and Kate Crawford. 2018. Datasheets for datasets. CoRR arXiv:1803.09010."},{"key":"e_1_3_2_2_41_1","unstructured":"Dave Gershgorn. 2018. America's biggest body-camera company says facial recognition isn't accurate enough for police. https:\/\/qz.com\/1351519\/facial-recognition-isnt-yet-accurateenough-for-policing-decisions\/. Accessed: 2018-08--30.  Dave Gershgorn. 2018. America's biggest body-camera company says facial recognition isn't accurate enough for police. https:\/\/qz.com\/1351519\/facial-recognition-isnt-yet-accurateenough-for-policing-decisions\/. Accessed: 2018-08--30."},{"key":"e_1_3_2_2_42_1","unstructured":"Dave Gershgorn. 2018. If AI is going to be the world's doctor it needs better textbooks. https:\/\/qz.com\/1367177\/if-ai-is-going-to-be-theworlds-doctor-it-needs-better-textbooks. Accessed: 2018-09--16.  Dave Gershgorn. 2018. If AI is going to be the world's doctor it needs better textbooks. https:\/\/qz.com\/1367177\/if-ai-is-going-to-be-theworlds-doctor-it-needs-better-textbooks. Accessed: 2018-09--16."},{"key":"e_1_3_2_2_43_1","unstructured":"Vivian Giang. 2018. The potential hidden bias in automated hiring systems. https:\/\/www.fastcompany.com\/40566971\/the-potentialhidden-bias-in-automated-hiring-systems. Accessed: 2018-09-03.  Vivian Giang. 2018. The potential hidden bias in automated hiring systems. https:\/\/www.fastcompany.com\/40566971\/the-potentialhidden-bias-in-automated-hiring-systems. Accessed: 2018-09-03."},{"key":"e_1_3_2_2_44_1","unstructured":"Google. 2018. The UX of AI - Library. https:\/\/design.google\/ library\/ux-ai\/. Accessed: 2018-08--28.  Google. 2018. The UX of AI - Library. https:\/\/design.google\/ library\/ux-ai\/. Accessed: 2018-08--28."},{"key":"e_1_3_2_2_45_1","unstructured":"Google. 2018. The What-If Tool: Code-free probing of machine learning models. https:\/\/ai.googleblog.com\/2018\/09\/the-what-if-tool-codefree-probing-of.html. Accessed: 2018-09--18.  Google. 2018. The What-If Tool: Code-free probing of machine learning models. https:\/\/ai.googleblog.com\/2018\/09\/the-what-if-tool-codefree-probing-of.html. Accessed: 2018-09--18."},{"key":"e_1_3_2_2_46_1","volume-title":"the ICML 2018 Debates Workshop.","author":"Green Ben","year":"2018","unstructured":"Ben Green and Lily Hu . 2018 . The myth in the methodology: Towards a recontextualization of fairness in machine learning . In the ICML 2018 Debates Workshop. Ben Green and Lily Hu. 2018. The myth in the methodology: Towards a recontextualization of fairness in machine learning. In the ICML 2018 Debates Workshop."},{"key":"e_1_3_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173582"},{"key":"e_1_3_2_2_48_1","volume-title":"Universal methods of design: 100 ways to research complex problems, develop innovative ideas, and design effective solutions","author":"Hanington Bruce","unstructured":"Bruce Hanington and Bella Martin . 2012. Universal methods of design: 100 ways to research complex problems, develop innovative ideas, and design effective solutions . Rockport Publishers . Bruce Hanington and Bella Martin. 2012. Universal methods of design: 100 ways to research complex problems, develop innovative ideas, and design effective solutions. Rockport Publishers."},{"key":"e_1_3_2_2_49_1","unstructured":"Moritz Hardt Eric Price Nati Srebro etal 2016. Equality of opportunity in supervised learning. In Advances in Neural Information Processing Systems (NeurIPS 2016). 3315--3323.   Moritz Hardt Eric Price Nati Srebro et al. 2016. Equality of opportunity in supervised learning. In Advances in Neural Information Processing Systems (NeurIPS 2016). 3315--3323."},{"key":"e_1_3_2_2_50_1","unstructured":"HireVue.com. 2018. Video interview software for recruiting & hiring. https:\/\/www.hirevue.com\/. Accessed: 2018-08--28.  HireVue.com. 2018. Video interview software for recruiting & hiring. https:\/\/www.hirevue.com\/. Accessed: 2018-08--28."},{"key":"e_1_3_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3170358.3170377"},{"key":"e_1_3_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3027385.3027451"},{"key":"e_1_3_2_2_53_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-93843-1_12"},{"key":"e_1_3_2_2_54_1","volume-title":"Contextual inquiry: A participatory technique for system design. Participatory design: Principles and practices","author":"Holtzblatt Karen","year":"1993","unstructured":"Karen Holtzblatt and Sandra Jones . 1993. Contextual inquiry: A participatory technique for system design. Participatory design: Principles and practices ( 1993 ), 177--210. Karen Holtzblatt and Sandra Jones. 1993. Contextual inquiry: A participatory technique for system design. Participatory design: Principles and practices (1993), 177--210."},{"key":"e_1_3_2_2_55_1","doi-asserted-by":"crossref","unstructured":"AI Now Institute. 2018. AI Now Institute. https:\/\/ainowinstitute.org. Accessed: 2018-08-03.  AI Now Institute. 2018. AI Now Institute. https:\/\/ainowinstitute.org. Accessed: 2018-08-03.","DOI":"10.12968\/cypn.2018.12.8"},{"key":"e_1_3_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3267851.3267916"},{"key":"e_1_3_2_2_57_1","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2005.314"},{"key":"e_1_3_2_2_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858515"},{"key":"e_1_3_2_2_59_1","unstructured":"Nathan Kallus and Angela Zhou. 2018. Residual unfairness in fair machine learning from prejudiced data. CoRR arXiv:1806.02887.  Nathan Kallus and Angela Zhou. 2018. Residual unfairness in fair machine learning from prejudiced data. CoRR arXiv:1806.02887."},{"key":"e_1_3_2_2_60_1","volume-title":"Proceedings of the 2016 International Joint Conference on Artificial Intelligence (IJCAI","author":"Kamar Ece","year":"2016","unstructured":"Ece Kamar . 2016 . Directions in hybrid intelligence: Complementing AI systems with human intelligence .. In Proceedings of the 2016 International Joint Conference on Artificial Intelligence (IJCAI 2016). 4070--4073. Ece Kamar. 2016. Directions in hybrid intelligence: Complementing AI systems with human intelligence.. In Proceedings of the 2016 International Joint Conference on Artificial Intelligence (IJCAI 2016). 4070--4073."},{"key":"e_1_3_2_2_61_1","volume-title":"Third AAAI Conference on Human Computation and Crowdsourcing (HCOMP","author":"Kamar Ece","year":"2015","unstructured":"Ece Kamar , Ashish Kapoor , and Eric Horvitz . 2015 . Identifying and accounting for task-dependent bias in crowdsourcing . In Third AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2015). Ece Kamar, Ashish Kapoor, and Eric Horvitz. 2015. Identifying and accounting for task-dependent bias in crowdsourcing. In Third AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2015)."},{"key":"e_1_3_2_2_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702520"},{"key":"e_1_3_2_2_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173748"},{"key":"e_1_3_2_2_64_1","volume-title":"Proceedings of the Thirty-Fifth International Conference on Machine Learning (ICML","author":"Kilbertus Niki","year":"2018","unstructured":"Niki Kilbertus , Adri\u00e0 Gasc\u00f3n , Matt J Kusner , Michael Veale , Krishna P Gummadi , and Adrian Weller . 2018 . Blind justice: Fairness with encrypted sensitive attributes . Proceedings of the Thirty-Fifth International Conference on Machine Learning (ICML 2018). Niki Kilbertus, Adri\u00e0 Gasc\u00f3n, Matt J Kusner, Michael Veale, Krishna P Gummadi, and Adrian Weller. 2018. Blind justice: Fairness with encrypted sensitive attributes. Proceedings of the Thirty-Fifth International Conference on Machine Learning (ICML 2018)."},{"key":"e_1_3_2_2_65_1","volume-title":"Proceedings of the Eighth Innovations in Theoretical Computer Science Conference (ITCS 2017)","author":"Kleinberg Jon","year":"2016","unstructured":"Jon Kleinberg , Sendhil Mullainathan , and Manish Raghavan . 2016 . Inherent trade-offs in the fair determination of risk scores . Proceedings of the Eighth Innovations in Theoretical Computer Science Conference (ITCS 2017) (2016). Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. 2016. Inherent trade-offs in the fair determination of risk scores. Proceedings of the Eighth Innovations in Theoretical Computer Science Conference (ITCS 2017) (2016)."},{"key":"e_1_3_2_2_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/2556288.2557238"},{"key":"e_1_3_2_2_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/2678025.2701399"},{"key":"e_1_3_2_2_68_1","volume-title":"Advances in Neural Information Processing Systems (NeurIPS","author":"Kusner Matt J","year":"2017","unstructured":"Matt J Kusner , Joshua Loftus , Chris Russell , and Ricardo Silva . 2017. Counterfactual fairness . In Advances in Neural Information Processing Systems (NeurIPS 2017 ). 4066--4076. Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. 2017. Counterfactual fairness. In Advances in Neural Information Processing Systems (NeurIPS 2017). 4066--4076."},{"key":"e_1_3_2_2_69_1","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence (AAAI","author":"Lakkaraju Himabindu","year":"2017","unstructured":"Himabindu Lakkaraju , Ece Kamar , Rich Caruana , and Eric Horvitz . 2017 . Identifying Unknown Unknowns in the Open World: Representations and Policies for Guided Exploration .. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI 2017). Himabindu Lakkaraju, Ece Kamar, Rich Caruana, and Eric Horvitz. 2017. Identifying Unknown Unknowns in the Open World: Representations and Policies for Guided Exploration.. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI 2017)."},{"key":"e_1_3_2_2_70_1","volume-title":"How we analyzed the COMPAS recidivism algorithm. ProPublica (5 2016) 9","author":"Larson Jeff","year":"2016","unstructured":"Jeff Larson , Surya Mattu , Lauren Kirchner , and Julia Angwin . 2016. How we analyzed the COMPAS recidivism algorithm. ProPublica (5 2016) 9 ( 2016 ). Jeff Larson, Surya Mattu, Lauren Kirchner, and Julia Angwin. 2016. How we analyzed the COMPAS recidivism algorithm. ProPublica (5 2016) 9 (2016)."},{"key":"e_1_3_2_2_71_1","doi-asserted-by":"publisher","DOI":"10.1177\/2053951718756684"},{"key":"e_1_3_2_2_72_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998230"},{"key":"e_1_3_2_2_73_1","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence (AAAI","author":"Liu Anqi","year":"2015","unstructured":"Anqi Liu , Lev Reyzin , and Brian D Ziebart . 2015 . Shift-pessimistic active learning using robust bias-aware prediction . In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI 2015). 2764--2770. Anqi Liu, Lev Reyzin, and Brian D Ziebart. 2015. Shift-pessimistic active learning using robust bias-aware prediction. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI 2015). 2764--2770."},{"key":"e_1_3_2_2_74_1","volume-title":"Proceedings of the Fourteenth Innovative Applications of Artificial Intelligence Conference (IAAI","author":"Liu Hugo","year":"2002","unstructured":"Hugo Liu and Push Singh . 2002 . MAKEBELIEVE: Using commonsense knowledge to generate stories . In Proceedings of the Fourteenth Innovative Applications of Artificial Intelligence Conference (IAAI 2002). 957--958. Hugo Liu and Push Singh. 2002. MAKEBELIEVE: Using commonsense knowledge to generate stories. In Proceedings of the Fourteenth Innovative Applications of Artificial Intelligence Conference (IAAI 2002). 957--958."},{"key":"e_1_3_2_2_75_1","volume-title":"Proceedings of the Thirty-fifth International Conference on Machine Learning (ICML 2018)","author":"Liu Lydia T","year":"2018","unstructured":"Lydia T Liu , Sarah Dean , Esther Rolf , Max Simchowitz , and Moritz Hardt . 2018 . Delayed impact of fair machine learning . Proceedings of the Thirty-fifth International Conference on Machine Learning (ICML 2018) (2018). Lydia T Liu, Sarah Dean, Esther Rolf, Max Simchowitz, and Moritz Hardt. 2018. Delayed impact of fair machine learning. Proceedings of the Thirty-fifth International Conference on Machine Learning (ICML 2018) (2018)."},{"key":"e_1_3_2_2_76_1","unstructured":"Natasha Lomas. 2018. Accenture wants to beat unfair AI with a professional toolkit. https:\/\/techcrun ch.com\/2018\/06\/09\/accenturewants-to-beat-unfair-ai-with-a-professional-toolkit\/. Accessed: 201806--14.  Natasha Lomas. 2018. Accenture wants to beat unfair AI with a professional toolkit. https:\/\/techcrun ch.com\/2018\/06\/09\/accenturewants-to-beat-unfair-ai-with-a-professional-toolkit\/. Accessed: 201806--14."},{"key":"e_1_3_2_2_77_1","unstructured":"Natasha Lomas. 2018. IBM launches cloud tool to detect AI bias and explain automated decisions. https:\/\/techcrunch.com\/2018\/09\/19\/ibmlaunches-cloud-tool-to-detect-ai-bias-and-explain-automateddecisions. Accessed: 2018-09--20.  Natasha Lomas. 2018. IBM launches cloud tool to detect AI bias and explain automated decisions. https:\/\/techcrunch.com\/2018\/09\/19\/ibmlaunches-cloud-tool-to-detect-ai-bias-and-explain-automateddecisions. Accessed: 2018-09--20."},{"key":"e_1_3_2_2_78_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1740-9713.2016.00960.x"},{"key":"e_1_3_2_2_79_1","volume-title":"International Journal of Data Science and Analytics","author":"Lyu Lingyu","year":"2018","unstructured":"Lingyu Lyu , Mehmed Kantardzic , and Tegjyot Singh Sethi . 2018. Sloppiness mitigation in crowdsourcing: detecting and correcting bias for crowd scoring tasks . International Journal of Data Science and Analytics ( 2018 ), 1--21. Lingyu Lyu, Mehmed Kantardzic, and Tegjyot Singh Sethi. 2018. Sloppiness mitigation in crowdsourcing: detecting and correcting bias for crowd scoring tasks. International Journal of Data Science and Analytics (2018), 1--21."},{"key":"e_1_3_2_2_80_1","volume-title":"Proceedings of the 2016 International Conference on Educational Data Mining (EDM","author":"Maclellan Christopher J","year":"2016","unstructured":"Christopher J Maclellan , Erik Harpstead , Rony Patel , and Kenneth R Koedinger . 2016 . The Apprentice Learner architecture: Closing the loop between learning theory and educational data .. In Proceedings of the 2016 International Conference on Educational Data Mining (EDM 2016). 151--158. Christopher J Maclellan, Erik Harpstead, Rony Patel, and Kenneth R Koedinger. 2016. The Apprentice Learner architecture: Closing the loop between learning theory and educational data.. In Proceedings of the 2016 International Conference on Educational Data Mining (EDM 2016). 151--158."},{"key":"e_1_3_2_2_81_1","doi-asserted-by":"publisher","DOI":"10.1177\/0004944116664618"},{"key":"e_1_3_2_2_82_1","unstructured":"MURAL. 2018. MURAL - Make remote design work. https:\/\/mural.co\/. Accessed: 2018-08-02.  MURAL. 2018. MURAL - Make remote design work. https:\/\/mural.co\/. Accessed: 2018-08-02."},{"key":"e_1_3_2_2_83_1","volume-title":"21 fairness definitions and their politics. FAT* 2018 tutorial","author":"Narayanan Arvind","year":"2018","unstructured":"Arvind Narayanan . 2018. 21 fairness definitions and their politics. FAT* 2018 tutorial ( 2018 ). Arvind Narayanan. 2018. 21 fairness definitions and their politics. FAT* 2018 tutorial (2018)."},{"key":"e_1_3_2_2_84_1","volume-title":"Algorithms of oppression: How search engines reinforce racism","author":"Noble Safiya Umoja","unstructured":"Safiya Umoja Noble . 2018. Algorithms of oppression: How search engines reinforce racism . NYU Press . Safiya Umoja Noble. 2018. Algorithms of oppression: How search engines reinforce racism. NYU Press."},{"key":"e_1_3_2_2_85_1","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence (AAAI","author":"Nushi Besmira","year":"2017","unstructured":"Besmira Nushi , Ece Kamar , Eric Horvitz , and Donald Kossmann . 2017 . On human intellect and machine failures: Troubleshooting integrative machine learning systems . In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI 2017). 1017--1025. Besmira Nushi, Ece Kamar, Eric Horvitz, and Donald Kossmann. 2017. On human intellect and machine failures: Troubleshooting integrative machine learning systems. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI 2017). 1017--1025."},{"key":"e_1_3_2_2_86_1","unstructured":"US Department of Education (ED). 2018. Family Educational Rights and Privacy Act (FERPA). https:\/\/www2.ed.gov\/policy\/gen\/guid\/fpco\/ferpa\/index.html. Accessed: 2018-09-04.  US Department of Education (ED). 2018. Family Educational Rights and Privacy Act (FERPA). https:\/\/www2.ed.gov\/policy\/gen\/guid\/fpco\/ferpa\/index.html. Accessed: 2018-09-04."},{"key":"e_1_3_2_2_87_1","unstructured":"Partnership on AI. 2018. The Partnership on AI. https:\/\/www.partnershiponai.org. Accessed: 2018-09-03.  Partnership on AI. 2018. The Partnership on AI. https:\/\/www.partnershiponai.org. Accessed: 2018-09-03."},{"key":"e_1_3_2_2_88_1","doi-asserted-by":"publisher","DOI":"10.1007\/s12553-017-0179-1"},{"key":"e_1_3_2_2_89_1","unstructured":"pymetrics. 2018. matching talent to opportunity. https:\/\/www.pymetrics.com\/. Accessed: 2018-08--28.  pymetrics. 2018. matching talent to opportunity. https:\/\/www.pymetrics.com\/. Accessed: 2018-08--28."},{"key":"e_1_3_2_2_90_1","unstructured":"Qualtrics. 2013. Qualtrics. Provo UT USA (2013).  Qualtrics. 2013. Qualtrics. Provo UT USA (2013)."},{"key":"e_1_3_2_2_91_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702174"},{"key":"e_1_3_2_2_92_1","volume-title":"Proceedings of the Thirty-first Annual Conference on Learning Theory (COLT","author":"Raghavan Manish","year":"2018","unstructured":"Manish Raghavan , Aleksandrs Slivkins , Jennifer Wortman Vaughan , and Zhiwei Steven Wu . 2018 . The externalities of exploration and how data diversity helps exploitation . In Proceedings of the Thirty-first Annual Conference on Learning Theory (COLT 2018). Manish Raghavan, Aleksandrs Slivkins, Jennifer Wortman Vaughan, and Zhiwei Steven Wu. 2018. The externalities of exploration and how data diversity helps exploitation. In Proceedings of the Thirty-first Annual Conference on Learning Theory (COLT 2018)."},{"key":"e_1_3_2_2_93_1","volume-title":"Algorithmic impact assessments: A practical framework for public agency accountability","author":"Reisman Dillon","year":"2018","unstructured":"Dillon Reisman , Jason Schultz , K Crawford , and M Whittaker . 2018. Algorithmic impact assessments: A practical framework for public agency accountability . AI Now Institute ( 2018 ). Dillon Reisman, Jason Schultz, K Crawford, and M Whittaker. 2018. Algorithmic impact assessments: A practical framework for public agency accountability. AI Now Institute (2018)."},{"key":"e_1_3_2_2_94_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173889"},{"key":"e_1_3_2_2_95_1","volume-title":"Advances in Neural Information Processing Systems (NeurIPS","author":"Sculley D.","year":"2015","unstructured":"D. Sculley , Gary Holt , Daniel Golovin , Eugene Davydov , Todd Phillips , Dietmar Ebner , Vinay Chaudhary , Michael Young , Jean-Francois Crespo , and Dan Dennison . 2015. Hidden technical debt in machine learning systems . In Advances in Neural Information Processing Systems (NeurIPS 2015 ). 2503--2511. D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-Francois Crespo, and Dan Dennison. 2015. Hidden technical debt in machine learning systems. In Advances in Neural Information Processing Systems (NeurIPS 2015). 2503--2511."},{"key":"e_1_3_2_2_96_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287598"},{"key":"e_1_3_2_2_97_1","volume-title":"Algorithmic accountability: A primer","author":"Data & Society","year":"2018","unstructured":"Data & Society . 2018. Algorithmic accountability: A primer . Data & Society ( 2018 ). Data & Society. 2018. Algorithmic accountability: A primer. Data & Society (2018)."},{"key":"e_1_3_2_2_98_1","volume-title":"Proceedings of the AAAI 2018 Spring Symposium: Designing the User Experience of Artificial Intelligence.","author":"Springer Aaron","year":"2018","unstructured":"Aaron Springer , J. Garcia-Gathright , and Henriette Cramer . 2018 . Assessing and addressing algorithmic bias-But before we get there . In Proceedings of the AAAI 2018 Spring Symposium: Designing the User Experience of Artificial Intelligence. Aaron Springer, J. Garcia-Gathright, and Henriette Cramer. 2018. Assessing and addressing algorithmic bias-But before we get there. In Proceedings of the AAAI 2018 Spring Symposium: Designing the User Experience of Artificial Intelligence."},{"key":"e_1_3_2_2_99_1","volume-title":"Pasteur's quadrant: Basic science and technological innovation","author":"Stokes Donald E","unstructured":"Donald E Stokes . 1997. Pasteur's quadrant: Basic science and technological innovation . Brookings Institution Press . Donald E Stokes. 1997. Pasteur's quadrant: Basic science and technological innovation. Brookings Institution Press."},{"key":"e_1_3_2_2_100_1","unstructured":"Sarah Tan Julius Adebayo Kori Inkpen and Ece Kamar. 2018. Investigating Human+Machine Complementarity for Recidivism Predictions. CoRR arXiv:1808.09123.  Sarah Tan Julius Adebayo Kori Inkpen and Ece Kamar. 2018. Investigating Human+Machine Complementarity for Recidivism Predictions. CoRR arXiv:1808.09123."},{"key":"e_1_3_2_2_101_1","unstructured":"Rob Thubron. 2018. IBM secretly used NYPD CCTV footage to train its facial recognition systems. https:\/\/www.techspot.com\/news\/76323ibm-secretly-used-nypd-cctv-footage-train-facial.html. Accessed: 2018-09--16.  Rob Thubron. 2018. IBM secretly used NYPD CCTV footage to train its facial recognition systems. https:\/\/www.techspot.com\/news\/76323ibm-secretly-used-nypd-cctv-footage-train-facial.html. Accessed: 2018-09--16."},{"key":"e_1_3_2_2_102_1","doi-asserted-by":"publisher","DOI":"10.1080\/02681102.2017.1310713"},{"key":"e_1_3_2_2_103_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025811"},{"key":"e_1_3_2_2_104_1","first-page":"1","article-title":"Making better use of the crowd","volume":"18","author":"Vaughan Jennifer Wortman","year":"2018","unstructured":"Jennifer Wortman Vaughan . 2018 . Making better use of the crowd . Journal of Machine Learning Research 18 , 193 (2018), 1 -- 46 . Jennifer Wortman Vaughan. 2018. Making better use of the crowd. Journal of Machine Learning Research 18, 193 (2018), 1--46.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_2_2_105_1","doi-asserted-by":"publisher","DOI":"10.1177\/2053951717743530"},{"key":"e_1_3_2_2_106_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174014"},{"key":"e_1_3_2_2_107_1","volume-title":"AI recruiting tools do not eliminate bias","author":"Wachter-Boettcher Sara","unstructured":"Sara Wachter-Boettcher . 2017. AI recruiting tools do not eliminate bias . http:\/\/time.com\/4993431\/ai-recruiting-tools-do-not-eliminatebias. Accessed: 2018-09-01. Sara Wachter-Boettcher. 2017. AI recruiting tools do not eliminate bias. http:\/\/time.com\/4993431\/ai-recruiting-tools-do-not-eliminatebias. Accessed: 2018-09-01."},{"key":"e_1_3_2_2_108_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174230"},{"key":"e_1_3_2_2_109_1","volume-title":"Proceedings of the AAAI 2018 Spring Symposium: Designing the User Experience of Artificial Intelligence.","author":"Yang Qian","year":"2018","unstructured":"Qian Yang . 2018 . Machine learning as a UX design material: How can we imagine beyond automation, recommenders, and reminders? . In Proceedings of the AAAI 2018 Spring Symposium: Designing the User Experience of Artificial Intelligence. Qian Yang. 2018. Machine learning as a UX design material: How can we imagine beyond automation, recommenders, and reminders?. In Proceedings of the AAAI 2018 Spring Symposium: Designing the User Experience of Artificial Intelligence."},{"key":"e_1_3_2_2_110_1","doi-asserted-by":"publisher","DOI":"10.1145\/3196709.3196729"},{"key":"e_1_3_2_2_111_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858373"},{"key":"e_1_3_2_2_112_1","unstructured":"Maggie Zhang. 2015. Google photos tags two AfricanAmericans as gorillas through facial recognition software. https:\/\/tinyurl.com\/Forbes-2015-07-01. Accessed: 2018-07--12.  Maggie Zhang. 2015. Google photos tags two AfricanAmericans as gorillas through facial recognition software. https:\/\/tinyurl.com\/Forbes-2015-07-01. Accessed: 2018-07--12."},{"key":"e_1_3_2_2_113_1","volume-title":"Proceedings of the Seventeenth International Conference on Autonomous Agents and Multiagent Systems (AAMAS","author":"Zhao Zian","year":"2018","unstructured":"Zian Zhao , Michael Madaio , Florian Pecune , Yoichi Matsuyama , and Justine Cassell . 2018 . Socially-conditioned task reasoning for a virtual tutoring agent . In Proceedings of the Seventeenth International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2018). 2265--2267. Zian Zhao, Michael Madaio, Florian Pecune, Yoichi Matsuyama, and Justine Cassell. 2018. Socially-conditioned task reasoning for a virtual tutoring agent. In Proceedings of the Seventeenth International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2018). 2265--2267."}],"event":{"name":"CHI '19: CHI Conference on Human Factors in Computing Systems","location":"Glasgow Scotland Uk","acronym":"CHI '19","sponsor":["SIGCHI ACM Special Interest Group on Computer-Human Interaction"]},"container-title":["Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3290605.3300830","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3290605.3300830","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:53:10Z","timestamp":1750204390000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3290605.3300830"}},"subtitle":["What Do Industry Practitioners Need?"],"short-title":[],"issued":{"date-parts":[[2019,5,2]]},"references-count":113,"alternative-id":["10.1145\/3290605.3300830","10.1145\/3290605"],"URL":"https:\/\/doi.org\/10.1145\/3290605.3300830","relation":{},"subject":[],"published":{"date-parts":[[2019,5,2]]},"assertion":[{"value":"2019-05-02","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}