{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T22:17:15Z","timestamp":1776118635081,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":94,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,10,5]],"date-time":"2021-10-05T00:00:00Z","timestamp":1633392000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-sa\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,10,5]]},"DOI":"10.1145\/3465416.3483294","type":"proceedings-article","created":{"date-parts":[[2021,11,4]],"date-time":"2021-11-04T22:07:16Z","timestamp":1636063636000},"page":"1-9","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":41,"title":["Algorithmic Auditing and Social Justice: Lessons from the History of Audit Studies"],"prefix":"10.1145","author":[{"given":"Briana","family":"Vecchione","sequence":"first","affiliation":[{"name":"Cornell University, USA"}]},{"given":"Karen","family":"Levy","sequence":"additional","affiliation":[{"name":"Cornell University, USA"}]},{"given":"Solon","family":"Barocas","sequence":"additional","affiliation":[{"name":"Microsoft Research \/ Cornell University, USA"}]}],"member":"320","published-online":{"date-parts":[[2021,11,4]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"Ada Lovelace Institute. 2020. Examining the Black Box. https:\/\/www.adalovelaceinstitute.org\/report\/examining-the-black-box-tools-for-assessing-algorithmic-systems\/.  Ada Lovelace Institute. 2020. Examining the Black Box. https:\/\/www.adalovelaceinstitute.org\/report\/examining-the-black-box-tools-for-assessing-algorithmic-systems\/."},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10115-017-1116-3"},{"key":"e_1_3_2_1_3_1","volume-title":"Biased Outcomes. Proceedings of the ACM on Human-Computer Interaction 3, CSCW(2019)","author":"Ali Muhammad","year":"2019","unstructured":"Muhammad Ali , Piotr Sapiezynski , Miranda Bogen , Aleksandra Korolova , Alan Mislove , and Aaron Rieke . 2019 . Discrimination through Optimization: How Facebook\u2019s Ad Delivery can Lead to Biased Outcomes. Proceedings of the ACM on Human-Computer Interaction 3, CSCW(2019) , 1\u201330. Muhammad Ali, Piotr Sapiezynski, Miranda Bogen, Aleksandra Korolova, Alan Mislove, and Aaron Rieke. 2019. Discrimination through Optimization: How Facebook\u2019s Ad Delivery can Lead to Biased Outcomes. Proceedings of the ACM on Human-Computer Interaction 3, CSCW(2019), 1\u201330."},{"key":"e_1_3_2_1_4_1","unstructured":"Julia Angwin Jeff Larson Surya Mattu and Lauren Kirchner. 2016. Machine Bias. https:\/\/www.propublica.org\/article\/machine-bias-risk-assessments-in-criminal-sentencing.  Julia Angwin Jeff Larson Surya Mattu and Lauren Kirchner. 2016. Machine Bias. https:\/\/www.propublica.org\/article\/machine-bias-risk-assessments-in-criminal-sentencing."},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1609\/icwsm.v14i1.7276"},{"key":"e_1_3_2_1_7_1","unstructured":"Jef Ausloos and Michael Veale. 2021. Researching with Data Rights. 136\u2013157\u00a0pages.  Jef Ausloos and Michael Veale. 2021. Researching with Data Rights. 136\u2013157\u00a0pages."},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-soc-073014-112445"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3449148"},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"crossref","unstructured":"Solon Barocas Anhong Guo Ece Kamar Jacquelyn Krones Meredith\u00a0Ringel Morris Jennifer\u00a0Wortman Vaughan Duncan Wadsworth and Hanna Wallach. 2021. Designing Disaggregated Evaluations of AI Systems: Choices Considerations and Tradeoffs. arXiv:arXiv:2103.06076  Solon Barocas Anhong Guo Ece Kamar Jacquelyn Krones Meredith\u00a0Ringel Morris Jennifer\u00a0Wortman Vaughan Duncan Wadsworth and Hanna Wallach. 2021. Designing Disaggregated Evaluations of AI Systems: Choices Considerations and Tradeoffs. arXiv:arXiv:2103.06076","DOI":"10.1145\/3461702.3462610"},{"key":"e_1_3_2_1_11_1","first-page":"2017","article-title":"Fairness in Machine Learning","volume":"1","author":"Barocas Solon","year":"2017","unstructured":"Solon Barocas , Moritz Hardt , and Arvind Narayanan . 2017 . Fairness in Machine Learning . Nips Tutorial 1 (2017), 2017 . Solon Barocas, Moritz Hardt, and Arvind Narayanan. 2017. Fairness in Machine Learning. Nips Tutorial 1(2017), 2017.","journal-title":"Nips Tutorial"},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1540-4560.2012.01747.x"},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1016\/bs.hefe.2016.08.004"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1257\/0002828042002561"},{"key":"e_1_3_2_1_15_1","volume-title":"Audit Studies: Behind the Scenes with Theory, Method, and Nuance","author":"Besbris Max","unstructured":"Max Besbris , Jacob\u00a0William Faber , Peter Rich , and Patrick Sharkey . 2018. The Geography of Stigma: Experimental Methods to Identify the Penalty of Place . In Audit Studies: Behind the Scenes with Theory, Method, and Nuance . Springer , Berlin, Germany , 159\u2013177. Max Besbris, Jacob\u00a0William Faber, Peter Rich, and Patrick Sharkey. 2018. The Geography of Stigma: Experimental Methods to Identify the Penalty of Place. In Audit Studies: Behind the Scenes with Theory, Method, and Nuance. Springer, Berlin, Germany, 159\u2013177."},{"key":"e_1_3_2_1_16_1","volume-title":"Help Wanted: An Examination of Hiring Algorithms, Equity, and Bias. https:\/\/www.upturn.org\/reports\/2018\/hiring-algorithms\/.","author":"Bogen Miranda","year":"2018","unstructured":"Miranda Bogen and Aaron Rieke . 2018 . Help Wanted: An Examination of Hiring Algorithms, Equity, and Bias. https:\/\/www.upturn.org\/reports\/2018\/hiring-algorithms\/. Miranda Bogen and Aaron Rieke. 2018. Help Wanted: An Examination of Hiring Algorithms, Equity, and Bias. https:\/\/www.upturn.org\/reports\/2018\/hiring-algorithms\/."},{"key":"e_1_3_2_1_17_1","unstructured":"Danah Boyd Karen Levy and Alice Marwick. 2014. The Networked Nature of Algorithmic Discrimination. 53\u201357\u00a0pages.  Danah Boyd Karen Levy and Alice Marwick. 2014. The Networked Nature of Algorithmic Discrimination. 53\u201357\u00a0pages."},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1177\/2053951720983865"},{"key":"e_1_3_2_1_19_1","unstructured":"Joy Buolamwini. 2018. When the Robot Doesn\u2019t See Dark Skin. https:\/\/www.nytimes.com\/2018\/06\/21\/opinion\/facial-analysis-technology-bias.html.  Joy Buolamwini. 2018. When the Robot Doesn\u2019t See Dark Skin. https:\/\/www.nytimes.com\/2018\/06\/21\/opinion\/facial-analysis-technology-bias.html."},{"key":"e_1_3_2_1_20_1","volume-title":"Conference on Fairness, Accountability and Transparency. ACM","author":"Buolamwini Joy","year":"2018","unstructured":"Joy Buolamwini and Timnit Gebru . 2018 . Gender shades: Intersectional Accuracy Disparities in Commercial Gender Classification . In Conference on Fairness, Accountability and Transparency. ACM , New York, NY, 77\u201391. Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In Conference on Fairness, Accountability and Transparency. ACM, New York, NY, 77\u201391."},{"key":"e_1_3_2_1_21_1","volume-title":"Audit Studies: Behind the Scenes with Theory, Method, and Nuance","author":"Cherry Frances","unstructured":"Frances Cherry and Marc Bendick . 2018. Making it count: Discrimination Auditing and the Activist Scholar Tradition . In Audit Studies: Behind the Scenes with Theory, Method, and Nuance . Springer , Berlin, Germany , 45\u201362. Frances Cherry and Marc Bendick. 2018. Making it count: Discrimination Auditing and the Activist Scholar Tradition. In Audit Studies: Behind the Scenes with Theory, Method, and Nuance. Springer, Berlin, Germany, 45\u201362."},{"key":"e_1_3_2_1_22_1","unstructured":"Rumman Chowdhury and Jutta Williams. 2021. Introducing Twitter\u2019s First Algorithmic Bias Bounty Challenge. https:\/\/blog.twitter.com\/engineering\/en_us\/topics\/insights\/2021\/algorithmic-bias-bounty-challenge.  Rumman Chowdhury and Jutta Williams. 2021. Introducing Twitter\u2019s First Algorithmic Bias Bounty Challenge. https:\/\/blog.twitter.com\/engineering\/en_us\/topics\/insights\/2021\/algorithmic-bias-bounty-challenge."},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.15195\/v5.a2"},{"key":"e_1_3_2_1_24_1","unstructured":"Jeffrey Dastin. 2018. Amazon Scraps Secret AI Recruiting Tool that Showed Bias against Women. https:\/\/www.reuters.com\/article\/us-amazon-com-jobs-automation-insight-idUSKCN1MK08G.  Jeffrey Dastin. 2018. Amazon Scraps Secret AI Recruiting Tool that Showed Bias against Women. https:\/\/www.reuters.com\/article\/us-amazon-com-jobs-automation-insight-idUSKCN1MK08G."},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1515\/popets-2015-0007"},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1080\/21670811.2014.976411"},{"key":"e_1_3_2_1_27_1","volume-title":"Data Feminism","author":"D\u2019Ignazio Catherine","unstructured":"Catherine D\u2019Ignazio and Lauren\u00a0 F Klein . 2020. Data Feminism . MIT Press , Cambridge, MA . Catherine D\u2019Ignazio and Lauren\u00a0F Klein. 2020. Data Feminism. MIT Press, Cambridge, MA."},{"key":"e_1_3_2_1_28_1","unstructured":"Melanie Ehrenkranz. 2019. Amazon\u2019s Face Recognition Tech Once Again Pegs Politicians as Criminals. https:\/\/gizmodo.com\/amazons-face-recognition-tech-once-again-pegs-politicia-1837215790.  Melanie Ehrenkranz. 2019. Amazon\u2019s Face Recognition Tech Once Again Pegs Politicians as Criminals. https:\/\/gizmodo.com\/amazons-face-recognition-tech-once-again-pegs-politicia-1837215790."},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.5465\/amj.2007.24160888"},{"key":"e_1_3_2_1_30_1","unstructured":"Amit Elazari Bar\u00a0On. 2018. We Need Bug Bounties for Bad Algorithms. https:\/\/www.vice.com\/en\/article\/8xkyj3\/we-need-bug-bounties-for-bad-algorithms.  Amit Elazari Bar\u00a0On. 2018. We Need Bug Bounties for Bad Algorithms. https:\/\/www.vice.com\/en\/article\/8xkyj3\/we-need-bug-bounties-for-bad-algorithms."},{"key":"e_1_3_2_1_31_1","volume-title":"Proceedings of the International AAAI Conference on Web and Social Media, Vol.\u00a011","author":"Eslami Motahhare","year":"2017","unstructured":"Motahhare Eslami , Kristen Vaccaro , Karrie Karahalios , and Kevin Hamilton . 2017 . \u201c Be Careful; Things can be Worse than they Appear\u201d: Understanding Biased Algorithms and Users\u2019 Behavior around Them in Rating Platforms . In Proceedings of the International AAAI Conference on Web and Social Media, Vol.\u00a011 . AAAI Press, Palo Alto, California, 62\u201371. Motahhare Eslami, Kristen Vaccaro, Karrie Karahalios, and Kevin Hamilton. 2017. \u201cBe Careful; Things can be Worse than they Appear\u201d: Understanding Biased Algorithms and Users\u2019 Behavior around Them in Rating Platforms. In Proceedings of the International AAAI Conference on Web and Social Media, Vol.\u00a011. AAAI Press, Palo Alto, California, 62\u201371."},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.15195\/v4.a19"},{"key":"e_1_3_2_1_33_1","volume-title":"An Introduction to Audit Studies in the Social Sciences","author":"Gaddis S\u00a0Michael","unstructured":"S\u00a0Michael Gaddis . 2018. An Introduction to Audit Studies in the Social Sciences . Springer , Berlin, Germany . 3\u201344 pages. S\u00a0Michael Gaddis. 2018. An Introduction to Audit Studies in the Social Sciences. Springer, Berlin, Germany. 3\u201344 pages."},{"key":"e_1_3_2_1_34_1","unstructured":"Clare Garvie. 2019. Garbage In Garbage Out. https:\/\/www.flawedfacedata.com\/.  Clare Garvie. 2019. Garbage In Garbage Out. https:\/\/www.flawedfacedata.com\/."},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1177\/1065912917749322"},{"key":"e_1_3_2_1_36_1","volume-title":"Getting a Job: A Study of Contacts and Careers","author":"Granovetter Mark","unstructured":"Mark Granovetter . 1995. Getting a Job: A Study of Contacts and Careers . University of Chicago Press , Chicago, IL . Mark Granovetter. 1995. Getting a Job: A Study of Contacts and Careers. University of Chicago Press, Chicago, IL."},{"key":"e_1_3_2_1_37_1","first-page":"2936","article-title":"A Survey of Accuracy Evaluation Metrics of Recommendation Tasks","volume":"10","author":"Gunawardana Asela","year":"2009","unstructured":"Asela Gunawardana and Guy Shani . 2009 . A Survey of Accuracy Evaluation Metrics of Recommendation Tasks . Journal of Machine Learning Research 10 , 12 (2009), 2936 \u2013 2962 . Asela Gunawardana and Guy Shani. 2009. A Survey of Accuracy Evaluation Metrics of Recommendation Tasks. Journal of Machine Learning Research 10, 12 (2009), 2936\u20132962.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1111\/ecoj.12080"},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998327"},{"key":"e_1_3_2_1_40_1","unstructured":"David\u00a0Heinemeier Hansson. 2019. The AppleCard is such a fucking sexist program. My wife and I filed joint tax returns live in a community-property state and have been married for a long time. Yet Apple\u2019s black box algorithm thinks I deserve 20x the credit limit she does. No appeals work.https:\/\/twitter.com\/dhh\/status\/1192540900393705474.  David\u00a0Heinemeier Hansson. 2019. The AppleCard is such a fucking sexist program. My wife and I filed joint tax returns live in a community-property state and have been married for a long time. Yet Apple\u2019s black box algorithm thinks I deserve 20x the credit limit she does. No appeals work.https:\/\/twitter.com\/dhh\/status\/1192540900393705474."},{"key":"e_1_3_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1080\/1369118X.2019.1573912"},{"key":"e_1_3_2_1_42_1","unstructured":"Lily Hu and Issa Kohler-Hausmann. 2020. What\u2019s Sex Got To Do With Fair Machine Learning. arXiv:arXiv:2006.01770  Lily Hu and Issa Kohler-Hausmann. 2020. What\u2019s Sex Got To Do With Fair Machine Learning. arXiv:arXiv:2006.01770"},{"key":"e_1_3_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3442381.3450077"},{"key":"e_1_3_2_1_44_1","unstructured":"Lucas Introna and Helen Nissenbaum. 2010. Facial Recognition Technology a Survey of Policy and Implementation Issues. 8\u201355\u00a0pages.  Lucas Introna and Helen Nissenbaum. 2010. Facial Recognition Technology a Survey of Policy and Implementation Issues. 8\u201355\u00a0pages."},{"key":"e_1_3_2_1_45_1","unstructured":"Jason Slotkin. 2020. Twitter Announces Changes To Image Cropping Amid Bias Concern. https:\/\/www.npr.org\/sections\/live-updates-protests-for-racial-justice\/2020\/10\/02\/919638417\/twitter-announces-changes-to-image-cropping-amid-bias-concern.  Jason Slotkin. 2020. Twitter Announces Changes To Image Cropping Amid Bias Concern. https:\/\/www.npr.org\/sections\/live-updates-protests-for-racial-justice\/2020\/10\/02\/919638417\/twitter-announces-changes-to-image-cropping-amid-bias-concern."},{"key":"e_1_3_2_1_46_1","first-page":"633","article-title":"Accountable Algorithms","volume":"165","author":"Kroll Joshua","year":"2017","unstructured":"Joshua Kroll , Joanna Huey , Solon Barocas , Edward Felten , Joel Reidenberg , David Robinson , and Harlan Yu . 2017 . Accountable Algorithms . University of Pennsylvania Law Review 165 , 3 (2017), 633 \u2013 705 . Joshua Kroll, Joanna Huey, Solon Barocas, Edward Felten, Joel Reidenberg, David Robinson, and Harlan Yu. 2017. Accountable Algorithms. University of Pennsylvania Law Review 165, 3 (2017), 633\u2013705.","journal-title":"University of Pennsylvania Law Review"},{"key":"e_1_3_2_1_47_1","doi-asserted-by":"crossref","unstructured":"Ren\u00e9 Mahieu and Jef Ausloos. 2020. Recognising and Enabling the Collective Dimension of the GDPR and the Right of Access.  Ren\u00e9 Mahieu and Jef Ausloos. 2020. Recognising and Enabling the Collective Dimension of the GDPR and the Right of Access.","DOI":"10.31228\/osf.io\/b5dwm"},{"key":"e_1_3_2_1_48_1","doi-asserted-by":"crossref","unstructured":"Manish Raghavan and Solon Barocas. 2019. Challenges for Mitigating Bias in Algorithmic Hiring. https:\/\/www.brookings.edu\/research\/challenges-for-mitigating-bias-in-algorithmic-hiring\/.  Manish Raghavan and Solon Barocas. 2019. Challenges for Mitigating Bias in Algorithmic Hiring. https:\/\/www.brookings.edu\/research\/challenges-for-mitigating-bias-in-algorithmic-hiring\/.","DOI":"10.1145\/3351095.3372828"},{"key":"e_1_3_2_1_49_1","unstructured":"The Markup. 2020. The Citizen Browser Project: Auditing the Algorithms of Disinformation. https:\/\/themarkup.org\/citizen-browser.  The Markup. 2020. The Citizen Browser Project: Auditing the Algorithms of Disinformation. https:\/\/themarkup.org\/citizen-browser."},{"key":"e_1_3_2_1_50_1","unstructured":"Emmanuel Martinez and Lauren Kirchner. 2021. The Secret Bias Hidden in Mortgage-Approval Algorithms. https:\/\/themarkup.org\/denied\/2021\/08\/25\/the-secret-bias-hidden-in-mortgage-approval-algorithms.  Emmanuel Martinez and Lauren Kirchner. 2021. The Secret Bias Hidden in Mortgage-Approval Algorithms. https:\/\/themarkup.org\/denied\/2021\/08\/25\/the-secret-bias-hidden-in-mortgage-approval-algorithms."},{"key":"e_1_3_2_1_51_1","volume-title":"Participatory Action Research","author":"McIntyre Alice","unstructured":"Alice McIntyre . 2007. Participatory Action Research . Sage Publications , New York, NY. Alice McIntyre. 2007. Participatory Action Research. Sage Publications, New York, NY."},{"key":"e_1_3_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1146\/annurev.soc.27.1.415"},{"key":"e_1_3_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1177\/2378023115621316"},{"key":"e_1_3_2_1_54_1","first-page":"4991","article-title":"Auditing for Transparency in Content Personalization Systems","volume":"10","author":"Mittelstadt Brent","year":"2016","unstructured":"Brent Mittelstadt . 2016 . Auditing for Transparency in Content Personalization Systems . International Journal of Communication 10 (2016), 4991 \u2013 5002 . Brent Mittelstadt. 2016. Auditing for Transparency in Content Personalization Systems. International Journal of Communication 10 (2016), 4991\u20135002.","journal-title":"International Journal of Communication"},{"key":"e_1_3_2_1_55_1","unstructured":"Shahien Nasiripour and Greg Farrell. 2021. Goldman Cleared of Bias in New York Review of Apple Card. https:\/\/www.bloomberg.com\/news\/articles\/2021-03-23\/goldman-didn-t-discriminate-with-apple-card-n-y-regulator-says.  Shahien Nasiripour and Greg Farrell. 2021. Goldman Cleared of Bias in New York Review of Apple Card. https:\/\/www.bloomberg.com\/news\/articles\/2021-03-23\/goldman-didn-t-discriminate-with-apple-card-n-y-regulator-says."},{"key":"e_1_3_2_1_56_1","unstructured":"Alfred Ng. 2021. Can Auditing Eliminate Bias from Algorithms?https:\/\/themarkup.org\/ask-the-markup\/2021\/02\/23\/can-auditing-eliminate-bias-from-algorithms  Alfred Ng. 2021. Can Auditing Eliminate Bias from Algorithms?https:\/\/themarkup.org\/ask-the-markup\/2021\/02\/23\/can-auditing-eliminate-bias-from-algorithms"},{"key":"e_1_3_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1177\/0002716206294796"},{"key":"e_1_3_2_1_58_1","unstructured":"Parag Agrawal and Dantley Davis. 2020. Transparency around Image Cropping and Changes to Come. https:\/\/blog.twitter.com\/official\/en_us\/topics\/product\/2020\/transparency-image-cropping.html.  Parag Agrawal and Dantley Davis. 2020. Transparency around Image Cropping and Changes to Come. https:\/\/blog.twitter.com\/official\/en_us\/topics\/product\/2020\/transparency-image-cropping.html."},{"key":"e_1_3_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1177\/0003122416630982"},{"key":"e_1_3_2_1_60_1","volume-title":"Audit Studies: Behind the Scenes with Theory, Method, and Nuance","author":"Pedulla S","unstructured":"David\u00a0 S Pedulla . 2018. Emerging Frontiers in Audit Study Research: Mechanisms, Variation, and Representativeness . In Audit Studies: Behind the Scenes with Theory, Method, and Nuance . Springer , New York, NY , 179\u2013195. David\u00a0S Pedulla. 2018. Emerging Frontiers in Audit Study Research: Mechanisms, Variation, and Representativeness. In Audit Studies: Behind the Scenes with Theory, Method, and Nuance. Springer, New York, NY, 179\u2013195."},{"key":"e_1_3_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1093\/sf\/soaa026"},{"key":"e_1_3_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372828"},{"key":"e_1_3_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/3375627.3375820"},{"key":"e_1_3_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372873"},{"key":"e_1_3_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1177\/1356389016683871"},{"key":"e_1_3_2_1_66_1","volume-title":"Essential Work: Analyzing the Hiring Technologies of Large Hourly Employers. https:\/\/www.upturn.org\/reports\/2021\/essential-work\/.","author":"Rieke Aaron","year":"2021","unstructured":"Aaron Rieke , Urmila Janardan , Mingwei Hsu , and Natasha Duarte . 2021 . Essential Work: Analyzing the Hiring Technologies of Large Hourly Employers. https:\/\/www.upturn.org\/reports\/2021\/essential-work\/. Aaron Rieke, Urmila Janardan, Mingwei Hsu, and Natasha Duarte. 2021. Essential Work: Analyzing the Hiring Technologies of Large Hourly Employers. https:\/\/www.upturn.org\/reports\/2021\/essential-work\/."},{"key":"e_1_3_2_1_67_1","volume-title":"Pedigree: How Elite Students get Elite Jobs","author":"Rivera A","year":"2016","unstructured":"Lauren\u00a0 A Rivera . 2016 . Pedigree: How Elite Students get Elite Jobs . Princeton University Press , Princeton, NJ . Lauren\u00a0A Rivera. 2016. Pedigree: How Elite Students get Elite Jobs. Princeton University Press, Princeton, NJ."},{"key":"e_1_3_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274417"},{"key":"e_1_3_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445557"},{"key":"e_1_3_2_1_70_1","volume-title":"Bit by Bit: Social Research in the Digital Age","author":"Salganik J","unstructured":"Matthew\u00a0 J Salganik . 2019. Bit by Bit: Social Research in the Digital Age . Princeton University Press , Princeton, NJ . Matthew\u00a0J Salganik. 2019. Bit by Bit: Social Research in the Digital Age. Princeton University Press, Princeton, NJ."},{"key":"e_1_3_2_1_71_1","first-page":"4349","article-title":"Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms. Data and discrimination","volume":"22","author":"Sandvig Christian","year":"2014","unstructured":"Christian Sandvig , Kevin Hamilton , Karrie Karahalios , and Cedric Langbort . 2014 . Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms. Data and discrimination : Converting Critical Concerns into Productive Inquiry 22 (2014), 4349 \u2013 4357 . Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. 2014. Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms. Data and discrimination: Converting Critical Concerns into Productive Inquiry 22(2014), 4349\u20134357.","journal-title":"Converting Critical Concerns into Productive Inquiry"},{"key":"e_1_3_2_1_72_1","doi-asserted-by":"crossref","unstructured":"Devansh Saxena Karla Badillo-Urquiola Pamela Wisniewski and Shion Guha. 2021. A Framework of High-Stakes Algorithmic Decision-Making for the Public Sector Developed through a Case Study of Child-Welfare. arXiv:arXiv:2107.03487  Devansh Saxena Karla Badillo-Urquiola Pamela Wisniewski and Shion Guha. 2021. A Framework of High-Stakes Algorithmic Decision-Making for the Public Sector Developed through a Case Study of Child-Welfare. arXiv:arXiv:2107.03487","DOI":"10.1145\/3476089"},{"key":"e_1_3_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287598"},{"key":"e_1_3_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-polisci-032015-010015"},{"key":"e_1_3_2_1_75_1","unstructured":"Mona Sloane. 2021. The Algorithmic Auditing Trap. https:\/\/onezero.medium.com\/the-algorithmic-auditing-trap-9a6f2d4d461d.  Mona Sloane. 2021. The Algorithmic Auditing Trap. https:\/\/onezero.medium.com\/the-algorithmic-auditing-trap-9a6f2d4d461d."},{"key":"e_1_3_2_1_76_1","doi-asserted-by":"crossref","unstructured":"Mona Sloane Emanuel Moss and Rumman Chowdhury. 2021. A Silicon Valley Love Triangle: Hiring Algorithms Pseudo-Science and the Quest for Auditability. arXiv:arXiv:2106.12403  Mona Sloane Emanuel Moss and Rumman Chowdhury. 2021. A Silicon Valley Love Triangle: Hiring Algorithms Pseudo-Science and the Quest for Auditability. arXiv:arXiv:2106.12403","DOI":"10.1016\/j.patter.2021.100425"},{"key":"e_1_3_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1257\/jep.34.2.49"},{"key":"e_1_3_2_1_78_1","volume-title":"Amazon\u2019s Face Recognition Falsely Matched 28 Members of Congress With Mugshots. https:\/\/www.aclu.org\/blog\/privacy-technology\/surveillance-technologies\/amazons-face-recognition-falsely-matched-28","author":"Snow Jacob","year":"2018","unstructured":"Jacob Snow . 2018 . Amazon\u2019s Face Recognition Falsely Matched 28 Members of Congress With Mugshots. https:\/\/www.aclu.org\/blog\/privacy-technology\/surveillance-technologies\/amazons-face-recognition-falsely-matched-28 . Jacob Snow. 2018. Amazon\u2019s Face Recognition Falsely Matched 28 Members of Congress With Mugshots. https:\/\/www.aclu.org\/blog\/privacy-technology\/surveillance-technologies\/amazons-face-recognition-falsely-matched-28."},{"key":"e_1_3_2_1_79_1","first-page":"303","article-title":"Assessing Risk Assessment in Action","volume":"103","author":"Stevenson Megan","year":"2018","unstructured":"Megan Stevenson . 2018 . Assessing Risk Assessment in Action . Minnesota Law Review 103 (2018), 303 . Megan Stevenson. 2018. Assessing Risk Assessment in Action. Minnesota Law Review 103(2018), 303.","journal-title":"Minnesota Law Review"},{"key":"e_1_3_2_1_81_1","volume-title":"Audit Cultures: Anthropological Studies in Accountability, Ethics, and the Academy","author":"Strathern Marilyn","year":"2000","unstructured":"Marilyn Strathern . 2000 . Audit Cultures: Anthropological Studies in Accountability, Ethics, and the Academy . Psychology Press , Hove, East Sussex, UK. Marilyn Strathern. 2000. Audit Cultures: Anthropological Studies in Accountability, Ethics, and the Academy. Psychology Press, Hove, East Sussex, UK."},{"key":"e_1_3_2_1_82_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.biocon.2013.11.003"},{"key":"e_1_3_2_1_83_1","doi-asserted-by":"publisher","DOI":"10.1145\/2447976.2447990"},{"key":"e_1_3_2_1_84_1","unstructured":"Taylor Telford. 2019. Apple Card Algorithm Sparks Gender Bias Allegations against Goldman Sachs. https:\/\/www.washingtonpost.com\/business\/2019\/11\/11\/apple-card-algorithm-sparks-gender-bias-allegations-against-goldman-sachs\/.  Taylor Telford. 2019. Apple Card Algorithm Sparks Gender Bias Allegations against Goldman Sachs. https:\/\/www.washingtonpost.com\/business\/2019\/11\/11\/apple-card-algorithm-sparks-gender-bias-allegations-against-goldman-sachs\/."},{"key":"e_1_3_2_1_85_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1751-9020.2010.00353.x"},{"key":"e_1_3_2_1_86_1","doi-asserted-by":"publisher","DOI":"10.1177\/0049124115570066"},{"key":"e_1_3_2_1_87_1","volume-title":"Audit Studies: Behind the Scenes with Theory, Method, and Nuance","author":"Vuolo Mike","unstructured":"Mike Vuolo , Christopher Uggen , and Sarah Lageson . 2018. To Match or Not to Match? Statistical and Substantive Considerations in Audit Design and Analysis . In Audit Studies: Behind the Scenes with Theory, Method, and Nuance . Springer , New York, NY , 119\u2013140. Mike Vuolo, Christopher Uggen, and Sarah Lageson. 2018. To Match or Not to Match? Statistical and Substantive Considerations in Audit Design and Analysis. In Audit Studies: Behind the Scenes with Theory, Method, and Nuance. Springer, New York, NY, 119\u2013140."},{"key":"e_1_3_2_1_88_1","first-page":"773","article-title":"Privacy Law\u2019s False Promise","volume":"97","author":"Waldman Ari\u00a0Ezra","year":"2020","unstructured":"Ari\u00a0Ezra Waldman . 2020 . Privacy Law\u2019s False Promise . Washington University Law Review 97 , 3 (2020), 773 \u2013 834 . Ari\u00a0Ezra Waldman. 2020. Privacy Law\u2019s False Promise. Washington University Law Review 97, 3 (2020), 773\u2013834.","journal-title":"Washington University Law Review"},{"key":"e_1_3_2_1_89_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF01115015"},{"key":"e_1_3_2_1_90_1","doi-asserted-by":"publisher","DOI":"10.1145\/3442188.3445928"},{"key":"e_1_3_2_1_91_1","volume-title":"How to Conduct a Community Self-Survey of Civil Rights","author":"Wormser Margot\u00a0Haas","unstructured":"Margot\u00a0Haas Wormser . 1950. How to Conduct a Community Self-Survey of Civil Rights . Association Press , New York, NY . Margot\u00a0Haas Wormser. 1950. How to Conduct a Community Self-Survey of Civil Rights. Association Press, New York, NY."},{"key":"e_1_3_2_1_92_1","unstructured":"Kyra Yee and Irene\u00a0Font Peradejordi. 2021. Sharing Learnings from the First Algorithmic Bias Bounty Challenge. https:\/\/blog.twitter.com\/engineering\/en_us\/topics\/insights\/2021\/learnings-from-the-first-algorithmic-bias-bounty-challenge.  Kyra Yee and Irene\u00a0Font Peradejordi. 2021. Sharing Learnings from the First Algorithmic Bias Bounty Challenge. https:\/\/blog.twitter.com\/engineering\/en_us\/topics\/insights\/2021\/learnings-from-the-first-algorithmic-bias-bounty-challenge."},{"key":"e_1_3_2_1_93_1","doi-asserted-by":"crossref","unstructured":"Kyra Yee Uthaipon Tantipongpipat and Shubhanshu Mishra. 2021. Image Cropping on Twitter: Fairness Metrics their Limitations and the Importance of Representation Design and Agency. arXiv:arXiv:2105.08667  Kyra Yee Uthaipon Tantipongpipat and Shubhanshu Mishra. 2021. Image Cropping on Twitter: Fairness Metrics their Limitations and the Importance of Representation Design and Agency. arXiv:arXiv:2105.08667","DOI":"10.1145\/3479594"},{"key":"e_1_3_2_1_94_1","doi-asserted-by":"publisher","DOI":"10.1257\/jep.12.2.23"},{"key":"e_1_3_2_1_95_1","volume-title":"White Logic","author":"Zuberi Tukufu","unstructured":"Tukufu Zuberi and Eduardo Bonilla-Silva . 2008. White Logic , White Methods : Racism and Methodology. Rowman & Littlefield Publishers , Lanham, MD, Chapter\u00a01, 3\u201330. Tukufu Zuberi and Eduardo Bonilla-Silva. 2008. White Logic, White Methods: Racism and Methodology. Rowman & Littlefield Publishers, Lanham, MD, Chapter\u00a01, 3\u201330."},{"key":"e_1_3_2_1_96_1","volume-title":"White Logic","author":"Zuberi Tukufu","unstructured":"Tukufu Zuberi and Eduardo Bonilla-Silva . 2008. White Logic , White Methods : Racism and Methodology. Rowman & Littlefield Publishers , Lanham, MD. Tukufu Zuberi and Eduardo Bonilla-Silva. 2008. White Logic, White Methods: Racism and Methodology. Rowman & Littlefield Publishers, Lanham, MD."}],"event":{"name":"EAAMO '21: Equity and Access in Algorithms, Mechanisms, and Optimization","location":"-- NY USA","acronym":"EAAMO '21","sponsor":["SIGAI ACM Special Interest Group on Artificial Intelligence","SIGecom Special Interest Group on Economics and Computation"]},"container-title":["Equity and Access in Algorithms, Mechanisms, and Optimization"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3465416.3483294","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3465416.3483294","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:18:31Z","timestamp":1750191511000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3465416.3483294"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,10,5]]},"references-count":94,"alternative-id":["10.1145\/3465416.3483294","10.1145\/3465416"],"URL":"https:\/\/doi.org\/10.1145\/3465416.3483294","relation":{},"subject":[],"published":{"date-parts":[[2021,10,5]]},"assertion":[{"value":"2021-11-04","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}