{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,2]],"date-time":"2026-09-02T14:01:32Z","timestamp":1788357692449,"version":"build-2803163510"},"publisher-location":"New York, NY, USA","reference-count":88,"publisher":"ACM","license":[{"start":{"date-parts":[[2022,7,26]],"date-time":"2022-07-26T00:00:00Z","timestamp":1658793600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Mozilla Foundation"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,7,26]]},"DOI":"10.1145\/3514094.3534181","type":"proceedings-article","created":{"date-parts":[[2022,7,27]],"date-time":"2022-07-27T22:25:13Z","timestamp":1658960713000},"page":"557-571","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":102,"title":["Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance"],"prefix":"10.1145","author":[{"given":"Inioluwa Deborah","family":"Raji","sequence":"first","affiliation":[{"name":"University of California, Berkeley, Berkeley, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Peggy","family":"Xu","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Colleen","family":"Honigsberg","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Daniel","family":"Ho","sequence":"additional","affiliation":[{"name":"Stanford University, Stanford, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2022,7,27]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"2020. Sandvig v. Barr. 73 pages.  2020. Sandvig v. Barr. 73 pages."},{"key":"e_1_3_2_1_2_1","unstructured":"2020. Van Buren v. United States. 2667 pages.  2020. Van Buren v. United States. 2667 pages."},{"key":"e_1_3_2_1_3_1","volume-title":"https:\/\/themarkup.org\/series\/impact Last accessed","year":"2021","unstructured":"2021. Impact. https:\/\/themarkup.org\/series\/impact Last accessed 25 October 2021 . 2021. Impact. https:\/\/themarkup.org\/series\/impact Last accessed 25 October 2021."},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359301"},{"key":"e_1_3_2_1_5_1","volume-title":"ProPublica","author":"Angwin Julia","year":"2016","unstructured":"Julia Angwin , Jeff Larson , Surya Mattu , and Lauren Kirchner . 2016 . Machine bias . ProPublica , May 23, 2016 (2016), 139--159. Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2016. Machine bias. ProPublica, May 23, 2016 (2016), 139--159."},{"key":"e_1_3_2_1_6_1","volume-title":"Baier and Lisa Ahramjian","author":"Ann","year":"2020","unstructured":"Ann H. Baier and Lisa Ahramjian . 2020 . Organic Certification of Farms and Businesses Producing Agricultural Products ., 8 pages. Ann H. Baier and Lisa Ahramjian. 2020. Organic Certification of Farms and Businesses Producing Agricultural Products., 8 pages."},{"key":"e_1_3_2_1_7_1","volume-title":"Strengthening the financial reporting system: can audit committees deliver? International journal of auditing 14, 2","author":"B\u00e9dard Jean","year":"2010","unstructured":"Jean B\u00e9dard and Yves Gendron . 2010. Strengthening the financial reporting system: can audit committees deliver? International journal of auditing 14, 2 ( 2010 ), 174--210. Jean B\u00e9dard and Yves Gendron. 2010. Strengthening the financial reporting system: can audit committees deliver? International journal of auditing 14, 2 (2010), 174--210."},{"key":"e_1_3_2_1_8_1","volume-title":"Paris","author":"Boffo Riccardo","year":"2020","unstructured":"Riccardo Boffo and Robert Patalano . 2020. Esg investing: Practices, progress and challenges. \u00c9ditions OCDE , Paris ( 2020 ). Riccardo Boffo and Robert Patalano. 2020. Esg investing: Practices, progress and challenges. \u00c9ditions OCDE, Paris (2020)."},{"key":"e_1_3_2_1_9_1","unstructured":"Russell Brandom. 2021. Facebook shut down German research on Instagram algorithm researchers say.  Russell Brandom. 2021. Facebook shut down German research on Instagram algorithm researchers say."},{"key":"e_1_3_2_1_10_1","volume-title":"Conference on fairness, accountability and transparency. PMLR, 77--91","author":"Buolamwini Joy","year":"2018","unstructured":"Joy Buolamwini and Timnit Gebru . 2018 . Gender shades: Intersectional accuracy disparities in commercial gender classification . In Conference on fairness, accountability and transparency. PMLR, 77--91 . Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency. PMLR, 77--91."},{"key":"e_1_3_2_1_11_1","volume-title":"Europe, Canada, and Taiwan. Therapeutic innovation & regulatory science 52, 1","author":"Chen Yi-Jung","year":"2018","unstructured":"Yi-Jung Chen , Chi-Ming Chiou , Yu-Wen Huang , Pei-Weng Tu , Yung-Chuan Lee , and Chia-Hung Chien . 2018. A comparative study of medical device regulations: US , Europe, Canada, and Taiwan. Therapeutic innovation & regulatory science 52, 1 ( 2018 ), 62--69. Yi-Jung Chen, Chi-Ming Chiou, Yu-Wen Huang, Pei-Weng Tu, Yung-Chuan Lee, and Chia-Hung Chien. 2018. A comparative study of medical device regulations: US, Europe, Canada, and Taiwan. Therapeutic innovation & regulatory science 52, 1 (2018), 62--69."},{"key":"e_1_3_2_1_12_1","volume-title":"Conference on Fairness, Accountability and Transparency. PMLR, 134--148","author":"Chouldechova Alexandra","year":"2018","unstructured":"Alexandra Chouldechova , Diana Benavides-Prado , Oleksandr Fialko , and Rhema Vaithianathan . 2018 . A case study of algorithm-assisted decision making in child maltreatment hotline screening decisions . In Conference on Fairness, Accountability and Transparency. PMLR, 134--148 . Alexandra Chouldechova, Diana Benavides-Prado, Oleksandr Fialko, and Rhema Vaithianathan. 2018. A case study of algorithm-assisted decision making in child maltreatment hotline screening decisions. In Conference on Fairness, Accountability and Transparency. PMLR, 134--148."},{"key":"e_1_3_2_1_13_1","volume-title":"Introducing Twitter's first algorithmic bias bounty challenge. https:\/\/blog.twitter.com\/engineering\/en_us\/topics\/insights\/2021\/algorithmic-bias-bounty-challenge Last accessed","author":"Chowdhury Rumman","year":"2021","unstructured":"Rumman Chowdhury and Jutta Williams . 2021. Introducing Twitter's first algorithmic bias bounty challenge. https:\/\/blog.twitter.com\/engineering\/en_us\/topics\/insights\/2021\/algorithmic-bias-bounty-challenge Last accessed 2 November 2021 . Rumman Chowdhury and Jutta Williams. 2021. Introducing Twitter's first algorithmic bias bounty challenge. https:\/\/blog.twitter.com\/engineering\/en_us\/topics\/insights\/2021\/algorithmic-bias-bounty-challenge Last accessed 2 November 2021."},{"key":"e_1_3_2_1_14_1","unstructured":"Craig Cochran. 2015. ISO 9001: 2015 in plain English. Paton Professional.  Craig Cochran. 2015. ISO 9001: 2015 in plain English. Paton Professional."},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3442188.3445881"},{"key":"e_1_3_2_1_16_1","unstructured":"\u00c1ngel D\u00edaz and Laura Hecht. 2021. Double Standards in Social Media Content Moderation. (2021).  \u00c1ngel D\u00edaz and Laura Hecht. 2021. Double Standards in Social Media Content Moderation. (2021)."},{"key":"e_1_3_2_1_17_1","volume-title":"Improving the confectionery industry supply chain through second party audits. British Food Journal","author":"Djekic Ilija","year":"2016","unstructured":"Ilija Djekic , Sladjana Dragojlovic , Zoran Miloradovic , Snezana Miljkovic-Zivanovic , Marija Savic , and Vesna Kekic . 2016. Improving the confectionery industry supply chain through second party audits. British Food Journal ( 2016 ). Ilija Djekic, Sladjana Dragojlovic, Zoran Miloradovic, Snezana Miljkovic-Zivanovic, Marija Savic, and Vesna Kekic. 2016. Improving the confectionery industry supply chain through second party audits. British Food Journal (2016)."},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1007\/s43681-021-00052-5"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1745-6622.2012.00380.x"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1177\/0093854818811379"},{"key":"e_1_3_2_1_21_1","volume-title":"Independent auditors are struggling to hold AI companies accountable","author":"Engler Alex C","unstructured":"Alex C Engler . 2021. Independent auditors are struggling to hold AI companies accountable . FastCompany . Alex C Engler. 2021. Independent auditors are struggling to hold AI companies accountable. FastCompany."},{"key":"e_1_3_2_1_22_1","first-page":"800","article-title":"Algorithmic accountability in the administrative state","volume":"37","author":"Engstrom David Freeman","year":"2020","unstructured":"David Freeman Engstrom and Daniel E Ho . 2020 . Algorithmic accountability in the administrative state . Yale J. on Reg. 37 (2020), 800 . David Freeman Engstrom and Daniel E Ho. 2020. Algorithmic accountability in the administrative state. Yale J. on Reg. 37 (2020), 800.","journal-title":"Yale J. on Reg."},{"key":"e_1_3_2_1_23_1","volume-title":"Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961","author":"Fedus William","year":"2021","unstructured":"William Fedus , Barret Zoph , and Noam Shazeer . 2021. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961 ( 2021 ). William Fedus, Barret Zoph, and Noam Shazeer. 2021. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961 (2021)."},{"key":"e_1_3_2_1_24_1","unstructured":"Foxglove. 2020. Home Office says it will abandon its racist visa algorithm -- after we sued them. https:\/\/www.foxglove.org.uk\/2020\/08\/04\/home-office-says-itwill-abandon-its-racist-visa-algorithm-after-we-sued-them\/ Last accessed 25 October 2021.  Foxglove. 2020. Home Office says it will abandon its racist visa algorithm -- after we sued them. https:\/\/www.foxglove.org.uk\/2020\/08\/04\/home-office-says-itwill-abandon-its-racist-visa-algorithm-after-we-sued-them\/ Last accessed 25 October 2021."},{"key":"e_1_3_2_1_25_1","volume-title":"We put a stop to the A Level grading algorithm! https:\/\/www.foxglove.org.uk\/2020\/08\/17\/we-put-a-stop-to-the-a-level-gradingalgorithm\/ Last accessed","year":"2021","unstructured":"Foxglove. 2020. We put a stop to the A Level grading algorithm! https:\/\/www.foxglove.org.uk\/2020\/08\/17\/we-put-a-stop-to-the-a-level-gradingalgorithm\/ Last accessed 25 October 2021 . Foxglove. 2020. We put a stop to the A Level grading algorithm! https:\/\/www.foxglove.org.uk\/2020\/08\/17\/we-put-a-stop-to-the-a-level-gradingalgorithm\/ Last accessed 25 October 2021."},{"key":"e_1_3_2_1_26_1","volume-title":"Face recognition vendor test (fvrt): Part 3, demographic effects","author":"Grother Patrick","unstructured":"Patrick Grother , Mei Ngan , and Kayee Hanaoka . 2019. Face recognition vendor test (fvrt): Part 3, demographic effects . National Institute of Standards and Technology . Patrick Grother, Mei Ngan, and Kayee Hanaoka. 2019. Face recognition vendor test (fvrt): Part 3, demographic effects. National Institute of Standards and Technology."},{"key":"e_1_3_2_1_27_1","volume-title":"We read the paper that forced Timnit Gebru out of Google. Here's what it says. Retrieved January 21","author":"Hao Karen","year":"2020","unstructured":"Karen Hao . 2020. We read the paper that forced Timnit Gebru out of Google. Here's what it says. Retrieved January 21 ( 2020 ), 2021. Karen Hao. 2020. We read the paper that forced Timnit Gebru out of Google. Here's what it says. Retrieved January 21 (2020), 2021."},{"key":"e_1_3_2_1_28_1","unstructured":"Karen Hao. 2021. How Facebook got addicted to spreading misinformation.  Karen Hao. 2021. How Facebook got addicted to spreading misinformation."},{"key":"e_1_3_2_1_29_1","volume-title":"The ACLU Slammed A Facial Recognition Company That Scrapes Photos From Instagram And Facebook. Buzzfeed News","author":"Haskins Caroline","year":"2020","unstructured":"Caroline Haskins , Ryan Mac , and Logan McDonald . 2020. The ACLU Slammed A Facial Recognition Company That Scrapes Photos From Instagram And Facebook. Buzzfeed News ( 2020 ). Caroline Haskins, Ryan Mac, and Logan McDonald. 2020. The ACLU Slammed A Facial Recognition Company That Scrapes Photos From Instagram And Facebook. Buzzfeed News (2020)."},{"key":"e_1_3_2_1_30_1","unstructured":"Rebecca Heilweil. 2020. Big tech companies back away from selling facial recognition to police. That's progress. Recode. Available at: https:\/\/www.vox.com\/recode\/2020\/6\/10\/21287194\/amazon-microsoft-ibm-facialrecognition-moratorium-police (accessed 18 June 2020) (2020).  Rebecca Heilweil. 2020. Big tech companies back away from selling facial recognition to police. That's progress. Recode. Available at: https:\/\/www.vox.com\/recode\/2020\/6\/10\/21287194\/amazon-microsoft-ibm-facialrecognition-moratorium-police (accessed 18 June 2020) (2020)."},{"key":"e_1_3_2_1_31_1","unstructured":"Hirevue. 2021. HireVue Leads the Industry with Commitment to Transparent and Ethical Use of AI in Hiring.  Hirevue. 2021. HireVue Leads the Industry with Commitment to Transparent and Ethical Use of AI in Hiring."},{"key":"e_1_3_2_1_32_1","unstructured":"ICO. 2020. AI Auditing Framework.  ICO. 2020. AI Auditing Framework."},{"key":"e_1_3_2_1_33_1","volume-title":"Digital Gatekeepers. Texas Law Review 99","author":"Kadri Thomas","year":"2020","unstructured":"Thomas Kadri . 2020. Digital Gatekeepers. Texas Law Review 99 ( 2020 ). Thomas Kadri. 2020. Digital Gatekeepers. Texas Law Review 99 (2020)."},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372874"},{"key":"e_1_3_2_1_35_1","volume-title":"An Algorithmic Equity Toolkit for Technology Audits by Community Advocates and Activists. arXiv preprint arXiv:1912.02943","author":"Katell Michael","year":"2019","unstructured":"Michael Katell , Meg Young , Bernease Herman , Dharma Dailey , Aaron Tam , Vivian Guetler , Corinne Binz , Daniella Raz , and PM Krafft . 2019. An Algorithmic Equity Toolkit for Technology Audits by Community Advocates and Activists. arXiv preprint arXiv:1912.02943 ( 2019 ). Michael Katell, Meg Young, Bernease Herman, Dharma Dailey, Aaron Tam, Vivian Guetler, Corinne Binz, Daniella Raz, and PM Krafft. 2019. An Algorithmic Equity Toolkit for Technology Audits by Community Advocates and Activists. arXiv preprint arXiv:1912.02943 (2019)."},{"key":"e_1_3_2_1_36_1","volume-title":"Ethical algorithm design should guide technology regulation. Retrieved from Brookings: https:\/\/www.brookings.edu\/research\/ethical-algorithm-designshould-guide-technology-regulation","author":"Kearns Michael","year":"2020","unstructured":"Michael Kearns and Aaron Roth . 2020. Ethical algorithm design should guide technology regulation. Retrieved from Brookings: https:\/\/www.brookings.edu\/research\/ethical-algorithm-designshould-guide-technology-regulation ( 2020 ). Michael Kearns and Aaron Roth. 2020. Ethical algorithm design should guide technology regulation. Retrieved from Brookings: https:\/\/www.brookings.edu\/research\/ethical-algorithm-designshould-guide-technology-regulation (2020)."},{"key":"e_1_3_2_1_37_1","volume-title":"Analyzing active managers' commitment to esg: Evidence from united nations principles for responsible investment. Available at SSRN 3555984","author":"Kim Soohun","year":"2020","unstructured":"Soohun Kim and Aaron Yoon . 2020. Analyzing active managers' commitment to esg: Evidence from united nations principles for responsible investment. Available at SSRN 3555984 ( 2020 ). Soohun Kim and Aaron Yoon. 2020. Analyzing active managers' commitment to esg: Evidence from united nations principles for responsible investment. Available at SSRN 3555984 (2020)."},{"key":"e_1_3_2_1_38_1","volume-title":"Auditing digital platforms for discrimination in economic opportunity advertising. arXiv preprint arXiv:2008.09656","author":"Kingsley Sara","year":"2020","unstructured":"Sara Kingsley , ClaraWang, Alex Mikhalenko , Proteeti Sinha , and Chinmay Kulkarni . 2020. Auditing digital platforms for discrimination in economic opportunity advertising. arXiv preprint arXiv:2008.09656 ( 2020 ). Sara Kingsley, ClaraWang, Alex Mikhalenko, Proteeti Sinha, and Chinmay Kulkarni. 2020. Auditing digital platforms for discrimination in economic opportunity advertising. arXiv preprint arXiv:2008.09656 (2020)."},{"key":"e_1_3_2_1_39_1","volume-title":"Access Denied: Faulty Automated Background Checks Freeze Out Renters.","author":"Kirchner Lauren","year":"2020","unstructured":"Lauren Kirchner and Matthew Goldstein . 2020 . Access Denied: Faulty Automated Background Checks Freeze Out Renters. (2020). Lauren Kirchner and Matthew Goldstein. 2020. Access Denied: Faulty Automated Background Checks Freeze Out Renters. (2020)."},{"key":"e_1_3_2_1_40_1","volume-title":"Despite a Civil Rights Settlement. ProPublica. Retrieved December 12","author":"Kofman Ava","year":"2019","unstructured":"Ava Kofman and Ariana Tobin . 2019. Facebook Ads Can Still Discriminate Against Women and Older Workers , Despite a Civil Rights Settlement. ProPublica. Retrieved December 12 ( 2019 ), 2020. Ava Kofman and Ariana Tobin. 2019. Facebook Ads Can Still Discriminate Against Women and Older Workers, Despite a Civil Rights Settlement. ProPublica. Retrieved December 12 (2019), 2020."},{"key":"e_1_3_2_1_41_1","volume-title":"Despite a Civil Rights Settlement.","author":"Kofman Ava","year":"2021","unstructured":"Ava Kofman and Ariana Tobin . 2021. Facebook Ads Can Still Discriminate Against Women and Older Workers , Despite a Civil Rights Settlement. ( 2021 ). Ava Kofman and Ariana Tobin. 2021. Facebook Ads Can Still Discriminate Against Women and Older Workers, Despite a Civil Rights Settlement. (2021)."},{"key":"e_1_3_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1111\/jacf.12346"},{"key":"e_1_3_2_1_43_1","volume-title":"Sunny Consolvo, Joshua Mason, Elie Bursztein, Zakir Durumeric, Kurt Thomas, and Michael Bailey.","author":"Kumar Deepak","year":"2021","unstructured":"Deepak Kumar , Patrick Gage Kelley , Sunny Consolvo, Joshua Mason, Elie Bursztein, Zakir Durumeric, Kurt Thomas, and Michael Bailey. 2021 . Designing Toxic Content Classification for a Diversity of Perspectives . arXiv preprint arXiv:2106.04511 (2021). Deepak Kumar, Patrick Gage Kelley, Sunny Consolvo, Joshua Mason, Elie Bursztein, Zakir Durumeric, Kurt Thomas, and Michael Bailey. 2021. Designing Toxic Content Classification for a Diversity of Perspectives. arXiv preprint arXiv:2106.04511 (2021)."},{"key":"e_1_3_2_1_44_1","unstructured":"Issie Lapowsky. 2021. Why Facebook's data-sharing project ballooned into a 2-year debacle. (2021).  Issie Lapowsky. 2021. Why Facebook's data-sharing project ballooned into a 2-year debacle. (2021)."},{"key":"e_1_3_2_1_45_1","unstructured":"Mark Latonero and Aaina Agarwal. 2021. Human Rights Impact Assessments for AI: Learning from Facebook's Failure in Myanmar. (2021) 18.  Mark Latonero and Aaina Agarwal. 2021. Human Rights Impact Assessments for AI: Learning from Facebook's Failure in Myanmar. (2021) 18."},{"key":"e_1_3_2_1_46_1","unstructured":"Alexandra Levine. 2021. 'Chilling': Facial recognition firm Clearview AI hits watchdog groups with subpoenas.  Alexandra Levine. 2021. 'Chilling': Facial recognition firm Clearview AI hits watchdog groups with subpoenas."},{"key":"e_1_3_2_1_48_1","volume-title":"An Examination of the Algorithmic Accountability Act of","author":"MacCarthy Mark","year":"2019","unstructured":"Mark MacCarthy . 2019. An Examination of the Algorithmic Accountability Act of 2019 . Available at SSRN 3615731 (2019). Mark MacCarthy. 2019. An Examination of the Algorithmic Accountability Act of 2019. Available at SSRN 3615731 (2019)."},{"key":"e_1_3_2_1_49_1","unstructured":"Markey and Matsui. 2021. Senator Markey Rep. Matsui Introduce Legislation to Combat Harmful Algorithms and Create New Online Transparency Regime.  Markey and Matsui. 2021. Senator Markey Rep. Matsui Introduce Legislation to Combat Harmful Algorithms and Create New Online Transparency Regime."},{"key":"e_1_3_2_1_50_1","unstructured":"Roy Maurer. 2021. New York City to Require Bias Audits of AI-Type HR Technology.  Roy Maurer. 2021. New York City to Require Bias Audits of AI-Type HR Technology."},{"key":"e_1_3_2_1_51_1","volume-title":"The Tampa Bay Times 3","author":"Bedi KATHLEEN","year":"2020","unstructured":"KATHLEEN McGRORY and NEIL Bedi . 2020. Targeted. The Tampa Bay Times 3 ( 2020 ). KATHLEEN McGRORY and NEIL Bedi. 2020. Targeted. The Tampa Bay Times 3 (2020)."},{"key":"e_1_3_2_1_52_1","volume-title":"Does facebook still sell discriminatory ads. The Markup","author":"Merrill JB","year":"2020","unstructured":"JB Merrill . 2020. Does facebook still sell discriminatory ads. The Markup ( 2020 ). JB Merrill. 2020. Does facebook still sell discriminatory ads. The Markup (2020)."},{"key":"e_1_3_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/3442188.3445935"},{"key":"e_1_3_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-019-0114-4"},{"key":"e_1_3_2_1_55_1","volume-title":"Madeleine Clare Elish, and Jacob Metcalf","author":"Moss Emanuel","year":"2021","unstructured":"Emanuel Moss , Elizabeth AnneWatkins , Ranjit Singh , Madeleine Clare Elish, and Jacob Metcalf . 2021 . Assembling Accountability : Algorithmic Impact Assessment for the Public Interest . Available at SSRN 3877437 (2021). Emanuel Moss, Elizabeth AnneWatkins, Ranjit Singh, Madeleine Clare Elish, and Jacob Metcalf. 2021. Assembling Accountability: Algorithmic Impact Assessment for the Public Interest. Available at SSRN 3877437 (2021)."},{"key":"e_1_3_2_1_56_1","volume-title":"Dissecting racial bias in an algorithm used to manage the health of populations. Science 366, 6464","author":"Obermeyer Ziad","year":"2019","unstructured":"Ziad Obermeyer , Brian Powers , Christine Vogeli , and Sendhil Mullainathan . 2019. Dissecting racial bias in an algorithm used to manage the health of populations. Science 366, 6464 ( 2019 ), 447--453. Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. 2019. Dissecting racial bias in an algorithm used to manage the health of populations. Science 366, 6464 (2019), 447--453."},{"key":"e_1_3_2_1_57_1","unstructured":"Will Oremus. 2021. Facebook keeps researching its own harms - and burying the findings. (2021).  Will Oremus. 2021. Facebook keeps researching its own harms - and burying the findings. (2021)."},{"key":"e_1_3_2_1_58_1","unstructured":"Barbara Ortutay. 2021. Facebook shuts out NYU academics' research on political ads.  Barbara Ortutay. 2021. Facebook shuts out NYU academics' research on political ads."},{"key":"e_1_3_2_1_59_1","volume-title":"An ESG Reckoning Is Coming. Harvard Business Review","author":"O'Leary Michael","year":"2021","unstructured":"Michael O'Leary and Warren Valdmanis . 2021. An ESG Reckoning Is Coming. Harvard Business Review ( 2021 ). Michael O'Leary and Warren Valdmanis. 2021. An ESG Reckoning Is Coming. Harvard Business Review (2021)."},{"key":"e_1_3_2_1_60_1","volume-title":"Facebook Hides Data Showing It Harms Users. Outside Scholars Need Access. (2021). Last accessed","author":"Persily Nathaniel","year":"2021","unstructured":"Nathaniel Persily . 2021. Facebook Hides Data Showing It Harms Users. Outside Scholars Need Access. (2021). Last accessed 25 October 2021 . Nathaniel Persily. 2021. Facebook Hides Data Showing It Harms Users. Outside Scholars Need Access. (2021). Last accessed 25 October 2021."},{"key":"e_1_3_2_1_61_1","volume-title":"ESG, and Compliance. Forthcoming","author":"Pollman Elizabeth","year":"2019","unstructured":"Elizabeth Pollman . 2019. Corporate Social Responsibility , ESG, and Compliance. Forthcoming , Cambridge Handbook of Compliance (D. Daniel Sokol & Benjamin van Rooij eds.), Loyola Law School, Los Angeles Legal Studies Research Paper 2019 --35 (2019). Elizabeth Pollman. 2019. Corporate Social Responsibility, ESG, and Compliance. Forthcoming, Cambridge Handbook of Compliance (D. Daniel Sokol & Benjamin van Rooij eds.), Loyola Law School, Los Angeles Legal Studies Research Paper 2019--35 (2019)."},{"key":"e_1_3_2_1_62_1","unstructured":"Michael Power. 1997. The audit society: Rituals of verification. OUP Oxford.  Michael Power. 1997. The audit society: Rituals of verification. OUP Oxford."},{"key":"e_1_3_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372828"},{"key":"e_1_3_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/3306618.3314244"},{"key":"e_1_3_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/3375627.3375820"},{"key":"e_1_3_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372873"},{"key":"e_1_3_2_1_67_1","volume-title":"Algorithmic impact assessments: A practical framework for public agency accountability","author":"Reisman Dillon","year":"2018","unstructured":"Dillon Reisman , Jason Schultz , Kate Crawford , and Meredith Whittaker . 2018. Algorithmic impact assessments: A practical framework for public agency accountability . AI Now Institute ( 2018 ), 1--22. Dillon Reisman, Jason Schultz, Kate Crawford, and Meredith Whittaker. 2018. Algorithmic impact assessments: A practical framework for public agency accountability. AI Now Institute (2018), 1--22."},{"key":"e_1_3_2_1_68_1","volume-title":"Forthcoming","author":"Richardson Rashida","year":"2021","unstructured":"Rashida Richardson . 2021. Defining and Demystifying Automated Decision Systems. Maryland Law Review , Forthcoming ( 2021 ). Rashida Richardson. 2021. Defining and Demystifying Automated Decision Systems. Maryland Law Review, Forthcoming (2021)."},{"key":"e_1_3_2_1_69_1","unstructured":"Rashida Richardson A Cahn A Kak A Diaz A Samant etal 2019. Confronting black boxes: A shadow report of the New York City automated decision system task force. AI Now Institute (2019).  Rashida Richardson A Cahn A Kak A Diaz A Samant et al. 2019. Confronting black boxes: A shadow report of the New York City automated decision system task force. AI Now Institute (2019)."},{"key":"e_1_3_2_1_70_1","volume-title":"Philip ME Garboden, and Jennifer E Cossyleon","author":"Rosen Eva","year":"2021","unstructured":"Eva Rosen , Philip ME Garboden, and Jennifer E Cossyleon . 2021 . Racial Discrimination in Housing : How Landlords Use Algorithms and Home Visits to Screen Tenants. American Sociological Review ( 2021), 00031224211029618. Eva Rosen, Philip ME Garboden, and Jennifer E Cossyleon. 2021. Racial Discrimination in Housing: How Landlords Use Algorithms and Home Visits to Screen Tenants. American Sociological Review (2021), 00031224211029618."},{"key":"e_1_3_2_1_71_1","unstructured":"Samantha Ross and Gordon Seymour. 2019. The Developing World of Assurance on Sustainability Reporting. unpublished manuscript.  Samantha Ross and Gordon Seymour. 2019. The Developing World of Assurance on Sustainability Reporting. unpublished manuscript."},{"key":"e_1_3_2_1_72_1","volume-title":"Algorithms that\" Don't See Color\": Comparing Biases in Lookalike and Special Ad Audiences. arXiv preprint arXiv:1912.07579","author":"Sapiezynski Piotr","year":"2019","unstructured":"Piotr Sapiezynski , Avijit Ghosh , Levi Kaplan , Alan Mislove , and Aaron Rieke . 2019. Algorithms that\" Don't See Color\": Comparing Biases in Lookalike and Special Ad Audiences. arXiv preprint arXiv:1912.07579 ( 2019 ). Piotr Sapiezynski, Avijit Ghosh, Levi Kaplan, Alan Mislove, and Aaron Rieke. 2019. Algorithms that\" Don't See Color\": Comparing Biases in Lookalike and Special Ad Audiences. arXiv preprint arXiv:1912.07579 (2019)."},{"key":"e_1_3_2_1_73_1","unstructured":"AndrewD Selbst. 2021. An Institutional ViewOf Algorithmic Impact Assessments. (2021).  AndrewD Selbst. 2021. An Institutional ViewOf Algorithmic Impact Assessments. (2021)."},{"key":"e_1_3_2_1_74_1","unstructured":"Nathan Sheard. 2021. Banning Government Use of Face Recognition Technology: 2020 Year in Review.  Nathan Sheard. 2021. Banning Government Use of Face Recognition Technology: 2020 Year in Review."},{"key":"e_1_3_2_1_75_1","unstructured":"G Sherwin and E Bhandari. 2019. Facebook settles civil rights cases by making sweeping changes to its online ad platform.  G Sherwin and E Bhandari. 2019. Facebook settles civil rights cases by making sweeping changes to its online ad platform."},{"key":"e_1_3_2_1_76_1","unstructured":"Mona Sloane. 2021. The Algorithmic Auditing Trap.  Mona Sloane. 2021. The Algorithmic Auditing Trap."},{"key":"e_1_3_2_1_77_1","volume-title":"Pseudo-Science, and the Quest for Auditability. arXiv preprint arXiv:2106.12403","author":"Sloane Mona","year":"2021","unstructured":"Mona Sloane , Emanuel Moss , and Rumman Chowdhury . 2021. A Silicon Valley Love Triangle: Hiring Algorithms , Pseudo-Science, and the Quest for Auditability. arXiv preprint arXiv:2106.12403 ( 2021 ). Mona Sloane, Emanuel Moss, and Rumman Chowdhury. 2021. A Silicon Valley Love Triangle: Hiring Algorithms, Pseudo-Science, and the Quest for Auditability. arXiv preprint arXiv:2106.12403 (2021)."},{"key":"e_1_3_2_1_78_1","unstructured":"Jacob Snow. 2018. Amazon's Face Recognition Falsely Matched 28 Members of Congress With Mugshots. https:\/\/www.aclu.org\/blog\/privacy-technology\/surveillance technologies\/amazons-face-recognition-falsely-matched-28 Last accessed 25 October 2021.  Jacob Snow. 2018. Amazon's Face Recognition Falsely Matched 28 Members of Congress With Mugshots. https:\/\/www.aclu.org\/blog\/privacy-technology\/surveillance technologies\/amazons-face-recognition-falsely-matched-28 Last accessed 25 October 2021."},{"key":"e_1_3_2_1_79_1","first-page":"925","article-title":"Contemporary housing discrimination: Facebook, targeted advertising, and the Fair Housing Act","volume":"57","author":"Spinks Chandler Nicholle","year":"2019","unstructured":"Chandler Nicholle Spinks . 2019 . Contemporary housing discrimination: Facebook, targeted advertising, and the Fair Housing Act . Hous. L. Rev. 57 (2019), 925 . Chandler Nicholle Spinks. 2019. Contemporary housing discrimination: Facebook, targeted advertising, and the Fair Housing Act. Hous. L. Rev. 57 (2019), 925.","journal-title":"Hous. L. Rev."},{"key":"e_1_3_2_1_80_1","doi-asserted-by":"publisher","DOI":"10.1145\/2447976.2447990"},{"key":"e_1_3_2_1_81_1","doi-asserted-by":"publisher","DOI":"10.1136\/amiajnl-2011-000214"},{"key":"e_1_3_2_1_82_1","volume-title":"ProPublica (March 28","author":"Tobin Ariana","year":"2019","unstructured":"Ariana Tobin . 2019 . HUD sues Facebook over housing discrimination and says the company's algorithms have made the problem worse . ProPublica (March 28 , 2019). Available at https:\/\/www.propublica.org\/article\/hud-sues-facebookhousingdiscrimination-advertising-algorithms (last accessed April 29, 2019) (2019). Ariana Tobin. 2019. HUD sues Facebook over housing discrimination and says the company's algorithms have made the problem worse. ProPublica (March 28, 2019). Available at https:\/\/www.propublica.org\/article\/hud-sues-facebookhousingdiscrimination-advertising-algorithms (last accessed April 29, 2019) (2019)."},{"key":"e_1_3_2_1_83_1","volume-title":"Proc. Computation+ Journalism Symposium.","author":"Trielli Daniel","year":"2017","unstructured":"Daniel Trielli , Jennifer Stark , and Nicholas Diakopoulos . 2017 . Algorithm tips: A resource for algorithmic accountability in Government . In Proc. Computation+ Journalism Symposium. Daniel Trielli, Jennifer Stark, and Nicholas Diakopoulos. 2017. Algorithm tips: A resource for algorithmic accountability in Government. In Proc. Computation+ Journalism Symposium."},{"key":"e_1_3_2_1_84_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174014"},{"key":"e_1_3_2_1_85_1","volume-title":"Google is using AI to help users explore the topics they're searching for - here's how. https:\/\/www.theverge.com\/2021\/9\/29\/22696268\/google-search-on-updates-ai-mum-explained Last accessed","author":"Vincent James","year":"2021","unstructured":"James Vincent . 2021. Google is using AI to help users explore the topics they're searching for - here's how. https:\/\/www.theverge.com\/2021\/9\/29\/22696268\/google-search-on-updates-ai-mum-explained Last accessed 2 November 2021 . James Vincent. 2021. Google is using AI to help users explore the topics they're searching for - here's how. https:\/\/www.theverge.com\/2021\/9\/29\/22696268\/google-search-on-updates-ai-mum-explained Last accessed 2 November 2021."},{"key":"e_1_3_2_1_86_1","doi-asserted-by":"publisher","DOI":"10.5555\/3152676"},{"key":"e_1_3_2_1_87_1","volume-title":"Google trained a trillion-parameter AI language model. https:\/\/venturebeat.com\/2021\/01\/12\/google-trained-a-trillion-parameterai-language-model\/ Last accessed","author":"Wiggers Kyle","year":"2021","unstructured":"Kyle Wiggers . 2021. Google trained a trillion-parameter AI language model. https:\/\/venturebeat.com\/2021\/01\/12\/google-trained-a-trillion-parameterai-language-model\/ Last accessed 2 November 2021 . Kyle Wiggers. 2021. Google trained a trillion-parameter AI language model. https:\/\/venturebeat.com\/2021\/01\/12\/google-trained-a-trillion-parameterai-language-model\/ Last accessed 2 November 2021."},{"key":"e_1_3_2_1_88_1","doi-asserted-by":"publisher","DOI":"10.1145\/3442188.3445928"},{"key":"e_1_3_2_1_89_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41591-021-01312-x"}],"event":{"name":"AIES '22: AAAI\/ACM Conference on AI, Ethics, and Society","location":"Oxford United Kingdom","acronym":"AIES '22","sponsor":["SIGAI ACM Special Interest Group on Artificial Intelligence","AAAI"]},"container-title":["Proceedings of the 2022 AAAI\/ACM Conference on AI, Ethics, and Society"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3514094.3534181","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3514094.3534181","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:02:36Z","timestamp":1750186956000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3514094.3534181"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,7,26]]},"references-count":88,"alternative-id":["10.1145\/3514094.3534181","10.1145\/3514094"],"URL":"https:\/\/doi.org\/10.1145\/3514094.3534181","relation":{},"subject":[],"published":{"date-parts":[[2022,7,26]]},"assertion":[{"value":"2022-07-27","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}