{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,28]],"date-time":"2026-08-28T11:34:54Z","timestamp":1787916894908,"version":"build-2784847793"},"publisher-location":"New York, NY, USA","reference-count":57,"publisher":"ACM","license":[{"start":{"date-parts":[[2024,6,3]],"date-time":"2024-06-03T00:00:00Z","timestamp":1717372800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"funder":[{"DOI":"10.13039\/https:\/\/doi.org\/10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["1704425"],"award-info":[{"award-number":["1704425"]}],"id":[{"id":"10.13039\/https:\/\/doi.org\/10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,6,3]]},"DOI":"10.1145\/3630106.3658959","type":"proceedings-article","created":{"date-parts":[[2024,6,5]],"date-time":"2024-06-05T09:14:21Z","timestamp":1717578861000},"page":"1107-1120","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":49,"title":["Auditing Work: Exploring the New York City algorithmic bias audit regime"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0009-0001-9687-3353","authenticated-orcid":false,"given":"Lara","family":"Groves","sequence":"first","affiliation":[{"name":"Ada Lovelace Institute, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2803-6625","authenticated-orcid":false,"given":"Jacob","family":"Metcalf","sequence":"additional","affiliation":[{"name":"Data & Society Research Institute, United States of America"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1894-3413","authenticated-orcid":false,"given":"Alayna","family":"Kennedy","sequence":"additional","affiliation":[{"name":"Independent researcher, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0828-8665","authenticated-orcid":false,"given":"Briana","family":"Vecchione","sequence":"additional","affiliation":[{"name":"Data & Society Research Institute, United States of America"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3683-2293","authenticated-orcid":false,"given":"Andrew","family":"Strait","sequence":"additional","affiliation":[{"name":"Ada Lovelace Institute, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,6,5]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"[n. d.]. IAAA - International Algorithmic Auditors Association. ([n. d.]). https:\/\/iaaa-algorithmicauditors.org\/"},{"key":"e_1_3_2_1_2_1","volume-title":"International Auditing and Assurance Standards Board","year":"2013","unstructured":"2013. ISAE 3000 Revised for IAASB. International Auditing and Assurance Standards Board (2013). https:\/\/www.iaasb.org\/publications\/international-standard-assurance-engagements-isae-3000-revised-assurance-engagements-other-audits-or"},{"key":"e_1_3_2_1_3_1","volume-title":"Comments Received by the Department of Consumer and Worker Protection on Proposed Rules related to Automated Employment Decision Tools. New York City Consumer and Worker Protection","year":"2023","unstructured":"2023. Comments Received by the Department of Consumer and Worker Protection on Proposed Rules related to Automated Employment Decision Tools. New York City Consumer and Worker Protection (2023). https:\/\/www.nyc.gov\/assets\/dca\/downloads\/pdf\/about\/PublicComments-Proposed-Rules-Related-to-Automated-Employment-Decision-Tools-Updated.pdf"},{"key":"e_1_3_2_1_4_1","volume-title":"Code of Federal Regulations. 29 CFR Part 1607 - General Principles","year":"2024","unstructured":"2024. Code of Federal Regulations. 29 CFR Part 1607 - General Principles (2024). https:\/\/www.wired.com\/story\/opinion-new-york-citys-surveillance-battle-offers-national-lessons\/"},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.2139\/ssrn.2746078"},{"key":"e_1_3_2_1_6_1","volume-title":"Big data\u2019s disparate impact. California law review","author":"Barocas Solon","year":"2016","unstructured":"Solon Barocas and Andrew\u00a0D Selbst. 2016. Big data\u2019s disparate impact. California law review (2016), 671\u2013732."},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1287\/mnsc.1120.1680"},{"key":"e_1_3_2_1_8_1","volume-title":"AI auditing: The broken bus on the road to AI accountability. arXiv preprint arXiv:2401.14462","author":"Birhane Abeba","year":"2024","unstructured":"Abeba Birhane, Ryan Steed, Victor Ojewale, Briana Vecchione, and Inioluwa\u00a0Deborah Raji. 2024. AI auditing: The broken bus on the road to AI accountability. arXiv preprint arXiv:2401.14462 (2024)."},{"key":"e_1_3_2_1_9_1","unstructured":"Jenny Brennan Lara Groves Elliot Jones and Andrew Strait. [n. d.]. AI Assurance?https:\/\/www.adalovelaceinstitute.org\/report\/risks-ai-systems\/"},{"key":"e_1_3_2_1_10_1","volume-title":"Conference on fairness, accountability and transparency. PMLR, 77\u201391","author":"Buolamwini Joy","year":"2018","unstructured":"Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency. PMLR, 77\u201391."},{"key":"e_1_3_2_1_11_1","volume-title":"New York City\u2019s Surveillance Battle Offers National Lessons. Wired","author":"Cahn Albert\u00a0Fox","year":"2021","unstructured":"Albert\u00a0Fox Cahn. 2021. New York City\u2019s Surveillance Battle Offers National Lessons. Wired (2021). https:\/\/www.wired.com\/story\/opinion-new-york-citys-surveillance-battle-offers-national-lessons\/"},{"key":"e_1_3_2_1_12_1","unstructured":"Pew\u00a0Research Center. 2023. AI in Hiring and Evaluation of Workers: What People Think. (2023). https:\/\/www.pewresearch.org\/internet\/2023\/04\/20\/ai-in-hiring-and-evaluating-workers-what-americans-think\/"},{"key":"e_1_3_2_1_13_1","unstructured":"Kathy Charmaz. 2023. Constructing grounded theory: A practical guide through qualitative analysis. sage."},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3593013.3594073"},{"key":"e_1_3_2_1_15_1","unstructured":"European Commission. 2024. The Digital Services Act package | Shaping Europe\u2019s digital future. (2024). https:\/\/digital- strategy.ec.europa.eu\/en\/policies\/digital-services-act-package"},{"key":"e_1_3_2_1_16_1","unstructured":"Equal Employment\u00a0Opportunity Commission. 2007. Employment Tests and Selection Procedures. (2007). https:\/\/www.eeoc.gov\/laws\/guidance\/employment-tests-and-selection-procedures"},{"key":"e_1_3_2_1_17_1","volume-title":"Definitions of Race and Ethnicity Categories. Data Collection","author":"Equal Employment\u00a0Opportunity Commission","year":"2022","unstructured":"Equal Employment\u00a0Opportunity Commission. 2022. Definitions of Race and Ethnicity Categories. Data Collection (2022)."},{"key":"e_1_3_2_1_18_1","volume-title":"Assessing Adverse Impact in Software, Algorithms, and Artificial Intelligence Used in Employment Selection Procedures Under Title VII of the Civil Rights Act of","author":"Equal Employment\u00a0Opportunity Commission","year":"1964","unstructured":"Equal Employment\u00a0Opportunity Commission. 2023. Assessing Adverse Impact in Software, Algorithms, and Artificial Intelligence Used in Employment Selection Procedures Under Title VII of the Civil Rights Act of 1964. (2023). https:\/\/www.eeoc.gov\/laws\/guidance\/select-issues-assessing-adverse-impact-software-algorithms-and-artificial"},{"key":"e_1_3_2_1_19_1","unstructured":"Australian Competition and Consumer Commission. 2020. Trivago misled consumers about hotel room rates. (2020). https:\/\/www.accc.gov.au\/media-release\/trivago-misled-consumers-about-hotel-room-rates"},{"key":"e_1_3_2_1_20_1","volume-title":"Congress. 2002","author":"US","year":"2002","unstructured":"US Congress. 2002. The Sarbanes Oxley Act. (2002). https:\/\/sarbanes-oxley-act.com\/"},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3531146.3533213"},{"key":"e_1_3_2_1_22_1","unstructured":"New York\u00a0City Council. 2021. A Local Law to amend the administrative code of the city of New York in relation to automated employment decision tools. (2021). https:\/\/legistar.council.nyc.gov\/LegislationDetail.aspx?ID=4344524&GUID=B051915D- A9AC-451E-81F8-6596032FA3F9&Options=Advanced&Search="},{"key":"e_1_3_2_1_23_1","volume-title":"Determinants of audit quality in the public sector. Accounting review","author":"Deis\u00a0Jr R","year":"1992","unstructured":"Donald\u00a0R Deis\u00a0Jr and Gary\u00a0A Giroux. 1992. Determinants of audit quality in the public sector. Accounting review (1992), 462\u2013479."},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2783258.2783311"},{"key":"e_1_3_2_1_25_1","unstructured":"Marissa Gerchick and Brooke Watson. 2023. Tracking Automated Employment Decision Tool Bias Audits. (2023). https:\/\/github.com\/aclu-national\/tracking-ll144-bias-audits"},{"key":"e_1_3_2_1_26_1","first-page":"289","article-title":"Algorithmic Auditing: Chasing AI Accountability","volume":"39","author":"Goodman P","year":"2022","unstructured":"Ellen\u00a0P Goodman and Julia Trehu. 2022. Algorithmic Auditing: Chasing AI Accountability. Santa Clara High Tech. LJ 39 (2022), 289.","journal-title":"Santa Clara High Tech. LJ"},{"key":"e_1_3_2_1_27_1","unstructured":"Lara Groves Jenny Brennan Inioluwa\u00a0Deborah Raji Aidan Peppin and Strait. 2022. Algorithmic impact assessment: a case study in healthcare. (2022). https:\/\/www.adalovelaceinstitute.org\/wp- content\/uploads\/2022\/02\/Algorithmic-impact-assessment-a-case-study-in-healthcare.pdf"},{"key":"e_1_3_2_1_28_1","unstructured":"Indeed. 2023. The Indeed Global AI Survey: Your Guide to the Future of Hiring. (2023). https:\/\/www.indeed.com\/lead\/the-indeed-ai-report?hl=en#form"},{"key":"e_1_3_2_1_29_1","volume-title":"New York City wants to restrict artificial intelligence in hiring. CBS News","author":"Ivanova Irina","year":"2020","unstructured":"Irina Ivanova. 2020. New York City wants to restrict artificial intelligence in hiring. CBS News (2020). https:\/\/www.cbsnews.com\/news\/new-york-city-artificial-intelligence-hiring-restriction\/"},{"key":"e_1_3_2_1_30_1","volume-title":"Keeping an eye on AI","author":"Jones Elliot","year":"2023","unstructured":"Elliot Jones, Jenny Brennan, Connor Dunlop, and Andrew Strait. 2023. Keeping an eye on AI. Ada Lovelace Institute (2023). https:\/\/www.adalovelaceinstitute.org\/wp-content\/uploads\/2023\/09\/ALI_Keeping-an-eye-on-AI-2023.pdf"},{"key":"e_1_3_2_1_31_1","volume-title":"New York City Moves to Create Accountability for Algorithms. ProPublica","author":"Kirchner Lauren","year":"2023","unstructured":"Lauren Kirchner. 2023. New York City Moves to Create Accountability for Algorithms. ProPublica (2023). https:\/\/www.propublica.org\/article\/new-york-city-moves-to-create-accountability-for-algorithms"},{"key":"e_1_3_2_1_32_1","volume-title":"ML and associated algorithms.","author":"Koshiyama Adriano","year":"2021","unstructured":"Adriano Koshiyama, Emre Kazim, Philip Treleaven, Pete Rai, Lukasz Szpruch, Giles Pavey, Ghazi Ahamat, Franziska Leutner, Randy Goebel, Andrew Knight, 2021. Towards algorithm auditing: a survey on managing legal, ethical and technological risks of AI, ML and associated algorithms. (2021)."},{"key":"e_1_3_2_1_33_1","volume-title":"Understanding artificial intelligence ethics and safety. arXiv preprint arXiv:1906.05684","author":"Leslie David","year":"2019","unstructured":"David Leslie. 2019. Understanding artificial intelligence ethics and safety. arXiv preprint arXiv:1906.05684 (2019)."},{"key":"e_1_3_2_1_34_1","volume-title":"A Hiring Law Blazes a Path for A.I. Regulation. The New York Times","author":"Lohr Steve","year":"2023","unstructured":"Steve Lohr. 2023. A Hiring Law Blazes a Path for A.I. Regulation. The New York Times (2023). https:\/\/www.nytimes.com\/2023\/05\/25\/technology\/ai-hiring-law-new-york.html"},{"key":"e_1_3_2_1_35_1","volume-title":"Assembling accountability: algorithmic impact assessment for the public interest. Available at SSRN 3877437","author":"Moss Emanuel","year":"2021","unstructured":"Emanuel Moss, Elizabeth\u00a0Anne Watkins, Ranjit Singh, Madeleine\u00a0Clare Elish, and Jacob Metcalf. 2021. Assembling accountability: algorithmic impact assessment for the public interest. Available at SSRN 3877437 (2021)."},{"key":"e_1_3_2_1_36_1","unstructured":"New York City\u00a0Department of Consumer and Worker Protections. 2023. Automated Employment Decision Tools: Frequently Asked Questions. (2023). https:\/\/www.nyc.gov\/assets\/dca\/downloads\/pdf\/about\/DCWP-AEDT-FAQ.pdf"},{"key":"e_1_3_2_1_37_1","unstructured":"New York City\u00a0Department of Consumer and Worker Protections. 2023. Automated Employment Decision Tools (Updated). (2023). https:\/\/rules.cityofnewyork.us\/rule\/automated-employment-decision-tools-updated\/"},{"key":"e_1_3_2_1_38_1","volume-title":"Use of External Consumer Data and Information Sources, Algorithms, and Predictive Models.","author":"Colorado\u00a0Division of Insurance. 2023.","year":"2023","unstructured":"Colorado\u00a0Division of Insurance. 2023. Regulation 10-1-1 Governance and Risk Management Framework Requirements for Life Insurers\u2019 Use of External Consumer Data and Information Sources, Algorithms, and Predictive Models. (2023). https:\/\/drive.google.com\/file\/d\/1dlPKJCDo76iHfJZDopQEhTDCmKbuYnNI\/view?usp=embed_facebook"},{"key":"e_1_3_2_1_39_1","volume-title":"https:\/\/www.justice.gov\/jm\/jm-9-48000-computer-fraud","author":"US\u00a0Department of Justice. 2015. 9-48.000 - Computer Fraud and Abuse Act.","year":"2015","unstructured":"US\u00a0Department of Justice. 2015. 9-48.000 - Computer Fraud and Abuse Act. (2015). https:\/\/www.justice.gov\/jm\/jm-9-48000-computer-fraud"},{"key":"e_1_3_2_1_40_1","unstructured":"Information\u00a0Commissioner\u2019s Office. 2020. Guidance on the AI auditing framework Draft guidance for consultation. Information Commissioner\u2019s Office. (2020). https:\/\/ico.org.uk\/media\/2617219\/guidance-on-the-ai-auditing-framework-draft-for-consultation.pdf"},{"key":"e_1_3_2_1_41_1","volume-title":"Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling. arXiv preprint arXiv:2402.17861","author":"Ojewale Victor","year":"2024","unstructured":"Victor Ojewale, Ryan Steed, Briana Vecchione, Abeba Birhane, and Inioluwa\u00a0Deborah Raji. 2024. Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling. arXiv preprint arXiv:2402.17861 (2024)."},{"key":"e_1_3_2_1_42_1","volume-title":"Online Safety Act","author":"Parliament UK","year":"2023","unstructured":"UK Parliament. 2022. Online Safety Act 2023. (2022). https:\/\/bills.parliament.uk\/bills\/3137"},{"key":"e_1_3_2_1_43_1","volume-title":"Colorado Aims to Prevent AI-Driven Discrimination in Insurance. GovTech","author":"Pattison-Gordon Jules","year":"2023","unstructured":"Jules Pattison-Gordon. 2023. Colorado Aims to Prevent AI-Driven Discrimination in Insurance. GovTech (2023). https:\/\/www.govtech.com\/policy\/colorado-aims-to-prevent-ai-driven-discrimination-in-insurance"},{"key":"e_1_3_2_1_44_1","unstructured":"PCAOB. [n. d.]. Driving improvement in audit quality to protect investors. ([n. d.]). https:\/\/pcaobus.org\/"},{"key":"e_1_3_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3593013.3594084"},{"key":"e_1_3_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372828"},{"key":"e_1_3_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3306618.3314244"},{"key":"e_1_3_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/3514094.3534181"},{"key":"e_1_3_2_1_49_1","unstructured":"Algemene Rekenkamer. 2022. An Audit of 9 Algorithms used by the Dutch Government - Report - Netherlands Court of Audit. (2022). https:\/\/english.rekenkamer.nl\/publications\/reports\/2022\/05\/18\/an-audit-of-9-algorithms-used-by-the-dutch-government"},{"key":"e_1_3_2_1_50_1","volume-title":"Algorithmic Accountability Act of","author":"Rep Yvette D.","year":"2022","unstructured":"Yvette D. [D-NY-9] Rep.\u00a0Clarke. 2022. Algorithmic Accountability Act of 2022. (2022). https:\/\/www.congress.gov\/bill\/117th-congress\/house-bill\/6580\/text"},{"key":"e_1_3_2_1_51_1","first-page":"109","article-title":"Disparate impact in big data policing. Ga","volume":"52","author":"Selbst D","year":"2017","unstructured":"Andrew\u00a0D Selbst. 2017. Disparate impact in big data policing. Ga. L. Rev. 52 (2017), 109.","journal-title":"L. Rev."},{"key":"e_1_3_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287598"},{"key":"e_1_3_2_1_53_1","unstructured":"SHRM. 2022. Automation and AI in HR. (2022). https:\/\/advocacy.shrm.org\/SHRM-2022-Automation-AI- Research.pdf?_ga=2.112869508.1029738808.1666019592-61357574.1655121608"},{"key":"e_1_3_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1177\/0148558X14544505"},{"key":"e_1_3_2_1_55_1","volume-title":"The four-fifths rule is not disparate impact: a woeful tale of epistemic trespassing in algorithmic fairness. arXiv preprint arXiv:2202.09519","author":"Watkins Elizabeth\u00a0Anne","year":"2022","unstructured":"Elizabeth\u00a0Anne Watkins, Michael McKenna, and Jiahao Chen. 2022. The four-fifths rule is not disparate impact: a woeful tale of epistemic trespassing in algorithmic fairness. arXiv preprint arXiv:2202.09519 (2022)."},{"key":"e_1_3_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3442188.3445928"},{"key":"e_1_3_2_1_57_1","volume-title":"ACM Conference on Fairness, Accountability and Transparency","author":"Wright Lucas","year":"2024","unstructured":"Lucas Wright, Roxana\u00a0Mika Muenster, Briana Vecchione, Tianyao Qu, Pika Cai, Alan Smith, COMM\/INFO 2450\u00a0Student Investigators, Jacob Metcalf, and J.\u00a0Nathan Matias. 2024. Null Compliance: NYC Local Law 144 and the Challenges of Algorithm Accountability. ACM Conference on Fairness, Accountability and Transparency (2024)."}],"event":{"name":"FAccT '24: The 2024 ACM Conference on Fairness, Accountability, and Transparency","location":"Rio de Janeiro Brazil","acronym":"FAccT '24"},"container-title":["The 2024 ACM Conference on Fairness, Accountability, and Transparency"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3630106.3658959","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3630106.3658959","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T18:50:58Z","timestamp":1750272658000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3630106.3658959"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6,3]]},"references-count":57,"alternative-id":["10.1145\/3630106.3658959","10.1145\/3630106"],"URL":"https:\/\/doi.org\/10.1145\/3630106.3658959","relation":{},"subject":[],"published":{"date-parts":[[2024,6,3]]},"assertion":[{"value":"2024-06-05","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}