{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,2]],"date-time":"2026-07-02T00:20:32Z","timestamp":1782951632960,"version":"3.54.5"},"reference-count":196,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2023,11,8]],"date-time":"2023-11-08T00:00:00Z","timestamp":1699401600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,11,8]],"date-time":"2023-11-08T00:00:00Z","timestamp":1699401600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Center for Information Technology Policy (CITP), Princeton University"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["DISO"],"published-print":{"date-parts":[[2023,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>AI auditing is a rapidly growing field of research and practice. This review article, which doubles as an editorial to Digital Society\u2019s topical collection on \u2018Auditing of AI\u2019, provides an overview of previous work in the field. Three key points emerge from the review. First, contemporary attempts to audit AI systems have much to learn from how audits have historically been structured and conducted in areas like financial accounting, safety engineering and the social sciences. Second, both policymakers and technology providers have an interest in promoting auditing as an AI governance mechanism. Academic researchers can thus fill an important role by studying the feasibility and effectiveness of different AI auditing procedures. Third, AI auditing is an inherently multidisciplinary undertaking, to which substantial contributions have been made by computer scientists and engineers as well as social scientists, philosophers, legal scholars and industry practitioners. Reflecting this diversity of perspectives, different approaches to AI auditing have different affordances and constraints. Specifically, a distinction can be made between technology-oriented audits, which focus on the properties and capabilities of AI systems, and process-oriented audits, which focus on technology providers\u2019 governance structures and quality management systems. The next step in the evolution of auditing as an AI governance mechanism, this article concludes, should be the interlinking of these available\u2014and complementary\u2014approaches into structured and holistic procedures to audit not only how AI systems are designed and used but also how they impact users, societies and the natural environment in applied settings over time.<\/jats:p>","DOI":"10.1007\/s44206-023-00074-y","type":"journal-article","created":{"date-parts":[[2023,11,8]],"date-time":"2023-11-08T12:01:55Z","timestamp":1699444915000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":88,"title":["Auditing of AI: Legal, Ethical and Technical Approaches"],"prefix":"10.1007","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8691-2582","authenticated-orcid":false,"given":"Jakob","family":"M\u00f6kander","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2023,11,8]]},"reference":[{"key":"74_CR1","doi-asserted-by":"publisher","first-page":"3","DOI":"10.48550\/arxiv.1806.05740","volume":"13","author":"R Abebe","year":"2019","unstructured":"Abebe, R., Hill, S., Vaughan, J. W., Small, P. M., & Schwartz, H. A. (2019). Using search queries to understand health information needs in Africa. Proceedings of the Thirteenth International AAAI Conference on Web and Social Media, 13, 3\u201314. https:\/\/doi.org\/10.48550\/arxiv.1806.05740","journal-title":"Proceedings of the Thirteenth International AAAI Conference on Web and Social Media"},{"key":"74_CR2","doi-asserted-by":"publisher","first-page":"95","DOI":"10.1007\/s10115-017-1116-3","volume":"54","author":"P Adler","year":"2018","unstructured":"Adler, P., Falk, C., Friedler, S. A., Nix, T., Rybeck, G., Scheidegger, C., Smith, B., & Venkatasubramanian, S. (2018). Auditing black-box models for indirect influence. Knowledge and Information Systems, 54, 95\u2013122. https:\/\/doi.org\/10.1007\/s10115-017-1116-3","journal-title":"Knowledge and Information Systems"},{"issue":"2","key":"74_CR3","doi-asserted-by":"publisher","first-page":"362","DOI":"10.1016\/j.jue.2008.02.004","volume":"64","author":"AM Ahmed","year":"2008","unstructured":"Ahmed, A. M., & Hammarstedt, M. (2008). Discrimination in the rental housing market: A field experiment on the Internet. Journal of Urban Economics, 64(2), 362\u2013372. https:\/\/doi.org\/10.1016\/j.jue.2008.02.004","journal-title":"Journal of Urban Economics"},{"key":"74_CR4","unstructured":"AI HLEG. (2019). Ethics guidelines for trustworthy AI. Retrieved July 20, 2023, from https:\/\/ec.europa.eu\/futurium\/en\/ai-alliance-consultation\/guidelines#Top"},{"key":"74_CR5","doi-asserted-by":"publisher","DOI":"10.48550\/arxiv.2204.10233","author":"N-J Akpinar","year":"2022","unstructured":"Akpinar, N.-J., Nagireddy, M., Stapleton, L., Cheng, H.-F., Zhu, H., Wu, S., & Heidari, H. (2022). A sandbox tool to bias(stress)-test fairness algorithms. ArXiv. https:\/\/doi.org\/10.48550\/arxiv.2204.10233","journal-title":"ArXiv"},{"key":"74_CR6","doi-asserted-by":"publisher","first-page":"199","DOI":"10.1145\/3359301","volume":"3","author":"M Ali","year":"2019","unstructured":"Ali, M., Sapiezynski, P., Mislove, A., Rieke, A., Bogen, M., & Korolova, A. (2019). Discrimination through optimization: How Facebook\u2019s ad delivery can lead to biased outcomes. Proceedings of the ACM on Human-Computer Interaction, 3, 199. https:\/\/doi.org\/10.1145\/3359301","journal-title":"Proceedings of the ACM on Human-Computer Interaction"},{"key":"74_CR7","unstructured":"Allford, L., & Carson, P. (2015). Safety practice safety, health, and environment audits with selected case histories. In Loss Prevention Bulletin (241). Retrieved July 20, 2023, from www.researchgate.net\/publication\/307978324"},{"key":"74_CR8","volume-title":"Algorithm audit: Why, what, and how?","author":"B Aragona","year":"2022","unstructured":"Aragona, B. (2022). Algorithm audit: Why, what, and how? (1st ed.). Routledge.","edition":"1"},{"issue":"4","key":"74_CR9","doi-asserted-by":"publisher","first-page":"343","DOI":"10.1007\/s10551-005-7888-5","volume":"61","author":"S Arjoon","year":"2005","unstructured":"Arjoon, S. (2005). Corporate governance: An ethical perspective. Journal of Business Ethics, 61(4), 343\u2013352. https:\/\/doi.org\/10.1007\/s10551-005-7888-5","journal-title":"Journal of Business Ethics"},{"issue":"3","key":"74_CR10","doi-asserted-by":"publisher","first-page":"405","DOI":"10.1007\/s43681-021-00084-x","volume":"2","author":"J Ayling","year":"2021","unstructured":"Ayling, J., & Chapman, A. (2021). Putting AI ethics to work: Are the tools fit for purpose? AI and Ethics, 2(3), 405\u2013429. https:\/\/doi.org\/10.1007\/s43681-021-00084-x","journal-title":"AI and Ethics"},{"key":"74_CR11","unstructured":"BABL AI. (2023). Boutique consultancy on responsible AI. Retrieved July 20, 2023, from https:\/\/babl.ai\/"},{"key":"74_CR12","doi-asserted-by":"publisher","first-page":"41","DOI":"10.1146\/ANNUREV-SOC-073014-112445","volume":"43","author":"D Baldassarri","year":"2017","unstructured":"Baldassarri, D., & Abascal, M. (2017). Field experiments across the social sciences. Annual Review of Sociology, 43, 41\u201373. https:\/\/doi.org\/10.1146\/ANNUREV-SOC-073014-112445","journal-title":"Annual Review of Sociology"},{"key":"74_CR13","volume-title":"Understanding regulation: Theory, strategy, and practice","author":"R Baldwin","year":"1999","unstructured":"Baldwin, R., & Cave, M. (1999). Understanding regulation: Theory, strategy, and practice. Oxford University Press."},{"issue":"1","key":"74_CR14","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3449148","volume":"5","author":"J Bandy","year":"2021","unstructured":"Bandy, J. (2021). Problematic machine behavior: A systematic literature review of algorithm audits. Proceedings of the ACM on Human-Computer Interaction, 5(1), 1\u201334. https:\/\/doi.org\/10.1145\/3449148","journal-title":"Proceedings of the ACM on Human-Computer Interaction"},{"key":"74_CR15","doi-asserted-by":"crossref","unstructured":"Bandy, J., & Diakopoulos, N. (2019). Auditing news curation systems: A case study examining algorithmic and editorial logic in Apple News. Proceedings of the 14th International AAAI Conference on Web and Social Media, ICWSM 2020, 2020, 36\u201347.","DOI":"10.1609\/icwsm.v14i1.7277"},{"key":"74_CR16","doi-asserted-by":"publisher","unstructured":"Barocas, S., & Selbst, A. D. (2016). Big Data\u2019s disparate impact. California Law Review, 104(3), 671\u2013732. https:\/\/doi.org\/10.15779\/Z38BG31","DOI":"10.15779\/Z38BG31"},{"key":"74_CR17","doi-asserted-by":"publisher","unstructured":"Bartley, N., Abeliuk, A., Ferrara, E., & Lerman, K. (2021). Auditing algorithmic bias on Twitter. ACM International Conference Proceeding Series, 65\u201373. https:\/\/doi.org\/10.1145\/3447535.3462491","DOI":"10.1145\/3447535.3462491"},{"key":"74_CR18","doi-asserted-by":"publisher","unstructured":"Baum, S. D. (2017). Social choice ethics in artificial intelligence. AI and Society, 1\u201312. https:\/\/doi.org\/10.1007\/s00146-017-0760-1","DOI":"10.1007\/s00146-017-0760-1"},{"key":"74_CR19","doi-asserted-by":"crossref","unstructured":"Berghout, E., Fijneman, R., Hendriks, L., de Boer, M., & Butijn, B.-J. (2023). Advanced digital auditing. Springer Nature.","DOI":"10.1007\/978-3-031-11089-4"},{"issue":"4","key":"74_CR20","doi-asserted-by":"publisher","first-page":"991","DOI":"10.1257\/0002828042002561","volume":"94","author":"M Bertrand","year":"2004","unstructured":"Bertrand, M., & Mullainathan, S. (2004). Are Emily and Greg more employable than Lakisha and Jamal: A field experiment on labor market discrimination. The American Economic Review, 94(4), 991\u20131013. https:\/\/doi.org\/10.1257\/0002828042002561","journal-title":"The American Economic Review"},{"key":"74_CR21","doi-asserted-by":"publisher","unstructured":"Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., Brynjolfsson, E., Buch, S., Card, D., \u2026 Liang, P. (2021). On the opportunities and risks of foundation models. ArXiv. https:\/\/doi.org\/10.48550\/arXiv.2108.07258","DOI":"10.48550\/arXiv.2108.07258"},{"key":"74_CR22","unstructured":"Brown, R. G. (1962). Changing audit objectives and techniques. The Accounting Review, 37(4), 696\u2013703. Retrieved July 20, 2023, from https:\/\/www.proquest.com\/docview\/1301318804"},{"key":"74_CR23","doi-asserted-by":"publisher","unstructured":"Brown, S., Davidovic, J., & Hasan, A. (2021). The algorithm audit: Scoring the algorithms that score us. Big Data & Society, 8.\u00a0https:\/\/doi.org\/10.1177\/2053951720983865","DOI":"10.1177\/2053951720983865"},{"key":"74_CR24","doi-asserted-by":"publisher","unstructured":"Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., \u2026 Amodei, D. (2020). Language models are few-shot learners. 34th Conference on Neural Information Processing Systems. https:\/\/doi.org\/10.48550\/arxiv.2005.14165","DOI":"10.48550\/arxiv.2005.14165"},{"key":"74_CR25","unstructured":"Brundage, M., Avin, S., Wang, J., Belfield, H., Krueger, G., Hadfield, G., \u2026 Anderljung, M. (2020). Toward trustworthy AI development: Mechanisms for supporting verifiable claims. ArXiv. Retrieved July 20, 2023, from http:\/\/arxiv.org\/abs\/2004.07213"},{"key":"74_CR26","doi-asserted-by":"publisher","first-page":"1","DOI":"10.2147\/OTT.S126905","volume":"1","author":"J Buolamwini","year":"2018","unstructured":"Buolamwini, J., & Gebru, T. (2018). Gender shades: Intersectional accuracy disparities in commercial gender classification. Conference on Fairness, Accountability, and Transparency, 1, 1\u201315. https:\/\/doi.org\/10.2147\/OTT.S126905","journal-title":"Conference on Fairness, Accountability, and Transparency"},{"issue":"5","key":"74_CR27","doi-asserted-by":"publisher","first-page":"825","DOI":"10.1111\/puar.13293","volume":"81","author":"M Busuioc","year":"2021","unstructured":"Busuioc, M. (2021). Accountable artificial intelligence: Holding algorithms to account. Public Administration Review, 81(5), 825\u2013836. https:\/\/doi.org\/10.1111\/puar.13293","journal-title":"Public Administration Review"},{"key":"74_CR28","doi-asserted-by":"publisher","first-page":"46","DOI":"10.1109\/VAST47406.2019.8986948","volume":"2019","author":"\u00c1A Cabrera","year":"2019","unstructured":"Cabrera, \u00c1. A., Epperson, W., Hohman, F., Kahng, M., Morgenstern, J., & Chau, D. H. (2019). FairVis: Visual analytics for discovering intersectional bias in machine learning. IEEE Conference on Visual Analytics Science and Technology, 2019, 46\u201356. https:\/\/doi.org\/10.1109\/VAST47406.2019.8986948","journal-title":"IEEE Conference on Visual Analytics Science and Technology"},{"issue":"1","key":"74_CR29","doi-asserted-by":"publisher","first-page":"92","DOI":"10.1007\/BF02691947","volume":"24","author":"FM Cancian","year":"1993","unstructured":"Cancian, F. M. (1993). Conflicts between activist research and academic success: Participatory research and alternative strategies. The American Sociologist, 24(1), 92\u2013106. https:\/\/doi.org\/10.1007\/BF02691947","journal-title":"The American Sociologist"},{"key":"74_CR30","unstructured":"Cartwright, N., & Montuschi, E. (2014). Philosophy of social science: A new introduction. Oxford University Press."},{"issue":"5\u20136","key":"74_CR31","doi-asserted-by":"publisher","first-page":"897","DOI":"10.1007\/S11186-020-09411-3\/METRICS","volume":"49","author":"A Christin","year":"2020","unstructured":"Christin, A. (2020). The ethnographer and the algorithm: Beyond the black box. Theory and Society, 49(5\u20136), 897\u2013918. https:\/\/doi.org\/10.1007\/S11186-020-09411-3\/METRICS","journal-title":"Theory and Society"},{"issue":"7","key":"74_CR32","doi-asserted-by":"publisher","first-page":"1","DOI":"10.3390\/info12070275","volume":"12","author":"P Cihon","year":"2021","unstructured":"Cihon, P., Schuett, J., & Baum, S. D. (2021). Corporate governance of artificial intelligence in the public interest. Information, 12(7), 1\u201330. https:\/\/doi.org\/10.3390\/info12070275","journal-title":"Information"},{"key":"74_CR33","doi-asserted-by":"publisher","unstructured":"Cobbe, J., Lee, M. S. A., & Singh, J. (2021). Reviewable automated decision-making: A framework for accountable algorithmic systems. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 598\u2013609. https:\/\/doi.org\/10.1145\/3442188.3445921","DOI":"10.1145\/3442188.3445921"},{"key":"74_CR34","unstructured":"Cosserat, G. W. (2004). Modern auditing (2nd ed.). John Wiley & Sons, Ltd."},{"key":"74_CR35","doi-asserted-by":"publisher","unstructured":"Costanza-Chock, S., Raji, I. D., & Buolamwini, J. (2022). Who audits the auditors? Recommendations from a field scan of the algorithmic auditing ecosystem. 2022 ACM Conference on Fairness, Accountability, and Transparency, 22, 1571\u20131583. https:\/\/doi.org\/10.1145\/3531146.3533213","DOI":"10.1145\/3531146.3533213"},{"key":"74_CR36","doi-asserted-by":"publisher","unstructured":"Coston, A., Guha, N., Ouyang, D., Lu, L., Chouldechova, A., & Ho, D. E. (2021). Leveraging administrative data for bias audits: Assessing disparate coverage with mobility data for COVID-19 Policy. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 173\u2013184. https:\/\/doi.org\/10.1145\/3442188.3445881","DOI":"10.1145\/3442188.3445881"},{"key":"74_CR37","doi-asserted-by":"publisher","unstructured":"Dafoe, A. (2017). AI Governance: A research agenda. American Journal of Psychiatry, 1\u201353. https:\/\/doi.org\/10.1176\/ajp.134.8.aj1348938","DOI":"10.1176\/ajp.134.8.aj1348938"},{"key":"74_CR38","doi-asserted-by":"publisher","unstructured":"Dash, A., Mukherjee, A., & Ghosh, S. (2019). A network-centric framework for auditing recommendation systems. IEEE INFOCOM 2019-IEEE Conference on Computer Communications, April, 1990\u20131998. https:\/\/doi.org\/10.1109\/INFOCOM.2019.8737486","DOI":"10.1109\/INFOCOM.2019.8737486"},{"key":"74_CR39","unstructured":"Dawson, M., Burrell, D. N., Rahim, E., & Brewster, S. (2010). Integrating software assurance into the software development life cycle (SDLC) meeting department of defense (DOD) demands. Journal of Information Systems Technology and Planning, 3(6), 49\u201353. Retrieved July 20, 2023, from www.academia.edu\/22484322"},{"issue":"3","key":"74_CR40","doi-asserted-by":"publisher","first-page":"305","DOI":"10.1007\/s10515-014-0168-9","volume":"23","author":"LA Dennis","year":"2016","unstructured":"Dennis, L. A., Fisher, M., Lincoln, N. K., Lisitsa, A., & Veres, S. M. (2016). Practical verification of decision-making in agent-based autonomous systems. Automated Software Engineering, 23(3), 305\u2013359. https:\/\/doi.org\/10.1007\/s10515-014-0168-9","journal-title":"Automated Software Engineering"},{"key":"74_CR41","doi-asserted-by":"publisher","unstructured":"Devos, A., Dhabalia, A., Shen, H., Holstein, K., & Eslami, M. (2022). Toward user-driven algorithm auditing: Investigating users\u2019 strategies for uncovering harmful algorithmic behavior. Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, 1\u201319. https:\/\/doi.org\/10.1145\/3491102.3517441","DOI":"10.1145\/3491102.3517441"},{"issue":"3","key":"74_CR42","doi-asserted-by":"publisher","first-page":"398","DOI":"10.1080\/21670811.2014.976411","volume":"3","author":"N Diakopoulos","year":"2015","unstructured":"Diakopoulos, N. (2015). Algorithmic accountability: Journalistic investigation of computational power structures. Digital Journalism, 3(3), 398\u2013415. https:\/\/doi.org\/10.1080\/21670811.2014.976411","journal-title":"Digital Journalism"},{"key":"74_CR43","doi-asserted-by":"publisher","DOI":"10.1201\/9781439822975","volume-title":"Auditing in the food industry: From safety and quality to environmental and other audits","author":"M Dillon","year":"2001","unstructured":"Dillon, M., & Griffith, C. J. (2001). Auditing in the food industry: From safety and quality to environmental and other audits. CRC Press."},{"key":"74_CR44","doi-asserted-by":"publisher","first-page":"93","DOI":"10.1007\/978-3-031-09846-8_7","volume-title":"The 2021 Yearbook of the Digital Ethics Lab","author":"M Durante","year":"2022","unstructured":"Durante, M., & Floridi, L. (2022). A legal principles-based framework for AI liability regulation. In J. M\u00f6kander & M. Ziosi (Eds.), The 2021 Yearbook of the Digital Ethics Lab (pp. 93\u2013112). Springer International Publishing."},{"key":"74_CR45","unstructured":"Economist Intelligence Unit. (2020). Staying ahead of the curve \u2013 The business case for responsible AI. Retrieved July 20, 2023, from https:\/\/www.eiu.com\/n\/staying-ahead-of-the-curve-the-business-case-for-responsible-ai\/"},{"issue":"3","key":"74_CR46","doi-asserted-by":"publisher","first-page":"46","DOI":"10.1109\/MSP.2018.2701152","volume":"16","author":"L Edwards","year":"2018","unstructured":"Edwards, L., & Veale, M. (2018). Enslaving the algorithm: From a \u201cright to an explanation\u201d to a \u201cright to better decisions\u201d? EEE Security & Privacy, 16(3), 46\u201354. https:\/\/doi.org\/10.1109\/MSP.2018.2701152","journal-title":"EEE Security & Privacy"},{"key":"74_CR47","unstructured":"Engler, A. C. (2021). Outside auditors are struggling to hold AI companies accountable. FastCompany. Retrieved July 20, 2023, from https:\/\/www.fastcompany.com\/90597594\/ai-algorithm-auditing-hirevue"},{"issue":"2","key":"74_CR48","doi-asserted-by":"publisher","first-page":"149","DOI":"10.1007\/s10676-016-9400-6","volume":"18","author":"A Etzioni","year":"2016","unstructured":"Etzioni, A., & Etzioni, O. (2016). AI assisted ethics. Ethics and Information Technology, 18(2), 149\u2013156. https:\/\/doi.org\/10.1007\/s10676-016-9400-6","journal-title":"Ethics and Information Technology"},{"key":"74_CR49","unstructured":"European Commission. (2021). Artificial Intelligence Act. Proposal for Regulation of the European Parliament and of the Council - Laying down Harmonised Rules on Artificial Intelligence and Amending Certain Union Legislative Acts. Retrieved July 20, 2023, from https:\/\/eur-lex.europa.eu\/-legal-content\/EN\/TXT\/?uri=celex%3A52021PC0206"},{"key":"74_CR50","unstructured":"European Parliament. (2016). Regulation (EU) 2016\/679 of the European Parliament and of the Council. In Official Journal of the European Union. Retrieved July 20, 2023, from https:\/\/eur-lex.europa.eu\/legal-content\/EN\/TXT\/PDF\/?uri=CELEX:32016R0679"},{"key":"74_CR51","doi-asserted-by":"publisher","unstructured":"European Parliamentary Research Service. (2019). A governance framework for algorithmic accountability and transparency. https:\/\/doi.org\/10.2861\/59990","DOI":"10.2861\/59990"},{"key":"74_CR52","unstructured":"European Parliamentary Research Service\u00a0(EPRS). (2022). Auditing the quality of datasets used in algorithmic decision-making systems. Retrieved July 20, 2023, from www.europarl.europa.eu\/regdata\/etudes-\/stud\/-2022\/729541\/eprs_stu(2022)729541_en.pdf"},{"key":"74_CR53","doi-asserted-by":"publisher","unstructured":"Evans, O., Cotton-Barratt, O., Finnveden, L., Bales, A., Balwit, A., Wills, P., Righetti, L., & Saunders, W. (2021). Truthful AI: Developing and governing AI that does not lie. ArXiv. https:\/\/doi.org\/10.48550\/arXiv.2110.06674","DOI":"10.48550\/arXiv.2110.06674"},{"key":"74_CR54","doi-asserted-by":"publisher","unstructured":"Falco, G., Shneiderman, B., Badger, J., Carrier, R., Dahbura, A., Danks, D., \u2026 Yeong, Z. K. (2021). Governing AI safety through independent audits. Nature Machine Intelligence 3(7), 566\u2013571. https:\/\/doi.org\/10.1038\/s42256-021-00370-7","DOI":"10.1038\/s42256-021-00370-7"},{"issue":"3","key":"74_CR55","doi-asserted-by":"publisher","first-page":"168","DOI":"10.7758\/rsf.2017.3.3.08","volume":"3","author":"HS Farber","year":"2017","unstructured":"Farber, H. S., Silverman, D., & Von Wachter, T. M. (2017). Factors determining callbacks to job applications by the unemployed: An audit study. Russell Sage Foundation Journal of the Social Sciences, 3(3), 168\u2013201. https:\/\/doi.org\/10.7758\/rsf.2017.3.3.08","journal-title":"Russell Sage Foundation Journal of the Social Sciences"},{"issue":"2","key":"74_CR56","doi-asserted-by":"publisher","first-page":"13","DOI":"10.1007\/s44206-022-00016-0","volume":"1","author":"A Fell\u00e4nder","year":"2022","unstructured":"Fell\u00e4nder, A., Rebane, J., Larsson, S., Wiggberg, M., & Heintz, F. (2022). Achieving a data-driven risk assessment methodology for ethical AI. Digital Society, 1(2), 13. https:\/\/doi.org\/10.1007\/s44206-022-00016-0","journal-title":"Digital Society"},{"key":"74_CR57","doi-asserted-by":"publisher","unstructured":"Fitzgerald, B., Stol, K. J., O\u2019Sullivan, R., & O\u2019Brien, D. (2013). Scaling agile methods to regulated environments: An industry case study. Proceedings - International Conference on Software Engineering, 863\u2013872. https:\/\/doi.org\/10.1109\/ICSE.2013.6606635","DOI":"10.1109\/ICSE.2013.6606635"},{"key":"74_CR58","unstructured":"Flint, D. (1988). Philosophy and principles of auditing: An introduction. Macmillan Education."},{"issue":"4","key":"74_CR59","doi-asserted-by":"publisher","first-page":"391","DOI":"10.1007\/s13347-017-0291-1","volume":"30","author":"L Floridi","year":"2017","unstructured":"Floridi, L. (2017). Infraethics\u2013on the conditions of possibility of morality. Philosophy and Technology, 30(4), 391\u2013394. https:\/\/doi.org\/10.1007\/s13347-017-0291-1","journal-title":"Philosophy and Technology"},{"key":"74_CR60","doi-asserted-by":"crossref","unstructured":"Floridi, L., & Chiriatti, M. (2020). GPT-3: Its nature, scope, limits, and consequences. In Minds and Machines (Vol. 30, Issue 4, pp. 681\u2013694). Springer.","DOI":"10.1007\/s11023-020-09548-1"},{"key":"74_CR61","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1162\/99608f92.8cd550d1","volume":"1","author":"L Floridi","year":"2019","unstructured":"Floridi, L., & Cowls, J. (2019). A unified framework of five principles for AI in society. Harvard Data Science Review, 1, 1\u201313. https:\/\/doi.org\/10.1162\/99608f92.8cd550d1","journal-title":"Harvard Data Science Review"},{"issue":"1","key":"74_CR62","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1007\/s11023-020-09521-y","volume":"30","author":"L Floridi","year":"2020","unstructured":"Floridi, L., & Strait, A. (2020). Ethical foresight analysis: What it is and why it is needed? Minds and Machines, 30(1), 77\u201397. https:\/\/doi.org\/10.1007\/s11023-020-09521-y","journal-title":"Minds and Machines"},{"key":"74_CR63","doi-asserted-by":"publisher","unstructured":"Floridi, L., Cowls, J., Beltrametti, M., Chatila, R., Chazerand, P., \u2026 Vayena, E. (2018). AI4People\u2014An ethical framework for a good AI society: Opportunities, risks, principles, and recommendations. Minds and Machines, 28(4), 689\u2013707. https:\/\/doi.org\/10.1007\/s11023-018-9482-5","DOI":"10.1007\/s11023-018-9482-5"},{"key":"74_CR64","doi-asserted-by":"publisher","unstructured":"Floridi, L., Holweg, M., Taddeo, M., Amaya Silva, J., M\u00f6kander, J., & Wen, Y. (2022). capAI \u2014 A procedure for conducting conformity assessment of AI systems in line with the EU Artificial Intelligence Act. SSRN Electronic Journal, 1\u201390. https:\/\/doi.org\/10.2139\/ssrn.4064091","DOI":"10.2139\/ssrn.4064091"},{"key":"74_CR65","unstructured":"Food and Drug Administration. (2021). Artificial intelligence and machine learning in software as a medical device. Retrieved July 20, 2023, from https:\/\/www.fda.gov\/medical-devices\/software-medical-device-samd\/artificial-intelligence-and-machine-learning-software-medical-device"},{"key":"74_CR66","doi-asserted-by":"publisher","DOI":"10.1515\/9780691191959","volume-title":"The technology trap: Capital, labor, and power in the age of automation","author":"CB Frey","year":"2019","unstructured":"Frey, C. B. (2019). The technology trap: Capital, labor, and power in the age of automation. Princeton University Press."},{"key":"74_CR67","doi-asserted-by":"crossref","unstructured":"Gaddis, S. M. (2018). An introduction to audit studies in the social sciences. Springer International Publishing.","DOI":"10.31235\/osf.io\/e5hfc"},{"issue":"7","key":"74_CR68","doi-asserted-by":"publisher","first-page":"471","DOI":"10.1093\/occmed\/49.7.471","volume":"49","author":"AS Gay","year":"1999","unstructured":"Gay, A. S., & New, N. H. (1999). Auditing health and safety management systems: A regulator\u2019s view. Occupational Medicine, 49(7), 471\u2013473. https:\/\/doi.org\/10.1093\/occmed\/49.7.471","journal-title":"Occupational Medicine"},{"issue":"12","key":"74_CR69","doi-asserted-by":"publisher","first-page":"86","DOI":"10.1145\/3458723","volume":"64","author":"T Gebru","year":"2021","unstructured":"Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Iii, H. D., & Crawford, K. (2021). Datasheets for datasets. Communications of the ACM, 64(12), 86\u201392. https:\/\/doi.org\/10.1145\/3458723","journal-title":"Communications of the ACM"},{"key":"74_CR70","doi-asserted-by":"crossref","unstructured":"Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. (2020). RealToxicityPrompts: Evaluating neural toxic degeneration in language models. Findings of the Association for Computational Linguistics: EMNLP, 3356\u20133369. Retrieved July 20, 2023, from http:\/\/arxiv.org\/abs\/2009.11462","DOI":"10.18653\/v1\/2020.findings-emnlp.301"},{"key":"74_CR71","doi-asserted-by":"publisher","first-page":"17","DOI":"10.1007\/s44206-022-00015-1","volume":"1","author":"D Gesmann-Nuissl","year":"2022","unstructured":"Gesmann-Nuissl, D., & Kunitz, S. (2022). Auditing of AI in railway technology \u2013 A European legal approach. DISO, 1, 17. https:\/\/doi.org\/10.1007\/s44206-022-00015-1","journal-title":"DISO"},{"key":"74_CR72","unstructured":"Gibson Dunn. (2023). New York city proposes rules to clarify upcoming artificial intelligence law for employers. Retrieved July 20, 2023, from https:\/\/www.gibsondunn.com\/new-york-city-proposes-rules-to-clarify-upcoming-artificial-intelligence-law-for-employers\/"},{"key":"74_CR73","doi-asserted-by":"publisher","unstructured":"Goel, K., Rajani, N., Vig, J., Taschdjian, Z., Bansal, M., & R\u00e9, C. (2021). Robustness gym: Unifying the NLP evaluation landscape. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Demonstrations, 42\u201355. https:\/\/doi.org\/10.18653\/V1\/2021.NAACL-DEMOS.6","DOI":"10.18653\/V1\/2021.NAACL-DEMOS.6"},{"key":"74_CR74","unstructured":"Government of Canada. (2019). Directive on Automated Decision-Making. Retrieved July 20, 2023, from www.tbs-sct.canada.ca\/pol\/doc-eng.aspx?id=32592"},{"key":"74_CR75","unstructured":"Government of Singapore. (2020). Model AI Governance Framework. Personal Data Protection Commission (PDPC). Retrieved July 20, 2023, from www.pdpc.gov.sg\/-\/media\/files\/pdpc\/pdf-files\/resource-for-organisation\/ai\/sgmodelaigovframework2.pdf"},{"key":"74_CR76","unstructured":"Grand View Research. (2017). Financial auditing professional services market report, 2025. Retrieved July 20, 2023, from https:\/\/www.grandviewresearch.com\/industry-analysis\/financial-auditing-professional-services-market"},{"key":"74_CR77","volume-title":"Comtemporary auditing","author":"K Gupta","year":"2004","unstructured":"Gupta, K. (2004). Comtemporary auditing. McGraw Hill."},{"key":"74_CR78","unstructured":"Guszcza, J., Rahwan, I., Bible, W., Cebrian, M., & Katyal, V. (2018). Why we need to audit algorithms. Harward Business Review. Retrieved July 20, 2023, from https:\/\/hbr.org\/2018\/11\/why-we-need-"},{"key":"74_CR79","unstructured":"Hale, C. (2017). What is activist research? Social Science Research Council. Retrieved July 20, 2023, from https:\/\/items.ssrc.org\/from-our-archives\/what-is-activist-research\/"},{"issue":"3","key":"74_CR80","doi-asserted-by":"publisher","first-page":"371","DOI":"10.1016\/0377-2217(86)90139-6","volume":"26","author":"JV Hansen","year":"1986","unstructured":"Hansen, J. V., & Messier, W. F. (1986). A knowledge-based expert system for auditing advanced computer systems. European Journal of Operational Research, 26(3), 371\u2013379. https:\/\/doi.org\/10.1016\/0377-2217(86)90139-6","journal-title":"European Journal of Operational Research"},{"issue":"2","key":"74_CR81","doi-asserted-by":"publisher","first-page":"14","DOI":"10.1007\/s44206-022-00017-z","volume":"1","author":"A Hasan","year":"2022","unstructured":"Hasan, A., Brown, S., Davidovic, J., Lange, B., & Regan, M. (2022). Algorithmic bias and risk assessments: Lessons from practice. Digital Society, 1(2), 14. https:\/\/doi.org\/10.1007\/s44206-022-00017-z","journal-title":"Digital Society"},{"key":"74_CR82","unstructured":"Hill, K. (2020). Twitter tells facial recognition trailblazer to stop using site\u02bcs photos. New York Tmes. Retrieved July 20, 2023, from https:\/\/www.nytimes.com\/2020\/01\/22\/technology\/clearview-ai-twitter-letter.html?searchResultPosition=11\/"},{"key":"74_CR83","unstructured":"Holland, S., Hosny, A., Newman, S., Joseph, J., & Chmielinski, K. (2018). The dataset nutrition label: A framework to drive higher data quality standards. ArXiv, May. Retrieved July 20, 2023, from http:\/\/arxiv.org\/abs\/1805.03677"},{"key":"74_CR84","doi-asserted-by":"publisher","first-page":"11","DOI":"10.1007\/978-3-030-12524-0_2","volume":"95","author":"IEEE Standard Association","year":"2019","unstructured":"IEEE Standard Association. (2019). Ethically aligned design. Intelligent Systems, Control and Automation: Science and Engineering, 95, 11\u201316. https:\/\/doi.org\/10.1007\/978-3-030-12524-0_2","journal-title":"Intelligent Systems, Control and Automation: Science and Engineering"},{"key":"74_CR85","unstructured":"Information Commissioner\u2019s Office\u00a0(ISO). (2020). Guidance on the AI auditing framework: Draft guidance for consultation. Retrieved July 20, 2023, from https:\/\/ico.org.uk\/media\/about-the-ico\/consultations\/-2617219\/guidance-on-the-ai-auditing-framework-draft-for-consultation.pdf"},{"key":"74_CR86","volume-title":"The IIA\u2019s artificial intelligence auditing framework","author":"Institute of Internal Auditors","year":"2018","unstructured":"Institute of Internal Auditors. (2018). The IIA\u2019s artificial intelligence auditing framework. Global Perspectives. Retrieved July 20, 2023, from https:\/\/www.nist.gov\/system\/files\/documents\/2021\/10\/04\/GPI-Artificial-Intelligence-Part-III.pdf"},{"key":"74_CR87","unstructured":"International Organization for Standardization. (2022). ISO\/IEC 38507:2022 - Information technology \u2014 Governance of IT \u2014 Governance implications of the use of artificial intelligence by organizations. Retrieved July 20, 2023, from https:\/\/www.iso.org\/standard\/56641.html?browse=tc"},{"key":"74_CR88","doi-asserted-by":"publisher","unstructured":"Jager, T., & Westhoek, E. (2023). Keeping control on deep learning image recognition algorithms. Advanced Digital Auditing, 121\u2013148. https:\/\/doi.org\/10.1007\/978-3-031-11089-4_6","DOI":"10.1007\/978-3-031-11089-4_6"},{"key":"74_CR89","doi-asserted-by":"publisher","unstructured":"Jobin, A., Ienca, M., & Vayena, E. (2019). The global landscape of AI ethics guidelines. Nature Machine Intelligence,\u00a01, 389\u2013399. https:\/\/doi.org\/10.1038\/s42256-019-0088-2","DOI":"10.1038\/s42256-019-0088-2"},{"key":"74_CR90","unstructured":"Kak, A., & West, S. M. (2023). Confronting tech power 2023 Landscape. AI Now Institute. Retrieved July 20, 2023, from https:\/\/ainowinstitute.org\/2023-landscape"},{"key":"74_CR91","doi-asserted-by":"publisher","unstructured":"Kassir, S., Baker, L., Dolphin, J., & Polli, F. (2022). AI for hiring in context: A perspective on overcoming the unique challenges of employment research to mitigate disparate impact. AI and Ethics, 1\u201324. https:\/\/doi.org\/10.1007\/s43681-022-00208-x","DOI":"10.1007\/s43681-022-00208-x"},{"key":"74_CR92","doi-asserted-by":"publisher","unstructured":"Kazim, E., & Koshiyama, A. (2020). AI assurance processes. SSRN Electronic Journal, 1\u20139. https:\/\/doi.org\/10.2139\/ssrn.3685087","DOI":"10.2139\/ssrn.3685087"},{"key":"74_CR93","unstructured":"Kearns, M., Neel, S., Roth, A., & Wu, Z. S. (2018). Preventing fairness gerrymandering: Auditing and learning for subgroup fairness. 35th International Conference on Machine Learning, ICML 2018, 6, 4008\u20134016. Retrieved July 20, 2023, from https:\/\/proceedings.mlr.press\/v80\/kearns18a.html"},{"key":"74_CR94","doi-asserted-by":"publisher","unstructured":"Keyes, O., Durbin, M., & Hutson, J. (2019). A mulching proposal: Analysing and improving an algorithmic system for turning the elderly into high-nutrient slurry. Conference on Human Factors in Computing Systems, 1\u201311. https:\/\/doi.org\/10.1145\/3290607.3310433","DOI":"10.1145\/3290607.3310433"},{"key":"74_CR95","first-page":"189","volume":"166","author":"P Kim","year":"2017","unstructured":"Kim, P. (2017). Auditing algorithms for discrimination. University of Pennsylvania Law Review, 166, 189\u2013203.","journal-title":"University of Pennsylvania Law Review"},{"key":"74_CR96","doi-asserted-by":"publisher","first-page":"2611","DOI":"10.48550\/arXiv.2102.04130","volume":"34","author":"HR Kirk","year":"2021","unstructured":"Kirk, H. R., Jun, Y., Iqbal, H., Benussi, E., Volpin, F., Dreyer, F. A., Shtedritski, A., & Asano, Y. M. (2021). Bias out-of-the-box: An empirical analysis of intersectional occupational biases in popular generative language models. Advances in Neural Information Processing Systems, 34, 2611\u20132642. https:\/\/doi.org\/10.48550\/arXiv.2102.04130","journal-title":"Advances in Neural Information Processing Systems"},{"key":"74_CR97","first-page":"663","volume-title":"Line operation safety audits: Definition and operating characteristics","author":"J Klinect","year":"2003","unstructured":"Klinect, J., Murray, P., Merritt, A., & Helmreich, R. (2003). Line operation safety audits: Definition and operating characteristics (pp. 663\u2013668). Proceedings of the 12th International Symposium on Aviation Psychology."},{"issue":"1","key":"74_CR98","doi-asserted-by":"publisher","first-page":"42","DOI":"10.1109\/MSP.2017.16","volume":"15","author":"M Kolhar","year":"2017","unstructured":"Kolhar, M., Abu-Alhaj, M. M., & Abd El-Atty, S. M. (2017). Cloud data auditing techniques with a focus on privacy and security. IEEE Security and Privacy, 15(1), 42\u201351. https:\/\/doi.org\/10.1109\/MSP.2017.16","journal-title":"IEEE Security and Privacy"},{"issue":"4","key":"74_CR99","doi-asserted-by":"publisher","first-page":"40","DOI":"10.1109\/MC.2021.3067225","volume":"55","author":"A Koshiyama","year":"2022","unstructured":"Koshiyama, A., Kazim, E., & Treleaven, P. (2022). Algorithm auditing: Managing the legal, ethical, and technological risks of artificial intelligence, machine learning, and associated algorithms. IEEE, 55(4), 40\u201350. https:\/\/doi.org\/10.1109\/MC.2021.3067225","journal-title":"IEEE"},{"issue":"2","key":"74_CR100","doi-asserted-by":"publisher","first-page":"168","DOI":"10.1177\/0022146516647098","volume":"57","author":"H Kugelmass","year":"2016","unstructured":"Kugelmass, H. (2016). \u201cSorry, I\u2019m Not Accepting New Patients\u201d: An audit study of access to mental health care. Journal of Health and Social Behavior, 57(2), 168\u2013183. https:\/\/doi.org\/10.1177\/0022146516647098","journal-title":"Journal of Health and Social Behavior"},{"key":"74_CR101","unstructured":"Kuusisto, A. (2001). Safety management systems Audit tools and reliability of auditing at 12 o\u2019clock noon [Doctoral dssertation, Tampere University of Technology]. Retrieved July 20, 2023, from https:\/\/publications.vtt.fi\/pdf\/publications\/2000\/P428.pdf"},{"key":"74_CR102","unstructured":"LaBrie, R. C., & Steinke, G. H. (2019). Towards a framework for ethical audits of AI algorithms. 25th Americas Conference on Information Systems, 1\u20135. Retrieved July 20, 2023, from https:\/\/dblp.org\/rec\/conf\/amcis\/LaBrieS19.html"},{"issue":"1","key":"74_CR103","doi-asserted-by":"publisher","first-page":"36","DOI":"10.1037\/amp0000972","volume":"78","author":"RN Landers","year":"2022","unstructured":"Landers, R. N., & Behrend, T. S. (2022). Auditing the AI auditors: A framework for evaluating fairness and bias in high stakes AI predictive models. American Psychologist, 78(1), 36\u201349. https:\/\/doi.org\/10.1037\/amp0000972","journal-title":"American Psychologist"},{"key":"74_CR104","doi-asserted-by":"publisher","unstructured":"Larsson, S., & Heintz, F. (2020). Transparency in artificial intelligence. New Media & Society, 9(2), 1\u201316. https:\/\/doi.org\/10.14763\/2020.2.1469","DOI":"10.14763\/2020.2.1469"},{"key":"74_CR105","doi-asserted-by":"publisher","DOI":"10.1016\/j.clsr.2021.105613","volume":"43","author":"J Laux","year":"2021","unstructured":"Laux, J., Wachter, S., & Mittelstadt, B. (2021). Taming the few: Platform regulation, independent audits, and the risks of capture created by the DMA and DSA. Computer Law & Security Review, 43, 105613. https:\/\/doi.org\/10.1016\/j.clsr.2021.105613","journal-title":"Computer Law & Security Review"},{"key":"74_CR106","doi-asserted-by":"publisher","unstructured":"Lee, S. C. (2021). Auditing algorithms: A rational counterfactual framework. Journal of International Technology and Information Management, 30(2), 2021. https:\/\/doi.org\/10.58729\/1941-6679.1464","DOI":"10.58729\/1941-6679.1464"},{"key":"74_CR107","unstructured":"Lee, T.-H., & Azham, M. A. (2008). The evolution of auditing: An analysis of the historical development. Journal of Modern Accounting and Auditing, 4(12), 1548\u20136583. Retrieved July 20, 2023, from https:\/\/www.researchgate.net\/publication\/339251518"},{"key":"74_CR108","doi-asserted-by":"crossref","unstructured":"Leveson, N. (2011). Engineering a safer world: Systems thinking applied to safety. MIT Press.","DOI":"10.7551\/mitpress\/8179.001.0001"},{"key":"74_CR109","doi-asserted-by":"publisher","first-page":"24","DOI":"10.1007\/s44206-022-00023-1","volume":"1","author":"R Light","year":"2022","unstructured":"Light, R., & Panai, E. (2022). The self-synchronisation of AI ethical principles. DISO, 1, 24. https:\/\/doi.org\/10.1007\/s44206-022-00023-1","journal-title":"DISO"},{"key":"74_CR110","doi-asserted-by":"publisher","unstructured":"Loi, M., Ferrario, A., & Vigan\u00f2, E. (2020). Transparency as design publicity: Explaining and justifying inscrutable algorithms. Ethics and Information Technology, Lipton 2018. https:\/\/doi.org\/10.1007\/s10676-020-09564-w","DOI":"10.1007\/s10676-020-09564-w"},{"key":"74_CR111","doi-asserted-by":"publisher","unstructured":"Luckcuck, M., Farrell, M., Dennis, L. A., Dixon, C., & Fisher, M. (2019). A summary of formal specification and verification of autonomous robotic systems. Integrated Formal Methods: 15th International Conference, IFM 2019, Bergen, Norway, December 2\u20136, 2019, Proceedings, 11918(5), 538\u2013541. https:\/\/doi.org\/10.1007\/978-3-030-34968-4_33","DOI":"10.1007\/978-3-030-34968-4_33"},{"key":"74_CR112","unstructured":"Lurie, E., & Mustafaraj, E. (2019). Opening up the black box: Auditing Google\u2019s top stories algorithm. 32nd FLAIRS Conference 2019, 376\u2013381. Retrieved July 20, 2023, from https:\/\/aaai.org\/ocs\/index.php\/FLAIRS\/FLAIRS19\/paper\/view\/18316\/17433"},{"issue":"1","key":"74_CR113","doi-asserted-by":"publisher","first-page":"132","DOI":"10.1016\/j.acra.2019.09.009","volume":"27","author":"V Mahajan","year":"2020","unstructured":"Mahajan, V., Venugopal, V. K., Murugavel, M., & Mahajan, H. (2020). The algorithmic audit: Working with vendors to validate radiology-AI algorithms\u2014How we do it. Academic Radiology, 27(1), 132\u2013135. https:\/\/doi.org\/10.1016\/j.acra.2019.09.009","journal-title":"Academic Radiology"},{"key":"74_CR114","doi-asserted-by":"publisher","DOI":"10.1007\/s43681-022-00143-x","author":"M M\u00e4ntym\u00e4ki","year":"2022","unstructured":"M\u00e4ntym\u00e4ki, M., Minkkinen, M., Birkstedt, T., & Viljanen, M. (2022). Defining organizational AI governance. AI and Ethics. https:\/\/doi.org\/10.1007\/s43681-022-00143-x","journal-title":"AI and Ethics"},{"key":"74_CR115","doi-asserted-by":"publisher","unstructured":"Marda, V., & Narayan, S. (2021). On the importance of ethnographic methods in AI research. In Nature Machine Intelligence (Vol. 3, Issue 3, pp. 187\u2013189). Nature Research. https:\/\/doi.org\/10.1038\/s42256-021-00323-0","DOI":"10.1038\/s42256-021-00323-0"},{"key":"74_CR116","unstructured":"Merrer, E. Le, Pons, R., & Tr\u00e9dan, G. (2022). Algorithmic audits of algorithms, and the law (hal-03583919). Retrieved July 20, 2023, from http:\/\/arxiv.org\/abs\/2203.03711"},{"issue":"4","key":"74_CR117","doi-asserted-by":"publisher","first-page":"272","DOI":"10.1561\/1100000083","volume":"14","author":"D Metaxa","year":"2021","unstructured":"Metaxa, D., Park, J. S., Robertson, R. E., Karahalios, K., Wilson, C., Hancock, J., & Sandvig, C. (2021). Auditing algorithms. Foundations and Trends in Human-Computer Interaction, 14(4), 272\u2013344. https:\/\/doi.org\/10.1561\/1100000083","journal-title":"Foundations and Trends in Human-Computer Interaction"},{"key":"74_CR118","doi-asserted-by":"publisher","unstructured":"Metcalf, J., Anne Watkins, E., Singh, R., Clare Elish, M., & Moss, E. (2021). Algorithmic impact assessments and accountability: The co-construction of impacts. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 735\u2013746. https:\/\/doi.org\/10.1145\/3442188.3445935","DOI":"10.1145\/3442188.3445935"},{"key":"74_CR119","doi-asserted-by":"crossref","unstructured":"Mikians, J., Gyarmati, L., Erramilli, V., & Laoutaris, N. (2012). Detecting price and search discrimination on the Internet. Hotnets. Retrieved July 20, 2023, from www.researchgate.net\/publication\/232321801","DOI":"10.1145\/2390231.2390245"},{"issue":"3","key":"74_CR120","doi-asserted-by":"publisher","first-page":"21","DOI":"10.1007\/s44206-022-00022-2","volume":"1","author":"M Minkkinen","year":"2022","unstructured":"Minkkinen, M., Laine, J., & M\u00e4ntym\u00e4ki, M. (2022). Continuous auditing of artificial intelligence: A conceptualization and assessment of tools and frameworks. Digital Society, 1(3), 21. https:\/\/doi.org\/10.1007\/s44206-022-00022-2","journal-title":"Digital Society"},{"key":"74_CR121","doi-asserted-by":"publisher","unstructured":"Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019). Model cards for model reporting. FAT* 2019 - Proceedings of the 2019 Conference on Fairness, Accountability, and Transparency, 220\u2013229. https:\/\/doi.org\/10.1145\/3287560.3287596","DOI":"10.1145\/3287560.3287596"},{"key":"74_CR122","unstructured":"Mittelstadt, B. (2016). Auditing for transparency in content personalization systems. International Journal of Communication, 10, 4991\u20135002. Retrieved July 20, 2023, from www.researchgate.net\/publication\/309136069"},{"key":"74_CR123","doi-asserted-by":"publisher","first-page":"2","DOI":"10.1007\/s11023-021-09557-8","volume":"0123456789","author":"J M\u00f6kander","year":"2021","unstructured":"M\u00f6kander, J., & Floridi, L. (2021). Ethics-based auditing to develop trustworthy AI. Minds and Machines, 0123456789, 2\u20136. https:\/\/doi.org\/10.1007\/s11023-021-09557-8","journal-title":"Minds and Machines"},{"key":"74_CR124","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s42256-022-00504-5","volume":"2022","author":"J M\u00f6kander","year":"2022","unstructured":"M\u00f6kander, J., & Floridi, L. (2022a). From algorithmic accountability to digital governance. Nature Machine Intelligence, 2022, 1\u20132. https:\/\/doi.org\/10.1038\/s42256-022-00504-5","journal-title":"Nature Machine Intelligence"},{"key":"74_CR125","doi-asserted-by":"publisher","unstructured":"M\u00f6kander, J., & Floridi, L. (2022b). Operationalising AI governance through ethics-based auditing: An industry case study. AI and Ethics, 1\u201318. https:\/\/doi.org\/10.1007\/s43681-022-00171-7","DOI":"10.1007\/s43681-022-00171-7"},{"key":"74_CR126","doi-asserted-by":"publisher","unstructured":"M\u00f6kander, J., Morley, J., Taddeo, M., & Floridi, L. (2021). Ethics-based auditing of automated decision-making systems: Nature, scope, and limitations. Science and Engineering Ethics, 1\u201330. https:\/\/doi.org\/10.1007\/s11948-021-00319-4","DOI":"10.1007\/s11948-021-00319-4"},{"issue":"2","key":"74_CR127","doi-asserted-by":"publisher","first-page":"241","DOI":"10.1007\/s11023-021-09577-4","volume":"32","author":"J M\u00f6kander","year":"2022","unstructured":"M\u00f6kander, J., Axente, M., Casolari, F., & Floridi, L. (2022a). Conformity assessments and post-market monitoring: A guide to the role of auditing in the proposed European AI regulation. Minds and Machines, 32(2), 241\u2013268. https:\/\/doi.org\/10.1007\/s11023-021-09577-4","journal-title":"Minds and Machines"},{"key":"74_CR128","doi-asserted-by":"publisher","first-page":"1068361","DOI":"10.3389\/fcomp.2022.1068361","volume":"4","author":"J M\u00f6kander","year":"2022","unstructured":"M\u00f6kander, J., Sheth, M., Gersbro-Sundler, M., Blomgren, P., & Floridi, L. (2022b). Challenges and best practices in corporate AI governance: Lessons from the biopharmaceutical industry. Frontiers in Computer Science, 4, 1068361. https:\/\/doi.org\/10.3389\/fcomp.2022.1068361","journal-title":"Frontiers in Computer Science"},{"key":"74_CR129","doi-asserted-by":"publisher","DOI":"10.1007\/s43681-023-00289-2","author":"J M\u00f6kander","year":"2023","unstructured":"M\u00f6kander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2023a). Auditing large language models: A three-layered approach. AI and Ethics. https:\/\/doi.org\/10.1007\/s43681-023-00289-2","journal-title":"AI and Ethics"},{"key":"74_CR130","doi-asserted-by":"publisher","first-page":"221","DOI":"10.1007\/s11023-022-09620-y","volume":"33","author":"J M\u00f6kander","year":"2023","unstructured":"M\u00f6kander, J., Sheth, M., Watson, D. S., et al. (2023b). The switch, the ladder, and the matrix: Models for classifying AI systems. Minds & Machines, 33, 221\u2013248. https:\/\/doi.org\/10.1007\/s11023-022-09620-y","journal-title":"Minds & Machines"},{"key":"74_CR131","doi-asserted-by":"publisher","unstructured":"Morina, G., Oliinyk, V., Waton, J., Marusic, I., & Georgatzis, K. (2019). Auditing and achieving intersectional fairness in classification problems. ArXiv. https:\/\/doi.org\/10.48550\/arXiv.1911.01468","DOI":"10.48550\/arXiv.1911.01468"},{"issue":"2","key":"74_CR132","doi-asserted-by":"publisher","first-page":"239","DOI":"10.1007\/s11023-021-09563-w","volume":"31","author":"J Morley","year":"2021","unstructured":"Morley, J., Elhalal, A., Garcia, F., Kinsey, L., Mokander, J., & Floridi, L. (2021). Ethics as a service: A pragmatic operationalisation of AI Ethics. Minds and Machines, 31(2), 239\u2013256. https:\/\/doi.org\/10.1007\/s11023-021-09563-w","journal-title":"Minds and Machines"},{"key":"74_CR133","doi-asserted-by":"publisher","DOI":"10.1016\/J.JSS.2021.111050","volume":"181","author":"L Myllyaho","year":"2021","unstructured":"Myllyaho, L., Raatikainen, M., M\u00e4nnist\u00f6, T., Mikkonen, T., & Nurminen, J. K. (2021). Systematic literature review of validation methods for AI systems. Journal of Systems and Software, 181, 111050. https:\/\/doi.org\/10.1016\/J.JSS.2021.111050","journal-title":"Journal of Systems and Software"},{"key":"74_CR134","unstructured":"Narula, N., Vasquez, W., & Virza, M. (2018). zkLedger: Privacy-preserving auditing for distributed ledgers. Proceedings of the 15th USENIX Symposium on Networked Systems Design and Implementation, 65\u201380. Retrieved July 20, 2023, from www.usenix.org\/system\/files\/conference\/nsdi18\/nsdi18-narula.pdf"},{"key":"74_CR135","unstructured":"National Institute of Standard and Technology (NIST). (2022). AI risk management framework. Retrieved July 20, 2023, from https:\/\/www.nist.gov\/itl\/ai-risk-management-framework"},{"issue":"3","key":"74_CR136","doi-asserted-by":"publisher","first-page":"915","DOI":"10.2307\/2946676","volume":"111","author":"D Neumark","year":"1996","unstructured":"Neumark, D., Bank, R. J., & Van Nort, K. D. (1996). Sex discrimination in restaurant hiring: An audit study. The Quarterly Journal of Economics, 111(3), 915\u2013941. https:\/\/doi.org\/10.2307\/2946676","journal-title":"The Quarterly Journal of Economics"},{"key":"74_CR137","doi-asserted-by":"publisher","unstructured":"Niemiec, E. (2022). Will the EU Medical Device Regulation help to improve the safety and performance of medical AI devices? Digital Health, 1\u20138. https:\/\/doi.org\/10.1177\/20552076221089079","DOI":"10.1177\/20552076221089079"},{"key":"74_CR138","unstructured":"O\u2019Neil, C. (2016). Weapons of math destruction: How big data increases inequality and threatens democracy. Crown Books."},{"key":"74_CR139","doi-asserted-by":"publisher","unstructured":"Organisation for Economic Co-operation and Development. (2015). Principles of Corporate Governance, 2015 In G20\/OECD Principles of Corporate Governance 2015 OECD Publishing https:\/\/doi.org\/10.1787\/9789264236882-EN","DOI":"10.1787\/9789264236882-EN"},{"key":"74_CR140","unstructured":"Organisation for Economic Co-operation and Development\u00a0(OECD). (2019). Recommendation of the council on artificial intelligence. Retrieved July 20, 2023, from https:\/\/legalinstruments.oecd.org\/en\/instruments\/oecd-legal-0449"},{"key":"74_CR141","doi-asserted-by":"publisher","unstructured":"Panigutti, C., Perotti, A., Panisson, A., Bajardi, P., & Pedreschi, D. (2021). FairLens: Auditing black-box clinical decision support systems. Information Processing and Management, 58(5). https:\/\/doi.org\/10.1016\/j.ipm.2021.102657","DOI":"10.1016\/j.ipm.2021.102657"},{"key":"74_CR142","doi-asserted-by":"publisher","unstructured":"Parikh, P. M., Shah, D. M., Parikh, K. P., Parikh, P. M., Shah, D. M., & Parikh, K. P. (2023). Judge Juan Manuel Padilla Garcia, ChatGPT, and a controversial medicolegal milestone. Indian Journal of Medical Sciences, 75(1), 3\u20138. https:\/\/doi.org\/10.25259\/IJMS_31_2023","DOI":"10.25259\/IJMS_31_2023"},{"issue":"3","key":"74_CR143","doi-asserted-by":"publisher","first-page":"457","DOI":"10.1086\/708691","volume":"87","author":"WS Parker","year":"2020","unstructured":"Parker, W. S. (2020). Model evaluation: An adequacy-for-purpose view. Philosophy of Science, 87(3), 457\u2013477. https:\/\/doi.org\/10.1086\/708691","journal-title":"Philosophy of Science"},{"key":"74_CR144","unstructured":"Pedreschi, D., Giannotti, F., Guidotti, R., Monreale, A., Pappalardo, L., Ruggieri, S., & Turini, F. (2018). Open the black box data-driven explanation of black box decision systems. Computer Science, 1(1), 1\u201315. Retrieved July 20, 2023, from http:\/\/arxiv.org\/abs\/1806.09936"},{"key":"74_CR145","unstructured":"Pentland, A. (2019). A perspective on legal algorithms. MIT Computational Law Report. Retrieved July 20, 2023, from https:\/\/law.mit.edu\/pub\/aperspectiveonlegalalgorithms\/release\/3"},{"key":"74_CR146","unstructured":"Perrault, R., Shoham, Y., Brynjolfsson, E., Clark, J., Etchemendy, J., Grosz, B., Lyons, T., Manyika, J., Mishra, S., & Niebles, J. (2019). The AI index 2019 annual report. Retrieved July 20, 2023, from https:\/\/hai.stanford.edu\/sites\/default\/files\/ai_index_2019_report.pdf"},{"key":"74_CR147","unstructured":"Peter, F. (2010). Political Legitimacy. In Stanford Encyclopedia of Philosophy. Stanford Univerity Press. https:\/\/plato.stanford.edu\/entries\/legitimacy\/"},{"issue":"1","key":"74_CR148","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/2193-8997-2-4","volume":"2","author":"G Piern\u00e9","year":"2013","unstructured":"Piern\u00e9, G. (2013). Hiring discrimination based on national origin and religious closeness: Results from a field experiment in the Paris area. IZA Journal of Labor Economics, 2(1), 1\u20134. https:\/\/doi.org\/10.1186\/2193-8997-2-4","journal-title":"IZA Journal of Labor Economics"},{"issue":"2","key":"74_CR149","doi-asserted-by":"publisher","first-page":"686","DOI":"10.1016\/J.FOODCONT.2012.07.044","volume":"30","author":"DA Powell","year":"2013","unstructured":"Powell, D. A., Erdozain, S., Dodd, C., Costa, R., Morley, K., & Chapman, B. J. (2013). Audits and inspections are never enough: A critique to enhance food safety. Food Control, 30(2), 686\u2013691. https:\/\/doi.org\/10.1016\/J.FOODCONT.2012.07.044","journal-title":"Food Control"},{"key":"74_CR150","unstructured":"Power, M. (1997). The audit society: Rituals of verification. Oxford University Press."},{"key":"74_CR151","doi-asserted-by":"publisher","unstructured":"Raji, I. D., & Buolamwini, J. (2019). Actionable auditing: Investigating the impact of publicly naming biased performance results of commercial AI products. AIES 2019 - Proceedings of the 2019 AAAI\/ACM Conference on AI, Ethics, and Society, 429\u2013435. https:\/\/doi.org\/10.1145\/3306618.3314244","DOI":"10.1145\/3306618.3314244"},{"key":"74_CR152","doi-asserted-by":"publisher","unstructured":"Raji, I. D., Kumar, I. E., Horowitz, A., & Selbst, A. (2022). The fallacy of AI functionality. ACM International Conference Proceeding Series, 959\u2013972. https:\/\/doi.org\/10.1145\/3531146.3533158","DOI":"10.1145\/3531146.3533158"},{"key":"74_CR153","doi-asserted-by":"crossref","unstructured":"Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020). Closing the AI accountability gap: Defining an end-to-end framework for internal algorithmic auditing. Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, 33\u201344.","DOI":"10.1145\/3351095.3372873"},{"issue":"6","key":"74_CR154","doi-asserted-by":"publisher","first-page":"2153","DOI":"10.1007\/S10618-022-00861-0\/FIGURES\/8","volume":"36","author":"AK Rhea","year":"2022","unstructured":"Rhea, A. K., Markey, K., D\u2019Arinzo, L., Schellmann, H., Sloane, M., Squires, P., Arif Khan, F., & Stoyanovich, J. (2022). An external stability audit framework to test the validity of personality prediction in AI hiring. Data Mining and Knowledge Discovery, 36(6), 2153\u20132193. https:\/\/doi.org\/10.1007\/S10618-022-00861-0\/FIGURES\/8","journal-title":"Data Mining and Knowledge Discovery"},{"key":"74_CR155","unstructured":"Robertson, A. (2022). Clearview AI agrees to permanent ban on selling facial recognition to private companies. The Verge. Retrieved July 20, 2023, from www.theverge.com\/2022\/5\/9\/23063952\/clearview-ai-aclu-settlement-illinois-bipa-injunction-private-companies"},{"key":"74_CR156","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3274417","volume":"2","author":"RE Robertson","year":"2018","unstructured":"Robertson, R. E., Jiang, S., Joseph, K., Friedland, L., Lazer, D., & Wilson, C. (2018). Auditing partisan audience bias within Google search. Proceedings of the ACM on Human-Computer Interaction, 2, 1\u201322. https:\/\/doi.org\/10.1145\/3274417","journal-title":"Proceedings of the ACM on Human-Computer Interaction"},{"issue":"4","key":"74_CR157","doi-asserted-by":"publisher","first-page":"105","DOI":"10.48550\/arXiv.1602.03506","volume":"36","author":"S Russell","year":"2015","unstructured":"Russell, S., Dewey, D., & Tegmark, M. (2015). Research priorities for robust and beneficial artificial intelligence. AI Magazine, 36(4), 105\u2013114. https:\/\/doi.org\/10.48550\/arXiv.1602.03506","journal-title":"AI Magazine"},{"key":"74_CR158","unstructured":"Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K. T., & Ghani, R. (2018). Aequitas: A bias and fairness audit toolkit. ArXiv. Retrieved July 20, 2023, from http:\/\/arxiv.org\/abs\/1811.05577"},{"issue":"7\/8","key":"74_CR159","doi-asserted-by":"publisher","first-page":"253","DOI":"10.5117\/MAB.96.90108","volume":"96","author":"I Sandu","year":"2022","unstructured":"Sandu, I., Wiersma, M., & Manichand, D. (2022). Time to audit your AI algorithms. Maandblad Voor Accountancy En Bedrijfseconomie, 96(7\/8), 253\u2013265. https:\/\/doi.org\/10.5117\/MAB.96.90108","journal-title":"Maandblad Voor Accountancy En Bedrijfseconomie"},{"key":"74_CR160","doi-asserted-by":"publisher","unstructured":"Sandvig, C., Hamilton, K., Karahalios, K., & Langbort, C. (2014). Auditing algorithms. ICA 2014 Data and Discrimination Preconference, 1\u201323. https:\/\/doi.org\/10.1109\/DEXA.2009.55","DOI":"10.1109\/DEXA.2009.55"},{"key":"74_CR161","unstructured":"Schonander, C. (2019). Enhancing trust in artificial intelligence: Audits and explanations can help. CIO. Retrieved July 20, 2023, from https:\/\/www.cio.com\/article\/220496"},{"key":"74_CR162","doi-asserted-by":"publisher","unstructured":"Schuett, J. (2022). Three lines of defense against risks from AI. ArXiv. https:\/\/doi.org\/10.48550\/arxiv.2212.08364","DOI":"10.48550\/arxiv.2212.08364"},{"key":"74_CR163","volume-title":"Capitalism, socialism, and democracy","author":"JA Schumpeter","year":"1942","unstructured":"Schumpeter, J. A. (1942). Capitalism, socialism, and democracy. Allen & Unwin."},{"key":"74_CR164","doi-asserted-by":"publisher","unstructured":"Seaver, N. (2017). Algorithms as culture: Some tactics for the ethnography of algorithmic systems. Big Data & Society, 4(2). https:\/\/doi.org\/10.1177\/2053951717738104","DOI":"10.1177\/2053951717738104"},{"key":"74_CR165","unstructured":"Selbst, A. D. (2021). An institutional view of algorithmic impact assessments. Harvard Journal of Law & Technology, 35."},{"key":"74_CR166","volume-title":"Information technology control and audit","author":"S Senft","year":"2009","unstructured":"Senft, S., & Gallegos, F. (2009). Information technology control and audit (3rd ed.). CRC Press.","edition":"3"},{"issue":"CSCW2","key":"74_CR167","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3479577","volume":"5","author":"H Shen","year":"2021","unstructured":"Shen, H., Devos, A., Eslami, M., & Holstein, K. (2021). Everyday algorithm auditing: Understanding the power of everyday users in surfacing harmful algorithmic behaviors. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW2), 1\u201329. https:\/\/doi.org\/10.1145\/3479577","journal-title":"Proceedings of the ACM on Human-Computer Interaction"},{"key":"74_CR168","doi-asserted-by":"publisher","first-page":"224","DOI":"10.1145\/3366423.3380109","volume":"2020","author":"M Silva","year":"2020","unstructured":"Silva, M., Santos De Oliveira, L., Andreou, A., Vaz De Melo, P. O., Goga, O., & Benevenuto, F. (2020). Facebook ads monitor: An independent auditing system for political ads on Facebook. Proceedings of the Web Conference, 2020, 224\u2013234. https:\/\/doi.org\/10.1145\/3366423.3380109","journal-title":"Proceedings of the Web Conference"},{"key":"74_CR169","unstructured":"Sloane, M. (2021). The algorithmic auditing trap. OneZero. Retrieved July 20, 2023, from https:\/\/onezero.medium.com\/the-algorithmic-auditing-trap-9a6f2d4d461d"},{"key":"74_CR170","volume-title":"Auditing: An international approach","author":"WJ Smieliauskas","year":"2010","unstructured":"Smieliauskas, W. J., & Bewley, K. (2010). Auditing: An international approach (5th ed.). McGraw-Hill Ryerson Higher Education.","edition":"5"},{"issue":"1","key":"74_CR171","doi-asserted-by":"publisher","first-page":"167","DOI":"10.1007\/S00146-021-01199-9\/METRICS","volume":"37","author":"M Smith","year":"2022","unstructured":"Smith, M., & Miller, S. (2022). The ethical application of biometric facial recognition technology. AI and Society, 37(1), 167\u2013175. https:\/\/doi.org\/10.1007\/S00146-021-01199-9\/METRICS","journal-title":"AI and Society"},{"issue":"1","key":"74_CR172","doi-asserted-by":"publisher","first-page":"57","DOI":"10.1080\/17579961.2021.1898300","volume":"13","author":"NA Smuha","year":"2021","unstructured":"Smuha, N. A. (2021). From a \u201crace to AI\u201d to a \u201crace to AI regulation\u201d: Regulatory competition for artificial intelligence. Law, Innovation and Technology, 13(1), 57\u201384. https:\/\/doi.org\/10.1080\/17579961.2021.1898300","journal-title":"Law, Innovation and Technology"},{"key":"74_CR173","doi-asserted-by":"publisher","DOI":"10.1016\/j.simpa.2022.100406","volume":"14","author":"K Sokol","year":"2022","unstructured":"Sokol, K., Santos-Rodriguez, R., & Flach, P. (2022). FAT Forensics: A Python toolbox for algorithmic fairness, accountability and transparency. Software Impacts, 14, 100406. https:\/\/doi.org\/10.1016\/j.simpa.2022.100406","journal-title":"Software Impacts"},{"key":"74_CR174","doi-asserted-by":"publisher","unstructured":"Sookhak, M., Akhunzada, A., Gani, A., Khurram Khan, M., & Anuar, N. B. (2014). Towards dynamic remote data auditing in computational clouds. Scientific World Journal, 2014. https:\/\/doi.org\/10.1155\/2014\/269357","DOI":"10.1155\/2014\/269357"},{"key":"74_CR175","doi-asserted-by":"publisher","DOI":"10.1145\/3219819.3220046","author":"T Speicher","year":"2018","unstructured":"Speicher, T., Heidari, H., Grgic-Hlaca, N., Gummadi, K. P., Singla, A., Weller, A., & Bilal Zafar, M. (2018). A Unified Approach to Quantifying Algorithmic Unfairness: Measuring Individual & Group Unfairness via Inequality Indices. https:\/\/doi.org\/10.1145\/3219819.3220046","journal-title":"A Unified Approach to Quantifying Algorithmic Unfairness: Measuring Individual & Group Unfairness via Inequality Indices."},{"issue":"1","key":"74_CR176","doi-asserted-by":"publisher","first-page":"60","DOI":"10.1016\/j.accinf.2011.11.001","volume":"13","author":"D Stoel","year":"2012","unstructured":"Stoel, D., Havelka, D., & Merhout, J. W. (2012). An analysis of attributes that impact information technology audit quality: A study of IT and financial audit practitioners. International Journal of Accounting Information Systems, 13(1), 60\u201379. https:\/\/doi.org\/10.1016\/j.accinf.2011.11.001","journal-title":"International Journal of Accounting Information Systems"},{"issue":"5","key":"74_CR177","doi-asserted-by":"publisher","first-page":"44","DOI":"10.1145\/2447976.2447990","volume":"56","author":"L Sweeney","year":"2013","unstructured":"Sweeney, L. (2013). Discrimination in online Ad delivery. Communications of the ACM, 56(5), 44\u201354. https:\/\/doi.org\/10.1145\/2447976.2447990","journal-title":"Communications of the ACM"},{"issue":"4","key":"74_CR178","doi-asserted-by":"publisher","first-page":"317","DOI":"10.1007\/s11023-016-9408-z","volume":"26","author":"M Taddeo","year":"2016","unstructured":"Taddeo, M. (2016). On the risks of relying on analogies to understand cyber conflicts. Minds and Machines, 26(4), 317\u2013321. https:\/\/doi.org\/10.1007\/s11023-016-9408-z","journal-title":"Minds and Machines"},{"issue":"6404","key":"74_CR179","doi-asserted-by":"publisher","first-page":"751","DOI":"10.1126\/science.aat5991","volume":"361","author":"M Taddeo","year":"2018","unstructured":"Taddeo, M., & Floridi, L. (2018). How AI can be a force for good. Science, 361(6404), 751\u2013752. https:\/\/doi.org\/10.1126\/science.aat5991","journal-title":"Science"},{"key":"74_CR180","unstructured":"Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., \u2026 Le, Q. (2022). LaMDA: Language models for dialog applications. Google. Retrieved July 20, 2023, from https:\/\/ai.googleblog.com\/2022\/01\/lamda-towards-safe-grounded-and-high.html?hl=fr&m=1"},{"key":"74_CR181","doi-asserted-by":"publisher","first-page":"4007","DOI":"10.48550\/arXiv.2110.11891","volume-title":"On the necessity of auditable algorithmic definitions for machine unlearning","author":"A Thudi","year":"2021","unstructured":"Thudi, A., Jia, H., Shumailov, I., & Papernot, N. (2021). On the necessity of auditable algorithmic definitions for machine unlearning (pp. 4007\u20134022). 31st USENIX Security Symposium. https:\/\/doi.org\/10.48550\/arXiv.2110.11891"},{"key":"74_CR182","doi-asserted-by":"publisher","unstructured":"Tolan, S. (2019). Fair and unbiased algorithmic decision making: Current state and future challenges. In JRC Working Papers on Digital Economy (2018\u201310). https:\/\/doi.org\/10.48550\/arxiv.1901.04730","DOI":"10.48550\/arxiv.1901.04730"},{"issue":"1","key":"74_CR183","doi-asserted-by":"publisher","first-page":"215","DOI":"10.1007\/S00146-021-01154-8","volume":"37","author":"A Tsamados","year":"2021","unstructured":"Tsamados, A., Aggarwal, N., Cowls, J., Morley, J., Roberts, H., Taddeo, M., & Floridi, L. (2021). The ethics of algorithms: Key problems and solutions. AI & Society, 37(1), 215\u2013230. https:\/\/doi.org\/10.1007\/S00146-021-01154-8","journal-title":"AI & Society"},{"key":"74_CR184","volume-title":"Auditing in the United Kingdom: A study of development in the audit methodologies of large accounting firms","author":"S Turley","year":"2005","unstructured":"Turley, S., & Cooper, M. (2005). Auditing in the United Kingdom: A study of development in the audit methodologies of large accounting firms. Prentice Hall."},{"issue":"1","key":"74_CR185","doi-asserted-by":"publisher","first-page":"199","DOI":"10.1007\/s43681-021-00117-5","volume":"2","author":"P Ugwudike","year":"2021","unstructured":"Ugwudike, P. (2021). AI audits for assessing design logics and building ethical systems: The case of predictive policing algorithms. AI and Ethics, 2(1), 199\u2013208. https:\/\/doi.org\/10.1007\/s43681-021-00117-5","journal-title":"AI and Ethics"},{"issue":"7","key":"74_CR186","first-page":"21","volume":"5","author":"R Ulloa","year":"2019","unstructured":"Ulloa, R., Makhortykh, M., & Urman, A. (2019). Algorithm auditing at a large-scale: Insights from search engine audits. Computer Science and Engineering, 5(7), 21\u201336.","journal-title":"Computer Science and Engineering"},{"key":"74_CR187","doi-asserted-by":"publisher","DOI":"10.9785\/cri-2021-220402","author":"M Veale","year":"2022","unstructured":"Veale, M., & Borgesius, F. Z. (2022). Demystifying the Draft EU Artificial Intelligence Act. Computer Law Review International. https:\/\/doi.org\/10.9785\/cri-2021-220402","journal-title":"Computer Law Review International"},{"key":"74_CR188","doi-asserted-by":"publisher","unstructured":"Vecchione, B., Levy, K., & Barocas, S. (2021). Algorithmic auditing and social justice: Lessons from the history of audit studies. ACM International Conference Proceeding Series, 1\u20139. https:\/\/doi.org\/10.1145\/3465416.3483294","DOI":"10.1145\/3465416.3483294"},{"key":"74_CR189","unstructured":"Verband Der Elektrotechnik (VDE). (2022). VCIO based description of systems for AI trustworthiness characterisation: (en). Retrieved July 20, 2023, from www.vde.com\/resource\/blob\/-2177870\/a24b13db01773747e6b7bba4ce20ea60\/vde-spec-90012-v1-0--en--data.pdf"},{"key":"74_CR190","doi-asserted-by":"publisher","first-page":"35","DOI":"10.1007\/s44206-023-00063-1","volume":"2","author":"D Vetter","year":"2023","unstructured":"Vetter, D., Amann, J., Bruneault, F., et al. (2023). Lessons learned from assessing trustworthy AI in practice. Digital Society, 2, 35. https:\/\/doi.org\/10.1007\/s44206-023-00063-1","journal-title":"Digital Society"},{"key":"74_CR191","unstructured":"Vlok, N. (2003). Technology auditing as a means of ensuring business continuity in a manufacturing organisation. Retrieved July 20, 2023, from https:\/\/core.ac.uk\/download\/pdf\/145048364.pdf"},{"issue":"2","key":"74_CR192","doi-asserted-by":"publisher","first-page":"841","DOI":"10.2139\/ssrn.3063289","volume":"31","author":"S Wachter","year":"2017","unstructured":"Wachter, S., Mittelstadt, B., & Russell, C. (2017). Counterfactual explanations without opening the black box: Automated decisions and the GDPR. Harvard Journal of Law and Technology, 31(2), 841\u2013888. https:\/\/doi.org\/10.2139\/ssrn.3063289","journal-title":"Harvard Journal of Law and Technology"},{"issue":"4","key":"74_CR193","doi-asserted-by":"publisher","first-page":"39","DOI":"10.2307\/248959","volume":"4","author":"IR Weiss","year":"1980","unstructured":"Weiss, I. R. (1980). Auditability of software: A survey of techniques and costs. MIS Quarterly: Management Information Systems, 4(4), 39\u201350. https:\/\/doi.org\/10.2307\/248959","journal-title":"MIS Quarterly: Management Information Systems"},{"key":"74_CR194","doi-asserted-by":"publisher","first-page":"666","DOI":"10.1145\/3442188.3445928","volume-title":"Building and auditing fair algorithms: A case study in candidate screening","author":"C Wilson","year":"2021","unstructured":"Wilson, C., Ghosh, A., Jiang, S., Mislove, A., Baker, L., Szary, J., Trindel, K., & Polli, F. (2021). Building and auditing fair algorithms: A case study in candidate screening (pp. 666\u2013677). FAccT 2021 - Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. https:\/\/doi.org\/10.1145\/3442188.3445928"},{"issue":"2","key":"74_CR195","doi-asserted-by":"publisher","first-page":"83","DOI":"10.1109\/tts.2021.3066209","volume":"2","author":"RV Zicari","year":"2021","unstructured":"Zicari, R. V., Brodersen, J., Brusseau, J., Dudder, B., Eichhorn, T., Ivanov, T., & Westerlund, M. (2021). Z-Inspection\u00ae: A process to assess trustworthy AI. IEEE Transactions on Technology and Society, 2(2), 83\u201397. https:\/\/doi.org\/10.1109\/tts.2021.3066209","journal-title":"IEEE Transactions on Technology and Society"},{"key":"74_CR196","volume-title":"The 2021 Yearbook of the Digital Ethics Lab","author":"N Zinda","year":"2021","unstructured":"Zinda, N. (2021). Ethics auditing framework for trustworthy AI: Lessons from the IT audit literature. In J. Mokander & M. Ziosi (Eds.), The 2021 Yearbook of the Digital Ethics Lab. Springer."}],"container-title":["Digital Society"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s44206-023-00074-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s44206-023-00074-y\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s44206-023-00074-y.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,2,2]],"date-time":"2024-02-02T11:13:01Z","timestamp":1706872381000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s44206-023-00074-y"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,11,8]]},"references-count":196,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2023,12]]}},"alternative-id":["74"],"URL":"https:\/\/doi.org\/10.1007\/s44206-023-00074-y","relation":{},"ISSN":["2731-4650","2731-4669"],"issn-type":[{"value":"2731-4650","type":"print"},{"value":"2731-4669","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,11,8]]},"assertion":[{"value":"28 August 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"25 September 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"8 November 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent to Participate"}},{"value":"The author declares no competing interests.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing Interests"}}],"article-number":"49"}}