{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,17]],"date-time":"2026-08-17T16:12:34Z","timestamp":1786983154832,"version":"3.56.0"},"reference-count":121,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2021,9,12]],"date-time":"2021-09-12T00:00:00Z","timestamp":1631404800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,9,12]],"date-time":"2021-09-12T00:00:00Z","timestamp":1631404800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100000266","name":"Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["AI Ethics"],"published-print":{"date-parts":[[2022,8]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Bias, unfairness and lack of transparency and accountability in Artificial Intelligence (AI) systems, and the potential for the misuse of predictive models for decision-making have raised concerns about the ethical impact and unintended consequences of new technologies for society across every sector where data-driven innovation is taking place. This paper reviews the landscape of suggested ethical frameworks with a focus on those which go beyond high-level statements of principles and offer practical tools for application of these principles in the production and deployment of systems. This work provides an assessment of these practical frameworks with the lens of known best practices for impact assessment and audit of technology. We review other historical uses of risk assessments and audits and create a typology that allows us to compare current AI ethics tools to Best Practices found in previous methodologies from technology, environment, privacy, finance and engineering. We analyse current AI ethics tools and their support for diverse stakeholders and components of the AI development and deployment lifecycle as well as the types of tools used to facilitate use. From this, we identify gaps in current AI ethics tools in auditing and risk assessment that should be considered going forward.<\/jats:p>","DOI":"10.1007\/s43681-021-00084-x","type":"journal-article","created":{"date-parts":[[2021,9,12]],"date-time":"2021-09-12T10:03:40Z","timestamp":1631441020000},"page":"405-429","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":174,"title":["Putting AI ethics to work: are the tools fit for purpose?"],"prefix":"10.1007","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2450-0881","authenticated-orcid":false,"given":"Jacqui","family":"Ayling","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3814-2587","authenticated-orcid":false,"given":"Adriane","family":"Chapman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2021,9,12]]},"reference":[{"issue":"2","key":"84_CR1","doi-asserted-by":"publisher","first-page":"56","DOI":"10.1145\/2844110","volume":"59","author":"N Diakopoulos","year":"2016","unstructured":"Diakopoulos, N.: Accountability in algorithmic decision making. Commun. ACM 59(2), 56\u201362 (2016). https:\/\/doi.org\/10.1145\/2844110","journal-title":"Commun. ACM"},{"key":"84_CR2","volume-title":"Automating inequality: how high-tech tools profile, police, and punish the poor","author":"V Eubanks","year":"2018","unstructured":"Eubanks, V.: Automating inequality: how high-tech tools profile, police, and punish the poor. St. Martin\u2019s Publishing Group (2018)"},{"key":"84_CR3","unstructured":"Council regulation (EU) 2016\/679: On the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95\/46\/EC (General Data Protection Regulation). Off. J. L119\/1 (2016) Available http:\/\/eur-lex.europa.eu\/legal-content\/EN\/TXT\/?uri=uriserv:OJ.L_.2016.119.01.0001.01.ENG&toc=OJ:L:2016:119:TOC. Accessed 23 Sep. 2017. (Online)."},{"issue":"1","key":"84_CR4","doi-asserted-by":"publisher","first-page":"99","DOI":"10.1007\/s11023-020-09517-8","volume":"30","author":"T Hagendorff","year":"2020","unstructured":"Hagendorff, T.: The ethics of AI ethics\u2014an evaluation of guidelines. Minds Mach. 30(1), 99\u2013120 (2020). https:\/\/doi.org\/10.1007\/s11023-020-09517-8","journal-title":"Minds Mach."},{"issue":"9","key":"84_CR5","doi-asserted-by":"publisher","first-page":"389","DOI":"10.1038\/s42256-019-0088-2","volume":"1","author":"A Jobin","year":"2019","unstructured":"Jobin, A., Ienca, M., Vayena, E.: The global landscape of AI ethics guidelines. Nat. Mach. Intell. 1(9), 389\u2013399 (2019). https:\/\/doi.org\/10.1038\/s42256-019-0088-2","journal-title":"Nat. Mach. Intell."},{"key":"84_CR6","doi-asserted-by":"crossref","unstructured":"Fjeld, J., Achten, N., Hilligoss, H., Nagy, A., Srikumar, M.: Principled artificial intelligence: mapping consensus in ethical and rights-based approaches to principles for AI. In: Social Science Research Network, Rochester, NY, SSRN Scholarly Paper ID 3518482 (2020). Available https:\/\/papers.ssrn.com\/abstract=3518482. Accessed 27 Jan. 2020. (Online)","DOI":"10.2139\/ssrn.3518482"},{"key":"84_CR7","doi-asserted-by":"publisher","DOI":"10.1007\/s11948-019-00165-5","author":"J Morley","year":"2019","unstructured":"Morley, J., Floridi, L., Kinsey, L., Elhalal, A.: From what to how: an initial review of publicly available AI ethics tools, methods and research to translate principles into practices. Sci. Eng. Ethics (2019). https:\/\/doi.org\/10.1007\/s11948-019-00165-5","journal-title":"Sci. Eng. Ethics"},{"issue":"3","key":"84_CR8","doi-asserted-by":"publisher","first-page":"477","DOI":"10.2307\/40041279","volume":"154","author":"DJ Solove","year":"2006","unstructured":"Solove, D.J.: A taxonomy of privacy. Univ. Pa Law Rev. 154(3), 477 (2006). https:\/\/doi.org\/10.2307\/40041279","journal-title":"Univ. Pa Law Rev."},{"key":"84_CR9","doi-asserted-by":"publisher","unstructured":"Citron. D.K., Solove, D.J.: Privacy harms. In: Social Science Research Network, Rochester, NY, SSRN Scholarly Paper ID 3782222 (2021). https:\/\/doi.org\/10.2139\/ssrn.3782222.","DOI":"10.2139\/ssrn.3782222"},{"issue":"2","key":"84_CR10","doi-asserted-by":"publisher","first-page":"205395171667967","DOI":"10.1177\/2053951716679679","volume":"3","author":"BD Mittelstadt","year":"2016","unstructured":"Mittelstadt, B.D., Allo, P., Taddeo, M., Wachter, S., Floridi, L.: The ethics of algorithms: mapping the debate. Big Data Soc. 3(2), 2053951716679679 (2016). https:\/\/doi.org\/10.1177\/2053951716679679","journal-title":"Big Data Soc."},{"key":"84_CR11","unstructured":"Hirsch, D., Bartley, T., Chandrasekaran, A., Norris, D., Parthasarathy, S., Turner, P. N.: Business data ethics: emerging trends in the governance of advanced analytics and AI. In: The Ohio State University, Ohio State Legal Studies Research Paper No. 628, 2020. Available https:\/\/cpb-us-w2.wpmucdn.com\/u.osu.edu\/dist\/3\/96132\/files\/2020\/10\/Final-Report-1.pdf. (Online)"},{"key":"84_CR12","doi-asserted-by":"publisher","first-page":"71","DOI":"10.2307\/1229546","volume":"53","author":"DJ Solove","year":"2001","unstructured":"Solove, D.J.: Privacy and power: computer databases and metaphors for information privacy. Stanford Law Rev. 53, 71 (2001)","journal-title":"Stanford Law Rev."},{"key":"84_CR13","doi-asserted-by":"publisher","DOI":"10.1016\/j.clsr.2020.105404","volume":"37","author":"CD Raab","year":"2020","unstructured":"Raab, C.D.: Information privacy, impact assessment, and the place of ethics. Comput. Law Secur. Rev. 37, 105404 (2020). https:\/\/doi.org\/10.1016\/j.clsr.2020.105404","journal-title":"Comput. Law Secur. Rev."},{"key":"84_CR14","doi-asserted-by":"publisher","unstructured":"Greene, D., Hoffmann, A.L., Stark, L.: Better, nicer, clearer, fairer: a critical assessment of the movement for ethical artificial intelligence and machine learning. In: Presented at the Hawaii International Conference on System Sciences (2019). https:\/\/doi.org\/10.24251\/HICSS.2019.258.","DOI":"10.24251\/HICSS.2019.258"},{"key":"84_CR15","doi-asserted-by":"publisher","unstructured":"Kazim, E., Koshiyama, A.: AI assurance processes. In: Social Science Research Network, Rochester, SSRN Scholarly Paper ID 3685087 (2020). https:\/\/doi.org\/10.2139\/ssrn.3685087.","DOI":"10.2139\/ssrn.3685087"},{"key":"84_CR16","unstructured":"Kind, C.: The term \u2018ethical AI\u2019 is finally starting to mean something. VentureBeat (2020). https:\/\/venturebeat.com\/2020\/08\/23\/the-term-ethical-ai-is-finally-starting-to-mean-something\/. Accessed 23 Aug. 2020"},{"key":"84_CR17","doi-asserted-by":"publisher","DOI":"10.1108\/JICES-12-2019-0138","author":"M Ryan","year":"2020","unstructured":"Ryan, M., Stahl, B.C.: Artificial intelligence ethics guidelines for developers and users: clarifying their content and normative implications. J. Inf. Commun. Ethics Soc. (2020). https:\/\/doi.org\/10.1108\/JICES-12-2019-0138","journal-title":"J. Inf. Commun. Ethics Soc."},{"key":"84_CR18","unstructured":"AlgorithmWatch: \u201cAI Ethics Guidelines Global Inventory by AlgorithmWatch,\u201d AI Ethics Guidelines Global Inventory (2020). https:\/\/inventory.algorithmwatch.org. Accessed 11 Aug. 2020."},{"key":"84_CR19","doi-asserted-by":"publisher","DOI":"10.1109\/TTS.2021.3052127","author":"D Schiff","year":"2021","unstructured":"Schiff, D., Borenstein, J., Biddle, J., Laas, K.: AI ethics in the public, private, and NGO sectors: a review of a global document collection. IEEE Trans. Technol. Soc. (2021). https:\/\/doi.org\/10.1109\/TTS.2021.3052127","journal-title":"IEEE Trans. Technol. Soc."},{"key":"84_CR20","unstructured":"Bird, S., et al.: Fairlearn: a toolkit for assessing and improving fairness in AI. Microsoft (2020). Available https:\/\/www.microsoft.com\/en-us\/research\/uploads\/prod\/2020\/05\/Fairlearn_WhitePaper-2020-09-22.pdf. Accessed 13 Oct. 2020. (Online)"},{"key":"84_CR21","doi-asserted-by":"publisher","first-page":"220","DOI":"10.1145\/3287560.3287596","volume":"19","author":"M Mitchell","year":"2019","unstructured":"Mitchell, M., et al.: Model cards for model reporting. Proc. Conf. Fairness Account. Transpar. FAT 19, 220\u2013229 (2019). https:\/\/doi.org\/10.1145\/3287560.3287596","journal-title":"Proc. Conf. Fairness Account. Transpar. FAT"},{"key":"84_CR22","unstructured":"Gebru, T., et al.: \u201cDatasheets for Datasets\u201d (2020). Available http:\/\/arxiv.org\/abs\/1803.09010. Accessed 03 Dec. 2020. (Online)"},{"key":"84_CR23","doi-asserted-by":"publisher","DOI":"10.2307\/j.ctv1ghv45t","volume-title":"Atlas of AI: power, politics, and the planetary costs of artificial intelligence","author":"K Crawford","year":"2021","unstructured":"Crawford, K.: Atlas of AI: power, politics, and the planetary costs of artificial intelligence. Yale University Press, New Haven (2021)"},{"issue":"1","key":"84_CR24","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1080\/14615517.2012.661557","volume":"30","author":"RK Morgan","year":"2012","unstructured":"Morgan, R.K.: Environmental impact assessment: the state of the art. Impact Assess. Proj. Apprais. 30(1), 5\u201314 (2012). https:\/\/doi.org\/10.1080\/14615517.2012.661557","journal-title":"Impact Assess. Proj. Apprais."},{"issue":"2","key":"84_CR25","doi-asserted-by":"publisher","first-page":"123","DOI":"10.1016\/j.clsr.2009.02.002","volume":"25","author":"R Clarke","year":"2009","unstructured":"Clarke, R.: Privacy impact assessment: Its origins and development. Comput. Law Secur. Rev. 25(2), 123\u2013135 (2009). https:\/\/doi.org\/10.1016\/j.clsr.2009.02.002","journal-title":"Comput. Law Secur. Rev."},{"key":"84_CR26","unstructured":"Information Commissioner\u2019s Office: Data protection impact assessments (2018). https:\/\/ico.org.uk\/for-organisations\/guide-to-the-general-data-protection-regulation-gdpr\/accountability-and-governance\/data-protection-impact-assessments\/. Accessed 07 Jun. 2018"},{"key":"84_CR27","unstructured":"The Danish Institute for Human Rights: Human rights impact assessment guidance and toolbox - road-testing version. The Danish Institute for Human Rights (2016). https:\/\/www.humanrights.dk\/business\/tools\/human-rights-impact-assessment-guidance-and-toolbox. Accessed 03 Feb. 2020"},{"key":"84_CR28","volume-title":"Risk Governance: Coping with Uncertainty in a Complex World","author":"O Renn","year":"2008","unstructured":"Renn, O.: Risk Governance: Coping with Uncertainty in a Complex World. Earthscan (2008)"},{"key":"84_CR29","doi-asserted-by":"publisher","first-page":"341","DOI":"10.1016\/0040-1625(74)90035-3","volume":"6","author":"JF Coates","year":"1974","unstructured":"Coates, J.F.: Some methods and techniques for comprehensive impact assessment. Technol. Forecast. Soc. Change 6, 341\u2013357 (1974). https:\/\/doi.org\/10.1016\/0040-1625(74)90035-3","journal-title":"Technol. Forecast. Soc. Change"},{"key":"84_CR30","unstructured":"IAIA: Technology Assessment (2009). https:\/\/www.iaia.org\/wiki-details.php?ID=26. Accessed 26 Jan. 2021"},{"issue":"5","key":"84_CR31","doi-asserted-by":"publisher","first-page":"543","DOI":"10.1016\/j.techfore.2005.06.002","volume":"73","author":"E Palm","year":"2006","unstructured":"Palm, E., Hansson, S.O.: The case for ethical technology assessment (eTA). Technol. Forecast. Soc. Change 73(5), 543\u2013558 (2006). https:\/\/doi.org\/10.1016\/j.techfore.2005.06.002","journal-title":"Technol. Forecast. Soc. Change"},{"key":"84_CR32","doi-asserted-by":"crossref","unstructured":"STOA: Centre for AI | Panel for the Future of Science and Technology (STOA) | European Parliament (2021). https:\/\/www.europarl.europa.eu\/stoa\/en\/centre-for-AI. Accessed 11 Feb. 2021","DOI":"10.25009\/st.2020.21.2596"},{"key":"84_CR33","doi-asserted-by":"publisher","first-page":"27","DOI":"10.1007\/s10202-012-0122-5","volume":"9","author":"L Hennen","year":"2012","unstructured":"Hennen, L.: Why do we still need participatory technology assessment? Poiesis Prax. 9, 27\u201341 (2012). https:\/\/doi.org\/10.1007\/s10202-012-0122-5","journal-title":"Poiesis Prax."},{"key":"84_CR34","unstructured":"CSPO: Participatory Technology Assessment | CSPO, Consortium for Science and Policy Outcomes (2021). https:\/\/cspo.org\/areas-of-focus\/pta\/. Accessed 12 Feb. 2021"},{"issue":"1","key":"84_CR35","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1080\/23299460.2014.992769","volume":"2","author":"A Kiran","year":"2015","unstructured":"Kiran, A., Oudshoorn, N.E.J., Verbeek, P.P.C.C.: Beyond checklists: toward an ethical-constructive technology assessment. J. Respons Innov. 2(1), 5\u201319 (2015). https:\/\/doi.org\/10.1080\/23299460.2014.992769","journal-title":"J. Respons Innov."},{"issue":"3","key":"84_CR36","doi-asserted-by":"publisher","first-page":"295","DOI":"10.1007\/BF01867157","volume":"11","author":"GW Suter","year":"1987","unstructured":"Suter, G.W., Barnthouse, L.W., O\u2019Neill, R.V.: Treatment of risk in environmental impact assessment. Environ. Manage. 11(3), 295\u2013303 (1987). https:\/\/doi.org\/10.1007\/BF01867157","journal-title":"Environ. Manage."},{"key":"84_CR37","unstructured":"UN Environment: \u201cAssessing Environmental Impacts A Global Review Of Legislation-UNEP-WCMC,\u201d In: UNEP-WCMC\u2019s official website\u2014Assessing Environmental Impacts A Global Review Of Legislation (2018). https:\/\/www.unep-wcmc.org\/assessing-environmental-impacts--a-global-review-of-legislation. Accessed 12 Feb. 2021."},{"key":"84_CR38","doi-asserted-by":"publisher","first-page":"104","DOI":"10.1016\/j.eiar.2013.06.003","volume":"43","author":"AN Glucker","year":"2013","unstructured":"Glucker, A.N., Driessen, P.P.J., Kolhoff, A., Runhaar, H.A.C.: Public participation in environmental impact assessment: why, who and how? Environ. Impact Assess. Rev. 43, 104\u2013111 (2013). https:\/\/doi.org\/10.1016\/j.eiar.2013.06.003","journal-title":"Environ. Impact Assess. Rev."},{"key":"84_CR39","unstructured":"IMA Europe: \u201cLife Cycle Assessment | IMA Europe,\u201d In: Industrial Mineral Association-Europe (2020). https:\/\/www.ima-europe.eu\/eu-policy\/environment\/life-cycle-assessment. Accessed 06 May 2021"},{"issue":"1","key":"84_CR40","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.ejor.2015.12.023","volume":"253","author":"T Aven","year":"2016","unstructured":"Aven, T.: Risk assessment and risk management: review of recent advances on their foundation. Eur. J. Oper. Res. 253(1), 1\u201313 (2016). https:\/\/doi.org\/10.1016\/j.ejor.2015.12.023","journal-title":"Eur. J. Oper. Res."},{"issue":"1","key":"84_CR41","doi-asserted-by":"publisher","first-page":"25","DOI":"10.1080\/01944360903310477","volume":"76","author":"MM Edwards","year":"2009","unstructured":"Edwards, M.M., Huddleston, J.R.: Prospects and perils of fiscal impact analysis. J. Am. Plann. Assoc. 76(1), 25\u201341 (2009). https:\/\/doi.org\/10.1080\/01944360903310477","journal-title":"J. Am. Plann. Assoc."},{"key":"84_CR42","volume-title":"Cost-Benefit Analysis","author":"DW Pearce","year":"2016","unstructured":"Pearce, D.W.: Cost-Benefit Analysis, 2nd edn. Macmillan International Higher Education (2016)","edition":"2"},{"issue":"2","key":"84_CR43","doi-asserted-by":"publisher","first-page":"86","DOI":"10.1080\/14615517.2013.782978","volume":"31","author":"D Kemp","year":"2013","unstructured":"Kemp, D., Vanclay, F.: Human rights and impact assessment: clarifying the connections in practice. Impact Assess. Proj. Apprais. 31(2), 86\u201396 (2013). https:\/\/doi.org\/10.1080\/14615517.2013.782978","journal-title":"Impact Assess. Proj. Apprais."},{"key":"84_CR44","doi-asserted-by":"crossref","unstructured":"Kende-Robbe, C.: Poverty and social impact analysis\u202f: linking macroeconomic policies to poverty outcomes: summary of early experiences. IMF (2003). https:\/\/www.imf.org\/en\/Publications\/WP\/Issues\/2016\/12\/30\/Poverty-and-Social-Impact-Analysis-Linking-Macroeconomic-Policies-to-Poverty-Outcomes-16248. Accessed 12 Feb. 2021","DOI":"10.2139\/ssrn.879118"},{"key":"84_CR45","doi-asserted-by":"publisher","unstructured":"Roessler, B.: New ways of thinking about privacy. (2008). https:\/\/doi.org\/10.1093\/oxfordhb\/9780199548439.003.0038.","DOI":"10.1093\/oxfordhb\/9780199548439.003.0038"},{"key":"84_CR46","volume-title":"Privacy and Freedom","author":"AF Westin","year":"1967","unstructured":"Westin, A.F.: Privacy and Freedom. Ig Publishing (1967)"},{"key":"84_CR47","doi-asserted-by":"publisher","DOI":"10.4159\/harvard.9780674436978","volume-title":"Information Technology in a Democracy","author":"AF Westin","year":"1971","unstructured":"Westin, A.F.: Information Technology in a Democracy. Harvard University Press (1971)"},{"key":"84_CR48","unstructured":"Stewart, B.: Privacy impact assessments. Priv. Law Policy Rep. 39(4), 1996 (2021). Available http:\/\/www.austlii.edu.au\/au\/journals\/PLPR\/1996\/39.html Accessed: Feb. 17, 2021. [Online]"},{"key":"84_CR49","unstructured":"Financial Reporting Council: Auditors I Audit and Assurance I Standards and Guidance for Auditors I Financial Reporting Council (2020). https:\/\/www.frc.org.uk\/auditors\/audit-assurance\/standards-and-guidance. Accessed 26 Apr. 2021"},{"key":"84_CR50","unstructured":"Rusby, R.: The interpretation and evaluation of assurance cases. In: Computer Science Laboratory, SRI International, Menlo Park CA 94025, USA, Technical Report SRI-CSL-15\u201301 (2015)"},{"issue":"9","key":"84_CR51","doi-asserted-by":"publisher","first-page":"82","DOI":"10.1109\/MC.2019.2914775","volume":"52","author":"R Bloomfield","year":"2019","unstructured":"Bloomfield, R., Khlaaf, H., Conmy, P.R., Fletcher, G.: Disruptive innovations and disruptive assurance: assuring machine learning and autonomy. Computer 52(9), 82\u201389 (2019). https:\/\/doi.org\/10.1109\/MC.2019.2914775","journal-title":"Computer"},{"key":"84_CR52","unstructured":"International Organization for Standardization: \u201cISO-Standards,\u201d ISO (2021). https:\/\/www.iso.org\/standards.html. Accessed 15 Jul. 2021"},{"key":"84_CR53","unstructured":"International Organization for Standardization: \u201cISO-Certification,\u201d ISO (2021). https:\/\/www.iso.org\/certification.html. Accessed 15 Jul. 2021"},{"key":"84_CR54","unstructured":"PwC UK: \u201cUnderstanding a financial statement audit,\u201d PricewaterhouseCooper, UK, (2013). Available https:\/\/www.pwc.com\/gx\/en\/audit-services\/publications\/assets\/pwc-understanding-financial-statement-audit.pdf. (Online)"},{"key":"84_CR55","unstructured":"Brundage, M., et al.: Toward trustworthy AI development: mechanisms for supporting verifiable claims (2020). Available http:\/\/arxiv.org\/abs\/2004.07213. Accessed 16 Nov. 2020. (Online)."},{"key":"84_CR56","doi-asserted-by":"publisher","DOI":"10.1007\/s11023-021-09557-8","author":"J M\u00f6kander","year":"2021","unstructured":"M\u00f6kander, J., Floridi, L.: Ethics-based auditing to develop trustworthy AI. Minds Mach. (2021). https:\/\/doi.org\/10.1007\/s11023-021-09557-8","journal-title":"Minds Mach."},{"issue":"3899","key":"84_CR57","doi-asserted-by":"publisher","first-page":"1232","DOI":"10.1126\/science.165.3899.1232","volume":"165","author":"C Starr","year":"1969","unstructured":"Starr, C.: Social benefit versus technological risk. Science 165(3899), 1232\u20131238 (1969)","journal-title":"Science"},{"issue":"6","key":"84_CR58","doi-asserted-by":"publisher","first-page":"1333","DOI":"10.1111\/j.1539-6924.2005.00702.x","volume":"25","author":"KM Thompson","year":"2005","unstructured":"Thompson, K.M., Deisler, P.F., Schwing, R.C.: Interdisciplinary vision: the first 25 years of the society for risk analysis (SRA), 1980\u20132005. Risk Anal. 25(6), 1333\u20131386 (2005). https:\/\/doi.org\/10.1111\/j.1539-6924.2005.00702.x","journal-title":"Risk Anal."},{"key":"84_CR59","volume-title":"Risk Society: Towards a New Modernity","author":"PU Beck","year":"1992","unstructured":"Beck, P.U.: Risk Society: Towards a New Modernity. SAGE (1992)"},{"key":"84_CR60","unstructured":"Moses, K., Malone, R.: Development of risk assessment matrix for NASA Engineering and safety center NASA technical reports server (NTRS). In: NASA Technical Reports Server (NTRS) (2004). https:\/\/ntrs.nasa.gov\/citations\/20050123548. Accessed 27 May 2021"},{"issue":"5","key":"84_CR61","doi-asserted-by":"publisher","first-page":"309","DOI":"10.1016\/j.aos.2014.05.002","volume":"39","author":"C Hayne","year":"2014","unstructured":"Hayne, C., Free, C.: Hybridized professional groups and institutional work: COSO and the rise of enterprise risk management. Account. Organ. Soc. 39(5), 309\u2013330 (2014). https:\/\/doi.org\/10.1016\/j.aos.2014.05.002","journal-title":"Account. Organ. Soc."},{"key":"84_CR62","unstructured":"Lauterbach, A., Bonime, A.: Environmental risk social risk governance risk. Risk Manage, 3 (2018)."},{"issue":"4","key":"84_CR63","doi-asserted-by":"publisher","first-page":"689","DOI":"10.1007\/s11023-018-9482-5","volume":"28","author":"L Floridi","year":"2018","unstructured":"Floridi, L., et al.: AI4People\u2014an ethical framework for a good AI society: opportunities, risks, principles, and recommendations. Minds Mach. 28(4), 689\u2013707 (2018). https:\/\/doi.org\/10.1007\/s11023-018-9482-5","journal-title":"Minds Mach."},{"key":"84_CR64","unstructured":"High Level Expert Group on AI: \u201cEthics guidelines for trustworthy AI,\u201d European Commission, Brussels, Text (2019). Available https:\/\/ec.europa.eu\/digital-single-market\/en\/news\/ethics-guidelines-trustworthy-ai. Accessed 23 May 2019. (Online)"},{"key":"84_CR65","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9781139192675","volume-title":"Strategic Management: A Stakeholder Approach","author":"RE Freeman","year":"2010","unstructured":"Freeman, R.E.: Strategic Management: A Stakeholder Approach. Cambridge University Press (2010)"},{"issue":"1","key":"84_CR66","doi-asserted-by":"publisher","first-page":"65","DOI":"10.2307\/258887","volume":"20","author":"T Donaldson","year":"1995","unstructured":"Donaldson, T., Preston, L.E.: The stakeholder theory of the corporation: concepts, evidence, and implications. Acad. Manage. Rev. 20(1), 65 (1995). https:\/\/doi.org\/10.2307\/258887","journal-title":"Acad. Manage. Rev."},{"key":"84_CR67","unstructured":"Business Roundtable: \u201cOur Commitment,\u201d Business Roundtable\u2014Opportunity Agenda (2020). https:\/\/opportunity.businessroundtable.org\/ourcommitment\/. Accessed 05 Feb. 2021"},{"key":"84_CR68","unstructured":"TensorFlow: \u201cResponsible AI,\u201d TensorFlow (2020). https:\/\/www.tensorflow.org\/resources\/responsible-ai. Accessed 02 Nov. 2020"},{"key":"84_CR69","unstructured":"Bantilan, N.: Themis-ml: a fairness-aware machine learning interface for end-to-end discrimination discovery and mitigation. (2017). Available http:\/\/arxiv.org\/abs\/1710.06921. Accessed 13 Nov. 2020. (Online)"},{"key":"84_CR70","unstructured":"Bellamy, R. K. E., et al.: AI Fairness 360: an extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias. (2018). Available http:\/\/arxiv.org\/abs\/1810.01943. Accessed 27 May 2021. (Online)"},{"key":"84_CR71","doi-asserted-by":"publisher","DOI":"10.1007\/s43681-021-00067-y","author":"MSA Lee","year":"2021","unstructured":"Lee, M.S.A., Floridi, L., Singh, J.: Formalising trade-offs beyond algorithmic fairness: lessons from ethical philosophy and welfare economics. AI Ethics (2021). https:\/\/doi.org\/10.1007\/s43681-021-00067-y","journal-title":"AI Ethics"},{"key":"84_CR72","doi-asserted-by":"publisher","unstructured":"Hutchinson, B., Mitchell, M.: 50 years of test (Un)fairness: lessons for machine learning. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, New York, pp. 49\u201358 (2019). https:\/\/doi.org\/10.1145\/3287560.3287600.","DOI":"10.1145\/3287560.3287600"},{"key":"84_CR73","doi-asserted-by":"publisher","unstructured":"Veale, M., Van Kleek, M., Binns, R.: Fairness and accountability design needs for algorithmic support in high-stakes public sector decision-making. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, New York, p. 440:1\u2013440:14 (2018). https:\/\/doi.org\/10.1145\/3173574.3174014.","DOI":"10.1145\/3173574.3174014"},{"issue":"7","key":"84_CR74","doi-asserted-by":"publisher","first-page":"900","DOI":"10.1080\/1369118X.2019.1573912","volume":"22","author":"AL Hoffmann","year":"2019","unstructured":"Hoffmann, A.L.: Where fairness fails: data, algorithms, and the limits of antidiscrimination discourse. Inf. Commun. Soc. 22(7), 900\u2013915 (2019). https:\/\/doi.org\/10.1080\/1369118X.2019.1573912","journal-title":"Inf. Commun. Soc."},{"key":"84_CR75","doi-asserted-by":"publisher","DOI":"10.3389\/fdata.2020.00018","author":"J Radford","year":"2020","unstructured":"Radford, J., Joseph, K.: Theory in theory out: the uses of social theory in machine learning for social science. Front. Big Data. (2020). https:\/\/doi.org\/10.3389\/fdata.2020.00018","journal-title":"Front. Big Data."},{"key":"84_CR76","unstructured":"Institute for the Future and Omidyar Network, \u201cEthical OS,\u201d (2018). https:\/\/ethicalos.org\/. Accessed 21 Jun. 2019"},{"key":"84_CR77","unstructured":"Doteveryone, Consequence Scanning\u2014doteveryone (2019). https:\/\/doteveryone.org.uk\/project\/consequence-scanning\/. Accessed 18 Jun 2019"},{"key":"84_CR78","doi-asserted-by":"publisher","unstructured":"Madaio, M. A., Stark, L., Wortman Vaughan, J., Wallach, H.: Co-designing checklists to understand organizational challenges and opportunities around fairness in AI. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, Honolulu, pp. 1\u201314 (2020). https:\/\/doi.org\/10.1145\/3313831.3376445.","DOI":"10.1145\/3313831.3376445"},{"issue":"14","key":"84_CR79","doi-asserted-by":"publisher","first-page":"1229","DOI":"10.1080\/10447318.2019.1619259","volume":"35","author":"C Stephanidis","year":"2019","unstructured":"Stephanidis, C., et al.: Seven HCI grand challenges. Int. J. Hum Comput Interact. 35(14), 1229\u20131269 (2019). https:\/\/doi.org\/10.1080\/10447318.2019.1619259","journal-title":"Int. J. Hum Comput Interact."},{"key":"84_CR80","unstructured":"Krippendorff, K.: Content analysis. In: International encyclopedia of communication, vol. 1. Oxford University Press, New York, pp 8 (1989). Available http:\/\/repository.upenn.edu\/asc_papers\/22. Accessed 08 Jul 2020 (Online)"},{"issue":"3","key":"84_CR81","doi-asserted-by":"publisher","first-page":"379","DOI":"10.1111\/j.1541-0072.2002.tb02153.x","volume":"30","author":"KB Smith","year":"2002","unstructured":"Smith, K.B.: Typologies, taxonomies, and the benefits of policy classification. Policy Stud. J. 30(3), 379\u2013395 (2002). https:\/\/doi.org\/10.1111\/j.1541-0072.2002.tb02153.x","journal-title":"Policy Stud. J."},{"key":"84_CR82","unstructured":"Singh, A., et al.: PriMP visualization\u2014principled artificial intelligence project. In: Harvard Law School, Berkman Klein Center for Internet and Society (2018). https:\/\/ai-hr.cyber.harvard.edu\/primp-viz.html. Accessed 24 Jun. 2019"},{"key":"84_CR83","unstructured":"Birhane, A., Kalluri, P., Card, D., Agnew, W., Dotan, R., Bao, M.: The values encoded in machine learning research (2021). Available http:\/\/arxiv.org\/abs\/2106.15590. Accessed 25 Jul. 2021 (Online)"},{"key":"84_CR84","unstructured":"International Standardization Organisation: \u201cISO 14001:2015,\u201d. ISO (2021). https:\/\/www.iso.org\/cms\/render\/live\/en\/sites\/isoorg\/contents\/data\/standard\/06\/08\/60857.html. Accessed 26 Jul. 2021"},{"key":"84_CR85","doi-asserted-by":"publisher","first-page":"8","DOI":"10.1016\/j.npls.2016.01.001","volume":"2","author":"M Bengtsson","year":"2016","unstructured":"Bengtsson, M.: How to plan and perform a qualitative study using content analysis. NursingPlus Open 2, 8\u201314 (2016). https:\/\/doi.org\/10.1016\/j.npls.2016.01.001","journal-title":"NursingPlus Open"},{"key":"84_CR86","doi-asserted-by":"publisher","unstructured":"Whittlestone, J., Nyrup, R., Alexandrova, A., Cave, S.: The role and limits of principles in AI ethics: towards a focus on tensions. In: Proceedings of the 2019 AAAI\/ACM Conference on AI, Ethics, and Society, Honolulu, pp. 195\u2013200 (2019). https:\/\/doi.org\/10.1145\/3306618.3314289.","DOI":"10.1145\/3306618.3314289"},{"issue":"5","key":"84_CR87","doi-asserted-by":"publisher","first-page":"598","DOI":"10.1111\/j.1467-8683.2005.00454.x","volume":"13","author":"T Clarke","year":"2005","unstructured":"Clarke, T.: Accounting for Enron: shareholder value and stakeholder interests. Corp. Gov. Int. Rev. 13(5), 598\u2013612 (2005). https:\/\/doi.org\/10.1111\/j.1467-8683.2005.00454.x","journal-title":"Corp. Gov. Int. Rev."},{"key":"84_CR88","doi-asserted-by":"publisher","DOI":"10.1017\/9781108329453","volume-title":"Principles of Contemporary Corporate Governance","author":"JJ du Plessis","year":"2018","unstructured":"du Plessis, J.J., Hargovan, A., Harris, J.: Principles of Contemporary Corporate Governance. Cambridge University Press (2018)"},{"key":"84_CR89","unstructured":"Freeman, R. E.: Strategic management: a stakeholder approach. Pitman (1984)"},{"key":"84_CR90","unstructured":"Foden, C.: Our structure. City of Lincoln Council (2019). https:\/\/www.lincoln.gov.uk\/council\/structure. Accessed 10 Jan. 2021"},{"key":"84_CR91","unstructured":"Stanley, M.: UK Civil Service\u2014Grades and Roles. In: Understanding Government (2020). https:\/\/www.civilservant.org.uk\/information-grades_and_roles.html. Accessed 10 Jan. 2021"},{"key":"84_CR92","unstructured":"National Crime Agency: Our leadership. In: National Crime Agency (2021). https:\/\/www.nationalcrimeagency.gov.uk\/who-we-are\/our-leadership. Accessed 10 Jan. 2021"},{"key":"84_CR93","unstructured":"Badr, W.: Evaluating machine learning models fairness and bias. Medium (2019). https:\/\/towardsdatascience.com\/evaluating-machine-learning-models-fairness-and-bias-4ec82512f7c3. Accessed 13 Nov. 2020"},{"key":"84_CR94","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-020-0186-1","author":"GA Kaissis","year":"2020","unstructured":"Kaissis, G.A., Makowski, M.R., R\u00fcckert, D., Braren, R.F.: Secure, privacy-preserving and federated machine learning in medical imaging. Mach. Intell Nat. (2020). https:\/\/doi.org\/10.1038\/s42256-020-0186-1","journal-title":"Mach. Intell Nat."},{"issue":"4","key":"84_CR95","doi-asserted-by":"publisher","first-page":"507","DOI":"10.14778\/3436905.3436911","volume":"14","author":"A Chapman","year":"2020","unstructured":"Chapman, A., Missier, P., Simonelli, G., Torlone, R.: Capturing and querying fine-grained provenance of preprocessing pipelines in data science. Proc. VLDB Endow. 14(4), 507\u2013520 (2020). https:\/\/doi.org\/10.14778\/3436905.3436911","journal-title":"Proc. VLDB Endow."},{"key":"84_CR96","unstructured":"Information Commissioner\u2019s Office: Guidance on the AI auditing framework Draft guidance for consultation p. 105 (2020)"},{"key":"84_CR97","doi-asserted-by":"publisher","DOI":"10.17169\/fqs-20.3.3343","author":"P Mayring","year":"2019","unstructured":"Mayring, P.: Qualitative content analysis: demarcation, varieties, developments. Forum Qual. Sozialforschung Forum Qual. Soc. Res. (2019). https:\/\/doi.org\/10.17169\/fqs-20.3.3343","journal-title":"Forum Qual. Sozialforschung Forum Qual. Soc. Res."},{"key":"84_CR98","unstructured":"Carrier, R., Brown, S.: Taxonomy: AI Audit, Assurance, and Assessment. For Humanity (2021). https:\/\/forhumanity.center\/blog\/taxonomy-ai-audit-assurance-and-assessment. Accessed 26 Apr. 2021"},{"key":"84_CR99","unstructured":"Ada Lovelace Institute and Data Kind UK: Examining the black box: tools for assessing algorithmic systems (2020). https:\/\/www.adalovelaceinstitute.org\/report\/examining-the-black-box-tools-for-assessing-algorithmic-systems\/. Accessed 23 Feb. 2021"},{"key":"84_CR100","volume-title":"Qualitative Research and Evaluation Methods: Integrating Theory and Practice","author":"MQ Patton","year":"2014","unstructured":"Patton, M.Q.: Qualitative Research and Evaluation Methods: Integrating Theory and Practice. SAGE Publications (2014)"},{"key":"84_CR101","volume-title":"Content Analysis: An Introduction to Its Methodology","author":"K Krippendorff","year":"2013","unstructured":"Krippendorff, K.: Content Analysis: An Introduction to Its Methodology. SAGE (2013)"},{"key":"84_CR102","doi-asserted-by":"publisher","unstructured":"Lee, M. S. A., Singh, J.: The landscape and gaps in open source fairness toolkits. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, New York, pp. 1\u201313 (2021) https:\/\/doi.org\/10.1145\/3411764.3445261.","DOI":"10.1145\/3411764.3445261"},{"issue":"6","key":"84_CR103","doi-asserted-by":"publisher","first-page":"115:1","DOI":"10.1145\/3457607","volume":"54","author":"N Mehrabi","year":"2021","unstructured":"Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., Galstyan, A.: A survey on bias and fairness in machine learning. ACM Comput. Surv. 54(6), 115:1-115:35 (2021). https:\/\/doi.org\/10.1145\/3457607","journal-title":"ACM Comput. Surv."},{"key":"84_CR104","unstructured":"Vakkuri, V., Kemell, K.-K., Kultanen, J., Siponen, M., Abrahamsson, P.: Ethically aligned design of autonomous systems: industry viewpoint and an empirical study, p. 18 (2019)"},{"key":"84_CR105","unstructured":"Mulgan, G.: AI ethics and the limits of code(s). In: nesta (2019). https:\/\/www.nesta.org.uk\/blog\/ai-ethics-and-limits-codes\/. Accessed 16 Sep. 2019"},{"key":"84_CR106","unstructured":"Floridi, L.: Why Information Matters. In: The New Atlantis (2017). http:\/\/www.thenewatlantis.com\/publications\/why-information-matters. Accessed 14 Oct 2020"},{"key":"84_CR107","unstructured":"Kitchin, R.: The ethics of smart cities (2019). Available https:\/\/www.rte.ie\/brainstorm\/2019\/0425\/1045602-the-ethics-of-smart-cities\/. Accessed 07 May 2019 (Online)"},{"key":"84_CR108","doi-asserted-by":"publisher","unstructured":"Bietti, E.: From ethics washing to ethics bashing: a view on tech ethics from within moral philosophy. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, Barcelona, Spain, pp 210\u2013219 (2020) https:\/\/doi.org\/10.1145\/3351095.3372860.","DOI":"10.1145\/3351095.3372860"},{"key":"84_CR109","doi-asserted-by":"crossref","unstructured":"Metcalf, J., Moss, E., Watkins, E. A., Singh, R., Elish, M. C.: Algorithmic impact assessments and accountability: the co-construction of impacts, p 19 (2021)","DOI":"10.1145\/3442188.3445935"},{"key":"84_CR110","unstructured":"European Commission: Proposal for a Regulation laying down harmonised rules on artificial intelligence | Shaping Europe\u2019s digital future. In: European Commission, Brussels, Proposal (2021). Available https:\/\/digital-strategy.ec.europa.eu\/en\/library\/proposal-regulation-laying-down-harmonised-rules-artificial-intelligence. Accessed 21 May 2021 (Online)"},{"key":"84_CR111","unstructured":"Webster, G.: Translation: personal information protection law of the People\u2019s Republic of China (Draft) (Second Review Draft) | DigiChina. In: Stanford DigiChina Cyber Policy Unit (2021). https:\/\/digichina.stanford.edu\/news\/translation-personal-information-protection-law-peoples-republic-china-draft-second-review. Accessed 21 May 2021"},{"key":"84_CR112","unstructured":"Lee, A., Sacks, S., Creemers, R., Shi, M., Webster, G.: China\u2019s draft privacy law adds platform self-governance, solidifies CAC\u2019s Role | DigiChina. In: Stanford DigiChina Cyber Policy Unit (2021). https:\/\/digichina.stanford.edu\/news\/chinas-draft-privacy-law-adds-platform-self-governance-solidifies-cacs-role. Accessed 21 May 2021"},{"key":"84_CR113","unstructured":"Jillson, E.: Aiming for truth, fairness, and equity in your company\u2019s use of AI. In: Federal Trade Commission (2021). https:\/\/www.ftc.gov\/news-events\/blogs\/business-blog\/2021\/04\/aiming-truth-fairness-equity-your-companys-use-ai. Accessed 20 Apr. 2021"},{"key":"84_CR114","first-page":"1","volume-title":"The Oxford Handbook of Ethics of AI","author":"JJ Bryson","year":"2020","unstructured":"Bryson, J.J.: The artificial intelligence of the ethics of artificial intelligence: an introductory overview for law and regulation. In: Dubber, M.D., Pasquale, F., Das, S. (eds.) The Oxford Handbook of Ethics of AI, pp. 1\u201325. Oxford University Press (2020)"},{"key":"84_CR115","unstructured":"CDEI: Types of assurance in AI and the role of standards. In: Centre for Data Ethics and Innovation Blog (2021). https:\/\/cdei.blog.gov.uk\/2021\/04\/17\/134\/. Accessed 26 May 2021"},{"key":"84_CR116","unstructured":"European Parliament: The adequate protection of personal data by the United Kingdom (2021) https:\/\/www.europarl.europa.eu\/doceo\/document\/TA-9-2021-0262_EN.html. Accessed 26 May 2021."},{"key":"84_CR117","doi-asserted-by":"publisher","DOI":"10.4324\/9780203108543","volume-title":"Routledge International Handbook of Participatory Design","author":"J Simonsen","year":"2012","unstructured":"Simonsen, J., Robertson, T.: Routledge International Handbook of Participatory Design. Routledge, London (2012)"},{"key":"84_CR118","unstructured":"Beck, E.: P for Political: Participation is not enough. Scand. J. Inf. Syst. 14(1) (2002). Available at https:\/\/aisel.aisnet.org\/sjis\/vol14\/iss1\/1. (Online)"},{"key":"84_CR119","unstructured":"Ada Lovelace Institute: Our Strategy (2020). https:\/\/www.adalovelaceinstitute.org\/about\/. Accessed 26 May 2021"},{"key":"84_CR120","unstructured":"Thuermer, G., Walker, J., Simperl, E., Carr, L.: When data meets citizens: an investigation of citizen engagement in data-driven innovation programmes. In: Presented at the 2nd Data Justice Conference, Cardiff University Online (2021)"},{"key":"84_CR121","unstructured":"Sloane, M., Moss, E., Awomolo, O., Forlano, L.: Participation is not a design fix for machine learning (2020). Available http:\/\/arxiv.org\/abs\/2007.02423. Accessed 26 May 2021. (Online)"}],"container-title":["AI and Ethics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s43681-021-00084-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s43681-021-00084-x\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s43681-021-00084-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,8,9]],"date-time":"2022-08-09T02:36:51Z","timestamp":1660012611000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s43681-021-00084-x"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,9,12]]},"references-count":121,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2022,8]]}},"alternative-id":["84"],"URL":"https:\/\/doi.org\/10.1007\/s43681-021-00084-x","relation":{},"ISSN":["2730-5953","2730-5961"],"issn-type":[{"value":"2730-5953","type":"print"},{"value":"2730-5961","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,9,12]]},"assertion":[{"value":"29 May 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"10 August 2021","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 September 2021","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"No conflicts.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}