{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,16]],"date-time":"2026-06-16T13:13:24Z","timestamp":1781615604157,"version":"3.54.5"},"reference-count":59,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2023,9,9]],"date-time":"2023-09-09T00:00:00Z","timestamp":1694217600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,9,9]],"date-time":"2023-09-09T00:00:00Z","timestamp":1694217600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100010661","name":"Horizon 2020 Framework Programme","doi-asserted-by":"publisher","award":["101016233"],"award-info":[{"award-number":["101016233"]}],"id":[{"id":"10.13039\/100010661","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100013294","name":"Connecting Europe Facility","doi-asserted-by":"publisher","award":["INEA\/CEF\/ICT\/A2020\/2276680"],"award-info":[{"award-number":["INEA\/CEF\/ICT\/A2020\/2276680"]}],"id":[{"id":"10.13039\/100013294","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000009","name":"Foundation for the National Institutes of Health","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100000009","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Johann Wolfgang Goethe-Universit\u00e4t, Frankfurt am Main"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["DISO"],"published-print":{"date-parts":[[2023,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Building artificial intelligence (AI) systems that adhere to ethical standards is a complex problem. Even though a multitude of guidelines for the design and development of such trustworthy AI systems exist, these guidelines focus on high-level and abstract requirements for AI systems, and it is often very difficult to assess if a specific system fulfills these requirements. The Z-Inspection\u00ae process provides a holistic and dynamic framework to evaluate the trustworthiness of specific AI systems at different stages of the AI lifecycle, including intended use, design, and development. It focuses, in particular, on the discussion and identification of ethical issues and tensions through the analysis of socio-technical scenarios and a requirement-based framework for ethical and trustworthy AI. This article is a methodological reflection on the Z-Inspection\u00ae process. We illustrate how high-level guidelines for ethical and trustworthy AI can be applied in practice and provide insights for both AI researchers and AI practitioners. We share the lessons learned from conducting a series of independent assessments to evaluate the trustworthiness of real-world AI systems, as well as key recommendations and practical suggestions on how to ensure a rigorous trustworthiness assessment throughout the lifecycle of an AI system. The results presented in this article are based on our assessments of AI systems in the healthcare sector and environmental monitoring, where we used the framework for trustworthy AI proposed in the <jats:italic>Ethics Guidelines for Trustworthy AI<\/jats:italic> by the European Commission\u2019s High-Level Expert Group on AI. However, the assessment process and the lessons learned can be adapted to other domains and include additional frameworks.<\/jats:p>","DOI":"10.1007\/s44206-023-00063-1","type":"journal-article","created":{"date-parts":[[2023,9,9]],"date-time":"2023-09-09T13:01:46Z","timestamp":1694264506000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":27,"title":["Lessons Learned from Assessing Trustworthy AI in Practice"],"prefix":"10.1007","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5977-5535","authenticated-orcid":false,"given":"Dennis","family":"Vetter","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Julia","family":"Amann","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Fr\u00e9d\u00e9rick","family":"Bruneault","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Megan","family":"Coffee","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Boris","family":"D\u00fcdder","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alessio","family":"Gallucci","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thomas Krendl","family":"Gilbert","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Thilo","family":"Hagendorff","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Irmhild","family":"van Halem","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Eleanore","family":"Hickman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Elisabeth","family":"Hildt","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sune","family":"Holm","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Georgios","family":"Kararigas","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pedro","family":"Kringen","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vince I.","family":"Madai","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Emilie","family":"Wiinblad Mathez","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jesmin Jahan","family":"Tithi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Magnus","family":"Westerlund","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Renee","family":"Wurth","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Roberto V.","family":"Zicari","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"name":"Z-Inspection\u00ae initiative (2022)","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2023,9,9]]},"reference":[{"key":"63_CR1","unstructured":"(AI HLEG) High-Level Expert Group on Artificial Intelligence. (2019). Ethics guidelines for trustworthy AI [Text]. European Commission. https:\/\/digital-strategy.ec.europa.eu\/en\/library\/ethics-guidelines-trustworthy-ai"},{"key":"63_CR2","unstructured":"(AI HLEG) High-Level Expert Group on Artificial Intelligence. (2020). Assessment List for Trustworthy Artificial Intelligence (ALTAI) for self-assessment [Text]. European Commission. https:\/\/ec.europa.eu\/newsroom\/dae\/document.cfm?doc_id=68342"},{"key":"63_CR3","doi-asserted-by":"publisher","unstructured":"Allahabadi, H., Amann, J., Balot, I., Beretta, A., Binkley, C., Bozenhard, J., Bruneault, F., Brusseau, J., Candemir, S., Cappellini, L. A., Chakraborty, S., Cherciu, N., Cociancig, C., Coffee, M., Ek, I., Espinosa-Leal, L., Farina, D., Fieux-Castagnet, G., Frauenfelder, T., & Zicari, R. V. (2022). Assessing Trustworthy AI in Times of COVID-19: Deep Learning for Predicting a Multiregional Score Conveying the Degree of Lung Compromise in COVID-19 Patients. IEEE Transactions on Technology and Society, 3(4), 272\u2013289. https:\/\/doi.org\/10.1109\/TTS.2022.3195114","DOI":"10.1109\/TTS.2022.3195114"},{"key":"63_CR4","doi-asserted-by":"publisher","unstructured":"Amann, J., Vetter, D., Blomberg, S. N., Christensen, H. C., Coffee, M., Gerke, S., Gilbert, T. K., Hagendorff, T., Holm, S., Livne, M., Spezzatti, A., Str\u00fcmke, I., Zicari, R. V., Madai, V. I., & on behalf of the Z-Inspection Initiative. (2022). To explain or not to explain?\u2014Artificial intelligence explainability in clinical decision support systems. PLOS Digital Health, 1(2), e0000016. https:\/\/doi.org\/10.1371\/journal.pdig.0000016","DOI":"10.1371\/journal.pdig.0000016"},{"key":"63_CR5","unstructured":"Angwin, J., Larson, J., Mattu, S., & Kirchner, L. (2016). Machine Bias. ProPublica. https:\/\/www.propublica.org\/article\/machine-bias-risk-assessments-in-criminal-sentencing?token=l0i8JndZRzf9U7hmG1DlFV6RjLJo1zYf"},{"key":"63_CR6","doi-asserted-by":"publisher","DOI":"10.1007\/s00146-021-01380-0","author":"J-C B\u00e9lisle-Pipon","year":"2022","unstructured":"B\u00e9lisle-Pipon, J.-C., Monteferrante, E., Roy, M.-C., & Couture, V. (2022). Artificial intelligence ethics has a black box problem. AI & SOCIETY. https:\/\/doi.org\/10.1007\/s00146-021-01380-0","journal-title":"AI & SOCIETY"},{"key":"63_CR7","doi-asserted-by":"publisher","first-page":"322","DOI":"10.1016\/j.resuscitation.2019.01.015","volume":"138","author":"SN Blomberg","year":"2019","unstructured":"Blomberg, S. N., Folke, F., Ersb\u00f8ll, A. K., Christensen, H. C., Torp-Pedersen, C., Sayre, M. R., Counts, C. R., & Lippert, F. K. (2019). Machine learning as a supportive tool to recognize cardiac arrest in emergency calls. Resuscitation, 138, 322\u2013329. https:\/\/doi.org\/10.1016\/j.resuscitation.2019.01.015","journal-title":"Resuscitation"},{"key":"63_CR8","doi-asserted-by":"publisher","first-page":"186","DOI":"10.1109\/ISSREW.2014.72","volume":"2014","author":"R Bloomfield","year":"2014","unstructured":"Bloomfield, R., & Netkachova, K. (2014). Building Blocks for Assurance Cases. IEEE International Symposium on Software Reliability Engineering Workshops, 2014, 186\u2013191. https:\/\/doi.org\/10.1109\/ISSREW.2014.72","journal-title":"IEEE International Symposium on Software Reliability Engineering Workshops"},{"key":"63_CR9","unstructured":"Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., Brynjolfsson, E., Buch, S., Card, D., Castellon, R., Chatterji, N., Chen, A., Creel, K., Davis, J. Q., Demszky, D., & Liang, P. (2021). On the Opportunities and Risks of Foundation Models. ArXiv:2108.07258 [Cs]. http:\/\/arxiv.org\/abs\/2108.07258"},{"issue":"1","key":"63_CR10","doi-asserted-by":"publisher","first-page":"205395172098386","DOI":"10.1177\/2053951720983865","volume":"8","author":"S Brown","year":"2021","unstructured":"Brown, S., Davidovic, J., & Hasan, A. (2021). The algorithm audit: Scoring the algorithms that score us. Big Data & Society, 8(1), 2053951720983865. https:\/\/doi.org\/10.1177\/2053951720983865","journal-title":"Big Data & Society"},{"key":"63_CR11","unstructured":"Brundage, M., Avin, S., Wang, J., Belfield, H., Krueger, G., Hadfield, G., Khlaaf, H., Yang, J., Toner, H., Fong, R., Maharaj, T., Koh, P. W., Hooker, S., Leung, J., Trask, A., Bluemke, E., Lebensold, J., O\u2019Keefe, C., Koren, M., & Anderljung, M. (2020). Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims. ArXiv:2004.07213 [Cs]. http:\/\/arxiv.org\/abs\/2004.07213"},{"key":"63_CR12","doi-asserted-by":"publisher","unstructured":"Brusseau, J. (2020). What a Philosopher Learned at an AI Ethics Evaluation. AI Ethics Journal, 1(1). https:\/\/doi.org\/10.47289\/AIEJ20201214","DOI":"10.47289\/AIEJ20201214"},{"key":"63_CR13","doi-asserted-by":"publisher","unstructured":"Chopra, A. K., & Singh, M. P. (2018). Sociotechnical Systems and Ethics in the Large. Proceedings of the 2018 AAAI\/ACM Conference on AI, Ethics, and Society, 48\u201353. https:\/\/doi.org\/10.1145\/3278721.3278740","DOI":"10.1145\/3278721.3278740"},{"key":"63_CR14","doi-asserted-by":"publisher","unstructured":"Cobbe, J., Lee, M. S. A., & Singh, J. (2021). Reviewable Automated Decision-Making: A Framework for Accountable Algorithmic Systems. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 598\u2013609. https:\/\/doi.org\/10.1145\/3442188.3445921","DOI":"10.1145\/3442188.3445921"},{"key":"63_CR15","doi-asserted-by":"publisher","unstructured":"Colquitt, J. A., & Rodell, J. B. (2015). Measuring Justice and Fairness. In R. S. Cropanzano & M. L. Ambrose (Eds.), The Oxford Handbook of Justice in the Workplace (p. 0). Oxford University Press. https:\/\/doi.org\/10.1093\/oxfordhb\/9780199981410.013.0008","DOI":"10.1093\/oxfordhb\/9780199981410.013.0008"},{"key":"63_CR16","doi-asserted-by":"publisher","unstructured":"Costanza-Chock, S., Raji, I. D., & Buolamwini, J. (2022). Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem. Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, 1571\u20131583. https:\/\/doi.org\/10.1145\/3531146.3533213","DOI":"10.1145\/3531146.3533213"},{"key":"63_CR17","unstructured":"Datenethikkommission. (2019). Opinion of the Data Ethics Commission (p. 238). Federal Ministry of Justice and Consumer Protection. https:\/\/www.bmjv.de\/SharedDocs\/Downloads\/DE\/Themen\/Fokusthemen\/Gutachten_DEK_EN_lang.pdf?__blob=publicationFile&v=3"},{"issue":"2","key":"63_CR18","doi-asserted-by":"publisher","first-page":"62","DOI":"10.1109\/TTS.2021.3074097","volume":"2","author":"S Dean","year":"2021","unstructured":"Dean, S., Gilbert, T. K., Lambert, N., & Zick, T. (2021). Axes for Sociotechnical Inquiry in AI Research. IEEE Transactions on Technology and Society, 2(2), 62\u201370. https:\/\/doi.org\/10.1109\/TTS.2021.3074097","journal-title":"IEEE Transactions on Technology and Society"},{"key":"63_CR19","doi-asserted-by":"publisher","unstructured":"Dobbe, R., Krendl Gilbert, T., & Mintz, Y. (2021). Hard choices in artificial intelligence. Artificial Intelligence, 300, 103555. https:\/\/doi.org\/10.1016\/j.artint.2021.103555","DOI":"10.1016\/j.artint.2021.103555"},{"key":"63_CR20","doi-asserted-by":"crossref","unstructured":"D\u00fcdder, B., M\u00f6slein, F., St\u00fcrtz, N., Westerlund, M., & Zicari, R. V. (2020). Ethical Maintenance of Artificial Intelligence Systems. In M. Pagani & R. Champion (Eds.), Artificial Intelligence for Sustainable Value Creation. Edward Elgar Publishing.","DOI":"10.4337\/9781839104398.00018"},{"key":"63_CR21","unstructured":"European Commission. (2021). Laying Down Harmonised Rules on Artificial Intelligence (Artificial Intelligence Act) and Amending Certain Union legislative Acts (COM(2021) 206 final). https:\/\/eur-lex.europa.eu\/legal-content\/EN\/TXT\/?uri=CELEX:52021PC0206"},{"key":"63_CR22","doi-asserted-by":"publisher","unstructured":"Falco, G., Shneiderman, B., Badger, J., Carrier, R., Dahbura, A., Danks, D., Eling, M., Goodloe, A., Gupta, J., Hart, C., Jirotka, M., Johnson, H., LaPointe, C., Llorens, A. J., Mackworth, A. K., Maple, C., P\u00e1lsson, S. E., Pasquale, F., Winfield, A., & Yeong, Z. K. (2021). Governing AI safety through independent audits. Nature Machine Intelligence, 3(7), Article 7. https:\/\/doi.org\/10.1038\/s42256-021-00370-7","DOI":"10.1038\/s42256-021-00370-7"},{"issue":"2","key":"63_CR23","doi-asserted-by":"publisher","first-page":"13","DOI":"10.1007\/s44206-022-00016-0","volume":"1","author":"A Fell\u00e4nder","year":"2022","unstructured":"Fell\u00e4nder, A., Rebane, J., Larsson, S., Wiggberg, M., & Heintz, F. (2022). Achieving a Data-Driven Risk Assessment Methodology for Ethical AI. Digital Society, 1(2), 13. https:\/\/doi.org\/10.1007\/s44206-022-00016-0","journal-title":"Digital Society"},{"key":"63_CR24","doi-asserted-by":"publisher","unstructured":"Floridi, L., Holweg, M., Taddeo, M., Amaya Silva, J., M\u00f6kander, J., & Wen, Y. (2022). CapAI - A Procedure for Conducting Conformity Assessment of AI Systems in Line with the EU Artificial Intelligence Act (SSRN Scholarly Paper No. 4064091). https:\/\/doi.org\/10.2139\/ssrn.4064091","DOI":"10.2139\/ssrn.4064091"},{"key":"63_CR25","unstructured":"ForHumanity. (2021). Independent Audit of AI Systems. https:\/\/forhumanity.center\/independent-audit-of-ai-systems\/"},{"key":"63_CR26","unstructured":"Gerards, J., Sch\u00e4fer, M. T., Vankan, A., & Muis, I. (2022). Impact Assessment\u2014Fundamental rights and algorithms (p. 99). Ministry of the Interior and Kingdom Relations. https:\/\/www.government.nl\/binaries\/government\/documenten\/reports\/2021\/07\/31\/impact-assessment-fundamental-rights-and-algorithms\/fundamental-rights-and-algorithms-impact-assessment-fraia.pdf"},{"key":"63_CR27","doi-asserted-by":"publisher","unstructured":"Gilbert, T. K., Dean, S., Lambert, N., Zick, T., & Snoswell, A. (2022). Reward Reports for Reinforcement Learning. (arXiv:2204.10817\n). arXiv. https:\/\/doi.org\/10.48550\/arXiv.2204.10817","DOI":"10.48550\/arXiv.2204.10817"},{"issue":"1","key":"63_CR28","doi-asserted-by":"publisher","first-page":"99","DOI":"10.1007\/s11023-020-09517-8","volume":"30","author":"T Hagendorff","year":"2020","unstructured":"Hagendorff, T. (2020). The Ethics of AI Ethics: An Evaluation of Guidelines. Minds and Machines, 30(1), 99\u2013120. https:\/\/doi.org\/10.1007\/s11023-020-09517-8","journal-title":"Minds and Machines"},{"key":"63_CR29","unstructured":"Hamilton, I. A. (2018). Amazon built an AI tool to hire people but had to shut it down because it was discriminating against women. Business Insider. https:\/\/www.businessinsider.com\/amazon-built-ai-to-hire-people-discriminated-against-women-2018-10"},{"issue":"4","key":"63_CR30","doi-asserted-by":"publisher","first-page":"593","DOI":"10.1007\/s40804-021-00224-0","volume":"22","author":"E Hickman","year":"2021","unstructured":"Hickman, E., & Petrin, M. (2021). Trustworthy AI and Corporate Governance: The EU\u2019s Ethics Guidelines for Trustworthy Artificial Intelligence from a Company Law Perspective. European Business Organization Law Review, 22(4), 593\u2013625. https:\/\/doi.org\/10.1007\/s40804-021-00224-0","journal-title":"European Business Organization Law Review"},{"key":"63_CR31","unstructured":"IEEE SA - The IEEE Standards Association. (n.d.). IEEE CertifAIEd\u2014The Mark of AI Ethics. Retrieved November 23, 2021, from https:\/\/engagestandards.ieee.org\/ieeecertifaied.html"},{"key":"63_CR32","unstructured":"Insight Centre. (n.d.). How to complete ALTAI - ALTAI. Retrieved March 2, 2022, from https:\/\/altai.insight-centre.org\/Home\/HowToComplete"},{"key":"63_CR33","doi-asserted-by":"publisher","unstructured":"Jobin, A., Ienca, M., & Vayena, E. (2019). The global landscape of AI ethics guidelines. Nature Machine Intelligence, 1(9), Article 9. https:\/\/doi.org\/10.1038\/s42256-019-0088-2","DOI":"10.1038\/s42256-019-0088-2"},{"issue":"3","key":"63_CR34","doi-asserted-by":"publisher","first-page":"388","DOI":"10.1080\/0960085X.2021.1927212","volume":"31","author":"N Kordzadeh","year":"2022","unstructured":"Kordzadeh, N., & Ghasemaghaei, M. (2022). Algorithmic bias: Review, synthesis, and future research directions. European Journal of Information Systems, 31(3), 388\u2013409. https:\/\/doi.org\/10.1080\/0960085X.2021.1927212","journal-title":"European Journal of Information Systems"},{"key":"63_CR35","doi-asserted-by":"publisher","unstructured":"Leikas, J., Koivisto, R., & Gotcheva, N. (2019). Ethical Framework for Designing Autonomous Intelligent Systems. Journal of Open Innovation: Technology, Market, and Complexity, 5(1), Article 1. https:\/\/doi.org\/10.3390\/joitmc5010018","DOI":"10.3390\/joitmc5010018"},{"key":"63_CR36","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1109\/IJCNN48605.2020.9206946","volume":"2020","author":"A Lucieri","year":"2020","unstructured":"Lucieri, A., Bajwa, M. N., Braun, S. A., Malik, M. I., Dengel, A., & Ahmed, S. (2020). On Interpretability of Deep Learning based Skin Lesion Classifiers using Concept Activation Vectors. International Joint Conference on Neural Networks (IJCNN), 2020, 1\u201310. https:\/\/doi.org\/10.1109\/IJCNN48605.2020.9206946","journal-title":"International Joint Conference on Neural Networks (IJCNN)"},{"key":"63_CR37","doi-asserted-by":"publisher","unstructured":"Lucivero, F. (2016). Ethical Assessments of Emerging Technologies: Appraising the moral plausibility of technological visions (1st ed. 2016). Springer International Publishing\u202f: Imprint: Springer. https:\/\/doi.org\/10.1007\/978-3-319-23282-9","DOI":"10.1007\/978-3-319-23282-9"},{"key":"63_CR38","unstructured":"Madiega, T. (2022). Briefing\u2014EU Legislation in Process. Artificial intelligence act. (p. 12). European Parliamentary Research Service. https:\/\/www.europarl.europa.eu\/thinktank\/en\/document\/EPRS_BRI(2021)698792"},{"key":"63_CR39","unstructured":"Ministerie van Binnenlandse Zaken en Koninkrijksrelaties. (2022). Pilot: Assessment voor verantwoorde Artificial Intelligence - Rijks ICT Gilde - UBRijk [Webpagina]. Ministerie van Algemene Zaken. https:\/\/www.rijksorganisatieodi.nl\/rijks-ict-gilde\/mycelia\/pilot-kunstmatige-intelligentie"},{"issue":"3","key":"63_CR40","doi-asserted-by":"publisher","first-page":"21","DOI":"10.1007\/s44206-022-00022-2","volume":"1","author":"M Minkkinen","year":"2022","unstructured":"Minkkinen, M., Laine, J., & M\u00e4ntym\u00e4ki, M. (2022). Continuous Auditing of Artificial Intelligence: A Conceptualization and Assessment of Tools and Frameworks. Digital Society, 1(3), 21. https:\/\/doi.org\/10.1007\/s44206-022-00022-2","journal-title":"Digital Society"},{"issue":"2","key":"63_CR41","doi-asserted-by":"publisher","first-page":"205395171667967","DOI":"10.1177\/2053951716679679","volume":"3","author":"BD Mittelstadt","year":"2016","unstructured":"Mittelstadt, B. D., Allo, P., Taddeo, M., Wachter, S., & Floridi, L. (2016). The ethics of algorithms: Mapping the debate. Big Data & Society, 3(2), 205395171667967. https:\/\/doi.org\/10.1177\/2053951716679679","journal-title":"Big Data & Society"},{"issue":"2","key":"63_CR42","doi-asserted-by":"publisher","first-page":"241","DOI":"10.1007\/s11023-021-09577-4","volume":"32","author":"J M\u00f6kander","year":"2022","unstructured":"M\u00f6kander, J., Axente, M., Casolari, F., & Floridi, L. (2022). Conformity Assessments and Post-market Monitoring: A Guide to the Role of Auditing in the Proposed European AI Regulation. Minds and Machines, 32(2), 241\u2013268. https:\/\/doi.org\/10.1007\/s11023-021-09577-4","journal-title":"Minds and Machines"},{"issue":"4","key":"63_CR43","doi-asserted-by":"publisher","first-page":"44","DOI":"10.1007\/s11948-021-00319-4","volume":"27","author":"J M\u00f6kander","year":"2021","unstructured":"M\u00f6kander, J., Morley, J., Taddeo, M., & Floridi, L. (2021). Ethics-Based Auditing of Automated Decision-Making Systems: Nature, Scope, and Limitations. Science and Engineering Ethics, 27(4), 44. https:\/\/doi.org\/10.1007\/s11948-021-00319-4","journal-title":"Science and Engineering Ethics"},{"key":"63_CR44","doi-asserted-by":"publisher","DOI":"10.1007\/s00146-021-01308-8","author":"J Morley","year":"2021","unstructured":"Morley, J., Kinsey, L., Elhalal, A., Garcia, F., Ziosi, M., & Floridi, L. (2021). Operationalising AI ethics: Barriers, enablers and next steps. AI & SOCIETY. https:\/\/doi.org\/10.1007\/s00146-021-01308-8","journal-title":"AI & SOCIETY"},{"key":"63_CR45","unstructured":"OECD. (2019). Recommendation of the Council on Artificial Intelligence (C\/MIN(2019)3\/FINAL). Organisation for Economic Co-operation and Development (OECD). https:\/\/legalinstruments.oecd.org\/en\/instruments\/OECD-LEGAL-0449"},{"key":"63_CR46","doi-asserted-by":"publisher","unstructured":"Schiff, D., Biddle, J., Borenstein, J., & Laas, K. (2020). What\u2019s Next for AI Ethics, Policy, and Governance? A Global Overview. Proceedings of the AAAI\/ACM Conference on AI, Ethics, and Society, 153\u2013158. https:\/\/doi.org\/10.1145\/3375627.3375804","DOI":"10.1145\/3375627.3375804"},{"key":"63_CR47","first-page":"117","volume":"35","author":"AD Selbst","year":"2021","unstructured":"Selbst, A. D. (2021). An Institutional View of Algorithmic Impact Assessments. Harvard Journal of Law & Technology (harvard JOLT), 35, 117.","journal-title":"Harvard Journal of Law & Technology (harvard JOLT)"},{"key":"63_CR48","doi-asserted-by":"publisher","unstructured":"Selbst, A. D., Boyd, D., Friedler, S. A., Venkatasubramanian, S., & Vertesi, J. (2019). Fairness and Abstraction in Sociotechnical Systems. Proceedings of the Conference on Fairness, Accountability, and Transparency, 59\u201368. https:\/\/doi.org\/10.1145\/3287560.3287598","DOI":"10.1145\/3287560.3287598"},{"key":"63_CR49","doi-asserted-by":"publisher","unstructured":"Signoroni, A., Savardi, M., Benini, S., Adami, N., Leonardi, R., Gibellini, P., Vaccher, F., Ravanelli, M., Borghesi, A., Maroldi, R., & Farina, D. (2021). BS-Net: Learning COVID-19 pneumonia severity on a large chest X-ray dataset. Medical Image Analysis, 71, 102046. https:\/\/doi.org\/10.1016\/j.media.2021.102046","DOI":"10.1016\/j.media.2021.102046"},{"key":"63_CR50","unstructured":"Thorbecke, C. (2019). New York probing Apple Card for alleged gender discrimination after viral tweet. ABC News. https:\/\/abcnews.go.com\/US\/york-probing-apple-card-alleged-gender-discrimination-viral\/story?id=66910300"},{"key":"63_CR51","unstructured":"UNESCO. (2021). Recommendation on the Ethics of Artificial Intelligence (SHS\/BIO\/PI\/2021\/1). United Nations Educational, Scientific and Cultural Organization (UNESCO). https:\/\/unesdoc.unesco.org\/ark:\/48223\/pf0000381137"},{"key":"63_CR52","doi-asserted-by":"publisher","unstructured":"Vetter, D., Tithi, J. J., Westerlund, M., Zicari, R. V., & Roig, G. (2022). Using Sentence Embeddings and Semantic Similarity for Seeking Consensus when Assessing Trustworthy AI (arXiv:2208.04608\n). arXiv. https:\/\/doi.org\/10.48550\/arXiv.2208.04608","DOI":"10.48550\/arXiv.2208.04608"},{"key":"63_CR53","unstructured":"Whittlestone, J., Nyrup, R., Alexandrova, A., Dihal, K., & Cave, S. (2019). Ethical and societal implications of algorithms, data, and artificial intelligence: A roadmap for research. Nuffield Foundation. https:\/\/www.nuffieldfoundation.org\/wp-content\/uploads\/2019\/02\/Ethical-and-Societal-Implications-of-Data-and-AI-report-Nuffield-Foundat.pdf"},{"key":"63_CR54","doi-asserted-by":"publisher","unstructured":"Zeng, Y., Lu, E., & Huangfu, C. (2018). Linking Artificial Intelligence Principles. (arXiv:1812.04814\n). arXiv. https:\/\/doi.org\/10.48550\/arXiv.1812.04814","DOI":"10.48550\/arXiv.1812.04814"},{"key":"63_CR55","doi-asserted-by":"publisher","unstructured":"Zicari, R. V., Ahmed, S., Amann, J., Braun, S. A., Brodersen, J., Bruneault, F., Brusseau, J., Campano, E., Coffee, M., Dengel, A., D\u00fcdder, B., Gallucci, A., Gilbert, T. K., Gottfrois, P., Goffi, E., Haase, C. B., Hagendorff, T., Hickman, E., Hildt, E., & Wurth, R. (2021a). Co-Design of a Trustworthy AI System in Healthcare: Deep Learning Based Skin Lesion Classifier. Frontiers in Human Dynamics, 3, 40. https:\/\/doi.org\/10.3389\/fhumd.2021.688152","DOI":"10.3389\/fhumd.2021.688152"},{"issue":"2","key":"63_CR56","doi-asserted-by":"publisher","first-page":"83","DOI":"10.1109\/TTS.2021.3066209","volume":"2","author":"RV Zicari","year":"2021","unstructured":"Zicari, R. V., Brodersen, J., Brusseau, J., D\u00fcdder, B., Eichhorn, T., Ivanov, T., Kararigas, G., Kringen, P., McCullough, M., M\u00f6slein, F., Mushtaq, N., Roig, G., St\u00fcrtz, N., Tolle, K., Tithi, J. J., van Halem, I., & Westerlund, M. (2021b). Z-Inspection\u00ae: A Process to Assess Trustworthy AI. IEEE Transactions on Technology and Society, 2(2), 83\u201397. https:\/\/doi.org\/10.1109\/TTS.2021.3066209","journal-title":"IEEE Transactions on Technology and Society"},{"key":"63_CR57","doi-asserted-by":"publisher","unstructured":"Zicari, R. V., Brusseau, J., Blomberg, S. N., Christensen, H. C., Coffee, M., Ganapini, M. B., Gerke, S., Gilbert, T. K., Hickman, E., Hildt, E., Holm, S., K\u00fchne, U., Madai, V. I., Osika, W., Spezzatti, A., Schnebel, E., Tithi, J. J., Vetter, D., Westerlund, M., & Kararigas, G. (2021c). On Assessing Trustworthy AI in Healthcare. Machine Learning as a Supportive Tool to Recognize Cardiac Arrest in Emergency Calls. Frontiers in Human Dynamics, 3, 30. https:\/\/doi.org\/10.3389\/fhumd.2021.673104","DOI":"10.3389\/fhumd.2021.673104"},{"key":"63_CR58","doi-asserted-by":"publisher","unstructured":"Zicari, R. V., Amann, J., Bruneault, F., Coffee, M., D\u00fcdder, B., Hickman, E., Gallucci, A., Gilbert, T. K., Hagendorff, T., van Halem, I., Hildt, E., Holm, S., Kararigas, G., Kringen, P., Madai, V. I., Mathez, E. W., Tithi, J. J., Vetter, D., Westerlund, M., & Wurth, R. (2022). How to Assess Trustworthy AI in Practice (arXiv:2206.09887). arXiv. https:\/\/doi.org\/10.48550\/arXiv.2206.09887","DOI":"10.48550\/arXiv.2206.09887"},{"key":"63_CR59","unstructured":"Z-Inspection\u00ae Initiative. (2023). Conference Reader. First World Z-Inspection Conference, Venice, Italy. https:\/\/z-inspection.org\/wp-content\/uploads\/2023\/05\/World-Z-inspection-conference-reader-.pdf"}],"container-title":["Digital Society"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s44206-023-00063-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s44206-023-00063-1\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s44206-023-00063-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,2,2]],"date-time":"2024-02-02T11:05:16Z","timestamp":1706871916000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s44206-023-00063-1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,9]]},"references-count":59,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2023,12]]}},"alternative-id":["63"],"URL":"https:\/\/doi.org\/10.1007\/s44206-023-00063-1","relation":{},"ISSN":["2731-4650","2731-4669"],"issn-type":[{"value":"2731-4650","type":"print"},{"value":"2731-4669","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,9,9]]},"assertion":[{"value":"5 January 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"22 August 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 September 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of Interest"}}],"article-number":"35"}}