{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,15]],"date-time":"2026-08-15T20:46:12Z","timestamp":1786826772667,"version":"build-2736575974"},"reference-count":127,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2026,8,7]],"date-time":"2026-08-07T00:00:00Z","timestamp":1786060800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,8,7]],"date-time":"2026-08-07T00:00:00Z","timestamp":1786060800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001782","name":"University of Melbourne","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100001782","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2027,2]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>With the advancement of AI models, more software systems are adopting AI as a component to facilitate automation. Pre-trained models (PTMs) have become a cornerstone of AI-based software, allowing for rapid integration and development with lower training cost. However, their adoption also introduces failure modes such as data leakage and biased outputs, that may require careful handling by downstream developers. While previous research has proposed taxonomies of these technical concerns and various mitigation strategies, how downstream developers address these issues during the development of general AI-based software when reusing PTMs remains unexplored. Understanding downstream developers\u2019 perspectives is essential because they directly influence how these potential failure concerns translate into practice, such as determining whether immediate risks like data leakage or model bias are recognised, mitigated, or inadvertently overlooked in real-world deployments. This study investigates downstream developers\u2019 concerns, practices and perceived challenges regarding practical AI failures during the development of AI-based software. To achieve this, we conducted a mixed-method study, including interviews with 16 participants, a survey of 86 practitioners, and an analysis of 874 AI incidents from the AI Incident Database. Our results reveal that while developers generally demonstrate strong awareness of potential AI failures, their practices, especially during the preparation and model selection phases, are often inadequate. The lack of concrete guidelines and policies leads to significant variability in the comprehensiveness of their approaches throughout the development lifecycle, with additional challenges such as poor documentation and knowledge gaps, further impeding effective implementation. Based on our findings, we offer suggestions for AI model contributors, developers of AI-based software, researchers, and policy makers to enhance the integration of failure mitigation measures aimed at mitigating direct harms from AI failures.<\/jats:p>","DOI":"10.1007\/s10664-026-10937-w","type":"journal-article","created":{"date-parts":[[2026,8,7]],"date-time":"2026-08-07T06:15:44Z","timestamp":1786083344000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["AI failures in the eyes of the downstream developer: a first look at concerns, practices, and challenges"],"prefix":"10.1007","volume":"32","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3274-8630","authenticated-orcid":false,"given":"Haoyu","family":"Gao","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mansooreh","family":"Zahedi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wenxin","family":"Jiang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hong Yi","family":"Lin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"James C.","family":"Davis","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christoph","family":"Treude","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,8,7]]},"reference":[{"issue":"2","key":"10937_CR1","doi-asserted-by":"publisher","first-page":"299","DOI":"10.1007\/s43681-023-00266-9","volume":"4","author":"A Agarwal","year":"2024","unstructured":"Agarwal A, Agarwal H (2024) A seven-layer model with checklists for standardising fairness assessment throughout the ai lifecycle. AI and Ethics 4(2):299\u2013314","journal-title":"AI and Ethics"},{"key":"10937_CR2","doi-asserted-by":"crossref","unstructured":"Amershi S, Begel A, Bird C, DeLine R, Gall H, Kamar E, Nagappan N, Nushi B, Zimmermann T (2019) Software engineering for machine learning: A case study. In: 2019 IEEE\/ACM 41st international conference on software engineering: software engineering in practice (ICSE-SEIP). IEEE, pp 291\u2013300","DOI":"10.1109\/ICSE-SEIP.2019.00042"},{"key":"10937_CR3","unstructured":"Amodei D, Olah C, Steinhardt J, Christiano P, Schulman J, Man\u00e9 D (2016) Concrete problems in ai safety. arXiv:1606.06565"},{"key":"10937_CR4","doi-asserted-by":"crossref","unstructured":"Anandayuvaraj D, Thulluri P, Figueroa J, Shandilya H, Davis JC (2023) Incorporating failure knowledge into design decisions for iot systems: A controlled experiment on novices. In: 2023 IEEE\/ACM 5th international workshop on software engineering research and practices for the IoT (SERP4IoT). IEEE, pp 33\u201337","DOI":"10.1109\/SERP4IoT59158.2023.00011"},{"key":"10937_CR5","doi-asserted-by":"crossref","unstructured":"Anandayuvaraj D, Campbell M, Tewari A, Davis JC (2024) Fail: Analyzing software failures from the news using llms. In: 39th IEEE\/ACM international conference on automated software engineering, pp 506\u2013518","DOI":"10.1145\/3691620.3695022"},{"key":"10937_CR6","unstructured":"Anandayuvaraj D, Singla T, Hammadeh Z, Lund A, Holloway A, Davis JC (2025) Learning from software failures: a case study at a national space research center. arXiv:2509.06301"},{"key":"10937_CR7","unstructured":"Karpathy A (2017) Software 2.0. https:\/\/karpathy.medium.com\/software-2-0-a64152b37c35. Accessed: 13 Mar 2025"},{"key":"10937_CR8","doi-asserted-by":"crossref","unstructured":"Aniche M, Treude C, Steinmacher I, Wiese I, Pinto G, Storey M-A, Gerosa MA (2018) How modern news aggregators help development communities shape and share knowledge. In: Proceedings of the 40th international conference on software engineering, pp 499\u2013510","DOI":"10.1145\/3180155.3180180"},{"key":"10937_CR9","unstructured":"Australian Government (2019). https:\/\/www.industry.gov.au\/publications\/australias-artificial-intelligence-ethics-principles\/australias-ai-ethics-principles. [Accessed 30 Jun 2025]"},{"key":"10937_CR10","doi-asserted-by":"crossref","unstructured":"Banyongrakkul P, Zahedi M, Thongtanunam P, Treude C, Gao H (2025) From release to adoption: Challenges in reusing pre-trained ai models for downstream developers. In: 2025 IEEE international conference on software maintenance and evolution (ICSME)","DOI":"10.1109\/ICSME64153.2025.00022"},{"issue":"2","key":"10937_CR11","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1191\/1478088706qp063oa","volume":"3","author":"V Braun","year":"2006","unstructured":"Braun V, Clarke V (2006) Using thematic analysis in psychology. Qual Res Psychol 3(2):77\u2013101","journal-title":"Qual Res Psychol"},{"key":"10937_CR12","doi-asserted-by":"crossref","unstructured":"Braz L, Aeberhard C, \u00c7alikli G, Bacchelli A (2022) Less is more: supporting developers in vulnerability detection during code review. In: 44th International conference on software engineering, pp 1317\u20131329","DOI":"10.1145\/3510003.3511560"},{"issue":"2","key":"10937_CR13","doi-asserted-by":"publisher","first-page":"131","DOI":"10.1145\/130844.130856","volume":"24","author":"CW Krueger","year":"1992","unstructured":"Krueger CW (1992) Software reuse. ACM Comput Surv (CSUR) 24(2):131\u2013183","journal-title":"ACM Comput Surv (CSUR)"},{"key":"10937_CR14","volume-title":"Securing the ai software supply chain","author":"S Chaudhuri","year":"2024","unstructured":"Chaudhuri S, Dasgupta K, Hepworth I, Le M, Lodato M, Maruseac M, Meiklejohn S, Minkus T, Olive K (2024) Securing the ai software supply chain. Technical report, Google"},{"key":"10937_CR15","first-page":"15411","volume":"37","author":"P-Y Chen","year":"2023","unstructured":"Chen P-Y, Liu S (2023) Holistic adversarial robustness of deep learning models. AAAI Conf Artif Intell 37:15411\u201315420","journal-title":"AAAI Conf Artif Intell"},{"issue":"4","key":"10937_CR16","doi-asserted-by":"publisher","first-page":"772","DOI":"10.1002\/spe.2949","volume":"51","author":"N Chong","year":"2021","unstructured":"Chong N, Cook B, Eidelman J, Kallas K, Khazem K, Monteiro FR, Schwartz-Narbonne D, Tasiran S, Tautschnig M, Tuttle MR (2021) Code-level model checking in the software development workflow at amazon web services. Softw Pract Exper 51(4):772\u2013797","journal-title":"Softw Pract Exper"},{"key":"10937_CR17","doi-asserted-by":"crossref","unstructured":"Ciolacu M, Tehrani AF, Binder L, Svasta PM (2018) Education 4.0-artificial intelligence assisted higher education: early recognition system with machine learning to support students\u2019 success. In: IEEE international symposium for design and technology in electronic packaging. IEEE, pp 23\u201330","DOI":"10.1109\/SIITME.2018.8599203"},{"key":"10937_CR18","doi-asserted-by":"crossref","unstructured":"Cruzes DS, Dyba T (2011) Recommended steps for thematic synthesis in software engineering. In: 2011 International symposium on empirical software engineering and measurement. IEEE, pp 275\u2013284","DOI":"10.1109\/ESEM.2011.36"},{"issue":"1","key":"10937_CR19","doi-asserted-by":"publisher","first-page":"101","DOI":"10.1080\/00223980.1989.10542966","volume":"123","author":"DA Beach","year":"1989","unstructured":"Beach DA (1989) Identifying the random responder. J Psychol 123(1):101\u2013103","journal-title":"J Psychol"},{"key":"10937_CR20","doi-asserted-by":"publisher","first-page":"311","DOI":"10.1007\/s10664-005-1290-x","volume":"10","author":"TC Lethbridge","year":"2005","unstructured":"Lethbridge TC, Sim SE, Singer J (2005) Studying: data collection techniques for software field studies software engineers. Empir Softw Eng 10:311\u2013341","journal-title":"Empir Softw Eng"},{"key":"10937_CR21","unstructured":"Yi DO, Mu R, Jin G, Qi Y, Hu J, Zhao X, Meng J, Ruan W, Huang X (2024) Position: Building guardrails for large language models requires systematic design. In: Forty-first international conference on machine learning"},{"key":"10937_CR22","doi-asserted-by":"crossref","unstructured":"Easterbrook S, Singer J, Storey M-A, Damian D (2008) Selecting empirical methods for software engineering research. In: Guide to advanced empirical software engineering. Springer, pp. 285\u2013311","DOI":"10.1007\/978-1-84800-044-5_11"},{"key":"10937_CR23","doi-asserted-by":"publisher","first-page":"71","DOI":"10.1016\/j.infsof.2016.04.002","volume":"75","author":"L Eduardo","year":"2016","unstructured":"Eduardo L, Martins G, Gorschek T (2016) Requirements engineering for safety-critical systems: a systematic literature review. Inf Softw Technol 75:71\u201389","journal-title":"Inf Softw Technol"},{"key":"10937_CR24","unstructured":"Weisstein EW (2004) Bonferroni correction. https:\/\/mathworld. wolfram. com\/"},{"key":"10937_CR25","unstructured":"European Commission (2020) Futurium | European AI Alliance - Welcome to the ALTAI portal! \u2014 futurium.ec.europa.eu. https:\/\/futurium.ec.europa.eu\/en\/european-ai-alliance\/pages\/welcome-altai-portal. [Accessed 24 Feb 2025]"},{"key":"10937_CR26","unstructured":"European Union (2024) The AI Act Explorer | EU Artificial Intelligence Act \u2014 artificialintelligenceact.eu. https:\/\/artificialintelligenceact.eu\/ai-act-explorer\/. [Accessed 30 Jun 2025]"},{"issue":"7","key":"10937_CR27","doi-asserted-by":"publisher","first-page":"566","DOI":"10.1038\/s42256-021-00370-7","volume":"3","author":"G Falco","year":"2021","unstructured":"Falco G, Shneiderman B, Badger J, Carrier R, Dahbura A, Danks D, Eling M, Goodloe A, Gupta J, Hart C et al (2021) Governing ai safety through independent audits. Nat Mach Intell 3(7):566\u2013571","journal-title":"Nat Mach Intell"},{"key":"10937_CR28","doi-asserted-by":"crossref","unstructured":"Fernandes M, Ferino S, Fernandes A, Kulesza U, Aranha E, Treude C (2022) Devops education: an interview study of challenges and recommendations. In: ACM\/IEEE 44th international conference on software engineering: software engineering education and training, pp 90\u2013101","DOI":"10.1145\/3510456.3514152"},{"key":"10937_CR29","doi-asserted-by":"crossref","unstructured":"Fleisig E, Smith G, Bossi M, Rustagi I, Yin X, Klein D (2024) Linguistic bias in ChatGPT: language models reinforce dialect discrimination. In: Al-Onaizan Y, Bansal M, Chen Y-N (eds) 2024 Conference on empirical methods in natural language processing. Miami, Florida, USA, November 2024. Association for Computational Linguistics, pp 13541\u201313564","DOI":"10.18653\/v1\/2024.emnlp-main.750"},{"key":"10937_CR30","doi-asserted-by":"crossref","unstructured":"Gao H, Treude C, Zahedi M (2023) Evaluating transfer learning for simplifying github readmes. ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering","DOI":"10.1145\/3611643.3616291"},{"key":"10937_CR31","doi-asserted-by":"crossref","unstructured":"Gao H, Zahedi M, Treude C, Rosenstock S, Cheong M (2024) Documenting ethical considerations in open source ai models. In: International symposium on empirical software engineering and measurement","DOI":"10.1145\/3674805.3686679"},{"key":"10937_CR32","doi-asserted-by":"crossref","unstructured":"Gao H, Treude C, Zahedi M (2025) Adapting installation instructions in rapidly evolving software ecosystems. IEEE Transactions on Software Engineering","DOI":"10.1109\/TSE.2025.3552614"},{"key":"10937_CR33","doi-asserted-by":"publisher","first-page":"92","DOI":"10.1016\/j.infsof.2016.04.015","volume":"76","author":"V Garousi","year":"2016","unstructured":"Garousi V, M\u00e4ntyl\u00e4 MV (2016) When and what to automate in software testing? a multi-vocal literature review. Inf Softw Technol 76:92\u2013117","journal-title":"Inf Softw Technol"},{"key":"10937_CR34","unstructured":"GitHub (n.d.) GitHub Acceptable Use Policies - GitHub Docs \u2014 docs.github.com. https:\/\/docs.github.com\/en\/site-policy\/acceptable-use-policies\/github-acceptable-use-policies. [Accessed 28 Feb 2025]"},{"key":"10937_CR35","doi-asserted-by":"crossref","unstructured":"Gladisch C, Heinz T, Heinzemann C, Oehlerking J, von Vietinghoff A, Pfitzer T (2019) Experience paper: Search-based testing in automated driving control applications. In: 2019 34th IEEE\/ACM international conference on automated software engineering (ASE). IEEE, pp 26\u201337","DOI":"10.1109\/ASE.2019.00013"},{"key":"10937_CR36","doi-asserted-by":"crossref","unstructured":"Gong Y, Liu G, Xue Y, Li R, Meng L (2023) A survey on dataset quality in machine learning. Information and Software Technology 107268","DOI":"10.1016\/j.infsof.2023.107268"},{"key":"10937_CR37","unstructured":"Goyal P (2017) Accurate, large minibatch sg d: training imagenet in 1 hour. arXiv:1706.02677"},{"issue":"1","key":"10937_CR38","doi-asserted-by":"publisher","first-page":"59","DOI":"10.1177\/1525822X05279903","volume":"18","author":"G Guest","year":"2006","unstructured":"Guest G, Bunce A, Johnson L (2006) How many interviews are enough? an experiment with data saturation and variability. Field Methods 18(1):59\u201382","journal-title":"Field Methods"},{"issue":"4","key":"10937_CR39","doi-asserted-by":"publisher","first-page":"531","DOI":"10.1038\/s42256-025-01020-y","volume":"7","author":"B Gyevnar","year":"2025","unstructured":"Gyevnar B, Kasirzadeh A (2025) Ai safety for everyone. Nat Mach Intell 7(4):531\u2013542","journal-title":"Nat Mach Intell"},{"key":"10937_CR40","doi-asserted-by":"crossref","unstructured":"Hacker P, Engel A, Mauer M (2023) Regulating chatgpt and other large generative ai models. In: Proceedings of the 2023 ACM conference on fairness, accountability, and transparency, pp 1112\u20131123","DOI":"10.1145\/3593013.3594067"},{"key":"10937_CR41","unstructured":"Inan H, Upasani K, Chi J, Rungta R, Iyer K, Mao Y, Tontchev M, Hu Q, Fuller B, Testuggine D et al (2023) Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv:2312.06674"},{"key":"10937_CR42","doi-asserted-by":"crossref","unstructured":"Hern\u00e1ndez-Orallo J, Mart\u00ednez-Plumed F, Avin S, Whittlestone J, \u00d3\u00a0h\u00c9igeartaigh S (2020) Ai paradigms and ai safety: mapping artefacts and techniques to safety issues. In: ECAI 2020. IOS Press, pp 2521\u20132528","DOI":"10.3233\/FAIA200386"},{"issue":"3","key":"10937_CR43","doi-asserted-by":"publisher","first-page":"422","DOI":"10.1109\/TSE.2012.30","volume":"39","author":"R Hoda","year":"2012","unstructured":"Hoda R, Noble J, Marshall S (2012) Self-organizing roles on agile software development teams. IEEE Trans Softw Eng 39(3):422\u2013444","journal-title":"IEEE Trans Softw Eng"},{"key":"10937_CR44","doi-asserted-by":"crossref","unstructured":"Hou X, Zhao Y, Wang S, Wang H (2025) Model context protocol (mcp): Landscape, security threats, and future research directions. arXiv:2503.23278","DOI":"10.1145\/3796519"},{"key":"10937_CR45","doi-asserted-by":"crossref","unstructured":"Hsueh P-Y, Melville P, Sindhwani V (2009) Data quality from crowdsourcing: a study of annotation selection criteria. In: NAACL HLT 2009 workshop on active learning for natural language processing, pp 27\u201335","DOI":"10.3115\/1564131.1564137"},{"key":"10937_CR46","unstructured":"Hugging Face (n.d.) Hugging face hub documentation. https:\/\/huggingface.co\/docs\/hub\/index. Accessed: 13 Mar 2025"},{"key":"10937_CR47","unstructured":"HuggingFace (n.d.) Safetensors \u2014 huggingface.co. https:\/\/huggingface.co\/docs\/safetensors\/en\/index [Accessed 05 Feb 2025]"},{"key":"10937_CR48","unstructured":"IEC (2010) IEC 61508-1:2010 \u2014 webstore.iec.ch. https:\/\/webstore.iec.ch\/en\/publication\/5515. [Accessed 06 Mar 2026]"},{"key":"10937_CR49","unstructured":"ISO. Information security, cybersecurity and privacy protection \u2014 evaluation criteria for it security (iso\/iec 15408-5). https:\/\/www.iso.org\/standard\/72917.html, 2022. [Accessed 12-03-2025]"},{"key":"10937_CR50","unstructured":"ISO (2023) Information technology \u2014 artificial intelligence \u2014 management system (iso 42001). https:\/\/www.iso.org\/standard\/81230.html. [Accessed 25 Feb 2025]"},{"key":"10937_CR51","doi-asserted-by":"crossref","unstructured":"J\u00e4\u00e4skel\u00e4inen P, Sanchez C, Holzapfel A (2025) Anticipatory technology ethics reflection by eliciting creative ai imaginaries through fictional research abstracts. In: Proceedings of the 2025 ACM conference on fairness, accountability, and transparency, pp 125\u2013136","DOI":"10.1145\/3715275.3732011"},{"key":"10937_CR52","doi-asserted-by":"crossref","unstructured":"Jajal P, Jiang W, Tewari A, Kocinare E, Woo J, Sarraf A, Lu Y-H, Thiruvathukal GK, Davis JC (2024) Interoperability in deep learning: a user survey and failure analysis of onnx model converters. In: Proceedings of the 33rd ACM SIGSOFT international symposium on software testing and analysis (ISSTA), pp 1466\u20131478","DOI":"10.1145\/3650212.3680374"},{"key":"10937_CR53","doi-asserted-by":"crossref","unstructured":"Davis JC, Jajal P, Jiang W, Schorlemmer TR, Synovic N, Thiruvathukal GK (2023) Reusing deep learning models: challenges and directions in software engineering. In: 2023 IEEE John Vincent Atanasoff international symposium on modern computing (JVA). IEEE, pp 17\u201330","DOI":"10.1109\/JVA60410.2023.00015"},{"key":"10937_CR54","doi-asserted-by":"crossref","unstructured":"Ji Z, Tiezheng Yu, Yan X, Lee N, Ishii E, Fung P (2023) Towards mitigating llm hallucination via self reflection. In: Findings of the association for computational linguistics EMNLP 2023, pp 1827\u20131843","DOI":"10.18653\/v1\/2023.findings-emnlp.123"},{"key":"10937_CR55","doi-asserted-by":"crossref","unstructured":"Jiang W, Synovic N, Sethi R, Indarapu A, Hyatt M, Schorlemmer TR, Thiruvathukal GK, Davis JC (2022) An empirical study of artifacts and security risks in the pre-trained model supply chain. In: Proceedings of the 2022 ACM workshop on software supply chain offensive research and ecosystem defenses, pp 105\u2013114","DOI":"10.1145\/3560835.3564547"},{"key":"10937_CR56","doi-asserted-by":"crossref","unstructured":"Jiang W, Synovic N, Hyatt M, Taylor R, Schorlemmer R, Sethi Y-H, Lu GK, Thiruvathukal JC, Davis (2023) An empirical study of pre-trained model reuse in the hugging face deep learning model registry. In: IEEE\/ACM 45th international conference on software engineering. IEEE","DOI":"10.1109\/ICSE48619.2023.00206"},{"key":"10937_CR57","doi-asserted-by":"crossref","unstructured":"Jiang W, Banna V, Vivek N, Goel A, Synovic N, Thiruvathukal GK, Davis JC (2024a) Challenges and practices of deep learning model reengineering: a case study on computer vision. Empirical Software Engineering (EMSE)","DOI":"10.1007\/s10664-024-10521-0"},{"key":"10937_CR58","doi-asserted-by":"crossref","unstructured":"Jiang W, Yasmin J, Jones J, Synovic N, Kuo J, Bielanski N, Tian Y, Thiruvathukal GK, Davis JC (2024b) Peatmoss: a dataset and initial analysis of pre-trained models in open-source software. In: 2024 IEEE\/ACM 21st international conference on mining software repositories (MSR). IEEE, pp 431\u2013443","DOI":"10.1145\/3643991.3644907"},{"key":"10937_CR59","doi-asserted-by":"crossref","unstructured":"Knight JC (2002) Safety critical systems: challenges and directions. In: Proceedings of the 24th international conference on software engineering, pp 547\u2013550","DOI":"10.1109\/ICSE.2002.1007998"},{"key":"10937_CR60","unstructured":"Kelly TP (1999) Arguing safety: a systematic approach to managing safety cases. University of York York, UK"},{"issue":"5","key":"10937_CR61","doi-asserted-by":"publisher","first-page":"81","DOI":"10.1109\/MS.2018.3571224","volume":"35","author":"F Khomh","year":"2018","unstructured":"Khomh F, Adams B, Cheng J, Fokaefs M, Antoniol G (2018) Software engineering for machine-learning applications: the road ahead. IEEE Softw 35(5):81\u201384","journal-title":"IEEE Softw"},{"key":"10937_CR62","unstructured":"Khot T, Trivedi H, Finlayson M, Fu Y, Richardson K, Clark P, Sabharwal A (2022) Decomposed prompting: a modular approach for solving complex tasks. arXiv:2210.02406"},{"issue":"11","key":"10937_CR63","doi-asserted-by":"publisher","first-page":"1024","DOI":"10.1109\/TSE.2017.2754374","volume":"44","author":"M Kim","year":"2017","unstructured":"Kim M, Zimmermann T, DeLine R, Begel A (2017) Data scientists in software teams: state of the art and challenges. IEEE Trans Softw Eng 44(11):1024\u20131038","journal-title":"IEEE Trans Softw Eng"},{"key":"10937_CR64","doi-asserted-by":"crossref","unstructured":"Lazar S, Nelson A (2023) Ai safety on whose terms?","DOI":"10.1126\/science.adi8982"},{"key":"10937_CR65","unstructured":"Lee H, Park C, Abel D, Jin M (2025) A black swan hypothesis: the role of human irrationality in AI safety. In: The thirteenth international conference on learning representations"},{"key":"10937_CR66","unstructured":"Becker LA (2000) Effect size (es)"},{"key":"10937_CR67","doi-asserted-by":"crossref","unstructured":"Salewski L, Alaniz S, Rio-Torto I, Schulz E, Akata Z (2023) In-context impersonation reveals large language models\u2019 strengths and biases. In: Thirty-seventh conference on neural information processing systems. https:\/\/openreview.net\/forum?id=CbsJ53LdKc","DOI":"10.52202\/075280-3152"},{"issue":"4","key":"10937_CR68","doi-asserted-by":"publisher","first-page":"237","DOI":"10.1016\/S0925-7535(03)00047-X","volume":"42","author":"N Leveson","year":"2004","unstructured":"Leveson N (2004) A new accident model for engineering safer systems. Saf Sci 42(4):237\u2013270","journal-title":"Saf Sci"},{"key":"10937_CR69","doi-asserted-by":"publisher","first-page":"569","DOI":"10.1109\/TSE.1983.235116","volume":"5","author":"NG Leveson","year":"1983","unstructured":"Leveson NG, Harvey PR (1983) Analyzing software safety. IEEE Trans Softw Eng (TSE) 5:569\u2013579","journal-title":"IEEE Trans Softw Eng (TSE)"},{"key":"10937_CR70","doi-asserted-by":"crossref","unstructured":"Li C, Hay M, Rastogi V, Miklau G, McGregor A (2010) Optimizing linear counting queries under differential privacy. In: Proceedings of the twenty-ninth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems, pp 123\u2013134","DOI":"10.1145\/1807085.1807104"},{"key":"10937_CR71","unstructured":"Lin C-Y(2004) Rouge: a package for automatic evaluation of summaries. In: Text summarization branches out, pp 74\u201381"},{"key":"10937_CR72","unstructured":"Liu Y, Yao Y, Ton J-F, Zhang X, Guo R, Cheng H, Klochkov Y, Taufiq MF, Li H (2023) Trustworthy llms: a survey and guideline for evaluating large language models\u2019 alignment. arXiv:2308.05374"},{"key":"10937_CR73","doi-asserted-by":"crossref","unstructured":"Lutz RR (2000) Software engineering for safety: a roadmap. In: Proceedings of the conference on the future of software engineering, ICSE \u201900, New York, NY, USA. Association for Computing Machinery, pp 213\u2013226. ISBN 1581132530","DOI":"10.1145\/336512.336556"},{"issue":"2","key":"10937_CR74","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3487043","volume":"31","author":"S Mart\u00ednez-Fern\u00e1ndez","year":"2022","unstructured":"Mart\u00ednez-Fern\u00e1ndez S, Bogner J, Franch X, Oriol M, Siebert J, Trendowicz A, Vollmer AM, Wagner S (2022) Software engineering for ai-based systems: a survey. ACM Trans Softw Eng Methodol (TOSEM) 31(2):1\u201359","journal-title":"ACM Trans Softw Eng Methodol (TOSEM)"},{"issue":"3","key":"10937_CR75","doi-asserted-by":"publisher","first-page":"276","DOI":"10.11613\/BM.2012.031","volume":"22","author":"ML McHugh","year":"2012","unstructured":"McHugh ML (2012) Interrater reliability: the kappa statistic. Biochemia Medica 22(3):276\u2013282","journal-title":"Biochemia Medica"},{"key":"10937_CR76","doi-asserted-by":"crossref","unstructured":"McGovern A, Bostrom A, McGraw M, Chase RJ, Gagne DJ, Ebert-Uphoff I, Musgrave KD, Schumacher A (2024) Identifying and categorizing bias in ai\/ml for earth sciences. Bull Am Meteor Soc 105(3):E567\u2013E583","DOI":"10.1175\/BAMS-D-23-0196.1"},{"key":"10937_CR77","first-page":"15458","volume":"35","author":"S McGregor","year":"2021","unstructured":"McGregor S (2021) Preventing repeated real world ai failures by cataloging incidents: the ai incident database. Proc AAAI Conf Artif Intell 35:15458\u201315463","journal-title":"Proc AAAI Conf Artif Intell"},{"key":"10937_CR78","doi-asserted-by":"crossref","unstructured":"Menze M, Geiger A (2015) Object scene flow for autonomous vehicles. In: IEEE conference on computer vision and pattern recognition, pp 3061\u20133070","DOI":"10.1109\/CVPR.2015.7298925"},{"key":"10937_CR79","unstructured":"MIT (2024) MIT AI Risk Repository. https:\/\/airisk.mit.edu\/. [Accessed 04 Apr 2026]"},{"key":"10937_CR80","doi-asserted-by":"crossref","unstructured":"Mitchell M, Wu S, Zaldivar A, Barnes P, Vasserman L, Hutchinson B, Spitzer E, Raji ID, Gebru T (2019) Model cards for model reporting. In: Proceedings of the conference on fairness, accountability, and transparency, pp 220\u2013229","DOI":"10.1145\/3287560.3287596"},{"key":"10937_CR81","doi-asserted-by":"crossref","unstructured":"Montes D, Peerapatanapokin P, Schultz J, Guo C, Jiang W, Davis JC (2022) Discrepancies among pre-trained deep neural networks: a new threat to model zoo reliability. In: Proceedings of the 30th ACM joint European software engineering conference and symposium on the foundations of software engineering (ESEC\/FSE-IVR), pp 1605\u20131609","DOI":"10.1145\/3540250.3560881"},{"key":"10937_CR82","doi-asserted-by":"crossref","unstructured":"Nahar N, Zhou S, Lewis G, K\u00e4stner C (2022) Collaboration challenges in building ml-enabled systems: communication, documentation, engineering, and process. In: 44th International conference on software engineering","DOI":"10.1145\/3510003.3510209"},{"key":"10937_CR83","doi-asserted-by":"crossref","unstructured":"Naveed H, Grundy J, Arora C, Khalajzadeh H, Haggag O (2024) Towards runtime monitoring for responsible machine learning using model-driven engineering. In: Proceedings of the ACM\/IEEE 27th international conference on model driven engineering languages and systems, pp 195\u2013202","DOI":"10.1145\/3640310.3674092"},{"key":"10937_CR84","doi-asserted-by":"publisher","first-page":"900","DOI":"10.1109\/ACCESS.2022.3232947","volume":"11","author":"N Ngadiman","year":"2022","unstructured":"Ngadiman N, Sulaiman S, Idris N, Samingan MR, Mohamed H (2022) Checklist approach for the development of educational applications by novice software developers. IEEE Access 11:900\u2013918","journal-title":"IEEE Access"},{"key":"10937_CR85","unstructured":"NIST (2023) AI Risk Management Framework \u2014 nist.gov. https:\/\/www.nist.gov\/itl\/ai-risk-management-framework. [Accessed 24 Feb 2025]"},{"key":"10937_CR86","doi-asserted-by":"crossref","unstructured":"Paleyes A, Urma R-G, Lawrence ND (2022) Challenges in deploying machine learning: a survey of case studies. ACM computing surveys (CSUR) 55(6):1\u201329","DOI":"10.1145\/3533378"},{"issue":"3","key":"10937_CR87","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3635715","volume":"33","author":"A Pant","year":"2024","unstructured":"Pant A, Hoda R, Spiegler SV, Tantithamthavorn C, Turhan B (2024) Ethics in the age of ai: an analysis of ai practitioners\u2019 awareness and challenges. ACM Trans Softw Eng Methodol 33(3):1\u201335","journal-title":"ACM Trans Softw Eng Methodol"},{"key":"10937_CR88","unstructured":"Patterson D, Gonzalez J, Le Q, Liang C, Munguia L-M, Rothchild D, So D, Texier M, Dean J (2021) Carbon emissions and large neural network training. arXiv:2104.10350"},{"key":"10937_CR89","unstructured":"Pouget H, Zuhdi R (2024) AI and Product Safety Standards Under the EU AI Act. Carnegie Endowment for International Peace"},{"key":"10937_CR90","unstructured":"Qi X, Huang Y, Zeng Y, Debenedetti E, Geiping J, He L, Huang K, Madhushani U, Sehwag V, Shi W et\u00a0al (2024) Ai risk management should incorporate both safety and security. arXiv:2405.19524"},{"issue":"7","key":"10937_CR91","first-page":"1","volume":"56","author":"L Qinghua","year":"2024","unstructured":"Qinghua L, Zhu L, Xiwei X, Whittle J, Zowghi D, Jacquet A (2024) Responsible ai pattern catalogue: a collection of best practices for ai governance and engineering. ACM Comput Surv 56(7):1\u201335","journal-title":"ACM Comput Surv"},{"key":"10937_CR92","doi-asserted-by":"crossref","unstructured":"Radanliev P, Santos O, Brandon-Jones A (2024) Capability hardware enhanced instructions and artificial intelligence bill of materials in trustworthy artificial intelligence systems: analyzing cybersecurity threats, exploits, and vulnerabilities in new software bills of materials with artificial intelligence. The Journal of Defense Modeling and Simulation 15485129241267919","DOI":"10.1177\/15485129241267919"},{"key":"10937_CR93","doi-asserted-by":"crossref","unstructured":"Rey D, Neuh\u00e4user M (2011) Wilcoxon-signed-rank test. International Encyclopedia of Statistical Science 1658\u20131659","DOI":"10.1007\/978-3-642-04898-2_616"},{"issue":"CSCW2","key":"10937_CR94","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3479499","volume":"5","author":"RY Wong","year":"2021","unstructured":"Wong RY(2021) Tactics of soft resistance in user experience professionals\u2019 values work. Proc ACM Human-Comput Interact 5(CSCW2):1\u201328","journal-title":"Proc ACM Human-Comput Interact"},{"issue":"CSCW1","key":"10937_CR95","first-page":"1","volume":"7","author":"RY Wong","year":"2023","unstructured":"Wong RY, Madaio MA, Merrill N (2023) Seeing like a toolkit: how toolkits envision the work of ai ethics. Proc ACM Human-Comput Interact 7(CSCW1):1\u201327","journal-title":"Proc ACM Human-Comput Interact"},{"key":"10937_CR96","unstructured":"Yampolskiy RV, Spellchecker MS (2016) Artificial intelligence safety and cybersecurity: A timeline of ai failures. arXiv:1610.07997"},{"issue":"5","key":"10937_CR97","doi-asserted-by":"publisher","first-page":"94","DOI":"10.1145\/129875.129885","volume":"35","author":"RE Anderson","year":"1992","unstructured":"Anderson RE (1992) Acm code of ethics and professional conduct. Commun ACM (CACM) 35(5):94\u201399","journal-title":"Commun ACM (CACM)"},{"key":"10937_CR98","doi-asserted-by":"crossref","unstructured":"Roselli D, Matthews J, Talagala N (2019) Managing bias in ai. In: Companion proceedings of the 2019 world wide web conference, pp 539\u2013544","DOI":"10.1145\/3308560.3317590"},{"issue":"12","key":"10937_CR99","doi-asserted-by":"publisher","first-page":"3354","DOI":"10.14778\/3415478.3415556","volume":"13","author":"L Rupprecht","year":"2020","unstructured":"Rupprecht L, Davis JC, Arnold C, Gur Y, Bhagwat D (2020) Improving reproducibility of data science pipelines through transparent provenance capture. VLDB Endowment 13(12):3354\u20133368","journal-title":"VLDB Endowment"},{"key":"10937_CR100","doi-asserted-by":"crossref","unstructured":"Salhab W, Ameyed D, Jaafar F, Mcheick H (2024) A systematic literature review on ai safety:Identifying trends, challenges and future directions. IEEE Access","DOI":"10.1109\/ACCESS.2024.3440647"},{"key":"10937_CR101","doi-asserted-by":"crossref","unstructured":"Ali SJ, Christin A, Smart A, Katila R (2023) Walking the walk of ai ethics: organizational challenges and the individualization of risk among ethics entrepreneurs. In: Proceedings of the 2023 ACM conference on fairness, accountability, and transparency, pp 217\u2013226","DOI":"10.1145\/3593013.3593990"},{"key":"10937_CR102","unstructured":"Schmidt J-H, Bartsch SC, Adam M, Benlian A (2023) Accountability incongruence and its effects on ai developers\u2019 job satisfaction. In: ECIS"},{"key":"10937_CR103","unstructured":"Sculley D, Holt G, Golovin D, Davydov E, Phillips T, Ebner D, Chaudhary V, Young M, Crespo J-F, Dennison D (2015) Hidden technical debt in machine learning systems. Advances in Neural Information Processing Systems 28"},{"issue":"4","key":"10937_CR104","doi-asserted-by":"publisher","first-page":"557","DOI":"10.1109\/32.799955","volume":"25","author":"CB Seaman","year":"1999","unstructured":"Seaman CB (1999) Qualitative methods in empirical studies of software engineering. IEEE Trans Software Eng 25(4):557\u2013572","journal-title":"IEEE Trans Software Eng"},{"key":"10937_CR105","doi-asserted-by":"crossref","unstructured":"Shaikh O, Zhang H, Held W, Bernstein M, Yang D (2022) On second thought, let\u2019s not think step by step! bias and toxicity in zero-shot reasoning. arXiv:2212.08061","DOI":"10.18653\/v1\/2023.acl-long.244"},{"key":"10937_CR106","doi-asserted-by":"crossref","unstructured":"Sheng E, Chang K-W, Natarajan P, Peng N (2021) Societal biases in language generation: progress and challenges. arXiv:2105.04054","DOI":"10.18653\/v1\/2021.acl-long.330"},{"issue":"4","key":"10937_CR107","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3419764","volume":"10","author":"B Shneiderman","year":"2020","unstructured":"Shneiderman B (2020) Bridging the gap between ethics and practice: guidelines for reliable, safe, and trustworthy human-centered ai systems. ACM Trans Interact Intell Syst (TiiS) 10(4):1\u201331","journal-title":"ACM Trans Interact Intell Syst (TiiS)"},{"issue":"2","key":"10937_CR108","doi-asserted-by":"publisher","first-page":"74","DOI":"10.4018\/JDM.2020040105","volume":"31","author":"K Siau","year":"2020","unstructured":"Siau K, Wang W (2020) Artificial intelligence (ai) ethics: ethics of ai and ethical ai. J Database Manag (JDM) 31(2):74\u201387","journal-title":"J Database Manag (JDM)"},{"key":"10937_CR109","doi-asserted-by":"crossref","unstructured":"Sillito J, Kutomi E (2020) Failures and fixes: a study of software system incident response. In: 2020 IEEE international conference on software maintenance and evolution (ICSME). IEEE, pp 185\u2013195","DOI":"10.1109\/ICSME46990.2020.00027"},{"key":"10937_CR110","doi-asserted-by":"crossref","unstructured":"Singer L, Figueira\u00a0Filho F, Storey M-A (2014) Software engineering at the speed of light: how developers stay current using twitter. In: Proceedings of the 36th international conference on software engineering, pp 211\u2013221","DOI":"10.1145\/2568225.2568305"},{"key":"10937_CR111","unstructured":"Hove SE, Anda B (2005) Experiences from conducting semi-structured interviews in empirical software engineering research. In: 11th IEEE international software metrics symposium (METRICS\u201905). IEEE, pp 10\u2013pp"},{"key":"10937_CR112","doi-asserted-by":"crossref","unstructured":"Lee SU, Perera H, Xia B, Liu Y, Lu Q, Zhu L, Salvado O, Whittle J (2024) Qb4aira: a question bank for responsible ai risk assessment. IEEE Software","DOI":"10.1109\/MS.2024.3512577"},{"key":"10937_CR113","doi-asserted-by":"crossref","unstructured":"Tan X, Gao K, Zhou M, Zhang L (2022) An exploratory study of deep learning supply chain. International Conference on Software Engineering","DOI":"10.1145\/3510003.3510199"},{"key":"10937_CR114","unstructured":"Zhuo TY, Huang Y, Chen C, Xing Z (2023) Red teaming chatgpt via jailbreaking: bias, robustness, reliability and toxicity. arXiv:2301.12867"},{"issue":"6","key":"10937_CR115","doi-asserted-by":"publisher","first-page":"565","DOI":"10.1109\/TSE.2014.2387172","volume":"41","author":"C Treude","year":"2014","unstructured":"Treude C, Robillard MP, Dagenais B (2014) Extracting development tasks to navigate software documentation. IEEE Trans Softw Eng 41(6):565\u2013581","journal-title":"IEEE Trans Softw Eng"},{"key":"10937_CR116","unstructured":"UK Government (2023) The Bletchley declaration by countries attending the AI safety summit, 1-2 november 2023. https:\/\/www.gov.uk\/government\/publications\/ai-safety-summit-2023-the-bletchley-declaration. Accessed: 23 Apr 2026"},{"key":"10937_CR117","unstructured":"Underwriters Laboratories (2026) UL 3115: outline of investigation for safety of AI-based products. https:\/\/store.accuristech.com\/standards\/ul-subject-3115. [Accessed 04 Apr 2026]"},{"key":"10937_CR118","doi-asserted-by":"crossref","unstructured":"Wang S, Levy B (2024) Beancounter: a low-toxicity, large-scale, and open dataset of business-oriented text. In: The Thirty-eight conference on neural information processing systems datasets and benchmarks track","DOI":"10.52202\/079017-2909"},{"key":"10937_CR119","unstructured":"Wang J, HU X, Hou W, Chen H, Zheng R, Wang Y, Yang L, Ye W, Huang H, Geng X, Jiao B, Zhang Y, Xie X (2023) On the robustness of chatGPT: an adversarial and out-of-distribution perspective. In: ICLR 2023 workshop on trustworthy and reliable large-scale machine learning models"},{"key":"10937_CR120","doi-asserted-by":"crossref","unstructured":"Wei A, Haghtalab N, Steinhardt J (2024) Jailbroken: how does llm safety training fail? Advances in Neural Information Processing Systems 36","DOI":"10.52202\/075280-3508"},{"key":"10937_CR121","unstructured":"Weidinger L, Mellor J, Rauh M, Griffin C, Uesato J, Huang P-S, Cheng M, Glaese M, Balle B, Kasirzadeh A et\u00a0al (2021) Ethical and social risks of harm from language models. arXiv:2112.04359"},{"key":"10937_CR122","doi-asserted-by":"crossref","unstructured":"Xia B, Bi T, Xing Z, Lu Q, Zhu L (2023) An empirical study on software bill of materials: where we stand and the road ahead. In: 2023 IEEE\/ACM 45th international conference on software engineering (ICSE). IEEE, pp 2630\u20132642","DOI":"10.1109\/ICSE48619.2023.00219"},{"key":"10937_CR123","doi-asserted-by":"publisher","first-page":"11809","DOI":"10.52202\/075280-0517","volume":"36","author":"S Yao","year":"2023","unstructured":"Yao S, Dian Yu, Zhao J, Shafran I, Griffiths T, Cao Y, Narasimhan K (2023) Tree of thoughts: deliberate problem solving with large language models. Adv Neural Inf Process Syst 36:11809\u201311822","journal-title":"Adv Neural Inf Process Syst"},{"key":"10937_CR124","doi-asserted-by":"crossref","unstructured":"Jung Y, Song J, Yang JY, Kim J-H, Kim S-Y Yang, E (2024) A simple remedy for dataset bias via self-influence: A mislabeled sample perspective. In: The Thirty-eighth annual conference on neural information processing systems. https:\/\/openreview.net\/forum?id=ZVrrPNqHFw","DOI":"10.52202\/079017-1383"},{"key":"10937_CR125","unstructured":"Zhang Y, Li Y, Cui L, Cai D, Liu L, Fu T, Huang X, Zhao E, Zhang Y, Chen Y et\u00a0al (2023) Siren\u2019s song in the ai ocean: a survey on hallucination in large language models. arXiv:2309.01219"},{"key":"10937_CR126","doi-asserted-by":"publisher","first-page":"46595","DOI":"10.52202\/075280-2020","volume":"36","author":"L Zheng","year":"2023","unstructured":"Zheng L, Chiang W-L, Sheng Y, Zhuang S, Zhanghao W, Zhuang Y, Lin Z, Li Z, Li D, Xing E et al (2023) Judging llm-as-a-judge with mt-bench and chatbot arena. Adv Neural Inf Process Syst 36:46595\u201346623","journal-title":"Adv Neural Inf Process Syst"},{"key":"10937_CR127","doi-asserted-by":"crossref","unstructured":"Zhou Z, Rahman\u00a0Siddiquee MM, Tajbakhsh N, Liang J (2018) Unet++: a nested u-net architecture for medical image segmentation. In: Deep learning in medical image analysis and multimodal learning for clinical decision support: 4th international workshop, DLMIA 2018, and 8th International Workshop, ML-CDS 2018, Held in Conjunction with MICCAI 2018. Springer, pp 3\u201311","DOI":"10.1007\/978-3-030-00889-5_1"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-026-10937-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10664-026-10937-w","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-026-10937-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,7]],"date-time":"2026-08-07T06:15:58Z","timestamp":1786083358000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10664-026-10937-w"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,8,7]]},"references-count":127,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2027,2]]}},"alternative-id":["10937"],"URL":"https:\/\/doi.org\/10.1007\/s10664-026-10937-w","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,7]]},"assertion":[{"value":"12 October 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"20 July 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"7 August 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Not applicable.","order":1,"name":"Ethics","label":"Conflict of Interest","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"This project is approved by the Ethics Committee at the University of Melbourne under ID 30642.","order":2,"name":"Ethics","label":"Ethical Approval","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"All participants provided informed consent before taking part in this research.","order":3,"name":"Ethics","label":"Informed Consent","group":{"name":"EthicsHeading","label":"Declarations"}}],"article-number":"20"}}