{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,12]],"date-time":"2026-05-12T18:35:12Z","timestamp":1778610912588,"version":"3.51.4"},"reference-count":80,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2020,11,19]],"date-time":"2020-11-19T00:00:00Z","timestamp":1605744000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"German Federal Ministry of Education and Research","award":["01IS18061D"],"award-info":[{"award-number":["01IS18061D"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["MAKE"],"abstract":"<jats:p>Machine Learning (ML) is increasingly applied for the control of safety-critical Cyber-Physical Systems (CPS) in application areas that cannot easily be mastered with traditional control approaches, such as autonomous driving. As a consequence, the safety of machine learning became a focus area for research in recent years. Despite very considerable advances in selected areas related to machine learning safety, shortcomings were identified on holistic approaches that take an end-to-end view on the risks associated to the engineering of ML-based control systems and their certification. Applying a classic technique of safety engineering, our paper provides a comprehensive and methodological analysis of the safety hazards that could be introduced along the ML lifecycle, and could compromise the safe operation of ML-based CPS. Identified hazards are illustrated and explained using a real-world application scenario\u2014an autonomous shop-floor transportation vehicle. The comprehensive analysis presented in this paper is intended as a basis for future holistic approaches for safety engineering of ML-based CPS in safety-critical applications, and aims to support the focus on research onto safety hazards that are not yet adequately addressed.<\/jats:p>","DOI":"10.3390\/make2040031","type":"journal-article","created":{"date-parts":[[2020,11,19]],"date-time":"2020-11-19T06:23:52Z","timestamp":1605767032000},"page":"579-602","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":66,"title":["Challenges of Machine Learning Applied to Safety-Critical Cyber-Physical Systems"],"prefix":"10.3390","volume":"2","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5213-3775","authenticated-orcid":false,"given":"Ana","family":"Pereira","sequence":"first","affiliation":[{"name":"School of Engineering, Hochschule f\u00fcr Technik und Wirtschaft Berlin, Wilhelminenhofstra\u00dfe 75A, 12459 Berlin, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0461-2058","authenticated-orcid":false,"given":"Carsten","family":"Thomas","sequence":"additional","affiliation":[{"name":"School of Engineering, Hochschule f\u00fcr Technik und Wirtschaft Berlin, Wilhelminenhofstra\u00dfe 75A, 12459 Berlin, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2020,11,19]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1031","DOI":"10.1007\/s10817-018-09509-5","article-title":"Compositional falsification of cyber-physical systems with machine learning components","volume":"63","author":"Dreossi","year":"2019","journal-title":"J. Autom. Reason."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Shi, J., Wan, J., Yan, H., and Suo, H. (2011, January 9\u201311). A survey of cyber-physical systems. Proceedings of the 2011 International Conference on Wireless Communications and Signal Processing (WCSP), Nanjing, China.","DOI":"10.1109\/WCSP.2011.6096958"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"1750014","DOI":"10.1142\/S2424862217500142","article-title":"Cyber physical system (CPS)-based industry 4.0: A survey","volume":"2","author":"Lu","year":"2017","journal-title":"J. Ind. Integr. Manag."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1007\/s11465-018-0499-5","article-title":"Smart manufacturing systems for Industry 4.0: Conceptual framework, scenarios, and future perspectives","volume":"13","author":"Zheng","year":"2018","journal-title":"Front. Mech. Eng."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"4","DOI":"10.20965\/ijat.2017.p0004","article-title":"\u201cIndustrie 4.0\u201d and smart manufacturing-a review of research issues and application examples","volume":"11","author":"Thoben","year":"2017","journal-title":"Int. J. Autom. Technol."},{"key":"ref_6","unstructured":"Heng, S. (2020, November 14). Industry 4.0: Upgrading of Germany\u2019s Industrial Capabilities on the Horizon. Available online: https:\/\/ssrn.com\/abstract=2656608."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"26754","DOI":"10.1109\/ACCESS.2017.2773127","article-title":"Working together: A review on safe human-robot collaboration in industrial environments","volume":"5","author":"Becerra","year":"2017","journal-title":"IEEE Access"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1016\/j.rcim.2018.07.006","article-title":"The effectiveness of virtual environments in developing collaborative strategies between industrial robots and humans","volume":"55","author":"Oyekan","year":"2019","journal-title":"Robot. Comput. Integr. Manuf."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"35","DOI":"10.1007\/s43154-020-00006-5","article-title":"Trends in Smart Manufacturing: Role of Humans and Industrial Robots in Smart Factories","volume":"1","author":"Evjemo","year":"2020","journal-title":"Curr. Robot. Rep."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Gharib, M., Lollini, P., Botta, M., Amparore, E., Donatelli, S., and Bondavalli, A. (2018, January 25\u201328). On the Safety of Automotive Systems Incorporating Machine Learning Based Components: A Position Paper. Proceedings of the 48th Annual IEEE\/IFIP International Conference on Dependable Systems and Networks Workshops (DSN-W), Luxembourg.","DOI":"10.1109\/DSN-W.2018.00074"},{"key":"ref_11","unstructured":"Faria, J.M. (2018, January 6\u20138). Machine learning safety: An overview. Proceedings of the 26th Safety-Critical Systems Symposium, York, UK."},{"key":"ref_12","unstructured":"Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Man\u00e9, D. (2016). Concrete problems in AI safety. arXiv."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Serban, A.C. (2019, January 25\u201329). Designing Safety Critical Software Systems to Manage Inherent Uncertainty. Proceedings of the 2019 IEEE International Conference on Software Architecture Companion (ICSA-C), Hamburg, Germany.","DOI":"10.1109\/ICSA-C.2019.00051"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Hains, G.J., Jakobsson, A., and Khmelevsky, Y. (2019). Formal methods and software engineering for DL. Security, safety and productivity for DL systems development. arXiv.","DOI":"10.1109\/SYSCON.2018.8369576"},{"key":"ref_15","unstructured":"Bishop, C.M. (2006). Pattern Recognition and Machine Learning, Springer."},{"key":"ref_16","unstructured":"Ashmore, R., Calinescu, R., and Paterson, C. (2019). Assuring the machine learning lifecycle: Desiderata, methods, and challenges. arXiv."},{"key":"ref_17","first-page":"463","article-title":"Sofware engneering challenges for machine learning applications: A literature review","volume":"13","author":"Kumeno","year":"2020","journal-title":"Intell. Decis. Technol."},{"key":"ref_18","unstructured":"Jenn, E., Albore, A., Mamalet, F., Flandin, G., Gabreau, C., Delseny, H., Gauffriau, A., Bonnin, H., Alecu, L., and Pirard, J. (2020, January 29\u201331). Identifying Challenges to the Certification of Machine Learning for Safety Critical Systems. Proceedings of the 10th European Congress on Embedded Real Time Systems (ERTS), Toulouse, France."},{"key":"ref_19","unstructured":"(2020, October 22). G-34, Artificial Intelligence in Aviation. Available online: https:\/\/www.sae.org\/works\/committeeHome.do?comtID=TEAG34."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Varshney, K.R. (February, January 31). Engineering safety in machine learning. Proceedings of the 2016 Information Theory and Applications Workshop (ITA), La Jolla, CA, USA.","DOI":"10.1109\/ITA.2016.7888195"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"246","DOI":"10.1089\/big.2016.0051","article-title":"On the safety of machine learning: Cyber-physical systems, decision sciences, and data products","volume":"5","author":"Varshney","year":"2017","journal-title":"Big Data"},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Kl\u00e4s, M., and Vollmer, A.M. (2018). Uncertainty in machine learning applications: A practice-driven classification of uncertainty. International Conference on Computer Safety, Reliability, and Security, Springer.","DOI":"10.1007\/978-3-319-99229-7_36"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Salay, R., Queiroz, R., and Czarnecki, K. (2017). An analysis of ISO 26262: Using machine learning safely in automotive software. arXiv.","DOI":"10.4271\/2018-01-1075"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Borg, M., Englund, C., Wnuk, K., Duran, B., Levandowski, C., Gao, S., Tan, Y., Kaijser, H., L\u00f6nn, H., and T\u00f6rnqvist, J. (2018). Safely entering the deep: A review of verification and validation for machine learning and a challenge elicitation in the automotive industry. arXiv.","DOI":"10.2991\/jase.d.190131.001"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Henriksson, J., Borg, M., and Englund, C. (2018, January 28). Automotive Safety and Machine Learning: Initial Results from a Study on How to Adapt the ISO 26262 Safety Standard. Proceedings of the 2018 IEEE\/ACM 1st International Workshop on Software Engineering for AI in Autonomous Systems (SEFAIAS), Gothenburg, Sweden.","DOI":"10.1145\/3194085.3194090"},{"key":"ref_26","unstructured":"Koopman, P., and Fratrik, F. (2019, January 27). How many operational design domains, objects, and events?. Proceedings of the SafeAI@ AAAI, Honolulu, HI, USA."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Schwalbe, G., and Schels, M. (2020, January 29\u201331). A Survey on Methods for the Safety Assurance of Machine Learning Based Systems. Proceedings of the 10th European Congress on Embedded Real Time Software and Systems (ERTS 2020), Toulouse, France.","DOI":"10.20378\/irb-47275"},{"key":"ref_28","unstructured":"Ma, L., Juefei-Xu, F., Xue, M., Hu, Q., Chen, S., Li, B., Liu, Y., Zhao, J., Yin, J., and See, S. (2018). Secure deep learning engineering: A software quality assurance perspective. arXiv."},{"key":"ref_29","unstructured":"Aravantinos, V., and Diehl, F. (2018). Traceability of deep neural networks. arXiv."},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1109\/TDSC.2004.2","article-title":"Basic concepts and taxonomy of dependable and secure computing","volume":"1","author":"Avizienis","year":"2004","journal-title":"IEEE Trans. Dependable Secur. Comput."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Smith, D., and Simpson, K. (2004). Functional Safety, Routledge.","DOI":"10.4324\/9780080477923"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"798","DOI":"10.1016\/j.ress.2007.03.031","article-title":"Principles of engineering safety: Risk and uncertainty reduction","volume":"93","author":"Hansson","year":"2008","journal-title":"Reliab. Eng. Syst. Saf."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"M\u00f6ller, N. (2012). The concepts of risk and safety. Handbook of Risk Theory: Epistemology, Decision Theory, Ethics, and Social Implications of Risk, Springer.","DOI":"10.1007\/978-94-007-1433-5_3"},{"key":"ref_34","unstructured":"Leveson, N.G. (2016). Engineering a Safer World: Systems Thinking Applied to Safety, The MIT Press."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Kelly, T. (1998). Arguing Safety\u2014A Systematic Approach to Safety Case Management. [Ph.D. Thesis, Department of Computer Science, University of York].","DOI":"10.1007\/3-540-48249-0_2"},{"key":"ref_36","unstructured":"G\u00e9ron, A. (2019). Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems, O\u2019Reilly Media."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"76","DOI":"10.1145\/507338.507355","article-title":"Data mining: Practical machine learning tools and techniques with Java implementations","volume":"31","author":"Witten","year":"2002","journal-title":"ACM Sigmod Rec."},{"key":"ref_38","first-page":"1174","article-title":"Machine learning algorithms: A review","volume":"7","author":"Dey","year":"2016","journal-title":"Int. J. Comput. Sci. Inf. Technol."},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"James, G., Witten, D., Hastie, T., and Tibshirani, R. (2013). An Introduction to Statistical Learning, Springer.","DOI":"10.1007\/978-1-4614-7138-7"},{"key":"ref_40","unstructured":"Patterson, J., and Gibson, A. (2017). Deep Learning: A Practitioner\u2019s Approach, O\u2019Reilly Media, Inc."},{"key":"ref_41","unstructured":"International Standards Organization (2018). ISO 26262:2018-12: Road Vehicles\u2014Functional Safety, International Standards Organization."},{"key":"ref_42","unstructured":"The International Electrotechnical Commission (2010). IEC 61508:2010: Functional Safety of Electrical\/Electronic\/ Programmable Electronic Safety-Related Systems, The International Electrotechnical Commission."},{"key":"ref_43","unstructured":"SAE International (2010). ARP4754A:2010: Guidelines for Development of Civil Aircraft and Systems, SAE International."},{"key":"ref_44","unstructured":"K\u00fcpper, D., Lorenz, M., Kuhlmann, K., Bouffault, O., Heng, L.Y., Van Wyck, J., K\u00f6cher, S., and Schlagete, J. (2018). AI in the Factory of the Future. The Ghost in the Machine, The Boston Consulting Group."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"117","DOI":"10.1016\/j.promfg.2018.04.003","article-title":"Rethinking Human-Machine Learning in Industry 4.0: How Does the Paradigm Shift Treat the Role of Human Learning?","volume":"23","author":"Ansari","year":"2018","journal-title":"Procedia Manuf."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"60","DOI":"10.1109\/MM.2015.133","article-title":"An Open Approach to Autonomous Vehicles","volume":"35","author":"Kato","year":"2015","journal-title":"IEEE Micro"},{"key":"ref_47","first-page":"39","article-title":"Accelerating the Machine Learning Lifecycle with MLflow","volume":"41","author":"Zaharia","year":"2018","journal-title":"IEEE Data Eng. Bull."},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Vogelsang, A., and Borg, M. (2019, January 23\u201327). Requirements Engineering for Machine Learning: Perspectives from Data Scientists. Proceedings of the 2019 IEEE 27th International Requirements Engineering Conference Workshops (REW), Jeju Island, Korea.","DOI":"10.1109\/REW.2019.00050"},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Ishikawa, F., and Yoshioka, N. (2019, January 28). How Do Engineers Perceive Difficulties in Engineering of Machine-Learning Systems?\u2014Questionnaire Survey. Proceedings of the 2019 IEEE\/ACM Joint 7th International Workshop on Conducting Empirical Studies in Industry (CESI) and 6th International Workshop on Software Engineering Research and Industrial Practice (SER IP), Montreal, QC, Canada.","DOI":"10.1109\/CESSER-IP.2019.00009"},{"key":"ref_50","unstructured":"Davis, A.M. (1993). Software Requirements: Objects, Functions, and States, Prentice-Hall, Inc."},{"key":"ref_51","unstructured":"Ellims, M., Barbier, E., and Botham, J. (2019, January 21\u201325). Safety Analysis Process for Machine Learning in Automated Vehicle Software. Proceedings of the 26th ITS World Congress, Singapore."},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Gharib, M., and Bondavalli, A. (2019, January 17\u201320). On the Evaluation Measures for Machine Learning Algorithms for Safety-Critical Systems. Proceedings of the 2019 15th European Dependable Computing Conference (EDCC), Naples, Italy.","DOI":"10.1109\/EDCC.2019.00035"},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Reschka, A. (2016). Safety concept for autonomous vehicles. Autonomous Driving, Springer.","DOI":"10.1007\/978-3-662-48847-8_23"},{"key":"ref_54","unstructured":"Roh, Y., Heo, G., and Whang, S.E. (2019). A Survey on Data Collection for Machine Learning: A Big Data\u2014AI Integration Perspective. IEEE Trans. Knowl. Data Eng."},{"key":"ref_55","unstructured":"Xiao, H., Rasul, K., and Vollgraf, R. (2017). Fashion-mnist: A novel image dataset for benchmarking machine learning algorithms. arXiv."},{"key":"ref_56","unstructured":"Chen, X., Fang, H., Lin, T.Y., Vedantam, R., Gupta, S., Doll\u00e1r, P., and Zitnick, C.L. (2015). Microsoft coco captions: Data collection and evaluation server. arXiv."},{"key":"ref_57","doi-asserted-by":"crossref","unstructured":"Patki, N., Wedge, R., and Veeramachaneni, K. (2016, January 17\u201319). The Synthetic Data Vault. Proceedings of the 2016 IEEE International Conference on Data Science and Advanced Analytics (DSAA), Montreal, QC, Canada.","DOI":"10.1109\/DSAA.2016.49"},{"key":"ref_58","doi-asserted-by":"crossref","first-page":"375","DOI":"10.1080\/713827180","article-title":"Data preparation for data mining","volume":"17","author":"Zhang","year":"2003","journal-title":"Appl. Artif. Intell."},{"key":"ref_59","doi-asserted-by":"crossref","unstructured":"Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll\u00e1r, P., and Zitnick, C.L. (2014). Microsoft Coco: Common Objects in Context, Springer. European Conference on Computer Vision.","DOI":"10.1007\/978-3-319-10602-1_48"},{"key":"ref_60","doi-asserted-by":"crossref","first-page":"429","DOI":"10.3233\/IDA-2002-6504","article-title":"The class imbalance problem: A systematic study","volume":"6","author":"Japkowicz","year":"2002","journal-title":"Intell. Data Anal."},{"key":"ref_61","doi-asserted-by":"crossref","first-page":"90","DOI":"10.1109\/MITS.2016.2583491","article-title":"Autonomous Vehicle Safety: An Interdisciplinary Challenge","volume":"9","author":"Koopman","year":"2017","journal-title":"IEEE Intell. Transp. Syst. Mag."},{"key":"ref_62","doi-asserted-by":"crossref","unstructured":"Goodfellow, I., McDaniel, P., and Papernot, N. (2018). Making Machine Learning Robust against Adversarial Inputs, Communications of the ACM.","DOI":"10.1145\/3134599"},{"key":"ref_63","unstructured":"Mitchel, M. (2020, November 18). Bias in the Vision and Language of Artificial Intelligence. Available online: https:\/\/web.stanford.edu\/class\/archive\/cs\/cs224n\/cs224n.1194\/slides\/cs224n-2019-lecture19-bias.pdf."},{"key":"ref_64","doi-asserted-by":"crossref","unstructured":"Gu, X., and Easwaran, A. (2019, January 16\u201318). Towards Safe Machine Learning for CPS: Infer Uncertainty from Training Data. Proceedings of the 10th ACM\/IEEE International Conference on Cyber-Physical Systems (ICCPS\u201919), Montreal, QC, Canada.","DOI":"10.1145\/3302509.3311038"},{"key":"ref_65","unstructured":"Goodfellow, I., Bengio, Y., Courville, A., and Bengio, Y. (2016). Deep Learning, MIT Press."},{"key":"ref_66","doi-asserted-by":"crossref","first-page":"42200","DOI":"10.1109\/ACCESS.2020.2976199","article-title":"Explainable machine learning for scientific insights and discoveries","volume":"8","author":"Roscher","year":"2020","journal-title":"IEEE Access"},{"key":"ref_67","doi-asserted-by":"crossref","unstructured":"Miao, H., Li, A., Davis, L.S., and Deshpande, A. (2017, January 19\u201322). Towards Unified Data and Lifecycle Management for Deep Learning. Proceedings of the 2017 IEEE 33rd International Conference on Data Engineering (ICDE), San Diego, CA, USA.","DOI":"10.1109\/ICDE.2017.112"},{"key":"ref_68","unstructured":"(2020, August 24). How to Select Algorithms for Azure Machine Learning. Available online: https:\/\/docs.microsoft.com\/en-us\/azure\/machine-learning\/how-to-select-algorithms."},{"key":"ref_69","unstructured":"Molnar, C. (2020). Interpretable Machine Learning, Lulu.com."},{"key":"ref_70","unstructured":"Marcus, G. (2018). Deep learning: A critical appraisal. arXiv."},{"key":"ref_71","doi-asserted-by":"crossref","first-page":"362","DOI":"10.1002\/rob.21918","article-title":"A survey of deep learning techniques for autonomous driving","volume":"37","author":"Grigorescu","year":"2020","journal-title":"J. Field Robot."},{"key":"ref_72","doi-asserted-by":"crossref","unstructured":"Kim, J., and Canny, J. (2017, January 22\u201329). Interpretable Learning for Self-Driving Cars by Visualizing Causal Attention. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy.","DOI":"10.1109\/ICCV.2017.320"},{"key":"ref_73","unstructured":"Zambaldi, V., Raposo, D., Santoro, A., Bapst, V., Li, Y., Babuschkin, I., Tuyls, K., Reichert, D., Lillicrap, T., and Lockhart, E. (2018). Relational deep reinforcement learning. arXiv."},{"key":"ref_74","doi-asserted-by":"crossref","unstructured":"Brown, A., Tuor, A., Hutchinson, B., and Nichols, N. (2018, January 12). Recurrent neural network attention mechanisms for interpretable system log anomaly detection. Proceedings of the First Workshop on Machine Learning for Computing Systems, Tempe, AZ, USA.","DOI":"10.1145\/3217871.3217872"},{"key":"ref_75","unstructured":"Zhang, J.M., Harman, M., Ma, L., and Liu, Y. (2020). Machine Learning Testing: Survey, Landscapes and Horizons. IEEE Trans. Softw. Eng."},{"key":"ref_76","first-page":"19","article-title":"System validation of highly automated vehicles with a database of relevant traffic scenarios","volume":"1","author":"Zlocki","year":"2017","journal-title":"Situations"},{"key":"ref_77","unstructured":"ISO (2019). PAS 21448-Road Vehicles-Safety of the Intended Functionality, International Organization for Standardization."},{"key":"ref_78","unstructured":"Mariani, R. (2020, November 14). Challenges in AI\/ML for Safety Critical Systems. Available online: http:\/\/www.dfts.org\/_2019\/DFT_2019-Mariani-v3.pdf."},{"key":"ref_79","unstructured":"Sutherland, G., and Hessami, A. (2020, January 11\u201313). Safety Critical Integrity Assurance in Large Datasets. Proceedings of the 28th Safety-Critical Systems Symposium, York, UK."},{"key":"ref_80","doi-asserted-by":"crossref","unstructured":"Huang, L., Joseph, A.D., Nelson, B., Rubinstein, B.I., and Tygar, J.D. (2011, January 17\u201321). Adversarial Machine Learning. Proceedings of the 4th ACM Workshop on Security and Artificial Intelligence (AISec \u201911), Chicago, IL, USA.","DOI":"10.1145\/2046684.2046692"}],"container-title":["Machine Learning and Knowledge Extraction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2504-4990\/2\/4\/31\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T10:34:17Z","timestamp":1760178857000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2504-4990\/2\/4\/31"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,11,19]]},"references-count":80,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2020,12]]}},"alternative-id":["make2040031"],"URL":"https:\/\/doi.org\/10.3390\/make2040031","relation":{},"ISSN":["2504-4990"],"issn-type":[{"value":"2504-4990","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,11,19]]}}}