{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,14]],"date-time":"2026-02-14T05:56:06Z","timestamp":1771048566616,"version":"3.50.1"},"reference-count":76,"publisher":"Springer Science and Business Media LLC","issue":"5","license":[{"start":{"date-parts":[[2020,4,23]],"date-time":"2020-04-23T00:00:00Z","timestamp":1587600000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2020,4,23]],"date-time":"2020-04-23T00:00:00Z","timestamp":1587600000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Mach Learn"],"published-print":{"date-parts":[[2020,5]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Fatal accidents are a major issue hindering the wide acceptance of safety-critical systems that employ machine learning and deep learning models, such as automated driving vehicles. In order to use machine learning in a safety-critical system, it is necessary to demonstrate the safety and security of the system through engineering processes. However, thus far, no such widely accepted engineering concepts or frameworks have been established for these systems. The key to using a machine learning model in a deductively engineered system is decomposing the data-driven training of machine learning models into requirement, design, and verification, particularly for machine learning models used in safety-critical systems. Simultaneously, open problems and relevant technical fields are not organized in a manner that enables researchers to select a theme and work on it. In this study, we identify, classify, and explore the open problems in engineering (safety-critical) machine learning systems\u2014that is, in terms of requirement, design, and verification of machine learning models and systems\u2014as well as discuss related works and research directions, using automated driving vehicles as an example. Our results show that machine learning models are characterized by a lack of requirements specification, lack of design specification, lack of interpretability, and lack of robustness. We also perform a gap analysis on a conventional system quality standard SQuaRE with the characteristics of machine learning models to study quality models for machine learning systems. We find that a lack of requirements specification and lack of robustness have the greatest impact on conventional quality models.<\/jats:p>","DOI":"10.1007\/s10994-020-05872-w","type":"journal-article","created":{"date-parts":[[2020,4,23]],"date-time":"2020-04-23T22:03:06Z","timestamp":1587679386000},"page":"1103-1126","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":67,"title":["Engineering problems in machine learning systems"],"prefix":"10.1007","volume":"109","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0731-8057","authenticated-orcid":false,"given":"Hiroshi","family":"Kuwajima","sequence":"first","affiliation":[]},{"given":"Hirotoshi","family":"Yasuoka","sequence":"additional","affiliation":[]},{"given":"Toshihiro","family":"Nakae","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2020,4,23]]},"reference":[{"key":"5872_CR65","unstructured":"Administration NHTS of\u00a0Transportation UD, (2017). Automated driving systems: A vision for safety 2.0."},{"key":"5872_CR1","doi-asserted-by":"publisher","unstructured":"Ali, G. G. M. N., & Chan, E. (2011). Co-operative data access in multiple road side units (rsus)-based vehicular ad hoc networks (VANETS). In Proceedings of the Australasian telecommunication networks and applications conference, ATNAC 2011, Melbourne, Australia, November 9\u201311, 2011 (pp. 1\u20136). IEEE. https:\/\/doi.org\/10.1109\/ATNAC.2011.6096651.","DOI":"10.1109\/ATNAC.2011.6096651"},{"key":"5872_CR2","unstructured":"Amodei, D., Olah, C., Steinhardt, J., Christiano, P.F., Schulman, J., & Man\u00e9, D. (2016). Concrete problems in AI safety. CoRR arXiv:1606.06565."},{"key":"5872_CR3","doi-asserted-by":"publisher","DOI":"10.1007\/978-94-015-9934-4","volume-title":"Introduction to mathematical logic and type theory: To truth through proof","author":"PB Andrews","year":"2002","unstructured":"Andrews, P. B. (2002). Introduction to mathematical logic and type theory: To truth through proof (2nd ed.). Norwell, MA: Kluwer Academic Publishers.","edition":"2"},{"key":"5872_CR4","doi-asserted-by":"publisher","first-page":"40","DOI":"10.1214\/09-SS054","volume":"4","author":"S Arlot","year":"2010","unstructured":"Arlot, S., Celisse, A., et al. (2010). A survey of cross-validation procedures for model selection. Statistics Surveys, 4, 40\u201379.","journal-title":"Statistics Surveys"},{"key":"5872_CR5","first-page":"2613","volume-title":"Advances in neural information processing systems 29","author":"O Bastani","year":"2016","unstructured":"Bastani, O., Ioannou, Y., Lampropoulos, L., Vytiniotis, D., Nori, A., & Criminisi, A. (2016). Measuring neural net robustness with constraints. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, & R. Garnett (Eds.), Advances in neural information processing systems 29 (pp. 2613\u20132621). Red Hook: Curran Associates Inc."},{"issue":"1","key":"5872_CR6","doi-asserted-by":"publisher","first-page":"45","DOI":"10.1023\/A:1007465907571","volume":"29","author":"S Ben-David","year":"1997","unstructured":"Ben-David, S., Kushilevitz, E., & Mansour, Y. (1997). Online learning versus offline learning. Machine Learning, 29(1), 45\u201363. https:\/\/doi.org\/10.1023\/A:1007465907571.","journal-title":"Machine Learning"},{"key":"5872_CR7","unstructured":"Bickel, S., Br\u00fcckner, M., & Scheffer, T. (2009). Discriminative learning under covariate shift. Journal of Machine Learning Research., 10, 2137\u20132155. https:\/\/dl.acm.org\/citation.cfm?id=1755858."},{"key":"5872_CR8","doi-asserted-by":"publisher","first-page":"63","DOI":"10.1007\/978-3-319-44781-0_8","volume-title":"Artificial Neural Networks and Machine Learning \u2013 ICANN 2016","author":"Alexander Binder","year":"2016","unstructured":"Binder, A., Montavon, G., Lapuschkin, S., M\u00fcller, K., & Samek, W. (2016). Layer-wise relevance propagation for neural networks with local renormalization layers. In A. E. P Villa, P. Masulli, A. J. P. Rivero (Eds.), Proceedings artificial neural networks and machine learning-ICANN 2016- 25th international conference on artificial neural networks, Barcelona, Spain, September 6\u20139, 2016, Part II, Lecture Notes in Computer Science. (vol. 9887, pp. 63\u201371). Springer. https:\/\/doi.org\/10.1007\/978-3-319-44781-0_8."},{"key":"5872_CR9","unstructured":"Bird, S., Crankshaw, D., Gibson, G., Gonzalez, J., Lakshmiratan, A., Li L.E., Re, C., & Sen, S. (2017). In Proceedings of the workshop on ml systems at nips 2017."},{"key":"5872_CR10","doi-asserted-by":"publisher","DOI":"10.3390\/app8020303","author":"R Borraz","year":"2018","unstructured":"Borraz, R., Navarro, P. J., Fern\u00e1ndez, C., & Alcover, P. M. (2018). Cloud incubator car: A reliable platform for autonomous driving. Applied Sciences. https:\/\/doi.org\/10.3390\/app8020303.","journal-title":"Applied Sciences"},{"key":"5872_CR11","doi-asserted-by":"publisher","first-page":"126","DOI":"10.1007\/978-3-030-01090-4_8","volume-title":"Automated Technology for Verification and Analysis","author":"Chih-Hong Cheng","year":"2018","unstructured":"Cheng, C., Huang, C., & Yasuoka, H. (2018) Quantitative projection coverage for testing ML-enabled autonomous systems. In S. K. Lahiri, C. Wang (Eds.) Proceedings of automated technology for verification and analysis - 16th international symposium, ATVA 2018, Los Angeles, CA, USA, October 7-10, 2018, Lecture Notes in Computer Science . (vol. 11138, pp. 126\u2013142). Springer. https:\/\/doi.org\/10.1007\/978-3-030-01090-4_8."},{"key":"5872_CR12","doi-asserted-by":"publisher","unstructured":"Cheng, C., N\u00fchrenberg, G., Huang, C., Ruess, H., & Yasuoka, H. (2018). Towards dependability metrics for neural networks. In Proceedings of 16th ACM\/IEEE international conference on formal methods and models for system design, MEMOCODE 2018, Beijing, China, October 15\u201318, 2018, (pp. 43\u201346). IEEE, https:\/\/doi.org\/10.1109\/MEMCOD.2018.8556962.","DOI":"10.1109\/MEMCOD.2018.8556962"},{"key":"5872_CR13","unstructured":"Cheng, C., N\u00fchrenberg, G., & Yasuoka, H. (2018). Runtime monitoring neuron activation patterns. CoRR abs\/1809.06573, arXiv:1809.06573."},{"key":"5872_CR14","doi-asserted-by":"crossref","unstructured":"Cheng, C. H., N\u00fchrenberg, G., & Ruess, H. (2017). Maximum resilience of artificial neural networks. In ATVA. Springer, Cham.","DOI":"10.1007\/978-3-319-68167-2_18"},{"key":"5872_CR15","doi-asserted-by":"publisher","unstructured":"Colwell, I., Phan, B., Saleem, S., Salay, R., & Czarnecki, K. (2018). An automated vehicle safety concept based on runtime restriction of the operational design domain (pp. 1910\u20131917). https:\/\/doi.org\/10.1109\/IVS.2018.8500530.","DOI":"10.1109\/IVS.2018.8500530"},{"key":"5872_CR16","doi-asserted-by":"publisher","unstructured":"Czarnecki, K. (2018). On-road safety of automated driving system (ads)\u2014Taxonomy and safety analysis methods. https:\/\/doi.org\/10.13140\/RG.2.2.28313.93287.","DOI":"10.13140\/RG.2.2.28313.93287"},{"key":"5872_CR17","unstructured":"Dantzig, G. B. (1987). Origins of the simplex method. Tech. rep.: stanford univ ca systems optimization lab."},{"key":"5872_CR18","unstructured":"De\u00a0Moura, L., & Bj\u00f8rner, N. (2008). Z3: An efficient smt solver. In Proceedings of the theory and practice of software, 14th international conference on tools and algorithms for the construction and analysis of systems, (pp. 337\u2013340). Springer, Berlin. TACAS\u201908\/ETAPS\u201908, http:\/\/dl.acm.org\/citation.cfm?id=1792734.1792766."},{"issue":"10","key":"5872_CR19","doi-asserted-by":"publisher","first-page":"78","DOI":"10.1145\/2347736.2347755","volume":"55","author":"P Domingos","year":"2012","unstructured":"Domingos, P. (2012). A few useful things to know about machine learning. Commun ACM, 55(10), 78\u201387. https:\/\/doi.org\/10.1145\/2347736.2347755.","journal-title":"Commun ACM"},{"key":"5872_CR20","doi-asserted-by":"publisher","first-page":"382","DOI":"10.1007\/978-3-642-40787-1_27","volume-title":"Runtime Verification","author":"Alexandre Donz\u00e9","year":"2013","unstructured":"Donz\u00e9 A (2013) On signal temporal logic. In Proceedings of the international conference on runtime verification, (pp. 382\u2013383). Springer, Berlin."},{"key":"5872_CR21","unstructured":"Dreossi, T., Ghosh, S., Sangiovanni-Vincentelli, A. L., & Seshia, S.A., (2017). Systematic testing of convolutional neural networks for autonomous driving. In Proceedings of the ICML workshop on reliable machine learning in the wild."},{"key":"5872_CR22","doi-asserted-by":"publisher","DOI":"10.1007\/s10817-018-09509-5","author":"T Dreossi","year":"2019","unstructured":"Dreossi, T., Donz\u00e9, A., & Seshia, A. S. (2019). Compositional falsification of cyber-physical systems with machine learning components. Journal of Automated Reasoning. https:\/\/doi.org\/10.1007\/s10817-018-09509-5.","journal-title":"Journal of Automated Reasoning"},{"key":"5872_CR23","doi-asserted-by":"publisher","unstructured":"Elkahky, A. M., Song, Y., & He, X. (2015). A multi-view deep learning approach for cross domain user modeling in recommendation systems. In Proceedings of the 24th international conference on world wide web, international world wide web conferences steering committee, Republic and Canton of Geneva, Switzerland, WWW \u201915, (pp. 278\u2013288). https:\/\/doi.org\/10.1145\/2736277.2741667.","DOI":"10.1145\/2736277.2741667"},{"key":"5872_CR24","doi-asserted-by":"publisher","first-page":"279","DOI":"10.1007\/978-3-319-67383-7_21","volume-title":"Software process improvement and capability determination","author":"F Falcini","year":"2017","unstructured":"Falcini, F., & Lami, G. (2017). Deep learning in automotive: Challenges and opportunities. In A. Mas, A. Mesquida, R. V. O\u2019Connor, T. Rout, & A. Dorling (Eds.), Software process improvement and capability determination (pp. 279\u2013288). Cham: Springer."},{"key":"5872_CR25","doi-asserted-by":"crossref","unstructured":"Garcia, F. A., & S\u00e1nchez, A. (2006) Formal verification of safety and liveness properties for logic controllers. a tool comparison.In Proceedings of the 3rd international conference on electrical and electronics engineering. (pp. 1\u20133).","DOI":"10.1109\/ICEEE.2006.251867"},{"key":"5872_CR26","unstructured":"Government U, of\u00a0Transportation UD (2018) 2018 federal guide to self-driving cars and automated driving: preparing for the future of transportation\u2014Automated vehicles 3.0 safety issues and role of the government in autonomous regulation."},{"key":"5872_CR27","doi-asserted-by":"publisher","unstructured":"Graves, A., Jaitly, N., & Mohamed, A. (2013). Hybrid speech recognition with deep bidirectional LSTM. In Proceedings of the IEEE workshop on automatic speech recognition and understanding, Olomouc, Czech Republic, December 8\u201312, 2013, (pp. 273\u2013278). IEEE https:\/\/doi.org\/10.1109\/ASRU.2013.6707742.","DOI":"10.1109\/ASRU.2013.6707742"},{"key":"5872_CR28","unstructured":"Gr\u00fcn, F., Rupprecht, C., Navab, N., & Federico, T. (2016). A taxonomy and library for visualizing learned features in convolutional neural networks. In Proceeding of the ICML workshop on visualization for deep learning."},{"key":"5872_CR29","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. (pp. 770\u2013778).","DOI":"10.1109\/CVPR.2016.90"},{"key":"5872_CR30","doi-asserted-by":"crossref","unstructured":"Huang, X., Kwiatkowska, M. Z., Wang, S., & Wu, M. (2017). Safety verification of deep neural networks. In Proceedings of the CAV.","DOI":"10.1007\/978-3-319-63387-9_1"},{"key":"5872_CR31","unstructured":"INCOSE (2015). Systems engineering handbook: A guide for system life cycle processes and activities, version 4.0 edn. Hoboken: Wiley."},{"key":"5872_CR32","doi-asserted-by":"publisher","first-page":"14","DOI":"10.1007\/978-3-319-99229-7_2","volume-title":"Developments in Language Theory","author":"Fuyuki Ishikawa","year":"2018","unstructured":"Ishikawa, F., & Matsuno, Y. (2018). Continuous argument engineering: Tackling uncertainty in machine learning based systems. In B. Gallina, A. Skavhaug, E. Schoitsch, & F. Bitsch (Eds.), Computer safety, reliability, and security\u2013SAFECOMP 2018 workshops, ASSURE, DECSoS, SASSUR, STRIVE, and WAISE, V\u00e4ster\u00e5s, Sweden, September 18, 2018, (vol. 11094, pp. 14\u201321). Proceedings, Springer, Lecture Notes in Computer Science. https:\/\/doi.org\/10.1007\/978-3-319-99229-7_2."},{"key":"5872_CR33","unstructured":"ISO 26262\u20131:2018, (2018). Road vehicles\u2013functional safety-part 1: Vocabulary. International organization for standardization: Tech. rep."},{"key":"5872_CR34","unstructured":"ISO IEC 25000:2014. (2014). Systems and software engineering\u2014Systems and software Quality Requirements and Evaluation (SQuaRE) - Guide to SQuaRE. International organization for standardization, international electrotechnical commission: Standard."},{"key":"5872_CR35","unstructured":"ISO IEC 9126:2001. (2001). Software engineering\u2014product quality. International organization for standardization, international electrotechnical commission: Tech. rep."},{"key":"5872_CR36","volume-title":"An introduction to statistical learning: With applications in R","author":"G James","year":"2014","unstructured":"James, G., Witten, D., Hastie, T., & Tibshirani, R. (2014). An introduction to statistical learning: With applications in R. Berlin: Springer."},{"key":"5872_CR37","doi-asserted-by":"publisher","unstructured":"Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014). Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the 22Nd ACM international conference on multimedia, (pp. 675\u2013678). ACM, New York, NY, USA, MM \u201914, https:\/\/doi.org\/10.1145\/2647868.2654889.","DOI":"10.1145\/2647868.2654889"},{"key":"5872_CR38","doi-asserted-by":"publisher","first-page":"182","DOI":"10.1016\/j.tra.2016.09.010","volume":"94","author":"N Kalra","year":"2016","unstructured":"Kalra, N., & Paddock, S. M. (2016). Driving to safety: How many miles of driving would it take to demonstrate autonomous vehicle reliability? Transportation Research Part A: Policy and Practice, 94, 182\u2013193. https:\/\/doi.org\/10.1016\/j.tra.2016.09.010.","journal-title":"Transportation Research Part A: Policy and Practice"},{"key":"5872_CR39","unstructured":"Katz, G., Barrett, C., Dill, D. L., Julian, K., & Kochenderfer, M. J. (2017). Towards proving the adversarial robustness of deep neural networks. In L. Bulwahn, M. Kamali, & S. Linker (Eds.), Proceedings of the first workshop on formal verification of autonomous vehicles (FVAV \u201917), electronic proceedings in theoretical computer science, (vol. 257, pp. 19\u201326). Turin, Italy. http:\/\/eptcs.web.cse.unsw.edu.au\/paper.cgi?FVAV2017.3."},{"key":"5872_CR40","volume-title":"Reluplex: An efficient smt solver for verifying deep neural networks","author":"G Katz","year":"2017","unstructured":"Katz, G., Barrett, C. W., Dill, D. L., Julian, K., & Kochenderfer, M. J. (2017). In CAV. Reluplex: An efficient smt solver for verifying deep neural networks. Cham: Springer."},{"key":"5872_CR41","unstructured":"Kelly, T., & Weaver, R. (2004). The goal structuring notation\u2014a safety argument notation. In Proceedings of dependable systems and networks 2004 workshop on assurance cases."},{"key":"5872_CR42","doi-asserted-by":"publisher","first-page":"15","DOI":"10.4271\/2016-01-0128","volume":"4","author":"P Koopman","year":"2016","unstructured":"Koopman, P., & Wagner, M. (2016). Challenges in autonomous vehicle testing and validation. SAE International Journal of Transportation Safety, 4, 15\u201324. https:\/\/doi.org\/10.4271\/2016-01-0128.","journal-title":"SAE International Journal of Transportation Safety"},{"key":"5872_CR43","first-page":"1097","volume-title":"Advances in Neural Information Processing Systems 25","author":"A Krizhevsky","year":"2012","unstructured":"Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In F. Pereira, C. J. C. Burges, L. Bottou, & K. Q. Weinberger (Eds.), Advances in Neural Information Processing Systems 25 (pp. 1097\u20131105). Red Hook: Curran Associates, Inc."},{"issue":"2","key":"5872_CR44","doi-asserted-by":"publisher","first-page":"273","DOI":"10.1007\/s13748-019-00179-x","volume":"8","author":"H Kuwajima","year":"2019","unstructured":"Kuwajima, H., Tanaka, M., & Okutomi, M. (2019). Improving transparency of deep neural inference process. Progress in Artificial Intelligence, 8(2), 273\u2013285. https:\/\/doi.org\/10.1007\/s13748-019-00179-x.","journal-title":"Progress in Artificial Intelligence"},{"key":"5872_CR45","volume-title":"Hardware design verification: simulation and formal method-based approaches","author":"WK Lam","year":"2008","unstructured":"Lam, W. K. (2008). Hardware design verification: simulation and formal method-based approaches (1st ed.). Upper Saddle River, NJ, USA: Prentice Hall PTR.","edition":"1"},{"key":"5872_CR46","unstructured":"Lemmer, K., & Mazzega, J. (2017). Pegasus: Effectively ensuring automated driving. In VDA technical congress."},{"key":"5872_CR47","unstructured":"Li, L. E., Dragan, A., Niebles, J. C., & Savarese, S. (2017). nips workshop on machine learning for intelligent transportation systems."},{"key":"5872_CR48","unstructured":"Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. A. (2013). Playing atari with deep reinforcement learning. CoRR abs\/1312.5602"},{"key":"5872_CR49","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1016\/j.patcog.2016.11.008","volume":"65","author":"G Montavon","year":"2017","unstructured":"Montavon, G., Lapuschkin, S., Binder, A., Samek, W., & M\u00fcller, K. (2017). Explaining nonlinear classification decisions with deep taylor decomposition. Pattern Recognition, 65, 211\u2013222. https:\/\/doi.org\/10.1016\/j.patcog.2016.11.008.","journal-title":"Pattern Recognition"},{"key":"5872_CR50","volume-title":"Machine learning: A probabilistic perspective","author":"KP Murphy","year":"2013","unstructured":"Murphy, K. P. (2013). Machine learning: A probabilistic perspective. Cambridge, MA: MIT Press."},{"key":"5872_CR51","unstructured":"Nair, V., & Hinton, G.E. (2010). Rectified linear units improve restricted boltzmann machines. In J. F\u00fcrnkranz, & T. Joachims (Eds.) Proceedings of the 27th international conference on machine learning (ICML-10), June 21\u201324, 2010, (pp. 807\u2013814). Haifa: Omnipress. http:\/\/www.icml2010.org\/papers\/432.pdf."},{"key":"5872_CR52","unstructured":"Ng, A. (2015). Deep learning. In nVIDIA GPU technology conference (GTC)."},{"key":"5872_CR53","doi-asserted-by":"crossref","unstructured":"Poggenhans, F., Pauls, J., Janosovits, J., Orf, S., Naumann, M., Kuhnt, F., & Mayr, M. (2018). Lanelet2: A high-definition map framework for the future of automated driving. In Proceedings of the ITSC, (pp. 1672\u20131679). IEEE.","DOI":"10.1109\/ITSC.2018.8569929"},{"key":"5872_CR54","doi-asserted-by":"crossref","unstructured":"Pulina, L., & Tacchella, A. (2010). An abstraction-refinement approach to verification of artificial neural networks. In Proceeding of the CAV.","DOI":"10.1007\/978-3-642-14295-6_24"},{"issue":"2","key":"5872_CR55","doi-asserted-by":"publisher","first-page":"117","DOI":"10.3233\/AIC-2012-0525","volume":"25","author":"L Pulina","year":"2012","unstructured":"Pulina, L., & Tacchella, A. (2012). Challenging smt solvers to verify neural networks. AI Communications, 25(2), 117\u2013135.","journal-title":"AI Communications"},{"key":"5872_CR56","unstructured":"Report of traffic collision involving an autonomous vehicle (ol 316). (2018). https:\/\/www.dmv.ca.gov\/portal\/dmv\/detail\/vr\/autonomous\/autonomousveh_ol316+."},{"key":"5872_CR57","doi-asserted-by":"crossref","unstructured":"Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). \u201cWhy should I trust you?\u201d: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, (pp. 1135\u20131144). San Francisco, CA, USA, August 13\u201317, 2016.","DOI":"10.1145\/2939672.2939778"},{"key":"5872_CR58","unstructured":"Salay, R., Queiroz, R., & Czarnecki, K. (2017). An analysis of ISO 26262: Using machine learning safely in automotive software. CoRR abs\/1709.02435, arXiv:1709.02435."},{"key":"5872_CR59","unstructured":"Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J. F., & Dennison, D. (2015). Hidden technical debt in machine learning systems. In Proceedings of the 28th international conference on neural information processing systems, (Vol. 2, pp. 2503\u20132511). Cambridge, MA: MIT Press. NIPS\u201915."},{"key":"5872_CR60","unstructured":"Shrikumar, A., Greenside, P., Shcherbina, A., Kundaje, A. (2016). Not just a black box: Learning important features through propagating activation differences. CoRR abs\/1605.01713"},{"key":"5872_CR61","unstructured":"Simonyan, K., Vedaldi, A., & Zisserman, A. (2014). Deep inside convolutional networks: Visualising image classification models and saliency maps. In Proceedings of the international conference on learning representations."},{"issue":"2","key":"5872_CR62","doi-asserted-by":"crossref","first-page":"111","DOI":"10.1111\/j.2517-6161.1974.tb00994.x","volume":"36","author":"M Stone","year":"1974","unstructured":"Stone, M. (1974). Cross-validatory choice and assessment of statistical predictions. Journal of the Royal Statistical Society: Series B (Methodological), 36(2), 111\u2013133.","journal-title":"Journal of the Royal Statistical Society: Series B (Methodological)"},{"key":"5872_CR63","first-page":"3104","volume-title":"Advances in Neural Information Processing Systems 27","author":"I Sutskever","year":"2014","unstructured":"Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence to sequence learning with neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, & K. Q. Weinberger (Eds.), Advances in Neural Information Processing Systems 27 (pp. 3104\u20133112). Red Hook: Curran Associates, Inc."},{"key":"5872_CR64","unstructured":"Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I. J., & Fergus, R. (2013). Intriguing Properties of Neural Networks. CoRR abs\/1312.6199."},{"key":"5872_CR66","unstructured":"Tsymbal, A. (2004). The problem of concept drift: Definitions and related work. Tech. rep."},{"issue":"6","key":"5872_CR67","doi-asserted-by":"publisher","first-page":"463","DOI":"10.1038\/s41573-019-0024-5","volume":"18","author":"J Vamathevan","year":"2019","unstructured":"Vamathevan, J., Clark, D., Czodrowski, P., Dunham, I., Ferran, E., Lee, G., et al. (2019). Applications of machine learning in drug discovery and development. Nature Reviews Drug Discovery, 18(6), 463\u2013477. https:\/\/doi.org\/10.1038\/s41573-019-0024-5.","journal-title":"Nature Reviews Drug Discovery"},{"key":"5872_CR68","unstructured":"VDA QMC Working Group 13\/Automotive SIG (2015). Automotive spice process assessment\/reference model version 3.0. Tech. rep. Automotive SPICE."},{"issue":"4","key":"5872_CR69","doi-asserted-by":"publisher","first-page":"964","DOI":"10.1007\/s10618-015-0448-4","volume":"30","author":"GI Webb","year":"2016","unstructured":"Webb, G. I., Hyde, R., Cao, H., Nguyen, H. L., & Petitjean, F. (2016). Characterizing concept drift. Data Mining and Knowledge Discovery, 30(4), 964\u2013994. https:\/\/doi.org\/10.1007\/s10618-015-0448-4.","journal-title":"Data Mining and Knowledge Discovery"},{"key":"5872_CR70","unstructured":"Wendorff, W. (2017). Quantitative sotif analysis for highly automated driving systems. In Safetronic."},{"key":"5872_CR71","unstructured":"Yu, F., Xian, W., Chen, Y., Liu, F., Liao, M., Madhavan, V., & Darrell, T. (2018). BDD100K: A diverse driving video database with scalable annotation tooling. CoRR abs\/1805.04687, arXiv:1805.04687."},{"key":"5872_CR72","doi-asserted-by":"publisher","first-page":"818","DOI":"10.1007\/978-3-319-10590-1_53","volume-title":"Computer Vision-ECCV 2014","author":"MD Zeiler","year":"2014","unstructured":"Zeiler, M. D., & Fergus, R. (2014). Visualizing and understanding convolutional networks. In D. Fleet, T. Pajdla, B. Schiele, & T. Tuytelaars (Eds.), Computer Vision-ECCV 2014 (pp. 818\u2013833). Cham: Springer."},{"key":"5872_CR73","doi-asserted-by":"publisher","unstructured":"Zendel, O., Murschitz, M., Humenberger, M., & Herzner, W. (2015). CV-HAZOP: Introducing test data validation for computer vision. In Proceedings of the international conference on computer vision, ICCV 2015, Santiago, Chile, December 7\u201313, 2015, IEEE Computer Society. (pp. 2066\u20132074). https:\/\/doi.org\/10.1109\/ICCV.2015.239.","DOI":"10.1109\/ICCV.2015.239"},{"key":"5872_CR74","unstructured":"Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016). Understanding deep learning requires rethinking generalization. CoRR abs\/1611.03530, arXiv:1611.03530."},{"key":"5872_CR75","unstructured":"Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., & Torralba, A. (2015). Object detectors emerge in deep scene cnns. In Proceedings of the international conference on learning representations."},{"key":"5872_CR76","doi-asserted-by":"crossref","unstructured":"Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., & Torralba, A. (2016). Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition.","DOI":"10.1109\/CVPR.2016.319"}],"container-title":["Machine Learning"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-020-05872-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10994-020-05872-w\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10994-020-05872-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,8,4]],"date-time":"2024-08-04T16:07:51Z","timestamp":1722787671000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10994-020-05872-w"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,4,23]]},"references-count":76,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2020,5]]}},"alternative-id":["5872"],"URL":"https:\/\/doi.org\/10.1007\/s10994-020-05872-w","relation":{},"ISSN":["0885-6125","1573-0565"],"issn-type":[{"value":"0885-6125","type":"print"},{"value":"1573-0565","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,4,23]]},"assertion":[{"value":"1 March 2019","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"1 July 2019","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"13 February 2020","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 April 2020","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Compliance with ethical standards"}},{"value":"The authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}