{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,17]],"date-time":"2026-06-17T15:15:24Z","timestamp":1781709324089,"version":"3.54.5"},"publisher-location":"Cham","reference-count":35,"publisher":"Springer Nature Switzerland","isbn-type":[{"value":"9783031737404","type":"print"},{"value":"9783031737411","type":"electronic"}],"license":[{"start":{"date-parts":[[2024,10,31]],"date-time":"2024-10-31T00:00:00Z","timestamp":1730332800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,10,31]],"date-time":"2024-10-31T00:00:00Z","timestamp":1730332800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Machine learning (ML) has improved dramatically over\u00a0the last decade. ML models have become a fundamental part of intelligent software systems, many of which are safety-critical. Since ML models have complex lifecycles, they require dedicated methods and tools, such as pipeline automation or experiment management. Unfortunately, the current state of the art is <jats:italic>model-centric<\/jats:italic>, disregarding the challenges of engineering systems with multiple ML models\u00a0that need to interact to realize complex functionality. Consider,\u00a0for instance, robotics or autonomous driving systems, where perception architectures can easily incorporate more than 30 ML models. Developing such multi-ML model systems requires architectures\u00a0that can integrate and chain ML components. Maintaining and evolving\u00a0them requires tackling the combinatorial explosion when re-training\u00a0ML components, often exploring different (hyper-)parameters, features, training algorithms, or other ML artifacts. Addressing\u00a0these problems requires <jats:italic>systems-centric<\/jats:italic> methods and tools. In this work, we discuss characteristics of multi-ML-model systems and challenges of engineering them. Inspired by such systems in\u00a0the autonomous driving domain, our focus is on experiment-management tooling, which supports tracking and reasoning about the training process for ML models. Our analysis reveals their concepts, but\u00a0also their limitations when engineering multi-ML-model systems, especially due to their model-centric focus. We discuss possible integration patterns and ML training\u00a0to facilitate the effective and efficient development, maintenance,\u00a0and evolution of multi-ML-model systems. Furthermore, we describe real-world multi-ML-model systems, providing early results from identifying and analyzing open-source systems from GitHub.<\/jats:p>","DOI":"10.1007\/978-3-031-73741-1_26","type":"book-chapter","created":{"date-parts":[[2024,10,30]],"date-time":"2024-10-30T14:54:58Z","timestamp":1730300098000},"page":"434-452","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Towards ML-Integration and Training Patterns for AI-Enabled Systems"],"prefix":"10.1007","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2604-0487","authenticated-orcid":false,"given":"Sven","family":"Peldszus","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-5666-2222","authenticated-orcid":false,"given":"Henriette","family":"Knopp","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-5845-5887","authenticated-orcid":false,"given":"Yorick","family":"Sens","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3870-5167","authenticated-orcid":false,"given":"Thorsten","family":"Berger","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,10,31]]},"reference":[{"key":"26_CR1","unstructured":"FSG Competition Handbook 2024. Tech. rep., Formula Student Germany (2023)"},{"issue":"3","key":"26_CR2","doi-asserted-by":"publisher","first-page":"120","DOI":"10.1109\/MS.2021.3134386","volume":"39","author":"S Apel","year":"2022","unstructured":"Apel, S., K\u00e4stner, C., Kang, E.: Feature interactions on steroids: on the composition of ML models. IEEE Softw. 39(3), 120\u2013124 (2022)","journal-title":"IEEE Softw."},{"key":"26_CR3","doi-asserted-by":"publisher","unstructured":"Arpteg, A., Brinne, B., Crnkovic-Friis, L., Bosch, J.: Software engineering challenges of deep learning. In: Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 50\u201359 (2018). https:\/\/doi.org\/10.1109\/SEAA.2018.00018","DOI":"10.1109\/SEAA.2018.00018"},{"key":"26_CR4","unstructured":"Baidu: Apollo Auto: An Open Autonomous Driving Platform (2024). https:\/\/github.com\/ApolloAuto"},{"key":"26_CR5","doi-asserted-by":"publisher","unstructured":"Bosch, J., Olsson, H.H., Crnkovic, I.: Artificial Intelligence Paradigms for Smart Cyber-Physical Systems, chap. Engineering AI Systems: A Research Agenda, pp. 1\u201319. IGI Global (2020). https:\/\/doi.org\/10.4018\/978-1-7998-5101-1.ch001","DOI":"10.4018\/978-1-7998-5101-1.ch001"},{"key":"26_CR6","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2020.110542","volume":"164","author":"HB Braiek","year":"2020","unstructured":"Braiek, H.B., Khomh, F.: On testing machine learning programs. J. Syst. Softw. (JSS) 164, 110542 (2020). https:\/\/doi.org\/10.1016\/j.jss.2020.110542","journal-title":"J. Syst. Softw. (JSS)"},{"key":"26_CR7","doi-asserted-by":"crossref","unstructured":"Cheng, C.H., Huang, C.H., Yasuoka, H.: Quantitative projection coverage for testing ML-enabled autonomous systems. In: International Symposium on Automated Technology for Verification and Analysis (ATVA), pp. 126\u2013142 (2018)","DOI":"10.1007\/978-3-030-01090-4_8"},{"key":"26_CR8","doi-asserted-by":"crossref","unstructured":"Chernikova, A., Oprea, A., Nita-Rotaru, C., Kim, B.: Are self-driving cars secure? Evasion attacks against deep neural networks for steering angle prediction. In: IEEE Security and Privacy Workshops (SPW), pp. 132\u2013137 (2019)","DOI":"10.1109\/SPW.2019.00033"},{"key":"26_CR9","unstructured":"Chollet, F., et\u00a0al.: Keras (2015). https:\/\/github.com\/fchollet\/keras"},{"key":"26_CR10","doi-asserted-by":"publisher","unstructured":"Gong, H., Feng, Y., Chen, T., Li, Z., Li, Y.: Fast and accurate: the perception system of a formula student driverless car. In: International Conference on Robotics, Control and Automation (ICRCA), pp. 45\u201349 (2022). https:\/\/doi.org\/10.1109\/ICRCA55033.2022.9828892","DOI":"10.1109\/ICRCA55033.2022.9828892"},{"key":"26_CR11","doi-asserted-by":"publisher","unstructured":"Hegiste, V., Legler, T., Ruskowski, M.: Federated Ensemble YOLOv5 \u2013 A Better Generalized Object Detection Algorithm (2023). arXiv (arXiv:2306.17829). https:\/\/doi.org\/10.48550\/ARXIV.2306.17829","DOI":"10.48550\/ARXIV.2306.17829"},{"key":"26_CR12","unstructured":"Heineke, K., Kampshoff, P., Mkrtchyan, A., Shao, E.: Self-Driving Car Technology: When Will the Robots Hit the Road? Tech. rep., McKinsey & Company (2017). https:\/\/www.mckinsey.com\/industries\/automotive-and-assembly\/our-insights\/self-driving-car-technology-when-will-the-robots-hit-the-road"},{"issue":"2","key":"26_CR13","doi-asserted-by":"publisher","first-page":"20:1","DOI":"10.1145\/3487890","volume":"55","author":"Y Hu","year":"2023","unstructured":"Hu, Y., et al.: Artificial intelligence security: threats and countermeasures. ACM Comput. Surv. 55(2), 20:1-20:36 (2023). https:\/\/doi.org\/10.1145\/3487890","journal-title":"ACM Comput. Surv."},{"key":"26_CR14","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s10664-023-10388-7","volume":"29","author":"S Idowu","year":"2024","unstructured":"Idowu, S., Osman, O., Strueber, D., Berger, T.: Machine learning experiment management tools: a mixed-methods empirical study. Empir. Softw. Eng. (EMSE) 29, 1 (2024)","journal-title":"Empir. Softw. Eng. (EMSE)"},{"key":"26_CR15","doi-asserted-by":"crossref","unstructured":"Idowu, S., Sens, Y., Berger, T., Kr\u00fcger, J., Vierhauser, M.: A large-scale study of ML-related python projects. In: Symposium On Applied Computing (SAC) (2024). https:\/\/api.semanticscholar.org\/CorpusID:267375897","DOI":"10.1145\/3605098.3636056"},{"key":"26_CR16","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3543847","volume":"55","author":"S Idowu","year":"2022","unstructured":"Idowu, S., Strueber, D., Berger, T.: Asset management in machine learning: state-of-research and state-of-practice. ACM Comput. Surv. 55, 1 (2022)","journal-title":"ACM Comput. Surv."},{"key":"26_CR17","doi-asserted-by":"publisher","unstructured":"Idowu, S., Strueber, D., Berger, T.: EMMM: a unified meta-model for tracking machine learning experiments. In: Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 48\u201355 (2022). https:\/\/doi.org\/10.1109\/SEAA56994.2022.00016","DOI":"10.1109\/SEAA56994.2022.00016"},{"key":"26_CR18","doi-asserted-by":"publisher","unstructured":"Idowu, S., Str\u00fcber, D., Berger, T.: Asset management in machine learning: a survey. In: International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 51\u201360 (2021). https:\/\/doi.org\/10.1109\/ICSE-SEIP52600.2021.00014","DOI":"10.1109\/ICSE-SEIP52600.2021.00014"},{"key":"26_CR19","unstructured":"Jing, P., et al.: Too Good to Be Safe: Tricking Lane Detection in Autonomous Driving with Crafted Perturbations. In: USENIX Security Symposium, pp. 3237\u20133254 (2021). https:\/\/www.usenix.org\/conference\/usenixsecurity21\/presentation\/jing"},{"key":"26_CR20","doi-asserted-by":"publisher","unstructured":"Jocher, G.: YOLOv5 by Ultralytics (2020). https:\/\/doi.org\/10.5281\/zenodo.3908559. https:\/\/github.com\/ultralytics\/yolov5","DOI":"10.5281\/zenodo.3908559"},{"issue":"6245","key":"26_CR21","doi-asserted-by":"publisher","first-page":"255","DOI":"10.1126\/science.aaa8415","volume":"349","author":"MI Jordan","year":"2015","unstructured":"Jordan, M.I., Mitchell, T.M.: Machine learning: trends, perspectives, and prospects. Science 349(6245), 255\u2013260 (2015). https:\/\/doi.org\/10.1126\/science.aaa8415","journal-title":"Science"},{"issue":"8","key":"26_CR22","doi-asserted-by":"publisher","first-page":"6469","DOI":"10.1109\/JIOT.2020.3043716","volume":"8","author":"L Liu","year":"2021","unstructured":"Liu, L., et al.: Computing systems for autonomous driving: state of the art and challenges. IEEE Internet Things J. 8(8), 6469\u20136486 (2021). https:\/\/doi.org\/10.1109\/JIOT.2020.3043716","journal-title":"IEEE Internet Things J."},{"issue":"2","key":"26_CR23","doi-asserted-by":"publisher","first-page":"80","DOI":"10.1109\/MSECP.2004.1281254","volume":"2","author":"G McGraw","year":"2004","unstructured":"McGraw, G.: Software security. IEEE Secur. Privacy 2(2), 80\u201383 (2004). https:\/\/doi.org\/10.1109\/MSECP.2004.1281254","journal-title":"IEEE Secur. Privacy"},{"key":"26_CR24","doi-asserted-by":"publisher","unstructured":"Nahar, N., Zhang, H., Lewis, G., Zhou, S., K\u00e4stner, C.: A meta-summary of challenges in building products with ml components - collecting experiences from 4758+ practitioners. In: International Conference on AI Engineering \u2013 Software Engineering for AI (CAIN), pp. 171\u2013183 (2023). https:\/\/doi.org\/10.1109\/CAIN58948.2023.00034","DOI":"10.1109\/CAIN58948.2023.00034"},{"key":"26_CR25","doi-asserted-by":"publisher","DOI":"10.1016\/J.JSS.2023.111860","volume":"207","author":"R Nazir","year":"2024","unstructured":"Nazir, R., Bucaioni, A., Pelliccione, P.: Architecting ML-enabled systems: challenges, best practices, and design decisions. J. Syst. Softw. (JSS) 207, 111860 (2024). https:\/\/doi.org\/10.1016\/J.JSS.2023.111860","journal-title":"J. Syst. Softw. (JSS)"},{"key":"26_CR26","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-658-37665-9","volume-title":"Security Compliance in Model-driven Development of Software Systems in Presence of Long-term Evolution and Variants","author":"S Peldszus","year":"2022","unstructured":"Peldszus, S.: Security Compliance in Model-driven Development of Software Systems in Presence of Long-term Evolution and Variants. Springer, Cham (2022). https:\/\/doi.org\/10.1007\/978-3-658-37665-9"},{"key":"26_CR27","doi-asserted-by":"crossref","unstructured":"Peng, Z., Yang, J., Chen, T.H., Ma, L.: A first look at the integration of machine learning models in complex autonomous driving systems: a case study on Apollo. In: Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE), pp. 1240\u20131250 (2020)","DOI":"10.1145\/3368089.3417063"},{"key":"26_CR28","unstructured":"PyTorch: Open Source Machine Learning Framework (2023). https:\/\/pytorch.org"},{"key":"26_CR29","doi-asserted-by":"publisher","unstructured":"Redmon, J., Divvala, S.K., Girshick, R.B., Farhadi, A.: You only look once: unified, real-time object detection. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 779\u2013788 (2016). https:\/\/doi.org\/10.1109\/CVPR.2016.91","DOI":"10.1109\/CVPR.2016.91"},{"issue":"2","key":"26_CR30","doi-asserted-by":"publisher","first-page":"30","DOI":"10.1109\/MDAT.2020.2971217","volume":"37","author":"M Shafique","year":"2020","unstructured":"Shafique, M., Naseer, M., Theocharides, T., Kyrkou, C., Mutlu, O., Orosa, L., Choi, J.: Robust machine learning systems: challenges, current trends, perspectives, and the road ahead. IEEE Des. Test 37(2), 30\u201357 (2020). https:\/\/doi.org\/10.1109\/MDAT.2020.2971217","journal-title":"IEEE Des. Test"},{"key":"26_CR31","doi-asserted-by":"publisher","unstructured":"Stancin, I., Jovic, A.: An overview and comparison of free Python libraries for data mining and big data analysis. In: International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO), pp. 977\u2013982 (2019). https:\/\/doi.org\/10.23919\/MIPRO.2019.8757088","DOI":"10.23919\/MIPRO.2019.8757088"},{"issue":"1","key":"26_CR32","doi-asserted-by":"publisher","first-page":"273","DOI":"10.1007\/S10270-022-00991-5","volume":"22","author":"K Tuma","year":"2023","unstructured":"Tuma, K., Peldszus, S., Str\u00fcber, D., Scandariato, R., J\u00fcrjens, J.: Checking Security Compliance between Models and Code. Softw. Syst. Model. (SoSyM) 22(1), 273\u2013296 (2023). https:\/\/doi.org\/10.1007\/S10270-022-00991-5","journal-title":"Softw. Syst. Model. (SoSyM)"},{"key":"26_CR33","doi-asserted-by":"publisher","unstructured":"Valls, M.I., Hendrikx, H.F., Reijgwart, V.J., Meier, F.V., Sa, I., Dub\u00e9, R., Gawel, A., B\u00fcrki, M., Siegwart, R.: Design of an autonomous racecar: perception, state estimation and system integration. In: International Conference on Robotics and Automation (ICRA), pp. 2048\u20132055 (2018). https:\/\/doi.org\/10.1109\/ICRA.2018.8462829","DOI":"10.1109\/ICRA.2018.8462829"},{"issue":"8","key":"26_CR34","doi-asserted-by":"publisher","first-page":"980","DOI":"10.1109\/TKDE.2004.29","volume":"16","author":"GI Webb","year":"2004","unstructured":"Webb, G.I., Zheng, Z.: Multistrategy ensemble learning: reducing error by combining ensemble learning techniques. IEEE Trans. Knowl. Data Eng. (TKDE) 16(8), 980\u2013991 (2004). https:\/\/doi.org\/10.1109\/TKDE.2004.29","journal-title":"IEEE Trans. Knowl. Data Eng. (TKDE)"},{"key":"26_CR35","doi-asserted-by":"publisher","first-page":"106775","DOI":"10.1016\/J.KNOSYS.2021.106775","volume":"216","author":"C Zhang","year":"2021","unstructured":"Zhang, C., Xie, Y., Bai, H., Yu, B., Li, W., Gao, Y.: A Survey on Federated Learning. Knowl. Based Syst. 216, 106775 (2021). https:\/\/doi.org\/10.1016\/J.KNOSYS.2021.106775","journal-title":"Knowl. Based Syst."}],"container-title":["Lecture Notes in Computer Science","Bridging the Gap Between AI and Reality"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-031-73741-1_26","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,30]],"date-time":"2024-10-30T15:02:02Z","timestamp":1730300522000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/978-3-031-73741-1_26"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,10,31]]},"ISBN":["9783031737404","9783031737411"],"references-count":35,"URL":"https:\/\/doi.org\/10.1007\/978-3-031-73741-1_26","relation":{},"ISSN":["0302-9743","1611-3349"],"issn-type":[{"value":"0302-9743","type":"print"},{"value":"1611-3349","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,10,31]]},"assertion":[{"value":"31 October 2024","order":1,"name":"first_online","label":"First Online","group":{"name":"ChapterHistory","label":"Chapter History"}},{"value":"AISoLA","order":1,"name":"conference_acronym","label":"Conference Acronym","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"International Conference on Bridging the Gap between AI and Reality","order":2,"name":"conference_name","label":"Conference Name","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Crete","order":3,"name":"conference_city","label":"Conference City","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Greece","order":4,"name":"conference_country","label":"Conference Country","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"2023","order":5,"name":"conference_year","label":"Conference Year","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"23 October 2023","order":7,"name":"conference_start_date","label":"Conference Start Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"28 October 2023","order":8,"name":"conference_end_date","label":"Conference End Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"1","order":9,"name":"conference_number","label":"Conference Number","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"aisola2023","order":10,"name":"conference_id","label":"Conference ID","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"https:\/\/2023-aisola.isola-conference.org\/","order":11,"name":"conference_url","label":"Conference URL","group":{"name":"ConferenceInfo","label":"Conference Information"}}]}}