{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,30]],"date-time":"2026-07-30T14:27:56Z","timestamp":1785421676790,"version":"3.56.0"},"reference-count":202,"publisher":"Association for Computing Machinery (ACM)","issue":"5","license":[{"start":{"date-parts":[[2025,1,22]],"date-time":"2025-01-22T00:00:00Z","timestamp":1737504000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Knowledge Foundation of Sweden (KKS) through the Synergy Project AIDA - A Holistic AI-driven Networking and Processing Framework for Industrial IoT","award":["Rek:20200067"],"award-info":[{"award-number":["Rek:20200067"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Comput. Surv."],"published-print":{"date-parts":[[2025,5,31]]},"abstract":"<jats:p>Artificial intelligence (AI), and especially its sub-field of Machine Learning (ML), are impacting the daily lives of everyone with their ubiquitous applications. In recent years, AI researchers and practitioners have introduced principles and guidelines to build systems that make reliable and trustworthy decisions. From a practical perspective, conventional ML systems process historical data to extract the features that are consequently used to train ML models that perform the desired task. However, in practice, a fundamental challenge arises when the system needs to be operationalized and deployed to evolve and operate in real-life environments continuously. To address this challenge, Machine Learning Operations (MLOps) have emerged as a potential recipe for standardizing ML solutions in deployment. Although MLOps demonstrated great success in streamlining ML processes, thoroughly defining the specifications of robust MLOps approaches remains of great interest to researchers and practitioners. In this paper, we provide a comprehensive overview of the trustworthiness property of MLOps systems. Specifically, we highlight technical practices to achieve robust MLOps systems. In addition, we survey the existing research approaches that address the robustness aspects of ML systems in production. We also review the tools and software available to build MLOps systems and summarize their support to handle the robustness aspects. Finally, we present the open challenges and propose possible future directions and opportunities within this emerging field. The aim of this paper is to provide researchers and practitioners working on practical AI applications with a comprehensive view to adopt robust ML solutions in production environments.<\/jats:p>","DOI":"10.1145\/3708497","type":"journal-article","created":{"date-parts":[[2024,12,18]],"date-time":"2024-12-18T10:51:46Z","timestamp":1734519106000},"page":"1-35","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":39,"title":["Towards Trustworthy Machine Learning in Production: An Overview of the Robustness in MLOps Approach"],"prefix":"10.1145","volume":"57","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0683-2783","authenticated-orcid":false,"given":"Firas","family":"Bayram","sequence":"first","affiliation":[{"name":"Karlstad University, Karlstad, Sweden"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9051-7609","authenticated-orcid":false,"given":"Bestoun S.","family":"Ahmed","sequence":"additional","affiliation":[{"name":"Computer Science and Mathematics, Karlstad University, Karlstad Sweden and Department of Computer Science, Faculty of Electrical Engineering, Czech Technical University, Prague, Czech Republic"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,1,22]]},"reference":[{"key":"e_1_3_2_2_2","article-title":"Concept drift detection in data stream mining: A literature review","author":"Agrahari Supriya","year":"2021","unstructured":"Supriya Agrahari and Anil Kumar Singh. 2021. Concept drift detection in data stream mining: A literature review. Journal of King Saud University-Computer and Information Sciences (2021).","journal-title":"Journal of King Saud University-Computer and Information Sciences"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/3292500.3330701"},{"issue":"1","key":"e_1_3_2_4_2","doi-asserted-by":"crossref","first-page":"372","DOI":"10.1109\/MNET.011.2000371","article-title":"Generalizing AI: Challenges and opportunities for plug and play AI solutions","volume":"35","author":"Ridhawi Ismaeel Al","year":"2020","unstructured":"Ismaeel Al Ridhawi, Safa Otoum, Moayad Aloqaily, and Azzedine Boukerche. 2020. Generalizing AI: Challenges and opportunities for plug and play AI solutions. IEEE Network 35, 1 (2020), 372\u2013379.","journal-title":"IEEE Network"},{"key":"e_1_3_2_5_2","first-page":"659","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Ali Hazrat","year":"2023","unstructured":"Hazrat Ali, Christer Gr\u00f6nlund, and Zubair Shah. 2023. Leveraging GANs for data scarcity of COVID-19: Beyond the hype. In Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition. 659\u2013667."},{"key":"e_1_3_2_6_2","doi-asserted-by":"crossref","DOI":"10.1007\/978-1-4842-6549-9","volume-title":"Beginning MLOps with MLFlow","author":"Alla Sridhar","year":"2021","unstructured":"Sridhar Alla and Suman Kalyan Adari. 2021. Beginning MLOps with MLFlow. Springer."},{"key":"e_1_3_2_7_2","doi-asserted-by":"crossref","first-page":"79","DOI":"10.1007\/978-1-4842-6549-9_3","volume-title":"Beginning MLOps with MLFlow","author":"Alla Sridhar","year":"2021","unstructured":"Sridhar Alla and Suman Kalyan Adari. 2021. What is MLOps? In Beginning MLOps with MLFlow. Springer, 79\u2013124."},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/3359591.3359735"},{"key":"e_1_3_2_9_2","first-page":"441","volume-title":"20th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN)","author":"Anguita Davide","year":"2012","unstructured":"Davide Anguita, Luca Ghelardoni, Alessandro Ghio, Luca Oneto, and Sandro Ridella. 2012. The \u2018K\u2019 in K-fold cross validation. In 20th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN). i6doc. com publ, 441\u2013446."},{"key":"e_1_3_2_10_2","first-page":"1","volume-title":"2022 IEEE International Conference on Evolving and Adaptive Intelligent Systems (EAIS)","author":"Antonini Mattia","year":"2022","unstructured":"Mattia Antonini, Miguel Pincheira, Massimo Vecchio, and Fabio Antonelli. 2022. Tiny-MLOps: A framework for orchestrating ML applications at the far edge of IoT systems. In 2022 IEEE International Conference on Evolving and Adaptive Intelligent Systems (EAIS). IEEE, 1\u20138."},{"key":"e_1_3_2_11_2","doi-asserted-by":"crossref","first-page":"215","DOI":"10.1007\/978-1-4842-5104-1_9","volume-title":"Practical DataOps","author":"Atwal Harvinder","year":"2020","unstructured":"Harvinder Atwal. 2020. DataOps technology. In Practical DataOps. Springer, 215\u2013247."},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-019-05791-5"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/3502287"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3097983.3098021"},{"key":"e_1_3_2_15_2","volume-title":"2022 37th IEEE\/ACM International Conference on Automated Software Engineering (ASE)","author":"Bayram Firas","year":"2022","unstructured":"Firas Bayram, Bestoun S. Ahmed, Erik Hallin, and Anton Engman. 2022. A drift handling approach for self-adaptive ML software in scalable industrial processes. In 2022 37th IEEE\/ACM International Conference on Automated Software Engineering (ASE). IEEE."},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2022.108632"},{"key":"e_1_3_2_17_2","first-page":"39","article-title":"Principles and practice of explainable machine learning","author":"Belle Vaishak","year":"2021","unstructured":"Vaishak Belle and Ioannis Papantonis. 2021. Principles and practice of explainable machine learning. Frontiers in Big Data (2021), 39.","journal-title":"Frontiers in Big Data"},{"issue":"2","key":"e_1_3_2_18_2","article-title":"Random search for hyper-parameter optimization.","volume":"13","author":"Bergstra James","year":"2012","unstructured":"James Bergstra and Yoshua Bengio. 2012. Random search for hyper-parameter optimization. Journal of Machine Learning Research 13, 2 (2012).","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_2_19_2","doi-asserted-by":"crossref","unstructured":"Daniel Berrar. 2019. Cross-Validation.Encyclopedia of Bioinformatics and Computational Biology 1 April (2019) 542\u2013545.","DOI":"10.1016\/B978-0-12-809633-8.20349-X"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/CISS.2018.8362326"},{"key":"e_1_3_2_21_2","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1007\/978-1-4842-4470-8_2","article-title":"An overview of Google Cloud Platform services","author":"Bisong Ekaba","year":"2019","unstructured":"Ekaba Bisong. 2019. An overview of Google Cloud Platform services. Building Machine Learning and Deep Learning Models on Google Cloud Platform (2019), 7\u201310.","journal-title":"Building Machine Learning and Deep Learning Models on Google Cloud Platform"},{"key":"e_1_3_2_22_2","article-title":"Performance metrics (error measures) in machine learning regression, forecasting and prognostics: Properties and typology","author":"Botchkarev Alexei","year":"2018","unstructured":"Alexei Botchkarev. 2018. Performance metrics (error measures) in machine learning regression, forecasting and prognostics: Properties and typology. arXiv preprint arXiv:1809.03006 (2018).","journal-title":"arXiv preprint arXiv:1809.03006"},{"issue":"2","key":"e_1_3_2_23_2","doi-asserted-by":"crossref","first-page":"144","DOI":"10.1111\/j.2044-8317.1978.tb00581.x","article-title":"Robustness?","volume":"31","author":"Bradley James V.","year":"1978","unstructured":"James V. Bradley. 1978. Robustness? Brit. J. Math. Statist. Psych. 31, 2 (1978), 144\u2013152.","journal-title":"Brit. J. Math. Statist. Psych."},{"key":"e_1_3_2_24_2","doi-asserted-by":"crossref","first-page":"1123","DOI":"10.1109\/BigData.2017.8258038","volume-title":"2017 IEEE International Conference on Big Data (Big Data)","author":"Breck Eric","year":"2017","unstructured":"Eric Breck, Shanqing Cai, Eric Nielsen, Michael Salib, and D. Sculley. 2017. The ML test score: A rubric for ML production readiness and technical debt reduction. In 2017 IEEE International Conference on Big Data (Big Data). IEEE, 1123\u20131132."},{"key":"e_1_3_2_25_2","volume-title":"MLSys","author":"Breck Eric","year":"2019","unstructured":"Eric Breck, Neoklis Polyzotis, Sudip Roy, Steven Whang, and Martin Zinkevich. 2019. Data validation for machine learning. In MLSys."},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1007\/BF01582063"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-39306-9_4"},{"key":"e_1_3_2_28_2","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1109\/SP.2017.49","volume-title":"2017 IEEE Symposium on Security and Privacy (SP\u201917)","author":"Carlini Nicholas","year":"2017","unstructured":"Nicholas Carlini and David Wagner. 2017. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP\u201917). IEEE, 39\u201357."},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/1541880.1541882"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.3390\/make1010020"},{"key":"e_1_3_2_31_2","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1007\/978-3-030-69128-8_2","volume-title":"Reflections on Artificial Intelligence for Humanity","author":"Chatila Raja","year":"2021","unstructured":"Raja Chatila, Virginia Dignum, Michael Fisher, Fosca Giannotti, Katharina Morik, Stuart Russell, and Karen Yeung. 2021. Trustworthy AI. In Reflections on Artificial Intelligence for Humanity. Springer, 13\u201339."},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3558943"},{"key":"e_1_3_2_33_2","article-title":"Did the model change? Efficiently assessing machine learning API shifts","author":"Chen Lingjiao","year":"2021","unstructured":"Lingjiao Chen, Tracy Cai, Matei Zaharia, and James Zou. 2021. Did the model change? Efficiently assessing machine learning API shifts. arXiv preprint arXiv:2107.14203 (2021).","journal-title":"arXiv preprint arXiv:2107.14203"},{"key":"e_1_3_2_34_2","first-page":"1617","volume-title":"International Conference on Machine Learning","author":"Chen Mayee","year":"2021","unstructured":"Mayee Chen, Karan Goel, Nimit S. Sohoni, Fait Poms, Kayvon Fatahalian, and Christopher R\u00e9. 2021. Mandoline: Model evaluation under distribution shift. In International Conference on Machine Learning. PMLR, 1617\u20131629."},{"key":"e_1_3_2_35_2","first-page":"012027","volume-title":"Journal of Physics: Conference Series","volume":"2327","author":"Chowdary Mandepudi Nobel","year":"2022","unstructured":"Mandepudi Nobel Chowdary, Bussa Sankeerth, Chennupati Kumar Chowdary, and Manu Gupta. 2022. Accelerating the machine learning model deployment using MLOps. In Journal of Physics: Conference Series, Vol. 2327. IOP Publishing, 012027."},{"key":"e_1_3_2_36_2","volume-title":"ICML\u20192003 Workshop on Learning from Imbalanced Data Sets (II), Washington, DC","author":"Chowdhury Abdur","year":"2003","unstructured":"Abdur Chowdhury and Joshua Alspector. 2003. Data duplication: An imbalance problem?. In ICML\u20192003 Workshop on Learning from Imbalanced Data Sets (II), Washington, DC. Citeseer."},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/2882903.2912574"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1037\/h0026256"},{"key":"e_1_3_2_39_2","volume-title":"Ethics Guidelines for Trustworthy AI","author":"Commission European","year":"2019","unstructured":"European Commission, Content Directorate-General for Communications Networks, and Technology. 2019. Ethics Guidelines for Trustworthy AI. Publications Office."},{"key":"e_1_3_2_40_2","article-title":"The missing piece in complex analytics: Low latency, scalable model management and serving with Velox","author":"Crankshaw Daniel","year":"2014","unstructured":"Daniel Crankshaw, Peter Bailis, Joseph E. Gonzalez, Haoyuan Li, Zhao Zhang, Michael J. Franklin, Ali Ghodsi, and Michael I. Jordan. 2014. The missing piece in complex analytics: Low latency, scalable model management and serving with Velox. arXiv preprint arXiv:1409.3809 (2014).","journal-title":"arXiv preprint arXiv:1409.3809"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/3453935"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jnca.2011.12.006"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2020.114060"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/2347736.2347755"},{"issue":"5","key":"e_1_3_2_45_2","doi-asserted-by":"crossref","first-page":"311","DOI":"10.1002\/sam.10054","article-title":"Adaptive concept drift detection","volume":"2","author":"Dries Anton","year":"2009","unstructured":"Anton Dries and Ulrich R\u00fcckert. 2009. Adaptive concept drift detection. Statistical Analysis and Data Mining 2, 5-6 (2009), 311\u2013327.","journal-title":"Statistical Analysis and Data Mining"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1145\/3359786"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cad.2021.103013"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1177\/1471082X15588398"},{"key":"e_1_3_2_49_2","first-page":"148","volume-title":"2018 Thirteenth International Conference on Digital Information Management (ICDIM\u201918)","author":"Ehrlinger Lisa","year":"2018","unstructured":"Lisa Ehrlinger, Thomas Grubinger, Bence Varga, Mario Pichler, Thomas Natschl\u00e4ger, and J\u00fcrgen Zeindl. 2018. Treating missing data in industrial data analytics. In 2018 Thirteenth International Conference on Digital Information Management (ICDIM\u201918). IEEE, 148\u2013155."},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1109\/TNN.2011.2160459"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.autcon.2022.104298"},{"issue":"3","key":"e_1_3_2_52_2","article-title":"Magician\u2019s corner: 9. Performance metrics for machine learning models","volume":"3","author":"Erickson Bradley J.","year":"2021","unstructured":"Bradley J. Erickson and Felipe Kitamura. 2021. Magician\u2019s corner: 9. Performance metrics for machine learning models. Radiology: Artificial Intelligence 3, 3 (2021).","journal-title":"Radiology: Artificial Intelligence"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-05318-5_1"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2013.2292894"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2014.10.085"},{"key":"e_1_3_2_56_2","first-page":"25","volume-title":"2021 IEEE Fourth International Conference on Artificial Intelligence and Knowledge Engineering (AIKE\u201921)","author":"Garg Satvik","year":"2021","unstructured":"Satvik Garg, Pradyumn Pundir, Geetanjali Rathee, P. K. Gupta, Somya Garg, and Saransh Ahlawat. 2021. On continuous integration\/continuous delivery for automated deployment of machine learning models using MLOps. In 2021 IEEE Fourth International Conference on Artificial Intelligence and Knowledge Engineering (AIKE\u201921). IEEE, 25\u201328."},{"issue":"14","key":"e_1_3_2_57_2","doi-asserted-by":"crossref","first-page":"5499","DOI":"10.1109\/JSEN.2019.2907398","article-title":"Aeromagnetic compensation algorithm robust to outliers of magnetic sensor based on Huber loss method","volume":"19","author":"Ge Jian","year":"2019","unstructured":"Jian Ge, Han Li, Hongpeng Wang, Haobin Dong, Huan Liu, Wenjie Wang, Zhiwen Yuan, Jun Zhu, and Haiyang Zhang. 2019. Aeromagnetic compensation algorithm robust to outliers of magnetic sensor based on Huber loss method. IEEE Sensors Journal 19, 14 (2019), 5499\u20135505.","journal-title":"IEEE Sensors Journal"},{"key":"e_1_3_2_58_2","doi-asserted-by":"crossref","first-page":"341","DOI":"10.1109\/ICDMW53433.2021.00049","volume-title":"2021 International Conference on Data Mining Workshops (ICDMW\u201921)","author":"Greco Salvatore","year":"2021","unstructured":"Salvatore Greco and Tania Cerquitelli. 2021. Drift Lens: Real-time unsupervised concept drift detection by evaluating per-label embedding distributions. In 2021 International Conference on Data Mining Workshops (ICDMW\u201921). IEEE, 341\u2013349."},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.01633"},{"key":"e_1_3_2_60_2","first-page":"192","volume-title":"2008 Fourth International Conference on Natural Computation","volume":"4","author":"Guo Xinjian","year":"2008","unstructured":"Xinjian Guo, Yilong Yin, Cailing Dong, Gongping Yang, and Guangtong Zhou. 2008. On the class imbalance problem. In 2008 Fourth International Conference on Natural Computation, Vol. 4. IEEE, 192\u2013201."},{"key":"e_1_3_2_61_2","article-title":"Robustness and explainability of artificial intelligence","author":"Hamon Ronan","year":"2020","unstructured":"Ronan Hamon, Henrik Junklewitz, and Ignacio Sanchez. 2020. Robustness and explainability of artificial intelligence. Publications Office of the European Union (2020).","journal-title":"Publications Office of the European Union"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA48506.2021.9561103"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.softx.2020.100591"},{"key":"e_1_3_2_64_2","article-title":"Machine learning operations: A survey on MLOps tool support","author":"Hewage Nipuni","year":"2022","unstructured":"Nipuni Hewage and Dulani Meedeniya. 2022. Machine learning operations: A survey on MLOps tool support. arXiv preprint arXiv:2202.10169 (2022).","journal-title":"arXiv preprint arXiv:2202.10169"},{"key":"e_1_3_2_65_2","first-page":"4249","volume-title":"International Conference on Machine Learning","author":"Hinder Fabian","year":"2020","unstructured":"Fabian Hinder, Andr\u00e9 Artelt, and Barbara Hammer. 2020. Towards non-parametric drift detection via dynamic adapting window independence drift detection (DAWIDD). In International Conference on Machine Learning. PMLR, 4249\u20134259."},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.patter.2020.100129"},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","DOI":"10.1007\/s13748-011-0008-0"},{"key":"e_1_3_2_68_2","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376177"},{"key":"e_1_3_2_69_2","first-page":"101","volume-title":"2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA\u201917)","author":"Hosseini Hossein","year":"2017","unstructured":"Hossein Hosseini, Baicen Xiao, and Radha Poovendran. 2017. Google\u2019s Cloud Vision API is not robust to noise. In 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA\u201917). IEEE, 101\u2013105."},{"key":"e_1_3_2_70_2","article-title":"ModelCI-e: Enabling continual learning in deep learning serving systems","author":"Huang Yizheng","year":"2021","unstructured":"Yizheng Huang, Huaizheng Zhang, Yonggang Wen, Peng Sun, and Nguyen Binh Duong Ta. 2021. ModelCI-e: Enabling continual learning in deep learning serving systems. arXiv preprint arXiv:2106.03122 (2021).","journal-title":"arXiv preprint arXiv:2106.03122"},{"key":"e_1_3_2_71_2","volume-title":"Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation","author":"Humble Jez","year":"2010","unstructured":"Jez Humble and David Farley. 2010. Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation. Pearson Education."},{"key":"e_1_3_2_72_2","doi-asserted-by":"crossref","first-page":"113","DOI":"10.1109\/IC2E.2019.00025","volume-title":"2019 IEEE International Conference on Cloud Engineering (IC2E\u201919)","author":"Hummer Waldemar","year":"2019","unstructured":"Waldemar Hummer, Vinod Muthusamy, Thomas Rausch, Parijat Dube, Kaoutar El Maghraoui, Anupama Murthi, and Punleuk Oum. 2019. ModelOps: Cloud-based lifecycle management for reliable and trusted AI. In 2019 IEEE International Conference on Cloud Engineering (IC2E\u201919). IEEE, 113\u2013120."},{"key":"e_1_3_2_73_2","doi-asserted-by":"publisher","DOI":"10.1145\/2791120"},{"key":"e_1_3_2_74_2","doi-asserted-by":"publisher","DOI":"10.1145\/2724719"},{"key":"e_1_3_2_75_2","doi-asserted-by":"publisher","DOI":"10.1145\/3310205"},{"key":"e_1_3_2_76_2","volume-title":"Information Technology \u2013 Artificial intelligence \u2013 Overview of Trustworthiness in Artificial Intelligence","year":"2022","unstructured":"ISO\/IEC. 2022. Information Technology \u2013 Artificial intelligence \u2013 Overview of Trustworthiness in Artificial Intelligence. Technical Report 24028:2020. Geneva, Switzerland."},{"key":"e_1_3_2_77_2","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-019-0088-2"},{"key":"e_1_3_2_78_2","first-page":"1","volume-title":"2021 47th Euromicro Conference on Software Engineering and Advanced Applications (SEAA\u201921)","author":"John Meenu Mary","year":"2021","unstructured":"Meenu Mary John, Helena Holmstr\u00f6m Olsson, and Jan Bosch. 2021. Towards MLOps: A framework and maturity model. In 2021 47th Euromicro Conference on Software Engineering and Advanced Applications (SEAA\u201921). IEEE, 1\u20138."},{"key":"e_1_3_2_79_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.aaa8415"},{"key":"e_1_3_2_80_2","doi-asserted-by":"publisher","DOI":"10.3390\/info11070363"},{"key":"e_1_3_2_81_2","doi-asserted-by":"publisher","DOI":"10.14778\/3430915.3430917"},{"key":"e_1_3_2_82_2","doi-asserted-by":"publisher","DOI":"10.1109\/TITS.2021.3059261"},{"key":"e_1_3_2_83_2","first-page":"105","volume-title":"International Conference on Network-Based Information Systems","author":"Kaur Davinder","year":"2020","unstructured":"Davinder Kaur, Suleyman Uslu, and Arjan Durresi. 2020. Requirements for trustworthy artificial intelligence\u2013A review. In International Conference on Network-Based Information Systems. Springer, 105\u2013115."},{"key":"e_1_3_2_84_2","doi-asserted-by":"publisher","DOI":"10.1145\/3491209"},{"key":"e_1_3_2_85_2","doi-asserted-by":"publisher","DOI":"10.1002\/sam.10124"},{"key":"e_1_3_2_86_2","article-title":"Improving generalization performance by switching from Adam to SGD","author":"Keskar Nitish Shirish","year":"2017","unstructured":"Nitish Shirish Keskar and Richard Socher. 2017. Improving generalization performance by switching from Adam to SGD. arXiv preprint arXiv:1712.07628 (2017).","journal-title":"arXiv preprint arXiv:1712.07628"},{"key":"e_1_3_2_87_2","doi-asserted-by":"crossref","unstructured":"Hiroaki Kitano. 2007. Towards a theory of biological robustness. Molecular Systems Biology 3 (2007) 137.","DOI":"10.1038\/msb4100179"},{"key":"e_1_3_2_88_2","article-title":"Monitoring and explainability of models in production","author":"Klaise Janis","year":"2020","unstructured":"Janis Klaise, Arnaud Van Looveren, Clive Cox, Giovanni Vacanti, and Alexandru Coca. 2020. Monitoring and explainability of models in production. arXiv preprint arXiv:2007.06299 (2020).","journal-title":"arXiv preprint arXiv:2007.06299"},{"key":"e_1_3_2_89_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2020.106223"},{"key":"e_1_3_2_90_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2023.3262138"},{"key":"e_1_3_2_91_2","first-page":"1","volume-title":"2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops\u201920)","author":"Kumar Abhishek","year":"2020","unstructured":"Abhishek Kumar, Tristan Braud, Sasu Tarkoma, and Pan Hui. 2020. Trustworthy AI in the age of pervasive computing and big data. In 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops\u201920). IEEE, 1\u20136."},{"key":"e_1_3_2_92_2","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1109\/BigDataService52369.2021.00008","volume-title":"2021 IEEE Seventh International Conference on Big Data Computing Service and Applications (BigDataService\u201921)","author":"Kurian John Joy","year":"2021","unstructured":"John Joy Kurian, Marcel Dix, Ido Amihai, Glenn Ceusters, and Ajinkya Prabhune. 2021. BOAT: A Bayesian optimization AutoML time-series framework for industrial applications. In 2021 IEEE Seventh International Conference on Big Data Computing Service and Applications (BigDataService\u201921). IEEE, 17\u201324."},{"key":"e_1_3_2_93_2","first-page":"2314","volume-title":"2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC\u201921)","author":"Larracy Robyn","year":"2021","unstructured":"Robyn Larracy, Angkoon Phinyomark, and Erik Scheme. 2021. Machine learning model validation for early stage studies with small sample sizes. In 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC\u201921). IEEE, 2314\u20132319."},{"key":"e_1_3_2_94_2","doi-asserted-by":"publisher","DOI":"10.1145\/3555803"},{"key":"e_1_3_2_95_2","doi-asserted-by":"publisher","DOI":"10.3390\/fi12060102"},{"key":"e_1_3_2_96_2","doi-asserted-by":"publisher","DOI":"10.1145\/3318464.3386126"},{"key":"e_1_3_2_97_2","article-title":"Trustworthy AI: A computational perspective","author":"Liu Haochen","year":"2022","unstructured":"Haochen Liu, Yiqi Wang, Wenqi Fan, Xiaorui Liu, Yaxin Li, Shaili Jain, Yunhao Liu, Anil K. Jain, and Jiliang Tang. 2022. Trustworthy AI: A computational perspective. ACM Trans. Intell. Syst. Technol. (June2022).","journal-title":"ACM Trans. Intell. Syst. Technol."},{"issue":"12","key":"e_1_3_2_98_2","first-page":"2346","article-title":"Learning under concept drift: A review","volume":"31","author":"Lu Jie","year":"2019","unstructured":"Jie Lu, Anjin Liu, Fan Dong, Feng Gu, Jo\u00e3o Gama, and Guangquan Zhang. 2019. Learning under concept drift: A review. IEEE Transactions on Knowledge and Data Engineering 31, 12 (2019), 2346\u20132363.","journal-title":"IEEE Transactions on Knowledge and Data Engineering"},{"key":"e_1_3_2_99_2","first-page":"2393","volume-title":"IJCAI","author":"Lu Yang","year":"2017","unstructured":"Yang Lu, Yiu-ming Cheung, and Yuan Yan Tang. 2017. Dynamic weighted majority for incremental learning of imbalanced data streams with concept drift. In IJCAI. 2393\u20132399."},{"key":"e_1_3_2_100_2","doi-asserted-by":"publisher","DOI":"10.1007\/s13721-016-0125-6"},{"key":"e_1_3_2_101_2","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocx133"},{"key":"e_1_3_2_102_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ejor.2017.07.044"},{"key":"e_1_3_2_103_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.conengprac.2021.104903"},{"key":"e_1_3_2_104_2","first-page":"002203452211060","article-title":"Towards trustworthy AI in dentistry","author":"Ma J.","year":"2022","unstructured":"J. Ma, L. Schneider, S. Lapuschkin, R. Achtibat, M. Duchrau, J. Krois, F. Schwendicke, and W. Samek. 2022. Towards trustworthy AI in dentistry. Journal of Dental Research (2022), 00220345221106086.","journal-title":"Journal of Dental Research"},{"key":"e_1_3_2_105_2","article-title":"The connection between out-of-distribution generalization and privacy of ML models","author":"Mahajan Divyat","year":"2021","unstructured":"Divyat Mahajan, Shruti Tople, and Amit Sharma. 2021. The connection between out-of-distribution generalization and privacy of ML models. arXiv preprint arXiv:2110.03369 (2021).","journal-title":"arXiv preprint arXiv:2110.03369"},{"key":"e_1_3_2_106_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.dcan.2017.10.002"},{"key":"e_1_3_2_107_2","doi-asserted-by":"publisher","DOI":"10.1109\/WAIN52551.2021.00024"},{"key":"e_1_3_2_108_2","first-page":"77","article-title":"Matchmaker: Data drift mitigation in machine learning for large-scale systems","volume":"4","author":"Mallick Ankur","year":"2022","unstructured":"Ankur Mallick, Kevin Hsieh, Behnaz Arzani, and Gauri Joshi. 2022. Matchmaker: Data drift mitigation in machine learning for large-scale systems. Proceedings of Machine Learning and Systems 4 (2022), 77\u201394.","journal-title":"Proceedings of Machine Learning and Systems"},{"key":"e_1_3_2_109_2","doi-asserted-by":"crossref","first-page":"359","DOI":"10.1007\/978-1-4842-4106-6_7","volume-title":"Cognitive Computing Recipes","author":"Masood Adnan","year":"2019","unstructured":"Adnan Masood and Adnan Hashmi. 2019. AIOps: Predictive analytics & machine learning in operations. In Cognitive Computing Recipes. Springer, 359\u2013382."},{"key":"e_1_3_2_110_2","first-page":"85","volume-title":"Joint European Conference on Machine Learning and Knowledge Discovery in Databases","author":"Mathew Jose","year":"2021","unstructured":"Jose Mathew, Meghana Negi, Rutvik Vijjali, and Jairaj Sathyanarayana. 2021. DeFraudNet: An end-to-end weak supervision framework to detect fraud in online food delivery. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 85\u201399."},{"key":"e_1_3_2_111_2","first-page":"45","volume-title":"2022 IEEE\/ACM 1st International Workshop on Software Engineering for Responsible Artificial Intelligence (SE4RAI\u201922)","author":"Matsui Beatriz M. A.","year":"2022","unstructured":"Beatriz M. A. Matsui and Denise H. Goya. 2022. MLOps: A guide to its adoption in the context of responsible AI. In 2022 IEEE\/ACM 1st International Workshop on Software Engineering for Responsible Artificial Intelligence (SE4RAI\u201922). IEEE, 45\u201349."},{"key":"e_1_3_2_112_2","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2020.2984868"},{"key":"e_1_3_2_113_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2020.3039931"},{"key":"e_1_3_2_114_2","doi-asserted-by":"crossref","first-page":"28","DOI":"10.1145\/3426745.3431338","volume-title":"Proceedings of the 1st Workshop on Distributed Machine Learning","author":"Meister Moritz","year":"2020","unstructured":"Moritz Meister, Sina Sheikholeslami, Amir H. Payberah, Vladimir Vlassov, and Jim Dowling. 2020. Maggy: Scalable asynchronous parallel hyperparameter search. In Proceedings of the 1st Workshop on Distributed Machine Learning. 28\u201333."},{"key":"e_1_3_2_115_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.inffus.2019.12.001"},{"key":"e_1_3_2_116_2","article-title":"SensiX++: Bringing MLOps and multi-tenant model serving to sensory edge devices","author":"Min Chulhong","year":"2021","unstructured":"Chulhong Min, Akhil Mathur, Utku Gunay Acer, Alessandro Montanari, and Fahim Kawsar. 2021. SensiX++: Bringing MLOps and multi-tenant model serving to sensory edge devices. arXiv preprint arXiv:2109.03947 (2021).","journal-title":"arXiv preprint arXiv:2109.03947"},{"key":"e_1_3_2_117_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10462-022-10246-w"},{"key":"e_1_3_2_118_2","first-page":"339","volume-title":"Joint European Conference on Machine Learning and Knowledge Discovery in Databases","author":"Moulton Richard Hugh","year":"2018","unstructured":"Richard Hugh Moulton, Herna L. Viktor, Nathalie Japkowicz, and Jo\u00e3o Gama. 2018. Clustering in the presence of concept drift. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 339\u2013355."},{"key":"e_1_3_2_119_2","doi-asserted-by":"publisher","DOI":"10.1145\/3379177.3388909"},{"key":"e_1_3_2_120_2","volume-title":"Data Quality: The Accuracy Dimension","author":"Olson Jack E.","year":"2003","unstructured":"Jack E. Olson. 2003. Data Quality: The Accuracy Dimension. Elsevier."},{"key":"e_1_3_2_121_2","doi-asserted-by":"publisher","DOI":"10.1109\/5254.769885"},{"key":"e_1_3_2_122_2","doi-asserted-by":"publisher","DOI":"10.1145\/3397481.3450637"},{"key":"e_1_3_2_123_2","first-page":"25","volume-title":"VLDB","author":"Qian Xiaolei","year":"1986","unstructured":"Xiaolei Qian and Gio Wiederhold. 1986. Knowledge-based integrity constraint validation. In VLDB, Vol. 86. Citeseer, 25\u201328."},{"key":"e_1_3_2_124_2","doi-asserted-by":"publisher","DOI":"10.5555\/1462129"},{"key":"e_1_3_2_125_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neunet.2021.05.004"},{"key":"e_1_3_2_126_2","volume-title":"Engineering MLOps: Rapidly Build, Test, and Manage Production-ready Machine Learning Life Cycles at Scale","author":"Raj Emmanuel","year":"2021","unstructured":"Emmanuel Raj. 2021. Engineering MLOps: Rapidly Build, Test, and Manage Production-ready Machine Learning Life Cycles at Scale. Packt Publishing Ltd."},{"key":"e_1_3_2_127_2","first-page":"191","volume-title":"2021 IEEE International Conference on Cloud Engineering (IC2E\u201921)","author":"Raj Emmanuel","year":"2021","unstructured":"Emmanuel Raj, David Buffoni, Magnus Westerlund, and Kimmo Ahola. 2021. Edge MLOps: An automation framework for AIoT applications. In 2021 IEEE International Conference on Cloud Engineering (IC2E\u201921). IEEE, 191\u2013200."},{"key":"e_1_3_2_128_2","volume-title":"Python Machine Learning: Machine Learning and Deep Learning with Python, Scikit-learn, and TensorFlow 2","author":"Raschka Sebastian","year":"2019","unstructured":"Sebastian Raschka and Vahid Mirjalili. 2019. Python Machine Learning: Machine Learning and Deep Learning with Python, Scikit-learn, and TensorFlow 2. Packt Publishing Ltd."},{"key":"e_1_3_2_129_2","doi-asserted-by":"publisher","DOI":"10.14778\/3157794.3157797"},{"key":"e_1_3_2_130_2","article-title":"Overton: A data system for monitoring and improving machine-learned products","author":"R\u00e9 Christopher","year":"2019","unstructured":"Christopher R\u00e9, Feng Niu, Pallavi Gudipati, and Charles Srisuwananukorn. 2019. Overton: A data system for monitoring and improving machine-learned products. arXiv preprint arXiv:1909.05372 (2019).","journal-title":"arXiv preprint arXiv:1909.05372"},{"key":"e_1_3_2_131_2","first-page":"4334","volume-title":"International Conference on Machine Learning","author":"Ren Mengye","year":"2018","unstructured":"Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. 2018. Learning to reweight examples for robust deep learning. In International Conference on Machine Learning. PMLR, 4334\u20134343."},{"key":"e_1_3_2_132_2","article-title":"A data quality-driven view of MLOps","author":"Renggli Cedric","year":"2021","unstructured":"Cedric Renggli, Luka Rimanic, Nezihe Merve G\u00fcrel, Bojan Karla\u0161, Wentao Wu, and Ce Zhang. 2021. A data quality-driven view of MLOps. arXiv preprint arXiv:2102.07750 (2021).","journal-title":"arXiv preprint arXiv:2102.07750"},{"key":"e_1_3_2_133_2","article-title":"Ease. ml\/snoopy: Towards automatic feasibility study for machine learning applications","author":"Renggli Cedric","year":"2020","unstructured":"Cedric Renggli, Luka Rimanic, Luka Kolar, Nora Hollenstein, Wentao Wu, and Ce Zhang. 2020. Ease. ml\/snoopy: Towards automatic feasibility study for machine learning applications. arXiv preprint arXiv:2010.08410 (2020).","journal-title":"arXiv preprint arXiv:2010.08410"},{"key":"e_1_3_2_134_2","first-page":"227","volume-title":"2016 15th IEEE International Conference on Machine Learning and Applications (ICMLA\u201916)","author":"Rozsa Andras","year":"2016","unstructured":"Andras Rozsa, Manuel G\u00fcnther, and Terrance E. Boult. 2016. Are accuracy and robustness correlated. In 2016 15th IEEE International Conference on Machine Learning and Applications (ICMLA\u201916). IEEE, 227\u2013232."},{"key":"e_1_3_2_135_2","doi-asserted-by":"crossref","unstructured":"Cynthia Rudin and Kiri L. Wagstaff. 2014. Machine learning for science and society. Mach Learn 95 (2014) 1\u20139.","DOI":"10.1007\/s10994-013-5425-9"},{"key":"e_1_3_2_136_2","doi-asserted-by":"publisher","DOI":"10.3390\/app11198861"},{"key":"e_1_3_2_137_2","doi-asserted-by":"publisher","DOI":"10.1007\/s42979-021-00592-x"},{"issue":"2","key":"e_1_3_2_138_2","first-page":"1","article-title":"AI-based modeling: Techniques, applications and research issues towards automation, intelligent and smart systems","volume":"3","author":"Sarker Iqbal H.","year":"2022","unstructured":"Iqbal H. Sarker. 2022. AI-based modeling: Techniques, applications and research issues towards automation, intelligent and smart systems. SN Computer Science 3, 2 (2022), 1\u201320.","journal-title":"SN Computer Science"},{"key":"e_1_3_2_139_2","unstructured":"Sebastian Schelter Felix Biessmann Tim Januschowski David Salinas Stephan Seufert and Gyuri Szarvas. 2018. On challenges in machine learning model management. IEEE Data Eng. Bull. 41 (2018) 5\u201315."},{"key":"e_1_3_2_140_2","doi-asserted-by":"publisher","DOI":"10.14778\/3229863.3229867"},{"key":"e_1_3_2_141_2","doi-asserted-by":"crossref","first-page":"59","DOI":"10.1109\/ICTAI.2009.25","volume-title":"2009 21st IEEE International Conference on Tools with Artificial Intelligence","author":"Seliya Naeem","year":"2009","unstructured":"Naeem Seliya, Taghi M. Khoshgoftaar, and Jason Van Hulse. 2009. A study on the relationships of classifier performance metrics. In 2009 21st IEEE International Conference on Tools with Artificial Intelligence. IEEE, 59\u201366."},{"key":"e_1_3_2_142_2","article-title":"Continual-Learning-as-a-Service (CLaaS): On-demand efficient adaptation of predictive models","author":"Semola Rudy","year":"2022","unstructured":"Rudy Semola, Vincenzo Lomonaco, and Davide Bacciu. 2022. Continual-Learning-as-a-Service (CLaaS): On-demand efficient adaptation of predictive models. arXiv preprint arXiv:2206.06957 (2022).","journal-title":"arXiv preprint arXiv:2206.06957"},{"key":"e_1_3_2_143_2","article-title":"Adversarially robust transfer learning","author":"Shafahi Ali","year":"2019","unstructured":"Ali Shafahi, Parsa Saadatpanah, Chen Zhu, Amin Ghiasi, Christoph Studer, David Jacobs, and Tom Goldstein. 2019. Adversarially robust transfer learning. arXiv preprint arXiv:1905.08232 (2019).","journal-title":"arXiv preprint arXiv:1905.08232"},{"key":"e_1_3_2_144_2","doi-asserted-by":"publisher","DOI":"10.1145\/3297662.3365807"},{"key":"e_1_3_2_145_2","first-page":"1644","volume-title":"2020 IEEE International Conference on Big Data (Big Data\u201920)","author":"Shrivastava Shrey","year":"2020","unstructured":"Shrey Shrivastava, Dhaval Patel, Nianjun Zhou, Arun Iyengar, and Anuradha Bhamidipaty. 2020. DQLearn: A toolkit for structured data quality learning. In 2020 IEEE International Conference on Big Data (Big Data\u201920). IEEE, 1644\u20131653."},{"key":"e_1_3_2_146_2","first-page":"626","volume-title":"2020 19th IEEE International Conference on Machine Learning and Applications (ICMLA\u201920)","author":"Silva Lucas Cardoso","year":"2020","unstructured":"Lucas Cardoso Silva, Fernando Rezende Zagatti, Bruno Silva Sette, Lucas Nildaimon dos Santos Silva, Daniel Lucr\u00e9dio, Diego Furtado Silva, and Helena de Medeiros Caseli. 2020. Benchmarking machine learning solutions in production. In 2020 19th IEEE International Conference on Machine Learning and Applications (ICMLA\u201920). IEEE, 626\u2013633."},{"key":"e_1_3_2_147_2","first-page":"259","volume-title":"Data Science Solutions on Azure","author":"Soh Julian","year":"2020","unstructured":"Julian Soh and Priyanshi Singh. 2020. Machine learning operations. In Data Science Solutions on Azure. Springer, 259\u2013279."},{"key":"e_1_3_2_148_2","article-title":"CheXstray: Real-time multi-modal data concordance for drift detection in medical imaging AI","author":"Soin Arjun","year":"2022","unstructured":"Arjun Soin, Jameson Merkow, Jin Long, Joesph Paul Cohen, Smitha Saligrama, Stephen Kaiser, Steven Borg, Ivan Tarapov, and Matthew P. Lungren. 2022. CheXstray: Real-time multi-modal data concordance for drift detection in medical imaging AI. arXiv preprint arXiv:2202.02833 (2022).","journal-title":"arXiv preprint arXiv:2202.02833"},{"key":"e_1_3_2_149_2","doi-asserted-by":"publisher","DOI":"10.1145\/3319535.3354211"},{"key":"e_1_3_2_150_2","article-title":"A segment-based drift adaptation method for data streams","author":"Song Yiliao","year":"2021","unstructured":"Yiliao Song, Jie Lu, Anjin Liu, Haiyan Lu, and Guangquan Zhang. 2021. A segment-based drift adaptation method for data streams. IEEE Transactions on Neural Networks and Learning Systems (2021).","journal-title":"IEEE Transactions on Neural Networks and Learning Systems"},{"issue":"9","key":"e_1_3_2_151_2","doi-asserted-by":"crossref","first-page":"1071","DOI":"10.1080\/17460441.2021.1932812","article-title":"The machine learning life cycle and the cloud: Implications for drug discovery","volume":"16","author":"Spjuth Ola","year":"2021","unstructured":"Ola Spjuth, Jens Frid, and Andreas Hellander. 2021. The machine learning life cycle and the cloud: Implications for drug discovery. Expert Opinion on Drug Discovery 16, 9 (2021), 1071\u20131079.","journal-title":"Expert Opinion on Drug Discovery"},{"issue":"11","key":"e_1_3_2_152_2","doi-asserted-by":"crossref","first-page":"4397","DOI":"10.1109\/TSE.2021.3119012","article-title":"Building maintainable software using abstraction layering","volume":"48","author":"Spray John","year":"2021","unstructured":"John Spray, Roopak Sinha, Arnab Sen, and Xingbin Cheng. 2021. Building maintainable software using abstraction layering. IEEE Transactions on Software Engineering 48, 11 (2021), 4397\u20134410.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_3_2_153_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2023.111615"},{"key":"e_1_3_2_154_2","first-page":"3054","volume-title":"2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC\u201913)","author":"Sugimoto Masahiro","year":"2013","unstructured":"Masahiro Sugimoto, Masahiro Takada, and Masakazu Toi. 2013. Comparison of robustness against missing values of alternative decision tree and multiple logistic regression for predicting clinical data in primary breast cancer. In 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC\u201913). IEEE, 3054\u20133057."},{"key":"e_1_3_2_155_2","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/9780262017091.001.0001"},{"key":"e_1_3_2_156_2","article-title":"Direct importance estimation with model selection and its application to covariate shift adaptation","volume":"20","author":"Sugiyama Masashi","year":"2007","unstructured":"Masashi Sugiyama, Shinichi Nakajima, Hisashi Kashima, Paul Buenau, and Motoaki Kawanabe. 2007. Direct importance estimation with model selection and its application to covariate shift adaptation. Advances in Neural Information Processing Systems 20 (2007).","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_157_2","first-page":"0453","volume-title":"2022 IEEE 12th Annual Computing and Communication Workshop and Conference (CCWC\u201922)","author":"Symeonidis Georgios","year":"2022","unstructured":"Georgios Symeonidis, Evangelos Nerantzis, Apostolos Kazakis, and George A. Papakostas. 2022. MLOps-definitions, tools and challenges. In 2022 IEEE 12th Annual Computing and Communication Workshop and Conference (CCWC\u201922). IEEE, 0453\u20130460."},{"key":"e_1_3_2_158_2","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1109\/SYNASC51798.2020.00015","volume-title":"2020 22nd International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC\u201920)","author":"Tamburri Damian A.","year":"2020","unstructured":"Damian A. Tamburri. 2020. Sustainable MLOps: Trends and challenges. In 2020 22nd International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC\u201920). IEEE, 17\u201323."},{"key":"e_1_3_2_159_2","article-title":"The risks of machine learning systems","author":"Tan Samson","year":"2022","unstructured":"Samson Tan, Araz Taeihagh, and Kathy Baxter. 2022. The risks of machine learning systems. arXiv preprint arXiv:2204.09852 (2022).","journal-title":"arXiv preprint arXiv:2204.09852"},{"key":"e_1_3_2_160_2","article-title":"MLOps: A taxonomy and a methodology","author":"Testi Matteo","year":"2022","unstructured":"Matteo Testi, Matteo Ballabio, Emanuele Frontoni, Giulio Iannello, Sara Moccia, Paolo Soda, and Gennaro Vessio. 2022. MLOps: A taxonomy and a methodology. IEEE Access (2022).","journal-title":"IEEE Access"},{"key":"e_1_3_2_161_2","doi-asserted-by":"publisher","DOI":"10.1145\/2487575.2487629"},{"key":"e_1_3_2_162_2","first-page":"278","volume-title":"2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA\u201921)","author":"Thulasidasan Sunil","year":"2021","unstructured":"Sunil Thulasidasan, Sushil Thapa, Sayera Dhaubhadel, Gopinath Chennupati, Tanmoy Bhattacharya, and Jeff Bilmes. 2021. An effective baseline for robustness to distributional shift. In 2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA\u201921). IEEE, 278\u2013285."},{"key":"e_1_3_2_163_2","volume-title":"Introducing MLOps","author":"Treveil Mark","year":"2020","unstructured":"Mark Treveil, Nicolas Omont, Cl\u00e9ment Stenac, Kenji Lefevre, Du Phan, Joachim Zentici, Adrien Lavoillotte, Makoto Miyazaki, and Lynn Heidmann. 2020. Introducing MLOps. O\u2019Reilly Media."},{"key":"e_1_3_2_164_2","article-title":"Robustness may be at odds with accuracy","author":"Tsipras Dimitris","year":"2018","unstructured":"Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. 2018. Robustness may be at odds with accuracy. arXiv preprint arXiv:1805.12152 (2018).","journal-title":"arXiv preprint arXiv:1805.12152"},{"key":"e_1_3_2_165_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.inffus.2006.11.002"},{"key":"e_1_3_2_166_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10845-019-01531-7"},{"key":"e_1_3_2_167_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0224365"},{"key":"e_1_3_2_168_2","first-page":"1","volume-title":"Proceedings of the 1st Workshop on Data Management for End-to-End Machine Learning","author":"Weide Tom van der","year":"2017","unstructured":"Tom van der Weide, Dimitris Papadopoulos, Oleg Smirnov, Michal Zielinski, and Tim van Kasteren. 2017. Versioning for end-to-end machine learning pipelines. In Proceedings of the 1st Workshop on Data Management for End-to-End Machine Learning. 1\u20139."},{"key":"e_1_3_2_169_2","first-page":"1","volume-title":"NIPS Workshop LearningSys","author":"Vartak Manasi","year":"2015","unstructured":"Manasi Vartak, Pablo Ortiz, Kathryn Siegel, Harihar Subramanyam, Samuel Madden, and Matei Zaharia. 2015. Supporting fast iteration in model building. In NIPS Workshop LearningSys. 1\u20136."},{"issue":"6","key":"e_1_3_2_170_2","doi-asserted-by":"crossref","first-page":"giab030","DOI":"10.1093\/gigascience\/giab030","article-title":"Driftage: A multi-agent system framework for concept drift detection","volume":"10","author":"Vieira Diogo Munaro","year":"2021","unstructured":"Diogo Munaro Vieira, Chrystinne Fernandes, Carlos Lucena, and S\u00e9rgio Lifschitz. 2021. Driftage: A multi-agent system framework for concept drift detection. GigaScience 10, 6 (2021), giab030.","journal-title":"GigaScience"},{"key":"e_1_3_2_171_2","unstructured":"St\u00e9phan Vincent-Lancrin and Reyer van der Vlies. 2020. Trustworthy Artificial Intelligence (AI) in Education: Promises and Challenges. OECD Education Working Papers No. 218 OECD Publishing Paris."},{"key":"e_1_3_2_172_2","first-page":"78","volume-title":"Fifth International Conference on the Innovative Computing Technology (intech\u201915)","author":"Virmani Manish","year":"2015","unstructured":"Manish Virmani. 2015. Understanding DevOps & bridging the gap from continuous integration to continuous delivery. In Fifth International Conference on the Innovative Computing Technology (intech\u201915). IEEE, 78\u201382."},{"key":"e_1_3_2_173_2","article-title":"Machine learning that matters","author":"Wagstaff Kiri","year":"2012","unstructured":"Kiri Wagstaff. 2012. Machine learning that matters. arXiv preprint arXiv:1206.4656 (2012).","journal-title":"arXiv preprint arXiv:1206.4656"},{"key":"e_1_3_2_174_2","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2022.3178128"},{"key":"e_1_3_2_175_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2019.2962152"},{"key":"e_1_3_2_176_2","doi-asserted-by":"publisher","DOI":"10.1145\/3448248"},{"key":"e_1_3_2_177_2","doi-asserted-by":"crossref","first-page":"335","DOI":"10.1016\/B978-0-12-804291-5.00009-X","volume-title":"Data Mining: Practical Machine Learning Tools and Techniques (4th ed.)","author":"Witten Ian H.","year":"2017","unstructured":"Ian H. Witten, Eibe Frank, Mark A. Hall, and Christopher J. Pal. 2017. Chapter 9 - Probabilistic methods. In Data Mining: Practical Machine Learning Tools and Techniques (4th ed.), Ian H. Witten, Eibe Frank, Mark A. Hall, and Christopher J. Pal (Eds.). Morgan Kaufmann, 335\u2013416."},{"key":"e_1_3_2_178_2","first-page":"163","volume-title":"International Conference on Society 5.0","author":"Woitsch Robert","year":"2021","unstructured":"Robert Woitsch, Wilfrid Utz, Anna Sumereder, Bernhard Dieber, Benjamin Breiling, Laura Crompton, Michael Funk, Karin Bruckm\u00fcller, and Stefan Schumann. 2021. Collaborative model-based process assessment for trustworthy AI in robotic platforms. In International Conference on Society 5.0. Springer, 163\u2013174."},{"key":"e_1_3_2_179_2","doi-asserted-by":"publisher","DOI":"10.1109\/JIOT.2020.2996784"},{"key":"e_1_3_2_180_2","first-page":"10355","volume-title":"International Conference on Machine Learning","author":"Wu Yinjun","year":"2020","unstructured":"Yinjun Wu, Edgar Dobriban, and Susan Davidson. 2020. DeltaGrad: Rapid retraining of machine learning models. In International Conference on Machine Learning. PMLR, 10355\u201310366."},{"key":"e_1_3_2_181_2","doi-asserted-by":"publisher","DOI":"10.1145\/3318464.3380571"},{"key":"e_1_3_2_182_2","first-page":"2691","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Xiao Tong","year":"2015","unstructured":"Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. 2015. Learning from massive noisy labeled data for image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2691\u20132699."},{"key":"e_1_3_2_183_2","first-page":"1","volume-title":"Proceedings of the Second Workshop on Data Management for End-to-end Machine Learning","author":"Xin Doris","year":"2018","unstructured":"Doris Xin, Litian Ma, Jialin Liu, Stephen Macke, Shuchen Song, and Aditya Parameswaran. 2018. Accelerating human-in-the-loop machine learning: Challenges and opportunities. In Proceedings of the Second Workshop on Data Management for End-to-end Machine Learning. 1\u20134."},{"key":"e_1_3_2_184_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2023.127061"},{"key":"e_1_3_2_185_2","first-page":"2327","volume-title":"30th USENIX Security Symposium (USENIX Security\u201921)","author":"Yang Limin","year":"2021","unstructured":"Limin Yang, Wenbo Guo, Qingying Hao, Arridhana Ciptadi, Ali Ahmadzadeh, Xinyu Xing, and Gang Wang. 2021. CADE: Detecting and explaining concept drift samples for security applications. In 30th USENIX Security Symposium (USENIX Security\u201921). 2327\u20132344."},{"key":"e_1_3_2_186_2","first-page":"8588","article-title":"A closer look at accuracy vs. robustness","volume":"33","author":"Yang Yao-Yuan","year":"2020","unstructured":"Yao-Yuan Yang, Cyrus Rashtchian, Hongyang Zhang, Russ R. Salakhutdinov, and Kamalika Chaudhuri. 2020. A closer look at accuracy vs. robustness. Advances in Neural Information Processing Systems 33 (2020), 8588\u20138601.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_187_2","first-page":"022022","volume-title":"Journal of Physics: Conference Series","volume":"1168","author":"Ying Xue","year":"2019","unstructured":"Xue Ying. 2019. An overview of overfitting and its solutions. In Journal of Physics: Conference Series, Vol. 1168. IOP Publishing, 022022."},{"key":"e_1_3_2_188_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jjimei.2022.100070"},{"key":"e_1_3_2_189_2","article-title":"DataOps-4G: On supporting generalists in data quality discovery","author":"Yu Shaochen","year":"2022","unstructured":"Shaochen Yu, Tianwa Chen, Lei Han, Gianluca Demartini, and Shazia Sadiq. 2022. DataOps-4G: On supporting generalists in data quality discovery. IEEE Transactions on Knowledge and Data Engineering (2022).","journal-title":"IEEE Transactions on Knowledge and Data Engineering"},{"issue":"4","key":"e_1_3_2_190_2","first-page":"39","article-title":"Accelerating the machine learning lifecycle with MLflow.","volume":"41","author":"Zaharia Matei","year":"2018","unstructured":"Matei Zaharia, Andrew Chen, Aaron Davidson, Ali Ghodsi, Sue Ann Hong, Andy Konwinski, Siddharth Murching, Tomas Nykodym, Paul Ogilvie, Mani Parkhe, et\u00a0al. 2018. Accelerating the machine learning lifecycle with MLflow. IEEE Data Eng. Bull. 41, 4 (2018), 39\u201345.","journal-title":"IEEE Data Eng. Bull."},{"key":"e_1_3_2_191_2","doi-asserted-by":"crossref","first-page":"206","DOI":"10.1109\/ICSA-C54293.2022.00047","volume-title":"2022 IEEE 19th International Conference on Software Architecture Companion (ICSA-C\u201922)","author":"Z\u00e1rate Gorka","year":"2022","unstructured":"Gorka Z\u00e1rate, Ra\u00fal Mi\u00f1\u00f3n, Josu D\u00edaz-de Arcaya, and Ana I. Torre-Bastida. 2022. K2E: Building MLOps environments for governing data and models catalogues while tracking versions. In 2022 IEEE 19th International Conference on Software Architecture Companion (ICSA-C\u201922). IEEE, 206\u2013209."},{"key":"e_1_3_2_192_2","first-page":"1","article-title":"Toward a safe MLOps process for the continuous development and safety assurance of ML-based systems in the railway domain","author":"Zeller Marc","year":"2024","unstructured":"Marc Zeller, Thomas Waschulzik, Reiner Schmid, and Claus Bahlmann. 2024. Toward a safe MLOps process for the continuous development and safety assurance of ML-based systems in the railway domain. AI and Ethics (2024), 1\u20138.","journal-title":"AI and Ethics"},{"key":"e_1_3_2_193_2","first-page":"3987","volume-title":"International Conference on Machine Learning","author":"Zenke Friedemann","year":"2017","unstructured":"Friedemann Zenke, Ben Poole, and Surya Ganguli. 2017. Continual learning through synaptic intelligence. In International Conference on Machine Learning. PMLR, 3987\u20133995."},{"key":"e_1_3_2_194_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2021.106775"},{"key":"e_1_3_2_195_2","doi-asserted-by":"publisher","DOI":"10.1109\/TMI.2020.2973595"},{"key":"e_1_3_2_196_2","unstructured":"Yizhen Zhao. 2021. Machine Learning in Production: A Literature Review. Universiteit van Amsterdam Technical Report."},{"issue":"3","key":"e_1_3_2_197_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3446905","article-title":"On the impact of sample duplication in machine-learning-based Android malware detection","volume":"30","author":"Zhao Yanjie","year":"2021","unstructured":"Yanjie Zhao, Li Li, Haoyu Wang, Haipeng Cai, Tegawend\u00e9 F. Bissyand\u00e9, Jacques Klein, and John Grundy. 2021. On the impact of sample duplication in machine-learning-based Android malware detection. ACM Transactions on Software Engineering and Methodology (TOSEM) 30, 3 (2021), 1\u201338.","journal-title":"ACM Transactions on Software Engineering and Methodology (TOSEM)"},{"key":"e_1_3_2_198_2","first-page":"914","article-title":"Bayesian adaptation for covariate shift","volume":"34","author":"Zhou Aurick","year":"2021","unstructured":"Aurick Zhou and Sergey Levine. 2021. Bayesian adaptation for covariate shift. Advances in Neural Information Processing Systems 34 (2021), 914\u2013927.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_199_2","article-title":"Domain generalization in vision: A survey","author":"Zhou Kaiyang","year":"2021","unstructured":"Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. 2021. Domain generalization in vision: A survey. arXiv preprint arXiv:2103.02503 (2021).","journal-title":"arXiv preprint arXiv:2103.02503"},{"key":"e_1_3_2_200_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2017.01.026"},{"key":"e_1_3_2_201_2","article-title":"Time series prediction method of industrial process with limited data based on transfer learning","author":"Zhou Xiaofeng","year":"2022","unstructured":"Xiaofeng Zhou, Naiju Zhai, Shuai Li, and Haibo Shi. 2022. Time series prediction method of industrial process with limited data based on transfer learning. IEEE Transactions on Industrial Informatics (2022).","journal-title":"IEEE Transactions on Industrial Informatics"},{"issue":"8","key":"e_1_3_2_202_2","doi-asserted-by":"crossref","first-page":"nwac123","DOI":"10.1093\/nsr\/nwac123","article-title":"Open-environment machine learning","volume":"9","author":"Zhou Zhi-Hua","year":"2022","unstructured":"Zhi-Hua Zhou. 2022. Open-environment machine learning. National Science Review 9, 8 (2022), nwac123.","journal-title":"National Science Review"},{"key":"e_1_3_2_203_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICDMW.2010.49"}],"container-title":["ACM Computing Surveys"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3708497","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3708497","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T01:09:54Z","timestamp":1750295394000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3708497"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,1,22]]},"references-count":202,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2025,5,31]]}},"alternative-id":["10.1145\/3708497"],"URL":"https:\/\/doi.org\/10.1145\/3708497","relation":{},"ISSN":["0360-0300","1557-7341"],"issn-type":[{"value":"0360-0300","type":"print"},{"value":"1557-7341","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,1,22]]},"assertion":[{"value":"2022-10-14","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-11-18","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-01-22","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}