{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,5]],"date-time":"2026-06-05T16:29:17Z","timestamp":1780676957433,"version":"3.54.1"},"reference-count":25,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2023,8,29]],"date-time":"2023-08-29T00:00:00Z","timestamp":1693267200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,8,29]],"date-time":"2023-08-29T00:00:00Z","timestamp":1693267200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100011950","name":"ITEA","doi-asserted-by":"publisher","award":["ITEA-2021-20219-IML4E"],"award-info":[{"award-number":["ITEA-2021-20219-IML4E"]}],"id":[{"id":"10.13039\/100011950","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100002765","name":"Bundesministerium f\u00fcr Wirtschaft und Technologie","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100002765","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Fraunhofer-Institut f\u00fcr Offene Kommunikationssysteme FOKUS"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Rev Socionetwork Strat"],"published-print":{"date-parts":[[2023,10]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Machine learning systems have gained widespread adoption across various industries. This includes highly regulated ones that need to match certain quality requirements based on a given risk exposure. The MLOps paradigm, following a similar approach to DevOps, promises major improvements in quality and speed, with a focus on deploying ML models at a fast pace with high quality on an automated basis. However, traditional point-in-time certifications with manual audits are inadequate for MLOps setups due to frequent changes to the ML system. To overcome this challenge, we propose Continuous Audit-Based Certification (CABC), which uses automated audits to issue or revoke certificates based on an automated assessment of artifacts from the MLOps lifecycle. Our approach utilizes artifacts from the MLOps lifecycle for quality measurements based on standards such as ISO 25012. We propose a risk-based measurement selection, an audit API for standardized retrieval of data for measurement, a tamper-proof data collection process, and an architecture for separation of duties in the certification process. CABC aims to improve efficiency, enhance trust in the ML system, and support highly regulated industries in achieving their quality goals.<\/jats:p>","DOI":"10.1007\/s12626-023-00148-w","type":"journal-article","created":{"date-parts":[[2023,8,29]],"date-time":"2023-08-29T09:02:22Z","timestamp":1693299742000},"page":"255-273","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":11,"title":["Towards a Risk-Based Continuous Auditing-Based Certification for Machine Learning"],"prefix":"10.1007","volume":"17","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-4900-8535","authenticated-orcid":false,"given":"Dorian","family":"Knoblauch","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6701-4720","authenticated-orcid":false,"given":"J\u00fcrgen","family":"Gro\u00dfmann","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2023,8,29]]},"reference":[{"key":"148_CR1","unstructured":"Ki-pruefkatalog. (2021). https:\/\/www.iais.fraunhofer.de\/content\/dam\/iais\/fb\/Kuenstliche_intelligenz\/ki-pruefkatalog\/202107_KI-Pruefkatalog.pdf. Accessed 27 Feb 2023"},{"key":"148_CR2","doi-asserted-by":"publisher","first-page":"79","DOI":"10.1007\/978-1-4842-6549-9_3","volume-title":"What Is MLOps?","author":"S Alla","year":"2021","unstructured":"Alla, S., & Adari, S. K. (2021). What Is MLOps? (pp. 79\u2013124). Berkeley, CA: Apress. https:\/\/doi.org\/10.1007\/978-1-4842-6549-9_3"},{"key":"148_CR3","first-page":"2","volume":"1","author":"S Barocas","year":"2017","unstructured":"Barocas, S., Hardt, M., & Narayanan, A. (2017). Fairness in machine learning. Nips Tutorial, 1, 2.","journal-title":"Nips Tutorial"},{"key":"148_CR4","unstructured":"Breck, E., Polyzotis, N., Roy, S., Whang, S., & Zinkevich, M. (2019). Data validation for machine learning. In: MLSys"},{"key":"148_CR5","doi-asserted-by":"publisher","unstructured":"Calefato, F., Lanubile, F., & Quaranta, L. (2022). A preliminary investigation of mlops practices in github. In: Proceedings of the 16th ACM \/ IEEE International Symposium on Empirical Software Engineering and Measurement. p. 283-288. ESEM \u201922, Association for Computing Machinery, New York, NY, USA https:\/\/doi.org\/10.1145\/3544902.3546636","DOI":"10.1145\/3544902.3546636"},{"key":"148_CR6","doi-asserted-by":"publisher","unstructured":"Garg, S., Pundir, P., Rathee, G., Gupta, P., Garg, S., & Ahlawat, S. (2021). On continuous integration\/continuous delivery for automated deployment of machine learning models using mlops. In: 2021 IEEE Fourth International Conference on Artificial Intelligence and Knowledge Engineering (AIKE). pp. 25\u201328. https:\/\/doi.org\/10.1109\/AIKE52691.2021.00010","DOI":"10.1109\/AIKE52691.2021.00010"},{"key":"148_CR7","doi-asserted-by":"publisher","unstructured":"Gokarna, M., & Singh, R. (2021). Devops: a historical review and future works. In: 2021 International Conference on Computing, Communication, and Intelligent Systems (ICCCIS). pp. 366\u2013371. https:\/\/doi.org\/10.1109\/ICCCIS51004.2021.9397235","DOI":"10.1109\/ICCCIS51004.2021.9397235"},{"issue":"5","key":"148_CR8","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/s42979-021-00726-1","volume":"2","author":"T Granlund","year":"2021","unstructured":"Granlund, T., Stirbu, V., & Mikkonen, T. (2021). Towards regulatory-compliant mlops: Oravizio\u2019s journey from a machine learning experiment to a deployed certified medical product. SN Computer Science, 2(5), 1\u201314.","journal-title":"SN Computer Science"},{"key":"148_CR9","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2021.110938","volume":"176","author":"F Gualo","year":"2021","unstructured":"Gualo, F., Rodriguez, M., Verdugo, J., Caballero, I., & Piattini, M. (2021). Data quality certification using ISO\/IEC 25012: industrial experiences. Journal of Systems and Software, 176, 110938. https:\/\/doi.org\/10.1016\/j.jss.2021.110938","journal-title":"Journal of Systems and Software"},{"key":"148_CR10","unstructured":"Hardt, M., Recht, B., & Singer, Y. (2016). Train faster, generalize better: stability of stochastic gradient descent. In: International conference on machine learning. pp. 1225\u20131234. PMLR"},{"issue":"9","key":"148_CR11","doi-asserted-by":"publisher","first-page":"1263","DOI":"10.1109\/TKDE.2008.239","volume":"21","author":"H He","year":"2009","unstructured":"He, H., & Garcia, E. A. (2009). Learning from imbalanced data. IEEE Transactions on Knowledge and Data Engineering, 21(9), 1263\u20131284.","journal-title":"IEEE Transactions on Knowledge and Data Engineering"},{"key":"148_CR12","unstructured":"For Information Security\u00a0(BSI). (2021). F.O.: Kriterienkatalog f\u00fcr ki-cloud-dienste - aic4. https:\/\/www.bsi.bund.de\/SharedDocs\/Downloads\/EN\/BSI\/CloudComputing\/AIC4\/AI-Cloud-Service-Compliance-Criteria-Catalogue_AIC4.pdf?__blob=publicationFile &v=4. Accessed 27 Feb 2023"},{"key":"148_CR13","unstructured":"International Organization for Standardization: Information technology - artificial intelligence - guidance on risk management. (2023). https:\/\/www.iso.org\/standard\/77304.html"},{"key":"148_CR14","unstructured":"ISO\/IEC: Systems and software engineering \u2013 systems and software quality requirements and evaluation (SQuaRE) \u2013 Data quality model. (2008). ISO\/IEC 25012, International Organization for Standardization, Geneva, Switzerland"},{"key":"148_CR15","unstructured":"ISO\/IEC: ISO\/IEC 25024, Systems and Software Engineering - Systems and Software Quality Requirements and Evaluation (SQuaRE) - Measurement of Data Quality. No. 25024, ISO, Geneva, Switzerland. (2015). https:\/\/books.google.de\/books?id=wYXKtAEACAAJ"},{"key":"148_CR16","doi-asserted-by":"crossref","unstructured":"Knoblauch, D., & Banse, C. (2019). Reducing implementation efforts in continuous auditing certification via an audit API. In: 2019 IEEE 28th International Conference on Enabling Technologies: Infrastructure for Collaborative Enterprises (WETICE). pp. 88\u201392. IEEE","DOI":"10.1109\/WETICE.2019.00025"},{"key":"148_CR17","unstructured":"Knoblauch, D., & Gro\u00dfmann, J. (2022). Towards continuous audit-based certification for mlops. In: Proceedings of the International Workshop on AI Compliance Mechanism. WAICOM 2022 Co-Chairs"},{"key":"148_CR18","unstructured":"Knoblauch, D., Gro\u00dfmann, J., Strick, L., & Pannetrat, A. (2019). Europ\u00e4isches rahmenwerk f\u00fcr continuous auditing based certification. In: Tagungsband zum 16. IT-Sicherheitskongress des BSI. pp. 495\u2013504. SecuMedia"},{"key":"148_CR19","doi-asserted-by":"publisher","first-page":"772","DOI":"10.1016\/j.procs.2021.01.327","volume":"180","author":"C Lettner","year":"2021","unstructured":"Lettner, C., Stumptner, R., Fragner, W., Rauchenzauner, F., & Ehrlinger, L. (2021). Daql 2.0: Measure data quality based on entity models. Procedia Computer Science, 180, 772\u2013777. https:\/\/doi.org\/10.1016\/j.procs.2021.01.327","journal-title":"Procedia Computer Science"},{"key":"148_CR20","unstructured":"Moreschi, S., Recupito, G., Lenarduzzi, V., Palomba, F., Hastbacka, D., & Taibi, D. (2023). Toward end-to-end mlops tools map: a preliminary study based on a multivocal literature review"},{"issue":"1","key":"148_CR21","doi-asserted-by":"publisher","first-page":"161","DOI":"10.1007\/s10444-004-7634-z","volume":"25","author":"S Mukherjee","year":"2006","unstructured":"Mukherjee, S., Niyogi, P., Poggio, T., & Rifkin, R. (2006). Learning theory: stability is sufficient for generalization and necessary and sufficient for consistency of empirical risk minimization. Advances in Computational Mathematics, 25(1), 161\u2013193.","journal-title":"Advances in Computational Mathematics"},{"key":"148_CR22","unstructured":"Rauber, J., Brendel, W., & Bethge, M. (2017). Foolbox v0.8.0: A python toolbox to benchmark the robustness of machine learning models. CoRR abs\/1707.04131, http:\/\/arxiv.org\/abs\/1707.04131"},{"issue":"3","key":"148_CR23","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0118432","volume":"10","author":"T Saito","year":"2015","unstructured":"Saito, T., & Rehmsmeier, M. (2015). The precision-recall plot is more informative than the roc plot when evaluating binary classifiers on imbalanced datasets. PloS One, 10(3), e0118432.","journal-title":"PloS One"},{"issue":"12","key":"148_CR24","doi-asserted-by":"publisher","first-page":"1781","DOI":"10.14778\/3229863.3229867","volume":"11","author":"S Schelter","year":"2018","unstructured":"Schelter, S., Lange, D., Schmidt, P., Celikel, M., Biessmann, F., & Grafberger, A. (2018). Automating large-scale data quality verification. Proceedings of the VLDB Endowment, 11(12), 1781\u20131794. https:\/\/doi.org\/10.14778\/3229863.3229867","journal-title":"Proceedings of the VLDB Endowment"},{"key":"148_CR25","unstructured":"Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., & Crespo, J. (2015). Hidden technical debt in machine learning systems. In: Advances in neural information processing systems. pp. 2503\u20132511"}],"container-title":["The Review of Socionetwork Strategies"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s12626-023-00148-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s12626-023-00148-w\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s12626-023-00148-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,10,27]],"date-time":"2023-10-27T07:36:17Z","timestamp":1698392177000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s12626-023-00148-w"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,8,29]]},"references-count":25,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2023,10]]}},"alternative-id":["148"],"URL":"https:\/\/doi.org\/10.1007\/s12626-023-00148-w","relation":{},"ISSN":["2523-3173","1867-3236"],"issn-type":[{"value":"2523-3173","type":"print"},{"value":"1867-3236","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,8,29]]},"assertion":[{"value":"4 March 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"1 August 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"29 August 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors of this paper, Dorian Knoblauch and J\u00fcrgen Gro\u00dfmann, declare the following potential conflicts of interest with respect to the research, authorship, and\/or publication of this article: - The work presented in this paper was partially funded by the German Federal Ministry of Education and Research (BMBF) under grant agreement ITEA-2021-20219-IML4E as well as the German Federal Ministry for Economic Affairs and Energy (BMWI) project KI-LOK. The authors confirm that this work is free from any other potential conflicts of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of Interest"}}]}}