{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,10]],"date-time":"2026-10-10T04:49:27Z","timestamp":1791607767325,"version":"4.3.4"},"reference-count":48,"publisher":"MDPI AG","issue":"19","license":[{"start":{"date-parts":[[2026,10,2]],"date-time":"2026-10-02T00:00:00Z","timestamp":1790899200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100004329","name":"The Slovenian Research and Innovation Agency","doi-asserted-by":"publisher","award":["P2-0057"],"award-info":[{"award-number":["P2-0057"]}],"id":[{"id":"10.13039\/501100004329","id-type":"DOI","asserted-by":"publisher"}]},{"award":["P2-0057"],"award-info":[{"award-number":["P2-0057"]}],"id":[{"id":"https:\/\/ror.org\/059bp8k51","id-type":"ROR","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Appl. Sci.","Applied Sciences"],"abstract":"<jats:p>Artificial intelligence (AI) components are increasingly embedded in software, challenging established assumptions of software testing practice. This article presents a systematic mapping review of testing approaches for AI-based systems, grounded in ISO\/IEC 29119 and ISO\/IEC TR 29119-11. The contribution is a structured, traceable mapping from challenges to approaches at the level of AI-based systems. From 1212 identified records, 26 studies were included in the final mapping, complemented by seven ISO\/IEC and ISO\/IEC\/IEEE normative documents that were analysed for normative guidance rather than as empirical evidence. The review identifies five recurring challenge categories\u2014weak specifications and the test oracle problem, data dependence and test inputs, non-functional and socio-technical properties, complex and dynamic environments, and continuous evolution and adaptation. It synthesises five corresponding approach categories: metric-based and comparative testing, scenario-based testing and simulation, architecture-driven testing, continuous validation and post-deployment testing, and standards-aligned and process-oriented testing. Across the mapped studies, classical testing concepts are reinterpreted and extended rather than replaced. However, evidence is unevenly distributed across challenge\u2013approach combinations: architecture-driven testing is represented by two primary studies, while socio-technical properties are addressed only by two secondary studies in the standards-aligned and process-oriented category. Seven of the twenty-five challenge\u2013approach combinations remain empty.<\/jats:p>","DOI":"10.3390\/app16199787","type":"journal-article","created":{"date-parts":[[2026,10,2]],"date-time":"2026-10-02T10:14:56Z","timestamp":1790936096000},"page":"9787","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Testing AI-Based Systems: A Systematic Mapping of Challenges and Approaches"],"prefix":"10.3390","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0009-0002-3215-8593","authenticated-orcid":false,"given":"Lidija","family":"Vincekovi\u010d","sequence":"first","affiliation":[{"name":"Faculty of Electrical Engineering and Computer Science, University of Maribor, 2000 Maribor, Slovenia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5477-4747","authenticated-orcid":false,"given":"Luka","family":"Pavli\u010d","sequence":"additional","affiliation":[{"name":"Faculty of Electrical Engineering and Computer Science, University of Maribor, 2000 Maribor, Slovenia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6518-5876","authenticated-orcid":false,"given":"Tina","family":"Berani\u010d","sequence":"additional","affiliation":[{"name":"Faculty of Electrical Engineering and Computer Science, University of Maribor, 2000 Maribor, Slovenia"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2026,10,2]]},"reference":[{"key":"ref_1","unstructured":"(2022). Software and Systems Engineering\u2014Software Testing\u2014Part 1: General Concepts (Standard No. ISO\/IEC\/IEEE 29119-1:2022)."},{"key":"ref_2","unstructured":"European Parliament, and Council of the European Union (2024). Regulation (EU) 2024\/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence and amending Regulations (EC) No 300\/2008, (EU) No 167\/2013, (EU) No 168\/2013, (EU) 2018\/858, (EU) 2018\/1139 and (EU) 2019\/2144 and Directives 2014\/90\/EU, (EU) 2016\/797 and (EU) 2020\/1828 (Artificial Intelligence Act) (Text with EEA Relevance). Off. J. Eur. Union, L 2024\/1689, 1\u2013144. Available online: https:\/\/eur-lex.europa.eu\/eli\/reg\/2024\/1689\/oj\/eng."},{"key":"ref_3","unstructured":"(2020). Software and Systems Engineering\u2014Software Testing\u2014Part 11: Guidelines on the Testing of AI-Based Systems (Standard No. ISO\/IEC TR 29119-11:2020)."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"911","DOI":"10.1109\/TSE.2024.3368208","article-title":"Software Testing with Large Language Models: Survey, Landscape, and Vision","volume":"50","author":"Wang","year":"2024","journal-title":"IEEE Trans. Softw. Eng."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"156","DOI":"10.1186\/s40537-024-01028-y","article-title":"Quality assurance strategies for machine learning applications in big data analytics: An overview","volume":"11","year":"2024","journal-title":"J. Big Data"},{"key":"ref_6","first-page":"1","article-title":"Software engineering for AI-based systems: A survey","volume":"31","author":"Bogner","year":"2022","journal-title":"ACM Trans. Softw. Eng. Methodol."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/TSE.2019.2962027","article-title":"Machine learning testing: Survey, landscapes and horizons","volume":"48","author":"Zhang","year":"2022","journal-title":"IEEE Trans. Softw. Eng."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"5193","DOI":"10.1007\/s10664-020-09881-0","article-title":"Testing machine learning-based systems: A systematic mapping","volume":"25","author":"Riccio","year":"2020","journal-title":"Empir. Softw. Eng."},{"key":"ref_9","first-page":"56","article-title":"Testing Artificial Intelligence System Towards Safety and Robustness: State of the Art","volume":"47","author":"Wu","year":"2020","journal-title":"IAENG Int. J. Comput. Sci."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1145\/3652155","article-title":"Fairness Testing: A Comprehensive Survey and Analysis of Trends","volume":"33","author":"Chen","year":"2024","journal-title":"ACM Trans. Softw. Eng. Methodol."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"e70032","DOI":"10.1111\/coin.70032","article-title":"Safety Monitoring of Machine Learning Perception Functions: A Survey","volume":"41","author":"Ferreira","year":"2025","journal-title":"Comput. Intell."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"107733","DOI":"10.1016\/j.infsof.2025.107733","article-title":"MLOps best practices, challenges and maturity models: A systematic literature review","volume":"183","author":"Zarour","year":"2025","journal-title":"Inf. Softw. Technol."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Yehudai, A., Eden, L., Li, A., Uziel, G., Zhao, Y., Bar-Haim, R., Cohan, A., and Shmueli-Scheuer, M. (2026, January 2\u20137). A Survey on Evaluation of LLM-based Agents. Proceedings of the Findings of the Association for Computational Linguistics: ACL 2026, San Diego, CA, USA.","DOI":"10.18653\/v1\/2026.findings-acl.1330"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Felderer, M., and Ramler, R. (2021). Quality assurance for AI-based systems: Overview and challenges. Software Quality: Future Perspectives on Software Engineering Quality (SWQD 2021), Springer. Lecture Notes in Business Information Processing.","DOI":"10.1007\/978-3-030-65854-0_3"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Marijan, D., Gotlieb, A., and Ahuja, M. (2019, January 4\u20139). Challenges of Testing Machine Learning Based Systems. Proceedings of the 2019 IEEE International Conference on Artificial Intelligence Testing (AITest), Newark, CA, USA.","DOI":"10.1109\/AITest.2019.00010"},{"key":"ref_16","first-page":"265","article-title":"Testing artificial intelligence by customers\u2019 needs","volume":"6","author":"Fehlmann","year":"2019","journal-title":"Athens J. Sci."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"De Angelis, E., De Angelis, G., and Proietti, M. (2023, January 17\u201320). A classification study on testing and verification of AI-based systems. Proceedings of the 2023 IEEE International Conference on Artificial Intelligence Testing (AITest), Athens, Greece.","DOI":"10.1109\/AITest58265.2023.00010"},{"key":"ref_18","unstructured":"(2021). Software and Systems Engineering\u2014Software Testing\u2014Part 4: Test Techniques (Standard No. ISO\/IEC\/IEEE 29119-4:2021)."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"n71","DOI":"10.1136\/bmj.n71","article-title":"The PRISMA 2020 statement: An updated guideline for reporting systematic reviews","volume":"372","author":"Page","year":"2021","journal-title":"BMJ"},{"key":"ref_20","unstructured":"(2021). Software and Systems Engineering\u2014Software Testing\u2014Part 3: Test Documentation (Standard No. ISO\/IEC\/IEEE 29119-3:2021)."},{"key":"ref_21","unstructured":"(2023). Systems and Software Engineering\u2014Systems and Software Quality Requirements and Evaluation (SQuaRE)\u2014Product Quality Model (Standard No. ISO\/IEC 25010:2023)."},{"key":"ref_22","unstructured":"(2023). Systems and Software Engineering\u2014Systems and Software Quality Requirements and Evaluation (SQuaRE)\u2014Quality-in-Use Model (Standard No. ISO\/IEC 25019:2023)."},{"key":"ref_23","unstructured":"(2024). Systems and Software Engineering\u2014Systems and Software Quality Requirements and Evaluation (SQuaRE)\u2014Guidance for Quality Evaluation of Artificial Intelligence (AI) Systems (Standard No. ISO\/IEC TS 25058:2024)."},{"key":"ref_24","unstructured":"(2022). Information Technology\u2014Artificial Intelligence\u2014Artificial Intelligence Concepts and Terminology (Standard No. ISO\/IEC 22989:2022)."},{"key":"ref_25","unstructured":"(2025). Artificial Intelligence\u2014Testing of AI \u2014Part 2: Overview of Testing AI Systems (Standard No. ISO\/IEC TS 42119-2:2025)."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., and Zimmermann, T. (2019, January 25\u201331). Software engineering for machine learning: A case study. Proceedings of the 2019 IEEE\/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), Montreal, QC, Canada.","DOI":"10.1109\/ICSE-SEIP.2019.00042"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Baylor, D., Breck, E., Cheng, H., Fiedel, N., Foo, C., Haque, Z., Haykal, S., Ispir, M., Jain, V., and Koc, L. (2017, January 13\u201317). TFX: A TensorFlow-based production-scale machine learning platform. Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), Halifax, NS, Canada.","DOI":"10.1145\/3097983.3098021"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Fremont, D., Kim, E., Pant, Y., Seshia, S., Acharya, A., Bruso, X., Wells, P., Lemke, S., Lu, Q., and Mehta, S. (2020, January 20\u201323). Formal Scenario-Based Testing of Autonomous Vehicles: From Simulation to the Real World. Proceedings of the 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), Rhodes, Greece.","DOI":"10.1109\/ITSC45102.2020.9294368"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"44","DOI":"10.1145\/2523813","article-title":"A survey on concept drift adaptation","volume":"46","author":"Gama","year":"2014","journal-title":"ACM Comput. Surv."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Grube, N., Massah, M., Tebbe, M., Wancura, P., Wiesbrock, H., Grossmann, J., and Kharma, S. (2024, January 15\u201318). On a systematic test of ML-based systems: Experiments on test statistics. Proceedings of the 2024 IEEE International Conference on Artificial Intelligence Testing (AITest), Shanghai, China.","DOI":"10.1109\/AITest62860.2024.00010"},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"112373","DOI":"10.1016\/j.jss.2025.112373","article-title":"Architectural tactics to achieve quality attributes of machine-learning-enabled systems: A systematic literature review","volume":"223","author":"Indykov","year":"2025","journal-title":"J. Syst. Softw."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"441","DOI":"10.1007\/s10462-018-9631-5","article-title":"Artificial intelligence test: A case study of intelligent vehicles","volume":"50","author":"Li","year":"2018","journal-title":"Artif. Intell. Rev."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Li, S., Guo, J., Lou, J., Fan, M., Liu, T., and Zhang, D. (2022, January 22\u201327). Testing machine learning systems in industry. Proceedings of the 44th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), Pittsburgh, PA, USA.","DOI":"10.1109\/ICSE-SEIP55303.2022.9793981"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Nayab, S., and Wotawa, F. (2025, January 16\u201320). Ontology-based testing for quality assurance of robotics applications. Proceedings of the 2025 25th International Conference on Software Quality, Reliability, and Security Companion (QRS-C), Hangzhou, China.","DOI":"10.1109\/QRS-C65679.2025.00114"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Neelofar, N., and Aleti, A. (2024, January 14\u201320). Towards reliable AI: Adequacy metrics for ensuring the quality of system-level testing of autonomous vehicles. Proceedings of the IEEE\/ACM 46th International Conference on Software Engineering (ICSE), Lisbon, Portugal.","DOI":"10.1145\/3597503.3623314"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Nishi, Y., Masuda, S., Ogawa, H., and Uetsuki, K. (2018, January 9\u201313). A test architecture for machine learning product. Proceedings of the 2018 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), V\u00e4ster\u00e5s, Sweden.","DOI":"10.1109\/ICSTW.2018.00060"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Osman, M.H., Kugele, S., and Shafaei, S. (2019, January 2\u20135). Run-Time Safety Monitoring Framework for AI-Based Systems: Automated Driving Cases. Proceedings of the 2019 26th Asia-Pacific Software Engineering Conference (APSEC), Putrajaya, Malaysia.","DOI":"10.1109\/APSEC48747.2019.00066"},{"key":"ref_38","unstructured":"Rabanser, S., G\u00fcnnemann, S., and Lipton, Z.C. (2019, January 8\u201314). Failing loudly: An empirical study of methods for detecting dataset shift. Proceedings of the 33rd International Conference on Neural Information Processing Systems, Vancouver, BC, Canada."},{"key":"ref_39","unstructured":"Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J.F., and Dennison, D. (2015, January 7\u201312). Hidden Technical Debt in Machine Learning Systems. Proceedings of the 29th Conference on Neural Information Processing Systems (NIPS 2015), Montreal, QC, Canada."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"67","DOI":"10.3390\/computers2020067","article-title":"A characterization of the utility of using artificial intelligence to test two artificial intelligence systems","volume":"2","author":"Straub","year":"2013","journal-title":"Computers"},{"key":"ref_41","doi-asserted-by":"crossref","unstructured":"Tuncali, C., Fainekos, G., Ito, H., and Kapinski, J. (2018, January 26\u201330). Simulation-based adversarial test generation for autonomous vehicles with machine learning components. Proceedings of the 2018 IEEE Intelligent Vehicles Symposium (IV), Suzhou, China.","DOI":"10.1109\/IVS.2018.8500421"},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"265","DOI":"10.1109\/TIV.2019.2955903","article-title":"Requirements-driven test generation for autonomous vehicles with machine learning components","volume":"5","author":"Tuncali","year":"2020","journal-title":"IEEE Trans. Intell. Veh."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Wiesbrock, H., and Grossmann, J. (2024, January 15\u201318). Outline of an independent systematic blackbox test for ML-based systems. Proceedings of the 2024 IEEE International Conference on Artificial Intelligence Testing (AITest), Shanghai, China.","DOI":"10.1109\/AITest62860.2024.00009"},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Min, H., Zhang, Z., Fan, T., Zhang, P., Zhang, C., and Qu, G. (2025). Full Coverage Testing Method for Automated Driving System in Logical Scenario Parameters Space. Sensors, 25.","DOI":"10.3390\/s25185764"},{"key":"ref_45","doi-asserted-by":"crossref","unstructured":"You, H., Wang, Z., Lin, B., and Chen, J. (May, January 26). Navigating the Testing of Evolving Deep Learning Systems: An Exploratory Interview Study. Proceedings of the 2025 IEEE\/ACM 47th International Conference on Software Engineering (ICSE), Ottawa, ON, Canada.","DOI":"10.1109\/ICSE55347.2025.00106"},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"305","DOI":"10.1109\/TSE.2024.3519159","article-title":"ChatAssert: LLM-Based Test Oracle Generation with External Tools Assistance","volume":"51","author":"Hayet","year":"2025","journal-title":"IEEE Trans. Softw. Eng."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Oviedo Lama, J., Guerrero Sosa, J., Rodr\u00edguez Monje, M., Romero Chicharo, F., and Piattini Velthuis, M. (2025, January 10\u201312). Towards Quality Assessment of AI Systems: A Case Study. Proceedings of the 20th International Conference on Software Technologies (ICSOFT), Bilbao, Spain.","DOI":"10.5220\/0013521700003964"},{"key":"ref_48","doi-asserted-by":"crossref","unstructured":"Wagner, M., Wittmann, M.T., and Borg, M. (2025, January 1\u20135). A Critical Look at the EU AI Act\u2019s Requirements and Affected Systems\u2014Who Must Explain What?. Proceedings of the 2025 IEEE 33rd International Requirements Engineering Conference Workshops (REW), Valencia, Spain.","DOI":"10.1109\/REW66121.2025.00074"}],"container-title":["Applied Sciences"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2076-3417\/16\/19\/9787\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,10]],"date-time":"2026-10-10T04:12:04Z","timestamp":1791605524000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2076-3417\/16\/19\/9787"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,10,2]]},"references-count":48,"journal-issue":{"issue":"19","published-online":{"date-parts":[[2026,10]]}},"alternative-id":["app16199787"],"URL":"https:\/\/doi.org\/10.3390\/app16199787","relation":{},"ISSN":["2076-3417"],"issn-type":[{"value":"2076-3417","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,10,2]]}}}