{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,25]],"date-time":"2026-03-25T21:32:05Z","timestamp":1774474325062,"version":"3.50.1"},"reference-count":35,"publisher":"MDPI AG","issue":"1","license":[{"start":{"date-parts":[[2024,12,31]],"date-time":"2024-12-31T00:00:00Z","timestamp":1735603200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100005416","name":"The Research Council of Norway","doi-asserted-by":"publisher","award":["321830"],"award-info":[{"award-number":["321830"]}],"id":[{"id":"10.13039\/501100005416","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Systems"],"abstract":"<jats:p>This paper investigated the effect of automation processes in an industrial company engineering complex cyber-physical systems. The authors used an industry-as-laboratory approach as the research method, exploring an ongoing development project. The automation efforts focused on four areas: (1) test setup, (2) test execution, (3) test result analysis, and (4) documentation. All four areas showed promising results on increased effectiveness and\/or efficiency. In particular, the automation of test result analysis will help the industrial company, KONGSBERG, reduce their main bottleneck in the test process, as well as reduce the risk of costly project delays. An automated system integration test process, facilitating iterative regression testing, will leverage the efficiency of the verification test process.<\/jats:p>","DOI":"10.3390\/systems13010017","type":"journal-article","created":{"date-parts":[[2024,12,31]],"date-time":"2024-12-31T07:34:19Z","timestamp":1735630459000},"page":"17","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Automation Processes for Efficient Verification of Complex Systems: An Empirical Case Study"],"prefix":"10.3390","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-5567-3179","authenticated-orcid":false,"given":"Rune Andre","family":"Haugen","sequence":"first","affiliation":[{"name":"Department of Science and Industry Systems, University of South-Eastern Norway, 3616 Kongsberg, Norway"},{"name":"Department of Research and Development, Kongsberg Defence and Aerospace, 3601 Kongsberg, Norway"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6358-9308","authenticated-orcid":false,"given":"Nils-Olav","family":"Skeie","sequence":"additional","affiliation":[{"name":"Department of Electrical Engineering, IT and Cybernetics, University of South-Eastern Norway, 3918 Porsgrunn, Norway"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1952-8377","authenticated-orcid":false,"given":"Gerrit","family":"Muller","sequence":"additional","affiliation":[{"name":"Department of Science and Industry Systems, University of South-Eastern Norway, 3616 Kongsberg, Norway"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2024,12,31]]},"reference":[{"key":"ref_1","unstructured":"(2023, December 27). KONGSBERG. Available online: www.kongsberg.com."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"463","DOI":"10.1002\/sys.21660","article-title":"Detecting emergence in engineered systems: A literature review and synthesis approach","volume":"26","author":"Haugen","year":"2023","journal-title":"Syst. Eng."},{"key":"ref_3","unstructured":"Haugen, R.A., and Mansouri, M. (2020, January 20\u201322). Applying Systems Thinking to Frame and Explore a Test System for Product Verification; a Case Study in Large Defence Projects. Proceedings of the INCOSE International Symposium 2020, Virtual."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Wechner, M.A., Marb, M.M., and Holzapfel, F. (2023, January 12\u201316). A Strategy for Efficient and Automated Validation and Verification of Maneuverability Requirements. Proceedings of the AIAA AVIATION 2023 Forum, San Diego, CA, USA.","DOI":"10.2514\/6.2023-3595"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Cho, E., Shin, Y.J., Hyun, S., Kim, H., and Bae, D.H. (2022, January 6\u20139). Automatic Generation of Metamorphic Relations for a Cyber-Physical System-of-Systems Using Genetic Algorithm. Proceedings of the 2022 29th Asia-Pacific Software Engineering Conference (APSEC), Virtual.","DOI":"10.1109\/APSEC57359.2022.00033"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"e1892","DOI":"10.1002\/stvr.1892","article-title":"Scenario-Driven Metamorphic Testing for Autonomous Driving Simulators","volume":"34","author":"Zhang","year":"2024","journal-title":"Softw. Test. Verif. Reliab."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"75337","DOI":"10.1109\/ACCESS.2024.3406510","article-title":"From Digital Twins to Digital Twin Prototypes: Concepts, Formalization, and Applications","volume":"12","author":"Barbie","year":"2024","journal-title":"IEEE Access"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Ross, W.D. (1924). Aristotle\u2019s Metaphysics: A Revised Text with Introduction and Commentary, Clarendon Press.","DOI":"10.1093\/actrade\/9780198847762.book.1"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Bondavalli, A., Bouchenak, S., and Kopetz, H. (2016). Emergence in Cyber-Physical Systems-of-Systems (Cpsoss). Cyber-Physical Systems of Systems: Foundations\u2014A Conceptual Model and Some Derivations: The Amadeos Legacy, Springer International Publishing.","DOI":"10.1007\/978-3-319-47590-5"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Mittal, S., Diallo, S., and Tolk, A. (2018). Emergent Behavior in Complex Systems Engineering: A Modeling and Simulation Approach, Wiley.","DOI":"10.1002\/9781119378952"},{"key":"ref_11","unstructured":"Snowden, D. (2020, November 25). Cognitive Edge. Available online: https:\/\/www.cognitive-edge.com\/."},{"key":"ref_12","unstructured":"Kjeldaas, K.A., Haugen, R.A., and Syverud, E. (2021, January 17\u201322). Challenges in Detecting Emergent Behavior in System Testing. Proceedings of the INCOSE International Symposium 2021, Virtual."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Skreddernes, O., Haugen, R.A., and Haskins, C. (2023, January 15\u201320). Coping with Verification in Complex Engineered Product Development. Proceedings of the INCOSE International Symposium, Honolulu, HI, USA.","DOI":"10.1002\/iis2.13034"},{"key":"ref_14","unstructured":"Raman, R., and Jeppu, Y. (2019, January 20\u201325). An Approach for Formal Verification of Machine Learning based Complex Systems. Proceedings of the INCOSE International Symposium 2019, Orlando, FL, USA."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Raman, R., and Jeppu, Y. (2020, January 24\u201327). Formal validation of emergent behavior in a machine learning based collision avoidance system. Proceedings of the 14th Annual IEEE International Systems Conference, SYSCON 2020, Virtual.","DOI":"10.1109\/SysCon47679.2020.9275870"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"538","DOI":"10.1002\/sys.21517","article-title":"Decision learning framework for architecture design decisions of complex systems and system-of-systems","volume":"22","author":"Raman","year":"2019","journal-title":"Syst. Eng."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Raman, R., and Jeppu, Y. (May, January 15). Does the Complex SoS Have Negative Emergent Behavior? Looking for Violations Formally. Proceedings of the 15th Annual IEEE International Systems Conference, SysCon 2021, Virtual.","DOI":"10.1109\/SysCon48628.2021.9447127"},{"key":"ref_18","unstructured":"Raman, R., Gupta, N., and Jeppu, Y. (2021, January 17\u201322). Framework for Formal Verification of Machine Learning Based Complex System-of-System. Proceedings of the INCOSE International Symposium 2021, Virtual."},{"key":"ref_19","unstructured":"Murugesan, A., and Raman, R. (2021, January 18\u201322). Reinforcement Learning for Emergent Behavior Evolution in Complex System-of-Systems. Proceedings of the ICONS: The Sixteenth International Conference on Systems, Porto, Portugal."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Enoiu, E., Sundmark, D., Causevic, A., and Pettersson, P. (2017, January 13\u201317). A Comparative Study of Manual and Automated Testing for Industrial Control Software. Proceedings of the IEEE International Conference on Software Testing, Verification and Validation (ICST), Tokyo, Japan.","DOI":"10.1109\/ICST.2017.44"},{"key":"ref_21","unstructured":"\u00d8vergaard, A., and Muller, G. (2013, January 24\u201327). System Verification by Automatic Testing. Proceedings of the INCOSE International Symposium, Philadelphia, PA, USA."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Giammarco, K. (2017, January 18\u201321). Practical modeling concepts for engineering emergence in systems of systems. Proceedings of the 12th System of Systems Engineering Conference (SoSE), Waikoloa, HI, USA.","DOI":"10.1109\/SYSOSE.2017.7994977"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Szabo, C., and Teo, Y. (2012, January 9\u201312). An integrated approach for the validation of emergence in component-based simulation models. Proceedings of the Winter Simulation Conference, Berlin, Germany.","DOI":"10.1109\/WSC.2012.6465059"},{"key":"ref_24","unstructured":"Muller, G., and Heemels, W.P.M. (2007, January 14\u201316). Five Years of Multi-Disciplinary Academic and Industrial Research: Lessons Learned. Proceedings of the Conference on Systems Engineering Research 2007, Hoboken, NJ, USA."},{"key":"ref_25","unstructured":"Muller, G. (2024, December 29). Industry-as-Laboratory Applied in Practice: The Boderc Project. Available online: www.gaudisite.nl\/IndustryAsLaboratoryAppliedPaper.pdf."},{"key":"ref_26","unstructured":"Saunders, M.N.K., Lewis, P., and Thornhill, A. (2019). Research Methods for Business Students, Pearson. [8th ed.]."},{"key":"ref_27","unstructured":"(2023, October 10). Dassault. Available online: www.3ds.com\/."},{"key":"ref_28","unstructured":"Kehoe, E.J. (2010). A Primer on the Taguchi Method, Society of Manufacturing Engineers. [2nd ed.]."},{"key":"ref_29","unstructured":"Haugen, R.A., and Ghaderi, A. (2020, January 22\u201324). Modelling and Simulation of Detection Rates of Emergent Behaviors in System Integration Test Regimes. Proceedings of the SIMS, Virtual."},{"key":"ref_30","unstructured":"(2023, March 29). Minitab. Available online: www.minitab.com."},{"key":"ref_31","unstructured":"(2023, March 29). Python. Available online: www.python.org."},{"key":"ref_32","unstructured":"Cloutier, R. (2023). System Integration, International Council on Systems Engineering (INCOSE)."},{"key":"ref_33","unstructured":"Feiler, P.H., Hanson, J., de Niz, D., and Wrage, L. (2009). System Architecture Virtual Integration: An Industrial Case Study, Software Engineering Institute."},{"key":"ref_34","unstructured":"Incose, and Wiley (2023). INCOSE Systems Engineering Handbook, John Wiley & Sons, Incorporated. [5th ed.]."},{"key":"ref_35","unstructured":"Kahlen, F.-J., Flumerfelt, S., and Alves, A. (2017). Digital Twin: Mitigating Unpredictable, Undesirable Emergent Behavior in Complex Systems. Transdisciplinary Perspectives on Complex Systems: New Findings and Approaches, Springer International Publishing."}],"container-title":["Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2079-8954\/13\/1\/17\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T16:57:48Z","timestamp":1760115468000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2079-8954\/13\/1\/17"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,12,31]]},"references-count":35,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2025,1]]}},"alternative-id":["systems13010017"],"URL":"https:\/\/doi.org\/10.3390\/systems13010017","relation":{},"ISSN":["2079-8954"],"issn-type":[{"value":"2079-8954","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,12,31]]}}}