{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,7]],"date-time":"2026-02-07T20:54:41Z","timestamp":1770497681126,"version":"3.49.0"},"reference-count":39,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2025,2,14]],"date-time":"2025-02-14T00:00:00Z","timestamp":1739491200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,2,14]],"date-time":"2025-02-14T00:00:00Z","timestamp":1739491200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100010661","name":"Horizon 2020 Framework Programme","doi-asserted-by":"publisher","award":["871319"],"award-info":[{"award-number":["871319"]}],"id":[{"id":"10.13039\/100010661","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100003086","name":"Eusko Jaurlaritza","doi-asserted-by":"publisher","award":["IT1519-22"],"award-info":[{"award-number":["IT1519-22"]}],"id":[{"id":"10.13039\/501100003086","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Softw Syst Model"],"published-print":{"date-parts":[[2025,6]]},"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>Cyber-physical systems (CPSs) blend digital and physical processes. CPS software is the key to realizing their functionalities. This software needs to evolve to deal with different aspects, such as the implementation of new functionalities or bug fixes. Because of this, design\u2013operation methods, colloquially known as \u201cDevOps,\u201d are paramount to be adopted within these systems. During DevOps phases, automating test execution at design time is a key enabler of streamlined software development and software quality improvement. Likewise, monitoring whether a CPS is behaving as expected at operation is similarly important. In DevOps, test oracles play an important role in enabling automated testing, ensuring the reliability of software deployments, providing feedback to developers, etc. However, defining and generating test oracles in the context of DevOps practices in CPSs need to accommodate aspects specific to CPSs, such as their time-continuous behavior and inherent uncertainties. To this end, in this paper, we propose a domain-specific language (DSL) to ease the definition of test oracles and an automated solution for generating a microservice encapsulating the defined test oracles, which is compatible with a DevOps ecosystem for CPSs. We evaluated our DSL with two industrial case study systems and 9 open-source CPSs. Our evaluation results suggest that our DSL can model around 98% of the requirements of these systems through test oracles. Furthermore, it is possible to generate a microservice to be applicable at different test levels within less than 20\u00a0min, being fast enough to be adopted in practice. \n<\/jats:p>","DOI":"10.1007\/s10270-025-01271-8","type":"journal-article","created":{"date-parts":[[2025,2,14]],"date-time":"2025-02-14T06:35:22Z","timestamp":1739514922000},"page":"679-704","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Defining and generating multi-level and uncertainty-wise test oracles for cyber-physical systems"],"prefix":"10.1007","volume":"24","author":[{"given":"Pablo","family":"Valle","sequence":"first","affiliation":[]},{"given":"Aitor","family":"Arrieta","sequence":"additional","affiliation":[]},{"given":"Liping","family":"Han","sequence":"additional","affiliation":[]},{"given":"Shaukat","family":"Ali","sequence":"additional","affiliation":[]},{"given":"Tao","family":"Yue","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2025,2,14]]},"reference":[{"issue":"1","key":"1271_CR1","doi-asserted-by":"publisher","first-page":"13","DOI":"10.1109\/JPROC.2011.2160929","volume":"100","author":"P Derler","year":"2011","unstructured":"Derler, P., Lee, E.A., Vincentelli, A.S.: Modeling cyber-physical systems. Proc. IEEE 100(1), 13\u201328 (2011)","journal-title":"Proc. IEEE"},{"issue":"1","key":"1271_CR2","first-page":"161","volume":"12","author":"R Baheti","year":"2011","unstructured":"Baheti, R., Gill, H.: Cyber-physical systems. Impact Control Technol. 12(1), 161\u2013166 (2011)","journal-title":"Impact Control Technol."},{"key":"1271_CR3","volume-title":"Principles of Cyber-physical Systems","author":"R Alur","year":"2015","unstructured":"Alur, R.: Principles of Cyber-physical Systems. MIT press, Cambridge, MA (2015)"},{"key":"1271_CR4","doi-asserted-by":"crossref","unstructured":"Ayerdi, J., Garciandia, A., Arrieta, A., Afzal, W., Enoiu, E., Agirre, A., Sagardui, G., Arratibel, M., Sellin, O.: Towards a taxonomy for eliciting design-operation continuum requirements of cyber-physical systems. In: 2020 IEEE 28th International Requirements Engineering Conference (RE), pp. 280\u2013290. IEEE (2020)","DOI":"10.1109\/RE48521.2020.00038"},{"issue":"3","key":"1271_CR5","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3571854","volume":"32","author":"F Zampetti","year":"2023","unstructured":"Zampetti, F., Tamburri, D., Panichella, S., Panichella, A., Canfora, G., Di Penta, M.: Continuous integration and delivery practices for cyber-physical systems: an interview-based study. ACM Trans. Softw. Eng. Methodol. 32(3), 1\u201344 (2023)","journal-title":"ACM Trans. Softw. Eng. Methodol."},{"key":"1271_CR6","doi-asserted-by":"publisher","first-page":"1379","DOI":"10.1007\/s10270-017-0609-6","volume":"18","author":"M Zhang","year":"2019","unstructured":"Zhang, M., Ali, S., Yue, T., Norgren, R., Okariz, O.: Uncertainty-wise cyber-physical system test modeling. Softw. Syst. Model. 18, 1379\u20131418 (2019). https:\/\/doi.org\/10.1007\/s10270-017-0609-6","journal-title":"Softw. Syst. Model."},{"key":"1271_CR7","doi-asserted-by":"publisher","first-page":"247","DOI":"10.1007\/978-3-319-42061-5_16","volume-title":"Modelling Foundations and Applications","author":"M Zhang","year":"2016","unstructured":"Zhang, M., Selic, B., Ali, S., Yue, T., Okariz, O., Norgren, R.: Understanding uncertainty in cyber-physical systems: a conceptual model. In: W\u0105sowski, A., L\u00f6nn, H. (eds.) Modelling Foundations and Applications, pp. 247\u2013264. Springer, Cham (2016)"},{"key":"1271_CR8","doi-asserted-by":"crossref","unstructured":"Arrieta, A., Wang, S., Arruabarrena, A., Markiegi, U., Sagardui, G., Etxeberria, L.: Multi-objective black-box test case selection for cost-effectively testing simulation models. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 1411\u20131418 (2018)","DOI":"10.1145\/3205455.3205490"},{"key":"1271_CR9","doi-asserted-by":"crossref","unstructured":"Menghi, C., Nejati, S., Briand, L., Parache, Y.I.: Approximation-refinement testing of compute-intensive cyber-physical models: An approach based on system identification. In: Proceedings of the ACM\/IEEE 42nd International Conference on Software Engineering, pp. 372\u2013384 (2020)","DOI":"10.1145\/3377811.3380370"},{"issue":"1","key":"1271_CR10","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3280987","volume":"28","author":"C Wang","year":"2018","unstructured":"Wang, C., Pastore, F., Briand, L.: Oracles for testing software timeliness with uncertainty. ACM Trans. Softw. Eng. Methodol. (TOSEM) 28(1), 1\u201330 (2018)","journal-title":"ACM Trans. Softw. Eng. Methodol. (TOSEM)"},{"key":"1271_CR11","doi-asserted-by":"crossref","unstructured":"Aldalur, I., Arrieta, A., Agirre, A., Sagardui, G., Arratibel, M.: A microservice-based framework for multi-level testing of cyber-physical systems. Softw. Quality J. 1\u201331 (2023)","DOI":"10.1007\/s11219-023-09639-z"},{"key":"1271_CR12","doi-asserted-by":"crossref","unstructured":"Boufaied, C., Menghi, C., Bianculli, D., Briand, L., Parache, Y.I.: Trace-checking signal-based temporal properties: A model-driven approach. In: Proceedings of the 35th IEEE\/ACM International Conference on Automated Software Engineering, pp. 1004\u20131015 (2020)","DOI":"10.1145\/3324884.3416631"},{"issue":"5","key":"1271_CR13","doi-asserted-by":"publisher","first-page":"3131","DOI":"10.1109\/TSE.2023.3242588","volume":"49","author":"C Boufaied","year":"2023","unstructured":"Boufaied, C., Menghi, C., Bianculli, D., Briand, L.C.: Trace diagnostics for signal-based temporal properties. IEEE Trans. Softw. Eng. 49(5), 3131\u20133154 (2023)","journal-title":"IEEE Trans. Softw. Eng."},{"key":"1271_CR14","doi-asserted-by":"crossref","unstructured":"Menghi, C., Nejati, S., Gaaloul, K., Briand, L.C.: Generating automated and online test oracles for simulink models with continuous and uncertain behaviors. In: Proceedings of the 2019 27th Acm Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 27\u201338 (2019)","DOI":"10.1145\/3338906.3338920"},{"key":"1271_CR15","doi-asserted-by":"crossref","unstructured":"Menghi, C., Vigan\u00f2, E., Bianculli, D., Briand, L.C.: Trace-checking cps properties: bridging the cyber-physical gap. In: 2021 IEEE\/ACM 43rd International Conference on Software Engineering (ICSE), pp. 847\u2013859. IEEE (2021)","DOI":"10.1109\/ICSE43902.2021.00082"},{"issue":"4","key":"1271_CR16","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3576041","volume":"32","author":"L Han","year":"2023","unstructured":"Han, L., Ali, S., Yue, T., Arrieta, A., Arratibel, M.: Uncertainty-aware robustness assessment of industrial elevator systems. ACM Trans. Softw. Eng. Methodol. 32(4), 1\u201351 (2023)","journal-title":"ACM Trans. Softw. Eng. Methodol."},{"issue":"11","key":"1271_CR17","first-page":"2465","volume":"34","author":"A Gartziandia","year":"2022","unstructured":"Gartziandia, A., Arrieta, A., Ayerdi, J., Illarramendi, M., Agirre, A., Sagardui, G., Arratibel, M.: Machine learning-based test oracles for performance testing of cyber-physical systems: an industrial case study on elevators dispatching algorithms. J. Softw.: Evol. Process 34(11), 2465 (2022)","journal-title":"J. Softw.: Evol. Process"},{"key":"1271_CR18","doi-asserted-by":"crossref","unstructured":"Arrieta, A., Ayerdi, J., Illarramendi, M., Agirre, A., Sagardui, G., Arratibel, M.: Using machine learning to build test oracles: an industrial case study on elevators dispatching algorithms. In: 2021 IEEE\/ACM International Conference on Automation of Software Test (AST), pp. 30\u201339. IEEE (2021)","DOI":"10.1109\/AST52587.2021.00012"},{"key":"1271_CR19","doi-asserted-by":"crossref","unstructured":"Ayerdi, J., Terragni, V., Arrieta, A., Tonella, P., Sagardui, G., Arratibel, M.: Generating metamorphic relations for cyber-physical systems with genetic programming: an industrial case study. In: Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 1264\u20131274 (2021)","DOI":"10.1145\/3468264.3473920"},{"key":"1271_CR20","doi-asserted-by":"crossref","unstructured":"Ayerdi, J., Segura, S., Arrieta, A., Sagardui, G., Arratibel, M.: Qos-aware metamorphic testing: an elevation case study. In: 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE), pp. 104\u2013114. IEEE (2020)","DOI":"10.1109\/ISSRE5003.2020.00019"},{"key":"1271_CR21","doi-asserted-by":"crossref","unstructured":"Arrieta, A., Otaegi, M., Han, L., Sagardui, G., Ali, S., Arratibel, M.: Automating test oracle generation in devops for industrial elevators. In: 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), pp. 284\u2013288. IEEE (2022)","DOI":"10.1109\/SANER53432.2022.00044"},{"key":"1271_CR22","doi-asserted-by":"crossref","unstructured":"Valle, P., Arrieta, A., Arratibel, M.: Applying and extending the delta debugging algorithm for elevator dispatching algorithms (experience paper). In: Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 1055\u20131067 (2023)","DOI":"10.1145\/3597926.3598117"},{"key":"1271_CR23","unstructured":"Xtext - Language Engineering Made Easy! https:\/\/eclipse.dev\/Xtext\/"},{"key":"1271_CR24","doi-asserted-by":"crossref","unstructured":"Nejati, S., Gaaloul, K., Menghi, C., Briand, L.C., Foster, S., Wolfe, D.: Evaluating model testing and model checking for finding requirements violations in simulink models. In: Proceedings of the 2019 27th Acm Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 1015\u20131025 (2019)","DOI":"10.1145\/3338906.3340444"},{"key":"1271_CR25","doi-asserted-by":"crossref","unstructured":"Mavridou, A., Bourbouh, H., Giannakopoulou, D., Pressburger, T., Hejase, M., Garoche, P.-L., Schumann, J.: The ten lockheed martin cyber-physical challenges: formalized, analyzed, and explained. In: 2020 IEEE 28th International Requirements Engineering Conference (RE), pp. 300\u2013310. IEEE (2020)","DOI":"10.1109\/RE48521.2020.00040"},{"key":"1271_CR26","volume-title":"Transportation Systems in Buildings: CIBSE Guide D: 2010","author":"G Barney","year":"2010","unstructured":"Barney, G.: Transportation Systems in Buildings: CIBSE Guide D: 2010. Chartered Institution of Building Services Engineers, London (2010)"},{"key":"1271_CR27","doi-asserted-by":"publisher","first-page":"110881","DOI":"10.1016\/j.jss.2020.110881","volume":"174","author":"C Boufaied","year":"2021","unstructured":"Boufaied, C., Jukss, M., Bianculli, D., Briand, L.C., Parache, Y.I.: Signal-based properties of cyber-physical systems: taxonomy and logic-based characterization. J. Syst. Softw. 174, 110881 (2021)","journal-title":"J. Syst. Softw."},{"key":"1271_CR28","doi-asserted-by":"crossref","unstructured":"Pradhan, S.M., Dubey, A., Gokhale, A., Lehofer, M.: Chariot: A domain specific language for extensible cyber-physical systems. In: Proceedings of the Workshop on Domain-specific Modeling, pp. 9\u201316 (2015)","DOI":"10.1145\/2846696.2846708"},{"key":"1271_CR29","doi-asserted-by":"crossref","unstructured":"Van Den\u00a0Berg, F., Garousi, V., Tekinerdogan, B., Haverkort, B.R.: Designing cyber-physical systems with adsl: a domain-specific language and tool support. In: 2018 13th Annual Conference on System of Systems Engineering (SoSE), pp. 225\u2013232. IEEE (2018)","DOI":"10.1109\/SYSOSE.2018.8428770"},{"key":"1271_CR30","doi-asserted-by":"crossref","unstructured":"Nandi, G.S., Pereira, D., Proen\u00e7a, J., Tovar, E.: Work-in-progress: a dsl for the safe deployment of runtime monitors in cyber-physical systems. In: 2020 IEEE Real-Time Systems Symposium (RTSS), pp. 395\u2013398. IEEE (2020)","DOI":"10.1109\/RTSS49844.2020.00047"},{"key":"1271_CR31","doi-asserted-by":"crossref","unstructured":"Klikovits, S., Linard, A., Buchs, D.: Crest-a dsl for reactive cyber-physical systems. In: System Analysis and Modeling. Languages, Methods, and Tools for Systems Engineering: 10th International Conference, SAM 2018, Copenhagen, Denmark, October 15\u201316, 2018, Proceedings 10, pp. 29\u201345. Springer (2018)","DOI":"10.1007\/978-3-030-01042-3_3"},{"key":"1271_CR32","doi-asserted-by":"crossref","unstructured":"Vernotte, A., Cretin, A., Legeard, B., Peureux, F.: A domain-specific language to design false data injection tests for air traffic control systems. Int. J. Softw. Tools Technol. Transfer, 1\u201332 (2022)","DOI":"10.1007\/s10009-021-00604-4"},{"key":"1271_CR33","unstructured":"Barreto, A.B., Hieb, M., Yano, E.: Developing a complex simulation environment for evaluating cyber attacks. In: Interservice\/Industry Training, Simulation, and Education Conference (I\/ITSEC), vol. 12248, pp. 1\u20139 (2012)"},{"key":"1271_CR34","doi-asserted-by":"crossref","unstructured":"Maciel, D., Paiva, A.C., Da\u00a0Silva, A.R.: From requirements to automated acceptance tests of interactive apps: An integrated model-based testing approach. In: ENASE, pp. 265\u2013272 (2019)","DOI":"10.5220\/0007679202650272"},{"key":"1271_CR35","doi-asserted-by":"crossref","unstructured":"Menzel, T., Bagschik, G., Maurer, M.: Scenarios for development, test and validation of automated vehicles. In: 2018 IEEE Intelligent Vehicles Symposium (IV), pp. 1821\u20131827. IEEE (2018)","DOI":"10.1109\/IVS.2018.8500406"},{"key":"1271_CR36","doi-asserted-by":"crossref","unstructured":"Queiroz, R., Berger, T., Czarnecki, K.: Geoscenario: An open dsl for autonomous driving scenario representation. In: 2019 IEEE Intelligent Vehicles Symposium (IV), pp. 287\u2013294. IEEE (2019)","DOI":"10.1109\/IVS.2019.8814107"},{"key":"1271_CR37","unstructured":"Afzal, A., Goues, C.L., Timperley, C.S.: Gzscenic: Automatic scene generation for gazebo simulator. arXiv preprint arXiv:2104.08625 (2021)"},{"key":"1271_CR38","doi-asserted-by":"crossref","unstructured":"Arrieta, A., Agirre, J.A., Sagardui, G.: A tool for the automatic generation of test cases and oracles for simulation models based on functional requirements. In: 2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 1\u20135. IEEE (2020)","DOI":"10.1109\/ICSTW50294.2020.00018"},{"key":"1271_CR39","doi-asserted-by":"crossref","unstructured":"Bernaerts, M., Oakes, B., Vanherpen, K., Aelvoet, B., Vangheluwe, H., Denil, J.: Validating industrial requirements with a contract-based approach. In: 2019 ACM\/IEEE 22nd International Conference on Model Driven Engineering Languages and Systems Companion (MODELS-C), pp. 18\u201327. IEEE (2019)","DOI":"10.1109\/MODELS-C.2019.00010"}],"container-title":["Software and Systems Modeling"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10270-025-01271-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10270-025-01271-8\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10270-025-01271-8.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,4]],"date-time":"2025-06-04T06:28:30Z","timestamp":1749018510000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10270-025-01271-8"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,2,14]]},"references-count":39,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2025,6]]}},"alternative-id":["1271"],"URL":"https:\/\/doi.org\/10.1007\/s10270-025-01271-8","relation":{},"ISSN":["1619-1366","1619-1374"],"issn-type":[{"value":"1619-1366","type":"print"},{"value":"1619-1374","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,2,14]]},"assertion":[{"value":"22 March 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"8 January 2025","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 January 2025","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"14 February 2025","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}