{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,28]],"date-time":"2026-04-28T15:34:12Z","timestamp":1777390452674,"version":"3.51.4"},"reference-count":90,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2023,4,26]],"date-time":"2023-04-26T00:00:00Z","timestamp":1682467200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Horizon 2020","award":["957254-COSMOS"],"award-info":[{"award-number":["957254-COSMOS"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2023,5,31]]},"abstract":"<jats:p>Continuous Integration and Delivery (CI\/CD) practices have shown several benefits for software development and operations, such as faster release cycles and early discovery of defects. For Cyber-Physical System (CPS) development, CI\/CD can help achieving required goals, such as high dependability, yet it may be challenging to apply. This article empirically investigates challenges, barriers, and their mitigation occurring when applying CI\/CD practices to develop CPSs in 10 organizations working in eight different domains. The study has been conducted through semi-structured interviews, by applying an open card sorting procedure together with a member-checking survey within the same organizations, and by validating the results through a further survey involving 55 professional developers. The study reveals several peculiarities in the application of CI\/CD to CPSs. These include the need for (i) combining continuous and periodic builds while balancing the use of Hardware-in-the-Loop and simulators, (ii) coping with difficulties in software deployment (iii) accounting for simulators and Hardware-in-the-Loop differing in their behavior, and (vi) combining hardware\/software expertise in the development team. Our findings open the road toward recommenders aimed at supporting the setting and evolution of CI\/CD pipelines, as well as university curricula requiring interdisciplinarity, such as knowledge about hardware, software, and their interplay.<\/jats:p>","DOI":"10.1145\/3571854","type":"journal-article","created":{"date-parts":[[2022,11,19]],"date-time":"2022-11-19T10:22:03Z","timestamp":1668853323000},"page":"1-44","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":30,"title":["Continuous Integration and Delivery Practices for Cyber-Physical Systems: An Interview-Based Study"],"prefix":"10.1145","volume":"32","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7098-8964","authenticated-orcid":false,"given":"Fiorella","family":"Zampetti","sequence":"first","affiliation":[{"name":"University of Sannio, Benevento, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1230-8961","authenticated-orcid":false,"given":"Damian","family":"Tamburri","sequence":"additional","affiliation":[{"name":"Eindhoven University of Technology, The Netherlands, DA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4120-626X","authenticated-orcid":false,"given":"Sebastiano","family":"Panichella","sequence":"additional","affiliation":[{"name":"Zurich University of Applied Sciences, Winterthur, Switzerland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7395-3588","authenticated-orcid":false,"given":"Annibale","family":"Panichella","sequence":"additional","affiliation":[{"name":"Technical University of Delft, Delft, The Netherlands"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0049-1279","authenticated-orcid":false,"given":"Gerardo","family":"Canfora","sequence":"additional","affiliation":[{"name":"University of Sannio, Benevento, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0340-9747","authenticated-orcid":false,"given":"Massimiliano","family":"Di Penta","sequence":"additional","affiliation":[{"name":"University of Sannio, Benevento, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2023,4,26]]},"reference":[{"key":"e_1_3_3_2_2","unstructured":"MISRA. 2004. Guidelines for the use of the C language in critical systems. Retrieved December 10 2022 from https:\/\/www.academia.edu\/26806881\/MISRA_C_2004_Guidelines_for_the_use_of_the_C_language_in_critical_systems.pdf."},{"key":"e_1_3_3_3_2","first-page":"194","volume-title":"Testing Software and Systems","author":"Asadollah Sara Abbaspour","unstructured":"Sara Abbaspour Asadollah, Rafia Inam, and Hans Hansson. 2015. A survey on testing for cyber physical system. In Testing Software and Systems. Lecture Notes in Computer Science, Vol. 9447. Springer, 194\u2013207."},{"key":"e_1_3_3_4_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2019.2897300"},{"key":"e_1_3_3_5_2","first-page":"143","volume-title":"Proceedings of the IEEE\/ACM International Conference on Automated Software Engineering (ASE\u201918)","author":"Abdessalem Raja Ben","unstructured":"Raja Ben Abdessalem, Annibale Panichella, Shiva Nejati, Lionel C. Briand, and Thomas Stifter. 2018. Testing autonomous cars for feature interaction failures using many-objective search. In Proceedings of the IEEE\/ACM International Conference on Automated Software Engineering (ASE\u201918). IEEE, Los Alamitos, CA, 143\u2013154."},{"key":"e_1_3_3_6_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-56441-4_24"},{"key":"e_1_3_3_7_2","first-page":"1411","volume-title":"Proceedings of the Genetic and Evolutionary Computation Conference","author":"Arrieta Aitor","unstructured":"Aitor Arrieta, Shuai Wang, Ainhoa Arruabarrena, Urtzi Markiegi, Goiuria Sagardui, and Leire Etxeberria. 2018. Multi-objective black-box test case selection for cost-effectively testing simulation models. In Proceedings of the Genetic and Evolutionary Computation Conference. 1411\u20131418."},{"key":"e_1_3_3_8_2","first-page":"5","volume-title":"Proceedings of the International Static Analysis Symposium","author":"Bagnara Roberto","unstructured":"Roberto Bagnara, Abramo Bagnara, and Patricia M. Hill. 2018. The MISRA C coding standard and its role in the development and analysis of safety- and security-critical embedded software. In Proceedings of the International Static Analysis Symposium. 5\u201323."},{"key":"e_1_3_3_9_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-021-10072-8"},{"key":"e_1_3_3_10_2","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2015.27"},{"key":"e_1_3_3_11_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2017.02.013"},{"key":"e_1_3_3_12_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-Companion.2019.00023"},{"key":"e_1_3_3_13_2","unstructured":"Alexey Dosovitskiy Germ\u00e1n Ros Felipe Codevilla Antonio M. L\u00f3pez and Vladlen Koltun. 2017. CARLA: An open urban driving simulator. In Proceedings of the 1st Annual Conference on Robot Learning (CoRL\u201917) . Proceedings of Machine Learning Research Vol. 78. PMLR 1\u201316."},{"key":"e_1_3_3_14_2","volume-title":"Continuous Integration: Improving Software Quality and Reducing Risk","author":"Duvall Paul","unstructured":"Paul Duvall, Stephen M. Matyas, and Andrew Glover. 2007. Continuous Integration: Improving Software Quality and Reducing Risk. Addison-Wesley."},{"key":"e_1_3_3_15_2","article-title":"Continuous integration patterns and antipatterns","author":"Duvall Paul M.","unstructured":"Paul M. Duvall. 2010. Continuous integration patterns and antipatterns. DZone Refcard #84. Retrieved November 26, 2022 from http:\/\/bit.ly\/l8rfVS.","journal-title":"DZone Refcard #84."},{"key":"e_1_3_3_16_2","article-title":"2011. Continuous delivery patterns and antipatterns","author":"Duvall Paul M.","unstructured":"Paul M. Duvall. 2011. Continuous delivery patterns and antipatterns. DZone Refcard #145. Retrieved November 26, 2022 from https:\/\/dzone.com\/refcardz\/continuous-delivery-patterns.","journal-title":"DZone Refcard #145."},{"key":"e_1_3_3_17_2","first-page":"830","volume-title":"Proceedings of the ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/SIGSOFT FSE\u201919).","author":"Eck Moritz","unstructured":"Moritz Eck, Fabio Palomba, Marco Castelluccio, and Alberto Bacchelli. 2019. Understanding flaky tests: The developer\u2019s perspective. In Proceedings of the ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/SIGSOFT FSE\u201919).830\u2013840."},{"key":"e_1_3_3_18_2","first-page":"26","volume-title":"Proceedings of the 43rd IEEE\/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE (NIER)\u201921).","author":"Elazhary Omar","unstructured":"Omar Elazhary, Margaret-Anne D. Storey, Neil A. Ernst, and Elise Paradis. 2021. ADEPT: A socio-technical theory of continuous integration. In Proceedings of the 43rd IEEE\/ACM International Conference on Software Engineering: New Ideas and Emerging Results (ICSE (NIER)\u201921).26\u201330."},{"key":"e_1_3_3_19_2","article-title":"2021. Uncovering the benefits and challenges of continuous integration practices","author":"Elazhary Omar","year":"2021","unstructured":"Omar Elazhary, Colin Werner, Ze Shi Li, Derek Lowlind, Neil A. Ernst, and Margaret-Anne Storey. 2021. Uncovering the benefits and challenges of continuous integration practices. arXiv:2103.04251 (2021).","journal-title":"arXiv:2103.04251"},{"key":"e_1_3_3_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2020.2982638"},{"key":"e_1_3_3_21_2","first-page":"238","volume-title":"Advances in Production Management Systems. Towards Smart and Digital Manufacturing.","author":"Flores-Garc\u00eda Erik","unstructured":"Erik Flores-Garc\u00eda, Goo-Young Kim, Jinho Yang, Magnus Wiktorsson, and Sang Do Noh. 2020. Analyzing the characteristics of digital twin and discrete event simulation in cyber physical systems. In Advances in Production Management Systems. Towards Smart and Digital Manufacturing.IFIP Advances in Information and Communication Technology, Vol. 592. Springer, 238\u2013244."},{"key":"e_1_3_3_22_2","unstructured":"Martin Fowler and Matthew Foemmel. 2000. Continuous Integration. Retrieved November 21 2021 from https:\/\/martinfowler.com\/articles\/originalContinuousIntegration.html."},{"key":"e_1_3_3_23_2","first-page":"1027","volume-title":"Proceedings of the 2008 IEEE\/ACS International Conference on Computer Systems and Applications","author":"Gall Heinz","unstructured":"Heinz Gall. 2008. Functional safety IEC 61508 \/ IEC 61511 the impact to certification and the user. In Proceedings of the 2008 IEEE\/ACS International Conference on Computer Systems and Applications. 1027\u20131031."},{"key":"e_1_3_3_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2018.2838131"},{"key":"e_1_3_3_25_2","first-page":"1","volume-title":"Proceedings of the 2018 IEEE 11th International Conference on Software Testing, Verification, and Validation (ICST\u201918)","author":"Gambi Alessio","unstructured":"Alessio Gambi, Jonathan Bell, and Andreas Zeller. 2018. Practical test dependency detection. In Proceedings of the 2018 IEEE 11th International Conference on Software Testing, Verification, and Validation (ICST\u201918). IEEE, Los Alamitos, CA, 1\u201311."},{"key":"e_1_3_3_26_2","first-page":"257","volume-title":"Proceedings of the ACM Joint Meeting on European Software Engineering Conference and the Symposium on the Foundations of Software Engineering (ESEC\/SIGSOFT FSE\u201919)","author":"Gambi Alessio","unstructured":"Alessio Gambi, Tri Huynh, and Gordon Fraser. 2019. Generating effective test cases for self-driving cars from police reports. In Proceedings of the ACM Joint Meeting on European Software Engineering Conference and the Symposium on the Foundations of Software Engineering (ESEC\/SIGSOFT FSE\u201919). ACM, New York, NY, 257\u2013267."},{"key":"e_1_3_3_27_2","first-page":"385","volume-title":"Proceedings of the 42nd International Conference on Software Engineering (ICSE\u201920)","author":"Garcia Joshua","unstructured":"Joshua Garcia, Yang Feng, Junjie Shen, Sumaya Almanee, Yuan Xia, and Qi Alfred Chen. 2020. A comprehensive study of autonomous vehicle bugs. In Proceedings of the 42nd International Conference on Software Engineering (ICSE\u201920). 385\u2013396."},{"key":"e_1_3_3_28_2","first-page":"597","volume-title":"Proceedings of the 2021 4th IEEE International Conference on Industrial Cyber-Physical Systems (ICPS\u201921)","author":"Gautham Smitha","unstructured":"Smitha Gautham, Athira Varma Jayakumar, Abhi Rajagopala, and Carl Elks. 2021. Realization of a model-based DevOps process for industrial safety critical cyber physical systems. In Proceedings of the 2021 4th IEEE International Conference on Industrial Cyber-Physical Systems (ICPS\u201921). 597\u2013604."},{"key":"e_1_3_3_29_2","first-page":"104","volume-title":"Proceedings of the 2018 Conference on Real-Time and Embedded Systems and Technologies (RTEST\u201918)","author":"Gholami Farshad","unstructured":"Farshad Gholami, Niousha Attar, Hassan Haghighi, Mojtaba Vahidi Asl, Meysam Valueian, and Saina Mohamadyari. 2018. A classifier-based test oracle for embedded software. In Proceedings of the 2018 Conference on Real-Time and Embedded Systems and Technologies (RTEST\u201918). 104\u2013111."},{"key":"e_1_3_3_30_2","first-page":"176","volume-title":"Proceedings of the 21st ACM\/IEEE International Conference on Model Driven Engineering Languages and Systems (MODELS\u201918)","author":"Gonz\u00e1lez Carlos A.","unstructured":"Carlos A. Gonz\u00e1lez, Mojtaba Varmazyar, Shiva Nejati, Lionel C. Briand, and Yago Isasi. 2018. Enabling model testing of cyber-physical systems. In Proceedings of the 21st ACM\/IEEE International Conference on Model Driven Engineering Languages and Systems (MODELS\u201918). ACM, New York, NY, 176\u2013186."},{"key":"e_1_3_3_31_2","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177705148"},{"key":"e_1_3_3_32_2","unstructured":"Robert M. Groves Floyd J. Fowler Jr. Mick P. Couyper James M. Lepkowski Eleanor Singer and Roger Tourangeau. 2009. Survey Methodology (2nd ed.). Wiley."},{"key":"e_1_3_3_33_2","first-page":"571","article-title":"2016. Testing of autonomous systems\u2014Challenges and current state-of-the-art","author":"Helle Philipp","unstructured":"Philipp Helle, Wladimir Schamai, and Carsten Strobel. 2016. Testing of autonomous systems\u2014Challenges and current state-of-the-art. In Proceedings of the INCOSE International Symposium.571\u2013584.","journal-title":"Proceedings of the INCOSE International Symposium."},{"key":"e_1_3_3_34_2","volume-title":"Proceedings of the 25th ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE\u201917)","author":"Hilton Michael","unstructured":"Michael Hilton, Nicholas Nelson, Timothy Tunnell, Darko Marinov, and Danny Dig. 2017. Trade-offs in continuous integration: Assurance, security, and flexibility. In Proceedings of the 25th ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE\u201917)."},{"key":"e_1_3_3_35_2","volume-title":"Proceedings of the 31st IEEE\/ACM International Conference on Automated Software Engineering (ASE\u201916)","author":"Hilton Michael","unstructured":"Michael Hilton, Timothy Tunnell, Kai Huang, Darko Marinov, and Danny Dig. 2016. Usage, costs, and benefits of continuous integration in open-source projects. In Proceedings of the 31st IEEE\/ACM International Conference on Automated Software Engineering (ASE\u201916)."},{"key":"e_1_3_3_36_2","unstructured":"David L. Hoover. 2016. Textual variation text-randomization and microanalysis. In Proceedings of the 11th Annual International Conference of the Alliance of Digital Humanities Organizations (DH\u201916) . 223\u2013225. http:\/\/dblp.uni-trier.de\/db\/conf\/dihu\/dh2016.html#Hoover16a."},{"key":"e_1_3_3_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/3460319.3464825"},{"key":"e_1_3_3_38_2","first-page":"194","volume-title":"Proceedings of the 2021 14th IEEE Conference on Software Testing, Verification, and Validation (ICST\u201921)","author":"Jahangirova Gunel","unstructured":"Gunel Jahangirova, Andrea Stocco, and Paolo Tonella. 2021. Quality metrics and oracles for autonomous vehicles testing. In Proceedings of the 2021 14th IEEE Conference on Software Testing, Verification, and Validation (ICST\u201921). IEEE, Los Alamitos, CA, 194\u2013204."},{"key":"e_1_3_3_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICST49551.2021.00030"},{"key":"e_1_3_3_40_2","volume-title":"Industrial DevOps: Applying DevOps and Continuous Delivery to Significant Cyber-Physical Systems","author":"Johnson Suzette","unstructured":"Suzette Johnson, Harry Koehneman, Diane LaFortune, Dean Leffingwell, Stephen Magill, Steve Mayner, Avigail Ofer, Robert Stroud, Anders Wallgren, and Robin Yeman. 2018. Industrial DevOps: Applying DevOps and Continuous Delivery to Significant Cyber-Physical Systems. National Academies Press. https:\/\/itrevolution.com\/book\/industrial-devops\/."},{"key":"e_1_3_3_41_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.tra.2016.09.010"},{"key":"e_1_3_3_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/566493.566495"},{"key":"e_1_3_3_43_2","doi-asserted-by":"publisher","DOI":"10.1145\/638574.638580"},{"key":"e_1_3_3_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/638574.638580"},{"key":"e_1_3_3_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/571681.571686"},{"issue":"3","key":"e_1_3_3_46_2","first-page":"411","article-title":"2004. Reliability in content analysis: Some common misconceptions and recommendations","volume":"30","author":"Krippendorff Klaus","year":"2004","unstructured":"Klaus Krippendorff. 2004. Reliability in content analysis: Some common misconceptions and recommendations. Journal of the Royal Statistical Society: Series B (Methodological) 30, 3 (2004), 411\u2013433.","journal-title":"Journal of the Royal Statistical Society: Series B (Methodological)"},{"key":"e_1_3_3_47_2","first-page":"101","volume-title":"Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA\u201919)","author":"Lam Wing","unstructured":"Wing Lam, Patrice Godefroid, Suman Nath, Anirudh Santhiar, and Suresh Thummalapenta. 2019. Root causing flaky tests in a large-scale industrial setting. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA\u201919). ACM, New York, NY, 101\u2013111."},{"key":"e_1_3_3_48_2","first-page":"403","volume-title":"Proceedings of the 31st IEEE International Symposium on Software Reliability Engineering (ISSRE\u201920)","author":"Lam Wing","unstructured":"Wing Lam, Stefan Winter, Angello Astorga, Victoria Stodden, and Darko Marinov. 2020. Understanding reproducibility and characteristics of flaky tests through test reruns in Java projects. In Proceedings of the 31st IEEE International Symposium on Software Reliability Engineering (ISSRE\u201920). 403\u2013413."},{"key":"e_1_3_3_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/3428270"},{"key":"e_1_3_3_50_2","first-page":"1381","volume-title":"Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering","author":"Lampel Johannes","unstructured":"Johannes Lampel, Sascha Just, Sven Apel, and Andreas Zeller. 2021. When life gives you oranges: Detecting and diagnosing intermittent job failures at Mozilla. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1381\u20131392."},{"key":"e_1_3_3_51_2","doi-asserted-by":"publisher","DOI":"10.1109\/TRO.2019.2942989"},{"key":"e_1_3_3_52_2","first-page":"31","volume-title":"Proceedings of the ACM\/IEEE 42nd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP\u201920)","author":"Malavolta Ivano","unstructured":"Ivano Malavolta, Grace Lewis, Bradley Schmerl, Patricia Lago, and David Garlan. 2020. How do you architect your robots? State of the practice and guidelines for ROS-based systems. In Proceedings of the ACM\/IEEE 42nd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP\u201920). ACM, New York, NY, 31\u201340."},{"key":"e_1_3_3_53_2","first-page":"448","volume-title":"Proceedings of the International Conference on Product-Focused Software Process Improvement","author":"M\u00e5rtensson Torvald","unstructured":"Torvald M\u00e5rtensson, Daniel St\u00e5hl, and Jan Bosch. 2016. Continuous integration applied to software-intensive embedded systems\u2014Problems and experiences. In Proceedings of the International Conference on Product-Focused Software Process Improvement. 448\u2013457."},{"key":"e_1_3_3_54_2","first-page":"372","volume-title":"Proceedings of the 2020 IEEE\/ACM 42nd International Conference on Software Engineering (ICSE\u201920)","author":"Menghi Claudio","unstructured":"Claudio Menghi, Shiva Nejati, Lionel Briand, and Yago Isasi Parache. 2020. Approximation-refinement testing of compute-intensive cyber-physical models: An approach based on system identification. In Proceedings of the 2020 IEEE\/ACM 42nd International Conference on Software Engineering (ICSE\u201920). IEEE, Los Alamitos, CA, 372\u2013384."},{"key":"e_1_3_3_55_2","first-page":"128","volume-title":"Proceedings of the 2021 IEEE International Conference on Artificial Intelligence Testing (AITest\u201921)","author":"Nguyen Vuong","unstructured":"Vuong Nguyen, Stefan Huber, and Alessio Gambi. 2021. SALVO: Automated generation of diversified tests for self-driving cars from existing maps. In Proceedings of the 2021 IEEE International Conference on Artificial Intelligence Testing (AITest\u201921). IEEE, Los Alamitos, CA, 128\u2013135."},{"key":"e_1_3_3_56_2","first-page":"392","volume-title":"Proceedings of the 2012 38th Euromicro Conference on Software Engineering and Advanced Applications (SEAA\u201912)","author":"Olsson Helena Holmstrom","unstructured":"Helena Holmstrom Olsson, Hiva Alahyari, and Jan Bosch. 2012. Climbing the \u201cStairway to Heaven\u2019\u2014A multiple-case study exploring barriers in the transition from agile development towards continuous deployment of software. In Proceedings of the 2012 38th Euromicro Conference on Software Engineering and Advanced Applications (SEAA\u201912). IEEE, Los Alamitos, CA, 392\u2013399."},{"key":"e_1_3_3_57_2","first-page":"28","volume-title":"Cyber Physical Systems. Model-Based Design","author":"Park Heejong","unstructured":"Heejong Park, Arvind Easwaran, and Sidharta Andalam. 2018. Challenges in digital twin development for cyber-physical production systems. In Cyber Physical Systems. Model-Based Design. Springer International, Cham, Switzerland, 28\u201348."},{"key":"e_1_3_3_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/505532.505535"},{"key":"e_1_3_3_59_2","first-page":"492","volume-title":"Proceedings of the 17th International Conference on Mining Software Repositories (MSR\u201920)","author":"Pinto Gustavo","unstructured":"Gustavo Pinto, Breno Miranda, Supun Dissanayake, Marcelo d\u2019Amorim, Christoph Treude, and Antonia Bertolino. 2020. What is the vocabulary of flaky tests? In Proceedings of the 17th International Conference on Mining Software Repositories (MSR\u201920). 492\u2013502."},{"key":"e_1_3_3_60_2","first-page":"492","volume-title":"Proceedings of the 17th International Conference on Mining Software Repositories","author":"Pinto Gustavo","unstructured":"Gustavo Pinto, Breno Miranda, Supun Dissanayake, Marcelo d\u2019Amorim, Christoph Treude, and Antonia Bertolino. 2020. What is the vocabulary of flaky tests? In Proceedings of the 17th International Conference on Mining Software Repositories. 492\u2013502."},{"key":"e_1_3_3_61_2","first-page":"164","volume-title":"Proceedings of the 41st International Conference on Software Engineering (ICSE\u201919)","author":"Rahman Akond","unstructured":"Akond Rahman, Chris Parnin, and Laurie Williams. 2019. The seven sins: Security smells in infrastructure as code scripts. In Proceedings of the 41st International Conference on Software Engineering (ICSE\u201919). IEEE, Los Alamitos, CA, 164\u2013175."},{"key":"e_1_3_3_62_2","first-page":"74","volume-title":"Proceedings of the 22nd IEEE\/ACM International Conference on Automated Software Engineering (ASE\u201907)","author":"Reiss Steven P.","unstructured":"Steven P. Reiss. 2007. Automatic code stylizing. In Proceedings of the 22nd IEEE\/ACM International Conference on Automated Software Engineering (ASE\u201907). 74\u201383."},{"key":"e_1_3_3_63_2","volume-title":"Proceedings of the ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE\u201920)","author":"Riccio Vincenzo","unstructured":"Vincenzo Riccio and Paolo Tonella. 2020. Model-based exploration of the frontier of behaviours for deep learning system testing. In Proceedings of the ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE\u201920). ACM, New York, NY."},{"key":"e_1_3_3_64_2","doi-asserted-by":"publisher","DOI":"10.1080\/08874417.2015.11645799"},{"key":"e_1_3_3_65_2","first-page":"21","volume-title":"Companion Proceedings of the 38th International Conference on Software Engineering (ICSE Companion\u201916)","author":"Savor Tony","unstructured":"Tony Savor, Mitchell Douglas, Michael Gentili, Laurie Williams, Kent Beck, and Michael Stumm. 2016. Continuous deployment at Facebook and OANDA. In Companion Proceedings of the 38th International Conference on Software Engineering (ICSE Companion\u201916). 21\u201330."},{"key":"e_1_3_3_66_2","first-page":"112","volume-title":"Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA\u201919)","author":"Shi August","unstructured":"August Shi, Jonathan Bell, and Darko Marinov. 2019. Mitigating the effects of flaky tests on mutation testing. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA\u201919). 112\u2013122."},{"key":"e_1_3_3_67_2","doi-asserted-by":"crossref","unstructured":"Hesham Shokry and Mike Hinchey. 2009. Model-based verification of embedded software. Computer 42 4 (2009) 53\u201359.","DOI":"10.1109\/MC.2009.125"},{"key":"e_1_3_3_68_2","doi-asserted-by":"publisher","DOI":"10.1109\/TITS.2017.2742141"},{"key":"e_1_3_3_69_2","volume-title":"Card Sorting: Designing Usable Categories","author":"Spencer Donna","unstructured":"Donna Spencer. 2009. Card Sorting: Designing Usable Categories. Rosenfeld Media."},{"key":"e_1_3_3_70_2","first-page":"54","volume-title":"Companion Proceedings of the 36th International Conference on Software Engineering","author":"St\u00e5hl Daniel","unstructured":"Daniel St\u00e5hl and Jan Bosch. 2014. Automated software integration flows in industry: A multiple-case study. In Companion Proceedings of the 36th International Conference on Software Engineering. ACM, New York, NY, 54\u201363."},{"key":"e_1_3_3_71_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2013.08.032"},{"key":"e_1_3_3_72_2","first-page":"68","volume-title":"Proceedings of the 42th Euromicro Conference on Software Engineering and Advanced Applications (SEAA\u201916)","author":"St\u00e5hl Daniel","unstructured":"Daniel St\u00e5hl, Kristofer Hall\u00e9n, and Jan Bosch. 2016. Continuous integration and delivery traceability in industry: Needs and practices. In Proceedings of the 42th Euromicro Conference on Software Engineering and Advanced Applications (SEAA\u201916). IEEE, Los Alamitos, CA, 68\u201372."},{"key":"e_1_3_3_73_2","first-page":"359","volume-title":"Proceedings of the 42nd International Conference on Software Engineering (ICSE\u201920)","author":"Stocco Andrea","unstructured":"Andrea Stocco, Michael Weiss, Marco Calzana, and Paolo Tonella. 2020. Misbehaviour prediction for autonomous driving systems. In Proceedings of the 42nd International Conference on Software Engineering (ICSE\u201920). 359\u2013371."},{"key":"e_1_3_3_74_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jcde.2019.04.002"},{"key":"e_1_3_3_75_2","first-page":"478","volume-title":"Complexity Challenges in Development of Cyber-Physical Systems","author":"T\u00f6rngren Martin","unstructured":"Martin T\u00f6rngren and Ulf Sellgren. 2018. Complexity Challenges in Development of Cyber-Physical Systems. Springer International, Cham, Switzerland, 478\u2013503."},{"key":"e_1_3_3_76_2","first-page":"805","volume-title":"Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (ESEC\/FSE\u201915)","author":"Vasilescu Bogdan","unstructured":"Bogdan Vasilescu, Yue Yu, Huaimin Wang, Premkumar T. Devanbu, and Vladimir Filkov. 2015. Quality and productivity outcomes relating to continuous integration in GitHub. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (ESEC\/FSE\u201915). ACM, New York, NY, 805\u2013816."},{"key":"e_1_3_3_77_2","doi-asserted-by":"publisher","DOI":"10.1145\/3106237.3106275"},{"key":"e_1_3_3_78_2","volume-title":"Proceedings of the 41st International Conference on Software Engineering (ICSE\u201919)","author":"Vassallo Carmine","unstructured":"Carmine Vassallo, Sebastian Proksch, Harald Gall, and Massimiliano Di Penta. 2019. Automated reporting of anti-patterns and decay in continuous integration. In Proceedings of the 41st International Conference on Software Engineering (ICSE\u201919). IEEE, Los Alamitos, CA."},{"key":"e_1_3_3_79_2","first-page":"327","volume-title":"Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering","author":"Vassallo Carmine","unstructured":"Carmine Vassallo, Sebastian Proksch, Anna Jancso, Harald C. Gall, and Massimiliano Di Penta. 2020. Configuration smells in continuous delivery pipelines: A linter and a six-month study on GitLab. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 327\u2013337."},{"key":"e_1_3_3_80_2","first-page":"41","volume-title":"Proceedings of the 32nd IEEE International Conference on Software Maintenance and Evolution (ICSME\u201916)","author":"Vassallo Carmine","unstructured":"Carmine Vassallo, Fiorella Zampetti, Daniele Romano, Moritz Beller, Annibale Panichella, Massimiliano Di Penta, and Andy Zaidman. 2016. Continuous delivery practices in a large financial organization. In Proceedings of the 32nd IEEE International Conference on Software Maintenance and Evolution (ICSME\u201916). 41\u201350."},{"key":"e_1_3_3_81_2","doi-asserted-by":"publisher","DOI":"10.1145\/3280987"},{"key":"e_1_3_3_82_2","first-page":"20","volume-title":"Proceedings of the 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE\u201921)","author":"Wang Dinghua","unstructured":"Dinghua Wang, Shuqing Li, Guanping Xiao, Yepang Liu, and Yulei Sui. 2021. An exploratory study of autopilot software bugs in unmanned aerial vehicles. In Proceedings of the 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC\/FSE\u201921). 20\u201331."},{"key":"e_1_3_3_83_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2020.3016826"},{"key":"e_1_3_3_84_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-021-10076-4"},{"key":"e_1_3_3_85_2","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.6337040"},{"key":"e_1_3_3_86_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-019-09785-8"},{"key":"e_1_3_3_87_2","first-page":"172","volume-title":"Runtime Verification","author":"Zapridou Eleni","unstructured":"Eleni Zapridou, Ezio Bartocci, and Panagiotis Katsaros. 2020. Runtime verification of autonomous driving systems in CARLA. In Runtime Verification. Lecture Notes in Computer Science, Vol. 12399. Springer, 172\u2013183."},{"key":"e_1_3_3_88_2","first-page":"50","volume-title":"Proceedings of the 43rd IEEE\/ACM International Conference on Software Engineering (ICSE\u201921)","author":"Zhang Peilun","unstructured":"Peilun Zhang, Yanjie Jiang, Anjiang Wei, Victoria Stodden, Darko Marinov, and August Shi. 2021. Domain-specific fixes for flaky tests with wrong assumptions on underdetermined specifications. In Proceedings of the 43rd IEEE\/ACM International Conference on Software Engineering (ICSE\u201921). 50\u201361."},{"key":"e_1_3_3_89_2","first-page":"385","volume-title":"Proceedings of the 2014 International Symposium on Software Testing and Analysis","author":"Zhang Sai","unstructured":"Sai Zhang, Darioush Jalali, Jochen Wuttke, K\u0131van\u00e7 Mu\u015flu, Wing Lam, Michael D. Ernst, and David Notkin. 2014. Empirically revisiting the test independence assumption. In Proceedings of the 2014 International Symposium on Software Testing and Analysis. 385\u2013396."},{"key":"e_1_3_3_90_2","first-page":"736","volume-title":"Proceedings of the 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME\u201920)","author":"Ziftci Celal","unstructured":"Celal Ziftci and Diego Cavalcanti. 2020. De-flake your tests: Automatically locating root causes of flaky tests in code at Google. In Proceedings of the 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME\u201920). 736\u2013745."},{"key":"e_1_3_3_91_2","doi-asserted-by":"publisher","DOI":"10.1002\/spe.2929"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3571854","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3571854","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T17:48:48Z","timestamp":1750182528000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3571854"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,4,26]]},"references-count":90,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2023,5,31]]}},"alternative-id":["10.1145\/3571854"],"URL":"https:\/\/doi.org\/10.1145\/3571854","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,4,26]]},"assertion":[{"value":"2022-03-08","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-10-18","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-04-26","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}