{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,1]],"date-time":"2026-07-01T03:17:06Z","timestamp":1782875826278,"version":"3.54.5"},"reference-count":92,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2024,4,17]],"date-time":"2024-04-17T00:00:00Z","timestamp":1713312000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"NSERC of Canada"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2024,5,31]]},"abstract":"<jats:p>\n            Test inputs fail not only when the system under test is faulty but also when the inputs are invalid or unrealistic. Failures resulting from invalid or unrealistic test inputs are spurious. Avoiding spurious failures improves the effectiveness of testing in exercising the main functions of a system, particularly for compute-intensive (CI) systems where a single test execution takes significant time. In this article, we propose to build failure models for inferring interpretable rules on test inputs that cause spurious failures. We examine two alternative strategies for building failure models: (1)\u00a0machine learning (ML)-guided test generation and (2)\u00a0surrogate-assisted test generation.\n            <jats:italic>ML-guided test generation<\/jats:italic>\n            infers boundary regions that separate passing and failing test inputs and samples test inputs from those regions.\n            <jats:italic>Surrogate-assisted test generation<\/jats:italic>\n            relies on surrogate models to predict labels for test inputs instead of exercising all the inputs. We propose a novel surrogate-assisted algorithm that uses multiple surrogate models simultaneously, and dynamically selects the prediction from the most accurate model. We empirically evaluate the accuracy of failure models inferred based on surrogate-assisted and ML-guided test generation algorithms. Using case studies from the domains of cyber-physical systems and networks, we show that our proposed surrogate-assisted approach generates failure models with an average accuracy of 83%, significantly outperforming ML-guided test generation and two baselines. Further, our approach learns failure-inducing rules that identify genuine spurious failures as validated against domain knowledge.\n          <\/jats:p>","DOI":"10.1145\/3638246","type":"journal-article","created":{"date-parts":[[2023,12,21]],"date-time":"2023-12-21T11:53:05Z","timestamp":1703159585000},"page":"1-32","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":6,"title":["Test Generation Strategies for Building Failure Models and Explaining Spurious Failures"],"prefix":"10.1145","volume":"33","author":[{"ORCID":"https:\/\/orcid.org\/0009-0006-0110-8488","authenticated-orcid":false,"given":"Baharin A.","family":"Jodat","sequence":"first","affiliation":[{"name":"University of Ottawa, Ottawa, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-5089-4059","authenticated-orcid":false,"given":"Abhishek","family":"Chandar","sequence":"additional","affiliation":[{"name":"University of Ottawa, Ottawa, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0281-8231","authenticated-orcid":false,"given":"Shiva","family":"Nejati","sequence":"additional","affiliation":[{"name":"University of Ottawa, Ottawa, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4711-8319","authenticated-orcid":false,"given":"Mehrdad","family":"Sabetzadeh","sequence":"additional","affiliation":[{"name":"University of Ottawa, Ottawa, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,4,17]]},"reference":[{"key":"e_1_3_1_2_2","volume-title":"Autopilot Online Benchmark","unstructured":"(Accessed: June 2023). Autopilot Online Benchmark. Retrieved from https:\/\/www.mathworks.com\/matlabcentral\/fileexchange\/41490-autopilot-demo-for-arp4754a-do-178c-and-do-331?focused=6796756&tab=model"},{"key":"e_1_3_1_3_2","volume-title":"Benchmark for Simulink Models","unstructured":"(Accessed: June 2023). Benchmark for Simulink Models. Retrieved from https:\/\/github.com\/anonpaper23\/testGenStrat\/tree\/main\/Benchmark\/Simulink%20Models"},{"key":"e_1_3_1_4_2","volume-title":"Code to Generate Results of each Research Questions","unstructured":"(Accessed: June 2023). Code to Generate Results of each Research Questions. Retrieved from https:\/\/github.com\/anonpaper23\/testGenStrat\/tree\/main\/Evaluation"},{"key":"e_1_3_1_5_2","volume-title":"Code to SoTA Implementation for NTSS Case Study","unstructured":"(Accessed: June 2023). Code to SoTA Implementation for NTSS Case Study. Retrieved from https:\/\/github.com\/anonpaper23\/testGenStrat\/blob\/main\/Code\/NTSS\/SoTA.py"},{"key":"e_1_3_1_6_2","volume-title":"Code to SoTA Implementation for Simulink Model Case Study","unstructured":"(Accessed: June 2023). Code to SoTA Implementation for Simulink Model Case Study. Retrieved from https:\/\/github.com\/anonpaper23\/testGenStrat\/blob\/main\/Code\/Simulink\/Algorithms\/decisiontreeSoTA.m"},{"key":"e_1_3_1_7_2","volume-title":"CPS and NTSS Requirements","unstructured":"(Accessed: June 2023). CPS and NTSS Requirements. Retrieved from https:\/\/github.com\/anonpaper23\/testGenStrat\/blob\/main\/Benchmark\/Formalization\/CPS_and_NTSS_Formalization.pdf"},{"key":"e_1_3_1_8_2","volume-title":"ENRICH \u2013 Non-robustnEss aNalysis for tRaffIC sHaping","unstructured":"(Accessed: June 2023). ENRICH \u2013 Non-robustnEss aNalysis for tRaffIC sHaping. Retrieved from https:\/\/github.com\/baharin\/ENRICH"},{"key":"e_1_3_1_9_2","volume-title":"Figure 16 to Figure 21 \u2013 Precision and Recall Results Obtained by Varying Time Budget in RQ2","unstructured":"(Accessed: June 2023). Figure 16 to Figure 21 \u2013 Precision and Recall Results Obtained by Varying Time Budget in RQ2. Retrieved from https:\/\/github.com\/anonpaper23\/testGenStrat\/blob\/main\/Supplementary_Material.pdf"},{"key":"e_1_3_1_10_2","volume-title":"Figure 9 \u2013 Comparing Dataset Sizes for Dynamic SA Algorithm and Seven Individual SA Algorithms in RQ1","unstructured":"(Accessed: June 2023). Figure 9 \u2013 Comparing Dataset Sizes for Dynamic SA Algorithm and Seven Individual SA Algorithms in RQ1. Retrieved from https:\/\/github.com\/anonpaper23\/testGenStrat\/blob\/main\/Supplementary_Material.pdf"},{"key":"e_1_3_1_11_2","volume-title":"Lockheed Martin","unstructured":"(Accessed: June 2023). Lockheed Martin. Retrieved from https:\/\/www.lockheedmartin.com"},{"key":"e_1_3_1_12_2","volume-title":"Logistic Regression","unstructured":"(Accessed: June 2023). Logistic Regression. Retrieved from http:\/\/faculty.cas.usf.edu\/mbrannick\/regression\/Logistic.html"},{"key":"e_1_3_1_13_2","volume-title":"OpenWrt","unstructured":"(Accessed: June 2023). OpenWrt. Retrieved from www.openwrt.org"},{"key":"e_1_3_1_14_2","volume-title":"Raw Datasets Obtained from each Algorithm for CPS and NTSS","unstructured":"(Accessed: June 2023). Raw Datasets Obtained from each Algorithm for CPS and NTSS. Retrieved from https:\/\/github.com\/anonpaper23\/testGenStrat\/tree\/main\/Data\/Dataset"},{"key":"e_1_3_1_15_2","volume-title":"Replication Package of Alhazen Framework","unstructured":"(Accessed: June 2023). Replication Package of Alhazen Framework. Retrieved from https:\/\/zenodo.org\/records\/3902142"},{"key":"e_1_3_1_16_2","volume-title":"Results of each Research Question.","unstructured":"(Accessed: June 2023). Results of each Research Question. Retrieved from https:\/\/github.com\/anonpaper23\/testGenStrat\/tree\/main\/Evaluation%20Results"},{"key":"e_1_3_1_17_2","volume-title":"Results of Statistical Analysis","unstructured":"(Accessed: June 2023). Results of Statistical Analysis. Retrieved from https:\/\/github.com\/anonpaper23\/testGenStrat\/blob\/main\/Evaluation%20Results\/RQ2\/RQ2StatisticalResults.xlsx"},{"key":"e_1_3_1_18_2","volume-title":"Rules Obtained for each CI Subject","unstructured":"(Accessed: June 2023). Rules Obtained for each CI Subject. Retrieved from https:\/\/github.com\/anonpaper23\/testGenStrat\/blob\/main\/Evaluation%20Results\/RQ4\/APandNTSS_Rules.xlsx"},{"key":"e_1_3_1_19_2","volume-title":"Source Codes of Algorithms for CPS and NTSS","unstructured":"(Accessed: June 2023). Source Codes of Algorithms for CPS and NTSS. Retrieved from https:\/\/github.com\/anonpaper23\/testGenStrat\/tree\/main\/Code"},{"key":"e_1_3_1_20_2","volume-title":"Table 15 to Table 20 \u2013 Average Accuracy, Recall and Precision Over all Runs of Algorithms by Varying Execution Time Budget in RQ2","unstructured":"(Accessed: June 2023). Table 15 to Table 20 \u2013 Average Accuracy, Recall and Precision Over all Runs of Algorithms by Varying Execution Time Budget in RQ2. Retrieved from https:\/\/github.com\/anonpaper23\/testGenStrat\/blob\/main\/Supplementary_Material.pdf"},{"key":"e_1_3_1_21_2","volume-title":"Table 21 to Table 24 \u2013 Full Set of Rules Obtained for NTSS, AP1, AP2 and AP3 in RQ4","unstructured":"(Accessed: June 2023). Table 21 to Table 24 \u2013 Full Set of Rules Obtained for NTSS, AP1, AP2 and AP3 in RQ4. Retrieved from https:\/\/github.com\/anonpaper23\/testGenStrat\/blob\/main\/Supplementary_Material.pdf"},{"key":"e_1_3_1_22_2","volume-title":"Table 3 \u2013 Parameter Names, Descriptions and Values used by SoTA.","unstructured":"(Accessed: June 2023). Table 3 \u2013 Parameter Names, Descriptions and Values used by SoTA. Retrieved from https:\/\/github.com\/anonpaper23\/testGenStrat\/blob\/main\/Supplementary_Material.pdf"},{"key":"e_1_3_1_23_2","volume-title":"Table 5 \u2013 Time Budgets Given to Non-CI Subjects in RQ1","unstructured":"(Accessed: June 2023). Table 5 \u2013 Time Budgets Given to Non-CI Subjects in RQ1. Retrieved from https:\/\/github.com\/anonpaper23\/testGenStrat\/blob\/main\/Supplementary_Material.pdf"},{"key":"e_1_3_1_24_2","volume-title":"Table 6 \u2013 Statistical Tests for Dataset Size and Percentage of Incorrect Labels Over Dataset Size in RQ1","unstructured":"(Accessed: June 2023). Table 6 \u2013 Statistical Tests for Dataset Size and Percentage of Incorrect Labels Over Dataset Size in RQ1. Retrieved from https:\/\/github.com\/anonpaper23\/testGenStrat\/blob\/main\/Supplementary_Material.pdf"},{"key":"e_1_3_1_25_2","volume-title":"Table 7 \u2013 Time Budget Considered for CI Subjects in RQ2","unstructured":"(Accessed: June 2023). Table 7 \u2013 Time Budget Considered for CI Subjects in RQ2. Retrieved from https:\/\/github.com\/anonpaper23\/testGenStrat\/blob\/main\/Supplementary_Material.pdf"},{"key":"e_1_3_1_26_2","volume-title":"Table 8 \u2013 Maximum Number of Test Executions for Non-CI Subjects in RQ2","unstructured":"(Accessed: June 2023). Table 8 \u2013 Maximum Number of Test Executions for Non-CI Subjects in RQ2. Retrieved from https:\/\/github.com\/anonpaper23\/testGenStrat\/blob\/main\/Supplementary_Material.pdf"},{"key":"e_1_3_1_27_2","volume-title":"Table 9 to Table 14 \u2013 Statistical Tests for Accuracy, Recall and Precision by Varying Execution Time Budget in RQ2","unstructured":"(Accessed: June 2023). Table 9 to Table 14 \u2013 Statistical Tests for Accuracy, Recall and Precision by Varying Execution Time Budget in RQ2. Retrieved from https:\/\/github.com\/anonpaper23\/testGenStrat\/blob\/main\/Supplementary_Material.pdf"},{"key":"e_1_3_1_28_2","unstructured":"(Accessed: June 2023). tc-cake. Retrieved from https:\/\/man7.org\/linux\/man-pages\/man8\/tc-cake.8.html"},{"key":"e_1_3_1_29_2","first-page":"254","volume-title":"Proceedings of the Tools and Algorithms for the Construction and Analysis of Systems.","author":"Annpureddy Yashwanth","year":"2011","unstructured":"Yashwanth Annpureddy, Che Liu, Georgios Fainekos, and Sriram Sankaranarayanan. 2011. S-TaLiRo: A tool for temporal logic falsification for hybrid systems. In Proceedings of the Tools and Algorithms for the Construction and Analysis of Systems.Parosh Aziz Abdulla and K. Rustan M. Leino (Eds.), Springer, 254\u2013257."},{"key":"e_1_3_1_30_2","doi-asserted-by":"crossref","first-page":"137","DOI":"10.1016\/j.infsof.2019.06.009","article-title":"Pareto efficient multi-objective black-box test case selection for simulation-based testing","volume":"114","author":"Arrieta Aitor","year":"2019","unstructured":"Aitor Arrieta, Shuai Wang, Urtzi Markiegi, Ainhoa Arruabarrena, Leire Etxeberria, and Goiuria Sagardui. 2019. Pareto efficient multi-objective black-box test case selection for simulation-based testing. Information and Software Technology 114 (2019), 137\u2013154.","journal-title":"Information and Software Technology"},{"key":"e_1_3_1_31_2","doi-asserted-by":"crossref","first-page":"688","DOI":"10.1109\/CEC.2017.7969377","volume-title":"Proceedings of the 2017 IEEE Congress on Evolutionary Computation","author":"Arrieta Aitor","year":"2017","unstructured":"Aitor Arrieta, Shuai Wang, Urtzi Markiegi, Goiuria Sagardui, and Leire Etxeberria. 2017. Search-based test case generation for cyber-physical systems. In Proceedings of the 2017 IEEE Congress on Evolutionary Computation. IEEE, 688\u2013697."},{"key":"e_1_3_1_32_2","volume-title":"Advanced Avionics Handbook","author":"(ASA) Federal Aviation Administration (FAA)\/Aviation Supplies & Academics","year":"2009","unstructured":"Federal Aviation Administration (FAA)\/Aviation Supplies & Academics (ASA). 2009. Advanced Avionics Handbook. Aviation Supplies & Academics, Incorporated. Retrieved from https:\/\/books.google.lu\/books?id=2xGuPwAACAAJ"},{"key":"e_1_3_1_33_2","volume-title":"Proceedings of the NDSS","author":"Aschermann Cornelius","year":"2019","unstructured":"Cornelius Aschermann, Tommaso Frassetto, Thorsten Holz, Patrick Jauernig, Ahmad-Reza Sadeghi, and Daniel Teuchert. 2019. NAUTILUS: Fishing for deep bugs with grammars. In Proceedings of the NDSS."},{"issue":"6","key":"e_1_3_1_34_2","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1145\/3140587.3062349","article-title":"Synthesizing program input grammars","volume":"52","author":"Bastani Osbert","year":"2017","unstructured":"Osbert Bastani, Rahul Sharma, Alex Aiken, and Percy Liang. 2017. Synthesizing program input grammars. ACM SIGPLAN Notices 52, 6 (2017), 95\u2013110.","journal-title":"ACM SIGPLAN Notices"},{"key":"e_1_3_1_35_2","first-page":"1","volume-title":"Proceedings of the 2017 IEEE 20th International Conference on Intelligent Transportation Systems","author":"Beglerovic Halil","year":"2017","unstructured":"Halil Beglerovic, Michael Stolz, and Martin Horn. 2017. Testing of autonomous vehicles using surrogate models and stochastic optimization. In Proceedings of the 2017 IEEE 20th International Conference on Intelligent Transportation Systems. IEEE, 1\u20136."},{"key":"e_1_3_1_36_2","doi-asserted-by":"crossref","first-page":"63","DOI":"10.1145\/2970276.2970311","volume-title":"Proceedings of the 31st IEEE\/ACM International Conference on Automated Software Engineering","author":"Abdessalem Raja Ben","year":"2016","unstructured":"Raja Ben Abdessalem, Shiva Nejati, Lionel C. Briand, and Thomas Stifter. 2016. Testing advanced driver assistance systems using multi-objective search and neural networks. In Proceedings of the 31st IEEE\/ACM International Conference on Automated Software Engineering. 63\u201374."},{"key":"e_1_3_1_37_2","first-page":"274","volume-title":"Proceedings of the 2020 IEEE 13th International Conference on Software Testing, Validation and Verification","author":"B\u00f6hme Marcel","year":"2020","unstructured":"Marcel B\u00f6hme, Charaka Geethal, and Van-Thuan Pham. 2020. Human-in-the-loop automatic program repair. In Proceedings of the 2020 IEEE 13th International Conference on Software Testing, Validation and Verification. IEEE, 274\u2013285."},{"key":"e_1_3_1_38_2","doi-asserted-by":"crossref","first-page":"801","DOI":"10.1145\/3377811.3380344","volume-title":"Proceedings of the ACM\/IEEE 42nd International Conference on Software Engineering","author":"Brindescu Caius","year":"2020","unstructured":"Caius Brindescu, Iftekhar Ahmed, Rafael Leano, and Anita Sarma. 2020. Planning for untangling: Predicting the difficulty of merge conflicts. In Proceedings of the ACM\/IEEE 42nd International Conference on Software Engineering. 801\u2013811."},{"key":"e_1_3_1_39_2","doi-asserted-by":"crossref","DOI":"10.1201\/9781315218335","volume-title":"Modeling and Simulation of Systems using MATLAB\u00ae and Simulink\u00ae","author":"Chaturvedi Devendra K.","year":"2017","unstructured":"Devendra K. Chaturvedi. 2017. Modeling and Simulation of Systems using MATLAB\u00ae and Simulink\u00ae. CRC press."},{"key":"e_1_3_1_40_2","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1613\/jair.953","article-title":"SMOTE: synthetic minority over-sampling technique","volume":"16","author":"Chawla Nitesh V.","year":"2002","unstructured":"Nitesh V. Chawla, Kevin W. Bowyer, Lawrence O. Hall, and W. Philip Kegelmeyer. 2002. SMOTE: synthetic minority over-sampling technique. Journal of Artificial Intelligence Research 16 (2002), 321\u2013357.","journal-title":"Journal of Artificial Intelligence Research"},{"key":"e_1_3_1_41_2","doi-asserted-by":"crossref","first-page":"115","DOI":"10.1016\/B978-1-55860-377-6.50023-2","volume-title":"Proceedings of the Machine Learning Proceedings 1995","author":"Cohen William W.","year":"1995","unstructured":"William W. Cohen. 1995. Fast effective rule induction. In Proceedings of the Machine Learning Proceedings 1995. Elsevier, 115\u2013123."},{"key":"e_1_3_1_42_2","first-page":"337","volume-title":"Proceedings of the International Conference on Tools and Algorithms for the Construction and Analysis of Systems","author":"Moura Leonardo De","year":"2008","unstructured":"Leonardo De Moura and Nikolaj Bj\u00f8rner. 2008. Z3: An efficient SMT solver. In Proceedings of the International Conference on Tools and Algorithms for the Construction and Analysis of Systems. Springer, 337\u2013340."},{"key":"e_1_3_1_43_2","doi-asserted-by":"crossref","first-page":"109","DOI":"10.1145\/2502524.2502540","volume-title":"Proceedings of the ACM\/IEEE 4th International Conference on Cyber-Physical Systems","author":"Derler Patricia","year":"2013","unstructured":"Patricia Derler, Edward A. Lee, Stavros Tripakis, and Martin T\u00f6rngren. 2013. Cyber-physical system design contracts. In Proceedings of the ACM\/IEEE 4th International Conference on Cyber-Physical Systems. 109\u2013118."},{"key":"e_1_3_1_44_2","article-title":"A review of surrogate assisted multiobjective evolutionary algorithms","volume":"2016","author":"D\u00edaz-Manr\u00edquez Alan","year":"2016","unstructured":"Alan D\u00edaz-Manr\u00edquez, Gregorio Toscano, Jose Hugo Barron-Zambrano, and Edgar Tello-Leal. 2016. A review of surrogate assisted multiobjective evolutionary algorithms. Computational Intelligence and Neuroscience 2016 (2016), 14 Pages.","journal-title":"Computational Intelligence and Neuroscience"},{"key":"e_1_3_1_45_2","doi-asserted-by":"crossref","first-page":"583","DOI":"10.1145\/3449639.3459306","volume-title":"Proceedings of the Genetic and Evolutionary Computation Conference","author":"Dushatskiy Arkadiy","year":"2021","unstructured":"Arkadiy Dushatskiy, Tanja Alderliesten, and Peter A. N. Bosman. 2021. A novel surrogate-assisted evolutionary algorithm applied to partition-based ensemble learning. In Proceedings of the Genetic and Evolutionary Computation Conference. 583\u2013591."},{"key":"e_1_3_1_46_2","first-page":"537","volume-title":"Proceedings of the 13th International Workshop on Search-Based Software Testing","author":"Feldt Robert","year":"2020","unstructured":"Robert Feldt and Shin Yoo. 2020. Flexible probabilistic modeling for search based test data generation. In Proceedings of the 13th International Workshop on Search-Based Software Testing. 537\u2013540."},{"key":"e_1_3_1_47_2","first-page":"131","article-title":"Building ensembles of surrogates by optimal convex combination","author":"Friese Martina","year":"2016","unstructured":"Martina Friese, Thomas Bartz-Beielstein, and Michael Emmerich. 2016. Building ensembles of surrogates by optimal convex combination. Bioinspired Optimization Methods and their Applications (2016), 131\u2013143.","journal-title":"Bioinspired Optimization Methods and their Applications"},{"key":"e_1_3_1_48_2","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1145\/3368089.3409737","volume-title":"Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering","author":"Gaaloul Khouloud","year":"2020","unstructured":"Khouloud Gaaloul, Claudio Menghi, Shiva Nejati, Lionel C. Briand, and David Wolfe. 2020. Mining assumptions for software components using machine learning. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 159\u2013171."},{"key":"e_1_3_1_49_2","doi-asserted-by":"crossref","first-page":"789","DOI":"10.1109\/ICSE.2015.91","volume-title":"Proceedings of the 2015 IEEE\/ACM 37th IEEE International Conference on Software Engineering","author":"Ghotra Baljinder","year":"2015","unstructured":"Baljinder Ghotra, Shane McIntosh, and Ahmed E. Hassan. 2015. Revisiting the impact of classification techniques on the performance of defect prediction models. In Proceedings of the 2015 IEEE\/ACM 37th IEEE International Conference on Software Engineering. IEEE, 789\u2013800."},{"key":"e_1_3_1_50_2","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1109\/ASE.2002.1114984","volume-title":"Proceedings of the International Conference on Automated Software Engineering","author":"Giannakopoulou Dimitra","year":"2002","unstructured":"Dimitra Giannakopoulou, Corina S. Pasareanu, and Howard Barringer. 2002. Assumption generation for software component verification. In Proceedings of the International Conference on Automated Software Engineering. IEEE, 3\u201312."},{"key":"e_1_3_1_51_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2021.106590"},{"key":"e_1_3_1_52_2","doi-asserted-by":"crossref","first-page":"237","DOI":"10.1145\/3395363.3397349","volume-title":"Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis","author":"Gopinath Rahul","year":"2020","unstructured":"Rahul Gopinath, Alexander Kampmann, Nikolas Havrikov, Ezekiel O. Soremekun, and Andreas Zeller. 2020. Abstracting failure-inducing inputs. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis. 237\u2013248."},{"issue":"4","key":"e_1_3_1_53_2","doi-asserted-by":"crossref","first-page":"242","DOI":"10.1147\/sj.94.0242","article-title":"Automatic generation of test cases","volume":"9","author":"Hanford Kenneth V.","year":"1970","unstructured":"Kenneth V. Hanford. 1970. Automatic generation of test cases. IBM Systems Journal 9, 4 (1970), 242\u2013257.","journal-title":"IBM Systems Journal"},{"issue":"5","key":"e_1_3_1_54_2","doi-asserted-by":"crossref","first-page":"90","DOI":"10.1007\/s10664-021-09982-4","article-title":"Can offline testing of deep neural networks replace their online testing? A case study of automated driving systems","volume":"26","author":"Haq Fitash Ul","year":"2021","unstructured":"Fitash Ul Haq, Donghwan Shin, Shiva Nejati, and Lionel Briand. 2021. Can offline testing of deep neural networks replace their online testing? A case study of automated driving systems. Empirical Software Engineering 26, 5 (2021), 90.","journal-title":"Empirical Software Engineering"},{"key":"e_1_3_1_55_2","doi-asserted-by":"crossref","first-page":"182","DOI":"10.1109\/ICSTW.2010.31","volume-title":"Proceedings of the 2010 3rd International Conference on Software Testing, Verification, and Validation Workshops","author":"Harman Mark","year":"2010","unstructured":"Mark Harman, Sung Gon Kim, Kiran Lakhotia, Phil McMinn, and Shin Yoo. 2010. Optimizing for the number of tests generated in search based test data generation with an application to the oracle cost problem. In Proceedings of the 2010 3rd International Conference on Software Testing, Verification, and Validation Workshops. IEEE, 182\u2013191."},{"issue":"2","key":"e_1_3_1_56_2","doi-asserted-by":"crossref","first-page":"226","DOI":"10.1109\/TSE.2009.71","article-title":"A theoretical and empirical study of search-based testing: Local, global, and hybrid search","volume":"36","author":"Harman Mark","year":"2009","unstructured":"Mark Harman and Phil McMinn. 2009. A theoretical and empirical study of search-based testing: Local, global, and hybrid search. IEEE Transactions on Software Engineering 36, 2 (2009), 226\u2013247.","journal-title":"IEEE Transactions on Software Engineering"},{"key":"e_1_3_1_57_2","doi-asserted-by":"crossref","first-page":"440","DOI":"10.1007\/BFb0028765","volume-title":"Proceedings of the Computer Aided Verification: 10th International Conference, CAV\u201998 Vancouver, BC, Canada, June 28\u2013July 2, 1998 Proceedings 10","author":"Henzinger Thomas A.","year":"1998","unstructured":"Thomas A. Henzinger, Shaz Qadeer, and Sriram K. Rajamani. 1998. You assume, we guarantee: Methodology and case studies. In Proceedings of the Computer Aided Verification: 10th International Conference, CAV\u201998 Vancouver, BC, Canada, June 28\u2013July 2, 1998 Proceedings 10. Springer, 440\u2013451."},{"key":"e_1_3_1_58_2","first-page":"37","volume-title":"Proceedings of the 2018 IEEE International Symposium on Local and Metropolitan Area Networks.","author":"H\u00f8iland-J\u00f8rgensen Toke","year":"2018","unstructured":"Toke H\u00f8iland-J\u00f8rgensen, Dave T\u00e4ht, and Jonathan Morton. 2018. Piece of CAKE: A comprehensive queue management solution for home gateways. In Proceedings of the 2018 IEEE International Symposium on Local and Metropolitan Area Networks.IEEE, 37\u201342."},{"key":"e_1_3_1_59_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cma.2022.114835"},{"key":"e_1_3_1_60_2","first-page":"1145","volume-title":"Proceedings of the 44th International Conference on Software Engineering.","author":"Hu Boyue Caroline","year":"2022","unstructured":"Boyue Caroline Hu, Lina Marsso, Krzysztof Czarnecki, Rick Salay, Huakun Shen, and Marsha Chechik. 2022. If a human can see it, so should your system: Reliability requirements for machine vision components. In Proceedings of the 44th International Conference on Software Engineering.Association for Computing Machinery, New York, NY, 1145\u20131156. DOI:DOI:10.1145\/3510003.3510109"},{"key":"e_1_3_1_61_2","first-page":"16","volume-title":"Proceedings of the 2021 IEEE\/ACM 14th International Workshop on Search-Based Software Testing","author":"Humeniuk Dmytro","year":"2021","unstructured":"Dmytro Humeniuk, Giuliano Antoniol, and Foutse Khomh. 2021. Data driven testing of cyber physical systems. In Proceedings of the 2021 IEEE\/ACM 14th International Workshop on Search-Based Software Testing. IEEE, 16\u201319."},{"key":"e_1_3_1_62_2","doi-asserted-by":"crossref","first-page":"106936","DOI":"10.1016\/j.infsof.2022.106936","article-title":"A search-based framework for automatic generation of testing environments for cyber-physical systems","author":"Humeniuk Dmytro","year":"2022","unstructured":"Dmytro Humeniuk, Foutse Khomh, and Giuliano Antoniol. 2022. A search-based framework for automatic generation of testing environments for cyber-physical systems. Information and Software Technology 149 (2022), 106936.","journal-title":"Information and Software Technology"},{"issue":"1","key":"e_1_3_1_63_2","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1007\/s00500-003-0328-5","article-title":"A comprehensive survey of fitness approximation in evolutionary computation","volume":"9","author":"Jin Yaochu","year":"2005","unstructured":"Yaochu Jin. 2005. A comprehensive survey of fitness approximation in evolutionary computation. Soft Computing 9, 1 (2005), 3\u201312.","journal-title":"Soft Computing"},{"key":"e_1_3_1_64_2","volume-title":"Proceedings of the GECCO","author":"Jin Yaochu","year":"2002","unstructured":"Yaochu Jin and Bernhard Sendhoff. 2002. Fitness approximation in evolutionary computation-a survey. In Proceedings of the GECCO. 1105\u201312."},{"key":"e_1_3_1_65_2","doi-asserted-by":"crossref","first-page":"386","DOI":"10.1109\/ICST57152.2023.00043","volume-title":"Proceedings of the 2023 IEEE Conference on Software Testing, Verification and Validation","author":"Jodat Baharin A.","year":"2023","unstructured":"Baharin A. Jodat, Shiva Nejati, Mehrdad Sabetzadeh, and Patricio Saavedra. 2023. Learning non-robustness using simulation-based testing: A network traffic-shaping case study. In Proceedings of the 2023 IEEE Conference on Software Testing, Verification and Validation. IEEE, 386\u2013397."},{"key":"e_1_3_1_66_2","doi-asserted-by":"crossref","first-page":"1228","DOI":"10.1145\/3368089.3409687","volume-title":"Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering","author":"Kampmann Alexander","year":"2020","unstructured":"Alexander Kampmann, Nikolas Havrikov, Ezekiel O. Soremekun, and Andreas Zeller. 2020. When does my program do this? learning circumstances of software behavior. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1228\u20131239."},{"key":"e_1_3_1_67_2","doi-asserted-by":"crossref","first-page":"215","DOI":"10.1145\/3533767.3534406","volume-title":"Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis","author":"Kapugama Charaka Geethal","year":"2022","unstructured":"Charaka Geethal Kapugama, Van-Thuan Pham, Aldeida Aleti, and Marcel B\u00f6hme. 2022. Human-in-the-loop oracle learning for semantic bugs in string processing programs. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis. 215\u2013226."},{"issue":"2","key":"e_1_3_1_68_2","doi-asserted-by":"crossref","first-page":"928","DOI":"10.1007\/s10664-015-9422-4","article-title":"Generating valid grammar-based test inputs by means of genetic programming and annotated grammars","volume":"22","author":"Kifetew Fitsum Meshesha","year":"2017","unstructured":"Fitsum Meshesha Kifetew, Roberto Tiella, and Paolo Tonella. 2017. Generating valid grammar-based test inputs by means of genetic programming and annotated grammars. Empirical Software Engineering 22, 2 (2017), 928\u2013961.","journal-title":"Empirical Software Engineering"},{"key":"e_1_3_1_69_2","first-page":"456","volume-title":"Proceedings of the 2021 36th IEEE\/ACM International Conference on Automated Software Engineering","author":"Kulkarni Neil","year":"2021","unstructured":"Neil Kulkarni, Caroline Lemieux, and Koushik Sen. 2021. Learning highly recursive input grammars. In Proceedings of the 2021 36th IEEE\/ACM International Conference on Automated Software Engineering. IEEE, 456\u2013467."},{"key":"e_1_3_1_70_2","article-title":"Estimating probabilistic safe WCET ranges of real-time systems at design stages","author":"Lee Jaekwon","year":"2022","unstructured":"Jaekwon Lee, Seung Yeob Shin, Shiva Nejati, Lionel C. Bsriand, and Yago Isasi Parache. 2022. Estimating probabilistic safe WCET ranges of real-time systems at design stages. ACM Transactions on Software Engineering and Methodology 32, 2 (2022), 1\u201333.","journal-title":"ACM Transactions on Software Engineering and Methodology"},{"key":"e_1_3_1_71_2","volume-title":"Essentials of Metaheuristics (2nd. ed.)","author":"Luke Sean","year":"2013","unstructured":"Sean Luke. 2013. Essentials of Metaheuristics (2nd. ed.). Lulu. Retrieved from http:\/\/cs.gmu.edu\/sean\/book\/metaheuristics\/"},{"key":"e_1_3_1_72_2","first-page":"938","volume-title":"Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC\/FSE 2017, Paderborn, Germany, September 4-8, 2017","author":"Matinnejad Reza","year":"2017","unstructured":"Reza Matinnejad, Shiva Nejati, and Lionel C. Briand. 2017. Automated testing of hybrid Simulink\/Stateflow controllers: Industrial case studies. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC\/FSE 2017, Paderborn, Germany, September 4-8, 2017. ACM, 938\u2013943."},{"key":"e_1_3_1_73_2","first-page":"1","article-title":"Mann-Whitney U test","author":"McKnight Patrick E.","year":"2010","unstructured":"Patrick E. McKnight and Julius Najab. 2010. Mann-Whitney U test. The Corsini Encyclopedia of Psychology (2010), 1\u20131.","journal-title":"The Corsini Encyclopedia of Psychology"},{"key":"e_1_3_1_74_2","first-page":"372","volume-title":"Proceedings of the 2020 IEEE\/ACM 42nd International Conference on Software Engineering","author":"Menghi Claudio","year":"2020","unstructured":"Claudio Menghi, Shiva Nejati, Lionel Briand, and Yago Isasi Parache. 2020. Approximation-refinement testing of compute-intensive cyber-physical models: An approach based on system identification. In Proceedings of the 2020 IEEE\/ACM 42nd International Conference on Software Engineering. IEEE, 372\u2013384."},{"key":"e_1_3_1_75_2","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1145\/3338906.3338920","volume-title":"Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering","author":"Menghi Claudio","year":"2019","unstructured":"Claudio Menghi, Shiva Nejati, Khouloud Gaaloul, and Lionel C. Briand. 2019. Generating automated and online test oracles for simulink models with continuous and uncertain behaviors. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 27\u201338."},{"issue":"12","key":"e_1_3_1_76_2","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1145\/96267.96279","article-title":"An empirical study of the reliability of UNIX utilities","volume":"33","author":"Miller Barton P.","year":"1990","unstructured":"Barton P. Miller, Lars Fredriksen, and Bryan So. 1990. An empirical study of the reliability of UNIX utilities. Communication of the ACM 33, 12 (1990), 32\u201344.","journal-title":"Communication of the ACM"},{"key":"e_1_3_1_77_2","volume-title":"Interpretable Machine Learning","author":"Molnar Christoph","year":"2020","unstructured":"Christoph Molnar. 2020. Interpretable Machine Learning. Lulu. com."},{"key":"e_1_3_1_78_2","doi-asserted-by":"crossref","first-page":"1015","DOI":"10.1145\/3338906.3340444","volume-title":"Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering","author":"Nejati Shiva","year":"2019","unstructured":"Shiva Nejati, Khouloud Gaaloul, Claudio Menghi, Lionel C. Briand, Stephen Foster, and David Wolfe. 2019. Evaluating model testing and model checking for finding requirements violations in Simulink models. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1015\u20131025."},{"key":"e_1_3_1_79_2","doi-asserted-by":"crossref","first-page":"107286","DOI":"10.1016\/j.infsof.2023.107286","article-title":"Reflections on surrogate-assisted search-based testing: A taxonomy and two replication studies based on industrial ADAS and simulink models","volume":"163","author":"Nejati Shiva","year":"2023","unstructured":"Shiva Nejati, Lev Sorokin, Damir Safin, Federico Formica, Mohammad Mahdi Mahboob, and Claudio Menghi. 2023. Reflections on surrogate-assisted search-based testing: A taxonomy and two replication studies based on industrial ADAS and simulink models. Information and Software Technology 163 (2023), 107286.","journal-title":"Information and Software Technology"},{"key":"e_1_3_1_80_2","article-title":"Machine learning yearning","author":"Ng Andrew","year":"2018","unstructured":"Andrew Ng. 2018. Machine learning yearning. Retrieved from http:\/\/www.mlyearning.org\/. Accessed June 2023.","journal-title":"R"},{"key":"e_1_3_1_81_2","doi-asserted-by":"crossref","first-page":"380","DOI":"10.1007\/978-3-030-05366-6_31","volume-title":"International Conference on Distributed Computing and Internet Technology","author":"Patgiri Ripon","year":"2019","unstructured":"Ripon Patgiri, Hemanth Katari, Ronit Kumar, and Dheeraj Sharma. 2019. Empirical study on malicious URL detection using machine learning. In Proceedings of theInternational Conference on Distributed Computing and Internet Technology. Springer, 380\u2013388."},{"key":"e_1_3_1_82_2","doi-asserted-by":"crossref","unstructured":"Vincenzo Riccio and Paolo Tonella. 2023. When and why test generators for deep learning produce invalid inputs: an empirical study. In IEEE\/ACM 45th International Conference on Software Engineering (ICSE\u201923) IEEE 1161\u20131173.","DOI":"10.1109\/ICSE48619.2023.00104"},{"issue":"3","key":"e_1_3_1_83_2","doi-asserted-by":"crossref","first-page":"217","DOI":"10.3166\/ejc.18.217-238","article-title":"Taming Dr. Frankenstein: Contract-based design for cyber-physical systems","volume":"18","author":"Sangiovanni-Vincentelli Alberto","year":"2012","unstructured":"Alberto Sangiovanni-Vincentelli, Werner Damm, and Roberto Passerone. 2012. Taming Dr. Frankenstein: Contract-based design for cyber-physical systems. European Journal of Control 18, 3 (2012), 217\u2013238.","journal-title":"European Journal of Control"},{"key":"e_1_3_1_84_2","first-page":"47","volume-title":"Proceedings of the International Conference on Model Driven Engineering Languages and Systems Companion Proceedings","author":"Schaap Alexander","year":"2018","unstructured":"Alexander Schaap, Gordon Marks, Vera Pantelic, Mark Lawford, Gehan Selim, Alan Wassyng, and Lucian Patcas. 2018. Documenting simulink designs of embedded systems. In Proceedings of the International Conference on Model Driven Engineering Languages and Systems Companion Proceedings. ACM, 47\u201351."},{"key":"e_1_3_1_85_2","article-title":"Practical bayesian optimization of machine learning algorithms","volume":"25","author":"Snoek Jasper","year":"2012","unstructured":"Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. 2012. Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (2012), 2960\u20132968.","journal-title":"Advances in Neural Information Processing Systems"},{"issue":"2","key":"e_1_3_1_86_2","doi-asserted-by":"crossref","first-page":"213","DOI":"10.1007\/s00530-014-0446-1","article-title":"Mean opinion score (MOS) revisited: Methods and applications, limitations and alternatives","volume":"22","author":"Streijl Robert C.","year":"2016","unstructured":"Robert C. Streijl, Stefan Winkler, and David S. Hands. 2016. Mean opinion score (MOS) revisited: Methods and applications, limitations and alternatives. Multimedia Systems 22, 2 (2016), 213\u2013227.","journal-title":"Multimedia Systems"},{"key":"e_1_3_1_87_2","doi-asserted-by":"crossref","first-page":"414","DOI":"10.1016\/j.ins.2021.03.002","article-title":"Surrogate models in evolutionary single-objective optimization: A new taxonomy and experimental study","volume":"562","author":"Tong Hao","year":"2021","unstructured":"Hao Tong, Changwu Huang, Leandro L. Minku, and Xin Yao. 2021. Surrogate models in evolutionary single-objective optimization: A new taxonomy and experimental study. Information Sciences 562 (2021), 414\u2013437.","journal-title":"Information Sciences"},{"issue":"2","key":"e_1_3_1_88_2","doi-asserted-by":"crossref","first-page":"265","DOI":"10.1109\/TIV.2019.2955903","article-title":"Requirements-driven test generation for autonomous vehicles with machine learning components","volume":"5","author":"Tuncali Cumhur Erkan","year":"2019","unstructured":"Cumhur Erkan Tuncali, Georgios Fainekos, Danil Prokhorov, Hisahiro Ito, and James Kapinski. 2019. Requirements-driven test generation for autonomous vehicles with machine learning components. IEEE Transactions on Intelligent Vehicles 5, 2 (2019), 265\u2013280.","journal-title":"IEEE Transactions on Intelligent Vehicles"},{"issue":"2","key":"e_1_3_1_89_2","first-page":"101","article-title":"A critique and improvement of the CL common language effect size statistics of McGraw and Wong","volume":"25","author":"Vargha Andr\u00e1s","year":"2000","unstructured":"Andr\u00e1s Vargha and Harold D. Delaney. 2000. A critique and improvement of the CL common language effect size statistics of McGraw and Wong. Journal of Educational and Behavioral Statistics 25, 2 (2000), 101\u2013132.","journal-title":"Journal of Educational and Behavioral Statistics"},{"key":"e_1_3_1_90_2","first-page":"724","volume-title":"Proceedings of the 2019 IEEE\/ACM 41st International Conference on Software Engineering","author":"Wang Junjie","year":"2019","unstructured":"Junjie Wang, Bihuan Chen, Lei Wei, and Yang Liu. 2019. Superion: Grammar-aware greybox fuzzing. In Proceedings of the 2019 IEEE\/ACM 41st International Conference on Software Engineering. IEEE, 724\u2013735."},{"issue":"8","key":"e_1_3_1_91_2","doi-asserted-by":"crossref","first-page":"e0237749","DOI":"10.1371\/journal.pone.0237749","article-title":"A systematic review of fuzzing based on machine learning techniques","volume":"15","author":"Wang Yan","year":"2020","unstructured":"Yan Wang, Peng Jia, Luping Liu, Cheng Huang, and Zhonglin Liu. 2020. A systematic review of fuzzing based on machine learning techniques. PloS One 15, 8 (2020), e0237749.","journal-title":"PloS One"},{"key":"e_1_3_1_92_2","volume-title":"Data Mining: Practical Machine Learning Tools and Techniques (3rd. ed.)","author":"Witten Ian H.","year":"2011","unstructured":"Ian H. Witten, Eibe Frank, and Mark A. Hall. 2011. Data Mining: Practical Machine Learning Tools and Techniques (3rd. ed.). Morgan Kaufmann, Amsterdam. Retrieved from http:\/\/www.sciencedirect.com\/science\/book\/9780123748560"},{"key":"e_1_3_1_93_2","article-title":"An ensemble of adaptive surrogate models based on local error expectations","author":"Xu Huanwei","year":"2021","unstructured":"Huanwei Xu, Xin Zhang, Hao Li, and Ge Xiang. 2021. An ensemble of adaptive surrogate models based on local error expectations. Mathematical Problems in Engineering 2021, Article ID 8857417 (2021), 14 pages.","journal-title":"Mathematical Problems in Engineering"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3638246","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3638246","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T22:53:35Z","timestamp":1750287215000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3638246"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,17]]},"references-count":92,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2024,5,31]]}},"alternative-id":["10.1145\/3638246"],"URL":"https:\/\/doi.org\/10.1145\/3638246","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,4,17]]},"assertion":[{"value":"2023-06-15","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-12-08","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-04-17","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}