{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,11]],"date-time":"2026-04-11T02:21:42Z","timestamp":1775874102034,"version":"3.50.1"},"reference-count":222,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2021,5,3]],"date-time":"2021-05-03T00:00:00Z","timestamp":1620000000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Comput. Surv."],"published-print":{"date-parts":[[2022,5,31]]},"abstract":"<jats:p>Robustness is the degree to which a certain system or component can operate correctly in the presence of invalid inputs or stressful environmental conditions. With the increasing complexity and widespread use of computer systems, obtaining assurances regarding their robustness has become of vital importance. This survey discusses the state of the art on software robustness assessment, with emphasis on key aspects like types of systems being evaluated, assessment techniques used, the target of the techniques, the types of faults used, and how system behavior is classified. The survey concludes with the identification of gaps and open challenges related with robustness assessment.<\/jats:p>","DOI":"10.1145\/3448977","type":"journal-article","created":{"date-parts":[[2021,5,4]],"date-time":"2021-05-04T03:42:46Z","timestamp":1620099766000},"page":"1-65","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":17,"title":["A Systematic Review on Software Robustness Assessment"],"prefix":"10.1145","volume":"54","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0011-9901","authenticated-orcid":false,"given":"Nuno","family":"Laranjeiro","sequence":"first","affiliation":[{"name":"University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Coimbra, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6051-3516","authenticated-orcid":false,"given":"Jo\u00e3o","family":"Agnelo","sequence":"additional","affiliation":[{"name":"University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Coimbra, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9660-2011","authenticated-orcid":false,"given":"Jorge","family":"Bernardino","sequence":"additional","affiliation":[{"name":"Polytechnic of Coimbra, Coimbra Institute of Engineering (ISEC), Portugal and University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Portugal"}]}],"member":"320","published-online":{"date-parts":[[2021,5,3]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.2006.29"},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCKE48569.2019.8964875"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/DSN.2004.1311957"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEIP.2017.17"},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/WCCAIS.2014.6916629"},{"key":"e_1_2_2_6_1","volume-title":"Proceedings of the 14th International Conference on Distributed Computing and Artificial Intelligence, Sigeru Omatu, Sara Rodr\u00edguez, Gabriel Villarrubia, Pedro Faria, Pawe\u0142 Sitek","author":"Arcile Johan","unstructured":"Johan Arcile , Raymond Devillers , Hanna Klaudel , Witold Klaudel , and Bo\u017cena Wo\u017ana-Szcze\u015bniak . 2018. Modeling and checking robustness of communicating autonomous vehicles . In Proceedings of the 14th International Conference on Distributed Computing and Artificial Intelligence, Sigeru Omatu, Sara Rodr\u00edguez, Gabriel Villarrubia, Pedro Faria, Pawe\u0142 Sitek , and Javier Prieto (Eds.). Springer International Publishing , Cham , 173--180. DOI:https:\/\/doi.org\/10.1007\/978-3-319-62410-5_21 10.1007\/978-3-319-62410-5_21 Johan Arcile, Raymond Devillers, Hanna Klaudel, Witold Klaudel, and Bo\u017cena Wo\u017ana-Szcze\u015bniak. 2018. Modeling and checking robustness of communicating autonomous vehicles. In Proceedings of the 14th International Conference on Distributed Computing and Artificial Intelligence, Sigeru Omatu, Sara Rodr\u00edguez, Gabriel Villarrubia, Pedro Faria, Pawe\u0142 Sitek, and Javier Prieto (Eds.). Springer International Publishing, Cham, 173--180. DOI:https:\/\/doi.org\/10.1007\/978-3-319-62410-5_21"},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3293455"},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.5220\/0006792405290536"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/12.980005"},{"key":"e_1_2_2_10_1","unstructured":"Rob Ashmore Radu Calinescu and Colin Paterson. 2021. Assuring the machine learning lifecycle: Desiderata methods and challenges. ACM Computing Surveys. Retrieved from http:\/\/arxiv.org\/abs\/1905.04223.  Rob Ashmore Radu Calinescu and Colin Paterson. 2021. Assuring the machine learning lifecycle: Desiderata methods and challenges. ACM Computing Surveys. Retrieved from http:\/\/arxiv.org\/abs\/1905.04223."},{"key":"e_1_2_2_11_1","volume-title":"ACM Digital Library. Retrieved","author":"Association for Computing Machinery. 2020.","year":"2020","unstructured":"Association for Computing Machinery. 2020. ACM Digital Library. Retrieved July 1, 2020 from https:\/\/dl.acm.org\/. Association for Computing Machinery. 2020. ACM Digital Library. Retrieved July 1, 2020 from https:\/\/dl.acm.org\/."},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00083"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/TDSC.2004.2"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1109\/PRDC.2013.53"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/780731.780750"},{"key":"e_1_2_2_16_1","volume-title":"Programming Embedded Systems: With C and GNU Development Tools (2 ed.). O\u2019Reilly Media","author":"Barr Michael","unstructured":"Michael Barr and Anthony Massa . 2006. Programming Embedded Systems: With C and GNU Development Tools (2 ed.). O\u2019Reilly Media , Inc . Michael Barr and Anthony Massa. 2006. Programming Embedded Systems: With C and GNU Development Tools (2 ed.). O\u2019Reilly Media, Inc."},{"key":"e_1_2_2_17_1","volume-title":"Douglas K. Barry. 2013. Web Services, Service-Oriented Architectures, and Cloud Computing","unstructured":"Douglas K. Barry. 2013. Web Services, Service-Oriented Architectures, and Cloud Computing ( 2 nd ed.). Morgan Kaufmann , San Francisco, CA . Douglas K. Barry. 2013. Web Services, Service-Oriented Architectures, and Cloud Computing (2nd ed.). Morgan Kaufmann, San Francisco, CA.","edition":"2"},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.actaastro.2018.11.011"},{"key":"e_1_2_2_19_1","volume-title":"Vos","author":"Bauersfeld Sebastian","year":"2012","unstructured":"Sebastian Bauersfeld and Tanja E. J . Vos . 2012 . GUITest: A Java library for fully automated GUI robustness testing. In Proceedings of the 27th IEEE\/ACM International Conference on Automated Software Engineering. IEEE Press , 330--333. DOI:https:\/\/doi.org\/10.1145\/2351676.2351739 10.1145\/2351676.2351739 Sebastian Bauersfeld and Tanja E. J. Vos. 2012. GUITest: A Java library for fully automated GUI robustness testing. In Proceedings of the 27th IEEE\/ACM International Conference on Automated Software Engineering. IEEE Press, 330--333. DOI:https:\/\/doi.org\/10.1145\/2351676.2351739"},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/SSIRI.2010.14"},{"key":"e_1_2_2_21_1","volume-title":"Proceedings of the International Conference on Autonomic Computing.IEEE Press, 62--69","author":"Mohamed","year":"2004","unstructured":"Mohamed N. Bennani and Daniel A. Menasc\u00e9. 2004. Assessing the robustness of self-managing computer systems under highly variable workloads . In Proceedings of the International Conference on Autonomic Computing.IEEE Press, 62--69 . DOI:https:\/\/doi.org\/10.1109\/ICAC. 2004 .1301348 10.1109\/ICAC.2004.1301348 Mohamed N. Bennani and Daniel A. Menasc\u00e9. 2004. Assessing the robustness of self-managing computer systems under highly variable workloads. In Proceedings of the International Conference on Autonomic Computing.IEEE Press, 62--69. DOI:https:\/\/doi.org\/10.1109\/ICAC.2004.1301348"},{"key":"e_1_2_2_22_1","volume-title":"Micael Gallego, Boni Garc\u00eda","author":"Bertolino Antonia","year":"2019","unstructured":"Antonia Bertolino , Guglielmo De Angelis , Micael Gallego, Boni Garc\u00eda , Francisco Gort\u00e1zar , Francesca Lonetti, and Eda March etti. 2019 . A systematic review on cloud testing. ACM Comput. Surv. 52, 5, Article 93 (Sep. 2019), 42 pages. DOI:https:\/\/doi.org\/10.1145\/3331447 10.1145\/3331447 Antonia Bertolino, Guglielmo De Angelis, Micael Gallego, Boni Garc\u00eda, Francisco Gort\u00e1zar, Francesca Lonetti, and Eda Marchetti. 2019. A systematic review on cloud testing. ACM Comput. Surv. 52, 5, Article 93 (Sep. 2019), 42 pages. DOI:https:\/\/doi.org\/10.1145\/3331447"},{"key":"e_1_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.5220\/0005540501910198"},{"key":"e_1_2_2_24_1","volume-title":"Fuzzing: Challenges and reflections","author":"Boehme Marcel","year":"2020","unstructured":"Marcel Boehme , Cristian Cadar , and Abhik Roychoudhury . 2020 . Fuzzing: Challenges and reflections . IEEE Software ( 2020). DOI:https:\/\/doi.org\/10.1109\/MS.2020.3016773 10.1109\/MS.2020.3016773 Marcel Boehme, Cristian Cadar, and Abhik Roychoudhury. 2020. Fuzzing: Challenges and reflections. IEEE Software (2020). DOI:https:\/\/doi.org\/10.1109\/MS.2020.3016773"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2006.07.009"},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.future.2020.06.049"},{"key":"e_1_2_2_27_1","volume-title":"Proceedings of the Workshop on Self-Healing, Adaptive and Self-Managed Systems.","author":"Broadwell Pete","year":"2002","unstructured":"Pete Broadwell , Naveen Sastry , and Jonathan Traupman . 2002 . FIG: A prototype tool for online verification of recovery . In Proceedings of the Workshop on Self-Healing, Adaptive and Self-Managed Systems. Pete Broadwell, Naveen Sastry, and Jonathan Traupman. 2002. FIG: A prototype tool for online verification of recovery. In Proceedings of the Workshop on Self-Healing, Adaptive and Self-Managed Systems."},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/JPROC.2006.888386"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/2554850.2554935"},{"key":"e_1_2_2_30_1","volume-title":"Proceedings of the 3rd International Conference on Web Information Systems and Technologies (Webist\u201907)","author":"Calori Ilaria Canova","year":"2007","unstructured":"Ilaria Canova Calori , Tor St\u00e5lhane , and Sven Ziemer . 2007 . Robustness analysis using fmea and bbn: Case study for a web-based application . Proceedings of the 3rd International Conference on Web Information Systems and Technologies (Webist\u201907) . Ilaria Canova Calori, Tor St\u00e5lhane, and Sven Ziemer. 2007. Robustness analysis using fmea and bbn: Case study for a web-based application. Proceedings of the 3rd International Conference on Web Information Systems and Technologies (Webist\u201907)."},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/LADC48089.2019.8995720"},{"key":"e_1_2_2_32_1","volume-title":"Proceedings of the III Brazilian Symposium on Systematic and Automated Software Testing (SAST\u201918)","author":"Wallace F.","unstructured":"Wallace F. F. Cardoso and Eliane Martins. 2018. Using a search and model based framework to improve robustness tests in cloud platforms . In Proceedings of the III Brazilian Symposium on Systematic and Automated Software Testing (SAST\u201918) . Association for Computing Machinery, New York, NY, 67--76. DOI:https:\/\/doi.org\/10.1145\/3266003.3266011 10.1145\/3266003.3266011 Wallace F. F. Cardoso and Eliane Martins. 2018. Using a search and model based framework to improve robustness tests in cloud platforms. In Proceedings of the III Brazilian Symposium on Systematic and Automated Software Testing (SAST\u201918). Association for Computing Machinery, New York, NY, 67--76. DOI:https:\/\/doi.org\/10.1145\/3266003.3266011"},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.666826"},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/LADCW.2011.15"},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSTW.2008.51"},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1186\/s13677-015-0043-7"},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE5003.2020.00047"},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/4236.991449"},{"key":"e_1_2_2_39_1","volume-title":"Proceedings of the 12th European Workshop on Dependable Computing (EWDC\u201909)","author":"Chu Hoang-Nam","year":"2009","unstructured":"Hoang-Nam Chu , Jean Arlat , Marc-Olivier Killijian , Benjamin Lussier , and David Powell . 2009 . Robustness testing of robot controller software . In Proceedings of the 12th European Workshop on Dependable Computing (EWDC\u201909) , H\u00e9l\u00e8ne Waeselynck (Ed.). Hoang-Nam Chu, Jean Arlat, Marc-Olivier Killijian, Benjamin Lussier, and David Powell. 2009. Robustness testing of robot controller software. In Proceedings of the 12th European Workshop on Dependable Computing (EWDC\u201909), H\u00e9l\u00e8ne Waeselynck (Ed.)."},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1063\/1.5033796"},{"key":"e_1_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/2771783.2771811"},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/PRDC.1999.816230"},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDSN.2000.857547"},{"key":"e_1_2_2_44_1","volume-title":"Roberto Natella, and Roberto Pietrantuono.","author":"Cotroneo Domenico","year":"2011","unstructured":"Domenico Cotroneo , Domenico Di Leo , Roberto Natella, and Roberto Pietrantuono. 2011 . A case study on state-based robustness testing of an operating system for the avionic domain. In Computer Safety, Reliability, and Security, Francesco Flammini, Sandro Bologna, and Valeria Vittorini (Eds.). Springer , Berlin, 213--227. Domenico Cotroneo, Domenico Di Leo, Roberto Natella, and Roberto Pietrantuono. 2011. A case study on state-based robustness testing of an operating system for the avionic domain. In Computer Safety, Reliability, and Security, Francesco Flammini, Sandro Bologna, and Valeria Vittorini (Eds.). Springer, Berlin, 213--227."},{"key":"e_1_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/2744210.2744213"},{"key":"e_1_2_2_46_1","first-page":"6","article-title":"Evolutionary fuzzing of Android OS vendor system services","volume":"24","author":"Cotroneo Domenico","year":"2019","unstructured":"Domenico Cotroneo , Antonio Ken Iannillo , and Roberto Natella . 2019 . Evolutionary fuzzing of Android OS vendor system services . Emp. Softw. Eng. 24 , 6 (Dec. 2019), 3630--3658. DOI:https:\/\/doi.org\/10.1007\/s10664-019-09725-6 10.1007\/s10664-019-09725-6 Domenico Cotroneo, Antonio Ken Iannillo, and Roberto Natella. 2019. Evolutionary fuzzing of Android OS vendor system services. Emp. Softw. Eng. 24, 6 (Dec. 2019), 3630--3658. DOI:https:\/\/doi.org\/10.1007\/s10664-019-09725-6","journal-title":"Emp. Softw. Eng."},{"key":"e_1_2_2_47_1","volume-title":"Roberto Natella, and Stefano Rosiello.","author":"Cotroneo Domenico","year":"2019","unstructured":"Domenico Cotroneo , Antonio Ken Iannillo , Roberto Natella, and Stefano Rosiello. 2019 . Dependability assessment of the Android OS through fault injection. IEEE Trans. Reliabil . (2019), 1--16. DOI:https:\/\/doi.org\/10.1109\/TR.2019.2954384 10.1109\/TR.2019.2954384 Domenico Cotroneo, Antonio Ken Iannillo, Roberto Natella, and Stefano Rosiello. 2019. Dependability assessment of the Android OS through fault injection. IEEE Trans. Reliabil. (2019), 1--16. DOI:https:\/\/doi.org\/10.1109\/TR.2019.2954384"},{"key":"e_1_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2013.6693073"},{"key":"e_1_2_2_49_1","volume-title":"JCrasher: An automatic robustness tester for Java. Softw. Prac. Exp. 34 (09","author":"Csallner Christoph","year":"2004","unstructured":"Christoph Csallner and Yannis Smaragdakis . 2004. JCrasher: An automatic robustness tester for Java. Softw. Prac. Exp. 34 (09 2004 ), 1025--1050. DOI:https:\/\/doi.org\/10.1002\/spe.602 10.1002\/spe.602 Christoph Csallner and Yannis Smaragdakis. 2004. JCrasher: An automatic robustness tester for Java. Softw. Prac. Exp. 34 (09 2004), 1025--1050. DOI:https:\/\/doi.org\/10.1002\/spe.602"},{"key":"e_1_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1109\/LADC.2013.17"},{"key":"e_1_2_2_51_1","volume-title":"Testing the robustness of controllers for self-adaptive systems. J. Brazil. Comput. Soc. 20","author":"C\u00e1mara Javier","year":"2014","unstructured":"Javier C\u00e1mara , Rog\u00e9rio de Lemos , Nuno Laranjeiro , Rafael Ventura , and Marco Vieira . 2014. Testing the robustness of controllers for self-adaptive systems. J. Brazil. Comput. Soc. 20 ( 2014 ). DOI:https:\/\/doi.org\/10.1186\/1678-4804-20-1 10.1186\/1678-4804-20-1 Javier C\u00e1mara, Rog\u00e9rio de Lemos, Nuno Laranjeiro, Rafael Ventura, and Marco Vieira. 2014. Testing the robustness of controllers for self-adaptive systems. J. Brazil. Comput. Soc. 20 (2014). DOI:https:\/\/doi.org\/10.1186\/1678-4804-20-1"},{"key":"e_1_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1109\/TDSC.2015.2429128"},{"key":"e_1_2_2_53_1","unstructured":"DBLP Team. 2020. DBLP. Retrieved July 1 2020 from https:\/\/dblp.uni-trier.de\/.  DBLP Team. 2020. DBLP. Retrieved July 1 2020 from https:\/\/dblp.uni-trier.de\/."},{"key":"e_1_2_2_54_1","volume-title":"Choffnes","author":"Deitel Harvey M.","year":"2003","unstructured":"Harvey M. Deitel , Paul J. Deitel , and David R . Choffnes . 2003 . Operating Systems (3rd Edition). Prentice Hall , Upper Saddle River, NJ. Harvey M. Deitel, Paul J. Deitel, and David R. Choffnes. 2003. Operating Systems (3rd Edition). Prentice Hall, Upper Saddle River, NJ."},{"key":"e_1_2_2_55_1","volume-title":"Proceedings of the 25th International Symposium on Fault-Tolerant Computing.522--527","author":"Dingman Christopher P.","year":"1995","unstructured":"Christopher P. Dingman , Joe Marshall , and Daniel P. Siewiorek . 1995. Measuring robustness of a fault tolerant aerospace system . In Proceedings of the 25th International Symposium on Fault-Tolerant Computing.522--527 . DOI:https:\/\/doi.org\/10.1109\/FTCS. 1995 .466945 10.1109\/FTCS.1995.466945 Christopher P. Dingman, Joe Marshall, and Daniel P. Siewiorek. 1995. Measuring robustness of a fault tolerant aerospace system. In Proceedings of the 25th International Symposium on Fault-Tolerant Computing.522--527. DOI:https:\/\/doi.org\/10.1109\/FTCS.1995.466945"},{"key":"e_1_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.1109\/PRDC.2002.1185639"},{"key":"e_1_2_2_57_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.2002.1173283"},{"key":"e_1_2_2_58_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2006.113"},{"key":"e_1_2_2_59_1","unstructured":"Elsevier BV. 2020. Scopus. Retrieved July 1 2020 from https:\/\/www.scopus.com\/home.uri.  Elsevier BV. 2020. Scopus. Retrieved July 1 2020 from https:\/\/www.scopus.com\/home.uri."},{"key":"e_1_2_2_60_1","volume-title":"Shin","author":"Feng Huan","year":"2016","unstructured":"Huan Feng and Kang G . Shin . 2016 . BinderCracker: Assessing the robustness of Android system services. arxiv:1604.06964.Retrieved from http:\/\/arxiv.org\/abs\/1604.06964. Huan Feng and Kang G. Shin. 2016. BinderCracker: Assessing the robustness of Android system services. arxiv:1604.06964.Retrieved from http:\/\/arxiv.org\/abs\/1604.06964."},{"key":"e_1_2_2_61_1","volume-title":"Testing of Communicating Systems","author":"Fernandez Jean-Claude","unstructured":"Jean-Claude Fernandez , Laurent Mounier , and Cyril Pachon . 2005. A model-based approach for robustness testing . In Testing of Communicating Systems , Ferhat Khendek and Rachida Dssouli (Eds.). Springer , Berlin , 333--348. Jean-Claude Fernandez, Laurent Mounier, and Cyril Pachon. 2005. A model-based approach for robustness testing. In Testing of Communicating Systems, Ferhat Khendek and Rachida Dssouli (Eds.). Springer, Berlin, 333--348."},{"key":"e_1_2_2_62_1","doi-asserted-by":"publisher","DOI":"10.5555\/851020.856196"},{"key":"#cr-split#-e_1_2_2_63_1.1","doi-asserted-by":"crossref","unstructured":"Christof Fetzer and Zhen Xiao. 2005. Automatic testing for robustness violations. In Testing Commercial-off-the-Shelf Components and Systems. DOI:https:\/\/doi.org\/10.1007\/3-540-27071-X_16 10.1007\/3-540-27071-X_16","DOI":"10.1007\/3-540-27071-X_16"},{"key":"#cr-split#-e_1_2_2_63_1.2","doi-asserted-by":"crossref","unstructured":"Christof Fetzer and Zhen Xiao. 2005. Automatic testing for robustness violations. In Testing Commercial-off-the-Shelf Components and Systems. DOI:https:\/\/doi.org\/10.1007\/3-540-27071-X_16","DOI":"10.1007\/3-540-27071-X_16"},{"key":"e_1_2_2_64_1","volume-title":"Proceedings of the 4th Conference on USENIX Windows Systems Symposium","volume":"4","author":"Justin","unstructured":"Justin E. Forrester and Barton P. Miller. 2000. An empirical study of the robustness of Windows NT applications using random testing . In Proceedings of the 4th Conference on USENIX Windows Systems Symposium , Volume 4 (WSS\u201900). USENIX Association, Berkeley, CA, 6--6. http:\/\/dl.acm.org\/citation.cfm?id&equals;1267102.1267108 Justin E. Forrester and Barton P. Miller. 2000. An empirical study of the robustness of Windows NT applications using random testing. In Proceedings of the 4th Conference on USENIX Windows Systems Symposium, Volume 4 (WSS\u201900). USENIX Association, Berkeley, CA, 6--6. http:\/\/dl.acm.org\/citation.cfm?id&equals;1267102.1267108"},{"key":"e_1_2_2_65_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-30577-4_19"},{"key":"e_1_2_2_66_1","first-page":"1","article-title":"Robustness testing of composed real-time systems. J, Comput, Methods Sci","volume":"10","author":"Fouchal Hac\u00e8ne","year":"2010","unstructured":"Hac\u00e8ne Fouchal , Antoine Rollet , and Abbas Tarhini . 2010 . Robustness testing of composed real-time systems. J, Comput, Methods Sci , Eng , 10 , 1 - 2 S2 (Sep. 2010), 135--148. Hac\u00e8ne Fouchal, Antoine Rollet, and Abbas Tarhini. 2010. Robustness testing of composed real-time systems. J, Comput, Methods Sci, Eng, 10, 1-2S2 (Sep. 2010), 135--148.","journal-title":"Eng"},{"key":"e_1_2_2_67_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2005.51"},{"key":"e_1_2_2_68_1","doi-asserted-by":"publisher","DOI":"10.1109\/JSYST.2012.2223532"},{"key":"e_1_2_2_69_1","volume-title":"Proceedings of the 10th International Symposium on Software Reliability Engineering. 166--174","author":"Anup","year":"1999","unstructured":"Anup K. Ghosh and Matthew Schmid. 1999. An approach to testing COTS software for robustness to operating system exceptions and errors . In Proceedings of the 10th International Symposium on Software Reliability Engineering. 166--174 . DOI:https:\/\/doi.org\/10.1109\/ISSRE. 1999 .809321 10.1109\/ISSRE.1999.809321 Anup K. Ghosh and Matthew Schmid. 1999. An approach to testing COTS software for robustness to operating system exceptions and errors. In Proceedings of the 10th International Symposium on Software Reliability Engineering. 166--174. DOI:https:\/\/doi.org\/10.1109\/ISSRE.1999.809321"},{"key":"e_1_2_2_70_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.1998.730886"},{"key":"e_1_2_2_71_1","doi-asserted-by":"publisher","DOI":"10.1109\/PRDC.2013.12"},{"key":"e_1_2_2_72_1","unstructured":"Google. 2020. Google Scholar. Retrieved July 1 2020 from https:\/\/scholar.google.com\/.  Google. 2020. Google Scholar. Retrieved July 1 2020 from https:\/\/scholar.google.com\/."},{"key":"e_1_2_2_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/3395363.3404366"},{"key":"#cr-split#-e_1_2_2_74_1.1","doi-asserted-by":"crossref","unstructured":"Samer Hanna and Malcolm Munro. 2008. An Approach for WSDL-Based Automated Robustness Testing of Web Services. 1093--1104. DOI:https:\/\/doi.org\/10.1007\/978-0-387-78578-3_40 10.1007\/978-0-387-78578-3_40","DOI":"10.1007\/978-0-387-78578-3_40"},{"key":"#cr-split#-e_1_2_2_74_1.2","doi-asserted-by":"crossref","unstructured":"Samer Hanna and Malcolm Munro. 2008. An Approach for WSDL-Based Automated Robustness Testing of Web Services. 1093--1104. DOI:https:\/\/doi.org\/10.1007\/978-0-387-78578-3_40","DOI":"10.1007\/978-0-387-78578-3_40"},{"key":"e_1_2_2_75_1","doi-asserted-by":"publisher","DOI":"10.18100\/ijamec.270378"},{"key":"e_1_2_2_76_1","doi-asserted-by":"publisher","DOI":"10.1145\/2480362.2480587"},{"key":"e_1_2_2_77_1","doi-asserted-by":"publisher","DOI":"10.1145\/3183519.3183534"},{"key":"e_1_2_2_78_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.2017.16"},{"key":"e_1_2_2_79_1","first-page":"12","article-title":"IEEE standard glossary of software engineering terminology","volume":"610","author":"IEEE.","year":"1990","unstructured":"IEEE. 1990 . IEEE standard glossary of software engineering terminology . IEEE Std 610 . 12 - 1990 (Dec 1990), 1--84. DOI:https:\/\/doi.org\/10.1109\/IEEESTD.1990.101064 10.1109\/IEEESTD.1990.101064 IEEE. 1990. IEEE standard glossary of software engineering terminology. IEEE Std 610.12-1990 (Dec 1990), 1--84. DOI:https:\/\/doi.org\/10.1109\/IEEESTD.1990.101064","journal-title":"IEEE Std"},{"key":"e_1_2_2_80_1","unstructured":"IEEE. 2020. IEEE Xplore. Retrieved July 1 2020 from https:\/\/ieeexplore.ieee.org\/Xplore\/home.jsp.  IEEE. 2020. IEEE Xplore. Retrieved July 1 2020 from https:\/\/ieeexplore.ieee.org\/Xplore\/home.jsp."},{"key":"e_1_2_2_81_1","doi-asserted-by":"publisher","DOI":"10.1109\/SOSE.2011.6139108"},{"key":"e_1_2_2_82_1","doi-asserted-by":"publisher","DOI":"10.1109\/PRDC.2002.1185618"},{"key":"e_1_2_2_83_1","doi-asserted-by":"publisher","DOI":"10.1109\/DSN-W.2018.00027"},{"key":"e_1_2_2_84_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2010.62"},{"key":"e_1_2_2_85_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2015.2445340"},{"key":"e_1_2_2_86_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICYCS.2008.329"},{"key":"e_1_2_2_87_1","volume-title":"Network and Parallel Computing, Jian Cao, Minglu Li, Min-You Wu","author":"Jing Chuanming","unstructured":"Chuanming Jing , Zhiliang Wang , Xia Yin , and Jianping Wu. 2008. A formal approach to robustness testing of network protocol . In Network and Parallel Computing, Jian Cao, Minglu Li, Min-You Wu , and Jinjun Chen (Eds.). Springer , Berlin , 24--37. Chuanming Jing, Zhiliang Wang, Xia Yin, and Jianping Wu. 2008. A formal approach to robustness testing of network protocol. In Network and Parallel Computing, Jian Cao, Minglu Li, Min-You Wu, and Jinjun Chen (Eds.). Springer, Berlin, 24--37."},{"key":"e_1_2_2_88_1","doi-asserted-by":"publisher","DOI":"10.1109\/DSN.2005.45"},{"key":"e_1_2_2_89_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.2007.23"},{"key":"e_1_2_2_90_1","doi-asserted-by":"publisher","DOI":"10.1109\/DSN.2007.71"},{"key":"e_1_2_2_91_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2014.45"},{"key":"e_1_2_2_92_1","doi-asserted-by":"publisher","DOI":"10.5555\/2560102"},{"key":"e_1_2_2_93_1","first-page":"5","article-title":"Web services business process execution language version 2.0","volume":"11","author":"Jordan Diane","year":"2007","unstructured":"Diane Jordan , John Evdemon , Alexandre Alves , Assaf Arkin , Sid Askary , Charlton Barreto , Ben Bloch , Francisco Curbera , Mark Ford , Yaron Goland , et\u00a0al. 2007 . Web services business process execution language version 2.0 . OASIS Stand. 11 , 120 (2007), 5 . Diane Jordan, John Evdemon, Alexandre Alves, Assaf Arkin, Sid Askary, Charlton Barreto, Ben Bloch, Francisco Curbera, Mark Ford, Yaron Goland, et\u00a0al. 2007. Web services business process execution language version 2.0. OASIS Stand. 11, 120 (2007), 5.","journal-title":"OASIS Stand."},{"key":"e_1_2_2_94_1","volume-title":"Software Security Assessment through Specification Mutations and Fault Injection","author":"Kaksonen Rauli","unstructured":"Rauli Kaksonen , Marko Laakso , and Ari Takanen . 2001. Software Security Assessment through Specification Mutations and Fault Injection . Springer US , Boston, MA , 173--183. DOI:https:\/\/doi.org\/10.1007\/978-0-387-35413-2_16 10.1007\/978-0-387-35413-2_16 Rauli Kaksonen, Marko Laakso, and Ari Takanen. 2001. Software Security Assessment through Specification Mutations and Fault Injection. Springer US, Boston, MA, 173--183. DOI:https:\/\/doi.org\/10.1007\/978-0-387-35413-2_16"},{"key":"e_1_2_2_95_1","doi-asserted-by":"publisher","DOI":"10.1145\/3407023.3407052"},{"key":"e_1_2_2_96_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.2005.13"},{"key":"e_1_2_2_97_1","doi-asserted-by":"publisher","DOI":"10.1109\/DSN-S50200.2020.00013"},{"key":"e_1_2_2_98_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA40945.2020.9197060"},{"key":"e_1_2_2_99_1","doi-asserted-by":"publisher","DOI":"10.1080\/10429247.2003.11415214"},{"key":"e_1_2_2_100_1","volume-title":"Procedures for performing systematic reviews. Keele Univ. 33 (08","author":"Kitchenham Barbara","year":"2004","unstructured":"Barbara Kitchenham . 2004. Procedures for performing systematic reviews. Keele Univ. 33 (08 2004 ). Barbara Kitchenham. 2004. Procedures for performing systematic reviews. Keele Univ. 33 (08 2004)."},{"key":"e_1_2_2_101_1","doi-asserted-by":"publisher","DOI":"10.1109\/FTCS.1999.781031"},{"key":"e_1_2_2_102_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.877845"},{"key":"e_1_2_2_103_1","doi-asserted-by":"publisher","DOI":"10.1109\/RELDIS.1997.632800"},{"key":"e_1_2_2_104_1","volume-title":"Proceedings of the 28th Annual International Symposium on Fault-Tolerant Computing (FTCS\u201998)","author":"Kropp Nathan P.","unstructured":"Nathan P. Kropp , Philip J. Koopman , and Daniel P. Siewiorek . 1998. Automated robustness testing of off-the-shelf software components . In Proceedings of the 28th Annual International Symposium on Fault-Tolerant Computing (FTCS\u201998) . IEEE Computer Society. Nathan P. Kropp, Philip J. Koopman, and Daniel P. Siewiorek. 1998. Automated robustness testing of off-the-shelf software components. In Proceedings of the 28th Annual International Symposium on Fault-Tolerant Computing (FTCS\u201998). IEEE Computer Society."},{"key":"e_1_2_2_105_1","doi-asserted-by":"publisher","DOI":"10.1109\/APSCC.2009.5394106"},{"key":"e_1_2_2_106_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDCSW.2010.73"},{"key":"#cr-split#-e_1_2_2_107_1.1","unstructured":"Nuno Laranjeiro Jo\u00e3o Agnelo and Jorge Bernardino. 2021. A Systematic Review on Software Robustness Assessment-Dataset. DOI:https:\/\/doi.org\/10.5281\/zenodo.4436319 10.5281\/zenodo.4436319"},{"key":"#cr-split#-e_1_2_2_107_1.2","unstructured":"Nuno Laranjeiro Jo\u00e3o Agnelo and Jorge Bernardino. 2021. A Systematic Review on Software Robustness Assessment-Dataset. DOI:https:\/\/doi.org\/10.5281\/zenodo.4436319"},{"key":"e_1_2_2_108_1","doi-asserted-by":"publisher","DOI":"10.1109\/SCC.2008.123"},{"key":"e_1_2_2_109_1","doi-asserted-by":"publisher","DOI":"10.1109\/LADC48089.2019.8995714"},{"key":"e_1_2_2_110_1","doi-asserted-by":"publisher","DOI":"10.1109\/SCC.2008.129"},{"key":"e_1_2_2_111_1","doi-asserted-by":"publisher","DOI":"10.1007\/s13174-012-0062-2"},{"key":"e_1_2_2_112_1","doi-asserted-by":"publisher","DOI":"10.1109\/TE.2013.2292570"},{"key":"e_1_2_2_113_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISORC.2008.25"},{"key":"e_1_2_2_114_1","volume-title":"Introduction to Embedded Systems: A Cyber-Physical Systems Approach","author":"Lee Edward Ashford","unstructured":"Edward Ashford Lee and Sanjit Arunkumar Seshia . 2016. Introduction to Embedded Systems: A Cyber-Physical Systems Approach ( 2 nd ed.). The MIT Press . Edward Ashford Lee and Sanjit Arunkumar Seshia. 2016. Introduction to Embedded Systems: A Cyber-Physical Systems Approach (2nd ed.). The MIT Press.","edition":"2"},{"key":"e_1_2_2_115_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.scico.2010.02.005"},{"key":"e_1_2_2_116_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.entcs.2009.12.037"},{"key":"e_1_2_2_117_1","volume-title":"Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC\u201917)","author":"Li Guanpeng","unstructured":"Guanpeng Li , Siva Kumar Sastry Hari , Michael Sullivan , Timothy Tsai , Karthik Pattabiraman , Joel Emer , and Stephen W. Keckler . 2017. Understanding error propagation in deep learning neural network (DNN) accelerators and applications . In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC\u201917) . ACM, New York, NY, 1--12. DOI:https:\/\/doi.org\/10.1145\/3126908.3126964 10.1145\/3126908.3126964 Guanpeng Li, Siva Kumar Sastry Hari, Michael Sullivan, Timothy Tsai, Karthik Pattabiraman, Joel Emer, and Stephen W. Keckler. 2017. Understanding error propagation in deep learning neural network (DNN) accelerators and applications. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC\u201917). ACM, New York, NY, 1--12. DOI:https:\/\/doi.org\/10.1145\/3126908.3126964"},{"key":"e_1_2_2_118_1","volume-title":"Proceedings of the 29th USENIX Security Symposium (USENIX Security 20)","author":"Liu Baozheng","year":"2020","unstructured":"Baozheng Liu , Chao Zhang , Guang Gong , Yishun Zeng , Haifeng Ruan , and Jianwei Zhuge . 2020 . FANS: Fuzzing Android native system services via automated interface analysis . In Proceedings of the 29th USENIX Security Symposium (USENIX Security 20) . USENIX Association, 307--323. Baozheng Liu, Chao Zhang, Guang Gong, Yishun Zeng, Haifeng Ruan, and Jianwei Zhuge. 2020. FANS: Fuzzing Android native system services via automated interface analysis. In Proceedings of the 29th USENIX Security Symposium (USENIX Security 20). USENIX Association, 307--323."},{"key":"e_1_2_2_119_1","volume-title":"Simulating errors in web services. J. Simul. 5 (12","author":"Looker Nicholas","year":"2004","unstructured":"Nicholas Looker , Malcolm Munro , and Jiudong xu. 2004. Simulating errors in web services. J. Simul. 5 (12 2004 ), 1473--8031. Nicholas Looker, Malcolm Munro, and Jiudong xu. 2004. Simulating errors in web services. J. Simul. 5 (12 2004), 1473--8031."},{"key":"e_1_2_2_120_1","doi-asserted-by":"publisher","DOI":"10.1109\/CMPSAC.2004.1342690"},{"key":"e_1_2_2_121_1","doi-asserted-by":"publisher","DOI":"10.1109\/COMPSAC.2005.16"},{"key":"e_1_2_2_122_1","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2006.114"},{"key":"e_1_2_2_123_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jii.2019.04.002"},{"key":"e_1_2_2_124_1","volume-title":"Proceedings of the International Conference on Dependable Systems and Networks. 325--330","author":"Madeira Henrique","unstructured":"Henrique Madeira , Raphael R. Some , Francisco Moreira , Diamantino Costa , and David A. Rennels . 2002. Experimental evaluation of a COTS system for space applications . In Proceedings of the International Conference on Dependable Systems and Networks. 325--330 . Henrique Madeira, Raphael R. Some, Francisco Moreira, Diamantino Costa, and David A. Rennels. 2002. Experimental evaluation of a COTS system for space applications. In Proceedings of the International Conference on Dependable Systems and Networks. 325--330."},{"key":"e_1_2_2_125_1","volume-title":"VI Test and Fault Tolerance Workshop.","author":"Maia Ricardo","year":"2005","unstructured":"Ricardo Maia , Luis Henriques , Ricardo Barbosa , Diamantino Costa , and Henrique Madeira . 2005 . Xception fault injection and robustness testing framework: A case-study of testing RTEMS . In VI Test and Fault Tolerance Workshop. Ricardo Maia, Luis Henriques, Ricardo Barbosa, Diamantino Costa, and Henrique Madeira. 2005. Xception fault injection and robustness testing framework: A case-study of testing RTEMS. In VI Test and Fault Tolerance Workshop."},{"key":"e_1_2_2_126_1","doi-asserted-by":"publisher","DOI":"10.1109\/DSN.2012.6263963"},{"key":"e_1_2_2_127_1","volume-title":"Manuel Egele, Edward J. Schwartz, and Maverick Woo.","author":"Marie Man\u00e8s Valentin Jean","year":"2019","unstructured":"Valentin Jean Marie Man\u00e8s , HyungSeok Han , Choongwoo Han , Sang Kil Cha , Manuel Egele, Edward J. Schwartz, and Maverick Woo. 2019 . The art, science, and engineering of fuzzing: A survey. IEEE Trans. Softw. Eng . (2019). DOI:https:\/\/doi.org\/10.1109\/TSE.2019.2946563 10.1109\/TSE.2019.2946563 Valentin Jean Marie Man\u00e8s, HyungSeok Han, Choongwoo Han, Sang Kil Cha, Manuel Egele, Edward J. Schwartz, and Maverick Woo. 2019. The art, science, and engineering of fuzzing: A survey. IEEE Trans. Softw. Eng. (2019). DOI:https:\/\/doi.org\/10.1109\/TSE.2019.2946563"},{"key":"e_1_2_2_128_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICWS.2007.49"},{"key":"e_1_2_2_129_1","volume-title":"Companion to the Proceedings of the 29th International Conference on Software Engineering (ICSE COMPANION\u201907)","author":"Martin Evan","unstructured":"Evan Martin , Suranjana Basu , and Tao Xie . 2007. WebSob: A tool for robustness testing of web services . In Companion to the Proceedings of the 29th International Conference on Software Engineering (ICSE COMPANION\u201907) . IEEE Computer Society , 65--66. DOI:https:\/\/doi.org\/10.1109\/ICSECOMPANION.2007.84 10.1109\/ICSECOMPANION.2007.84 Evan Martin, Suranjana Basu, and Tao Xie. 2007. WebSob: A tool for robustness testing of web services. In Companion to the Proceedings of the 29th International Conference on Software Engineering (ICSE COMPANION\u201907). IEEE Computer Society, 65--66. DOI:https:\/\/doi.org\/10.1109\/ICSECOMPANION.2007.84"},{"key":"e_1_2_2_130_1","volume-title":"Proceedings of the International Conference on Dependable Systems and Networks. 483--487","author":"Martins Eliane","unstructured":"Eliane Martins , Cec\u00edlia M. F. Rubira , and Nelson G. M. Leme . 2002. Jaca: A reflective fault injection tool based on patterns . In Proceedings of the International Conference on Dependable Systems and Networks. 483--487 . Eliane Martins, Cec\u00edlia M. F. Rubira, and Nelson G. M. Leme. 2002. Jaca: A reflective fault injection tool based on patterns. In Proceedings of the International Conference on Dependable Systems and Networks. 483--487."},{"key":"e_1_2_2_131_1","volume-title":"Ana Rosa Cavalli, and Edgar Toshiro Yano","author":"Mattiello-Francisco F\u00e1tima","year":"2012","unstructured":"F\u00e1tima Mattiello-Francisco , Eliane Martins , Ana Rosa Cavalli, and Edgar Toshiro Yano . 2012 . InRob: An approach for testing interoperability and robustness of real-time embedded software. J. Syst. Softw . 85 (01 2012), 3--15. DOI:https:\/\/doi.org\/10.1016\/j.jss.2011.02.034 10.1016\/j.jss.2011.02.034 F\u00e1tima Mattiello-Francisco, Eliane Martins, Ana Rosa Cavalli, and Edgar Toshiro Yano. 2012. InRob: An approach for testing interoperability and robustness of real-time embedded software. J. Syst. Softw. 85 (01 2012), 3--15. DOI:https:\/\/doi.org\/10.1016\/j.jss.2011.02.034"},{"key":"e_1_2_2_132_1","doi-asserted-by":"publisher","DOI":"10.1109\/LATINCOM.2009.5304903"},{"key":"e_1_2_2_133_1","doi-asserted-by":"publisher","DOI":"10.1109\/EDCC.2010.14"},{"key":"e_1_2_2_134_1","doi-asserted-by":"publisher","DOI":"10.1109\/DSN.2007.85"},{"key":"e_1_2_2_135_1","volume-title":"Advances in Computers","author":"Menzies Tim","unstructured":"Tim Menzies and Charles Pecheur . 2005. Verification and validation and artificial intelligence . In Advances in Computers , Vol. 65 . Elsevier , 153--201. DOI:https:\/\/doi.org\/10.1016\/S0065-2458(05)65004-8 10.1016\/S0065-2458(05)65004-8 Tim Menzies and Charles Pecheur. 2005. Verification and validation and artificial intelligence. In Advances in Computers, Vol. 65. Elsevier, 153--201. DOI:https:\/\/doi.org\/10.1016\/S0065-2458(05)65004-8"},{"key":"e_1_2_2_136_1","volume-title":"Resilience Assessment and Evaluation of Computing Systems","author":"Micskei Zolt\u00e1n","unstructured":"Zolt\u00e1n Micskei , Henrique Madeira , Alberto Avritzer , Istv\u00e1n Majzik , Marco Vieira , and Nuno Antunes . 2012. Robustness testing techniques and tools . In Resilience Assessment and Evaluation of Computing Systems . Springer , 323--339. Zolt\u00e1n Micskei, Henrique Madeira, Alberto Avritzer, Istv\u00e1n Majzik, Marco Vieira, and Nuno Antunes. 2012. Robustness testing techniques and tools. In Resilience Assessment and Evaluation of Computing Systems. Springer, 323--339."},{"key":"e_1_2_2_137_1","volume-title":"Proceedings of the International Workshop on Engineering of Fault Tolerant Systems (EFTS\u201906)","author":"Micskei Zolt\u00e1n","year":"2006","unstructured":"Zolt\u00e1n Micskei , Istv\u00e1n Majzik , and Francis Tam . 2006 . Robustness testing techniques for high availability middleware solutions . In Proceedings of the International Workshop on Engineering of Fault Tolerant Systems (EFTS\u201906) . Zolt\u00e1n Micskei, Istv\u00e1n Majzik, and Francis Tam. 2006. Robustness testing techniques for high availability middleware solutions. In Proceedings of the International Workshop on Engineering of Fault Tolerant Systems (EFTS\u201906)."},{"key":"e_1_2_2_138_1","volume-title":"Comparing robustness of AIS-based middleware implementations","author":"Micskei Zolt\u00e1n","unstructured":"Zolt\u00e1n Micskei , Istv\u00e1n Majzik , and Francis Tam . 2007. Comparing robustness of AIS-based middleware implementations . In Service Availability, Miroslaw Malek, Manfred Reitenspie\u00df, and Aad van Moorsel (Eds.). Springer , Berlin , 20--30. Zolt\u00e1n Micskei, Istv\u00e1n Majzik, and Francis Tam. 2007. Comparing robustness of AIS-based middleware implementations. In Service Availability, Miroslaw Malek, Manfred Reitenspie\u00df, and Aad van Moorsel (Eds.). Springer, Berlin, 20--30."},{"key":"e_1_2_2_139_1","doi-asserted-by":"publisher","DOI":"10.1145\/1145735.1145743"},{"key":"e_1_2_2_140_1","doi-asserted-by":"publisher","DOI":"10.1145\/96267.96279"},{"key":"e_1_2_2_142_1","doi-asserted-by":"publisher","DOI":"10.1109\/LASCAS.2014.6820257"},{"key":"e_1_2_2_143_1","doi-asserted-by":"publisher","DOI":"10.1109\/DFT.2014.6962080"},{"key":"e_1_2_2_144_1","doi-asserted-by":"publisher","DOI":"10.1109\/EDCC.2006.16"},{"key":"e_1_2_2_145_1","doi-asserted-by":"publisher","DOI":"10.1109\/32.601075"},{"key":"e_1_2_2_146_1","volume-title":"Proceedings of the International Conference on Automation, Control, Engineering and Computer Science (ACECS\u201914)","author":"Naceur Maha","year":"2014","unstructured":"Maha Naceur , Lilia Sfaxi , and Riadh Robbana . 2014 . Robustness testing of secure wireless sensor networks . In Proceedings of the International Conference on Automation, Control, Engineering and Computer Science (ACECS\u201914) . Maha Naceur, Lilia Sfaxi, and Riadh Robbana. 2014. Robustness testing of secure wireless sensor networks. In Proceedings of the International Conference on Automation, Control, Engineering and Computer Science (ACECS\u201914)."},{"key":"e_1_2_2_147_1","doi-asserted-by":"publisher","DOI":"10.1109\/PRDC.2011.51"},{"key":"e_1_2_2_148_1","doi-asserted-by":"publisher","DOI":"10.1145\/2841425"},{"key":"e_1_2_2_149_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSC.2018.2847344"},{"key":"e_1_2_2_150_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380334"},{"key":"e_1_2_2_151_1","doi-asserted-by":"publisher","DOI":"10.1109\/RADECS.2005.4365596"},{"key":"e_1_2_2_152_1","doi-asserted-by":"publisher","DOI":"10.1109\/WORDS.2003.1267504"},{"key":"e_1_2_2_153_1","first-page":"25","article-title":"Software dysfunction: Why do software fail?J","volume":"2014","author":"Ogheneovo Edward E.","year":"2014","unstructured":"Edward E. Ogheneovo . 2014 . Software dysfunction: Why do software fail?J . Comput. Chem. 2014 (2014), 25 -- 35 . Edward E. Ogheneovo. 2014. Software dysfunction: Why do software fail?J. Comput. Chem. 2014 (2014), 25--35.","journal-title":"Comput. Chem."},{"key":"e_1_2_2_154_1","doi-asserted-by":"publisher","DOI":"10.1109\/DepCoS-RELCOMEX.2009.41"},{"key":"e_1_2_2_155_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDSN.2000.857548"},{"key":"e_1_2_2_156_1","volume-title":"Proceedings of the International Conference on Dependable Systems and Networks. 141--150","author":"Pan Jiantao","year":"2001","unstructured":"Jiantao Pan , Philip Koopman , Yennun Huang , Robert Gruber , and Mimi Ling Jiang . 2001 . Robustness testing and hardening of CORBA ORB implementations . In Proceedings of the International Conference on Dependable Systems and Networks. 141--150 . DOI:https:\/\/doi.org\/10.1109\/DSN.2001.941400 10.1109\/DSN.2001.941400 Jiantao Pan, Philip Koopman, Yennun Huang, Robert Gruber, and Mimi Ling Jiang. 2001. Robustness testing and hardening of CORBA ORB implementations. In Proceedings of the International Conference on Dependable Systems and Networks. 141--150. DOI:https:\/\/doi.org\/10.1109\/DSN.2001.941400"},{"key":"e_1_2_2_157_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.2010.17"},{"key":"e_1_2_2_158_1","volume-title":"Communication Protocol Engineering","author":"Popovic Miroslav","unstructured":"Miroslav Popovic . 2006. Communication Protocol Engineering . CRC Press, Boca Raton , FL. Miroslav Popovic. 2006. Communication Protocol Engineering. CRC Press, Boca Raton, FL."},{"key":"e_1_2_2_159_1","doi-asserted-by":"publisher","DOI":"10.1109\/ECBS.2007.13"},{"key":"e_1_2_2_160_1","doi-asserted-by":"publisher","DOI":"10.1109\/EDCC.2012.16"},{"key":"e_1_2_2_161_1","volume-title":"A software-implemented fault injector on windows NT platform. Inf. Technol. J. 9 (03","author":"Qing-He Pan","year":"2010","unstructured":"Pan Qing-He , Hong Rong , and Pan Shu . 2010. A software-implemented fault injector on windows NT platform. Inf. Technol. J. 9 (03 2010 ). DOI:https:\/\/doi.org\/10.3923\/itj.2010.576.582 10.3923\/itj.2010.576.582 Pan Qing-He, Hong Rong, and Pan Shu. 2010. A software-implemented fault injector on windows NT platform. Inf. Technol. J. 9 (03 2010). DOI:https:\/\/doi.org\/10.3923\/itj.2010.576.582"},{"key":"e_1_2_2_162_1","doi-asserted-by":"publisher","DOI":"10.1109\/HPCC.2012.90"},{"key":"e_1_2_2_163_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2018.2803055"},{"key":"e_1_2_2_164_1","doi-asserted-by":"publisher","DOI":"10.1109\/DAC.2018.8465834"},{"key":"e_1_2_2_165_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-020-09881-0"},{"key":"e_1_2_2_166_1","doi-asserted-by":"publisher","DOI":"10.1109\/DSN.2002.1028909"},{"key":"e_1_2_2_167_1","doi-asserted-by":"publisher","DOI":"10.1109\/SERA.2007.14"},{"key":"e_1_2_2_168_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISCC.2009.5202405"},{"key":"e_1_2_2_169_1","doi-asserted-by":"publisher","DOI":"10.1109\/AQTR.2008.4588704"},{"key":"e_1_2_2_170_1","doi-asserted-by":"publisher","DOI":"10.1109\/PRDC.2018.00016"},{"key":"e_1_2_2_171_1","doi-asserted-by":"publisher","DOI":"10.1109\/PRDC.2005.44"},{"key":"e_1_2_2_172_1","volume-title":"Proceedings of the 14th International Conference on Enterprise Information Systems (ICEIS\u201912)","author":"Rychl\u00fd Marek","year":"2012","unstructured":"Marek Rychl\u00fd and Martin \u017dou\u017eelka . 2012 . Fault injection for web-services . In Proceedings of the 14th International Conference on Enterprise Information Systems (ICEIS\u201912) . INSTICC, SciTePress, 377--383. DOI:https:\/\/doi.org\/10.5220\/0004153003770383 10.5220\/0004153003770383 Marek Rychl\u00fd and Martin \u017dou\u017eelka. 2012. Fault injection for web-services. In Proceedings of the 14th International Conference on Enterprise Information Systems (ICEIS\u201912). INSTICC, SciTePress, 377--383. DOI:https:\/\/doi.org\/10.5220\/0004153003770383"},{"key":"e_1_2_2_173_1","doi-asserted-by":"publisher","DOI":"10.1145\/1244002.1244315"},{"key":"e_1_2_2_174_1","doi-asserted-by":"publisher","DOI":"10.1109\/SERVICES.2015.53"},{"key":"e_1_2_2_175_1","doi-asserted-by":"publisher","DOI":"10.1145\/1516533.1516538"},{"key":"e_1_2_2_176_1","volume-title":"Proceedings of the 12th European Workshop on Dependable Computing.","author":"Salva S\u00e9bastien","year":"2009","unstructured":"S\u00e9bastien Salva and Issam Rabhi . 2009 . Automatic web service robustness testing from WSDL descriptions . In Proceedings of the 12th European Workshop on Dependable Computing. S\u00e9bastien Salva and Issam Rabhi. 2009. Automatic web service robustness testing from WSDL descriptions. In Proceedings of the 12th European Workshop on Dependable Computing."},{"key":"e_1_2_2_177_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICIW.2010.32"},{"key":"e_1_2_2_178_1","doi-asserted-by":"publisher","DOI":"10.1145\/2632168.2632169"},{"key":"e_1_2_2_179_1","volume-title":"Proceedings of the DARPA Information Survivability Conference and Exposition. DISCEX","volume":"2","author":"Schmid Matthew","year":"2000","unstructured":"Matthew Schmid , Anup K. Ghosh , and Frank Hill . 2000 . Techniques for evaluating the robustness of Windows NT software . In Proceedings of the DARPA Information Survivability Conference and Exposition. DISCEX , Vol. 2 . 347--360. DOI:https:\/\/doi.org\/10.1109\/DISCEX.2000.821532 10.1109\/DISCEX.2000.821532 Matthew Schmid, Anup K. Ghosh, and Frank Hill. 2000. Techniques for evaluating the robustness of Windows NT software. In Proceedings of the DARPA Information Survivability Conference and Exposition. DISCEX, Vol. 2. 347--360. DOI:https:\/\/doi.org\/10.1109\/DISCEX.2000.821532"},{"key":"e_1_2_2_180_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2016.2544951"},{"key":"e_1_2_2_181_1","article-title":"A state-aware approach for robustness testing of embedded real-time operating systems","volume":"2","author":"Shahpasand Raheleh","year":"2019","unstructured":"Raheleh Shahpasand , Samad Paydar , Yasser Sedaghat , and Reza Ramezani . 2019 . A state-aware approach for robustness testing of embedded real-time operating systems . J. Comput. Knowl. Eng. 2 , 1 (2019). Raheleh Shahpasand, Samad Paydar, Yasser Sedaghat, and Reza Ramezani. 2019. A state-aware approach for robustness testing of embedded real-time operating systems. J. Comput. Knowl. Eng. 2, 1 (2019).","journal-title":"J. Comput. Knowl. Eng."},{"key":"e_1_2_2_182_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCKE.2016.7802133"},{"key":"e_1_2_2_183_1","doi-asserted-by":"publisher","DOI":"10.1109\/APSEC.2011.58"},{"key":"e_1_2_2_184_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2012.06.002"},{"key":"e_1_2_2_185_1","volume-title":"Web Application Architecture: Principles, Protocols and Practices","author":"Shklar Leon","unstructured":"Leon Shklar and Rich Rosen . 2009. Web Application Architecture: Principles, Protocols and Practices ( 2 nd ed.). Wiley , Chichester, UK . Leon Shklar and Rich Rosen. 2009. Web Application Architecture: Principles, Protocols and Practices (2nd ed.). Wiley, Chichester, UK.","edition":"2"},{"key":"e_1_2_2_186_1","doi-asserted-by":"publisher","DOI":"10.5555\/1249246.1249622"},{"key":"e_1_2_2_187_1","doi-asserted-by":"publisher","DOI":"10.1109\/FTCS.1993.627311"},{"key":"e_1_2_2_188_1","doi-asserted-by":"crossref","unstructured":"Theodore John Socolofsky and Claudia Jeanne Kale. 1991. TCP\/IP Tutorial. RFC 1180. RFC Editor. Retrieved from http:\/\/www.rfc-editor.org\/rfc\/rfc1180.txt.  Theodore John Socolofsky and Claudia Jeanne Kale. 1991. TCP\/IP Tutorial. RFC 1180. RFC Editor. Retrieved from http:\/\/www.rfc-editor.org\/rfc\/rfc1180.txt.","DOI":"10.17487\/rfc1180"},{"key":"e_1_2_2_189_1","unstructured":"Springer Nature Switzerland AG. 2020. Springer Link. Retrieved July 1 2020 from https:\/\/link.springer.com\/.  Springer Nature Switzerland AG. 2020. Springer Link. Retrieved July 1 2020 from https:\/\/link.springer.com\/."},{"key":"e_1_2_2_190_1","volume-title":"Proceedings of the 3rd International Symposium on Software Reliability Engineering. 237--245","author":"Suh Byung-Hoon","year":"1992","unstructured":"Byung-Hoon Suh , John Hudak , Daniel Siewiorek , and Zary Segall . 1992 . Development of a benchmark to measure system robustness: Experiences and lessons learned . In Proceedings of the 3rd International Symposium on Software Reliability Engineering. 237--245 . Byung-Hoon Suh, John Hudak, Daniel Siewiorek, and Zary Segall. 1992. Development of a benchmark to measure system robustness: Experiences and lessons learned. In Proceedings of the 3rd International Symposium on Software Reliability Engineering. 237--245."},{"key":"e_1_2_2_191_1","doi-asserted-by":"publisher","DOI":"10.1109\/AICCSA.2005.1387132"},{"key":"e_1_2_2_192_1","doi-asserted-by":"publisher","DOI":"10.1109\/IVS.2018.8500421"},{"key":"e_1_2_2_193_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11219-017-9359-5"},{"key":"e_1_2_2_194_1","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.456"},{"key":"e_1_2_2_195_1","volume-title":"Proceedings of the Australian Software Engineering Conference. 241--250","author":"Vasan Arunchandar","year":"2004","unstructured":"Arunchandar Vasan and A. M. Memon . 2004. ASPIRE: Automated systematic protocol implementation robustness evaluation . In Proceedings of the Australian Software Engineering Conference. 241--250 . DOI:https:\/\/doi.org\/10.1109\/ASWEC. 2004 .1290477 10.1109\/ASWEC.2004.1290477 Arunchandar Vasan and A. M. Memon. 2004. ASPIRE: Automated systematic protocol implementation robustness evaluation. In Proceedings of the Australian Software Engineering Conference. 241--250. DOI:https:\/\/doi.org\/10.1109\/ASWEC.2004.1290477"},{"key":"e_1_2_2_196_1","volume-title":"KITO tool: A fault injection environment in Linux kernel data structures. Microelectr. Reliabil. 60","author":"Velasco Alejandro","year":"2016","unstructured":"Alejandro Velasco , Bartolomeo Montrucchio , and Maurizio Rebaudengo . 2016. KITO tool: A fault injection environment in Linux kernel data structures. Microelectr. Reliabil. 60 ( 2016 ). DOI:https:\/\/doi.org\/10.1016\/j.microrel.2016.02.011 10.1016\/j.microrel.2016.02.011 Alejandro Velasco, Bartolomeo Montrucchio, and Maurizio Rebaudengo. 2016. KITO tool: A fault injection environment in Linux kernel data structures. Microelectr. Reliabil. 60 (2016). DOI:https:\/\/doi.org\/10.1016\/j.microrel.2016.02.011"},{"key":"e_1_2_2_197_1","doi-asserted-by":"publisher","DOI":"10.1109\/DSN.2007.16"},{"key":"e_1_2_2_198_1","doi-asserted-by":"publisher","DOI":"10.1109\/PRDC.2007.56"},{"key":"e_1_2_2_199_1","volume-title":"Dean","author":"Vigder Mark","year":"1996","unstructured":"Mark Vigder , W. Morven Gentleman , and John C . Dean . 1996 . COTS Software Integration: State of the Art. Vol. 39190 . National Research Council Canada, Institute for Information Technology . Mark Vigder, W. Morven Gentleman, and John C. Dean. 1996. COTS Software Integration: State of the Art. Vol. 39190. National Research Council Canada, Institute for Information Technology."},{"key":"e_1_2_2_200_1","doi-asserted-by":"publisher","DOI":"10.1109\/4236.991448"},{"key":"e_1_2_2_201_1","volume-title":"The Web Services Specification Reference Book","author":"Walsh Aaron E.","unstructured":"Aaron E. Walsh . 2002. UDDI , SOAP, and WSDL : The Web Services Specification Reference Book . Prentice Hall Professional Technical Reference . Aaron E. Walsh. 2002. UDDI, SOAP, and WSDL: The Web Services Specification Reference Book. Prentice Hall Professional Technical Reference."},{"key":"e_1_2_2_203_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2019.00126"},{"key":"e_1_2_2_204_1","doi-asserted-by":"publisher","DOI":"10.1109\/AST.2015.22"},{"key":"e_1_2_2_205_1","doi-asserted-by":"publisher","DOI":"10.1145\/1985793.1985801"},{"key":"e_1_2_2_206_1","doi-asserted-by":"publisher","DOI":"10.1109\/DSN.2013.6575310"},{"key":"e_1_2_2_207_1","doi-asserted-by":"publisher","DOI":"10.1145\/2601248.2601268"},{"key":"e_1_2_2_208_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVCBT.2018.00011"},{"key":"e_1_2_2_209_1","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3417064"},{"key":"e_1_2_2_210_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377811.3380388"},{"key":"e_1_2_2_211_1","volume-title":"Multi-layered system robustness testing strategy based on abnormal parameter. J. Comput. 8","author":"Xiang Lin","year":"2013","unstructured":"Lin Xiang , Zhan Zhang , Decheng Zuo , and Xiaozong Yang . 2013. Multi-layered system robustness testing strategy based on abnormal parameter. J. Comput. 8 ( 2013 ). DOI:https:\/\/doi.org\/10.4304\/jcp.8.7.1882-1891 10.4304\/jcp.8.7.1882-1891 Lin Xiang, Zhan Zhang, Decheng Zuo, and Xiaozong Yang. 2013. Multi-layered system robustness testing strategy based on abnormal parameter. J. Comput. 8 (2013). DOI:https:\/\/doi.org\/10.4304\/jcp.8.7.1882-1891"},{"key":"e_1_2_2_212_1","doi-asserted-by":"publisher","DOI":"10.1109\/WAINA.2009.19"},{"key":"e_1_2_2_213_1","doi-asserted-by":"publisher","DOI":"10.1145\/3293882.3330579"},{"key":"e_1_2_2_214_1","doi-asserted-by":"publisher","DOI":"10.1109\/ITCS.2009.164"},{"key":"e_1_2_2_215_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cja.2013.05.004"},{"key":"e_1_2_2_216_1","doi-asserted-by":"publisher","DOI":"10.1145\/2536853.2536881"},{"key":"e_1_2_2_217_1","volume-title":"Proceedings of the 2018 48th Annual IEEE\/IFIP International Conference on Dependable Systems and Networks (DSN\u201918)","author":"Yi Edgardo Barsallo","year":"2018","unstructured":"Edgardo Barsallo Yi , Amiya Maji , and Saurabh Bagchi . 2018 . How reliable is my wearable: A fuzz testing-based study . In Proceedings of the 2018 48th Annual IEEE\/IFIP International Conference on Dependable Systems and Networks (DSN\u201918) . 410--417. DOI:https:\/\/doi.org\/10.1109\/DSN.2018.00050 10.1109\/DSN.2018.00050 Edgardo Barsallo Yi, Amiya Maji, and Saurabh Bagchi. 2018. How reliable is my wearable: A fuzz testing-based study. In Proceedings of the 2018 48th Annual IEEE\/IFIP International Conference on Dependable Systems and Networks (DSN\u201918). 410--417. DOI:https:\/\/doi.org\/10.1109\/DSN.2018.00050"},{"key":"e_1_2_2_218_1","doi-asserted-by":"publisher","DOI":"10.1145\/3386901.3388916"},{"key":"e_1_2_2_219_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICOCI.2006.5276549"},{"key":"e_1_2_2_220_1","volume-title":"Machine learning testing: Survey, landscapes and horizons","author":"Zhang Jie M.","year":"2020","unstructured":"Jie M. Zhang , Mark Harman , Lei Ma , and Yang Liu . 2020. Machine learning testing: Survey, landscapes and horizons . IEEE Trans. Softw. Eng . ( 2020 ). DOI:https:\/\/doi.org\/10.1109\/TSE.2019.2962027 10.1109\/TSE.2019.2962027 Jie M. Zhang, Mark Harman, Lei Ma, and Yang Liu. 2020. Machine learning testing: Survey, landscapes and horizons. IEEE Trans. Softw. Eng. (2020). DOI:https:\/\/doi.org\/10.1109\/TSE.2019.2962027"},{"key":"e_1_2_2_221_1","doi-asserted-by":"publisher","DOI":"10.1109\/HPCC.and.EUC.2013.292"}],"container-title":["ACM Computing Surveys"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3448977","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3448977","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:47:55Z","timestamp":1750193275000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3448977"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,5,3]]},"references-count":222,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2022,5,31]]}},"alternative-id":["10.1145\/3448977"],"URL":"https:\/\/doi.org\/10.1145\/3448977","relation":{},"ISSN":["0360-0300","1557-7341"],"issn-type":[{"value":"0360-0300","type":"print"},{"value":"1557-7341","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,5,3]]},"assertion":[{"value":"2020-07-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-02-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-05-03","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}