{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,10]],"date-time":"2026-02-10T17:45:34Z","timestamp":1770745534094,"version":"3.49.0"},"reference-count":104,"publisher":"Association for Computing Machinery (ACM)","issue":"5","license":[{"start":{"date-parts":[[2021,5,25]],"date-time":"2021-05-25T00:00:00Z","timestamp":1621900800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Italian MIUR PRIN 2015","award":["GAUSS - Prot. 2015KWREMX"],"award-info":[{"award-number":["GAUSS - Prot. 2015KWREMX"]}]},{"DOI":"10.13039\/501100000781","name":"European Research Council","doi-asserted-by":"publisher","award":["PRECRIME - Advanced Grant 2017 Agreement n. 787703"],"award-info":[{"award-number":["PRECRIME - Advanced Grant 2017 Agreement n. 787703"]}],"id":[{"id":"10.13039\/501100000781","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Comput. Surv."],"published-print":{"date-parts":[[2022,6,30]]},"abstract":"<jats:p>\n            Field testing refers to testing techniques that operate in the field to reveal those faults that escape in-house testing. Field testing techniques are becoming increasingly popular with the growing complexity of contemporary software systems. In this article, we present the first systematic survey of field testing approaches over a body of 80 collected studies, and propose their categorization based on the environment and the system on which field testing is performed. We discuss four research questions addressing\n            <jats:italic>how<\/jats:italic>\n            software is tested in the field,\n            <jats:italic>what<\/jats:italic>\n            is tested in the field, which are the\n            <jats:italic>requirements<\/jats:italic>\n            , and how field tests are\n            <jats:italic>managed<\/jats:italic>\n            , and identify many challenging research directions.\n          <\/jats:p>","DOI":"10.1145\/3447240","type":"journal-article","created":{"date-parts":[[2021,5,25]],"date-time":"2021-05-25T13:02:30Z","timestamp":1621947750000},"page":"1-39","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":31,"title":["A Survey of Field-based Testing Techniques"],"prefix":"10.1145","volume":"54","author":[{"given":"Antonia","family":"Bertolino","sequence":"first","affiliation":[{"name":"ISTI-CNR, Italy"}]},{"given":"Pietro","family":"Braione","sequence":"additional","affiliation":[{"name":"University of Milano-Bicocca, Italy"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1076-0076","authenticated-orcid":false,"given":"Guglielmo De","family":"Angelis","sequence":"additional","affiliation":[{"name":"IASI-CNR, Italy"}]},{"given":"Luca","family":"Gazzola","sequence":"additional","affiliation":[{"name":"University of Milano-Bicocca, Italy"}]},{"given":"Fitsum","family":"Kifetew","sequence":"additional","affiliation":[{"name":"Fondazione Bruno Kessler (FBK), Italy"}]},{"given":"Leonardo","family":"Mariani","sequence":"additional","affiliation":[{"name":"University of Milano-Bicocca, Italy"}]},{"given":"Matteo","family":"Orr\u00f9","sequence":"additional","affiliation":[{"name":"University of Milano-Bicocca, Italy"}]},{"given":"Mauro","family":"Pezz\u00e8","sequence":"additional","affiliation":[{"name":"Universit\u00e0 della Svizzera Italiana and Schaffhausen Institute of Technology, Switzerland"}]},{"given":"Roberto","family":"Pietrantuono","sequence":"additional","affiliation":[{"name":"University of Naples Federico II, Italy"}]},{"given":"Stefano","family":"Russo","sequence":"additional","affiliation":[{"name":"University of Naples Federico II, Italy"}]},{"given":"Paolo","family":"Tonella","sequence":"additional","affiliation":[{"name":"Universit\u00e0 della Svizzera italiana, Switzerland"}]}],"member":"320","published-online":{"date-parts":[[2021,5,25]]},"reference":[{"key":"e_1_2_2_1_1","volume-title":"Proceedings of the 18th International Conference on the Engineering of Computer-Based Systems (ECBS\u201911)","author":"Akour Mohammed","year":"2011","unstructured":"Mohammed Akour , Akanksha Jaidev , and Tariq M. King . 2011. Towards change propagating test models in autonomic and adaptive systems . In Proceedings of the 18th International Conference on the Engineering of Computer-Based Systems (ECBS\u201911) . IEEE, 89--96. DOI:https:\/\/doi.org\/10.1109\/ECBS. 2011 .23 10.1109\/ECBS.2011.23 Mohammed Akour, Akanksha Jaidev, and Tariq M. King. 2011. Towards change propagating test models in autonomic and adaptive systems. In Proceedings of the 18th International Conference on the Engineering of Computer-Based Systems (ECBS\u201911). IEEE, 89--96. DOI:https:\/\/doi.org\/10.1109\/ECBS.2011.23"},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2013.407"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEIP.2017.17"},{"key":"e_1_2_2_4_1","volume-title":"Article 106363 (Nov.","author":"Alyahya Sultan","year":"2020","unstructured":"Sultan Alyahya . 2020. Crowdsourced software testing: A systematic literature review. Inf. Softw. Technol. 127 , Article 106363 (Nov. 2020 ). DOI:https:\/\/doi.org\/10.1016\/j.infsof.2020.106363 10.1016\/j.infsof.2020.106363 Sultan Alyahya. 2020. Crowdsourced software testing: A systematic literature review. Inf. Softw. Technol. 127, Article 106363 (Nov. 2020). DOI:https:\/\/doi.org\/10.1016\/j.infsof.2020.106363"},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICWS.2008.111"},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/COMPSAC.2007.106"},{"key":"e_1_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/1882362.1882367"},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2010.10.024"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2016.60"},{"key":"e_1_2_2_10_1","first-page":"9","article-title":"Metamorphic runtime checking of applications without test oracles","volume":"28","author":"Bell Jonathan","year":"2015","unstructured":"Jonathan Bell , Christian Murphy , and Gail Kaiser . 2015 . Metamorphic runtime checking of applications without test oracles . J. Defense Softw. Engineering 28 , 2 (2015), 9 -- 13 . Jonathan Bell, Christian Murphy, and Gail Kaiser. 2015. Metamorphic runtime checking of applications without test oracles. J. Defense Softw. Engineering 28, 2 (2015), 9--13.","journal-title":"J. Defense Softw. Engineering"},{"key":"e_1_2_2_11_1","volume-title":"Micael Gallego, Boni Garc\u00eda","author":"Bertolino Antonia","year":"2019","unstructured":"Antonia Bertolino , Guglielmo De Angelis , Micael Gallego, Boni Garc\u00eda , Francisco Gort\u00e1zar , Francesca Lonetti, and Eda March etti. 2019 . A systematic review on cloud testing. Comput. Surv. 52, 5, Article 93 (Sept. 2019). DOI:https:\/\/doi.org\/10.1145\/3331447 10.1145\/3331447 Antonia Bertolino, Guglielmo De Angelis, Micael Gallego, Boni Garc\u00eda, Francisco Gort\u00e1zar, Francesca Lonetti, and Eda Marchetti. 2019. A systematic review on cloud testing. Comput. Surv. 52, 5, Article 93 (Sept. 2019). DOI:https:\/\/doi.org\/10.1145\/3331447"},{"key":"e_1_2_2_12_1","volume-title":"Lars Frantzen, and Andrea Polini.","author":"Bertolino Antonia","year":"2007","unstructured":"Antonia Bertolino , Guglielmo De Angelis , Lars Frantzen, and Andrea Polini. 2007 . The PLASTIC framework and tools for testing service-oriented applications. In International Summer Schools: Software Engineering (ISSSE 2006-2008), Andrea De Lucia and Filomena Ferrucci (Eds.). Lecture Notes in Computer Science, Vol. 5413 . Springer , 106--139. DOI:https:\/\/doi.org\/10.1007\/978-3-540-95888-8_5 10.1007\/978-3-540-95888-8_5 Antonia Bertolino, Guglielmo De Angelis, Lars Frantzen, and Andrea Polini. 2007. The PLASTIC framework and tools for testing service-oriented applications. In International Summer Schools: Software Engineering (ISSSE 2006-2008), Andrea De Lucia and Filomena Ferrucci (Eds.). Lecture Notes in Computer Science, Vol. 5413. Springer, 106--139. DOI:https:\/\/doi.org\/10.1007\/978-3-540-95888-8_5"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2011.227"},{"key":"e_1_2_2_14_1","volume-title":"Proceedings of the 8th International Conference on Web Information Systems and Technologies (WEBIST\u201912)","volume":"140","author":"Bertolino Antonia","year":"2012","unstructured":"Antonia Bertolino , Guglielmo De Angelis , and Andrea Polini . 2012 . Governance policies for verification and validation of service choreographies . In Proceedings of the 8th International Conference on Web Information Systems and Technologies (WEBIST\u201912) , J. Cordeiro and K.H. Krempels (Eds.), Lecture Notes in Business Information Processing , Vol. 140 . Springer, 86--102. DOI:https:\/\/doi.org\/10.1007\/978-3-642-36608-6_6 10.1007\/978-3-642-36608-6_6 Antonia Bertolino, Guglielmo De Angelis, and Andrea Polini. 2012. Governance policies for verification and validation of service choreographies. In Proceedings of the 8th International Conference on Web Information Systems and Technologies (WEBIST\u201912), J. Cordeiro and K.H. Krempels (Eds.), Lecture Notes in Business Information Processing, Vol. 140. Springer, 86--102. DOI:https:\/\/doi.org\/10.1007\/978-3-642-36608-6_6"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSTW.2009.39"},{"key":"e_1_2_2_16_1","first-page":"1","article-title":"A comprehensive architecture for dynamic evolution of online testing of an embedded system","volume":"80","author":"Bhanu J. Sasi","year":"2015","unstructured":"Smt. J. Sasi Bhanu , A. Vinaya Babu , and P. Trimurthy . 2015 . A comprehensive architecture for dynamic evolution of online testing of an embedded system . J. Theor. Appl. Inf. Technol. 80 , 1 (Oct. 2015), 160--172. http:\/\/www.jatit.org\/volumes\/Vol80No1\/17Vol80No1.pdf Smt. J. Sasi Bhanu, A. Vinaya Babu, and P. Trimurthy. 2015. A comprehensive architecture for dynamic evolution of online testing of an embedded system. J. Theor. Appl. Inf. Technol. 80, 1 (Oct. 2015), 160--172. http:\/\/www.jatit.org\/volumes\/Vol80No1\/17Vol80No1.pdf","journal-title":"J. Theor. Appl. Inf. Technol."},{"key":"e_1_2_2_17_1","volume-title":"Proceedings of the 18th International Conference on Parallel Architectures and Compilation Techniques (PACT\u201909)","author":"Bobba Jayaram","year":"2009","unstructured":"Jayaram Bobba , Weiwei Xiong , Luke Yen , Mark D. Hill , and David A. Wood . 2009. StealthTest: Low overhead online software testing using transactional memory . In Proceedings of the 18th International Conference on Parallel Architectures and Compilation Techniques (PACT\u201909) . IEEE, 146--155. DOI:https:\/\/doi.org\/10.1109\/PACT. 2009 .15 10.1109\/PACT.2009.15 Jayaram Bobba, Weiwei Xiong, Luke Yen, Mark D. Hill, and David A. Wood. 2009. StealthTest: Low overhead online software testing using transactional memory. In Proceedings of the 18th International Conference on Parallel Architectures and Compilation Techniques (PACT\u201909). IEEE, 146--155. DOI:https:\/\/doi.org\/10.1109\/PACT.2009.15"},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/EDOC.2006.44"},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2004.07.006"},{"key":"e_1_2_2_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2010.11"},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.4018\/jwsr.2007040103"},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSC.2014.2312953"},{"key":"e_1_2_2_23_1","volume-title":"Proceedings of the 2010 ACM Symposium on Applied Computing (SAC\u201910)","author":"da Costa Andrew Diniz","unstructured":"Andrew Diniz da Costa , Camila Nunes , Viviane Torres da Silva , Baldoino Fonseca , and Carlos J. P . de Lucena. 2010. JAAF+T: A framework to implement self-adaptive agents that apply self-test . In Proceedings of the 2010 ACM Symposium on Applied Computing (SAC\u201910) . ACM, 928--935. DOI:https:\/\/doi.org\/10.1145\/1774088.1774280 10.1145\/1774088.1774280 Andrew Diniz da Costa, Camila Nunes, Viviane Torres da Silva, Baldoino Fonseca, and Carlos J. P. de Lucena. 2010. JAAF+T: A framework to implement self-adaptive agents that apply self-test. In Proceedings of the 2010 ACM Symposium on Applied Computing (SAC\u201910). ACM, 928--935. DOI:https:\/\/doi.org\/10.1145\/1774088.1774280"},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/ARES.2010.22"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.4018\/jsse.2010070103"},{"key":"e_1_2_2_26_1","volume-title":"Proceedings of the 7th International Conference on Web Information Systems and Technologies (WEBIST\u201911)","author":"Angelis Guglielmo De","year":"2011","unstructured":"Guglielmo De Angelis , Antonia Bertolino , and Andrea Polini . 2011 . (role)CAST: A framework for on-line service testing . In Proceedings of the 7th International Conference on Web Information Systems and Technologies (WEBIST\u201911) . SciTePress, Noordwijkerhout, The Netherlands, 13--18. DOI:https:\/\/doi.org\/10.5220\/0003340500130018 10.5220\/0003340500130018 Guglielmo De Angelis, Antonia Bertolino, and Andrea Polini. 2011. (role)CAST: A framework for on-line service testing. In Proceedings of the 7th International Conference on Web Information Systems and Technologies (WEBIST\u201911). SciTePress, Noordwijkerhout, The Netherlands, 13--18. DOI:https:\/\/doi.org\/10.5220\/0003340500130018"},{"key":"e_1_2_2_27_1","volume-title":"Proceedings of the 8th International Conference on Web Information Systems and Technologies (WEBIST\u201912)","author":"Angelis Guglielmo De","year":"2012","unstructured":"Guglielmo De Angelis , Antonia Bertolino , and Andrea Polini . 2012 . Validation and verification policies for governance of service choreographies . In Proceedings of the 8th International Conference on Web Information Systems and Technologies (WEBIST\u201912) . SciTePress, 58--70. DOI:https:\/\/doi.org\/10.5220\/0003936 200580070 10.5220\/0003936200580070 Guglielmo De Angelis, Antonia Bertolino, and Andrea Polini. 2012. Validation and verification policies for governance of service choreographies. In Proceedings of the 8th International Conference on Web Information Systems and Technologies (WEBIST\u201912). SciTePress, 58--70. DOI:https:\/\/doi.org\/10.5220\/0003936200580070"},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1109\/SBESC.2014.27"},{"key":"e_1_2_2_29_1","volume-title":"Article e1733 (May","author":"Derakhshanfar Pouria","year":"2020","unstructured":"Pouria Derakhshanfar , Xavier Devroey , Gilles Perrouin , Andy Zaidman , and Arie van Deursen . 2020. Search-based crash reproduction using behavioural model seeding. Softw. Test. Verif. Reliab. 30, 3 , Article e1733 (May 2020 ). DOI:https:\/\/doi.org\/10.1002\/stvr.1733 10.1002\/stvr.1733 Pouria Derakhshanfar, Xavier Devroey, Gilles Perrouin, Andy Zaidman, and Arie van Deursen. 2020. Search-based crash reproduction using behavioural model seeding. Softw. Test. Verif. Reliab. 30, 3, Article e1733 (May 2020). DOI:https:\/\/doi.org\/10.1002\/stvr.1733"},{"key":"e_1_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.5555\/829505.830124"},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.5555\/850965.854686"},{"key":"e_1_2_2_32_1","volume-title":"Web Services Regression Testing","author":"Penta Massimiliano Di","unstructured":"Massimiliano Di Penta , Marcello Bruno , Gianpiero Esposito , Valentina Mazza , and Gerardo Canfora . 2007. Web Services Regression Testing . Springer , 205--234. DOI:https:\/\/doi.org\/10.1007\/978-3-540-72912-9_8 10.1007\/978-3-540-72912-9_8 Massimiliano Di Penta, Marcello Bruno, Gianpiero Esposito, Valentina Mazza, and Gerardo Canfora. 2007. Web Services Regression Testing. Springer, 205--234. DOI:https:\/\/doi.org\/10.1007\/978-3-540-72912-9_8"},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-17694-4_6"},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/1007512.1007522"},{"key":"e_1_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2015.06.063"},{"key":"e_1_2_2_36_1","volume-title":"Proceedings of 4th International Workshop on Formal Approaches to Software Testing (FATES\u201904)","author":"Frantzen Lars","year":"1848","unstructured":"Lars Frantzen , Jan Tretmans , and Tim A. C. Willemse . 2004. Test generation based on symbolic specifications . In Proceedings of 4th International Workshop on Formal Approaches to Software Testing (FATES\u201904) , Jens Grabowski and Brian Nielsen (Eds.), Lecture Notes in Computer Science,Vol. 3395. Springer, 1--15. DOI:https:\/\/doi.org\/10.1007\/978-3-540-3 1848 -4_1 10.1007\/978-3-540-31848-4_1 Lars Frantzen, Jan Tretmans, and Tim A. C. Willemse. 2004. Test generation based on symbolic specifications. In Proceedings of 4th International Workshop on Formal Approaches to Software Testing (FATES\u201904), Jens Grabowski and Brian Nielsen (Eds.), Lecture Notes in Computer Science,Vol. 3395. Springer, 1--15. DOI:https:\/\/doi.org\/10.1007\/978-3-540-31848-4_1"},{"key":"e_1_2_2_37_1","volume-title":"Proceedings of the 10th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS\u201915)","author":"Erik","year":"2015","unstructured":"Erik M. Fredericks and Betty H. C. Cheng. 2015. Automated generation of adaptive test plans for self-adaptive systems . In Proceedings of the 10th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS\u201915) . IEEE, 157--167. DOI:https:\/\/doi.org\/10.1109\/SEAMS. 2015 .15 10.1109\/SEAMS.2015.15 Erik M. Fredericks and Betty H. C. Cheng. 2015. Automated generation of adaptive test plans for self-adaptive systems. In Proceedings of the 10th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS\u201915). IEEE, 157--167. DOI:https:\/\/doi.org\/10.1109\/SEAMS.2015.15"},{"key":"e_1_2_2_38_1","volume-title":"Proceedings of the 9th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS\u201914)","author":"Fredericks Erik M.","unstructured":"Erik M. Fredericks , Byron DeVries , and Betty H. C. Cheng . 2014. Towards run-time adaptation of test cases for self-adaptive systems in the face of uncertainty . In Proceedings of the 9th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS\u201914) . ACM, 17--26. DOI:https:\/\/doi.org\/10.1145\/2593929.2593937 10.1145\/2593929.2593937 Erik M. Fredericks, Byron DeVries, and Betty H. C. Cheng. 2014. Towards run-time adaptation of test cases for self-adaptive systems in the face of uncertainty. In Proceedings of the 9th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS\u201914). ACM, 17--26. DOI:https:\/\/doi.org\/10.1145\/2593929.2593937"},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.2017.10"},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-77092-3_63"},{"key":"e_1_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASEW.2008.4686292"},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/QSIC.2009.56"},{"key":"e_1_2_2_43_1","volume-title":"Proceedings of the 2nd International Conference on Advances in System Testing and Validation Lifecycle (VALID\u201910)","author":"Gonzalez-Sanchez Alberto","year":"2010","unstructured":"Alberto Gonzalez-Sanchez , Eric Piel , Hans-Gerhard Gross , and Arjan J. C . van Gemund. 2010. Runtime testability in dynamic high-availability component-based systems . In Proceedings of the 2nd International Conference on Advances in System Testing and Validation Lifecycle (VALID\u201910) . IEEE, 37--42. DOI:https:\/\/doi.org\/10.1109\/VALID. 2010 .13 10.1109\/VALID.2010.13 Alberto Gonzalez-Sanchez, Eric Piel, Hans-Gerhard Gross, and Arjan J. C. van Gemund. 2010. Runtime testability in dynamic high-availability component-based systems. In Proceedings of the 2nd International Conference on Advances in System Testing and Validation Lifecycle (VALID\u201910). IEEE, 37--42. DOI:https:\/\/doi.org\/10.1109\/VALID.2010.13"},{"key":"e_1_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/2970276.2970360"},{"key":"e_1_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/2694344.2694390"},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.3233\/JIFS-169076"},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2011.55"},{"key":"e_1_2_2_48_1","first-page":"6","article-title":"Testing of data-centric and event-based dynamic service compositions","volume":"23","author":"Hummer Waldemar","year":"2013","unstructured":"Waldemar Hummer , Orna Raz , Onn Shehory , Philipp Leitner , and Schahram Dustdar . 2013 . Testing of data-centric and event-based dynamic service compositions . Softw. Test. Verif. Reliabil. 23 , 6 (Sep. 2013), 465--497. DOI:https:\/\/doi.org\/10.1002\/stvr.1493 10.1002\/stvr.1493 Waldemar Hummer, Orna Raz, Onn Shehory, Philipp Leitner, and Schahram Dustdar. 2013. Testing of data-centric and event-based dynamic service compositions. Softw. Test. Verif. Reliabil. 23, 6 (Sep. 2013), 465--497. DOI:https:\/\/doi.org\/10.1002\/stvr.1493","journal-title":"Softw. Test. Verif. Reliabil."},{"key":"e_1_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/503112.503114"},{"key":"e_1_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.5555\/2337223.2337279"},{"key":"e_1_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1109\/CMPSAC.1989.65053"},{"key":"e_1_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2014.29"},{"key":"e_1_2_2_53_1","series-title":"Lecture Notes in Computer Science","volume-title":"Clarke","author":"King Tariq M.","year":"2011","unstructured":"Tariq M. King , Andrew A. Allen , Rodolfo Cruz , and Peter J . Clarke . 2011 . Safe runtime validation of behavioral adaptations in autonomic software. In Proceedings of the 8th International Conference on Autonomic and Trusted Computing (ATC\u201911), Jose M. Alcaraz Calero, Laurence T. Yang, F\u00e9lix G\u00f3mez M\u00e1rmol, and Luis Javier Garc\u00eda Villalba (Eds.), Lecture Notes in Computer Science , Vol. 6906 . Springer , 31--46. DOI:https:\/\/doi.org\/10.1007\/978-3-642-23496-5_3 10.1007\/978-3-642-23496-5_3 Tariq M. King, Andrew A. Allen, Rodolfo Cruz, and Peter J. Clarke. 2011. Safe runtime validation of behavioral adaptations in autonomic software. In Proceedings of the 8th International Conference on Autonomic and Trusted Computing (ATC\u201911), Jose M. Alcaraz Calero, Laurence T. Yang, F\u00e9lix G\u00f3mez M\u00e1rmol, and Luis Javier Garc\u00eda Villalba (Eds.), Lecture Notes in Computer Science, Vol. 6906. Springer, 31--46. DOI:https:\/\/doi.org\/10.1007\/978-3-642-23496-5_3"},{"key":"e_1_2_2_54_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISADS.2007.75"},{"key":"e_1_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.4304\/jcp.2.9.37-49"},{"key":"e_1_2_2_57_1","volume-title":"Encyclopedia of Machine Learning and Data Mining, Claude Sammut and Geoffrey I","author":"Kohavi Ron","unstructured":"Ron Kohavi and Roger Longbotham . 2017. Online controlled experiments and A\/B testing . In Encyclopedia of Machine Learning and Data Mining, Claude Sammut and Geoffrey I . Webb (Eds.). Springer , 922--929. DOI:https:\/\/doi.org\/10.1007\/978-1-4899-7687-1_891 10.1007\/978-1-4899-7687-1_891 Ron Kohavi and Roger Longbotham. 2017. Online controlled experiments and A\/B testing. In Encyclopedia of Machine Learning and Data Mining, Claude Sammut and Geoffrey I. Webb (Eds.). Springer, 922--929. DOI:https:\/\/doi.org\/10.1007\/978-1-4899-7687-1_891"},{"key":"e_1_2_2_58_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-34691-0_7"},{"key":"e_1_2_2_59_1","doi-asserted-by":"publisher","DOI":"10.1109\/WETICE.2013.62"},{"key":"e_1_2_2_60_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-34691-0_9"},{"key":"e_1_2_2_61_1","volume-title":"Avanc\u00e9es R\u00e9centes Dans le Domaine Des Architectures Logicielles: Articles S\u00e9lectionn\u00e9s et Etendus de (CAL'11) (Revue des Nouvelles Technologies de l'Information), Philippe Aniort\u00e9 (Ed.)","author":"Lahami Mariam","unstructured":"Mariam Lahami , Moez Krichen , and Mohamed Jmaiel . 2012. A distributed test architecture for adaptable and distributed real-time systems . In Avanc\u00e9es R\u00e9centes Dans le Domaine Des Architectures Logicielles: Articles S\u00e9lectionn\u00e9s et Etendus de (CAL'11) (Revue des Nouvelles Technologies de l'Information), Philippe Aniort\u00e9 (Ed.) , Vol. L-6 . Hermann , 732--92. Mariam Lahami, Moez Krichen, and Mohamed Jmaiel. 2012. A distributed test architecture for adaptable and distributed real-time systems. In Avanc\u00e9es R\u00e9centes Dans le Domaine Des Architectures Logicielles: Articles S\u00e9lectionn\u00e9s et Etendus de (CAL'11) (Revue des Nouvelles Technologies de l'Information), Philippe Aniort\u00e9 (Ed.), Vol. L-6. Hermann, 732--92."},{"key":"e_1_2_2_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/2489300.2489335"},{"key":"e_1_2_2_63_1","doi-asserted-by":"publisher","DOI":"10.1504\/IJCAT.2015.070489"},{"key":"e_1_2_2_64_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.scico.2016.02.002"},{"key":"e_1_2_2_65_1","doi-asserted-by":"publisher","DOI":"10.1109\/SC.2008.5218557"},{"key":"e_1_2_2_66_1","doi-asserted-by":"publisher","DOI":"10.1109\/RTCSA.2005.8"},{"key":"e_1_2_2_67_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-981-10-5041-1_109"},{"key":"e_1_2_2_68_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jlap.2008.08.004"},{"key":"e_1_2_2_69_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICMCIS.2016.7496557"},{"key":"e_1_2_2_70_1","doi-asserted-by":"publisher","DOI":"10.1109\/QSIC.2010.69"},{"key":"e_1_2_2_71_1","doi-asserted-by":"publisher","DOI":"10.1109\/COMPSACW.2012.86"},{"key":"e_1_2_2_72_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSC.2014.2322621"},{"key":"e_1_2_2_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/1808984.1808987"},{"key":"e_1_2_2_74_1","volume-title":"Proceedings of the 2017 International Conference on Software Quality, Reliability and Security (QRS\u201917)","author":"Mor\u00e1n Jes\u00fas","year":"2017","unstructured":"Jes\u00fas Mor\u00e1n , Antonia Bertolino , Claudio de la Riva, and Javier Tuya. 2017. Towards ex Vivo testing of mapreduce applications . In Proceedings of the 2017 International Conference on Software Quality, Reliability and Security (QRS\u201917) . IEEE, 73--80. DOI:https:\/\/doi.org\/10.1109\/QRS. 2017 .17 10.1109\/QRS.2017.17 Jes\u00fas Mor\u00e1n, Antonia Bertolino, Claudio de la Riva, and Javier Tuya. 2017. Towards ex Vivo testing of mapreduce applications. In Proceedings of the 2017 International Conference on Software Quality, Reliability and Security (QRS\u201917). IEEE, 73--80. DOI:https:\/\/doi.org\/10.1109\/QRS.2017.17"},{"key":"e_1_2_2_75_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2009.18"},{"key":"e_1_2_2_76_1","doi-asserted-by":"publisher","DOI":"10.1145\/1572272.1572295"},{"key":"e_1_2_2_77_1","doi-asserted-by":"publisher","DOI":"10.1145\/1808266.1808269"},{"key":"e_1_2_2_78_1","doi-asserted-by":"publisher","DOI":"10.1504\/IJES.2016.080388"},{"key":"e_1_2_2_79_1","volume-title":"Proceedings of the 20th Iberoamerican Conference on Software Engineering (CibSE\u201917)","volume":"1","author":"de Oliveira Neves V\u00e2nia","year":"2017","unstructured":"V\u00e2nia de Oliveira Neves , M\u00e1rcio Eduardo Delamaro , and Paulo Cesar Masiero . 2017 . Automated structural software testing of autonomous vehicles . In Proceedings of the 20th Iberoamerican Conference on Software Engineering (CibSE\u201917) , Vol. 1 . Curran Associates, Buenos Aires, Argentina, 15--28. V\u00e2nia de Oliveira Neves, M\u00e1rcio Eduardo Delamaro, and Paulo Cesar Masiero. 2017. Automated structural software testing of autonomous vehicles. In Proceedings of the 20th Iberoamerican Conference on Software Engineering (CibSE\u201917), Vol. 1. Curran Associates, Buenos Aires, Argentina, 15--28."},{"key":"e_1_2_2_80_1","volume-title":"Proceedings of the 3rd IEEE International Conference on Self-Adaptive and Self-Organizing Systems (SASO\u201909)","author":"Niebuhr Dirk","year":"2009","unstructured":"Dirk Niebuhr , Andreas Rausch , Cornel Klein , Juergen Reichmann , and Reiner N. Schmid . 2009. Achieving dependable component bindings in dynamic adaptive systems - A runtime testing approach . In Proceedings of the 3rd IEEE International Conference on Self-Adaptive and Self-Organizing Systems (SASO\u201909) . IEEE, 186--197. DOI:https:\/\/doi.org\/10.1109\/SASO. 2009 .40 10.1109\/SASO.2009.40 Dirk Niebuhr, Andreas Rausch, Cornel Klein, Juergen Reichmann, and Reiner N. Schmid. 2009. Achieving dependable component bindings in dynamic adaptive systems - A runtime testing approach. In Proceedings of the 3rd IEEE International Conference on Self-Adaptive and Self-Organizing Systems (SASO\u201909). IEEE, 186--197. DOI:https:\/\/doi.org\/10.1109\/SASO.2009.40"},{"key":"e_1_2_2_81_1","doi-asserted-by":"publisher","DOI":"10.1109\/RELDIS.1996.559698"},{"key":"e_1_2_2_82_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2015.03.027"},{"key":"e_1_2_2_83_1","doi-asserted-by":"publisher","DOI":"10.1109\/COMPSAC.2009.24"},{"key":"e_1_2_2_84_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2015.03.007"},{"key":"e_1_2_2_85_1","volume-title":"Software Testing and Analysis: Process, Principles, and Techniques","author":"Pezz\u00e8 Mauro","unstructured":"Mauro Pezz\u00e8 and Michal Young . 2008. Software Testing and Analysis: Process, Principles, and Techniques . John Wiley & Sons . Mauro Pezz\u00e8 and Michal Young. 2008. Software Testing and Analysis: Process, Principles, and Techniques. John Wiley & Sons."},{"key":"e_1_2_2_86_1","doi-asserted-by":"publisher","DOI":"10.1145\/1596495.1596499"},{"key":"e_1_2_2_87_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2007.70719"},{"key":"e_1_2_2_88_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2013.18"},{"key":"e_1_2_2_89_1","doi-asserted-by":"publisher","DOI":"10.1109\/COMPSAC.2011.81"},{"key":"e_1_2_2_90_1","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2018.111094748"},{"key":"e_1_2_2_91_1","volume-title":"Gerber","author":"Smidts Carol","year":"2014","unstructured":"Carol Smidts , Chetan Mutha , Manuel Rodriguez , and Matthew J . Gerber . 2014 . Software testing with an operational profile: OP definition. Comput. Surv. 46, 3, Article 39 (Feb. 2014). DOI:https:\/\/doi.org\/10.1145\/2518106 10.1145\/2518106 Carol Smidts, Chetan Mutha, Manuel Rodriguez, and Matthew J. Gerber. 2014. Software testing with an operational profile: OP definition. Comput. Surv. 46, 3, Article 39 (Feb. 2014). DOI:https:\/\/doi.org\/10.1145\/2518106"},{"key":"e_1_2_2_92_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-019-09762-1"},{"key":"e_1_2_2_93_1","volume-title":"Proceedings of the 45th Annual Southeast Regional Conference (ACM-SE 45)","author":"Stevens Ronald","unstructured":"Ronald Stevens , Brittany Parsons , and Tariq M. King . 2007. A self-testing autonomic container . In Proceedings of the 45th Annual Southeast Regional Conference (ACM-SE 45) . ACM, 1--6. DOI:https:\/\/doi.org\/10.1145\/1233341.1233343 10.1145\/1233341.1233343 Ronald Stevens, Brittany Parsons, and Tariq M. King. 2007. A self-testing autonomic container. In Proceedings of the 45th Annual Southeast Regional Conference (ACM-SE 45). ACM, 1--6. DOI:https:\/\/doi.org\/10.1145\/1233341.1233343"},{"key":"e_1_2_2_94_1","doi-asserted-by":"publisher","DOI":"10.1109\/COMPSAC.2006.169"},{"key":"e_1_2_2_95_1","volume-title":"Proceedings of the 3rd International Workshop on Theory of Reliability and Markov Modeling for Information Technologies (TheRMIT\u201917)","author":"Vain J\u00fcri","year":"2017","unstructured":"J\u00fcri Vain , Leonidas Tsiopoulos , Vyacheslav Kharchenko , Apneet Kaur , Maksim Jenihhin , and Jaan Raik . 2017 . Multi-fragment Markov model guided online test generation for MPSoC . In Proceedings of the 3rd International Workshop on Theory of Reliability and Markov Modeling for Information Technologies (TheRMIT\u201917) . 594--607. J\u00fcri Vain, Leonidas Tsiopoulos, Vyacheslav Kharchenko, Apneet Kaur, Maksim Jenihhin, and Jaan Raik. 2017. Multi-fragment Markov model guided online test generation for MPSoC. In Proceedings of the 3rd International Workshop on Theory of Reliability and Markov Modeling for Information Technologies (TheRMIT\u201917). 594--607."},{"key":"e_1_2_2_96_1","doi-asserted-by":"publisher","DOI":"10.1145\/1081706.1081751"},{"key":"e_1_2_2_97_1","doi-asserted-by":"publisher","DOI":"10.1145\/1868914.1868973"},{"key":"e_1_2_2_98_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISADS.2007.54"},{"key":"e_1_2_2_99_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1018990322378"},{"key":"e_1_2_2_100_1","doi-asserted-by":"publisher","DOI":"10.1109\/FTDCS.1993.344143"},{"key":"e_1_2_2_101_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2013.20"},{"key":"e_1_2_2_102_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE.2004.4"},{"key":"e_1_2_2_103_1","doi-asserted-by":"publisher","DOI":"10.1109\/COMPSAC.2006.95"},{"key":"e_1_2_2_104_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSC.2010.54"},{"key":"e_1_2_2_105_1","volume-title":"A test automation framework for collaborative testing of web service dynamic compositions","author":"Zhu Hong","unstructured":"Hong Zhu and Yufeng Zhang . 2014. A test automation framework for collaborative testing of web service dynamic compositions . In Advanced Web Services, Athman Bouguettaya, Quan Z. Sheng, and Florian Daniel (Eds.). Springer , 171--197. DOI:https:\/\/doi.org\/10.1007\/978-1-4614-7535-4_8 10.1007\/978-1-4614-7535-4_8 Hong Zhu and Yufeng Zhang. 2014. A test automation framework for collaborative testing of web service dynamic compositions. In Advanced Web Services, Athman Bouguettaya, Quan Z. Sheng, and Florian Daniel (Eds.). Springer, 171--197. DOI:https:\/\/doi.org\/10.1007\/978-1-4614-7535-4_8"}],"container-title":["ACM Computing Surveys"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3447240","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3447240","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T21:28:27Z","timestamp":1750195707000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3447240"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,5,25]]},"references-count":104,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2022,6,30]]}},"alternative-id":["10.1145\/3447240"],"URL":"https:\/\/doi.org\/10.1145\/3447240","relation":{},"ISSN":["0360-0300","1557-7341"],"issn-type":[{"value":"0360-0300","type":"print"},{"value":"1557-7341","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,5,25]]},"assertion":[{"value":"2020-04-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-01-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-05-25","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}