{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,26]],"date-time":"2026-03-26T15:46:42Z","timestamp":1774540002712,"version":"3.50.1"},"reference-count":168,"publisher":"Association for Computing Machinery (ACM)","issue":"5","license":[{"start":{"date-parts":[[2019,9,13]],"date-time":"2019-09-13T00:00:00Z","timestamp":1568332800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Spanish Government under project LERNIM","award":["RTC-2016-4674-7"],"award-info":[{"award-number":["RTC-2016-4674-7"]}]},{"name":"Regional Government of Madrid (CM) under project Cloud4BigData","award":["S2013\/ICE-2894"],"award-info":[{"award-number":["S2013\/ICE-2894"]}]},{"name":"Italian MIUR PRIN 2015 Project GAUSS"},{"name":"FSE 8 FEDER"},{"name":"European Project H2020","award":["731535"],"award-info":[{"award-number":["731535"]}]},{"name":"Ministry of Economy and Competitiveness FEDER 8 AEI"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Comput. Surv."],"published-print":{"date-parts":[[2020,9,30]]},"abstract":"<jats:p>A systematic literature review is presented that surveyed the topic of cloud testing over the period 2012--2017. Cloud testing can refer either to testing cloud-based systems (testing of the cloud) or to leveraging the cloud for testing purposes (testing in the cloud): both approaches (and their combination into testing of the cloud in the cloud) have drawn research interest. An extensive paper search was conducted by both automated query of popular digital libraries and snowballing, which resulted in the final selection of 147 primary studies. Along the survey, a framework has been incrementally derived that classifies cloud testing research among six main areas and their topics. The article includes a detailed analysis of the selected primary studies to identify trends and gaps, as well as an extensive report of the state-of-the-art as it emerges by answering the identified Research Questions. We find that cloud testing is an active research field, although not all topics have received enough attention and conclude by presenting the most relevant open research challenges for each area of the classification framework.<\/jats:p>","DOI":"10.1145\/3331447","type":"journal-article","created":{"date-parts":[[2019,9,13]],"date-time":"2019-09-13T12:28:56Z","timestamp":1568377736000},"page":"1-42","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":32,"title":["A Systematic Review on Cloud Testing"],"prefix":"10.1145","volume":"52","author":[{"given":"Antonia","family":"Bertolino","sequence":"first","affiliation":[{"name":"CNR-ISTI, Pisa, Italy"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1076-0076","authenticated-orcid":false,"given":"Guglielmo De","family":"Angelis","sequence":"additional","affiliation":[{"name":"CNR-IASI, Roma, Italy"}]},{"given":"Micael","family":"Gallego","sequence":"additional","affiliation":[{"name":"Universidad Rey Juan Carlos, Madrid, Spain"}]},{"given":"Boni","family":"Garc\u00eda","sequence":"additional","affiliation":[{"name":"Universidad Rey Juan Carlos, Madrid, Spain"}]},{"given":"Francisco","family":"Gort\u00e1zar","sequence":"additional","affiliation":[{"name":"Universidad Rey Juan Carlos, Madrid, Spain"}]},{"given":"Francesca","family":"Lonetti","sequence":"additional","affiliation":[{"name":"CNR-ISTI, Pisa, Italy"}]},{"given":"Eda","family":"Marchetti","sequence":"additional","affiliation":[{"name":"CNR-ISTI, Pisa, Italy"}]}],"member":"320","published-online":{"date-parts":[[2019,9,13]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/AINA.2014.117"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/SERVICES.2014.81"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/CLOUD.2015.36"},{"key":"e_1_2_1_4_1","volume-title":"Proceedings of the IEEE International Conference on Software Quality, Reliability and Security. IEEE, 555--562","author":"Al-Said Ahmad Amro","year":"2017","unstructured":"Amro Al-Said Ahmad , Pearl Brereton , and Peter Andras . 2017 . A systematic mapping study of empirical studies on software cloud testing methods . In Proceedings of the IEEE International Conference on Software Quality, Reliability and Security. IEEE, 555--562 . Amro Al-Said Ahmad, Pearl Brereton, and Peter Andras. 2017. A systematic mapping study of empirical studies on software cloud testing methods. In Proceedings of the IEEE International Conference on Software Quality, Reliability and Security. IEEE, 555--562."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3023147.3023149"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/CLOUD.2017.56"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2851613.2852004"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2859889.2859890"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.5220\/0006308905230530"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCITechnol.2012.6285773"},{"key":"e_1_2_1_11_1","volume-title":"Proceedings of the 7th IEEE International Conference on Intelligent Computing and Information Systems (ICICIS\u201915)","author":"Ali Amira","year":"2016","unstructured":"Amira Ali and Nagwa Badr . 2016 . Performance testing as a service for web applications . In Proceedings of the 7th IEEE International Conference on Intelligent Computing and Information Systems (ICICIS\u201915) . 356--361. Amira Ali and Nagwa Badr. 2016. Performance testing as a service for web applications. In Proceedings of the 7th IEEE International Conference on Intelligent Computing and Information Systems (ICICIS\u201915). 356--361."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/2637365.2517216"},{"key":"e_1_2_1_13_1","volume-title":"A semi-automatic and trustworthy scheme for continuous cloud service certification","author":"Anisetti Marco","year":"2017","unstructured":"Marco Anisetti , Claudio Ardagna , Ernesto Damiani , and Filippo Gaudenzi . 2017. A semi-automatic and trustworthy scheme for continuous cloud service certification . IEEE Trans. Serv. Comput . ( 2017 ). Marco Anisetti, Claudio Ardagna, Ernesto Damiani, and Filippo Gaudenzi. 2017. A semi-automatic and trustworthy scheme for continuous cloud service certification. IEEE Trans. Serv. Comput. (2017)."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/1721654.1721672"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.22266\/ijies2017.0430.11"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/SOSE.2011.6139087"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.5555\/2662413.2662417"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10009-016-0430-5"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-40972-1_22"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/IMIS.2014.2"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/GLOCOMW.2013.6825037"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/FOSE.2007.25"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3183440.3183497"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2006.07.009"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1002\/stvr.1544"},{"key":"e_1_2_1_27_1","volume-title":"Software Engineering Frameworks for the Cloud Computing Paradigm","author":"Chana Inderveer","unstructured":"Inderveer Chana and Priyanka Chawla . 2013. Testing perspectives for cloud-based applications . In Software Engineering Frameworks for the Cloud Computing Paradigm . Springer , 145--164. Inderveer Chana and Priyanka Chawla. 2013. Testing perspectives for cloud-based applications. In Software Engineering Frameworks for the Cloud Computing Paradigm. Springer, 145--164."},{"key":"e_1_2_1_28_1","volume-title":"Muhammad Ali Babar, and Christian W. Probst","author":"Chauhan Muhammad Aufeef","year":"2016","unstructured":"Muhammad Aufeef Chauhan , Muhammad Ali Babar, and Christian W. Probst . 2016 . A process framework for designing software reference architectures for providing tools as a service. In Proceedings of the International Conference on Product-Focused Software Process Improvement. Springer , 111--126. Muhammad Aufeef Chauhan, Muhammad Ali Babar, and Christian W. Probst. 2016. A process framework for designing software reference architectures for providing tools as a service. In Proceedings of the International Conference on Product-Focused Software Process Improvement. Springer, 111--126."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jcss.2015.12.001"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/IPDPS.2014.41"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/2970276.2970294"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICS.2016.0046"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2744210.2744213"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/CLOUD.2013.12"},{"key":"e_1_2_1_35_1","volume-title":"Cloud Crawler: A declarative performance evaluation environment for infrastructure-as-a-service clouds. Concur. Comput.: Pract. Exper. 29, 1","author":"Cunha Matheus","year":"2017","unstructured":"Matheus Cunha , N. C. Mendon\u00e7a , and Am\u00e9rico Sampaio . 2017 . Cloud Crawler: A declarative performance evaluation environment for infrastructure-as-a-service clouds. Concur. Comput.: Pract. Exper. 29, 1 (2017). Matheus Cunha, N. C. Mendon\u00e7a, and Am\u00e9rico Sampaio. 2017. Cloud Crawler: A declarative performance evaluation environment for infrastructure-as-a-service clouds. Concur. Comput.: Pract. Exper. 29, 1 (2017)."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/IECON.2014.7048856"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/INCoS.2014.62"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/PDCAT.2013.61"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/IC2E.2017.30"},{"key":"e_1_2_1_40_1","doi-asserted-by":"crossref","unstructured":"Sergio Di Martino Filomena Ferrucci Valerio Maggio and Federica Sarro. 2012. Towards migrating genetic algorithms for test data generation to the cloud. In Software Testing in the Cloud: Perspectives on an Emerging Discipline. IGI Global 113--135.   Sergio Di Martino Filomena Ferrucci Valerio Maggio and Federica Sarro. 2012. Towards migrating genetic algorithms for test data generation to the cloud. In Software Testing in the Cloud: Perspectives on an Emerging Discipline. IGI Global 113--135.","DOI":"10.4018\/978-1-4666-2536-5.ch006"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/SERVICES.2016.25"},{"key":"e_1_2_1_42_1","volume-title":"Proceedings of the Technology Conference on Performance Evaluation and Benchmarking. Springer, 173--188","author":"Folkerts Enno","year":"2012","unstructured":"Enno Folkerts , Alexander Alexandrov , Kai Sachs , Alexandru Iosup , Volker Markl , and Cafer Tosun . 2012 . Benchmarking in the cloud: What it should, can, and cannot be . In Proceedings of the Technology Conference on Performance Evaluation and Benchmarking. Springer, 173--188 . Enno Folkerts, Alexander Alexandrov, Kai Sachs, Alexandru Iosup, Volker Markl, and Cafer Tosun. 2012. Benchmarking in the cloud: What it should, can, and cannot be. In Proceedings of the Technology Conference on Performance Evaluation and Benchmarking. Springer, 173--188."},{"key":"e_1_2_1_43_1","volume-title":"Proceedings of the International Conference on Collaboration Technologies and Systems (CTS\u201912)","author":"Fox Geoffrey C.","unstructured":"Geoffrey C. Fox , Supun Kamburugamuve , and Ryan D. Hartman . 2012. Architecture and measured characteristics of a cloud based internet of things . In Proceedings of the International Conference on Collaboration Technologies and Systems (CTS\u201912) . IEEE, 6--12. Geoffrey C. Fox, Supun Kamburugamuve, and Ryan D. Hartman. 2012. Architecture and measured characteristics of a cloud based internet of things. In Proceedings of the International Conference on Collaboration Technologies and Systems (CTS\u201912). IEEE, 6--12."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2017.51"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2013.6693140"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1109\/MIC.2013.119"},{"key":"e_1_2_1_47_1","first-page":"9","article-title":"Cloud testing-issues, challenges, needs and practice","volume":"1","author":"Gao Jerry","year":"2011","unstructured":"Jerry Gao , Xiaoying Bai , and Wei-Tek Tsai . 2011 . Cloud testing-issues, challenges, needs and practice . Softw. Eng.: Int. J. 1 , 1 (2011), 9 -- 23 . Jerry Gao, Xiaoying Bai, and Wei-Tek Tsai. 2011. Cloud testing-issues, challenges, needs and practice. Softw. Eng.: Int. J. 1, 1 (2011), 9--23.","journal-title":"Softw. Eng.: Int. J."},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/SOSE.2013.66"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2013.445"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1109\/CloudCom.2012.6427555"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1109\/HASE.2014.30"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1109\/SOSE.2013.59"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2017.3641116"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1109\/INTECH.2013.6653653"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1109\/CLOUD.2015.112"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1109\/CLOUD.2015.126"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICWS.2016.24"},{"key":"e_1_2_1_58_1","volume-title":"Proceedings of the International Conference on Software Testing, Verification and Validation (ICST\u201915)","author":"Herbold Steffen","unstructured":"Steffen Herbold , Alberto De Francesco , Jens Grabowski , Patrick Harms , Lom M. Hillah , Fabrice Kordon , Ariele-Paolo Maesano , Libero Maesano , Claudia Di Napoli , Fabio De Rosa et al. 2015. The MIDAS cloud platform for testing SOA applications . In Proceedings of the International Conference on Software Testing, Verification and Validation (ICST\u201915) . IEEE, 1--8. Steffen Herbold, Alberto De Francesco, Jens Grabowski, Patrick Harms, Lom M. Hillah, Fabrice Kordon, Ariele-Paolo Maesano, Libero Maesano, Claudia Di Napoli, Fabio De Rosa et al. 2015. The MIDAS cloud platform for testing SOA applications. In Proceedings of the International Conference on Software Testing, Verification and Validation (ICST\u201915). IEEE, 1--8."},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10009-016-0437-y"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10009-016-0440-3"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1109\/IS3C.2014.17"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISPDC.2016.28"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1109\/MobileCloud.2014.13"},{"key":"e_1_2_1_64_1","volume-title":"Proceedings of the 4th IEEE International Conference on Cloud Computing Technology and Science (CloudCom\u201912)","author":"Gong Huang","year":"2012","unstructured":"Jun-fei Huang and Yun-zhan Gong . 2012 . Remote mobile test system: A mobile phone cloud for application testing . In Proceedings of the 4th IEEE International Conference on Cloud Computing Technology and Science (CloudCom\u201912) . IEEE, 1--4. Jun-fei Huang and Yun-zhan Gong. 2012. Remote mobile test system: A mobile phone cloud for application testing. In Proceedings of the 4th IEEE International Conference on Cloud Computing Technology and Science (CloudCom\u201912). IEEE, 1--4."},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1109\/SERVICES.2013.39"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1109\/WAINA.2013.94"},{"key":"e_1_2_1_67_1","volume-title":"Roland K\u0171bert et al.","author":"Hume Alastair C.","year":"2012","unstructured":"Alastair C. Hume , Yahya Al-Hazmi , Bartosz Belter , Konrad Campowsky , Luis M. Carril , Gino Carrozzo , Vegard Engen , David Garc\u00eda-P\u00e9rez , Jordi Jofre Ponsat\u00ed , Roland K\u0171bert et al. 2012 . BonFIRE: A multi-cloud test facility for internet of services experimentation. In Proceedings of the International Conference on Testbeds and Research Infrastructures. Springer , 81--96. Alastair C. Hume, Yahya Al-Hazmi, Bartosz Belter, Konrad Campowsky, Luis M. Carril, Gino Carrozzo, Vegard Engen, David Garc\u00eda-P\u00e9rez, Jordi Jofre Ponsat\u00ed, Roland K\u0171bert et al. 2012. BonFIRE: A multi-cloud test facility for internet of services experimentation. In Proceedings of the International Conference on Testbeds and Research Infrastructures. Springer, 81--96."},{"key":"e_1_2_1_68_1","volume-title":"Saifan","author":"Husni Hind","year":"2017","unstructured":"Hind Husni and Ahmad A . Saifan . 2017 . Cloud testing: Steps, tools, challenges. In Proceedings of the New Trends in Information Technology (NTIT'17). The University of Jordan , Amman, Jordan, Paper 16. Hind Husni and Ahmad A. Saifan. 2017. Cloud testing: Steps, tools, challenges. In Proceedings of the New Trends in Information Technology (NTIT'17). The University of Jordan, Amman, Jordan, Paper 16."},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSESS.2014.6933530"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2015.2398438"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1109\/SERE-C.2012.32"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1109\/QSIC.2013.65"},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCC.2013.10"},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.17485\/ijst\/2015\/v8i35\/76200"},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.1109\/CLOUD.2012.98"},{"key":"e_1_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2015.01.058"},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICWS.2015.54"},{"key":"e_1_2_1_78_1","first-page":"6","article-title":"Software testing in cloud platform: A survey","volume":"46","author":"Vanitha Katherine A.","year":"2012","unstructured":"A. Vanitha Katherine and K. Alagarsamy . 2012 . Software testing in cloud platform: A survey . Int. J. Comput. Appl. 46 , 6 (May 2012), 21--25. A. Vanitha Katherine and K. Alagarsamy. 2012. Software testing in cloud platform: A survey. Int. J. Comput. Appl. 46, 6 (May 2012), 21--25.","journal-title":"Int. J. Comput. Appl."},{"key":"e_1_2_1_79_1","doi-asserted-by":"publisher","DOI":"10.1109\/CCAA.2016.7813826"},{"key":"e_1_2_1_80_1","doi-asserted-by":"publisher","DOI":"10.1109\/CloudCom.2013.156"},{"key":"e_1_2_1_81_1","volume-title":"Schwach","author":"Kiran Mariam","year":"2014","unstructured":"Mariam Kiran , Andreas Friesen , Anthony J. H. Simons , and Wolfgang K. R . Schwach . 2014 . Model-based Testing in Cloud Brokerage Scenarios. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Vol. 8377 LNCS. 192--208. Mariam Kiran, Andreas Friesen, Anthony J. H. Simons, and Wolfgang K. R. Schwach. 2014. Model-based Testing in Cloud Brokerage Scenarios. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Vol. 8377 LNCS. 192--208."},{"key":"e_1_2_1_82_1","volume-title":"Simons","author":"Kiran Mariam","year":"2014","unstructured":"Mariam Kiran and Anthony J. H . Simons . 2014 . Model-based testing for composite web services in cloud brokerage scenarios. In Proceedings of the European Conference on Service-Oriented and Cloud Computing. Springer , 190--205. Mariam Kiran and Anthony J. H. Simons. 2014. Model-based testing for composite web services in cloud brokerage scenarios. In Proceedings of the European Conference on Service-Oriented and Cloud Computing. Springer, 190--205."},{"key":"e_1_2_1_84_1","doi-asserted-by":"publisher","DOI":"10.1109\/CISIS.2013.85"},{"key":"e_1_2_1_85_1","doi-asserted-by":"publisher","DOI":"10.1109\/SOSE.2013.78"},{"key":"e_1_2_1_86_1","doi-asserted-by":"publisher","DOI":"10.1109\/MobServ.2015.36"},{"key":"e_1_2_1_87_1","volume-title":"Proceedings of the 4th International Conference on Advances in System Testing and Validation Lifecycle (VALID\u201912)","author":"Lamas Etiene","year":"2012","unstructured":"Etiene Lamas , Luis A. V. Dias , and Adilson Marques Da Cunha . 2012 . Software architectural drivers for cloud testing . In Proceedings of the 4th International Conference on Advances in System Testing and Validation Lifecycle (VALID\u201912) . 114--120. Etiene Lamas, Luis A. V. Dias, and Adilson Marques Da Cunha. 2012. Software architectural drivers for cloud testing. In Proceedings of the 4th International Conference on Advances in System Testing and Validation Lifecycle (VALID\u201912). 114--120."},{"key":"e_1_2_1_88_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11036-012-0375-1"},{"key":"e_1_2_1_89_1","volume-title":"Simons","author":"Lefticaru Raluca","year":"2014","unstructured":"Raluca Lefticaru and Anthony J. H . Simons . 2014 . X-Machine based testing for cloud services. In Proceedings of the European Conference on Service-Oriented and Cloud Computing. Springer , 175--189. Raluca Lefticaru and Anthony J. H. Simons. 2014. X-Machine based testing for cloud services. In Proceedings of the European Conference on Service-Oriented and Cloud Computing. Springer, 175--189."},{"key":"e_1_2_1_90_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASI.2016.7539857"},{"key":"e_1_2_1_91_1","doi-asserted-by":"publisher","DOI":"10.1109\/MobServ.2015.37"},{"key":"e_1_2_1_92_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICASI.2017.7988384"},{"key":"e_1_2_1_93_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICS.2016.0125"},{"key":"e_1_2_1_94_1","doi-asserted-by":"publisher","DOI":"10.1145\/2811681.2811691"},{"key":"e_1_2_1_95_1","doi-asserted-by":"publisher","DOI":"10.5555\/2429759.2430064"},{"key":"e_1_2_1_96_1","doi-asserted-by":"publisher","DOI":"10.1109\/COMPSAC.2015.271"},{"key":"e_1_2_1_97_1","doi-asserted-by":"publisher","DOI":"10.1109\/IC3.2014.6897235"},{"key":"e_1_2_1_98_1","doi-asserted-by":"publisher","DOI":"10.5555\/2663608.2663613"},{"key":"e_1_2_1_99_1","doi-asserted-by":"publisher","DOI":"10.1145\/2635868.2635896"},{"key":"e_1_2_1_100_1","volume-title":"Proceedings of the International Conference on Computing and Communication Technologies (ICCCT\u201914)","author":"Malini A.","unstructured":"A. Malini , N. Venkatesh , K. Sundarakantham , and S. Mercyshalinie . 2014. Mobile application testing on smart devices using MTAAS framework in cloud . In Proceedings of the International Conference on Computing and Communication Technologies (ICCCT\u201914) . A. Malini, N. Venkatesh, K. Sundarakantham, and S. Mercyshalinie. 2014. Mobile application testing on smart devices using MTAAS framework in cloud. In Proceedings of the International Conference on Computing and Communication Technologies (ICCCT\u201914)."},{"key":"e_1_2_1_101_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCCRI.2015.10"},{"key":"e_1_2_1_102_1","volume-title":"Murthy","author":"Markande Krishna","year":"2013","unstructured":"Krishna Markande and Sridhar J . Murthy . 2013 . Leveraging potential of cloud for software performance testing. In Cloud Computing. Springer , 293--322. Krishna Markande and Sridhar J. Murthy. 2013. Leveraging potential of cloud for software performance testing. In Cloud Computing. Springer, 293--322."},{"key":"e_1_2_1_103_1","doi-asserted-by":"publisher","DOI":"10.1145\/3030207.3044530"},{"key":"e_1_2_1_104_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.procs.2014.09.006"},{"key":"e_1_2_1_105_1","volume-title":"Proceedings of the 8th IEEE International Conference on Informatics and Systems (INFOS\u201912)","author":"Mohamed Eman M.","unstructured":"Eman M. Mohamed , Sherif El-Etriby , and Hatem S . Abdul-kader. 2012. Randomness testing of modern encryption techniques in cloud environment . In Proceedings of the 8th IEEE International Conference on Informatics and Systems (INFOS\u201912) . IEEE, CC--1. Eman M. Mohamed, Sherif El-Etriby, and Hatem S. Abdul-kader. 2012. Randomness testing of modern encryption techniques in cloud environment. In Proceedings of the 8th IEEE International Conference on Informatics and Systems (INFOS\u201912). IEEE, CC--1."},{"key":"e_1_2_1_106_1","doi-asserted-by":"publisher","DOI":"10.1109\/FiCloud.2015.106"},{"key":"e_1_2_1_107_1","doi-asserted-by":"publisher","DOI":"10.1109\/NGCT.2015.7375075"},{"key":"e_1_2_1_108_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.procs.2015.04.018"},{"key":"e_1_2_1_109_1","doi-asserted-by":"publisher","DOI":"10.1145\/2460999.2461037"},{"key":"e_1_2_1_110_1","doi-asserted-by":"publisher","DOI":"10.1109\/SAMI.2015.7061877"},{"key":"e_1_2_1_111_1","doi-asserted-by":"publisher","DOI":"10.1109\/MIC.2010.113"},{"key":"e_1_2_1_112_1","volume-title":"Proceedings of the International Conference on Model-Driven Engineering and Software Development. Springer, 114--131","author":"Petrova-Antonova Dessislava","year":"2016","unstructured":"Dessislava Petrova-Antonova , Sylvia Ilieva , and Denitsa Manova . 2016 . Automated web service composition testing as a service . In Proceedings of the International Conference on Model-Driven Engineering and Software Development. Springer, 114--131 . Dessislava Petrova-Antonova, Sylvia Ilieva, and Denitsa Manova. 2016. Automated web service composition testing as a service. In Proceedings of the International Conference on Model-Driven Engineering and Software Development. Springer, 114--131."},{"key":"e_1_2_1_113_1","doi-asserted-by":"publisher","DOI":"10.5220\/0005844400330042"},{"key":"e_1_2_1_114_1","volume-title":"Proceedings of the International Conference on Advanced Communication Control and Computing Technologies (ICACCCT\u201914)","author":"Prathibhan C. Mano","unstructured":"C. Mano Prathibhan , A. Malini , N. Venkatesh , and K. Sundarakantham . 2014. An automated testing framework for testing Android mobile applications in the cloud . In Proceedings of the International Conference on Advanced Communication Control and Computing Technologies (ICACCCT\u201914) . IEEE, 1216--1219. C. Mano Prathibhan, A. Malini, N. Venkatesh, and K. Sundarakantham. 2014. An automated testing framework for testing Android mobile applications in the cloud. In Proceedings of the International Conference on Advanced Communication Control and Computing Technologies (ICACCCT\u201914). IEEE, 1216--1219."},{"key":"e_1_2_1_115_1","doi-asserted-by":"publisher","DOI":"10.1145\/2180921.2180938"},{"key":"e_1_2_1_116_1","doi-asserted-by":"publisher","DOI":"10.1109\/SOSE.2015.45"},{"key":"e_1_2_1_117_1","doi-asserted-by":"publisher","DOI":"10.1145\/2593793.2593794"},{"key":"e_1_2_1_118_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCCN.2014.6911762"},{"key":"e_1_2_1_119_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11219-014-9256-0"},{"key":"e_1_2_1_120_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11219-014-9256-0"},{"key":"e_1_2_1_121_1","doi-asserted-by":"publisher","DOI":"10.1145\/2993288.2993301"},{"key":"e_1_2_1_122_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.simpat.2013.04.002"},{"key":"e_1_2_1_123_1","doi-asserted-by":"publisher","DOI":"10.1109\/MobServ.2015.57"},{"key":"e_1_2_1_124_1","doi-asserted-by":"publisher","DOI":"10.1145\/2757667.2757677"},{"key":"e_1_2_1_125_1","article-title":"Cloud based testing techniques (CTT)","volume":"104","author":"Shrivastva Akash","year":"2014","unstructured":"Akash Shrivastva , Shubham Gupta , and Rinki Tiwari . 2014 . Cloud based testing techniques (CTT) . Int. J. Comput. Appl. 104 , 5 (2014). Akash Shrivastva, Shubham Gupta, and Rinki Tiwari. 2014. Cloud based testing techniques (CTT). Int. J. Comput. Appl. 104, 5 (2014).","journal-title":"Int. J. Comput. Appl."},{"key":"e_1_2_1_126_1","first-page":"397","article-title":"Cloud testing\u2014A systematic review","volume":"2","author":"Siddiqui Tamanna","year":"2015","unstructured":"Tamanna Siddiqui and Riaz Ahmad . 2015 . Cloud testing\u2014A systematic review . Int. Res. J. Eng. Technol. 2 , 3 (2015), 397 -- 406 . Tamanna Siddiqui and Riaz Ahmad. 2015. Cloud testing\u2014A systematic review. Int. Res. J. Eng. Technol. 2, 3 (2015), 397--406.","journal-title":"Int. Res. J. Eng. Technol."},{"key":"e_1_2_1_127_1","doi-asserted-by":"publisher","DOI":"10.1109\/IC2E.2013.33"},{"key":"e_1_2_1_128_1","doi-asserted-by":"publisher","DOI":"10.1109\/AINA.2017.57"},{"key":"e_1_2_1_129_1","doi-asserted-by":"publisher","DOI":"10.5555\/2662413.2662415"},{"key":"e_1_2_1_130_1","doi-asserted-by":"publisher","DOI":"10.1145\/2380552.2380579"},{"key":"e_1_2_1_131_1","article-title":"The design and execution of performance testing strategy for cloud-based system","volume":"1","author":"Mohamed Suffian Muhammad Dhiauddin","year":"2014","unstructured":"Muhammad Dhiauddin Mohamed Suffian , Fairul Rizal Fahrurazi , and Suhaimi Ibrahim . 2014 . The design and execution of performance testing strategy for cloud-based system . Int. J. Softw. Eng. Technol. 1 , 2 (2014). Muhammad Dhiauddin Mohamed Suffian, Fairul Rizal Fahrurazi, and Suhaimi Ibrahim. 2014. The design and execution of performance testing strategy for cloud-based system. Int. J. Softw. Eng. Technol. 1, 2 (2014).","journal-title":"Int. J. Softw. Eng. Technol."},{"key":"e_1_2_1_132_1","doi-asserted-by":"publisher","DOI":"10.1142\/S0218194016500078"},{"key":"e_1_2_1_133_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2016.11.016"},{"key":"e_1_2_1_134_1","volume-title":"Proceedings of the 3rd International Conference on Advanced Cloud and Big Data (CBD\u201915)","author":"Tao Chuanqi","year":"2016","unstructured":"Chuanqi Tao , Jerry Gao , and Bixin Li . 2016 . Cloud-based infrastructure for mobile testing as a service . In Proceedings of the 3rd International Conference on Advanced Cloud and Big Data (CBD\u201915) . 133--140. Chuanqi Tao, Jerry Gao, and Bixin Li. 2016. Cloud-based infrastructure for mobile testing as a service. In Proceedings of the 3rd International Conference on Advanced Cloud and Big Data (CBD\u201915). 133--140."},{"key":"e_1_2_1_135_1","first-page":"537","article-title":"Cloud platform based automated security testing system for mobile internet. Tsinghua Sci","volume":"20","author":"Tao Dan","year":"2015","unstructured":"Dan Tao , Zhaowen Lin , and Cheng Lu . 2015 . Cloud platform based automated security testing system for mobile internet. Tsinghua Sci . Technol. 20 , 6 (2015), 537 -- 544 . Dan Tao, Zhaowen Lin, and Cheng Lu. 2015. Cloud platform based automated security testing system for mobile internet. Tsinghua Sci. Technol. 20, 6 (2015), 537--544.","journal-title":"Technol."},{"key":"e_1_2_1_136_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSTW.2014.43"},{"key":"e_1_2_1_137_1","doi-asserted-by":"publisher","DOI":"10.1145\/3107091.3107093"},{"key":"e_1_2_1_138_1","doi-asserted-by":"publisher","DOI":"10.1109\/IC2E.2013.44"},{"key":"e_1_2_1_139_1","doi-asserted-by":"publisher","DOI":"10.1109\/SOSE.2014.11"},{"key":"e_1_2_1_140_1","doi-asserted-by":"publisher","DOI":"10.1109\/SOSE.2015.12"},{"key":"e_1_2_1_141_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.simpat.2016.08.003"},{"key":"e_1_2_1_142_1","volume-title":"Combinatorial Testing in Cloud Computing","author":"Tsai Wei-Tek","unstructured":"Wei-Tek Tsai and Guanqiu Qi. 2017. Adaptive fault detection in multi-tenancy SaaS systems . In Combinatorial Testing in Cloud Computing . Springer , 25--36. Wei-Tek Tsai and Guanqiu Qi. 2017. Adaptive fault detection in multi-tenancy SaaS systems. In Combinatorial Testing in Cloud Computing. Springer, 25--36."},{"key":"e_1_2_1_143_1","volume-title":"Combinatorial Testing in Cloud Computing","author":"Tsai Wei-Tek","unstructured":"Wei-Tek Tsai and Guanqiu Qi. 2017. Adaptive reasoning algorithm with automated test cases generation and test algebra in SaaS system . In Combinatorial Testing in Cloud Computing . Springer , 83--99. Wei-Tek Tsai and Guanqiu Qi. 2017. Adaptive reasoning algorithm with automated test cases generation and test algebra in SaaS system. In Combinatorial Testing in Cloud Computing. Springer, 83--99."},{"key":"e_1_2_1_144_1","volume-title":"Combinatorial Testing in Cloud Computing","author":"Tsai Wei-Tek","unstructured":"Wei-Tek Tsai and Guanqiu Qi. 2017. Integrated TaaS with fault detection and test algebra . In Combinatorial Testing in Cloud Computing . Springer , 115--128. Wei-Tek Tsai and Guanqiu Qi. 2017. Integrated TaaS with fault detection and test algebra. In Combinatorial Testing in Cloud Computing. Springer, 115--128."},{"key":"e_1_2_1_145_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISADS.2015.53"},{"key":"e_1_2_1_146_1","doi-asserted-by":"publisher","DOI":"10.1109\/SERE.2014.26"},{"key":"e_1_2_1_147_1","doi-asserted-by":"publisher","DOI":"10.1109\/SOSE.2014.36"},{"key":"e_1_2_1_148_1","doi-asserted-by":"publisher","DOI":"10.1109\/APNOMS.2015.7275375"},{"key":"e_1_2_1_149_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2012.335"},{"key":"e_1_2_1_150_1","volume-title":"Press release: Gartner forecasts worldwide public cloud services revenue to reach $260 billion","author":"van der Meulen Rob","year":"2017","unstructured":"Rob van der Meulen and Christy Pettey . 2017. Press release: Gartner forecasts worldwide public cloud services revenue to reach $260 billion in 2017 . Retrieved from: https:\/\/www.gartner.com\/newsroom\/id\/3815165. Rob van der Meulen and Christy Pettey. 2017. Press release: Gartner forecasts worldwide public cloud services revenue to reach $260 billion in 2017. Retrieved from: https:\/\/www.gartner.com\/newsroom\/id\/3815165."},{"key":"e_1_2_1_151_1","doi-asserted-by":"publisher","DOI":"10.1145\/2361999.2362008"},{"key":"e_1_2_1_152_1","first-page":"213","article-title":"Cloud testing: A state-of-the-art review","volume":"28","author":"Vilkomir Sergiy","year":"2012","unstructured":"Sergiy Vilkomir . 2012 . Cloud testing: A state-of-the-art review . Inform. Sec. 28 , 2 (2012), 213 -- 222 . Sergiy Vilkomir. 2012. Cloud testing: A state-of-the-art review. Inform. Sec. 28, 2 (2012), 213--222.","journal-title":"Inform. Sec."},{"key":"e_1_2_1_153_1","doi-asserted-by":"publisher","DOI":"10.1109\/SERVICES.2015.20"},{"key":"e_1_2_1_154_1","doi-asserted-by":"publisher","DOI":"10.1109\/COMPSAC.2017.24"},{"key":"e_1_2_1_155_1","doi-asserted-by":"publisher","DOI":"10.1145\/2601248.2601268"},{"key":"e_1_2_1_156_1","doi-asserted-by":"publisher","DOI":"10.1109\/SOSE.2014.13"},{"key":"e_1_2_1_157_1","volume-title":"Proceedings of the 25th Euromicro International Conference on Parallel, Distributed and Network-based Processing (PDP\u201917)","author":"Xavier Miguel G.","unstructured":"Miguel G. Xavier , Kassiano J. Matteussi , Gabriel R. Fran\u00e7a , Wagner P. Pereira , and Cesar A. F . De Rose. 2017. Mobile application testing on clouds: Challenges, opportunities and architectural elements . In Proceedings of the 25th Euromicro International Conference on Parallel, Distributed and Network-based Processing (PDP\u201917) . IEEE, 181--185. Miguel G. Xavier, Kassiano J. Matteussi, Gabriel R. Fran\u00e7a, Wagner P. Pereira, and Cesar A. F. De Rose. 2017. Mobile application testing on clouds: Challenges, opportunities and architectural elements. In Proceedings of the 25th Euromicro International Conference on Parallel, Distributed and Network-based Processing (PDP\u201917). IEEE, 181--185."},{"key":"e_1_2_1_158_1","first-page":"65","article-title":"URMG: Enhanced CBMG-based method for automatically testing web applications in the cloud. TSINGHUA Sci","volume":"19","author":"Xu Xiaolin","year":"2014","unstructured":"Xiaolin Xu , Hai Jin , Song Wu , Lixiang Tang , and Yihong Wang . 2014 . URMG: Enhanced CBMG-based method for automatically testing web applications in the cloud. TSINGHUA Sci . Technol. 19 , 1 (2014), 65 -- 75 . Xiaolin Xu, Hai Jin, Song Wu, Lixiang Tang, and Yihong Wang. 2014. URMG: Enhanced CBMG-based method for automatically testing web applications in the cloud. TSINGHUA Sci. Technol. 19, 1 (2014), 65--75.","journal-title":"Technol."},{"key":"e_1_2_1_159_1","doi-asserted-by":"publisher","DOI":"10.1109\/CLUSTER.2012.20"},{"key":"e_1_2_1_160_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICPADS.2012.69"},{"key":"e_1_2_1_161_1","doi-asserted-by":"publisher","DOI":"10.1002\/sec.352"},{"key":"e_1_2_1_162_1","doi-asserted-by":"publisher","DOI":"10.1109\/SOSE.2013.77"},{"key":"e_1_2_1_163_1","doi-asserted-by":"publisher","DOI":"10.1109\/SERE-C.2012.11"},{"key":"e_1_2_1_164_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2016.03.065"},{"key":"e_1_2_1_165_1","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2011.158"},{"key":"e_1_2_1_166_1","doi-asserted-by":"publisher","DOI":"10.1109\/SOSE.2015.27"},{"key":"e_1_2_1_167_1","doi-asserted-by":"publisher","DOI":"10.1109\/CLOUDCOM-ASIA.2013.57"},{"key":"e_1_2_1_168_1","doi-asserted-by":"publisher","DOI":"10.1109\/SNPD.2014.6888746"},{"key":"e_1_2_1_169_1","volume-title":"Proceedings of the 6th USENIX Workshop on Hot Topics in Cloud Computing. 1, S2","author":"Zhi Junji","year":"2014","unstructured":"Junji Zhi , Sahil Suneja , and Eyal De Lara . 2014 . The case for system testing with swift hierarchical vm fork . In Proceedings of the 6th USENIX Workshop on Hot Topics in Cloud Computing. 1, S2 (2014), S3. Junji Zhi, Sahil Suneja, and Eyal De Lara. 2014. The case for system testing with swift hierarchical vm fork. In Proceedings of the 6th USENIX Workshop on Hot Topics in Cloud Computing. 1, S2 (2014), S3."},{"key":"e_1_2_1_170_1","doi-asserted-by":"publisher","DOI":"10.1145\/2462307.2462321"}],"container-title":["ACM Computing Surveys"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3331447","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3331447","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T00:43:28Z","timestamp":1750207408000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3331447"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,9,13]]},"references-count":168,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2020,9,30]]}},"alternative-id":["10.1145\/3331447"],"URL":"https:\/\/doi.org\/10.1145\/3331447","relation":{},"ISSN":["0360-0300","1557-7341"],"issn-type":[{"value":"0360-0300","type":"print"},{"value":"1557-7341","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,9,13]]},"assertion":[{"value":"2018-10-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-05-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-09-13","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}