{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,18]],"date-time":"2025-12-18T14:08:34Z","timestamp":1766066914628,"version":"3.41.0"},"reference-count":76,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2018,8,25]],"date-time":"2018-08-25T00:00:00Z","timestamp":1535155200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"publisher","award":["KO 3445\/11-1"],"award-info":[{"award-number":["KO 3445\/11-1"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100007601","name":"Horizon 2020 Framework Programme","doi-asserted-by":"publisher","award":["ICT-06-2016: Cloud Computing No 73225"],"award-info":[{"award-number":["ICT-06-2016: Cloud Computing No 73225"]}],"id":[{"id":"10.13039\/501100007601","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100003246","name":"Nederlandse Organisatie voor Wetenschappelijk Onderzoek","doi-asserted-by":"publisher","award":["MagnaData & Commit"],"award-info":[{"award-number":["MagnaData & Commit"]}],"id":[{"id":"10.13039\/501100003246","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Model. Perform. Eval. Comput. Syst."],"published-print":{"date-parts":[[2018,12,31]]},"abstract":"<jats:p>\n            In only a decade, cloud computing has emerged from a pursuit for a service-driven information and communication technology (ICT), becoming a significant fraction of the ICT market. Responding to the growth of the market, many alternative cloud services and their underlying systems are currently vying for the attention of cloud users and providers. To make informed choices between competing cloud service providers, permit the cost-benefit analysis of cloud-based systems, and enable system DevOps to evaluate and tune the performance of these complex ecosystems, appropriate performance metrics, benchmarks, tools, and methodologies are necessary. This requires re-examining old system properties and considering new system properties, possibly leading to the re-design of classic benchmarking metrics such as expressing performance as throughput and latency (response time). In this work, we address these requirements by focusing on four system properties: (i)\n            <jats:italic>elasticity<\/jats:italic>\n            of the cloud service, to accommodate large variations in the amount of service requested, (ii)\u00a0\n            <jats:italic>performance isolation<\/jats:italic>\n            between the tenants of shared cloud systems and resulting\n            <jats:italic>performance variability<\/jats:italic>\n            , (iii)\u00a0\n            <jats:italic>availability<\/jats:italic>\n            of cloud services and systems, and (iv) the\n            <jats:italic>operational risk<\/jats:italic>\n            of running a production system in a cloud environment. Focusing on key metrics for each of these properties, we review the state-of-the-art, then select or propose new metrics together with measurement approaches. We see the presented metrics as a foundation toward upcoming, future industry-standard cloud benchmarks.\n          <\/jats:p>","DOI":"10.1145\/3236332","type":"journal-article","created":{"date-parts":[[2018,8,27]],"date-time":"2018-08-27T12:13:23Z","timestamp":1535372003000},"page":"1-36","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":28,"title":["Quantifying Cloud Performance and Dependability"],"prefix":"10.1145","volume":"3","member":"320","published-online":{"date-parts":[[2018,8,25]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/3030207.3030229"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","unstructured":"Giuseppe Aceto et al. 2013. Cloud monitoring: A survey. Comput. Netw. 57 9 (2013). 10.1016\/j.comnet.2013.04.001","DOI":"10.1016\/j.comnet.2013.04.001"},{"key":"e_1_2_1_3_1","volume-title":"Machado","author":"Almeida Rodrigo F.","year":"2013","unstructured":"Rodrigo F. Almeida, Fl\u00e1vio R. C. Sousa, S\u00e9rgio Lifschitz, and Javam C. Machado. 2013. On defining metrics for elasticity of cloud databases. In Proceedings of the SBBD. Retrieved from http:\/\/sbbd2013.cin.ufpe.br\/Proceedings\/artigos\/sbbd_shp_12.html."},{"key":"e_1_2_1_4_1","unstructured":"Amazon. 2017. EC2 Compute SLA. Retrieved from http:\/\/aws.amazon.com\/ec2\/sla\/."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2016.64"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/945445.945462"},{"volume-title":"On the value of service demand estimation for auto-scaling","author":"Bauer Andr\u00e9","key":"e_1_2_1_7_1","unstructured":"Andr\u00e9 Bauer, Johannes Grohmann, Nikolas Herbst, and Samuel Kounev. 2018. On the value of service demand estimation for auto-scaling. In Proceedings of GI\/ITG MMB. Springer."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2668930.2688043"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/MCC.2014.51"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/1594156.1594168"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICIW.2010.91"},{"key":"e_1_2_1_12_1","unstructured":"Dean Chandler et al. 2012. Report on Cloud Computing to the OSG Steering Committee. Technical Report. Retrieved from http:\/\/www.spec.org\/osgcloud\/docs\/osgcloudwgreport20120410.pdf."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.5555\/1247415.1247462"},{"key":"e_1_2_1_14_1","unstructured":"CloudSleuth. 2017. CloudSleuth monitoring network. Retrieved from https:\/\/cloud.spring.io\/spring-cloud-sleuth\/."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/1807128.1807152"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/1551609.1551635"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.5555\/1191828.1192595"},{"volume-title":"Proceedings of ACM\/IEEE CCGrid.","author":"Thibault","key":"e_1_2_1_18_1","unstructured":"Thibault Dory et al. 2011. Measuring elasticity for cloud databases. In Proceedings of ACM\/IEEE CCGrid. Retrieved from http:\/\/www.info.ucl.ac.be\/ pvr\/CC2011elasticityCRfinal.pdf."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/1287624.1287679"},{"volume-title":"Uptake of Cloud in Europe. Final Report. Digital Agenda for Europe report","author":"European Commission","key":"e_1_2_1_20_1","unstructured":"European Commission. 2014. Uptake of Cloud in Europe. Final Report. Digital Agenda for Europe report. Publications Office of the European Union, Luxembourg."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.future.2016.11.002"},{"key":"e_1_2_1_22_1","article-title":"A survey of Cloud monitoring tools: Taxonomy, capabilities and objectives","volume":"74","author":"Kaniz Fatema","year":"2014","unstructured":"Kaniz Fatema et al. 2014. A survey of Cloud monitoring tools: Taxonomy, capabilities and objectives. J. Parallel Distrib. Comput. 74, 10 (2014).","journal-title":"J. Parallel Distrib. Comput."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISPASS.2015.7095802"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.future.2011.05.022"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/5666.5673"},{"key":"e_1_2_1_26_1","volume-title":"Selected Topics in Performance Evaluation and Benchmarking. LNCS","volume":"7755","author":"Folkerts Enno","year":"2012","unstructured":"Enno Folkerts, Alexander Alexandrov, Kai Sachs, Alexandru Iosup, Volker Markl, and Cafer Tosun. 2012. Benchmarking in the cloud: What it should, can, and cannot be. In Selected Topics in Performance Evaluation and Benchmarking. LNCS, Vol. 7755."},{"key":"e_1_2_1_27_1","unstructured":"N. Forsgren Velasquez et al. 2015. State of DevOps report 2015. Puppet Labs IT Revolut. (2015). https:\/\/puppet.com\/resources\/whitepaper\/2015-state-devops-report."},{"key":"e_1_2_1_28_1","unstructured":"Martin Fowler. 2013. Continuous delivery. Retrieved from https:\/\/martinfowler.com\/bliki\/ContinuousDelivery.html."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","unstructured":"Saurabh Kumar Garg et al. 2013. A framework for ranking of cloud computing services. Elsevier FGCS 29 4 (2013). 10.1016\/j.future.2012.06.006","DOI":"10.1016\/j.future.2012.06.006"},{"key":"e_1_2_1_30_1","unstructured":"Google. Compute level SLA. Retrieved from https:\/\/cloud.google.com\/compute\/sla."},{"volume-title":"Proceedings of IEEE ICSA Workshops. IEEE, 243--246","author":"Hasselbring W.","key":"e_1_2_1_31_1","unstructured":"W. Hasselbring and G. Steinacker. 2017. Microservice architectures for scalability, agility and reliability in E-commerce. In Proceedings of IEEE ICSA Workshops. IEEE, 243--246."},{"key":"e_1_2_1_32_1","volume-title":"Proceedings of USENIX ICAC. USENIX.","author":"Herbst Nikolas","year":"2013","unstructured":"Nikolas Herbst, Samuel Kounev, and Ralf Reussner. 2013. Elasticity in cloud computing: What it is, and what it is not. In Proceedings of USENIX ICAC. USENIX. Retrieved from https:\/\/www.usenix.org\/conference\/icac13\/elasticity-cloud-computing-what-it-and-what-it-not."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.5555\/2821357.2821366"},{"volume-title":"Performance Evaluation and Benchmarking","author":"Huppler Karl","key":"e_1_2_1_34_1","unstructured":"Karl Huppler. 2009. Performance Evaluation and Benchmarking. Springer-Verlag, Berlin, 18--30."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-32627-1_7"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3030207.3030214"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2011.66"},{"volume-title":"Proceedings of IEEE Network Operations and Management Symposium (NOMS'10)","author":"Isci C.","key":"e_1_2_1_39_1","unstructured":"C. Isci, J. E. Hanson, I. Whalley, M. Steinder, and J. O. Kephart. 2010. Runtime demand estimation for effective dynamic resource management. In Proceedings of IEEE Network Operations and Management Symposium (NOMS'10). IEEE, 381--388."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/2188286.2188301"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10922-014-9307-7"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/71.862209"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISCC.2011.5984009"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.5220\/0004975407140721"},{"key":"e_1_2_1_45_1","first-page":"116","article-title":"Metrics and techniques for quantifying performance isolation in cloud environments","volume":"90","author":"Krebs Rouven","year":"2014","unstructured":"Rouven Krebs, Christof Momm, and Samuel Kounev. 2014. Metrics and techniques for quantifying performance isolation in cloud environments. Elsevier SciCo Vol. 90, Part B (2014), 116--134.","journal-title":"Elsevier SciCo"},{"key":"e_1_2_1_46_1","unstructured":"Michael Kuperberg et al. 2011. Defining and Quantifying Elasticity of Resources in Cloud Computing and Scalable Platforms. Technical Report. KIT Germany. Retrieved from http:\/\/digbib.ubka.uni-karlsruhe.de\/volltexte\/1000023476."},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/2885497"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/3184407.3184415"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/1879141.1879143"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1109\/Grid.2012.15"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10723-014-9314-7"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/1998582.1998606"},{"volume-title":"Proceedings of ACM\/IEEE CCGrid.","author":"Mao Ming","key":"e_1_2_1_53_1","unstructured":"Ming Mao, Jie Li, and M. Humphrey. 2010. Cloud auto-scaling with deadline and budget constraints. In Proceedings of ACM\/IEEE CCGrid."},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/1064979.1064984"},{"key":"e_1_2_1_55_1","unstructured":"Microsoft. 2017. Azure compute level SLA. Retrieved from https:\/\/azure.microsoft.com\/en-us\/support\/legal\/sla\/virtual-machines\/v1_2\/."},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/1383519.1383526"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1145\/2930659"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.comnet.2015.09.037"},{"volume-title":"Proceedings of IEEE GLOBECOM.","author":"Persico V.","key":"e_1_2_1_59_1","unstructured":"V. Persico, P. Marchetta, A. Botta, and A. Pescape. 2015. On network throughput variability in Microsoft azure cloud. In Proceedings of IEEE GLOBECOM."},{"key":"e_1_2_1_60_1","volume-title":"Study: Five Refining Attributes of Public and Private Cloud Computing. Technical Report. Gartner.","author":"Plummer D. C.","year":"2009","unstructured":"D. C. Plummer et al. 2009. Study: Five Refining Attributes of Public and Private Cloud Computing. Technical Report. Gartner."},{"key":"e_1_2_1_61_1","unstructured":"Mike Roberts. 2016. Serverless architectures. Retrieved from https:\/\/martinfowler.com\/articles\/serverless.html."},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.14778\/1920841.1920902"},{"volume-title":"Proceedings of ICSCS. 1--5.","author":"Shawky D. M.","key":"e_1_2_1_63_1","unstructured":"D. M. Shawky and A. F. Ali. 2012. Defining a measure of cloud computing elasticity. In Proceedings of ICSCS. 1--5."},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1109\/CCGrid.2015.58"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1109\/SRII.2012.51"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1109\/SCC.2012.65"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/2024723.2000099"},{"key":"e_1_2_1_68_1","volume-title":"Proceedings of DBKDA. 125--131","author":"Tinnefeld Christian","year":"2014","unstructured":"Christian Tinnefeld, Daniel Taschik, and Hasso Plattner. 2014. Quantifying the elasticity of a database management system. In Proceedings of DBKDA. 125--131. Retrieved from http:\/\/www.thinkmind.org\/index.php?view&equals;article8articleid&equals;dbkda_2014_5_30_50076."},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/3154847.3154848"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1109\/CCGrid.2012.46"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1145\/3019596"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1145\/1646468.1646475"},{"key":"e_1_2_1_73_1","unstructured":"Joe Weinman. 2011. Time is Money: The Value of \u201cOn-Demand.\u201d Retrieved from http:\/\/www.joeweinman.com\/resources\/Joe_Weinman_Time_Is_Money.pdf."},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1109\/CCGRID.2009.40"},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.1109\/UCC.2011.33"},{"key":"e_1_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.1145\/1037949.1024415"},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1145\/1736020.1736036"}],"container-title":["ACM Transactions on Modeling and Performance Evaluation of Computing Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3236332","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3236332","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T01:39:39Z","timestamp":1750210779000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3236332"}},"subtitle":["Taxonomy, Metric Design, and Emerging Challenges"],"editor":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3462-6426","authenticated-orcid":false,"given":"Nikolas","family":"Herbst","sequence":"first","affiliation":[]},{"given":"Andr\u00e9","family":"Bauer","sequence":"additional","affiliation":[]},{"given":"Samuel","family":"Kounev","sequence":"additional","affiliation":[]},{"given":"Giorgos","family":"Oikonomou","sequence":"additional","affiliation":[]},{"given":"Erwin Van","family":"Eyk","sequence":"additional","affiliation":[]},{"given":"George","family":"Kousiouris","sequence":"additional","affiliation":[]},{"given":"Athanasia","family":"Evangelinou","sequence":"additional","affiliation":[]},{"given":"Rouven","family":"Krebs","sequence":"additional","affiliation":[]},{"given":"Tim","family":"Brecht","sequence":"additional","affiliation":[]},{"given":"Cristina L.","family":"Abad","sequence":"additional","affiliation":[]},{"given":"Alexandru","family":"Iosup","sequence":"additional","affiliation":[]}],"short-title":[],"issued":{"date-parts":[[2018,8,25]]},"references-count":76,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2018,12,31]]}},"alternative-id":["10.1145\/3236332"],"URL":"https:\/\/doi.org\/10.1145\/3236332","relation":{},"ISSN":["2376-3639","2376-3647"],"issn-type":[{"type":"print","value":"2376-3639"},{"type":"electronic","value":"2376-3647"}],"subject":[],"published":{"date-parts":[[2018,8,25]]},"assertion":[{"value":"2017-09-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-07-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-08-25","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}