{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,1]],"date-time":"2026-05-01T17:12:34Z","timestamp":1777655554605,"version":"3.51.4"},"reference-count":54,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2016,4,19]],"date-time":"2016-04-19T00:00:00Z","timestamp":1461024000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"European Community's Seventh Framework Programme","award":["610802 (CloudWave)"],"award-info":[{"award-number":["610802 (CloudWave)"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Internet Technol."],"published-print":{"date-parts":[[2016,8,22]]},"abstract":"<jats:p>Benchmarking the performance of public cloud providers is a common research topic. Previous work has already extensively evaluated the performance of different cloud platforms for different use cases, and under different constraints and experiment setups. In this article, we present a principled, large-scale literature review to collect and codify existing research regarding the predictability of performance in public Infrastructure-as-a-Service (IaaS) clouds. We formulate 15 hypotheses relating to the nature of performance variations in IaaS systems, to the factors of influence of performance variations, and how to compare different instance types. In a second step, we conduct extensive real-life experimentation on four cloud providers to empirically validate those hypotheses. We show that there are substantial differences between providers. Hardware heterogeneity is today less prevalent than reported in earlier research, while multitenancy has a dramatic impact on performance and predictability, but only for some cloud providers. We were unable to discover a clear impact of the time of the day or the day of the week on cloud performance.<\/jats:p>","DOI":"10.1145\/2885497","type":"journal-article","created":{"date-parts":[[2016,4,22]],"date-time":"2016-04-22T13:53:06Z","timestamp":1461333186000},"page":"1-23","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":161,"title":["Patterns in the Chaos\u2014A Study of Performance Variation and Predictability in Public IaaS Clouds"],"prefix":"10.1145","volume":"16","author":[{"given":"Philipp","family":"Leitner","sequence":"first","affiliation":[{"name":"Department of Informatics, University of Zurich, Zurich, Switzerland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"J\u00fcrgen","family":"Cito","sequence":"additional","affiliation":[{"name":"Department of Informatics, University of Zurich, Zurich, Switzerland"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2016,4,19]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/WAINA.2010.166"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/1721654.1721672"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.compeleceng.2014.03.017"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/1730836.1730842"},{"key":"e_1_2_1_5_1","volume-title":"Proceedings of the 18th IEEE International Enterprise Distributed Object Computing Conference (EDOC\u201914)","author":"Borhani Amir Hossein","year":"2014"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.future.2008.12.001"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/QEST.2012.23"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/CloudCom.2014.71"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/CLOUD.2013.12"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.5555\/1926618.1926641"},{"key":"e_1_2_1_11_1","first-page":"3","article-title":"Running scientific codes on Amazon EC2: A performance analysis of five high-end instances","volume":"13","author":"Exposito Roberto R.","year":"2013","journal-title":"Computer Science and Technology"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/2391229.2391249"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/MESOCA.2012.6392599"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00607-014-0410-0"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2087522.2087535"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1186\/2192-113X-2-6"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/MWC.2013.6549280"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/1456659.1456671"},{"key":"e_1_2_1_19_1","doi-asserted-by":"crossref","unstructured":"Zach Hill and Marty Humphrey. 2009. A quantitative analysis of high performance computing with Amazon\u2019s EC2 infrastructure: The death of the local cluster? In GRID. IEEE 26--33.  Zach Hill and Marty Humphrey. 2009. A quantitative analysis of high performance computing with Amazon\u2019s EC2 infrastructure: The death of the local cluster? In GRID. IEEE 26--33.","DOI":"10.1109\/GRID.2009.5353067"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/1851476.1851532"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/UCC.2013.40"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPDS.2011.66"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/CCGrid.2011.22"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/CloudCom.2010.69"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/CLOUD.2012.98"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.5555\/998675.999432"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/1807167.1807231"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1080\/0952813X.2013.784812"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/2245276.2245360"},{"key":"e_1_2_1_30_1","volume-title":"Proceedings of the 8th IEEE\/ACM International Conference on Utility and Cloud Computing (UCC\u201915)","author":"Leitner Philipp","year":"2015"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/CLOUD.2011.80"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/1879141.1879143"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/CloudCom.2013.7"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/SERVICES.2013.73"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1109\/Grid.2012.15"},{"key":"e_1_2_1_36_1","article-title":"On the conceptualization of performance evaluation of IaaS services","author":"Li Zheng","year":"2013","journal-title":"IEEE Transactions on Services Computing."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/CLOUD.2012.103"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/2287036.2287045"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/CloudCom.2013.69"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCC.2013.12"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/CloudCom.2011.114"},{"key":"e_1_2_1_42_1","first-page":"2404821","article-title":"Understanding the performance and potential of cloud computing for scientific applications","volume":"2015","author":"Sadooghi Iman","year":"2015","journal-title":"IEEE Transactions on Cloud Computing PP, 99. DOI:http:\/\/dx.doi.org\/10.1109\/TCC."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.14778\/1920841.1920902"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/2740908.2742833"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/CloudCom.2014.98"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2015.43"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-89652-4_32"},{"key":"e_1_2_1_48_1","volume-title":"Cloudstone: Multi-platform, multi-language benchmark and measurement tools for Web 2.0. In Cloud Computing and Its Applications.","author":"Sobel Will","year":"2008"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/2382196.2382228"},{"key":"e_1_2_1_50_1","first-page":"18","article-title":"Benchmarking Amazon EC2 for high-performance scientific computing","volume":"33","author":"Walker Edward","year":"2008","journal-title":"LOGIN"},{"key":"e_1_2_1_51_1","volume-title":"Proceedings of the 29th Conference on Information Communications (INFOCOM\u201910)","author":"Wang Guohui","year":"1833"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1109\/SKG.2011.10"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1109\/CLOUD.2015.50"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/2792980"}],"container-title":["ACM Transactions on Internet Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2885497","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2885497","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T19:05:14Z","timestamp":1750273514000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2885497"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2016,4,19]]},"references-count":54,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2016,8,22]]}},"alternative-id":["10.1145\/2885497"],"URL":"https:\/\/doi.org\/10.1145\/2885497","relation":{},"ISSN":["1533-5399","1557-6051"],"issn-type":[{"value":"1533-5399","type":"print"},{"value":"1557-6051","type":"electronic"}],"subject":[],"published":{"date-parts":[[2016,4,19]]},"assertion":[{"value":"2015-01-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2016-01-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2016-04-19","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}