{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,8]],"date-time":"2026-10-08T08:12:42Z","timestamp":1791447162200,"version":"4.3.3"},"reference-count":66,"publisher":"Springer Science and Business Media LLC","issue":"6","license":[{"start":{"date-parts":[[2022,8,6]],"date-time":"2022-08-06T00:00:00Z","timestamp":1659744000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,8,6]],"date-time":"2022-08-06T00:00:00Z","timestamp":1659744000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100002869","name":"Christian-Albrechts-Universit\u00e4t zu Kiel","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100002869","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2022,11]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Cloud-native applications constitute a recent trend for designing large-scale software systems. However, even though several cloud-native tools and patterns have emerged to support scalability, there is no commonly accepted method to empirically benchmark their scalability. In this study, we present a benchmarking method, allowing researchers and practitioners to conduct empirical scalability evaluations of cloud-native applications, frameworks, and deployment options. Our benchmarking method consists of scalability metrics, measurement methods, and an architecture for a scalability benchmarking tool, particularly suited for cloud-native applications. Following fundamental scalability definitions and established benchmarking best practices, we propose to quantify scalability by performing isolated experiments for different load and resource combinations, which asses whether specified service level objectives (SLOs) are achieved. To balance usability and reproducibility, our benchmarking method provides configuration options, controlling the trade-off between overall execution time and statistical grounding. We perform an extensive experimental evaluation of our method\u2019s configuration options for the special case of event-driven microservices. For this purpose, we use benchmark implementations of the two stream processing frameworks Kafka Streams and Flink and run our experiments in two public clouds and one private cloud. We find that, independent of the cloud platform, it only takes a few repetitions (\u2264\u20095) and short execution times (\u2264\u20095 minutes) to assess whether SLOs are achieved. Combined with our findings from evaluating different search strategies, we conclude that our method allows to benchmark scalability in reasonable time.<\/jats:p>","DOI":"10.1007\/s10664-022-10162-1","type":"journal-article","created":{"date-parts":[[2022,8,6]],"date-time":"2022-08-06T07:14:24Z","timestamp":1659770064000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":41,"title":["A configurable method for benchmarking scalability of cloud-native applications"],"prefix":"10.1007","volume":"27","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6912-2549","authenticated-orcid":false,"given":"S\u00f6ren","family":"Henning","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wilhelm","family":"Hasselbring","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2022,8,6]]},"reference":[{"key":"10162_CR1","doi-asserted-by":"crossref","unstructured":"Abedi A, Brecht T (2017) Conducting repeatable experiments in highly variable cloud computing environments. Proceedings of the 8th ACM\/SPEC on International Conference on Performance Engineering.","DOI":"10.1145\/3030207.3030229"},{"issue":"1","key":"10162_CR2","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s13677-019-0134-y","volume":"8","author":"A Al-Said Ahmad","year":"2019","unstructured":"Al-Said Ahmad A, Andras P (2019) Scalability analysis comparisons of cloud-based software services. Journal of Cloud Computing 8(1):1\u201317. https:\/\/doi.org\/10.1186\/s13677-019-0134-y","journal-title":"Journal of Cloud Computing"},{"key":"10162_CR3","doi-asserted-by":"publisher","unstructured":"Avritzer A, Camilli M, Janes A, Russo B, Jah\u010d J, Hoorn A, Britto R, Trubiani C (2021) PPTAM\u03bb: What, where, and how of cross-domain scalability assessment. In: 2021 IEEE 18th international conference on software architecture companion (ICSA-C). https:\/\/doi.org\/10.1109\/ICSA-C52384.2021.00016https:\/\/doi.org\/10.1109\/ICSA-C52384.2021.00016","DOI":"10.1109\/ICSA-C52384.2021.00016 10.1109\/ICSA-C52384.2021.00016"},{"key":"10162_CR4","doi-asserted-by":"publisher","first-page":"110564","DOI":"10.1016\/j.jss.2020.110564","volume":"165","author":"A Avritzer","year":"2020","unstructured":"Avritzer A, Ferme V, Janes A, Russo B, van Hoorn A, Schulz H, Menasch\u00e9 D, Rufino V (2020) Scalability assessment of microservice architecture deployment configurations: A domain-based approach leveraging operational profiles and load tests. J Syst Softw 165:110564. https:\/\/doi.org\/10.1016\/j.jss.2020.110564","journal-title":"J Syst Softw"},{"key":"10162_CR5","doi-asserted-by":"publisher","unstructured":"Balalaie A, Heydarnoori A, Jamshidi P (May 2016) Microservices architecture enables DevOps: Migration to a cloud-native architecture. IEEE Software 33(3):42\u201352. https:\/\/doi.org\/10.1109\/ms.2016.64","DOI":"10.1109\/ms.2016.64"},{"key":"10162_CR6","doi-asserted-by":"publisher","unstructured":"Becker M, Lehrig S, Becker S (2015) Systematically deriving quality metrics for cloud computing systems. In: Proceedings of the 6th ACM\/SPEC International Conference on Performance Engineering. https:\/\/doi.org\/10.1145\/2668930.2688043","DOI":"10.1145\/2668930.2688043"},{"key":"10162_CR7","doi-asserted-by":"crossref","unstructured":"Bermbach D, Wittern E, Tai S (2017) Cloud Service Benchmarking: Measuring Quality of Cloud Services from a Client Perspective, 1st edn. Springer Publishing Company, Incorporated","DOI":"10.1007\/978-3-319-55483-9"},{"key":"10162_CR8","doi-asserted-by":"publisher","unstructured":"Bondi AB (2000) Characteristics of scalability and their impact on performance. In: Proceedings International Workshop on Software and Performance. https:\/\/doi.org\/10.1145\/350391.350432","DOI":"10.1145\/350391.350432"},{"key":"10162_CR9","doi-asserted-by":"publisher","unstructured":"Brataas G, Herbst N, Ivan\u0161ek S, Polutnik J (2017) Scalability analysis of cloud software services. In: Proceedings International Conference on Autonomic Computing (ICAC). https:\/\/doi.org\/10.1109\/ICAC.2017.34","DOI":"10.1109\/ICAC.2017.34"},{"key":"10162_CR10","doi-asserted-by":"publisher","first-page":"111064","DOI":"10.1016\/j.jss.2021.111064","volume":"182","author":"G Brataas","year":"2021","unstructured":"Brataas G, Martini A, Hanssen G K, R\u00e6der G (2021) Agile elicitation of scalability requirements for open systems: A case study. J Syst Softw 182:111064. https:\/\/doi.org\/10.1016\/j.jss.2021.111064, https:\/\/www.sciencedirect.com\/science\/article\/pii\/S0164121221001618","journal-title":"J Syst Softw"},{"key":"10162_CR11","doi-asserted-by":"publisher","unstructured":"Brunner S, Bl\u00f6chlinger M, Toffetti G, Spillner J, Bohnert T M (2015) Experimental evaluation of the cloud-native application design. In: 2015 IEEE\/ACM 8th international conference on utility and cloud computing (UCC). https:\/\/doi.org\/10.1109\/UCC.2015.87","DOI":"10.1109\/UCC.2015.87"},{"key":"10162_CR12","doi-asserted-by":"publisher","unstructured":"Bulej L, Hork\u00fd V, Tuma P, Farquet F, Prokopec A (2020) Duet benchmarking: Improving measurement accuracy in the cloud. In: Proceedings of the ACM\/SPEC international conference on performance engineering. https:\/\/doi.org\/10.1145\/3358960.3379132","DOI":"10.1145\/3358960.3379132"},{"issue":"5","key":"10162_CR13","doi-asserted-by":"publisher","first-page":"50","DOI":"10.1145\/2890784","volume":"59","author":"B Burns","year":"2016","unstructured":"Burns B, Grant B, Oppenheimer D, Brewer E, Wilkes J (2016) Borg, Omega, and Kubernetes. Commun. ACM 59(5):50\u201357. https:\/\/doi.org\/10.1145\/2890784","journal-title":"Commun. ACM"},{"key":"10162_CR14","unstructured":"Carbone P, Katsifodimos A, Ewen S, Markl V, Haridi S, Tzoumas K (2015) Apache Flink: Stream and batch processing in a single engine. Bulletin of the IEEE Computer Society Technical Committee on Data Engineering 36(4)"},{"key":"10162_CR15","unstructured":"Cloud Native Computing Foundation (2018) CNCF cloud native definition v1.0. https:\/\/github.com\/cncf\/toc\/blob\/main\/DEFINITION.md"},{"issue":"1","key":"10162_CR16","doi-asserted-by":"publisher","first-page":"e3825","DOI":"10.1002\/cpe.3825","volume":"29","author":"M Cunha","year":"2017","unstructured":"Cunha M, Mendon\u00e7a NC, Sampaio A (2017) Cloud Crawler: a declarative performance evaluation environment for infrastructure-as-a-service clouds. Concurrency and Computation: Practice and Experience 29 (1):e3825. https:\/\/doi.org\/10.1002\/cpe.3825","journal-title":"Concurrency and Computation: Practice and Experience"},{"issue":"3","key":"10162_CR17","doi-asserted-by":"publisher","first-page":"225","DOI":"10.1145\/356733.356735","volume":"10","author":"PJ Denning","year":"1978","unstructured":"Denning PJ, Buzen JP (1978) The operational analysis of queueing network models. ACM Comput Surv 10(3):225\u2013261. https:\/\/doi.org\/10.1145\/356733.356735","journal-title":"ACM Comput Surv"},{"key":"10162_CR18","doi-asserted-by":"publisher","unstructured":"Duboc L, Rosenblum D, Wicks T (2007) A framework for characterization and analysis of software system scalability. In: Proceedings European software engineering conference\/symposium on the foundations of software engineering. https:\/\/doi.org\/10.1145\/1287624.1287679","DOI":"10.1145\/1287624.1287679"},{"key":"10162_CR19","doi-asserted-by":"crossref","unstructured":"Folkerts E, Alexandrov A, Sachs K, Iosup A, Markl V, Tosun C (2013) Benchmarking in the cloud: What it should, can, and cannot be. In: Nambiar R, Poess M (eds) Selected topics in performance evaluation and benchmarking. Springer, Berlin, Heidelberg, pp 173\u2013188","DOI":"10.1007\/978-3-642-36727-4_12"},{"key":"10162_CR20","doi-asserted-by":"crossref","unstructured":"Fragkoulis M, Carbone P, Kalavri V, Katsifodimos A (2020) A survey on the evolution of stream processing systems. arXiv:https:\/\/arxiv.org\/abs\/2008.00842","DOI":"10.1145\/3318464.3383131"},{"issue":"5","key":"10162_CR21","doi-asserted-by":"publisher","first-page":"16","DOI":"10.1109\/MCC.2017.4250939","volume":"4","author":"D Gannon","year":"2017","unstructured":"Gannon D, Barga R, Sundaresan N (2017) Cloud-native applications. IEEE Cloud Computing 4(5):16\u201321. https:\/\/doi.org\/10.1109\/MCC.2017.4250939","journal-title":"IEEE Cloud Computing"},{"key":"10162_CR22","unstructured":"Gorton I (2022) Foundations of Scalable Systems, 1st edn. O\u2019Reilly"},{"key":"10162_CR23","volume-title":"Guerrilla capacity planning: A tactical approach to planning for highly scalable applications and services","author":"NJ Gunther","year":"2007","unstructured":"Gunther NJ (2007) Guerrilla capacity planning: A tactical approach to planning for highly scalable applications and services, 1st edn. Springer, Berlin","edition":"1st edn."},{"key":"10162_CR24","doi-asserted-by":"publisher","unstructured":"Gunther NJ, Puglia P, Tomasette K (March 2015) Hadoop superlinear scalability. Commun ACM 58(4):46\u201355. https:\/\/doi.org\/10.1145\/2719919","DOI":"10.1145\/2719919"},{"key":"10162_CR25","doi-asserted-by":"publisher","unstructured":"Hasselbring W (2021) Benchmarking as empirical standard in software engineering research. In: Proceedings of the Evaluation and Assessment in Software Engineering. https:\/\/doi.org\/10.1145\/3463274.3463361","DOI":"10.1145\/3463274.3463361"},{"issue":"8","key":"10162_CR26","doi-asserted-by":"publisher","first-page":"84","DOI":"10.1109\/MC.2020.2998235","volume":"53","author":"W Hasselbring","year":"2020","unstructured":"Hasselbring W, Carr L, Hettrick S, Packer H, Tiropanis T (2020) Open source research software. Computer 53(8):84\u201388. https:\/\/doi.org\/10.1109\/MC.2020.2998235","journal-title":"Computer"},{"key":"10162_CR27","doi-asserted-by":"publisher","unstructured":"He S, Liu T, Lama P, Lee J, Kim IK, Wang W (2021) Performance testing for cloud computing with dependent data bootstrapping. In: 2021 36th IEEE\/ACM international conference on automated software engineering (ASE). https:\/\/doi.org\/10.1109\/ASE51524.2021.9678687","DOI":"10.1109\/ASE51524.2021.9678687"},{"key":"10162_CR28","doi-asserted-by":"publisher","unstructured":"He S, Manns G, Saunders J, Wang W, Pollock L, Soffa M L (2019) A statistics-based performance testing methodology for cloud applications. In: Proceedings of the 2019 27th ACM Joint Meeting on European software Engineering Conference and Symposium on the Foundations of Software Engineering. https:\/\/doi.org\/10.1145\/3338906.3338912","DOI":"10.1145\/3338906.3338912"},{"key":"10162_CR29","doi-asserted-by":"publisher","unstructured":"Henning S, Hasselbring W (2020) Scalable and reliable multi-dimensional sensor data aggregation in data-streaming architectures. Data-Enabled Discovery and Applications 4(1). https:\/\/doi.org\/10.1007\/s41688-020-00041-3https:\/\/doi.org\/10.1007\/s41688-020-00041-3","DOI":"10.1007\/s41688-020-00041-3 10.1007\/s41688-020-00041-3"},{"key":"10162_CR30","doi-asserted-by":"publisher","unstructured":"Henning S, Hasselbring W (2021) How to measure scalability of distributed stream processing engines?. In: Companion of the ACM\/SPEC International Conference on Performance Engineering. https:\/\/doi.org\/10.1145\/3447545.3451190https:\/\/doi.org\/10.1145\/3447545.3451190","DOI":"10.1145\/3447545.3451190 10.1145\/3447545.3451190"},{"key":"10162_CR31","doi-asserted-by":"publisher","unstructured":"Henning S, Hasselbring W (2021) Replication package for: A configurable method for benchmarking scalability of cloud-native applications. https:\/\/doi.org\/10.5281\/zenodo.5596982","DOI":"10.5281\/zenodo.5596982"},{"key":"10162_CR32","doi-asserted-by":"publisher","first-page":"100209","DOI":"10.1016\/j.bdr.2021.100209","volume":"25","author":"S Henning","year":"2021","unstructured":"Henning S, Hasselbring W (2021) Theodolite: Scalability benchmarking of distributed stream processing engines in microservice architectures. Big Data Research 25:100209. https:\/\/doi.org\/10.1016\/j.bdr.2021.100209","journal-title":"Big Data Research"},{"issue":"1","key":"10162_CR33","doi-asserted-by":"publisher","first-page":"65","DOI":"10.1007\/s42488-021-00043-5","volume":"3","author":"S Henning","year":"2021","unstructured":"Henning S, Hasselbring W, Burmester H, M\u00f6bius A, Wojcieszak M (2021) Goals and measures for analyzing power consumption data in manufacturing enterprises. Journal of Data, Information and Management 3(1):65\u201382. https:\/\/doi.org\/10.1007\/s42488-021-00043-5","journal-title":"Journal of Data, Information and Management"},{"key":"10162_CR34","unstructured":"Herbst N R, Kounev S, Reussner R (2013) Elasticity in cloud computing: What it is, and what it is not. In: Proceedings International Conference on Autonomic Computing (ICAC 13)"},{"key":"10162_CR35","doi-asserted-by":"publisher","unstructured":"Hoefler T, Belli R (2015) Scientific benchmarking of parallel computing systems: Twelve ways to tell the masses when reporting performance results. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. https:\/\/doi.org\/10.1145\/2807591.2807644","DOI":"10.1145\/2807591.2807644"},{"key":"10162_CR36","doi-asserted-by":"crossref","unstructured":"Huppler K (2009) The art of building a good benchmark. In: Nambiar R, Poess M (eds) Performance evaluation and benchmarking","DOI":"10.1007\/978-3-642-10424-4_3"},{"key":"10162_CR37","unstructured":"Ibryam B, Huss R (2019) Kubernetes Patterns: Reusable Elements for Designing Cloud Native Applications, 1st edn. O\u2019Reilly"},{"key":"10162_CR38","doi-asserted-by":"publisher","unstructured":"Iosup A, Prodan R, Epema D (2014) IaaS cloud benchmarking: Approaches, challenges, and experience. In: Li X, Qiu J (eds) Cloud Computing for Data-intensive Applications. https:\/\/doi.org\/10.1007\/978-1-4939-1905-5\u2216_4","DOI":"10.1007\/978-1-4939-1905-5\u2216_4"},{"key":"10162_CR39","doi-asserted-by":"publisher","unstructured":"Iosup A, Yigitbasi N, Epema D (2011) On the performance variability of production cloud services. In: 2011 11th IEEE\/ACM international symposium on cluster, cloud and grid computing. https:\/\/doi.org\/10.1109\/CCGrid.2011.22","DOI":"10.1109\/CCGrid.2011.22"},{"key":"10162_CR40","doi-asserted-by":"publisher","unstructured":"Islam S, Lee K, Fekete A, Liu A (2012) How a consumer can measure elasticity for cloud platforms. In: Proceedings of the 3rd ACM\/SPEC international conference on performance engineering. https:\/\/doi.org\/10.1145\/2188286.2188301https:\/\/doi.org\/10.1145\/2188286.2188301","DOI":"10.1145\/2188286.2188301 10.1145\/2188286.2188301"},{"issue":"6","key":"10162_CR41","doi-asserted-by":"publisher","first-page":"589","DOI":"10.1109\/71.862209","volume":"11","author":"P Jogalekar","year":"2000","unstructured":"Jogalekar P, Woodside M (2000) Evaluating the scalability of distributed systems. IEEE Transactions on Parallel and Distributed Systems 11(6):589\u2013603. https:\/\/doi.org\/10.1109\/71.862209","journal-title":"IEEE Transactions on Parallel and Distributed Systems"},{"key":"10162_CR42","doi-asserted-by":"publisher","unstructured":"Karakaya Z, Yazici A, Alayyoub M (2017) A comparison of stream processing frameworks. In: Proceedings International conference on computer and applications (ICCA). https:\/\/doi.org\/10.1109\/COMAPP.2017.8079733https:\/\/doi.org\/10.1109\/COMAPP.2017.8079733","DOI":"10.1109\/COMAPP.2017.8079733 10.1109\/COMAPP.2017.8079733"},{"key":"10162_CR43","doi-asserted-by":"publisher","unstructured":"Karimov J, Rabl T, Katsifodimos A, Samarev R, Heiskanen H, Markl V (2018) Benchmarking distributed stream data processing systems. In: Proceedings International conference on data engineering (ICDE). https:\/\/doi.org\/10.1109\/ICDE.2018.00169","DOI":"10.1109\/ICDE.2018.00169"},{"key":"10162_CR44","unstructured":"Kleppmann M (2017) Designing Data-Intensive Applications, 1st edn. O\u2019Reilly"},{"issue":"1","key":"10162_CR45","doi-asserted-by":"publisher","first-page":"1","DOI":"10.18417\/emisa.14.1","volume":"14","author":"H Knoche","year":"2019","unstructured":"Knoche H, Hasselbring W (2019) Drivers and barriers for microservice adoption \u2013 a survey among professionals in Germany. Enterprise Modelling and Information Systems Architectures (EMISAJ) \u2013 International Journal of Conceptual Modeling 14(1):1\u201335. https:\/\/doi.org\/10.18417\/emisa.14.1","journal-title":"Enterprise Modelling and Information Systems Architectures (EMISAJ) \u2013 International Journal of Conceptual Modeling"},{"key":"10162_CR46","doi-asserted-by":"publisher","unstructured":"Kossmann D, Kraska T, Loesing S (2010) An evaluation of alternative architectures for transaction processing in the cloud. In: Proceedings SIGMOD International Conference on Management of Data. https:\/\/doi.org\/10.1145\/1807167.1807231https:\/\/doi.org\/10.1145\/1807167.1807231","DOI":"10.1145\/1807167.1807231 10.1145\/1807167.1807231"},{"key":"10162_CR47","doi-asserted-by":"crossref","unstructured":"Kounev S, Lange K-D, von Kistowski J (2020) Systems Benchmarking: For Scientists and Engineers, 1st edn. Springer Publishing Company, Incorporated","DOI":"10.1007\/978-3-030-41705-5"},{"key":"10162_CR48","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.jss.2017.01.001","volume":"126","author":"N Kratzke","year":"2017","unstructured":"Kratzke N, Quint P-C (2017) Understanding cloud-native applications after 10 years of cloud computing - a systematic mapping study. J Syst Softw 126:1\u201316. https:\/\/doi.org\/10.1016\/j.jss.2017.01.001","journal-title":"J Syst Softw"},{"key":"10162_CR49","doi-asserted-by":"publisher","unstructured":"Kuhlenkamp J, Klems M, R\u00f6ss O (August 2014) Benchmarking scalability and elasticity of distributed database systems. Proc VLDB Endow 7 (12):1219\u20131230. https:\/\/doi.org\/10.14778\/2732977.2732995","DOI":"10.14778\/2732977.2732995"},{"key":"10162_CR50","doi-asserted-by":"publisher","unstructured":"Laaber C, Scheuner J, Leitner P (August 2019) Software microbenchmarking in the cloud. how bad is it really?. Empirical Softw Engg 24(4):2469\u20132508. https:\/\/doi.org\/10.1007\/s10664-019-09681-1","DOI":"10.1007\/s10664-019-09681-1"},{"key":"10162_CR51","doi-asserted-by":"publisher","unstructured":"Lehrig S, Eikerling H, Becker S (2015) Scalability, elasticity, and efficiency in cloud computing: A systematic literature review of definitions and metrics. In: Int Conf Quality of Software Architectures. https:\/\/doi.org\/10.1145\/2737182.2737185","DOI":"10.1145\/2737182.2737185"},{"key":"10162_CR52","doi-asserted-by":"publisher","first-page":"115","DOI":"10.1016\/j.future.2017.04.018","volume":"78","author":"S Lehrig","year":"2018","unstructured":"Lehrig S, Sanders R, Brataas G, Cecowski M, Ivan\u0161ek S, Polutnik J (2018) CloudStore \u2013 towards scalability, elasticity, and efficiency benchmarking and analysis in cloud computing. Futur Gener Comput Syst 78:115\u2013126. https:\/\/doi.org\/10.1016\/j.future.2017.04.018","journal-title":"Futur Gener Comput Syst"},{"key":"10162_CR53","doi-asserted-by":"publisher","unstructured":"Leitner P, Cito J (2016) Patterns in the chaos\u2014a study of performance variation and predictability in public iaas clouds. ACM Trans Internet Technol 16(3). https:\/\/doi.org\/10.1145\/2885497","DOI":"10.1145\/2885497"},{"key":"10162_CR54","unstructured":"Maricq A, Duplyakin D, Jimenez I, Maltzahn C, Stutsman R, Ricci R (2018) Taming performance variability. In: Proceedings of the 13th USENIX Conference on Operating Systems Design and Implementation"},{"key":"10162_CR55","unstructured":"Merenstein A, Tarasov V, Anwar A, Bhagwat D, Rupprecht L, Skourtis D, Zadok E (2020) The case for benchmarking control operations in cloud native storage. In: 12th USENIX workshop on hot topics in storage and file systems (HotStorage 20)"},{"key":"10162_CR56","doi-asserted-by":"publisher","unstructured":"Michael M, Moreira JE, Shiloach D, Wisniewski RW (2007) Scale-up x scale-out: A case study using nutch\/lucene. In: 2007 IEEE international parallel and distributed processing symposium. https:\/\/doi.org\/10.1109\/IPDPS.2007.370631","DOI":"10.1109\/IPDPS.2007.370631"},{"key":"10162_CR57","doi-asserted-by":"publisher","unstructured":"Nasiri H, Nasehi S, Goudarzi M (2019) Evaluation of distributed stream processing frameworks for iot applications in smart cities. Journal of Big Data 6(52). https:\/\/doi.org\/10.1186\/s40537-019-0215-2","DOI":"10.1186\/s40537-019-0215-2"},{"issue":"8","key":"10162_CR58","doi-asserted-by":"publisher","first-page":"1528","DOI":"10.1109\/TSE.2019.2927908","volume":"47","author":"AV Papadopoulos","year":"2021","unstructured":"Papadopoulos AV, Versluis L, Bauer A, Herbst N, Kistowski J, Ali-Eldin A, Abad CL, Amaral JN, Tuma P, Iosup A (2021) Methodological principles for reproducible performance evaluation in cloud computing. IEEE Trans Softw Eng 47(8):1528\u20131543. https:\/\/doi.org\/10.1109\/TSE.2019.2927908","journal-title":"IEEE Trans Softw Eng"},{"key":"10162_CR59","unstructured":"Ralph P, bin Ali N, Baltes S, Bianculli D, Diaz J, Dittrich Y, Ernst N, Felderer M, Feldt R, Filieri A, de Fran\u00e7a BBN, Furia C A, Gay G, Gold N, Graziotin D, He P, Hoda R, Juristo N, Kitchenham B, Lenarduzzi V, Mart\u00ednez J, Melegati J, Mendez D, Menzies T, Molleri J, Pfahl D, Robbes R, Russo D, Saarim\u00e4ki N, Sarro F, Taibi D, Siegmund J, Spinellis D, Staron M, Stol K, Storey M-A, Taibi D, Tamburri D, Torchiano M, Treude C, Turhan B, Wang X, Vegas S (2021) Empirical standards for software engineering research. arXiv:2010.03525, Version 0.2.0"},{"key":"10162_CR60","doi-asserted-by":"publisher","unstructured":"Sim SE, Easterbrook S, Holt R C (2003) Using benchmarking to advance research: A challenge to software engineering. In: 25th international conference on software engineering. https:\/\/doi.org\/10.1109\/icse.2003.1201189https:\/\/doi.org\/10.1109\/icse.2003.1201189. IEEE","DOI":"10.1109\/icse.2003.1201189 10.1109\/icse.2003.1201189"},{"key":"10162_CR61","doi-asserted-by":"publisher","first-page":"215","DOI":"10.1016\/j.jss.2018.09.082","volume":"146","author":"J Soldani","year":"2018","unstructured":"Soldani J, Tamburri DA, Van Den Heuvel W-J (2018) The pains and gains of microservices: A systematic grey literature review. J Syst Softw 146:215\u2013232. https:\/\/doi.org\/10.1016\/j.jss.2018.09.082","journal-title":"J Syst Softw"},{"key":"10162_CR62","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/2590528.2590529","volume":"2014","author":"WF Tichy","year":"2014","unstructured":"Tichy WF (2014) Where\u2019s the science in software engineering? ubiquity symposium: The science in computer science. Ubiquity 2014:1\u20136. https:\/\/doi.org\/10.1145\/2590528.2590529","journal-title":"Ubiquity"},{"key":"10162_CR63","doi-asserted-by":"publisher","unstructured":"Tsai W-T, Huang Y, Shao Q (2011) Testing the scalability of SaaS applications. In: 2011 IEEE international conference on service-oriented computing and applications (SOCA). https:\/\/doi.org\/10.1109\/SOCA.2011.6166245https:\/\/doi.org\/10.1109\/SOCA.2011.6166245","DOI":"10.1109\/SOCA.2011.6166245 10.1109\/SOCA.2011.6166245"},{"key":"10162_CR64","doi-asserted-by":"publisher","unstructured":"v Kistowski J, Arnold JA, Huppler K, Lange K-D, Henning JL, Cao P (2015) How to build a benchmark. In: Proceedings ACM\/SPEC international conference on performance engineering. https:\/\/doi.org\/10.1145\/2668930.2688819https:\/\/doi.org\/10.1145\/2668930.2688819","DOI":"10.1145\/2668930.2688819 10.1145\/2668930.2688819"},{"key":"10162_CR65","doi-asserted-by":"publisher","unstructured":"Wang G, Chen L, Dikshit A, Gustafson J, Chen B, Sax MJ, Roesler J, Blee-Goldman S, Cadonna B, Mehta A, Madan V, Rao J (2021) Consistency and completeness: Rethinking distributed stream processing in apache kafka. In: Proceedings of the 2021 International Conference on Management of Data. https:\/\/doi.org\/10.1145\/3448016.3457556","DOI":"10.1145\/3448016.3457556"},{"key":"10162_CR66","doi-asserted-by":"publisher","unstructured":"Weber A, Herbst N, Groenda H, Kounev S (2014) Towards a resource elasticity benchmark for cloud environments. In: Proceedings International workshop on hot topics in cloud service scalability. HotTopiCS \u201914. https:\/\/doi.org\/10.1145\/2649563.2649571","DOI":"10.1145\/2649563.2649571"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-022-10162-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10664-022-10162-1\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-022-10162-1.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,9,26]],"date-time":"2022-09-26T08:25:59Z","timestamp":1664180759000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10664-022-10162-1"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,8,6]]},"references-count":66,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2022,11]]}},"alternative-id":["10162"],"URL":"https:\/\/doi.org\/10.1007\/s10664-022-10162-1","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,8,6]]},"assertion":[{"value":"30 March 2022","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"6 August 2022","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"<!--Emphasis Type='Bold' removed-->Competing Interests"}}],"article-number":"143"}}