{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,27]],"date-time":"2026-02-27T23:00:08Z","timestamp":1772233208674,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":54,"publisher":"ACM","license":[{"start":{"date-parts":[[2020,4,20]],"date-time":"2020-04-20T00:00:00Z","timestamp":1587340800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"German Federal Ministry of Education and Research","award":["01IS17010"],"award-info":[{"award-number":["01IS17010"]}]},{"name":"European Union's Horizon 2020 research and innovation programme","award":["825040"],"award-info":[{"award-number":["825040"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2020,4,20]]},"DOI":"10.1145\/3358960.3379124","type":"proceedings-article","created":{"date-parts":[[2020,5,4]],"date-time":"2020-05-04T07:48:26Z","timestamp":1588578506000},"page":"138-149","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":34,"title":["Microservices: A Performance Tester's Dream or Nightmare?"],"prefix":"10.1145","author":[{"given":"Simon","family":"Eismann","sequence":"first","affiliation":[{"name":"University of W\u00fcrzburg, W\u00fcrzburg, Germany"}]},{"given":"Cor-Paul","family":"Bezemer","sequence":"additional","affiliation":[{"name":"University of Alberta, Edmonton, Canada"}]},{"given":"Weiyi","family":"Shang","sequence":"additional","affiliation":[{"name":"Concordia University, Montreal, Canada"}]},{"given":"Du\u0161an","family":"Okanovi\u0107","sequence":"additional","affiliation":[{"name":"University of Stuttgart, Stuttgart, Germany"}]},{"given":"Andr\u00e9","family":"van Hoorn","sequence":"additional","affiliation":[{"name":"University of Stuttgart, Stuttgart, Germany"}]}],"member":"320","published-online":{"date-parts":[[2020,4,20]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.5555\/3101282.3101285"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"crossref","unstructured":"Tarek M. Ahmed Cor-Paul Bezemer Tse-Hsun Chen Ahmed E. Hassan and Weiyi Shang. 2016. Studying the Effectiveness of Application Performance Management (APM) Tools for Detecting Performance Regressions for Web Applications: An Experience Report. In International Conference on Mining Software Repositories (MSR). ACM 1--12.  Tarek M. Ahmed Cor-Paul Bezemer Tse-Hsun Chen Ahmed E. Hassan and Weiyi Shang. 2016. Studying the Effectiveness of Application Performance Management (APM) Tools for Detecting Performance Regressions for Web Applications: An Experience Report. In International Conference on Mining Software Repositories (MSR). ACM 1--12.","DOI":"10.1145\/2901739.2901774"},{"key":"e_1_3_2_1_3_1","volume-title":"Ann. Math. Statist.","volume":"23","author":"Anderson T. W.","year":"1952","unstructured":"T. W. Anderson and D. A. Darling . 1952. Asymptotic Theory of Certain 'Goodness of Fit' Criteria Based on Stochastic Processes . Ann. Math. Statist. , Vol. 23 , 2 (06 1952 ), 193--212. T. W. Anderson and D. A. Darling. 1952. Asymptotic Theory of Certain 'Goodness of Fit' Criteria Based on Stochastic Processes. Ann. Math. Statist., Vol. 23, 2 (06 1952), 193--212."},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-017-9553-x"},{"key":"e_1_3_2_1_5_1","volume-title":"DevOps: A Software Architect's Perspective","author":"Bass Len","unstructured":"Len Bass , Ingo Weber , and Liming Zhu . 2015. DevOps: A Software Architect's Perspective . Addison-Wesley . Len Bass, Ingo Weber, and Liming Zhu. 2015. DevOps: A Software Architect's Perspective .Addison-Wesley."},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1002\/smr.1657"},{"key":"e_1_3_2_1_7_1","first-page":"169","article-title":"The DaCapo Benchmarks","volume":"41","author":"Blackburn Stephen M.","year":"2006","unstructured":"Stephen M. Blackburn , Robin Garner , Chris Hoffmann , Asjad M. Khang , Kathryn S. McKinley , Rotem Bentzur , Amer Diwan , Daniel Feinberg , Daniel Frampton , Samuel Z. Guyer , Martin Hirzel , Antony Hosking , Maria Jump , Han Lee , J. Eliot B. Moss , Aashish Phansalkar , Darko Stefanovi\u0107 , Thomas VanDrunen , Daniel von Dincklage , and Ben Wiedermann . 2006 . The DaCapo Benchmarks : Java Benchmarking Development and Analysis. SIGPLAN Not. , Vol. 41 , 10 (2006), 169 -- 190 . Stephen M. Blackburn, Robin Garner, Chris Hoffmann, Asjad M. Khang, Kathryn S. McKinley, Rotem Bentzur, Amer Diwan, Daniel Feinberg, Daniel Frampton, Samuel Z. Guyer, Martin Hirzel, Antony Hosking, Maria Jump, Han Lee, J. Eliot B. Moss, Aashish Phansalkar, Darko Stefanovi\u0107, Thomas VanDrunen, Daniel von Dincklage, and Ben Wiedermann. 2006. The DaCapo Benchmarks: Java Benchmarking Development and Analysis. SIGPLAN Not., Vol. 41, 10 (2006), 169--190.","journal-title":"Java Benchmarking Development and Analysis. SIGPLAN Not."},{"key":"e_1_3_2_1_8_1","volume-title":"Performance Modeling, Requirements, Testing, Scalability, and Practice","author":"Bondi Andr\u00e9 B.","unstructured":"Andr\u00e9 B. Bondi . 2014. Foundations of Software and System Performance Engineering: Process , Performance Modeling, Requirements, Testing, Scalability, and Practice . Addison-Wesley Professional . Andr\u00e9 B. Bondi. 2014. Foundations of Software and System Performance Engineering: Process, Performance Modeling, Requirements, Testing, Scalability, and Practice .Addison-Wesley Professional."},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2856127"},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/583854.582443"},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-SEIP.2017.26"},{"key":"e_1_3_2_1_12_1","unstructured":"D. E. Damasceno Costa C. Bezemer P. Leitner and A. Andrzejak. 2019. What's Wrong With My Benchmark Results? Studying Bad Practices in JMH Benchmarks. IEEE Transactions on Software Engineering (2019) 1--14.  D. E. Damasceno Costa C. Bezemer P. Leitner and A. Andrzejak. 2019. What's Wrong With My Benchmark Results? Studying Bad Practices in JMH Benchmarks. IEEE Transactions on Software Engineering (2019) 1--14."},{"key":"e_1_3_2_1_13_1","unstructured":"Sandeep Dinesh. 2018. Kubernetes best practices: Resource requests and limits. https:\/\/cloud.google.com\/blog\/products\/gcp\/kubernetes-best-practices-resource-requests-and-limits .  Sandeep Dinesh. 2018. Kubernetes best practices: Resource requests and limits. https:\/\/cloud.google.com\/blog\/products\/gcp\/kubernetes-best-practices-resource-requests-and-limits ."},{"key":"e_1_3_2_1_14_1","volume-title":"Manuel Mazzara, Fabrizio Montesi, Ruslan Mustafin, and Larisa Safina.","author":"Dragoni Nicola","year":"2017","unstructured":"Nicola Dragoni , Saverio Giallorenzo , Alberto Lluch Lafuente , Manuel Mazzara, Fabrizio Montesi, Ruslan Mustafin, and Larisa Safina. 2017 . Microservices: yesterday, today, and tomorrow. In Present and Ulterior Software Engineering. Springer , Cham, 195--216. Nicola Dragoni, Saverio Giallorenzo, Alberto Lluch Lafuente, Manuel Mazzara, Fabrizio Montesi, Ruslan Mustafin, and Larisa Safina. 2017. Microservices: yesterday, today, and tomorrow. In Present and Ulterior Software Engineering. Springer, Cham, 195--216."},{"key":"e_1_3_2_1_15_1","volume-title":"Proc. IEEE International Conference on Software Architecture (ICSA 2017) Workshops. IEEE, 294--297","author":"Thomas F.","year":"2017","unstructured":"Thomas F. D\u00fc llmann, Robert Heinrich , Andr\u00e9 van Hoorn , Teerat Pitakrat , J\u00fc rgen Walter , and Felix Willnecker . 2017 . CASPA: A Platform for Comparability of Architecture-Based Software Performance Engineering Approaches . In Proc. IEEE International Conference on Software Architecture (ICSA 2017) Workshops. IEEE, 294--297 . Thomas F. D\u00fc llmann, Robert Heinrich, Andr\u00e9 van Hoorn, Teerat Pitakrat, J\u00fc rgen Walter, and Felix Willnecker. 2017. CASPA: A Platform for Comparability of Architecture-Based Software Performance Engineering Approaches. In Proc. IEEE International Conference on Software Architecture (ICSA 2017) Workshops. IEEE, 294--297."},{"key":"e_1_3_2_1_16_1","volume-title":"Microservices: A Performance Tester's Dream or Nightmare? - Replication package. https:\/\/doi.org\/10.5281\/zenodo.3582707 .","author":"Eismann Simon","year":"2019","unstructured":"Simon Eismann , Cor-Paul Bezemer , Weiyi Shang , Duvsan Okanovi\u0107 , and Andr\u00e9 van Hoorn . 2019 . Microservices: A Performance Tester's Dream or Nightmare? - Replication package. https:\/\/doi.org\/10.5281\/zenodo.3582707 . 10.5281\/zenodo.3582707 Simon Eismann, Cor-Paul Bezemer, Weiyi Shang, Duvsan Okanovi\u0107, and Andr\u00e9 van Hoorn. 2019. Microservices: A Performance Tester's Dream or Nightmare? - Replication package. https:\/\/doi.org\/10.5281\/zenodo.3582707 ."},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/MCC.2016.105"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2015.144"},{"key":"e_1_3_2_1_19_1","volume-title":"2016 IEEE International Conference on Software Testing, Verification and Validation (ICST). 22--32","author":"Gao R.","unstructured":"R. Gao , Z. M. Jiang , C. Barna , and M. Litoiu . 2016. A Framework to Evaluate the Effectiveness of Different Load Testing Analysis Techniques . In 2016 IEEE International Conference on Software Testing, Verification and Validation (ICST). 22--32 . R. Gao, Z. M. Jiang, C. Barna, and M. Litoiu. 2016. A Framework to Evaluate the Effectiveness of Different Load Testing Analysis Techniques. In 2016 IEEE International Conference on Software Testing, Verification and Validation (ICST). 22--32."},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338912"},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3053600.3053653"},{"key":"e_1_3_2_1_22_1","volume-title":"Proc. 10th International Conference on Autonomic Computing (ICAC'13)","author":"Herbst Nikolas Roman","unstructured":"Nikolas Roman Herbst , Samuel Kounev , and Ralf H. Reussner . 2013. Elasticity in Cloud Computing: What It Is, and What It Is Not . In Proc. 10th International Conference on Autonomic Computing (ICAC'13) . 23--27. Nikolas Roman Herbst, Samuel Kounev, and Ralf H. Reussner. 2013. Elasticity in Cloud Computing: What It Is, and What It Is Not. In Proc. 10th International Conference on Autonomic Computing (ICAC'13). 23--27."},{"key":"e_1_3_2_1_23_1","unstructured":"International Organization for Standardization (ISO). 2005. ISO\/IEC 25000:2005 Software Engineering - Software Product Quality Requirements and Evaluation (SQuaRE) .  International Organization for Standardization (ISO). 2005. ISO\/IEC 25000:2005 Software Engineering - Software Product Quality Requirements and Evaluation (SQuaRE) ."},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2018.2141039"},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2015.2445340"},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2851553.2892039"},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3196398.3196407"},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-019-09681-1"},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-019-09681-1"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3030207.3030213"},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/2885497"},{"key":"e_1_3_2_1_32_1","volume-title":"https:\/\/martinfowler.com\/articles\/microservices.html Retrieved","author":"Lewis James","year":"2018","unstructured":"James Lewis and Martin Fowler . 2014. Microservices. https:\/\/martinfowler.com\/articles\/microservices.html Retrieved June 3, 2018 from James Lewis and Martin Fowler. 2014. Microservices. https:\/\/martinfowler.com\/articles\/microservices.html Retrieved June 3, 2018 from"},{"key":"e_1_3_2_1_33_1","volume-title":"Measuring Computer Performance: A Practitioner's Guide","author":"Lilja David J.","unstructured":"David J. Lilja . 2005. Measuring Computer Performance: A Practitioner's Guide . Cambridge University Press . David J. Lilja. 2005. Measuring Computer Performance: A Practitioner's Guide .Cambridge University Press."},{"key":"e_1_3_2_1_34_1","volume-title":"Ordinal Analysis of Behavioral Data","author":"Long Jeffrey D.","unstructured":"Jeffrey D. Long , Du Feng , and Norman Cliff . 2003. Ordinal Analysis of Behavioral Data . John Wiley & Sons, Inc. Jeffrey D. Long, Du Feng, and Norman Cliff. 2003. Ordinal Analysis of Behavioral Data .John Wiley & Sons, Inc."},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/2635868.2635920"},{"key":"e_1_3_2_1_36_1","volume-title":"Ann. Math. Statist.","volume":"18","author":"Mann H. B.","year":"1947","unstructured":"H. B. Mann and D. R. Whitney . 1947. On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other . Ann. Math. Statist. , Vol. 18 , 1 (03 1947 ), 50--60. H. B. Mann and D. R. Whitney. 1947. On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other. Ann. Math. Statist., Vol. 18, 1 (03 1947), 50--60."},{"key":"e_1_3_2_1_37_1","volume-title":"13th $$USENIX$$ Symposium on Operating Systems Design and Implementation (OSDI 18). 409--425.","author":"Maricq Aleksander","unstructured":"Aleksander Maricq , Dmitry Duplyakin , Ivo Jimenez , Carlos Maltzahn , Ryan Stutsman , and Robert Ricci . 2018. Taming performance variability . In 13th $$USENIX$$ Symposium on Operating Systems Design and Implementation (OSDI 18). 409--425. Aleksander Maricq, Dmitry Duplyakin, Ivo Jimenez, Carlos Maltzahn, Ryan Stutsman, and Robert Ricci. 2018. Taming performance variability. In 13th $$USENIX$$ Symposium on Operating Systems Design and Implementation (OSDI 18). 409--425."},{"key":"e_1_3_2_1_38_1","volume-title":"Building Microservices","author":"Newman Sam","unstructured":"Sam Newman . 2015. Building Microservices 1 st ed.). O'Reilly Media, Inc. , Sebastopol, California, USA. Sam Newman. 2015. Building Microservices 1st ed.). O'Reilly Media, Inc., Sebastopol, California, USA.","edition":"1"},{"key":"e_1_3_2_1_39_1","unstructured":"A. V. Papadopoulos L. Versluis A. Bauer N. Herbst J. Von Kistowski A. Alieldin C. Abad J. N. Amaral P. Tima and A. Iosup. 2019. Methodological Principles for Reproducible Performance Evaluation in Cloud Computing. IEEE Transactions on Software Engineering (2019) 1--1.  A. V. Papadopoulos L. Versluis A. Bauer N. Herbst J. Von Kistowski A. Alieldin C. Abad J. N. Amaral P. Tima and A. Iosup. 2019. Methodological Principles for Reproducible Performance Evaluation in Cloud Computing. IEEE Transactions on Software Engineering (2019) 1--1."},{"key":"e_1_3_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2018.2141028"},{"key":"e_1_3_2_1_41_1","volume-title":"Annual meeting of the Southern Association for Institutional Research .","author":"Romano Jeanine","year":"2006","unstructured":"Jeanine Romano , Jeffrey D Kromrey , Jesse Coraggio , Jeff Skowronek , and Linda Devine . 2006 . Exploring methods for evaluating group differences on the NSSE and other surveys: Are the t-test and Cohen's d indices the most appropriate choices . In Annual meeting of the Southern Association for Institutional Research . Jeanine Romano, Jeffrey D Kromrey, Jesse Coraggio, Jeff Skowronek, and Linda Devine. 2006. Exploring methods for evaluating group differences on the NSSE and other surveys: Are the t-test and Cohen's d indices the most appropriate choices. In Annual meeting of the Southern Association for Institutional Research ."},{"key":"e_1_3_2_1_42_1","volume-title":"Monitoring Kubernetes performance metrics. https:\/\/www.datadoghq.com\/blog\/monitoring-kubernetes-performance-metrics\/ Retrieved","author":"Saponaro Jean-Mathieu","year":"2018","unstructured":"Jean-Mathieu Saponaro . 2016. Monitoring Kubernetes performance metrics. https:\/\/www.datadoghq.com\/blog\/monitoring-kubernetes-performance-metrics\/ Retrieved June 6, 2018 from Jean-Mathieu Saponaro. 2016. Monitoring Kubernetes performance metrics. https:\/\/www.datadoghq.com\/blog\/monitoring-kubernetes-performance-metrics\/ Retrieved June 6, 2018 from"},{"key":"e_1_3_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3185768.3186286"},{"key":"e_1_3_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/2668930.2688052"},{"key":"e_1_3_2_1_45_1","volume-title":"Proceedings of the 25th International Conference on Software Engineering (ICSE). 74--83","author":"Sim S. Elliot","unstructured":"S. Elliot Sim , Steve Easterbrook , and Richard C. Holt . 2003. Using benchmarking to advance research: a challenge to software engineering . In Proceedings of the 25th International Conference on Software Engineering (ICSE). 74--83 . S. Elliot Sim, Steve Easterbrook, and Richard C. Holt. 2003. Using benchmarking to advance research: a challenge to software engineering. In Proceedings of the 25th International Conference on Software Engineering (ICSE). 74--83."},{"key":"e_1_3_2_1_46_1","first-page":"3","article-title":"Estimate of deviation between empirical distribution functions in two independent samples","volume":"2","author":"Smirnov Nikolai V","year":"1939","unstructured":"Nikolai V Smirnov . 1939 . Estimate of deviation between empirical distribution functions in two independent samples . Bulletin Moscow University , Vol. 2 , 2 (1939), 3 -- 16 . Nikolai V Smirnov. 1939. Estimate of deviation between empirical distribution functions in two independent samples. Bulletin Moscow University, Vol. 2, 2 (1939), 3--16.","journal-title":"Bulletin Moscow University"},{"key":"e_1_3_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3030207.3030226"},{"key":"e_1_3_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/3185768.3186299"},{"key":"e_1_3_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10270-016-0566-5"},{"key":"e_1_3_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICAC.2018.00025"},{"key":"e_1_3_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1109\/MASCOTS.2018.00030"},{"key":"e_1_3_2_1_52_1","volume-title":"Individual comparisons by ranking methods. Biometrics bulletin","author":"Wilcoxon Frank","year":"1945","unstructured":"Frank Wilcoxon . 1945. Individual comparisons by ranking methods. Biometrics bulletin , Vol. 1 , 6 ( 1945 ), 80--83. Frank Wilcoxon. 1945. Individual comparisons by ranking methods. Biometrics bulletin, Vol. 1, 6 (1945), 80--83."},{"key":"e_1_3_2_1_53_1","volume-title":"Experimentation in Software Engineering","author":"Wohlin Claes","unstructured":"Claes Wohlin , Per Runeson , Martin H\u00f6st , Magnus C. Ohlsson , and Bj\u00f6 rn Regnell . 2012. Experimentation in Software Engineering . Springer . Claes Wohlin, Per Runeson, Martin H\u00f6st, Magnus C. Ohlsson, and Bj\u00f6 rn Regnell. 2012. Experimentation in Software Engineering. Springer."},{"key":"e_1_3_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/2479871.2479909"}],"event":{"name":"ICPE '20: ACM\/SPEC International Conference on Performance Engineering","location":"Edmonton AB Canada","acronym":"ICPE '20","sponsor":["SIGMETRICS ACM Special Interest Group on Measurement and Evaluation","SIGSOFT ACM Special Interest Group on Software Engineering"]},"container-title":["Proceedings of the ACM\/SPEC International Conference on Performance Engineering"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3358960.3379124","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3358960.3379124","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:13:26Z","timestamp":1750202006000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3358960.3379124"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,4,20]]},"references-count":54,"alternative-id":["10.1145\/3358960.3379124","10.1145\/3358960"],"URL":"https:\/\/doi.org\/10.1145\/3358960.3379124","relation":{},"subject":[],"published":{"date-parts":[[2020,4,20]]},"assertion":[{"value":"2020-04-20","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}