{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,27]],"date-time":"2026-10-27T20:03:23Z","timestamp":1793131403522,"version":"4.5.0"},"reference-count":53,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2022,11,28]],"date-time":"2022-11-28T00:00:00Z","timestamp":1669593600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,11,28]],"date-time":"2022-11-28T00:00:00Z","timestamp":1669593600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Empir Software Eng"],"published-print":{"date-parts":[[2023,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Microbenchmarking is a widely used form of performance testing in Java software. A microbenchmark repeatedly executes a small chunk of code while collecting measurements related to its performance. Due to Java Virtual Machine optimizations, microbenchmarks are usually subject to severe performance fluctuations in the first phase of their execution (also known as warmup). For this reason, software developers typically discard measurements of this phase and focus their analysis when benchmarks reach a steady state of performance. Developers estimate the end of the warmup phase based on their expertise, and configure their benchmarks accordingly. Unfortunately, this approach is based on two strong assumptions: (i) benchmarks always reach a steady state of performance and (ii) developers accurately estimate warmup. In this paper, we show that Java microbenchmarks do not always reach a steady state, and often developers fail to accurately estimate the end of the warmup phase. We found that a considerable portion of studied benchmarks do not hit the steady state, and warmup estimates provided by software developers are often inaccurate (with a large error). This has significant implications both in terms of results quality and time-effort. Furthermore, we found that dynamic reconfiguration significantly improves warmup estimation accuracy, but still it induces suboptimal warmup estimates and relevant side-effects. We envision this paper as a starting point for supporting the introduction of more sophisticated automated techniques that can ensure results quality in a timely fashion.<\/jats:p>","DOI":"10.1007\/s10664-022-10247-x","type":"journal-article","created":{"date-parts":[[2022,11,28]],"date-time":"2022-11-28T04:08:54Z","timestamp":1669608534000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":39,"title":["Towards effective assessment of steady state performance in Java software: are we there yet?"],"prefix":"10.1007","volume":"28","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3676-0645","authenticated-orcid":false,"given":"Luca","family":"Traini","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4507-464X","authenticated-orcid":false,"given":"Vittorio","family":"Cortellessa","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2041-7375","authenticated-orcid":false,"given":"Daniele","family":"Di Pompeo","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0329-1101","authenticated-orcid":false,"given":"Michele","family":"Tucci","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2022,11,28]]},"reference":[{"key":"10247_CR1","doi-asserted-by":"publisher","unstructured":"AlGhamdi H M, Bezemer C P, Shang W, Hassan A E, Flora P (2020) Towards reducing the time needed for load testing. J Softw: Evol Process e2276. https:\/\/doi.org\/10.1002\/smr.2276. https:\/\/onlinelibrary.wiley.com\/doi\/abs\/10.1002\/smr.2276, smr.2276","DOI":"10.1002\/smr.2276"},{"issue":"2","key":"10247_CR2","doi-asserted-by":"publisher","first-page":"291","DOI":"10.1016\/S0378-3758(96)00138-3","volume":"60","author":"J Antoch","year":"1997","unstructured":"Antoch J, Hu\u0161kova M, Pr\u00e1\u0161kov\u00e1 Z (1997) Effect of dependence on statistics for determination of change. J Stat Plan Inference 60(2):291\u2013310. https:\/\/doi.org\/10.1016\/S0378-3758(96)00138-3. https:\/\/www.sciencedirect.com\/science\/article\/pii\/S0378375896001383","journal-title":"J Stat Plan Inference"},{"key":"10247_CR3","unstructured":"Bagley D, Fulgham B, Gouy I (2004) The computer language benchmarks game. https:\/\/benchmarksgame-team.pages.debian.net\/benchmarksgame. Accessed: 2021-10-12"},{"key":"10247_CR4","doi-asserted-by":"publisher","unstructured":"Barrett E, Bolz-Tereick C F, Killick R, Mount S, Tratt L (2017) Virtual machine warmup blows hot and cold. Proc ACM Program Lang 1(OOPSLA). https:\/\/doi.org\/10.1145\/3133876","DOI":"10.1145\/3133876"},{"key":"10247_CR5","doi-asserted-by":"publisher","unstructured":"Beller M, Gousios G, Zaidman A (2017) Oops, my tests broke the build: an explorative analysis of travis ci with github. In: 2017 IEEE\/ACM 14th international conference on mining software repositories (MSR). https:\/\/doi.org\/10.1109\/MSR.2017.62, pp 356\u2013367","DOI":"10.1109\/MSR.2017.62"},{"issue":"P3","key":"10247_CR6","doi-asserted-by":"publisher","first-page":"408","DOI":"10.1016\/j.scico.2013.02.001","volume":"98","author":"CF Bolz","year":"2015","unstructured":"Bolz C F, Tratt L (2015) The impact of meta-tracing on vm design and implementation. Sci Comput Program 98(P3):408\u2013421. https:\/\/doi.org\/10.1016\/j.scico.2013.02.001","journal-title":"Sci Comput Program"},{"issue":"1","key":"10247_CR7","doi-asserted-by":"publisher","first-page":"139","DOI":"10.1007\/s10515-015-0188-0","volume":"24","author":"L Bulej","year":"2017","unstructured":"Bulej L, Bures T, Hork\u00fd V, Kotrc J, Marek L, Troj\u00e1nek T, Tuma P (2017) Unit testing performance with stochastic performance logic. Autom Softw Eng 24(1):139\u2013187. https:\/\/doi.org\/10.1007\/s10515-015-0188-0","journal-title":"Autom Softw Eng"},{"key":"10247_CR8","doi-asserted-by":"publisher","unstructured":"Chen J, Shang W (2017) An exploratory study of performance regression introducing code changes. In: 2017 IEEE International conference on software maintenance and evolution, ICSME 2017, Shanghai, China, September 17\u201322, 2017. https:\/\/doi.org\/10.1109\/ICSME.2017.13. IEEE Computer Society, pp 341\u2013352","DOI":"10.1109\/ICSME.2017.13"},{"key":"10247_CR9","doi-asserted-by":"crossref","unstructured":"Cohen J (2013) Statistical power analysis for the behavioral sciences. Taylor & Francis","DOI":"10.4324\/9780203771587"},{"key":"10247_CR10","doi-asserted-by":"publisher","first-page":"111084","DOI":"10.1016\/j.jss.2021.111084","volume":"183","author":"V Cortellessa","year":"2022","unstructured":"Cortellessa V, Di Pompeo D, Eramo R, Tucci M (2022) A model-driven approach for continuous performance engineering in microservice-based systems. J Syst Softw 183:111084. https:\/\/doi.org\/10.1016\/j.jss.2021.111084. https:\/\/www.sciencedirect.com\/science\/article\/pii\/S0164121221001813","journal-title":"J Syst Softw"},{"issue":"7","key":"10247_CR11","doi-asserted-by":"publisher","first-page":"1452","DOI":"10.1109\/TSE.2019.2925345","volume":"47","author":"D Costa","year":"2021","unstructured":"Costa D, Bezemer C P, Leitner P, Andrzejak A (2021) What\u2019s wrong with my benchmark results? Studying bad practices in jmh benchmarks. IEEE Trans Softw Eng 47(7):1452\u20131467. https:\/\/doi.org\/10.1109\/TSE.2019.2925345","journal-title":"IEEE Trans Softw Eng"},{"key":"10247_CR12","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511802843","volume-title":"Bootstrap methods and their application. Cambridge Series in Statistical and Probabilistic Mathematics","author":"AC Davison","year":"1997","unstructured":"Davison A C, Hinkley D V (1997) Bootstrap methods and their application. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, Cambridge. https:\/\/doi.org\/10.1017\/CBO9780511802843"},{"key":"10247_CR13","doi-asserted-by":"publisher","unstructured":"Ding Z, Chen J, Shang W (2020) Towards the use of the readily available tests from the release pipeline as performance tests: are we there yet? In: Rothermel G, Bae D (eds) ICSE \u201920: 42nd international conference on software engineering, Seoul, South Korea, 27 June\u201319 July, 2020. https:\/\/doi.org\/10.1145\/3377811.3380351. ACM, pp 1435\u20131446","DOI":"10.1145\/3377811.3380351"},{"key":"10247_CR14","doi-asserted-by":"publisher","first-page":"205","DOI":"10.1017\/CBO9780511984679.011","volume-title":"Analysis of changepoint models","author":"IA Eckley","year":"2011","unstructured":"Eckley I A, Fearnhead P, Killick R (2011) Analysis of changepoint models. Cambridge University Press, Cambridge, pp 205\u2013224. https:\/\/doi.org\/10.1017\/CBO9780511984679.011"},{"issue":"525","key":"10247_CR15","doi-asserted-by":"publisher","first-page":"169","DOI":"10.1080\/01621459.2017.1385466","volume":"114","author":"P Fearnhead","year":"2019","unstructured":"Fearnhead P, Rigaill G (2019) Changepoint detection in the presence of outliers. J Am Stat Assoc 114(525):169\u2013183","journal-title":"J Am Stat Assoc"},{"issue":"2","key":"10247_CR16","doi-asserted-by":"crossref","first-page":"175","DOI":"10.1111\/j.2517-6161.1954.tb00159.x","volume":"16","author":"EC Fieller","year":"1954","unstructured":"Fieller E C (1954) Some problems in interval estimation. J R Stat Soc B: Stat (Methodol) 16(2):175\u2013185. http:\/\/www.jstor.org\/stable\/2984043","journal-title":"J R Stat Soc B: Stat (Methodol)"},{"key":"10247_CR17","unstructured":"Fowler M (2006) Continuous integration. https:\/\/www.martinfowler.com\/articles\/continuousIntegration.html. Accessed: 25 Jan 2022"},{"key":"10247_CR18","doi-asserted-by":"publisher","unstructured":"Georges A, Buytaert D, Eeckhout L (2007) Statistically rigorous java performance evaluation. In: Proceedings of the 22nd annual ACM SIGPLAN conference on object-oriented programming systems, languages and applications, OOPSLA \u201907. https:\/\/doi.org\/10.1145\/1297027.1297033. Association for Computing Machinery, New York, pp 57\u201376","DOI":"10.1145\/1297027.1297033"},{"key":"10247_CR19","doi-asserted-by":"publisher","unstructured":"Giese H, Lambers L, Z\u00f6llner C (2020) From classic to agile: experiences from more than a decade of project-based modeling education. In: Guerra E, Iovino L (eds) MODELS \u201920: ACM\/IEEE 23rd international conference on model driven engineering languages and systems, virtual event, Canada, 18\u201323 October, 2020, companion proceedings. https:\/\/doi.org\/10.1145\/3417990.3418743. ACM, pp 22:1\u201322:10","DOI":"10.1145\/3417990.3418743"},{"key":"10247_CR20","unstructured":"Haynes K, Eckley I A, Fearnhead P (2014) Efficient penalty search for multiple changepoint problems. 1412.3617"},{"key":"10247_CR21","doi-asserted-by":"publisher","unstructured":"He S, Manns G, Saunders J, Wang W, Pollock L, Soffa M L (2019) A statistics-based performance testing methodology for cloud applications. In: Proceedings of the 2019 27th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, ESEC\/FSE 2019. https:\/\/doi.org\/10.1145\/3338906.3338912. Association for Computing Machinery, New York, pp 188\u2013199","DOI":"10.1145\/3338906.3338912"},{"issue":"11","key":"10247_CR22","doi-asserted-by":"publisher","first-page":"1091","DOI":"10.1109\/TSE.2015.2445340","volume":"41","author":"ZM Jiang","year":"2015","unstructured":"Jiang Z M, Hassan A E (2015) A survey on load testing of large-scale software systems. IEEE Trans Softw Eng 41(11):1091\u20131118. https:\/\/doi.org\/10.1109\/TSE.2015.2445340","journal-title":"IEEE Trans Softw Eng"},{"key":"10247_CR23","doi-asserted-by":"publisher","unstructured":"Kalibera T, Jones R (2013) Rigorous benchmarking in reasonable time. In: Proceedings of the 2013 international symposium on memory management, ISMM \u201913, pp 63\u201374. https:\/\/doi.org\/10.1145\/2491894.2464160. Association for Computing Machinery, New York","DOI":"10.1145\/2491894.2464160"},{"key":"10247_CR24","unstructured":"Kalibera T, Jones R (2020) Quantifying performance changes with effect size confidence intervals. 2007.10899"},{"issue":"500","key":"10247_CR25","doi-asserted-by":"publisher","first-page":"1590","DOI":"10.1080\/01621459.2012.737745","volume":"107","author":"R Killick","year":"2012","unstructured":"Killick R, Fearnhead P, Eckley I A (2012) Optimal detection of changepoints with a linear computational cost. J Am Stat Assoc 107(500):1590\u20131598. https:\/\/doi.org\/10.1080\/01621459.2012.737745","journal-title":"J Am Stat Assoc"},{"issue":"1","key":"10247_CR26","doi-asserted-by":"publisher","first-page":"79","DOI":"10.1214\/aoms\/1177729694","volume":"22","author":"S Kullback","year":"1951","unstructured":"Kullback S, Leibler R A (1951) On information and sufficiency. Ann Math Stat 22(1):79\u201386","journal-title":"Ann Math Stat"},{"key":"10247_CR27","doi-asserted-by":"publisher","unstructured":"Laaber C, Leitner P (2018) An evaluation of open-source software microbenchmark suites for continuous performance assessment. In: Proceedings of the 15th international conference on mining software repositories, MSR \u201918. https:\/\/doi.org\/10.1145\/3196398.3196407. Association for Computing Machinery, New York, pp 119\u2013130","DOI":"10.1145\/3196398.3196407"},{"issue":"4","key":"10247_CR28","doi-asserted-by":"publisher","first-page":"2469","DOI":"10.1007\/s10664-019-09681-1","volume":"24","author":"C Laaber","year":"2019","unstructured":"Laaber C, Scheuner J, Leitner P (2019) Software microbenchmarking in the cloud. how bad is it really? Empir Softw Eng 24(4):2469\u20132508. https:\/\/doi.org\/10.1007\/s10664-019-09681-1","journal-title":"Empir Softw Eng"},{"key":"10247_CR29","doi-asserted-by":"publisher","unstructured":"Laaber C, W\u00fcrsten S, Gall H C, Leitner P (2020) Dynamically reconfiguring software microbenchmarks: reducing execution time without sacrificing result quality. In: Proceedings of the 28th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, ESEC\/FSE 2020. https:\/\/doi.org\/10.1145\/3368089.3409683. Association for Computing Machinery, New York, pp 989\u20131001","DOI":"10.1145\/3368089.3409683"},{"issue":"6","key":"10247_CR30","doi-asserted-by":"publisher","first-page":"133","DOI":"10.1007\/s10664-021-10037-x","volume":"26","author":"C Laaber","year":"2021","unstructured":"Laaber C, Gall H C, Leitner P (2021) Applying test case prioritization to software microbenchmarks. Empir Softw Eng 26(6):133. https:\/\/doi.org\/10.1007\/s10664-021-10037-x","journal-title":"Empir Softw Eng"},{"issue":"8","key":"10247_CR31","doi-asserted-by":"publisher","first-page":"1501","DOI":"10.1016\/j.sigpro.2005.01.012","volume":"85","author":"M Lavielle","year":"2005","unstructured":"Lavielle M (2005) Using penalized contrasts for the change-point problem. Signal Process 85(8):1501\u20131510. https:\/\/doi.org\/10.1016\/j.sigpro.2005.01.012","journal-title":"Signal Process"},{"key":"10247_CR32","doi-asserted-by":"publisher","unstructured":"Leitner P, Bezemer C P (2017) An exploratory study of the state of practice of performance testing in java-based open source projects. In: Proceedings of the 8th ACM\/SPEC on international conference on performance engineering, ICPE \u201917. https:\/\/doi.org\/10.1145\/3030207.3030213. Association for Computing Machinery, New York, pp 373\u2013384","DOI":"10.1145\/3030207.3030213"},{"key":"10247_CR33","unstructured":"Maricq A, Duplyakin D, Jimenez I, Maltzahn C, Stutsman R, Ricci R (2018) Taming performance variability. In: 13th USENIX symposium on operating systems design and implementation (OSDI 18). https:\/\/www.usenix.org\/conference\/osdi18\/presentation\/maricq. USENIX Association, Carlsbad, pp 409\u2013425"},{"key":"10247_CR34","doi-asserted-by":"publisher","unstructured":"Mostafa S, Wang X, Xie T (2017) Perfranker: prioritization of performance regression tests for collection-intensive software. In: Bultan T, Sen K (eds) Proceedings of the 26th ACM SIGSOFT international symposium on software testing and analysis, Santa Barbara, CA, USA, July 10\u201314, 2017. https:\/\/doi.org\/10.1145\/3092703.3092725. ACM, pp 23\u201334","DOI":"10.1145\/3092703.3092725"},{"key":"10247_CR35","doi-asserted-by":"publisher","unstructured":"Mytkowicz T, Diwan A, Hauswirth M, Sweeney P F (2009a) Producing wrong data without doing anything obviously wrong! In: Soffa ML, Irwin MJ (eds) Proceedings of the 14th international conference on architectural support for programming languages and operating systems, ASPLOS 2009, Washington, DC, USA, March 7\u201311, 2009. https:\/\/doi.org\/10.1145\/1508244.1508275. ACM, pp 265\u2013276","DOI":"10.1145\/1508244.1508275"},{"key":"10247_CR36","doi-asserted-by":"publisher","unstructured":"Mytkowicz T, Diwan A, Hauswirth M, Sweeney P F (2009b) Producing wrong data without doing anything obviously wrong!. In: Soffa ML, Irwin MJ (eds) Proceedings of the 14th international conference on architectural support for programming languages and operating systems, ASPLOS 2009, Washington, DC, USA, March 7\u201311, 2009. https:\/\/doi.org\/10.1145\/1508244.1508275. ACM, pp 265\u2013276","DOI":"10.1145\/1508244.1508275"},{"key":"10247_CR37","doi-asserted-by":"crossref","unstructured":"Neumann G, Harman M, Poulding S Barros M, Labiche Y (eds) (2015) Transformed vargha-delaney effect size. Springer International Publishing, Cham","DOI":"10.1007\/978-3-319-22183-0_29"},{"key":"10247_CR38","unstructured":"Oaks S (2014) Java performance\u2014the definitive guide: getting the most out of your code. O\u2019Reilly. http:\/\/shop.oreilly.com\/product\/0636920028499.do"},{"issue":"8","key":"10247_CR39","doi-asserted-by":"publisher","first-page":"1528","DOI":"10.1109\/TSE.2019.2927908","volume":"47","author":"AV Papadopoulos","year":"2021","unstructured":"Papadopoulos A V, Versluis L, Bauer A, Herbst N, von Kistowski J, Ali-Eldin A, Abad C L, Amaral J N, Tuma P, Iosup A (2021) Methodological principles for reproducible performance evaluation in cloud computing. IEEE Trans Softw Eng 47(8):1528\u20131543. https:\/\/doi.org\/10.1109\/TSE.2019.2927908","journal-title":"IEEE Trans Softw Eng"},{"key":"10247_CR40","unstructured":"Ratanaworabhan P, Livshits B, Simmons D, Ba Zorn (2009) Jsmeter: characterizing real-world behavior of javascript programs. Tech. Rep. MSR-TR-2009-173. https:\/\/www.microsoft.com\/en-us\/research\/publication\/jsmeter-characterizing-real-world-behavior-of-javascript-programs\/"},{"key":"10247_CR41","doi-asserted-by":"publisher","unstructured":"Rausch T, Hummer W, Leitner P, Schulte S (2017) An empirical analysis of build failures in the continuous integration workflows of java-based open-source software. In: 2017 IEEE\/ACM 14th international conference on mining software repositories (MSR). https:\/\/doi.org\/10.1109\/MSR.2017.54, pp 345\u2013355","DOI":"10.1109\/MSR.2017.54"},{"key":"10247_CR42","doi-asserted-by":"publisher","unstructured":"Reichelt D G, K\u00fchne S, Hasselbring W (2019) Peass: a tool for identifying performance changes at code level. In: 34th IEEE\/ACM international conference on automated software engineering, ASE 2019, San Diego, CA, USA, November 11\u201315, 2019. https:\/\/doi.org\/10.1109\/ASE.2019.00123. IEEE, pp 1146\u20131149","DOI":"10.1109\/ASE.2019.00123"},{"key":"10247_CR43","doi-asserted-by":"publisher","unstructured":"Rubin J, Rinard M (2016) The challenges of staying together while moving fast: an exploratory study. In: Proceedings of the 38th international conference on software engineering, ICSE \u201916. https:\/\/doi.org\/10.1145\/2884781.2884871. Association for Computing Machinery, New York, pp 982\u2013993","DOI":"10.1145\/2884781.2884871"},{"key":"10247_CR44","doi-asserted-by":"publisher","unstructured":"Samoaa H, Leitner P (2021) An exploratory study of the impact of parameterization on jmh measurement results in open-source projects. In: Proceedings of the ACM\/SPEC international conference on performance engineering, ICPE \u201921. https:\/\/doi.org\/10.1145\/3427921.3450243. Association for Computing Machinery, New York, pp 213\u2013224","DOI":"10.1145\/3427921.3450243"},{"key":"10247_CR45","doi-asserted-by":"publisher","unstructured":"Sarro F, Petrozziello A, Harman M (2016) Multi-objective software effort estimation. In: Proceedings of the 38th international conference on software engineering, ICSE \u201916. https:\/\/doi.org\/10.1145\/2884781.2884830. Association for Computing Machinery, New York, pp 619\u2013630","DOI":"10.1145\/2884781.2884830"},{"key":"10247_CR46","doi-asserted-by":"publisher","unstructured":"Satopaa V, Albrecht J R, Irwin D E, Raghavan B (2011) Finding a \u201ckneedle\u201d in a haystack: detecting knee points in system behavior. In: 31st IEEE international conference on distributed computing systems workshops (ICDCS 2011 workshops), 20\u201324 June 2011, Minneapolis, Minnesota, USA. https:\/\/doi.org\/10.1109\/ICDCSW.2011.20. IEEE Computer Society, pp 166\u2013171","DOI":"10.1109\/ICDCSW.2011.20"},{"key":"10247_CR47","doi-asserted-by":"publisher","unstructured":"Stefan P, Hork\u00fd V, Bulej L, Tuma P (2017) Unit testing performance in java projects: are we there yet? In: Binder W, Cortellessa V, Koziolek A, Smirni E, Poess M (eds) Proceedings of the 8th ACM\/SPEC on international conference on performance engineering, ICPE 2017, L\u2019Aquila, Italy, April 22\u201326, 2017. https:\/\/doi.org\/10.1145\/3030207.3030226. ACM, pp 401\u2013412","DOI":"10.1145\/3030207.3030226"},{"key":"10247_CR48","unstructured":"Suchanek M, Navratil M, Bailey L, Boyle C (2017) Performance tuning guide (red hat enterprise Linux 7). https:\/\/access.redhat.com\/documentation\/en-us\/red_hat_enterprise_linux\/7\/html\/performance_tuning_guide\/, (Online; accessed 28 June 2021)"},{"issue":"3","key":"10247_CR49","doi-asserted-by":"publisher","first-page":"74","DOI":"10.1007\/s10664-021-10069-3","volume":"27","author":"L Traini","year":"2022","unstructured":"Traini L (2022) Exploring performance assurance practices and challenges in agile software development: an ethnographic study. Empir Softw Eng 27(3):74. https:\/\/doi.org\/10.1007\/s10664-021-10069-3","journal-title":"Empir Softw Eng"},{"key":"10247_CR50","doi-asserted-by":"publisher","unstructured":"Traini L, Di Pompeo D, Tucci M, Lin B, Scalabrino S, Bavota G, Lanza M, Oliveto R, Cortellessa V (2021) How software refactoring impacts execution time. ACM Trans Softw Eng Methodol 31(2). https:\/\/doi.org\/10.1145\/3485136","DOI":"10.1145\/3485136"},{"key":"10247_CR51","unstructured":"Tukey J W et al (1977) Exploratory data analysis, vol 2. Reading"},{"issue":"2","key":"10247_CR52","first-page":"101","volume":"25","author":"A Vargha","year":"2000","unstructured":"Vargha A, Delaney H D (2000) A critique and improvement of the \u201ccl\u201d common language effect size statistics of Mcgraw and Wong. J Educ Behav Stat 25 (2):101\u2013132. http:\/\/www.jstor.org\/stable\/1165329","journal-title":"J Educ Behav Stat"},{"key":"10247_CR53","doi-asserted-by":"publisher","unstructured":"Vassallo C, Schermann G, Zampetti F, Romano D, Leitner P, Zaidman A, Di Penta M, Panichella S (2017) A tale of ci build failures: an open source and a financial organization perspective. In: 2017 IEEE International conference on software maintenance and evolution (ICSME). https:\/\/doi.org\/10.1109\/ICSME.2017.67, pp 183\u2013193","DOI":"10.1109\/ICSME.2017.67"}],"container-title":["Empirical Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-022-10247-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10664-022-10247-x\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10664-022-10247-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,9]],"date-time":"2024-10-09T12:23:59Z","timestamp":1728476639000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10664-022-10247-x"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,11,28]]},"references-count":53,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2023,1]]}},"alternative-id":["10247"],"URL":"https:\/\/doi.org\/10.1007\/s10664-022-10247-x","relation":{},"ISSN":["1382-3256","1573-7616"],"issn-type":[{"value":"1382-3256","type":"print"},{"value":"1573-7616","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,11,28]]},"assertion":[{"value":"30 September 2022","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"28 November 2022","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}],"article-number":"13"}}