{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,25]],"date-time":"2025-12-25T12:38:28Z","timestamp":1766666308736,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":28,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,4,9]],"date-time":"2021-04-09T00:00:00Z","timestamp":1617926400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,4,9]]},"DOI":"10.1145\/3427921.3450234","type":"proceedings-article","created":{"date-parts":[[2021,4,10]],"date-time":"2021-04-10T07:37:01Z","timestamp":1618040221000},"page":"33-41","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":21,"title":["Creating a Virtuous Cycle in Performance Testing at MongoDB"],"prefix":"10.1145","author":[{"given":"David","family":"Daly","sequence":"first","affiliation":[{"name":"MongoDB Inc, New York, NY, USA"}]}],"member":"320","published-online":{"date-parts":[[2021,4,9]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"[n.d.]. 1.3.5.17. Detection of Outliers. https:\/\/www.itl.nist.gov\/div898\/handbook\/eda\/section3\/eda35h.htm  [n.d.]. 1.3.5.17. Detection of Outliers. https:\/\/www.itl.nist.gov\/div898\/handbook\/eda\/section3\/eda35h.htm"},{"key":"e_1_3_2_1_2_1","unstructured":"[n.d.]. 1.3.5.17.3. Generalized Extreme Studentized Deviate Test for Outliers. https:\/\/www.itl.nist.gov\/div898\/handbook\/eda\/section3\/eda35h3.htm  [n.d.]. 1.3.5.17.3. Generalized Extreme Studentized Deviate Test for Outliers. https:\/\/www.itl.nist.gov\/div898\/handbook\/eda\/section3\/eda35h3.htm"},{"key":"e_1_3_2_1_3_1","unstructured":"[n.d.]. Evergreen Continuous Integration: Why We Reinvented The Wheel. https:\/\/engineering.mongodb.com\/post\/evergreen-continuous-integration-why-we-reinvented-the-wheel  [n.d.]. Evergreen Continuous Integration: Why We Reinvented The Wheel. https:\/\/engineering.mongodb.com\/post\/evergreen-continuous-integration-why-we-reinvented-the-wheel"},{"key":"e_1_3_2_1_4_1","unstructured":"[n.d.]. Evergreen Ecosystem. https:\/\/github.com\/evergreen-ci\/evergreen  [n.d.]. Evergreen Ecosystem. https:\/\/github.com\/evergreen-ci\/evergreen"},{"key":"e_1_3_2_1_5_1","unstructured":"[n.d.]. Package cedar. https:\/\/godoc.org\/github.com\/evergreen-ci\/cedar  [n.d.]. Package cedar. https:\/\/godoc.org\/github.com\/evergreen-ci\/cedar"},{"key":"e_1_3_2_1_6_1","unstructured":"[n.d.]. Package poplar. https:\/\/godoc.org\/github.com\/evergreen-ci\/poplar  [n.d.]. Package poplar. https:\/\/godoc.org\/github.com\/evergreen-ci\/poplar"},{"key":"e_1_3_2_1_7_1","unstructured":"2020. Genny workload generator. https:\/\/github.com\/mongodb\/genny original-date: 2018-02--12T19:23:44Z.  2020. Genny workload generator. https:\/\/github.com\/mongodb\/genny original-date: 2018-02--12T19:23:44Z."},{"key":"e_1_3_2_1_8_1","unstructured":"2020. YCSB. https:\/\/github.com\/mongodb-labs\/YCSB original-date: 2015-03--17T18:10:30Z.  2020. YCSB. https:\/\/github.com\/mongodb-labs\/YCSB original-date: 2015-03--17T18:10:30Z."},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1807128.1807152"},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3358960.3375791"},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICST.2017.17"},{"key":"e_1_3_2_1_12_1","volume-title":"The Only Constant Is Change. In 2020 20th IEEE\/ACM International Symposium on Cluster, Cloud and Internet Computing (CCGRID). 370--379","author":"Duplyakin Dmitry","year":"2020","unstructured":"Dmitry Duplyakin , Alexandru Uta , Aleksander Maricq , and Robert Ricci . 2020 . In Datacenter Performance , The Only Constant Is Change. In 2020 20th IEEE\/ACM International Symposium on Cluster, Cloud and Internet Computing (CCGRID). 370--379 . https:\/\/doi.org\/10.1109\/CCGrid49817.2020.00--56 10.1109\/CCGrid49817.2020.00--56 Dmitry Duplyakin, Alexandru Uta, Aleksander Maricq, and Robert Ricci. 2020. In Datacenter Performance, The Only Constant Is Change. In 2020 20th IEEE\/ACM International Symposium on Cluster, Cloud and Internet Computing (CCGRID). 370--379. https:\/\/doi.org\/10.1109\/CCGrid49817.2020.00--56"},{"key":"e_1_3_2_1_13_1","volume-title":"Testing with Fewer Resources: An Adaptive Approach to Performance-Aware Test Case Generation","author":"Grano Giovanni","year":"2019","unstructured":"Giovanni Grano , Christoph Laaber , Annibale Panichella , and Sebastiano Panichella . 2019. Testing with Fewer Resources: An Adaptive Approach to Performance-Aware Test Case Generation .IEEE Transactions on Software Engineering( 2019 ), 1--1. https:\/\/doi.org\/10.1109\/TSE.2019.2946773 arXiv: 1907.08578. 10.1109\/TSE.2019.2946773 Giovanni Grano, Christoph Laaber, Annibale Panichella, and Sebastiano Panichella. 2019. Testing with Fewer Resources: An Adaptive Approach to Performance-Aware Test Case Generation.IEEE Transactions on Software Engineering(2019), 1--1. https:\/\/doi.org\/10.1109\/TSE.2019.2946773 arXiv: 1907.08578."},{"key":"e_1_3_2_1_14_1","unstructured":"Henrik Ingo and David Daly. 2019. Reducing variability in performance tests on EC2: Setup and Key Results. https:\/\/engineering.mongodb.com\/post\/reducing-variability-in-performance-tests-on-ec2-setup-and-key-results  Henrik Ingo and David Daly. 2019. Reducing variability in performance tests on EC2: Setup and Key Results. https:\/\/engineering.mongodb.com\/post\/reducing-variability-in-performance-tests-on-ec2-setup-and-key-results"},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2568225.2568232"},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3395032.3395323"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3196398.3196407"},{"key":"e_1_3_2_1_18_1","volume-title":"Software micro benchmarking in the cloud. How bad is it really? Empirical Software Engineering 24, 4(Aug","author":"Laaber Christoph","year":"2019","unstructured":"Christoph Laaber , Joel Scheuner , and Philipp Leitner . 2019. Software micro benchmarking in the cloud. How bad is it really? Empirical Software Engineering 24, 4(Aug . 2019 ), 2469--2508. https:\/\/doi.org\/10.1007\/s10664-019-09681--1 10.1007\/s10664-019-09681--1 Christoph Laaber, Joel Scheuner, and Philipp Leitner. 2019. Software micro benchmarking in the cloud. How bad is it really? Empirical Software Engineering 24, 4(Aug. 2019), 2469--2508. https:\/\/doi.org\/10.1007\/s10664-019-09681--1"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177730491"},{"key":"e_1_3_2_1_20_1","volume-title":"Taming Performance Variability. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18)","author":"Maricq Aleksander","year":"2018","unstructured":"Aleksander Maricq , Dmitry Duplyakin , Ivo Jimenez , Carlos Maltzahn , Ryan Stutsman , and Robert Ricci . 2018 . Taming Performance Variability. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18) . 409--425.https:\/\/www.usenix.org\/conference\/osdi18\/presentation\/maricq Aleksander Maricq, Dmitry Duplyakin, Ivo Jimenez, Carlos Maltzahn, Ryan Stutsman, and Robert Ricci. 2018. Taming Performance Variability. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18). 409--425.https:\/\/www.usenix.org\/conference\/osdi18\/presentation\/maricq"},{"key":"e_1_3_2_1_21_1","unstructured":"Mat Keep and Dan Pasette. 2020. Accelerating Delivery with a New Quarterly Release Cycle Starting with MongoDB 5.0 | MongoDB Blog. https:\/\/www.mongodb.com\/blog\/post\/new-quarterly-releases-starting-with-mongodb-5-0  Mat Keep and Dan Pasette. 2020. Accelerating Delivery with a New Quarterly Release Cycle Starting with MongoDB 5.0 | MongoDB Blog. https:\/\/www.mongodb.com\/blog\/post\/new-quarterly-releases-starting-with-mongodb-5-0"},{"key":"e_1_3_2_1_22_1","volume-title":"Accurate Modeling of Performance Histories for Evolving Software Systems. In 2019 34th IEEE\/ACM International Conference on Automated Software Engineering (ASE). IEEE, SanDiego, CA, USA, 640--652","author":"Muhlbauer Stefan","year":"2019","unstructured":"Stefan Muhlbauer , Sven Apel , and Norbert Siegmund . 2019 . Accurate Modeling of Performance Histories for Evolving Software Systems. In 2019 34th IEEE\/ACM International Conference on Automated Software Engineering (ASE). IEEE, SanDiego, CA, USA, 640--652 . https:\/\/doi.org\/10.1109\/ASE.2019.00065 10.1109\/ASE.2019.00065 Stefan Muhlbauer, Sven Apel, and Norbert Siegmund. 2019. Accurate Modeling of Performance Histories for Evolving Software Systems. In 2019 34th IEEE\/ACM International Conference on Automated Software Engineering (ASE). IEEE, SanDiego, CA, USA, 640--652. https:\/\/doi.org\/10.1109\/ASE.2019.00065"},{"key":"e_1_3_2_1_23_1","volume-title":"Identifying Software Performance Changes Across Variants and Versions. In 2020 35th IEEE\/ACM International Conference on Automated Software Engineering (ASE). 12","author":"M\u00fchlbauer Stefan","year":"2020","unstructured":"Stefan M\u00fchlbauer , Sven Apel , and Norbert Siegmund . 2020 . Identifying Software Performance Changes Across Variants and Versions. In 2020 35th IEEE\/ACM International Conference on Automated Software Engineering (ASE). 12 . https:\/\/doi.org\/10.1145\/3324884.3416573 10.1145\/3324884.3416573 Stefan M\u00fchlbauer, Sven Apel, and Norbert Siegmund. 2020. Identifying Software Performance Changes Across Variants and Versions. In 2020 35th IEEE\/ACM International Conference on Automated Software Engineering (ASE). 12. https:\/\/doi.org\/10.1145\/3324884.3416573"},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3358960.3379142"},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.2017.1385466"},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2897356.2897362"},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1080\/00401706.1983.10487848"},{"key":"e_1_3_2_1_28_1","volume-title":"The Generalization of 'Student's' Problem When Several Different Population Variances are Involved. Biometrika 34, 1--2 (Jan","author":"Welch B. L.","year":"1947","unstructured":"B. L. Welch . 1947. The Generalization of 'Student's' Problem When Several Different Population Variances are Involved. Biometrika 34, 1--2 (Jan . 1947 ), 28--35. https:\/\/doi.org\/10.1093\/biomet\/34.1--2.28 Publisher : Oxford Academic . 10.1093\/biomet B. L. Welch. 1947. The Generalization of 'Student's' Problem When Several Different Population Variances are Involved. Biometrika 34, 1--2 (Jan. 1947), 28--35. https:\/\/doi.org\/10.1093\/biomet\/34.1--2.28 Publisher: Oxford Academic."}],"event":{"name":"ICPE '21: ACM\/SPEC International Conference on Performance Engineering","sponsor":["SIGMETRICS ACM Special Interest Group on Measurement and Evaluation","SIGSOFT ACM Special Interest Group on Software Engineering"],"location":"Virtual Event France","acronym":"ICPE '21"},"container-title":["Proceedings of the ACM\/SPEC International Conference on Performance Engineering"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3427921.3450234","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3427921.3450234","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:02:32Z","timestamp":1750197752000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3427921.3450234"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,4,9]]},"references-count":28,"alternative-id":["10.1145\/3427921.3450234","10.1145\/3427921"],"URL":"https:\/\/doi.org\/10.1145\/3427921.3450234","relation":{},"subject":[],"published":{"date-parts":[[2021,4,9]]},"assertion":[{"value":"2021-04-09","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}