{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,24]],"date-time":"2026-04-24T02:06:40Z","timestamp":1776996400688,"version":"3.51.4"},"publisher-location":"New York, NY, USA","reference-count":19,"publisher":"ACM","license":[{"start":{"date-parts":[[2020,4,20]],"date-time":"2020-04-20T00:00:00Z","timestamp":1587340800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2020,4,20]]},"DOI":"10.1145\/3358960.3375791","type":"proceedings-article","created":{"date-parts":[[2020,5,4]],"date-time":"2020-05-04T07:48:26Z","timestamp":1588578506000},"page":"67-75","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":39,"title":["The Use of Change Point Detection to Identify Software Performance Regressions in a Continuous Integration System"],"prefix":"10.1145","author":[{"given":"David","family":"Daly","sequence":"first","affiliation":[{"name":"MongoDB Inc, New York, NY, USA"}]},{"given":"William","family":"Brown","sequence":"additional","affiliation":[{"name":"Columbia University & MongoDB Inc, New York, NY, USA"}]},{"given":"Henrik","family":"Ingo","sequence":"additional","affiliation":[{"name":"MongoDB Inc, New York, NY, USA"}]},{"given":"Jim","family":"O'Leary","sequence":"additional","affiliation":[{"name":"MongoDB Inc, New York, NY, USA"}]},{"given":"David","family":"Bradford","sequence":"additional","affiliation":[{"name":"MongoDB Inc, New York, NY, USA"}]}],"member":"320","published-online":{"date-parts":[[2020,4,20]]},"reference":[{"key":"e_1_3_2_1_1_1","unstructured":"[n.d.]. Best Buy APIs open data set. https:\/\/bestbuyapis.github.io\/api-documentation\/#overview  [n.d.]. Best Buy APIs open data set. https:\/\/bestbuyapis.github.io\/api-documentation\/#overview"},{"key":"e_1_3_2_1_2_1","unstructured":"[n.d.]. (Chromium) Regression Detection for Performance Tests. wiki. https:\/\/www.chromium.org\/chromium-os\/testing\/perf-regression-detection  [n.d.]. (Chromium) Regression Detection for Performance Tests. wiki. https:\/\/www.chromium.org\/chromium-os\/testing\/perf-regression-detection"},{"key":"e_1_3_2_1_3_1","unstructured":"[n.d.]. Circle CI. https:\/\/circleci.com\/docs\/  [n.d.]. Circle CI. https:\/\/circleci.com\/docs\/"},{"key":"e_1_3_2_1_4_1","unstructured":"[n.d.]. Evergreen Continuous Integration. https:\/\/github.com\/evergreen-ci\/evergreen\/wiki  [n.d.]. Evergreen Continuous Integration. https:\/\/github.com\/evergreen-ci\/evergreen\/wiki"},{"key":"e_1_3_2_1_5_1","unstructured":"[n.d.]. Gitlab CI\/CD. https:\/\/docs.gitlab.com\/ee\/ci\/  [n.d.]. Gitlab CI\/CD. https:\/\/docs.gitlab.com\/ee\/ci\/"},{"key":"e_1_3_2_1_6_1","unstructured":"[n.d.]. Jenkins Performance Plugin. wiki.https:\/\/wiki.jenkins.io\/display\/JENKINS\/Performance+Plugin  [n.d.]. Jenkins Performance Plugin. wiki.https:\/\/wiki.jenkins.io\/display\/JENKINS\/Performance+Plugin"},{"key":"e_1_3_2_1_7_1","unstructured":"[n.d.]. (LUCI) A Tour of Continuous Integration UI. https:\/\/chromium.googlesource.com\/chromium\/src.git\/+\/master\/docs\/tour_of_luci_ui.md  [n.d.]. (LUCI) A Tour of Continuous Integration UI. https:\/\/chromium.googlesource.com\/chromium\/src.git\/+\/master\/docs\/tour_of_luci_ui.md"},{"key":"e_1_3_2_1_8_1","unstructured":"[n.d.]. Travis CI. https:\/\/docs.travis-ci.com  [n.d.]. Travis CI. https:\/\/docs.travis-ci.com"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/ROMAN.2007.4415217"},{"key":"e_1_3_2_1_10_1","volume-title":"A survey of methods for timeseries change point detection.Knowledge and information systems 51, 2","author":"Aminikhanghahi Samaneh","year":"2017","unstructured":"Samaneh Aminikhanghahi and Diane J Cook . 2017. A survey of methods for timeseries change point detection.Knowledge and information systems 51, 2 ( 2017 ),339--367. Samaneh Aminikhanghahi and Diane J Cook. 2017. A survey of methods for timeseries change point detection.Knowledge and information systems 51, 2 (2017),339--367."},{"key":"e_1_3_2_1_11_1","volume-title":"Time series analysis: forecasting and control","author":"Box George EP","unstructured":"George EP Box , Gwilym M Jenkins , Gregory C Reinsel , and Greta M Ljung . 2015. Time series analysis: forecasting and control . John Wiley & Sons . George EP Box, Gwilym M Jenkins, Gregory C Reinsel, and Greta M Ljung. 2015. Time series analysis: forecasting and control. John Wiley & Sons."},{"key":"e_1_3_2_1_12_1","unstructured":"Kyle Erf. 2016. Evergreen Continuous Integration: Why We Rein-vented The Wheel. Blog Post. https:\/\/engineering.mongodb.com\/post\/evergreen-continuous-integration-why-we-reinvented-the-wheel  Kyle Erf. 2016. Evergreen Continuous Integration: Why We Rein-vented The Wheel. Blog Post. https:\/\/engineering.mongodb.com\/post\/evergreen-continuous-integration-why-we-reinvented-the-wheel"},{"key":"e_1_3_2_1_13_1","unstructured":"Henrik Ingo and David Daly. 2019. Reducing variability in performance tests onEC2: Setup and Key Results. Blog Post. https:\/\/engineering.mongodb.com\/post\/reducing-variability-in-performance-tests-on-ec2-setup-and-key-results  Henrik Ingo and David Daly. 2019. Reducing variability in performance tests onEC2: Setup and Key Results. Blog Post. https:\/\/engineering.mongodb.com\/post\/reducing-variability-in-performance-tests-on-ec2-setup-and-key-results"},{"key":"e_1_3_2_1_14_1","first-page":"1","article-title":"ecp: An R Package for NonparametricMultiple Change Point Analysis of Multivariate Data","volume":"62","author":"James Nicholas","year":"2015","unstructured":"Nicholas James and David Matteson . 2015 . ecp: An R Package for NonparametricMultiple Change Point Analysis of Multivariate Data . Journal of Statistical Software, Articles 62 , 7 (2015), 1 -- 25 . https:\/\/doi.org\/10.18637\/jss.v062.i07 10.18637\/jss.v062.i07 Nicholas James and David Matteson. 2015. ecp: An R Package for NonparametricMultiple Change Point Analysis of Multivariate Data. Journal of Statistical Software, Articles 62, 7 (2015), 1--25. https:\/\/doi.org\/10.18637\/jss.v062.i07","journal-title":"Journal of Statistical Software, Articles"},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.2013.849605"},{"key":"e_1_3_2_1_16_1","unstructured":"NIST\/SEMATECH. 2012. e-Handbook of Statistical Methods. http:\/\/www.itl.nist.gov\/div898\/handbook\/note.  NIST\/SEMATECH. 2012. e-Handbook of Statistical Methods. http:\/\/www.itl.nist.gov\/div898\/handbook\/note."},{"key":"e_1_3_2_1_17_1","unstructured":"John Ferguson Smart. 2011.Jenkins: The Definitive Guide: Continuous Integrationfor the Masses. \"O'Reilly Media Inc.\".  John Ferguson Smart. 2011.Jenkins: The Definitive Guide: Continuous Integrationfor the Masses. \"O'Reilly Media Inc.\"."},{"key":"e_1_3_2_1_18_1","volume-title":"Filed","author":"Vallone Anthony","year":"2018","unstructured":"Anthony Vallone . 2018 . Window Deviation Analyzer. Patent Publication 2018\/0150373 A1 , Filed Nov. 28, 2016, Published May 31, 2018. Anthony Vallone. 2018. Window Deviation Analyzer. Patent Publication 2018\/0150373 A1, Filed Nov. 28, 2016, Published May 31, 2018."},{"key":"e_1_3_2_1_19_1","unstructured":"DJ Walker-Morgan. 2019. MongoDB 4.2 Performance Boosts. Blog.https:\/\/www.mongodb.com\/blog\/post\/mongodb-42-performance-boosts  DJ Walker-Morgan. 2019. MongoDB 4.2 Performance Boosts. Blog.https:\/\/www.mongodb.com\/blog\/post\/mongodb-42-performance-boosts"}],"event":{"name":"ICPE '20: ACM\/SPEC International Conference on Performance Engineering","location":"Edmonton AB Canada","acronym":"ICPE '20","sponsor":["SIGMETRICS ACM Special Interest Group on Measurement and Evaluation","SIGSOFT ACM Special Interest Group on Software Engineering"]},"container-title":["Proceedings of the ACM\/SPEC International Conference on Performance Engineering"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3358960.3375791","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3358960.3375791","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:13:26Z","timestamp":1750202006000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3358960.3375791"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,4,20]]},"references-count":19,"alternative-id":["10.1145\/3358960.3375791","10.1145\/3358960"],"URL":"https:\/\/doi.org\/10.1145\/3358960.3375791","relation":{},"subject":[],"published":{"date-parts":[[2020,4,20]]},"assertion":[{"value":"2020-04-20","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}