{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,4]],"date-time":"2026-02-04T20:38:40Z","timestamp":1770237520021,"version":"3.49.0"},"reference-count":66,"publisher":"Association for Computing Machinery (ACM)","issue":"FSE","license":[{"start":{"date-parts":[[2024,7,12]],"date-time":"2024-07-12T00:00:00Z","timestamp":1720742400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["No.62272495"],"award-info":[{"award-number":["No.62272495"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100021171","name":"Guangdong Basic and Applied Basic Research Foundation","doi-asserted-by":"crossref","award":["No.2023B1515020054"],"award-info":[{"award-number":["No.2023B1515020054"]}],"id":[{"id":"10.13039\/501100021171","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Softw. Eng."],"published-print":{"date-parts":[[2024,7,12]]},"abstract":"<jats:p>\n                    In large-scale online service systems, the occurrence of software changes is inevitable and frequent. Despite rigorous pre-deployment testing practices, the presence of defective software changes in the online environment cannot be completely eliminated. Consequently, there is a pressing need for automated techniques that can effectively identify these defective changes. However, the current abnormal change detection (ACD) approaches fall short in accurately pinpointing defective changes, primarily due to their disregard for the propagation of faults. To address the limitations of ACD, we propose a novel concept called root cause change analysis (RCCA) to identify the underlying root causes of change-inducing incidents. In order to apply the RCCA concept to practical scenarios, we have devised an intelligent RCCA framework named\n                    <jats:italic toggle=\"yes\">ChangeRCA<\/jats:italic>\n                    . This framework aims to localize the defective change associated with change-inducing incidents among multiple changes. To assess the effectiveness of\n                    <jats:italic toggle=\"yes\">ChangeRCA<\/jats:italic>\n                    , we have conducted an extensive evaluation utilizing a real-world dataset from WeChat and a simulated dataset encompassing 81 diverse defective changes. The evaluation results demonstrate that\n                    <jats:italic toggle=\"yes\">ChangeRCA<\/jats:italic>\n                    outperforms the state-of-the-art ACD approaches, achieving an impressive Top-1 Hit Rate of\n                    <jats:inline-formula>\n                      <mml:math xmlns:mml=\"http:\/\/www.w3.org\/1998\/Math\/MathML\" display=\"inline\">\n                        <mml:mn>85<\/mml:mn>\n                        <mml:mo>%<\/mml:mo>\n                      <\/mml:math>\n                    <\/jats:inline-formula>\n                    and significantly reducing the time required to identify defective changes.\n                  <\/jats:p>","DOI":"10.1145\/3643728","type":"journal-article","created":{"date-parts":[[2024,7,12]],"date-time":"2024-07-12T10:22:09Z","timestamp":1720779729000},"page":"24-46","source":"Crossref","is-referenced-by-count":3,"title":["ChangeRCA: Finding Root Causes from Software Changes in Large Online Systems"],"prefix":"10.1145","volume":"1","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-6195-9088","authenticated-orcid":false,"given":"Guangba","family":"Yu","sequence":"first","affiliation":[{"name":"Sun Yat-sen University, Guangzhou, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0972-6900","authenticated-orcid":false,"given":"Pengfei","family":"Chen","sequence":"additional","affiliation":[{"name":"Sun Yat-sen University, Guangzhou, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7963-082X","authenticated-orcid":false,"given":"Zilong","family":"He","sequence":"additional","affiliation":[{"name":"Sun Yat-sen University, Guangzhou, China"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-9667-9276","authenticated-orcid":false,"given":"Qiuyu","family":"Yan","sequence":"additional","affiliation":[{"name":"Tencent, Guangzhou, China"}]},{"ORCID":"https:\/\/orcid.org\/0009-0008-2652-6138","authenticated-orcid":false,"given":"Yu","family":"Luo","sequence":"additional","affiliation":[{"name":"Tencent, Guangzhou, China"}]},{"ORCID":"https:\/\/orcid.org\/0009-0004-4961-1053","authenticated-orcid":false,"given":"Fangyuan","family":"Li","sequence":"additional","affiliation":[{"name":"Tencent, Guangzhou, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7878-4330","authenticated-orcid":false,"given":"Zibin","family":"Zheng","sequence":"additional","affiliation":[{"name":"Sun Yat-sen University, Guangzhou, China"}]}],"member":"320","published-online":{"date-parts":[[2024,7,12]]},"reference":[{"key":"e_1_3_1_2_2","unstructured":"Amazon. 2017. Summary of the Amazon S3 Service Disruption in the Northern Virginia (US-EAST-1) Region. https:\/\/aws.amazon.com\/message\/41926\/. Accessed February 6 2023."},{"key":"e_1_3_1_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/2810146.2810152"},{"key":"e_1_3_1_4_2","doi-asserted-by":"publisher","DOI":"10.34028\/iajit\/18\/2\/14"},{"key":"e_1_3_1_5_2","doi-asserted-by":"publisher","DOI":"10.1109\/CLOUD53861.2021.00025"},{"key":"e_1_3_1_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/3098822.3098834"},{"key":"e_1_3_1_7_2","unstructured":"ChangeRCA. 2024. ChangeRCA. https:\/\/github.com\/IntelligentDDS\/ChangeRCA. Accessed Feb. 6 2024."},{"key":"e_1_3_1_8_2","doi-asserted-by":"publisher","DOI":"10.2307\/2347162"},{"key":"e_1_3_1_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/3460319.3464810"},{"key":"e_1_3_1_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338916"},{"key":"e_1_3_1_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2016.2553032"},{"key":"e_1_3_1_12_2","unstructured":"Dropbox. 2014. Dropbox change failure. https:\/\/dropbox.tech\/infrastructure\/outage-post-mortem. Accessed February 6 2023."},{"key":"e_1_3_1_13_2","doi-asserted-by":"publisher","DOI":"10.5555\/1656980.1657005"},{"key":"e_1_3_1_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3542929.3563482"},{"key":"e_1_3_1_15_2","unstructured":"Gitlab. 2023. Gitlab. https:\/\/gitlab.com. Accessed Sep. 6 2023."},{"key":"e_1_3_1_16_2","unstructured":"GoogleCloudPlatform. 2023. OnlineBoutique. https:\/\/github.com\/GoogleCloudPlatform\/microservices-demo. Accessed Sep.6 2023."},{"key":"e_1_3_1_17_2","unstructured":"Lynn Greiner. 2020. The great 2020 Gmail outage: A tale of two blackouts and lessons learned. https:\/\/www.itworldcanada.com\/article\/the-great-2020-gmail-outage-a-tale-of-two-blackouts-and-lessons-learned\/439924. Accessed February 6 2023."},{"key":"e_1_3_1_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3556904"},{"key":"e_1_3_1_19_2","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1997.9.8.1735"},{"key":"e_1_3_1_20_2","unstructured":"Istio. 2023. Istio. https:\/\/istio.io. Accessed Sep. 6 2023."},{"key":"e_1_3_1_21_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510152"},{"key":"e_1_3_1_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE43902.2021.00066"},{"key":"e_1_3_1_23_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE55969.2022.00022"},{"key":"e_1_3_1_24_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-20984-0_15"},{"key":"e_1_3_1_25_2","first-page":"389","volume-title":"NSDI 2020","author":"Li Ze","year":"2020","unstructured":"Ze Li, Qian Cheng, Ken Hsieh, Yingnong Dang, Peng Huang, Pankaj Singh, Xinsheng Yang, Qingwei Lin, Youjiang Wu, Sebastien Levy, and Murali Chintalapati. 2020. Gandalf: An Intelligent, End-To-End Analytics Service for Safe Deployment in Large-Scale Cloud Infrastructure. In NSDI 2020. USENIX Association, 389\u2013402. https:\/\/www.usenix.org\/conference\/nsdi20\/presentation\/li"},{"key":"e_1_3_1_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/3540250.3549092"},{"key":"e_1_3_1_27_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-03596-9_1"},{"key":"e_1_3_1_28_2","first-page":"559","volume-title":"NSDI 2020","author":"Lou Chang","year":"2020","unstructured":"Chang Lou, Peng Huang, and Scott Smith. 2020. Understanding, Detecting and Localizing Partial Failures in Large System Software. In NSDI 2020. USENIX Association, 559\u2013574. https:\/\/www.usenix.org\/conference\/nsdi20\/presentation\/lou"},{"key":"e_1_3_1_29_2","unstructured":"Kim Lyons. 2021. Facebook says \u2018configuration change\u2019 caused some users to be logged out unexpectedly. https:\/\/www.theverge.com\/2021\/1\/23\/22245842\/facebook-logged-out-configuration-change-ios-app-security. Accessed February 6 2024."},{"key":"e_1_3_1_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/3452296.3472901"},{"key":"e_1_3_1_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/2079296.2079309"},{"key":"e_1_3_1_32_2","doi-asserted-by":"publisher","DOI":"10.1145\/1851182.1851219"},{"key":"e_1_3_1_33_2","first-page":"435","volume-title":"NSDI 2020","author":"Mehta Sonu","year":"2022","unstructured":"Sonu Mehta, Ranjita Bhagwan, Rahul Kumar, Chetan Bansal, Chandra Shekhar Maddila, Balasubramanyan Ashok, Sumit Asthana, Christian Bird, and Aditya Kumar. 2022. Rex: Preventing Bugs and Misconfiguration in Large Services Using Correlated Change Analysis. In NSDI 2020. USENIX Association, 435\u2013448. https:\/\/www.usenix.org\/conference\/nsdi20\/presentation\/mehta"},{"key":"e_1_3_1_34_2","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2019.2955937"},{"key":"e_1_3_1_35_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2022.111505"},{"key":"e_1_3_1_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/3460319.3464805"},{"key":"e_1_3_1_37_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE.2017.62"},{"key":"e_1_3_1_38_2","unstructured":"Pivotal. 2023. Spring Boot. https:\/\/spring.io. Accessed Sep. 6 2023."},{"key":"e_1_3_1_39_2","doi-asserted-by":"publisher","DOI":"10.1145\/2889160.2889223"},{"key":"e_1_3_1_40_2","unstructured":"Toby Sterling. 2021. Fastly blames software bug for major global internet outage. https:\/\/www.reuters.com\/business\/media-telecom\/fastly-blames-software-bug-major-global-internet-outage-2021-06-09\/. Accessed February 6 2023."},{"key":"e_1_3_1_41_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10742-014-0123-z"},{"issue":"5","key":"e_1_3_1_42_2","first-page":"360","article-title":"Understanding interobserver agreement: the kappa statistic","volume":"37","author":"Viera Anthony J","year":"2005","unstructured":"Anthony J Viera, Joanne M Garrett, et al. 2005. Understanding interobserver agreement: the kappa statistic. Fam med 37, 5 (2005), 360\u2013363. https:\/\/pubmed.ncbi.nlm.nih.gov\/15883903\/","journal-title":"Fam med"},{"key":"e_1_3_1_43_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE51524.2021.9678708"},{"key":"e_1_3_1_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/3534678.3539127"},{"key":"e_1_3_1_45_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE55969.2022.00043"},{"key":"e_1_3_1_46_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2019.2948158"},{"key":"e_1_3_1_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/2970276.2970359"},{"key":"e_1_3_1_48_2","doi-asserted-by":"publisher","DOI":"10.1109\/NOMS47738.2020.9110353"},{"key":"e_1_3_1_49_2","doi-asserted-by":"publisher","DOI":"10.1145\/3180155.3182516"},{"key":"e_1_3_1_50_2","first-page":"619","volume-title":"OSDI 2016","author":"Xu Tianyin","year":"2016","unstructured":"Tianyin Xu, Xinxin Jin, Peng Huang, Yuanyuan Zhou, Shan Lu, Long Jin, and Shankar Pasupathy. 2016. Early Detection of Configuration Errors to Reduce Failure Damage. In OSDI 2016. USENIX Association, 619\u2013634. https:\/\/www.usenix.org\/conference\/osdi16\/technical-sessions\/presentation\/xu"},{"key":"e_1_3_1_51_2","doi-asserted-by":"publisher","DOI":"10.1145\/3447851.3458737"},{"key":"e_1_3_1_52_2","doi-asserted-by":"publisher","DOI":"10.1109\/CCGrid51090.2021.00051"},{"key":"e_1_3_1_53_2","doi-asserted-by":"publisher","DOI":"10.1145\/3442381.3449905"},{"key":"e_1_3_1_54_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE48619.2023.00151"},{"key":"e_1_3_1_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/3611643.3616249"},{"key":"e_1_3_1_56_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICWS.2019.00023"},{"key":"e_1_3_1_57_2","doi-asserted-by":"publisher","DOI":"10.1002\/smr.2413"},{"key":"e_1_3_1_58_2","first-page":"575","volume-title":"NSDI 2020","author":"Zhai Ennan","year":"2020","unstructured":"Ennan Zhai, Ang Chen, Ruzica Piskac, Mahesh Balakrishnan, Bingchuan Tian, Bo Song, and Haoliang Zhang. 2020. Check before You Change: Preventing Correlated Failures in Service Updates. In NSDI 2020. USENIX Association, 575\u2013589. https:\/\/www.usenix.org\/conference\/nsdi20\/presentation\/zhai"},{"key":"e_1_3_1_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/3510003.3510132"},{"key":"e_1_3_1_60_2","doi-asserted-by":"publisher","DOI":"10.1145\/3092703.3092731"},{"key":"e_1_3_1_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/2716281.2836087"},{"key":"e_1_3_1_62_2","doi-asserted-by":"publisher","DOI":"10.1145\/3459637.3481903"},{"key":"e_1_3_1_63_2","doi-asserted-by":"publisher","DOI":"10.1145\/3477132.3483577"},{"key":"e_1_3_1_64_2","doi-asserted-by":"publisher","DOI":"10.1145\/3468264.3468543"},{"key":"e_1_3_1_65_2","doi-asserted-by":"publisher","DOI":"10.1109\/ISSRE59848.2023.00028"},{"key":"e_1_3_1_66_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2018.2887384"},{"key":"e_1_3_1_67_2","doi-asserted-by":"publisher","DOI":"10.1145\/3338906.3338961"}],"container-title":["Proceedings of the ACM on Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3643728","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3643728","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,2,4]],"date-time":"2026-02-04T07:51:54Z","timestamp":1770191514000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3643728"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,12]]},"references-count":66,"journal-issue":{"issue":"FSE","published-print":{"date-parts":[[2024,7,12]]}},"alternative-id":["10.1145\/3643728"],"URL":"https:\/\/doi.org\/10.1145\/3643728","relation":{},"ISSN":["2994-970X"],"issn-type":[{"value":"2994-970X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,7,12]]}}}