{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,30]],"date-time":"2026-03-30T20:54:27Z","timestamp":1774904067299,"version":"3.50.1"},"reference-count":87,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2021,11,15]],"date-time":"2021-11-15T00:00:00Z","timestamp":1636934400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Softw. Eng. Methodol."],"published-print":{"date-parts":[[2022,1,31]]},"abstract":"<jats:p>Artificial Intelligence for IT Operations (AIOps) has been adopted in organizations in various tasks, including interpreting models to identify indicators of service failures. To avoid misleading practitioners, AIOps model interpretations should be consistent (i.e., different AIOps models on the same task agree with one another on feature importance). However, many AIOps studies violate established practices in the machine learning community when deriving interpretations, such as interpreting models with suboptimal performance, though the impact of such violations on the interpretation consistency has not been studied.<\/jats:p>\n          <jats:p>In this article, we investigate the consistency of AIOps model interpretation along three dimensions: internal consistency, external consistency, and time consistency. We conduct a case study on two AIOps tasks: predicting Google cluster job failures and Backblaze hard drive failures. We find that the randomness from learners, hyperparameter tuning, and data sampling should be controlled to generate consistent interpretations. AIOps models with AUCs greater than 0.75 yield more consistent interpretation compared to low-performing models. Finally, AIOps models that are constructed with the Sliding Window or Full History approaches have the most consistent interpretation with the trends presented in the entire datasets. Our study provides valuable guidelines for practitioners to derive consistent AIOps model interpretation.<\/jats:p>","DOI":"10.1145\/3488269","type":"journal-article","created":{"date-parts":[[2021,11,15]],"date-time":"2021-11-15T17:44:52Z","timestamp":1636998292000},"page":"1-38","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":22,"title":["Towards a Consistent Interpretation of AIOps Models"],"prefix":"10.1145","volume":"31","author":[{"given":"Yingzhe","family":"Lyu","sequence":"first","affiliation":[{"name":"Software Analysis and Intelligence Lab (SAIL), Queen\u2019s University, Kingston, Ontario, Canada"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1812-5365","authenticated-orcid":false,"given":"Gopi Krishnan","family":"Rajbahadur","sequence":"additional","affiliation":[{"name":"Centre for Software Excellence, Huawei, Canada"}]},{"given":"Dayi","family":"Lin","sequence":"additional","affiliation":[{"name":"Centre for Software Excellence, Huawei, Canada"}]},{"given":"Boyuan","family":"Chen","sequence":"additional","affiliation":[{"name":"Centre for Software Excellence, Huawei, Canada"}]},{"given":"Zhen Ming (Jack)","family":"Jiang","sequence":"additional","affiliation":[{"name":"Lassonde School of Engineering, York University, Toronto, Ontario, Canada"}]}],"member":"320","published-online":{"date-parts":[[2021,11,15]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"Mikhail Korobov Konstantin Lopuhin. 2021. ELI5. Read The Docs. Retrieved from https:\/\/eli5.readthedocs.io\/en\/latest\/overview.html.   Mikhail Korobov Konstantin Lopuhin. 2021. ELI5. Read The Docs. Retrieved from https:\/\/eli5.readthedocs.io\/en\/latest\/overview.html."},{"key":"e_1_2_1_2_1","unstructured":"Ender Celik. 2021. PIMP - R implementation for permutation importance score. RDocumentation. Retrieved from https:\/\/www.rdocumentation.org\/packages\/vita\/versions\/1.0.0\/topics\/PIMP.   Ender Celik. 2021. PIMP - R implementation for permutation importance score. RDocumentation. Retrieved from https:\/\/www.rdocumentation.org\/packages\/vita\/versions\/1.0.0\/topics\/PIMP."},{"key":"e_1_2_1_3_1","volume-title":"Proceedings of the Conference on Neural Information Processing Systems.","author":"Adebayo Julius","year":"2018","unstructured":"Julius Adebayo , Justin Gilmer , Michael Muelly , Ian J. Goodfellow , Moritz Hardt , and Been Kim . 2018 . Sanity checks for saliency maps . In Proceedings of the Conference on Neural Information Processing Systems. Julius Adebayo, Justin Gilmer, Michael Muelly, Ian J. Goodfellow, Moritz Hardt, and Been Kim. 2018. Sanity checks for saliency maps. In Proceedings of the Conference on Neural Information Processing Systems."},{"key":"e_1_2_1_4_1","volume-title":"Global hourly enterprise server downtime cost","author":"Alsop Thomas","year":"2019","unstructured":"Thomas Alsop . 2020. Global hourly enterprise server downtime cost 2019 . Retrieved from https:\/\/www.statista.com\/statistics\/753938\/worldwide-enterprise-serverhourly-downtime-cost\/. Thomas Alsop. 2020. Global hourly enterprise server downtime cost 2019. Retrieved from https:\/\/www.statista.com\/statistics\/753938\/worldwide-enterprise-serverhourly-downtime-cost\/."},{"key":"e_1_2_1_5_1","unstructured":"Backblaze. 2020. Hard Drive Data and Stats. Retrieved from https:\/\/www.backblaze.com\/b2\/hard-drive-test-data.html.  Backblaze. 2020. Hard Drive Data and Stats. Retrieved from https:\/\/www.backblaze.com\/b2\/hard-drive-test-data.html."},{"key":"e_1_2_1_6_1","volume-title":"Proceedings of the USENIX Conference on Operational Machine Learning.","author":"Banerjee Amitabha","year":"2020","unstructured":"Amitabha Banerjee , Chien-Chia Chen , Chien-Chun Hung , Xiaobo Huang , Yifan Wang , and Razvan Chevesaran . 2020 . Challenges and experiences with MLOps for performance diagnostics in hybrid-cloud enterprise software deployments . In Proceedings of the USENIX Conference on Operational Machine Learning. Amitabha Banerjee, Chien-Chia Chen, Chien-Chun Hung, Xiaobo Huang, Yifan Wang, and Razvan Chevesaran. 2020. Challenges and experiences with MLOps for performance diagnostics in hybrid-cloud enterprise software deployments. In Proceedings of the USENIX Conference on Operational Machine Learning."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-020-09878-9"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3377813.3381353"},{"key":"e_1_2_1_9_1","unstructured":"James Bergstra and Yoshua Bengio. 2012. Random search for hyper-parameter optimization. J. Mach. Learn. Res. (2012).  James Bergstra and Yoshua Bengio. 2012. Random search for hyper-parameter optimization. J. Mach. Learn. Res. (2012)."},{"key":"e_1_2_1_10_1","volume-title":"Statistics review 13: Receiver operating characteristic curves. Crit. Care","author":"Bewick Viv","year":"2004","unstructured":"Viv Bewick , Liz Cheek , and Jonathan Ball . 2004. Statistics review 13: Receiver operating characteristic curves. Crit. Care ( 2004 ). Viv Bewick, Liz Cheek, and Jonathan Ball. 2004. Statistics review 13: Receiver operating characteristic curves. Crit. Care (2004)."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939699"},{"key":"e_1_2_1_12_1","volume-title":"Proceedings of the 29th International Conference on Machine Learning.","author":"Boyd Kendrick","unstructured":"Kendrick Boyd , Vitor Santos Costa , Jesse Davis , and C. David Page . 2012. Unachievable region in precision-recall space and its effect on empirical evaluation . In Proceedings of the 29th International Conference on Machine Learning. Kendrick Boyd, Vitor Santos Costa, Jesse Davis, and C. David Page. 2012. Unachievable region in precision-recall space and its effect on empirical evaluation. In Proceedings of the 29th International Conference on Machine Learning."},{"key":"e_1_2_1_13_1","doi-asserted-by":"crossref","unstructured":"Johan Bring. 1995. Variable importance by partitioning R 2. Qual. Quant. (1995).  Johan Bring. 1995. Variable importance by partitioning R 2. Qual. Quant. (1995).","DOI":"10.1007\/BF01101897"},{"key":"e_1_2_1_14_1","volume-title":"Can I trust the explainer? Verifying post-hoc explanatory methods. CoRR abs\/1910.02065","author":"Camburu Oana-Maria","year":"2019","unstructured":"Oana-Maria Camburu , Eleonora Giunchiglia , Jakob Foerster , Thomas Lukasiewicz , and Phil Blunsom . 2019. Can I trust the explainer? Verifying post-hoc explanatory methods. CoRR abs\/1910.02065 ( 2019 ). Oana-Maria Camburu, Eleonora Giunchiglia, Jakob Foerster, Thomas Lukasiewicz, and Phil Blunsom. 2019. Can I trust the explainer? Verifying post-hoc explanatory methods. CoRR abs\/1910.02065 (2019)."},{"key":"e_1_2_1_15_1","doi-asserted-by":"crossref","unstructured":"Boyuan Chen and Zhen Ming Jack Jiang. 2019. Extracting and studying the logging-code-issue-introducing changes in Java-based large-scale open source software systems. Empir. Softw. Eng. (2019).  Boyuan Chen and Zhen Ming Jack Jiang. 2019. Extracting and studying the logging-code-issue-introducing changes in Java-based large-scale open source software systems. Empir. Softw. Eng. (2019).","DOI":"10.1007\/s10664-019-09690-0"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00042"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416624"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409768"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3308558.3313501"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3417055"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICSE-Companion.2019.00023"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDCS.2017.317"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIFS.2020.3021924"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-28645-5_29"},{"key":"e_1_2_1_25_1","volume-title":"Hanna M. Wallach, Hal Daum\u00e9 III, and Kate Crawford.","author":"Gebru Timnit","year":"2018","unstructured":"Timnit Gebru , Jamie Morgenstern , Briana Vecchione , Jennifer Wortman Vaughan , Hanna M. Wallach, Hal Daum\u00e9 III, and Kate Crawford. 2018 . Datasheets for datasets. CoRR abs\/1803.09010 (2018). Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna M. Wallach, Hal Daum\u00e9 III, and Kate Crawford. 2018. Datasheets for datasets. CoRR abs\/1803.09010 (2018)."},{"key":"e_1_2_1_26_1","unstructured":"Odd Erik Gundersen and Sigbj\u00f8rn Kjensmo. 2018. State of the art: Reproducibility in artificial intelligence. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence (AAAI-18) the 30th Innovative Applications of Artificial Intelligence (IAAI-18) and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18).  Odd Erik Gundersen and Sigbj\u00f8rn Kjensmo. 2018. State of the art: Reproducibility in artificial intelligence. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence (AAAI-18) the 30th Innovative Applications of Artificial Intelligence (IAAI-18) and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18)."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3243734.3243792"},{"key":"e_1_2_1_28_1","volume-title":"Proceedings of the 31st International Conference on Machine Learning.","author":"Harel Maayan","year":"2014","unstructured":"Maayan Harel , Shie Mannor , Ran El-Yaniv , and Koby Crammer . 2014 . Concept drift detection through resampling . In Proceedings of the 31st International Conference on Machine Learning. Maayan Harel, Shie Mannor, Ran El-Yaniv, and Koby Crammer. 2014. Concept drift detection through resampling. In Proceedings of the 31st International Conference on Machine Learning."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3236083"},{"key":"e_1_2_1_30_1","unstructured":"Reyhane Askari Hemmat and Abdelhakim Hafid. 2016. SLA violation prediction in cloud computing: A machine learning perspective. CoRR arXiv:1611.10338. https:\/\/arxiv.org\/abs\/1611.10338.  Reyhane Askari Hemmat and Abdelhakim Hafid. 2016. SLA violation prediction in cloud computing: A machine learning perspective. CoRR arXiv:1611.10338. https:\/\/arxiv.org\/abs\/1611.10338."},{"key":"e_1_2_1_31_1","volume-title":"Artificial intelligence faces reproducibility crisis. Science","author":"Hutson Matthew","year":"2018","unstructured":"Matthew Hutson . 2018. Artificial intelligence faces reproducibility crisis. Science ( 2018 ). Matthew Hutson. 2018. Artificial intelligence faces reproducibility crisis. Science (2018)."},{"key":"e_1_2_1_32_1","unstructured":"IDG. 2020. 2020 Cloud Computing Study. Retrieved from https:\/\/www.idg.com\/tools-for-marketers\/2020-cloud-computing-study\/.  IDG. 2020. 2020 Cloud Computing Study. Retrieved from https:\/\/www.idg.com\/tools-for-marketers\/2020-cloud-computing-study\/."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/CLOUD.2014.129"},{"key":"e_1_2_1_34_1","unstructured":"George Jenks. 1967. The data model concept in statistical mapping. In International Yearbook of Cartography. George Philip.  George Jenks. 1967. The data model concept in statistical mapping. In International Yearbook of Cartography. George Philip."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3417054"},{"key":"e_1_2_1_36_1","volume-title":"Hoa Khanh Dam, and John Grundy","author":"Jiarpakdee Jirayus","year":"2020","unstructured":"Jirayus Jiarpakdee , Chakkrit Tantithamthavorn , Hoa Khanh Dam, and John Grundy . 2020 . An empirical study of model-agnostic techniques for defect prediction models. IEEE Trans. Softw. Eng . (2020). Jirayus Jiarpakdee, Chakkrit Tantithamthavorn, Hoa Khanh Dam, and John Grundy. 2020. An empirical study of model-agnostic techniques for defect prediction models. IEEE Trans. Softw. Eng. (2020)."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-020-09848-1"},{"key":"e_1_2_1_38_1","volume-title":"Bellwethers: A baseline method for transfer learning","author":"Krishna Rahul","year":"2018","unstructured":"Rahul Krishna and Tim Menzies . 2018 . Bellwethers: A baseline method for transfer learning . IEEE Trans. Softw. Eng . (2018). Rahul Krishna and Tim Menzies. 2018. Bellwethers: A baseline method for transfer learning. IEEE Trans. Softw. Eng. (2018)."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1952.10483441"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1109\/DSN.2014.44"},{"key":"e_1_2_1_41_1","unstructured":"Jing Li Rebecca J. Stones Gang Wang Xiaoguang Liu Zhongwei Li and Ming Xu. 2017. Hard drive failure prediction using Decision Trees. Reliab. Eng. Syst. Saf. (2017).  Jing Li Rebecca J. Stones Gang Wang Xiaoguang Liu Zhongwei Li and Ming Xu. 2017. Hard drive failure prediction using Decision Trees. Reliab. Eng. Syst. Saf. (2017)."},{"key":"e_1_2_1_42_1","unstructured":"Yangguang Li Zhen Ming (Jack) Jiang Heng Li Ahmed E. Hassan Cheng He Ruirui Huang Zhengda Zeng Mian Wang and Pinan Chen. 2020. Predicting node failures in an ultra-large-scale cloud computing platform: An AIOps solution. ACM Trans. Softw. Eng. Methodol. (2020).  Yangguang Li Zhen Ming (Jack) Jiang Heng Li Ahmed E. Hassan Cheng He Ruirui Huang Zhengda Zeng Mian Wang and Pinan Chen. 2020. Predicting node failures in an ultra-large-scale cloud computing platform: An AIOps solution. ACM Trans. Softw. Eng. Methodol. (2020)."},{"key":"e_1_2_1_43_1","volume-title":"Liem and Annibale Panichella","author":"Cynthia C.","year":"2020","unstructured":"Cynthia C. S. Liem and Annibale Panichella . 2020 . Run, forest, run? On randomization and reproducibility in predictive software engineering. arXiv:2012.08387 (2020). Cynthia C. S. Liem and Annibale Panichella. 2020. Run, forest, run? On randomization and reproducibility in predictive software engineering. arXiv:2012.08387 (2020)."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2014.96"},{"key":"e_1_2_1_45_1","volume-title":"Hassan","author":"Lin Dayi","year":"2018","unstructured":"Dayi Lin , Cor-Paul Bezemer , and Ahmed E . Hassan . 2018 . An empirical study of early access games on the Steam platform. Empir. Softw. Eng . (2018). Dayi Lin, Cor-Paul Bezemer, and Ahmed E. Hassan. 2018. An empirical study of early access games on the Steam platform. Empir. Softw. Eng. (2018)."},{"key":"e_1_2_1_46_1","volume-title":"Hassan","author":"Lin Dayi","year":"2019","unstructured":"Dayi Lin , Cor-Paul Bezemer , and Ahmed E . Hassan . 2019 . Identifying gameplay videos that exhibit bugs in computer games. Empir. Softw. Eng . (2019). Dayi Lin, Cor-Paul Bezemer, and Ahmed E. Hassan. 2019. Identifying gameplay videos that exhibit bugs in computer games. Empir. Softw. Eng. (2019)."},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3236024.3236060"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/2884781.2884795"},{"key":"e_1_2_1_49_1","volume-title":"The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery.Queue","author":"Lipton Zachary C.","year":"2018","unstructured":"Zachary C. Lipton . 2018. The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery.Queue ( 2018 ). Zachary C. Lipton. 2018. The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery.Queue (2018)."},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2013.6693105"},{"key":"e_1_2_1_51_1","unstructured":"Jian-Guang Lou Qingwei Lin Rui Ding Qiang Fu Dongmei Zhang and Tao Xie. 2017. Experience report on applying software analytics in incident management of online service. Autom. Softw. Eng. (2017).  Jian-Guang Lou Qingwei Lin Rui Ding Qiang Fu Dongmei Zhang and Tao Xie. 2017. Experience report on applying software analytics in incident management of online service. Autom. Softw. Eng. (2017)."},{"key":"e_1_2_1_52_1","volume-title":"Proceedings of the 31st International Conference on Neural Information Processing Systems.","author":"Scott","unstructured":"Scott M. Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions . In Proceedings of the 31st International Conference on Neural Information Processing Systems. Scott M. Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. In Proceedings of the 31st International Conference on Neural Information Processing Systems."},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/2623330.2623374"},{"key":"e_1_2_1_54_1","unstructured":"Yingzhe Lyu. 2021. Replication Package for an Empirical Study of the Impact of Data Splitting Decisions on the Performance of AIOps Solutions. Retrieved from https:\/\/github.com\/SAILResearch\/suppmaterial-19-yingzhe-aiops_data_splitting.  Yingzhe Lyu. 2021. Replication Package for an Empirical Study of the Impact of Data Splitting Decisions on the Performance of AIOps Solutions. Retrieved from https:\/\/github.com\/SAILResearch\/suppmaterial-19-yingzhe-aiops_data_splitting."},{"key":"e_1_2_1_55_1","volume-title":"Hassan","author":"Lyu Yingzhe","year":"2021","unstructured":"Yingzhe Lyu , Heng Li , Mohammed Sayagh , Zhenming (Jack) Jiang , and Ahmed E . Hassan . 2021 . An empirical study of the impact of data splitting decisions on the performance of AIOps solutions. ACM Trans. Softw. Eng. Methodol . (2021). Yingzhe Lyu, Heng Li, Mohammed Sayagh, Zhenming (Jack) Jiang, and Ahmed E. Hassan. 2021. An empirical study of the impact of data splitting decisions on the performance of AIOps solutions. ACM Trans. Softw. Eng. Methodol. (2021)."},{"key":"e_1_2_1_56_1","volume-title":"Proceedings of the USENIX Annual Technical Conference. 391\u2013402","author":"Mahdisoltani Farzaneh","year":"2017","unstructured":"Farzaneh Mahdisoltani , Ioan A. Stefanovici , and Bianca Schroeder . 2017 . Proactive error prediction to improve storage system reliability . In Proceedings of the USENIX Annual Technical Conference. 391\u2013402 . Farzaneh Mahdisoltani, Ioan A. Stefanovici, and Bianca Schroeder. 2017. Proactive error prediction to improve storage system reliability. In Proceedings of the USENIX Annual Technical Conference. 391\u2013402."},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2019.2954841"},{"key":"e_1_2_1_58_1","volume-title":"Council Post: How to Optimize Gross Margins For Digital Services.","author":"Merker Kit","year":"2020","unstructured":"Kit Merker . 2020 . Council Post: How to Optimize Gross Margins For Digital Services. Retrieved from https:\/\/www.forbes.com\/sites\/forbesbusinessdevelopmentcouncil\/2020\/12\/03\/how-to-optimize-gross-margins-for-digital-services\/?sh=13d62bb130c3. Kit Merker. 2020. Council Post: How to Optimize Gross Margins For Digital Services. Retrieved from https:\/\/www.forbes.com\/sites\/forbesbusinessdevelopmentcouncil\/2020\/12\/03\/how-to-optimize-gross-margins-for-digital-services\/?sh=13d62bb130c3."},{"key":"e_1_2_1_59_1","volume-title":"Simultaneous Statistical Inference","author":"Miller Rupert G.","unstructured":"Rupert G. Miller . 2012. Simultaneous Statistical Inference . Springer Science & Business Media . Rupert G. Miller. 2012. Simultaneous Statistical Inference. Springer Science & Business Media."},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/3287560.3287596"},{"key":"e_1_2_1_61_1","unstructured":"Christoph Molnar. 2019. Interpretable Machine Learning. Retrieved from https:\/\/christophm.github.io\/interpretable-ml-book\/.  Christoph Molnar. 2019. Interpretable Machine Learning. Retrieved from https:\/\/christophm.github.io\/interpretable-ml-book\/."},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1109\/CCGRID.2019.00038"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v33i01.33019780"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/3324884.3416545"},{"key":"e_1_2_1_65_1","unstructured":"Joelle Pineau Philippe Vincent-Lamarre Koustuv Sinha Vincent Larivi\u00e8re Alina Beygelzimer Florence d\u2019Alch\u00e9 Buc Emily Fox and Hugo Larochelle. 2020. Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program). arxiv:2003.12206 [cs.LG]  Joelle Pineau Philippe Vincent-Lamarre Koustuv Sinha Vincent Larivi\u00e8re Alina Beygelzimer Florence d\u2019Alch\u00e9 Buc Emily Fox and Hugo Larochelle. 2020. Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program). arxiv:2003.12206 [cs.LG]"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2021.3056941"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/2884781.2884848"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939778"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v32i1.11491"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1109\/IWQoS.2015.7404739"},{"key":"e_1_2_1_72_1","doi-asserted-by":"crossref","unstructured":"Cynthia Rudin. 2019. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat. Mach. Intell. (2019).  Cynthia Rudin. 2019. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat. Mach. Intell. (2019).","DOI":"10.1038\/s42256-019-0048-x"},{"key":"e_1_2_1_73_1","volume-title":"Proceedings of the 7th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.","author":"Nick Street W.","year":"2001","unstructured":"W. Nick Street and YongSeog Kim . 2001 . A streaming ensemble algorithm (SEA) for large-scale classification . In Proceedings of the 7th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. W. Nick Street and YongSeog Kim. 2001. A streaming ensemble algorithm (SEA) for large-scale classification. In Proceedings of the 7th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining."},{"key":"e_1_2_1_74_1","volume-title":"Proceedings of the International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP\u201918)","author":"Tantithamthavorn Chakkrit","unstructured":"Chakkrit Tantithamthavorn and Ahmed E. Hassan . 2018. An experience report on defect modelling in practice: Pitfalls and challenges . In Proceedings of the International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP\u201918) . Chakkrit Tantithamthavorn and Ahmed E. Hassan. 2018. An experience report on defect modelling in practice: Pitfalls and challenges. In Proceedings of the International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP\u201918)."},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2018.2876537"},{"key":"e_1_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2018.2794977"},{"key":"e_1_2_1_77_1","volume-title":"Zumbo","author":"Thomas D. Roland","year":"1998","unstructured":"D. Roland Thomas , Edward Hughes , and Bruno D . Zumbo . 1998 . On variable importance in linear regression. Soc. Indic. Res . (1998). D. Roland Thomas, Edward Hughes, and Bruno D. Zumbo. 1998. On variable importance in linear regression. Soc. Indic. Res. (1998)."},{"key":"e_1_2_1_78_1","volume-title":"Who belongs in the family. Psychometrika","author":"Thorndike Robert L.","year":"1953","unstructured":"Robert L. Thorndike . 1953. Who belongs in the family. Psychometrika ( 1953 ). Robert L. Thorndike. 1953. Who belongs in the family. Psychometrika (1953)."},{"key":"e_1_2_1_79_1","unstructured":"Burak Turhan. 2012. On the dataset shift problem in software engineering prediction models. Empir. Softw. Eng. (2012).  Burak Turhan. 2012. On the dataset shift problem in software engineering prediction models. Empir. Softw. Eng. (2012)."},{"key":"e_1_2_1_80_1","doi-asserted-by":"publisher","DOI":"10.1145\/956750.956778"},{"key":"e_1_2_1_81_1","doi-asserted-by":"publisher","DOI":"10.1109\/EuroSP48549.2020.00018"},{"key":"e_1_2_1_82_1","volume-title":"Proceedings of the USENIX Annual Technical Conference.","author":"Xu Yong","year":"2018","unstructured":"Yong Xu , Kaixin Sui , Randolph Yao , Hongyu Zhang , Qingwei Lin , Yingnong Dang , Peng Li , Keceng Jiang , Wenchi Zhang , Jian-Guang Lou , Murali Chintalapati , and Dongmei Zhang . 2018 . Improving service availability of cloud systems by predicting disk error . In Proceedings of the USENIX Annual Technical Conference. Yong Xu, Kaixin Sui, Randolph Yao, Hongyu Zhang, Qingwei Lin, Yingnong Dang, Peng Li, Keceng Jiang, Wenchi Zhang, Jian-Guang Lou, Murali Chintalapati, and Dongmei Zhang. 2018. Improving service availability of cloud systems by predicting disk error. In Proceedings of the USENIX Annual Technical Conference."},{"key":"e_1_2_1_83_1","doi-asserted-by":"publisher","DOI":"10.1109\/DSN.2016.38"},{"key":"e_1_2_1_84_1","doi-asserted-by":"publisher","DOI":"10.1109\/TNSM.2018.2794409"},{"key":"e_1_2_1_85_1","volume-title":"Proceedings of the Annual Conference on Neural Information Processing Systems.","author":"Yeh Chih-Kuan","year":"2019","unstructured":"Chih-Kuan Yeh , Cheng-Yu Hsieh , Arun Sai Suggala , David I. Inouye , and Pradeep Ravikumar . 2019 . On the (In)fidelity and sensitivity of explanations . In Proceedings of the Annual Conference on Neural Information Processing Systems. Chih-Kuan Yeh, Cheng-Yu Hsieh, Arun Sai Suggala, David I. Inouye, and Pradeep Ravikumar. 2019. On the (In)fidelity and sensitivity of explanations. In Proceedings of the Annual Conference on Neural Information Processing Systems."},{"key":"e_1_2_1_86_1","volume-title":"Deepak Puthal, Albert Y. Zomaya, and Rajiv Ranjan.","author":"Zeng Xuezhi","year":"2021","unstructured":"Xuezhi Zeng , Saurabh Garg , Mutaz Barika , Sanat Kumar Bista , Deepak Puthal, Albert Y. Zomaya, and Rajiv Ranjan. 2021 . Detection of SLA violation for big data analytics applications in cloud. IEEE Trans. Comput . (2021). Xuezhi Zeng, Saurabh Garg, Mutaz Barika, Sanat Kumar Bista, Deepak Puthal, Albert Y. Zomaya, and Rajiv Ranjan. 2021. Detection of SLA violation for big data analytics applications in cloud. IEEE Trans. Comput. (2021)."},{"key":"e_1_2_1_87_1","doi-asserted-by":"publisher","DOI":"10.1109\/DSN.2005.44"},{"key":"e_1_2_1_88_1","doi-asserted-by":"publisher","DOI":"10.1145\/3368089.3409672"}],"container-title":["ACM Transactions on Software Engineering and Methodology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3488269","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3488269","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:30:23Z","timestamp":1750188623000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3488269"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,11,15]]},"references-count":87,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2022,1,31]]}},"alternative-id":["10.1145\/3488269"],"URL":"https:\/\/doi.org\/10.1145\/3488269","relation":{},"ISSN":["1049-331X","1557-7392"],"issn-type":[{"value":"1049-331X","type":"print"},{"value":"1557-7392","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,11,15]]},"assertion":[{"value":"2021-03-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-08-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-11-15","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}