{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,6]],"date-time":"2026-04-06T22:58:49Z","timestamp":1775516329232,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":36,"publisher":"ACM","license":[{"start":{"date-parts":[[2022,8,14]],"date-time":"2022-08-14T00:00:00Z","timestamp":1660435200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2022,8,14]]},"DOI":"10.1145\/3534678.3539145","type":"proceedings-article","created":{"date-parts":[[2022,8,12]],"date-time":"2022-08-12T19:06:12Z","timestamp":1660331172000},"page":"3671-3681","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":35,"title":["Amazon SageMaker Model Monitor: A System for Real-Time Insights into Deployed Machine Learning Models"],"prefix":"10.1145","author":[{"given":"David","family":"Nigenda","sequence":"first","affiliation":[{"name":"Amazon AWS AI, Seattle, WA, USA"}]},{"given":"Zohar","family":"Karnin","sequence":"additional","affiliation":[{"name":"Amazon AWS AI, Haifa, Israel"}]},{"given":"Muhammad Bilal","family":"Zafar","sequence":"additional","affiliation":[{"name":"Amazon AWS AI, Berlin, Germany"}]},{"given":"Raghu","family":"Ramesha","sequence":"additional","affiliation":[{"name":"Amazon AWS AI, Santa Clara, CA, USA"}]},{"given":"Alan","family":"Tan","sequence":"additional","affiliation":[{"name":"Amazon AWS AI, Vancouver, Canada"}]},{"given":"Michele","family":"Donini","sequence":"additional","affiliation":[{"name":"Amazon AWS AI, Santa Clara, CA, USA"}]},{"given":"Krishnaram","family":"Kenthapadi","sequence":"additional","affiliation":[{"name":"Fiddler AI, Palo Alto, CA, USA"}]}],"member":"320","published-online":{"date-parts":[[2022,8,14]]},"reference":[{"key":"e_1_3_2_1_1_1","first-page":"671","article-title":"Big data's disparate impact","volume":"104","author":"Barocas Solon","year":"2016","unstructured":"Solon Barocas and Andrew D Selbst. 2016. Big data's disparate impact. Calif. L. Rev. 104 (2016), 671.","journal-title":"Calif. L. Rev."},{"key":"e_1_3_2_1_2_1","unstructured":"Eric Breck Neoklis Polyzotis Sudip Roy Steven Whang and Martin Zinkevich. 2019. Data Validation for Machine Learning.. In MLSys."},{"key":"e_1_3_2_1_3_1","unstructured":"SM Clarify. 2021. Create Feature Attribute Baselines and Explainability Reports. https:\/\/docs.aws.amazon.com\/sagemaker\/latest\/dg\/clarify-featureattribute-baselines-reports.html Accessed: 2022-02."},{"key":"e_1_3_2_1_4_1","unstructured":"Graham Cormode Zohar Karnin Edo Liberty Justin Thaler and Pavel Vesel"},{"key":"e_1_3_2_1_5_1","unstructured":"y. 2021. Relative Error Streaming Quantiles. In PODS."},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.3905\/jfds.2021.1.075"},{"key":"e_1_3_2_1_7_1","volume-title":"BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805","author":"Devlin Jacob","year":"2018","unstructured":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)."},{"key":"e_1_3_2_1_8_1","unstructured":"Dheeru Dua and Casey Graff. 2017. UCI Machine Learning Repository. http:\/\/archive.ics.uci.edu\/ml"},{"key":"e_1_3_2_1_9_1","unstructured":"Jordan Edwards et al. 2022. MLOps: Model management deployment lineage and monitoring with Azure Machine Learning. https:\/\/tinyurl.com\/57y8rrec"},{"key":"e_1_3_2_1_10_1","volume-title":"An introduction to the bootstrap","author":"Efron Bradley","unstructured":"Bradley Efron and Robert J Tibshirani. 1994. An introduction to the bootstrap. CRC press."},{"key":"e_1_3_2_1_11_1","volume-title":"Evidently AI: Open-Source Machine Learning Monitoring. https:\/\/evidentlyai.com","year":"2022","unstructured":"Evidently. 2022. Evidently AI: Open-Source Machine Learning Monitoring. https:\/\/evidentlyai.com"},{"key":"e_1_3_2_1_12_1","unstructured":"Fiddler. 2022. Explainable Monitoring. https:\/\/www.fiddler.ai\/ml-monitoring"},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2523813"},{"key":"e_1_3_2_1_14_1","unstructured":"Saurabh Garg Yifan Wu Sivaraman Balakrishnan and Zachary Lipton. 2020. A Unified View of Label Shift Estimation. In NeurIPS."},{"key":"e_1_3_2_1_15_1","first-page":"723","article-title":"A kernel two-sample test","volume":"13","author":"Gretton Arthur","year":"2012","unstructured":"Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Sch\u00f6lkopf, and Alexander Smola. 2012. A kernel two-sample test. The Journal of Machine Learning Research 13, 1 (2012), 723--773.","journal-title":"The Journal of Machine Learning Research"},{"key":"e_1_3_2_1_16_1","unstructured":"Sudipto Guha Nina Mishra Gourav Roy and Okke Schrijvers. 2016. Robust random cut forest based anomaly detection on streams. In ICML."},{"key":"e_1_3_2_1_17_1","volume-title":"Sanjiv Das, Kevin Haas, Tyler Hill, and Krishnaram Kenthapadi.","author":"Hardt Michaela","year":"2021","unstructured":"Michaela Hardt, Xiaoguang Chen, Xiaoyi Cheng, Michele Donini, Jason Gelman, Satish Gollaprolu, John He, Pedro Larroy, Xinyu Liu, Nick McCarthy, Ashish Rathi, Scott Rees, Ankit Siva, ErhYuan Tsai, Keerthan Vasist, Pinar Yilmaz, Muhammad Bilal Zafar, Sanjiv Das, Kevin Haas, Tyler Hill, and Krishnaram Kenthapadi. 2021. Amazon SageMaker Clarify: Machine Learning Bias Detection and Explainability in the Cloud. In KDD."},{"key":"e_1_3_2_1_18_1","unstructured":"Moritz Hardt Eric Price and Nati Srebro. 2016. Equality of opportunity in supervised learning. In NeurIPS."},{"key":"e_1_3_2_1_19_1","volume-title":"Meet Michelangelo: Uber's Machine Learning Platform. https:\/\/eng.uber.com\/michelangelo-machine-learningplatform\/","author":"Hermann Jeremy","year":"2017","unstructured":"Jeremy Hermann and Mike Del Balso. 2017. Meet Michelangelo: Uber's Machine Learning Platform. https:\/\/eng.uber.com\/michelangelo-machine-learningplatform\/"},{"key":"e_1_3_2_1_20_1","unstructured":"IBM. 2022. Validating and monitoring AI models with Watson OpenScale. https:\/\/tinyurl.com\/5yzybu44"},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"crossref","unstructured":"Zohar Karnin Kevin Lang and Edo Liberty. 2016. Optimal quantile approximation in streams. In FOCS.","DOI":"10.1109\/FOCS.2016.17"},{"key":"e_1_3_2_1_22_1","unstructured":"Zachary Lipton Yu-Xiang Wang and Alexander Smola. 2018. Detecting and correcting for label shift with black box predictors. In ICML."},{"key":"e_1_3_2_1_23_1","volume-title":"RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907.11692","author":"Liu Yinhan","year":"2019","unstructured":"Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907.11692 (2019)."},{"key":"e_1_3_2_1_24_1","unstructured":"Scott M Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. In NeurIPS."},{"key":"e_1_3_2_1_25_1","unstructured":"Tomas Mikolov Ilya Sutskever Kai Chen Greg S Corrado and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In NeurIPS."},{"key":"e_1_3_2_1_26_1","volume-title":"Machine learning: A probabilistic perspective","author":"Murphy Kevin P","unstructured":"Kevin P Murphy. 2012. Machine learning: A probabilistic perspective. MIT press."},{"key":"e_1_3_2_1_27_1","volume-title":"Raghu Ramesha, Alan Tan, Michele Donini, and Krishnaram Kenthapadi.","author":"Nigenda David","year":"2021","unstructured":"David Nigenda, Zohar Karnin, Muhammad Bilal Zafar, Raghu Ramesha, Alan Tan, Michele Donini, and Krishnaram Kenthapadi. 2021. Amazon SageMaker Model Monitor: A System for Real-Time Insights into Deployed Machine Learning Models. arXiv preprint arXiv:2111.13657 (2021)."},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"crossref","unstructured":"Sashank Reddi Barnabas Poczos and Alex Smola. 2015. Doubly robust covariate shift correction. In AAAI.","DOI":"10.1609\/aaai.v29i1.9576"},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"crossref","unstructured":"Marco Tulio Ribeiro Sameer Singh and Carlos Guestrin. 2016. \"Why should I trust you?\" Explaining the predictions of any classifier. In KDD.","DOI":"10.1145\/2939672.2939778"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"crossref","unstructured":"Sebastian Schelter Dustin Lange Philipp Schmidt Meltem Celikel Felix Biessmann and Andreas Grafberger. 2018. Automating large-scale data quality verification. In VLDB.","DOI":"10.14778\/3229863.3229867"},{"key":"e_1_3_2_1_31_1","unstructured":"David Sculley Gary Holt Daniel Golovin Eugene Davydov Todd Phillips Dietmar Ebner Vinay Chaudhary Michael Young Jean-Francois Crespo and Dan Dennison. 2015. Hidden technical debt in machine learning systems. In NeurIPS."},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-32381-3_16"},{"key":"e_1_3_2_1_33_1","unstructured":"Ankur Taly Kaz Sato and Claudiu Gruia. 2021. Monitoring feature attributions: How Google saved one of the largest ML services in trouble. https:\/\/tinyurl.com\/awt3f5ex Google Cloud Blog.."},{"key":"e_1_3_2_1_34_1","first-page":"58","article-title":"The problem of concept drift: definitions and related work","volume":"106","author":"Tsymbal Alexey","year":"2004","unstructured":"Alexey Tsymbal. 2004. The problem of concept drift: definitions and related work. Computer Science Department, Trinity College Dublin 106, 2 (2004), 58.","journal-title":"Computer Science Department, Trinity College Dublin"},{"key":"e_1_3_2_1_35_1","volume-title":"All of statistics: A concise course in statistical inference","author":"Wasserman Larry","unstructured":"Larry Wasserman. 2004. All of statistics: A concise course in statistical inference. Vol. 26. Springer."},{"key":"e_1_3_2_1_36_1","unstructured":"Yifan Wu Ezra Winston Divyansh Kaushik and Zachary Lipton. 2019. Domain adaptation with asymmetrically-relaxed distribution alignment. In ICML."}],"event":{"name":"KDD '22: The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining","location":"Washington DC USA","acronym":"KDD '22","sponsor":["SIGMOD ACM Special Interest Group on Management of Data","SIGKDD ACM Special Interest Group on Knowledge Discovery in Data"]},"container-title":["Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3534678.3539145","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3534678.3539145","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:02:58Z","timestamp":1750186978000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3534678.3539145"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,8,14]]},"references-count":36,"alternative-id":["10.1145\/3534678.3539145","10.1145\/3534678"],"URL":"https:\/\/doi.org\/10.1145\/3534678.3539145","relation":{},"subject":[],"published":{"date-parts":[[2022,8,14]]},"assertion":[{"value":"2022-08-14","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}