{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,2]],"date-time":"2026-07-02T16:36:16Z","timestamp":1783010176963,"version":"3.54.5"},"publisher-location":"New York, NY, USA","reference-count":30,"publisher":"ACM","license":[{"start":{"date-parts":[[2020,5,31]],"date-time":"2020-05-31T00:00:00Z","timestamp":1590883200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2020,6,11]]},"DOI":"10.1145\/3318464.3380604","type":"proceedings-article","created":{"date-parts":[[2020,5,29]],"date-time":"2020-05-29T17:12:33Z","timestamp":1590772353000},"page":"1289-1299","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":25,"title":["Learning to Validate the Predictions of Black Box Classifiers on Unseen Data"],"prefix":"10.1145","author":[{"given":"Sebastian","family":"Schelter","sequence":"first","affiliation":[{"name":"New York University, New York, NY, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tammo","family":"Rukat","sequence":"additional","affiliation":[{"name":"Amazon Research, Berlin, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Felix","family":"Biessmann","sequence":"additional","affiliation":[{"name":"Beuth University Berlin, Berlin, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2020,5,31]]},"reference":[{"key":"e_1_3_2_2_1_1","unstructured":"Denis Baylor Eric Breck Heng-Tze Cheng Noah Fiedel Chuan Yu Foo Zakaria Haque Salem Haykal Mustafa Ispir Vihan Jain Levent Koc Chiu Yuen Koo Lukasz Lew Clemens Mewald Akshay Naresh Modi Neoklis Polyzotis Sukriti Ramesh Sudip Roy Steven Euijong Whang Martin Wicke Jarek Wilkiewicz Xin Zhang and Martin Zinkevich. 2017. TFX: A TensorFlow-Based Production-Scale Machine Learning Platform. KDD 1387--1395. Denis Baylor Eric Breck Heng-Tze Cheng Noah Fiedel Chuan Yu Foo Zakaria Haque Salem Haykal Mustafa Ispir Vihan Jain Levent Koc Chiu Yuen Koo Lukasz Lew Clemens Mewald Akshay Naresh Modi Neoklis Polyzotis Sukriti Ramesh Sudip Roy Steven Euijong Whang Martin Wicke Jarek Wilkiewicz Xin Zhang and Martin Zinkevich. 2017. TFX: A TensorFlow-Based Production-Scale Machine Learning Platform. KDD 1387--1395."},{"key":"e_1_3_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.5555\/1577069.1755858"},{"key":"e_1_3_2_2_3_1","unstructured":"Eric Breck Neoklis Polyzotis Sudip Roy Steven Whang and Martin Zinkevich. 2019. Data Validation for Machine Learning. SysML. Eric Breck Neoklis Polyzotis Sudip Roy Steven Whang and Martin Zinkevich. 2019. Data Validation for Machine Learning. SysML."},{"key":"e_1_3_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939785"},{"key":"e_1_3_2_2_5_1","unstructured":"Francois Chollet et al. 2015. Keras tensorflow.org\/guide\/keras. Francois Chollet et al. 2015. Keras tensorflow.org\/guide\/keras."},{"key":"e_1_3_2_2_6_1","volume-title":"Steven Euijong Whang, and Neoklis Polyzotis","author":"Chung Yeounoh","year":"2018","unstructured":"Yeounoh Chung , Tim Kraska , Steven Euijong Whang, and Neoklis Polyzotis . 2018 . Slice finder: Automated data slicing for model interpretability. SysML. Yeounoh Chung, Tim Kraska, Steven Euijong Whang, and Neoklis Polyzotis. 2018. Slice finder: Automated data slicing for model interpretability. SysML."},{"key":"e_1_3_2_2_7_1","unstructured":"Matthias Feurer Aaron Klein Katharina Eggensperger Jost Springenberg Manuel Blum and Frank Hutter. 2015. Efficient and robust automated machine learning. NeurIPS 2962--2970. Matthias Feurer Aaron Klein Katharina Eggensperger Jost Springenberg Manuel Blum and Frank Hutter. 2015. Efficient and robust automated machine learning. NeurIPS 2962--2970."},{"key":"e_1_3_2_2_8_1","unstructured":"Joseph M Hellerstein. 2008. Quantitative data cleaning for large databases. United Nations Economic Commission for Europe (UNECE). Joseph M Hellerstein. 2008. Quantitative data cleaning for large databases. United Nations Economic Commission for Europe (UNECE)."},{"key":"e_1_3_2_2_9_1","doi-asserted-by":"crossref","unstructured":"Jiayuan Huang Arthur Gretton Karsten M Borgwardt Bernhard Sch\u00f6lkopf and Alex J Smola. 2007. Correcting sample selection bias by unlabeled data. NeurIPS 601--608. Jiayuan Huang Arthur Gretton Karsten M Borgwardt Bernhard Sch\u00f6lkopf and Alex J Smola. 2007. Correcting sample selection bias by unlabeled data. NeurIPS 601--608.","DOI":"10.7551\/mitpress\/7503.003.0080"},{"key":"e_1_3_2_2_10_1","volume-title":"Automated Sanity Checking for ML Data Sets. ML Systems Workshop @ NeurIPS.","author":"Hynes Nick","year":"2017","unstructured":"Nick Hynes , D Sculley , and Michael Terry . 2017 . The Data Linter: Lightweight , Automated Sanity Checking for ML Data Sets. ML Systems Workshop @ NeurIPS. Nick Hynes, D Sculley, and Michael Terry. 2017. The Data Linter: Lightweight, Automated Sanity Checking for ML Data Sets. ML Systems Workshop @ NeurIPS."},{"key":"e_1_3_2_2_11_1","unstructured":"Haifeng Jin Qingquan Song and Xia Hu. 2018. Auto-Keras: Efficient Neural Architecture Search with Network Morphism. [arXiv]cs.LG\/cs.LG\/1806.10282 Haifeng Jin Qingquan Song and Xia Hu. 2018. Auto-Keras: Efficient Neural Architecture Search with Network Morphism. [arXiv]cs.LG\/cs.LG\/1806.10282"},{"key":"e_1_3_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/2935694.2935698"},{"key":"e_1_3_2_2_13_1","unstructured":"Zachary C Lipton Yu-Xiang Wang and Alex Smola. 2018. Detecting and Correcting for Label Shift with Black Box Predictors. ICML. Zachary C Lipton Yu-Xiang Wang and Alex Smola. 2018. Detecting and Correcting for Label Shift with Black Box Predictors. ICML."},{"key":"e_1_3_2_2_14_1","first-page":"51","article-title":"Data structures for statistical computing in python","volume":"445","author":"McKinney Wes","year":"2010","unstructured":"Wes McKinney 2010 . Data structures for statistical computing in python . Python in Science , Vol. 445 , 51 -- 56 . Wes McKinney et almbox. 2010. Data structures for statistical computing in python. Python in Science, Vol. 445, 51--56.","journal-title":"Python in Science"},{"key":"e_1_3_2_2_15_1","volume-title":"Tensorfuzz: Debugging neural networks with coverage-guided fuzzing. arXiv preprint arXiv:1807.10875.","author":"Odena Augustus","year":"2018","unstructured":"Augustus Odena and Ian Goodfellow . 2018 . Tensorfuzz: Debugging neural networks with coverage-guided fuzzing. arXiv preprint arXiv:1807.10875. Augustus Odena and Ian Goodfellow. 2018. Tensorfuzz: Debugging neural networks with coverage-guided fuzzing. arXiv preprint arXiv:1807.10875."},{"key":"e_1_3_2_2_16_1","volume-title":"La Creis Kidd, and Jason H. Moore","author":"Olson Randal S.","year":"2016","unstructured":"Randal S. Olson , Ryan J. Urbanowicz , Peter C. Andrews , Nicole A. Lavender , La Creis Kidd, and Jason H. Moore . 2016 . Automating Biomedical Data Science Through Tree-Based Pipeline Optimization. EvoApplications . Randal S. Olson, Ryan J. Urbanowicz, Peter C. Andrews, Nicole A. Lavender, La Creis Kidd, and Jason H. Moore. 2016. Automating Biomedical Data Science Through Tree-Based Pipeline Optimization. EvoApplications."},{"key":"e_1_3_2_2_17_1","volume-title":"et almbox","author":"Pedregosa Fabian","year":"2011","unstructured":"Fabian Pedregosa , Ga\u00ebl Varoquaux , Alexandre Gramfort , Vincent Michel , Bertrand Thirion , Olivier Grisel , Mathieu Blondel , Peter Prettenhofer , Ron Weiss , Vincent Dubourg , et almbox . 2011 . Scikit-learn : Machine learning in Python. JMLR , Vol. 12 , Oct, 2825--2830. Fabian Pedregosa, Ga\u00ebl Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et almbox. 2011. Scikit-learn: Machine learning in Python. JMLR, Vol. 12, Oct, 2825--2830."},{"key":"e_1_3_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132747.3132785"},{"key":"e_1_3_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3299887.3299891"},{"key":"e_1_3_2_2_20_1","volume-title":"Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift. NeurIPS.","author":"Rabanser Stephan","year":"2019","unstructured":"Stephan Rabanser , Stephan G\u00fcnnemann , and Zachary C Lipton . 2019 . Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift. NeurIPS. Stephan Rabanser, Stephan G\u00fcnnemann, and Zachary C Lipton. 2019. Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift. NeurIPS."},{"key":"e_1_3_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3328519.3329126"},{"key":"e_1_3_2_2_22_1","volume-title":"IEEE Data Engineering Bulletin","volume":"41","author":"Schelter Sebastian","year":"2018","unstructured":"Sebastian Schelter , Felix Biessmann , Tim Januschowski , David Salinas , Stephan Seufert , and Gyuri Szarvas . 2018 a. On Challenges in Machine Learning Model Management . IEEE Data Engineering Bulletin , Vol. 41 . Sebastian Schelter, Felix Biessmann, Tim Januschowski, David Salinas, Stephan Seufert, and Gyuri Szarvas. 2018a. On Challenges in Machine Learning Model Management. IEEE Data Engineering Bulletin, Vol. 41."},{"key":"e_1_3_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.14778\/3229863.3229867"},{"key":"e_1_3_2_2_24_1","doi-asserted-by":"crossref","unstructured":"S. Schelter J. Soto V. Markl D. Burdick B. Reinwald and A. Evfimievski. 2015. Efficient sample generation for scalable meta learning. ICDE 1191--1202. S. Schelter J. Soto V. Markl D. Burdick B. Reinwald and A. Evfimievski. 2015. Efficient sample generation for scalable meta learning. ICDE 1191--1202.","DOI":"10.1109\/ICDE.2015.7113367"},{"key":"e_1_3_2_2_25_1","unstructured":"D Sculley Gary Holt Daniel Golovin Eugene Davydov Todd Phillips Dietmar Ebner Vinay Chaudhary Michael Young Jean-Francois Crespo and Dan Dennison. 2015. Hidden technical debt in machine learning systems. NeurIPS 2503--2511. D Sculley Gary Holt Daniel Golovin Eugene Davydov Todd Phillips Dietmar Ebner Vinay Chaudhary Michael Young Jean-Francois Crespo and Dan Dennison. 2015. Hidden technical debt in machine learning systems. NeurIPS 2503--2511."},{"key":"e_1_3_2_2_27_1","volume-title":"Machine Learning in Non-Stationary Environments - Introduction to Covariate Shift Adaptation","author":"Sugiyama Masashi","unstructured":"Masashi Sugiyama and Motoaki Kawanabe . 2012. Machine Learning in Non-Stationary Environments - Introduction to Covariate Shift Adaptation . MIT Press . Masashi Sugiyama and Motoaki Kawanabe. 2012. Machine Learning in Non-Stationary Environments - Introduction to Covariate Shift Adaptation .MIT Press."},{"key":"e_1_3_2_2_28_1","volume-title":"et almbox","author":"Sugiyama Masashi","year":"2017","unstructured":"Masashi Sugiyama , Neil D Lawrence , Anton Schwaighofer , et almbox . 2017 . Dataset shift in machine learning .MIT Press . Masashi Sugiyama, Neil D Lawrence, Anton Schwaighofer, et almbox. 2017. Dataset shift in machine learning .MIT Press."},{"key":"e_1_3_2_2_29_1","article-title":"Finding Stationary Subspaces in Multivariate Time","volume":"103","author":"von B\u00fcnau Paul","year":"2009","unstructured":"Paul von B\u00fcnau , Frank C. Meinecke , Franz C. Kir\u00e1ly , and Klaus-Robert M\u00fcller . 2009 . Finding Stationary Subspaces in Multivariate Time Series. Phys. Rev. Lett. , Vol. 103. Issue 21. Paul von B\u00fcnau, Frank C. Meinecke, Franz C. Kir\u00e1ly, and Klaus-Robert M\u00fcller. 2009. Finding Stationary Subspaces in Multivariate Time Series. Phys. Rev. Lett., Vol. 103. Issue 21.","journal-title":"Series. Phys. Rev. Lett."},{"key":"e_1_3_2_2_30_1","doi-asserted-by":"crossref","unstructured":"Pei Wang and Yeye He. 2019. Uni-Detect: A Unified Approach to Automated Error Detection in Tables. SIGMOD 811--828. Pei Wang and Yeye He. 2019. Uni-Detect: A Unified Approach to Automated Error Detection in Tables. SIGMOD 811--828.","DOI":"10.1145\/3299869.3319855"},{"key":"e_1_3_2_2_31_1","first-page":"819","article-title":"Domain Adaptation under Target and Conditional Shift","volume":"28","author":"Zhang Kun","year":"2013","unstructured":"Kun Zhang , Bernhard Sch\u00f6lkopf , Krikamol Muandet , and Zhikun Wang . 2013 . Domain Adaptation under Target and Conditional Shift . ICML , Vol. 28 , 819 -- 827 . Kun Zhang, Bernhard Sch\u00f6lkopf, Krikamol Muandet, and Zhikun Wang. 2013. Domain Adaptation under Target and Conditional Shift. ICML, Vol. 28, 819--827.","journal-title":"ICML"}],"event":{"name":"SIGMOD\/PODS '20: International Conference on Management of Data","location":"Portland OR USA","acronym":"SIGMOD\/PODS '20","sponsor":["SIGMOD ACM Special Interest Group on Management of Data"]},"container-title":["Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3318464.3380604","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3318464.3380604","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:38:23Z","timestamp":1750199903000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3318464.3380604"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,5,31]]},"references-count":30,"alternative-id":["10.1145\/3318464.3380604","10.1145\/3318464"],"URL":"https:\/\/doi.org\/10.1145\/3318464.3380604","relation":{},"subject":[],"published":{"date-parts":[[2020,5,31]]},"assertion":[{"value":"2020-05-31","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}