{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,21]],"date-time":"2026-03-21T03:37:55Z","timestamp":1774064275150,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":42,"publisher":"ACM","license":[{"start":{"date-parts":[[2020,8,20]],"date-time":"2020-08-20T00:00:00Z","timestamp":1597881600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2020,8,23]]},"DOI":"10.1145\/3394486.3403379","type":"proceedings-article","created":{"date-parts":[[2020,8,20]],"date-time":"2020-08-20T23:04:00Z","timestamp":1597964640000},"page":"3272-3280","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":26,"title":["An Empirical Analysis of Backward Compatibility in Machine Learning Systems"],"prefix":"10.1145","author":[{"given":"Megha","family":"Srivastava","sequence":"first","affiliation":[{"name":"Microsoft Research, Redmond, WA, USA"}]},{"given":"Besmira","family":"Nushi","sequence":"additional","affiliation":[{"name":"Microsoft Research, Redmond, WA, USA"}]},{"given":"Ece","family":"Kamar","sequence":"additional","affiliation":[{"name":"Microsoft Research, Redmond, WA, USA"}]},{"given":"Shital","family":"Shah","sequence":"additional","affiliation":[{"name":"Microsoft Research, Redmond, WA, USA"}]},{"given":"Eric","family":"Horvitz","sequence":"additional","affiliation":[{"name":"Microsoft Research, Redmond, WA, USA"}]}],"member":"320","published-online":{"date-parts":[[2020,8,20]]},"reference":[{"key":"e_1_3_2_1_1_1","volume-title":"Software engineering for machine learning: A case study","author":"Amershi Saleema","unstructured":"Saleema Amershi , Andrew Begel , Christian Bird , Robert DeLine , Harald Gall , Ece Kamar , Nachiappan Nagappan , Besmira Nushi , and Thomas Zimmermann . 2019. Software engineering for machine learning: A case study . In ICSE-SEIP. IEEE. Saleema Amershi, Andrew Begel, Christian Bird, Robert DeLine, Harald Gall, Ece Kamar, Nachiappan Nagappan, Besmira Nushi, and Thomas Zimmermann. 2019. Software engineering for machine learning: A case study. In ICSE-SEIP. IEEE."},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"crossref","unstructured":"Sean Andrist Dan Bohus Ece Kamar and Eric Horvitz. 2017. What went wrong and why? diagnosing situated interaction failures in the wild. In ICSR.  Sean Andrist Dan Bohus Ece Kamar and Eric Horvitz. 2017. What went wrong and why? diagnosing situated interaction failures in the wild. In ICSR.","DOI":"10.1007\/978-3-319-70022-9_29"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v33i01.33012429"},{"key":"e_1_3_2_1_4_1","unstructured":"Jan Bosch. 2009. From software product lines to software ecosystems. In SPLC.  Jan Bosch. 2009. From software product lines to software ecosystems. In SPLC."},{"key":"e_1_3_2_1_5_1","volume-title":"Not-so-supervised: a survey of semi-supervised, multi-instance, and transfer learning in medical image analysis. Medical image analysis","author":"Cheplygina Veronika","year":"2019","unstructured":"Veronika Cheplygina , Marleen de Bruijne , and Josien PW Pluim . 2019. Not-so-supervised: a survey of semi-supervised, multi-instance, and transfer learning in medical image analysis. Medical image analysis , Vol. 54 ( 2019 ), 280--296. Veronika Cheplygina, Marleen de Bruijne, and Josien PW Pluim. 2019. Not-so-supervised: a survey of semi-supervised, multi-instance, and transfer learning in medical image analysis. Medical image analysis, Vol. 54 (2019), 280--296."},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/JBHI.2017.2769800"},{"key":"e_1_3_2_1_7_1","volume-title":"Ki Hyun Tae, and Steven Euijong Whang","author":"Chung Yeounoh","year":"2019","unstructured":"Yeounoh Chung , Tim Kraska , Neoklis Polyzotis , Ki Hyun Tae, and Steven Euijong Whang . 2019 . Slice finder: Automated data slicing for model validation. In ICDE. Yeounoh Chung, Tim Kraska, Neoklis Polyzotis, Ki Hyun Tae, and Steven Euijong Whang. 2019. Slice finder: Automated data slicing for model validation. In ICDE."},{"key":"e_1_3_2_1_8_1","volume-title":"SIGMOD,, Kenneth A","author":"Dallachiesa Michele","unstructured":"Michele Dallachiesa , Amr Ebaid , Ahmed Eldawy , Ahmed K. Elmagarmid , Ihab F. Ilyas , Mourad Ouzzani , and Nan Tang . 2013. NADEEF: a commodity data cleaning system . In SIGMOD,, Kenneth A . Ross, Divesh Srivastava, and Dimitris Papadias (Eds.). ACM , 541--552. https:\/\/doi.org\/10.1145\/2463676.2465327 10.1145\/2463676.2465327 Michele Dallachiesa, Amr Ebaid, Ahmed Eldawy, Ahmed K. Elmagarmid, Ihab F. Ilyas, Mourad Ouzzani, and Nan Tang. 2013. NADEEF: a commodity data cleaning system. In SIGMOD,, Kenneth A. Ross, Divesh Srivastava, and Dimitris Papadias (Eds.). ACM, 541--552. https:\/\/doi.org\/10.1145\/2463676.2465327"},{"key":"e_1_3_2_1_9_1","unstructured":"T. E. de Campos B. R. Babu and M. Varma. 2009. Character recognition in natural images. In VISAPP.  T. E. de Campos B. R. Babu and M. Varma. 2009. Character recognition in natural images. In VISAPP."},{"key":"e_1_3_2_1_10_1","unstructured":"FICO. 2018 (accessed February 13 2020). Explainable machine learning challenge. https:\/\/community.fico.com\/s\/explainable-machine-learning-challenge?tabset-3158a=4fbc8.  FICO. 2018 (accessed February 13 2020). Explainable machine learning challenge. https:\/\/community.fico.com\/s\/explainable-machine-learning-challenge?tabset-3158a=4fbc8."},{"key":"e_1_3_2_1_11_1","volume-title":"Classification in the presence of label noise: a survey","author":"Fr\u00e9nay Beno\u00eet","year":"2013","unstructured":"Beno\u00eet Fr\u00e9nay and Michel Verleysen . 2013. Classification in the presence of label noise: a survey . IEEE transactions on neural networks and learning systems, Vol. 25 , 5 ( 2013 ), 845--869. Beno\u00eet Fr\u00e9nay and Michel Verleysen. 2013. Classification in the presence of label noise: a survey. IEEE transactions on neural networks and learning systems, Vol. 25, 5 (2013), 845--869."},{"key":"e_1_3_2_1_12_1","volume-title":"Catastrophic forgetting in connectionist networks. Trends in cognitive sciences","author":"French Robert M","year":"1999","unstructured":"Robert M French . 1999. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences , Vol. 3 , 4 ( 1999 ), 128--135. Robert M French. 1999. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, Vol. 3, 4 (1999), 128--135."},{"key":"e_1_3_2_1_13_1","volume-title":"An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211","author":"Goodfellow Ian J","year":"2013","unstructured":"Ian J Goodfellow , Mehdi Mirza , Da Xiao , Aaron Courville , and Yoshua Bengio . 2013. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211 ( 2013 ). Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. 2013. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211 (2013)."},{"key":"e_1_3_2_1_14_1","unstructured":"Dan Hendrycks and Thomas Dietterich. 2019. Benchmarking neural network robustness to common corruptions and perturbations. ICLR.  Dan Hendrycks and Thomas Dietterich. 2019. Benchmarking neural network robustness to common corruptions and perturbations. ICLR."},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/72.105415"},{"key":"e_1_3_2_1_16_1","unstructured":"Lu Jiang Zhengyuan Zhou Thomas Leung Li-Jia Li and Li Fei-Fei. 2017. MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels. In ICML .  Lu Jiang Zhengyuan Zhou Thomas Leung Li-Jia Li and Li Fei-Fei. 2017. MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels. In ICML ."},{"key":"e_1_3_2_1_17_1","unstructured":"Andrej Karpathy. 2017. Software 2.0. https:\/\/medium.com\/@karpathy\/software-2-0-a64152b37c35  Andrej Karpathy. 2017. Software 2.0. https:\/\/medium.com\/@karpathy\/software-2-0-a64152b37c35"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"crossref","unstructured":"Ronald Kemker Marc McClure Angelina Abitino Tyler L Hayes and Christopher Kanan. 2018. Measuring catastrophic forgetting in neural networks. In AAAI.  Ronald Kemker Marc McClure Angelina Abitino Tyler L Hayes and Christopher Kanan. 2018. Measuring catastrophic forgetting in neural networks. In AAAI.","DOI":"10.1609\/aaai.v32i1.11651"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"crossref","unstructured":"Jonathan Krause Benjamin Sapp Andrew Howard Howard Zhou Alexander Toshev Tom Duerig James Philbin and Li Fei-Fei. 2016. The unreasonable effectiveness of noisy data for fine-grained recognition. In ECCV.  Jonathan Krause Benjamin Sapp Andrew Howard Howard Zhou Alexander Toshev Tom Duerig James Philbin and Li Fei-Fei. 2016. The unreasonable effectiveness of noisy data for fine-grained recognition. In ECCV.","DOI":"10.1007\/978-3-319-46487-9_19"},{"key":"e_1_3_2_1_20_1","unstructured":"Alex Krizhevsky. 2009. Learning multiple layers of features from tiny images. (2009).  Alex Krizhevsky. 2009. Learning multiple layers of features from tiny images. (2009)."},{"key":"e_1_3_2_1_21_1","volume-title":"Patent 0 298","author":"Kruppa Robert William","year":"2008","unstructured":"Robert William Kruppa and Ravinder Prakash . U. S. Patent 0 298 668, Dec. 2008 . Method for fraud detection using multiple scan technologies. Robert William Kruppa and Ravinder Prakash. U.S. Patent 0 298 668, Dec. 2008. Method for fraud detection using multiple scan technologies."},{"key":"e_1_3_2_1_22_1","unstructured":"Alexey Kurakin Ian J. Goodfellow and Samy Bengio. 2017. Adversarial Machine Learning at Scale. ICLR.  Alexey Kurakin Ian J. Goodfellow and Samy Bengio. 2017. Adversarial Machine Learning at Scale. ICLR."},{"key":"e_1_3_2_1_23_1","volume-title":"Human computation. Synthesis lectures on artificial intelligence and machine learning","author":"Law Edith","year":"2011","unstructured":"Edith Law and Luis von Ahn . 2011. Human computation. Synthesis lectures on artificial intelligence and machine learning , Vol. 5 , 3 ( 2011 ), 1--121. Edith Law and Luis von Ahn. 2011. Human computation. Synthesis lectures on artificial intelligence and machine learning, Vol. 5, 3 (2011), 1--121."},{"key":"e_1_3_2_1_24_1","volume-title":"MNIST handwritten digit database. ATT Labs [Online]. Available: http:\/\/yann.lecun.com\/exdb\/mnist","author":"LeCun Yann","year":"2010","unstructured":"Yann LeCun , Corinna Cortes , and CJ Burges . 2010. MNIST handwritten digit database. ATT Labs [Online]. Available: http:\/\/yann.lecun.com\/exdb\/mnist , Vol. 2 ( 2010 ). Yann LeCun, Corinna Cortes, and CJ Burges. 2010. MNIST handwritten digit database. ATT Labs [Online]. Available: http:\/\/yann.lecun.com\/exdb\/mnist, Vol. 2 (2010)."},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.211"},{"key":"e_1_3_2_1_26_1","unstructured":"Andrew L. Maas Raymond E. Daly Peter T. Pham Dan Huang Andrew Y. Ng and Christopher Potts. 2011. Learning Word Vectors for Sentiment Analysis. In ACL.  Andrew L. Maas Raymond E. Daly Peter T. Pham Dan Huang Andrew Y. Ng and Christopher Potts. 2011. Learning Word Vectors for Sentiment Analysis. In ACL."},{"key":"e_1_3_2_1_27_1","unstructured":"Aleksander Madry Aleksandar Makelov Ludwig Schmidt Dimitris Tsipras and Adrian Vladu. 2017. Towards Deep Learning Models Resistant to Adversarial Attacks. ICLR.  Aleksander Madry Aleksandar Makelov Ludwig Schmidt Dimitris Tsipras and Adrian Vladu. 2017. Towards Deep Learning Models Resistant to Adversarial Attacks. ICLR."},{"key":"e_1_3_2_1_28_1","volume-title":"Psychology of learning and motivation.","author":"McCloskey Michael","unstructured":"Michael McCloskey and Neal J Cohen . 1989. Catastrophic interference in connectionist networks: The sequential learning problem . In Psychology of learning and motivation. Vol. 24 . Elsevier , 109--165. Michael McCloskey and Neal J Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation. Vol. 24. Elsevier, 109--165."},{"key":"e_1_3_2_1_29_1","volume-title":"Improving Speech Recognition and Keyword Search for Low Resource Languages Using Web Data. ISCA","author":"Mendels Gideon","year":"2015","unstructured":"Gideon Mendels , Erica Cooper , Victor Soto , Julia Hirschberg , Mark Gales , Kate Knill , Anton Ragni , and Haipeng Wang . 2015. Improving Speech Recognition and Keyword Search for Low Resource Languages Using Web Data. ISCA ( 2015 ). Gideon Mendels, Erica Cooper, Victor Soto, Julia Hirschberg, Mark Gales, Kate Knill, Anton Ragni, and Haipeng Wang. 2015. Improving Speech Recognition and Keyword Search for Low Resource Languages Using Web Data. ISCA (2015)."},{"key":"e_1_3_2_1_30_1","unstructured":"Nagarajan Natarajan Inderjit S. Dhillon Pradeep Ravikumar and Ambuj Tewari. 2013. Learning with Noisy Labels. In NeurIPS .  Nagarajan Natarajan Inderjit S. Dhillon Pradeep Ravikumar and Ambuj Tewari. 2013. Learning with Noisy Labels. In NeurIPS ."},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"crossref","unstructured":"Besmira Nushi Ece Kamar and Eric Horvitz. 2018. Towards accountable ai: Hybrid human-machine analyses for characterizing system failure. In HCOMP.  Besmira Nushi Ece Kamar and Eric Horvitz. 2018. Towards accountable ai: Hybrid human-machine analyses for characterizing system failure. In HCOMP.","DOI":"10.1609\/hcomp.v6i1.13337"},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"crossref","unstructured":"Besmira Nushi Ece Kamar Eric Horvitz and Donald Kossmann. 2017. On human intellect and machine failures: Troubleshooting integrative machine learning systems. In AAAI.  Besmira Nushi Ece Kamar Eric Horvitz and Donald Kossmann. 2017. On human intellect and machine failures: Troubleshooting integrative machine learning systems. In AAAI.","DOI":"10.1609\/aaai.v31i1.10633"},{"key":"e_1_3_2_1_33_1","first-page":"3","article-title":"Data cleaning: Problems and current approaches","volume":"23","author":"Rahm Erhard","year":"2000","unstructured":"Erhard Rahm and Hong Hai Do . 2000 . Data cleaning: Problems and current approaches . IEEE Data Eng. Bull. , Vol. 23 , 4 (2000), 3 -- 13 . Erhard Rahm and Hong Hai Do. 2000. Data cleaning: Problems and current approaches. IEEE Data Eng. Bull., Vol. 23, 4 (2000), 3--13.","journal-title":"IEEE Data Eng. Bull."},{"key":"e_1_3_2_1_34_1","volume-title":"Snorkel: Rapid training data creation with weak supervision. The VLDB Journal","author":"Ratner Alexander","year":"2019","unstructured":"Alexander Ratner , Stephen H Bach , Henry Ehrenberg , Jason Fries , Sen Wu , and Christopher R\u00e9 . 2019 . Snorkel: Rapid training data creation with weak supervision. The VLDB Journal (2019), 1--22. Alexander Ratner, Stephen H Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher R\u00e9. 2019. Snorkel: Rapid training data creation with weak supervision. The VLDB Journal (2019), 1--22."},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.14778\/3137628.3137631"},{"key":"e_1_3_2_1_36_1","unstructured":"David Sculley Gary Holt Daniel Golovin Eugene Davydov Todd Phillips Dietmar Ebner Vinay Chaudhary Michael Young Jean-Francois Crespo and Dan Dennison. 2015. Hidden technical debt in machine learning systems. In NeurIPS.  David Sculley Gary Holt Daniel Golovin Eugene Davydov Todd Phillips Dietmar Ebner Vinay Chaudhary Michael Young Jean-Francois Crespo and Dan Dennison. 2015. Hidden technical debt in machine learning systems. In NeurIPS."},{"key":"e_1_3_2_1_37_1","volume-title":"Training Convolutional Networks with Noisy Labels. In ICLR","author":"Sukhbaatar Sainbayar","year":"2015","unstructured":"Sainbayar Sukhbaatar , Joan Bruna , Manohar Paluri , Lubomir D. Bourdev , and Rob Fergus . 2015 . Training Convolutional Networks with Noisy Labels. In ICLR 2015. Sainbayar Sukhbaatar, Joan Bruna, Manohar Paluri, Lubomir D. Bourdev, and Rob Fergus. 2015. Training Convolutional Networks with Noisy Labels. In ICLR 2015."},{"key":"e_1_3_2_1_38_1","volume-title":"Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon.","author":"Toneva Mariya","year":"2019","unstructured":"Mariya Toneva , Alessandro Sordoni , Remi Tachet des Combes , Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. 2019 . An empirical study of example forgetting during deep neural network learning. ICLR. Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. 2019. An empirical study of example forgetting during deep neural network learning. ICLR."},{"key":"e_1_3_2_1_39_1","volume-title":"CAPTCHA: Using hard AI problems for security","author":"Ahn Luis Von","year":"2003","unstructured":"Luis Von Ahn , Manuel Blum , Nicholas J Hopper , and John Langford . 2003 . CAPTCHA: Using hard AI problems for security . In EUROCRYPT. Springer . Luis Von Ahn, Manuel Blum, Nicholas J Hopper, and John Langford. 2003. CAPTCHA: Using hard AI problems for security. In EUROCRYPT. Springer."},{"key":"e_1_3_2_1_40_1","unstructured":"Junfeng Wen Chun-Nam Yu and Russell Greiner. 2014. Robust Learning under Uncertain Test Distributions: Relating Covariate Shift to Model Misspecification. ICML.  Junfeng Wen Chun-Nam Yu and Russell Greiner. 2014. Robust Learning under Uncertain Test Distributions: Relating Covariate Shift to Model Misspecification. ICML."},{"key":"e_1_3_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2018.2864499"},{"key":"e_1_3_2_1_42_1","doi-asserted-by":"crossref","unstructured":"Tianyi Zhang Cuiyun Gao Lei Ma Michael R Lyu and Miryung Kim. 2019. An empirical study of common challenges in developing deep learning applications. In ISSRE.  Tianyi Zhang Cuiyun Gao Lei Ma Michael R Lyu and Miryung Kim. 2019. An empirical study of common challenges in developing deep learning applications. In ISSRE.","DOI":"10.1109\/ISSRE.2019.00020"}],"event":{"name":"KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining","location":"Virtual Event CA USA","acronym":"KDD '20","sponsor":["SIGMOD ACM Special Interest Group on Management of Data","SIGKDD ACM Special Interest Group on Knowledge Discovery in Data"]},"container-title":["Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3394486.3403379","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3394486.3403379","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T21:31:30Z","timestamp":1750195890000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3394486.3403379"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,8,20]]},"references-count":42,"alternative-id":["10.1145\/3394486.3403379","10.1145\/3394486"],"URL":"https:\/\/doi.org\/10.1145\/3394486.3403379","relation":{},"subject":[],"published":{"date-parts":[[2020,8,20]]},"assertion":[{"value":"2020-08-20","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}