{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,28]],"date-time":"2026-10-28T08:32:38Z","timestamp":1793176358971,"version":"4.5.0"},"reference-count":74,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2018,12,11]],"date-time":"2018-12-11T00:00:00Z","timestamp":1544486400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGMOD Rec."],"published-print":{"date-parts":[[2018,12,11]]},"abstract":"<jats:p>Machine learning has become an essential tool for gleaning knowledge from data and tackling a diverse set of computationally hard tasks. However, the accuracy of a machine learned model is deeply tied to the data that it is trained on. Designing and building robust processes and tools that make it easier to analyze, validate, and transform data that is fed into large-scale machine learning systems poses data management challenges. Drawn from our experience in developing data-centric infrastructure for a production machine learning platform at Google, we summarize some of the interesting research challenges that we encountered, and survey some of the relevant literature from the data management and machine learning communities. Specifically, we explore challenges in three main areas of focus - data understanding, data validation and cleaning, and data preparation. In each of these areas, we try to explore how different constraints are imposed on the solutions depending on where in the lifecycle of a model the problems are encountered and who encounters them.<\/jats:p>","DOI":"10.1145\/3299887.3299891","type":"journal-article","created":{"date-parts":[[2018,12,12]],"date-time":"2018-12-12T07:49:32Z","timestamp":1544600972000},"page":"17-28","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":204,"title":["Data Lifecycle Challenges in Production Machine Learning"],"prefix":"10.1145","volume":"47","author":[{"given":"Neoklis","family":"Polyzotis","sequence":"first","affiliation":[{"name":"Google Research, Mountain View, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sudip","family":"Roy","sequence":"additional","affiliation":[{"name":"Google Research, Mountain View, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Steven Euijong","family":"Whang","sequence":"additional","affiliation":[{"name":"KAIST, Daejeon, South Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Martin","family":"Zinkevich","sequence":"additional","affiliation":[{"name":"Google Research, Mountain View, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2018,12,11]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"Deep learning for detection of diabetic eye disease. https:\/\/research.googleblog.com\/2016\/11\/ deep-learning-for-detection-of-diabetic.html.  Deep learning for detection of diabetic eye disease. https:\/\/research.googleblog.com\/2016\/11\/ deep-learning-for-detection-of-diabetic.html."},{"key":"e_1_2_1_2_1","unstructured":"Kaggle. https:\/\/www.kaggle.com\/.  Kaggle. https:\/\/www.kaggle.com\/."},{"key":"e_1_2_1_3_1","unstructured":"Keras. https:\/\/keras.io\/.  Keras. https:\/\/keras.io\/."},{"key":"e_1_2_1_4_1","unstructured":"Mxnet. https:\/\/mxnet.incubator.apache.org\/.  Mxnet. https:\/\/mxnet.incubator.apache.org\/."},{"key":"e_1_2_1_5_1","unstructured":"Tensorflow. https:\/\/www.tensorflow.org\/.  Tensorflow. https:\/\/www.tensorflow.org\/."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2465351.2465355"},{"key":"e_1_2_1_7_1","volume-title":"CIDR","author":"Anderson M. R.","year":"2013","unstructured":"M. R. Anderson , D. Antenucci , V. Bittorf , M. Burgess , M. J. Cafarella , A. Kumar , F. Niu , Y. Park , C. R\u00b4e , and C. Zhang . Brainwash: A data system for feature engineering . In CIDR , 2013 . M. R. Anderson, D. Antenucci, V. Bittorf, M. Burgess, M. J. Cafarella, A. Kumar, F. Niu, Y. Park, C. R\u00b4e, and C. Zhang. Brainwash: A data system for feature engineering. In CIDR, 2013."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2016.7498272"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3035918.3035928"},{"key":"e_1_2_1_10_1","volume-title":"Detection of Abrupt Changes: Theory and Application","author":"Basseville M.","year":"1993","unstructured":"M. Basseville and I. V. Nikiforov . Detection of Abrupt Changes: Theory and Application . Prentice-Hall, Inc. , 1993 . M. Basseville and I. V. Nikiforov. Detection of Abrupt Changes: Theory and Application. Prentice-Hall, Inc., 1993."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3097983.3098021"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2013.50"},{"key":"e_1_2_1_13_1","volume-title":"Datahub: Collaborative data science & dataset version management at scale. CoRR, abs\/1409.0798","author":"Bhardwaj A. P.","year":"2014","unstructured":"A. P. Bhardwaj , S. Bhattacherjee , A. Chavan , A. Deshpande , A. J. Elmore , S. Madden , and A. G. Parameswaran . Datahub: Collaborative data science & dataset version management at scale. CoRR, abs\/1409.0798 , 2014 . A. P. Bhardwaj, S. Bhattacherjee, A. Chavan, A. Deshpande, A. J. Elmore, S. Madden, and A. G. Parameswaran. Datahub: Collaborative data science & dataset version management at scale. CoRR, abs\/1409.0798, 2014."},{"key":"e_1_2_1_14_1","volume-title":"CIDR","author":"Binnig C.","year":"2017","unstructured":"C. Binnig , L. D. Stefani , T. Kraska , E. Upfal , E. Zgraggen , and Z. Zhao . Toward sustainable insights, or why polygamy is bad for you . In CIDR , 2017 . C. Binnig, L. D. Stefani, T. Kraska, E. Upfal, E. Zgraggen, and Z. Zhao. Toward sustainable insights, or why polygamy is bad for you. In CIDR, 2017."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.14778\/3007263.3007279"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/1066157.1066175"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.14778\/3137765.3137775"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1214\/14-AOAS788"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.14778\/1453856.1453916"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3035918.3058740"},{"key":"e_1_2_1_21_1","first-page":"982","volume-title":"PVLDB","author":"Chen B.-C.","year":"2005","unstructured":"B.-C. Chen , L. Chen , Y. Lin , and R. Ramakrishnan . Prediction cubes . In PVLDB , pages 982 -- 993 , 2005 . B.-C. Chen, L. Chen, Y. Lin, and R. Ramakrishnan. Prediction cubes. In PVLDB, pages 982--993, 2005."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2011.5767833"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2013.6544847"},{"key":"e_1_2_1_24_1","volume-title":"CIDR","author":"Crankshaw D.","year":"2015","unstructured":"D. Crankshaw , P. Bailis , J. E. Gonzalez , H. Li , Z. Zhang , M. J. Franklin , A. Ghodsi , and M. I. Jordan . The missing piece in complex analytics: Low latency, scalable model management and serving with velox . In CIDR , 2015 . D. Crankshaw, P. Bailis, J. E. Gonzalez, H. Li, Z. Zhang, M. J. Franklin, A. Ghodsi, and M. I. Jordan. The missing piece in complex analytics: Low latency, scalable model management and serving with velox. In CIDR, 2015."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10619-014-7163-9"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/1553374.1553559"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.14778\/1454159.1454226"},{"key":"e_1_2_1_28_1","volume-title":"A roadmap for a rigorous science of interpretability. CoRR, abs\/1702.08608","author":"Doshi-Velez F.","year":"2017","unstructured":"F. Doshi-Velez and B. Kim . A roadmap for a rigorous science of interpretability. CoRR, abs\/1702.08608 , 2017 . F. Doshi-Velez and B. Kim. A roadmap for a rigorous science of interpretability. CoRR, abs\/1702.08608, 2017."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/2948674.2948675"},{"key":"e_1_2_1_30_1","first-page":"3","article-title":"On the probable error of a coefficient of correlation deduced from a small sample","volume":"1","author":"Fisher R. A.","year":"1921","unstructured":"R. A. Fisher . On the probable error of a coefficient of correlation deduced from a small sample . Metron , 1 : 3 -- 32 , 1921 . R. A. Fisher. On the probable error of a coefficient of correlation deduced from a small sample. Metron, 1:3--32, 1921.","journal-title":"Metron"},{"key":"e_1_2_1_31_1","doi-asserted-by":"crossref","unstructured":"R. A. Fisher. Statistical Methods for Research Workers pages 66--70. Springer New York 1992.  R. A. Fisher. Statistical Methods for Research Workers pages 66--70. Springer New York 1992.","DOI":"10.1007\/978-1-4612-4380-9_6"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1751-5823.2002.tb00178.x"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2013.6544854"},{"key":"e_1_2_1_34_1","volume-title":"Explaining and harnessing adversarial examples. CoRR, abs\/1412.6572","author":"Goodfellow I. J.","year":"2014","unstructured":"I. J. Goodfellow , J. Shlens , and C. Szegedy . Explaining and harnessing adversarial examples. CoRR, abs\/1412.6572 , 2014 . I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. CoRR, abs\/1412.6572, 2014."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/2882903.2903730"},{"key":"e_1_2_1_36_1","volume-title":"CIDR","author":"Hellerstein J. M.","year":"2017","unstructured":"J. M. Hellerstein , V. Sreekanti , J. E. Gonzales , Sudhansku, Arora, A. Bhattacharyya , S. Das , A. Dey , M. Donsky , G. Fierro , S. Nag , K. Ramachandran , C. She , E. Sun , C. Steinbach , and V. Subramanian . Establishing common ground with data context . In CIDR , 2017 . J. M. Hellerstein, V. Sreekanti, J. E. Gonzales, Sudhansku, Arora, A. Bhattacharyya, S. Das, A. Dey, M. Donsky, G. Fierro, S. Nag, K. Ramachandran, C. She, E. Sun, C. Steinbach, and V. Subramanian. Establishing common ground with data context. In CIDR, 2017."},{"key":"e_1_2_1_37_1","volume-title":"Beyond segmentation. Journal of Targeting, Measurement and Analysis for Marketing, (1):60--72","author":"Jenkinson A.","year":"1994","unstructured":"A. Jenkinson . Beyond segmentation. Journal of Targeting, Measurement and Analysis for Marketing, (1):60--72 , 1994 . A. Jenkinson. Beyond segmentation. Journal of Targeting, Measurement and Analysis for Marketing, (1):60--72, 1994."},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2016.7498300"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939502.2939503"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2747646"},{"key":"e_1_2_1_41_1","volume-title":"PP(99):1--1","author":"Kim M.","year":"2017","unstructured":"M. Kim , T. Zimmermann , R. DeLine , and A. Begel . Data scientists in software teams: State of the art and challenges. TSE , PP(99):1--1 , 2017 . M. Kim, T. Zimmermann, R. DeLine, and A. Begel. Data scientists in software teams: State of the art and challenges. TSE, PP(99):1--1, 2017."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/1514894.1514901"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.14778\/2536274.2536302"},{"key":"e_1_2_1_44_1","volume-title":"CIDR","author":"Kraska T.","year":"2013","unstructured":"T. Kraska , A. Talwalkar , J. C. Duchi , R. Griffith , M. J. Franklin , and M. I. Jordan . Mlbase: A distributed machine-learning system . In CIDR , 2013 . T. Kraska, A. Talwalkar, J. C. Duchi, R. Griffith, M. J. Franklin, and M. I. Jordan. Mlbase: A distributed machine-learning system. In CIDR, 2013."},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.14778\/2994509.2994514"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/2935694.2935698"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/2882903.2882952"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.14778\/1920841.1920886"},{"key":"e_1_2_1_49_1","volume-title":"Provdb: A system for lifecycle management of collaborative analysis workflows. CoRR, abs\/1610.04963","author":"Miao H.","year":"2016","unstructured":"H. Miao , A. Chavan , and A. Deshpande . Provdb: A system for lifecycle management of collaborative analysis workflows. CoRR, abs\/1610.04963 , 2016 . H. Miao, A. Chavan, and A. Deshpande. Provdb: A system for lifecycle management of collaborative analysis workflows. CoRR, abs\/1610.04963, 2016."},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2017.112"},{"key":"e_1_2_1_51_1","volume-title":"Efficient estimation of word representations in vector space. CoRR, abs\/1301.3781","author":"Mikolov T.","year":"2013","unstructured":"T. Mikolov , K. Chen , G. Corrado , and J. Dean . Efficient estimation of word representations in vector space. CoRR, abs\/1301.3781 , 2013 . T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of word representations in vector space. CoRR, abs\/1301.3781, 2013."},{"key":"e_1_2_1_52_1","unstructured":"F.\n      Olsson\n    . \n      A\n     literature survey of active machine learning in the context of natural language processing.\n   volume T\n  2009\n   of \n  SICS Technical Report\n  . \n  Swedish Institute of Computer Science 2009\n  .  F. Olsson. A literature survey of active machine learning in the context of natural language processing. volume T2009 of SICS Technical Report. Swedish Institute of Computer Science 2009."},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/1989323.1989459"},{"key":"e_1_2_1_54_1","volume-title":"CIDR","author":"Palkar S.","year":"2017","unstructured":"S. Palkar , J. J. Thomas , A. Shanbhag , M. Schwarzkopt , S. P. Amarasinghe , and M. Zaharia . A common runtime for high performance data analysis . In CIDR , 2017 . S. Palkar, J. J. Thomas, A. Shanbhag, M. Schwarzkopt, S. P. Amarasinghe, and M. Zaharia. A common runtime for high performance data analysis. In CIDR, 2017."},{"key":"e_1_2_1_55_1","doi-asserted-by":"crossref","unstructured":"K. Pearson. On the Criterion that a Given System of Deviations from the Probable in the Case of a Correlated System of Variables is Such that it Can be Reasonably Supposed to have Arisen from Random Sampling pages 11--28. Springer New York 1992.  K. Pearson. On the Criterion that a Given System of Deviations from the Probable in the Case of a Correlated System of Variables is Such that it Can be Reasonably Supposed to have Arisen from Random Sampling pages 11--28. Springer New York 1992.","DOI":"10.1007\/978-1-4612-4380-9_2"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.14778\/3157794.3157797"},{"key":"e_1_2_1_57_1","first-page":"3567","volume-title":"NIPS","author":"Ratner A. J.","year":"2016","unstructured":"A. J. Ratner , C. D. Sa , S. Wu , D. Selsam , and C. R\u00b4e . Data programming: Creating large training sets, quickly . In NIPS , pages 3567 -- 3575 , 2016 . A. J. Ratner, C. D. Sa, S. Wu, D. Selsam, and C. R\u00b4e. Data programming: Creating large training sets, quickly. In NIPS, pages 3567--3575, 2016."},{"issue":"3","key":"e_1_2_1_58_1","first-page":"26","article-title":"Feature engineering for knowledge base construction","volume":"37","author":"R\u00b4e C.","year":"2014","unstructured":"C. R\u00b4e , A. A. Sadeghian , Z. Shan , J. Shin , F. Wang , S. Wu , and C. Zhang . Feature engineering for knowledge base construction . IEEE Data Eng. Bull. , 37 ( 3 ): 26 -- 40 , 2014 . C. R\u00b4e, A. A. Sadeghian, Z. Shan, J. Shin, F. Wang, S. Wu, and C. Zhang. Feature engineering for knowledge base construction. IEEE Data Eng. Bull., 37(3):26--40, 2014.","journal-title":"IEEE Data Eng. Bull."},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1017\/S0269888913000039"},{"key":"e_1_2_1_60_1","first-page":"531","volume-title":"VLDB","author":"Sathe G.","year":"2001","unstructured":"G. Sathe and S. Sarawagi . Intelligent rollups in multidimensional olap data . In VLDB , pages 531 -- 540 , 2001 . G. Sathe and S. Sarawagi. Intelligent rollups in multidimensional olap data. In VLDB, pages 531--540, 2001."},{"key":"e_1_2_1_61_1","volume-title":"Workshop on ML Systems at NIPS 2017","author":"Schelter S.","year":"2017","unstructured":"S. Schelter , J.-H. Boese , J. Kirschnick , T. Klein , and S. Seufert . Automatically tracking metadata and provenance of machine learning experiments . In Workshop on ML Systems at NIPS 2017 , 2017 . S. Schelter, J.-H. Boese, J. Kirschnick, T. Klein, and S. Seufert. Automatically tracking metadata and provenance of machine learning experiments. In Workshop on ML Systems at NIPS 2017, 2017."},{"key":"e_1_2_1_62_1","first-page":"2503","volume-title":"NIPS","author":"Sculley D.","year":"2015","unstructured":"D. Sculley , G. Holt , D. Golovin , E. Davydov , T. Phillips , D. Ebner , V. Chaudhary , M. Young , J.-F. Crespo , and D. Dennison . Hidden technical debt in machine learning systems . In NIPS , pages 2503 -- 2511 , 2015 . D. Sculley, G. Holt, D. Golovin, E. Davydov, T. Phillips, D. Ebner, V. Chaudhary, M. Young, J.-F. Crespo, and D. Dennison. Hidden technical debt in machine learning systems. In NIPS, pages 2503--2511, 2015."},{"key":"e_1_2_1_63_1","volume-title":"Morgan & Claypool","author":"Settles B.","year":"2012","unstructured":"B. Settles . Active Learning . Synthesis Lectures on Artificial Intelligence and Machine Learning . Morgan & Claypool , 2012 . B. Settles. Active Learning. Synthesis Lectures on Artificial Intelligence and Machine Learning. Morgan & Claypool, 2012."},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.14778\/3157794.3157804"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/1401890.1401965"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.14778\/3025111.3025126"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2017.109"},{"key":"e_1_2_1_68_1","volume-title":"CIDR","author":"Stonebraker M.","year":"2013","unstructured":"M. Stonebraker , D. Bruckner , I. F. Ilyas , G. Beskales , M. Cherniack , S. B. Zdonik , A. Pagan , and S. Xu . Data curation at scale: The data tamer system . In CIDR , 2013 . M. Stonebraker, D. Bruckner, I. F. Ilyas, G. Beskales, M. Cherniack, S. B. Zdonik, A. Pagan, and S. Xu. Data curation at scale: The data tamer system. In CIDR, 2013."},{"key":"e_1_2_1_69_1","volume-title":"CIDR","author":"Vartak M.","year":"2017","unstructured":"M. Vartak . MODELDB : A system for machine learning model management . In CIDR , 2017 . M. Vartak. MODELDB: A system for machine learning model management. In CIDR, 2017."},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.14778\/2831360.2831371"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2014.6816655"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2750549"},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1145\/2877204"},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.1145\/3035918.3064019"}],"container-title":["ACM SIGMOD Record"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3299887.3299891","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3299887.3299891","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:53:38Z","timestamp":1750190018000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3299887.3299891"}},"subtitle":["A Survey"],"short-title":[],"issued":{"date-parts":[[2018,12,11]]},"references-count":74,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2018,12,11]]}},"alternative-id":["10.1145\/3299887.3299891"],"URL":"https:\/\/doi.org\/10.1145\/3299887.3299891","relation":{},"ISSN":["0163-5808"],"issn-type":[{"value":"0163-5808","type":"print"}],"subject":[],"published":{"date-parts":[[2018,12,11]]},"assertion":[{"value":"2018-12-11","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}