{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,1]],"date-time":"2026-11-01T04:48:48Z","timestamp":1793508528266,"version":"4.5.0"},"reference-count":54,"publisher":"Association for Computing Machinery (ACM)","issue":"3","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2017,11]]},"abstract":"<jats:p>Labeling training data is increasingly the largest bottleneck in deploying machine learning systems. We present Snorkel, a first-of-its-kind system that enables users to train state-of-the-art models without hand labeling any training data. Instead, users write labeling functions that express arbitrary heuristics, which can have unknown accuracies and correlations. Snorkel denoises their outputs without access to ground truth by incorporating the first end-to-end implementation of our recently proposed machine learning paradigm, data programming. We present a flexible interface layer for writing labeling functions based on our experience over the past year collaborating with companies, agencies, and research labs. In a user study, subject matter experts build models 2.8X faster and increase predictive performance an average 45.5% versus seven hours of hand labeling. We study the modeling tradeoffs in this new setting and propose an optimizer for automating tradeoff decisions that gives up to 1.8X speedup per pipeline execution. In two collaborations, with the U.S. Department of Veterans Affairs and the U.S. Food and Drug Administration, and on four open-source text and image data sets representative of other deployments, Snorkel provides 132% average improvements to predictive performance over prior heuristic approaches and comes within an average 3.60% of the predictive performance of large hand-curated training sets.<\/jats:p>","DOI":"10.14778\/3157794.3157797","type":"journal-article","created":{"date-parts":[[2017,12,12]],"date-time":"2017-12-12T13:33:38Z","timestamp":1513085618000},"page":"269-282","source":"Crossref","is-referenced-by-count":521,"title":["Snorkel"],"prefix":"10.14778","volume":"11","author":[{"given":"Alexander","family":"Ratner","sequence":"first","affiliation":[{"name":"Stanford University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Stephen H.","family":"Bach","sequence":"additional","affiliation":[{"name":"Stanford University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Henry","family":"Ehrenberg","sequence":"additional","affiliation":[{"name":"Stanford University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jason","family":"Fries","sequence":"additional","affiliation":[{"name":"Stanford University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sen","family":"Wu","sequence":"additional","affiliation":[{"name":"Stanford University"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christopher","family":"R\u00e9","sequence":"additional","affiliation":[{"name":"Stanford University"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2017,11]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Technical report","author":"Worldwide","year":"2017","unstructured":"Worldwide semiannual cognitive\/artificial intelligence systems spending guide. Technical report , International Data Corporation , 2017 . Worldwide semiannual cognitive\/artificial intelligence systems spending guide. Technical report, International Data Corporation, 2017."},{"key":"e_1_2_1_2_1","volume-title":"USENIX Symposium on Operating Systems Design and Implementation (OSDI)","author":"Abadi M.","year":"2016","unstructured":"M. Abadi , P. Barham , J. Chen , Z. Chen , A. Davis , J. Dean , M. Devin , S. Ghemawat , G. Irving , M. Isard , : A system for large-scale machine learning . In USENIX Symposium on Operating Systems Design and Implementation (OSDI) , 2016 . M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, et al. TensorFlow: A system for large-scale machine learning. In USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2016."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIT.1970.1054472"},{"key":"e_1_2_1_4_1","volume-title":"Meeting of the Association for Computational Linguistics (ACL)","author":"Alfonseca E.","year":"2012","unstructured":"E. Alfonseca , K. Filippova , J.-Y. Delort , and G. Garrido . Pattern learning for relation extraction with a hierarchical topic model . In Meeting of the Association for Computational Linguistics (ACL) , 2012 . E. Alfonseca, K. Filippova, J.-Y. Delort, and G. Garrido. Pattern learning for relation extraction with a hierarchical topic model. In Meeting of the Association for Computational Linguistics (ACL), 2012."},{"key":"e_1_2_1_5_1","volume-title":"International Conference on Machine Learning (ICML)","author":"Bach S. H.","year":"2017","unstructured":"S. H. Bach , B. He , A. Ratner , and C. R\u00e9 . Learning the structure of generative models without labeled data . In International Conference on Machine Learning (ICML) , 2017 . S. H. Bach, B. He, A. Ratner, and C. R\u00e9. Learning the structure of generative models without labeled data. In International Conference on Machine Learning (ICML), 2017."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/279943.279962"},{"key":"e_1_2_1_7_1","volume-title":"Meeting of the Association for Computational Linguistics (ACL)","author":"Bunescu R. C.","year":"2007","unstructured":"R. C. Bunescu and R. J. Mooney . Learning to extract relations from the Web using minimal supervision . In Meeting of the Association for Computational Linguistics (ACL) , 2007 . R. C. Bunescu and R. J. Mooney. Learning to extract relations from the Web using minimal supervision. In Meeting of the Association for Computational Linguistics (ACL), 2007."},{"key":"e_1_2_1_8_1","volume-title":"The MetaCyc database of metabolic pathways and enzymes and the BioCyc collection of pathway\/genome databases. Nucleic Acids Research, 44(D1):D471--D480","author":"Caspi R.","year":"2016","unstructured":"R. Caspi , R. Billington , L. Ferrer , H. Foerster , C. A. Fulcher , I. M. Keseler , A. Kothari , M. Krummenacker , M. Latendresse , L. A. Mueller , Q. Ong , S. Paley , P. Subhraveti , D. S. Weaver , and P. D. Karp . The MetaCyc database of metabolic pathways and enzymes and the BioCyc collection of pathway\/genome databases. Nucleic Acids Research, 44(D1):D471--D480 , 2016 . R. Caspi, R. Billington, L. Ferrer, H. Foerster, C. A. Fulcher, I. M. Keseler, A. Kothari, M. Krummenacker, M. Latendresse, L. A. Mueller, Q. Ong, S. Paley, P. Subhraveti, D. S. Weaver, and P. D. Karp. The MetaCyc database of metabolic pathways and enzymes and the BioCyc collection of pathway\/genome databases. Nucleic Acids Research, 44(D1):D471--D480, 2016."},{"key":"e_1_2_1_9_1","volume-title":"Semi-Supervised Learning. Adaptive Computation and Machine Learning","author":"Chapelle O.","year":"2009","unstructured":"O. Chapelle , B. Sch\u00f6lkopf , and A. Zien , editors . Semi-Supervised Learning. Adaptive Computation and Machine Learning . MIT Press , 2009 . O. Chapelle, B. Sch\u00f6lkopf, and A. Zien, editors. Semi-Supervised Learning. Adaptive Computation and Machine Learning. MIT Press, 2009."},{"key":"e_1_2_1_10_1","volume-title":"Workshop on Recent Trends in News Information Retrieval","author":"Corney D.","year":"2016","unstructured":"D. Corney , D. Albakour , M. Martinez , and S. Moussa . What do a million news articles look like ? In Workshop on Recent Trends in News Information Retrieval , 2016 . D. Corney, D. Albakour, M. Martinez, and S. Moussa. What do a million news articles look like? In Workshop on Recent Trends in News Information Retrieval, 2016."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2488388.2488414"},{"key":"e_1_2_1_12_1","volume-title":"Database","author":"Davis A. P.","year":"2013","unstructured":"A. P. Davis A CTD-Pfizer collaboration: Manual curation of 88,000 scientific articles text mined for drug-disease and drug-phenotype interactions . Database , 2013 . A. P. Davis et al. A CTD-Pfizer collaboration: Manual curation of 88,000 scientific articles text mined for drug-disease and drug-phenotype interactions. Database, 2013."},{"issue":"1","key":"e_1_2_1_13_1","first-page":"20","article-title":"Maximum likelihood estimation of observer error-rates using the EM algorithm","volume":"28","author":"Dawid A. P.","year":"1979","unstructured":"A. P. Dawid and A. M. Skene . Maximum likelihood estimation of observer error-rates using the EM algorithm . Journal of the Royal Statistical Society C , 28 ( 1 ): 20 -- 28 , 1979 . A. P. Dawid and A. M. Skene. Maximum likelihood estimation of observer error-rates using the EM algorithm. Journal of the Royal Statistical Society C, 28(1):20--28, 1979.","journal-title":"Journal of the Royal Statistical Society C"},{"key":"e_1_2_1_14_1","volume-title":"IEEE Conference on (CVPR)","author":"Deng J.","year":"2009","unstructured":"J. Deng , W. Dong , R. Socher , L.-J. Li , K. Li , and L. Fei-Fei . Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition , IEEE Conference on (CVPR) , 2009 . J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, IEEE Conference on (CVPR), 2009."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-01853-4"},{"key":"e_1_2_1_16_1","volume-title":"Time {Online","author":"Eadicicco L.","year":"2017","unstructured":"L. Eadicicco . Baidu's Andrew Ng on the future of artificial intelligence, 2017. Time {Online ; posted 11- January - 2017 }. L. Eadicicco. Baidu's Andrew Ng on the future of artificial intelligence, 2017. Time {Online; posted 11-January-2017}."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neunet.2005.06.042"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/W14-1611"},{"key":"e_1_2_1_19_1","unstructured":"K. He X. Zhang S. Ren and J. Sun. Deep residual learning for image recognition. CoRR abs\/1512.03385 2015.  K. He X. Zhang S. Ren and J. Sun. Deep residual learning for image recognition. CoRR abs\/1512.03385 2015."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.3115\/992133.992154"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1162\/089976602760128018"},{"key":"e_1_2_1_22_1","volume-title":"Meeting of the Association for Computational Linguistics (ACL)","author":"Hoffmann R.","year":"2011","unstructured":"R. Hoffmann , C. Zhang , X. Ling , L. Zettlemoyer , and D. S. Weld . Knowledge-based weak supervision for information extraction of overlapping relations . In Meeting of the Association for Computational Linguistics (ACL) , 2011 . R. Hoffmann, C. Zhang, X. Ling, L. Zettlemoyer, and D. S. Weld. Knowledge-based weak supervision for information extraction of overlapping relations. In Meeting of the Association for Computational Linguistics (ACL), 2011."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2015.7113284"},{"key":"e_1_2_1_24_1","volume-title":"Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980","author":"Kingma D.","year":"2014","unstructured":"D. Kingma and J. Ba . Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014 . D. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1093\/jamia\/ocv071"},{"key":"e_1_2_1_26_1","volume-title":"Semantic Web Journal","author":"Lehmann J.","year":"2014","unstructured":"J. Lehmann , R. Isele , M. Jakob , A. Jentzsch , D. Kontokostas , P. Mendes , S. Hellmann , M. Morsey , P. van Kleef , S. Auer , and C. Bizer . DBpedia - A large-scale, multilingual knowledge base extracted from Wikipedia . Semantic Web Journal , 2014 . J. Lehmann, R. Isele, M. Jakob, A. Jentzsch, D. Kontokostas, P. Mendes, S. Hellmann, M. Morsey, P. van Kleef, S. Auer, and C. Bizer. DBpedia - A large-scale, multilingual knowledge base extracted from Wikipedia. Semantic Web Journal, 2014."},{"key":"e_1_2_1_27_1","volume-title":"ICML Workshop: Machine Learning Meets Crowdsourcing","author":"Li H.","year":"2013","unstructured":"H. Li , B. Yu , and D. Zhou . Error rate analysis of labeling by crowdsourcing . In ICML Workshop: Machine Learning Meets Crowdsourcing . Atalanta, Georgia, USA , 2013 . H. Li, B. Yu, and D. Zhou. Error rate analysis of labeling by crowdsourcing. In ICML Workshop: Machine Learning Meets Crowdsourcing. Atalanta, Georgia, USA, 2013."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2897350.2897352"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/1553374.1553457"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.5555\/1756006.1756038"},{"key":"e_1_2_1_31_1","volume-title":"Google's hand-fed AI now gives answers, not just search results","author":"Metz C.","year":"2016","unstructured":"C. Metz . Google's hand-fed AI now gives answers, not just search results , 2016 . Wired {Online; posted 29-November-2016}. C. Metz. Google's hand-fed AI now gives answers, not just search results, 2016. Wired {Online; posted 29-November-2016}."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.5555\/1690219.1690287"},{"key":"e_1_2_1_33_1","volume-title":"The comparative toxicogenomics database: update","author":"Grondin D. A. P., C. J.","year":"2017","unstructured":"D. A. P., C. J. Grondin , R. J. Johnson , D. Sciaky , B. L. King , R. McMorran , J. Wiegers , T. Wiegers , and C. J. Mattingly . The comparative toxicogenomics database: update 2017 . Nucleic Acids Research , 2016. D. A. P., C. J. Grondin, R. J. Johnson, D. Sciaky, B. L. King, R. McMorran, J. Wiegers, T. Wiegers, and C. J. Mattingly. The comparative toxicogenomics database: update 2017. Nucleic Acids Research, 2016."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2009.191"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1219097111"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2588555.2593674"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/1978942.1979148"},{"key":"e_1_2_1_38_1","volume-title":"Neural Information Processing Systems (NIPS)","author":"Ratner A.","year":"2016","unstructured":"A. Ratner , C. De Sa , S. Wu , D. Selsam , and C. R\u00e9 . Data programming: Creating large training sets, quickly . In Neural Information Processing Systems (NIPS) , 2016 . A. Ratner, C. De Sa, S. Wu, D. Selsam, and C. R\u00e9. Data programming: Creating large training sets, quickly. In Neural Information Processing Systems (NIPS), 2016."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.14778\/3137628.3137631"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3035918.3035951"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.5555\/1889788.1889799"},{"key":"e_1_2_1_42_1","volume-title":"Conference on Empirical Methods on Natural Language Processing (EMNLP)","author":"Roth B.","year":"2013","unstructured":"B. Roth and D. Klakow . Combining generative and discriminative model scores for distant supervision . In Conference on Empirical Methods on Natural Language Processing (EMNLP) , 2013 . B. Roth and D. Klakow. Combining generative and discriminative model scores for distant supervision. In Conference on Empirical Methods on Natural Language Processing (EMNLP), 2013."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDCSW.2011.20"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIT.1965.1053799"},{"key":"e_1_2_1_45_1","volume-title":"Active Learning. Synthesis Lectures on Artificial Intelligence and Machine Learning","author":"Settles B.","year":"2012","unstructured":"B. Settles . Active Learning. Synthesis Lectures on Artificial Intelligence and Machine Learning . Morgan & Claypool Publishers , 2012 . B. Settles. Active Learning. Synthesis Lectures on Artificial Intelligence and Machine Learning. Morgan & Claypool Publishers, 2012."},{"key":"e_1_2_1_46_1","volume-title":"AAAI Conference on Artificial Intelligence (AAAI)","author":"Stewart R.","year":"2017","unstructured":"R. Stewart and S. Ermon . Label-free supervision of neural networks with physics and other domain knowledge . In AAAI Conference on Artificial Intelligence (AAAI) , 2017 . R. Stewart and S. Ermon. Label-free supervision of neural networks with physics and other domain knowledge. In AAAI Conference on Artificial Intelligence (AAAI), 2017."},{"key":"e_1_2_1_47_1","volume-title":"Revisiting unreasonable effectiveness of data in deep learning era. arXiv preprint arXiv:1707.02968","author":"Sun C.","year":"2017","unstructured":"C. Sun , A. Shrivastava , S. Singh , and A. Gupta . Revisiting unreasonable effectiveness of data in deep learning era. arXiv preprint arXiv:1707.02968 , 2017 . C. Sun, A. Shrivastava, S. Singh, and A. Gupta. Revisiting unreasonable effectiveness of data in deep learning era. arXiv preprint arXiv:1707.02968, 2017."},{"key":"e_1_2_1_48_1","volume-title":"Meeting of the Association for Computational Linguistics (ACL)","author":"Takamatsu S.","year":"2012","unstructured":"S. Takamatsu , I. Sato , and H. Nakagawa . Reducing wrong labels in distant supervision for relation extraction . In Meeting of the Association for Computational Linguistics (ACL) , 2012 . S. Takamatsu, I. Sato, and H. Nakagawa. Reducing wrong labels in distant supervision for relation extraction. In Meeting of the Association for Computational Linguistics (ACL), 2012."},{"key":"e_1_2_1_49_1","volume-title":"BioCreative Challenge Evaluation Workshop","author":"Wei C.-H.","year":"2015","unstructured":"C.-H. Wei , Y. Peng , R. Leaman , D. A. P., C. J. Mattingly , J. Li , T. Wiegers , and Z. Lu . Overview of the BioCreative Vchemical disease relation (CDR)task . In BioCreative Challenge Evaluation Workshop , 2015 . C.-H. Wei, Y. Peng, R. Leaman, D. A. P., C. J. Mattingly, J. Li, T. Wiegers, and Z. Lu. Overview of the BioCreative Vchemical disease relation (CDR)task. In BioCreative Challenge Evaluation Workshop, 2015."},{"key":"e_1_2_1_50_1","volume-title":"Risk and Trust (PASSAT) and Inernational Conference on Social Computing (SocialCom)","author":"Yuen M.-C.","year":"2011","unstructured":"M.-C. Yuen , I. King , and K.-S. Leung . A survey of crowdsourcing systems. In Privacy, Security , Risk and Trust (PASSAT) and Inernational Conference on Social Computing (SocialCom) , 2011 . M.-C. Yuen, I. King, and K.-S. Leung. A survey of crowdsourcing systems. In Privacy, Security, Risk and Trust (PASSAT) and Inernational Conference on Social Computing (SocialCom), 2011."},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.5555\/1613715.1613721"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3060586"},{"key":"e_1_2_1_53_1","first-page":"1","article-title":"Spectral methods meet EM: A provably optimal algorithm for crowdsourcing","volume":"17","author":"Zhang Y.","year":"2016","unstructured":"Y. Zhang , X. Chen , D. Zhou , and M. I. Jordan . Spectral methods meet EM: A provably optimal algorithm for crowdsourcing . Journal of Machine Learning Research , 17 : 1 -- 44 , 2016 . Y. Zhang, X. Chen, D. Zhou, and M. I. Jordan. Spectral methods meet EM: A provably optimal algorithm for crowdsourcing. Journal of Machine Learning Research, 17:1--44, 2016.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.14778\/2168651.2168656"}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3157794.3157797","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,28]],"date-time":"2022-12-28T04:36:37Z","timestamp":1672202197000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3157794.3157797"}},"subtitle":["rapid training data creation with weak supervision"],"short-title":[],"issued":{"date-parts":[[2017,11]]},"references-count":54,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2017,11]]}},"alternative-id":["10.14778\/3157794.3157797"],"URL":"https:\/\/doi.org\/10.14778\/3157794.3157797","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2017,11]]}}}