{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,13]],"date-time":"2026-06-13T15:29:21Z","timestamp":1781364561678,"version":"3.54.1"},"reference-count":41,"publisher":"Association for Computing Machinery (ACM)","issue":"11","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2021,7]]},"abstract":"<jats:p>\n            High-quality labels are expensive to obtain for many machine learning tasks, such as medical image classification tasks. Therefore, probabilistic (weak) labels produced by weak supervision tools are used to seed a process in which influential samples with weak labels are identified and cleaned by several human annotators to improve the model performance. To lower the overall cost and computational overhead of this process, we propose a solution called CHEF (CHEap and Fast label cleaning), which consists of the following three components. First, to reduce the cost of human annotators, we use INFL, which prioritizes the\n            <jats:italic>most influential<\/jats:italic>\n            training samples for cleaning and provides cleaned labels to save the cost of one human annotator. Second, to accelerate the sample selector phase and the model constructor phase, we use Increm-INFL to\n            <jats:italic>incrementally<\/jats:italic>\n            produce influential samples, and DeltaGrad-L to\n            <jats:italic>incrementally<\/jats:italic>\n            update the model. Third, we redesign the typical label cleaning pipeline so that human annotators iteratively clean smaller batch of samples rather than one big batch of samples. This yields better overall model performance and enables possible early termination when the expected model performance has been achieved. Extensive experiments show that our approach gives good model prediction performance while achieving significant speed-ups.\n          <\/jats:p>","DOI":"10.14778\/3476249.3476290","type":"journal-article","created":{"date-parts":[[2021,10,27]],"date-time":"2021-10-27T16:46:23Z","timestamp":1635353183000},"page":"2410-2418","source":"Crossref","is-referenced-by-count":4,"title":["CHEF"],"prefix":"10.14778","volume":"14","author":[{"given":"Yinjun","family":"Wu","sequence":"first","affiliation":[{"name":"University of Pennsylvania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"James","family":"Weimer","sequence":"additional","affiliation":[{"name":"University of Pennsylvania"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Susan B.","family":"Davidson","sequence":"additional","affiliation":[{"name":"University of Pennsylvania"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2021,10,27]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"11th Annual Conference on Innovative Data Systems Research (CIDR 2021)","author":"Melgar Leonel Aguilar","year":"2021","unstructured":"Leonel Aguilar Melgar , David Dao , Shaoduo Gan , Nezihe M G\u00fcrel , Nora Hollenstein , Jiawei Jiang , Bojan Karla\u0161 , Thomas Lemmin , Tian Li , Yang Li , 2021 . Ease. ML: A Lifecycle Management system for Machine Learning . In 11th Annual Conference on Innovative Data Systems Research (CIDR 2021) (virtual). CIDR. Leonel Aguilar Melgar, David Dao, Shaoduo Gan, Nezihe M G\u00fcrel, Nora Hollenstein, Jiawei Jiang, Bojan Karla\u0161, Thomas Lemmin, Tian Li, Yang Li, et al. 2021. Ease. ML: A Lifecycle Management system for Machine Learning. In 11th Annual Conference on Innovative Data Systems Research (CIDR 2021)(virtual). CIDR."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3299869.3314036"},{"key":"e_1_2_1_3_1","volume-title":"Interactive Weak Supervision: Learning Useful Heuristics for Data Labeling. arXiv preprint arXiv:2012.06046","author":"Boecking Benedikt","year":"2020","unstructured":"Benedikt Boecking , Willie Neiswanger , Eric Xing , and Artur Dubrawski . 2020. Interactive Weak Supervision: Learning Useful Heuristics for Data Labeling. arXiv preprint arXiv:2012.06046 ( 2020 ). Benedikt Boecking, Willie Neiswanger, Eric Xing, and Artur Dubrawski. 2020. Interactive Weak Supervision: Learning Useful Heuristics for Data Labeling. arXiv preprint arXiv:2012.06046 (2020)."},{"key":"e_1_2_1_4_1","volume-title":"Error and discrepancy in radiology: inevitable or avoidable? Insights into imaging 8, 1","author":"Brady Adrian P","year":"2017","unstructured":"Adrian P Brady . 2017. Error and discrepancy in radiology: inevitable or avoidable? Insights into imaging 8, 1 ( 2017 ), 171--182. Adrian P Brady. 2017. Error and discrepancy in radiology: inevitable or avoidable? Insights into imaging 8, 1 (2017), 171--182."},{"key":"e_1_2_1_5_1","volume-title":"DART: Data Addition and Removal Trees. arXiv preprint arXiv:2009.05567","author":"Brophy Jonathan","year":"2020","unstructured":"Jonathan Brophy and Daniel Lowd . 2020 . DART: Data Addition and Removal Trees. arXiv preprint arXiv:2009.05567 (2020). Jonathan Brophy and Daniel Lowd. 2020. DART: Data Addition and Removal Trees. arXiv preprint arXiv:2009.05567 (2020)."},{"key":"e_1_2_1_6_1","volume-title":"Data Programming using Continuous and Quality-Guided Labeling Functions. arXiv preprint arXiv:1911.09860","author":"Chatterjee Oishik","year":"2019","unstructured":"Oishik Chatterjee , Ganesh Ramakrishnan , and Sunita Sarawagi . 2019. Data Programming using Continuous and Quality-Guided Labeling Functions. arXiv preprint arXiv:1911.09860 ( 2019 ). Oishik Chatterjee, Ganesh Ramakrishnan, and Sunita Sarawagi. 2019. Data Programming using Continuous and Quality-Guided Labeling Functions. arXiv preprint arXiv:1911.09860 (2019)."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3318464.3380592"},{"key":"e_1_2_1_8_1","volume-title":"Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805","author":"Devlin Jacob","year":"2018","unstructured":"Jacob Devlin , Ming-Wei Chang , Kenton Lee , and Kristina Toutanova . 2018 . Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018). Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.14778\/3297753.3297758"},{"key":"e_1_2_1_10_1","volume-title":"International Conference on Machine Learning. PMLR, 2242--2251","author":"Ghorbani Amirata","year":"2019","unstructured":"Amirata Ghorbani and James Zou . 2019 . Data shapley: Equitable valuation of data for machine learning . In International Conference on Machine Learning. PMLR, 2242--2251 . Amirata Ghorbani and James Zou. 2019. Data shapley: Equitable valuation of data for machine learning. In International Conference on Machine Learning. PMLR, 2242--2251."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.5555\/3454287.3454603"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0377-0427(00)00413-1"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1001\/jama.2016.17216"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.5555\/3327757.3327944"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00342"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v33i01.3301590"},{"key":"e_1_2_1_18_1","volume-title":"The 22nd International Conference on Artificial Intelligence and Statistics. PMLR, 1167--1176","author":"Jia Ruoxi","year":"2019","unstructured":"Ruoxi Jia , David Dao , Boxin Wang , Frances Ann Hubis , Nick Hynes , Nezihe Merve G\u00fcrel , Bo Li , Ce Zhang , Dawn Song , and Costas J Spanos . 2019 . Towards efficient data valuation based on the shapley value . In The 22nd International Conference on Artificial Intelligence and Statistics. PMLR, 1167--1176 . Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nick Hynes, Nezihe Merve G\u00fcrel, Bo Li, Ce Zhang, Dawn Song, and Costas J Spanos. 2019. Towards efficient data valuation based on the shapley value. In The 22nd International Conference on Artificial Intelligence and Statistics. PMLR, 1167--1176."},{"key":"e_1_2_1_19_1","volume-title":"a large publicly available database of labeled chest radiographs. arXiv preprint arXiv:1901.07042","author":"Johnson Alistair EW","year":"2019","unstructured":"Alistair EW Johnson , Tom J Pollard , Nathaniel R Greenbaum , Matthew P Lungren , Chih-ying Deng, Yifan Peng , Zhiyong Lu , Roger G Mark , Seth J Berkowitz , and Steven Horng . 2019. MIMIC-CXR-JPG , a large publicly available database of labeled chest radiographs. arXiv preprint arXiv:1901.07042 ( 2019 ). Alistair EW Johnson, Tom J Pollard, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Yifan Peng, Zhiyong Lu, Roger G Mark, Seth J Berkowitz, and Steven Horng. 2019. MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs. arXiv preprint arXiv:1901.07042 (2019)."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.5555\/3305381.3305576"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.14778\/2994509.2994514"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1080\/00029890.1978.11994526"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/2557642.2563675"},{"key":"e_1_2_1_24_1","first-page":"16","article-title":"Towards automated data cleaning workflows","volume":"15","author":"Mahdavi Mohammad","year":"2019","unstructured":"Mohammad Mahdavi , Felix Neutatz , Larysa Visengeriyeva , and Ziawasch Abedjan . 2019 . Towards automated data cleaning workflows . Machine Learning 15 (2019), 16 . Mohammad Mahdavi, Felix Neutatz, Larysa Visengeriyeva, and Ziawasch Abedjan. 2019. Towards automated data cleaning workflows. Machine Learning 15 (2019), 16.","journal-title":"Machine Learning"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.5555\/3104322.3104416"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.5555\/343374"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.24251\/HICSS.2020.028"},{"key":"e_1_2_1_28_1","volume-title":"Updating quasi-Newton matrices with limited storage. Mathematics of computation 35, 151","author":"Nocedal Jorge","year":"1980","unstructured":"Jorge Nocedal . 1980. Updating quasi-Newton matrices with limited storage. Mathematics of computation 35, 151 ( 1980 ), 773--782. Jorge Nocedal. 1980. Updating quasi-Newton matrices with limited storage. Mathematics of computation 35, 151 (1980), 773--782."},{"key":"e_1_2_1_29_1","unstructured":"Adam Paszke Sam Gross Soumith Chintala Gregory Chanan Edward Yang Zachary DeVito Zeming Lin Alban Desmaison Luca Antiga and Adam Lerer. 2017. Automatic differentiation in pytorch. (2017).  Adam Paszke Sam Gross Soumith Chintala Gregory Chanan Edward Yang Zachary DeVito Zeming Lin Alban Desmaison Luca Antiga and Adam Lerer. 2017. Automatic differentiation in pytorch. (2017)."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.5555\/3454287.3454588"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.14778\/3157794.3157797"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1016\/S1361-3723(16)30028-8"},{"key":"e_1_2_1_33_1","unstructured":"Burr Settles. 2009. Active learning literature survey. (2009).  Burr Settles. 2009. Active learning literature survey. (2009)."},{"key":"e_1_2_1_34_1","doi-asserted-by":"crossref","unstructured":"L Smyth. 2020. Training-ValueNet: A new approach for label cleaning on weakly-supervised datasets. (2020).  L Smyth. 2020. Training-ValueNet: A new approach for label cleaning on weakly-supervised datasets. (2020).","DOI":"10.1109\/DEVLRN.2019.8850689"},{"key":"e_1_2_1_35_1","volume-title":"Learning from noisy labels with deep neural networks. arXiv preprint arXiv:1406.2080 2, 3","author":"Sukhbaatar Sainbayar","year":"2014","unstructured":"Sainbayar Sukhbaatar and Rob Fergus . 2014. Learning from noisy labels with deep neural networks. arXiv preprint arXiv:1406.2080 2, 3 ( 2014 ), 4. Sainbayar Sukhbaatar and Rob Fergus. 2014. Learning from noisy labels with deep neural networks. arXiv preprint arXiv:1406.2080 2, 3 (2014), 4."},{"key":"e_1_2_1_36_1","article-title":"Visualizing data using t-SNE","volume":"9","author":"der Maaten Laurens Van","year":"2008","unstructured":"Laurens Van der Maaten and Geoffrey Hinton . 2008 . Visualizing data using t-SNE . Journal of machine learning research 9 , 11 (2008). Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of machine learning research 9, 11 (2008).","journal-title":"Journal of machine learning research"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.14778\/3291264.3291268"},{"key":"e_1_2_1_38_1","volume-title":"International Conference on Machine Learning. PMLR, 10355--10366","author":"Wu Yinjun","year":"2020","unstructured":"Yinjun Wu , Edgar Dobriban , and Susan Davidson . 2020 . DeltaGrad: Rapid retraining of machine learning models . In International Conference on Machine Learning. PMLR, 10355--10366 . Yinjun Wu, Edgar Dobriban, and Susan Davidson. 2020. DeltaGrad: Rapid retraining of machine learning models. In International Conference on Machine Learning. PMLR, 10355--10366."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3318464.3380571"},{"key":"e_1_2_1_40_1","volume-title":"CHEF: A Cheap and Fast Pipeline for Iteratively Cleaning Label Uncertainties (Technical Report). arXiv preprint arXiv:2107.08588","author":"Wu Yinjun","year":"2021","unstructured":"Yinjun Wu , James Weimer , and Susan B Davidson . 2021 . CHEF: A Cheap and Fast Pipeline for Iteratively Cleaning Label Uncertainties (Technical Report). arXiv preprint arXiv:2107.08588 (2021). Yinjun Wu, James Weimer, and Susan B Davidson. 2021. CHEF: A Cheap and Fast Pipeline for Iteratively Cleaning Label Uncertainties (Technical Report). arXiv preprint arXiv:2107.08588 (2021)."},{"key":"e_1_2_1_41_1","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","volume":"32","author":"Zhang Xuezhou","year":"2018","unstructured":"Xuezhou Zhang , Xiaojin Zhu , and Stephen Wright . 2018 . Training set debugging using trusted items . In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 32 . Xuezhou Zhang, Xiaojin Zhu, and Stephen Wright. 2018. Training set debugging using trusted items. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32."}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3476249.3476290","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,28]],"date-time":"2022-12-28T10:04:39Z","timestamp":1672221879000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3476249.3476290"}},"subtitle":["a cheap and fast pipeline for iteratively cleaning label uncertainties"],"short-title":[],"issued":{"date-parts":[[2021,7]]},"references-count":41,"journal-issue":{"issue":"11","published-print":{"date-parts":[[2021,7]]}},"alternative-id":["10.14778\/3476249.3476290"],"URL":"https:\/\/doi.org\/10.14778\/3476249.3476290","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2021,7]]}}}