{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,3,7]],"date-time":"2024-03-07T12:10:32Z","timestamp":1709813432112},"reference-count":35,"publisher":"Association for Computing Machinery (ACM)","issue":"10","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2017,6]]},"abstract":"<jats:p>Data cleaning, whether manual or algorithmic, is rarely perfect leaving a dataset with an unknown number of false positives and false negatives after cleaning. In many scenarios, quantifying the number of remaining errors is challenging because our data integrity rules themselves may be incomplete, or the available gold-standard datasets may be too small to extrapolate. As the use of inherently fallible crowds becomes more prevalent in data cleaning problems, it is important to have estimators to quantify the extent of such errors. We propose novel species estimators to estimate the number of distinct remaining errors in a dataset after it has been cleaned by a set of crowd workers -- essentially, quantifying the utility of hiring additional workers to clean the dataset. This problem requires new estimators that are robust to false positives and false negatives, and we empirically show on three real-world datasets that existing species estimators are unstable for this problem, while our proposed techniques quickly converge.<\/jats:p>","DOI":"10.14778\/3115404.3115414","type":"journal-article","created":{"date-parts":[[2017,9,7]],"date-time":"2017-09-07T13:35:53Z","timestamp":1504791353000},"page":"1094-1105","source":"Crossref","is-referenced-by-count":10,"title":["A data quality metric (DQM)"],"prefix":"10.14778","volume":"10","author":[{"given":"Yeounoh","family":"Chung","sequence":"first","affiliation":[{"name":"Brown University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sanjay","family":"Krishnan","sequence":"additional","affiliation":[{"name":"UC Berkeley"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tim","family":"Kraska","sequence":"additional","affiliation":[{"name":"Brown University"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2017,6]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"For big-data scientists 'janitor work' is key hurdle to insights. https:\/\/nyti.ms\/2jNUfHo.  For big-data scientists 'janitor work' is key hurdle to insights. https:\/\/nyti.ms\/2jNUfHo."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.14778\/2994509.2994518"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2465351.2465355"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.14778\/2732967.2732975"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/1066157.1066175"},{"key":"e_1_2_1_6_1","doi-asserted-by":"crossref","unstructured":"A. Chao and S.-M. Lee. Estimating the number of classes via sample coverage. J. the American statistical Association 87(417):210--217 1992.  A. Chao and S.-M. Lee. Estimating the number of classes via sample coverage. J. the American statistical Association 87(417):210--217 1992.","DOI":"10.1080\/01621459.1992.10475194"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2665069"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ic.2004.04.007"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2882903.2912574"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/2882903.2882909"},{"key":"e_1_2_1_11_1","first-page":"315","volume-title":"PVLDB","author":"Cong G.","year":"2007"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/1659225.1659228"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2010.154"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/1989323.1989331"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.14778\/2367502.2367564"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2588555.2588576"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.14778\/2732939.2732943"},{"key":"e_1_2_1_18_1","first-page":"311","volume-title":"PVLDB","author":"Haas P. J.","year":"1995"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/2661829.2661885"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/1710115.1710121"},{"key":"e_1_2_1_21_1","first-page":"32","volume-title":"AAAI Spring Symposium: Wisdom of the Crowd","author":"Lange R.","year":"2012"},{"key":"e_1_2_1_22_1","first-page":"701","volume-title":"Proceedings of NIPS","author":"Liu Q.","year":"2012"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2007.367867"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1561\/1900000044"},{"key":"e_1_2_1_25_1","unstructured":"D. Oleson A. Sorokin G. P. Laughlin V. Hester J. Le and L. Biewald. Programmatic gold: Targeted and scalable quality assurance in crowdsourcing. Human computation 11(11) 2011.   D. Oleson A. Sorokin G. P. Laughlin V. Hester J. Le and L. Biewald. Programmatic gold: Targeted and scalable quality assurance in crowdsourcing. Human computation 11(11) 2011."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2014.2359666"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/505248.506010"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/1659225.1659227"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/2845644"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/1993636.1993727"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.14778\/2350229.2350263"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2588555.2610505"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00778-013-0315-0"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.14778\/1952376.1952378"},{"key":"e_1_2_1_35_1","first-page":"1260","volume-title":"Proceedings of NIPS","author":"Zhang Y.","year":"2014"}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3115404.3115414","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,28]],"date-time":"2022-12-28T09:48:46Z","timestamp":1672220926000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3115404.3115414"}},"subtitle":["how to estimate the number of undetected errors in data sets"],"short-title":[],"issued":{"date-parts":[[2017,6]]},"references-count":35,"journal-issue":{"issue":"10","published-print":{"date-parts":[[2017,6]]}},"alternative-id":["10.14778\/3115404.3115414"],"URL":"https:\/\/doi.org\/10.14778\/3115404.3115414","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2017,6]]}}}