{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,17]],"date-time":"2026-02-17T15:01:35Z","timestamp":1771340495670,"version":"3.50.1"},"reference-count":29,"publisher":"Association for Computing Machinery (ACM)","issue":"13","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2014,8]]},"abstract":"<jats:p>Large-scale classification is an increasingly critical Big Data problem. So far, however, very little has been published on how this is done in practice. In this paper we describe Chimera, our solution to classify tens of millions of products into 5000+ product types at WalmartLabs. We show that at this scale, many conventional assumptions regarding learning and crowdsourcing break down, and that existing solutions cease to work. We describe how Chimera employs a combination of learning, rules (created by in-house analysts), and crowdsourcing to achieve accurate, continuously improving, and cost-effective classification. We discuss a set of lessons learned for other similar Big Data systems. In particular, we argue that at large scales crowdsourcing is critical, but must be used in combination with learning, rules, and in-house analysts. We also argue that using rules (in conjunction with learning) is a must, and that more research attention should be paid to helping analysts create and manage (tens of thousands of) rules more effectively.<\/jats:p>","DOI":"10.14778\/2733004.2733024","type":"journal-article","created":{"date-parts":[[2015,5,12]],"date-time":"2015-05-12T15:37:52Z","timestamp":1431445072000},"page":"1529-1540","source":"Crossref","is-referenced-by-count":63,"title":["Chimera"],"prefix":"10.14778","volume":"7","author":[{"given":"Chong","family":"Sun","sequence":"first","affiliation":[{"name":"@WalmartLabs"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Narasimhan","family":"Rampalli","sequence":"additional","affiliation":[{"name":"@WalmartLabs"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Frank","family":"Yang","sequence":"additional","affiliation":[{"name":"@WalmartLabs"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"AnHai","family":"Doan","sequence":"additional","affiliation":[{"name":"@WalmartLabs and University of Wisconsin-Madison"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2014,8]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"reCAPTCHA. http:\/\/en.wikipedia.org\/wiki\/ReCAPTCHA. reCAPTCHA. http:\/\/en.wikipedia.org\/wiki\/ReCAPTCHA."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/1480506.1480508"},{"key":"e_1_2_1_3_1","volume-title":"LREC","author":"Ambati V.","year":"2010","unstructured":"V. Ambati , S. Vogel , and J. G. Carbonell . Active learning and crowdsourcing for machine translation . In LREC , 2010 . V. Ambati, S. Vogel, and J. G. Carbonell. Active learning and crowdsourcing for machine translation. In LREC, 2010."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/2020408.2020449"},{"key":"e_1_2_1_5_1","first-page":"163","volume-title":"NIPS","author":"Bengio S.","year":"2010","unstructured":"S. Bengio , J. Weston , and D. Grangier . Label embedding trees for large multi-class tasks . In NIPS , pages 163 -- 171 , 2010 . S. Bengio, J. Weston, and D. Grangier. Label embedding trees for large multi-class tasks. In NIPS, pages 163--171, 2010."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.5555\/1888089.1888123"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1016\/B978-1-55860-377-6.50023-2"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/1924421.1924442"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2588555.2588576"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.5555\/645326.649721"},{"key":"e_1_2_1_11_1","volume-title":"AAMAS","author":"Kamar E.","year":"2012","unstructured":"E. Kamar , S. Hacker , and E. Horvitz . Combining human and machine intelligence in large-scale crowdsourcing . In AAMAS , 2012 . E. Kamar, S. Hacker, and E. Horvitz. Combining human and machine intelligence in large-scale crowdsourcing. In AAMAS, 2012."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2011.79"},{"key":"e_1_2_1_13_1","volume-title":"NIPS","author":"Karger D. R.","year":"2011","unstructured":"D. R. Karger , S. Oh , and D. Shah . Iterative learning for reliable crowdsourcing systems . In NIPS , 2011 . D. R. Karger, S. Oh, and D. Shah. Iterative learning for reliable crowdsourcing systems. In NIPS, 2011."},{"key":"e_1_2_1_14_1","volume-title":"SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation","author":"Le J.","year":"2010","unstructured":"J. Le , A. Edmonds , V. Hester , and L. Biewald . Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution . In SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation , 2010 . J. Le, A. Edmonds, V. Hester, and L. Biewald. Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution. In SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation, 2010."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/133160.133172"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/1089815.1089821"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.5555\/311445"},{"key":"e_1_2_1_19_1","volume-title":"AAAI-98 Workshop","author":"McCallum A.","year":"1998","unstructured":"A. McCallum and K. Nigam . A comparison of event models for naive bayes text classification . In AAAI-98 Workshop , 1998 . A. McCallum and K. Nigam. A comparison of event models for naive bayes text classification. In AAAI-98 Workshop, 1998."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.5555\/541177"},{"key":"e_1_2_1_21_1","volume-title":"Database management systems (3. ed.)","author":"Ramakrishnan R.","year":"2003","unstructured":"R. Ramakrishnan and J. Gehrke . Database management systems (3. ed.) . McGraw-Hill , 2003 . R. Ramakrishnan and J. Gehrke. Database management systems (3. ed.). McGraw-Hill, 2003."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.5555\/1756006.1859894"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.5555\/235222"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/505282.505283"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2396761.2396838"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2063576.2063855"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10618-010-0175-9"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/1571941.1572131"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2011.5995430"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/312624.312647"}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/2733004.2733024","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,28]],"date-time":"2022-12-28T09:41:46Z","timestamp":1672220506000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/2733004.2733024"}},"subtitle":["large-scale classification using machine learning, rules, and crowdsourcing"],"short-title":[],"issued":{"date-parts":[[2014,8]]},"references-count":29,"journal-issue":{"issue":"13","published-print":{"date-parts":[[2014,8]]}},"alternative-id":["10.14778\/2733004.2733024"],"URL":"https:\/\/doi.org\/10.14778\/2733004.2733024","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2014,8]]}}}