{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,19]],"date-time":"2026-05-19T07:13:37Z","timestamp":1779174817043,"version":"3.51.4"},"reference-count":56,"publisher":"Association for Computing Machinery (ACM)","issue":"4","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. VLDB Endow."],"published-print":{"date-parts":[[2016,11]]},"abstract":"<jats:p>\n            Crowdsourcing is a new computing paradigm that harnesses human effort to solve computer-hard problems, such as entity resolution and photo tagging. The crowd (or workers) have diverse qualities and it is important to effectively model a worker's quality. Most of existing worker models assume that workers have the same quality on different tasks. In practice, however, tasks belong to a variety of diverse domains, and workers have different qualities on different domains. For example, a worker who is a basketball fan should have better quality for the task of labeling a photo related to '\n            <jats:italic>Stephen Curry<\/jats:italic>\n            ' than the one related to '\n            <jats:italic>Leonardo DiCaprio<\/jats:italic>\n            '. In this paper, we study how to leverage domain knowledge to accurately model a worker's quality. We examine using\n            <jats:italic>knowledge base<\/jats:italic>\n            (KB), e.g., Wikipedia and Freebase, to detect the domains of tasks and workers. We develop\n            <jats:italic>Domain Vector Estimation<\/jats:italic>\n            , which analyzes the domains of a task with respect to the KB. We also study\n            <jats:italic>Truth Inference<\/jats:italic>\n            , which utilizes the domain-sensitive worker model to accurately infer the true answer of a task. We design an\n            <jats:italic>Online Task Assignment<\/jats:italic>\n            algorithm, which judiciously and efficiently assigns tasks to appropriate workers. To implement these solutions, we have built DOCS, a system deployed on the Amazon Mechanical Turk. Experiments show that DOCS performs much better than the state-of-the-art approaches.\n          <\/jats:p>","DOI":"10.14778\/3025111.3025118","type":"journal-article","created":{"date-parts":[[2017,1,24]],"date-time":"2017-01-24T15:29:41Z","timestamp":1485271781000},"page":"361-372","source":"Crossref","is-referenced-by-count":79,"title":["DOCS"],"prefix":"10.14778","volume":"10","author":[{"given":"Yudian","family":"Zheng","sequence":"first","affiliation":[{"name":"The University of Hong Kong"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Guoliang","family":"Li","sequence":"additional","affiliation":[{"name":"Tsinghua University"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Reynold","family":"Cheng","sequence":"additional","affiliation":[{"name":"The University of Hong Kong"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2016,11]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"https:\/\/docs.aws.amazon.com\/AWSMechTurk\/latest\/RequesterUI\/amt-ui.pdf.  https:\/\/docs.aws.amazon.com\/AWSMechTurk\/latest\/RequesterUI\/amt-ui.pdf."},{"key":"e_1_2_1_2_1","unstructured":"http:\/\/answers.yahoo.com\/question\/index?qid=20071211155603AAKwtyr.  http:\/\/answers.yahoo.com\/question\/index?qid=20071211155603AAKwtyr."},{"key":"e_1_2_1_3_1","unstructured":"Amazon mechanical turk. https:\/\/www.mturk.com\/.  Amazon mechanical turk. https:\/\/www.mturk.com\/."},{"key":"e_1_2_1_4_1","unstructured":"Chinacrowd. http:\/\/www.chinacrowds.com.  Chinacrowd. http:\/\/www.chinacrowds.com."},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2588555.2610514"},{"key":"e_1_2_1_6_1","volume-title":"Latent dirichlet allocation. Journal of Machine Learning Research, 3(Jan):993--1022","author":"Blei D. M.","year":"2003","unstructured":"D. M. Blei , A. Y. Ng , and M. I. Jordan . Latent dirichlet allocation. Journal of Machine Learning Research, 3(Jan):993--1022 , 2003 . D. M. Blei, A. Y. Ng, and M. I. Jordan. Latent dirichlet allocation. Journal of Machine Learning Research, 3(Jan):993--1022, 2003."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0022-0000(73)80033-9"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2012.122"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/2882903.2915252"},{"key":"e_1_2_1_10_1","first-page":"1787","volume-title":"EMNLP","author":"Cheng X.","year":"2013","unstructured":"X. Cheng and D. Roth . Relational inference for wikification . In EMNLP , pages 1787 -- 1796 , 2013 . X. Cheng and D. Roth. Relational inference for wikification. In EMNLP, pages 1787--1796, 2013."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2749431"},{"key":"e_1_2_1_12_1","unstructured":"Compositions. http:\/\/mathworld.wolfram.com\/Composition.html.  Compositions. http:\/\/mathworld.wolfram.com\/Composition.html."},{"key":"e_1_2_1_13_1","unstructured":"CrowdFlower. http:\/\/crowdflower.com\/.  CrowdFlower. http:\/\/crowdflower.com\/."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/2448496.2448524"},{"key":"e_1_2_1_15_1","first-page":"20","volume-title":"Maximum likelihood estimation of observer error-rates using the em algorithm. Applied statistics","author":"Dawid A. P.","year":"1979","unstructured":"A. P. Dawid and A. M. Skene . Maximum likelihood estimation of observer error-rates using the em algorithm. Applied statistics , pages 20 -- 28 , 1979 . A. P. Dawid and A. M. Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Applied statistics, pages 20--28, 1979."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2187836.2187900"},{"key":"e_1_2_1_17_1","series-title":"Series B (methodological)","first-page":"1","volume-title":"Maximum likelihood from incomplete data via the em algorithm. Journal of the royal statistical society","author":"Dempster A. P.","year":"1977","unstructured":"A. P. Dempster , N. M. Laird , and D. B. Rubin . Maximum likelihood from incomplete data via the em algorithm. Journal of the royal statistical society . Series B (methodological) , pages 1 -- 38 , 1977 . A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the royal statistical society. Series B (methodological), pages 1--38, 1977."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2750550"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/1989323.1989331"},{"key":"e_1_2_1_20_1","unstructured":"Freebase. https:\/\/www.freebase.com\/.  Freebase. https:\/\/www.freebase.com\/."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2567948.2577311"},{"key":"e_1_2_1_22_1","first-page":"45","volume-title":"AAAI","author":"Ho C.-J.","year":"2012","unstructured":"C.-J. Ho and J. W. Vaughan . Online task assignment in crowdsourcing markets . In AAAI , pages 45 -- 51 , 2012 . C.-J. Ho and J. W. Vaughan. Online task assignment in crowdsourcing markets. In AAAI, pages 45--51, 2012."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2016.7498298"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDE.2016.7498229"},{"key":"e_1_2_1_25_1","volume-title":"TAC","author":"Ji H.","year":"2010","unstructured":"H. Ji , R. Grishman , H. T. Dang , K. Griffitt , and J. Ellis . Overview of the tac 2010 knowledge base population track . In TAC , 2010 . H. Ji, R. Grishman, H. T. Dang, K. Griffitt, and J. Ellis. Overview of the tac 2010 knowledge base population track. In TAC, 2010."},{"key":"e_1_2_1_26_1","volume-title":"On information and sufficiency. The annals of mathematical statistics, 22(1):79--86","author":"Kullback S.","year":"1951","unstructured":"S. Kullback and R. A. Leibler . On information and sufficiency. The annals of mathematical statistics, 22(1):79--86 , 1951 . S. Kullback and R. A. Leibler. On information and sufficiency. The annals of mathematical statistics, 22(1):79--86, 1951."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2016.2535242"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2897350.2897352"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.14778\/2336664.2336676"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2783258.2783314"},{"key":"e_1_2_1_31_1","first-page":"211","volume-title":"CIDR","author":"Marcus A.","year":"2011","unstructured":"A. Marcus , E. Wu , S. Madden , and R. C. Miller . Crowdsourced databases: Query processing with people . In CIDR , pages 211 -- 214 , 2011 . A. Marcus, E. Wu, S. Madden, and R. C. Miller. Crowdsourced databases: Query processing with people. In CIDR, pages 211--214, 2011."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2872427.2883070"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.5555\/42805"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2213836.2213878"},{"key":"e_1_2_1_35_1","unstructured":"QA. https:\/\/webscope.sandbox.yahoo.com\/catalog.php?datatype=l&did=76.  QA. https:\/\/webscope.sandbox.yahoo.com\/catalog.php?datatype=l&did=76."},{"key":"e_1_2_1_36_1","first-page":"1375","volume-title":"ACL","author":"Ratinov L.","year":"2011","unstructured":"L. Ratinov , D. Roth , D. Downey , and M. Anderson . Local and global algorithms for disambiguation to wikipedia . In ACL , pages 1375 -- 1384 , 2011 . L. Ratinov, D. Roth, D. Downey, and M. Anderson. Local and global algorithms for disambiguation to wikipedia. In ACL, pages 1375--1384, 2011."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00778-015-0385-2"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/584091.584093"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2014.2327028"},{"key":"e_1_2_1_40_1","unstructured":"Technical Report. http:\/\/i.cs.hku.hk\/~ydzheng2\/docs_full.pdf.  Technical Report. http:\/\/i.cs.hku.hk\/~ydzheng2\/docs_full.pdf."},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2014.04.005"},{"key":"e_1_2_1_42_1","first-page":"417","volume-title":"ICWS","author":"Trani S.","year":"2014","unstructured":"S. Trani , D. Ceccarelli , C. Lucchese , S. Orlando , and R. Perego . Dexter 2.0-an open source tool for semantically enriching data . In ICWS , pages 417 -- 420 , 2014 . S. Trani, D. Ceccarelli, C. Lucchese, S. Orlando, and R. Perego. Dexter 2.0-an open source tool for semantically enriching data. In ICWS, pages 417--420, 2014."},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.14778\/2350229.2350263"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/2463676.2465280"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.14778\/2536336.2536337"},{"key":"e_1_2_1_46_1","first-page":"2035","volume-title":"NIPS","author":"Whitehill J.","year":"2009","unstructured":"J. Whitehill , T.-f. Wu , J. Bergsma , J. R. Movellan , and P. L. Ruvolo . Whose vote should count more: Optimal integration of labels from labelers of unknown expertise . In NIPS , pages 2035 -- 2043 , 2009 . J. Whitehill, T.-f. Wu, J. Bergsma, J. R. Movellan, and P. L. Ruvolo. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In NIPS, pages 2035--2043, 2009."},{"key":"e_1_2_1_47_1","unstructured":"Wikipedia. https:\/\/en.wikipedia.org\/wiki\/Category:Main_topic_classifications.  Wikipedia. https:\/\/en.wikipedia.org\/wiki\/Category:Main_topic_classifications."},{"key":"e_1_2_1_48_1","unstructured":"Yahoo Answers. https:\/\/answers.yahoo.com\/dir\/index.  Yahoo Answers. https:\/\/answers.yahoo.com\/dir\/index."},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/2505515.2505720"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.14778\/2921558.2921559"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.5555\/1996889.1996934"},{"key":"e_1_2_1_52_1","first-page":"397","volume-title":"EDBT","author":"Zhao Z.","year":"2015","unstructured":"Z. Zhao , F. Wei , M. Zhou , W. Chen , and W. Ng . Crowd-selection query processing in crowdsourcing databases: A task-driven approach . In EDBT , pages 397 -- 408 , 2015 . Z. Zhao, F. Wei, M. Zhou, W. Chen, and W. Ng. Crowd-selection query processing in crowdsourcing databases: A task-driven approach. In EDBT, pages 397--408, 2015."},{"key":"e_1_2_1_53_1","first-page":"193","volume-title":"EDBT","author":"Zheng Y.","year":"2015","unstructured":"Y. Zheng , R. Cheng , S. Maniu , and L. Mo . On optimality of jury selection in crowdsourcing . In EDBT , pages 193 -- 204 , 2015 . Y. Zheng, R. Cheng, S. Maniu, and L. Mo. On optimality of jury selection in crowdsourcing. In EDBT, pages 193--204, 2015."},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2749430"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/2783258.2783339"},{"key":"e_1_2_1_56_1","first-page":"2195","volume-title":"NIPS","author":"Zhou D.","year":"2012","unstructured":"D. Zhou , S. Basu , Y. Mao , and J. C. Platt . Learning from the wisdom of crowds by minimax entropy . In NIPS , pages 2195 -- 2203 , 2012 . D. Zhou, S. Basu, Y. Mao, and J. C. Platt. Learning from the wisdom of crowds by minimax entropy. In NIPS, pages 2195--2203, 2012."}],"container-title":["Proceedings of the VLDB Endowment"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.14778\/3025111.3025118","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,28]],"date-time":"2022-12-28T10:19:47Z","timestamp":1672222787000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.14778\/3025111.3025118"}},"subtitle":["a domain-aware crowdsourcing system using knowledge bases"],"short-title":[],"issued":{"date-parts":[[2016,11]]},"references-count":56,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2016,11]]}},"alternative-id":["10.14778\/3025111.3025118"],"URL":"https:\/\/doi.org\/10.14778\/3025111.3025118","relation":{},"ISSN":["2150-8097"],"issn-type":[{"value":"2150-8097","type":"print"}],"subject":[],"published":{"date-parts":[[2016,11]]}}}