{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,4]],"date-time":"2026-05-04T10:19:43Z","timestamp":1777889983903,"version":"3.51.4"},"reference-count":47,"publisher":"SAGE Publications","issue":"3","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["SW"],"published-print":{"date-parts":[[2018,4,12]]},"DOI":"10.3233\/sw-170261","type":"journal-article","created":{"date-parts":[[2017,1,24]],"date-time":"2017-01-24T12:28:54Z","timestamp":1485260934000},"page":"337-354","source":"Crossref","is-referenced-by-count":0,"title":["Using microtasks to crowdsource DBpedia entity classification: A study in workflow design"],"prefix":"10.1177","volume":"9","author":[{"given":"Qiong","family":"Bu","sequence":"first","affiliation":[{"name":"School of Electronics and Computer Science, University of Southampton, UK. E-mails:\u00a0qb1g13@soton.ac.uk,\u00a0e.simperl@soton.ac.uk,\u00a0s.zerr@soton.ac.uk,\u00a0yl2@soton.ac.uk"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Elena","family":"Simperl","sequence":"additional","affiliation":[{"name":"School of Electronics and Computer Science, University of Southampton, UK. E-mails:\u00a0qb1g13@soton.ac.uk,\u00a0e.simperl@soton.ac.uk,\u00a0s.zerr@soton.ac.uk,\u00a0yl2@soton.ac.uk"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sergej","family":"Zerr","sequence":"additional","affiliation":[{"name":"School of Electronics and Computer Science, University of Southampton, UK. E-mails:\u00a0qb1g13@soton.ac.uk,\u00a0e.simperl@soton.ac.uk,\u00a0s.zerr@soton.ac.uk,\u00a0yl2@soton.ac.uk"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yunjia","family":"Li","sequence":"additional","affiliation":[{"name":"School of Electronics and Computer Science, University of Southampton, UK. E-mails:\u00a0qb1g13@soton.ac.uk,\u00a0e.simperl@soton.ac.uk,\u00a0s.zerr@soton.ac.uk,\u00a0yl2@soton.ac.uk"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"179","reference":[{"key":"10.3233\/SW-170261_ref1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-41338-4_17"},{"issue":"4","key":"10.3233\/SW-170261_ref2","doi-asserted-by":"publisher","first-page":"570","DOI":"10.3758\/BF03194727","article-title":"What makes a categorization task difficult?","volume":"64","author":"Alfonso-Reese","year":"2002","journal-title":"Perception & Psychophysics"},{"key":"10.3233\/SW-170261_ref3","unstructured":"L.\u00a0Aroyo and C.\u00a0Welty, Crowd truth: Harnessing disagreement in crowdsourcing a relation extraction gold standard, in: Proceedings of the 3rd Annual ACM Web Science Conference, 2013 Paris, France, WebSci\u201913, ACM, 2013."},{"key":"10.3233\/SW-170261_ref4","doi-asserted-by":"publisher","DOI":"10.1145\/1866029.1866078"},{"key":"10.3233\/SW-170261_ref5","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0100647"},{"key":"10.3233\/SW-170261_ref6","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2466265"},{"key":"10.3233\/SW-170261_ref7","unstructured":"M.\u00a0Collins and Y.\u00a0Singer, Unsupervised models for named entity classification, in: Proceedings of the 1999 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora, P.\u00a0Fung and J.\u00a0Zhou, eds, University of Maryland College Park, MD, USA, 21\u201322 June, 1999, Association for Computational Linguistics, 1999, pp.\u00a0100\u2013110, http:\/\/aclweb.org\/anthology\/W\/W99\/W99-0613.pdf."},{"key":"10.3233\/SW-170261_ref8","doi-asserted-by":"publisher","DOI":"10.1109\/ISDA.2011.6121700"},{"issue":"5","key":"10.3233\/SW-170261_ref9","doi-asserted-by":"publisher","first-page":"665","DOI":"10.1007\/s00778-013-0324-z","article-title":"Large-scale linked data integration using probabilistic reasoning and crowdsourcing","volume":"22","author":"Demartini","year":"2013","journal-title":"The VLDB Journal"},{"key":"10.3233\/SW-170261_ref10","doi-asserted-by":"publisher","DOI":"10.1145\/2187836.2187900"},{"key":"10.3233\/SW-170261_ref11","doi-asserted-by":"publisher","DOI":"10.1145\/1816123.1816143"},{"key":"10.3233\/SW-170261_ref12","doi-asserted-by":"publisher","DOI":"10.1145\/2736277.2741685"},{"key":"10.3233\/SW-170261_ref13","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-18818-8_32"},{"key":"10.3233\/SW-170261_ref16","unstructured":"J.S.\u00a0Hare, M.\u00a0Acosta, A.\u00a0Weston, E.\u00a0Simperl, S.\u00a0Samangooei, D.\u00a0Dupplaw and P.H.\u00a0Lewis, An investigation of techniques that aim to improve the quality of labels provided by the crowd, in: Proceedings of the MediaEval 2013 Multimedia Benchmark Workshop, M.A.\u00a0Larson, X.\u00a0Anguera, T.\u00a0Reuter, G.J.F.\u00a0Jones, B.\u00a0Ionescu, M.\u00a0Schedl, T.\u00a0Piatrik, C.\u00a0Hauff and M.\u00a0Soleymani, eds, Barcelona, Spain, October 18\u201319, 2013, CEUR Workshop Proceedings, Vol.\u00a01043, 2013, http:\/\/ceur-ws.org\/Vol-1043\/mediaeval2013_submission_44.pdf."},{"issue":"8","key":"10.3233\/SW-170261_ref17","doi-asserted-by":"publisher","first-page":"1509","DOI":"10.1093\/bioinformatics\/bti171","article-title":"Optimal number of features as a function of sample size for various classification rules","volume":"21","author":"Hua","year":"2005","journal-title":"Bioinformatics"},{"issue":"2","key":"10.3233\/SW-170261_ref18","doi-asserted-by":"publisher","first-page":"16","DOI":"10.1145\/1869086.1869094","article-title":"Analyzing the Amazon mechanical turk marketplace","volume":"17","author":"Ipeirotis","year":"2010","journal-title":"ACM Crossroads"},{"key":"10.3233\/SW-170261_ref19","doi-asserted-by":"publisher","DOI":"10.1145\/1837885.1837906"},{"issue":"6","key":"10.3233\/SW-170261_ref20","doi-asserted-by":"publisher","first-page":"995","DOI":"10.1037\/0022-3514.79.6.995","article-title":"When choice is demotivating: Can one desire too much of a good thing?","volume":"79","author":"Iyengar","year":"2000","journal-title":"Journal of Personality and Social Psychology"},{"key":"10.3233\/SW-170261_ref21","unstructured":"E.\u00a0Kamar, S.\u00a0Hacker and E.\u00a0Horvitz, Combining human and machine intelligence in large-scale crowdsourcing, in: International Conference on Autonomous Agents and Multiagent Systems (3 Volumes), W.\u00a0van der Hoek, L.\u00a0Padgham, V.\u00a0Conitzer and M.\u00a0Winikoff, eds, AAMAS 2012, Valencia, Spain, June 4\u20138, 2012, IFAAMAS, 2012, pp.\u00a0467\u2013474. http:\/\/dl.acm.org\/citation.cfm?id=2343643."},{"key":"10.3233\/SW-170261_ref22","doi-asserted-by":"crossref","unstructured":"J.-H.\u00a0Kim, I.-H.\u00a0Kang and K.-S.\u00a0Choi, Unsupervised named entity classification models and their ensembles, in: 19th International Conference on Computational Linguistics, COLING 2002, Howard International House and Academia Sinica, Taipei, Taiwan, August 24\u2013September 1, 2002, Association for Computational Linguistics, 2002. http:\/\/aclweb.org\/anthology\/C02-1088.","DOI":"10.3115\/1072228.1072316"},{"key":"10.3233\/SW-170261_ref23","doi-asserted-by":"publisher","DOI":"10.1145\/1357054.1357127"},{"issue":"1","key":"10.3233\/SW-170261_ref25","doi-asserted-by":"publisher","first-page":"159","DOI":"10.2307\/2529310","article-title":"The measurement of observer agreement for categorical data","volume":"33","author":"Landis","year":"1977","journal-title":"Biometrics"},{"issue":"2","key":"10.3233\/SW-170261_ref26","doi-asserted-by":"publisher","first-page":"167","DOI":"10.3233\/SW-140134","article-title":"Dbpedia \u2013 A large-scale, multilingual knowledge base extracted from Wikipedia","volume":"6","author":"Lehmann","year":"2015","journal-title":"Semantic Web"},{"issue":"3","key":"10.3233\/SW-170261_ref27","doi-asserted-by":"publisher","first-page":"1179","DOI":"10.1111\/j.1365-2966.2008.13689.x","article-title":"Galaxy Zoo: Morphologies derived from visual inspection of galaxies from the Sloan Digital Sky Survey","volume":"389","author":"Lintott","year":"2008","journal-title":"Monthly Notices of the Royal Astronomical Society"},{"key":"10.3233\/SW-170261_ref28","doi-asserted-by":"publisher","DOI":"10.1145\/1866029.1866040"},{"key":"10.3233\/SW-170261_ref29","doi-asserted-by":"publisher","DOI":"10.1145\/2660114.2660123"},{"issue":"2","key":"10.3233\/SW-170261_ref30","doi-asserted-by":"publisher","first-page":"81","DOI":"10.1037\/h0043158","article-title":"The magical number seven, plus or minus two: Some limits on our capacity for processing information","volume":"63","author":"Miller","year":"1956","journal-title":"Psychological Review"},{"key":"10.3233\/SW-170261_ref31","doi-asserted-by":"crossref","unstructured":"C.\u00a0Niu, W.\u00a0Li, J.\u00a0Ding and R.K.\u00a0Srihari, A bootstrapping approach to named entity classification using successive learners, in: Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, E.W.\u00a0Hinrichs and D.\u00a0Roth, eds, Sapporo Convention Center, Sapporo, Japan, 7\u201312 July, 2003, ACL, 2003, pp.\u00a0335\u2013342. http:\/\/aclweb.org\/anthology\/P\/P03\/P03-1043.pdf.","DOI":"10.3115\/1075096.1075139"},{"issue":"3","key":"10.3233\/SW-170261_ref32","doi-asserted-by":"publisher","first-page":"382","DOI":"10.1016\/0010-0285(76)90013-X","article-title":"Basic objects in natural categories","volume":"8","author":"Rosch","year":"1976","journal-title":"Cognitive Osychology"},{"key":"10.3233\/SW-170261_ref33","doi-asserted-by":"publisher","DOI":"10.1145\/1719970.1720006"},{"key":"10.3233\/SW-170261_ref34","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-35176-1_33"},{"issue":"3","key":"10.3233\/SW-170261_ref35","doi-asserted-by":"publisher","first-page":"229","DOI":"10.1002\/mar.20271","article-title":"What moderates the too-much-choice effect?","volume":"26","author":"Scheibehenne","year":"2009","journal-title":"Psychology and Marketing"},{"key":"10.3233\/SW-170261_ref36","doi-asserted-by":"crossref","unstructured":"B.\u00a0Schwartz, The Paradox of Choice, Ecco, 2004.","DOI":"10.1037\/e597322010-001"},{"key":"10.3233\/SW-170261_ref37","unstructured":"D.\u00a0Shahaf and E.\u00a0Horvitz, Generalized task markets for human and machine computation, in: Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2010, M.\u00a0Fox and D.\u00a0Poole, eds, Atlanta, Georgia, USA, July 11\u201315, 2010, AAAI Press, 2010. http:\/\/www.aaai.org\/ocs\/index.php\/AAAI\/AAAI10\/paper\/view\/1951."},{"key":"10.3233\/SW-170261_ref38","doi-asserted-by":"publisher","DOI":"10.1145\/1401890.1401965"},{"key":"10.3233\/SW-170261_ref39","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-36406-8_1"},{"issue":"1\u20132","key":"10.3233\/SW-170261_ref40","doi-asserted-by":"publisher","first-page":"33","DOI":"10.1007\/s11280-009-0078-0","article-title":"Human intelligence in the process of semantic content creation","volume":"13","author":"Siorpaes","year":"2010","journal-title":"World Wide Web"},{"key":"10.3233\/SW-170261_ref41","doi-asserted-by":"crossref","unstructured":"R.\u00a0Snow, B.\u00a0O\u2019Connor, D.\u00a0Jurafsky and A.Y.\u00a0Ng, Cheap and fast \u2013 But is it good? Evaluating non-expert annotations for natural language tasks, in: A Meeting of SIGDAT, a Special Interest Group of the ACL, 2008 Conference on Empirical Methods in Natural Language Processing, EMNLP 2008, Proceedings of the Conference, Honolulu, Hawaii, USA, 25\u201327 October, 2008, ACL, 2008, pp.\u00a0254\u2013263. http:\/\/www.aclweb.org\/anthology\/D08-1027.","DOI":"10.3115\/1613715.1613751"},{"issue":"13","key":"10.3233\/SW-170261_ref42","doi-asserted-by":"publisher","first-page":"1529","DOI":"10.14778\/2733004.2733024","article-title":"Chimera: Large-scale classification using machine learning, rules, and crowdsourcing","volume":"7","author":"Sun","year":"2014","journal-title":"Proceedings of the VLDB Endowment"},{"issue":"3","key":"10.3233\/SW-170261_ref43","doi-asserted-by":"publisher","first-page":"457","DOI":"10.1016\/0010-0285(91)90016-H","article-title":"Object categories and expertise: Is the basic level in the eye of the beholder?","volume":"23","author":"Tanaka","year":"1991","journal-title":"Cognitive Psychology"},{"key":"10.3233\/SW-170261_ref44","doi-asserted-by":"crossref","unstructured":"D.\u00a0Vickrey, A.\u00a0Bronzan, W.\u00a0Choi, A.\u00a0Kumar, J.\u00a0Turner-Maier, A.\u00a0Wang and D.\u00a0Koller, Online word games for semantic data collection, in: A Meeting of SIGDAT, a Special Interest Group of the ACL, 2008 Conference on Empirical Methods in Natural Language Processing, EMNLP 2008, Proceedings of the Conference, Honolulu, Hawaii, USA, 25\u201327 October 2008, ACL, 2008, pp.\u00a0533\u2013542. http:\/\/www.aclweb.org\/anthology\/D08-1056.","DOI":"10.3115\/1613715.1613781"},{"issue":"6","key":"10.3233\/SW-170261_ref45","doi-asserted-by":"publisher","first-page":"92","DOI":"10.1109\/MC.2006.196","article-title":"Games with a purpose","volume":"39","author":"von Ahn","year":"2006","journal-title":"IEEE Computer"},{"key":"10.3233\/SW-170261_ref46","doi-asserted-by":"publisher","DOI":"10.1145\/985692.985733"},{"issue":"11","key":"10.3233\/SW-170261_ref47","doi-asserted-by":"publisher","first-page":"1483","DOI":"10.14778\/2350229.2350263","article-title":"Crowder: Crowdsourcing entity resolution","volume":"5","author":"Wang","year":"2012","journal-title":"Proceedings of the VLDB Endowment"},{"key":"10.3233\/SW-170261_ref48","unstructured":"J.\u00a0Whitehill, P.\u00a0Ruvolo, T.\u00a0Wu, J.\u00a0Bergsma and J.R.\u00a0Movellan, Whose vote should count more: Optimal integration of labels from labelers of unknown expertise, in: Advances in Neural Information Processing Systems 22: 23rd Annual Conference on Neural Information Processing Systems 2009, Y.\u00a0Bengio, D.\u00a0Schuurmans, J.D.\u00a0Lafferty, C.K.I.\u00a0Williams and A.\u00a0Culotta, eds, 7\u201310 December, 2009, Vancouver, British Columbia, Canada, Curran Associates, Inc., 2009, pp.\u00a02035\u20132043. http:\/\/papers.nips.cc\/paper\/3644-whose-vote-should-count-more-optimal-integration-of-labels-from-labelers-of-unknown-expertise."},{"key":"10.3233\/SW-170261_ref49","doi-asserted-by":"publisher","DOI":"10.1007\/11683568_8"},{"key":"10.3233\/SW-170261_ref50","doi-asserted-by":"publisher","DOI":"10.1145\/2506182.2506195"}],"container-title":["Semantic Web"],"original-title":[],"link":[{"URL":"https:\/\/content.iospress.com\/download?id=10.3233\/SW-170261","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,1]],"date-time":"2026-05-01T05:23:39Z","timestamp":1777613019000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/full\/10.3233\/SW-170261"}},"subtitle":[],"editor":[{"given":"Marta","family":"Sabou","sequence":"additional","affiliation":[{"name":"Technische Universit\u00e4t Vienna, Austria"}],"role":[{"role":"editor","vocabulary":"crossref"}]},{"given":"Lora","family":"Aroyo","sequence":"additional","affiliation":[{"name":"Vrije Universiteit Amsterdam, Netherlands"}],"role":[{"role":"editor","vocabulary":"crossref"}]},{"given":"Kalina","family":"Bontcheva","sequence":"additional","affiliation":[{"name":"University of Sheffield, UK"}],"role":[{"role":"editor","vocabulary":"crossref"}]},{"given":"Alessandro","family":"Bozzon","sequence":"additional","affiliation":[{"name":"Technische Universiteit Delft, Netherlands"}],"role":[{"role":"editor","vocabulary":"crossref"}]}],"short-title":[],"issued":{"date-parts":[[2018,4,12]]},"references-count":47,"journal-issue":{"issue":"3"},"URL":"https:\/\/doi.org\/10.3233\/sw-170261","relation":{},"ISSN":["2210-4968","1570-0844"],"issn-type":[{"value":"2210-4968","type":"electronic"},{"value":"1570-0844","type":"print"}],"subject":[],"published":{"date-parts":[[2018,4,12]]}}}