{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2024,7,25]],"date-time":"2024-07-25T05:11:21Z","timestamp":1721884281080},"reference-count":64,"publisher":"Tsinghua University Press","issue":"3","license":[{"start":{"date-parts":[[2019,9,2]],"date-time":"2019-09-02T00:00:00Z","timestamp":1567382400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.emerald.com\/insight\/site-policies"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["IJCS"],"published-print":{"date-parts":[[2019,9,2]]},"abstract":"Purpose<\/jats:title>Ensuring quality is one of the most significant challenges in microtask crowdsourcing tasks. Aggregation of the collected data from the crowd is one of the important steps to infer the correct answer, but the existing study seems to be limited to the single-step task. This study aims to look at multiple-step classification tasks and understand aggregation in such cases; hence, it is useful for assessing the classification quality.<\/jats:p><\/jats:sec>Design\/methodology\/approach<\/jats:title>The authors present a model to capture the information of the workflow, questions and answers for both single- and multiple-question classification tasks. They propose an adapted approach on top of the classic approach so that the model can handle tasks with several multiple-choice questions in general instead of a specific domain or any specific hierarchical classifications. They evaluate their approach with three representative tasks from existing citizen science projects in which they have the gold standard created by experts.<\/jats:p><\/jats:sec>Findings<\/jats:title>The results show that the approach can provide significant improvements to the overall classification accuracy. The authors\u2019 analysis also demonstrates that all algorithms can achieve higher accuracy for the volunteer- versus paid-generated data sets for the same task. Furthermore, the authors observed interesting patterns in the relationship between the performance of different algorithms and workflow-specific factors including the number of steps and the number of available options in each step.<\/jats:p><\/jats:sec>Originality\/value<\/jats:title>Due to the nature of crowdsourcing, aggregating the collected data is an important process to understand the quality of crowdsourcing results. Different inference algorithms have been studied for simple microtasks consisting of single questions with two or more answers. However, as classification tasks typically contain many questions, the proposed method can be applied to a wide range of tasks including both single- and multiple-question classification tasks.<\/jats:p><\/jats:sec>","DOI":"10.1108\/ijcs-06-2019-0017","type":"journal-article","created":{"date-parts":[[2019,10,17]],"date-time":"2019-10-17T12:51:06Z","timestamp":1571316666000},"page":"222-248","source":"Crossref","is-referenced-by-count":3,"title":["Quality assessment in crowdsourced classification tasks"],"prefix":"10.26599","volume":"3","author":[{"given":"Qiong","family":"Bu","sequence":"first","affiliation":[]},{"given":"Elena","family":"Simperl","sequence":"additional","affiliation":[]},{"given":"Adriane","family":"Chapman","sequence":"additional","affiliation":[]},{"given":"Eddy","family":"Maddalena","sequence":"additional","affiliation":[]}],"member":"11138","reference":[{"key":"ref39","doi-asserted-by":"publisher","DOI":"10.1109\/EMR.2010.5559142"},{"key":"ref38","first-page":"69","article-title":"Getting by with a little help from the crowd: practical approaches to social image labeling","author":"loni","year":"0","journal-title":"Proceedings of the 2014 International ACM Workshop on Crowdsourcing for Multimedia"},{"key":"ref33","first-page":"43","article-title":"CrowdForge: Crowdsourcing complex work","author":"kittur","year":"0","journal-title":"Proceedings of the 24th annual ACM symposium on User interface software and technology - UIST '11"},{"key":"ref32","doi-asserted-by":"crossref","first-page":"1301","DOI":"10.1145\/2441776.2441923","article-title":"The future of crowd work","author":"kittur","year":"2013","journal-title":"Proceedings of the 2013 Conference on Computer Supported Cooperative Work-CSCW'13"},{"key":"ref31","doi-asserted-by":"crossref","first-page":"453","DOI":"10.1145\/1357054.1357127","article-title":"Crowdsourcing user studies with mechanical Turk","author":"kittur","year":"2008","journal-title":"Proceedings of the SIGCHI conference on Human factors in computing systems"},{"key":"ref30","doi-asserted-by":"publisher","DOI":"10.3115\/1072228.1072316"},{"key":"ref37","doi-asserted-by":"crossref","first-page":"1040","DOI":"10.14778\/2336664.2336676","article-title":"CDAS: a crowdsourcing data analytics system","volume":"5","author":"liu","year":"0","journal-title":"Proceedings of the VLDB Endowment"},{"key":"ref36","doi-asserted-by":"publisher","DOI":"10.1145\/1600150.1600159"},{"key":"ref35","doi-asserted-by":"crossref","first-page":"166","DOI":"10.1111\/j.1365-2966.2010.17432.x","article-title":"Galaxy zoo 1: data release of morphological classifications for nearly 900 000 galaxies","volume":"410","author":"lintott","year":"2011","journal-title":"Monthly Notices of the Royal Astronomical Society"},{"key":"ref34","first-page":"2053","article-title":"Turkomatic","author":"kulkarni","year":"0","journal-title":"Proceedings of the 2011 annual conference extended abstracts on Human factors in computing systems – CHI EA '11"},{"key":"ref60","article-title":"Quality assessment methodologies for linked open data","author":"zaveri","year":"2013","journal-title":"Semantic Web"},{"key":"ref62","doi-asserted-by":"publisher","DOI":"10.14778\/3055540.3055547"},{"key":"ref61","doi-asserted-by":"publisher","DOI":"10.1016\/j.ins.2016.12.026"},{"key":"ref63","author":"wang","year":"2015","journal-title":"Cost-effective quality assurance in crowd labeling"},{"key":"ref28","first-page":"1953","article-title":"Iterative learning for reliable crowdsourcing systems","author":"karger","year":"2011","journal-title":"Advances in neural information processing systems"},{"key":"ref64","author":"yoram","year":"2012","journal-title":"How to grade a test without knowing the answers - a bayesian graphical model for adaptive crowdsourcing and aptitude testing"},{"key":"ref27","first-page":"2030","article-title":"Planning for crowdsourcing hierarchical tasks","author":"kamar","year":"0","journal-title":"Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems"},{"key":"ref29","first-page":"1","article-title":"Quality control of crowd labeling through expert evaluation","author":"khattak","year":"0","journal-title":"Second Workshop Comput Soc Sci Wisdom Crowds"},{"key":"ref2","author":"bachrach","year":"2012","journal-title":"How to grade a test without knowing the answers – a Bayesian graphical model for adaptive crowdsourcing and aptitude testing"},{"key":"ref1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-41338-4_17"},{"key":"ref20","first-page":"78","article-title":"Interpretation of crowdsourced activities using provenance network analysis","author":"huynh","year":"0","journal-title":"The First AAAI Conference on Human Computation and Crowdsourcing"},{"key":"ref22","doi-asserted-by":"publisher","DOI":"10.1145\/1837885.1837906"},{"key":"ref21","doi-asserted-by":"publisher","DOI":"10.1007\/s10618-013-0306-1"},{"key":"ref24","first-page":"1042008","article-title":"International vocabulary of metrology? Basic and general concepts and associated terms (VIM) vocabulaire international de metrologie? Concepts fondamentaux et généraux et termes associés (VIM)","volume":"3","year":"2008","journal-title":"International Organization for Standardization Geneva"},{"key":"ref23","doi-asserted-by":"publisher","DOI":"10.1145\/1837885.1837906"},{"key":"ref26","first-page":"467","article-title":"Combining human and machine intelligence in large-scale crowdsourcing","volume":"1","author":"kamar","year":"0","journal-title":"Proceedings of the 11 th International Conference on Autonomous Agents and Multiagent Systems International Foundation for Autonomous Agents and Multiagent Systems"},{"key":"ref25","doi-asserted-by":"publisher","DOI":"10.1145\/505248.506007"},{"key":"ref50","article-title":"Bayesian combination of multiple, imperfect classifiers","author":"simpson","year":"2011","journal-title":"Proceedings of the 25th Conference on Neural Information Processing Systems"},{"key":"ref51","first-page":"254","article-title":"Cheap and fast – but is it good? Evaluating non-expert annotations for natural language tasks","author":"snow","year":"0","journal-title":"Proceedings of the Conference on Empirical Methods in Natural Language Processing Association for Computational Linguistics"},{"key":"ref59","author":"yang","year":"2016","journal-title":"Modeling task complexity in crowdsourcing"},{"key":"ref58","first-page":"1713","article-title":"Sembler: ensembling crowd sequential labeling for improved quality","volume":"2","author":"wu","year":"0","journal-title":"Proceedings of the National Conference on Artificial Intelligence"},{"key":"ref57","doi-asserted-by":"crossref","first-page":"2835","DOI":"10.1093\/mnras\/stt1458","article-title":"Galaxy zoo 2: detailed morphological classifications for 304 122 galaxies from the sloan digital sky survey","volume":"435","author":"willett","year":"2013","journal-title":"Monthly Notices of the Royal Astronomical Society"},{"key":"ref56","doi-asserted-by":"crossref","first-page":"14","DOI":"10.1109\/eScienceW.2011.27","article-title":"Mechanisms for data quality and validation in citizen science","author":"wiggins","year":"2011","journal-title":"e-Science Workshops (eScience W) 20111 IEEE Seventh International Conference on"},{"key":"ref55","first-page":"1","article-title":"Whose vote should count more: optimal integration of labels from labelers of unknown expertise","volume":"22","author":"whitehill","year":"2009","journal-title":"Advances in neural information processing systems"},{"key":"ref54","doi-asserted-by":"publisher","DOI":"10.3115\/1613715.1613781"},{"key":"ref53","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v29i1.9338"},{"key":"ref52","doi-asserted-by":"publisher","DOI":"10.1007\/11912873_23"},{"key":"ref10","first-page":"238","article-title":"Ipeirotis, and Philippe Cudré-Mauroux","author":"difallah","year":"2015","journal-title":"The dynamics of micro-task crowdsourcing the case of Amazon MTurk"},{"key":"ref11","first-page":"679","article-title":"Crowdsourcing the general public for large scale molecular pathology studies in cancer","volume":"2","author":"dos reis","year":"2015","journal-title":"EBioMedicine"},{"key":"ref40","first-page":"94","article-title":"Volunteering versus work for pay: incentives and tradeoffs in crowdsourcing","author":"mao","year":"0","journal-title":"First AAAI Conference on Human Computation and Crowdsourcing"},{"key":"ref12","first-page":"256","author":"dumais","year":"2000","journal-title":"Hierarchical classification of web content"},{"key":"ref13","first-page":"11","article-title":"How crowdsourcable is your task","author":"eickhoff","year":"0","journal-title":"Proceedings of the workshop on crowdsourcing for search and data mining (CSDM) at the fourth ACM international conference on web search and data mining (WSDM)"},{"key":"ref14","doi-asserted-by":"publisher","DOI":"10.1109\/MIS.2015.66"},{"key":"ref15","doi-asserted-by":"publisher","DOI":"10.1145\/2631775.2631819"},{"key":"ref16","first-page":"3065","author":"gelas","year":"2011","journal-title":"Quality assessment of crowdsourcing transcriptions for African languages"},{"key":"ref17","article-title":"An investigation of techniques that aim to improve the quality of labels provided by the crowd","volume":"1043","author":"hare","year":"2013","journal-title":"Proceedings of the MediaEval 2013 Multimedia Benchmark Workshop"},{"key":"ref18","first-page":"1","article-title":"An evaluation of aggregation techniques in crowdsourcing","volume":"8181","author":"hung","year":"2013","journal-title":"Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)"},{"key":"ref19","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2723731"},{"key":"ref4","doi-asserted-by":"publisher","DOI":"10.1145\/1866029.1866078"},{"key":"ref3","doi-asserted-by":"publisher","DOI":"10.1145\/1541880.1541883"},{"key":"ref6","article-title":"Crowdsourcing multi-label classification for taxonomy creation","author":"bragg","year":"0","journal-title":"HCOMP 2013 First AAAI Conference on Human Computation and Crowdsourcing"},{"key":"ref5","doi-asserted-by":"publisher","DOI":"10.1145\/1866029.1866078"},{"key":"ref8","doi-asserted-by":"publisher","DOI":"10.1145\/2187836.2187900"},{"key":"ref7","doi-asserted-by":"publisher","DOI":"10.2307\/2346806"},{"key":"ref49","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-36406-8_1"},{"key":"ref9","first-page":"238","author":"difallah","year":"2015","journal-title":"The dynamics of micro-task crowdsourcing the case of Amazon MTurk"},{"key":"ref46","doi-asserted-by":"publisher","DOI":"10.1145\/1719970.1720006"},{"key":"ref45","doi-asserted-by":"publisher","DOI":"10.1145\/2660114.2660118"},{"key":"ref48","first-page":"156","article-title":"SQUARE: a benchmark for research on computing crowd consensus","author":"sheshadri","year":"0","journal-title":"First AAAI Conference on Human Computation and …"},{"key":"ref47","article-title":"Generalized task markets for human and machine computation","author":"shahaf","year":"0","journal-title":"AAAI"},{"key":"ref42","doi-asserted-by":"publisher","DOI":"10.14778\/1952376.1952377"},{"key":"ref41","first-page":"937","article-title":"Quality control for crowdsourced hierarchical classification","volume":"2016","author":"otani","year":"2016","journal-title":"Proceedings of the IEEE International Conference on Data Mining"},{"key":"ref44","doi-asserted-by":"crossref","first-page":"88","DOI":"10.1080\/00031305.1994.10476030","article-title":"The three sigma rule","volume":"48","author":"pukelsheim","year":"1994","journal-title":"The American Statistician"},{"key":"ref43","doi-asserted-by":"publisher","DOI":"10.4018\/ijswis.2014040104"}],"container-title":["International Journal of Crowd Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/IJCS-06-2019-0017\/full\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/IJCS-06-2019-0017\/full\/html","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,7,25]],"date-time":"2024-07-25T04:32:24Z","timestamp":1721881944000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/IJCS-06-2019-0017\/full\/html"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,9,2]]},"references-count":64,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2019,9,2]]}},"alternative-id":["10.1108\/IJCS-06-2019-0017"],"URL":"https:\/\/doi.org\/10.1108\/ijcs-06-2019-0017","relation":{},"ISSN":["2398-7294","2398-7294"],"issn-type":[{"value":"2398-7294","type":"print"},{"value":"2398-7294","type":"print"}],"subject":[],"published":{"date-parts":[[2019,9,2]]}}}