{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,29]],"date-time":"2026-05-29T12:10:23Z","timestamp":1780056623133,"version":"3.54.0"},"reference-count":220,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2018,1,4]],"date-time":"2018-01-04T00:00:00Z","timestamp":1515024000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"ARC","award":["DP1601104515"],"award-info":[{"award-number":["DP1601104515"]}]},{"name":"Integrating Quality Control into Crowd-Sourcing Services"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Comput. Surv."],"published-print":{"date-parts":[[2019,1,31]]},"abstract":"<jats:p>Crowdsourcing enables one to leverage on the intelligence and wisdom of potentially large groups of individuals toward solving problems. Common problems approached with crowdsourcing are labeling images, translating or transcribing text, providing opinions or ideas, and similar\u2014all tasks that computers are not good at or where they may even fail altogether. The introduction of humans into computations and\/or everyday work, however, also poses critical, novel challenges in terms of quality control, as the crowd is typically composed of people with unknown and very diverse abilities, skills, interests, personal objectives, and technological resources. This survey studies quality in the context of crowdsourcing along several dimensions, so as to define and characterize it and to understand the current state of the art. Specifically, this survey derives a quality model for crowdsourcing tasks, identifies the methods and techniques that can be used to assess the attributes of the model, and the actions and strategies that help prevent and mitigate quality problems. An analysis of how these features are supported by the state of the art further identifies open issues and informs an outlook on hot future research directions.<\/jats:p>","DOI":"10.1145\/3148148","type":"journal-article","created":{"date-parts":[[2018,1,4]],"date-time":"2018-01-04T16:27:31Z","timestamp":1515083251000},"page":"1-40","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":280,"title":["Quality Control in Crowdsourcing"],"prefix":"10.1145","volume":"51","author":[{"given":"Florian","family":"Daniel","sequence":"first","affiliation":[{"name":"Politecnico di Milano, Milano, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pavel","family":"Kucherbaev","sequence":"additional","affiliation":[{"name":"Delft University of Technology"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cinzia","family":"Cappiello","sequence":"additional","affiliation":[{"name":"Politecnico di Milano, Milano, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Boualem","family":"Benatallah","sequence":"additional","affiliation":[{"name":"University of New South Wales, CSE, Sydney, NSW, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mohammad","family":"Allahbakhsh","sequence":"additional","affiliation":[{"name":"University of Zabol, Zabol, Iran"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2018,1,4]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2911514"},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/1242572.1242608"},{"key":"e_1_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-19437-5_23"},{"key":"e_1_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4614-6396-2_1"},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047203"},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2006.196"},{"key":"e_1_2_2_7_1","volume-title":"CHI","author":"Sampath Harini Alagarai","year":"2014","unstructured":"Harini Alagarai Sampath , Rajeev Rajeshuni , and Bipin Indurkhya . 2014 . Cognitively inspired task design to improve user performance on crowdsourcing platforms . In CHI 2014. 3665--3674. Harini Alagarai Sampath, Rajeev Rajeshuni, and Bipin Indurkhya. 2014. Cognitively inspired task design to improve user performance on crowdsourcing platforms. In CHI 2014. 3665--3674."},{"key":"e_1_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/MIC.2013.20"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.4108\/icst.collaboratecom.2012.250499"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/SOCA.2014.26"},{"key":"e_1_2_2_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145382"},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/2814573"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2488388.2488398"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/2505515.2507884"},{"key":"e_1_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2556288.2557158"},{"key":"e_1_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.websem.2007.03.002"},{"key":"e_1_2_2_17_1","volume-title":"26th IAAI Conference.","author":"Aydin Bahadir Ismail","year":"2014","unstructured":"Bahadir Ismail Aydin , Yavuz Selim Yilmaz , Yaliang Li , Qi Li , Jing Gao , and Murat Demirbas . 2014 . Crowdsourcing for multiple-choice question answering . In 26th IAAI Conference. Bahadir Ismail Aydin, Yavuz Selim Yilmaz, Yaliang Li, Qi Li, Jing Gao, and Murat Demirbas. 2014. Crowdsourcing for multiple-choice question answering. In 26th IAAI Conference."},{"key":"e_1_2_2_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2983323.2983716"},{"key":"e_1_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/1541880.1541883"},{"key":"e_1_2_2_20_1","volume-title":"Analytic methods for optimizing realtime crowdsourcing. CoRR abs\/1204.2995","author":"Bernstein Michael S.","year":"2012","unstructured":"Michael S. Bernstein , David R. Karger , Robert C. Miller , and Joel Brandt . 2012. Analytic methods for optimizing realtime crowdsourcing. CoRR abs\/1204.2995 ( 2012 ). http:\/\/arxiv.org\/abs\/1204.2995 Michael S. Bernstein, David R. Karger, Robert C. Miller, and Joel Brandt. 2012. Analytic methods for optimizing realtime crowdsourcing. CoRR abs\/1204.2995 (2012). http:\/\/arxiv.org\/abs\/1204.2995"},{"key":"e_1_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/1866029.1866078"},{"key":"e_1_2_2_22_1","volume-title":"UIST 2010 (UIST\u201910)","author":"Bigham Jeffrey P.","year":"2010","unstructured":"Jeffrey P. Bigham , Chandrika Jayant , Hanjie Ji , Greg Little , Andrew Miller , Robert C. Miller , Robin Miller , Aubrey Tatarowicz , Brandyn White , Samual White , and Tom Yeh . 2010 . VizWiz: Nearly real-time answers to visual questions . In UIST 2010 (UIST\u201910) . 333--342. Jeffrey P. Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C. Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, and Tom Yeh. 2010. VizWiz: Nearly real-time answers to visual questions. In UIST 2010 (UIST\u201910). 333--342."},{"key":"e_1_2_2_23_1","volume-title":"Enhancing Learning Through Self-assessment","author":"Boud David","unstructured":"David Boud . 2013. Enhancing Learning Through Self-assessment . Routledge . David Boud. 2013. Enhancing Learning Through Self-assessment. Routledge."},{"key":"e_1_2_2_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2971648.2971741"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2187836.2187971"},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/2488388.2488403"},{"key":"e_1_2_2_27_1","volume-title":"1st AAAI Conference on Human Computation and Crowdsourcing.","author":"Bragg Jonathan","year":"2013","unstructured":"Jonathan Bragg , Daniel S. Weld , and others. 2013 . Crowdsourcing multi-label classification for taxonomy creation . In 1st AAAI Conference on Human Computation and Crowdsourcing. Jonathan Bragg, Daniel S. Weld, and others. 2013. Crowdsourcing multi-label classification for taxonomy creation. In 1st AAAI Conference on Human Computation and Crowdsourcing."},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/2623330.2623717"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-22233-7_10"},{"key":"e_1_2_2_30_1","volume-title":"AAAI","author":"Caragiannis Ioannis","year":"2014","unstructured":"Ioannis Caragiannis , Ariel D. Procaccia , and Nisarg Shah . 2014 . Modal ranking: A uniquely robust voting rule . In AAAI 2014. 616--622. Ioannis Caragiannis, Ariel D. Procaccia, and Nisarg Shah. 2014. Modal ranking: A uniquely robust voting rule. In AAAI 2014. 616--622."},{"key":"e_1_2_2_31_1","volume-title":"Proceedings of AAMAS 2012 -","volume":"2","author":"Cavallo Ruggiero","year":"2012","unstructured":"Ruggiero Cavallo and Shaili Jain . 2012 . Efficient crowdsourcing contests . In Proceedings of AAMAS 2012 - Volume 2 . 677--686. Ruggiero Cavallo and Shaili Jain. 2012. Efficient crowdsourcing contests. In Proceedings of AAMAS 2012 - Volume 2. 677--686."},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4614-8806-4_30"},{"key":"e_1_2_2_33_1","volume-title":"ICML","volume":"28","author":"Chen Xi","year":"2013","unstructured":"Xi Chen , Qihang Lin , and Dengyong Zhou . 2013 . Optimistic knowledge gradient policy for optimal budget allocation in crowdsourcing . In ICML 2013, Vol. 28 , 64--72. Xi Chen, Qihang Lin, and Dengyong Zhou. 2013. Optimistic knowledge gradient policy for optimal budget allocation in crowdsourcing. In ICML 2013, Vol. 28, 64--72."},{"key":"e_1_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702145"},{"key":"e_1_2_2_35_1","volume-title":"CHI","author":"Cheng Justin","year":"2015","unstructured":"Justin Cheng , Jaime Teevan , Shamsi T. Iqbal , and Michael S. Bernstein . 2015b. Break it down: A comparison of macro- and microtasks . In CHI 2015 . 4061--4064. http:\/\/doi.acm.org\/10.1145\/2702123.2702146 Justin Cheng, Jaime Teevan, Shamsi T. Iqbal, and Michael S. Bernstein. 2015b. Break it down: A comparison of macro- and microtasks. In CHI 2015. 4061--4064. http:\/\/doi.acm.org\/10.1145\/2702123.2702146"},{"key":"e_1_2_2_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/1837885.1837889"},{"key":"e_1_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2675133.2675260"},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/2488388.2488414"},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2806416.2806460"},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/1978542.1978560"},{"key":"e_1_2_2_41_1","volume-title":"HCOMP","author":"Alfaro Luca De","year":"2015","unstructured":"Luca De Alfaro , Vassilis Polychronopoulos , and Michael Shavlovsky . 2015 . Reliable aggregation of boolean crowdsourced tasks . In HCOMP 2015. Luca De Alfaro, Vassilis Polychronopoulos, and Michael Shavlovsky. 2015. Reliable aggregation of boolean crowdsourced tasks. In HCOMP 2015."},{"key":"e_1_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00778-013-0324-z"},{"key":"e_1_2_2_43_1","volume-title":"HCOMP","author":"Difallah Djellel Eddine","year":"2014","unstructured":"Djellel Eddine Difallah , Michele Catasta , Gianluca Demartini , and Philippe Cudr\u00e9-Mauroux . 2014 . Scaling-up the crowd: Micro-task pricing schemes for worker retention and latency improvement . In HCOMP 2014. Djellel Eddine Difallah, Michele Catasta, Gianluca Demartini, and Philippe Cudr\u00e9-Mauroux. 2014. Scaling-up the crowd: Micro-task pricing schemes for worker retention and latency improvement. In HCOMP 2014."},{"key":"e_1_2_2_44_1","unstructured":"Djellel Eddine Difallah Gianluca Demartini and Philippe Cudr\u00e9-Mauroux. 2012. Mechanical cheat: Spamming schemes and adversarial techniques on crowdsourcing platforms. In CrowdSearch. 26--30.  Djellel Eddine Difallah Gianluca Demartini and Philippe Cudr\u00e9-Mauroux. 2012. Mechanical cheat: Spamming schemes and adversarial techniques on crowdsourcing platforms. In CrowdSearch. 26--30."},{"key":"e_1_2_2_45_1","volume-title":"WWW","author":"Difallah Djellel Eddine","year":"2013","unstructured":"Djellel Eddine Difallah , Gianluca Demartini , and Philippe Cudr\u00e9-Mauroux . 2013 . Pick-a-crowd: Tell me what you like, and I\u2019ll tell you what to do . In WWW 2013. 367--374. Djellel Eddine Difallah, Gianluca Demartini, and Philippe Cudr\u00e9-Mauroux. 2013. Pick-a-crowd: Tell me what you like, and I\u2019ll tell you what to do. In WWW 2013. 367--374."},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/2872427.2883030"},{"key":"e_1_2_2_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/2556288.2557217"},{"key":"e_1_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/MIC.2012.33"},{"key":"e_1_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858268"},{"key":"e_1_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145355"},{"key":"e_1_2_2_51_1","volume-title":"HCOMP","author":"Drapeau Ryan","year":"2016","unstructured":"Ryan Drapeau , Lydia B. Chilton , Jonathan Bragg , and Daniel S. Weld . 2016. MicroTalk: Using argumentation to improve crowdsourcing accuracy . In HCOMP 2016 . Ryan Drapeau, Lydia B. Chilton, Jonathan Bragg, and Daniel S. Weld. 2016. MicroTalk: Using argumentation to improve crowdsourcing accuracy. In HCOMP 2016."},{"key":"e_1_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-011-9181-9"},{"key":"e_1_2_2_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/2348283.2348400"},{"key":"e_1_2_2_54_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-26190-4_20"},{"key":"e_1_2_2_55_1","volume-title":"HCOMP","author":"Faltings Boi","year":"2014","unstructured":"Boi Faltings , Radu Jurca , Pearl Pu , and Bao Duy Tran . 2014 . Incentives to counter bias in human computation . In HCOMP 2014. http:\/\/www.aaai.org\/ocs\/index.php\/HCOMP\/HCOMP14\/paper\/view\/8945. Boi Faltings, Radu Jurca, Pearl Pu, and Bao Duy Tran. 2014. Incentives to counter bias in human computation. In HCOMP 2014. http:\/\/www.aaai.org\/ocs\/index.php\/HCOMP\/HCOMP14\/paper\/view\/8945."},{"key":"e_1_2_2_56_1","volume-title":"28th AAAI Conference on Artificial Intelligence.","author":"Fang Meng","year":"2014","unstructured":"Meng Fang , Jie Yin , and Dacheng Tao . 2014 . Active learning for crowdsourcing using knowledge transfer . In 28th AAAI Conference on Artificial Intelligence. Meng Fang, Jie Yin, and Dacheng Tao. 2014. Active learning for crowdsourcing using knowledge transfer. In 28th AAAI Conference on Artificial Intelligence."},{"key":"e_1_2_2_57_1","volume-title":"Ipeirotis","author":"Faradani Siamak","year":"2011","unstructured":"Siamak Faradani , Bj\u00f6rn Hartmann , and Panagiotis G . Ipeirotis . 2011 . What\u2019s the right price? Pricing tasks for finishing on time.Human Computation 11 (2011). Siamak Faradani, Bj\u00f6rn Hartmann, and Panagiotis G. Ipeirotis. 2011. What\u2019s the right price? Pricing tasks for finishing on time.Human Computation 11 (2011)."},{"key":"e_1_2_2_58_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-38791-8_26"},{"key":"e_1_2_2_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/2736277.2741639"},{"key":"e_1_2_2_60_1","doi-asserted-by":"publisher","DOI":"10.2307\/3033543"},{"key":"e_1_2_2_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702443"},{"key":"e_1_2_2_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/2984511.2984542"},{"key":"e_1_2_2_63_1","volume-title":"ICML","author":"Gao Chao","year":"2016","unstructured":"Chao Gao , Yu Lu , and Denny Zhou . 2016 . Exact exponent in optimal rates for crowdsourcing . In ICML 2016. 603--611. Chao Gao, Yu Lu, and Denny Zhou. 2016. Exact exponent in optimal rates for crowdsourcing. In ICML 2016. 603--611."},{"key":"e_1_2_2_64_1","doi-asserted-by":"publisher","DOI":"10.1145\/2396761.2398554"},{"key":"e_1_2_2_65_1","volume-title":"CSCW","author":"Hansen Derek L.","year":"2013","unstructured":"Derek L. Hansen , Patrick J. Schone , Douglas Corey , Matthew Reid , and Jake Gehring . 2013 . Quality control mechanisms for crowdsourcing: Peer review, arbitration, 8 expertise at familysearch indexing . In CSCW 2013. 649--660. Derek L. Hansen, Patrick J. Schone, Douglas Corey, Matthew Reid, and Jake Gehring. 2013. Quality control mechanisms for crowdsourcing: Peer review, arbitration, 8 expertise at familysearch indexing. In CSCW 2013. 649--660."},{"key":"e_1_2_2_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2470744"},{"key":"e_1_2_2_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/1460355.1460357"},{"key":"e_1_2_2_68_1","volume-title":"CSCW","author":"Hata Kenji","year":"2017","unstructured":"Kenji Hata , Ranjay Krishna , Li Fei-Fei , and Michael S. Bernstein . 2017. A glimpse far into the future: Understanding long-term crowd worker quality . In CSCW 2017 . 889--901. Kenji Hata, Ranjay Krishna, Li Fei-Fei, and Michael S. Bernstein. 2017. A glimpse far into the future: Understanding long-term crowd worker quality. In CSCW 2017. 889--901."},{"key":"e_1_2_2_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/1753326.1753357"},{"key":"e_1_2_2_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2208619"},{"key":"e_1_2_2_71_1","doi-asserted-by":"publisher","DOI":"10.1145\/255656.255692"},{"key":"e_1_2_2_72_1","volume-title":"WWW","author":"Heymann Paul","year":"2011","unstructured":"Paul Heymann and Hector Garcia-Molina . 2011 . Turkalytics: Analytics for human computation . In WWW 2011. 477--486. Paul Heymann and Hector Garcia-Molina. 2011. Turkalytics: Analytics for human computation. In WWW 2011. 477--486."},{"key":"e_1_2_2_73_1","volume-title":"NIPS","author":"Ho Chien-Ju","year":"2016","unstructured":"Chien-Ju Ho , Rafael Frongillo , and Yiling Chen . 2016 . Eliciting categorical data for optimal aggregation . In NIPS 2016. Curran Associates, Inc., 2450--2458. Chien-Ju Ho, Rafael Frongillo, and Yiling Chen. 2016. Eliciting categorical data for optimal aggregation. In NIPS 2016. Curran Associates, Inc., 2450--2458."},{"key":"e_1_2_2_74_1","doi-asserted-by":"publisher","DOI":"10.1145\/2736277.2741102"},{"key":"e_1_2_2_75_1","first-page":"45","article-title":"Online task assignment in crowdsourcing markets","volume":"12","author":"Ho Chien-Ju","year":"2012","unstructured":"Chien-Ju Ho and Jennifer Wortman Vaughan . 2012 . Online task assignment in crowdsourcing markets . In AAAI , Vol. 12. 45 -- 51 . Chien-Ju Ho and Jennifer Wortman Vaughan. 2012. Online task assignment in crowdsourcing markets. In AAAI, Vol. 12. 45--51.","journal-title":"AAAI"},{"key":"e_1_2_2_76_1","doi-asserted-by":"publisher","DOI":"10.1145\/2642918.2647362"},{"key":"e_1_2_2_77_1","doi-asserted-by":"publisher","DOI":"10.1109\/MC.2014.245"},{"key":"e_1_2_2_78_1","volume-title":"The rise of crowdsourcing. Wired (June","year":"2006","unstructured":"Jeff. Howe. 2006. The rise of crowdsourcing. Wired (June 2006 ). Jeff. Howe. 2006. The rise of crowdsourcing. Wired (June 2006)."},{"key":"e_1_2_2_79_1","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2208700"},{"key":"e_1_2_2_80_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2470743"},{"key":"e_1_2_2_81_1","doi-asserted-by":"publisher","DOI":"10.1145\/2441776.2441847"},{"key":"e_1_2_2_82_1","doi-asserted-by":"publisher","DOI":"10.1145\/2484028.2484199"},{"key":"e_1_2_2_83_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-41154-0_1"},{"key":"e_1_2_2_84_1","volume-title":"SIGMOD","author":"Viet Hung Nguyen Quoc","year":"2015","unstructured":"Nguyen Quoc Viet Hung , Duong Chi Thang , Matthias Weidlich , and Karl Aberer . 2015 . Minimizing efforts in validating crowd answers . In SIGMOD 2015. 999--1014. Nguyen Quoc Viet Hung, Duong Chi Thang, Matthias Weidlich, and Karl Aberer. 2015. Minimizing efforts in validating crowd answers. In SIGMOD 2015. 999--1014."},{"key":"e_1_2_2_85_1","doi-asserted-by":"publisher","DOI":"10.1002\/cpe.2923"},{"key":"e_1_2_2_86_1","volume-title":"HCOMP","author":"Huynh Trung Dong","year":"2013","unstructured":"Trung Dong Huynh , Mark Ebden , Matteo Venanzi , Sarvapali D. Ramchurn , Stephen J. Roberts , and Luc Moreau . 2013 . Interpretation of crowdsourced activities using provenance network analysis . In HCOMP 2013. Trung Dong Huynh, Mark Ebden, Matteo Venanzi, Sarvapali D. Ramchurn, Stephen J. Roberts, and Luc Moreau. 2013. Interpretation of crowdsourced activities using provenance network analysis. In HCOMP 2013."},{"key":"e_1_2_2_87_1","doi-asserted-by":"publisher","DOI":"10.1109\/WIIAT.2008.14"},{"key":"e_1_2_2_88_1","volume-title":"CHI","author":"Ikeda Kazushi","year":"2016","unstructured":"Kazushi Ikeda and Michael S. Bernstein . 2016. Pay it backward: Per-task payments on crowdsourcing platforms reduce productivity . In CHI 2016 . 4111--4121. http:\/\/doi.acm.org\/10.1145\/2858036.2858327 Kazushi Ikeda and Michael S. Bernstein. 2016. Pay it backward: Per-task payments on crowdsourcing platforms reduce productivity. In CHI 2016. 4111--4121. http:\/\/doi.acm.org\/10.1145\/2858036.2858327"},{"key":"e_1_2_2_89_1","doi-asserted-by":"publisher","DOI":"10.1145\/1869086.1869094"},{"key":"e_1_2_2_90_1","volume-title":"Ipeirotis and Evgeniy Gabrilovich","author":"Panagiotis","year":"2014","unstructured":"Panagiotis G. Ipeirotis and Evgeniy Gabrilovich . 2014 . Quizz : Targeted crowdsourcing with a billion (potential) users. In WWW 2014. 143--154. Panagiotis G. Ipeirotis and Evgeniy Gabrilovich. 2014. Quizz: Targeted crowdsourcing with a billion (potential) users. In WWW 2014. 143--154."},{"key":"e_1_2_2_91_1","volume-title":"CHI","author":"Irani Lilly C.","year":"2013","unstructured":"Lilly C. Irani and M. Silberman . 2013. Turkopticon: Interrupting worker invisibility in Amazon Mechanical Turk . In CHI 2013 . 611--620. Lilly C. Irani and M. Silberman. 2013. Turkopticon: Interrupting worker invisibility in Amazon Mechanical Turk. In CHI 2013. 611--620."},{"key":"e_1_2_2_92_1","volume-title":"NIPS","author":"Jagabathula Srikanth","year":"2014","unstructured":"Srikanth Jagabathula , Lakshminarayanan Subramanian , and Ashwin Venkataraman . 2014 . Reputation-based worker filtering in crowdsourcing . In NIPS 2014. Curran Associates, Inc., 2492--2500. Srikanth Jagabathula, Lakshminarayanan Subramanian, and Ashwin Venkataraman. 2014. Reputation-based worker filtering in crowdsourcing. In NIPS 2014. Curran Associates, Inc., 2492--2500."},{"key":"e_1_2_2_93_1","doi-asserted-by":"publisher","DOI":"10.1145\/2487575.2487595"},{"key":"e_1_2_2_94_1","volume-title":"Soto","author":"John Oliver P.","year":"2008","unstructured":"Oliver P. John , Laura P. Naumann , and Christopher J . Soto . 2008 . Paradigm shift to the integrative big five trait taxonomy. Handbook of Personality: Theory and Research (3rd ed.). Guilford Press , New York, 114--158. Oliver P. John, Laura P. Naumann, and Christopher J. Soto. 2008. Paradigm shift to the integrative big five trait taxonomy. Handbook of Personality: Theory and Research (3rd ed.). Guilford Press, New York, 114--158."},{"key":"e_1_2_2_95_1","volume-title":"John and Sanjay Srivastava","author":"Oliver","year":"1999","unstructured":"Oliver P. John and Sanjay Srivastava . 1999 . The big five trait taxonomy: History, measurement, and theoretical perspectives. Handbook of Personality: Theory and Research (2nd ed.). Guilford Press , New York, 102--138. Oliver P. John and Sanjay Srivastava. 1999. The big five trait taxonomy: History, measurement, and theoretical perspectives. Handbook of Personality: Theory and Research (2nd ed.). Guilford Press, New York, 102--138."},{"key":"e_1_2_2_96_1","unstructured":"Hyun Joon Jung and Matthew Lease. 2011. Improving consensus accuracy via Z-score and weighted voting. In Human Computation.  Hyun Joon Jung and Matthew Lease. 2011. Improving consensus accuracy via Z-score and weighted voting. In Human Computation."},{"key":"e_1_2_2_97_1","doi-asserted-by":"publisher","DOI":"10.1145\/2348283.2348486"},{"key":"e_1_2_2_98_1","volume-title":"HCOMP","author":"Jung Hyun Joon","year":"2014","unstructured":"Hyun Joon Jung , Yubin Park , and Matthew Lease . 2014 . Predicting next label quality: A time-series model of crowdwork . In HCOMP 2014. Hyun Joon Jung, Yubin Park, and Matthew Lease. 2014. Predicting next label quality: A time-series model of crowdwork. In HCOMP 2014."},{"key":"e_1_2_2_99_1","doi-asserted-by":"publisher","DOI":"10.1109\/MS.2004.1331309"},{"key":"e_1_2_2_100_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2820016"},{"key":"e_1_2_2_101_1","volume-title":"NIPS 2011","author":"Karger David R.","year":"2011","unstructured":"David R. Karger , Sewoong Oh , and Devavrat Shah . 2011 . Iterative learning for reliable crowdsourcing systems . In NIPS 2011 . Curran Associates, Inc. , 1953--1961. David R. Karger, Sewoong Oh, and Devavrat Shah. 2011. Iterative learning for reliable crowdsourcing systems. In NIPS 2011. Curran Associates, Inc., 1953--1961."},{"key":"e_1_2_2_102_1","doi-asserted-by":"publisher","DOI":"10.1287\/opre.2013.1235"},{"key":"e_1_2_2_103_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858588"},{"key":"e_1_2_2_104_1","doi-asserted-by":"publisher","DOI":"10.1145\/2009916.2009947"},{"key":"e_1_2_2_105_1","doi-asserted-by":"publisher","DOI":"10.1145\/2063576.2063860"},{"key":"e_1_2_2_106_1","doi-asserted-by":"publisher","DOI":"10.1145\/2396761.2398697"},{"key":"e_1_2_2_107_1","doi-asserted-by":"publisher","DOI":"10.1145\/2835776.2835835"},{"key":"e_1_2_2_108_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-16985-4_39"},{"key":"e_1_2_2_109_1","doi-asserted-by":"publisher","DOI":"10.1145\/1926180.1926195"},{"key":"e_1_2_2_110_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-31095-9_30"},{"key":"e_1_2_2_111_1","volume-title":"HCOMP","author":"KhudaBukhsh Ashiqur R.","year":"2014","unstructured":"Ashiqur R. KhudaBukhsh , Jaime G. Carbonell , and Peter J. Jansen . 2014. Detecting non-adversarial collusion in crowdsourcing . In HCOMP 2014 . Ashiqur R. KhudaBukhsh, Jaime G. Carbonell, and Peter J. Jansen. 2014. Detecting non-adversarial collusion in crowdsourcing. In HCOMP 2014."},{"key":"e_1_2_2_112_1","doi-asserted-by":"publisher","DOI":"10.1145\/1869086.1869096"},{"key":"e_1_2_2_113_1","doi-asserted-by":"publisher","DOI":"10.1145\/1357054.1357127"},{"key":"e_1_2_2_114_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145357"},{"key":"e_1_2_2_115_1","doi-asserted-by":"publisher","DOI":"10.1145\/2441776.2441923"},{"key":"e_1_2_2_116_1","volume-title":"Kraut","author":"Kittur Aniket","year":"2011","unstructured":"Aniket Kittur , Boris Smus , Susheel Khamkar , and Robert E . Kraut . 2011 . Crowdforge : Crowdsourcing complex work. In UIST\u2019 11. 43--52. Aniket Kittur, Boris Smus, Susheel Khamkar, and Robert E. Kraut. 2011. Crowdforge: Crowdsourcing complex work. In UIST\u201911. 43--52."},{"key":"e_1_2_2_117_1","doi-asserted-by":"publisher","DOI":"10.1145\/2675133.2675255"},{"key":"e_1_2_2_118_1","volume-title":"Panagiotis Ipeirotis, and Evgeniy Gabrilovich.","author":"Kobren Ari","year":"2015","unstructured":"Ari Kobren , Chun How Tan , Panagiotis Ipeirotis, and Evgeniy Gabrilovich. 2015 . Getting more for less: Optimized crowdsourcing with dynamic tasks and goals. In WWW 2015. 592--602. Ari Kobren, Chun How Tan, Panagiotis Ipeirotis, and Evgeniy Gabrilovich. 2015. Getting more for less: Optimized crowdsourcing with dynamic tasks and goals. In WWW 2015. 592--602."},{"key":"e_1_2_2_119_1","volume-title":"HCOMP","author":"Krause Markus","year":"2015","unstructured":"Markus Krause and Ren\u00e9 F. Kizilcec . 2015. To play or not to play: Interactions between response quality and task complexity in games and paid crowdsourcing . In HCOMP 2015 . 102--109. Markus Krause and Ren\u00e9 F. Kizilcec. 2015. To play or not to play: Interactions between response quality and task complexity in games and paid crowdsourcing. In HCOMP 2015. 102--109."},{"key":"e_1_2_2_120_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858115"},{"key":"e_1_2_2_121_1","doi-asserted-by":"publisher","DOI":"10.1145\/2522968.2522969"},{"key":"e_1_2_2_122_1","doi-asserted-by":"publisher","DOI":"10.1109\/MIC.2015.96"},{"key":"e_1_2_2_123_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2820005"},{"key":"e_1_2_2_124_1","volume-title":"CSCW\u201912","author":"Kulkarni Anand","unstructured":"Anand Kulkarni , Matthew Can , and Bj\u00f6rn Hartmann . 2012a. Collaboratively crowdsourcing workflows with Turkomatic . In CSCW\u201912 . ACM , New York , 1003--1012. Anand Kulkarni, Matthew Can, and Bj\u00f6rn Hartmann. 2012a. Collaboratively crowdsourcing workflows with Turkomatic. In CSCW\u201912. ACM, New York, 1003--1012."},{"key":"e_1_2_2_125_1","doi-asserted-by":"publisher","DOI":"10.1109\/MIC.2012.72"},{"key":"e_1_2_2_126_1","volume-title":"HCOMP","author":"Kulkarni Anand","year":"2014","unstructured":"Anand Kulkarni , Prayag Narula , David Rolnitzky , and Nathan Kontny . 2014 . Wish: Amplifying creative ability with expert crowds . In HCOMP 2014. Anand Kulkarni, Prayag Narula, David Rolnitzky, and Nathan Kontny. 2014. Wish: Amplifying creative ability with expert crowds. In HCOMP 2014."},{"key":"e_1_2_2_127_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2466269"},{"key":"e_1_2_2_128_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702594"},{"key":"e_1_2_2_129_1","doi-asserted-by":"publisher","DOI":"10.1145\/2441776.2441912"},{"key":"e_1_2_2_130_1","doi-asserted-by":"publisher","DOI":"10.1145\/2531602.2531733"},{"key":"e_1_2_2_131_1","doi-asserted-by":"publisher","DOI":"10.1209\/epl\/i2006-10204-8"},{"key":"e_1_2_2_132_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858144"},{"key":"e_1_2_2_133_1","volume-title":"SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation. 21--26","author":"Le John","year":"2010","unstructured":"John Le , Andy Edmonds , Vaughn Hester , and Lukas Biewald . 2010 . Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution . In SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation. 21--26 . John Le, Andy Edmonds, Vaughn Hester, and Lukas Biewald. 2010. Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution. In SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation. 21--26."},{"key":"e_1_2_2_134_1","volume-title":"Booms","author":"Lewis Robert C.","year":"1983","unstructured":"Robert C. Lewis and Bernhard H . Booms . 1983 . Emerging Perspectives on Service Marketing. American Marketing , 99--107. Robert C. Lewis and Bernhard H. Booms. 1983. Emerging Perspectives on Service Marketing. American Marketing, 99--107."},{"key":"e_1_2_2_135_1","volume-title":"WWW","author":"Li Hongwei","year":"2014","unstructured":"Hongwei Li , Bo Zhao , and Ariel Fuxman . 2014 . The wisdom of minority: Discovering and targeting the right group of workers for crowdsourcing . In WWW 2014. 165--176. Hongwei Li, Bo Zhao, and Ariel Fuxman. 2014. The wisdom of minority: Discovering and targeting the right group of workers for crowdsourcing. In WWW 2014. 165--176."},{"key":"e_1_2_2_136_1","doi-asserted-by":"publisher","DOI":"10.1145\/2835776.2835797"},{"key":"e_1_2_2_137_1","volume-title":"AAAI","author":"Lin Christopher H.","year":"2014","unstructured":"Christopher H. Lin , Ece Kamar , and Eric Horvitz . 2014 . Signals in the silence: Models of implicit feedback in a recommendation system for crowdsourcing . In AAAI 2014. 908--915. Christopher H. Lin, Ece Kamar, and Eric Horvitz. 2014. Signals in the silence: Models of implicit feedback in a recommendation system for crowdsourcing. In AAAI 2014. 908--915."},{"key":"e_1_2_2_138_1","doi-asserted-by":"publisher","DOI":"10.1145\/1837885.1837907"},{"key":"e_1_2_2_139_1","volume-title":"UIST","author":"Little Greg","year":"2010","unstructured":"Greg Little , Lydia B. Chilton , Max Goldman , and Robert C. Miller . 2010a. Turkit: Human computation algorithms on Mechanical Turk . In UIST 2010 . ACM, 57--66. Greg Little, Lydia B. Chilton, Max Goldman, and Robert C. Miller. 2010a. Turkit: Human computation algorithms on Mechanical Turk. In UIST 2010. ACM, 57--66."},{"key":"e_1_2_2_140_1","doi-asserted-by":"publisher","DOI":"10.1145\/1866029.1866040"},{"key":"e_1_2_2_141_1","volume-title":"ICML","author":"Liu Chao","year":"2012","unstructured":"Chao Liu and Yi-Min Wang . 2012 . TrueLabel + confusions: A spectrum of probabilistic models in analyzing multiple ratings .. In ICML 2012. icml.cc\/ Omnipress. Chao Liu and Yi-Min Wang. 2012. TrueLabel + confusions: A spectrum of probabilistic models in analyzing multiple ratings.. In ICML 2012. icml.cc\/ Omnipress."},{"key":"e_1_2_2_142_1","volume-title":"NIPS 2013","author":"Liu Qiang","year":"2013","unstructured":"Qiang Liu , Alexander T. Ihler , and Mark Steyvers . 2013 . Scoring workers in crowdsourcing: How many control questions are enough? In NIPS 2013 . Curran Associates, Inc. , 1914--1922. Qiang Liu, Alexander T. Ihler, and Mark Steyvers. 2013. Scoring workers in crowdsourcing: How many control questions are enough? In NIPS 2013. Curran Associates, Inc., 1914--1922."},{"key":"e_1_2_2_143_1","volume-title":"HCOMP","author":"Livshits Benjamin","year":"2014","unstructured":"Benjamin Livshits and Todd Mytkowicz . 2014 . Saving money while polling with interpoll using power analysis . In HCOMP 2014. Benjamin Livshits and Todd Mytkowicz. 2014. Saving money while polling with interpoll using power analysis. In HCOMP 2014."},{"key":"e_1_2_2_144_1","doi-asserted-by":"publisher","DOI":"10.1109\/EMR.2010.5559142"},{"key":"e_1_2_2_145_1","volume-title":"HCOMP","author":"Mao Andrew","year":"2013","unstructured":"Andrew Mao , Ece Kamar , Yiling Chen , Eric Horvitz , Megan E. Schwamb , Chris J. Lintott , and Arfon M. Smith . 2013. Volunteering versus work for pay: Incentives and tradeoffs in crowdsourcing . In HCOMP 2013 . Andrew Mao, Ece Kamar, Yiling Chen, Eric Horvitz, Megan E. Schwamb, Chris J. Lintott, and Arfon M. Smith. 2013. Volunteering versus work for pay: Incentives and tradeoffs in crowdsourcing. In HCOMP 2013."},{"key":"e_1_2_2_146_1","doi-asserted-by":"publisher","DOI":"10.14778\/2535568.2448944"},{"key":"e_1_2_2_147_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2470708"},{"key":"e_1_2_2_148_1","doi-asserted-by":"publisher","DOI":"10.1145\/2872427.2883070"},{"key":"e_1_2_2_149_1","volume-title":"HCOMP","author":"McDonnell Tyler","year":"2016","unstructured":"Tyler McDonnell , Matthew Lease , Mucahid Kutlu , and Tamer Elsayed . 2016 . Why is that relevant? Collecting annotator rationales for relevance judgments . In HCOMP 2016. Tyler McDonnell, Matthew Lease, Mucahid Kutlu, and Tamer Elsayed. 2016. Why is that relevant? Collecting annotator rationales for relevance judgments. In HCOMP 2016."},{"key":"e_1_2_2_150_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-35386-4_10"},{"key":"e_1_2_2_151_1","doi-asserted-by":"publisher","DOI":"10.1145\/2783446.2783580"},{"key":"e_1_2_2_152_1","doi-asserted-by":"publisher","DOI":"10.1145\/2487575.2487593"},{"key":"e_1_2_2_153_1","doi-asserted-by":"publisher","DOI":"10.1109\/MIC.2012.68"},{"key":"e_1_2_2_154_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2914756"},{"key":"e_1_2_2_155_1","volume-title":"Zou","author":"Nath Swaprava","year":"2012","unstructured":"Swaprava Nath , Pankaj Dayama , Dinesh Garg , Y. Narahari , and James Y . Zou . 2012 . Threats and trade-offs in resource critical crowdsourcing tasks over networks. In AAAI 2012. Swaprava Nath, Pankaj Dayama, Dinesh Garg, Y. Narahari, and James Y. Zou. 2012. Threats and trade-offs in resource critical crowdsourcing tasks over networks. In AAAI 2012."},{"key":"e_1_2_2_156_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858490"},{"key":"e_1_2_2_157_1","doi-asserted-by":"publisher","DOI":"10.1145\/2661829.2661918"},{"key":"e_1_2_2_158_1","volume-title":"Homepage Usability: 50 Websites Deconstructed","author":"Nielsen Jakob","unstructured":"Jakob Nielsen , Marie Tahir , and Marie Tahir . 2002. Homepage Usability: 50 Websites Deconstructed . Vol. 50 . New Riders Indianapolis , IN. Jakob Nielsen, Marie Tahir, and Marie Tahir. 2002. Homepage Usability: 50 Websites Deconstructed. Vol. 50. New Riders Indianapolis, IN."},{"key":"e_1_2_2_159_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-38791-8_50"},{"key":"e_1_2_2_160_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047198"},{"key":"e_1_2_2_161_1","volume-title":"HCOMP","author":"Nushi Besmira","year":"2015","unstructured":"Besmira Nushi , Adish Singla , Anja Gruenheid , Erfan Zamanian , Andreas Krause , and Donald Kossmann . 2015 . Crowd access path optimization: Diversity matters . In HCOMP 2015. Besmira Nushi, Adish Singla, Anja Gruenheid, Erfan Zamanian, Andreas Krause, and Donald Kossmann. 2015. Crowd access path optimization: Diversity matters. In HCOMP 2015."},{"key":"e_1_2_2_162_1","volume-title":"ICML","author":"Ok Jungseul","year":"2016","unstructured":"Jungseul Ok , Sewoong Oh , Jinwoo Shin , and Yung Yi . 2016 . Optimality of belief propagation for crowdsourced classification . In ICML 2016. JMLR.org, 535--544. Jungseul Ok, Sewoong Oh, Jinwoo Shin, and Yung Yi. 2016. Optimality of belief propagation for crowdsourced classification. In ICML 2016. JMLR.org, 535--544."},{"key":"e_1_2_2_163_1","volume-title":"HCOMP 2011 11","author":"Oleson David","year":"2011","unstructured":"David Oleson , Alexander Sorokin , Greg P. Laughlin , Vaughn Hester , John Le , and Lukas Biewald . 2011 . Programmatic gold: Targeted and scalable quality assurance in crowdsourcing . HCOMP 2011 11 , 11 (2011). David Oleson, Alexander Sorokin, Greg P. Laughlin, Vaughn Hester, John Le, and Lukas Biewald. 2011. Programmatic gold: Targeted and scalable quality assurance in crowdsourcing. HCOMP 2011 11, 11 (2011)."},{"key":"e_1_2_2_164_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-38791-8_27"},{"key":"e_1_2_2_165_1","unstructured":"Lawrence Page Sergey Brin Rajeev Motwani and Terry Winograd. 1999. The PageRank citation ranking: Bringing order to the Web. (1999).  Lawrence Page Sergey Brin Rajeev Motwani and Terry Winograd. 1999. The PageRank citation ranking: Bringing order to the Web. (1999)."},{"key":"e_1_2_2_166_1","volume-title":"CSCW","author":"Preist Chris","year":"2014","unstructured":"Chris Preist , Elaine Massung , and David Coyle . 2014 . Competing or aiming to be average?: Normification as a means of engaging digital volunteers . In CSCW 2014. 1222--1233. Chris Preist, Elaine Massung, and David Coyle. 2014. Competing or aiming to be average?: Normification as a means of engaging digital volunteers. In CSCW 2014. 1222--1233."},{"key":"e_1_2_2_167_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICRIIS.2011.6125717"},{"key":"e_1_2_2_168_1","volume-title":"CSCW","author":"Alexander","year":"2014","unstructured":"Alexander J. Quinn and Benjamin B. Bederson. 2014. AskSheet: Efficient human computation for decision making with spreadsheets . In CSCW 2014 . 1456--1466. Alexander J. Quinn and Benjamin B. Bederson. 2014. AskSheet: Efficient human computation for decision making with spreadsheets. In CSCW 2014. 1456--1466."},{"key":"e_1_2_2_169_1","volume-title":"HCOMP","author":"Radanovic Goran","year":"2016","unstructured":"Goran Radanovic and Boi Faltings . 2016 . Learning to scale payments in crowdsourcing with properboost . In HCOMP 2016. Goran Radanovic and Boi Faltings. 2016. Learning to scale payments in crowdsourcing with properboost. In HCOMP 2016."},{"key":"e_1_2_2_170_1","volume-title":"SEKE","author":"Rajasekharan Karthikeyan","year":"2013","unstructured":"Karthikeyan Rajasekharan , Aditya P. Mathur , and See-Kiong Ng . 2013 . Effective crowdsourcing for software feature ideation in online co-creation forums . In SEKE 2013. 119--124. Karthikeyan Rajasekharan, Aditya P. Mathur, and See-Kiong Ng. 2013. Effective crowdsourcing for software feature ideation in online co-creation forums. In SEKE 2013. 119--124."},{"key":"e_1_2_2_171_1","volume-title":"HCOMP","author":"Rao Huaming","year":"2013","unstructured":"Huaming Rao , Shih-Wen Huang , and Wai-Tat Fu . 2013 . What will others choose? How a majority vote reward scheme can improve human computation in a spatial location identification task . In HCOMP 2013. Huaming Rao, Shih-Wen Huang, and Wai-Tat Fu. 2013. What will others choose? How a majority vote reward scheme can improve human computation in a spatial location identification task. In HCOMP 2013."},{"key":"e_1_2_2_172_1","volume-title":"NIPS 2011","author":"Vikas","year":"1809","unstructured":"Vikas C. Raykar and Shipeng Yu. 2011. Ranking annotators for crowdsourced labeling tasks . In NIPS 2011 . Curran Associates Inc. , 1809 --1817. Vikas C. Raykar and Shipeng Yu. 2011. Ranking annotators for crowdsourced labeling tasks. In NIPS 2011. Curran Associates Inc., 1809--1817."},{"key":"e_1_2_2_173_1","volume-title":"Bernstein","author":"Retelny Daniela","year":"2014","unstructured":"Daniela Retelny , S\u00e9bastien Robaszkiewicz , Alexandra To , Walter S. Lasecki , Jay Patel , Negar Rahmati , Tulsee Doshi , Melissa Valentine , and Michael S . Bernstein . 2014 . Expert crowdsourcing with flash teams. In UIST. ACM , 75--85. Daniela Retelny, S\u00e9bastien Robaszkiewicz, Alexandra To, Walter S. Lasecki, Jay Patel, Negar Rahmati, Tulsee Doshi, Melissa Valentine, and Michael S. Bernstein. 2014. Expert crowdsourcing with flash teams. In UIST. ACM, 75--85."},{"key":"e_1_2_2_174_1","unstructured":"Jakob Rogstadius Vassilis Kostakos Aniket Kittur Boris Smus Jim Laredo and Maja Vukovic. 2011. An assessment of intrinsic and extrinsic motivation on task performance in crowdsourcing markets. In ICWSM.  Jakob Rogstadius Vassilis Kostakos Aniket Kittur Boris Smus Jim Laredo and Maja Vukovic. 2011. An assessment of intrinsic and extrinsic motivation on task performance in crowdsourcing markets. In ICWSM."},{"key":"e_1_2_2_175_1","doi-asserted-by":"publisher","DOI":"10.1145\/2661829.2661946"},{"key":"e_1_2_2_176_1","doi-asserted-by":"publisher","DOI":"10.1145\/2736277.2741097"},{"key":"e_1_2_2_177_1","doi-asserted-by":"publisher","DOI":"10.1007\/s00778-015-0385-2"},{"key":"e_1_2_2_178_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047199"},{"key":"e_1_2_2_179_1","doi-asserted-by":"publisher","DOI":"10.1145\/2380116.2380125"},{"key":"e_1_2_2_180_1","volume-title":"HCOMP","author":"Sakurai Yuko","year":"2013","unstructured":"Yuko Sakurai , Tenda Okimoto , Masaaki Oka , Masato Shinoda , and Makoto Yokoo . 2013 . Ability grouping of crowd workers via reward discrimination . In HCOMP 2013. Yuko Sakurai, Tenda Okimoto, Masaaki Oka, Masato Shinoda, and Makoto Yokoo. 2013. Ability grouping of crowd workers via reward discrimination. In HCOMP 2013."},{"key":"e_1_2_2_181_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.is.2012.09.003"},{"key":"e_1_2_2_182_1","doi-asserted-by":"publisher","DOI":"10.1145\/2461256.2461275"},{"key":"e_1_2_2_183_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-38709-8_44"},{"key":"e_1_2_2_184_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11280-012-0180-6"},{"key":"e_1_2_2_185_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSC.2011.2"},{"key":"e_1_2_2_186_1","volume-title":"AMCIS","author":"Schulze Thimo","year":"2013","unstructured":"Thimo Schulze , Dennis Nordheimer , and Martin Schader . 2013 . Worker perception of quality assurance mechanisms in crowdsourcing and human computation markets . In AMCIS 2013. Thimo Schulze, Dennis Nordheimer, and Martin Schader. 2013. Worker perception of quality assurance mechanisms in crowdsourcing and human computation markets. In AMCIS 2013."},{"key":"e_1_2_2_187_1","volume-title":"NIPS","author":"Shah Nihar Bhadresh","year":"2015","unstructured":"Nihar Bhadresh Shah and Denny Zhou . 2015 . Double or nothing: Multiplicative incentive mechanisms for crowdsourcing . In NIPS 2015. Curran Associates, Inc., 1--9. Nihar Bhadresh Shah and Denny Zhou. 2015. Double or nothing: Multiplicative incentive mechanisms for crowdsourcing. In NIPS 2015. Curran Associates, Inc., 1--9."},{"key":"e_1_2_2_188_1","volume-title":"ICML","author":"Shah Nihar Bhadresh","year":"2016","unstructured":"Nihar Bhadresh Shah and Dengyong Zhou . 2016 . No oops, you won\u2019t do it again: Mechanisms for self-correction in crowdsourcing . In ICML 2016. 1--10. Nihar Bhadresh Shah and Dengyong Zhou. 2016. No oops, you won\u2019t do it again: Mechanisms for self-correction in crowdsourcing. In ICML 2016. 1--10."},{"key":"e_1_2_2_189_1","volume-title":"HCOMP","author":"Sheshadri Aashish","year":"2013","unstructured":"Aashish Sheshadri and Matthew Lease . 2013 . SQUARE: A benchmark for research on computing crowd consensus . In HCOMP 2013. Aashish Sheshadri and Matthew Lease. 2013. SQUARE: A benchmark for research on computing crowd consensus. In HCOMP 2013."},{"key":"e_1_2_2_190_1","doi-asserted-by":"crossref","unstructured":"Yaron Singer and Manas Mittal. 2013. Pricing mechanisms for crowdsourcing markets. In WWW. 1157--1166.  Yaron Singer and Manas Mittal. 2013. Pricing mechanisms for crowdsourcing markets. In WWW. 1157--1166.","DOI":"10.1145\/2488388.2488489"},{"key":"e_1_2_2_191_1","volume-title":"ICML","author":"Singla Adish","year":"2014","unstructured":"Adish Singla , Ilija Bogunovic , G\u00e1bor Bart\u00f3k , Amin Karbasi , and Andreas Krause . 2014 . Near-optimally teaching the crowd to classify . In ICML 2014. JMLR.org, II-154--II-162. Adish Singla, Ilija Bogunovic, G\u00e1bor Bart\u00f3k, Amin Karbasi, and Andreas Krause. 2014. Near-optimally teaching the crowd to classify. In ICML 2014. JMLR.org, II-154--II-162."},{"key":"e_1_2_2_192_1","doi-asserted-by":"publisher","DOI":"10.1145\/2568225.2568249"},{"key":"e_1_2_2_193_1","volume-title":"When majority voting fails: Comparing quality assurance methods for noisy human computation environment. arXiv:1204.3516","author":"Sun Yu-An","year":"2012","unstructured":"Yu-An Sun and Christopher Dance . 2012. When majority voting fails: Comparing quality assurance methods for noisy human computation environment. arXiv:1204.3516 ( 2012 ). Yu-An Sun and Christopher Dance. 2012. When majority voting fails: Comparing quality assurance methods for noisy human computation environment. arXiv:1204.3516 (2012)."},{"key":"e_1_2_2_194_1","volume-title":"The Wisdom of Crowds","author":"Surowiecki James","unstructured":"James Surowiecki . 2005. The Wisdom of Crowds . Anchor Books . James Surowiecki. 2005. The Wisdom of Crowds. Anchor Books."},{"key":"e_1_2_2_195_1","doi-asserted-by":"publisher","DOI":"10.1109\/MIC.2012.66"},{"key":"e_1_2_2_196_1","first-page":"242","article-title":"Transfer learning","volume":"1","author":"Torrey Lisa","year":"2009","unstructured":"Lisa Torrey and Jude Shavlik . 2009 . Transfer learning . Handbook of Research on Machine Learning Applications and Trends: Algorithms, Methods, and Techniques 1 (2009), 242 . Lisa Torrey and Jude Shavlik. 2009. Transfer learning. Handbook of Research on Machine Learning Applications and Trends: Algorithms, Methods, and Techniques 1 (2009), 242.","journal-title":"Handbook of Research on Machine Learning Applications and Trends: Algorithms, Methods, and Techniques"},{"key":"e_1_2_2_197_1","volume-title":"Avi Rosenfeld, Sarvapali D. Ramchurn, and Nicholas R. Jennings.","author":"Tran-Thanh Long","year":"2015","unstructured":"Long Tran-Thanh , Trung Dong Huynh , Avi Rosenfeld, Sarvapali D. Ramchurn, and Nicholas R. Jennings. 2015 . Crowdsourcing complex workflows under budget constraints. In AAAI 2015. 1298--1304. Long Tran-Thanh, Trung Dong Huynh, Avi Rosenfeld, Sarvapali D. Ramchurn, and Nicholas R. Jennings. 2015. Crowdsourcing complex workflows under budget constraints. In AAAI 2015. 1298--1304."},{"key":"e_1_2_2_198_1","volume-title":"HCOMP","author":"Ukkonen Antti","year":"2015","unstructured":"Antti Ukkonen , Behrouz Derakhshan , and Hannes Heikinheimo . 2015 . Crowdsourced nonparametric density estimation using relative distances . In HCOMP 2015. Antti Ukkonen, Behrouz Derakhshan, and Hannes Heikinheimo. 2015. Crowdsourced nonparametric density estimation using relative distances. In HCOMP 2015."},{"key":"e_1_2_2_199_1","volume-title":"CHI","author":"Vaish Rajan","year":"2014","unstructured":"Rajan Vaish , Keith Wyngarden , Jingshu Chen , Brandon Cheung , and Michael S. Bernstein . 2014. Twitch crowdsourcing: Crowd contributions in short bursts of time . In CHI 2014 . 3645--3654. Rajan Vaish, Keith Wyngarden, Jingshu Chen, Brandon Cheung, and Michael S. Bernstein. 2014. Twitch crowdsourcing: Crowd contributions in short bursts of time. In CHI 2014. 3645--3654."},{"key":"e_1_2_2_200_1","doi-asserted-by":"publisher","DOI":"10.14778\/2732977.2732982"},{"key":"e_1_2_2_201_1","volume-title":"McKeon","author":"Vi\u00e9gas Fernanda B.","year":"2007","unstructured":"Fernanda B. Vi\u00e9gas , Martin Wattenberg , and Matthew M . McKeon . 2007 . The hidden order of Wikipedia. Online Communities and Social Computing. Springer , 445--454. Fernanda B. Vi\u00e9gas, Martin Wattenberg, and Matthew M. McKeon. 2007. The hidden order of Wikipedia. Online Communities and Social Computing. Springer, 445--454."},{"key":"e_1_2_2_202_1","doi-asserted-by":"publisher","DOI":"10.1126\/science.1160379"},{"key":"e_1_2_2_203_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-16558-0_36"},{"key":"e_1_2_2_204_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-16132-2_10"},{"key":"e_1_2_2_205_1","volume-title":"HCOMP","author":"Waggoner Bo","year":"2014","unstructured":"Bo Waggoner and Yiling Chen . 2014 . Output agreement mechanisms and common knowledge . In HCOMP 2014. Bo Waggoner and Yiling Chen. 2014. Output agreement mechanisms and common knowledge. In HCOMP 2014."},{"key":"e_1_2_2_206_1","volume-title":"Zhao","author":"Wang Gang","year":"2012","unstructured":"Gang Wang , Christo Wilson , Xiaohan Zhao , Yibo Zhu , Manish Mohanlal , Haitao Zheng , and Ben Y . Zhao . 2012 . Serf and turf: Crowdturfing for fun and profit. In WWW 2012. 679--688. Gang Wang, Christo Wilson, Xiaohan Zhao, Yibo Zhu, Manish Mohanlal, Haitao Zheng, and Ben Y. Zhao. 2012. Serf and turf: Crowdturfing for fun and profit. In WWW 2012. 679--688."},{"key":"e_1_2_2_207_1","volume-title":"NIPS 2011","author":"Fabian","year":"1800","unstructured":"Fabian L. Wauthier and Michael I. Jordan. 2011. Bayesian bias mitigation for crowdsourcing . In NIPS 2011 . Curran Associates, Inc. , 1800 --1808. Fabian L. Wauthier and Michael I. Jordan. 2011. Bayesian bias mitigation for crowdsourcing. In NIPS 2011. Curran Associates, Inc., 1800--1808."},{"key":"e_1_2_2_208_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998234"},{"key":"e_1_2_2_209_1","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2207709"},{"key":"e_1_2_2_210_1","doi-asserted-by":"publisher","DOI":"10.1002\/meet.2011.14504801135"},{"key":"e_1_2_2_211_1","volume-title":"ICML","author":"Yan Yan","year":"2011","unstructured":"Yan Yan , Glenn M. Fung , R\u00f3mer Rosales , and Jennifer G. Dy . 2011. Active learning from crowds . In ICML 2011 . 1161--1168. Yan Yan, Glenn M. Fung, R\u00f3mer Rosales, and Jennifer G. Dy. 2011. Active learning from crowds. In ICML 2011. 1161--1168."},{"key":"e_1_2_2_212_1","volume-title":"HCOMP","author":"Yang Jie","year":"2016","unstructured":"Jie Yang , Judith Redi , Gianluca DeMartini , and Alessandro Bozzon . 2016 . Modeling task complexity in crowdsourcing . In HCOMP 2016. 249--258. Jie Yang, Judith Redi, Gianluca DeMartini, and Alessandro Bozzon. 2016. Modeling task complexity in crowdsourcing. In HCOMP 2016. 249--258."},{"key":"e_1_2_2_213_1","volume-title":"HCOMP","author":"Yin Ming","year":"2014","unstructured":"Ming Yin , Yiling Chen , and Yu-An Sun . 2014 . Monetary interventions in crowdsourcing task switching . In HCOMP 2014. Ming Yin, Yiling Chen, and Yu-An Sun. 2014. Monetary interventions in crowdsourcing task switching. In HCOMP 2014."},{"key":"e_1_2_2_214_1","volume-title":"CSCW","author":"Yu Lixiu","year":"2014","unstructured":"Lixiu Yu , Paul Andr\u00e9 , Aniket Kittur , and Robert Kraut . 2014 . A comparison of social, learning, and financial strategies on crowd engagement and output quality . In CSCW 2014. 967--978. Lixiu Yu, Paul Andr\u00e9, Aniket Kittur, and Robert Kraut. 2014. A comparison of social, learning, and financial strategies on crowd engagement and output quality. In CSCW 2014. 967--978."},{"key":"e_1_2_2_215_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.physa.2006.04.057"},{"key":"e_1_2_2_216_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11063-014-9343-z"},{"key":"e_1_2_2_217_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2014.2327039"},{"key":"e_1_2_2_218_1","doi-asserted-by":"publisher","DOI":"10.1145\/2487575.2487708"},{"key":"e_1_2_2_219_1","volume-title":"CSCW","author":"Zhu Haiyi","year":"2014","unstructured":"Haiyi Zhu , Steven P. Dow , Robert E. Kraut , and Aniket Kittur . 2014 . Reviewing versus doing: Learning and performance in crowd assessment . In CSCW 2014. 1445--1455. Haiyi Zhu, Steven P. Dow, Robert E. Kraut, and Aniket Kittur. 2014. Reviewing versus doing: Learning and performance in crowd assessment. In CSCW 2014. 1445--1455."},{"key":"e_1_2_2_220_1","doi-asserted-by":"publisher","DOI":"10.1145\/2684822.2685301"}],"container-title":["ACM Computing Surveys"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3148148","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3148148","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T02:26:34Z","timestamp":1750213594000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3148148"}},"subtitle":["A Survey of Quality Attributes, Assessment Techniques, and Assurance Actions"],"short-title":[],"issued":{"date-parts":[[2018,1,4]]},"references-count":220,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2019,1,31]]}},"alternative-id":["10.1145\/3148148"],"URL":"https:\/\/doi.org\/10.1145\/3148148","relation":{},"ISSN":["0360-0300","1557-7341"],"issn-type":[{"value":"0360-0300","type":"print"},{"value":"1557-7341","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,1,4]]},"assertion":[{"value":"2016-06-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2017-09-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-01-04","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}