{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,1]],"date-time":"2025-12-01T02:53:40Z","timestamp":1764557620553,"version":"3.41.2"},"reference-count":86,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW2","license":[{"start":{"date-parts":[[2021,10,13]],"date-time":"2021-10-13T00:00:00Z","timestamp":1634083200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2021,10,13]]},"abstract":"<jats:p>We consider a class of variable effort human annotation tasks in which the number of labels required per item can greatly vary (e.g., finding all faces in an image, named entities in a text, bird calls in an audio recording, etc.). In such tasks, some items require far more effort than others to annotate. Furthermore, the per-item annotation effort is not known until after each item is annotated since determining the number of labels required is an implicit part of the annotation task itself. On an image bounding-box task with crowdsourced annotators, we show that annotator accuracy and recall consistently drop as effort increases. We hypothesize reasons for this drop and investigate a set of approaches to counteract it. Firstly, we benchmark on this task a set of general best-practice methods for quality crowdsourcing. Notably, only one of these methods actually improves quality: the use of visible gold questions that provide periodic feedback to workers on their accuracy as they work. Given these promising results, we then investigate and evaluate variants of the visible gold approach, yielding further improvement. Final results show a 7% improvement in bounding-box accuracy over the baseline. We discuss the generality of the visible gold approach and promising directions for future research.<\/jats:p>","DOI":"10.1145\/3476073","type":"journal-article","created":{"date-parts":[[2021,10,19]],"date-time":"2021-10-19T02:30:11Z","timestamp":1634610611000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":16,"title":["The Challenge of Variable Effort Crowdsourcing and How Visible Gold Can Help"],"prefix":"10.1145","volume":"5","author":[{"given":"Danula","family":"Hettiachchi","sequence":"first","affiliation":[{"name":"RMIT University, Melbourne, Australia"}]},{"given":"Mike","family":"Schaekermann","sequence":"additional","affiliation":[{"name":"Amazon, Toronto, ON, Canada"}]},{"given":"Tristan J.","family":"McKinney","sequence":"additional","affiliation":[{"name":"Amazon, Palo Alto, CA, USA"}]},{"given":"Matthew","family":"Lease","sequence":"additional","affiliation":[{"name":"Amazon &amp; University of Texas at Austin, Seattle, WA, USA"}]}],"member":"320","published-online":{"date-parts":[[2021,10,18]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Iterative Bounding Box Annotation for Object Detection. In International Conference on Pattern Recognition (ICPR) .","author":"Adhikari Bishwo","year":"2020","unstructured":"Bishwo Adhikari and Heikki Huttunen. 2020. Iterative Bounding Box Annotation for Object Detection. In International Conference on Pattern Recognition (ICPR) ."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1109\/EUVIP.2018.8611732"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3209542.3209558"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145382"},{"key":"e_1_2_1_5_1","volume-title":"Proceedings of the AAAI Conference on Human Computation (HCOMP). AAAI Press, 2--7.","author":"Attenberg Josh","year":"2011","unstructured":"Josh Attenberg, Panagiotis G. Ipeirotis, and Foster Provost. 2011. Beat the Machine: Challenging Workers to Find the Unknown Unknowns. In Proceedings of the AAAI Conference on Human Computation (HCOMP). AAAI Press, 2--7."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/1866029.1866078"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1007\/s41095-019-0149--9"},{"key":"e_1_2_1_8_1","volume-title":"Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS","author":"Bragg Jonathan","year":"2016","unstructured":"Jonathan Bragg, Mausam, and Daniel S. Weld. 2016. Optimal testing for crowd workers. Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS (2016), 966--974."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1177\/1745691610393980"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858237"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3026044"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v6i1.13332"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1613\/jair.1.11332"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/2702123.2702145"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2013.06.002"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2675133.2675260"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3148148"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2882903.2882953"},{"key":"e_1_2_1_19_1","volume-title":"Scaling-up the Crowd: Micro-Task Pricing Schemes for Worker Retention and Latency Improvement. Second AAAI Conference on Human Computation and Crowdsourcing Hcomp","author":"Difallah Djellel Eddine","year":"2014","unstructured":"Djellel Eddine Difallah, Michele Catasta, Gianluca Demartini, Philippe Cudr, and Philippe Cudr\u00e9 -Mauroux. 2014. Scaling-up the Crowd: Micro-Task Pricing Schemes for Worker Retention and Latency Improvement. Second AAAI Conference on Human Computation and Crowdsourcing Hcomp (2014), 50--58."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858268"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145355"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2750550"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3027385.3027402"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1007\/978--3--319--24258--3_8"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3078714.3078715"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v4i1.13289"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.5555\/1866696.1866723"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/3336191.3371857"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3289600.3291035"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2470744"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3415181"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/2736277.2741102"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/1807342.1807376"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2441776.2441847"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/2723372.2723731"},{"key":"e_1_2_1_36_1","unstructured":"Panos Ipeirotis. 2011. Pay Enough or Don't Pay at All. May 13. https:\/\/www.behind-the-enemy-lines.com\/2011\/05\/pay-enough-or-dont-pay-at-all.html."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3308560.3317081"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","unstructured":"Gabriella Kazai. 2011. In Search of Quality in Crowdsourcing for Search Engine Evaluation. In Advances in Information Retrieval. Springer Berlin Heidelberg 165--176. https:\/\/doi.org\/10.1007\/978--3--642--20161--5_17","DOI":"10.1007\/978--3--642--20161--5_17"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v34i1.2431"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3035918.3064055"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/1458082.1458160"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145357"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047202"},{"volume-title":"Crowdsourcing in Computer Vision","author":"Kovashka Adriana","key":"e_1_2_1_44_1","unstructured":"Adriana Kovashka, Olga Russakovsky, and Li Fei-Fei. 2016. Crowdsourcing in Computer Vision .Now Publishers Inc., Hanover, MA, USA."},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145354"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1613\/jair.1.12012"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-020-01316-z"},{"key":"e_1_2_1_48_1","volume-title":"SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation. 21--26","author":"Le John","year":"2010","unstructured":"John Le, Andy Edmonds, Vaughn Hester, and Lukas Biewald. 2010. Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution. In SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation. 21--26."},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1365-2966.2008.13689.x"},{"volume-title":"Proceedings of the ACM SIGKDD workshop on human computation. 68--76","author":"Little Greg","key":"e_1_2_1_50_1","unstructured":"Greg Little, Lydia B. Chilton, Max Goldman, and Robert C. Miller. 2010. Exploring iterative and parallel human computation processes. In Proceedings of the ACM SIGKDD workshop on human computation. 68--76."},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-019-01247--4"},{"key":"e_1_2_1_52_1","volume-title":"Advances in Neural Information Processing Systems","volume":"26","author":"Liu Qiang","year":"2013","unstructured":"Qiang Liu, Alexander T Ihler, and Mark Steyvers. 2013. Scoring Workers in Crowdsourcing: How Many Control Questions are Enough?. In Advances in Neural Information Processing Systems, Vol. 26. Curran Associates, Inc., 1914--1922."},{"volume-title":"Sixth AAAI Conference on Human Computation and Crowdsourcing (HCOMP","author":"Chaithanya Manam V.K.","key":"e_1_2_1_53_1","unstructured":"V.K. Chaithanya Manam and Alexander J. Quinn. 2018. WingIt: Efficient refinement of unclear task instructions. In Sixth AAAI Conference on Human Computation and Crowdsourcing (HCOMP, Vol. 6). AAAI Press."},{"key":"e_1_2_1_54_1","volume-title":"Design Activism for Minimum Wage Crowd Work. In Fifth AAAI Conference on Human Computation and Crowdsourcing (HCOMP): Works-in-Progress Track .","author":"Mankar Akash","year":"2017","unstructured":"Akash Mankar, Riddhi J. Shah, and Matthew Lease. 2017. Design Activism for Minimum Wage Crowd Work. In Fifth AAAI Conference on Human Computation and Crowdsourcing (HCOMP): Works-in-Progress Track ."},{"volume-title":"Proceedings of the ACM SIGKDD Workshop on Human Computation. ACM, 77--85","author":"Mason Winter","key":"e_1_2_1_55_1","unstructured":"Winter Mason and Duncan J. Watts. 2009. Financial incentives and the performance of crowds. In Proceedings of the ACM SIGKDD Workshop on Human Computation. ACM, 77--85."},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858490"},{"key":"e_1_2_1_57_1","volume-title":"Proceedings of the AAAI Conference on Human Computation (HCOMP). AAAI Press.","author":"Oleson David","year":"2011","unstructured":"David Oleson, Alexander Sorokin, Greg Laughlin, Vaughn Hester, John Le, and Lukas Biewald. 2011. Programmatic gold: Targeted and scalable quality assurance in crowdsourcing. In Proceedings of the AAAI Conference on Human Computation (HCOMP). AAAI Press."},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.99"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.528"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.27"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/2557500.2557512"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/3134724"},{"key":"e_1_2_1_63_1","volume-title":"Proceedings of the International AAAI Conference on Web and Social Media","volume":"5","author":"Rogstadius Jakob","year":"2011","unstructured":"Jakob Rogstadius, Vassilis Kostakos, Aniket Kittur, Boris Smus, Jim Laredo, and Maja Vukovic. 2011. An assessment of intrinsic and extrinsic motivation on task performance in crowdsourcing markets. In Proceedings of the International AAAI Conference on Web and Social Media, Vol. 5. AAAI Press."},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.24251\/HICSS.2019.637"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298824"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v3i1.13234"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/3366423.3380200"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376506"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274423"},{"key":"e_1_2_1_70_1","unstructured":"SetiHome. 2021. SetiHome. https:\/\/setiathome.berkeley.edu"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2019.00852"},{"key":"e_1_2_1_72_1","volume-title":"Crowdsourcing Annotations for Visual Object Detection. In Workshops at the Twenty-Sixth AAAI Conference on Artificial Intelligence. AAAI Press.","author":"Su Hao","year":"2012","unstructured":"Hao Su, Jia Deng, and Li Fei-Fei. 2012. Crowdsourcing Annotations for Visual Object Detection. In Workshops at the Twenty-Sixth AAAI Conference on Artificial Intelligence. AAAI Press."},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/1924421.1924441"},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858108"},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2018.2797962"},{"key":"e_1_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.5555\/3122009.3242050"},{"key":"e_1_2_1_77_1","unstructured":"Werner Vogels. 2007. Help Find Jim Gray. https:\/\/www.thingsdistributed.com\/2007\/02\/help_find_jim_gray.html."},{"key":"e_1_2_1_78_1","doi-asserted-by":"publisher","DOI":"10.1109\/DS-RT.2009.36"},{"key":"e_1_2_1_79_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998234"},{"key":"e_1_2_1_80_1","volume-title":"Fair Work: Crowd Work Minimum Wage with One Line of Code. In Proceedings of the Seventh AAAI Conference on Human Computation and Crowdsourcing (HCOMP","volume":"206","author":"Whiting Mark E.","unstructured":"Mark E. Whiting, Grant Hugh, and Michael S. Bernstein. 2019. Fair Work: Crowd Work Minimum Wage with One Line of Code. In Proceedings of the Seventh AAAI Conference on Human Computation and Crowdsourcing (HCOMP, Vol. 7). 197--206."},{"key":"e_1_2_1_81_1","unstructured":"Wikipedia. 2020. Where's Wally? waldourl."},{"key":"e_1_2_1_82_1","volume-title":"Modeling Task Complexity in Crowdsourcing. The Fourth AAAI Conference on Human Computation and Crowdsourcing October, 249--258","author":"Yang Jie","year":"2016","unstructured":"Jie Yang, Judith Redi, Gianluca Demartini, and Alessandro Bozzon. 2016. Modeling Task Complexity in Crowdsourcing. The Fourth AAAI Conference on Human Computation and Crowdsourcing October, 249--258. https:\/\/aaai.org\/ocs\/index.php\/HCOMP\/HCOMP16\/paper\/viewFile\/14039\/13653"},{"key":"e_1_2_1_83_1","volume-title":"Examining the Role of Perceived Fairness in Pay on the Performance Quality of Crowdworkers. Proceedings of the International AAAI Conference on Web and Social Media","volume":"11","author":"Ye Teng","year":"2017","unstructured":"Teng Ye, Sangseok You, and Lionel Robert Jr. 2017. When Does More Money Work? Examining the Role of Perceived Fairness in Pay on the Performance Quality of Crowdworkers. Proceedings of the International AAAI Conference on Web and Social Media, Vol. 11, 1."},{"key":"e_1_2_1_84_1","doi-asserted-by":"publisher","DOI":"10.5555\/2832249.2832277"},{"key":"e_1_2_1_85_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v4i1.13282"},{"key":"e_1_2_1_86_1","doi-asserted-by":"publisher","DOI":"10.1145\/2531602.2531718"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3476073","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3476073","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,14]],"date-time":"2025-07-14T04:53:09Z","timestamp":1752468789000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3476073"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,10,13]]},"references-count":86,"journal-issue":{"issue":"CSCW2","published-print":{"date-parts":[[2021,10,13]]}},"alternative-id":["10.1145\/3476073"],"URL":"https:\/\/doi.org\/10.1145\/3476073","relation":{},"ISSN":["2573-0142"],"issn-type":[{"type":"electronic","value":"2573-0142"}],"subject":[],"published":{"date-parts":[[2021,10,13]]},"assertion":[{"value":"2021-10-18","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}