{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,18]],"date-time":"2026-05-18T11:31:03Z","timestamp":1779103863550,"version":"3.51.4"},"reference-count":82,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW","license":[{"start":{"date-parts":[[2019,11,7]],"date-time":"2019-11-07T00:00:00Z","timestamp":1573084800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100015599","name":"Toyota Research Institute","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100015599","id-type":"DOI","asserted-by":"crossref"}]},{"name":"the Next-Generation Information Computing Development Program through the National Research Foundation of Korea(NRF) funded by the Ministry of Science and ICT","award":["RF2017M3C4A7065960"],"award-info":[{"award-number":["RF2017M3C4A7065960"]}]},{"name":"the IBM AI Horizons Network"},{"name":"a DARPA Young Faculty Award"},{"name":"USDOT Center for Connected and Automated Transportation at the University of Michigan"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2019,11,7]]},"abstract":"<jats:p>When crowdsourcing the creation of machine learning datasets, statistical distributions that capture diverse answers can represent ambiguous data better than a single best answer. Unfortunately, collecting distributions is expensive because a large number of responses need to be collected to form a stable distribution. Despite this, the efficient collection of answer distributions-that is, ways to use less human effort to collect estimates of the eventual distribution that would be formed by a large group of responses-is an under-studied topic. In this paper, we demonstrate that this type of estimation is possible and characterize different elicitation approaches to guide the development of future systems. We investigate eight elicitation approaches along two dimensions: annotation granularity and estimation perspective. Annotation granularity is varied by annotating i) a single \"best\" label, ii) all relevant labels, iii) a ranking of all relevant labels, or iv) real-valued weights for all relevant labels. Estimation perspective is varied by prompting workers to either respond with their own answer or an estimate of the answer(s) that they expect other workers would provide. Our study collected ordinal annotations on the emotional valence of facial images from 1,960 crowd workers and found that, surprisingly, the most fine-grained elicitation methods were not the most accurate, despite workers spending more time to provide answers. Instead, the most efficient approach was to ask workers to choose all relevant classes that others would have selected. This resulted in a 21.4% reduction in the human time required to reach the same performance as the baseline (i.e., selecting a single answer with their own perspective). By analyzing cases in which finer-grained annotations degraded performance, we contribute to a better understanding of the trade-offs between answer elicitation approaches. Our work makes it more tractable to use answer distributions in large-scale tasks such as ML training, and aims to spark future work on techniques that can efficiently estimate answer distributions.<\/jats:p>","DOI":"10.1145\/3359164","type":"journal-article","created":{"date-parts":[[2019,11,8]],"date-time":"2019-11-08T13:40:21Z","timestamp":1573220421000},"page":"1-25","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":30,"title":["Efficient Elicitation Approaches to Estimate Collective Crowd Answers"],"prefix":"10.1145","volume":"3","author":[{"given":"John Joon Young","family":"Chung","sequence":"first","affiliation":[{"name":"University of Michigan, Ann Arbor, MI, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jean Y.","family":"Song","sequence":"additional","affiliation":[{"name":"University of Michigan, Ann Arbor, MI, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sindhu","family":"Kutty","sequence":"additional","affiliation":[{"name":"University of Michigan, Ann Arbor, MI, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sungsoo (Ray)","family":"Hong","sequence":"additional","affiliation":[{"name":"New York University, New York City, NY, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Juho","family":"Kim","sequence":"additional","affiliation":[{"name":"Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Walter S.","family":"Lasecki","sequence":"additional","affiliation":[{"name":"University of Michigan, Ann Arbor, MI, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2019,11,7]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v36i1.2564"},{"key":"e_1_2_1_2_1","volume-title":"Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI'17)","author":"Augustin Alexandry","year":"2017","unstructured":"Alexandry Augustin , Matteo Venanzi , Alex Rogers , and Nicholas R. Jennings . 2017. Bayesian Aggregation of Categorical Distributions with Applications in Crowdsourcing . In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI'17) . AAAI Press, Menlo Park, CA, USA, 1411--1417. https:\/\/doi.org\/10.24963\/ijcai. 2017 \/195 10.24963\/ijcai.2017 Alexandry Augustin, Matteo Venanzi, Alex Rogers, and Nicholas R. Jennings. 2017. Bayesian Aggregation of Categorical Distributions with Applications in Crowdsourcing. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI'17). AAAI Press, Menlo Park, CA, USA, 1411--1417. https:\/\/doi.org\/10.24963\/ijcai.2017\/195"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/FG.2018.00033"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2006.11.016"},{"key":"e_1_2_1_5_1","volume-title":"First AAAI conference on human computation and crowdsourcing. AAAI","author":"Bragg Jonathan","year":"2013","unstructured":"Jonathan Bragg , Daniel S Weld , 2013 . Crowdsourcing multi-label classification for taxonomy creation . In First AAAI conference on human computation and crowdsourcing. AAAI , Palo Alto, CA, USA. Jonathan Bragg, Daniel S Weld, et almbox. 2013. Crowdsourcing multi-label classification for taxonomy creation. In First AAAI conference on human computation and crowdsourcing. AAAI, Palo Alto, CA, USA."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3026044"},{"key":"e_1_2_1_7_1","volume-title":"Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI'19)","author":"Chen Quanze","unstructured":"Quanze Chen , Jonathan Bragg , Lydia B. Chilton , and Dan S. Weld . 2019. Cicero: Multi-Turn, Contextual Argumentation for Accurate Crowdsourcing . In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI'19) . ACM, New York, NY, USA, Paper No. 531. https:\/\/doi.org\/10.1145\/3290605.3300761 10.1145\/3290605.3300761 Quanze Chen, Jonathan Bragg, Lydia B. Chilton, and Dan S. Weld. 2019. Cicero: Multi-Turn, Contextual Argumentation for Accurate Crowdsourcing. In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI'19). ACM, New York, NY, USA, Paper No. 531. https:\/\/doi.org\/10.1145\/3290605.3300761"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2470654.2466265"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300460"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.2466\/pr0.1997.80.2.355"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1177\/147078530805000106"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1093\/poq\/nfn062"},{"key":"e_1_2_1_13_1","volume-title":"phone, mail, and mixed-mode surveys: the tailored design method","author":"Dillman Don A","unstructured":"Don A Dillman , Jolene D Smyth , and Leah Melani Christian . 2014. Internet , phone, mail, and mixed-mode surveys: the tailored design method . John Wiley & Sons . Don A Dillman, Jolene D Smyth, and Leah Melani Christian. 2014. Internet, phone, mail, and mixed-mode surveys: the tailored design method .John Wiley & Sons."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858268"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-18818-8_43"},{"key":"e_1_2_1_16_1","volume-title":"Proceedings of the Sixth AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2018","author":"Dumitrache Anca","year":"2018","unstructured":"Anca Dumitrache , Lora Aroyo , and Chris Welty . 2018 a. Capturing Ambiguity in Crowdsourcing Frame Disambiguation . In Proceedings of the Sixth AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2018 , Z\u00fc rich, Switzerland, July 5--8 , 2018. AAAI, Palo Alto, CA, USA, 12--20. Anca Dumitrache, Lora Aroyo, and Chris Welty. 2018a. Capturing Ambiguity in Crowdsourcing Frame Disambiguation. In Proceedings of the Sixth AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2018, Z\u00fc rich, Switzerland, July 5--8, 2018. AAAI, Palo Alto, CA, USA, 12--20."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3152889"},{"key":"e_1_2_1_18_1","volume-title":"Quality Metrics for Crowdsourcing with Disagreement. arXiv preprint arXiv:1808.06080","author":"Dumitrache Anca","year":"2018","unstructured":"Anca Dumitrache , Oana Inel , Lora Aroyo , Benjamin Timmermans , and Chris Welty . 2018c. CrowdTruth 2.0 : Quality Metrics for Crowdsourcing with Disagreement. arXiv preprint arXiv:1808.06080 ( 2018 ). Anca Dumitrache, Oana Inel, Lora Aroyo, Benjamin Timmermans, and Chris Welty. 2018c. CrowdTruth 2.0: Quality Metrics for Crowdsourcing with Disagreement. arXiv preprint arXiv:1808.06080 (2018)."},{"key":"e_1_2_1_19_1","volume-title":"FACES--A database of facial expressions in young, middle-aged, and older women and men: Development and validation. Behavior research methods","author":"Ebner Natalie C","year":"2010","unstructured":"Natalie C Ebner , Michaela Riediger , and Ulman Lindenberger . 2010. FACES--A database of facial expressions in young, middle-aged, and older women and men: Development and validation. Behavior research methods , Vol. 42 , 1 ( 2010 ), 351--362. https:\/\/doi.org\/10.3758\/BRM.42.1.351 10.3758\/BRM.42.1.351 Natalie C Ebner, Michaela Riediger, and Ulman Lindenberger. 2010. FACES--A database of facial expressions in young, middle-aged, and older women and men: Development and validation. Behavior research methods, Vol. 42, 1 (2010), 351--362. https:\/\/doi.org\/10.3758\/BRM.42.1.351"},{"key":"e_1_2_1_20_1","volume-title":"An introduction to the bootstrap","author":"Efron Bradley","unstructured":"Bradley Efron and Robert J Tibshirani . 1994. An introduction to the bootstrap . CRC press . Bradley Efron and Robert J Tibshirani. 1994. An introduction to the bootstrap .CRC press."},{"key":"e_1_2_1_21_1","volume-title":"On the universality and cultural specificity of emotion recognition: a meta-analysis. Psychological bulletin","author":"Elfenbein Hillary Anger","year":"2002","unstructured":"Hillary Anger Elfenbein and Nalini Ambady . 2002. On the universality and cultural specificity of emotion recognition: a meta-analysis. Psychological bulletin , Vol. 128 , 2 ( 2002 ), 203. Hillary Anger Elfenbein and Nalini Ambady. 2002. On the universality and cultural specificity of emotion recognition: a meta-analysis. Psychological bulletin, Vol. 128, 2 (2002), 203."},{"key":"e_1_2_1_22_1","volume-title":"Perspective Taking: Misstepping Into Others' Shoes. Handbook of imagination and mental simulation","author":"Epley Nicholas","year":"2008","unstructured":"Nicholas Epley and Eugene M Caruso . 2008 . Perspective Taking: Misstepping Into Others' Shoes. Handbook of imagination and mental simulation (2008), 295. Nicholas Epley and Eugene M Caruso. 2008. Perspective Taking: Misstepping Into Others' Shoes. Handbook of imagination and mental simulation (2008), 295."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1037\/0022-3514.87.3.327"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2017.2689998"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2016.2545658"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2013.51"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3325480.3325497"},{"key":"e_1_2_1_28_1","volume-title":"Judgment & Decision Making","volume":"9","author":"Goldstein Daniel G","year":"2014","unstructured":"Daniel G Goldstein and David Rothschild . 2014 . Lay understanding of probability distributions . Judgment & Decision Making , Vol. 9 , 1 (2014). Daniel G Goldstein and David Rothschild. 2014. Lay understanding of probability distributions. Judgment & Decision Making, Vol. 9, 1 (2014)."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/RE.2014.6912269"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274329"},{"key":"e_1_2_1_31_1","volume-title":"Tenenbaum","author":"Griffiths Thomas L.","year":"2008","unstructured":"Thomas L. Griffiths , Charles Kemp , and Joshua B . Tenenbaum . 2008 . Bayesian Models of Cognition .Cambridge University Press , 59--100. https:\/\/doi.org\/10.1017\/CBO9780511816772.006 10.1017\/CBO9780511816772.006 Thomas L. Griffiths, Charles Kemp, and Joshua B. Tenenbaum. 2008. Bayesian Models of Cognition .Cambridge University Press, 59--100. https:\/\/doi.org\/10.1017\/CBO9780511816772.006"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025781"},{"key":"e_1_2_1_33_1","volume-title":"Margrit Betke, and Kristen Grauman.","author":"Gurari Danna","year":"2019","unstructured":"Danna Gurari , Yinan Zhao , Suyog Dutt Jain , Margrit Betke, and Kristen Grauman. 2019 . Predicting How to Distribute Work Between Algorithms and Humans to Segment an Image Batch. International Journal of Computer Vision ( 2019), 1--19. Danna Gurari, Yinan Zhao, Suyog Dutt Jain, Margrit Betke, and Kristen Grauman. 2019. Predicting How to Distribute Work Between Algorithms and Humans to Segment an Image Batch. International Journal of Computer Vision (2019), 1--19."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173640"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359208"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2441776.2441847"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P14-1105"},{"key":"e_1_2_1_38_1","volume-title":"Psychology today","author":"Irving L Janis. 1971. Group","year":"1971","unstructured":"Irving L Janis. 1971. Group think. Psychology today , Vol. 5 , 6 ( 1971 ), 43--46. Irving L Janis. 1971. Groupthink. Psychology today, Vol. 5, 6 (1971), 43--46."},{"key":"e_1_2_1_39_1","volume-title":"Belief polarization is not always irrational. Psychological review","author":"Jern Alan","year":"2014","unstructured":"Alan Jern , Kai-Min K Chang , and Charles Kemp . 2014. Belief polarization is not always irrational. Psychological review , Vol. 121 , 2 ( 2014 ), 206. https:\/\/doi.org\/10.1037\/a0035941 10.1037\/a0035941 Alan Jern, Kai-Min K Chang, and Charles Kemp. 2014. Belief polarization is not always irrational. Psychological review, Vol. 121, 2 (2014), 206. https:\/\/doi.org\/10.1037\/a0035941"},{"key":"e_1_2_1_40_1","volume-title":"Kai min Chang, and Charles Kemp","author":"Jern Alan","year":"2009","unstructured":"Alan Jern , Kai min Chang, and Charles Kemp . 2009 . Bayesian belief polarization. In Advances in Neural Information Processing Systems 22, Y. Bengio, D. Schuurmans, J. D. Lafferty, C. K. I. Williams, and A. Culotta (Eds.). Curran Associates, Inc ., 853--861. http:\/\/papers.nips.cc\/paper\/3725-bayesian-belief-polarization.pdf Alan Jern, Kai min Chang, and Charles Kemp. 2009. Bayesian belief polarization. In Advances in Neural Information Processing Systems 22, Y. Bengio, D. Schuurmans, J. D. Lafferty, C. K. I. Williams, and A. Culotta (Eds.). Curran Associates, Inc., 853--861. http:\/\/papers.nips.cc\/paper\/3725-bayesian-belief-polarization.pdf"},{"key":"e_1_2_1_41_1","volume-title":"Thirty-Second AAAI Conference on Artificial Intelligence. AAAI","author":"Jia Xiuyi","year":"2018","unstructured":"Xiuyi Jia , Weiwei Li , Junyu Liu , and Yu Zhang . 2018 . Label Distribution Learning by Exploiting Label Correlations . In Thirty-Second AAAI Conference on Artificial Intelligence. AAAI , Palo Alto, CA, USA. Xiuyi Jia, Weiwei Li, Junyu Liu, and Yu Zhang. 2018. Label Distribution Learning by Exploiting Label Correlations. In Thirty-Second AAAI Conference on Artificial Intelligence. AAAI, Palo Alto, CA, USA."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N18-2099"},{"key":"e_1_2_1_43_1","volume-title":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics","author":"Jiang Youxuan","year":"1865","unstructured":"Youxuan Jiang , Jonathan K. Kummerfeld , and Walter S. Lasecki . 2017. Understanding Task Design Trade-offs in Crowdsourced Paraphrase Collection . In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics , Vancouver, Canada, 103--109. https:\/\/doi.org\/10. 1865 3\/v1\/P17--2017 10.18653\/v1 Youxuan Jiang, Jonathan K. Kummerfeld, and Walter S. Lasecki. 2017. Understanding Task Design Trade-offs in Crowdsourced Paraphrase Collection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, Vancouver, Canada, 103--109. https:\/\/doi.org\/10.18653\/v1\/P17--2017"},{"key":"e_1_2_1_44_1","first-page":"1","article-title":"Likert items and scales","volume":"1","author":"Johns Rob","year":"2010","unstructured":"Rob Johns . 2010 . Likert items and scales . Survey Question Bank: Methods Fact Sheet , Vol. 1 (2010), 1 -- 11 . Rob Johns. 2010. Likert items and scales. Survey Question Bank: Methods Fact Sheet, Vol. 1 (2010), 1--11.","journal-title":"Survey Question Bank: Methods Fact Sheet"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1037\/0022-3514.80.4.557"},{"key":"e_1_2_1_46_1","volume-title":"Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics","author":"Jurgens David","year":"2013","unstructured":"David Jurgens . 2013 . Embracing Ambiguity: A Comparison of Annotation Methodologies for Crowdsourcing Word Sense Labels . In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics , Atlanta, Georgia, USA, 556--562. David Jurgens. 2013. Embracing Ambiguity: A Comparison of Annotation Methodologies for Crowdsourcing Word Sense Labels. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Atlanta, Georgia, USA, 556--562."},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2820016"},{"key":"e_1_2_1_48_1","volume-title":"Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16)","author":"Kay Matthew","unstructured":"Matthew Kay , Tara Kola , Jessica R. Hullman , and Sean A. Munson . 2016. When (Ish) is My Bus?: User-centered Visualizations of Uncertainty in Everyday, Mobile Predictive Systems . In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16) . ACM, New York, NY, USA, 5092--5103. https:\/\/doi.org\/10.1145\/2858036.2858558 10.1145\/2858036.2858558 Matthew Kay, Tara Kola, Jessica R. Hullman, and Sean A. Munson. 2016. When (Ish) is My Bus?: User-centered Visualizations of Uncertainty in Everyday, Mobile Predictive Systems. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 5092--5103. https:\/\/doi.org\/10.1145\/2858036.2858558"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.5840\/jphil20081051024"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290607.3312769"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/1357054.1357127"},{"key":"e_1_2_1_52_1","volume-title":"On information and sufficiency. The annals of mathematical statistics","author":"Kullback Solomon","year":"1951","unstructured":"Solomon Kullback and Richard A Leibler . 1951. On information and sufficiency. The annals of mathematical statistics , Vol. 22 , 1 ( 1951 ), 79--86. http:\/\/www.jstor.org\/stable\/2236703 Solomon Kullback and Richard A Leibler. 1951. On information and sufficiency. The annals of mathematical statistics, Vol. 22, 1 (1951), 79--86. http:\/\/www.jstor.org\/stable\/2236703"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/2212776.2223835"},{"key":"e_1_2_1_54_1","volume-title":"Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST '14)","author":"Lasecki Walter S.","unstructured":"Walter S. Lasecki , Mitchell Gordon , Danai Koutra , Malte F. Jung , Steven P. Dow , and Jeffrey P. Bigham . 2014. Glance: Rapidly Coding Behavioral Video with the Crowd . In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST '14) . ACM, New York, NY, USA, 551--562. https:\/\/doi.org\/10.1145\/2642918.2647367 10.1145\/2642918.2647367 Walter S. Lasecki, Mitchell Gordon, Danai Koutra, Malte F. Jung, Steven P. Dow, and Jeffrey P. Bigham. 2014. Glance: Rapidly Coding Behavioral Video with the Crowd. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST '14). ACM, New York, NY, USA, 551--562. https:\/\/doi.org\/10.1145\/2642918.2647367"},{"key":"e_1_2_1_55_1","volume-title":"SIGIR 2010 workshop on crowdsourcing for search evaluation","volume":"2126","author":"Le John","year":"2010","unstructured":"John Le , Andy Edmonds , Vaughn Hester , and Lukas Biewald . 2010 . Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution . In SIGIR 2010 workshop on crowdsourcing for search evaluation , Vol. 2126 . ACM, New York, NY, USA. John Le, Andy Edmonds, Vaughn Hester, and Lukas Biewald. 2010. Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution. In SIGIR 2010 workshop on crowdsourcing for search evaluation, Vol. 2126. ACM, New York, NY, USA."},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274373"},{"key":"e_1_2_1_57_1","volume-title":"Twenty-Sixth AAAI Conference on Artificial Intelligence .","author":"Lin Christopher H","year":"2012","unstructured":"Christopher H Lin , Mausam Mausam , and Daniel S Weld . 2012 . Dynamically Switching between Synergistic Workflows for Crowdsourcing . In Twenty-Sixth AAAI Conference on Artificial Intelligence . Christopher H Lin, Mausam Mausam, and Daniel S Weld. 2012. Dynamically Switching between Synergistic Workflows for Crowdsourcing. In Twenty-Sixth AAAI Conference on Artificial Intelligence ."},{"key":"e_1_2_1_58_1","volume-title":"Bever","author":"Mackay Donald G.","year":"1967","unstructured":"Donald G. Mackay and Thomas G . Bever . 1967 . In search of ambiguity. Perception & Psychophysics , Vol. 2 , 5 (01 May 1967), 193--200. https:\/\/doi.org\/10.3758\/BF03213049 10.3758\/BF03213049 Donald G. Mackay and Thomas G. Bever. 1967. In search of ambiguity. Perception & Psychophysics, Vol. 2, 5 (01 May 1967), 193--200. https:\/\/doi.org\/10.3758\/BF03213049"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/1600150.1600175"},{"key":"e_1_2_1_60_1","volume-title":"Fourth AAAI Conference on Human Computation and Crowdsourcing. AAAI","author":"McDonnell Tyler","year":"2016","unstructured":"Tyler McDonnell , Matthew Lease , Mucahid Kutlu , and Tamer Elsayed . 2016 . Why is that relevant? Collecting annotator rationales for relevance judgments . In Fourth AAAI Conference on Human Computation and Crowdsourcing. AAAI , Palo Alto, CA, USA. Tyler McDonnell, Matthew Lease, Mucahid Kutlu, and Tamer Elsayed. 2016. Why is that relevant? Collecting annotator rationales for relevance judgments. In Fourth AAAI Conference on Human Computation and Crowdsourcing. AAAI, Palo Alto, CA, USA."},{"key":"e_1_2_1_61_1","volume-title":"The Meaning of Meaning: A Study of the Influence of Language upon Thought and of the Science of Symbolism","author":"Ogden Charles Kay","unstructured":"Charles Kay Ogden and Ivor Armstrong Richards . 1923. The Meaning of Meaning: A Study of the Influence of Language upon Thought and of the Science of Symbolism . Vol. 29 . K. Paul, Trench, Trubner & Company , Limited. Charles Kay Ogden and Ivor Armstrong Richards. 1923. The Meaning of Meaning: A Study of the Influence of Language upon Thought and of the Science of Symbolism. Vol. 29. K. Paul, Trench, Trubner & Company, Limited."},{"key":"e_1_2_1_62_1","volume-title":"Programmatic Gold: Targeted and Scalable Quality Assurance in Crowdsourcing. Human computation","author":"Oleson David","year":"2011","unstructured":"David Oleson , Alexander Sorokin , Greg P Laughlin , Vaughn Hester , John Le , and Lukas Biewald . 2011 . Programmatic Gold: Targeted and Scalable Quality Assurance in Crowdsourcing. Human computation , Vol. 11 , 11 (2011). David Oleson, Alexander Sorokin, Greg P Laughlin, Vaughn Hester, John Le, and Lukas Biewald. 2011. Programmatic Gold: Targeted and Scalable Quality Assurance in Crowdsourcing. Human computation, Vol. 11, 11 (2011)."},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1016\/0024-3795(82)90112-4"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P14-2083"},{"key":"e_1_2_1_65_1","volume-title":"Science","volume":"306","author":"Prelec Dravzen","year":"2004","unstructured":"Dravzen Prelec . 2004 . A Bayesian Truth Serum for Subjective Data . Science , Vol. 306 , 5695 (2004), 462--466. https:\/\/doi.org\/10.1126\/science.1102081 10.1126\/science.1102081 Dravzen Prelec. 2004. A Bayesian Truth Serum for Subjective Data. Science, Vol. 306, 5695 (2004), 462--466. https:\/\/doi.org\/10.1126\/science.1102081"},{"key":"e_1_2_1_66_1","volume-title":"Nature","volume":"541","author":"Prelec Dravz","year":"2017","unstructured":"Dravz en Prelec , H Sebastian Seung , and John McCoy . 2017 . A solution to the single-question crowd wisdom problem . Nature , Vol. 541 , 7638 (2017), 532. Dravz en Prelec, H Sebastian Seung, and John McCoy. 2017. A solution to the single-question crowd wisdom problem. Nature, Vol. 541, 7638 (2017), 532."},{"key":"e_1_2_1_67_1","first-page":"19 pages. https","volume-title":"Irresolvable Disagreement: A Study on Worker Deliberation in Crowd Work. Proc. ACM Hum.-Comput. Interact.","volume":"2","author":"Schaekermann Mike","year":"2018","unstructured":"Mike Schaekermann , Joslin Goh , Kate Larson , and Edith Law . 2018 . Resolvable vs . Irresolvable Disagreement: A Study on Worker Deliberation in Crowd Work. Proc. ACM Hum.-Comput. Interact. , Vol. 2 , CSCW, Article 154 (Nov. 2018), bibinfonumpages 19 pages. https:\/\/doi.org\/10 .1145\/3274423 10.1145\/3274423 Mike Schaekermann, Joslin Goh, Kate Larson, and Edith Law. 2018. Resolvable vs. Irresolvable Disagreement: A Study on Worker Deliberation in Crowd Work. Proc. ACM Hum.-Comput. Interact., Vol. 2, CSCW, Article 154 (Nov. 2018), bibinfonumpages19 pages. https:\/\/doi.org\/10.1145\/3274423"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/2675133.2675285"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/2675133.2675239"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/2757226.2757230"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.3115\/1613715.1613751"},{"key":"e_1_2_1_72_1","volume-title":"Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics","author":"Socher Richard","year":"2013","unstructured":"Richard Socher , Alex Perelygin , Jean Wu , Jason Chuang , Christopher D. Manning , Andrew Ng , and Christopher Potts . 2013 . Recursive deep models for semantic compositionality over a sentiment treebank . In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics , Seattle, Washington, USA, 1631--1642. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Seattle, Washington, USA, 1631--1642."},{"key":"e_1_2_1_73_1","article-title":"FourEyes: Leveraging Tool Diversity as a Means to Improve Aggregate Accuracy in Crowdsourcing","volume":"19","author":"Song Jean Y","year":"2019","unstructured":"Jean Y Song , Raymond Fok , Juho Kim , and Walter S Lasecki . 2019 . FourEyes: Leveraging Tool Diversity as a Means to Improve Aggregate Accuracy in Crowdsourcing . ACM Transactions on Interactive Intelligent Systems (TiiS) , Vol. 19 , 1 (2019), Article No.3. https:\/\/doi.org\/10.1145\/3237188 10.1145\/3237188 Jean Y Song, Raymond Fok, Juho Kim, and Walter S Lasecki. 2019. FourEyes: Leveraging Tool Diversity as a Means to Improve Aggregate Accuracy in Crowdsourcing. ACM Transactions on Interactive Intelligent Systems (TiiS), Vol. 19, 1 (2019), Article No.3. https:\/\/doi.org\/10.1145\/3237188","journal-title":"ACM Transactions on Interactive Intelligent Systems (TiiS)"},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1145\/3172944.3172948"},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.1145\/3301275.3302305"},{"key":"e_1_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.97"},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1145\/3059454.3059467"},{"key":"e_1_2_1_78_1","doi-asserted-by":"publisher","DOI":"10.1145\/2187836.2187969"},{"key":"e_1_2_1_79_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1467-9280.2008.02136.x"},{"key":"e_1_2_1_80_1","doi-asserted-by":"publisher","DOI":"10.1109\/SLT.2012.6424200"},{"key":"e_1_2_1_81_1","volume-title":"Visual Question Answer Diversity. In Sixth AAAI Conference on Human Computation and Crowdsourcing. AAAI","author":"Yang Chun-Ju","year":"2018","unstructured":"Chun-Ju Yang , Kristen Grauman , and Danna Gurari . 2018 . Visual Question Answer Diversity. In Sixth AAAI Conference on Human Computation and Crowdsourcing. AAAI , Palo Alto, CA, USA. Chun-Ju Yang, Kristen Grauman, and Danna Gurari. 2018. Visual Question Answer Diversity. In Sixth AAAI Conference on Human Computation and Crowdsourcing. AAAI, Palo Alto, CA, USA."},{"key":"e_1_2_1_82_1","doi-asserted-by":"publisher","DOI":"10.1145\/3136755.3136792"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3359164","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3359164","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:12:53Z","timestamp":1750201973000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3359164"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,11,7]]},"references-count":82,"journal-issue":{"issue":"CSCW","published-print":{"date-parts":[[2019,11,7]]}},"alternative-id":["10.1145\/3359164"],"URL":"https:\/\/doi.org\/10.1145\/3359164","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,11,7]]},"assertion":[{"value":"2019-11-07","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}