{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,7]],"date-time":"2026-02-07T18:46:52Z","timestamp":1770490012356,"version":"3.49.0"},"reference-count":40,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW","license":[{"start":{"date-parts":[[2018,11,1]],"date-time":"2018-11-01T00:00:00Z","timestamp":1541030400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100000038","name":"Natural Sciences and Engineering Research Council of Canada","doi-asserted-by":"publisher","award":["CHRP 478468-15"],"award-info":[{"award-number":["CHRP 478468-15"]}],"id":[{"id":"10.13039\/501100000038","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000024","name":"Canadian Institutes of Health Research","doi-asserted-by":"publisher","award":["CPG-140200"],"award-info":[{"award-number":["CPG-140200"]}],"id":[{"id":"10.13039\/501100000024","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2018,11]]},"abstract":"<jats:p>Crowdsourced classification of data typically assumes that objects can be unambiguously classified into categories. In practice, many classification tasks are ambiguous due to various forms of disagreement. Prior work shows that exchanging verbal justifications can significantly improve answer accuracy over aggregation techniques. In this work, we study how worker deliberation affects resolvability and accuracy using case studies with both an objective and a subjective task. Results show that case resolvability depends on various factors, including the level and reasons for the initial disagreement, as well as the amount and quality of deliberation activities. Our work reinforces the finding that deliberation can increase answer accuracy and the importance of verbal discussion in this process. We contribute a new public data set on worker deliberation for text classification tasks, and discuss considerations for the design of deliberation workflows for classification.<\/jats:p>","DOI":"10.1145\/3274423","type":"journal-article","created":{"date-parts":[[2018,11,1]],"date-time":"2018-11-01T21:21:27Z","timestamp":1541107287000},"page":"1-19","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":39,"title":["Resolvable vs. Irresolvable Disagreement"],"prefix":"10.1145","volume":"2","author":[{"given":"Mike","family":"Schaekermann","sequence":"first","affiliation":[{"name":"University of Waterloo, Waterloo, ON, Canada"}]},{"given":"Joslin","family":"Goh","sequence":"additional","affiliation":[{"name":"University of Waterloo, Waterloo, ON, Canada"}]},{"given":"Kate","family":"Larson","sequence":"additional","affiliation":[{"name":"University of Waterloo, Waterloo, ON, Canada"}]},{"given":"Edith","family":"Law","sequence":"additional","affiliation":[{"name":"University of Waterloo, Waterloo, ON, Canada"}]}],"member":"320","published-online":{"date-parts":[[2018,11]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Loughin","author":"Bilder Christopher R.","year":"2004","unstructured":"Christopher R. Bilder and Thomas M. Loughin. 2004. Testing for Marginal Independence between Two Categorical Variables with Multiple Responses. Biometrics 60, 1 (3 2004), 241--248."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3026044"},{"key":"e_1_2_1_3_1","volume-title":"An Experimental Application of the DELPHI Method to the Use of Experts. Management Science 9, 3 (4","author":"Dalkey Norman","year":"1963","unstructured":"Norman Dalkey and Olaf Helmer. 1963. An Experimental Application of the DELPHI Method to the Use of Experts. Management Science 9, 3 (4 1963), 458--467."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1365-2869.2008.00700.x"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2011.6126394"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858268"},{"key":"e_1_2_1_7_1","volume-title":"Proceedings of the 4th AAAI Conference on Human Computation and Crowdsourcing (HCOMP).","author":"Drapeau Ryan","unstructured":"Ryan Drapeau, Lydia B. Chilton, Jonathan Bragg, and Daniel S. Weld. 2016. MicroTalk: Using Argumentation to Improve Crowdsourcing Accuracy. In Proceedings of the 4th AAAI Conference on Human Computation and Crowdsourcing (HCOMP)."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3152889"},{"key":"e_1_2_1_9_1","volume-title":"Proceedings of the Eight International Conference on Language Resources and Evaluation - LREC '12","author":"Filatova Elena","year":"2012","unstructured":"Elena Filatova. 2012. Irony and Sarcasm: Corpus Generation and Analysis Using Crowdsourcing. In Proceedings of the Eight International Conference on Language Resources and Evaluation - LREC '12, Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Mehmet Ugur Dogan, Bente Maegaard, Joseph Mariani, Jan Odijk, and Stelios Piperidis (Eds.). European Language Resources Association (ELRA), 392--398."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1080\/19331681.2012.665755"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2984511.2984542"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.5555\/2584544"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v32i1.11756"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025781"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1061\/(ASCE)0887-3801(1995)9:4(244)"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1080\/03610928008827941"},{"key":"e_1_2_1_17_1","volume-title":"Victims of Groupthink: A Psychological Study of Foreign Policy Decisions and Fiascoes. The ANNALS of the American Academy of Political and Social Science 407, 1 (5","author":"Jones Alan M.","year":"1973","unstructured":"Alan M. Jones. 1973. Victims of Groupthink: A Psychological Study of Foreign Policy Decisions and Fiascoes. The ANNALS of the American Academy of Political and Social Science 407, 1 (5 1973), 179--180."},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2820016"},{"key":"e_1_2_1_19_1","volume-title":"Group decision making and communication technology. Organizational Behavior and Human Decision Processes 52, 1 (6","author":"Kiesler Sara","year":"1992","unstructured":"Sara Kiesler and Lee Sproull. 1992. Group decision making and communication technology. Organizational Behavior and Human Decision Processes 52, 1 (6 1992), 96--123."},{"key":"e_1_2_1_20_1","volume-title":"Webster","author":"Krause Jonathan","year":"2018","unstructured":"Jonathan Krause, Varun Gulshan, Ehsan Rahimy, Peter Karth, Kasumi Widner, Greg S. Corrado, Lily Peng, and Dale R. Webster. 2018. Grader Variability and the Importance of Reference Standards for Evaluating Machine Learning Models for Diabetic Retinopathy. Ophthalmology (3 2018)."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2531602.2531677"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/2145204.2145249"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2208621"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3159649"},{"key":"e_1_2_1_25_1","volume-title":"Generalized Linear Models (2 ed.)","author":"McCullagh Peter","unstructured":"Peter McCullagh and John Nelder. 1989. Generalized Linear Models (2 ed.). Chapman & Hall\/CRC."},{"key":"e_1_2_1_26_1","volume-title":"Proceedings of the 4th AAAI Conference on Human Computation and Crowdsourcing (HCOMP).","author":"McDonnell Tyler","year":"2016","unstructured":"Tyler McDonnell, Matthew Lease, Tamer Elsayad, and Mucahid Kutlu. 2016. Why Is That Relevant? Collecting Annotator Rationales for Relevance Judgments. In Proceedings of the 4th AAAI Conference on Human Computation and Crowdsourcing (HCOMP)."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1080\/135467896394500"},{"key":"e_1_2_1_28_1","volume-title":"Aggregated knowledge from a small number of debates outperforms the wisdom of large crowds. Nature Human Behaviour (1","author":"Navajas Joaquin","year":"2018","unstructured":"Joaquin Navajas, Tamara Niella, Gerry Garbulsky, Bahador Bahrami, and Mariano Sigman. 2018. Aggregated knowledge from a small number of debates outperforms the wisdom of large crowds. Nature Human Behaviour (1 2018)."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1559-1816.1977.tb02416.x"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1992.10476271"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.5555\/3061053.3061154"},{"key":"e_1_2_1_32_1","volume-title":"Ng","author":"Rajpurkar Pranav","year":"2017","unstructured":"Pranav Rajpurkar, Awni Y. Hannun, Masoumeh Haghpanahi, Codie Bourn, and Andrew Y. Ng. 2017. Cardiologist-Level Arrhythmia Detection with Convolutional Neural Networks. (7 2017). http:\/\/arxiv.org\/abs\/1707.01836"},{"key":"e_1_2_1_33_1","volume-title":"Rosenberg and Steven van Hout","author":"Richard","year":"2013","unstructured":"Richard S. Rosenberg and Steven van Hout. 2013. The American Academy of Sleep Medicine Inter-scorer Reliability Program: Sleep Stage Scoring. Journal of Clinical Sleep Medicine (1 2013)."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/N15-1047"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.2041-6962.2006.tb00028.x"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1988.10478613"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.5555\/1858842.1858904"},{"key":"e_1_2_1_39_1","volume-title":"Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HLT). 260--267","author":"Zaidan Omar F.","unstructured":"Omar F. Zaidan, Jason Eisner, and Christine D. Piatko. 2007. Using \"Annotator Rationales\" to Improve Machine Learning for Text Categorization. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HLT). 260--267."},{"key":"e_1_2_1_40_1","volume-title":"Proceedings of the NIPS 2008 Workshop on Cost Sensitive Learning.","author":"Zaidan Omar F.","unstructured":"Omar F. Zaidan, Jason Eisner, and Christine D. Piatko. 2008. Machine learning with annotator rationales to reduce annotation cost. In Proceedings of the NIPS 2008 Workshop on Cost Sensitive Learning."},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998235"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3274423","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3274423","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T00:44:35Z","timestamp":1750207475000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3274423"}},"subtitle":["A Study on Worker Deliberation in Crowd Work"],"short-title":[],"issued":{"date-parts":[[2018,11]]},"references-count":40,"journal-issue":{"issue":"CSCW","published-print":{"date-parts":[[2018,11]]}},"alternative-id":["10.1145\/3274423"],"URL":"https:\/\/doi.org\/10.1145\/3274423","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,11]]},"assertion":[{"value":"2018-11-01","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}