{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,24]],"date-time":"2026-10-24T01:10:50Z","timestamp":1792804250541,"version":"4.5.0"},"publisher-location":"New York, NY, USA","reference-count":86,"publisher":"ACM","license":[{"start":{"date-parts":[[2022,4,27]],"date-time":"2022-04-27T00:00:00Z","timestamp":1651017600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,4,27]]},"DOI":"10.1145\/3491101.3519614","type":"proceedings-article","created":{"date-parts":[[2022,4,29]],"date-time":"2022-04-29T16:49:48Z","timestamp":1651250988000},"page":"1-8","source":"Crossref","is-referenced-by-count":11,"title":["Debiased Label Aggregation for Subjective Crowdsourcing Tasks"],"prefix":"10.1145","author":[{"given":"Shaun","family":"Wallace","sequence":"first","affiliation":[{"name":"Computer Science, Brown University, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tianyuan","family":"Cai","sequence":"additional","affiliation":[{"name":"Adobe, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Brendan","family":"Le","sequence":"additional","affiliation":[{"name":"Computer Science, Brown University, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Luis A.","family":"Leiva","sequence":"additional","affiliation":[{"name":"University of Luxembourg, Luxembourg"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2022,4,28]]},"reference":[{"key":"e_1_3_2_2_1_1","volume-title":"Proc. WWW Companion. 1294\u20131295","author":"Aroyo Lora","year":"2019","unstructured":"Lora Aroyo , Anca Dumitrache , Oana Inel , Zolt\u00e1n Szl\u00e1vik , Benjamin Timmermans , and Chris Welty . 2019 . Crowdsourcing Inclusivity: Dealing with Diversity of Opinions, Perspectives and Ambiguity in Annotated Data . In Proc. WWW Companion. 1294\u20131295 . Lora Aroyo, Anca Dumitrache, Oana Inel, Zolt\u00e1n Szl\u00e1vik, Benjamin Timmermans, and Chris Welty. 2019. Crowdsourcing Inclusivity: Dealing with Diversity of Opinions, Perspectives and Ambiguity in Annotated Data. In Proc. WWW Companion. 1294\u20131295."},{"key":"e_1_3_2_2_2_1","volume-title":"Designing Evaluations of Machine Learning Models for Subjective Inference: The Case of Sentence Toxicity. In REAIS Workshop a HCOMP.","author":"Balayn Agathe","year":"2019","unstructured":"Agathe Balayn and Alessandro Bozzon . 2019 . Designing Evaluations of Machine Learning Models for Subjective Inference: The Case of Sentence Toxicity. In REAIS Workshop a HCOMP. Agathe Balayn and Alessandro Bozzon. 2019. Designing Evaluations of Machine Learning Models for Subjective Inference: The Case of Sentence Toxicity. In REAIS Workshop a HCOMP."},{"key":"e_1_3_2_2_3_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-58452-8_21"},{"key":"e_1_3_2_2_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3242587.3242598"},{"key":"e_1_3_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICPR.2014.756"},{"key":"e_1_3_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3026044"},{"key":"e_1_3_2_2_7_1","doi-asserted-by":"publisher","DOI":"10.1613\/jair.953"},{"key":"e_1_3_2_2_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2939672.2939785"},{"key":"e_1_3_2_2_9_1","volume-title":"Proc. ACM Hum. Comput. Interact. 3, CSCW","author":"Joon\u00a0Young Chung John","year":"2019","unstructured":"John Joon\u00a0Young Chung , Jean\u00a0 Y Song , Sindhu Kutty , Sungsoo Hong , Juho Kim , and Walter\u00a0 S Lasecki . 2019 . Efficient elicitation approaches to estimate collective crowd answers . Proc. ACM Hum. Comput. Interact. 3, CSCW (2019), 1\u201325. John Joon\u00a0Young Chung, Jean\u00a0Y Song, Sindhu Kutty, Sungsoo Hong, Juho Kim, and Walter\u00a0S Lasecki. 2019. Efficient elicitation approaches to estimate collective crowd answers. Proc. ACM Hum. Comput. Interact. 3, CSCW (2019), 1\u201325."},{"key":"e_1_3_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/VLHCC.2004.3"},{"key":"e_1_3_2_2_11_1","volume-title":"Fostering creativity in the classroom: General principles. The creativity research handbook 1, 84.114","author":"Cropley J","year":"1997","unstructured":"Arthur\u00a0 J Cropley . 1997. Fostering creativity in the classroom: General principles. The creativity research handbook 1, 84.114 ( 1997 ), 1\u201346. Arthur\u00a0J Cropley. 1997. Fostering creativity in the classroom: General principles. The creativity research handbook 1, 84.114 (1997), 1\u201346."},{"key":"e_1_3_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/2488388.2488414"},{"key":"e_1_3_2_2_13_1","doi-asserted-by":"crossref","unstructured":"Burcu\u00a0F. Darst Kristen\u00a0C. Malecki and Corinne\u00a0D. Engelman. 2018. Using recursive feature elimination in random forest to account for correlated variables in high dimensional data. BMC Genetics 19(2018).  Burcu\u00a0F. Darst Kristen\u00a0C. Malecki and Corinne\u00a0D. Engelman. 2018. Using recursive feature elimination in random forest to account for correlated variables in high dimensional data. BMC Genetics 19(2018).","DOI":"10.1186\/s12863-018-0633-8"},{"key":"e_1_3_2_2_14_1","volume-title":"Proc. HCOMP.","author":"De\u00a0Alfaro Luca","year":"2015","unstructured":"Luca De\u00a0Alfaro , Vassilis Polychronopoulos , and Michael Shavlovsky . 2015 . Reliable aggregation of boolean crowdsourced tasks . In Proc. HCOMP. Luca De\u00a0Alfaro, Vassilis Polychronopoulos, and Michael Shavlovsky. 2015. Reliable aggregation of boolean crowdsourced tasks. In Proc. HCOMP."},{"key":"e_1_3_2_2_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173986"},{"key":"e_1_3_2_2_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/1557019.1557053"},{"key":"e_1_3_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-18818-8_43"},{"key":"e_1_3_2_2_18_1","volume-title":"Proc. HCOMP.","author":"Dumitrache Anca","year":"2018","unstructured":"Anca Dumitrache , Lora Aroyo , and Chris Welty . 2018 . Capturing ambiguity in crowdsourcing frame disambiguation . In Proc. HCOMP. Anca Dumitrache, Lora Aroyo, and Chris Welty. 2018. Capturing ambiguity in crowdsourcing frame disambiguation. In Proc. HCOMP."},{"key":"e_1_3_2_2_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3152889"},{"key":"e_1_3_2_2_20_1","volume-title":"Comparing sets of patterns with the Jaccard index. Australas. J. Inf. Syst. 22","author":"Fletcher Sam","year":"2018","unstructured":"Sam Fletcher and Md\u00a0Zahidul Islam . 2018. Comparing sets of patterns with the Jaccard index. Australas. J. Inf. Syst. 22 ( 2018 ). Sam Fletcher and Md\u00a0Zahidul Islam. 2018. Comparing sets of patterns with the Jaccard index. Australas. J. Inf. Syst. 22 (2018)."},{"key":"e_1_3_2_2_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/223904.223972"},{"key":"e_1_3_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2011.01.017"},{"key":"e_1_3_2_2_23_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2017.2689998"},{"key":"e_1_3_2_2_24_1","volume-title":"Proc. AAAI. 1575\u20131581","author":"Gao W.","year":"2016","unstructured":"W. Gao , L. Wang , Y.-F. Li , and Z.-H. Zhou . 2016 . Risk minimization in the presence of label noise . In Proc. AAAI. 1575\u20131581 . W. Gao, L. Wang, Y.-F. Li, and Z.-H. Zhou. 2016. Risk minimization in the presence of label noise. In Proc. AAAI. 1575\u20131581."},{"key":"e_1_3_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445423"},{"key":"e_1_3_2_2_26_1","volume-title":"Proc. AAAI.","author":"Guan Melody","year":"2018","unstructured":"Melody Guan , Varun Gulshan , Andrew Dai , and Geoffrey Hinton . 2018 . Who said what: Modeling individual labelers improves classification . In Proc. AAAI. Melody Guan, Varun Gulshan, Andrew Dai, and Geoffrey Hinton. 2018. Who said what: Modeling individual labelers improves classification. In Proc. AAAI."},{"key":"e_1_3_2_2_27_1","unstructured":"Andrew Hard Chlo\u00e9\u00a0M Kiddon Daniel Ramage Francoise Beaufays Hubert Eichner Kanishka Rao Rajiv Mathews and Sean Augenstein. 2018. Federated Learning for Mobile Keyboard Prediction. In arXiv:1811.03604.  Andrew Hard Chlo\u00e9\u00a0M Kiddon Daniel Ramage Francoise Beaufays Hubert Eichner Kanishka Rao Rajiv Mathews and Sean Augenstein. 2018. Federated Learning for Mobile Keyboard Prediction. In arXiv:1811.03604."},{"key":"e_1_3_2_2_28_1","first-page":"11","article-title":"Machine learning-based sentiment analysis for twitter accounts","volume":"23","author":"Hasan Ali","year":"2018","unstructured":"Ali Hasan , Sana Moin , Ahmad Karim , and Shahaboddin Shamshirband . 2018 . Machine learning-based sentiment analysis for twitter accounts . Appl. Math. Comput. 23 , 1 (2018), 11 . Ali Hasan, Sana Moin, Ahmad Karim, and Shahaboddin Shamshirband. 2018. Machine learning-based sentiment analysis for twitter accounts. Appl. Math. Comput. 23, 1 (2018), 11.","journal-title":"Appl. Math. Comput."},{"key":"e_1_3_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/2766462.2767804"},{"key":"e_1_3_2_2_30_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.mcm.2012.01.006"},{"key":"e_1_3_2_2_31_1","volume-title":"Proc. UIST. 63\u201372","author":"I.","unstructured":"Jason\u00a0 I. Hong and James\u00a0A. Landay. 2000. SATIN: A Toolkit for Informal Ink-Based Applications . In Proc. UIST. 63\u201372 . Jason\u00a0I. Hong and James\u00a0A. Landay. 2000. SATIN: A Toolkit for Informal Ink-Based Applications. In Proc. UIST. 63\u201372."},{"key":"e_1_3_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/ISDA.2007.31"},{"key":"e_1_3_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300637"},{"key":"e_1_3_2_2_34_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10618-013-0306-1"},{"key":"e_1_3_2_2_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2820016"},{"key":"e_1_3_2_2_36_1","volume-title":"Proc. NeurIPS. 1953\u20131961","author":"Karger R","year":"2011","unstructured":"David\u00a0 R Karger , Sewoong Oh , and Devavrat Shah . 2011 . Iterative learning for reliable crowdsourcing systems . In Proc. NeurIPS. 1953\u20131961 . David\u00a0R Karger, Sewoong Oh, and Devavrat Shah. 2011. Iterative learning for reliable crowdsourcing systems. In Proc. NeurIPS. 1953\u20131961."},{"key":"e_1_3_2_2_37_1","doi-asserted-by":"publisher","DOI":"10.1109\/FG.2017.96"},{"key":"e_1_3_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274373"},{"key":"e_1_3_2_2_39_1","volume-title":"Context-Aware Gestures for Mixed-Initiative Text Editing UIs. Interact. Comput. 27, 6","author":"Leiva A.","year":"2014","unstructured":"Luis\u00a0 A. Leiva , Vicent Alabau , Ver\u00f3nica Romero , Alejandro\u00a0 H. Toselli , and Enrique Vidal . 2014. Context-Aware Gestures for Mixed-Initiative Text Editing UIs. Interact. Comput. 27, 6 ( 2014 ). Luis\u00a0A. Leiva, Vicent Alabau, Ver\u00f3nica Romero, Alejandro\u00a0H. Toselli, and Enrique Vidal. 2014. Context-Aware Gestures for Mixed-Initiative Text Editing UIs. Interact. Comput. 27, 6 (2014)."},{"key":"e_1_3_2_2_40_1","volume-title":"Proc. AAAI Workshops.","author":"Lin H","year":"2012","unstructured":"Christopher\u00a0 H Lin , Mausam Mausam , and Daniel\u00a0 S Weld . 2012 . Crowdsourcing control: Moving beyond multiple choice . In Proc. AAAI Workshops. Christopher\u00a0H Lin, Mausam Mausam, and Daniel\u00a0S Weld. 2012. Crowdsourcing control: Moving beyond multiple choice. In Proc. AAAI Workshops."},{"key":"e_1_3_2_2_41_1","doi-asserted-by":"publisher","DOI":"10.1016\/0031-3203(91)90009-T"},{"key":"e_1_3_2_2_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3308560.3317082"},{"key":"e_1_3_2_2_43_1","volume-title":"International Workshop on Graphics Recognition. 291\u2013304","author":"Liu Wenyin","year":"2003","unstructured":"Wenyin Liu . 2003 . On-line graphics recognition: State-of-the-art . In International Workshop on Graphics Recognition. 291\u2013304 . Wenyin Liu. 2003. On-line graphics recognition: State-of-the-art. In International Workshop on Graphics Recognition. 291\u2013304."},{"key":"e_1_3_2_2_44_1","volume-title":"Proc. AAAI. 937\u2013943","author":"Loftin Robert\u00a0Tyler","year":"2014","unstructured":"Robert\u00a0Tyler Loftin , James MacGlashan , Bei Peng , Matthew\u00a0 E Taylor , Michael\u00a0 L Littman , Jeff Huang , and David\u00a0 L Roberts . 2014 . A Strategy-Aware Technique for Learning Behaviors from Discrete Human Feedback .. In Proc. AAAI. 937\u2013943 . Robert\u00a0Tyler Loftin, James MacGlashan, Bei Peng, Matthew\u00a0E Taylor, Michael\u00a0L Littman, Jeff Huang, and David\u00a0L Roberts. 2014. A Strategy-Aware Technique for Learning Behaviors from Discrete Human Feedback.. In Proc. AAAI. 937\u2013943."},{"key":"e_1_3_2_2_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/332040.332458"},{"key":"e_1_3_2_2_46_1","volume-title":"Proc. HCOMP.","author":"Manam K\u00a0Chaithanya","year":"2018","unstructured":"V K\u00a0Chaithanya Manam and Alexander\u00a0 J Quinn . 2018 . Wingit: Efficient refinement of unclear task instructions . In Proc. HCOMP. VK\u00a0Chaithanya Manam and Alexander\u00a0J Quinn. 2018. Wingit: Efficient refinement of unclear task instructions. In Proc. HCOMP."},{"key":"e_1_3_2_2_47_1","volume-title":"Proc. ESWC. 88\u201399","author":"Maynard Diana","year":"2011","unstructured":"Diana Maynard and Adam Funk . 2011 . Automatic detection of political opinions in tweets . In Proc. ESWC. 88\u201399 . Diana Maynard and Adam Funk. 2011. Automatic detection of political opinions in tweets. In Proc. ESWC. 88\u201399."},{"key":"e_1_3_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174188"},{"key":"e_1_3_2_2_49_1","volume-title":"The Meaning of Meaning: A Study of the Influence of Language upon Thought and of the Science of Symbolism. Nature 111, 566","author":"Ogden Charles\u00a0Kay","year":"1923","unstructured":"Charles\u00a0Kay Ogden and Ivor\u00a0Armstrong Richards . 1923. The Meaning of Meaning: A Study of the Influence of Language upon Thought and of the Science of Symbolism. Nature 111, 566 ( 1923 ). Charles\u00a0Kay Ogden and Ivor\u00a0Armstrong Richards. 1923. The Meaning of Meaning: A Study of the Influence of Language upon Thought and of the Science of Symbolism. Nature 111, 566 (1923)."},{"key":"e_1_3_2_2_50_1","doi-asserted-by":"publisher","DOI":"10.3389\/fnhum.2014.00436"},{"key":"e_1_3_2_2_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3448110"},{"key":"e_1_3_2_2_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/1378773.1378775"},{"key":"e_1_3_2_2_53_1","volume-title":"HCC Workshop: Sketch Tools for Diagramming. 57\u201363","author":"Paulson Brandon","year":"2008","unstructured":"Brandon Paulson , Pankaj Rajan , Pedro Davalos , Ricardo Gutierrez-Osuna , and Tracy Hammond . 2008 . What!?! no Rubine features?: using geometric-based features to produce normalized confidence values for sketch recognition . In HCC Workshop: Sketch Tools for Diagramming. 57\u201363 . Brandon Paulson, Pankaj Rajan, Pedro Davalos, Ricardo Gutierrez-Osuna, and Tracy Hammond. 2008. What!?! no Rubine features?: using geometric-based features to produce normalized confidence values for sketch recognition. In HCC Workshop: Sketch Tools for Diagramming. 57\u201363."},{"key":"e_1_3_2_2_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835449.1835617"},{"key":"e_1_3_2_2_55_1","article-title":"Evaluation: from Precision, Recall and F-measure to ROC, Informedness, Markedness and Correlation","volume":"2","author":"Powers D.M.W.","year":"2011","unstructured":"D.M.W. Powers . 2011 . Evaluation: from Precision, Recall and F-measure to ROC, Informedness, Markedness and Correlation . J. Mach. Learn. Technol. 2 , 1 (2011). D.M.W. Powers. 2011. Evaluation: from Precision, Recall and F-measure to ROC, Informedness, Markedness and Correlation. J. Mach. Learn. Technol. 2, 1 (2011).","journal-title":"J. Mach. Learn. Technol."},{"key":"e_1_3_2_2_56_1","doi-asserted-by":"publisher","DOI":"10.5555\/1756006.1859894"},{"key":"e_1_3_2_2_57_1","doi-asserted-by":"publisher","DOI":"10.5555\/1005332.1005336"},{"key":"e_1_3_2_2_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/127719.122753"},{"key":"e_1_3_2_2_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274423"},{"key":"e_1_3_2_2_60_1","volume-title":"Proc. CSCW. 826\u2013838","author":"Sen Shilad","year":"2015","unstructured":"Shilad Sen , Margaret\u00a0 E Giesel , Rebecca Gold , Benjamin Hillmann , Matt Lesicko , Samuel Naden , Jesse Russell , Zixiao Wang , and Brent Hecht . 2015 . Turkers, Scholars,\u201d Arafat\u201d and\u201d Peace\u201d Cultural Communities and Algorithmic Gold Standards . In Proc. CSCW. 826\u2013838 . Shilad Sen, Margaret\u00a0E Giesel, Rebecca Gold, Benjamin Hillmann, Matt Lesicko, Samuel Naden, Jesse Russell, Zixiao Wang, and Brent Hecht. 2015. Turkers, Scholars,\u201d Arafat\u201d and\u201d Peace\u201d Cultural Communities and Algorithmic Gold Standards. In Proc. CSCW. 826\u2013838."},{"key":"e_1_3_2_2_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/1401890.1401965"},{"key":"e_1_3_2_2_62_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDAR.2007.4377056"},{"key":"e_1_3_2_2_63_1","volume-title":"Proc. NeurIPS. 1085\u20131092","author":"Smyth Padhraic","year":"1994","unstructured":"Padhraic Smyth , Usama Fayyad , Michael Burl , Pietro Perona , and Pierre Baldi . 1994 . Inferring Ground Truth from Subjective Labelling of Venus Images . In Proc. NeurIPS. 1085\u20131092 . Padhraic Smyth, Usama Fayyad, Michael Burl, Pietro Perona, and Pierre Baldi. 1994. Inferring Ground Truth from Subjective Labelling of Venus Images. In Proc. NeurIPS. 1085\u20131092."},{"key":"e_1_3_2_2_64_1","volume-title":"Proc. EMNLP. 254\u2013263","author":"Snow Rion","year":"2008","unstructured":"Rion Snow , Brendan O\u2019connor , Dan Jurafsky , and Andrew\u00a0 Y Ng . 2008 . Cheap and fast\u2013but is it good? evaluating non-expert annotations for natural language tasks . In Proc. EMNLP. 254\u2013263 . Rion Snow, Brendan O\u2019connor, Dan Jurafsky, and Andrew\u00a0Y Ng. 2008. Cheap and fast\u2013but is it good? evaluating non-expert annotations for natural language tasks. In Proc. EMNLP. 254\u2013263."},{"key":"e_1_3_2_2_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/2872518.2891113"},{"key":"e_1_3_2_2_66_1","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2018.2836969"},{"key":"e_1_3_2_2_67_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10115-020-01475-y"},{"key":"e_1_3_2_2_68_1","doi-asserted-by":"publisher","DOI":"10.1109\/34.57669"},{"key":"e_1_3_2_2_69_1","doi-asserted-by":"publisher","DOI":"10.1037\/a0017907"},{"key":"e_1_3_2_2_70_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2018.2860987"},{"key":"e_1_3_2_2_71_1","doi-asserted-by":"publisher","DOI":"10.1145\/3476060"},{"key":"e_1_3_2_2_72_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICDM.2018.00067"},{"key":"e_1_3_2_2_73_1","volume-title":"SemEval-2021 Task 12: Learning with Disagreements. In ACL Workshop on Semantic Evaluation.","author":"Uma Alexandra","year":"2021","unstructured":"Alexandra Uma , Tommaso Fornaciari , Anca Dumitrache , Tristan Miller , Jon Chamberlain , Barbara Plank , Edwin Simpson , and Massimo Poesio . 2021 . SemEval-2021 Task 12: Learning with Disagreements. In ACL Workshop on Semantic Evaluation. Alexandra Uma, Tommaso Fornaciari, Anca Dumitrache, Tristan Miller, Jon Chamberlain, Barbara Plank, Edwin Simpson, and Massimo Poesio. 2021. SemEval-2021 Task 12: Learning with Disagreements. In ACL Workshop on Semantic Evaluation."},{"key":"e_1_3_2_2_74_1","doi-asserted-by":"publisher","DOI":"10.1145\/3415243"},{"key":"e_1_3_2_2_75_1","doi-asserted-by":"publisher","DOI":"10.1145\/3479592"},{"key":"e_1_3_2_2_76_1","volume-title":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol.\u00a05. 187\u2013196","author":"Wallace Shaun","year":"2017","unstructured":"Shaun Wallace , Lucy Van\u00a0Kleunen , Marianne Aubin-Le\u00a0Quere , Abraham Peterkin , Yirui Huang , and Jeff Huang . 2017 . Drafty: Enlisting Users to be Editors who Maintain Structured Data . In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol.\u00a05. 187\u2013196 . Shaun Wallace, Lucy Van\u00a0Kleunen, Marianne Aubin-Le\u00a0Quere, Abraham Peterkin, Yirui Huang, and Jeff Huang. 2017. Drafty: Enlisting Users to be Editors who Maintain Structured Data. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol.\u00a05. 187\u2013196."},{"key":"e_1_3_2_2_77_1","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2019\/515"},{"key":"e_1_3_2_2_78_1","volume-title":"Proc. NeurIPS. 2424\u20132432","author":"Welinder Peter","year":"2010","unstructured":"Peter Welinder , Steve Branson , Pietro Perona , and Serge Belongie . 2010 . The multidimensional wisdom of crowds . In Proc. NeurIPS. 2424\u20132432 . Peter Welinder, Steve Branson, Pietro Perona, and Serge Belongie. 2010. The multidimensional wisdom of crowds. In Proc. NeurIPS. 2424\u20132432."},{"key":"e_1_3_2_2_79_1","volume-title":"Proc. NeurIPS. 2035\u20132043","author":"Whitehill Jacob","year":"2009","unstructured":"Jacob Whitehill , Ting-fan Wu, Jacob Bergsma , Javier Movellan , and Paul Ruvolo . 2009 . Whose vote should count more: Optimal integration of labels from labelers of unknown expertise . In Proc. NeurIPS. 2035\u20132043 . Jacob Whitehill, Ting-fan Wu, Jacob Bergsma, Javier Movellan, and Paul Ruvolo. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Proc. NeurIPS. 2035\u20132043."},{"key":"e_1_3_2_2_80_1","doi-asserted-by":"publisher","DOI":"10.1109\/PCCGA.2002.1167869"},{"key":"e_1_3_2_2_81_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3025496"},{"key":"e_1_3_2_2_82_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376870"},{"key":"e_1_3_2_2_83_1","doi-asserted-by":"publisher","DOI":"10.1145\/3136755.3136792"},{"key":"e_1_3_2_2_84_1","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2019\/204"},{"key":"e_1_3_2_2_85_1","volume-title":"Proc. SAC. 1331\u20131340","author":"Zhao Qian","year":"2018","unstructured":"Qian Zhao , F\u00a0Maxwell Harper , Gediminas Adomavicius , and Joseph\u00a0 A Konstan . 2018 . Explicit or implicit feedback? Engagement or satisfaction? A field experiment on machine-learning-based recommender systems . In Proc. SAC. 1331\u20131340 . Qian Zhao, F\u00a0Maxwell Harper, Gediminas Adomavicius, and Joseph\u00a0A Konstan. 2018. Explicit or implicit feedback? Engagement or satisfaction? A field experiment on machine-learning-based recommender systems. In Proc. SAC. 1331\u20131340."},{"key":"e_1_3_2_2_86_1","doi-asserted-by":"publisher","DOI":"10.14778\/3055540.3055547"}],"event":{"name":"CHI '22: CHI Conference on Human Factors in Computing Systems","location":"New Orleans LA USA","acronym":"CHI '22","sponsor":["SIGCHI ACM Special Interest Group on Computer-Human Interaction"]},"container-title":["CHI Conference on Human Factors in Computing Systems Extended Abstracts"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3491101.3519614","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3491101.3519614","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:31:01Z","timestamp":1750188661000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3491101.3519614"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,4,27]]},"references-count":86,"alternative-id":["10.1145\/3491101.3519614","10.1145\/3491101"],"URL":"https:\/\/doi.org\/10.1145\/3491101.3519614","relation":{},"subject":[],"published":{"date-parts":[[2022,4,27]]}}}