{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T06:35:01Z","timestamp":1782801301075,"version":"3.54.5"},"reference-count":96,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW2","license":[{"start":{"date-parts":[[2023,9,28]],"date-time":"2023-09-28T00:00:00Z","timestamp":1695859200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-sa\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2023,9,28]]},"abstract":"<jats:p>When groups of people are tasked with making a judgment, the issue of uncertainty often arises. Existing methods to reduce uncertainty typically focus on iteratively improving specificity in the overall task instruction. However, uncertainty can arise from multiple sources, such as ambiguity of the item being judged due to limited context, or disagreements among the participants due to different perspectives and an under-specified task. A one-size-fits-all intervention may be ineffective if it is not targeted to the right source of uncertainty. In this paper we introduce a new workflow, Judgment Sieve, to reduce uncertainty in tasks involving group judgment in a targeted manner. By utilizing measurements that separate different sources of uncertainty during an initial round of judgment elicitation, we can then select a targeted intervention adding context or deliberation to most effectively reduce uncertainty on each item being judged. We test our approach on two tasks: rating word pair similarity and toxicity of online comments, showing that targeted interventions reduced uncertainty for the most uncertain cases. In the top 10% of cases, we saw an ambiguity reduction of 21.4% and 25.7%, and a disagreement reduction of 22.2% and 11.2% for the two tasks respectively. We also found through a simulation that our targeted approach reduced the average uncertainty scores for both sources of uncertainty as opposed to uniform approaches where reductions in average uncertainty from one source came with an increase for the other.<\/jats:p>","DOI":"10.1145\/3610074","type":"journal-article","created":{"date-parts":[[2023,10,4]],"date-time":"2023-10-04T15:54:10Z","timestamp":1696434850000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":9,"title":["Judgment Sieve: Reducing Uncertainty in Group Judgments through Interventions Targeting Ambiguity versus Disagreement"],"prefix":"10.1145","volume":"7","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6500-8922","authenticated-orcid":false,"given":"Quan Ze","family":"Chen","sequence":"first","affiliation":[{"name":"University of Washington, Seattle, WA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9462-9835","authenticated-orcid":false,"given":"Amy X.","family":"Zhang","sequence":"additional","affiliation":[{"name":"University of Washington, Seattle, WA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,10,4]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"ACM","volume":"2013","author":"Aroyo Lora","year":"2013","unstructured":"Lora Aroyo and Chris Welty. 2013. Crowd truth: Harnessing disagreement in crowdsourcing a relation extraction gold standard. WebSci2013. ACM, Vol. 2013, 2013 (2013)."},{"key":"e_1_2_1_2_1","volume-title":"What is the Will of the People? Moderation Preferences for Misinformation. ArXiv","author":"Atreja Shubham","year":"2022","unstructured":"Shubham Atreja, Libby Hemphill, and Paul Resnick. 2022. What is the Will of the People? Moderation Preferences for Misinformation. ArXiv, Vol. abs\/2202.00799 (2022)."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1177\/1329878X20951301"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047201"},{"key":"e_1_2_1_5_1","volume-title":"Are we done with ImageNet? ArXiv","author":"Beyer Lucas","year":"2020","unstructured":"Lucas Beyer, Olivier J. H'enaff, Alexander Kolesnikov, Xiaohua Zhai, and A\"aron van den Oord. 2020. Are we done with ImageNet? ArXiv, Vol. abs\/2006.07159 (2020)."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3461702.3462571"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1162\/artl_a_00336"},{"key":"e_1_2_1_8_1","volume-title":"Christ\u00e8le Gras-Le Guen, et al","author":"Blangis Flora","year":"2021","unstructured":"Flora Blangis, Slimane Allali, J\u00e9r\u00e9mie F Cohen, Nathalie Vabres, Catherine Adamsbaum, Caroline Rey-Salmon, Andreas Werner, Yacine Refes, Pauline Adnot, Christ\u00e8le Gras-Le Guen, et al. 2021. Variations in guidelines for diagnosis of child physical abuse in high-income countries: a systematic review. JAMA network open, Vol. 4, 11 (2021), e2129068--e2129068."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3242587.3242598"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3432929"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3026044"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300761"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3476076"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359164"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v10i1.21986"},{"key":"e_1_2_1_16_1","volume-title":"Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experiment. ArXiv","volume":"2109","author":"Cortes Corinna","year":"2021","unstructured":"Corinna Cortes and Neil D. Lawrence. 2021. Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experiment. ArXiv, Vol. abs\/2109.09774 (2021)."},{"key":"e_1_2_1_17_1","volume-title":"Introduction to Statistics in Metrology","author":"Crowder Stephen","unstructured":"Stephen Crowder, Collin Delker, Eric Forrest, and Nevin Martin. 2020. Introduction to Statistics in Metrology. Springer."},{"key":"e_1_2_1_18_1","volume-title":"Online deliberation design: Choices, criteria, and evidence. arXiv preprint arXiv:1302.5177","author":"Davies Todd","year":"2013","unstructured":"Todd Davies and Reid Chandler. 2013. Online deliberation design: Choices, criteria, and evidence. arXiv preprint arXiv:1302.5177 (2013)."},{"key":"e_1_2_1_19_1","first-page":"20","article-title":"Maximum Likelihood Estimation of Observer Error?Rates Using the EM Algorithm","volume":"28","author":"Philip Dawid A.","year":"1979","unstructured":"A. Philip Dawid and Allan Skene. 1979. Maximum Likelihood Estimation of Observer Error?Rates Using the EM Algorithm. Journal of The Royal Statistical Society Series C-applied Statistics, Vol. 28 (1979), 20--28.","journal-title":"Journal of The Royal Statistical Society Series C-applied Statistics"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3159652.3159661"},{"key":"e_1_2_1_22_1","unstructured":"Djellel Eddine Difallah Gianluca Demartini and Philippe Cudr\u00e9-Mauroux. 2012. Mechanical cheat: Spamming schemes and adversarial techniques on crowdsourcing platforms. In CrowdSearch."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v4i1.13270"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3152889"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376293"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/503104.503110"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.naacl-main.204"},{"key":"e_1_2_1_28_1","volume-title":"Craig R. and G\u00fclden \u00dclk\u00fcmen","author":"Fox Craig R","year":"2011","unstructured":"Craig R Fox and G\u00fclden \u00dclk\u00fcmen. 2011. Distinguishing two dimensions of uncertainty. Fox, Craig R. and G\u00fclden \u00dclk\u00fcmen (2011),?Distinguishing Two Dimensions of Uncertainty,\" in Essays in Judgment and Decision Making, Brun, W., Kirkeb\u00f8en, G. and Montgomery, H., eds. Oslo: Universitetsforlaget (2011)."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-24258-3_8"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3078714.3078715"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3458723"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372862"},{"key":"e_1_2_1_33_1","volume-title":"Custodians of the Internet: Platforms, content moderation, and the hidden decisions that shape social media","author":"Gillespie Tarleton","unstructured":"Tarleton Gillespie. 2018. Custodians of the Internet: Platforms, content moderation, and the hidden decisions that shape social media. Yale University Press."},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445423"},{"key":"e_1_2_1_35_1","volume-title":"Understanding Crowdsourcing Workflow: Modeling and Optimizing Iterative and Parallel Processes. In AAAI Conference on Human Computation & Crowdsourcing.","author":"Goto Shinsuke","year":"2016","unstructured":"Shinsuke Goto, Toru Ishida, and Donghui Lin. 2016. Understanding Crowdsourcing Workflow: Modeling and Optimizing Iterative and Parallel Processes. In AAAI Conference on Human Computation & Crowdsourcing."},{"key":"e_1_2_1_36_1","volume-title":"Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial. Tutorials in quantitative methods for psychology","author":"Hallgren Kevin A.","year":"2012","unstructured":"Kevin A. Hallgren. 2012. Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial. Tutorials in quantitative methods for psychology, Vol. 8 1 (2012), 23--34."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3476073"},{"key":"e_1_2_1_38_1","volume-title":"Toward a synthesis of cognitive biases: how noisy information processing can bias human decision making. Psychological bulletin","author":"Hilbert Martin","year":"2012","unstructured":"Martin Hilbert. 2012. Toward a synthesis of cognitive biases: how noisy information processing can bias human decision making. Psychological bulletin, Vol. 138 2 (2012), 211--37."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0951-8320(96)00077-4"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.5555\/2390524.2390645"},{"key":"e_1_2_1_41_1","volume-title":"Machine Learning: An Introduction to Concepts and Methods. arXiv: Learning","author":"Hullermeier E.","year":"2019","unstructured":"E. Hullermeier and W. Waegeman. 2019. Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods. arXiv: Learning (2019)."},{"key":"e_1_2_1_42_1","doi-asserted-by":"crossref","unstructured":"Oana Inel and Lora Aroyo. 2017. Harnessing Diversity in Crowds and Machines for Better NER Performance. In ESWC.","DOI":"10.1007\/978-3-319-58068-5_18"},{"key":"e_1_2_1_43_1","unstructured":"Matthew Ingram. [n. d.]. Here's Why Facebook Removing That Vietnam War Photo Is So Important. Fortune ([n. d.]). https:\/\/fortune.com\/2016\/09\/09\/facebook-napalm-photo-vietnam-war\/"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/1837885.1837906"},{"key":"e_1_2_1_45_1","volume-title":"Casey Fiesler, and Jed R. Brubaker.","author":"Jiang Jialun Aaron","year":"2021","unstructured":"Jialun Aaron Jiang, Morgan Klaus Scheuerman, Casey Fiesler, and Jed R. Brubaker. 2021. Understanding international perceptions of the severity of harmful content online. PLoS ONE, Vol. 16 (2021)."},{"key":"e_1_2_1_46_1","volume-title":"Companion Proceedings of The 2019 World Wide Web Conference. 1121--1130","author":"Chaithanya Manam V K.","unstructured":"V K. Chaithanya Manam, Dwarakanath Jampani, Mariam Zaim, Meng-Han Wu, and Alexander J. Quinn. 2019. TaskMate: A Mechanism to Improve the Quality of Instructions in Crowdsourcing. In Companion Proceedings of The 2019 World Wide Web Conference. 1121--1130."},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2820016"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.strusafe.2008.06.020"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/1979742.1979869"},{"key":"e_1_2_1_50_1","volume-title":"Annotation Curricula to Implicitly Train Non-Expert Annotators. ArXiv","author":"Lee Ji-Ung","year":"2022","unstructured":"Ji-Ung Lee, Jan-Christoph Klie, and Iryna Gurevych. 2022. Annotation Curricula to Implicitly Train Non-Expert Annotators. ArXiv, Vol. abs\/2106.02382 (2022)."},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.822"},{"key":"e_1_2_1_52_1","volume-title":"A richly annotated pedestrian dataset for person retrieval in real surveillance scenarios","author":"Li Dangwei","year":"2018","unstructured":"Dangwei Li, Zhang Zhang, Xiaotang Chen, and Kaiqi Huang. 2018. A richly annotated pedestrian dataset for person retrieval in real surveillance scenarios. IEEE transactions on image processing, Vol. 28, 4 (2018), 1575--1590."},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.2307\/1287749"},{"key":"e_1_2_1_54_1","volume-title":"Proceedings of NAACL and HLT","author":"Liu Angli","year":"2016","unstructured":"Angli Liu, Stephen Soderland, Jonathan Bragg, Christopher H. Lin, Xiao Ling, and Daniel S. Weld. 2016. Effective Crowd Annotation for Relation Extraction. In Proceedings of NAACL and HLT 2016."},{"key":"e_1_2_1_55_1","volume-title":"Sixth AAAI Conference on Human Computation and Crowdsourcing.","author":"Chaithanya Manam VK","year":"2018","unstructured":"VK Chaithanya Manam and Alexander J Quinn. 2018. Wingit: Efficient refinement of unclear task instructions. In Sixth AAAI Conference on Human Computation and Crowdsourcing."},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3308560.3317081"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1139\/cjp-2017-0030"},{"key":"e_1_2_1_58_1","volume-title":"Controlling Bad Behavior in Online Communities: An Examination of Moderation Work. In International Conference on Interaction Sciences.","author":"McGillicuddy Aiden R.","year":"2020","unstructured":"Aiden R. McGillicuddy, Jean-Gr\u00e9goire Bernard, and Jocelyn Cranefield. 2020. Controlling Bad Behavior in Online Communities: An Examination of Moderation Work. In International Conference on Interaction Sciences."},{"key":"e_1_2_1_59_1","volume-title":"Alessio Palmero Aprosio, and Sara Tonelli","author":"Menini Stefano","year":"2021","unstructured":"Stefano Menini, Alessio Palmero Aprosio, and Sara Tonelli. 2021. Abuse is Contextual, What about NLP? The Role of Context in Abusive Language Annotation and Detection. ArXiv, Vol. abs\/2103.14916 (2021)."},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/219717.219748"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/3301275.3302301"},{"key":"e_1_2_1_62_1","unstructured":"Jethro Mullen and Charles Riley. [n. d.]. After outcry Facebook will reinstate iconic Vietnam War photo. CNN Business ([n. d.]). https:\/\/money.cnn.com\/2016\/09\/09\/technology\/facebook-censorship-vietnam-war-photo\/index.html"},{"key":"e_1_2_1_63_1","doi-asserted-by":"crossref","unstructured":"Alexandra Papoutsaki Hua Guo Dana\u00eb Metaxa-Kakavouli Connor Gramazio Jeff Rasley Wenting Xie Guan Wang and Jeff Huang. 2015. Crowdsourcing from Scratch: A Pragmatic Experiment in Data Collection by Novice Requesters. In HCOMP.","DOI":"10.1609\/hcomp.v3i1.13230"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00185"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00293"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.396"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.law-1.14"},{"key":"e_1_2_1_68_1","doi-asserted-by":"crossref","unstructured":"Vivek Pradhan Mike Schaekermann and Matthew Lease. 2021. In Search of Ambiguity: A Three-Stage Workflow Design to Clarify Annotation Guidelines for Crowd Workers. ArXiv Vol. abs\/2112.02255 (2021).","DOI":"10.3389\/frai.2022.828187"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF01499143"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-015-0816-y"},{"key":"e_1_2_1_71_1","unstructured":"Maarten Sap Dallas Card Saadia Gabriel Yejin Choi and Noah A Smith. 2019. The Risk of Racial Bias in Hate Speech Detection. In ACL. https:\/\/www.aclweb.org\/anthology\/P19--1163.pdf"},{"key":"e_1_2_1_72_1","volume-title":"Annotators with attitudes: How annotator beliefs and identities bias toxic language detection. arXiv preprint arXiv:2111.07997","author":"Sap Maarten","year":"2021","unstructured":"Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A Smith. 2021. Annotators with attitudes: How annotator beliefs and identities bias toxic language detection. arXiv preprint arXiv:2111.07997 (2021)."},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359178"},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274423"},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.1145\/3051457.3051466"},{"key":"e_1_2_1_76_1","volume-title":"Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics","author":"Snow Rion","year":"2008","unstructured":"Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Ng. 2008. Cheap and Fast -- But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Honolulu, Hawaii, 254--263. https:\/\/aclanthology.org\/D08--1027"},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1561\/1100000085"},{"key":"e_1_2_1_78_1","volume-title":"Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)","author":"Solorio Thamar","year":"2014","unstructured":"Thamar Solorio, Ragib Hasan, and Mainul Mizan. 2014. Sockpuppet Detection in Wikipedia: A Corpus of Real-World Deceptive Writing for Linking Identities. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14). European Language Resources Association (ELRA), Reykjavik, Iceland, 1355--1358. http:\/\/www.lrec-conf.org\/proceedings\/lrec2014\/pdf\/1007_Paper.pdf"},{"key":"e_1_2_1_79_1","volume-title":"Re-TACRED: Addressing Shortcomings of the TACRED Dataset. In AAAI Conference on Artificial Intelligence.","author":"Stoica George","year":"2021","unstructured":"George Stoica, Emmanouil Antonios Platanios, and Barnab'as P'oczos. 2021. Re-TACRED: Addressing Shortcomings of the TACRED Dataset. In AAAI Conference on Artificial Intelligence."},{"key":"e_1_2_1_80_1","doi-asserted-by":"publisher","DOI":"10.4101\/jvwr.v6i3.6409"},{"key":"e_1_2_1_81_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.746"},{"key":"e_1_2_1_82_1","doi-asserted-by":"publisher","DOI":"10.1086\/467688"},{"key":"e_1_2_1_83_1","doi-asserted-by":"publisher","DOI":"10.1111\/lcrp.12174"},{"key":"e_1_2_1_84_1","first-page":"1124","article-title":"Judgment under Uncertainty","volume":"185","author":"Tversky Amos","year":"1974","unstructured":"Amos Tversky and Daniel Kahneman. 1974. Judgment under Uncertainty: Heuristics and Biases. Science, Vol. 185 (1974), 1124--1131.","journal-title":"Heuristics and Biases. Science"},{"key":"e_1_2_1_85_1","volume-title":"Proc. ACM SIGIR Workshop on Crowdsourcing for Information Retrieval (CIR'11)","author":"Vuurens Jeroen","year":"2011","unstructured":"Jeroen Vuurens, Arjen P de Vries, and Carsten Eickhoff. 2011. How much spam can you take? an analysis of crowdsourcing results to increase accuracy. In Proc. ACM SIGIR Workshop on Crowdsourcing for Information Retrieval (CIR'11). 21--26."},{"key":"e_1_2_1_86_1","doi-asserted-by":"publisher","DOI":"10.1076\/iaij.4.1.5.16466"},{"key":"e_1_2_1_87_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.comnet.2021.108227"},{"key":"e_1_2_1_88_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W16-5618"},{"key":"e_1_2_1_89_1","volume-title":"Lora Mois Aroyo, and Praveen Kumar Paritosh","author":"Welty Chris","year":"2019","unstructured":"Chris Welty, Lora Mois Aroyo, and Praveen Kumar Paritosh. 2019. A Metrological Framework for Evaluating Crowd-powered Instruments. In HCOMP-2019: AAAI Conference on Human Computation."},{"key":"e_1_2_1_90_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359311"},{"key":"e_1_2_1_91_1","doi-asserted-by":"publisher","DOI":"10.3115\/1034678.1034721"},{"key":"e_1_2_1_92_1","volume-title":"Quinn","author":"Wu Meng-Han","year":"2017","unstructured":"Meng-Han Wu and Alexander J. Quinn. 2017. Confusing the Crowd: Task Instruction Quality on Amazon Mechanical Turk. In HCOMP."},{"key":"e_1_2_1_93_1","volume-title":"Ranking with Uncertain Labels. 2007 IEEE International Conference on Multimedia and Expo (2007)","author":"Yan Shuicheng","year":"2007","unstructured":"Shuicheng Yan, Huan Wang, Thomas S. Huang, Qiong Yang, and Xiaoou Tang. 2007. Ranking with Uncertain Labels. 2007 IEEE International Conference on Multimedia and Expo (2007), 96--99."},{"key":"e_1_2_1_94_1","volume-title":"Learn To Be Uncertain: Leveraging Uncertain Labels In Chest X-rays With Bayesian Neural Networks. In CVPR Workshops.","author":"Yang Hao-Yu","year":"2019","unstructured":"Hao-Yu Yang, Junling Yang, Yue Pan, Kunlin Cao, Qi Song, Feng Gao, and Youbing Yin. 2019. Learn To Be Uncertain: Leveraging Uncertain Labels In Chest X-rays With Bayesian Neural Networks. In CVPR Workshops."},{"key":"e_1_2_1_95_1","doi-asserted-by":"publisher","DOI":"10.1609\/icwsm.v11i1.14886"},{"key":"e_1_2_1_96_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D17--1004"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3610074","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3610074","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,8,21]],"date-time":"2025-08-21T04:17:44Z","timestamp":1755749864000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3610074"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,28]]},"references-count":96,"journal-issue":{"issue":"CSCW2","published-print":{"date-parts":[[2023,9,28]]}},"alternative-id":["10.1145\/3610074"],"URL":"https:\/\/doi.org\/10.1145\/3610074","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,9,28]]},"assertion":[{"value":"2023-10-04","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}