{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,18]],"date-time":"2026-05-18T10:49:18Z","timestamp":1779101358137,"version":"3.51.4"},"reference-count":81,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW2","license":[{"start":{"date-parts":[[2022,11,7]],"date-time":"2022-11-07T00:00:00Z","timestamp":1667779200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2022,11,7]]},"abstract":"<jats:p>Human data labeling with multiple labelers and the resulting conflict resolution remains the norm for many enterprise machine learning pipelines. Conflict resolution can be a time-intensive and costly process. Our goal was to study how human-AI collaboration can improve conflict resolution, by enabling users to automate groups of conflict resolution tasks. However, little is known about whether and how people will rely on automation during conflict resolution. Currently, automation commonly uses labelers' majority vote labels for conflict resolution, as the top chosen label by most labelers is often correct. We envisioned a system where an AI would assist in finding cases where the labeler majority vote was wrong and where automation is supported for batches or groups of conflicts. In order to understand whether humans could use labeler and AI information effectively, we investigated how and when users rely on labeler and AI information and on automated group conflict resolution. We ran a study with 144 Mechanical Turk workers. We found that automation increased users' accuracy\/time, use of automated conflict resolution was relatively similar regardless of whether the automation was based on labeler or AI selected labels, and providing labeler and AI selected labels may reduce inappropriate reliance on automation.<\/jats:p>","DOI":"10.1145\/3555212","type":"journal-article","created":{"date-parts":[[2022,11,11]],"date-time":"2022-11-11T22:59:06Z","timestamp":1668207546000},"page":"1-27","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":11,"title":["Reliance and Automation for Human-AI Collaborative Data Labeling Conflict Resolution"],"prefix":"10.1145","volume":"6","author":[{"given":"Michelle","family":"Brachman","sequence":"first","affiliation":[{"name":"IBM Research, Cambridge, MA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zahra","family":"Ashktorab","sequence":"additional","affiliation":[{"name":"IBM Research, Yorktown Heights, NY, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Michael","family":"Desmond","sequence":"additional","affiliation":[{"name":"IBM Research, Yorktown Heights, NY, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Evelyn","family":"Duesterwald","sequence":"additional","affiliation":[{"name":"IBM Research, Yorktown Heights, NY, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Casey","family":"Dugan","sequence":"additional","affiliation":[{"name":"IBM Research, Cambridge, MA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Narendra Nath","family":"Joshi","sequence":"additional","affiliation":[{"name":"Adobe, San Jose, CA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Qian","family":"Pan","sequence":"additional","affiliation":[{"name":"IBM Research, Cambridge, MA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Aabhas","family":"Sharma","sequence":"additional","affiliation":[{"name":"IBM Research, Cambridge, MA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2022,11,11]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/1978942.1978966"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300233"},{"key":"e_1_2_1_3_1","volume-title":"Michelle Brachman, Aabhas Sharma, Kristina Brimijoin, Qian Pan, Christine T Wolf, et al.","author":"Ashktorab Zahra","year":"2021","unstructured":"Zahra Ashktorab , Michael Desmond , Josh Andres , Michael Muller , Narendra Nath Joshi , Michelle Brachman, Aabhas Sharma, Kristina Brimijoin, Qian Pan, Christine T Wolf, et al. 2021 . AI-Assisted Human Labeling: Batching for Efficiency without Overreliance. Proceedings of the ACM on Human-Computer Interaction, Vol. 5 , CSCW 1 (2021), 1--27. Zahra Ashktorab, Michael Desmond, Josh Andres, Michael Muller, Narendra Nath Joshi, Michelle Brachman, Aabhas Sharma, Kristina Brimijoin, Qian Pan, Christine T Wolf, et al. 2021. AI-Assisted Human Labeling: Batching for Efficiency without Overreliance. Proceedings of the ACM on Human-Computer Interaction, Vol. 5, CSCW1 (2021), 1--27."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3415167"},{"key":"e_1_2_1_5_1","volume-title":"Optimizing AI for Teamwork. arXiv preprint arXiv:2004.13102","author":"Bansal Gagan","year":"2020","unstructured":"Gagan Bansal , Besmira Nushi , Ece Kamar , Eric Horvitz , and Daniel S Weld . 2020a. Optimizing AI for Teamwork. arXiv preprint arXiv:2004.13102 ( 2020 ). Gagan Bansal, Besmira Nushi, Ece Kamar, Eric Horvitz, and Daniel S Weld. 2020a. Optimizing AI for Teamwork. arXiv preprint arXiv:2004.13102 (2020)."},{"key":"e_1_2_1_6_1","volume-title":"Optimizing AI for Teamwork. In Proceedings of the AAAI Conference on Artificial Intelligence","volume":"35","author":"Bansal Gagan","year":"2021","unstructured":"Gagan Bansal , Besmira Nushi , Ece Kamar , Eric Horvitz , and Daniel S Weld . 2021 . Is the Most Accurate AI the Best Teammate ? Optimizing AI for Teamwork. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 35 . 11405--11414. Gagan Bansal, Besmira Nushi, Ece Kamar, Eric Horvitz, and Daniel S Weld. 2021. Is the Most Accurate AI the Best Teammate? Optimizing AI for Teamwork. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 11405--11414."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v7i1.5285"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v33i01.33012429"},{"key":"e_1_2_1_9_1","volume-title":"Marco Tulio Ribeiro, and Daniel S Weld","author":"Bansal Gagan","year":"2020","unstructured":"Gagan Bansal , Tongshuang Wu , Joyce Zhu , Raymond Fok , Besmira Nushi , Ece Kamar , Marco Tulio Ribeiro, and Daniel S Weld . 2020 b. Does the whole exceed its parts? the effect of ai explanations on complementary team performance. arXiv preprint arXiv:2006.14779 (2020). Gagan Bansal, Tongshuang Wu, Joyce Zhu, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel S Weld. 2020b. Does the whole exceed its parts? the effect of ai explanations on complementary team performance. arXiv preprint arXiv:2006.14779 (2020)."},{"key":"e_1_2_1_10_1","volume-title":"Think twice before jumping on the bandwagon: Clarifying concepts in research on the bandwagon effect. Political studies review","author":"Barnfield Matthew","year":"2020","unstructured":"Matthew Barnfield . 2020. Think twice before jumping on the bandwagon: Clarifying concepts in research on the bandwagon effect. Political studies review , Vol. 18 , 4 ( 2020 ), 553--574. Matthew Barnfield. 2020. Think twice before jumping on the bandwagon: Clarifying concepts in research on the bandwagon effect. Political studies review, Vol. 18, 4 (2020), 553--574."},{"key":"e_1_2_1_11_1","volume-title":"Comparing visual-interactive labeling with active learning: An experimental study","author":"Bernard J\u00fcrgen","year":"2017","unstructured":"J\u00fcrgen Bernard , Marco Hutter , Matthias Zeppelzauer , Dieter Fellner , and Michael Sedlmair . 2017. Comparing visual-interactive labeling with active learning: An experimental study . IEEE transactions on visualization and computer graphics, Vol. 24 , 1 ( 2017 ), 298--308. J\u00fcrgen Bernard, Marco Hutter, Matthias Zeppelzauer, Dieter Fellner, and Michael Sedlmair. 2017. Comparing visual-interactive labeling with active learning: An experimental study. IEEE transactions on visualization and computer graphics, Vol. 24, 1 (2017), 298--308."},{"key":"e_1_2_1_12_1","unstructured":"Virginia Braun and Victoria Clarke. 2012. Thematic analysis. (2012).  Virginia Braun and Victoria Clarke. 2012. Thematic analysis. (2012)."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3366423.3380250"},{"key":"e_1_2_1_14_1","volume-title":"Proceedings of the 25th International Conference on Intelligent User Interfaces. 454--464","author":"Zana Bucc","year":"2020","unstructured":"Zana Bucc inca, Phoebe Lin , Krzysztof Z Gajos , and Elena L Glassman . 2020 . Proxy tasks and subjective measures can be misleading in evaluating explainable ai systems . In Proceedings of the 25th International Conference on Intelligent User Interfaces. 454--464 . Zana Bucc inca, Phoebe Lin, Krzysztof Z Gajos, and Elena L Glassman. 2020. Proxy tasks and subjective measures can be misleading in evaluating explainable ai systems. In Proceedings of the 25th International Conference on Intelligent User Interfaces. 454--464."},{"key":"e_1_2_1_15_1","volume-title":"Maja Barbara Malaya, and Krzysztof Z Gajos","author":"Zana Bucc","year":"2021","unstructured":"Zana Bucc inca , Maja Barbara Malaya, and Krzysztof Z Gajos . 2021 . To Trust or to Think : Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making . arXiv preprint arXiv:2102.09692 (2021). Zana Bucc inca, Maja Barbara Malaya, and Krzysztof Z Gajos. 2021. To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making. arXiv preprint arXiv:2102.09692 (2021)."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICHI.2015.26"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.3115\/1699510.1699548"},{"key":"e_1_2_1_18_1","volume-title":"Noah Constant, Mario Guajardo-C\u00e9spedes, Steve Yuan, Chris Tar, et al.","author":"Cer Daniel","year":"2018","unstructured":"Daniel Cer , Yinfei Yang , Sheng-yi Kong, Nan Hua , Nicole Limtiaco , Rhomni St John , Noah Constant, Mario Guajardo-C\u00e9spedes, Steve Yuan, Chris Tar, et al. 2018 . Universal sentence encoder. arXiv preprint arXiv:1803.11175 (2018). Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-C\u00e9spedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder. arXiv preprint arXiv:1803.11175 (2018)."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3026044"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300761"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300460"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359164"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1002\/meet.14504701328"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v4i1.13270"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/PACIFICVIS.2017.8031598"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v6i1.13330"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1177\/1071181319631264"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1080\/09298215.2016.1200631"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376316"},{"key":"e_1_2_1_30_1","volume-title":"Proceedings of the ACM on Human-Computer Interaction","volume":"4","author":"Ghai Bhavya","year":"2021","unstructured":"Bhavya Ghai , Q Vera Liao , Yunfeng Zhang , Rachel Bellamy , and Klaus Mueller . 2021 . Explainable Active Learning (XAL) Toward AI Explanations as Interfaces for Machine Teachers . Proceedings of the ACM on Human-Computer Interaction , Vol. 4 , CSCW3 (2021), 1--28. Bhavya Ghai, Q Vera Liao, Yunfeng Zhang, Rachel Bellamy, and Klaus Mueller. 2021. Explainable Active Learning (XAL) Toward AI Explanations as Interfaces for Machine Teachers. Proceedings of the ACM on Human-Computer Interaction, Vol. 4, CSCW3 (2021), 1--28."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/1378773.1378804"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359152"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1109\/mis.2013.24"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2856767.2856811"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.5555\/1564131.1564137"},{"key":"e_1_2_1_36_1","volume-title":"Do We Trust the Crowd? Effects of Crowdsourcing on Perceived Credibility of Online Health Information. Health Communication","author":"Huang Yan","year":"2020","unstructured":"Yan Huang and S Shyam Sundar . 2020. Do We Trust the Crowd? Effects of Crowdsourcing on Perceived Credibility of Online Health Information. Health Communication ( 2020 ), 1--10. Yan Huang and S Shyam Sundar. 2020. Do We Trust the Crowd? Effects of Crowdsourcing on Perceived Credibility of Online Health Information. Health Communication (2020), 1--10."},{"key":"e_1_2_1_37_1","volume-title":"How machine-learning recommendations influence clinician treatment selections: the example of the antidepressant selection. Translational psychiatry","author":"Jacobs Maia","year":"2021","unstructured":"Maia Jacobs , Melanie F Pradier , Thomas H McCoy , Roy H Perlis , Finale Doshi-Velez , and Krzysztof Z Gajos . 2021. How machine-learning recommendations influence clinician treatment selections: the example of the antidepressant selection. Translational psychiatry , Vol. 11 , 1 ( 2021 ), 1--9. Maia Jacobs, Melanie F Pradier, Thomas H McCoy, Roy H Perlis, Finale Doshi-Velez, and Krzysztof Z Gajos. 2021. How machine-learning recommendations influence clinician treatment selections: the example of the antidepressant selection. Translational psychiatry, Vol. 11, 1 (2021), 1--9."},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2820016"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858402"},{"key":"e_1_2_1_40_1","volume-title":"Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrations. 5--9.","author":"Klie Jan-Christoph","year":"2018","unstructured":"Jan-Christoph Klie , Michael Bugert , Beto Boullosa , Richard Eckart de Castilho , and Iryna Gurevych . 2018 . The inception platform: Machine-assisted and knowledge-oriented interactive annotation . In Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrations. 5--9. Jan-Christoph Klie, Michael Bugert, Beto Boullosa, Richard Eckart de Castilho, and Iryna Gurevych. 2018. The inception platform: Machine-assisted and knowledge-oriented interactive annotation. In Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrations. 5--9."},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1207\/s15506878jobem4903_3"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.31234\/osf.io\/nfc45"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/2556288.2557238"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300717"},{"key":"e_1_2_1_45_1","volume-title":"The conflict detection and resolution in knowledge merging for image annotation. Information processing & management","author":"Lee Cheng-Yu","year":"2006","unstructured":"Cheng-Yu Lee and Von-Wun Soo . 2006. The conflict detection and resolution in knowledge merging for image annotation. Information processing & management , Vol. 42 , 4 ( 2006 ), 1030--1055. Cheng-Yu Lee and Von-Wun Soo. 2006. The conflict detection and resolution in knowledge merging for image annotation. Information processing & management, Vol. 42, 4 (2006), 1030--1055."},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1518\/hfes.46.1.50.30392"},{"key":"e_1_2_1_47_1","volume-title":"Automation failures on tasks easily performed by operators undermine trust in automated aids. Human factors","author":"Madhavan Poornima","year":"2006","unstructured":"Poornima Madhavan , Douglas A Wiegmann , and Frank C Lacson . 2006. Automation failures on tasks easily performed by operators undermine trust in automated aids. Human factors , Vol. 48 , 2 ( 2006 ), 241--256. Poornima Madhavan, Douglas A Wiegmann, and Frank C Lacson. 2006. Automation failures on tasks easily performed by operators undermine trust in automated aids. Human factors, Vol. 48, 2 (2006), 241--256."},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173922"},{"key":"e_1_2_1_49_1","volume-title":"Supporting trust calibration and the effective use of decision aids by presenting dynamic system confidence information. Human factors","author":"McGuirl John M","year":"2006","unstructured":"John M McGuirl and Nadine B Sarter . 2006. Supporting trust calibration and the effective use of decision aids by presenting dynamic system confidence information. Human factors , Vol. 48 , 4 ( 2006 ), 656--665. John M McGuirl and Nadine B Sarter. 2006. Supporting trust calibration and the effective use of decision aids by presenting dynamic system confidence information. Human factors, Vol. 48, 4 (2006), 656--665."},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jbusres.2020.09.004"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290607.3313054"},{"key":"e_1_2_1_52_1","doi-asserted-by":"crossref","unstructured":"Debra Meyerson Karl E Weick Roderick M Kramer etal 1996. Swift trust and temporary groups. Trust in organizations: Frontiers of theory and research Vol. 166 (1996) 195.  Debra Meyerson Karl E Weick Roderick M Kramer et al. 1996. Swift trust and temporary groups. Trust in organizations: Frontiers of theory and research Vol. 166 (1996) 195.","DOI":"10.4135\/9781452243610.n9"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445402"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/1743384.1743478"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0229132"},{"key":"e_1_2_1_56_1","volume-title":"Individual differences in the calibration of trust in automation. Human factors","author":"Pop Vlad L","year":"2015","unstructured":"Vlad L Pop , Alex Shrewsbury , and Francis T Durso . 2015. Individual differences in the calibration of trust in automation. Human factors , Vol. 57 , 4 ( 2015 ), 545--556. Vlad L Pop, Alex Shrewsbury, and Francis T Durso. 2015. Individual differences in the calibration of trust in automation. Human factors, Vol. 57, 4 (2015), 545--556."},{"key":"e_1_2_1_57_1","volume-title":"Squares: Supporting interactive performance analysis for multiclass classifiers","author":"Ren Donghao","year":"2016","unstructured":"Donghao Ren , Saleema Amershi , Bongshin Lee , Jina Suh , and Jason D Williams . 2016 . Squares: Supporting interactive performance analysis for multiclass classifiers . IEEE transactions on visualization and computer graphics, Vol. 23 , 1 (2016), 61--70. Donghao Ren, Saleema Amershi, Bongshin Lee, Jina Suh, and Jason D Williams. 2016. Squares: Supporting interactive performance analysis for multiclass classifiers. IEEE transactions on visualization and computer graphics, Vol. 23, 1 (2016), 61--70."},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445591"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359178"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376506"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274423"},{"key":"e_1_2_1_62_1","unstructured":"Burr Settles. 2009. Active learning literature survey. (2009).  Burr Settles. 2009. Active learning literature survey. (2009)."},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1108\/IMR-09-2014-0302"},{"key":"e_1_2_1_64_1","volume-title":"Proceedings of the 2008 conference on empirical methods in natural language processing. 254--263","author":"Snow Rion","year":"2008","unstructured":"Rion Snow , Brendan O'connor , Dan Jurafsky , and Andrew Y Ng . 2008 . Cheap and fast--but is it good? evaluating non-expert annotations for natural language tasks . In Proceedings of the 2008 conference on empirical methods in natural language processing. 254--263 . Rion Snow, Brendan O'connor, Dan Jurafsky, and Andrew Y Ng. 2008. Cheap and fast--but is it good? evaluating non-expert annotations for natural language tasks. In Proceedings of the 2008 conference on empirical methods in natural language processing. 254--263."},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.coling-main.422"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025171.3025208"},{"key":"e_1_2_1_67_1","volume-title":"The MAIN model: A heuristic approach to understanding technology effects on credibility","author":"Sundar S Shyam","unstructured":"S Shyam Sundar . 2008. The MAIN model: A heuristic approach to understanding technology effects on credibility . MacArthur Foundation Digital Media and Learning Initiative . S Shyam Sundar. 2008. The MAIN model: A heuristic approach to understanding technology effects on credibility. MacArthur Foundation Digital Media and Learning Initiative."},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.5555\/1228983.1228994"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/1358628.1358873"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/3334480.3381069"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359313"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.chb.2020.106278"},{"key":"e_1_2_1_73_1","volume-title":"Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. Advances in neural information processing systems","author":"Whitehill Jacob","year":"2009","unstructured":"Jacob Whitehill , Ting-fan Wu, Jacob Bergsma , Javier Movellan , and Paul Ruvolo . 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. Advances in neural information processing systems , Vol. 22 ( 2009 ), 2035--2043. Jacob Whitehill, Ting-fan Wu, Jacob Bergsma, Javier Movellan, and Paul Ruvolo. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. Advances in neural information processing systems, Vol. 22 (2009), 2035--2043."},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376301"},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300509"},{"key":"e_1_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.1145\/2930238.2930290"},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1145\/3301275.3302277"},{"key":"e_1_2_1_78_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998572"},{"key":"e_1_2_1_79_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173733"},{"key":"e_1_2_1_80_1","volume-title":"Sheng","author":"Zhang Jing","year":"2016","unstructured":"Jing Zhang , Xindong Wu , and Victor S . Sheng . 2016 . Learning from crowdsourced labeled data: a survey. Artificial Intelligence Review ( 2016). https:\/\/doi.org\/10.1007\/s10462-016--9491--9 10.1007\/s10462-016--9491--9 Jing Zhang, Xindong Wu, and Victor S. Sheng. 2016. Learning from crowdsourced labeled data: a survey. Artificial Intelligence Review (2016). https:\/\/doi.org\/10.1007\/s10462-016--9491--9"},{"key":"e_1_2_1_81_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3372852"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3555212","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3555212","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T17:49:14Z","timestamp":1750182554000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3555212"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,11,7]]},"references-count":81,"journal-issue":{"issue":"CSCW2","published-print":{"date-parts":[[2022,11,7]]}},"alternative-id":["10.1145\/3555212"],"URL":"https:\/\/doi.org\/10.1145\/3555212","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,11,7]]},"assertion":[{"value":"2022-11-11","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}