{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,17]],"date-time":"2025-10-17T00:32:52Z","timestamp":1760661172109,"version":"build-2065373602"},"reference-count":123,"publisher":"Association for Computing Machinery (ACM)","issue":"7","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2025,10,18]]},"abstract":"<jats:p>Data annotation underpins the success of modern AI, but the aggregation of crowd-collected datasets can harm the preservation of diverse perspectives in data. Difficult and ambiguous tasks cannot easily be collapsed into unitary labels. Prior work has shown that deliberation and discussion improve data quality and preserve diverse perspectives--however, synchronous deliberation through crowdsourcing platforms is time-intensive and costly. In this work, we create a Socratic dialog system using Large Language Models (LLMs) to act as a deliberation partner in place of other crowdworkers. Against a benchmark of synchronous deliberation on two tasks (Sarcasm and Relation detection), our Socratic LLM encouraged participants to consider alternate annotation perspectives, update their labels as needed (with higher confidence), and resulted in higher annotation accuracy (for the Relation task where ground truth is available). Qualitative findings show that our agent's Socratic approach was effective at encouraging reasoned arguments from our participants, and that the intervention was well-received. Our methodology lays the groundwork for building scalable systems that preserve individual perspectives in generating more representative datasets.<\/jats:p>","DOI":"10.1145\/3757707","type":"journal-article","created":{"date-parts":[[2025,10,16]],"date-time":"2025-10-16T17:32:00Z","timestamp":1760635920000},"page":"1-35","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Wisdom of the Crowd, Without the Crowd: A Socratic LLM for Asynchronous Deliberation on Perspectivist Data"],"prefix":"10.1145","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9013-5595","authenticated-orcid":false,"given":"Malik","family":"Khadar","sequence":"first","affiliation":[{"name":"University of Minnesota, Minneapolis, MN, USA"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-2599-5286","authenticated-orcid":false,"given":"Daniel","family":"Runningen","sequence":"additional","affiliation":[{"name":"University of Minnesota, Minneapolis, MN, USA"}]},{"ORCID":"https:\/\/orcid.org\/0009-0007-4933-2267","authenticated-orcid":false,"given":"Julia","family":"Tang","sequence":"additional","affiliation":[{"name":"University of Minnesota, Minneapolis, MN, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0620-0903","authenticated-orcid":false,"given":"Stevie","family":"Chancellor","sequence":"additional","affiliation":[{"name":"University of Minnesota, Minneapolis, MN, USA"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-8239-937X","authenticated-orcid":false,"given":"Harmanpreet","family":"Kaur","sequence":"additional","affiliation":[{"name":"University of Minnesota, Minneapolis, MN, USA"}]}],"member":"320","published-online":{"date-parts":[[2025,10,16]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Reframing the Socratic Method","author":"Abrams Jamie R.","year":"2015","unstructured":"Jamie R. Abrams. 2015. Reframing the Socratic Method. Vol. 64, 4 (2015), 562-585. showeprint[jstor]24716713 https:\/\/www.jstor.org\/stable\/24716713"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3626252.3630799"},{"key":"e_1_2_1_3_1","volume-title":"2024 IEEE Symposium on Visual Languages and Human-Centric Computing (VL\/HCC). IEEE, 412-416","author":"Tamime Reham Al","year":"2024","unstructured":"Reham Al Tamime, Joni Salminen, Soon-Gyo Jung, and Bernard Jansen. 2024. Evaluating LLM-Generated Topics from Survey Responses: Identifying Challenges in Recruiting Participants through Crowdsourcing. In 2024 IEEE Symposium on Visual Languages and Human-Centric Computing (VL\/HCC). IEEE, 412-416."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1037\/stl0000121"},{"key":"e_1_2_1_5_1","volume-title":"Using crowdsourcing for TREC relevance assessment. Information processing & management","author":"Alonso Omar","year":"2012","unstructured":"Omar Alonso and Stefano Mizzaro. 2012. Using crowdsourcing for TREC relevance assessment. Information processing & management, Vol. 48, 6 (2012), 1053-1066."},{"key":"e_1_2_1_6_1","volume-title":"Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. 989-998","author":"Andr\u00e9 Paul","year":"2014","unstructured":"Paul Andr\u00e9, Aniket Kittur, and Steven P Dow. 2014. Crowd synthesis: Extracting categories and clusters from complex data. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. 989-998."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1609\/aimag.v36i1.2564"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3442188.3445922"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3613904.3642406"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-67256-4_32"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.kjss.2018.01.001"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3242587.3242598"},{"key":"e_1_2_1_13_1","volume-title":"Using thematic analysis in psychology. Qualitative research in psychology","author":"Braun Virginia","year":"2006","unstructured":"Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology, Vol. 3, 2 (2006), 77-101."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3449287"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v37i6.25840"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858237"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3501998"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025453.3026044"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3531146.3533071"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1080\/10447318.2020.1841438"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300761"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3476076"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3610074"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.5555\/3545946.3599105"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3148148"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00449"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1609\/icwsm.v11i1.14955"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","unstructured":"Valerio De Stefano. 2015. The Rise of the 'Just-in-Time Workforce': On-Demand Work Crowd Work and Labour Protection in the 'Gig-Economy'. showeprint[Social Science Research Network]2682602 doi:10.2139\/ssrn.2682602.","DOI":"10.2139\/ssrn.2682602"},{"key":"e_1_2_1_29_1","first-page":"85","article-title":"A review of mass media campaigns in road safety","volume":"220","author":"Delaney Amanda","year":"2004","unstructured":"Amanda Delaney, Bella Lough, Michelle Whelan, Max Cameron, et al., 2004. A review of mass media campaigns in road safety. Monash University Accident Research Centre Reports, Vol. 220 (2004), 85.","journal-title":"Monash University Accident Research Centre Reports"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.13187\/er.2016.111.511"},{"key":"e_1_2_1_31_1","doi-asserted-by":"crossref","unstructured":"Yuyang Ding Hanglei Hu Jie Zhou Qin Chen Bo Jiang and Liang He. 2024. Boosting Large Language Models with Socratic Method for Conversational Mathematics Teaching. arXiv:2407.17349 [cs]","DOI":"10.1145\/3627673.3679881"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/8732.001.0001"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v4i1.13270"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/MTS.2021.3056293"},{"key":"e_1_2_1_35_1","first-page":"392","article-title":"Irony and Sarcasm","author":"Filatova Elena","year":"2012","unstructured":"Elena Filatova. 2012. Irony and Sarcasm: Corpus Generation and Analysis Using Crowdsourcing. In Lrec. Citeseer, 392-398.","journal-title":"Corpus Generation and Analysis Using Crowdsourcing. In Lrec. Citeseer"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2405.05860"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.17220\/mojet.2019.02.003"},{"key":"e_1_2_1_38_1","doi-asserted-by":"crossref","first-page":"80","DOI":"10.3390\/ijgi7030080","article-title":"Increasing the accuracy of crowdsourced information on land cover via a voting procedure weighted by information inferred from the contributed data","volume":"7","author":"Fritz Steffen","year":"2018","unstructured":"Giles Foody, Linda See, Steffen Fritz, Inian Moorthy, Christoph Perger, Christian Schill, and Doreen Boyd. 2018. Increasing the accuracy of crowdsourced information on land cover via a voting procedure weighted by information inferred from the contributed data. ISPRS International Journal of Geo-Information, Vol. 7, 3 (2018), 80.","journal-title":"ISPRS International Journal of Geo-Information"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2013.2292894"},{"key":"e_1_2_1_40_1","volume-title":"Cristina Marco, and Davide Bernardi.","author":"Frenda Simona","year":"2024","unstructured":"Simona Frenda, Gavin Abercrombie, Valerio Basile, Alessandro Pedrani, Raffaella Panizzon, Alessandra Teresa Cignarella, Cristina Marco, and Davide Bernardi. 2024. Perspectivist approaches to natural language processing: a survey. Language Resources and Evaluation (2024), 1-28."},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/3491102.3502004"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445423"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v6i1.13331"},{"key":"e_1_2_1_44_1","volume-title":"of Crowdsourced Noisy Annotations, with an Instance-Dependent Noise Transition Model","author":"Guo Hui","year":"2023","unstructured":"Hui Guo, Boyu Wang, and Grace Yi. 2023. Label Correction, of Crowdsourced Noisy Annotations, with an Instance-Dependent Noise Transition Model. Vol. 36 (2023), 347-386. https:\/\/proceedings.neurips.cc\/paper_files\/paper\/2023\/hash\/015a8c69bedcb0a7b2ed2e1678f34399-Abstract-Conference.html"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2022.3207292"},{"key":"e_1_2_1_46_1","unstructured":"Jawad Haqbeen Takayuki Ito Rafik Hadfi Tomohiro Nishida Zoia Sahab Sofia Sahab Shafiq Roghmal and Mohammad Amiryar. 2020. Promoting Discussion with AI-based Facilitation: Urban Dialogue with Kabul City."},{"key":"e_1_2_1_47_1","volume-title":"Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Human mental workload\/Elsevier","author":"Hart Sandra G","year":"1988","unstructured":"Sandra G Hart. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Human mental workload\/Elsevier (1988)."},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1186\/s12909-023-04134-2"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/353485.353487"},{"key":"e_1_2_1_50_1","first-page":"1","volume-title":"Proceedings of the ACM on Human-Computer Interaction","volume":"7","author":"Houtti Mo","year":"2023","unstructured":"Mo Houtti, Moyan Zhou, Loren Terveen, and Stevie Chancellor. 2023. '' All of the White People Went First'': How Video Conferencing Consolidates Control and Exacerbates Workplace Bias. Proceedings of the ACM on Human-Computer Interaction, Vol. 7, CSCW1 (2023), 1-25."},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3657604.3664661"},{"key":"e_1_2_1_52_1","volume-title":"How a cockpit remembers its speeds. Cognitive science","author":"Hutchins Edwin","year":"1995","unstructured":"Edwin Hutchins. 1995. How a cockpit remembers its speeds. Cognitive science, Vol. 19, 3 (1995), 265-288."},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2407.06902"},{"key":"e_1_2_1_54_1","volume-title":"Proceedings of the third workshop on abusive language online. 46-57","author":"Ibrohim Muhammad Okky","year":"2019","unstructured":"Muhammad Okky Ibrohim and Indra Budi. 2019. Multi-label hate speech and abusive language detection in Indonesian Twitter. In Proceedings of the third workshop on abusive language online. 46-57."},{"key":"e_1_2_1_55_1","volume-title":"Crowdtruth: Machine-human computation framework for harnessing disagreement in gathering annotated data. In The Semantic Web-ISWC 2014: 13th International Semantic Web Conference, Riva del Garda","author":"Inel Oana","year":"2014","unstructured":"Oana Inel, Khalid Khamkham, Tatiana Cristea, Anca Dumitrache, Arne Rutjes, Jelle van der Ploeg, Lukasz Romaszko, Lora Aroyo, and Robert-Jan Sips. 2014. Crowdtruth: Machine-human computation framework for harnessing disagreement in gathering annotated data. In The Semantic Web-ISWC 2014: 13th International Semantic Web Conference, Riva del Garda, Italy, October 19-23, 2014. Proceedings, Part II 13. Springer, 486-504."},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/2566486.2567988"},{"key":"e_1_2_1_57_1","volume-title":"CHI'05 extended abstracts on Human factors in computing systems. 1489-1492","author":"Iqbal Shamsi T","year":"2005","unstructured":"Shamsi T Iqbal and Brian P Bailey. 2005. Investigating the effectiveness of mental workload as a predictor of opportune moments for interruption. In CHI'05 extended abstracts on Human factors in computing systems. 1489-1492."},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.15346\/hc.v1i2.14"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/3308560.3317081"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/3544548.3580645"},{"key":"e_1_2_1_61_1","doi-asserted-by":"crossref","first-page":"347","DOI":"10.1177\/1368430299024003","article-title":"Effects of differential status of group members on process and outcome of deliberation","volume":"2","author":"Kaplan Martin F","year":"1999","unstructured":"Martin F Kaplan and Ana M Martin. 1999. Effects of differential status of group members on process and outcome of deliberation. Group Processes & Intergroup Relations, Vol. 2, 4 (1999), 347-364.","journal-title":"Group Processes & Intergroup Relations"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/3610056"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2406.11709"},{"key":"e_1_2_1_64_1","first-page":"1","volume-title":"Proceedings of the ACM on Human-Computer Interaction","volume":"2","author":"Kaur Harmanpreet","year":"2018","unstructured":"Harmanpreet Kaur, Alex C Williams, Anne Loomis Thompson, Walter S Lasecki, Shamsi T Iqbal, and Jaime Teevan. 2018. Creating better action plans for writing tasks via vocabulary-based planning. Proceedings of the ACM on Human-Computer Interaction, Vol. 2, CSCW (2018), 1-22."},{"key":"e_1_2_1_65_1","volume-title":"Proceedings of ICLR","author":"Khetan Ashish","year":"2018","unstructured":"Ashish Khetan, Zachary C Lipton, and Anima Anandkumar. 2018. Learning from noisy singly-labeled data. In Proceedings of ICLR 2018."},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1145\/2441776.2441923"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","unstructured":"Charles Koutcheme Nicola Dainese Arto Hellas Sami Sarsa Juho Leinonen Syed Ashraf and Paul Denny. 2024. Evaluating Language Models for Generating and Judging Programming Feedback. arXiv:2407.04873 [cs] doi:10.48550\/arXiv.2407.04873.","DOI":"10.48550\/arXiv.2407.04873"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/2207676.2208621"},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/3613904.3642278"},{"key":"e_1_2_1_70_1","volume-title":"Deliberation and disagreement: Problem solving, prediction, and positive dissensus. Politics, philosophy & economics","author":"Landemore H\u00e9l\u00e8ne","year":"2015","unstructured":"H\u00e9l\u00e8ne Landemore and Scott E Page. 2015. Deliberation and disagreement: Problem solving, prediction, and positive dissensus. Politics, philosophy & economics, Vol. 14, 3 (2015), 229-254."},{"key":"e_1_2_1_71_1","volume-title":"This is not a boundary object: Reflections on the origin of a concept. Science, technology, & human values","author":"Star Susan Leigh","year":"2010","unstructured":"Susan Leigh Star. 2010. This is not a boundary object: Reflections on the origin of a concept. Science, technology, & human values, Vol. 35, 5 (2010), 601-617."},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.822"},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2016.2535242"},{"key":"e_1_2_1_74_1","volume-title":"The dark side of chatgpt: Legal and ethical challenges from stochastic parrots and hallucination. arXiv preprint arXiv:2304.14347","author":"Zihao Li.","year":"2023","unstructured":"Zihao Li. 2023. The dark side of chatgpt: Legal and ethical challenges from stochastic parrots and hallucination. arXiv preprint arXiv:2304.14347 (2023)."},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.1145\/3579607"},{"key":"e_1_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-43687-2_17"},{"key":"e_1_2_1_77_1","volume-title":"Proceedings of the 37th International Conference, on Machine Learning (ICML'20","volume":"6458","author":"Lukasik Michal","year":"2020","unstructured":"Michal Lukasik, Srinadh Bhojanapalli, Aditya Krishna Menon, and Sanjiv Kumar. 2020. Does Label Smoothing Mitigate Label Noise?. In Proceedings of the 37th International Conference, on Machine Learning (ICML'20, Vol. 119). JMLR.org, 6448-6458."},{"key":"e_1_2_1_78_1","doi-asserted-by":"publisher","DOI":"10.1145\/2783258.2783314"},{"key":"e_1_2_1_79_1","doi-asserted-by":"publisher","DOI":"10.1145\/3650400.3650635"},{"key":"e_1_2_1_80_1","doi-asserted-by":"publisher","unstructured":"Shuai Ma Qiaoyi Chen Xinru Wang Chengbo Zheng Zhenhui Peng Ming Yin and Xiaojuan Ma. 2024. Towards Human-AI Deliberation: Design and Evaluation of LLM-Empowered Deliberative AI for AI-Assisted Decision-Making. arXiv:2403.16812 doi:10.48550\/arXiv.2403.16812.","DOI":"10.48550\/arXiv.2403.16812"},{"key":"e_1_2_1_81_1","volume-title":"Quantifying public response towards Islam on Twitter after Paris attacks. arXiv preprint arXiv:1512.04570","author":"Magdy Walid","year":"2015","unstructured":"Walid Magdy, Kareem Darwish, and Norah Abokhodair. 2015. Quantifying public response towards Islam on Twitter after Paris attacks. arXiv preprint arXiv:1512.04570 (2015)."},{"key":"e_1_2_1_82_1","volume-title":"Paper delivered at the program in ethics and public affairs seminar","author":"Manin Bernard","unstructured":"Bernard Manin. 2005. Democratic deliberation: Why we should promote debate rather than discussion. In Paper delivered at the program in ethics and public affairs seminar, Princeton University, Vol. 13."},{"key":"e_1_2_1_83_1","doi-asserted-by":"publisher","DOI":"10.1145\/3630106.3658907"},{"key":"e_1_2_1_84_1","first-page":"35","volume-title":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing","volume":"9","author":"Melis David Alvarez","year":"2021","unstructured":"David Alvarez Melis, Harmanpreet Kaur, Hal Daum\u00e9 III, Hanna Wallach, and Jennifer Wortman Vaughan. 2021. From human explanation to model interpretability: A framework based on weight of evidence. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 9. 35-47."},{"key":"e_1_2_1_85_1","first-page":"18","volume-title":"Perspectives on Politics","volume":"12","author":"Mendelberg Tali","unstructured":"Tali Mendelberg, Christopher F Karpowitz, and J Baxter Oliphant. [n.d.]. Gender inequality in deliberation: Unpacking the black box of interaction. Perspectives on Politics, Vol. 12, 1 ([n.d.]), 18-44."},{"key":"e_1_2_1_86_1","volume-title":"Is deliberative democracy unfair to disadvantaged groups? In Democracy as Public Deliberation","author":"Miller David","unstructured":"David Miller. 2018. Is deliberative democracy unfair to disadvantaged groups? In Democracy as Public Deliberation. Routledge, 201-226."},{"key":"e_1_2_1_87_1","volume-title":"or Agonistic Pluralism?","author":"Mouffe Chantal","year":"1999","unstructured":"Chantal Mouffe. 1999. Deliberative Democracy, or Agonistic Pluralism? Vol. 66, 3 (1999), 745-758. showeprint[jstor]40971349 https:\/\/www.jstor.org\/stable\/40971349"},{"key":"e_1_2_1_88_1","volume-title":"Concise thoughts: Impact of output length on llm reasoning and cost. arXiv preprint arXiv:2407.19825","author":"Nayab Sania","year":"2024","unstructured":"Sania Nayab, Giulio Rossolini, Marco Simoni, Andrea Saracino, Giorgio Buttazzo, Nicolamaria Manes, and Fabrizio Giacomelli. 2024. Concise thoughts: Impact of output length on llm reasoning and cost. arXiv preprint arXiv:2407.19825 (2024)."},{"key":"e_1_2_1_89_1","doi-asserted-by":"publisher","DOI":"10.1145\/1743384.1743478"},{"key":"e_1_2_1_90_1","doi-asserted-by":"publisher","DOI":"10.1145\/3640543.3645157"},{"key":"e_1_2_1_91_1","first-page":"1182","volume-title":"Proceedings of The Web Conference","author":"Pei Weiping","year":"2020","unstructured":"Weiping Pei, Arthur Mayer, Kaylynn Tu, and Chuan Yue. 2020. Attention please: Your attention check questions in survey studies can be automatically answered. In Proceedings of The Web Conference 2020. 1182-1193."},{"key":"e_1_2_1_92_1","volume-title":"Distributed cognition. HCI models, theories, and frameworks: Toward a multidisciplinary science","author":"Perry Mark","year":"2003","unstructured":"Mark Perry. 2003. Distributed cognition. HCI models, theories, and frameworks: Toward a multidisciplinary science (2003), 193-223."},{"key":"e_1_2_1_93_1","doi-asserted-by":"publisher","unstructured":"Barbara Plank. 2022. The 'Problem' of Human Label Variation: On Ground Truth in Data Modeling and Evaluation. arXiv. doi:10.48550\/ARXIV.2211.02570.","DOI":"10.48550\/ARXIV.2211.02570"},{"key":"e_1_2_1_94_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-41154-0_1"},{"key":"e_1_2_1_95_1","doi-asserted-by":"publisher","DOI":"10.5555\/1866696.1866717"},{"key":"e_1_2_1_96_1","volume-title":"Sen Wu, Daniel Selsam, and Christopher R\u00e9.","author":"Ratner Alexander J","year":"2016","unstructured":"Alexander J Ratner, Christopher M De Sa, Sen Wu, Daniel Selsam, and Christopher R\u00e9. 2016. Data programming: Creating large training sets, quickly. Advances in neural information processing systems, Vol. 29 (2016)."},{"key":"e_1_2_1_97_1","doi-asserted-by":"publisher","DOI":"10.1145\/3637300"},{"key":"e_1_2_1_98_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445518"},{"key":"e_1_2_1_99_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.eacl-main.178"},{"key":"e_1_2_1_100_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-96957-8_36"},{"key":"e_1_2_1_101_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274423"},{"key":"e_1_2_1_102_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v1i1.13088"},{"key":"e_1_2_1_103_1","doi-asserted-by":"publisher","DOI":"10.1145\/3526113.3545671"},{"key":"e_1_2_1_104_1","volume-title":"Institutional ecology,translations' and boundary objects: Amateurs and professionals in Berkeley's Museum of Vertebrate Zoology","author":"Star Susan Leigh","year":"1907","unstructured":"Susan Leigh Star and James R Griesemer. 1989. Institutional ecology,translations' and boundary objects: Amateurs and professionals in Berkeley's Museum of Vertebrate Zoology, 1907-39. Social studies of science, Vol. 19, 3 (1989), 387-420."},{"key":"e_1_2_1_105_1","doi-asserted-by":"publisher","DOI":"10.1145\/3613904.3642513"},{"key":"e_1_2_1_106_1","volume-title":"Domain-weighted majority voting for crowdsourcing","author":"Tao Dapeng","year":"2018","unstructured":"Dapeng Tao, Jun Cheng, Zhengtao Yu, Kun Yue, and Lizhen Wang. 2018. Domain-weighted majority voting for crowdsourcing. IEEE transactions on neural networks and learning systems, Vol. 30, 1 (2018), 163-174."},{"key":"e_1_2_1_107_1","volume-title":"The uses of argument","author":"Toulmin Stephen E","unstructured":"Stephen E Toulmin. 2003. The uses of argument. Cambridge university press."},{"key":"e_1_2_1_108_1","doi-asserted-by":"publisher","DOI":"10.1145\/3579605"},{"key":"e_1_2_1_109_1","volume-title":"Use of mass media campaigns to change health behaviour. The lancet","author":"Wakefield Melanie A","year":"2010","unstructured":"Melanie A Wakefield, Barbara Loken, and Robert C Hornik. 2010. Use of mass media campaigns to change health behaviour. The lancet, Vol. 376, 9748 (2010), 1261-1271."},{"key":"e_1_2_1_110_1","first-page":"263","article-title":"Active Learning Strategies, to Promote Critical Thinking","volume":"38","author":"Walker Stacy E.","year":"2003","unstructured":"Stacy E. Walker. 2003. Active Learning Strategies, to Promote Critical Thinking. Journal of Athletic Training, Vol. 38, 3 (2003), 263-267.","journal-title":"Journal of Athletic Training"},{"key":"e_1_2_1_111_1","doi-asserted-by":"publisher","DOI":"10.1145\/3491101.3519614"},{"key":"e_1_2_1_112_1","volume-title":"The new dialectic: Conversational contexts of argument","author":"Walton Douglas N","unstructured":"Douglas N Walton. 1998. The new dialectic: Conversational contexts of argument. University of Toronto Press."},{"key":"e_1_2_1_113_1","volume-title":"Detecting coherent groups in crowd scenes by multiview clustering","author":"Wang Qi","year":"2018","unstructured":"Qi Wang, Mulin Chen, Feiping Nie, and Xuelong Li. 2018. Detecting coherent groups in crowd scenes by multiview clustering. IEEE transactions on pattern analysis and machine intelligence, Vol. 42, 1 (2018), 46-58."},{"key":"e_1_2_1_114_1","first-page":"4679","article-title":"Disagreement matters: Preserving label diversity by jointly modeling item and annotator label distributions with DisCo","volume":"2023","author":"Weerasooriya Tharindu Cyril","year":"2023","unstructured":"Tharindu Cyril Weerasooriya, Alexander Ororbia, Raj Bhensadadia, Ashiqur KhudaBukhsh, and Christopher Homan. 2023. Disagreement matters: Preserving label diversity by jointly modeling item and annotator label distributions with DisCo. In Findings of the Association for Computational Linguistics: ACL 2023. 4679-4695.","journal-title":"Findings of the Association for Computational Linguistics: ACL"},{"key":"e_1_2_1_115_1","unstructured":"Guy Williams et al. 2014. Harkness learning: principles of a radical American pedagogy. (2014)."},{"key":"e_1_2_1_116_1","unstructured":"Xuansheng Wu Haiyan Zhao Yaochen Zhu Yucheng Shi Fan Yang Tianming Liu Xiaoming Zhai Wenlin Yao Jundong Li Mengnan Du et al. 2024. Usable XAI: 10 strategies towards exploiting explainability in the LLM era. arXiv preprint arXiv:2403.08946 (2024)."},{"key":"e_1_2_1_117_1","doi-asserted-by":"publisher","DOI":"10.14778\/3275536.3275541"},{"key":"e_1_2_1_118_1","doi-asserted-by":"publisher","DOI":"10.1609\/hcomp.v4i1.13282"},{"key":"e_1_2_1_119_1","unstructured":"Omar Zaidan Jason Eisner and Christine Piatko. 2007. Using ''annotator rationales'' to improve machine learning for text categorization. In Human language technologies 2007: The conference of the North American chapter of the association for computational linguistics; proceedings of the main conference. 260-267."},{"key":"e_1_2_1_120_1","doi-asserted-by":"publisher","DOI":"10.1145\/3579601"},{"key":"e_1_2_1_121_1","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2017.2677468"},{"key":"e_1_2_1_122_1","doi-asserted-by":"publisher","DOI":"10.1145\/2684822.2685301"},{"key":"e_1_2_1_123_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.tsc.2019.02.006"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3757707","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,16]],"date-time":"2025-10-16T17:50:13Z","timestamp":1760637013000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3757707"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,16]]},"references-count":123,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2025,10,18]]}},"alternative-id":["10.1145\/3757707"],"URL":"https:\/\/doi.org\/10.1145\/3757707","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,10,16]]},"assertion":[{"value":"2025-10-16","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}