{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,17]],"date-time":"2025-10-17T00:31:59Z","timestamp":1760661119211,"version":"build-2065373602"},"reference-count":74,"publisher":"Association for Computing Machinery (ACM)","issue":"7","funder":[{"name":"NUS Department of Communications and New Media"},{"name":"NUS Centre for Trusted Internet and Community"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2025,10,18]]},"abstract":"<jats:p>Supervised machine-learning models often underperform in predicting user behaviors from conversational text, hindered by poor crowdsourced label quality and low NLP task accuracy. We introduce the Metadata-Sensitive Weighted-Encoding Ensemble Model (MSWEEM), which integrates annotator meta-features like fatigue and speeding. First, our results show MSWEEM outperforms standard ensembles by 14% on held-out data and 12% on an alternative dataset. Second, we find that incorporating signals of annotator behavior, such as speed and fatigue, significantly boosts model performance. Third, we find that annotators with higher qualifications, such as Master's, deliver more consistent and faster annotations. Given the increasing uncertainty over annotation quality, our experiments show that understanding annotator patterns is crucial for enhancing model accuracy in user behavior prediction.<\/jats:p>","DOI":"10.1145\/3757484","type":"journal-article","created":{"date-parts":[[2025,10,16]],"date-time":"2025-10-16T17:32:00Z","timestamp":1760635920000},"page":"1-35","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Improving User Behavior Prediction: Leveraging Annotator Metadata in Supervised Machine Learning Models"],"prefix":"10.1145","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2740-7818","authenticated-orcid":false,"given":"Lynnette Hui Xian","family":"Ng","sequence":"first","affiliation":[{"name":"Carnegie Mellon University, Pittsburgh, PA, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8127-1157","authenticated-orcid":false,"given":"Kokil","family":"Jaidka","sequence":"additional","affiliation":[{"name":"NUS Centre for Trusted Internet&amp;Community, National University of Singapore, Singapore, Singapore"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0465-6387","authenticated-orcid":false,"given":"Kai Yuan","family":"Tay","sequence":"additional","affiliation":[{"name":"National University of Singapore, Singapore, Singapore"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3586-7240","authenticated-orcid":false,"given":"Niyati","family":"Chhaya","sequence":"additional","affiliation":[{"name":"Adobe Research, Bangalore, India"}]}],"member":"320","published-online":{"date-parts":[[2025,10,16]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/S15-2045"},{"key":"e_1_2_1_2_1","volume-title":"Inter-coder agreement for computational linguistics. Computational linguistics","author":"Artstein Ron","year":"2008","unstructured":"Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational linguistics, Vol. 34, 4 (2008), 555-596."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1162\/COLI_a_00074"},{"key":"e_1_2_1_4_1","unstructured":"Jason Brownlee. 2018. Better deep learning: train faster reduce overfitting and make better predictions. Machine Learning Mastery."},{"key":"e_1_2_1_5_1","volume-title":"Building a discourse-tagged corpus in the framework of rhetorical structure theory. Current and new directions in discourse and dialogue","author":"Carlson Lynn","year":"2003","unstructured":"Lynn Carlson, Daniel Marcu, and Mary Ellen Okurowski. 2003. Building a discourse-tagged corpus in the framework of rhetorical structure theory. Current and new directions in discourse and dialogue (2003), 85-112."},{"key":"e_1_2_1_6_1","volume-title":"Nonna\u00efvet\u00e9 among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior research methods","author":"Chandler Jesse","year":"2014","unstructured":"Jesse Chandler, Pam Mueller, and Gabriele Paolacci. 2014. Nonna\u00efvet\u00e9 among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior research methods, Vol. 46 (2014), 112-130."},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ins.2016.09.036"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/TR.2022.3156126"},{"key":"e_1_2_1_9_1","volume-title":"Weakly-Supervised Hierarchical Models for Predicting Persuasive Strategies in Good-faith Textual Requests. arXiv preprint arXiv:2101.06351","author":"Chen Jiaao","year":"2021","unstructured":"Jiaao Chen and Diyi Yang. 2021. Weakly-Supervised Hierarchical Models for Predicting Persuasive Strategies in Good-faith Textual Requests. arXiv preprint arXiv:2101.06351 (2021)."},{"key":"e_1_2_1_10_1","volume-title":"A computational approach to politeness with application to social factors. arXiv preprint arXiv:1306.6078","author":"Danescu-Niculescu-Mizil Cristian","year":"2013","unstructured":"Cristian Danescu-Niculescu-Mizil, Moritz Sudhof, Dan Jurafsky, Jure Leskovec, and Christopher Potts. 2013. A computational approach to politeness with application to social factors. arXiv preprint arXiv:1306.6078 (2013)."},{"key":"e_1_2_1_11_1","volume-title":"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805","author":"Devlin Jacob","year":"2018","unstructured":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805 (2018)."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3531146.3534647"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3159652.3159661"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0279720"},{"key":"e_1_2_1_15_1","volume-title":"Increasing cheat robustness of crowdsourcing tasks. Information retrieval","author":"Eickhoff Carsten","year":"2013","unstructured":"Carsten Eickhoff and Arjen P de Vries. 2013. Increasing cheat robustness of crowdsourcing tasks. Information retrieval, Vol. 16 (2013), 121-137."},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.naacl-main.204"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.engappai.2022.105151"},{"key":"e_1_2_1_18_1","volume-title":"Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets. arXiv preprint arXiv:1908.07898","author":"Geva Mor","year":"2019","unstructured":"Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019. Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets. arXiv preprint arXiv:1908.07898 (2019)."},{"key":"e_1_2_1_19_1","first-page":"41","volume-title":"Proceedings of the AAAI Conference on Human Computation and Crowdsourcing","volume":"6","author":"Goyal Tanya","year":"2018","unstructured":"Tanya Goyal, Tyler McDonnell, Mucahid Kutlu, Tamer Elsayed, and Matthew Lease. 2018. Your behavior signals your reliability: Modeling crowd behavioral traces to ensure quality relevance annotations. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 6. 41-49."},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","unstructured":"X. Guo L. Polania B. Zhu C. Boncelet and K. Barner. 2019. Graph neural networks for image understanding based on multiple cues: group emotion recognition and event recognition as use cases. (2019). https:\/\/doi.org\/10.48550\/arxiv.1909.12911","DOI":"10.48550\/arxiv.1909.12911"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174023"},{"key":"e_1_2_1_22_1","first-page":"1","volume-title":"Proceedings of the ACM on Human-Computer Interaction","volume":"7","author":"He Wanrong","year":"2023","unstructured":"Wanrong He, Mitchell L Gordon, Lindsay Popowski, and Michael S Bernstein. 2023. Cura: Curation at Social Media Scale. Proceedings of the ACM on Human-Computer Interaction, Vol. 7, CSCW2 (2023), 1-33."},{"key":"e_1_2_1_23_1","volume-title":"Shaurya Rohatgi, and Ting-Hao'Kenneth' Huang.","author":"He Zeyu","year":"2024","unstructured":"Zeyu He, Chieh-Yang Huang, Chien-Kuang Cornelia Ding, Shaurya Rohatgi, and Ting-Hao'Kenneth' Huang. 2024. If in a Crowdsourced Data Annotation Pipeline, a GPT-4. arXiv preprint arXiv:2402.16795 (2024)."},{"key":"e_1_2_1_24_1","volume-title":"Long short-term memory. Neural computation","author":"Hochreiter Sepp","year":"1997","unstructured":"Sepp Hochreiter and J\u00fcrgen Schmidhuber. 1997. Long short-term memory. Neural computation, Vol. 9, 8 (1997), 1735-1780."},{"key":"e_1_2_1_25_1","volume-title":"Proceedings of the 2013 Conference of the North American","author":"Hovy Dirk","unstructured":"Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. Learning Whom to Trust with MACE. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Lucy Vanderwende, Hal Daum\u00e9 III, and Katrin Kirchhoff (Eds.). Association for Computational Linguistics, Atlanta, Georgia, 1120-1130. https:\/\/aclanthology.org\/N13-1132"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.naacl-main.49"},{"key":"e_1_2_1_27_1","volume-title":"INTERNATIONAL JOURNAL OF TRANSLATION","volume":"22","author":"Hovy Eduard","year":"2010","unstructured":"Eduard Hovy and Julia Lavid. 2010. Towards a 'Science'of Corpus Annotation: A New Methodological Challenge for Corpus Linguistics. INTERNATIONAL JOURNAL OF TRANSLATION, Vol. 22, 1 (2010)."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.3115\/1564131.1564137"},{"key":"e_1_2_1_29_1","first-page":"27 09","volume-title":"Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing. Association for Computational Linguistics","author":"Hsueh Pei-Yun","year":"2009","unstructured":"Pei-Yun Hsueh, Prem Melville, and Vikas Sindhwani. 2009b. Data Quality from Crowdsourcing: A Study of Annotation Selection Criteria. In Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing. Association for Computational Linguistics, Boulder, Colorado, 27-35. https:\/\/www.aclweb.org\/anthology\/W09-1904"},{"key":"e_1_2_1_30_1","unstructured":"Panagiotis G Ipeirotis. 2010. Demographics of mechanical turk. (2010)."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/1837885.1837906"},{"key":"e_1_2_1_32_1","first-page":"1","volume-title":"Proceedings of the ACM on Human-Computer Interaction","volume":"8","author":"Jaidka Kokil","year":"2024","unstructured":"Kokil Jaidka, Hansin Ahuja, and Lynnette Hui Xian Ng. 2024. It Takes Two to Negotiate: Modeling Social Exchange in Online Multiplayer Games. Proceedings of the ACM on Human-Computer Interaction, Vol. 8, CSCW1 (2024), 1-22."},{"key":"e_1_2_1_33_1","volume-title":"Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2191-2200","author":"Jaidka Kokil","year":"2021","unstructured":"Kokil Jaidka, Andrea Ceolin, Iknoor Singh, Niyati Chhaya, and Lyle Ungar. 2021a. WikiTalkEdit: A Dataset for modeling Editors' behaviors on Wikipedia. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2191-2200."},{"key":"e_1_2_1_34_1","volume-title":"Editorial for the 4th AAAI-21 Workshop on Affective Content Analysis.","author":"Jaidka Kokil","year":"2021","unstructured":"Kokil Jaidka, Niyati Chhaya, Lyle Ungar, Jennifer Healey, and Atanu Sinha. 2021b. Editorial for the 4th AAAI-21 Workshop on Affective Content Analysis. (2021)."},{"key":"e_1_2_1_35_1","unstructured":"Kokil Jaidka Iknoor Singh Jiahui Liu Niyati Chhaya and Lyle Ungar. 2020a. A report of the CL-Aff OffMyChest Shared Task: Modeling Supportiveness and Disclosure.. In AffCon@ AAAI. 118-129."},{"key":"e_1_2_1_36_1","volume-title":"Proceedings of the 3rd Workshop on Affective Content Analysis (AffCon 2020) co-located with Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI","author":"Jaidka Kokil","year":"2020","unstructured":"Kokil Jaidka, Iknoor Singh, Jiahui Liu, Niyati Chhaya, and Lyle Ungar. 2020b. A report of the CL-Aff OffMyChest Shared Task: Modeling Supportiveness and Disclosure. In Proceedings of the 3rd Workshop on Affective Content Analysis (AffCon 2020) co-located with Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI 2020), New York, USA, February 7, 2020 (CEUR Workshop Proceedings, Vol. 2614), Niyati Chhaya, Kokil Jaidka, Jennifer Healey, Lyle Ungar, and Atanu R. Sinha (Eds.). CEUR-WS.org, 118-129. http:\/\/ceur-ws.org\/Vol-2614\/AffCon20_session3_claffoverview.pdf"},{"key":"e_1_2_1_37_1","unstructured":"Daniel Kahneman. 1973. Attention and Effort. Prentice-Hall."},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/2818048.2820016"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2063576.2063860"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/D14-1181"},{"key":"e_1_2_1_41_1","volume-title":"The relationship between motivation, monetary compensation, and data quality among US-and India-based workers on Mechanical Turk. Behavior research methods","author":"Litman Leib","year":"2015","unstructured":"Leib Litman, Jonathan Robinson, and Cheskie Rosenzweig. 2015. The relationship between motivation, monetary compensation, and data quality among US-and India-based workers on Mechanical Turk. Behavior research methods, Vol. 47, 2 (2015), 519-528."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D15-1168"},{"key":"e_1_2_1_43_1","volume-title":"RoBERTa: A Robustly Optimized BERT Pretraining Approach. CoRR","author":"Liu Yinhan","year":"2019","unstructured":"Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. CoRR, Vol. abs\/1907.11692 (2019). arXiv:1907.11692 http:\/\/arxiv.org\/abs\/1907.11692"},{"key":"e_1_2_1_44_1","volume-title":"Your Cursor Reveals: On Analyzing Workers' Browsing Behavior and Annotation Quality In Crowdsourcing Tasks","author":"Lo Pei-Chi","year":"2022","unstructured":"Pei-Chi Lo and Ee-Peng Lim. 2022. Your Cursor Reveals: On Analyzing Workers' Browsing Behavior and Annotation Quality In Crowdsourcing Tasks. IEEE Access (2022)."},{"key":"e_1_2_1_45_1","unstructured":"Ismini Lourentzou. 2019. Data quality in the deep learning ERA: Active semi-supervised learning and text normalization for natural language understanding. (2019)."},{"key":"e_1_2_1_46_1","first-page":"1","volume-title":"Proceedings of the ACM on Human-Computer Interaction","volume":"7","author":"Narimanzadeh Hasti","year":"2023","unstructured":"Hasti Narimanzadeh, Arash Badie-Modiri, Iuliia G Smirnova, and Ted Hsuan Yun Chen. 2023. Crowdsourcing subjective annotations using pairwise comparisons reduces bias and error compared to the majority-vote method. Proceedings of the ACM on Human-Computer Interaction, Vol. 7, CSCW2 (2023), 1-29."},{"key":"e_1_2_1_47_1","first-page":"686","volume-title":"Proceedings of the International AAAI Conference on Web and Social Media","volume":"17","author":"Xian Ng Lynnette Hui","year":"2023","unstructured":"Lynnette Hui Xian Ng and Kathleen M Carley. 2023. Botbuster: Multi-platform bot detection using a mixture of experts. In Proceedings of the International AAAI Conference on Web and Social Media, Vol. 17. 686-697."},{"key":"e_1_2_1_48_1","volume-title":"Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 1650-1659","author":"Niculae Vlad","year":"2015","unstructured":"Vlad Niculae, Srijan Kumar, Jordan Boyd-Graber, and Cristian Danescu-Niculescu-Mizil. 2015. Linguistic Harbingers of Betrayal: A Case Study on an Online Strategy Game. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 1650-1659."},{"key":"e_1_2_1_49_1","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 1642-1652","author":"Parikh Pulkit","year":"2019","unstructured":"Pulkit Parikh, Harika Abburi, Pinkesh Badjatiya, Radhika Krishnan, Niyati Chhaya, Manish Gupta, and Vasudeva Varma. 2019. Multi-label Categorization of Accounts of Sexism using a Neural Framework. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 1642-1652."},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00185"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00040"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.engappai.2016.01.012"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.353"},{"key":"e_1_2_1_54_1","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 3811-3854","author":"Peskov Denis","year":"2020","unstructured":"Denis Peskov, Benny Cheng, Ahmed Elgohary, Joe Barrow, Cristian Danescu-Niculescu-Mizil, and Jordan Boyd-Graber. 2020b. It Takes Two to Lie: One to Lie, and One to Listen. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 3811-3854."},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v32i1.11506"},{"key":"e_1_2_1_56_1","volume-title":"2018 13th International Conference on Computer Engineering and Systems (ICCES). IEEE, 633-639","author":"Saeed Nagwa MK","year":"2018","unstructured":"Nagwa MK Saeed, Nivin A Helal, Nagwa L Badr, and Tarek F Gharib. 2018. The impact of spam reviews on feature-based sentiment analysis. In 2018 13th International Conference on Computer Engineering and Systems (ICCES). IEEE, 633-639."},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jksuci.2019.10.002"},{"key":"e_1_2_1_58_1","volume-title":"a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108","author":"Sanh Victor","year":"2019","unstructured":"Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 (2019)."},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-90238-4_12"},{"key":"e_1_2_1_60_1","volume-title":"Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems. 21-33","author":"Sedoc Jo","year":"2020","unstructured":"Jo ao Sedoc and Lyle Ungar. 2020. Item Response Theory for Efficient Human Evaluation of Chatbots. In Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems. 21-33."},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.procs.2022.12.023"},{"key":"e_1_2_1_62_1","volume-title":"Domain-weighted majority voting for crowdsourcing","author":"Tao Dapeng","year":"2018","unstructured":"Dapeng Tao, Jun Cheng, Zhengtao Yu, Kun Yue, and Lizhen Wang. 2018. Domain-weighted majority voting for crowdsourcing. IEEE transactions on neural networks and learning systems, Vol. 30, 1 (2018), 163-174."},{"key":"e_1_2_1_63_1","volume-title":"Knowledge & Information Systems","volume":"62","author":"Tao Fangna","year":"2020","unstructured":"Fangna Tao, Liangxiao Jiang, and Chaoqun Li. 2020. Label similarity-based weighted soft majority voting and pairing for crowdsourcing. Knowledge & Information Systems, Vol. 62, 7 (2020)."},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.5555\/3137854.3138127"},{"key":"e_1_2_1_65_1","unstructured":"Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N. Gomez Lukasz Kaiser and Illia Polosukhin. 2017. Attention is All You Need. https:\/\/arxiv.org\/pdf\/1706.03762.pdf"},{"key":"e_1_2_1_66_1","volume-title":"Auditing Counterfire: Evaluating Advanced Counterargument Generation with Evidence and Style. arXiv preprint arXiv:2402.08498","author":"Verma Preetika","year":"2024","unstructured":"Preetika Verma, Kokil Jaidka, and Svetlana Churina. 2024. Auditing Counterfire: Evaluating Advanced Counterargument Generation with Evidence and Style. arXiv preprint arXiv:2402.08498 (2024)."},{"key":"e_1_2_1_67_1","volume-title":"Manoel Horta Ribeiro, and Robert West","author":"Veselovsky Veniamin","year":"2023","unstructured":"Veniamin Veselovsky, Manoel Horta Ribeiro, and Robert West. 2023. Artificial artificial artificial intelligence: Crowd workers widely use large language models for text production tasks. arXiv preprint arXiv:2306.07899 (2023)."},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W16-5618"},{"key":"e_1_2_1_69_1","volume-title":"Proceedings of the 3rd Workshop on Affective Content Analysis (AffCon 2020) co-located with Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI","author":"Xin Weizhao","year":"2020","unstructured":"Weizhao Xin and Diana Inkpen. 2020. [CL-Aff Shared Task] Detecting Disclosure and Support via Deep Multi-Task Learning. In Proceedings of the 3rd Workshop on Affective Content Analysis (AffCon 2020) co-located with Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI 2020), New York, USA, February 7, 2020 (CEUR Workshop Proceedings, Vol. 2614). CEUR-WS.org, 118-129. http:\/\/ceur-ws.org\/Vol-2614\/AffCon20_session3_detecting.pdf"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0127281"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N19-1364"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10462-022-10283-5"},{"key":"e_1_2_1_73_1","volume-title":"Le","author":"Yang Zhilin","year":"2019","unstructured":"Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019b. XLNet: Generalized Autoregressive Pretraining for Language Understanding. CoRR, Vol. abs\/1906.08237 (2019). arXiv:1906.08237 http:\/\/arxiv.org\/abs\/1906.08237"},{"key":"e_1_2_1_74_1","volume-title":"Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC","author":"Zimmerman Steven","year":"2018","unstructured":"Steven Zimmerman, Udo Kruschwitz, and Chris Fox. 2018. Improving hate speech detection with deep learning ensembles. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)."}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3757484","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,16]],"date-time":"2025-10-16T17:45:52Z","timestamp":1760636752000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3757484"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,16]]},"references-count":74,"journal-issue":{"issue":"7","published-print":{"date-parts":[[2025,10,18]]}},"alternative-id":["10.1145\/3757484"],"URL":"https:\/\/doi.org\/10.1145\/3757484","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,10,16]]},"assertion":[{"value":"2025-10-16","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}