{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,23]],"date-time":"2026-09-23T11:33:52Z","timestamp":1790163232337,"version":"4.1.0"},"reference-count":83,"publisher":"Association for Computing Machinery (ACM)","issue":"CSCW2","license":[{"start":{"date-parts":[[2022,11,7]],"date-time":"2022-11-07T00:00:00Z","timestamp":1667779200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["IIS-1911230"],"award-info":[{"award-number":["IIS-1911230"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000006","name":"Office of Naval Research","doi-asserted-by":"publisher","award":["N00014-19-1-2554, N00174-19-1-0024"],"award-info":[{"award-number":["N00014-19-1-2554, N00174-19-1-0024"]}],"id":[{"id":"10.13039\/100000006","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Hum.-Comput. Interact."],"published-print":{"date-parts":[[2022,11,7]]},"abstract":"<jats:p>As machine learning methods become more powerful and capture more nuances of human behavior, biases in the dataset can shape what the model learns and is evaluated on. This paper explores and attempts to quantify the uncertainties and biases due to annotator demographics when creating sentiment analysis datasets. We ask &gt;1000 crowdworkers to provide their demographic information and annotations for multimodal sentiment data and its component modalities. We show that demographic differences among annotators impute a significant effect on their ratings, and that these effects also occur in each component modality. We compare predictions of different state-of-the-art multimodal machine learning algorithms against annotations provided by different demographic groups, and find that changing annotator demographics can cause &gt;4.5 in accuracy difference when determining positive versus negative sentiment. Our findings underscore the importance of accounting for crowdworker attributes, such as demographics, when building datasets, evaluating algorithms, and interpreting results for sentiment analysis.<\/jats:p>","DOI":"10.1145\/3555632","type":"journal-article","created":{"date-parts":[[2022,11,11]],"date-time":"2022-11-11T17:59:06Z","timestamp":1668189546000},"page":"1-22","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":21,"title":["Impact of Annotator Demographics on Sentiment Dataset Labeling"],"prefix":"10.1145","volume":"6","author":[{"given":"Yi","family":"Ding","sequence":"first","affiliation":[{"name":"Georgia State University, Atlanta, GA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jacob","family":"You","sequence":"additional","affiliation":[{"name":"Westlake High School, Thousand Oaks, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tonja-Katrin","family":"Machulla","sequence":"additional","affiliation":[{"name":"TU Dortmund, Dortmund, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jennifer","family":"Jacobs","sequence":"additional","affiliation":[{"name":"University of California, Santa Barbara, Santa Barbara, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pradeep","family":"Sen","sequence":"additional","affiliation":[{"name":"University of California, Santa Barbara, Santa Barbara, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tobias","family":"H\u00f6llerer","sequence":"additional","affiliation":[{"name":"University of California, Santa Barbara, Santa Barbara, CA, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2022,11,11]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1468-5584.2004.00264.x"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.alw-1.21"},{"key":"e_1_2_1_3_1","volume-title":"Gender and job status as contextual cues for the interpretation of facial expression of emotion. Sex roles","author":"Algoe Sara B","year":"2000","unstructured":"Sara B Algoe , Brenda N Buswell , and John D DeLamater . 2000. Gender and job status as contextual cues for the interpretation of facial expression of emotion. Sex roles , Vol. 42 , 3 ( 2000 ), 183--208. Sara B Algoe, Brenda N Buswell, and John D DeLamater. 2000. Gender and job status as contextual cues for the interpretation of facial expression of emotion. Sex roles, Vol. 42, 3 (2000), 183--208."},{"key":"e_1_2_1_4_1","volume-title":"Multimodal machine learning: A survey and taxonomy","author":"Tadas Baltruvs","year":"2018","unstructured":"Tadas Baltruvs aitis, Chaitanya Ahuja , and Louis-Philippe Morency . 2018. Multimodal machine learning: A survey and taxonomy . IEEE transactions on pattern analysis and machine intelligence, Vol. 41 , 2 ( 2018 ), 423--443. Tadas Baltruvs aitis, Chaitanya Ahuja, and Louis-Philippe Morency. 2018. Multimodal machine learning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence, Vol. 41, 2 (2018), 423--443."},{"key":"e_1_2_1_5_1","volume-title":"Race after technology: Abolitionist tools for the new jim code. Social forces","author":"Benjamin Ruha","year":"2019","unstructured":"Ruha Benjamin . 2019. Race after technology: Abolitionist tools for the new jim code. Social forces ( 2019 ). Ruha Benjamin. 2019. Race after technology: Abolitionist tools for the new jim code. Social forces (2019)."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3415164"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1540-4560.1997.tb02448.x"},{"key":"e_1_2_1_8_1","volume-title":"IEMOCAP: Interactive emotional dyadic motion capture database. Language resources and evaluation","author":"Busso Carlos","year":"2008","unstructured":"Carlos Busso , Murtaza Bulut , Chi-Chun Lee , Abe Kazemzadeh , Emily Mower , Samuel Kim , Jeannette N Chang , Sungbok Lee , and Shrikanth S Narayanan . 2008 . IEMOCAP: Interactive emotional dyadic motion capture database. Language resources and evaluation , Vol. 42 , 4 (2008), 335--359. Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. IEMOCAP: Interactive emotional dyadic motion capture database. Language resources and evaluation, Vol. 42, 4 (2008), 335--359."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3476076"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359164"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-1418"},{"key":"e_1_2_1_12_1","volume-title":"Sex differences: A study of the eye of the beholder. Child development","author":"Condry John","year":"1976","unstructured":"John Condry and Sandra Condry . 1976. Sex differences: A study of the eye of the beholder. Child development ( 1976 ), 812--819. John Condry and Sandra Condry. 1976. Sex differences: A study of the eye of the beholder. Child development (1976), 812--819."},{"key":"e_1_2_1_13_1","volume-title":"Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860","author":"Dai Zihang","year":"2019","unstructured":"Zihang Dai , Zhilin Yang , Yiming Yang , Jaime Carbonell , Quoc V Le , and Ruslan Salakhutdinov . 2019 . Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860 (2019). Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860 (2019)."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1080\/00207594.2011.626043"},{"key":"e_1_2_1_15_1","volume-title":"GoEmotions: A Dataset of Fine-Grained Emotions. (July","author":"Demszky Dorottya","year":"2020","unstructured":"Dorottya Demszky , Dana Movshovitz-Attias , Jeongwoo Ko , Alan Cowen , Gaurav Nemade , and Sujith Ravi . 2020. GoEmotions: A Dataset of Fine-Grained Emotions. (July 2020 ), 4040--4054. Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020. GoEmotions: A Dataset of Fine-Grained Emotions. (July 2020), 4040--4054."},{"key":"e_1_2_1_16_1","volume-title":"Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805","author":"Devlin Jacob","year":"2018","unstructured":"Jacob Devlin , Ming-Wei Chang , Kenton Lee , and Kristina Toutanova . 2018 . Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018). Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3173986"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/3159652.3159661"},{"key":"e_1_2_1_19_1","volume-title":"Multimodal Classification of EEG During Physical Activity. In 2019 International Conference on Multimodal Interaction. 185--194","author":"Ding Yi","year":"2019","unstructured":"Yi Ding , Brandon Huynh , Aiwen Xu , Tom Bullock , Hubert Cecotti , Matthew Turk , Barry Giesbrecht , and Tobias H\u00f6llerer . 2019 . Multimodal Classification of EEG During Physical Activity. In 2019 International Conference on Multimodal Interaction. 185--194 . Yi Ding, Brandon Huynh, Aiwen Xu, Tom Bullock, Hubert Cecotti, Matthew Turk, Barry Giesbrecht, and Tobias H\u00f6llerer. 2019. Multimodal Classification of EEG During Physical Activity. In 2019 International Conference on Multimodal Interaction. 185--194."},{"key":"e_1_2_1_20_1","volume-title":"Sparse Fusion for Multimodal Transformers. arXiv preprint arXiv:2111.11992","author":"Ding Yi","year":"2021","unstructured":"Yi Ding , Alex Rich , Mason Wang , Noah Stier , Pradeep Sen , Matthew Turk , and Tobias H\u00f6llerer . 2021. Sparse Fusion for Multimodal Transformers. arXiv preprint arXiv:2111.11992 ( 2021 ). Yi Ding, Alex Rich, Mason Wang, Noah Stier, Pradeep Sen, Matthew Turk, and Tobias H\u00f6llerer. 2021. Sparse Fusion for Multimodal Transformers. arXiv preprint arXiv:2111.11992 (2021)."},{"key":"e_1_2_1_21_1","volume-title":"An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. CoRR","author":"Dosovitskiy Alexey","year":"1929","unstructured":"Alexey Dosovitskiy , Lucas Beyer , Alexander Kolesnikov , Dirk Weissenborn , Xiaohua Zhai , Thomas Unterthiner , Mostafa Dehghani , Matthias Minderer , Georg Heigold , Sylvain Gelly , Jakob Uszkoreit , and Neil Houlsby . 2020. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. CoRR , Vol. abs\/ 2010 .1 1929 (2020). Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2020. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. CoRR, Vol. abs\/2010.11929 (2020)."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1162\/rest.89.1.100"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1037\/1528-3542.4.1.87"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3130914"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/3119930"},{"key":"e_1_2_1_26_1","volume-title":"Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets. arXiv preprint arXiv:1908.07898","author":"Geva Mor","year":"2019","unstructured":"Mor Geva , Yoav Goldberg , and Jonathan Berant . 2019. Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets. arXiv preprint arXiv:1908.07898 ( 2019 ). Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019. Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets. arXiv preprint arXiv:1908.07898 (2019)."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1382"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.3389\/fdgth.2021.645232"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445423"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300830"},{"key":"e_1_2_1_31_1","unstructured":"Jeff Howe et al. 2006. The rise of crowdsourcing. Wired magazine Vol. 14 6 (2006) 1--4.  Jeff Howe et al. 2006. The rise of crowdsourcing. Wired magazine Vol. 14 6 (2006) 1--4."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3290605.3300637"},{"key":"e_1_2_1_33_1","unstructured":"David R Karger Sewoong Oh and Devavrat Shah. 2011. Iterative learning for reliable crowdsourcing systems. Neural Information Processing Systems.  David R Karger Sewoong Oh and Devavrat Shah. 2011. Iterative learning for reliable crowdsourcing systems. Neural Information Processing Systems."},{"key":"e_1_2_1_34_1","volume-title":"Power, approach, and inhibition. Psychological review","author":"Keltner Dacher","year":"2003","unstructured":"Dacher Keltner , Deborah H Gruenfeld , and Cameron Anderson . 2003. Power, approach, and inhibition. Psychological review , Vol. 110 , 2 ( 2003 ), 265. Dacher Keltner, Deborah H Gruenfeld, and Cameron Anderson. 2003. Power, approach, and inhibition. Psychological review, Vol. 110, 2 (2003), 265."},{"key":"e_1_2_1_35_1","volume-title":"Leigh Yeh, Ali Omrani, Yehsong Kim, Kris Coombs, Shreya Havaldar, Gwenyth Portillo-Wightman, Elaine Gonzalez, et al.","author":"Kennedy Brendan","year":"2018","unstructured":"Brendan Kennedy , Mohammad Atari , Aida Mostafazadeh Davani , Leigh Yeh, Ali Omrani, Yehsong Kim, Kris Coombs, Shreya Havaldar, Gwenyth Portillo-Wightman, Elaine Gonzalez, et al. 2018 . The Gab Hate Corpus : A collection of 27k posts annotated for hate speech. (2018). Brendan Kennedy, Mohammad Atari, Aida Mostafazadeh Davani, Leigh Yeh, Ali Omrani, Yehsong Kim, Kris Coombs, Shreya Havaldar, Gwenyth Portillo-Wightman, Elaine Gonzalez, et al. 2018. The Gab Hate Corpus: A collection of 27k posts annotated for hate speech. (2018)."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3461702.3462609"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1080\/17439760903271025"},{"key":"e_1_2_1_38_1","volume-title":"Deap: A database for emotion analysis","author":"Koelstra Sander","year":"2011","unstructured":"Sander Koelstra , Christian Muhl , Mohammad Soleymani , Jong-Seok Lee , Ashkan Yazdani , Touradj Ebrahimi , Thierry Pun , Anton Nijholt , and Ioannis Patras . 2011 . Deap: A database for emotion analysis ; using physiological signals. IEEE transactions on affective computing, Vol. 3 , 1 (2011), 18--31. Sander Koelstra, Christian Muhl, Mohammad Soleymani, Jong-Seok Lee, Ashkan Yazdani, Touradj Ebrahimi, Thierry Pun, Anton Nijholt, and Ioannis Patras. 2011. Deap: A database for emotion analysis; using physiological signals. IEEE transactions on affective computing, Vol. 3, 1 (2011), 18--31."},{"key":"e_1_2_1_39_1","unstructured":"Jean Kossaifi Robert Walecki Yannis Panagakis Jie Shen Maximilian Schmitt Fabien Ringeval Jing Han Vedhas Pandit Antoine Toisoul Bjoern W Schuller etal 2019. Sewa db: A rich database for audio-visual emotion and sentiment research in the wild. IEEE transactions on pattern analysis and machine intelligence (2019).  Jean Kossaifi Robert Walecki Yannis Panagakis Jie Shen Maximilian Schmitt Fabien Ringeval Jing Han Vedhas Pandit Antoine Toisoul Bjoern W Schuller et al. 2019. Sewa db: A rich database for audio-visual emotion and sentiment research in the wild. IEEE transactions on pattern analysis and machine intelligence (2019)."},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.socialnlp-1.7"},{"key":"e_1_2_1_41_1","volume-title":"First Workshop and Grand Challenge on Computational Modeling of Human Multimodal Language.","author":"Liang Paul Pu","year":"2018","unstructured":"Paul Pu Liang , Ruslan Salakhutdinov , and Louis-Philippe Morency . 2018 . Computational modeling of human multimodal language: The mosei dataset and interpretable dynamic fusion . In First Workshop and Grand Challenge on Computational Modeling of Human Multimodal Language. Paul Pu Liang, Ruslan Salakhutdinov, and Louis-Philippe Morency. 2018. Computational modeling of human multimodal language: The mosei dataset and interpretable dynamic fusion. In First Workshop and Grand Challenge on Computational Modeling of Human Multimodal Language."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.paid.2020.109976"},{"key":"e_1_2_1_43_1","volume-title":"Conducting behavioral research on Amazon's Mechanical Turk. Behavior research methods","author":"Mason Winter","year":"2012","unstructured":"Winter Mason and Siddharth Suri . 2012. Conducting behavioral research on Amazon's Mechanical Turk. Behavior research methods , Vol. 44 , 1 ( 2012 ), 1--23. Winter Mason and Siddharth Suri. 2012. Conducting behavioral research on Amazon's Mechanical Turk. Behavior research methods, Vol. 44, 1 (2012), 1--23."},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/1600150.1600175"},{"key":"e_1_2_1_45_1","volume-title":"The specificity of age-related decline in interpretation of emotion cues from prosody. Psychology and aging","author":"Mitchell Rachel LC","year":"2011","unstructured":"Rachel LC Mitchell , Rachel A Kingston , and Sofia L Barbosa Boucc as. 2011. The specificity of age-related decline in interpretation of emotion cues from prosody. Psychology and aging , Vol. 26 , 2 ( 2011 ), 406. Rachel LC Mitchell, Rachel A Kingston, and Sofia L Barbosa Boucc as. 2011. The specificity of age-related decline in interpretation of emotion cues from prosody. Psychology and aging, Vol. 26, 2 (2011), 406."},{"key":"e_1_2_1_46_1","doi-asserted-by":"crossref","unstructured":"Louis-Philippe Morency Rada Mihalcea and Payal Doshi. 2011. Towards Multimodal Sentiment Analysis: Harvesting Opinions from The Web. Alicante Spain.  Louis-Philippe Morency Rada Mihalcea and Payal Doshi. 2011. Towards Multimodal Sentiment Analysis: Harvesting Opinions from The Web. Alicante Spain.","DOI":"10.1145\/2070481.2070509"},{"key":"e_1_2_1_47_1","volume-title":"Attention Bottlenecks for Multimodal Fusion. arXiv preprint arXiv:2107.00135","author":"Nagrani Arsha","year":"2021","unstructured":"Arsha Nagrani , Shan Yang , Anurag Arnab , Aren Jansen , Cordelia Schmid , and Chen Sun . 2021. Attention Bottlenecks for Multimodal Fusion. arXiv preprint arXiv:2107.00135 ( 2021 ). Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun. 2021. Attention Bottlenecks for Multimodal Fusion. arXiv preprint arXiv:2107.00135 (2021)."},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.00793"},{"key":"e_1_2_1_49_1","volume-title":"Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 973--982","author":"P\u00e9rez-Rosas Ver\u00f3nica","year":"2013","unstructured":"Ver\u00f3nica P\u00e9rez-Rosas , Rada Mihalcea , and Louis-Philippe Morency . 2013 . Utterance-level multimodal sentiment analysis . In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 973--982 . Ver\u00f3nica P\u00e9rez-Rosas, Rada Mihalcea, and Louis-Philippe Morency. 2013. Utterance-level multimodal sentiment analysis. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 973--982."},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.1471-6402.2000.tb01024.x"},{"key":"e_1_2_1_51_1","volume-title":"The influence of gender and social role on the interpretation of facial expressions. Sex roles","author":"Plant E Ashby","year":"2004","unstructured":"E Ashby Plant , Kristen C Kling , and Ginny L Smith . 2004. The influence of gender and social role on the interpretation of facial expressions. Sex roles , Vol. 51 , 3 ( 2004 ), 187--196. E Ashby Plant, Kristen C Kling, and Ginny L Smith. 2004. The influence of gender and social role on the interpretation of facial expressions. Sex roles, Vol. 51, 3 (2004), 187--196."},{"key":"e_1_2_1_52_1","volume-title":"The circumplex model of affect: An integrative approach to affective neuroscience, cognitive development, and psychopathology. Development and psychopathology","author":"Posner Jonathan","year":"2005","unstructured":"Jonathan Posner , James A Russell , and Bradley S Peterson . 2005. The circumplex model of affect: An integrative approach to affective neuroscience, cognitive development, and psychopathology. Development and psychopathology , Vol. 17 , 3 ( 2005 ), 715--734. Jonathan Posner, James A Russell, and Bradley S Peterson. 2005. The circumplex model of affect: An integrative approach to affective neuroscience, cognitive development, and psychopathology. Development and psychopathology, Vol. 17, 3 (2005), 715--734."},{"key":"e_1_2_1_53_1","volume-title":"Aida Mostafazadeh Davani, and Mark D'iaz","author":"Prabhakaran Vinodkumar","year":"2021","unstructured":"Vinodkumar Prabhakaran , Aida Mostafazadeh Davani, and Mark D'iaz . 2021 . On releasing annotator-level labels and information in datasets. arXiv preprint arXiv:2110.05699 (2021). Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, and Mark D'iaz. 2021. On releasing annotator-level labels and information in datasets. arXiv preprint arXiv:2110.05699 (2021)."},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1177\/0956797610371963"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.214"},{"key":"e_1_2_1_56_1","volume-title":"angry, or sad?\": Age-of-poser and age-of-rater effects on multi-dimensional emotion perception. Cognition & emotion","author":"Riediger Michaela","year":"2011","unstructured":"Michaela Riediger , Manuel C Voelkle , Natalie C Ebner , and Ulman Lindenberger . 2011. Beyond ?happy , angry, or sad?\": Age-of-poser and age-of-rater effects on multi-dimensional emotion perception. Cognition & emotion , Vol. 25 , 6 ( 2011 ), 968--982. Michaela Riediger, Manuel C Voelkle, Natalie C Ebner, and Ulman Lindenberger. 2011. Beyond ?happy, angry, or sad?\": Age-of-poser and age-of-rater effects on multi-dimensional emotion perception. Cognition & emotion, Vol. 25, 6 (2011), 968--982."},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.1207\/s15324834basp2003_3"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neubiorev.2008.01.001"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/3449241"},{"key":"e_1_2_1_60_1","volume-title":"Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection. arXiv preprint arXiv:2111.07997","author":"Sap Maarten","year":"2021","unstructured":"Maarten Sap , Swabha Swayamdipta , Laura Vianna , Xuhui Zhou , Yejin Choi , and Noah A Smith . 2021. Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection. arXiv preprint arXiv:2111.07997 ( 2021 ). Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A Smith. 2021. Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection. arXiv preprint arXiv:2111.07997 (2021)."},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274423"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445742"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/1958824.1958865"},{"key":"e_1_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.1037\/0022-3514.49.1.146"},{"key":"e_1_2_1_65_1","volume-title":"Proceedings of the 2008 conference on empirical methods in natural language processing. 254--263","author":"Snow Rion","year":"2008","unstructured":"Rion Snow , Brendan O'connor , Dan Jurafsky , and Andrew Y Ng . 2008 . Cheap and fast--but is it good? evaluating non-expert annotations for natural language tasks . In Proceedings of the 2008 conference on empirical methods in natural language processing. 254--263 . Rion Snow, Brendan O'connor, Dan Jurafsky, and Andrew Y Ng. 2008. Cheap and fast--but is it good? evaluating non-expert annotations for natural language tasks. In Proceedings of the 2008 conference on empirical methods in natural language processing. 254--263."},{"key":"e_1_2_1_66_1","volume-title":"Proceedings of the conference. Association for Computational Linguistics. Meeting","volume":"2019","author":"Hubert Tsai Yao-Hung","year":"2019","unstructured":"Yao-Hung Hubert Tsai , Shaojie Bai , Paul Pu Liang , J Zico Kolter , Louis-Philippe Morency , and Ruslan Salakhutdinov . 2019 . Multimodal transformer for unaligned multimodal language sequences . In Proceedings of the conference. Association for Computational Linguistics. Meeting , Vol. 2019 . NIH Public Access, 6558. Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the conference. Association for Computational Linguistics. Meeting, Vol. 2019. NIH Public Access, 6558."},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/3411764.3445291"},{"key":"e_1_2_1_68_1","unstructured":"Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N Gomez \u0141ukasz Kaiser and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems. 5998--6008.  Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N Gomez \u0141ukasz Kaiser and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems. 5998--6008."},{"key":"e_1_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.1145\/3449083"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v33i01.33017216"},{"key":"e_1_2_1_71_1","volume-title":"Bayesian bias mitigation for crowdsourcing. Advances in neural information processing systems","author":"Wauthier Fabian L","year":"2011","unstructured":"Fabian L Wauthier and Michael Jordan . 2011. Bayesian bias mitigation for crowdsourcing. Advances in neural information processing systems , Vol. 24 ( 2011 ), 1800--1808. Fabian L Wauthier and Michael Jordan. 2011. Bayesian bias mitigation for crowdsourcing. Advances in neural information processing systems, Vol. 24 (2011), 1800--1808."},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.alw-1.22"},{"key":"e_1_2_1_73_1","volume-title":"Gender and preschoolers' perception of emotion. Merrill-Palmer Quarterly (1982-)","author":"Widen Sherri C","year":"2002","unstructured":"Sherri C Widen and James A Russell . 2002. Gender and preschoolers' perception of emotion. Merrill-Palmer Quarterly (1982-) ( 2002 ), 248--262. Sherri C Widen and James A Russell. 2002. Gender and preschoolers' perception of emotion. Merrill-Palmer Quarterly (1982-) (2002), 248--262."},{"key":"e_1_2_1_74_1","volume-title":"Linear models and linear mixed effects models in R with linguistic applications. arXiv preprint arXiv:1308.5499","author":"Winter Bodo","year":"2013","unstructured":"Bodo Winter . 2013. Linear models and linear mixed effects models in R with linguistic applications. arXiv preprint arXiv:1308.5499 ( 2013 ). Bodo Winter. 2013. Linear models and linear mixed effects models in R with linguistic applications. arXiv preprint arXiv:1308.5499 (2013)."},{"key":"e_1_2_1_75_1","doi-asserted-by":"publisher","DOI":"10.1145\/3359245"},{"key":"e_1_2_1_76_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-65414-6_35"},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376447"},{"key":"e_1_2_1_78_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W19-3502"},{"key":"e_1_2_1_79_1","doi-asserted-by":"publisher","DOI":"10.1145\/3351095.3375709"},{"key":"e_1_2_1_80_1","volume-title":"Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems","author":"Yang Zhilin","year":"2019","unstructured":"Zhilin Yang , Zihang Dai , Yiming Yang , Jaime Carbonell , Russ R Salakhutdinov , and Quoc V Le . 2019 a. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems , Vol. 32 (2019). Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019a. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, Vol. 32 (2019)."},{"key":"e_1_2_1_81_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.343"},{"key":"e_1_2_1_82_1","volume-title":"German and French. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing","volume":"2020","author":"Zadeh Amir","year":"2020","unstructured":"Amir Zadeh , Yan Sheng Cao , Simon Hessner , Paul Pu Liang , Soujanya Poria , and Louis-Philippe Morency . 2020 . Carnegie Mellon University-MOSEAS: A multimodal language dataset for Spanish, Portuguese , German and French. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing , Vol. 2020 . NIH Public Access , 1801. Amir Zadeh, Yan Sheng Cao, Simon Hessner, Paul Pu Liang, Soujanya Poria, and Louis-Philippe Morency. 2020. Carnegie Mellon University-MOSEAS: A multimodal language dataset for Spanish, Portuguese, German and French. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing, Vol. 2020. NIH Public Access, 1801."},{"key":"e_1_2_1_83_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-1208"}],"container-title":["Proceedings of the ACM on Human-Computer Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3555632","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3555632","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3555632","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T13:49:18Z","timestamp":1750168158000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3555632"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,11,7]]},"references-count":83,"journal-issue":{"issue":"CSCW2","published-print":{"date-parts":[[2022,11,7]]}},"alternative-id":["10.1145\/3555632"],"URL":"https:\/\/doi.org\/10.1145\/3555632","relation":{},"ISSN":["2573-0142"],"issn-type":[{"value":"2573-0142","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,11,7]]},"assertion":[{"value":"2022-11-11","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}