{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,6]],"date-time":"2026-01-06T13:25:28Z","timestamp":1767705928063,"version":"build-2065373602"},"reference-count":32,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2018,5,11]],"date-time":"2018-05-11T00:00:00Z","timestamp":1525996800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001602","name":"Science Foundation Ireland","doi-asserted-by":"publisher","award":["16\/IFB\/4336","SFI\/12\/RC\/2289"],"award-info":[{"award-number":["16\/IFB\/4336","SFI\/12\/RC\/2289"]}],"id":[{"id":"10.13039\/501100001602","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000780","name":"European Union","doi-asserted-by":"publisher","award":["H2020 - 644632 (MixedEmotions)"],"award-info":[{"award-number":["H2020 - 644632 (MixedEmotions)"]}],"id":[{"id":"10.13039\/501100000780","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>While the recognition of positive\/negative sentiment in text is an established task with many standard data sets and well developed methodologies, the recognition of a more nuanced affect has received less attention: there are few publicly available annotated resources and there are a number of competing emotion representation schemes with as yet no clear approach to choose between them. To address this lack, we present a series of emotion annotation studies on tweets, providing methods for comparisons between annotation methods (relative vs. absolute) and between different representation schemes. We find improved annotator agreement with a relative annotation scheme (comparisons) on a dimensional emotion model over a categorical annotation scheme on Ekman\u2019s six basic emotions; however, when we compare inter-annotator agreement for comparisons with agreement for a rating scale annotation scheme (both with the same dimensional emotion model), we find improved inter-annotator agreement with rating scales, challenging a common belief that relative judgements are more reliable. To support these studies and as a contribution in itself, we further present a publicly available collection of 2019 tweets annotated with scores on each of four emotion dimensions: valence, arousal, dominance and surprise, following the emotion representation model identified by Fontaine et al. in 2007.<\/jats:p>","DOI":"10.3390\/info9050117","type":"journal-article","created":{"date-parts":[[2018,5,14]],"date-time":"2018-05-14T02:57:20Z","timestamp":1526266640000},"page":"117","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":10,"title":["A Comparison of Emotion Annotation Approaches for Text"],"prefix":"10.3390","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6094-0358","authenticated-orcid":false,"given":"Ian D.","family":"Wood","sequence":"first","affiliation":[{"name":"Insight Centre for Data Analytics, National University of Ireland Galway, Galway H91 AEX4, Ireland"},{"name":"Aylien Ltd., Dublin D02 RH68, Irenland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"John P.","family":"McCrae","sequence":"additional","affiliation":[{"name":"Insight Centre for Data Analytics, National University of Ireland Galway, Galway H91 AEX4, Ireland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Vladimir","family":"Andryushechkin","sequence":"additional","affiliation":[{"name":"Insight Centre for Data Analytics, National University of Ireland Galway, Galway H91 AEX4, Ireland"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Paul","family":"Buitelaar","sequence":"additional","affiliation":[{"name":"Insight Centre for Data Analytics, National University of Ireland Galway, Galway H91 AEX4, Ireland"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2018,5,11]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1561\/1500000011","article-title":"Opinion Mining and Sentiment Analysis","volume":"2","author":"Pang","year":"2008","journal-title":"Found. Trends Inf. Retr."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Meiselman, H.L. (2016). Sentiment Analysis: Detecting Valence, Emotions, and Other Affectual States from Text. Emotion Measurement, Woodhead Publishing.","DOI":"10.1016\/B978-0-08-100508-8.00026-6"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Mohammad, S., and Bravo-Marquez, F. (2017, January 8). WASSA-2017 Shared Task on Emotion Intensity. Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, Copenhagen, Denmark.","DOI":"10.18653\/v1\/W17-5205"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Mohammad, S.M., Bravo-Marquez, F., Salameh, M., and Kiritchenko, S. (2018, January 5\u20136). SemEval-2018 Task 1: Affect in Tweets. Proceedings of the International Workshop on Semantic Evaluation (SemEval-2018), New Orleans, LA, USA.","DOI":"10.18653\/v1\/S18-1001"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Felbo, B., Mislove, A., S\u00f8gaard, A., Rahwan, I., and Lehmann, S. (2017, January 7\u201311). Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark.","DOI":"10.18653\/v1\/D17-1169"},{"key":"ref_6","first-page":"578","article-title":"EMOBANK: Studying the Impact of Annotation Perspective and Representation Format on Dimensional Emotion Analysis","volume":"2017","author":"Buechel","year":"2017","journal-title":"EACL"},{"key":"ref_7","unstructured":"Abdul-Mageed, M., and Ungar, L. (August, January 30). EmoNet: Fine-Grained Emotion Detection with Gated Recurrent Neural Networks. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vancouver, BC, Canada."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"712","DOI":"10.1037\/0022-3514.53.4.712","article-title":"Universals and cultural differences in the judgments of facial expressions of emotion","volume":"53","author":"Ekman","year":"1987","journal-title":"J. Pers. Soc. Psychol."},{"key":"ref_9","unstructured":"Osgood, C.E., May, W.H., and Miron, M.S. (1975). Cross-Cultural Universals of Affective Meaning, University of Illinois Press."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"273","DOI":"10.1016\/0092-6566(77)90037-X","article-title":"Evidence for a three-factor theory of emotions","volume":"11","author":"Russell","year":"1977","journal-title":"J. Res. Pers."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"1050","DOI":"10.1111\/j.1467-9280.2007.02024.x","article-title":"The World of Emotions is not Two-Dimensional","volume":"18","author":"Fontaine","year":"2007","journal-title":"Psychol. Sci."},{"key":"ref_12","unstructured":"Bradley, M.M., and Lang, P.J. (2007). Affective Norms for English Text (ANET): Affective Ratings of Texts and Instruction Manual, University of Florida. Technical Report."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Preotiuc-Pietro, D., Schwartz, H.A., Park, G., Eichstaedt, J.C., Kern, M., Ungar, L., and Shulman, E.P. (2016, January 12\u201317). Modelling Valence and Arousal in Facebook posts. Proceedings of the NAACL-HLT, San Diego, CA, USA.","DOI":"10.18653\/v1\/W16-0404"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Yu, L.C., Lee, L.H., Hao, S., Wang, J., He, Y., Hu, J., Lai, K.R., and Zhang, X. (2016, January 12\u201317). Building Chinese Affective Resources in Valence-Arousal Dimensions. Proceedings of the NAACL-HLT, San Diego, CA, USA.","DOI":"10.18653\/v1\/N16-1066"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1016\/0005-7916(94)90063-9","article-title":"Measuring Emotion: The self-assessment manikin and the semantic differential","volume":"25","author":"Bradley","year":"1994","journal-title":"J. Behav. Theory Exp. Psychiatry"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Metallinou, A., and Narayanan, S. (2013, January 22\u201326). Annotation and processing of continuous emotional attributes: Challenges and opportunities. Proceedings of the 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), Shanghai, China.","DOI":"10.1109\/FG.2013.6553804"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Yannakakis, G.N., Cowie, R., and Busso, C. (2017, January 23\u201326). The Ordinal Nature of Emotions. Proceedings of the International Conference on Affective Computing and Intelligent Interaction, San Antonio, TX, USA.","DOI":"10.1109\/ACII.2017.8273608"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"762","DOI":"10.1109\/TASL.2010.2064164","article-title":"Ranking-Based Emotion Recognition for Music Organization and Retrieval","volume":"19","author":"Yang","year":"2011","journal-title":"IEEE Trans. Audio Speech Lang. Process."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"314","DOI":"10.1109\/TAFFC.2014.2352268","article-title":"Don\u2019t classify ratings of affect; rank them!","volume":"5","author":"Martinez","year":"2014","journal-title":"IEEE Trans. Affect. Comput."},{"key":"ref_20","unstructured":"Kiritchenko, S., and Mohammad, S.M. (August, January 30). Best-Worst Scaling More Reliable than Rating Scales: A Case Study on Sentiment Intensity Annotation. Proceedings of the 32nd International Conference on Computational Linguistics and the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, BC, Canada."},{"key":"ref_21","unstructured":"Wood, I., and Ruder, S. (2016, January 23). Emoji as Emotion Tags for Tweets. Proceedings of the Emotion and Sentiment Analysis Workshop LREC2016, Portoro\u017e, Slovenia."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"37","DOI":"10.1111\/j.1469-8137.1912.tb05611.x","article-title":"The distribution of the flora in the alpine zone","volume":"11","author":"Jaccard","year":"1912","journal-title":"New Phytol."},{"key":"ref_23","unstructured":"Passonneau, R.J. (2004, January 26\u201331). Computing Reliability for Coreference Annotation. Proceedings of the LREC, Reykjavik, Iceland."},{"key":"ref_24","unstructured":"Passonneau, R. (2006). Measuring Agreement on Set-Valued Items (MASI) for Semantic and Pragmatic Annotation, Columbia University."},{"key":"ref_25","unstructured":"Louviere, J.J., and Woodworth, G. (1991). Best-Worst Scaling: A Model for the Largest Difference Judgments, University of Alberta. Working Paper."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Louviere, J.J., Flynn, T.N., and Marley, A.A. (2015). Best-Worst Scaling: Theory, Methods and Applications, Cambridge University Press.","DOI":"10.1017\/CBO9781107337855"},{"key":"ref_27","unstructured":"Wood, I.D., McCrae, J.P., and Buitelaar, P. (2018, January 7\u201312). A Comparison Of Emotion Annotation Schemes And A New Annotated Data Set. Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan."},{"key":"ref_28","unstructured":"Bollen, J., Mao, H., and Pepe, A. (2011). Modeling Public Mood and Emotion: Twitter Sentiment and Socio-Economic Phenomena, AAAI Press. Icwsm."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"96","DOI":"10.1145\/2818717","article-title":"The Rise of Social Bots","volume":"59","author":"Ferrara","year":"2016","journal-title":"Commun. ACM"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Andryushechkin, V., Wood, I., and O\u2019Neill, J. (2017, January 18). NUIG at EmoInt-2017: BiLSTM and SVR ensemble to detect emotion intensity. Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, Copenhagen, Denmark.","DOI":"10.18653\/v1\/W17-5223"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Pennington, J., Socher, R., and Manning, C.D. (2014, January 25\u201329). GloVe: Global Vectors for Word Representation. Proceedings of the Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.","DOI":"10.3115\/v1\/D14-1162"},{"key":"ref_32","unstructured":"Chopra, S., Hadsell, R., and LeCun, Y. (2005, January 20\u201325). Learning a similarity metric discriminatively, with application to face verification. Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR\u201905), San Diego, CA, USA."}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/9\/5\/117\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T15:03:51Z","timestamp":1760195031000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/9\/5\/117"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,5,11]]},"references-count":32,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2018,5]]}},"alternative-id":["info9050117"],"URL":"https:\/\/doi.org\/10.3390\/info9050117","relation":{},"ISSN":["2078-2489"],"issn-type":[{"type":"electronic","value":"2078-2489"}],"subject":[],"published":{"date-parts":[[2018,5,11]]}}}