{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,13]],"date-time":"2026-04-13T12:34:34Z","timestamp":1776083674029,"version":"3.50.1"},"reference-count":66,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2016,5,2]],"date-time":"2016-05-02T00:00:00Z","timestamp":1462147200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Intell. Syst. Technol."],"published-print":{"date-parts":[[2016,7,14]]},"abstract":"<jats:p>Unobtrusive recognition of the user's mood is an essential capability for affect-adaptive systems. Mood is a subtle, long-term affective state, often misrecognized even by humans. The challenge to train a machine to recognize it from, for example, a video of the user, is significant, and already begins with the lack of ground truth for supervised learning. Existing affective databases consist mainly of short videos, annotated in terms of expressed emotions rather than mood. In very few cases, we encounter perceived mood annotations, of questionable reliability, however, due to the subjectivity of mood estimation and the small number of coders involved. In this work, we introduce a new database for mood recognition from video. Our database contains 180 long, acted videos, depicting typical daily scenarios, and subtle facial and bodily expressions. The videos cover three visual modalities (face, body, Kinect data), and are annotated in terms of emotions (via G-trace) and mood (via the Self-Assessment Manikin and the AffectButton). To annotate the database exhaustively, we exploit crowdsourcing to reach out to an extensive number of nonexpert coders. We validate the reliability of our crowdsourced annotations by (1) adopting a number of criteria to filter out unreliable coders, and (2) comparing the annotations of a subset of our videos with those collected in a controlled lab setting.<\/jats:p>","DOI":"10.1145\/2897369","type":"journal-article","created":{"date-parts":[[2016,5,3]],"date-time":"2016-05-03T13:08:37Z","timestamp":1462280917000},"page":"1-27","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":11,"title":["Crowdsourcing Empathetic Intelligence"],"prefix":"10.1145","volume":"7","author":[{"given":"Christina","family":"Katsimerou","sequence":"first","affiliation":[{"name":"Delft University of Technology"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Joris","family":"Albeda","sequence":"additional","affiliation":[{"name":"Delft University of Technology"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alina","family":"Huldtgren","sequence":"additional","affiliation":[{"name":"Eindhoven University of Technology"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ingrid","family":"Heynderickx","sequence":"additional","affiliation":[{"name":"Eindhoven University of Technology"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Judith A.","family":"Redi","sequence":"additional","affiliation":[{"name":"Delft University of Technology"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2016,5,2]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/2660114.2660116"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-012-9204-1"},{"key":"e_1_2_1_3_1","unstructured":"O. Alonso C. Marshall and M. Najork. 2014. Crowdsourcing a Subjective Labelling Task: A human-Centered Framework to Ensure Reliable Results. Technical Report. Microsoft Research San Francisco Bay Area. O. Alonso C. Marshall and M. Najork. 2014. Crowdsourcing a Subjective Labelling Task: A human-Centered Framework to Ensure Reliable Results. Technical Report. Microsoft Research San Francisco Bay Area."},{"key":"e_1_2_1_4_1","volume-title":"GEMEP-GEneva Multimodal Emotion Portrayals: A Corpus for the Study of Multimodal Emotional Expressions. Proceedings of LREC","volume":"6","author":"B\u00e4nziger T."},{"key":"e_1_2_1_5_1","volume-title":"Proceedings of LREC.","author":"Batliner A.","year":"2004"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/ACII.2013.9"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2660114.2660115"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1080\/02699930541000057"},{"key":"e_1_2_1_9_1","doi-asserted-by":"crossref","unstructured":"J. Benedetti. 2013. Stanislavski and the Actor: The Method of Physical Action. Routledge New York NY. J. Benedetti. 2013. Stanislavski and the Actor: The Method of Physical Action. Routledge New York NY.","DOI":"10.4324\/9780203821534"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1016\/0005-7916(94)90063-9"},{"key":"e_1_2_1_11_1","doi-asserted-by":"crossref","DOI":"10.1080\/01621459.1993.10594284","article-title":"Approximate inference in generalized linear mixed models","volume":"88","author":"Breslow N. E.","year":"1993","journal-title":"Journal of the American Statistical Association"},{"key":"e_1_2_1_12_1","volume-title":"IEEE International Conference on Affective Computing and Intelligent Interaction.","author":"Broekens J."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10579-008-9076-6"},{"key":"e_1_2_1_14_1","volume-title":"Proceedings of LREC, Marrakech.","author":"Busso C."},{"key":"e_1_2_1_15_1","doi-asserted-by":"crossref","unstructured":"C. Busso and S. Narayanan. 2008b. Scripted Dialogs Versus Improvisation: Lessons Learned About Emotional Elicitation Techniques from the IEMOCAP Database. Interspeech 2008 -- Eurospeech Brisbane Australia. C. Busso and S. Narayanan. 2008b. Scripted Dialogs Versus Improvisation: Lessons Learned About Emotional Elicitation Techniques from the IEMOCAP Database. Interspeech 2008 -- Eurospeech Brisbane Australia.","DOI":"10.21437\/Interspeech.2008-463"},{"key":"e_1_2_1_16_1","volume-title":"LREC Workshop on Multimodal Corpora: Advances in Capturing, Coding and Analyzing Multimodality, Malta.","author":"Caridakis G.","year":"2010"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neunet.2005.03.002"},{"key":"e_1_2_1_18_1","doi-asserted-by":"crossref","unstructured":"R. Cowie etal 2011. Issues in Data Collection. Emotion-Oriented Systems. Springer Berlin 197--212. R. Cowie et al. 2011. Issues in Data Collection. Emotion-Oriented Systems. Springer Berlin 197--212.","DOI":"10.1007\/978-3-642-15184-2_12"},{"key":"e_1_2_1_20_1","unstructured":"R. Cowie and M. Sawey. 2011. GTrace-General Trace Program from Queen's Belfast. R. Cowie and M. Sawey. 2011. GTrace-General Trace Program from Queen's Belfast."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1007\/BF02310555"},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0167-6393(02)00070-5"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-74889-2_43"},{"key":"e_1_2_1_24_1","volume-title":"IEEE International Conference on Affective Computing and Intelligent Interaction","author":"Eckhardt M."},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-011-9181-9"},{"key":"e_1_2_1_26_1","unstructured":"P. Ekman. 1984. Expression and the Nature of Emotion. Approaches to Emotion 3 19--344. P. Ekman. 1984. Expression and the Nature of Emotion. Approaches to Emotion 3 19--344."},{"key":"e_1_2_1_27_1","doi-asserted-by":"crossref","unstructured":"P. Ekman. 1992. An argument for basic emotions. Cognition and Emotion 6.3--4 169--200. P. Ekman. 1992. An argument for basic emotions. Cognition and Emotion 6.3--4 169--200.","DOI":"10.1080\/02699939208411068"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1111\/j.2044-8295.1994.tb02508.x"},{"key":"e_1_2_1_29_1","volume-title":"International Conference on Intelligent Systems and Computing (ISYC\u201906)","author":"Grimm M."},{"key":"e_1_2_1_30_1","volume-title":"IEEE International Conference on Multimedia and Expo, Hannover.","author":"Grimm M."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10919-010-0094-x"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICPR.2006.39"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1080\/19312450709336664"},{"key":"e_1_2_1_34_1","volume-title":"2015 Seventh International Workshop on Quality of Multimedia Experience (QoMEX). IEEE.","author":"Ho\u00dffeld T."},{"key":"e_1_2_1_35_1","first-page":"1","article-title":"The rise of crowdsourcing","volume":"14","author":"Howe J.","year":"2006","journal-title":"Wired Magazine"},{"key":"e_1_2_1_36_1","volume-title":"Proceedings of the NAACL HLT Workshop on Active Learning for Natural Language Processing. Association for Computational Linguistics","author":"Hsueh P.-Y."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-19312-0_2"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/1837885.1837906"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jml.2007.11.007"},{"key":"e_1_2_1_40_1","volume-title":"Human Emotions: A Reader","author":"Jenkins J.","year":"1998"},{"key":"e_1_2_1_41_1","volume-title":"International Conference on User Modelling, Adaptation and Personalization.","author":"Katsimerou C."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/1357054.1357127"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1080\/10413200008404211"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/2464464.2464485"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1016\/0272-7358(90)90075-L"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1109\/T-AFFC.2011.20"},{"key":"e_1_2_1_47_1","doi-asserted-by":"crossref","unstructured":"A. Mehrabian. 1996. Pleasure-arousal-dominance: A general framework for describing and measuring individual differences in temperament. Current Psychology 261--292. A. Mehrabian. 1996. Pleasure-arousal-dominance: A general framework for describing and measuring individual differences in temperament. Current Psychology 261--292.","DOI":"10.1007\/BF02686918"},{"key":"e_1_2_1_48_1","volume-title":"Multimodal Corpora: Advances in Capturing, Coding and Analyzing Multimodality.","author":"Metallinou A.","year":"2010"},{"key":"e_1_2_1_49_1","volume-title":"IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, Shanghai. 1--8.","author":"Metallinou A."},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1109\/T-AFFC.2011.40"},{"key":"e_1_2_1_51_1","volume-title":"Handbook of Affective Computing","author":"Morris R. R."},{"key":"e_1_2_1_52_1","doi-asserted-by":"crossref","unstructured":"M. M\u00fcller. 2007. Dynamic time warping. In Information Retrieval for Music and Motion 69--84. M. M\u00fcller. 2007. Dynamic time warping. In Information Retrieval for Music and Motion 69--84.","DOI":"10.1007\/978-3-540-74048-3_4"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1109\/T-AFFC.2011.9"},{"key":"e_1_2_1_54_1","unstructured":"K. Oatley and J. Jenkins. 1996. Understanding Emotions. Blackwell Publishers Ltd Oxford UK. K. Oatley and J. Jenkins. 1996. Understanding Emotions. Blackwell Publishers Ltd Oxford UK."},{"key":"e_1_2_1_55_1","doi-asserted-by":"crossref","unstructured":"J. Redi and I. Heynderickx. 2012. Image integrity and Aesthetics: Towards a More Encompassing Definition of Visual Quality. SPIE Electronic imaging Symposium\u2014Human Vision and Electronic Imaging XVII. J. Redi and I. Heynderickx. 2012. Image integrity and Aesthetics: Towards a More Encompassing Definition of Visual Quality. SPIE Electronic imaging Symposium\u2014Human Vision and Electronic Imaging XVII.","DOI":"10.1117\/12.911894"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/2660114.2660118"},{"key":"e_1_2_1_57_1","doi-asserted-by":"crossref","unstructured":"L. D. Riek M. F. O\u2019connor and P. Robinson. 2011. Guess what&quest; A game for affective annotation of video using crowd-sourcing. Affective Computing and Intelligent Interaction 277--285. L. D. Riek M. F. O\u2019connor and P. Robinson. 2011. Guess what&quest; A game for affective annotation of video using crowd-sourcing. Affective Computing and Intelligent Interaction 277--285.","DOI":"10.1007\/978-3-642-24600-5_31"},{"key":"e_1_2_1_58_1","doi-asserted-by":"crossref","unstructured":"J. Russell. 1980. A circumplex model of affect. Personality and Social Psychology 1161--1178. J. Russell. 1980. A circumplex model of affect. Personality and Social Psychology 1161--1178.","DOI":"10.1037\/h0077714"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1037\/0033-295X.110.1.145"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/2047196.2047199"},{"key":"e_1_2_1_61_1","doi-asserted-by":"crossref","unstructured":"K. R. Scherer. 2005. What are emotions&quest; And how can they be measured&quest; Social Science Information 44 4 695--729. K. R. Scherer. 2005. What are emotions&quest; And how can they be measured&quest; Social Science Information 44 4 695--729.","DOI":"10.1177\/0539018405058216"},{"key":"e_1_2_1_62_1","volume-title":"Proceedings of the Conference on Empirical Methods in Natural Language Processing.","author":"Snow R."},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/2506364.2506365"},{"key":"e_1_2_1_64_1","unstructured":"J. Surowiecki. 2005. The Wisdom of Crowds. Random House LLC New York NY. J. Surowiecki. 2005. The Wisdom of Crowds. Random House LLC New York NY."},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.1145\/2512530.2512533"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.1177\/10298649020050S107"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2008.52"}],"container-title":["ACM Transactions on Intelligent Systems and Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2897369","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2897369","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:39:02Z","timestamp":1750221542000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2897369"}},"subtitle":["The Case of the Annotation of EMMA Database for Emotion and Mood Recognition"],"short-title":[],"issued":{"date-parts":[[2016,5,2]]},"references-count":66,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2016,7,14]]}},"alternative-id":["10.1145\/2897369"],"URL":"https:\/\/doi.org\/10.1145\/2897369","relation":{},"ISSN":["2157-6904","2157-6912"],"issn-type":[{"value":"2157-6904","type":"print"},{"value":"2157-6912","type":"electronic"}],"subject":[],"published":{"date-parts":[[2016,5,2]]},"assertion":[{"value":"2015-01-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2015-11-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2016-05-02","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}