{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,1]],"date-time":"2026-05-01T17:16:22Z","timestamp":1777655782904,"version":"3.51.4"},"reference-count":35,"publisher":"MIT Press","license":[{"start":{"date-parts":[[2023,3,16]],"date-time":"2023-03-16T00:00:00Z","timestamp":1678924800000},"content-version":"vor","delay-in-days":74,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,3,14]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>While the link between color and emotion has been widely studied, how context-based changes in color impact the intensity of perceived emotions is not well understood. In this work, we present a new multimodal dataset for exploring the emotional connotation of color as mediated by line, stroke, texture, shape, and language. Our dataset, FeelingBlue, is a collection of 19,788 4-tuples of abstract art ranked by annotators according to their evoked emotions and paired with rationales for those annotations. Using this corpus, we present a baseline for a new task: Justified Affect Transformation. Given an image I, the task is to 1) recolor I to enhance a specified emotion e and 2) provide a textual justification for the change in e. Our model is an ensemble of deep neural networks which takes I, generates an emotionally transformed color palette p conditioned on I, applies p to I, and then justifies the color transformation in text via a visual-linguistic model. Experimental results shed light on the emotional connotation of color in context, demonstrating both the promise of our approach on this challenging task and the considerable potential for future investigations enabled by our corpus.1<\/jats:p>","DOI":"10.1162\/tacl_a_00540","type":"journal-article","created":{"date-parts":[[2023,3,16]],"date-time":"2023-03-16T14:38:17Z","timestamp":1678977497000},"page":"176-190","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":6,"title":["FeelingBlue: A Corpus for Understanding the Emotional Connotation of Color in Context"],"prefix":"10.1162","volume":"11","author":[{"given":"Amith","family":"Ananthram","sequence":"first","affiliation":[{"name":"Department of Computer Science, Columbia University, USA. amith@cs.columbia.edu"}]},{"given":"Olivia","family":"Winn","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Columbia University, USA. olivia@cs.columbia.edu"}]},{"given":"Smaranda","family":"Muresan","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Columbia University, USA"},{"name":"Data Science Institute, Columbia University, USA. smara@cs.columbia.edu"}]}],"member":"281","published-online":{"date-parts":[[2023,3,14]]},"reference":[{"key":"2023031614352494500_","unstructured":"2016. IdeelArt: The online gallerist."},{"key":"2023031614352494500_","doi-asserted-by":"publisher","first-page":"11569","DOI":"10.1109\/CVPR46437.2021.01140","article-title":"Artemis: Affective language for visual art","volume-title":"Proceedings of the IEEE\/ CVF Conference on Computer Vision and Pattern Recognition","author":"Achlioptas","year":"2021"},{"key":"2023031614352494500_","doi-asserted-by":"publisher","first-page":"5240","DOI":"10.1109\/CVPR.2016.566","article-title":"Recognizing emotions from abstract paintings using non-linear matrix completion","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Alameda-Pineda","year":"2016"},{"key":"2023031614352494500_","doi-asserted-by":"publisher","first-page":"431","DOI":"10.1007\/978-3-030-01258-8_27","article-title":"Coloring with words: Guiding image colorization through text-based palette generation","volume-title":"Proceedings of the European Conference on Computer Vision (ECCV)","author":"Bahng","year":"2018"},{"issue":"2","key":"2023031614352494500_","doi-asserted-by":"publisher","first-page":"219","DOI":"10.1207\/s15516709cog2102_4","article-title":"The texture lexicon: Understanding the categorization of visual texture terms and their relationship to texture images","volume":"21","author":"Bhushan","year":"1997","journal-title":"Cognitive Science"},{"issue":"3","key":"2023031614352494500_","doi-asserted-by":"publisher","first-page":"904","DOI":"10.3758\/s13428-013-0403-5","article-title":"Concreteness ratings for 40 thousand generally known English word lemmas","volume":"46","author":"Brysbaert","year":"2014","journal-title":"Behavior Research Methods"},{"key":"2023031614352494500_","doi-asserted-by":"publisher","first-page":"1058","DOI":"10.1109\/CVPRW.2017.143","article-title":"PaletteNet: Image recolorization with given color palette","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops","author":"Cho","year":"2017"},{"key":"2023031614352494500_","doi-asserted-by":"publisher","first-page":"248","DOI":"10.1109\/CVPR.2009.5206848","article-title":"ImageNet: A large-scale hierarchical image database","volume-title":"2009 IEEE Conference on Computer Vision and Pattern Recognition","author":"Deng","year":"2009"},{"key":"2023031614352494500_","doi-asserted-by":"publisher","DOI":"10.4337\/9781781003152.00014","article-title":"Best-Worst Scaling: Theory and methods","volume-title":"Handbook of Choice Modelling","author":"Flynn","year":"2014"},{"key":"2023031614352494500_","first-page":"770","article-title":"Deep residual learning for image recognition","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"He","year":"2016"},{"key":"2023031614352494500_","article-title":"GANs trained by a two time-scale update rule converge to a local Nash Equilibrium","volume":"30","author":"Heusel","year":"2017","journal-title":"Advances in Neural Information Processing Systems"},{"issue":"11","key":"2023031614352494500_","doi-asserted-by":"publisher","first-page":"2980","DOI":"10.1109\/TMM.2018.2827782","article-title":"Building emotional machines: Recognizing image emotions through deep neural networks","volume":"20","author":"Kim","year":"2018","journal-title":"IEEE Transactions on Multimedia"},{"key":"2023031614352494500_","article-title":"Adam: A method for stochastic optimization","volume-title":"3rd International Conference for Learning Representations (ICLR)","author":"Kingma","year":"2015"},{"key":"2023031614352494500_","doi-asserted-by":"publisher","first-page":"811","DOI":"10.18653\/v1\/N16-1095","article-title":"Capturing reliable fine-grained sentiment associations by crowdsourcing and Best-Worst Scaling","volume-title":"NAACL-HLT","author":"Kiritchenko","year":"2016"},{"key":"2023031614352494500_","article-title":"The role of ImageNet classes in Fr\u00e9chet Inception Distance","author":"Kynk\u00e4\u00e4nniemi","year":"2022","journal-title":"arXiv preprint arXiv:2203.06026"},{"key":"2023031614352494500_","doi-asserted-by":"publisher","first-page":"83","DOI":"10.1145\/1873951.1873965","article-title":"Affective image classification using features inspired by psychology and art theory","volume-title":"Proceedings of the 18th ACM International Conference on Multimedia","author":"Machajdik","year":"2010"},{"key":"2023031614352494500_","doi-asserted-by":"publisher","first-page":"105","DOI":"10.1533\/9780857095534.1.105","article-title":"Colour psychology: The emotional effects of colour perception","volume-title":"Colour Design","author":"Mikellides","year":"2012"},{"key":"2023031614352494500_","first-page":"368","article-title":"Even the abstract have color: Consensus in word-colour associations","volume-title":"Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies","author":"Mohammad","year":"2011"},{"key":"2023031614352494500_","doi-asserted-by":"publisher","first-page":"65","DOI":"10.18653\/v1\/S17-1007","article-title":"Emotion intensities in tweets","volume-title":"Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (*SEM 2017)","author":"Mohammad","year":"2017"},{"key":"2023031614352494500_","article-title":"WikiArt Emotions: An annotated dataset of emotions evoked by art","volume-title":"Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)","author":"Mohammad","year":"2018"},{"key":"2023031614352494500_","article-title":"Why links between colors and emotions may be universal","author":"Mohr","year":"2022"},{"key":"2023031614352494500_","article-title":"Color survey results","author":"Monroe","year":"2010"},{"key":"2023031614352494500_","article-title":"Connotative meaning in English and Italian colour-word metaphors","volume":"10","author":"Philip","year":"2006","journal-title":"Metaphorik"},{"key":"2023031614352494500_","first-page":"8748","article-title":"Learning transferable visual models from natural language supervision","volume-title":"International Conference on Machine Learning","author":"Radford","year":"2021"},{"key":"2023031614352494500_","article-title":"Hierarchical text-conditional image generation with CLIP latents","author":"Ramesh","year":"2022","journal-title":"arXiv preprint arXiv:2204.06125"},{"key":"2023031614352494500_","first-page":"1","article-title":"Learning multi-level deep representations for image emotion classification","author":"Rao","year":"2019","journal-title":"Neural Processing Letters"},{"issue":"2","key":"2023031614352494500_","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/2768209","article-title":"Affective analysis of professional and amateur abstract paintings using statistical analysis and art theory","volume":"5","author":"Sartori","year":"2015","journal-title":"ACM Transactions on Interactive Intelligent Systems (TiiS)"},{"issue":"1","key":"2023031614352494500_","first-page":"1929","article-title":"Dropout: A simple way to prevent neural networks from overfitting","volume":"15","author":"Srivastava","year":"2014","journal-title":"The Journal of Machine Learning Research"},{"issue":"2","key":"2023031614352494500_","doi-asserted-by":"publisher","first-page":"686","DOI":"10.3758\/s13428-015-0598-8","article-title":"Color associations to emotion and emotion-laden words: A collection of norms for stimulus construction and selection","volume":"48","author":"Sutton","year":"2016","journal-title":"Behavior Research Methods"},{"key":"2023031614352494500_","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/2346676.2346684","article-title":"Finding emotion in image descriptions","volume-title":"Proceedings of the First International Workshop on Issues of Sentiment Discovery and Opinion Mining","author":"Ulinski","year":"2012"},{"key":"2023031614352494500_","first-page":"292","article-title":"Emo2Vec: Learning generalized emotion representation by multi-task training","volume-title":"Proceedings of the 9th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis","author":"Peng","year":"2018"},{"key":"2023031614352494500_","doi-asserted-by":"publisher","first-page":"56","DOI":"10.1016\/j.neucom.2018.03.078","article-title":"Image captioning by incorporating affective concepts learned from both visual and textual components","volume":"328","author":"Yang","year":"2019","journal-title":"Neurocomputing"},{"key":"2023031614352494500_","doi-asserted-by":"publisher","first-page":"413","DOI":"10.1007\/978-3-642-24800-9_38","article-title":"Analyzing emotional semantics of abstract art using low-level image features","volume-title":"International Symposium on Intelligent Data Analysis","author":"He","year":"2011"},{"issue":"1","key":"2023031614352494500_","doi-asserted-by":"publisher","first-page":"4669","DOI":"10.24963\/ijcai.2017\/651","article-title":"Approximating discrete probability distribution of image emotions by multi-modal features fusion","volume":"1000","author":"Zhao","year":"2017","journal-title":"Transfer"},{"key":"2023031614352494500_","doi-asserted-by":"publisher","first-page":"1319","DOI":"10.1145\/3240508.3240591","article-title":"EmotionGAN: Unsupervised domain adaptation for learning discrete probability distributions of image emotions","volume-title":"Proceedings of the 26th ACM International Conference on Multimedia","author":"Zhao","year":"2018"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00540\/2074909\/tacl_a_00540.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00540\/2074909\/tacl_a_00540.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,3,16]],"date-time":"2023-03-16T14:38:17Z","timestamp":1678977497000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00540\/115241\/FeelingBlue-A-Corpus-for-Understanding-the"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023]]},"references-count":35,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00540","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2023]]},"published":{"date-parts":[[2023]]}}}