{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T04:50:16Z","timestamp":1760244616483,"version":"build-2065373602"},"reference-count":52,"publisher":"MDPI AG","issue":"12","license":[{"start":{"date-parts":[[2022,11,22]],"date-time":"2022-11-22T00:00:00Z","timestamp":1669075200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"FCT\u2014Foundation for Science and Technology","award":["SFRH\/BD\/139775\/2018","SFRH\/BD\/133415\/2017","UIDB\/00326\/2020","UIDP\/00326\/2020"],"award-info":[{"award-number":["SFRH\/BD\/139775\/2018","SFRH\/BD\/133415\/2017","UIDB\/00326\/2020","UIDP\/00326\/2020"]}]},{"name":"national funds (PIDDAC)","award":["SFRH\/BD\/139775\/2018","SFRH\/BD\/133415\/2017","UIDB\/00326\/2020","UIDP\/00326\/2020"],"award-info":[{"award-number":["SFRH\/BD\/139775\/2018","SFRH\/BD\/133415\/2017","UIDB\/00326\/2020","UIDP\/00326\/2020"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"<jats:p>The development of computational artifacts to study cross-modal associations has been a growing research topic, as they allow new degrees of abstraction. In this context, we propose a novel approach to the computational exploration of relationships between music and abstract images, grounded by findings from cognitive sciences (emotion and perception). Due to the problem\u2019s high-level nature, we rely on evolutionary programming techniques to evolve this audio\u2013visual dialogue. To articulate the complexity of the problem, we develop a framework with four modules: (i) vocabulary set, (ii) music generator, (iii) image generator, and (iv) evolutionary engine. We test our approach by evolving a given music set to a corresponding set of images, steered by the expression of four emotions (angry, calm, happy, sad). Then, we perform preliminary user tests to evaluate if the user\u2019s perception is consistent with the system\u2019s expression. Results suggest an agreement between the user\u2019s emotional perception of the music\u2013image pairs and the system outcomes, favoring the integration of cognitive science knowledge. We also discuss the benefit of employing evolutionary strategies, such as genetic programming on multi-modal problems of a creative nature. Overall, this research contributes to a better understanding of the foundations of auditory\u2013visual associations mediated by emotions and perception.<\/jats:p>","DOI":"10.3390\/e24121706","type":"journal-article","created":{"date-parts":[[2022,11,22]],"date-time":"2022-11-22T08:25:28Z","timestamp":1669105528000},"page":"1706","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["\u201cFound in Translation\u201d: An Evolutionary Framework for Auditory\u2013Visual Relationships"],"prefix":"10.3390","volume":"24","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4787-1260","authenticated-orcid":false,"given":"Ana","family":"Rodrigues","sequence":"first","affiliation":[{"name":"Department of Informatics Engineering, Centre for Informatics and Systems of the University of Coimbra, University of Coimbra, 3004-531 Coimbra, Portugal"},{"name":"Intelligent Systems Associate Laboratory (LASI), University of Minho, 4800-058 Guimar\u00e3es, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6611-2468","authenticated-orcid":false,"given":"Bruna","family":"Sousa","sequence":"additional","affiliation":[{"name":"Department of Informatics Engineering, Centre for Informatics and Systems of the University of Coimbra, University of Coimbra, 3004-531 Coimbra, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6916-8811","authenticated-orcid":false,"given":"Am\u00edlcar","family":"Cardoso","sequence":"additional","affiliation":[{"name":"Department of Informatics Engineering, Centre for Informatics and Systems of the University of Coimbra, University of Coimbra, 3004-531 Coimbra, Portugal"},{"name":"Intelligent Systems Associate Laboratory (LASI), University of Minho, 4800-058 Guimar\u00e3es, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6308-6484","authenticated-orcid":false,"given":"Penousal","family":"Machado","sequence":"additional","affiliation":[{"name":"Department of Informatics Engineering, Centre for Informatics and Systems of the University of Coimbra, University of Coimbra, 3004-531 Coimbra, Portugal"},{"name":"Intelligent Systems Associate Laboratory (LASI), University of Minho, 4800-058 Guimar\u00e3es, Portugal"}]}],"member":"1968","published-online":{"date-parts":[[2022,11,22]]},"reference":[{"key":"ref_1","unstructured":"Grill, T., and Flexer, A. (2012, January 9\u201312). Visualization of perceptual qualities in textural sounds. Proceedings of the ICMC, Ljubljana, Slovenia."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Lindborg, P., and Friberg, A.K. (2015). Colour association with music is mediated by emotion: Evidence from an experiment using a CIE Lab interface and interviews. PLoS ONE, 10.","DOI":"10.1371\/journal.pone.0144013"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Moroni, A., Maiolla, R., Manzolli, J., and Zuben, F.V. (2006). ArTVox: Evolutionary Composition in Visual and Sound Domains. Smart Graphics, Springer.","DOI":"10.1007\/11795018_20"},{"key":"ref_4","first-page":"8","article-title":"Transgenic visual-and-sound compositions","volume":"9","author":"Moroni","year":"2014","journal-title":"NICS Rep."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"191","DOI":"10.1016\/B978-0-444-63287-6.00010-5","article-title":"Perception of emotion in abstract artworks: A multidisciplinary approach","volume":"204","author":"Melcher","year":"2013","journal-title":"Prog. Brain Res."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Gabrielsson, A., and Juslin, P.N. (2003). Emotional Expression in Music, Oxford University Press.","DOI":"10.1093\/oso\/9780195126013.003.0026"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Schindler, I., Hosoya, G., Menninghaus, W., Beermann, U., Wagner, V., Eid, M., and Scherer, K.R. (2017). Measuring aesthetic emotions: A review of the literature and a new assessment tool. PLoS ONE, 12.","DOI":"10.1037\/t79669-000"},{"key":"ref_8","first-page":"23","article-title":"Computer Models of Creativity","volume":"30","author":"Boden","year":"2009","journal-title":"AI Mag."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1080\/14626260902867915","article-title":"What is generative art?","volume":"20","author":"Boden","year":"2009","journal-title":"Digit. Creat."},{"key":"ref_10","first-page":"64","article-title":"Computers and the visual arts","volume":"66\u201367","author":"Noll","year":"1966","journal-title":"Des. Q."},{"key":"ref_11","unstructured":"Sims, K. (1992). Interactive evolution of dynamical systems. Toward a Practice of Autonomous Systems: Proceedings of the First European Conference on Artificial Life, MIT Press."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"1161","DOI":"10.1037\/h0077714","article-title":"A circumplex model of affect","volume":"39","author":"Russell","year":"1980","journal-title":"J. Personal. Soc. Psychol."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Koza, J.R., and Poli, R. (2005). Genetic programming. Search Methodologies, Springer.","DOI":"10.1007\/0-387-28356-0_5"},{"key":"ref_14","unstructured":"Arnheim, R. (1965). Art and Visual Perception: A Psychology of the Creative Eye, University of California Press."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Juslin, P.N., and Sloboda, J.A. (2001). Music and emotion. Theory and Research, Elsevier Inc.","DOI":"10.1093\/oso\/9780192631886.001.0001"},{"key":"ref_16","first-page":"29","article-title":"Mary ellen bute: Seeing sound","volume":"1","author":"Moritz","year":"1996","journal-title":"Animat. World"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1162\/014892605775179955","article-title":"Foundations of a visual music","volume":"29","author":"Evans","year":"2005","journal-title":"Comput. Music J."},{"key":"ref_18","first-page":"39","article-title":"On perceptual metaphors","volume":"11","author":"Marks","year":"1996","journal-title":"Metaphor Symb."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"29","DOI":"10.2307\/1575616","article-title":"Toward the automatic generation of visual music","volume":"25","year":"1992","journal-title":"Leonardo"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Collopy, F., Fuhrer, R., and Jameson, D. (1999, January 13\u201316). Visual music in a visual programming language. Proceedings of the IEEE Symposium on Visual Languages, Tokyo, Japan.","DOI":"10.1109\/VL.1999.795882"},{"key":"ref_21","unstructured":"Kandinsky, W., and Rebay, H. (1979). Point and Line to Plane, Courier Corporation."},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"45","DOI":"10.2307\/3681184","article-title":"To paint on water: The audiovisual duet of complementarity","volume":"18","author":"Whitney","year":"1994","journal-title":"Comput. Music J."},{"key":"ref_23","first-page":"367-44","article-title":"The role of structure in the musical expression of emotions","volume":"367400","author":"Gabrielsson","year":"2010","journal-title":"Handb. Music Emot. Theory Res. Appl."},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"235","DOI":"10.1016\/j.plrev.2013.05.008","article-title":"From everyday emotions to aesthetic emotions: Towards a unified theory of musical emotions","volume":"10","author":"Juslin","year":"2013","journal-title":"Phys. Life Rev."},{"key":"ref_25","first-page":"3","article-title":"Synaesthesia\u2013a window into perception, thought and language","volume":"8","author":"Ramachandran","year":"2001","journal-title":"J. Conscious. Stud."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"193","DOI":"10.2190\/A704-5647-5245-R47P","article-title":"The color of music: Correspondence through emotion","volume":"25","author":"Barbiere","year":"2007","journal-title":"Empir. Stud. Arts"},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1430","DOI":"10.1080\/02699931.2015.1070119","article-title":"Perceptual dimensions differentiate emotions","volume":"30","author":"Cavanaugh","year":"2016","journal-title":"Cogn. Emot."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1080\/14640747608400532","article-title":"Visual perception of musical notation: Registering pitch symbols in memory","volume":"28","author":"Sloboda","year":"1976","journal-title":"Q. J. Exp. Psychol."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"43","DOI":"10.2307\/40285811","article-title":"A cross-cultural investigation of the perception of emotion in music: Psychophysical and cultural cues","volume":"17","author":"Balkwill","year":"1999","journal-title":"Music Percept."},{"key":"ref_30","unstructured":"Cariani, P. (2019). Musical Intervals, Scales, and Tunings: Auditory Representations and Neural Codes, MIT Press."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"229","DOI":"10.1163\/156856808784532563","article-title":"Implicit and explicit features of paintings","volume":"21","year":"2008","journal-title":"Spat. Vis."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"63","DOI":"10.1162\/002409404772828157","article-title":"A universal grammar for visual composition?","volume":"37","author":"Stebbing","year":"2004","journal-title":"Leonardo"},{"key":"ref_33","unstructured":"Lockton, D., Lenzi, S., Hekkert, P., Oak, A., S\u00e1daba, J., and Lloyd, P. (July, January 25). Audiovisual Sonifications: A Design Map for Multisensory Integration in Data Representation. Proceedings of the DRS2022, Bilbao, Spain."},{"key":"ref_34","first-page":"141","article-title":"The further exploits of AARON, painter","volume":"4","author":"Cohen","year":"1995","journal-title":"Stanf. Humanit. Rev."},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"McCormack, J. (2001). Eden: An evolutionary sonic ecosystem. European Conference on Artificial Life, Springer.","DOI":"10.1007\/3-540-44811-X_13"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Cope, D. (2015). Algorithmic music composition. Patterns of Intuition, Springer.","DOI":"10.1007\/978-94-017-9561-6_19"},{"key":"ref_37","unstructured":"Machado, P., and Cardoso, A. (2000, January 17\u201318). NEvAr\u2013the assessment of an evolutionary art tool. Proceedings of the Symposium on Creative & Cultural Aspects and Applications of AI & Cognitive Science, Birmingham, UK."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"433","DOI":"10.1007\/s10710-017-9307-y","article-title":"Affective evolutionary music composition with MetaCompose","volume":"18","author":"Scirea","year":"2017","journal-title":"Genet. Program. Evolvable Mach."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1162\/COMJ_a_00282","article-title":"Techniques for generative melodies inspired by music cognition","volume":"39","author":"Brown","year":"2015","journal-title":"Comput. Music J."},{"key":"ref_40","unstructured":"Poli, R., Langdon, W.B., and McPhee, N.F. (2022, November 01). A Field Guide to Genetic Programming. Available online: https:\/\/www.researchgate.net\/publication\/216301261_A_Field_Guide_to_Genetic_Programming."},{"key":"ref_41","unstructured":"Temperley, D. (2004). The Cognition of Basic Musical Structures, MIT Press."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Bregman, A.S. (1994). Auditory Scene Analysis: The Perceptual Organization of Sound, MIT Press.","DOI":"10.1121\/1.408434"},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1080\/14626260701253606","article-title":"Controlling musical emotionality: An affective computational architecture for influencing musical emotions","volume":"18","author":"Livingstone","year":"2007","journal-title":"Digit. Creat."},{"key":"ref_44","unstructured":"Leborg, C. (2006). Visual Grammar: A Design Handbook, Princeton Architectural Press."},{"key":"ref_45","unstructured":"Gerstner, K., and Stephenson, D.Q. (1964). Designing Programmes, Hastings House."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"489","DOI":"10.1348\/0007126042369811","article-title":"A model of aesthetic appreciation and aesthetic judgments","volume":"95","author":"Leder","year":"2004","journal-title":"Br. J. Psychol."},{"key":"ref_47","unstructured":"Sei\u00e7a, M., Rodrigues, A., Cardoso, F.A., Martins, P., and Machado, P. (2019, January 14\u201318). Computer Generation and Perception Evaluation of Music-Emotion Associations. Proceedings of the 14th International Symposium on Computer Music Multidisciplinary Research, Marseille, France."},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"283","DOI":"10.1006\/jvlc.2001.0209","article-title":"A visual programming language for expressing rhythmic visuals","volume":"12","author":"Collopy","year":"2001","journal-title":"J. Vis. Lang. Comput."},{"key":"ref_49","unstructured":"Rodrigues, A., Cardoso, A., and Machado, P. (2019, January 17\u201321). A Dynamic Approach for the Generation of Perceptual Associations. Proceedings of the ICCC, Charlotte, NC, USA."},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Bentley, P.J., and Corne, D.W. (2002). Creative Evolutionary Systems, Morgan Kaufmann.","DOI":"10.1016\/B978-155860673-9\/50035-5"},{"key":"ref_51","unstructured":"Stiny, G., and Gips, J. (1971, January 23\u201328). Shape grammars and the generative specification of painting and sculpture. Proceedings of the IFIP Congress (2), Ljubljana, Yugoslavia."},{"key":"ref_52","unstructured":"Llano, M.T., d\u2019Inverno, M., Yee-King, M., McCormack, J., Ilsar, A., Pease, A., and Colton, S. (2022). Explainable computational creativity. arXiv."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/24\/12\/1706\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T01:24:14Z","timestamp":1760145854000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/24\/12\/1706"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,11,22]]},"references-count":52,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2022,12]]}},"alternative-id":["e24121706"],"URL":"https:\/\/doi.org\/10.3390\/e24121706","relation":{},"ISSN":["1099-4300"],"issn-type":[{"type":"electronic","value":"1099-4300"}],"subject":[],"published":{"date-parts":[[2022,11,22]]}}}