{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,3,13]],"date-time":"2025-03-13T04:07:18Z","timestamp":1741838838806,"version":"3.38.0"},"reference-count":44,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2025,3,12]],"date-time":"2025-03-12T00:00:00Z","timestamp":1741737600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,3,12]],"date-time":"2025-03-12T00:00:00Z","timestamp":1741737600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"Funda\u00e7\u00e3o para a Ci\u00eancia e Tecnologia","award":["PTDC\/CCI\/ 29234\/2017"],"award-info":[{"award-number":["PTDC\/CCI\/ 29234\/2017"]}]},{"DOI":"10.13039\/501100005765","name":"Universidade de Lisboa","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100005765","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["SN COMPUT. SCI."],"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>Largely due to the significant emotional impact they have on viewers and in their lives, movies are a powerful vehicle for culture and education and one of the most important and impactful forms of entertainment. By making a huge amount of movies more accessible in pervasive services and devices, and helping in emotion recognition and classification, technology has been playing an important role, and it is becoming more pertinent the ability to search, visualize and access movies based on their emotional impact, although emotions are seldom taken into account in these systems. In this paper, we characterize the challenges and approaches in this scenario, then present and evaluate, at two different stages, interactive means to visualize and search movies based on their dominant and actual emotional impact along the movie, with different and personalizable models and modalities, in As Movies Go By. In particular through emotional highlights in words, colors, emojis and trajectories, by drawing emotional blueprints or through users\u2019 emotional states, with the ability to get us into a movie in serendipitous moments.<\/jats:p>","DOI":"10.1007\/s42979-025-03690-2","type":"journal-article","created":{"date-parts":[[2025,3,12]],"date-time":"2025-03-12T05:30:44Z","timestamp":1741757444000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Multimodal Search and Visualisation of Movies Based on Emotions Along Time in As Movies Go By"],"prefix":"10.1007","volume":"6","author":[{"given":"Francisco","family":"Caldeira","sequence":"first","affiliation":[]},{"given":"Jo\u00e3o","family":"Louren\u00e7o","sequence":"additional","affiliation":[]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0306-3352","authenticated-orcid":false,"given":"Teresa","family":"Chambel","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2025,3,12]]},"reference":[{"key":"3690_CR1","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-85729-079-3","volume-title":"Visualization of time-oriented data","author":"W Aigner","year":"2011","unstructured":"Aigner W, Miksch S, Schumann H, Tominski C. Visualization of time-oriented data, vol. 4. London: Springer; 2011. https:\/\/doi.org\/10.1007\/978-0-85729-079-3."},{"issue":"1","key":"3690_CR2","doi-asserted-by":"publisher","first-page":"8","DOI":"10.3390\/bs10010008","volume":"10","author":"P Arriaga","year":"2019","unstructured":"Arriaga P, Alexandre J, Postolache O, Fonseca MJ, Langlois T, Chambel T. Why do we watch? The role of emotion gratifications and individual differences in predicting rewatchability and movie recommendation. Behav Sci. 2019;10(1):8. https:\/\/doi.org\/10.3390\/bs10010008.","journal-title":"Behav Sci"},{"issue":"2","key":"3690_CR3","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1016\/S0376-6357(02)00078-5","volume":"17","author":"P Aurier","year":"2015","unstructured":"Aurier P, Guintcheva G. The dynamics of emotions in movie consumption: a spectator-centred approach. Int J Arts Manag. 2015;17(2):5\u201318.","journal-title":"Int J Arts Manag"},{"key":"3690_CR4","doi-asserted-by":"publisher","unstructured":"Bader N, Mokryn O, Lanir J. Exploring emotions in online movie reviews for online browsing. In 22nd International Conference on Intelligent User Interfaces Companion. 2017, 35\u201338. https:\/\/doi.org\/10.1145\/3030024.3040982.","DOI":"10.1145\/3030024.3040982."},{"issue":"2","key":"3690_CR5","doi-asserted-by":"publisher","first-page":"69","DOI":"10.1016\/S0376-6357(02)00078-5","volume":"60","author":"M Cabanac","year":"2002","unstructured":"Cabanac M. What is emotion? Behav Proc. 2002;60(2):69\u201383. https:\/\/doi.org\/10.1016\/S0376-6357(02)00078-5.","journal-title":"Behav Proc"},{"key":"3690_CR6","doi-asserted-by":"publisher","unstructured":"Caldeira F, Louren\u00e7o J, Tavares Silva N, Chambel T. Towards multimodal search and visualization of movies based on emotions. In\u00a0ACM International Conference on Interactive Media Experiences (IMX 2022), 2022, 349\u2013356. https:\/\/doi.org\/10.1145\/3505284.3532987.","DOI":"10.1145\/3505284.3532987."},{"key":"3690_CR7","doi-asserted-by":"publisher","unstructured":"Caldeira F, Louren\u00e7o J, Chambel T. Happy or sad, smiling or drawing: multimodal search and visualisation of movies based on emotions along time. In\u00a0Proc. of the 18th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications VISIGRAPP\/HUCAPP - Human Computer Interaction Theory and Applications 2023, SciTePress, 2023, pp. 85\u201397. Best Paper Award. https:\/\/doi.org\/10.5220\/0011896400003417.","DOI":"10.5220\/0011896400003417."},{"key":"3690_CR8","unstructured":"Card SK, Mackinlay JD, Shneiderman B. Information visualization. Readings in information visualization: using vision to think, 1999, 1\u201334."},{"issue":"1","key":"3690_CR9","doi-asserted-by":"publisher","first-page":"58","DOI":"10.1504\/IJAMC.2013.053674","volume":"5","author":"T Chambel","year":"2013","unstructured":"Chambel T, Langlois T, Martins P, Gil N, Silva N, Duarte E. Content-based search overviews and exploratory browsing of movies with MovieClouds. Int J Adv Media Commun. 2013;5(1):58\u201379. https:\/\/doi.org\/10.1504\/IJAMC.2013.053674.","journal-title":"Int J Adv Media Commun"},{"key":"3690_CR10","unstructured":"Chen Y-X. Exploratory browsing: enhancing the browsing experience with media collections. Ph. D. Dissertation. Citeseer. 2010."},{"issue":"11","key":"3690_CR11","doi-asserted-by":"publisher","first-page":"14663","DOI":"10.1007\/s11042-021-10803-5","volume":"81","author":"M Cohen-Kalaf","year":"2021","unstructured":"Cohen-Kalaf M, Lanir J, Bak P, Mokryn O. Movie emotion map: an interactive tool for exploring movies according to their emotional signature. Multimed Tools Appl. 2021;81(11):14663\u201384. https:\/\/doi.org\/10.1007\/s11042-021-10803-5.","journal-title":"Multimed Tools Appl"},{"key":"3690_CR12","unstructured":"Cowie R, Douglas-Cowie E, Savvidou S, McMahon E, Sawey M, Schroder M. \u201c'FEELTRACE': an instrument for recording perceived emotion in real time\u201d. In\u00a0ISCA tutorial and research workshop (ITRW) on speech and emotion. 2000."},{"issue":"2","key":"3690_CR13","doi-asserted-by":"publisher","first-page":"16","DOI":"10.1109\/MCG.2016.29","volume":"36","author":"P Cruz","year":"2016","unstructured":"Cruz P, Machado P. Pulsing blood vessels: a figurative approach to traffic visualization. IEEE Comput Graphics Appl. 2016;36(2):16\u201321. https:\/\/doi.org\/10.1109\/MCG.2016.29.","journal-title":"IEEE Comput Graphics Appl"},{"key":"3690_CR14","doi-asserted-by":"publisher","DOI":"10.1080\/17588928.2020.1846027","author":"A Dam\u00e1sio","year":"2021","unstructured":"Dam\u00e1sio A. Feeling & knowing: making minds conscious. Pantheon. 2021. https:\/\/doi.org\/10.1080\/17588928.2020.1846027.","journal-title":"Pantheon"},{"issue":"3\u20134","key":"3690_CR15","doi-asserted-by":"publisher","first-page":"240","DOI":"10.1057\/PALGRAVE.IVS.9500182","volume":"7","author":"S Dodge","year":"2008","unstructured":"Dodge S, Weibel R, Lautensch\u00fctz A-K. Towards a taxonomy of movement patterns. Inf Vis. 2008;7(3\u20134):240\u201352. https:\/\/doi.org\/10.1057\/PALGRAVE.IVS.9500182.","journal-title":"Inf Vis"},{"issue":"3\u20134","key":"3690_CR16","doi-asserted-by":"publisher","first-page":"169","DOI":"10.1080\/02699939208411068","volume":"6","author":"P Ekman","year":"1992","unstructured":"Ekman P. An argument for basic emotions. Cogn Emot. 1992;6(3\u20134):169\u2013200. https:\/\/doi.org\/10.1080\/02699939208411068.","journal-title":"Cogn Emot"},{"key":"3690_CR17","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyg.2021.645173","author":"B Fischer","year":"2021","unstructured":"Fischer B, Herbert C. Emoji as affective symbols: affective judgments of emoji, emoticons, and human faces varying in emotional content. Front Psychol. 2021. https:\/\/doi.org\/10.3389\/fpsyg.2021.645173.","journal-title":"Front Psychol"},{"key":"3690_CR18","doi-asserted-by":"publisher","first-page":"201","DOI":"10.1145\/332040.332432","volume":"2000","author":"M Hassenzahl","year":"2000","unstructured":"Hassenzahl M, Platz A, Burmester M, Lehner K. Hedonic and ergonomic quality aspects determine a software\u2019s appeal. ACM CHI. 2000;2000:201\u20138. https:\/\/doi.org\/10.1145\/332040.332432.","journal-title":"ACM CHI"},{"key":"3690_CR19","doi-asserted-by":"publisher","unstructured":"Jorge A, Serra S, Chambel T.\u00a0Interactive visualizations of video tours in space and time. In\u00a028th Int. BCS Human Computer Interaction Conference (HCI 2014), 2014, pp. 329\u2013334. https:\/\/doi.org\/10.5555\/2742941.2742989.","DOI":"10.5555\/2742941.2742989."},{"issue":"4","key":"3690_CR20","doi-asserted-by":"publisher","first-page":"345","DOI":"10.1007\/BF00992553","volume":"5","author":"PR Kleinginna","year":"1981","unstructured":"Kleinginna PR, Kleinginna AM. A categorized list of emotion definitions, with suggestions for a consensual definition. Motiv Emot. 1981;5(4):345\u201379. https:\/\/doi.org\/10.1007\/BF00992553.","journal-title":"Motiv Emot"},{"key":"3690_CR21","doi-asserted-by":"publisher","unstructured":"Knautz K, Siebenlist T, Stock WG. Memose: search engine for emotions in multimedia documents. In Proceedings of the 33rd international ACM SIGIR conference on research and development in information retrieval. 2010, 791\u2013792. https:\/\/doi.org\/10.1145\/1835449.1835618.","DOI":"10.1145\/1835449.1835618."},{"issue":"2","key":"3690_CR22","first-page":"3","volume":"8","author":"AM Lund","year":"2001","unstructured":"Lund AM. Measuring usability with the USE questionnaire. Usability User Exp. 2001;8(2):3\u20136.","journal-title":"Usability User Exp"},{"key":"3690_CR23","doi-asserted-by":"publisher","unstructured":"Martinho J, Chambel T. ColorsInMotion: interactive visualization and exploration of video spaces. In Proc. of the 13th International MindTrek Conference: Everyday Life in the Ubiquitous Era. 2009, 190\u2013197. https:\/\/doi.org\/10.1145\/1621841.1621876.","DOI":"10.1145\/1621841.1621876."},{"key":"3690_CR24","doi-asserted-by":"publisher","first-page":"81","DOI":"10.1037\/h0043158","volume":"63","author":"GA Miller","year":"1956","unstructured":"Miller GA. The magical number seven, plus or minus two: some limits on our capacity for processing information. Psychol Rev. 1956;63:81\u201397. https:\/\/doi.org\/10.1037\/h0043158.","journal-title":"Psychol Rev"},{"key":"3690_CR25","doi-asserted-by":"publisher","unstructured":"Moreira A, Chambel T. As Music goes by in versions and movies along time. In Proc. of ACM International Conference on Interactive Experiences for TV and Online Video (TVX 2018), 2018, pp. 239\u2013244. https:\/\/doi.org\/10.1145\/3210825.3213567.","DOI":"10.1145\/3210825.3213567."},{"key":"3690_CR26","doi-asserted-by":"publisher","unstructured":"Moreira A, Chambel T. This Music reminds me of a movie, or is it an old song? An interactive audiovisual journey to find out, explore and play. In VISIGRAPP (1:GRAPP). 2019, 145\u2013158. https:\/\/doi.org\/10.5220\/0007692401450158.","DOI":"10.5220\/0007692401450158."},{"key":"3690_CR27","unstructured":"Nefkens MW. MovieWall: a novel interface concept for visual exploration of large movie collections. Master\u2019s thesis, Utrecht University; 2017. https:\/\/studenttheses.uu.nl\/handle\/20.500.12932\/27906. Accessed Aug 2023."},{"key":"3690_CR28","doi-asserted-by":"publisher","unstructured":"Nunes L, Ribeiro C, Chambel T. Emotional and Engaging movie annotation with gamification.\u00a0VISIGRAPP (2: HUCAPP)\u00a02022:\u00a0262\u2013272. https:\/\/doi.org\/10.5220\/0010991500003124.","DOI":"10.5220\/0010991500003124."},{"key":"3690_CR29","doi-asserted-by":"publisher","unstructured":"Oliveira E, Martins P, Chambel T. iFelt: accessing movies through our emotions. In Proc. of 9th European Conference on Interactive TV and Video, 2011, 105\u2013114. https:\/\/doi.org\/10.1145\/2000119.2000141.","DOI":"10.1145\/2000119.2000141."},{"key":"3690_CR30","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1016\/B978-0-12-558701-3.50007-7","volume-title":"Theories of emotion","author":"R Plutchik","year":"1980","unstructured":"Plutchik R. A general psychoevolutionary theory of emotion. In: Theories of emotion. Amsterdam: Elsevier; 1980. p. 3\u201333."},{"issue":"4","key":"3690_CR31","doi-asserted-by":"publisher","first-page":"344","DOI":"10.1511\/2001.28.344","volume":"89","author":"R Plutchik","year":"2001","unstructured":"Plutchik R. The nature of emotions: Human emotions have deep evolutionary roots, a fact that may explain their complexity and provide tools for clinical practice. Am Sci. 2001;89(4):344\u201350. https:\/\/doi.org\/10.1511\/2001.28.344.","journal-title":"Am Sci"},{"key":"3690_CR32","doi-asserted-by":"publisher","unstructured":"Ramalho J, Chambel T. Immersive 360\u00ba mobile video with an emotional perspective. 2013 ACM international workshop on Immersive Media Experiences. 2013. https:\/\/doi.org\/10.1145\/2512142.2512144.","DOI":"10.1145\/2512142.2512144."},{"issue":"6","key":"3690_CR33","doi-asserted-by":"publisher","first-page":"1161","DOI":"10.1037\/h0077714","volume":"39","author":"JA Russell","year":"1980","unstructured":"Russell JA. A circumplex model of affect. J Pers Soc Psychol. 1980;39(6):1161\u201378. https:\/\/doi.org\/10.1037\/h0077714.","journal-title":"J Pers Soc Psychol"},{"key":"3690_CR34","unstructured":"Sacharin V, Schlegel K, Scherer KR. Geneva emotion wheel rating study. Archive ouverte UNIGE; 2012. https:\/\/archive-ouverte.unige.ch\/unige:97849. Accessed Aug 2023."},{"issue":"7","key":"3690_CR35","doi-asserted-by":"publisher","first-page":"1307","DOI":"10.1080\/02699930902928969","volume":"23","author":"KR Scherer","year":"2009","unstructured":"Scherer KR. The dynamic architecture of emotion: evidence for the component process model. Cogn Emot. 2009;23(7):1307\u201351. https:\/\/doi.org\/10.1080\/02699930902928969.","journal-title":"Cogn Emot"},{"issue":"9","key":"3690_CR36","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0256503","volume":"16","author":"A Semeraro","year":"2021","unstructured":"Semeraro A, Vilella S, Ruffo G. PyPlutchik: visualizing and comparing emotion-annotated corpora. PLoS ONE. 2021;16(9): e0256503. https:\/\/doi.org\/10.1371\/journal.pone.0256503.","journal-title":"PLoS ONE"},{"key":"3690_CR37","doi-asserted-by":"publisher","unstructured":"Serra S, Jorge A, Chambel T. Multimodal access to georeferenced mobile video through shape, speed and time. In 28th Int. BCS Human Computer Interaction Conference (HCI 2014). 2014, 347\u2013352. https:\/\/doi.org\/10.5555\/2742941.2742992.","DOI":"10.5555\/2742941.2742992."},{"key":"3690_CR38","doi-asserted-by":"publisher","unstructured":"Serra V, Chambel T. Quote surfing in music and movies with an emotional flavor. In VISIGRAPP (2: HUCAPP). 2020, 75\u201385. https:\/\/doi.org\/10.5220\/0009177300750085.","DOI":"10.5220\/0009177300750085."},{"key":"3690_CR39","unstructured":"Shah V. The role of film in society. ThoughtEconomics; 2011. https:\/\/thoughteconomics.com\/the-role-of-film-in-society\/. Accessed 29 May 2023."},{"key":"3690_CR40","unstructured":"Tkalcic M, Kosir A, Tasic J. Affective recommender systems: the role of emotions in recommender systems. In: The RecSys 2011 workshop on human decision making in recommender systems. Citeseer; 2011, pp. 9\u201313. https:\/\/api.semanticscholar.org\/CorpusID:16782471. Accessed Aug 2023."},{"key":"3690_CR41","doi-asserted-by":"publisher","unstructured":"Tominski C, Abello J, Schumann H. Axes-based visualizations with radial layouts. In\u00a0Proceeding of 2004 ACM symposium on Applied Computing, 2004, 242\u20131247. https:\/\/doi.org\/10.1145\/967900.968153.","DOI":"10.1145\/967900.968153."},{"key":"3690_CR42","volume-title":"The visual display of quantitative information","author":"E Tufte","year":"1983","unstructured":"Tufte E. The visual display of quantitative information. Cheshire: Graphics Press LLC; 1983."},{"key":"3690_CR43","unstructured":"Uhrig SC.\u00a0Cinema is good for you: the effects of cinema attendance on self reported anxiety or depression and 'happiness'. ISER Working Paper Series\u00a02005\u201314. Institute for Social and Economic Research; 2005. https:\/\/ideas.repec.org\/p\/ese\/iserwp\/2005-14.html. Accessed Aug 2023."},{"key":"3690_CR44","doi-asserted-by":"publisher","unstructured":"Zhang S, Tian Q, Huang Q. Gao W, Li S. Utilizing affective analysis for efficient movie browsing. In 16th IEEE International Conference on Image Processing (ICIP). IEEE, 2009, 1853\u20131856. https:\/\/doi.org\/10.1109\/ICIP.2009.5413590.","DOI":"10.1109\/ICIP.2009.5413590."}],"container-title":["SN Computer Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s42979-025-03690-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s42979-025-03690-2\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s42979-025-03690-2.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,3,12]],"date-time":"2025-03-12T05:30:52Z","timestamp":1741757452000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s42979-025-03690-2"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,3,12]]},"references-count":44,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2025,3]]}},"alternative-id":["3690"],"URL":"https:\/\/doi.org\/10.1007\/s42979-025-03690-2","relation":{},"ISSN":["2661-8907"],"issn-type":[{"value":"2661-8907","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,3,12]]},"assertion":[{"value":"6 September 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"22 December 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 March 2025","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Francisco Caldeira declares that he has no conflict of interest. Jo\u00e3o Louren\u00e7o declares that he has no conflict of interest. Teresa Chambel declares that she has no conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"This article does not contain any studies with animals performed by any of the authors. All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and\/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethical Approval"}},{"value":"Informed consent was obtained from all individual participants included in the study.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Informed Consent"}}],"article-number":"272"}}