{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,25]],"date-time":"2026-08-25T10:33:52Z","timestamp":1787654032099,"version":"build-2736575974"},"reference-count":63,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2022,3,4]],"date-time":"2022-03-04T00:00:00Z","timestamp":1646352000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Multimedia Comput. Commun. Appl."],"published-print":{"date-parts":[[2022,8,31]]},"abstract":"<jats:p>Considerable attention has been paid to physiological signal-based emotion recognition in the field of affective computing. For reliability and user-friendly acquisition, electrodermal activity (EDA) has a great advantage in practical applications. However, EDA-based emotion recognition with large-scale subjects is still a tough problem. The traditional well-designed classifiers with hand-crafted features produce poorer results because of their limited representation abilities. And the deep learning models with auto feature extraction suffer the overfitting drop-off because of large-scale individual differences. Since music has a strong correlation with human emotion, static music can be involved as the external benchmark to constrain various dynamic EDA signals. In this article, we make an attempt by fusing the subject\u2019s individual EDA features and the external evoked music features. And we propose an end-to-end multimodal framework, the one-dimensional residual temporal and channel attention network (RTCAN-1D). For EDA features, the channel-temporal attention mechanism for EDA-based emotion recognition is first involved in mine the temporal and channel-wise dynamic and steady features. The comparisons with single EDA-based SOTA models on DEAP and AMIGOS datasets prove the effectiveness of RTCAN-1D to mine EDA features. For music features, we simply process the music signal with the open-source toolkit openSMILE to obtain external feature vectors. We conducted systematic and extensive evaluations. The experiments on the current largest music emotion dataset PMEmo validate that the fusion of EDA and music is a reliable and efficient solution for large-scale emotion recognition.<\/jats:p>","DOI":"10.1145\/3490686","type":"journal-article","created":{"date-parts":[[2022,3,4]],"date-time":"2022-03-04T05:26:32Z","timestamp":1646371592000},"page":"1-23","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":46,"title":["A Multimodal Framework for Large-Scale Emotion Recognition by Fusing Music and Electrodermal Activity Signals"],"prefix":"10.1145","volume":"18","author":[{"given":"Guanghao","family":"Yin","sequence":"first","affiliation":[{"name":"Zhejiang University, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shouqian","family":"Sun","sequence":"additional","affiliation":[{"name":"Zhejiang University, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dian","family":"Yu","sequence":"additional","affiliation":[{"name":"Zhejiang University, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dejian","family":"Li","sequence":"additional","affiliation":[{"name":"Zhejiang University, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kejun","family":"Zhang","sequence":"additional","affiliation":[{"name":"Zhejiang University and Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies, Hangzhou, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2022,3,4]]},"reference":[{"key":"e_1_3_1_2_2","doi-asserted-by":"publisher","DOI":"10.3390\/s19071659"},{"key":"e_1_3_1_3_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jneumeth.2005.02.001"},{"key":"e_1_3_1_4_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0173392"},{"key":"e_1_3_1_5_2","doi-asserted-by":"publisher","DOI":"10.1109\/MeMeA.2018.8438595"},{"key":"e_1_3_1_6_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2018.2798607"},{"key":"e_1_3_1_7_2","volume-title":"Deep Listeners: Music, Emotion, and Trancing","author":"Becker Judith","year":"2004","unstructured":"Judith Becker. 2004. Deep Listeners: Music, Emotion, and Trancing, Vol. 1. Indiana University Press."},{"key":"e_1_3_1_8_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jneumeth.2010.04.028"},{"issue":"4","key":"e_1_3_1_9_2","first-page":"647","article-title":"Decomposition of skin conductance data by means of nonnegative deconvolution","volume":"47","author":"Benedek Mathias","year":"2010","unstructured":"Mathias Benedek and Christian Kaernbach. 2010. Decomposition of skin conductance data by means of nonnegative deconvolution. Psychophysiology 47, 4 (2010), 647\u2013658.","journal-title":"Psychophysiology"},{"key":"e_1_3_1_10_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4614-1126-0"},{"key":"e_1_3_1_11_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2005.38"},{"key":"e_1_3_1_12_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICASSP.2016.7472151"},{"key":"e_1_3_1_13_2","article-title":"Amigos: A dataset for affect, personality and mood research on individuals and groups","author":"Correa Juan Abdon Miranda","year":"2018","unstructured":"Juan Abdon Miranda Correa, Mojtaba Khomami Abadi, Niculae Sebe, and Ioannis Patras. 2018. Amigos: A dataset for affect, personality and mood research on individuals and groups. IEEE Transactions on Affective Computing 12, 2 (2018), 479\u2013493.","journal-title":"IEEE Transactions on Affective Computing"},{"key":"e_1_3_1_14_2","volume-title":"Descartes\u2019 error: Emotion, reason, and the human brain","author":"Damasio Antonio R.","year":"1994","unstructured":"Antonio R. Damasio. 1994. Descartes\u2019 error: Emotion, reason, and the human brain. Optometry and Vision Science 72, 11 (1994), 847\u2013848."},{"key":"e_1_3_1_15_2","first-page":"200","article-title":"The electrodermal system","volume":"2","author":"Dawson Michael E.","year":"2007","unstructured":"Michael E. Dawson, Anne M. Schell, and Diane L. Filion. 2007. The electrodermal system. Handbook of Psychophysiology 2 (2007), 200\u2013223.","journal-title":"Handbook of Psychophysiology"},{"key":"e_1_3_1_16_2","doi-asserted-by":"publisher","DOI":"10.1080\/02699939208411068"},{"key":"e_1_3_1_17_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2010.09.020"},{"key":"e_1_3_1_18_2","doi-asserted-by":"publisher","DOI":"10.1109\/TITB.2009.2038481"},{"key":"e_1_3_1_19_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2020.113571"},{"key":"e_1_3_1_20_2","doi-asserted-by":"publisher","DOI":"10.1109\/JSEN.2016.2623677"},{"issue":"4","key":"e_1_3_1_21_2","first-page":"797","article-title":"cvxEDA: A convex optimization approach to electrodermal activity processing","volume":"63","author":"Greco Alberto","year":"2015","unstructured":"Alberto Greco, Gaetano Valenza, Antonio Lanata, Enzo Pasquale Scilingo, and Luca Citi. 2015. cvxEDA: A convex optimization approach to electrodermal activity processing. IEEE Transactions on Biomedical Engineering 63, 4 (2015), 797\u2013804.","journal-title":"IEEE Transactions on Biomedical Engineering"},{"key":"e_1_3_1_22_2","doi-asserted-by":"publisher","DOI":"10.4108\/icst.pervasivehealth.2013.252133"},{"key":"e_1_3_1_23_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.tics.2012.07.006"},{"key":"e_1_3_1_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2016.90"},{"key":"e_1_3_1_25_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00745"},{"key":"e_1_3_1_26_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.243"},{"key":"e_1_3_1_27_2","doi-asserted-by":"publisher","DOI":"10.1111\/j.1745-6916.2007.00044.x"},{"key":"e_1_3_1_28_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cviu.2006.10.019"},{"key":"e_1_3_1_29_2","doi-asserted-by":"publisher","DOI":"10.1109\/CSPA.2011.5759912"},{"key":"e_1_3_1_30_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSMCA.2008.918624"},{"key":"e_1_3_1_31_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.bspc.2017.08.024"},{"key":"e_1_3_1_32_2","first-page":"280","article-title":"Bimodal emotion recognition using speech and physiological changes","volume":"265","author":"Kim Jonghwa","year":"2007","unstructured":"Jonghwa Kim. 2007. Bimodal emotion recognition using speech and physiological changes. Robust Speech Recognition and Understanding 265 (2007), 280.","journal-title":"Robust Speech Recognition and Understanding"},{"key":"e_1_3_1_33_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2008.26"},{"key":"e_1_3_1_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/2808204"},{"key":"e_1_3_1_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/T-AFFC.2011.15"},{"key":"e_1_3_1_36_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.imavis.2012.10.002"},{"key":"e_1_3_1_37_2","doi-asserted-by":"publisher","DOI":"10.1111\/j.1469-8986.1993.tb03352.x"},{"key":"e_1_3_1_38_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0140-6736(11)60325-5"},{"key":"e_1_3_1_39_2","doi-asserted-by":"publisher","DOI":"10.1109\/IJCNN.2000.857823"},{"key":"e_1_3_1_40_2","first-page":"540","volume-title":"Proceedings of the AAAI\/IAAI","author":"Lawrence Steve","year":"1997","unstructured":"Steve Lawrence, C. Lee Giles, and Ah Chung Tsoi. 1997. Lessons in neural network training: Overfitting may be harder than expected. In Proceedings of the AAAI\/IAAI. Citeseer, 540\u2013545."},{"key":"e_1_3_1_41_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-71589-6_33"},{"issue":"1","key":"e_1_3_1_42_2","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2037676.2037683","article-title":"Exploiting online music tags for music emotion classification","volume":"7","author":"Lin Yu-Ching","year":"2011","unstructured":"Yu-Ching Lin, Yi-Hsuan Yang, and Homer H. Chen. 2011. Exploiting online music tags for music emotion classification. ACM Transactions on Multimedia Computing, Communications, and Applications 7, 1 (2011), 1\u201316.","journal-title":"ACM Transactions on Multimedia Computing, Communications, and Applications"},{"key":"e_1_3_1_43_2","first-page":"194","volume-title":"Proceedings of the Pacific Rim Conference on Multimedia","author":"Liu Jiamin","year":"2017","unstructured":"Jiamin Liu, Yuanqi Su, and Yuehu Liu. 2017. Multi-modal emotion recognition with temporal-band attention based on lstm-rnn. In Proceedings of the Pacific Rim Conference on Multimedia. Springer, 194\u2013204."},{"key":"e_1_3_1_44_2","doi-asserted-by":"publisher","DOI":"10.1145\/3313831.3376366"},{"key":"e_1_3_1_45_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.94"},{"key":"e_1_3_1_46_2","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/1140.001.0001"},{"key":"e_1_3_1_47_2","doi-asserted-by":"publisher","DOI":"10.1109\/34.954607"},{"key":"e_1_3_1_48_2","doi-asserted-by":"publisher","DOI":"10.1177\/053901882021004003"},{"key":"e_1_3_1_49_2","doi-asserted-by":"publisher","DOI":"10.1037\/h0077714"},{"key":"e_1_3_1_50_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACII.2013.117"},{"key":"e_1_3_1_51_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2018.2883213"},{"key":"e_1_3_1_52_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2017.74"},{"key":"e_1_3_1_53_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.bspc.2018.08.024"},{"key":"e_1_3_1_54_2","article-title":"Feature extraction and selection for emotion recognition from electrodermal activity","author":"Shukla Jainendra","year":"2019","unstructured":"Jainendra Shukla, Miguel Barreda-Angeles, Joan Oliver, G. C. Nandi, and Domenec Puig. 2019. Feature extraction and selection for emotion recognition from electrodermal activity. IEEE Transactions on Affective Computing 12, 4 (2019), 857\u2013869.","journal-title":"IEEE Transactions on Affective Computing"},{"key":"e_1_3_1_55_2","unstructured":"Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. In the 3rd International Conference on Learning Representations (ICLR\u201915) ."},{"key":"e_1_3_1_56_2","doi-asserted-by":"publisher","DOI":"10.5555\/2627435.2670313"},{"key":"e_1_3_1_57_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v31i1.11112"},{"key":"e_1_3_1_58_2","doi-asserted-by":"publisher","DOI":"10.1109\/EMBC.2013.6610504"},{"key":"e_1_3_1_59_2","first-page":"5998","volume-title":"Proceedings of the Advances in Neural Information Processing Systems","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the Advances in Neural Information Processing Systems. 5998\u20136008."},{"key":"e_1_3_1_60_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00813"},{"key":"e_1_3_1_61_2","doi-asserted-by":"publisher","DOI":"10.2307\/1413483"},{"key":"e_1_3_1_62_2","doi-asserted-by":"publisher","DOI":"10.1016\/S1566-2535(02)00055-6"},{"key":"e_1_3_1_63_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICIP.2019.8803627"},{"key":"e_1_3_1_64_2","doi-asserted-by":"publisher","DOI":"10.1145\/3206025.3206037"}],"container-title":["ACM Transactions on Multimedia Computing, Communications, and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3490686","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3490686","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T15:30:29Z","timestamp":1750174229000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3490686"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3,4]]},"references-count":63,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2022,8,31]]}},"alternative-id":["10.1145\/3490686"],"URL":"https:\/\/doi.org\/10.1145\/3490686","relation":{},"ISSN":["1551-6857","1551-6865"],"issn-type":[{"value":"1551-6857","type":"print"},{"value":"1551-6865","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,3,4]]},"assertion":[{"value":"2021-01-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-10-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-03-04","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}