{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,5]],"date-time":"2026-06-05T04:54:41Z","timestamp":1780635281357,"version":"3.54.1"},"reference-count":80,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2025,3,10]],"date-time":"2025-03-10T00:00:00Z","timestamp":1741564800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62276242"],"award-info":[{"award-number":["62276242"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"National Aviation Science Foundation","award":["2022Z071078001"],"award-info":[{"award-number":["2022Z071078001"]}]},{"name":"CAAI-Huawei MindSpore Open Fund","award":["CAAIXSJLJJ-2022-001A"],"award-info":[{"award-number":["CAAIXSJLJJ-2022-001A"]}]},{"name":"Anhui Province Key Research and Development Program","award":["202104a05020007"],"award-info":[{"award-number":["202104a05020007"]}]},{"name":"Dreams Foundation of Jianghuai Advance Technology Center","award":["2023-ZM01Z001"],"award-info":[{"award-number":["2023-ZM01Z001"]}]},{"name":"Beijing Municipal Science and Technology Commission, and Administrative Commission of Zhongguancun Science Park","award":["Z231100005923035"],"award-info":[{"award-number":["Z231100005923035"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Multimedia Comput. Commun. Appl."],"published-print":{"date-parts":[[2025,4,30]]},"abstract":"<jats:p>As a focal point of research in various fields, human body language understanding has long been a subject of intense interest. Within this realm, the exploration of emotion recognition through the analysis of facial expressions, voice patterns, and physiological signals holds significant practical value. Compared with unimodal approaches, multimodal emotion recognition models leverage complementary information from vision, acoustic, and language modalities to robust perceive the human sentiment attitudes. However, the heterogeneity among modality signals leads to significant domain shifts, posing challenges for achieving balanced fusion. In this article, we propose a Domain-Separated Bottleneck Attention (DBA) Fusion Framework for human multimodal emotion recognition with lower computational complexity. Specifically, we partition each modality into two distinct domains: the invariant\/private domain. The invariant domain contains crucial shared information, while the private domain aims to capture modality-specific representations. For the decomposed features, we introduce two sets of bottleneck cross-attention modules to effectively utilize the complementarity between domains to reduce redundant information. In each module, we interweave two Fusion Adapter blocks into the Self-Attention Transformer backbone. Each Fusion Adapter block integrates a small group of latent tokens as bridges for inter-modal and inter-domain interactions, mitigating the adverse effects of modality distribution differences and lowering computational costs. Extensive experimental results demonstrate that our method outperforms State-of-the-Art (SOTA) approaches across three widely used benchmark datasets.<\/jats:p>","DOI":"10.1145\/3711865","type":"journal-article","created":{"date-parts":[[2025,1,10]],"date-time":"2025-01-10T14:38:01Z","timestamp":1736519881000},"page":"1-21","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":12,"title":["Domain-Separated Bottleneck Attention Fusion Framework for Multimodal Emotion Recognition"],"prefix":"10.1145","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0009-0004-5455-3812","authenticated-orcid":false,"given":"Peng","family":"He","sequence":"first","affiliation":[{"name":"University of Science and Technology of China, Hefei, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3197-8103","authenticated-orcid":false,"given":"Jun","family":"Yu","sequence":"additional","affiliation":[{"name":"Department of Automation, University of Science and Technology of China, Hefei, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chengjie","family":"Ge","sequence":"additional","affiliation":[{"name":"University of Science and Technology of China, Hefei, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ye","family":"Yu","sequence":"additional","affiliation":[{"name":"Hefei University of Technology, Hefei, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wei","family":"Xu","sequence":"additional","affiliation":[{"name":"Division of Life Sciences and Medicine, The First Affiliated Hospital, University of Science and Technology of China, Hefei, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lei","family":"Wang","sequence":"additional","affiliation":[{"name":"USTC, Hefei, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tianyu","family":"Liu","sequence":"additional","affiliation":[{"name":"Jianghuai Advance Technology Center, Hefei, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhen","family":"Kan","sequence":"additional","affiliation":[{"name":"University of Science and Technology of China, Hefei, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2025,3,10]]},"reference":[{"key":"e_1_3_1_2_2","first-page":"718","article-title":"Emonet: Fine-grained emotion detection with gated recurrent neural networks","author":"Abdul-Mageed Muhammad","year":"2017","unstructured":"Muhammad Abdul-Mageed and Lyle Ungar. 2017. Emonet: Fine-grained emotion detection with gated recurrent neural networks. In ACL, 718\u2013728.","journal-title":"ACL"},{"key":"e_1_3_1_3_2","first-page":"24206","article-title":"Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text","volume":"34","author":"Akbari Hassan","year":"2021","unstructured":"Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. 2021. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. In NeurIPS, Vol. 34, 24206\u201324221.","journal-title":"NeurIPS"},{"key":"e_1_3_1_4_2","doi-asserted-by":"publisher","DOI":"10.5555\/1776334.1776363"},{"key":"e_1_3_1_5_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.1224313"},{"key":"e_1_3_1_6_2","first-page":"12449","article-title":"wav2vec 2.0: A framework for self-supervised learning of speech representations","author":"Baevski Alexei","year":"2020","unstructured":"Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. In Advances in Neural Information Processing Systems, 12449\u201312460.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_7_2","first-page":"1","volume-title":"WACV","author":"Baltru\u0161aitis Tadas","year":"2016","unstructured":"Tadas Baltru\u0161aitis, Peter Robinson, and Louis-Philippe Morency. 2016. Openface: An open source facial behavior analysis toolkit. In WACV. IEEE, 1\u201310."},{"key":"e_1_3_1_8_2","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1609\/aaai.v36i1.19873","article-title":"Learning unseen emotions from gestures via semantically-conditioned zero-shot perception with adversarial autoencoders","volume":"36","author":"Banerjee Abhishek","year":"2022","unstructured":"Abhishek Banerjee, Uttaran Bhattacharya, and Aniket Bera. 2022. Learning unseen emotions from gestures via semantically-conditioned zero-shot perception with adversarial autoencoders. In AAAI, Vol. 36, 3\u201310.","journal-title":"AAAI"},{"key":"e_1_3_1_9_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10579-008-9076-6"},{"key":"e_1_3_1_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3640343"},{"key":"e_1_3_1_11_2","unstructured":"Jacob Devlin Ming-Wei Chang Kenton Lee and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805."},{"key":"e_1_3_1_12_2","unstructured":"Soumya Dutta and Sriram Ganapathy. 2023. HCAM\u2013Hierarchical cross attention model for multi-modal emotion recognition. arXiv:2304.06910."},{"key":"e_1_3_1_13_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.patcog.2010.09.020"},{"key":"e_1_3_1_14_2","doi-asserted-by":"crossref","first-page":"255","DOI":"10.1017\/CBO9780511659911.013","article-title":"Spontaneous facial behavior during intense emotional episodes: Artistic truth and optical truth","volume":"2","author":"Fern\u00e1ndez-Dols Jos\u00e9 Miguel","year":"1997","unstructured":"Jos\u00e9 Miguel Fern\u00e1ndez-Dols and Mar\u00eda-Angeles Ruiz-Belda. 1997. Spontaneous facial behavior during intense emotional episodes: Artistic truth and optical truth. The Psychology of Facial Expression 2 (1997), 255\u2013274.","journal-title":"The Psychology of Facial Expression"},{"issue":"5","key":"e_1_3_1_15_2","doi-asserted-by":"crossref","first-page":"2018","DOI":"10.1093\/cercor\/bhv020","article-title":"Concept representation reflects multimodal abstraction: A framework for embodied semantics","volume":"26","author":"Fernandino Leonardo","year":"2016","unstructured":"Leonardo Fernandino, Jeffrey R. Binder, Rutvik H. Desai, Suzanne L. Pendl, Colin J. Humphries, William L. Gross, Lisa L. Conant, and Mark S. Seidenberg. 2016. Concept representation reflects multimodal abstraction: A framework for embodied semantics. Cerebral Cortex 26, 5 (2016), 2018\u20132034.","journal-title":"Cerebral Cortex"},{"key":"e_1_3_1_16_2","first-page":"1878","volume-title":"AAAI","volume":"38","author":"Ge Chengjie","year":"2024","unstructured":"Chengjie Ge, Xueyang Fu, Peng He, Kunyu Wang, Chengzhi Cao, and Zheng-Jun Zha. 2024. Neuromorphic event signal-driven network for video de-raining. In AAAI, Vol. 38, 1878\u20131886."},{"key":"e_1_3_1_17_2","first-page":"1510","volume-title":"WWW","author":"Ge Shiping","year":"2023","unstructured":"Shiping Ge, Zhiwei Jiang, Zifeng Cheng, Cong Wang, Yafeng Yin, and Qing Gu. 2023. Learning robust multi-modal representation for multi-label emotion recognition via adversarial masking and perturbation. In WWW, 1510\u20131518."},{"key":"e_1_3_1_18_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.inffus.2023.102218"},{"key":"e_1_3_1_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/1143844.1143891"},{"key":"e_1_3_1_20_2","first-page":"1122","volume-title":"MM","author":"Hazarika Devamanyu","year":"2020","unstructured":"Devamanyu Hazarika, Roger Zimmermann, and Soujanya Poria. 2020. Misa: Modality-invariant and-specific representations for multimodal sentiment analysis. In MM, 1122\u20131131."},{"key":"e_1_3_1_21_2","first-page":"2790","volume-title":"ICML","author":"Houlsby Neil","year":"2019","unstructured":"Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In ICML. PMLR, 2790\u20132799."},{"key":"e_1_3_1_22_2","first-page":"1439","volume-title":"ICCV","author":"Hu Ronghang","year":"2021","unstructured":"Ronghang Hu and Amanpreet Singh. 2021. Unit: Multimodal multitask learning with a unified transformer. In ICCV, 1439\u20131449."},{"key":"e_1_3_1_23_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neunet.2005.03.004"},{"key":"e_1_3_1_24_2","first-page":"133","volume-title":"Emotion-Oriented Systems: The Humaine Handbook","author":"Knapp R. Benjamin","year":"2010","unstructured":"R. Benjamin Knapp, Jonghwa Kim, and Elisabeth Andr\u00e9. 2010. Physiological signals and their use in augmenting emotion recognition for human\u2013machine interaction. In Emotion-Oriented Systems: The Humaine Handbook. Cowie Roddy, Pelachaud Catherine, and Petta Paolo (Eds.), Springer, 133\u2013159."},{"key":"e_1_3_1_25_2","first-page":"79399","article-title":"CAST: Cross-attention in space and time for video action recognition","volume":"36","author":"Lee Dongho","year":"2024","unstructured":"Dongho Lee, Jongseo Lee, and Jinwoo Choi. 2024. CAST: Cross-attention in space and time for video action recognition. In Advances in Neural Information Processing Systems, Vol. 36, 79399\u201379425.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_26_2","first-page":"1","article-title":"Cross-attentional audio-visual fusion for weakly-supervised action localization","author":"Lee Jun-Tae","year":"2020","unstructured":"Jun-Tae Lee, Mihir Jain, Hyoungwoo Park, and Sungrack Yun. 2020. Cross-attentional audio-visual fusion for weakly-supervised action localization. In ICLR, 1\u201317.","journal-title":"ICLR"},{"key":"e_1_3_1_27_2","first-page":"19108","volume-title":"CVPR","author":"Li Guangyao","year":"2022","unstructured":"Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, and Di Hu. 2022. Learning to answer questions in dynamic audio-visual scenarios. In CVPR, 19108\u201319118."},{"key":"e_1_3_1_28_2","first-page":"9694","article-title":"Align before fuse: Vision and language representation learning with momentum distillation","volume":"34","author":"Li Junnan","year":"2021","unstructured":"Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021. Align before fuse: Vision and language representation learning with momentum distillation. In Advances in Neural Information Processing Systems, Vol. 34, 9694\u20139705.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_1_29_2","first-page":"6631","volume-title":"CVPR","author":"Li Yong","year":"2023","unstructured":"Yong Li, Yuanzhi Wang, and Zhen Cui. 2023. Decoupled multimodal distilling for emotion recognition. In CVPR, 6631\u20136640."},{"issue":"1","key":"e_1_3_1_30_2","first-page":"302","article-title":"Learning representations for facial actions from unlabeled videos","volume":"44","author":"Li Yong","year":"2020","unstructured":"Yong Li, Jiabei Zeng, and Shiguang Shan. 2020. Learning representations for facial actions from unlabeled videos. IEEE Transactions on Pattern Analysis and Machine Intelligence 44, 1 (2020), 302\u2013317.","journal-title":"IEEE Transactions on Pattern Analysis and Machine Intelligence"},{"key":"e_1_3_1_31_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.01118"},{"key":"e_1_3_1_32_2","doi-asserted-by":"publisher","DOI":"10.1109\/TASLP.2021.3065823"},{"key":"e_1_3_1_33_2","unstructured":"Zheng Lian Haiyang Sun Licai Sun Jiangyan Yi Bin Liu and Jianhua Tao. 2024. AffectGPT: Dataset and framework for explainable multimodal emotion recognition. arXiv:2407.07653."},{"key":"e_1_3_1_34_2","first-page":"8148","volume-title":"ICCV","author":"Liang Tao","year":"2021","unstructured":"Tao Liang, Guosheng Lin, Lei Feng, Yan Zhang, and Fengmao Lv. 2021. Attention is not enough: Mitigating the distribution discrepancy in asynchronous multimodal sequence fusion. In ICCV, 8148\u20138156."},{"key":"e_1_3_1_35_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52729.2023.00228"},{"issue":"3","key":"e_1_3_1_36_2","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1017\/S0140525X11000446","article-title":"The brain basis of emotion: A meta-analytic review","volume":"35","author":"Lindquist Kristen A.","year":"2012","unstructured":"Kristen A. Lindquist, Tor D. Wager, Hedy Kober, Eliza Bliss-Moreau, and Lisa Feldman Barrett. 2012. The brain basis of emotion: A meta-analytic review. Behavioral and Brain Sciences 35, 3 (2012), 121\u2013143.","journal-title":"Behavioral and Brain Sciences"},{"key":"e_1_3_1_37_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2023.110372"},{"key":"e_1_3_1_38_2","doi-asserted-by":"crossref","unstructured":"Zhun Liu Ying Shen Varun Bharadhwaj Lakshminarasimhan Paul Pu Liang Amir Zadeh and Louis-Philippe Morency. 2018. Efficient low-rank multimodal fusion with modality-specific factors. arXiv:1806.00064.","DOI":"10.18653\/v1\/P18-1209"},{"key":"e_1_3_1_39_2","unstructured":"Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv:1711.05101."},{"key":"e_1_3_1_40_2","first-page":"2554","volume-title":"CVPR","author":"Lv Fengmao","year":"2021","unstructured":"Fengmao Lv, Xiang Chen, Yanyong Huang, Lixin Duan, and Guosheng Lin. 2021. Progressive modality reinforcement for human multimodal emotion recognition from unaligned multimodal sequences. In CVPR, 2554\u20132562."},{"key":"e_1_3_1_41_2","article-title":"Multimodal information bottleneck: Learning minimal sufficient unimodal and multimodal representations","volume":"25","author":"Mai Sijie","year":"2022","unstructured":"Sijie Mai, Ying Zeng, and Haifeng Hu. 2022. Multimodal information bottleneck: Learning minimal sufficient unimodal and multimodal representations. IEEE Transactions on Multimedia 25 (2022).","journal-title":"IEEE Transactions on Multimedia"},{"key":"e_1_3_1_42_2","doi-asserted-by":"publisher","DOI":"10.1080\/02699930802204677"},{"issue":"45","key":"e_1_3_1_43_2","doi-asserted-by":"crossref","first-page":"16518","DOI":"10.1073\/pnas.0507650102","article-title":"Rapid perceptual integration of facial expression and emotional body language","volume":"102","author":"Meeren Hanneke K. M.","year":"2005","unstructured":"Hanneke K. M. Meeren, Corn\u00e9 C. R. J. van Heijnsbergen, and Beatrice de Gelder. 2005. Rapid perceptual integration of facial expression and emotional body language. PNAS 102, 45 (2005), 16518\u201316523.","journal-title":"PNAS"},{"key":"e_1_3_1_44_2","first-page":"2227","volume-title":"ICASSP","author":"Mirsamadi Seyedmahdad","year":"2017","unstructured":"Seyedmahdad Mirsamadi, Emad Barsoum, and Cha Zhang. 2017. Automatic speech emotion recognition using recurrent neural networks with local attention. In ICASSP. IEEE, 2227\u20132231."},{"key":"e_1_3_1_45_2","doi-asserted-by":"crossref","first-page":"1359","DOI":"10.1609\/aaai.v34i02.5492","article-title":"M3er: Multiplicative multimodal emotion recognition using facial, textual, and speech cues","volume":"34","author":"Mittal Trisha","year":"2020","unstructured":"Trisha Mittal, Uttaran Bhattacharya, Rohan Chandra, Aniket Bera, and Dinesh Manocha. 2020. M3er: Multiplicative multimodal emotion recognition using facial, textual, and speech cues. In AAAI, Vol. 34, 1359\u20131367.","journal-title":"AAAI"},{"key":"e_1_3_1_46_2","first-page":"14234","volume-title":"CVPR","author":"Mittal Trisha","year":"2020","unstructured":"Trisha Mittal, Pooja Guhan, Uttaran Bhattacharya, Rohan Chandra, Aniket Bera, and Dinesh Manocha. 2020. Emoticon: Context-aware multimodal emotion recognition using Frege\u2019s principle. In CVPR, 14234\u201314243."},{"key":"e_1_3_1_47_2","first-page":"14200","article-title":"Attention bottlenecks for multimodal fusion","volume":"34","author":"Nagrani Arsha","year":"2021","unstructured":"Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun. 2021. Attention bottlenecks for multimodal fusion. In NeurIPS, Vol. 34, 14200\u201314213.","journal-title":"NeurIPS"},{"key":"e_1_3_1_48_2","first-page":"8200","volume-title":"IROS","author":"Narayanan Venkatraman","year":"2020","unstructured":"Venkatraman Narayanan, Bala Murali Manoghar, Vishnu Sashank Dorbala, Dinesh Manocha, and Aniket Bera. 2020. Proxemo: Gait-based emotion learning and multi-view proxemic fusion for socially-aware robot navigation. In IROS. IEEE, 8200\u20138207."},{"key":"e_1_3_1_49_2","first-page":"669","volume-title":"ACM MM","author":"Pantic Maja","year":"2005","unstructured":"Maja Pantic, Nicu Sebe, Jeffrey F. Cohn, and Thomas Huang. 2005. Affective multimodal human-computer interaction. In ACM MM, 669\u2013676."},{"key":"e_1_3_1_50_2","article-title":"Deep contextualized word representations","author":"Peters Matthew E.","year":"2018","unstructured":"Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv:1802.05365.","journal-title":"arXiv:"},{"key":"e_1_3_1_51_2","first-page":"439","volume-title":"ICDM","author":"Poria Soujanya","year":"2016","unstructured":"Soujanya Poria, Iti Chaturvedi, Erik Cambria, and Amir Hussain. 2016. Convolutional MKL based multimodal emotion recognition and sentiment analysis. In ICDM. IEEE, 439\u2013448."},{"key":"e_1_3_1_52_2","first-page":"II","volume-title":"ICASSP","author":"Schuller Bj\u00f6rn","year":"2003","unstructured":"Bj\u00f6rn Schuller, Gerhard Rigoll, and Manfred Lang. 2003. Hidden Markov model-based speech emotion recognition. In ICASSP, Vol. 2. IEEE, II\u20131."},{"key":"e_1_3_1_53_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2023.111149"},{"key":"e_1_3_1_54_2","article-title":"Dynamic causal disentanglement model for dialogue emotion detection","volume":"58","author":"Su Yuting","year":"2024","unstructured":"Yuting Su, Yichen Wei, Weizhi Nie, Sicheng Zhao, and Anan Liu. 2024. Dynamic causal disentanglement model for dialogue emotion detection. IEEE Transactions on Affective Computing 58 (2024).","journal-title":"IEEE Transactions on Affective Computing"},{"key":"e_1_3_1_55_2","doi-asserted-by":"crossref","unstructured":"Hao Tan and Mohit Bansal. 2019. Lxmert: Learning cross-modality encoder representations from transformers. arXiv:1908.07490. Retrieved from https:\/\/arxiv.org\/abs\/1908.07490","DOI":"10.18653\/v1\/D19-1514"},{"key":"e_1_3_1_56_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.procs.2017.05.025"},{"key":"e_1_3_1_57_2","first-page":"436","volume-title":"ECCV","author":"Tian Yapeng","year":"2020","unstructured":"Yapeng Tian, Dingzeyu Li, and Chenliang Xu. 2020. Unified multisensory perception: Weakly-supervised audio-visual video parsing. In ECCV. Springer, 436\u2013454."},{"key":"e_1_3_1_58_2","first-page":"6558","article-title":"Multimodal transformer for unaligned multimodal language sequences","volume":"2019","author":"Tsai Yao-Hung Hubert","year":"2019","unstructured":"Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J. Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019. Multimodal transformer for unaligned multimodal language sequences. In ACL Proceedings, Vol. 2019. NIH Public Access, 6558.","journal-title":"In ACL Proceedings"},{"key":"e_1_3_1_59_2","unstructured":"Yao-Hung Hubert Tsai Paul Pu Liang Amir Zadeh Louis-Philippe Morency and Ruslan Salakhutdinov. 2018. Learning factorized multimodal representations. arXiv:1806.06176."},{"issue":"11","key":"e_1_3_1_60_2","first-page":"2579","article-title":"Visualizing data using t-SNE","volume":"9","author":"Van der Maaten Laurens","year":"2008","unstructured":"Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of Machine Learning Research 9, 11 (2008), 2579\u20132605.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_1_61_2","first-page":"1","article-title":"Attention is all you need","volume":"30","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, \u0141ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS, Vol. 30, 1\u201311.","journal-title":"NeurIPS"},{"key":"e_1_3_1_62_2","first-page":"7216","volume-title":"AAAI","volume":"33","author":"Wang Yansen","year":"2019","unstructured":"Yansen Wang, Ying Shen, Zhun Liu, Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. 2019. Words can shift: Dynamically adjusting word representations using nonverbal behaviors. In AAAI, Vol. 33, 7216\u20137223."},{"key":"e_1_3_1_63_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.inffus.2022.03.009"},{"key":"e_1_3_1_64_2","first-page":"10941","volume-title":"CVPR","author":"Wei Xi","year":"2020","unstructured":"Xi Wei, Tianzhu Zhang, Yan Li, Yongdong Zhang, and Feng Wu. 2020. Multi-modality cross attention network for image and sentence matching. In CVPR, 10941\u201310950."},{"key":"e_1_3_1_65_2","first-page":"6269","volume-title":"ICASSP","author":"Wu Wen","year":"2021","unstructured":"Wen Wu, Chao Zhang, and Philip C. Woodland. 2021. Emotion recognition by fusing time synchronous and time asynchronous representations. In ICASSP. IEEE, 6269\u20136273."},{"key":"e_1_3_1_66_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01936"},{"key":"e_1_3_1_67_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.knosys.2023.110370"},{"key":"e_1_3_1_68_2","first-page":"799","volume-title":"MICCAI\u201819","author":"Yang Huzheng","year":"2019","unstructured":"Huzheng Yang, Xiaoxiao Li, Yifan Wu, Siyi Li, Su Lu, James S. Duncan, James C. Gee, and Shi Gu. 2019. Interpretable multimodality embedding of cerebral cortex using attention graph network for identifying bipolar disorder. In MICCAI\u201819. Springer, 799\u2013807."},{"key":"e_1_3_1_69_2","doi-asserted-by":"crossref","first-page":"10790","DOI":"10.1609\/aaai.v35i12.17289","article-title":"Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis","volume":"35","author":"Yu Wenmeng","year":"2021","unstructured":"Wenmeng Yu, Hua Xu, Ziqi Yuan, and Jiele Wu. 2021. Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis. In AAAI, Vol. 35, 10790\u201310797.","journal-title":"AAAI"},{"key":"e_1_3_1_70_2","doi-asserted-by":"publisher","DOI":"10.1121\/1.2935783"},{"key":"e_1_3_1_71_2","first-page":"4400","volume-title":"ACM MM","author":"Yuan Ziqi","year":"2021","unstructured":"Ziqi Yuan, Wei Li, Hua Xu, and Wenmeng Yu. 2021. Transformer-based feature reconstruction network for robust multimodal sentiment analysis. In ACM MM, 4400\u20134407."},{"key":"e_1_3_1_72_2","doi-asserted-by":"crossref","unstructured":"Amir Zadeh Minghai Chen Soujanya Poria Erik Cambria and Louis-Philippe Morency. 2017. Tensor fusion network for multimodal sentiment analysis. arXiv:1707.07250.","DOI":"10.18653\/v1\/D17-1115"},{"key":"e_1_3_1_73_2","doi-asserted-by":"publisher","DOI":"10.1109\/MIS.2016.94"},{"key":"e_1_3_1_74_2","first-page":"2236","volume-title":"ACL","author":"Zadeh AmirAli Bagher","year":"2018","unstructured":"AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018. Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph. In ACL, 2236\u20132246."},{"key":"e_1_3_1_75_2","first-page":"2428","volume-title":"CVPR","author":"Zhang Wei","year":"2022","unstructured":"Wei Zhang, Feng Qiu, Suzhen Wang, Hao Zeng, Zhimeng Zhang, Rudong An, Bowen Ma, and Yu Ding. 2022. Transformer-based multimodal information fusion for facial expression analysis. In CVPR, 2428\u20132437."},{"key":"e_1_3_1_76_2","doi-asserted-by":"crossref","first-page":"9100","DOI":"10.1609\/aaai.v36i8.20895","article-title":"Tailor versatile multi-modal learning for multi-label emotion recognition","volume":"36","author":"Zhang Yi","year":"2022","unstructured":"Yi Zhang, Mingyuan Chen, Jundong Shen, and Chongjun Wang. 2022. Tailor versatile multi-modal learning for multi-label emotion recognition. In AAAI, Vol. 36, 9100\u20139108.","journal-title":"AAAI"},{"key":"e_1_3_1_77_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01342"},{"key":"e_1_3_1_78_2","doi-asserted-by":"crossref","unstructured":"Jinming Zhao Tenggan Zhang Jingwen Hu Yuchen Liu Qin Jin Xinchao Wang and Haizhou Li. 2022. M3ED: Multi-modal multi-scene multi-label emotional dialogue database. arXiv:2205.10237.","DOI":"10.18653\/v1\/2022.acl-long.391"},{"key":"e_1_3_1_79_2","article-title":"Emotional chatting machine: Emotional conversation generation with internal and external memory","volume":"32","author":"Zhou Hao","year":"2018","unstructured":"Hao Zhou, Minlie Huang, Tianyang Zhang, Xiaoyan Zhu, and Bing Liu. 2018. Emotional chatting machine: Emotional conversation generation with internal and external memory. In AAAI, Vol. 32.","journal-title":"AAAI"},{"issue":"24","key":"e_1_3_1_80_2","doi-asserted-by":"crossref","first-page":"2200481","DOI":"10.1002\/adma.202200481","article-title":"A heterogeneously integrated spiking neuron array for multimode-fused perception and object classification","volume":"34","author":"Zhu Jiaxue","year":"2022","unstructured":"Jiaxue Zhu, Xumeng Zhang, Rui Wang, Ming Wang, Pei Chen, Lingli Cheng, Zuheng Wu, Yongzhou Wang, Qi Liu, and Ming Liu. 2022. A heterogeneously integrated spiking neuron array for multimode-fused perception and object classification. Advanced Materials 34, 24 (2022), 2200481.","journal-title":"Advanced Materials"},{"issue":"4","key":"e_1_3_1_81_2","doi-asserted-by":"crossref","first-page":"331","DOI":"10.1097\/WCO.0b013e32832d95db","article-title":"Receptor mapping: architecture of the human cerebral cortex","volume":"22","author":"Zilles Karl","year":"2009","unstructured":"Karl Zilles and Katrin Amunts. 2009. Receptor mapping: architecture of the human cerebral cortex. Current Opinion in Neurology 22, 4 (2009), 331\u2013339.","journal-title":"Current Opinion in Neurology"}],"container-title":["ACM Transactions on Multimedia Computing, Communications, and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3711865","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3711865","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T01:19:15Z","timestamp":1750295955000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3711865"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,3,10]]},"references-count":80,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2025,4,30]]}},"alternative-id":["10.1145\/3711865"],"URL":"https:\/\/doi.org\/10.1145\/3711865","relation":{},"ISSN":["1551-6857","1551-6865"],"issn-type":[{"value":"1551-6857","type":"print"},{"value":"1551-6865","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,3,10]]},"assertion":[{"value":"2024-04-30","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-12-13","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-03-10","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}