{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,13]],"date-time":"2026-11-13T07:03:45Z","timestamp":1794553425957,"version":"4.9.1"},"reference-count":47,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2023,11,8]],"date-time":"2023-11-08T00:00:00Z","timestamp":1699401600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["U1936203"],"award-info":[{"award-number":["U1936203"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100004543","name":"Program of China Scholarship Council","doi-asserted-by":"crossref","award":["202106220122"],"award-info":[{"award-number":["202106220122"]}],"id":[{"id":"10.13039\/501100004543","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2024,3,31]]},"abstract":"<jats:p>\n                    Multimodal information (e.g., visual, acoustic, and textual) has been widely used to enhance representation learning for micro-video recommendation. For integrating multimodal information into a joint representation of micro-video, multimodal fusion plays a vital role in the existing micro-video recommendation approaches. However, the static multimodal fusion used in previous studies is insufficient to model the various relationships among multimodal information of different micro-videos. In this article, we develop a novel meta-learning-based multimodal fusion framework called\n                    <jats:italic>Meta Multimodal Fusion<\/jats:italic>\n                    (MetaMMF), which dynamically assigns parameters to the multimodal fusion function for each micro-video during its representation learning. Specifically, MetaMMF regards the multimodal fusion of each micro-video as an independent task. Based on the meta information extracted from the multimodal features of the input task, MetaMMF parameterizes a neural network as the item-specific fusion function via a meta learner. We perform extensive experiments on three benchmark datasets, demonstrating the significant improvements over several state-of-the-art multimodal recommendation models, like MMGCN, LATTICE, and InvRL. Furthermore, we lighten our model by adopting canonical polyadic decomposition to improve the training efficiency, and validate its effectiveness through experimental results. Codes are available at\n                    <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" xlink:href=\"https:\/\/github.com\/hanliu95\/MetaMMF\">https:\/\/github.com\/hanliu95\/MetaMMF<\/jats:ext-link>\n                    .\n                  <\/jats:p>","DOI":"10.1145\/3617827","type":"journal-article","created":{"date-parts":[[2023,8,30]],"date-time":"2023-08-30T06:33:16Z","timestamp":1693377196000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":34,"title":["Dynamic Multimodal Fusion via Meta-Learning Towards Micro-Video Recommendation"],"prefix":"10.1145","volume":"42","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-5202-8740","authenticated-orcid":false,"given":"Han","family":"Liu","sequence":"first","affiliation":[{"name":"Shandong University, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1791-3159","authenticated-orcid":false,"given":"Yinwei","family":"Wei","sequence":"additional","affiliation":[{"name":"National University of Singapore, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4547-3982","authenticated-orcid":false,"given":"Fan","family":"Liu","sequence":"additional","affiliation":[{"name":"National University of Singapore, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5199-1428","authenticated-orcid":false,"given":"Wenjie","family":"Wang","sequence":"additional","affiliation":[{"name":"National University of Singapore, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1476-0273","authenticated-orcid":false,"given":"Liqiang","family":"Nie","sequence":"additional","affiliation":[{"name":"Harbin Institute of Technology (Shenzhen), China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6097-7807","authenticated-orcid":false,"given":"Tat-Seng","family":"Chua","sequence":"additional","affiliation":[{"name":"National University of Singapore, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,11,8]]},"reference":[{"key":"e_1_3_2_2_2","first-page":"1","volume-title":"ICLR","author":"Arora Sanjeev","year":"2017","unstructured":"Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A simple but tough-to-beat baseline for sentence embeddings. In ICLR. 1\u201316."},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2018.2798607"},{"key":"e_1_3_2_4_2","first-page":"805","article-title":"Heterogeneous hierarchical feature aggregation network for personalized micro-video recommendation","volume":"24","author":"Cai Desheng","year":"2021","unstructured":"Desheng Cai, Shengsheng Qian, Quan Fang, and Changsheng Xu. 2021. Heterogeneous hierarchical feature aggregation network for personalized micro-video recommendation. TMM 24 (2021), 805\u2013818.","journal-title":"TMM"},{"key":"e_1_3_2_5_2","first-page":"335","volume-title":"SIGIR","author":"Chen Jingyuan","year":"2017","unstructured":"Jingyuan Chen, Hanwang Zhang, Xiangnan He, Liqiang Nie, Wei Liu, and Tat-Seng Chua. 2017. Attentive collaborative filtering: Multimedia recommendation with item-and component-level attention. In SIGIR. 335\u2013344."},{"issue":"1","key":"e_1_3_2_6_2","first-page":"32","article-title":"Emotion recognition in human-computer interaction","volume":"18","author":"Cowie Roddy","year":"2001","unstructured":"Roddy Cowie, Ellen Douglas-Cowie, Nicolas Tsapatsoulis, George Votsis, Stefanos Kollias, Winfried Fellenz, and John G. Taylor. 2001. Emotion recognition in human-computer interaction. SPM 18, 1 (2001), 32\u201380.","journal-title":"SPM"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/3460198"},{"key":"e_1_3_2_8_2","first-page":"3469","volume-title":"MM","author":"Du Xiaoyu","year":"2020","unstructured":"Xiaoyu Du, Xiang Wang, Xiangnan He, Zechao Li, Jinhui Tang, and Tat-Seng Chua. 2020. How to learn item representation for cold-start multimedia recommendation?. In MM. 3469\u20133477."},{"key":"e_1_3_2_9_2","first-page":"619","volume-title":"MM","author":"Du Xiaoyu","year":"2022","unstructured":"Xiaoyu Du, Zike Wu, Fuli Feng, Xiangnan He, and Jinhui Tang. 2022. Invariant representation learning for multimedia recommendation. In MM. 619\u2013628."},{"key":"e_1_3_2_10_2","first-page":"1126","volume-title":"ICML","author":"Finn Chelsea","year":"2017","unstructured":"Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML. 1126\u20131135."},{"key":"e_1_3_2_11_2","first-page":"1142","volume-title":"MM","author":"Fu Yuqian","year":"2020","unstructured":"Yuqian Fu, Li Zhang, Junke Wang, Yanwei Fu, and Yu-Gang Jiang. 2020. Depth guided adaptive meta-fusion network for few-shot video recognition. In MM. 1142\u20131151."},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.5555\/1953048.2021071"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/3295822"},{"key":"e_1_3_2_14_2","first-page":"1024","volume-title":"NeurIPS","author":"Hamilton Will","year":"2017","unstructured":"Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive representation learning on large graphs. In NeurIPS. 1024\u20131034."},{"key":"e_1_3_2_15_2","first-page":"770","volume-title":"CVPR","author":"He Kaiming","year":"2016","unstructured":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR. 770\u2013778."},{"key":"e_1_3_2_16_2","first-page":"144","volume-title":"AAAI","author":"He Ruining","year":"2016","unstructured":"Ruining He and Julian McAuley. 2016. VBPR: Visual bayesian personalized ranking from implicit feedback. In AAAI. 144\u2013150."},{"key":"e_1_3_2_17_2","first-page":"639","volume-title":"SIGIR","author":"He Xiangnan","year":"2020","unstructured":"Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. 2020. Lightgcn: Simplifying and powering graph convolution network for recommendation. In SIGIR. 639\u2013648."},{"key":"e_1_3_2_18_2","first-page":"131","volume-title":"ICASSP","author":"Hershey Shawn","year":"2017","unstructured":"Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et\u00a0al. 2017. CNN architectures for large-scale audio classification. In ICASSP. 131\u2013135."},{"key":"e_1_3_2_19_2","first-page":"1440","volume-title":"ICIP","author":"Hu Xinxin","year":"2019","unstructured":"Xinxin Hu, Kailun Yang, Lei Fei, and Kaiwei Wang. 2019. Acnet: Attention based network to exploit complementary features for rgbd semantic segmentation. In ICIP. 1440\u20131444."},{"key":"e_1_3_2_20_2","first-page":"2333","volume-title":"CIKM","author":"Huang Po-Sen","year":"2013","unstructured":"Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning deep structured semantic models for web search using click through data. In CIKM. 2333\u20132338."},{"key":"e_1_3_2_21_2","first-page":"3487","volume-title":"MM","author":"Jiang Hao","year":"2020","unstructured":"Hao Jiang, Wenjie Wang, Yinwei Wei, Zan Gao, Yinglong Wang, and Liqiang Nie. 2020. What aspect do you like: Multi-scale time-aware user interest modeling for micro-video recommendation. In MM. 3487\u20133495."},{"key":"e_1_3_2_22_2","volume-title":"ICLR","author":"Kingma Diederik P","year":"2015","unstructured":"Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR."},{"issue":"1","key":"e_1_3_2_23_2","first-page":"333","article-title":"Multimedia classification and event detection using double fusion","volume":"71","author":"Lan Zhen-zhong","year":"2014","unstructured":"Zhen-zhong Lan, Lei Bao, Shoou-I Yu, Wei Liu, and Alexander G Hauptmann. 2014. Multimedia classification and event detection using double fusion. MTA 71, 1 (2014), 333\u2013347.","journal-title":"MTA"},{"key":"e_1_3_2_24_2","first-page":"1073","volume-title":"KDD","author":"Lee Hoyeop","year":"2019","unstructured":"Hoyeop Lee, Jinbae Im, Seongwon Jang, Hyunsouk Cho, and Sehee Chung. 2019. Melu: Meta-learned user preference estimator for cold-start recommendation. In KDD. 1073\u20131082."},{"key":"e_1_3_2_25_2","first-page":"1","article-title":"Disentangled multimodal representation learning for recommendation","author":"Liu Fan","year":"2022","unstructured":"Fan Liu, Zhiyong Cheng, Huilin Chen, Anan Liu, Liqiang Nie, and Mohan Kankanhalli. 2022. Disentangled multimodal representation learning for recommendation. TMM (2022), 1\u201311.","journal-title":"TMM"},{"key":"e_1_3_2_26_2","first-page":"1526","volume-title":"MM","author":"Liu Fan","year":"2019","unstructured":"Fan Liu, Zhiyong Cheng, Changchang Sun, Yinglong Wang, Liqiang Nie, and Mohan Kankanhalli. 2019. User diverse preference modeling by multimodal attentive metric learning. In MM. 1526\u20131534."},{"key":"e_1_3_2_27_2","first-page":"1296","volume-title":"WWW","author":"Liu Fan","year":"2021","unstructured":"Fan Liu, Zhiyong Cheng, Lei Zhu, Zan Gao, and Liqiang Nie. 2021. Interest-aware message-passing GCN for recommendation. In WWW. 1296\u20131305."},{"key":"e_1_3_2_28_2","first-page":"3449","volume-title":"IJCAI","author":"Liu Han","year":"2018","unstructured":"Han Liu, Xiangnan He, Fuli Feng, Liqiang Nie, Rui Liu, and Hanwang Zhang. 2018. Discrete factorization machines for fast feature-based recommendation. In IJCAI. 3449\u20133455."},{"key":"e_1_3_2_29_2","article-title":"HS-GCN: Hamming spatial graph convolutional networks for recommendation","author":"Liu Han","year":"2022","unstructured":"Han Liu, Yinwei Wei, Jianhua Yin, and Liqiang Nie. 2022. HS-GCN: Hamming spatial graph convolutional networks for recommendation. TKDE (2022).","journal-title":"TKDE"},{"key":"e_1_3_2_30_2","first-page":"3020","volume-title":"WWW","author":"Liu Shang","year":"2019","unstructured":"Shang Liu, Zhenzhong Chen, Hongyi Liu, and Xinghai Hu. 2019. User-video co-attention network for personalized micro-video recommendation. In WWW. 3020\u20133026."},{"issue":"3","key":"e_1_3_2_31_2","first-page":"2991","article-title":"LGA: Latent genre aware micro-video recommendation on social media","volume":"77","author":"Ma Jingwei","year":"2018","unstructured":"Jingwei Ma, Guang Li, Mingyang Zhong, Xin Zhao, Lei Zhu, and Xue Li. 2018. LGA: Latent genre aware micro-video recommendation on social media. MTA 77, 3 (2018), 2991\u20133008.","journal-title":"MTA"},{"key":"e_1_3_2_32_2","first-page":"2130","volume-title":"ICASSP","author":"Mroueh Youssef","year":"2015","unstructured":"Youssef Mroueh, Etienne Marcheret, and Vaibhava Goel. 2015. Deep multimodal learning for audio-visual speech recognition. In ICASSP. IEEE, 2130\u20132134."},{"key":"e_1_3_2_33_2","first-page":"2554","volume-title":"ICML","author":"Munkhdalai Tsendsuren","year":"2017","unstructured":"Tsendsuren Munkhdalai and Hong Yu. 2017. Meta networks. In ICML. 2554\u20132563."},{"key":"e_1_3_2_34_2","first-page":"689","volume-title":"ICML","author":"Ngiam Jiquan","year":"2011","unstructured":"Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y. Ng. 2011. Multimodal deep learning. In ICML. 689\u2013696."},{"key":"e_1_3_2_35_2","first-page":"452","volume-title":"UAI","author":"Rendle Steffen","year":"2009","unstructured":"Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2009. BPR: Bayesian personalized ranking from implicit feedback. In UAI. 452\u2013461."},{"key":"e_1_3_2_36_2","first-page":"1842","volume-title":"ICML","author":"Santoro Adam","year":"2016","unstructured":"Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. 2016. Meta-learning with memory-augmented neural networks. In ICML. 1842\u20131850."},{"key":"e_1_3_2_37_2","first-page":"4080","volume-title":"NeurIPS","author":"Snell Jake","year":"2017","unstructured":"Jake Snell, Kevin Swersky, and Richard Zemel. 2017. Prototypical networks for few-shot learning. In NeurIPS. 4080\u20134090."},{"issue":"11","key":"e_1_3_2_38_2","article-title":"Visualizing data using t-SNE.","volume":"9","author":"Maaten Laurens Van der","year":"2008","unstructured":"Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. JMLR 9, 11 (2008).","journal-title":"JMLR"},{"key":"e_1_3_2_39_2","first-page":"6907","volume-title":"NeurIPS","author":"Vartak Manasi","year":"2017","unstructured":"Manasi Vartak, Arvind Thiagarajan, Conrado Miranda, Jeshua Bratman, and Hugo Larochelle. 2017. A meta-learning perspective on cold-start recommendations for items. In NeurIPS. 6907\u20136917."},{"key":"e_1_3_2_40_2","first-page":"1","volume-title":"ICLR","author":"Veli\u010dkovi\u0107 Petar","year":"2018","unstructured":"Petar Veli\u010dkovi\u0107, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Li\u00f2, and Yoshua Bengio. 2018. Graph attention networks. In ICLR. 1\u201312."},{"key":"e_1_3_2_41_2","first-page":"165","volume-title":"SIGIR","author":"Wang Xiang","year":"2019","unstructured":"Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua. 2019. Neural graph collaborative filtering. In SIGIR. 165\u2013174."},{"key":"e_1_3_2_42_2","first-page":"1","volume-title":"ICME","author":"Wang Yifan","year":"2021","unstructured":"Yifan Wang, Xing Xu, Wei Yu, Ruicong Xu, Zuo Cao, and Heng Tao Shen. 2021. Combine early and late fusion together: A hybrid fusion framework for image-text matching. In ICME. 1\u20136."},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2019.2923608"},{"key":"e_1_3_2_44_2","first-page":"3541","volume-title":"MM","author":"Wei Yinwei","year":"2020","unstructured":"Yinwei Wei, Xiang Wang, Liqiang Nie, Xiangnan He, and Tat-Seng Chua. 2020. Graph-refined convolutional network for multimedia recommendation with implicit feedback. In MM. 3541\u20133549."},{"key":"e_1_3_2_45_2","first-page":"1437","volume-title":"MM","author":"Wei Yinwei","year":"2019","unstructured":"Yinwei Wei, Xiang Wang, Liqiang Nie, Xiangnan He, Richang Hong, and Tat-Seng Chua. 2019. MMGCN: Multi-modal graph convolution network for personalized recommendation of micro-video. In MM. 1437\u20131445."},{"issue":"5","key":"e_1_3_2_46_2","first-page":"1","article-title":"Scaling learning algorithms towards AI","volume":"34","author":"Yoshua Bengio","year":"2007","unstructured":"Bengio Yoshua and Lecun Yann. 2007. Scaling learning algorithms towards AI. Large-scale Kernel Machines 34, 5 (2007), 1\u201341.","journal-title":"Large-scale Kernel Machines"},{"key":"e_1_3_2_47_2","first-page":"3872","volume-title":"MM","author":"Zhang Jinghao","year":"2021","unstructured":"Jinghao Zhang, Yanqiao Zhu, Qiang Liu, Shu Wu, Shuhui Wang, and Liang Wang. 2021. Mining latent structures for multimedia recommendation. In MM. 3872\u20133880."},{"key":"e_1_3_2_48_2","first-page":"1167","volume-title":"SIGIR","author":"Zhu Yongchun","year":"2021","unstructured":"Yongchun Zhu, Ruobing Xie, Fuzhen Zhuang, Kaikai Ge, Ying Sun, Xu Zhang, Leyu Lin, and Juan Cao. 2021. Learning to warm up cold item embeddings for cold-start recommendation with meta scaling and shifting networks. In SIGIR. 1167\u20131176."}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3617827","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3617827","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T12:37:57Z","timestamp":1750163877000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3617827"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,11,8]]},"references-count":47,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2024,3,31]]}},"alternative-id":["10.1145\/3617827"],"URL":"https:\/\/doi.org\/10.1145\/3617827","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"value":"1046-8188","type":"print"},{"value":"1558-2868","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,11,8]]},"assertion":[{"value":"2022-12-28","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-08-14","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-11-08","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}