{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,19]],"date-time":"2026-06-19T15:45:43Z","timestamp":1781883943727,"version":"3.54.5"},"publisher-location":"California","reference-count":0,"publisher":"International Joint Conferences on Artificial Intelligence Organization","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,7]]},"abstract":"<jats:p>As deep learning advances, there is an ever-growing demand for models capable of synthesizing information from multi-modal resources to address the complex tasks raised from real-life applications. Recently, many large multi-modal datasets have been collected, on which researchers actively explore different methods of fusing multi-modal information. However, little attention has been paid to quantifying the contribution of different modalities within the proposed models. \n\nIn this paper, we propose the SHapley vAlue-based PErceptual (SHAPE) scores that measure the marginal contribution of individual modalities and the degree of cooperation across modalities. Using these scores, we systematically evaluate different fusion methods on different multi-modal datasets for different tasks. Our experiments suggest that for some tasks where different modalities are complementary, the multi-modal models still tend to use the dominant modality alone and ignore the cooperation across modalities. On the other hand, models learn to exploit cross-modal cooperation when different modalities are indispensable for the task. In this case, the scores indicate it is better to fuse different modalities at relatively early stages. We hope our scores can help improve the understanding of how the present multi-modal models operate on different modalities and encourage more sophisticated methods of integrating multiple modalities.<\/jats:p>","DOI":"10.24963\/ijcai.2022\/425","type":"proceedings-article","created":{"date-parts":[[2022,7,15]],"date-time":"2022-07-15T22:55:56Z","timestamp":1657925756000},"page":"3064-3070","source":"Crossref","is-referenced-by-count":11,"title":["SHAPE: An Unified Approach to Evaluate the Contribution and Cooperation of Individual Modalities"],"prefix":"10.24963","author":[{"given":"Pengbo","family":"Hu","sequence":"first","affiliation":[{"name":"School of Mathematical Sciences, University of Science and Technology of China, Hefei, China"},{"name":"Shanghai Center for Brain Science and Brain-Inspired Technology, Shanghai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xingyu","family":"Li","sequence":"additional","affiliation":[{"name":"Shanghai Center for Brain Science and Brain-Inspired Technology, Shanghai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yi","family":"Zhou","sequence":"additional","affiliation":[{"name":"National Engineering Laboratory for Brain-inspired Intelligence Technology and Application, School of Information Science and Technology, University of Science and Technology of China, Hefei, China"},{"name":"Shanghai Center for Brain Science and Brain-Inspired Technology, Shanghai, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"10584","event":{"name":"Thirty-First International Joint Conference on Artificial Intelligence {IJCAI-22}","theme":"Artificial Intelligence","location":"Vienna, Austria","acronym":"IJCAI-2022","number":"31","sponsor":["International Joint Conferences on Artificial Intelligence Organization (IJCAI)"],"start":{"date-parts":[[2022,7,23]]},"end":{"date-parts":[[2022,7,29]]}},"container-title":["Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence"],"original-title":[],"deposited":{"date-parts":[[2022,7,18]],"date-time":"2022-07-18T07:09:39Z","timestamp":1658128179000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.ijcai.org\/proceedings\/2022\/425"}},"subtitle":[],"proceedings-subject":"Artificial Intelligence Research Articles","short-title":[],"issued":{"date-parts":[[2022,7]]},"references-count":0,"URL":"https:\/\/doi.org\/10.24963\/ijcai.2022\/425","relation":{},"subject":[],"published":{"date-parts":[[2022,7]]}}}