{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,17]],"date-time":"2026-04-17T16:31:21Z","timestamp":1776443481214,"version":"3.51.2"},"publisher-location":"New York, NY, USA","reference-count":73,"publisher":"ACM","license":[{"start":{"date-parts":[[2023,10,9]],"date-time":"2023-10-09T00:00:00Z","timestamp":1696809600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["1722822, 1750439"],"award-info":[{"award-number":["1722822, 1750439"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000002","name":"National Institutes of Health","doi-asserted-by":"publisher","award":["R01MH125740, R01MH132225, R01MH096951, R21MH130767"],"award-info":[{"award-number":["R01MH125740, R01MH132225, R01MH096951, R21MH130767"]}],"id":[{"id":"10.13039\/100000002","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000185","name":"Defense Advanced Research Projects Agency","doi-asserted-by":"publisher","award":["FA87502321015"],"award-info":[{"award-number":["FA87502321015"]}],"id":[{"id":"10.13039\/100000185","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100000006","name":"Office of Naval Research","doi-asserted-by":"publisher","award":["N000141812861, N000142312368"],"award-info":[{"award-number":["N000141812861, N000142312368"]}],"id":[{"id":"10.13039\/100000006","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2023,10,9]]},"DOI":"10.1145\/3577190.3614151","type":"proceedings-article","created":{"date-parts":[[2023,10,7]],"date-time":"2023-10-07T22:30:48Z","timestamp":1696717848000},"page":"425-435","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":5,"title":["Multimodal Fusion Interactions: A Study of Human and Automatic Quantification"],"prefix":"10.1145","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7768-3610","authenticated-orcid":false,"given":"Paul Pu","family":"Liang","sequence":"first","affiliation":[{"name":"Machine Learning Department, Carnegie Mellon University, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-8916-6667","authenticated-orcid":false,"given":"Yun","family":"Cheng","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3752-2756","authenticated-orcid":false,"given":"Ruslan","family":"Salakhutdinov","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6376-7696","authenticated-orcid":false,"given":"Louis-Philippe","family":"Morency","sequence":"additional","affiliation":[{"name":"Language Technologies Institute, Carnegie Mellon University, United States"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2023,10,9]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR42600.2020.01006"},{"key":"e_1_3_2_1_2_1","volume-title":"VQA: Visual Question Answering. International Journal of Computer Vision","author":"Agrawal Aishwarya","year":"2017","unstructured":"Aishwarya Agrawal , Jiasen Lu , Stanislaw Antol , Margaret Mitchell , C.\u00a0 Lawrence Zitnick , Devi Parikh , and Dhruv Batra . 2017 . VQA: Visual Question Answering. International Journal of Computer Vision (2017). Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C.\u00a0Lawrence Zitnick, Devi Parikh, and Dhruv Batra. 2017. VQA: Visual Question Answering. International Journal of Computer Vision (2017)."},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-14400-4_20"},{"key":"e_1_3_2_1_4_1","volume-title":"Co-training and expansion: Towards bridging theory and practice. Advances in neural information processing systems 17","author":"Balcan Maria-Florina","year":"2004","unstructured":"Maria-Florina Balcan , Avrim Blum , and Ke Yang . 2004. Co-training and expansion: Towards bridging theory and practice. Advances in neural information processing systems 17 ( 2004 ). Maria-Florina Balcan, Avrim Blum, and Ke Yang. 2004. Co-training and expansion: Towards bridging theory and practice. Advances in neural information processing systems 17 (2004)."},{"key":"e_1_3_2_1_5_1","volume-title":"Text and image: A critical introduction to the visual\/verbal divide","author":"Bateman John","unstructured":"John Bateman . 2014. Text and image: A critical introduction to the visual\/verbal divide . Routledge . John Bateman. 2014. Text and image: A critical introduction to the visual\/verbal divide. Routledge."},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1056\/NEJM200006223422506"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.3390\/e16042161"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/279943.279962"},{"key":"e_1_3_2_1_9_1","doi-asserted-by":"crossref","unstructured":"Santiago Castro Devamanyu Hazarika Ver\u00f3nica P\u00e9rez-Rosas Roger Zimmermann Rada Mihalcea and Soujanya Poria. 2019. Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper). In ACL. 4619\u20134629.  Santiago Castro Devamanyu Hazarika Ver\u00f3nica P\u00e9rez-Rosas Roger Zimmermann Rada Mihalcea and Soujanya Poria. 2019. Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper). In ACL. 4619\u20134629.","DOI":"10.18653\/v1\/P19-1455"},{"key":"e_1_3_2_1_10_1","volume-title":"Gene regulatory network inference from single-cell data using multivariate information measures. Cell systems 5, 3","author":"Chan E","year":"2017","unstructured":"Thalia\u00a0 E Chan , Michael\u00a0 PH Stumpf , and Ann\u00a0 C Babtie . 2017. Gene regulatory network inference from single-cell data using multivariate information measures. Cell systems 5, 3 ( 2017 ), 251\u2013267. Thalia\u00a0E Chan, Michael\u00a0PH Stumpf, and Ann\u00a0C Babtie. 2017. Gene regulatory network inference from single-cell data using multivariate information measures. Cell systems 5, 3 (2017), 251\u2013267."},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.5555\/3023476.3023487"},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuroimage.2020.116699"},{"key":"e_1_3_2_1_13_1","first-page":"1","article-title":"CVXPY: A Python-embedded modeling language for convex optimization","volume":"17","author":"Diamond Steven","year":"2016","unstructured":"Steven Diamond and Stephen Boyd . 2016 . CVXPY: A Python-embedded modeling language for convex optimization . Journal of Machine Learning Research 17 , 83 (2016), 1 \u2013 5 . Steven Diamond and Stephen Boyd. 2016. CVXPY: A Python-embedded modeling language for convex optimization. Journal of Machine Learning Research 17, 83 (2016), 1\u20135.","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_3_2_1_14_1","volume-title":"A review and meta-analysis of multimodal affect detection systems. ACM computing surveys (CSUR) 47, 3","author":"D\u2019mello K","year":"2015","unstructured":"Sidney\u00a0 K D\u2019mello and Jacqueline Kory . 2015. A review and meta-analysis of multimodal affect detection systems. ACM computing surveys (CSUR) 47, 3 ( 2015 ), 1\u201336. Sidney\u00a0K D\u2019mello and Jacqueline Kory. 2015. A review and meta-analysis of multimodal affect detection systems. ACM computing surveys (CSUR) 47, 3 (2015), 1\u201336."},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1177\/1754073917696583"},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1063\/1.3638449"},{"key":"e_1_3_2_1_17_1","volume-title":"The development of infant discrimination of affect in multimodal and unimodal stimulation: The role of intersensory redundancy.Developmental psychology 43, 1","author":"Flom Ross","year":"2007","unstructured":"Ross Flom and Lorraine\u00a0 E Bahrick . 2007. The development of infant discrimination of affect in multimodal and unimodal stimulation: The role of intersensory redundancy.Developmental psychology 43, 1 ( 2007 ), 238. Ross Flom and Lorraine\u00a0E Bahrick. 2007. The development of infant discrimination of affect in multimodal and unimodal stimulation: The role of intersensory redundancy.Developmental psychology 43, 1 (2007), 238."},{"key":"e_1_3_2_1_18_1","volume-title":"Predictive learning via rule ensembles. The annals of applied statistics 2, 3","author":"Friedman H","year":"2008","unstructured":"Jerome\u00a0 H Friedman and Bogdan\u00a0 E Popescu . 2008. Predictive learning via rule ensembles. The annals of applied statistics 2, 3 ( 2008 ), 916\u2013954. Jerome\u00a0H Friedman and Bogdan\u00a0E Popescu. 2008. Predictive learning via rule ensembles. The annals of applied statistics 2, 3 (2008), 916\u2013954."},{"key":"e_1_3_2_1_19_1","unstructured":"Wendell\u00a0R Garner. 1962. Uncertainty and structure as psychological concepts. (1962).  Wendell\u00a0R Garner. 1962. Uncertainty and structure as psychological concepts. (1962)."},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.670"},{"key":"e_1_3_2_1_21_1","volume-title":"International Conference on Machine Learning. PMLR, 2376\u20132384","author":"Goyal Yash","year":"2019","unstructured":"Yash Goyal , Ziyan Wu , Jan Ernst , Dhruv Batra , Devi Parikh , and Stefan Lee . 2019 . Counterfactual visual explanations . In International Conference on Machine Learning. PMLR, 2376\u20132384 . Yash Goyal, Ziyan Wu, Jan Ernst, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Counterfactual visual explanations. In International Conference on Machine Learning. PMLR, 2376\u20132384."},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1103\/PhysRevE.87.012130"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"crossref","unstructured":"Md\u00a0Kamrul Hasan Wasifur Rahman AmirAli\u00a0Bagher Zadeh Jianyuan Zhong Md\u00a0Iftekhar Tanveer Louis-Philippe Morency and Mohammed\u00a0Ehsan Hoque. 2019. UR-FUNNY: A Multimodal Language Dataset for Understanding Humor. In EMNLP. 2046\u20132056.  Md\u00a0Kamrul Hasan Wasifur Rahman AmirAli\u00a0Bagher Zadeh Jianyuan Zhong Md\u00a0Iftekhar Tanveer Louis-Philippe Morency and Mohammed\u00a0Ehsan Hoque. 2019. UR-FUNNY: A Multimodal Language Dataset for Understanding Humor. In EMNLP. 2046\u20132056.","DOI":"10.18653\/v1\/D19-1211"},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"crossref","unstructured":"Jack Hessel and Lillian Lee. 2020. Does my multimodal model learn cross-modal interactions? It\u2019s harder to tell than you might think!. In EMNLP.  Jack Hessel and Lillian Lee. 2020. Does my multimodal model learn cross-modal interactions? It\u2019s harder to tell than you might think!. In EMNLP.","DOI":"10.18653\/v1\/2020.emnlp-main.62"},{"key":"e_1_3_2_1_25_1","unstructured":"Aleks Jakulin and Ivan Bratko. 2003. Quantifying and visualizing attribute interactions: An approach based on entropy. (2003).  Aleks Jakulin and Ivan Bratko. 2003. Quantifying and visualizing attribute interactions: An approach based on entropy. (2003)."},{"key":"e_1_3_2_1_26_1","volume-title":"International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=rylnK6VtDH","author":"Jayakumar M.","year":"2020","unstructured":"Siddhant\u00a0 M. Jayakumar , Wojciech\u00a0 M. Czarnecki , Jacob Menick , Jonathan Schwarz , Jack Rae , Simon Osindero , Yee\u00a0Whye Teh , Tim Harley , and Razvan Pascanu . 2020 . Multiplicative Interactions and Where to Find Them . In International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=rylnK6VtDH Siddhant\u00a0M. Jayakumar, Wojciech\u00a0M. Czarnecki, Jacob Menick, Jonathan Schwarz, Jack Rae, Simon Osindero, Yee\u00a0Whye Teh, Tim Harley, and Razvan Pascanu. 2020. Multiplicative Interactions and Where to Find Them. In International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=rylnK6VtDH"},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.215"},{"key":"e_1_3_2_1_28_1","volume-title":"International Conference on Learning Representations","author":"Kaushik Divyansh","year":"2019","unstructured":"Divyansh Kaushik , Eduard Hovy , and Zachary\u00a0 C Lipton . 2019 . Learning the difference that makes a difference with counterfactually-augmented data . International Conference on Learning Representations (2019). Divyansh Kaushik, Eduard Hovy, and Zachary\u00a0C Lipton. 2019. Learning the difference that makes a difference with counterfactually-augmented data. International Conference on Learning Representations (2019)."},{"key":"e_1_3_2_1_29_1","volume-title":"International Conference on Machine Learning. PMLR, 5583\u20135594","author":"Kim Wonjae","year":"2021","unstructured":"Wonjae Kim , Bokyung Son , and Ildoo Kim . 2021 . Vilt: Vision-and-language transformer without convolution or region supervision . In International Conference on Machine Learning. PMLR, 5583\u20135594 . Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021. Vilt: Vision-and-language transformer without convolution or region supervision. In International Conference on Machine Learning. PMLR, 5583\u20135594."},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-1469"},{"key":"e_1_3_2_1_31_1","volume-title":"An Information Decomposition Framework. arXiv preprint arXiv:2302.12247","author":"Liang Paul\u00a0Pu","year":"2023","unstructured":"Paul\u00a0Pu Liang , Yun Cheng , Xiang Fan , Chun\u00a0Kai Ling , Suzanne Nie , Richard Chen , Zihao Deng , Faisal Mahmood , Ruslan Salakhutdinov , and Louis-Philippe Morency . 2023. Quantifying & Modeling Feature Interactions : An Information Decomposition Framework. arXiv preprint arXiv:2302.12247 ( 2023 ). Paul\u00a0Pu Liang, Yun Cheng, Xiang Fan, Chun\u00a0Kai Ling, Suzanne Nie, Richard Chen, Zihao Deng, Faisal Mahmood, Ruslan Salakhutdinov, and Louis-Philippe Morency. 2023. Quantifying & Modeling Feature Interactions: An Information Decomposition Framework. arXiv preprint arXiv:2302.12247 (2023)."},{"key":"e_1_3_2_1_32_1","volume-title":"Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applications. arXiv preprint arXiv:2306.04539","author":"Liang Paul\u00a0Pu","year":"2023","unstructured":"Paul\u00a0Pu Liang , Chun\u00a0Kai Ling , Yun Cheng , Alex Obolenskiy , Yudong Liu , Rohan Pandey , Alex Wilf , Louis-Philippe Morency , and Ruslan Salakhutdinov . 2023. Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applications. arXiv preprint arXiv:2306.04539 ( 2023 ). Paul\u00a0Pu Liang, Chun\u00a0Kai Ling, Yun Cheng, Alex Obolenskiy, Yudong Liu, Rohan Pandey, Alex Wilf, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2023. Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applications. arXiv preprint arXiv:2306.04539 (2023)."},{"key":"e_1_3_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1014"},{"key":"e_1_3_2_1_34_1","volume-title":"MultiViz: Towards Visualizing and Understanding Multimodal Models. In International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=i2_TvOFmEml","author":"Liang Paul\u00a0Pu","year":"2023","unstructured":"Paul\u00a0Pu Liang , Yiwei Lyu , Gunjan Chhablani , Nihal Jain , Zihao Deng , Xingbo Wang , Louis-Philippe Morency , and Ruslan Salakhutdinov . 2023 . MultiViz: Towards Visualizing and Understanding Multimodal Models. In International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=i2_TvOFmEml Paul\u00a0Pu Liang, Yiwei Lyu, Gunjan Chhablani, Nihal Jain, Zihao Deng, Xingbo Wang, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2023. MultiViz: Towards Visualizing and Understanding Multimodal Models. In International Conference on Learning Representations. https:\/\/openreview.net\/forum?id=i2_TvOFmEml"},{"key":"e_1_3_2_1_35_1","volume-title":"Foundations and recent trends in multimodal machine learning: Principles, challenges, and open questions. arXiv preprint arXiv:2209.03430","author":"Liang Paul\u00a0Pu","year":"2022","unstructured":"Paul\u00a0Pu Liang , Amir Zadeh , and Louis-Philippe Morency . 2022. Foundations and recent trends in multimodal machine learning: Principles, challenges, and open questions. arXiv preprint arXiv:2209.03430 ( 2022 ). Paul\u00a0Pu Liang, Amir Zadeh, and Louis-Philippe Morency. 2022. Foundations and recent trends in multimodal machine learning: Principles, challenges, and open questions. arXiv preprint arXiv:2209.03430 (2022)."},{"key":"e_1_3_2_1_36_1","volume-title":"DIME: Fine-Grained Interpretations of Multimodal Models via Disentangled Local Explanations(AIES \u201922)","author":"Lyu Yiwei","year":"2022","unstructured":"Yiwei Lyu , Paul\u00a0Pu Liang , Zihao Deng , Ruslan Salakhutdinov , and Louis-Philippe Morency . 2022 . DIME: Fine-Grained Interpretations of Multimodal Models via Disentangled Local Explanations(AIES \u201922) . Association for Computing Machinery , New York, NY, USA , 455\u2013467. https:\/\/doi.org\/10.1145\/3514094.3534148 10.1145\/3514094.3534148 Yiwei Lyu, Paul\u00a0Pu Liang, Zihao Deng, Ruslan Salakhutdinov, and Louis-Philippe Morency. 2022. DIME: Fine-Grained Interpretations of Multimodal Models via Disentangled Local Explanations(AIES \u201922). Association for Computing Machinery, New York, NY, USA, 455\u2013467. https:\/\/doi.org\/10.1145\/3514094.3534148"},{"key":"e_1_3_2_1_37_1","volume-title":"A taxonomy of relationships between images and text. Journal of documentation","author":"Marsh E","year":"2003","unstructured":"Emily\u00a0 E Marsh and Marilyn Domas\u00a0White . 2003. A taxonomy of relationships between images and text. Journal of documentation ( 2003 ). Emily\u00a0E Marsh and Marilyn Domas\u00a0White. 2003. A taxonomy of relationships between images and text. Journal of documentation (2003)."},{"key":"e_1_3_2_1_38_1","volume-title":"Proceedings of The 24th International Conference on Artificial Intelligence and Statistics(Proceedings of Machine Learning Research, Vol.\u00a0130)","author":"Mazzetto Alessio","year":"2021","unstructured":"Alessio Mazzetto , Dylan Sam , Andrew Park , Eli Upfal , and Stephen Bach . 2021 . Semi-Supervised Aggregation of Dependent Weak Supervision Sources With Performance Guarantees . In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics(Proceedings of Machine Learning Research, Vol.\u00a0130) , Arindam Banerjee and Kenji Fukumizu (Eds.). PMLR, 3196\u20133204. https:\/\/proceedings.mlr.press\/v130\/mazzetto21a.html Alessio Mazzetto, Dylan Sam, Andrew Park, Eli Upfal, and Stephen Bach. 2021. Semi-Supervised Aggregation of Dependent Weak Supervision Sources With Performance Guarantees. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics(Proceedings of Machine Learning Research, Vol.\u00a0130), Arindam Banerjee and Kenji Fukumizu (Eds.). PMLR, 3196\u20133204. https:\/\/proceedings.mlr.press\/v130\/mazzetto21a.html"},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIT.1954.1057469"},{"key":"e_1_3_2_1_40_1","volume-title":"Proceedings of the 28th international conference on machine learning (ICML-11)","author":"Ngiam Jiquan","year":"2011","unstructured":"Jiquan Ngiam , Aditya Khosla , Mingyu Kim , Juhan Nam , Honglak Lee , and Andrew\u00a0 Y Ng . 2011 . Multimodal deep learning . In Proceedings of the 28th international conference on machine learning (ICML-11) . 689\u2013696. Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew\u00a0Y Ng. 2011. Multimodal deep learning. In Proceedings of the 28th international conference on machine learning (ICML-11). 689\u2013696."},{"key":"e_1_3_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1007\/s13735-019-00187-6"},{"key":"e_1_3_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/1101149.1101299"},{"key":"e_1_3_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1126\/science.283.5406.1272"},{"key":"e_1_3_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1086\/431246"},{"key":"e_1_3_2_1_45_1","volume-title":"Quantifying how much sensory information in a neural code is relevant for behavior. Advances in Neural Information Processing Systems 30","author":"Pica Giuseppe","year":"2017","unstructured":"Giuseppe Pica , Eugenio Piasini , Houman Safaai , Caroline Runyan , Christopher Harvey , Mathew Diamond , Christoph Kayser , Tommaso Fellin , and Stefano Panzeri . 2017. Quantifying how much sensory information in a neural code is relevant for behavior. Advances in Neural Information Processing Systems 30 ( 2017 ). Giuseppe Pica, Eugenio Piasini, Houman Safaai, Caroline Runyan, Christopher Harvey, Mathew Diamond, Christoph Kayser, Tommaso Fellin, and Stefano Panzeri. 2017. Quantifying how much sensory information in a neural code is relevant for behavior. Advances in Neural Information Processing Systems 30 (2017)."},{"key":"e_1_3_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1109\/TAFFC.2015.2407898"},{"key":"e_1_3_2_1_47_1","volume-title":"International Conference on Machine Learning. PMLR, 8748\u20138763","author":"Radford Alec","year":"2021","unstructured":"Alec Radford , Jong\u00a0Wook Kim , Chris Hallacy , Aditya Ramesh , Gabriel Goh , Sandhini Agarwal , Girish Sastry , Amanda Askell , Pamela Mishkin , Jack Clark , 2021 . Learning transferable visual models from natural language supervision . In International Conference on Machine Learning. PMLR, 8748\u20138763 . Alec Radford, Jong\u00a0Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. PMLR, 8748\u20138763."},{"key":"e_1_3_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/1228175.1228254"},{"key":"e_1_3_2_1_49_1","volume-title":"A mathematical theory of communication. The Bell system technical journal 27, 3","author":"Shannon Claude\u00a0Elwood","year":"1948","unstructured":"Claude\u00a0Elwood Shannon . 1948. A mathematical theory of communication. The Bell system technical journal 27, 3 ( 1948 ), 379\u2013423. Claude\u00a0Elwood Shannon. 1948. A mathematical theory of communication. The Bell system technical journal 27, 3 (1948), 379\u2013423."},{"key":"e_1_3_2_1_50_1","volume-title":"Multimodal emotion recognition in response to videos","author":"Soleymani Mohammad","year":"2011","unstructured":"Mohammad Soleymani , Maja Pantic , and Thierry Pun . 2011. Multimodal emotion recognition in response to videos . IEEE transactions on affective computing 3, 2 ( 2011 ), 211\u2013223. Mohammad Soleymani, Maja Pantic, and Thierry Pun. 2011. Multimodal emotion recognition in response to videos. IEEE transactions on affective computing 3, 2 (2011), 211\u2013223."},{"key":"e_1_3_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390156.1390282"},{"key":"e_1_3_2_1_52_1","volume-title":"Conference on Learning Theory.","author":"Sridharan Karthik","year":"2008","unstructured":"Karthik Sridharan and Sham\u00a0 M Kakade . 2008 . An information theoretic framework for multi-view learning . In Conference on Learning Theory. Karthik Sridharan and Sham\u00a0M Kakade. 2008. An information theoretic framework for multi-view learning. In Conference on Learning Theory."},{"key":"e_1_3_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0019-9958(80)90478-7"},{"key":"e_1_3_2_1_54_1","volume-title":"What makes for good views for contrastive learning?Advances in Neural Information Processing Systems 33","author":"Tian Yonglong","year":"2020","unstructured":"Yonglong Tian , Chen Sun , Ben Poole , Dilip Krishnan , Cordelia Schmid , and Phillip Isola . 2020. What makes for good views for contrastive learning?Advances in Neural Information Processing Systems 33 ( 2020 ), 6827\u20136839. Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. 2020. What makes for good views for contrastive learning?Advances in Neural Information Processing Systems 33 (2020), 6827\u20136839."},{"key":"e_1_3_2_1_55_1","volume-title":"High-degree neurons feed cortical computations. PLoS computational biology 12, 5","author":"Timme M","year":"2016","unstructured":"Nicholas\u00a0 M Timme , Shinya Ito , Maxym Myroshnychenko , Sunny Nigam , Masanori Shimono , Fang-Chin Yeh , Pawel Hottowy , Alan\u00a0 M Litke , and John\u00a0 M Beggs . 2016. High-degree neurons feed cortical computations. PLoS computational biology 12, 5 ( 2016 ), e1004858. Nicholas\u00a0M Timme, Shinya Ito, Maxym Myroshnychenko, Sunny Nigam, Masanori Shimono, Fang-Chin Yeh, Pawel Hottowy, Alan\u00a0M Litke, and John\u00a0M Beggs. 2016. High-degree neurons feed cortical computations. PLoS computational biology 12, 5 (2016), e1004858."},{"key":"e_1_3_2_1_56_1","unstructured":"Christopher Tosh Akshay Krishnamurthy and Daniel Hsu. 2021. Contrastive learning multi-view redundancy and linear models. In Algorithmic Learning Theory. PMLR 1179\u20131206.  Christopher Tosh Akshay Krishnamurthy and Daniel Hsu. 2021. Contrastive learning multi-view redundancy and linear models. In Algorithmic Learning Theory. PMLR 1179\u20131206."},{"key":"e_1_3_2_1_57_1","volume-title":"Learning Factorized Multimodal Representations. In International Conference on Learning Representations.","author":"Tsai Hung\u00a0Hubert","year":"2019","unstructured":"Yao- Hung\u00a0Hubert Tsai , Paul\u00a0Pu Liang , Amir Zadeh , Louis-Philippe Morency , and Ruslan Salakhutdinov . 2019 . Learning Factorized Multimodal Representations. In International Conference on Learning Representations. Yao-Hung\u00a0Hubert Tsai, Paul\u00a0Pu Liang, Amir Zadeh, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019. Learning Factorized Multimodal Representations. In International Conference on Learning Representations."},{"key":"e_1_3_2_1_58_1","volume-title":"International Conference on Learning Representations.","author":"Tsai Hung\u00a0Hubert","year":"2020","unstructured":"Yao- Hung\u00a0Hubert Tsai , Yue Wu , Ruslan Salakhutdinov , and Louis-Philippe Morency . 2020 . Self-supervised Learning from a Multi-view Perspective . In International Conference on Learning Representations. Yao-Hung\u00a0Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-Philippe Morency. 2020. Self-supervised Learning from a Multi-view Perspective. In International Conference on Learning Representations."},{"key":"e_1_3_2_1_59_1","volume-title":"International Conference on Learning Representations.","author":"Tsang Michael","year":"2019","unstructured":"Michael Tsang , Dehua Cheng , Hanpeng Liu , Xue Feng , Eric Zhou , and Yan Liu . 2019 . Feature Interaction Interpretability: A Case for Explaining Ad-Recommendation Systems via Neural Interaction Detection . In International Conference on Learning Representations. Michael Tsang, Dehua Cheng, Hanpeng Liu, Xue Feng, Eric Zhou, and Yan Liu. 2019. Feature Interaction Interpretability: A Case for Explaining Ad-Recommendation Systems via Neural Interaction Detection. In International Conference on Learning Representations."},{"key":"e_1_3_2_1_60_1","volume-title":"International Conference on Learning Representations.","author":"Tsang Michael","year":"2018","unstructured":"Michael Tsang , Dehua Cheng , and Yan Liu . 2018 . Detecting Statistical Interactions from Neural Network Weights . In International Conference on Learning Representations. Michael Tsang, Dehua Cheng, and Yan Liu. 2018. Detecting Statistical Interactions from Neural Network Weights. In International Conference on Learning Representations."},{"key":"e_1_3_2_1_61_1","volume-title":"Multimodality and reading: The construction of meaning through image-text interaction","author":"Unsworth Len","unstructured":"Len Unsworth and Chris Cl\u00e9irigh . 2014. Multimodality and reading: The construction of meaning through image-text interaction . Routledge . Len Unsworth and Chris Cl\u00e9irigh. 2014. Multimodality and reading: The construction of meaning through image-text interaction. Routledge."},{"key":"e_1_3_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1109\/TVCG.2021.3114794"},{"key":"e_1_3_2_1_63_1","volume-title":"Information theoretical analysis of multivariate correlation. IBM Journal of research and development 4, 1","author":"Watanabe Satosi","year":"1960","unstructured":"Satosi Watanabe . 1960. Information theoretical analysis of multivariate correlation. IBM Journal of research and development 4, 1 ( 1960 ), 66\u201382. Satosi Watanabe. 1960. Information theoretical analysis of multivariate correlation. IBM Journal of research and development 4, 1 (1960), 66\u201382."},{"key":"e_1_3_2_1_64_1","doi-asserted-by":"publisher","DOI":"10.3389\/frobt.2015.00005"},{"key":"e_1_3_2_1_65_1","volume-title":"Partial information decomposition as a unified approach to the specification of neural goal functions. Brain and cognition 112","author":"Wibral Michael","year":"2017","unstructured":"Michael Wibral , Viola Priesemann , Jim\u00a0 W Kay , Joseph\u00a0 T Lizier , and William\u00a0 A Phillips . 2017. Partial information decomposition as a unified approach to the specification of neural goal functions. Brain and cognition 112 ( 2017 ), 25\u201338. Michael Wibral, Viola Priesemann, Jim\u00a0W Kay, Joseph\u00a0T Lizier, and William\u00a0A Phillips. 2017. Partial information decomposition as a unified approach to the specification of neural goal functions. Brain and cognition 112 (2017), 25\u201338."},{"key":"e_1_3_2_1_66_1","volume-title":"Nonnegative decomposition of multivariate information. arXiv preprint arXiv:1004.2515","author":"Williams L","year":"2010","unstructured":"Paul\u00a0 L Williams and Randall\u00a0 D Beer . 2010. Nonnegative decomposition of multivariate information. arXiv preprint arXiv:1004.2515 ( 2010 ). Paul\u00a0L Williams and Randall\u00a0D Beer. 2010. Nonnegative decomposition of multivariate information. arXiv preprint arXiv:1004.2515 (2010)."},{"key":"e_1_3_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.findings-emnlp.344"},{"key":"e_1_3_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1145\/956750.956840"},{"key":"e_1_3_2_1_69_1","doi-asserted-by":"publisher","DOI":"10.5555\/1005332.1044700"},{"key":"e_1_3_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR46437.2021.00692"},{"key":"e_1_3_2_1_71_1","doi-asserted-by":"crossref","unstructured":"Amir Zadeh Minghai Chen Soujanya Poria Erik Cambria and Louis-Philippe Morency. 2017. Tensor Fusion Network for Multimodal Sentiment Analysis. In EMNLP.  Amir Zadeh Minghai Chen Soujanya Poria Erik Cambria and Louis-Philippe Morency. 2017. Tensor Fusion Network for Multimodal Sentiment Analysis. In EMNLP.","DOI":"10.18653\/v1\/D17-1115"},{"key":"e_1_3_2_1_72_1","doi-asserted-by":"crossref","unstructured":"AmirAli\u00a0Bagher Zadeh Paul\u00a0Pu Liang Soujanya Poria Erik Cambria and Louis-Philippe Morency. 2018. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In ACL.  AmirAli\u00a0Bagher Zadeh Paul\u00a0Pu Liang Soujanya Poria Erik Cambria and Louis-Philippe Morency. 2018. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In ACL.","DOI":"10.18653\/v1\/P18-1208"},{"key":"e_1_3_2_1_73_1","volume-title":"Equal But Not The Same: Understanding the Implicit Relationship Between Persuasive Images and Text. In British Machine Vision Conference (BMVC).","author":"Zhang Mingda","year":"2018","unstructured":"Mingda Zhang , Rebecca Hwa , and Adriana Kovashka . 2018 . Equal But Not The Same: Understanding the Implicit Relationship Between Persuasive Images and Text. In British Machine Vision Conference (BMVC). Mingda Zhang, Rebecca Hwa, and Adriana Kovashka. 2018. Equal But Not The Same: Understanding the Implicit Relationship Between Persuasive Images and Text. In British Machine Vision Conference (BMVC)."}],"event":{"name":"ICMI '23: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION","location":"Paris France","acronym":"ICMI '23","sponsor":["SIGCHI ACM Special Interest Group on Computer-Human Interaction"]},"container-title":["INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3577190.3614151","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3577190.3614151","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:37:01Z","timestamp":1750178221000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3577190.3614151"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,10,9]]},"references-count":73,"alternative-id":["10.1145\/3577190.3614151","10.1145\/3577190"],"URL":"https:\/\/doi.org\/10.1145\/3577190.3614151","relation":{},"subject":[],"published":{"date-parts":[[2023,10,9]]},"assertion":[{"value":"2023-10-09","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}