{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,16]],"date-time":"2025-10-16T06:58:54Z","timestamp":1760597934762,"version":"3.41.0"},"reference-count":43,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2020,2,17]],"date-time":"2020-02-17T00:00:00Z","timestamp":1581897600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["61771025"],"award-info":[{"award-number":["61771025"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Multimedia Comput. Commun. Appl."],"published-print":{"date-parts":[[2020,2,29]]},"abstract":"<jats:p>\n            Entailment recognition is an important paradigm of reasoning that judges if a hypothesis can be inferred from given premises. However, previous efforts mainly concentrate on text-based reasoning as\n            <jats:italic>recognizing textual entailment (RTE)<\/jats:italic>\n            , where the hypotheses and premises are both textual. In fact, humans\u2019 reasoning process has the characteristic of\n            <jats:italic>cross-media reasoning<\/jats:italic>\n            . It is naturally based on the joint inference with different sensory organs, which represent complementary reasoning cues from unique perspectives as language, vision, and audition. How to realize cross-media reasoning has been a significant challenge to achieve the breakthrough for width and depth of entailment recognition. Therefore, this article extends RTE to a novel reasoning paradigm:\n            <jats:bold>\n              <jats:italic>recognizing cross-media entailment (RCE)<\/jats:italic>\n            <\/jats:bold>\n            , and proposes\n            <jats:bold>\n              <jats:italic>heterogeneous interactive learning (HIL)<\/jats:italic>\n            <\/jats:bold>\n            approach. Specifically, HIL recognizes entailment relationships via cross-media joint inference, from image-text premises to text hypotheses. It is an end-to-end architecture with two parts: (1) Cross-media hybrid embedding is proposed to perform cross embedding of premises and hypotheses for generating their fine-grained representations. It aims to achieve the alignment of cross-media inference cues via image-text and text-text interactive attention. (2) Heterogeneous joint inference is proposed to construct a heterogeneous interaction tensor space and extract semantic features for entailment recognition. It aims to simultaneously capture the interaction between cross-media premises and hypotheses and distinguish their entailment relationships. Experimental results on widely used Stanford natural language inference (SNLI) dataset with image premises from Flickr30K dataset verify the effectiveness of HIL and the intrinsic inter-media complementarity in reasoning.\n          <\/jats:p>","DOI":"10.1145\/3365003","type":"journal-article","created":{"date-parts":[[2020,3,4]],"date-time":"2020-03-04T10:23:32Z","timestamp":1583317412000},"page":"1-21","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["RCE-HIL"],"prefix":"10.1145","volume":"16","author":[{"given":"Xin","family":"Huang","sequence":"first","affiliation":[{"name":"Peking University, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yuxin","family":"Peng","sequence":"additional","affiliation":[{"name":"Peking University, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zhang","family":"Wen","sequence":"additional","affiliation":[{"name":"Peking University, Beijing, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2020,2,17]]},"reference":[{"volume-title":"Proceedings of the International Conference on Machine Learning (ICML\u201910)","year":"2010","author":"Andrew Galen","key":"e_1_2_1_1_1"},{"volume-title":"Proceedings of the Conference on Empirical Methods on Natural Language Processing (EMNLP\u201915)","author":"Bowman Samuel R.","key":"e_1_2_1_2_1"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P16-1139"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/1236471.1236472"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-1224"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-1152"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D16-1053"},{"volume-title":"Proceedings of the Learning Methods for Text Understanding and Mining Workshop.","year":"2004","author":"Dagan Ido","key":"e_1_2_1_8_1"},{"key":"e_1_2_1_9_1","article-title":"Correspondence autoencoders for cross-modal retrieval","volume":"12","author":"Feng Fangxiang","year":"2015","journal-title":"ACM Trans. Multim. Comput. Commun. Applic."},{"volume-title":"Natural language inference over interaction space. arXiv preprint arXiv: abs\/1709.04348","year":"2017","author":"Gong Yichen","key":"e_1_2_1_10_1"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D17-1305"},{"volume-title":"Proceedings of the 21st International Conference on Computational Linguistics and 44th Meeting of the Association for Computational Linguistics. 905--912","author":"Sanda","key":"e_1_2_1_12_1"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1093\/biomet\/28.3-4.321"},{"volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR\u201917)","author":"Huang Gao","key":"e_1_2_1_14_1"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2018.00921"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2017\/263"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1109\/TMM.2015.2390499"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-016-0981-7"},{"volume-title":"Proceedings of the International Conference on Advances in Neural Information Processing Systems (NIPS\u201912)","author":"Hinton G.","key":"e_1_2_1_19_1"},{"key":"e_1_2_1_20_1","article-title":"Learning label preserving binary codes for multimedia retrieval: A general approach","volume":"14","author":"Li Kai","year":"2018","journal-title":"ACM Trans. Multim. Comput. Commun. Applic."},{"volume-title":"Proceedings of the International Conference on Advances in Neural Information Processing Systems (NIPS\u201916)","year":"2016","author":"Lu Jiasen","key":"e_1_2_1_21_1"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/219717.219748"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.3115\/1690219.1690257"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P16-2022"},{"volume-title":"Proceedings of the International Conference on Machine Learning (ICML\u201911)","author":"Ngiam Jiquan","key":"e_1_2_1_26_1"},{"volume-title":"A neural architecture mimicking humans end-to-end for natural language inference. arXiv preprint arXiv: abs\/1611.04741","year":"2016","author":"Paria Biswajit","key":"e_1_2_1_27_1"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D16-1244"},{"volume-title":"Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI\u201916)","year":"2016","author":"Peng Yuxin","key":"e_1_2_1_29_1"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1109\/TCSVT.2017.2705068"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1109\/TMM.2017.2742704"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-016-0965-7"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/1873951.1873987"},{"volume-title":"Proceedings of the International Conference on Computational Linguistics (COLING\u201916)","year":"2016","author":"Sha Lei","key":"e_1_2_1_34_1"},{"volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence (AAAI\u201918)","year":"2018","author":"Shen Tao","key":"e_1_2_1_35_1"},{"volume-title":"Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409.1556","year":"2014","author":"Simonyan Karen","key":"e_1_2_1_36_1"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1185"},{"volume-title":"Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI\u201917)","author":"Wang Peng","key":"e_1_2_1_38_1"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N16-1170"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.5555\/3171837.3171865"},{"key":"e_1_2_1_41_1","first-page":"449","article-title":"Cross-modal retrieval with CNN visual features: A new baseline","volume":"47","author":"Wei Yunchao","year":"2017","journal-title":"IEEE Trans. Cyber."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2017.2746267"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00097"},{"volume-title":"Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics (EACL\u201917)","year":"2017","author":"Yu Hong","key":"e_1_2_1_44_1"}],"container-title":["ACM Transactions on Multimedia Computing, Communications, and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3365003","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3365003","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:01:29Z","timestamp":1750197689000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3365003"}},"subtitle":["Recognizing Cross-media Entailment with Heterogeneous Interactive Learning"],"short-title":[],"issued":{"date-parts":[[2020,2,17]]},"references-count":43,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2020,2,29]]}},"alternative-id":["10.1145\/3365003"],"URL":"https:\/\/doi.org\/10.1145\/3365003","relation":{},"ISSN":["1551-6857","1551-6865"],"issn-type":[{"type":"print","value":"1551-6857"},{"type":"electronic","value":"1551-6865"}],"subject":[],"published":{"date-parts":[[2020,2,17]]},"assertion":[{"value":"2019-01-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-09-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-02-17","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}