{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,13]],"date-time":"2026-05-13T15:28:06Z","timestamp":1778686086721,"version":"3.51.4"},"reference-count":52,"publisher":"Springer Science and Business Media LLC","issue":"2-4","license":[{"start":{"date-parts":[[2017,2,20]],"date-time":"2017-02-20T00:00:00Z","timestamp":1487548800000},"content-version":"unspecified","delay-in-days":0,"URL":"http:\/\/www.springer.com\/tdm"}],"funder":[{"name":"Defense Advanced Research Projects Agency (US)"},{"name":"Defense Advanced Research Projects Agency (US)"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Int J Comput Vis"],"published-print":{"date-parts":[[2018,4]]},"DOI":"10.1007\/s11263-017-0997-7","type":"journal-article","created":{"date-parts":[[2017,2,19]],"date-time":"2017-02-19T21:25:22Z","timestamp":1487539522000},"page":"440-456","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":32,"title":["Deep Multimodal Fusion: A Hybrid Approach"],"prefix":"10.1007","volume":"126","author":[{"given":"Mohamed R.","family":"Amer","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Timothy","family":"Shields","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Behjat","family":"Siddiquie","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Amir","family":"Tamrakar","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ajay","family":"Divakaran","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sek","family":"Chai","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2017,2,20]]},"reference":[{"key":"997_CR1","doi-asserted-by":"crossref","unstructured":"Amer, M., Siddiquie, B., Khan, S., Divakaran, A., & Sawhney, H. (2014). Multimodal fusion using dynamic hybrid models. In WACV.","DOI":"10.1109\/WACV.2014.6836053"},{"key":"997_CR2","doi-asserted-by":"crossref","unstructured":"Bengio, Y. (2009). Learning deep architectures for ai. In FTML.","DOI":"10.1561\/2200000006"},{"key":"997_CR3","unstructured":"Camgoz, N., Kindiroglu, A., & Akarun, L. (2014). Gesture recognition using templatebased random forest classifiers. In ECCV."},{"key":"997_CR4","unstructured":"Chang, J. (2014). Nonparametric gesture labeling from multi-modal data. In ECCV-W."},{"key":"997_CR5","unstructured":"Chen, G., Clarke, D., Giuliani, M., Weikersdorfer, D., & Knoll, A. (2014). Multi-modality gesture detection and recognition with un-supervision, randomization and discrimination. In ECCV-W."},{"key":"997_CR6","unstructured":"Cox, S., Harvey, R., Lan, Y., & Newman, J. (2008). The challenge of multispeaker lip-reading. In AVSP."},{"key":"997_CR7","unstructured":"Druck, G., & McCallum, A. (2010). High-performance semi-supervised learning using discriminatively constrained generative models. In ICML."},{"key":"997_CR8","unstructured":"Escalera, S., Baro, X., Gonzalez, J., Bautista, M., Madadi, M., Reyes, M., Ponce, V., Escalante, H., Shotton, J., & Guyon, I. (2014). Chalearn looking at people challenge 2014: Dataset and results. In ECCV-W."},{"key":"997_CR9","unstructured":"Evangelidis, G., Singh, G., & Horaud, R. (2014). Continuous gesture recognition from articulated poses. In ECCV-W."},{"key":"997_CR10","doi-asserted-by":"crossref","unstructured":"Fujino, A., Ueda, N., & Saito, K. (2008). Semi-supervised learning for a hybrid generative\/discriminative classifier based on the maximum entropy principle. In TPAMI.","DOI":"10.1109\/TPAMI.2007.70710"},{"key":"997_CR11","unstructured":"Garg, N., & Henderson, J. (2011). Temporal restricted Boltzmann machines for dependency parsing. In ACL."},{"key":"997_CR12","doi-asserted-by":"crossref","unstructured":"Glodek, M., et al. (2011). Multiple classifier systems for the classification of audio-visual emotional states. In ACII.","DOI":"10.1007\/978-3-642-24571-8_47"},{"key":"997_CR13","doi-asserted-by":"crossref","first-page":"4765","DOI":"10.1109\/TSP.2009.2026513","volume":"57","author":"M Gurban","year":"2009","unstructured":"Gurban, M., & Thiran, J. P. (2009). Information theoretic feature extraction for audio-visual speech recognition. IEEE Transactions on Signal Processing, 57, 4765\u20134776.","journal-title":"IEEE Transactions on Signal Processing"},{"key":"997_CR14","unstructured":"Hausler, C., & Susemihl, A. (2012). Temporal autoencoding restricted Boltzmann machine. In CoRR."},{"key":"997_CR15","doi-asserted-by":"crossref","unstructured":"Hinton, G. E. (2002). Training products of experts by minimizing contrastive divergence. In NC.","DOI":"10.1162\/089976602760128018"},{"key":"997_CR16","doi-asserted-by":"crossref","unstructured":"Hinton, G. E., Osindero, S., & Teh, Y. W. (2006). A fast learning algorithm for deep belief nets. In NC.","DOI":"10.1162\/neco.2006.18.7.1527"},{"key":"997_CR17","doi-asserted-by":"crossref","unstructured":"Larochelle, H., & Bengio, Y. (2008). Classification using discriminative restricted Boltzmann machines. In ICML.","DOI":"10.1145\/1390156.1390224"},{"key":"997_CR18","unstructured":"Lewandowski, N. B., Bengio, Y., & Vincent, P. (2012). Modeling temporal dependencies in high-dimensional sequences: Application to polyphonic music generation and transcription. In ICML."},{"key":"997_CR19","doi-asserted-by":"crossref","unstructured":"Li, X., Lee, T., & Liu, Y. (2011). Hybrid generative-discriminative classification using posterior divergence. In CVPR.","DOI":"10.1109\/CVPR.2011.5995584"},{"key":"997_CR20","unstructured":"Lucey, P., & Sridharan, S. (2006). Patch based representation of visual speech. In HCSnet workshop on the use of vision in human-computer interaction."},{"key":"997_CR21","unstructured":"Matthews, I., et al. (2002). Extraction of visual features for lipreading. In: TPAMI."},{"key":"997_CR22","doi-asserted-by":"crossref","unstructured":"Memisevic, R. & Hinton, G. E. (2007). Unsupervised learning of image transformations. In CVPR.","DOI":"10.1109\/CVPR.2007.383036"},{"key":"997_CR23","unstructured":"Mohamed, A. R., & Hinton, G. E. (2009). Phone recognition using restricted Boltzmann machines. In ICASSP."},{"key":"997_CR24","unstructured":"Monnier, C., German, S., & Ost, A. (2014). A multi-scale boosted detector for efficient and robust gesture recognition. In ECCV-W."},{"key":"997_CR25","unstructured":"Neverova, N., Wolf, C., Taylor, G., & Nebout, F. (2014). Moddrop: Adaptive multi-modal gesture recognition. In PAMI."},{"key":"997_CR26","unstructured":"Neverova, N., Wolf, C., Taylor, G. W., & Nebout, F. (2014). Multi-scale deep learning for gesture detection and localization. In ECCV-W."},{"key":"997_CR27","unstructured":"Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., & Ng, A. (2011). Multimodal deep learning. In ICML."},{"issue":"3","key":"997_CR28","doi-asserted-by":"crossref","first-page":"299","DOI":"10.1007\/s11263-007-0122-4","volume":"79","author":"J Niebles","year":"2008","unstructured":"Niebles, J., Wang, H., & Fei-Fei, L. (2008). Unsupervised learning of human action categories using spatial-temporal words. IJCV, 79(3), 299\u2013318.","journal-title":"IJCV"},{"key":"997_CR29","doi-asserted-by":"crossref","unstructured":"Papandreou, G., Katsamanis, A., Pitsikalis, V., & Maragos, P. (2009). Adaptive multimodal fusion by uncertainty compensation with application to audiovisual speech recognition. In TASLP.","DOI":"10.1109\/TASL.2008.2011515"},{"key":"997_CR30","doi-asserted-by":"crossref","unstructured":"Patterson, E., et al. (2002). Cuave: A new audio-visual database for multimodal human-computer interface research. In ICASSP.","DOI":"10.1109\/ICASSP.2002.5745028"},{"key":"997_CR31","unstructured":"Peng, X., Wang, L., & Cai, Z. (2014). Action and gesture temporal spotting with super vector representation. In ECCV-W."},{"key":"997_CR32","doi-asserted-by":"crossref","unstructured":"Perina, A., et al. (2012). Free energy score spaces: Using generative information in discriminative classifiers. In TPAMI.","DOI":"10.1109\/TPAMI.2011.241"},{"key":"997_CR33","unstructured":"Pigou, L., Dieleman, S., & Kindermans, P. J. (2014). Sign language recognition using convolutional neural networks. In ECCV-W."},{"key":"997_CR34","doi-asserted-by":"crossref","unstructured":"Ramirez, G., Baltrusaitis, T., & Morency, L. P. (2011). Modeling latent discriminative dynamic of multi-dimensional affective signals. In ACII.","DOI":"10.1007\/978-3-642-24571-8_51"},{"key":"997_CR35","doi-asserted-by":"crossref","unstructured":"Ranzato, M. A., et al. (2011). On deep generative models with applications to recognition. In CVPR.","DOI":"10.1109\/CVPR.2011.5995710"},{"key":"997_CR36","doi-asserted-by":"crossref","unstructured":"Rehg, J. M., et al. (2013). Decoding children\u2019s social behavior. In CVPR.","DOI":"10.1109\/CVPR.2013.438"},{"key":"997_CR37","unstructured":"Salakhutdinov, R., & Hinton, G. E. (2006). Reducing the dimensionality of data with neural networks. In Science."},{"key":"997_CR38","doi-asserted-by":"crossref","unstructured":"Salter, D. A., Tamrakar, A., Behjat\u00a0Siddiquie, M. R. A., Divakaran, A., Lande, B., & Mehri, D. (2015). The tower game dataset: A multimodal dataset for analyzing social interaction predicates. In ACII.","DOI":"10.1109\/ACII.2015.7344639"},{"key":"997_CR39","doi-asserted-by":"crossref","unstructured":"Schuller, B., et al. (2011). Avec 2011\u2014the first international audio visual emotion challenge. In ACII.","DOI":"10.1007\/978-3-642-24571-8_53"},{"key":"997_CR40","doi-asserted-by":"crossref","unstructured":"Siddiquie, B., Khan, S., Divakaran, A., & Sawhney, H. (2013). Affect analysis in natural human interactions using joint hidden conditional random fields. In ICME.","DOI":"10.1109\/ICME.2013.6607590"},{"key":"997_CR41","doi-asserted-by":"crossref","unstructured":"Sminchisescu, C., Kanaujia, A., & Metaxas, D. (2006). Learning joint top-down and bottom-up processes for 3d visual inference. In CVPR.","DOI":"10.1109\/CVPR.2006.169"},{"key":"997_CR42","unstructured":"Srivastava, N., & Salakhutdinov, R. (2012). Multimodal learning with deep Boltzmann machines. In NIPS."},{"key":"997_CR43","doi-asserted-by":"crossref","unstructured":"Sun, X., Lichtenauer, J., Valstar, M. F., Nijholt, A., & Pantic., M. (2011). A multimodal database for mimicry analysis. In ACII.","DOI":"10.1007\/978-3-642-24600-5_40"},{"key":"997_CR44","unstructured":"Sutskever, I., & Hinton, G. E. (2007). Learning multilevel distributed representations for high-dimensional sequences. In AISTATS."},{"key":"997_CR45","unstructured":"Sutskever, I., Hinton, G., & Taylor, G. (2008). The recurrent temporal restricted Boltzmann machine. In NIPS."},{"key":"997_CR46","doi-asserted-by":"crossref","unstructured":"Taylor, G. W., et al. (2010). Dynamical binary latent variable models for 3d human pose tracking. In CVPR.","DOI":"10.1109\/CVPR.2010.5540157"},{"key":"997_CR47","first-page":"1025","volume":"12","author":"GW Taylor","year":"2011","unstructured":"Taylor, G. W., Hinton, G. E., & Roweis, S. T. (2011). Two distributed-state models for generating high-dimensional time series. Journal of Machine Learning Research, 12, 1025\u20131068.","journal-title":"Journal of Machine Learning Research"},{"key":"997_CR48","unstructured":"Wu, D. (2014). Deep dynamic neural networks for gesture segmentation and recognition. In ECCV-W."},{"key":"997_CR49","doi-asserted-by":"crossref","unstructured":"Zanfir, M., Leordeanu, M., & Sminchisescu, C. (2013). The moving pose: An efficient 3d kinematics descriptor for low-latency action recognition and detection. In ICCV.","DOI":"10.1109\/ICCV.2013.342"},{"key":"997_CR50","unstructured":"Zeiler, M. D., & Fergus, R. (2014). A multimodal database for mimicry analysis. In ECCV."},{"key":"997_CR51","unstructured":"Zeiler, M. D., Taylor, G. W., Sigal, L., Matthews, I., & Fergus, R. (2011). Facial expression transfer with input\u2013output temporal restricted Boltzmann machines. In NIPS."},{"key":"997_CR52","doi-asserted-by":"crossref","first-page":"1254","DOI":"10.1109\/TMM.2009.2030637","volume":"11","author":"G Zhao","year":"2009","unstructured":"Zhao, G., & Barnard, M. (2009). Lipreading with local spatiotemporal descriptors. Transactions of Multimedia, 11, 1254\u20131265.","journal-title":"Transactions of Multimedia"}],"container-title":["International Journal of Computer Vision"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/link.springer.com\/article\/10.1007\/s11263-017-0997-7\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s11263-017-0997-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"http:\/\/link.springer.com\/content\/pdf\/10.1007\/s11263-017-0997-7.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2019,9,18]],"date-time":"2019-09-18T17:32:14Z","timestamp":1568827934000},"score":1,"resource":{"primary":{"URL":"http:\/\/link.springer.com\/10.1007\/s11263-017-0997-7"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,2,20]]},"references-count":52,"journal-issue":{"issue":"2-4","published-print":{"date-parts":[[2018,4]]}},"alternative-id":["997"],"URL":"https:\/\/doi.org\/10.1007\/s11263-017-0997-7","relation":{},"ISSN":["0920-5691","1573-1405"],"issn-type":[{"value":"0920-5691","type":"print"},{"value":"1573-1405","type":"electronic"}],"subject":[],"published":{"date-parts":[[2017,2,20]]}}}