{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,4]],"date-time":"2025-11-04T10:50:51Z","timestamp":1762253451827,"version":"3.41.0"},"reference-count":37,"publisher":"Association for Computing Machinery (ACM)","issue":"2s","license":[{"start":{"date-parts":[[2020,4,30]],"date-time":"2020-04-30T00:00:00Z","timestamp":1588204800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100007834","name":"Natural Science Foundation of Ningbo","doi-asserted-by":"crossref","award":["2018A610165"],"award-info":[{"award-number":["2018A610165"]}],"id":[{"id":"10.13039\/100007834","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/100007219","name":"Natural Science Foundation of Shanghai","doi-asserted-by":"crossref","award":["14ZR1429800,15ZR1430000,18ZR1427400,18ZR1427500"],"award-info":[{"award-number":["14ZR1429800,15ZR1430000,18ZR1427400,18ZR1427500"]}],"id":[{"id":"10.13039\/100007219","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Science Foundation of Department of Education of Zhejiang","award":["Y201941227"],"award-info":[{"award-number":["Y201941227"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Multimedia Comput. Commun. Appl."],"published-print":{"date-parts":[[2020,4,30]]},"abstract":"<jats:p>Understanding temporal dependencies of videos is fundamental for vision problems, but deep learning\u2013based models are still insufficient in this field. In this article, we propose a novel deep multiplicative neural network (DMNN) for learning hierarchical long-term representations from video. The DMNN is built upon the multiplicative block that remembers the pairwise transformations between consecutive frames using multiplicative interactions rather than the regular weighted-sum ones. The block is slided over the timesteps to update the memory of the networks on the frame pairs. Deep architecture can be implemented by stacking multiple layers of the sliding blocks. The multiplicative interactions lead to exact, rather than approximate, modeling of temporal dependencies. The memory mechanism can remember the temporal dependencies for an arbitrary length of time. The multiple layers output multiple-level representations that reflect the multi-timescale structure of video. Moreover, to address the difficulty of training DMNNs, we derive a theoretically sound convergent method, which leads to a fast and stable convergence. We demonstrate a new state-of-the-art classification performance with proposed networks on the UCF101 dataset and the effectiveness of capturing complicate temporal dependencies on a variety of synthetic datasets.<\/jats:p>","DOI":"10.1145\/3357797","type":"journal-article","created":{"date-parts":[[2020,7,7]],"date-time":"2020-07-07T12:38:12Z","timestamp":1594125492000},"page":"1-19","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Modeling Long-Term Dependencies from Videos Using Deep Multiplicative Neural Networks"],"prefix":"10.1145","volume":"16","author":[{"given":"Wen","family":"Si","sequence":"first","affiliation":[{"name":"Shanghai Business Schooland Fudan University, Shanghai, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Cong","family":"Liu","sequence":"additional","affiliation":[{"name":"Zhejiang University, Ningbo, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zhongqin","family":"Bi","sequence":"additional","affiliation":[{"name":"Shanghai University of Electric Power, Shanghai, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Meijing","family":"Shan","sequence":"additional","affiliation":[{"name":"East China University of Political Science and Law, Shanghai, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2020,7,14]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"Samy Bengio Oriol Vinyals Navdeep Jaitly and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances in Neural Information Processing Systems. 1171--1179.  Samy Bengio Oriol Vinyals Navdeep Jaitly and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances in Neural Information Processing Systems. 1171--1179."},{"key":"e_1_2_1_2_1","unstructured":"Guillaume Desjardins Karen Simonyan Razvan Pascanu and Koray Kavukcuoglu. 2015. Natural neural networks. In Advances in Neural Information Processing Systems. 2071--2079.  Guillaume Desjardins Karen Simonyan Razvan Pascanu and Koray Kavukcuoglu. 2015. Natural neural networks. In Advances in Neural Information Processing Systems. 2071--2079."},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298878"},{"key":"e_1_2_1_4_1","volume-title":"Proceedings of the International Conference on Artificial Intelligence and Statistics. 249--256","author":"Glorot Xavier","year":"2010","unstructured":"Xavier Glorot and Yoshua Bengio . 2010 . Understanding the difficulty of training deep feedforward neural networks . In Proceedings of the International Conference on Artificial Intelligence and Statistics. 249--256 . Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the International Conference on Artificial Intelligence and Statistics. 249--256."},{"key":"e_1_2_1_5_1","volume-title":"Proceedings of the International Conference on Artificial Intelligence and Statistics. 315--323","author":"Glorot Xavier","year":"2011","unstructured":"Xavier Glorot , Antoine Bordes , and Yoshua Bengio . 2011 . Deep sparse rectifier neural networks . In Proceedings of the International Conference on Artificial Intelligence and Statistics. 315--323 . Xavier Glorot, Antoine Bordes, and Yoshua Bengio. 2011. Deep sparse rectifier neural networks. In Proceedings of the International Conference on Artificial Intelligence and Statistics. 315--323."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2013.337"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.123"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/MSP.2012.2205597"},{"key":"e_1_2_1_9_1","volume-title":"Proceedings of the 32nd International Conference on Machine Learning","volume":"37","author":"Ioffe Sergey","year":"2015","unstructured":"Sergey Ioffe and Christian Szegedy . 2015 . Batch normalization: Accelerating deep network training by reducing internal covariate shift . In Proceedings of the 32nd International Conference on Machine Learning , Vol. 37 . 448--456. http:\/\/proceedings.mlr.press\/v37\/ioffe15.html. Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning, Vol. 37. 448--456. http:\/\/proceedings.mlr.press\/v37\/ioffe15.html."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2012.59"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2014.223"},{"key":"e_1_2_1_12_1","volume-title":"Zemel","author":"Kiros Ryan","year":"2014","unstructured":"Ryan Kiros , Ruslan Salakhutdinov , and Richard S . Zemel . 2014 . Unifying visual-semantic embeddings with multimodal neural language models. arXiv:1411.2539. Ryan Kiros, Ruslan Salakhutdinov, and Richard S. Zemel. 2014. Unifying visual-semantic embeddings with multimodal neural language models. arXiv:1411.2539."},{"key":"e_1_2_1_13_1","unstructured":"Kishore Reddy Konda Roland Memisevic and Vincent Michalski. 2013. Learning to encode motion using spatio-temporal synchrony. arXiv:1306.3162.  Kishore Reddy Konda Roland Memisevic and Vincent Michalski. 2013. Learning to encode motion using spatio-temporal synchrony. arXiv:1306.3162."},{"key":"e_1_2_1_14_1","volume-title":"Hinton","author":"Krizhevsky Alex","year":"2012","unstructured":"Alex Krizhevsky , Ilya Sutskever , and Geoffrey E . Hinton . 2012 . ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems . 1097--1105. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems. 1097--1105."},{"key":"e_1_2_1_15_1","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR\u201915)","author":"Lan Zhengzhong","year":"2015","unstructured":"Zhengzhong Lan , Ming Lin , Xuanchong Li , Alex G. Hauptmann , and Bhiksha Raj . 2015 . Beyond Gaussian pyramid: Multi-skip feature stacking for action recognition . In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR\u201915) . 204--212. Zhengzhong Lan, Ming Lin, Xuanchong Li, Alex G. Hauptmann, and Bhiksha Raj. 2015. Beyond Gaussian pyramid: Multi-skip feature stacking for action recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR\u201915). 204--212."},{"key":"e_1_2_1_16_1","volume-title":"Proceedings of the 30th International Conference on Machine Learning (ICML\u201913)","volume":"30","author":"Maas Andrew L.","unstructured":"Andrew L. Maas , Awni Y. Hannun , and Andrew Y. Ng . 2013. Rectifier nonlinearities improve neural network acoustic models . In Proceedings of the 30th International Conference on Machine Learning (ICML\u201913) , Vol. 30 . Andrew L. Maas, Awni Y. Hannun, and Andrew Y. Ng. 2013. Rectifier nonlinearities improve neural network acoustic models. In Proceedings of the 30th International Conference on Machine Learning (ICML\u201913), Vol. 30."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-21735-7_7"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2011.6126419"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/TPAMI.2013.53"},{"key":"e_1_2_1_20_1","unstructured":"Vincent Michalski Roland Memisevic and Kishore Konda. 2014. Modeling deep temporal dependencies with recurrent grammar cells. In Advances in Neural Information Processing Systems. 1925--1933.  Vincent Michalski Roland Memisevic and Kishore Konda. 2014. Modeling deep temporal dependencies with recurrent grammar cells. In Advances in Neural Information Processing Systems. 1925--1933."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/1553374.1553469"},{"volume-title":"Neural Networks: Tricks of the Trade","author":"Montavon Gr\u00e9goire","key":"e_1_2_1_22_1","unstructured":"Gr\u00e9goire Montavon and Klaus-Robert M\u00fcller . 2012. Deep Boltzmann machines and the centering trick . In Neural Networks: Tricks of the Trade . Springer , 621--637. Gr\u00e9goire Montavon and Klaus-Robert M\u00fcller. 2012. Deep Boltzmann machines and the centering trick. In Neural Networks: Tricks of the Trade. Springer, 621--637."},{"volume-title":"Proceedings of the 27th International Conference on Machine Learning (ICML\u201910)","author":"Nair Vinod","key":"e_1_2_1_23_1","unstructured":"Vinod Nair and Geoffrey E. Hinton . 2010. Rectified linear units improve restricted Boltzmann machines . In Proceedings of the 27th International Conference on Machine Learning (ICML\u201910) . 807--814. Vinod Nair and Geoffrey E. Hinton. 2010. Rectified linear units improve restricted Boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning (ICML\u201910). 807--814."},{"key":"e_1_2_1_24_1","volume-title":"Proceedings of the International Conference on Artificial Intelligence and Statistics. 924--932","author":"Raiko Tapani","year":"2012","unstructured":"Tapani Raiko , Harri Valpola , and Yann LeCun . 2012 . Deep learning made easier by linear transformations in perceptrons . In Proceedings of the International Conference on Artificial Intelligence and Statistics. 924--932 . Tapani Raiko, Harri Valpola, and Yann LeCun. 2012. Deep learning made easier by linear transformations in perceptrons. In Proceedings of the International Conference on Artificial Intelligence and Statistics. 924--932."},{"key":"e_1_2_1_25_1","unstructured":"Karen Simonyan and Andrew Zisserman. 2014. Two-stream convolutional networks for action recognition in videos. In Advances in Neural Information Processing Systems. 568--576.  Karen Simonyan and Andrew Zisserman. 2014. Two-stream convolutional networks for action recognition in videos. In Advances in Neural Information Processing Systems. 568--576."},{"key":"e_1_2_1_26_1","volume-title":"Proceedings of the International Conference on Learning Representations. 2017--2027","author":"Simonyan Karen","year":"2015","unstructured":"Karen Simonyan and Andrew Zisserman . 2015 . Very deep convolutional networks for large-scale image recognition . In Proceedings of the International Conference on Learning Representations. 2017--2027 . Karen Simonyan and Andrew Zisserman. 2015. Very deep convolutional networks for large-scale image recognition. In Proceedings of the International Conference on Learning Representations. 2017--2027."},{"key":"e_1_2_1_27_1","volume-title":"Amir Roshan Zamir, and Mubarak Shah","author":"Soomro Khurram","year":"2012","unstructured":"Khurram Soomro , Amir Roshan Zamir, and Mubarak Shah . 2012 . UCF101: A Dataset of 101 Human Actions Classes from Videos in the Wild. CRCV-TR12-01. Center for Research in Computer Vision, University of Central Floriday , Orlando, FL. Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. 2012. UCF101: A Dataset of 101 Human Actions Classes from Videos in the Wild. CRCV-TR12-01. Center for Research in Computer Vision, University of Central Floriday, Orlando, FL."},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.5555\/2627435.2670313"},{"key":"e_1_2_1_29_1","volume-title":"Proceedings of the 32nd International Conference on Machine Learning (ICML\u201915)","author":"Srivastava Nitish","year":"2015","unstructured":"Nitish Srivastava , Elman Mansimov , and Ruslan Salakhutdinov . 2015 . Unsupervised learning of video representations using LSTMs . In Proceedings of the 32nd International Conference on Machine Learning (ICML\u201915) . 843\u2013852. Nitish Srivastava, Elman Mansimov, and Ruslan Salakhutdinov. 2015. Unsupervised learning of video representations using LSTMs. In Proceedings of the 32nd International Conference on Machine Learning (ICML\u201915). 843\u2013852."},{"key":"e_1_2_1_30_1","volume-title":"Le","author":"Sutskever Ilya","year":"2014","unstructured":"Ilya Sutskever , Oriol Vinyals , and Quoc V. V . Le . 2014 . Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems . 3104--3112. Ilya Sutskever, Oriol Vinyals, and Quoc V. V. Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems. 3104--3112."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-15567-3_11"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2015.510"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/N15-1173"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/TIP.2015.2403231"},{"key":"e_1_2_1_35_1","unstructured":"Naiyan Wang and Dit-Yan Yeung. 2013. Learning a deep compact image representation for visual tracking. In Advances in Neural Information Processing Systems. 809--817.  Naiyan Wang and Dit-Yan Yeung. 2013. Learning a deep compact image representation for visual tracking. In Advances in Neural Information Processing Systems. 809--817."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/2566486.2568032"},{"key":"e_1_2_1_37_1","volume-title":"Deep learning. Nature 521, 7553 (05","author":"Hinton Geoffrey","year":"2015","unstructured":"Geoffrey Hinton , Yann LeCun , and Yoshua Bengio . 2015. Deep learning. Nature 521, 7553 (05 2015 ), 436--444. Geoffrey Hinton, Yann LeCun, and Yoshua Bengio. 2015. Deep learning. Nature 521, 7553 (05 2015), 436--444."}],"container-title":["ACM Transactions on Multimedia Computing, Communications, and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3357797","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3357797","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T22:32:58Z","timestamp":1750199578000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3357797"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,4,30]]},"references-count":37,"journal-issue":{"issue":"2s","published-print":{"date-parts":[[2020,4,30]]}},"alternative-id":["10.1145\/3357797"],"URL":"https:\/\/doi.org\/10.1145\/3357797","relation":{},"ISSN":["1551-6857","1551-6865"],"issn-type":[{"type":"print","value":"1551-6857"},{"type":"electronic","value":"1551-6865"}],"subject":[],"published":{"date-parts":[[2020,4,30]]},"assertion":[{"value":"2019-05-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-08-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2020-07-14","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}