{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,5]],"date-time":"2026-06-05T15:54:38Z","timestamp":1780674878782,"version":"3.54.1"},"reference-count":38,"publisher":"MIT Press","issue":"12","content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2024,11,19]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>In recent years, using orthogonal matrices has been shown to be a promising approach to improving recurrent neural networks (RNNs) with training, stability, and convergence, particularly to control gradients. While gated recurrent unit (GRU) and long short-term memory (LSTM) architectures address the vanishing gradient problem by using a variety of gates and memory cells, they are still prone to the exploding gradient problem. In this work, we analyze the gradients in GRU and propose the use of orthogonal matrices to prevent exploding gradient problems and enhance long-term memory. We study where to use orthogonal matrices and propose a Neumann series\u2013based scaled Cayley transformation for training orthogonal matrices in GRU, which we call Neumann-Cayley orthogonal GRU (NC-GRU). We present detailed experiments of our model on several synthetic and real-world tasks, which show that NC-GRU significantly outperforms GRU and several other RNNs.<\/jats:p>","DOI":"10.1162\/neco_a_01710","type":"journal-article","created":{"date-parts":[[2024,9,23]],"date-time":"2024-09-23T17:07:37Z","timestamp":1727111257000},"page":"2651-2676","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":3,"title":["Orthogonal Gated Recurrent Unit With Neumann-Cayley Transformation"],"prefix":"10.1162","volume":"36","author":[{"given":"Vasily","family":"Zadorozhnyy","sequence":"first","affiliation":[{"name":"SRI International, Princeton, NJ 08540, U.S.A. vasily.zadorozhnyy@sri.com"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Edison","family":"Mucllari","sequence":"additional","affiliation":[{"name":"Department of Mathematics, University of Kentucky, Lexington, KY 40506, U.S.A. edison.mucllari@uky.edu"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cole","family":"Pospisil","sequence":"additional","affiliation":[{"name":"NSWC Crane, Crane, IN 47522, U.S.A. pospisilcole@gmail.com"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Duc","family":"Nguyen","sequence":"additional","affiliation":[{"name":"The University of Tennessee, Knoxville, TN 37996, U.S.A. ducnguyen@utk.edu"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Qiang","family":"Ye","sequence":"additional","affiliation":[{"name":"Department of Mathematics, University of Kentucky, Lexington, KY 40506, U.S.A. qye3@uky.edu"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2024,11,19]]},"reference":[{"key":"2024111519195567600_bib1","first-page":"1120","article-title":"Unitary evolution recurrent neural networks","volume-title":"Proceedings of the 33rd International Conference on Machine Learning","author":"Arjovsky","year":"2016"},{"key":"2024111519195567600_bib2","author":"Ba","year":"2016","journal-title":"Layer normalization"},{"key":"2024111519195567600_bib3","author":"Bai","year":"2018","journal-title":"An empirical evaluation of generic convolutional and recurrent networks for sequence modeling"},{"key":"2024111519195567600_bib4","doi-asserted-by":"crossref","first-page":"1183","DOI":"10.1109\/ICNN.1993.298725","article-title":"The problem of learning long-term dependencies in recurrent networks","volume-title":"Proceedings of 1993 IEEE International Conference on Neural Networks","author":"Bengio","year":"1993"},{"key":"2024111519195567600_bib5","author":"Cho","year":"2014","journal-title":"Learning phrase representations using RNN encoder-decoder for statistical machine translation"},{"key":"2024111519195567600_bib6","article-title":"Recurrent batch normalization","volume-title":"Proceedings of the 5th International Conference on Learning Representations","author":"Cooijmans","year":"2017"},{"key":"2024111519195567600_bib7","doi-asserted-by":"crossref","DOI":"10.1137\/1.9781611971446","volume-title":"Applied numerical linear algebra","author":"Demmel","year":"1997"},{"key":"2024111519195567600_bib8","article-title":"Orthogonal Laguerre current neural networks","author":"Dorado-Rojas","year":"2020","journal-title":"Proceedings of the 34th Conference on Neural Information Processing Systems"},{"key":"2024111519195567600_bib9","author":"Foerster","year":"2016","journal-title":"Input switched affine networks: An MN architecture designed for interpretability"},{"key":"2024111519195567600_bib10","article-title":"Orthogonal recurrent neural networks with scaled Cayley transform","volume-title":"Proceedings of ICML 2018","author":"Helfrich","year":"2018"},{"key":"2024111519195567600_bib11","first-page":"4115","article-title":"Eigenvalue normalized recurrent neural networks for short term memory","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","author":"Helfrich","year":"2020"},{"key":"2024111519195567600_bib12","article-title":"Recurrent orthogonal networks and long-memory tasks","volume-title":"Proceedings of the 33rd International Conference on Machine Learning","author":"Henaff","year":"2017"},{"issue":"8","key":"2024111519195567600_bib13","doi-asserted-by":"publisher","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Computation"},{"issue":"8","key":"2024111519195567600_bib14","doi-asserted-by":"publisher","first-page":"2554","DOI":"10.1073\/pnas.79.8.2554","article-title":"Neural networks and physical systems with emergent collective computational abilities","volume":"79","author":"Hopfield","year":"1982","journal-title":"Proceedings of the National Academy of Sciences"},{"key":"2024111519195567600_bib15","first-page":"2050","article-title":"Learning unitary operators with help from u(n)","volume-title":"Proceedings of the 31st AAAI Conference on Artificial Intelligence","author":"Hyland","year":"2017"},{"key":"2024111519195567600_bib16","author":"Ioffe","year":"2015","journal-title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift"},{"issue":"4","key":"2024111519195567600_bib17","doi-asserted-by":"publisher","first-page":"765","DOI":"10.1162\/neco_a_01174","article-title":"Gated orthogonal recurrent units: On learning to forget","volume":"31","author":"Jing","year":"2019","journal-title":"Neural Computation"},{"key":"2024111519195567600_bib18","first-page":"1733","article-title":"Tunable efficient unitary neural networks (EUNN) and their application to RNNs","volume-title":"Proceedings of the 34th International Conference on Machine Learning","author":"Jing","year":"2017"},{"issue":"2","key":"2024111519195567600_bib19","doi-asserted-by":"publisher","first-page":"335","DOI":"10.1016\/j.laa.2005.08.027","article-title":"Is there a small skew Cayley transform with zero diagonal?","volume":"417","author":"Kahan","year":"2006","journal-title":"Linear Algebra and Its Applications"},{"key":"2024111519195567600_bib20","article-title":"Adam: A method for stochastic optimization","author":"Kingma","year":"2014"},{"key":"2024111519195567600_bib21","first-page":"3794","article-title":"Cheap orthogonal constraints in neural networks: A simple parameterization of the orthogonal and unitary group","volume-title":"Proceedings of the 36th International Conference on Machine Learning","author":"Lezcano-Casado","year":"2019"},{"key":"2024111519195567600_bib22","first-page":"4528","article-title":"Complex unitary recurrent neural networks using scaled Cayley transform","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","author":"Maduranga","year":"2019"},{"issue":"2","key":"2024111519195567600_bib23","first-page":"313","article-title":"Building a large annotated corpus of English: The Penn Treebank","volume":"19","author":"Marcus","year":"1993","journal-title":"Computational Linguistics"},{"issue":"2","key":"2024111519195567600_bib24","first-page":"313","article-title":"Building a large annotated corpus of English: The Penn Treebank","volume":"19","author":"Marcus","year":"1993","journal-title":"Computational Linguistics"},{"key":"2024111519195567600_bib25","article-title":"Regularizing and optimizing LSTM language models","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Merity","year":"2018"},{"key":"2024111519195567600_bib26","author":"Merity","year":"2016","journal-title":"Pointer sentinel mixture models"},{"key":"2024111519195567600_bib27","first-page":"2401","article-title":"Efficient orthogonal parameterisation of recurrent neural networks using householder reflections","volume-title":"Proceedings of ICML 2017","author":"Mhammedi","year":"2017"},{"key":"2024111519195567600_bib28","first-page":"1924","article-title":"MomentumRNN: Integrating momentum into recurrent neural networks","volume-title":"Advances in neural information processing systems","author":"Nguyen","year":"2020"},{"key":"2024111519195567600_bib29","volume-title":"Learning internal representations by error propagation","author":"Rumelhart","year":"1986"},{"key":"2024111519195567600_bib30","article-title":"Weight normalization: A simple reparameterization to accelerate training of deep neural networks","volume":"29","author":"Salimans","year":"2016","journal-title":"Advances in neural information processing systems"},{"key":"2024111519195567600_bib31","author":"Saxe","year":"2014","journal-title":"Exact solutions to nonlinear dynamics of learning in deep linear neural networks"},{"key":"2024111519195567600_bib32","author":"Tagare","year":"2011","journal-title":"Notes on optimization on Stiefel manifolds"},{"issue":"2","key":"2024111519195567600_bib33","article-title":"Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude","volume":"4","author":"Tieleman","year":"2012","journal-title":"COURSERA: Neural Networks for Machine Learning"},{"key":"2024111519195567600_bib34","author":"Ulyanov","year":"2017","journal-title":"Instance normalization: The missing ingredient for fast stylization"},{"key":"2024111519195567600_bib35","first-page":"3570","article-title":"On orthogonality and learning recurrent networks with long term dependencies","volume-title":"Proceedings of the 34th International Conference on Machine Learning","author":"Vorontsov","year":"2017"},{"key":"2024111519195567600_bib36","first-page":"4880","article-title":"Full-capacity unitary recurrent neural networks","volume-title":"Advances in neural information processing systems","author":"Wisdom","year":"2016"},{"key":"2024111519195567600_bib37","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-030-01261-8_1","article-title":"Group normalization","volume-title":"Proceedings of the European Conference on Computer Vision","author":"Wu","year":"2018"},{"key":"2024111519195567600_bib38","article-title":"Understanding and improving layer normalization","volume-title":"Advances in neural information processing systems","author":"Xu","year":"2019"}],"container-title":["Neural Computation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/direct.mit.edu\/neco\/article-pdf\/36\/12\/2651\/2479594\/neco_a_01710.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/direct.mit.edu\/neco\/article-pdf\/36\/12\/2651\/2479594\/neco_a_01710.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,11,15]],"date-time":"2024-11-15T19:20:15Z","timestamp":1731698415000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/neco\/article\/36\/12\/2651\/124541\/Orthogonal-Gated-Recurrent-Unit-With-Neumann"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,11,19]]},"references-count":38,"journal-issue":{"issue":"12","published-online":{"date-parts":[[2024,11,19]]},"published-print":{"date-parts":[[2024,11,19]]}},"URL":"https:\/\/doi.org\/10.1162\/neco_a_01710","relation":{},"ISSN":["0899-7667","1530-888X"],"issn-type":[{"value":"0899-7667","type":"print"},{"value":"1530-888X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,12]]},"published":{"date-parts":[[2024,11,19]]}}}