{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,13]],"date-time":"2026-08-13T03:49:55Z","timestamp":1786592995484,"version":"build-2736575974"},"reference-count":27,"publisher":"MIT Press - Journals","issue":"4","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Neural Computation"],"published-print":{"date-parts":[[2019,4]]},"abstract":"<jats:p> We present a novel recurrent neural network (RNN)\u2013based model that combines the remembering ability of unitary evolution RNNs with the ability of gated RNNs to effectively forget redundant or irrelevant information in its memory. We achieve this by extending restricted orthogonal evolution RNNs with a gating mechanism similar to gated recurrent unit RNNs with a reset gate and an update gate. Our model is able to outperform long short-term memory, gated recurrent units, and vanilla unitary or orthogonal RNNs on several long-term-dependency benchmark tasks. We empirically show that both orthogonal and unitary RNNs lack the ability to forget. This ability plays an important role in RNNs. We provide competitive results along with an analysis of our model on many natural sequential tasks, including question answering, speech spectrum prediction, character-level language modeling, and synthetic tasks that involve long-term dependencies such as algorithmic, denoising, and copying tasks. <\/jats:p>","DOI":"10.1162\/neco_a_01174","type":"journal-article","created":{"date-parts":[[2019,2,15]],"date-time":"2019-02-15T02:01:44Z","timestamp":1550196104000},"page":"765-783","source":"Crossref","is-referenced-by-count":60,"title":["Gated Orthogonal Recurrent Units: On Learning to Forget"],"prefix":"10.1162","volume":"31","author":[{"given":"Li","family":"Jing","sequence":"first","affiliation":[{"name":"Massachusetts Institute of Technology, Cambridge, MA 02139, U.S.A."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Caglar","family":"Gulcehre","sequence":"additional","affiliation":[{"name":"University of Montreal, Montreal H3T, 1J4, Quebec, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"John","family":"Peurifoy","sequence":"additional","affiliation":[{"name":"Massachusetts Institute of Technology, Cambridge, MA 02139, U.S.A."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yichen","family":"Shen","sequence":"additional","affiliation":[{"name":"Massachusetts Institute of Technology, Cambridge, MA 02139, U.S.A."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Max","family":"Tegmark","sequence":"additional","affiliation":[{"name":"Massachusetts Institute of Technology, Cambridge, MA 02139, U.S.A."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marin","family":"Soljacic","sequence":"additional","affiliation":[{"name":"Massachusetts Institute of Technology, Cambridge, MA 02139, U.S.A."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yoshua","family":"Bengio","sequence":"additional","affiliation":[{"name":"University of Montreal, Montreal H3T 1J4, Quebec, Canada"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","reference":[{"key":"B1","first-page":"1120","author":"Arjovsky M.","year":"2016","journal-title":"Proceedings of the 33rd International Conference on Machine Learning"},{"key":"B2","doi-asserted-by":"publisher","DOI":"10.1109\/72.279181"},{"key":"B3","author":"Chan W.","year":"2016","journal-title":"Proceedings of the IEEE Conference on Acoustics, Speech, and Signal Processing"},{"key":"B4","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/W14-4012"},{"key":"B5","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/D14-1179"},{"key":"B6","first-page":"577","volume-title":"Advances in neural information processing systems","author":"Chorowski J. K.","year":"2015"},{"key":"B7","doi-asserted-by":"publisher","DOI":"10.6028\/NIST.IR.4930"},{"key":"B8","author":"Gers F.","year":"2001","journal-title":"Long short-term memory in recurrent neural networks"},{"key":"B9","doi-asserted-by":"publisher","DOI":"10.1162\/089976600300015015"},{"key":"B10","author":"Graves A.","year":"2014","journal-title":"Neural Turing Machines"},{"key":"B11","first-page":"2034","volume-title":"Proceedings of the 33rd International Conference on Machine Learning","author":"Henaff M.","year":"2016"},{"key":"B12","author":"Hochreiter S.","year":"1991","journal-title":"Untersuchungen zu dynamischen neuronalen netzen"},{"key":"B13","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1997.9.8.1735"},{"key":"B14","first-page":"2050","author":"Hyland S.","year":"2017","journal-title":"Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence"},{"key":"B15","author":"Jing L.","year":"2016","journal-title":"Tunable efficient unitary neural networks (EUNN) and their application to RNNS"},{"key":"B16","doi-asserted-by":"publisher","DOI":"10.1016\/S0166-4115(97)80111-2"},{"key":"B17","author":"Karpathy A.","year":"2015","journal-title":"Visualizing and understanding recurrent networks"},{"key":"B18","author":"Kingma D. P.","year":"2014","journal-title":"CoRR"},{"key":"B19","author":"Le Q. V.","year":"2015","journal-title":"A simple way to initialize recurrent networks of rectified linear units"},{"key":"B20","author":"Li Y.","year":"2015","journal-title":"Gated graph sequence neural networks"},{"key":"B21","first-page":"313","volume":"19","author":"Marcus M.","year":"1993","journal-title":"Computational Linguistics"},{"key":"B22","author":"Mhammedi Z.","year":"2017","journal-title":"Proceedings of the International Conference on Machine Learning"},{"key":"B23","author":"Pascanu R.","year":"2013","journal-title":"Proceedings of the International Conference on Machine Learning"},{"key":"B24","first-page":"3104","volume-title":"Advances in neural information processing systems","volume":"27","author":"Sutskever I.","year":"2014"},{"issue":"2","key":"B25","first-page":"26","volume":"4","author":"Tieleman T.","year":"2012","journal-title":"COURSERA: Neural Networks for Machine Learning"},{"key":"B26","author":"Weston J.","year":"2015","journal-title":"Towards AI-complete question answering: A set of prerequisite toy tasks"},{"key":"B27","first-page":"4880","volume":"29","author":"Wisdom S.","year":"2016","journal-title":"Advances in neural information processing systems"}],"container-title":["Neural Computation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/neco_a_01174","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,3,12]],"date-time":"2021-03-12T21:43:03Z","timestamp":1615585383000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/neco\/article\/31\/4\/765-783\/8458"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,4]]},"references-count":27,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2019,4]]}},"alternative-id":["10.1162\/neco_a_01174"],"URL":"https:\/\/doi.org\/10.1162\/neco_a_01174","relation":{},"ISSN":["0899-7667","1530-888X"],"issn-type":[{"value":"0899-7667","type":"print"},{"value":"1530-888X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,4]]}}}