{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,9]],"date-time":"2026-07-09T15:20:50Z","timestamp":1783610450067,"version":"3.55.0"},"reference-count":63,"publisher":"MIT Press - Journals","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Transactions of the Association for Computational Linguistics"],"published-print":{"date-parts":[[2019,11]]},"abstract":"<jats:p> Stacking long short-term memory (LSTM) cells or gated recurrent units (GRUs) as part of a recurrent neural network (RNN) has become a standard approach to solving a number of tasks ranging from language modeling to text summarization. Although LSTMs and GRUs were designed to model long-range dependencies more accurately than conventional RNNs, they nevertheless have problems copying or recalling information from the long distant past. Here, we derive a phase-coded representation of the memory state, Rotational Unit of Memory (RUM), that unifies the concepts of unitary learning and associative memory. We show experimentally that RNNs based on RUMs can solve basic sequential tasks such as memory copying and memory recall much better than LSTMs\/GRUs. We further demonstrate that by replacing LSTM\/GRU with RUM units we can apply neural networks to real-world problems such as language modeling and text summarization, yielding results comparable to the state of the art. <\/jats:p>","DOI":"10.1162\/tacl_a_00258","type":"journal-article","created":{"date-parts":[[2019,4,18]],"date-time":"2019-04-18T14:32:46Z","timestamp":1555597966000},"page":"121-138","source":"Crossref","is-referenced-by-count":13,"title":["Rotational Unit of Memory: A Novel Representation Unit for RNNs with Scalable Applications"],"prefix":"10.1162","volume":"7","author":[{"given":"Rumen","family":"Dangovski","sequence":"first","affiliation":[{"name":"Massachusetts Institute of Technology."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Li","family":"Jing","sequence":"additional","affiliation":[{"name":"Massachusetts Institute of Technology."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Preslav","family":"Nakov","sequence":"additional","affiliation":[{"name":"Qatar Computing Research Institute, HBKU."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mi\u0107o","family":"Tatalovi\u0107","sequence":"additional","affiliation":[{"name":"Massachusetts Institute of Technology"},{"name":"Association of British Science Writers."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Marin","family":"Solja\u010di\u0107","sequence":"additional","affiliation":[{"name":"Massachusetts Institute of Technology."}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","reference":[{"key":"bib2","first-page":"1120","volume-title":"Proceedings of the 33rd International Conference on International Conference on Machine Learning","author":"Arjovsky Martin","year":"2016"},{"key":"bib3","volume-title":"Algebra","author":"Artin Michael","year":"2011"},{"key":"bib4","first-page":"4331","volume-title":"Proceedings of the Annual Conference on Neural Information Processing Systems: Advances in Neural Information Processing Systems 29","author":"Ba Jimmy","year":"2016"},{"key":"bib5","first-page":"4880","volume-title":"Proceedings of the Annual Conference on Neural Information Processing Systems: Advances in Neural Information Processing Systems 29","author":"Ba Jimmy Lei","year":"2016"},{"key":"bib6","volume-title":"Proceedings of the 3rd International Conference on Learning Representations","author":"Bahdanau Dzmitry","year":"2015"},{"key":"bib7","first-page":"1292","volume-title":"Proceedings of the 33rd International Conference on Machine Learning","author":"Balduzzi David","year":"2016"},{"issue":"2","key":"bib8","doi-asserted-by":"crossref","first-page":"157","DOI":"10.1109\/72.279181","volume":"5","author":"Bengio Yoshua","year":"1994","journal-title":"IEEE Transactions on Neural Networks"},{"key":"bib9","first-page":"1662","volume-title":"Proceedings of the 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Celikyilmaz Asli","year":"2018"},{"key":"bib10","first-page":"675","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics","author":"Chen Yen-Chun","year":"2018"},{"key":"bib11","first-page":"1724","volume-title":"Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing","author":"Cho Kyunghyun","year":"2014"},{"key":"bib12","volume-title":"Proceedings of the 5th International Conference on Learning Representations","author":"Chung Junyoung","year":"2017"},{"key":"bib13","doi-asserted-by":"crossref","first-page":"1460","DOI":"10.1364\/OPTICA.3.001460","volume":"3","author":"Clements William R.","year":"2016","journal-title":"Optica"},{"key":"bib14","volume-title":"Proceedings of the 6th International Conference on Learning Representations","author":"Cohen Taco S.","year":"2018"},{"key":"bib15","first-page":"1986","volume-title":"Proceedings of the 33rd International Conference on Machine Learning, ICMvL \u201916","author":"Danihelka Ivo","year":"2016"},{"key":"bib16","author":"Graves Alex","year":"2013","journal-title":"arXiv preprint arXiv:1308.0850"},{"key":"bib17","author":"Graves Alex","year":"2014","journal-title":"arXiv preprint arXiv:1410.5401"},{"key":"bib18","doi-asserted-by":"crossref","first-page":"471","DOI":"10.1038\/nature20101","volume":"538","author":"Graves Alex","year":"2016","journal-title":"Nature"},{"key":"bib19","volume-title":"Proceedings of the 4th International Conference on Learning Representations","author":"Ha David","year":"2016"},{"key":"bib20","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-319-13467-3","volume-title":"Lie Groups, Lie Algebras, and Representations","author":"Hall Brian C.","year":"2015"},{"key":"bib21","first-page":"2034","volume-title":"Proceedings of the 33rd International Conference on International Conference on Machine Learning","author":"Henaff Mikael","year":"2016"},{"key":"bib22","volume-title":"Proceedings of the 5th International Conference on Learning Representations","author":"Henaff Mikael","year":"2017"},{"key":"bib23","first-page":"1693","volume-title":"Proceedings of the Annual Conference on Neural Information Processing Systems: Advances in Neural Information Processing Systems 28","author":"Hermann Karl Moritz","year":"2015"},{"key":"bib24","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1997.9.8.1735"},{"key":"bib25","author":"Li Jing","year":"2017","journal-title":"arXiv preprint arXiv:1706.02761"},{"key":"bib26","first-page":"1733","volume-title":"Proceedings of the 34th International Conference on Machine Learning","author":"Li Jing","year":"2017"},{"key":"bib27","volume-title":"Computational Rigid Vehicle Dynamics","author":"Katz Amnon","year":"2001"},{"key":"bib28","volume-title":"Proceedings of the 3rd International Conference on Learning Representations","author":"Kingma Diederik P.","year":"2015"},{"key":"bib29","doi-asserted-by":"crossref","first-page":"444","DOI":"10.1109\/T-C.1974.223960","volume":"23","author":"Kohonen Teuvo","year":"1974","journal-title":"IEEE Transactions on Computers"},{"key":"bib30","first-page":"1172","volume-title":"Proceedings of the Annual Conference on Neural Information Processing Systems: Advances in Neural Information Processing Systems 29","author":"Krotov Dmitry","year":"2016"},{"key":"bib31","volume-title":"Proceedings of the 5th International Conference on Learning Representations","author":"Krueger David","year":"2017"},{"key":"bib32","volume-title":"Quaternions and Rotation Sequences. A Primer with Applications to Orbits, Aerospace and Virtual Reality","author":"Kuipers Jack B.","year":"2002"},{"key":"bib33","first-page":"6391","volume-title":"Proceedings of the Annual Conference on Neural Information Processing Systems: Advances in Neural Information Processing Systems 31","author":"Li Hao","year":"2018"},{"key":"bib34","first-page":"71","volume-title":"Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics","author":"Lin Chin-Yew","year":"2003"},{"key":"bib35","doi-asserted-by":"crossref","first-page":"1412","DOI":"10.18653\/v1\/D15-1166","volume-title":"Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing","author":"Luong Minh-Thang","year":"2015"},{"key":"bib36","first-page":"313","volume":"19","author":"Marcus Mitchell P.","year":"1993","journal-title":"Computational Linguistics"},{"key":"bib37","author":"Merity Stephen","year":"2018","journal-title":"arXiv preprint arXiv:1803.08240"},{"key":"bib38","first-page":"5915","volume-title":"Proceedings of the Annual Conference on Neural Information Processing Systems: Advances in Neural Information Processing Systems 30","author":"Mujika Asier","year":"2017"},{"key":"bib39","first-page":"3075","volume-title":"Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence","author":"Nallapati Ramesh","year":"2017"},{"key":"bib40","doi-asserted-by":"crossref","first-page":"280","DOI":"10.18653\/v1\/K16-1028","volume-title":"Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning","author":"Nallapati Ramesh","year":"2016"},{"key":"bib41","first-page":"1747","volume-title":"Proceedings of the 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Narayan Shashi","year":"2018"},{"key":"bib42","doi-asserted-by":"crossref","first-page":"103","DOI":"10.1561\/1500000015","volume":"5","author":"Nenkova Ani","year":"2011","journal-title":"Foundations and Trends in Information Retrieval"},{"key":"bib44","first-page":"1","volume-title":"Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics","author":"Perez Julien","year":"2017"},{"key":"bib45","volume-title":"Holographic Reduced Representation: Distributed Representation for Cognitive Structures","author":"Plate Tony A.","year":"2003"},{"key":"bib46","doi-asserted-by":"crossref","first-page":"318","DOI":"10.7551\/mitpress\/5236.001.0001","volume-title":"Parallel Distributed Processing: Explorations in the Microstructure of Cognition","volume":"1","author":"Rumelhart David E.","year":"1986"},{"key":"bib47","volume-title":"Modern Quantum Mechanics","author":"Sakurai Jun J.","year":"2010"},{"key":"bib48","first-page":"1073","volume-title":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics","author":"See Abigail","year":"2017"},{"key":"bib49","volume-title":"Proceedings of the 5th International Conference on Learning Representations","author":"Seo Minjoon","year":"2017"},{"key":"bib50","volume-title":"Computer Vision","author":"Shapiro Linda G.","year":"2001"},{"issue":"15","key":"bib51","first-page":"1929","volume":"1","author":"Srivastava Nitish","year":"2014","journal-title":"Journal of Machine Learning Research"},{"key":"bib52","volume-title":"Proceedings of the Annual Conference on Neural Information Processing Systems: Advances in Neural Information Processing Systems 28","author":"Sukhbaatar Sainbayar","year":"2015"},{"key":"bib53","first-page":"3104","volume-title":"Proceedings of the Annual Conference on Neural Information Processing Systems: Advances in Neural Information Processing Systems 27","author":"Sutskever Ilya","year":"2014"},{"issue":"01","key":"bib54","volume":"17","author":"Tatalovi\u0107 Mi\u0107o","year":"2018","journal-title":"Journal of Science Communication"},{"key":"bib55","doi-asserted-by":"publisher","DOI":"10.1162\/089120102762671936"},{"key":"bib56","first-page":"163","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Vadapalli Raghuram","year":"2018"},{"key":"bib57","volume-title":"Proceedings of the Annual Conference on Neural Information Processing Systems: Advances in Neural Information Processing Systems 28","author":"Vinyals Oriol","year":"2015"},{"key":"bib58","first-page":"3570","volume-title":"Proceedings of the 34th International Conference on Machine Learning","author":"Vorontsov Eugene","year":"2017"},{"key":"bib59","doi-asserted-by":"crossref","first-page":"849","DOI":"10.1109\/CVPR.2018.00095","volume-title":"Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition","author":"Weiler Maurice","year":"2018"},{"key":"bib60","volume-title":"Proceedings of the 4th International Conference on Learning Representations","author":"Weston Jason","year":"2016"},{"key":"bib61","first-page":"4880","volume-title":"Proceedings of the Annual Conference on Neural Information Processing Systems: Advances in Neural Information Processing Systems 29","author":"Wisdom Scott","year":"2016"},{"key":"bib62","first-page":"5028","volume-title":"Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition","author":"Worrall Daniel E.","year":"2017"},{"key":"bib63","first-page":"2397","volume-title":"Proceedings of the 33rd International Conference on International Conference on Machine Learning","author":"Xiong Caiming","year":"2016"},{"key":"bib64","author":"Zhang Wei","year":"2017","journal-title":"arXiv preprint arXiv: 1709.06493"},{"key":"bib65","volume-title":"Proceedings of the 5th International Conference on Learning Representations","author":"Zoph Barret","year":"2017"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/tacl_a_00258","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,3,12]],"date-time":"2021-03-12T21:39:19Z","timestamp":1615585159000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/43512"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,11]]},"references-count":63,"alternative-id":["10.1162\/tacl_a_00258"],"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00258","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,11]]}}}