{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,10]],"date-time":"2026-02-10T19:26:55Z","timestamp":1770751615495,"version":"3.50.0"},"reference-count":38,"publisher":"MIT Press","issue":"9","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Neural Computation"],"published-print":{"date-parts":[[2018,9]]},"abstract":"<jats:p>Rule extraction from black box models is critical in domains that require model validation before implementation, as can be the case in credit scoring and medical diagnosis. Though already a challenging problem in statistical learning in general, the difficulty is even greater when highly nonlinear, recursive models, such as recurrent neural networks (RNNs), are fit to data. Here, we study the extraction of rules from second-order RNNs trained to recognize the Tomita grammars. We show that production rules can be stably extracted from trained RNNs and that in certain cases, the rules outperform the trained RNNs.<\/jats:p>","DOI":"10.1162\/neco_a_01111","type":"journal-article","created":{"date-parts":[[2018,7,18]],"date-time":"2018-07-18T23:43:25Z","timestamp":1531957405000},"page":"2568-2591","source":"Crossref","is-referenced-by-count":31,"title":["An Empirical Evaluation of Rule Extraction from Recurrent Neural Networks"],"prefix":"10.1162","volume":"30","author":[{"given":"Qinglong","family":"Wang","sequence":"first","affiliation":[{"name":"McGill University, Montreal, Quebec H3A 0E9, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kaixuan","family":"Zhang","sequence":"additional","affiliation":[{"name":"Pennsylvania State University, University Park, PA 16802, U.S.A."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alexander G.","family":"Ororbia II","sequence":"additional","affiliation":[{"name":"Pennsylvania State University, University Park, PA 16802, U.S.A."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xinyu","family":"Xing","sequence":"additional","affiliation":[{"name":"Pennsylvania State University, University Park, PA 16802, U.S.A."}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xue","family":"Liu","sequence":"additional","affiliation":[{"name":"McGill University, Montreal, Quebec H3A 0E9, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"C. Lee","family":"Giles","sequence":"additional","affiliation":[{"name":"Pennsylvania State University, University Park, PA 16802, U.S.A."}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"281","reference":[{"key":"B1","doi-asserted-by":"publisher","DOI":"10.1109\/TNN.2011.2170180"},{"key":"B2","volume-title":"Theory of finite automata with an introduction to formal languages","author":"Carroll J.","year":"1989"},{"key":"B3","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1996.8.6.1135"},{"key":"B4","author":"Cho K.","year":"2014","journal-title":"On the properties of neural machine translation: Encoder-decoder approaches"},{"key":"B5","doi-asserted-by":"publisher","DOI":"10.1109\/TIT.1956.1056813"},{"key":"B6","author":"Dhingra B.","year":"2017","journal-title":"Linguistic knowledge as memory for recurrent neural networks"},{"key":"B7","author":"Du J.","year":"2016","journal-title":"Convergence analysis of distributed inference with vector-valued gaussian belief propagation"},{"key":"B8","author":"Du J.","year":"2017","journal-title":"Topology adaptive graph convolutional networks"},{"key":"B9","doi-asserted-by":"publisher","DOI":"10.1207\/s15516709cog1402_1"},{"key":"B10","doi-asserted-by":"publisher","DOI":"10.1007\/BF00116897"},{"key":"B11","doi-asserted-by":"publisher","DOI":"10.1109\/IJCNN.1991.155350"},{"key":"B12","doi-asserted-by":"publisher","DOI":"10.1023\/A:1010884214864"},{"key":"B13","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1992.4.3.393"},{"key":"B14","first-page":"380","volume-title":"Advances in neural information processing systems","volume":"2","author":"Giles C. L.","year":"1989"},{"key":"B15","doi-asserted-by":"publisher","DOI":"10.1109\/72.668898"},{"key":"B16","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1997.9.8.1735"},{"key":"B17","volume-title":"Introduction to automata theory, languages, and computation","author":"Hopcroft J. E.","year":"2013"},{"key":"B18","doi-asserted-by":"publisher","DOI":"10.1162\/0899766053630350"},{"key":"B19","first-page":"501","volume-title":"Advances in neural information processing systems","volume":"7","author":"Kolen J. F.","year":"1994"},{"key":"B20","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2015.2418323"},{"key":"B21","doi-asserted-by":"publisher","DOI":"10.1109\/72.548162"},{"key":"B22","unstructured":"Minsky, M. L. (1967). Computation: Finite and infinite machines. Upper Saddle River, NJ: Prentice Hall."},{"key":"B23","author":"Murdoch W. J.","year":"2017","journal-title":"Automatic rule extraction from long short term memory networks"},{"key":"B24","doi-asserted-by":"publisher","DOI":"10.1145\/235809.235811"},{"key":"B25","doi-asserted-by":"publisher","DOI":"10.1016\/0893-6080(95)00086-0"},{"key":"B26","doi-asserted-by":"crossref","first-page":"63","DOI":"10.7551\/mitpress\/4070.003.0005","volume-title":"Knowledge based neurocomputing","author":"Omlin C. W.","year":"2000"},{"key":"B27","first-page":"1310","author":"Pascanu R.","year":"2013","journal-title":"Proceedings of the International Conference on Machine Learning"},{"key":"B28","doi-asserted-by":"publisher","DOI":"10.1007\/BF00114845"},{"key":"B29","volume-title":"Shape, structure and pattern recognition","author":"Sanfeliu A.","year":"1994"},{"key":"B30","doi-asserted-by":"publisher","DOI":"10.3115\/1603899.1603912"},{"key":"B31","first-page":"2440","volume-title":"Advances in neural information processing systems","volume":"28","author":"Sukhbaatar S.","year":"2015"},{"issue":"2","key":"B32","first-page":"26","volume":"4","author":"Tieleman T.","year":"2012","journal-title":"COURSERA: Neural Networks for Machine Learning"},{"key":"B33","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1995.7.4.822"},{"key":"B34","first-page":"105","volume-title":"Proceedings of the Fourth Annual Cognitive Science Conference","author":"Tomita M.","year":"1982"},{"key":"B35","author":"Wang Q.","year":"2018","journal-title":"A comparison of rule extraction for different recurrent neural network models and grammatical complexity"},{"key":"B36","first-page":"309","volume-title":"Advances in neural information processing systems","volume":"4","author":"Watrous R. L.","year":"1992"},{"key":"B37","author":"Weston J.","year":"2014","journal-title":"Memory networks"},{"key":"B38","doi-asserted-by":"publisher","DOI":"10.1162\/neco.1993.5.6.976"}],"container-title":["Neural Computation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/neco_a_01111","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,7,8]],"date-time":"2024-07-08T10:54:40Z","timestamp":1720436080000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/neco\/article\/30\/9\/2568-2591\/8390"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,9]]},"references-count":38,"journal-issue":{"issue":"9","published-print":{"date-parts":[[2018,9]]}},"alternative-id":["10.1162\/neco_a_01111"],"URL":"https:\/\/doi.org\/10.1162\/neco_a_01111","relation":{},"ISSN":["0899-7667","1530-888X"],"issn-type":[{"value":"0899-7667","type":"print"},{"value":"1530-888X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,9]]}}}