{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T03:38:43Z","timestamp":1760240323063,"version":"build-2065373602"},"reference-count":32,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2019,5,13]],"date-time":"2019-05-13T00:00:00Z","timestamp":1557705600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Beijing Science and Technology Planning Program of China","award":["Z171100004717001"],"award-info":[{"award-number":["Z171100004717001"]}]},{"DOI":"10.13039\/501100004826","name":"Beijing Natural Science Foundation","doi-asserted-by":"publisher","award":["4172002"],"award-info":[{"award-number":["4172002"]}],"id":[{"id":"10.13039\/501100004826","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61701009"],"award-info":[{"award-number":["61701009"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>This paper presents a new algorithm based on the theory of mutual information and information geometry. This algorithm places emphasis on adaptive mutual information estimation and maximum likelihood estimation. With the theory of information geometry, we adjust the mutual information along the geodesic line. Finally, we evaluate our proposal using empirical datasets that are dedicated for classification and regression. The results show that our algorithm contributes to a significant improvement over existing methods.<\/jats:p>","DOI":"10.3390\/a12050103","type":"journal-article","created":{"date-parts":[[2019,5,14]],"date-time":"2019-05-14T10:42:33Z","timestamp":1557830553000},"page":"103","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Improved Neural Networks Based on Mutual Information via Information Geometry"],"prefix":"10.3390","volume":"12","author":[{"given":"Meng","family":"Wang","sequence":"first","affiliation":[{"name":"College of Computer Science, Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chuang-Bai","family":"Xiao","sequence":"additional","affiliation":[{"name":"College of Computer Science, Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Zhen-Hu","family":"Ning","sequence":"additional","affiliation":[{"name":"College of Computer Science, Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jing","family":"Yu","sequence":"additional","affiliation":[{"name":"College of Computer Science, Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ya-Hao","family":"Zhang","sequence":"additional","affiliation":[{"name":"State Grid Information &amp; Telecommunication Co.,Ltd., 1401 Main Building No 2 BaiGuang Avenue, Xi Cheng District, Beijing 100031, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jin","family":"Pang","sequence":"additional","affiliation":[{"name":"State Grid Information &amp; Telecommunication Co.,Ltd., 1401 Main Building No 2 BaiGuang Avenue, Xi Cheng District, Beijing 100031, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2019,5,13]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"2093","DOI":"10.1109\/TPAMI.2009.67","article-title":"Fisher information nonparametric embedding","volume":"31","author":"Carter","year":"2009","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_2","first-page":"312","article-title":"Nonlinear System Identification Using Neural Networks","volume":"13","author":"Vapourware","year":"2007","journal-title":"Int. J. Res."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"494","DOI":"10.1109\/TNNLS.2016.2617239","article-title":"Guest Editorial Special Issue on New Developments in Neural Network Structures for Signal Processing, Autonomous Decision, and Adaptive Control","volume":"28","author":"Song","year":"2017","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_4","unstructured":"Pardey, J., Roberts, S., and Tarassenko, L. (1994, January 15). Application of artificial neural networks to medical signal processing. Proceedings of the Application of Artificial Neural Networks to Medical Signal Processing, London, UK."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"504","DOI":"10.1126\/science.1127647","article-title":"Reducing the dimensionality of data with neural networks","volume":"313","author":"Hinton","year":"2006","journal-title":"Science"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Larochelle, H., and Bengio, Y. (2008, January 5\u20139). Classification using discriminative restricted Boltzmann machines (PDF). Proceedings of the 25th International Conference on Machine Learning, Helsinki, Finland.","DOI":"10.1145\/1390156.1390224"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"18","DOI":"10.4249\/scholarpedia.1888","article-title":"Recurrent neural networks","volume":"8","author":"Grossberg","year":"2013","journal-title":"Scholarpedia."},{"key":"ref_8","first-page":"12","article-title":"Evolving Neural Network Ensembles by Minimization of Mutual Information","volume":"1","author":"Yao","year":"2004","journal-title":"Int. J. Hybrid Intell. Syst."},{"key":"ref_9","unstructured":"Zhang, Z., Chen, Q., and Qiao, L. (2010, January 23\u201326). A Merging and Splitting Algorithm Based on Mutual Information for Design Neural Networks. Proceedings of the IEEE Fifth International Conference on Bio-inspired Computing, Theories & Applications, Changsha, China."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1016\/j.neunet.2014.09.004","article-title":"Measuring the usefulness of hidden units in Boltzmann machines with mutual information","volume":"12","author":"Berglund","year":"2015","journal-title":"Neural Netw."},{"key":"ref_11","unstructured":"Niles, L.T., Silverman, H.F., and Bush, M.A. (1990, January 3\u20136). Neural networks, Maximum Mutual Information Training, and Maximum Likelihood Training. Proceedings of the International Conference on Acoustics, Speech, and Signal Processing, Albuquerque, NM, USA."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Arwini, K., and Dodson, C.T. (2008). Information Geometry: Near Randomness and Near Independence, Springer.","DOI":"10.1007\/978-3-540-69393-2"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Frieden, B.R. (2004). Science from Fisher Information: A Unification, Cambridge University Press.","DOI":"10.1017\/CBO9780511616907"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Kass, R.E., and Vos, P.W. (1997). Geometrical Foundations of Asymptotic Inference, Wiley.","DOI":"10.1002\/9781118165980"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"29","DOI":"10.2307\/3318651","article-title":"Information geometry of estimating functions in semiparametric statistical models","volume":"3","author":"Amari","year":"1997","journal-title":"Bernoulli"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"251","DOI":"10.1162\/089976698300017746","article-title":"Natural gradient works efficiently in learning","volume":"10","author":"Amari","year":"1998","journal-title":"Neural Comput."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1016\/0020-0255(85)90050-7","article-title":"The relation between information theory and the differential, geometry approach to statistics","volume":"35","author":"Campbell","year":"1985","journal-title":"Inf. Sci."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1701","DOI":"10.1109\/18.930911","article-title":"Information geometry on hierarchy of probability distributions","volume":"47","author":"Amari","year":"2001","journal-title":"IEEE Trans. Inf. Theory"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"443","DOI":"10.1016\/0893-6080(91)90040-C","article-title":"Dualistic Geometry of the Manifold of Higher-Order Neurons","volume":"4","author":"Amari","year":"1991","journal-title":"Neural Netw."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"481","DOI":"10.1162\/NECO_a_00689","article-title":"Natural Gradient Learning Algorithms for RBF Networks","volume":"27","author":"Zhao","year":"2015","journal-title":"Neural Comput."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1309","DOI":"10.1016\/j.neunet.2005.01.008","article-title":"A novel information geometric approach to variable selection in MLP networks","volume":"18","author":"Eleuteri","year":"2005","journal-title":"Neural Netw."},{"key":"ref_22","first-page":"145","article-title":"Distribution of Mutual Information","volume":"10","author":"Hutter","year":"2001","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1016\/0024-3795(80)90226-8","article-title":"Conjugate gradient type methods for unsymmetric and inconsistent systems of linear equations","volume":"34","author":"Axelsson","year":"1980","journal-title":"Lin. Alg. Appl."},{"key":"ref_24","unstructured":"Press, W.H., Teukolsky, S.A., Vetterling, W.T., and Flannery, B.P. (2007). Kullback\u2013Leibler Distance. Numerical Recipes. The Art of Scientific Computing (3rd ed.), Cambridge University Press."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Cox, D.R., and Hinkley, D.V. (1974). Theoritical Statistics, Chapman & Hall\/CRC.","DOI":"10.1007\/978-1-4899-2887-0"},{"key":"ref_26","unstructured":"Ravanbakhsh, S., Poczos, B., Schneider, J., Schuurmans, D., and Greiner, R. (2016, January 9\u201311). Stochastic Neural Networks with Monotonic Activation Functions. Proceedings of the 19th International Conference on Artificial Intelligence and Statistics (AISTATS), Cadiz, Spain."},{"key":"ref_27","unstructured":"Amari, S. (2002, January 20\u201325). Information geometry of statistical inference-an overview. Proceedings of the IEEE Information Theory Workshop, Bangalore, India."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1016\/0893-9659(94)00112-P","article-title":"Salicrij. Statistical tests based on geodesic distances","volume":"8","author":"Menendez","year":"1995","journal-title":"Appl. Math. Lett."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Cam, L.L. (1986). Asymptotic Methods in Statistical Decision Theory, Springer.","DOI":"10.1007\/978-1-4612-4946-7"},{"key":"ref_30","first-page":"785","article-title":"Step-sizes for the gradient method","volume":"42","author":"Yuan","year":"1999","journal-title":"AMS\/IP Stud. Adv. Math."},{"key":"ref_31","first-page":"32","article-title":"A fast learning algorithm for deep belief nets","volume":"3","author":"Hinton","year":"2006","journal-title":"Neural Comput."},{"key":"ref_32","first-page":"67","article-title":"Optimizing number of hidden neurons in neural networks","volume":"2","author":"Liu","year":"2007","journal-title":"Artif. Intell. Appl."}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/12\/5\/103\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T12:51:36Z","timestamp":1760187096000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/12\/5\/103"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,5,13]]},"references-count":32,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2019,5]]}},"alternative-id":["a12050103"],"URL":"https:\/\/doi.org\/10.3390\/a12050103","relation":{},"ISSN":["1999-4893"],"issn-type":[{"type":"electronic","value":"1999-4893"}],"subject":[],"published":{"date-parts":[[2019,5,13]]}}}