{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,23]],"date-time":"2026-08-23T16:53:30Z","timestamp":1787504010375,"version":"build-2736575974"},"reference-count":59,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,6,7]],"date-time":"2023-06-07T00:00:00Z","timestamp":1686096000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100012330","name":"Cognitive Sciences and Technologies Council","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100012330","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Comput. Neurosci."],"abstract":"<jats:p>Hierarchical Temporal Memory (HTM) is an unsupervised algorithm in machine learning. It models several fundamental neocortical computational principles. Spatial Pooler (SP) is one of the main components of the HTM, which continuously encodes streams of binary input from various layers and regions into sparse distributed representations. In this paper, the goal is to evaluate the sparsification in the SP algorithm from the perspective of information theory by the information bottleneck (IB), Cramer-Rao lower bound, and Fisher information matrix. This paper makes two main contributions. First, we introduce a new upper bound for the standard information bottleneck relation, which we refer to as modified-IB in this paper. This measure is used to evaluate the performance of the SP algorithm in different sparsity levels and various amounts of noise. The MNIST, Fashion-MNIST and NYC-Taxi datasets were fed to the SP algorithm separately. The SP algorithm with learning was found to be resistant to noise. Adding up to 40% noise to the input resulted in no discernible change in the output. Using the probabilistic mapping method and Hidden Markov Model, the sparse SP output representation was reconstructed in the input space. In the modified-IB relation, it is numerically calculated that a lower noise level and a higher sparsity level in the SP algorithm lead to a more effective reconstruction and SP with 2% sparsity produces the best results. Our second contribution is to prove mathematically that more sparsity leads to better performance of the SP algorithm. The data distribution was considered the Cauchy distribution, and the Cramer\u2013Rao lower bound was analyzed to estimate SP\u2019s output at different sparsity levels.<\/jats:p>","DOI":"10.3389\/fncom.2023.1140782","type":"journal-article","created":{"date-parts":[[2023,6,7]],"date-time":"2023-06-07T02:43:49Z","timestamp":1686105829000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":5,"title":["Information-theoretic analysis of Hierarchical Temporal Memory-Spatial Pooler algorithm with a new upper bound for the standard information bottleneck method"],"prefix":"10.3389","volume":"17","author":[{"given":"Shiva","family":"Sanati","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Modjtaba","family":"Rouhani","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ghosheh Abed","family":"Hodtani","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2023,6,7]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"308","DOI":"10.1016\/j.patcog.2019.01.032","article-title":"SSR2: Sparse signal recovery for single-image super-resolution on faces with extreme low resolutions.","volume":"90","author":"Abiantun","year":"2019","journal-title":"Pattern Recognit."},{"key":"B2","first-page":"1947","article-title":"Emergence of invariance and disentanglement in deep representations.","volume":"19","author":"Achille","year":"2018","journal-title":"J. Mach. Learn. Res."},{"key":"B3","article-title":"How do neurons operate on sparse distributed representations? A mathematical theory of sparsity, neurons and active dendrites.","author":"Ahmad","year":"2016","journal-title":"arXiv [Preprint]"},{"key":"B4","article-title":"How can we be so dense? The benefits of using highly sparse representations.","author":"Ahmad","year":"2019","journal-title":"arXiv [Preprint]"},{"key":"B5","article-title":"Deep variational information bottleneck.","author":"Alemi","year":"2016","journal-title":"arXiv [Preprint]"},{"key":"B6","doi-asserted-by":"publisher","first-page":"359","DOI":"10.1038\/s41583-021-00448-6","article-title":"Two views on the cognitive brain.","volume":"22","author":"Barack","year":"2021","journal-title":"Nat. Rev. Neurosci."},{"key":"B7","doi-asserted-by":"publisher","first-page":"1587","DOI":"10.3390\/e15051587","article-title":"Function identification in neuron populations via information bottleneck.","volume":"15","author":"Buddha","year":"2013","journal-title":"Entropy"},{"key":"B8","doi-asserted-by":"publisher","first-page":"524","DOI":"10.1016\/S1364-6613(02)02041-7","article-title":"Classical conditioning, awareness, and brain systems.","volume":"6","author":"Clark","year":"2002","journal-title":"Trends Cogn. Sci."},{"key":"B9","doi-asserted-by":"publisher","DOI":"10.3389\/fncom.2017.00111","article-title":"The HTM spatial pooler\u2014a neocortical algorithm for online sparse distributed coding.","volume":"11","author":"Cui","year":"2017","journal-title":"Front. Comput. Neurosci."},{"key":"B10","doi-asserted-by":"publisher","first-page":"6819","DOI":"10.1109\/TCOMM.2022.3200679","article-title":"Frequency estimation by interpolation of two fourier coefficients: Cram\u00e9r-Rao bound and maximum likelihood solution.","volume":"70","author":"D\u2019Amico","year":"2022","journal-title":"IEEE Trans. Commun."},{"key":"B11","doi-asserted-by":"publisher","first-page":"2365","DOI":"10.1049\/iet-ipr.2019.1312","article-title":"Sparse representation based computed tomography images reconstruction by coupled dictionary learning algorithm.","volume":"14","author":"Deeba","year":"2020","journal-title":"IET Image Process."},{"key":"B12","doi-asserted-by":"publisher","first-page":"1122","DOI":"10.1109\/78.923295","article-title":"Cramer-Rao bounds for estimating range, velocity, and direction with an active array.","volume":"49","author":"Dogandzic","year":"2001","journal-title":"IEEE Trans. Signal Process."},{"key":"B13","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1000062","article-title":"Synaptic learning rules and sparse coding in a model sensory system.","volume":"4","author":"Finelli","year":"2008","journal-title":"PLoS Comput. Biol."},{"key":"B14","doi-asserted-by":"publisher","DOI":"10.3390\/e22090999","article-title":"The conditional entropy bottleneck.","volume":"22","author":"Fischer","year":"2020","journal-title":"Entropy"},{"key":"B15","article-title":"Sparse coding in the primate cortex","author":"Foldiak","year":"2003","journal-title":"The handbook of brain theory and neural networks"},{"key":"B16","doi-asserted-by":"publisher","first-page":"500","DOI":"10.1016\/j.tics.2016.05.001","article-title":"The functional anatomy of time: What and when in the brain.","volume":"20","author":"Friston","year":"2016","journal-title":"Trends Cogn. Sci."},{"key":"B17","doi-asserted-by":"publisher","first-page":"278","DOI":"10.1016\/j.tics.2006.04.008","article-title":"Is neocortex essentially multisensory?","volume":"10","author":"Ghazanfar","year":"2006","journal-title":"Trends Cogn. Sci."},{"key":"B18","doi-asserted-by":"publisher","first-page":"19","DOI":"10.1109\/JSAIT.2020.2991561","article-title":"The information bottleneck problem and its applications in machine learning.","volume":"1","author":"Goldfeld","year":"2020","journal-title":"IEEE J. Sel. Areas Inf. Theory"},{"key":"B19","doi-asserted-by":"publisher","first-page":"121014","DOI":"10.1109\/ACCESS.2020.3006569","article-title":"Sentiment analysis via deep multichannel neural networks with variational information bottleneck.","volume":"8","author":"Gu","year":"2020","journal-title":"IEEE Access"},{"key":"B20","author":"Hawkins","year":"2011","journal-title":"Cortical learning algorithm and hierarchical temporal memory"},{"key":"B21","author":"Hawkins","year":"2016","journal-title":"Biological and machine intelligence (bami). Initial online release 0.4"},{"key":"B22","author":"Hoffmann","year":"1971","journal-title":"Linear algebra"},{"key":"B23","doi-asserted-by":"publisher","first-page":"104","DOI":"10.1162\/neco_a_01453","article-title":"Bridging the functional and wiring properties of V1 neurons through sparse coding.","volume":"34","author":"Hu","year":"2022","journal-title":"Neural Comput."},{"key":"B24","doi-asserted-by":"publisher","first-page":"35136","DOI":"10.1109\/ACCESS.2020.2974320","article-title":"Connectivity-based localization in ultra-dense networks: CRLB, theoretical variance, and MLE.","volume":"8","author":"Huang","year":"2020","journal-title":"IEEE Access"},{"key":"B25","doi-asserted-by":"publisher","first-page":"10655","DOI":"10.1073\/pnas.1201892109","article-title":"Evolution of columns, modules, and domains in the neocortex of primates.","volume":"109","author":"Kaas","year":"2012","journal-title":"Proc. Natl. Acad. Sci."},{"key":"B26","doi-asserted-by":"publisher","first-page":"781","DOI":"10.1007\/s11760-019-01605-2","article-title":"Decreasing Cramer\u2013Rao lower bound by preprocessing steps.","volume":"14","author":"Khorasani","year":"2020","journal-title":"Signal Image Video Process."},{"key":"B27","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1007\/s10772-021-09941-w","article-title":"Sparse representation and reproduction of speech signals in complex Fourier basis.","volume":"25","author":"Kwek","year":"2022","journal-title":"Int. J. Speech Technol."},{"key":"B28","article-title":"Efficient sparse coding algorithms.","volume":"19","author":"Lee","year":"2006","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"B29","doi-asserted-by":"publisher","first-page":"1385","DOI":"10.1007\/s11063-021-10445-6","article-title":"Information bottleneck theory on convolutional neural networks.","volume":"53","author":"Li","year":"2021","journal-title":"Neural Process. Lett."},{"key":"B30","doi-asserted-by":"publisher","DOI":"10.1016\/j.compbiomed.2021.104239","article-title":"Medical image fusion using segment graph filter and sparse representation.","volume":"131","author":"Li","year":"2021","journal-title":"Comput. Biol. Med."},{"key":"B31","doi-asserted-by":"publisher","first-page":"352","DOI":"10.1007\/978-3-642-38457-8_38","article-title":"Sparse representation for machine learning","author":"Li","year":"2013","journal-title":"Proceedings of the Canadian conference on artificial intelligence"},{"key":"B32","doi-asserted-by":"publisher","first-page":"296","DOI":"10.1016\/j.tics.2005.04.010","article-title":"Capacity limits of information processing in the brain.","volume":"9","author":"Marois","year":"2005","journal-title":"Trends Cogn. Sci."},{"key":"B33","doi-asserted-by":"publisher","first-page":"641","DOI":"10.1016\/j.tics.2016.06.008","article-title":"Ensemble coding and two conceptions of perceptual sparsity.","volume":"20","author":"McClelland","year":"2016","journal-title":"Trends Cogn. Sci."},{"key":"B34","doi-asserted-by":"publisher","first-page":"62","DOI":"10.1016\/S1364-6613(00)01601-6","article-title":"Cognitive architecture of a mini-brain: The honeybee.","volume":"5","author":"Menzel","year":"2001","journal-title":"Trends Cogn. Sci."},{"key":"B35","doi-asserted-by":"publisher","DOI":"10.3389\/frobt.2016.00081","article-title":"A mathematical formalization of hierarchical temporal memory\u2019s spatial pooler.","volume":"3","author":"Mnatzaganian","year":"2017","journal-title":"Front. Robot. AI"},{"key":"B36","doi-asserted-by":"publisher","DOI":"10.15837\/ijccc.2022.1.4650","article-title":"Information bottleneck in deep learning-a semiotic approach.","volume":"17","author":"Musat","year":"2022","journal-title":"Int. J. Comput. Commun. Control"},{"key":"B37","doi-asserted-by":"publisher","first-page":"427","DOI":"10.1007\/978-3-030-95157-3_23","article-title":"Sparse representation for sampled-data filters","author":"Nagahara","year":"2022","journal-title":"Realization and model reduction of dynamical systems"},{"key":"B38","doi-asserted-by":"publisher","first-page":"607","DOI":"10.1038\/381607a0","article-title":"Wavelet-like receptive fields emerge from a network that learns sparse codes for natural images.","volume":"381","author":"Olshausen","year":"1996","journal-title":"Nature"},{"key":"B39","doi-asserted-by":"publisher","first-page":"481","DOI":"10.1016\/j.conb.2004.07.007","article-title":"Sparse coding of sensory inputs.","volume":"14","author":"Olshausen","year":"2004","journal-title":"Curr. Opin. Neurobiol."},{"key":"B40","doi-asserted-by":"publisher","first-page":"505","DOI":"10.1016\/S1364-6613(02)02005-3","article-title":"Hippocampal and neocortical contributions to memory: Advances in the complementary learning systems framework.","volume":"6","author":"O\u2019Reilly","year":"2002","journal-title":"Trends Cogn. Sci."},{"key":"B41","doi-asserted-by":"publisher","DOI":"10.1167\/jov.20.12.10","article-title":"Selectivity and robustness of sparse coding networks.","volume":"20","author":"Paiton","year":"2020","journal-title":"J. Vis."},{"key":"B42","doi-asserted-by":"publisher","first-page":"1790","DOI":"10.1109\/TNNLS.2018.2874432","article-title":"Maximum likelihood estimation-based joint sparse representation for the classification of hyperspectral remote sensing images.","volume":"30","author":"Peng","year":"2018","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"B43","article-title":"Understanding learning dynamics of binary neural networks via information bottleneck.","author":"Raj","year":"2020","journal-title":"arXiv [Preprint]"},{"key":"B44","doi-asserted-by":"publisher","DOI":"10.1002\/9780470316436","author":"Rao","year":"1973","journal-title":"Linear statistical inference and its applications"},{"key":"B45","doi-asserted-by":"publisher","first-page":"1072","DOI":"10.1109\/TSP.2012.2226449","article-title":"Learning sparsifying transforms.","volume":"61","author":"Ravishankar","year":"2012","journal-title":"IEEE Trans. Signal Process."},{"key":"B46","doi-asserted-by":"publisher","first-page":"89","DOI":"10.1007\/s10994-011-5247-6","article-title":"Applying the information bottleneck to statistical relational learning.","volume":"86","author":"Riguzzi","year":"2012","journal-title":"Mach. Learn."},{"key":"B47","article-title":"Analyzing neural codes using the information bottleneck method","author":"Schneidman","year":"2001","journal-title":"Advances in neural information processing systems"},{"key":"B48","article-title":"Opening the black box of deep neural networks via information.","author":"Shwartz-Ziv","year":"2017","journal-title":"arXiv [Preprint]"},{"key":"B49","doi-asserted-by":"publisher","first-page":"796","DOI":"10.1109\/LSP.2014.2316008","article-title":"A lower bound for the fisher information measure.","volume":"21","author":"Stein","year":"2014","journal-title":"IEEE Signal Process. Lett."},{"key":"B50","article-title":"The information bottleneck method.","author":"Tishby","year":"2000","journal-title":"arXiv [Preprint]"},{"key":"B51","doi-asserted-by":"publisher","DOI":"10.1109\/ITW.2015.7133169","article-title":"Deep learning and the information bottleneck principle","author":"Tishby","year":"2015","journal-title":"Proceedings of the 2015 IEEE information theory workshop (ITW)"},{"key":"B52","article-title":"Towards human-agent communication via the information bottleneck principle.","author":"Tucker","year":"2022","journal-title":"arXiv [Preprint]"},{"key":"B53","doi-asserted-by":"publisher","first-page":"5543","DOI":"10.1109\/TSP.2012.2204258","article-title":"Computing constrained Cram\u00e9r-Rao bounds.","volume":"60","author":"Tune","year":"2012","journal-title":"IEEE Trans. Signal Process."},{"key":"B54","doi-asserted-by":"publisher","first-page":"235","DOI":"10.1016\/j.neucom.2022.08.014","article-title":"Information flow in deep restricted Boltzmann machines: An analysis of mutual information between inputs and outputs.","volume":"507","author":"Vera","year":"2022","journal-title":"Neurocomputing"},{"key":"B55","doi-asserted-by":"publisher","first-page":"712","DOI":"10.1109\/TIM.2019.2906334","article-title":"A novel weighted sparse representation classification strategy based on dictionary learning for rotating machinery.","volume":"69","author":"Wang","year":"2019","journal-title":"IEEE Trans. Instrum. Meas."},{"key":"B56","doi-asserted-by":"publisher","first-page":"533","DOI":"10.1146\/annurev-neuro-110920-035434","article-title":"Theory of the multiregional neocortex: Large-scale neural dynamics and distributed cognition.","volume":"45","author":"Wang","year":"2022","journal-title":"Annu. Rev. Neurosci."},{"key":"B57","doi-asserted-by":"publisher","first-page":"44247","DOI":"10.1007\/s11042-022-13203-5","article-title":"Improved image representation and sparse representation for face recognition.","volume":"81","author":"Wei","year":"2022","journal-title":"Multimed. Tools Appl."},{"key":"B58","doi-asserted-by":"publisher","DOI":"10.1016\/j.measurement.2021.110633","article-title":"A hybrid denoising model using deep learning and sparse representation with application in bearing weak fault diagnosis.","volume":"189","author":"Zhou","year":"2022","journal-title":"Measurement"},{"key":"B59","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuroimage.2021.118569","article-title":"Unsupervised MR harmonization by learning disentangled representations using information bottleneck theory.","volume":"243","author":"Zuo","year":"2021","journal-title":"Neuroimage"}],"container-title":["Frontiers in Computational Neuroscience"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fncom.2023.1140782\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,6,7]],"date-time":"2023-06-07T02:44:00Z","timestamp":1686105840000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fncom.2023.1140782\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,7]]},"references-count":59,"alternative-id":["10.3389\/fncom.2023.1140782"],"URL":"https:\/\/doi.org\/10.3389\/fncom.2023.1140782","relation":{},"ISSN":["1662-5188"],"issn-type":[{"value":"1662-5188","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,6,7]]},"article-number":"1140782"}}