{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,18]],"date-time":"2026-09-18T09:02:26Z","timestamp":1789722146875,"version":"4.0.1"},"reference-count":53,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2025,1,29]],"date-time":"2025-01-29T00:00:00Z","timestamp":1738108800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Neurosci."],"abstract":"<jats:p>Energy-efficient spikformer has been proposed by integrating the biologically plausible spiking neural network (SNN) and artificial transformer, whereby the spiking self-attention (SSA) is used to achieve both higher accuracy and lower computational cost. However, it seems that self-attention is not always necessary, especially in sparse spike-form calculation manners. In this article, we innovatively replace vanilla SSA (using dynamic bases calculating from Query and Key) with spike-form Fourier transform, wavelet transform, and their combinations (using fixed triangular or wavelets bases), based on a key hypothesis that both of them use a set of basis functions for information transformation. Hence, the Fourier-or-Wavelet-based spikformer (FWformer) is proposed and verified in visual classification tasks, including both static image and event-based video datasets. The FWformer can achieve comparable or even higher accuracies (0.4%\u20131.5%), higher running speed (9%\u201351% for training and 19%\u201370% for inference), reduced theoretical energy consumption (20%\u201325%), and reduced graphic processing unit (GPU) memory usage (4%\u201326%), compared to the standard spikformer. Our result indicates the continuous refinement of new transformers that are inspired either by biological discovery (spike-form), or information theory (Fourier or Wavelet transform), is promising.<\/jats:p>","DOI":"10.3389\/fnins.2024.1516868","type":"journal-article","created":{"date-parts":[[2025,1,29]],"date-time":"2025-01-29T06:59:55Z","timestamp":1738133995000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["Fourier or Wavelet bases as counterpart self-attention in spikformer for efficient visual classification"],"prefix":"10.3389","volume":"18","author":[{"given":"Qingyu","family":"Wang","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Duzhen","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xinyuan","family":"Cai","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tielin","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Bo","family":"Xu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2025,1,29]]},"reference":[{"key":"B1","first-page":"213","article-title":"\u201cEnd-to-end object detection with transformers,\u201d","author":"Carion","year":"2020"},{"key":"B2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2301.11929","article-title":"Training full spike neural networks via auxiliary accumulation pathway","author":"Chen","year":"2023","journal-title":"arXiv"},{"key":"B3","doi-asserted-by":"publisher","first-page":"12299","DOI":"10.1109\/CVPR46437.2021.01212","article-title":"\u201cPre-trained image processing transformer,\u201d","author":"Chen","year":"2021"},{"key":"B4","first-page":"9355","article-title":"Twins: revisiting the design of spatial attention in vision transformers","volume":"34","author":"Chu","year":"2021","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"B5","doi-asserted-by":"publisher","first-page":"297","DOI":"10.1090\/S0025-5718-1965-0178586-1","article-title":"An algorithm for the machine calculation of complex Fourier series","volume":"19","author":"Cooley","year":"1965","journal-title":"Math. Comput"},{"key":"B6","doi-asserted-by":"publisher","first-page":"82","DOI":"10.1109\/MM.2018.112130359","article-title":"Loihi: a neuromorphic manycore processor with on-chip learning","volume":"38","author":"Davies","year":"2018","journal-title":"IEEE Micro"},{"key":"B7","volume-title":"Theoretical Neuroscience: Computational and Mathematical Modeling of Neural Systems","author":"Dayan","year":"2005"},{"key":"B8","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2202.11946","article-title":"Temporal efficient training of spiking neural network via gradient re-weighting","author":"Deng","year":"2022","journal-title":"arXiv"},{"key":"B9","article-title":"\u201cAn image is worth 16x16 words: transformers for image recognition at scale,\u201d","author":"Dosovitskiy","year":"2020"},{"key":"B10","doi-asserted-by":"publisher","first-page":"eadi1480","DOI":"10.1126\/sciadv.adi1480","article-title":"Spikingjelly: an open-source machine learning infrastructure platform for spike-based intelligence","volume":"9","author":"Fang","year":"2023","journal-title":"Sci. Adv"},{"key":"B11","first-page":"21056","article-title":"Deep residual learning in spiking neural networks","volume":"34","author":"Fang","year":"","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"B12","first-page":"2661","article-title":"\u201cIncorporating learnable membrane time constant to enhance learning of spiking neural networks,\u201d","author":"Fang","year":""},{"key":"B13","volume-title":"Digital Image Processing","author":"Gonzales","year":"1987"},{"key":"B14","first-page":"13558","article-title":"\u201cRMP-SNN: residual membrane potential neuron for enabling deeper high-accuracy and low-latency spiking neural network,\u201d","author":"Han","year":"2020"},{"key":"B15","first-page":"10","article-title":"\u201c1.1 computing's energy problem (and what we can do about it),\u201d","author":"Horowitz","year":"2014"},{"key":"B16","article-title":"\u201cSpiking deep residual networks,\u201d","author":"Hu","year":"2021"},{"key":"B17","doi-asserted-by":"publisher","first-page":"933","DOI":"10.1093\/cercor\/bhh053","article-title":"Spike-timing dynamics of neuronal groups","volume":"14","author":"Izhikevich","year":"2004","journal-title":"Cereb Cortex"},{"key":"B18","doi-asserted-by":"publisher","first-page":"424","DOI":"10.3389\/fnins.2020.00424","article-title":"Synaptic plasticity dynamics for deep continuous local learning (DECOLLE)","volume":"14","author":"Kaiser","year":"2020","journal-title":"Front. Neurosci"},{"key":"B19","first-page":"5156","article-title":"\u201cTransformers are RNNs: fast autoregressive transformers with linear attention,\u201d","author":"Katharopoulos","year":"2020"},{"key":"B20","doi-asserted-by":"publisher","first-page":"686","DOI":"10.1016\/j.neunet.2021.09.022","article-title":"Optimizing deeper spiking neural networks for dynamic vision sensing","volume":"144","author":"Kim","year":"2021","journal-title":"Neural Netw"},{"key":"B21","doi-asserted-by":"publisher","first-page":"439","DOI":"10.3389\/fnins.2020.00439","article-title":"Efficient processing of spatio-temporal data streams with spiking neural networks","volume":"14","author":"Kugele","year":"2020","journal-title":"Front. Neurosci"},{"key":"B22","doi-asserted-by":"publisher","first-page":"119","DOI":"10.3389\/fnins.2020.00119","article-title":"Enabling spike-based backpropagation for training deep neural network architectures","volume":"14","author":"Lee","year":"2020","journal-title":"Front. Neurosci"},{"key":"B23","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2105.03824","article-title":"FNET: mixing tokens with Fourier transforms","author":"Lee-Thorp","year":"2021","journal-title":"arXiv"},{"key":"B24","first-page":"23426","article-title":"Differentiable spike: rethinking gradient-descent for training spiking neural networks","volume":"34","author":"Li","year":"2021","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"B25","first-page":"631","article-title":"\u201cNeuromorphic data augmentation for training spiking neural networks,\u201d","author":"Li","year":"2022"},{"key":"B26","first-page":"1","article-title":"\u201cLong short-term memory spiking networks and their applications,\u201d","author":"Lotfi Rezaabad","year":"2020"},{"key":"B27","doi-asserted-by":"publisher","first-page":"1659","DOI":"10.1016\/S0893-6080(97)00011-7","article-title":"Networks of spiking neurons: the third generation of neural network models","volume":"10","author":"Maass","year":"1997","journal-title":"Neural Netw"},{"key":"B28","doi-asserted-by":"publisher","first-page":"12444","DOI":"10.1109\/CVPR52688.2022.01212","article-title":"\u201cTraining high-performance low-latency spiking neural networks by differentiation on spike representation,\u201d","author":"Meng","year":"2022","journal-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition"},{"key":"B29","first-page":"1","article-title":"\u201cSpiking transformer networks: a rate coded approach for processing sequential data,\u201d","author":"Mueller","year":"2021"},{"key":"B30","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2008.03658","article-title":"Diet-SNN: direct input encoding with leakage and threshold optimization in deep spiking neural networks","author":"Rathi","year":"2020","journal-title":"arXiv"},{"key":"B31","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2005.01807","article-title":"Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation","author":"Rathi","year":"2020","journal-title":"arXiv"},{"key":"B32","doi-asserted-by":"publisher","first-page":"682","DOI":"10.3389\/fnins.2017.00682","article-title":"Conversion of continuous-valued deep networks to efficient event-driven networks for image classification","volume":"11","author":"Rueckauer","year":"2017","journal-title":"Front. Neurosci"},{"key":"B33","article-title":"Slayer: Spike layer error reassignment in time","author":"Shrestha","year":"2018","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"B34","first-page":"2016","article-title":"\u201cDeep reinforcement learning with population-coded spiking neural network for continuous control,\u201d","author":"Tang","year":"2021"},{"key":"B35","doi-asserted-by":"publisher","first-page":"109050","DOI":"10.1016\/j.patcog.2022.109050","article-title":"Multi-stage image denoising with the wavelet transform","volume":"134","author":"Tian","year":"2023","journal-title":"Pattern Recognit"},{"key":"B36","article-title":"Attention is all you need","author":"Vaswani","year":"2017","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"B37","doi-asserted-by":"publisher","first-page":"102","DOI":"10.1609\/aaai.v37i1.25081","article-title":"Complex dynamic neurons improved spiking transformer network for efficient automatic speech recognition","volume":"37","author":"Wang","year":"2023","journal-title":"Proc. AAAI Conf. Artif. Intell"},{"key":"B38","first-page":"568","article-title":"\u201cPyramid vision transformer: a versatile backbone for dense prediction without convolutions,\u201d","author":"Wang","year":"2021"},{"key":"B39","doi-asserted-by":"publisher","first-page":"331","DOI":"10.3389\/fnins.2018.00331","article-title":"Spatio-temporal backpropagation for training high-performance spiking neural networks","volume":"12","author":"Wu","year":"2018","journal-title":"Front. Neurosci"},{"key":"B40","doi-asserted-by":"publisher","first-page":"1311","DOI":"10.1609\/aaai.v33i01.33011311","article-title":"Direct training for spiking neural networks: Faster, larger, better","volume":"33","author":"Wu","year":"2019","journal-title":"Proc. AAAI Conf. Artif. Intell"},{"key":"B41","doi-asserted-by":"publisher","first-page":"6249","DOI":"10.1109\/TNNLS.2021.3073016","article-title":"Liaf-net: leaky integrate and analog fire network for lightweight and efficient spatiotemporal information processing","volume":"33","author":"Wu","year":"2021","journal-title":"IEEE Trans. Neural Netw. Learn. Syst"},{"key":"B42","first-page":"30392","article-title":"Early convolutions help transformers see better","volume":"34","author":"Xiao","year":"2021","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"B43","doi-asserted-by":"publisher","first-page":"3207","DOI":"10.24963\/ijcai.2021\/441","article-title":"\u201cExploiting spiking dynamics with spatial-temporal feature normalization in graph learning,\u201d","author":"Xu","year":"2021"},{"key":"B44","first-page":"10221","article-title":"\u201cTemporal-wise attention spiking neural networks for event streams classification,\u201d","author":"Yao","year":"2021"},{"key":"B45","article-title":"Spike-driven transformer","author":"Yao","year":"2024","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"B46","doi-asserted-by":"publisher","first-page":"1837","DOI":"10.3390\/app13031837","article-title":"Research on image denoising in edge detection based on wavelet transform","volume":"13","author":"You","year":"2023","journal-title":"Appl. Sci"},{"key":"B47","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2005.00742","article-title":"Hard-coded Gaussian attention for neural machine translation","author":"You","year":"2020","journal-title":"arXiv"},{"key":"B48","doi-asserted-by":"publisher","first-page":"558","DOI":"10.1109\/ICCV48922.2021.00060","article-title":"\u201cTokens-to-token VIT: Training vision transformers from scratch on imagenet,\u201d","author":"Yuan","year":"2021","journal-title":"Proceedings of the IEEE\/CVF international conference on computer vision"},{"key":"B49","doi-asserted-by":"publisher","first-page":"40838","DOI":"10.1109\/ACCESS.2020.2976798","article-title":"Image compression and encryption scheme based on compressive sensing and Fourier transform","volume":"8","author":"Zhang","year":"2020","journal-title":"IEEE Access"},{"key":"B50","doi-asserted-by":"publisher","first-page":"eabh0146","DOI":"10.1126\/sciadv.abh0146","article-title":"Self-backpropagation of synaptic modifications elevates the efficiency of spiking and artificial neural networks","volume":"7","author":"Zhang","year":"2021","journal-title":"Sci. Adv"},{"key":"B51","first-page":"12022","article-title":"Temporal spike sequence learning via backpropagation for deep spiking neural networks","volume":"33","author":"Zhang","year":"2020","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"B52","doi-asserted-by":"publisher","first-page":"11062","DOI":"10.1609\/aaai.v35i12.17320","article-title":"Going deeper with directly-trained larger spiking neural networks","volume":"35","author":"Zheng","year":"2021","journal-title":"Proc. AAAI Conf. Artif. Intell"},{"key":"B53","article-title":"\u201cSpikformer: when Spiking Neural Network Meets Transformer,\u201d","author":"Zhou","year":"2023"}],"container-title":["Frontiers in Neuroscience"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnins.2024.1516868\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,1,29]],"date-time":"2025-01-29T07:00:02Z","timestamp":1738134002000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnins.2024.1516868\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,1,29]]},"references-count":53,"alternative-id":["10.3389\/fnins.2024.1516868"],"URL":"https:\/\/doi.org\/10.3389\/fnins.2024.1516868","relation":{},"ISSN":["1662-453X"],"issn-type":[{"value":"1662-453X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,1,29]]},"article-number":"1516868"}}