{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,8]],"date-time":"2026-05-08T16:45:18Z","timestamp":1778258718066,"version":"3.51.4"},"reference-count":32,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,8,24]],"date-time":"2023-08-24T00:00:00Z","timestamp":1692835200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Neurorobot."],"abstract":"<jats:p>Artificial Intelligence (AI) is driving advancements across various fields by simulating and enhancing human intelligence. In Natural Language Processing (NLP), transformer models like the Kerformer, a linear transformer based on a kernel approach, have garnered success. However, traditional attention mechanisms in these models have quadratic calculation costs linked to input sequence lengths, hampering efficiency in tasks with extended orders. To tackle this, Kerformer introduces a nonlinear reweighting mechanism, transforming maximum attention into feature-based dot product attention. By exploiting the non-negativity and non-linear weighting traits of softmax computation, separate non-negativity operations for <jats:italic>Query<\/jats:italic>(<jats:italic>Q<\/jats:italic>) and <jats:italic>Key<\/jats:italic>(<jats:italic>K<\/jats:italic>) computations are performed. The inclusion of the SE Block further enhances model performance. Kerformer significantly reduces attention matrix time complexity from <jats:italic>O<\/jats:italic>(<jats:italic>N<\/jats:italic><jats:sup>2<\/jats:sup>) to <jats:italic>O<\/jats:italic>(<jats:italic>N<\/jats:italic>), with <jats:italic>N<\/jats:italic> representing sequence length. This transformation results in remarkable efficiency and scalability gains, especially for prolonged tasks. Experimental results demonstrate Kerformer's superiority in terms of time and memory consumption, yielding higher average accuracy (83.39%) in NLP and vision tasks. In tasks with long sequences, Kerformer achieves an average accuracy of 58.94% and exhibits superior efficiency and convergence speed in visual tasks. This model thus offers a promising solution to the limitations posed by conventional attention mechanisms in handling lengthy tasks.<\/jats:p>","DOI":"10.3389\/fnbot.2023.1214203","type":"journal-article","created":{"date-parts":[[2023,8,24]],"date-time":"2023-08-24T13:42:01Z","timestamp":1692884521000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":7,"title":["A novel approach to attention mechanism using kernel functions: Kerformer"],"prefix":"10.3389","volume":"17","author":[{"given":"Yao","family":"Gan","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yanyun","family":"Fu","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Deyong","family":"Wang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yongming","family":"Li","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1965","published-online":{"date-parts":[[2023,8,24]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","first-page":"12449","DOI":"10.48550\/arXiv.2006.11477","article-title":"wav2vec 2.0: a framework for self-supervised learning of speech representations","volume":"33","author":"Baevski","year":"2020","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"B2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2004.05150","article-title":"Longformer: the long-document transformer","author":"Beltagy","year":"2020","journal-title":"arXiv"},{"key":"B3","article-title":"Efficient 8-bit quantization of transformer neural machine language translation model","author":"Bhandare","year":"2019","journal-title":"arXiv preprint arXiv:1906.00532"},{"key":"B4","doi-asserted-by":"publisher","first-page":"213","DOI":"10.1007\/978-3-030-58452-8_13","author":"Carion","year":"2020"},{"key":"B5","doi-asserted-by":"publisher","first-page":"2122","DOI":"10.48550\/arXiv.2111.00035","article-title":"Skyformer: remodel self-attention with Gaussian kernel and Nystr\u00f6m method","volume":"34","author":"Chen","year":"2021","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"B6","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1904.10509","article-title":"Generating long sequences with sparse transformers","author":"Child","year":"2019","journal-title":"arXiv"},{"key":"B7","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2009.14794","article-title":"Rethinking attention with performers","author":"Choromanski","year":"2020","journal-title":"arXiv"},{"key":"B8","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1810.04805","article-title":"Bert: pre-training of deep bidirectional transformers for language understanding","author":"Devlin","year":"2018","journal-title":"arXiv"},{"key":"B9","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2010.11929","article-title":"An image is worth 16x16 words: transformers for image recognition at scale","author":"Dosovitskiy","year":"2020","journal-title":"arXiv"},{"key":"B10","doi-asserted-by":"publisher","first-page":"122302","DOI":"10.1109\/ACCESS.2022.3223705","article-title":"Attention-based multi-learning approach for speech emotion recognition with dilated convolution","volume":"10","author":"Kakuba","year":"","journal-title":"IEEE Access"},{"key":"B11","doi-asserted-by":"publisher","first-page":"125538","DOI":"10.1109\/ACCESS.2022.3225684","article-title":"Deep learning-based speech emotion recognition using multi-level fusion of concurrent features","volume":"10","author":"Kakuba","year":"","journal-title":"IEEE Access"},{"key":"B12","first-page":"5156","author":"Katharopoulos","year":"2020"},{"key":"B13","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2001.04451","article-title":"Reformer: the efficient transformer","author":"Kitaev","year":"2020","journal-title":"arXiv"},{"key":"B14","doi-asserted-by":"publisher","DOI":"10.32470\/CCN.2018.1116-0","article-title":"Learning long-range spatial dependencies with horizontal gated recurrent units","author":"Linsley","year":"2018","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"B15","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1907.11692","article-title":"RoBERTa: a robustly optimized BERT pretraining approach","author":"Liu","year":"2019","journal-title":"arXiv"},{"key":"B16","first-page":"142","author":"Maas","year":"2011","journal-title":"Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies"},{"key":"B17","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1804.06028","article-title":"ListOps: a diagnostic dataset for latent tree learning","author":"Nangia","year":"2018","journal-title":"arXiv"},{"key":"B18","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-1018","author":"Ni","year":"2019","journal-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)"},{"key":"B19","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2103.02143","article-title":"Random feature attention","author":"Peng","year":"2021","journal-title":"arXiv"},{"key":"B20","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2202.08791","article-title":"cosFormer: rethinking softmax in attention","author":"Qin","year":"2022","journal-title":"arXiv"},{"key":"B21","doi-asserted-by":"publisher","first-page":"919","DOI":"10.1007\/s10579-012-9211-2","article-title":"The ACL anthology network corpus","volume":"47","author":"Radev","year":"2013","journal-title":"Lang. Resour. Eval"},{"key":"B22","doi-asserted-by":"publisher","DOI":"10.1109\/WACV48630.2021.00176","author":"Rymarczyk","year":"2021","journal-title":"Proceedings of the IEEE\/CVF Winter Conference on Applications of Computer Vision"},{"key":"B23","doi-asserted-by":"publisher","DOI":"10.1007\/s10915-023-02125-5","article-title":"Dct-former: efficient self-attention with discrete cosine transform","author":"Scribano","year":"2023","journal-title":"J. Sci. Comput"},{"key":"B24","first-page":"3531","author":"Shen","year":"2021","journal-title":"Proceedings of the IEEE\/CVF Winter Conference on Applications of Computer Vision"},{"key":"B25","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2011.04006","article-title":"Long range arena: a benchmark for efficient transformers","author":"Tay","year":"2020","journal-title":"arXiv"},{"key":"B26","article-title":"Attention is all you need","author":"Vaswani","year":"2017","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"B27","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1804.07461","article-title":"GLUE: a multi-task benchmark and analysis platform for natural language understanding","author":"Wang","year":"2018","journal-title":"arXiv"},{"key":"B28","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2006.04768","article-title":"Linformer: self-attention with linear complexity","author":"Wang","year":"2020","journal-title":"arXiv"},{"key":"B29","doi-asserted-by":"publisher","author":"Wu","year":"2021","DOI":"10.1145\/3442381.3449946"},{"key":"B30","doi-asserted-by":"publisher","first-page":"14138","DOI":"10.1609\/aaai.v35i16.17664","article-title":"Nystr\u00f6mformer: a nystr\u00f6m-based algorithm for approximating self-attention","volume":"35","author":"Xiong","year":"2021","journal-title":"Proc. AAAI Conf. Artif. Intell"},{"key":"B31","doi-asserted-by":"publisher","first-page":"17283","DOI":"10.48550\/arXiv.2007.14062","article-title":"Big bird: transformers for longer sequences","volume":"33","author":"Zaheer","year":"2020","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"B32","first-page":"12321","author":"Zeng","year":"2021"}],"container-title":["Frontiers in Neurorobotics"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2023.1214203\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,8,24]],"date-time":"2023-08-24T13:42:22Z","timestamp":1692884542000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2023.1214203\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,8,24]]},"references-count":32,"alternative-id":["10.3389\/fnbot.2023.1214203"],"URL":"https:\/\/doi.org\/10.3389\/fnbot.2023.1214203","relation":{},"ISSN":["1662-5218"],"issn-type":[{"value":"1662-5218","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,8,24]]},"article-number":"1214203"}}