{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,30]],"date-time":"2025-10-30T06:28:11Z","timestamp":1761805691627,"version":"build-2065373602"},"reference-count":36,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2025,10,30]],"date-time":"2025-10-30T00:00:00Z","timestamp":1761782400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Neurorobot."],"abstract":"<jats:p>Under the influence of Masked Language Modeling (MLM), Masked Image Modeling (MIM) employs an attention mechanism to perform masked training on images. However, processing a single image requires numerous iterations and substantial computational resources to reconstruct the masked regions, resulting in high computational complexity and significant time costs. To address this issue, we propose an Effective and Efficient self-supervised Masked model based on Mixed feature training (EESMM). First, we stack two images for encoding and input the fused features into the network, which not only reduces computational complexity but also enables the learning of more features. Second, during decoding, we obtain the decoding features corresponding to the original images based on the decoding features of the two input original images and the mixed images, and then construct a corresponding loss function to enhance feature representation. EESMM significantly reduces pre-training time without sacrificing accuracy, achieving 83% accuracy on ImageNet in just 363 h using four V100 GPUs\u2013only one-tenth of the training time required by SimMIM. This validates that the method can substantially accelerate the pre-training process without noticeable performance degradation.<\/jats:p>","DOI":"10.3389\/fnbot.2025.1705970","type":"journal-article","created":{"date-parts":[[2025,10,30]],"date-time":"2025-10-30T06:23:06Z","timestamp":1761805386000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Effective and efficient self-supervised masked model based on mixed feature training"],"prefix":"10.3389","volume":"19","author":[{"given":"Qingjiu","family":"Kang","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Feng","family":"Liu","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chunliu","family":"Cai","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1965","published-online":{"date-parts":[[2025,10,30]]},"reference":[{"key":"B1","first-page":"1298","article-title":"\u201cData2vec: a general framework for self-supervised learning in speech, vision and language,\u201d","volume-title":"International Conference on Machine Learning","author":"Baevski","year":"2022"},{"key":"B2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2106.08254","article-title":"Beit: bert pre-training of image transformers","author":"Bao","year":"2021","journal-title":"arXiv [preprint"},{"key":"B3","first-page":"9650","article-title":"\u201cEmerging properties in self-supervised vision transformers,\u201d","volume-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision","author":"Caron","year":"2021"},{"key":"B4","doi-asserted-by":"crossref","first-page":"8046","DOI":"10.1109\/WACV61041.2025.00781","article-title":"\u201cLocal masked reconstruction for efficient self-supervised learning on high-resolution images,\u201d","volume-title":"2025 IEEE\/CVF Winter Conference on Applications of Computer Vision (WACV)","author":"Chen","year":"2025"},{"key":"B5","first-page":"1597","article-title":"\u201cA simple framework for contrastive learning of visual representations,\u201d","volume-title":"International Conference on Machine Learning","author":"Chen","year":"2020"},{"key":"B6","doi-asserted-by":"publisher","first-page":"208","DOI":"10.1007\/s11263-023-01852-4","article-title":"Context autoencoder for self-supervised representation learning","volume":"132","author":"Chen","year":"2024","journal-title":"Int. J. Comput. Vis"},{"key":"B7","first-page":"9640","article-title":"\u201cAn empirical study of training self-supervised vision transformers,\u201d","volume-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision","author":"Chen","year":"2021"},{"key":"B8","first-page":"702","article-title":"\u201cRandaugment: practical automated data augmentation with a reduced search space,\u201d","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops","author":"Cubuk","year":"2020"},{"key":"B9","first-page":"4171","article-title":"\u201cBert: pre-training of deep bidirectional transformers for language understanding,\u201d","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Devlin","year":"2019"},{"key":"B10","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv:2010.11929","article-title":"An image is worth 16x16 words: transformers for image recognition at scale","author":"Dosovitskiy","year":"2020","journal-title":"arXiv [preprint"},{"key":"B11","doi-asserted-by":"publisher","first-page":"21271","DOI":"10.48550\/arXiv.2006.07733","article-title":"Bootstrap your own latent-a new approach to self-supervised learning","volume":"33","author":"Grill","year":"2020","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"B12","first-page":"16000","article-title":"\u201cMasked autoencoders are scalable vision learners,\u201d","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"He","year":"2022"},{"key":"B13","first-page":"646","article-title":"\u201cDeep networks with stochastic depth,\u201d","volume-title":"European Conference on Computer Vision","author":"Huang","year":"2016"},{"key":"B14","first-page":"19997","article-title":"Green hierarchical vision transformer for masked image modeling","volume":"35","author":"Huang","year":"2022","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"B15","first-page":"300","article-title":"\u201cWhat to hide from your students: attention-guided masked image modeling,\u201d","volume-title":"European Conference on Computer Vision","author":"Kakogeorgiou","year":"2022"},{"key":"B16","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv:2106.09785","article-title":"Efficient self-supervised vision transformers for representation learning","author":"Li","year":"2021","journal-title":"arXiv [preprint"},{"key":"B17","first-page":"2142","article-title":"\u201cMage: masked generative encoder to unify representation learning and image synthesis,\u201d","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Li","year":"2023"},{"key":"B18","doi-asserted-by":"publisher","first-page":"1799","DOI":"10.1609\/aaai.v37i2.25269","article-title":"Good helper is around you: attention-driven masked image modeling","volume":"37","author":"Liu","year":"2023","journal-title":"Proc. AAAI Conf. Artif. Intell"},{"key":"B19","first-page":"10012","article-title":"\u201cSwin transformer: Hierarchical vision transformer using shifted windows,\u201d","volume-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision","author":"Liu","year":"2021"},{"key":"B20","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1608.03983","article-title":"Sgdr: Stochastic gradient descent with warm restarts","author":"Loshchilov","year":"2016","journal-title":"arXiv [preprint"},{"key":"B21","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1711.05101","article-title":"Decoupled weight decay regularization","author":"Loshchilov","year":"2017","journal-title":"arXiv [preprint"},{"key":"B22","first-page":"8821","article-title":"\u201cZero-shot text-to-image generation,\u201d","volume-title":"International Conference on Machine Learning","author":"Ramesh","year":"2021"},{"key":"B23","doi-asserted-by":"crossref","first-page":"879","DOI":"10.1109\/WACV61041.2025.00095","article-title":"\u201cDeepmim: deep supervision for masked image modeling,\u201d","volume-title":"2025 IEEE\/CVF Winter Conference on Applications of Computer Vision (WACV)","author":"Ren","year":"2025"},{"key":"B24","first-page":"2818","article-title":"\u201cRethinking the inception architecture for computer vision,\u201d","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Szegedy","year":"2016"},{"key":"B25","first-page":"6105","article-title":"\u201cEfficientnet: rethinking model scaling for convolutional neural networks,\u201d","volume-title":"International Conference on Machine Learning","author":"Tan","year":"2019"},{"key":"B26","first-page":"10347","article-title":"\u201cTraining data-efficient image transformers & distillation through attention,\u201d","volume-title":"International Conference on Machine Learning","author":"Touvron","year":"2021"},{"key":"B27","doi-asserted-by":"crossref","DOI":"10.1109\/CVPR52729.2023.00211","article-title":"\u201cMasked image modeling with local multi-scale reconstruction,\u201d","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Wang","year":"2023"},{"key":"B28","first-page":"14668","article-title":"\u201cMasked feature prediction for self-supervised visual pre-training,\u201d","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Wei","year":""},{"key":"B29","first-page":"337","article-title":"\u201cMVP: multimodality-guided visual pre-training,\u201d","volume-title":"European Conference on Computer Vision","author":"Wei","year":""},{"key":"B30","first-page":"9653","article-title":"\u201cSimMIM: a simple framework for masked image modeling,\u201d","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Xie","year":"2022"},{"key":"B31","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2112.07658","article-title":"Adavit: Adaptive tokens for efficient vision transformer","author":"Yin","year":"2021","journal-title":"arXiv [preprint"},{"key":"B32","first-page":"579","article-title":"\u201cIncorporating convolution designs into visual transformers,\u201d","volume-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision","author":"Yuan","year":"2021"},{"key":"B33","first-page":"6023","article-title":"\u201cCutmix: regularization strategy to train strong classifiers with localizable features,\u201d","volume-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision","author":"Yun","year":"2019"},{"key":"B34","doi-asserted-by":"publisher","first-page":"13001","DOI":"10.1609\/aaai.v34i07.7000","article-title":"Random erasing data augmentation","volume":"34","author":"Zhong","year":"2020","journal-title":"Proc. AAAI Conf. Artif. Intell"},{"key":"B35","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2111.07832","article-title":"IBOT: image bert pre-training with online tokenizer","author":"Zhou","year":"2021","journal-title":"arXiv [preprint"},{"key":"B36","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2401.02020","article-title":"Spikformer v2: join the high accuracy club on imagenet with an SNN ticket","author":"Zhou","year":"2024","journal-title":"arXiv [preprint"}],"container-title":["Frontiers in Neurorobotics"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2025.1705970\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,30]],"date-time":"2025-10-30T06:23:13Z","timestamp":1761805393000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2025.1705970\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,30]]},"references-count":36,"alternative-id":["10.3389\/fnbot.2025.1705970"],"URL":"https:\/\/doi.org\/10.3389\/fnbot.2025.1705970","relation":{},"ISSN":["1662-5218"],"issn-type":[{"type":"electronic","value":"1662-5218"}],"subject":[],"published":{"date-parts":[[2025,10,30]]},"article-number":"1705970"}}