{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,20]],"date-time":"2026-08-20T16:12:13Z","timestamp":1787242333614,"version":"build-2736575974"},"reference-count":56,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2026,4,1]],"date-time":"2026-04-01T00:00:00Z","timestamp":1775001600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>We introduce Self-Competitive Distillation (SCD), a parameter-neutral training strategy aimed at influencing optimization dynamics without increasing model size or relying on external teachers. Two identical instances of the same architecture, initialized with different random seeds, are trained jointly and dynamically exchange asymmetric teacher\u2013student roles based on instantaneous performance, enabling knowledge transfer between diverging optimization trajectories. Under fixed parameter and training budgets, SCD is observed to improve the realized effective capacity of lightweight architectures, yielding a higher test accuracy at matched epochs. Across multiple lightweight vision models and datasets, SCD demonstrates gains in both in-domain performance and cross-domain generalization, as measured by xScore. These results suggest that, within the evaluated experimental conditions, SCD can help mobile models make more effective use of training dynamics, while the underlying architecture remains the primary determinant of effective capacity in resource-constrained settings.<\/jats:p>","DOI":"10.3390\/a19040262","type":"journal-article","created":{"date-parts":[[2026,4,1]],"date-time":"2026-04-01T08:08:34Z","timestamp":1775030914000},"page":"262","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Accelerating Realization of Effective Capacity in Lightweight Vision Models via Self-Competitive Distillation"],"prefix":"10.3390","volume":"19","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6180-9909","authenticated-orcid":false,"given":"Weidong","family":"Zhang","sequence":"first","affiliation":[{"name":"School of Computing and AI, Arizona State University, Tempe, AZ 85281, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Baoxin","family":"Li","sequence":"additional","affiliation":[{"name":"School of Computing and AI, Arizona State University, Tempe, AZ 85281, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3264-7904","authenticated-orcid":false,"given":"Huan","family":"Liu","sequence":"additional","affiliation":[{"name":"School of Computing and AI, Arizona State University, Tempe, AZ 85281, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Pak Lun Kevin","family":"Ding","sequence":"additional","affiliation":[{"name":"School of Computing and AI, Arizona State University, Tempe, AZ 85281, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ahmet Arda","family":"Dalyanci","sequence":"additional","affiliation":[{"name":"School of Computing and AI, Arizona State University, Tempe, AZ 85281, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2026,4,1]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 3\u20137). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Denver, CO, USA.","DOI":"10.1109\/CVPR.2016.90"},{"key":"ref_2","unstructured":"Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012, January 12\u201315). ImageNet Classification with Deep Convolutional Neural Networks. Proceedings of the 25th International Conference on Neural Information Processing Systems (NeurIPS), Granada, Spain."},{"key":"ref_3","unstructured":"Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2021, January 3\u20137). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. Proceedings of the International Conference on Learning Representations (ICLR), Virtual."},{"key":"ref_4","unstructured":"Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and J\u00e9gou, H. (2021, January 18\u201324). Training data-efficient image transformers & distillation through attention. Proceedings of the International Conference on Machine Learning, Virtual."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Yuan, L., Chen, Y., Wang, T., Yu, W., Shi, Y., Jiang, Z.H., Tay, F.E., Feng, J., and Yan, S. (2021, January 11\u201317). Tokens-to-token ViT: Training vision transformers from scratch on ImageNet. Proceedings of the ICCV, Virtual.","DOI":"10.1109\/ICCV48922.2021.00060"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. (2021, January 11\u201317). Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows. Proceedings of the ICCV, Virtual.","DOI":"10.1109\/ICCV48922.2021.00986"},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"200","DOI":"10.1145\/3505244","article-title":"Transformers in Vision: A Survey","volume":"54","author":"Khan","year":"2022","journal-title":"ACM Comput. Surv."},{"key":"ref_8","unstructured":"Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017). MobileNets: Efficient convolutional neural networks for mobile vision applications. arXiv."},{"key":"ref_9","unstructured":"Ma, N., Zhang, X., Zheng, H.T., and Sun, J. (2018, January 18\u201322). ShuffleNet v1: An extremely efficient convolutional neural network for mobile devices. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA."},{"key":"ref_10","unstructured":"Han, S., Mao, H., and Dally, W.J. (2016, January 2\u20134). Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding. Proceedings of the International Conference on Learning Representations (ICLR), San Juan, Puerto Rico."},{"key":"ref_11","unstructured":"Mehta, S., and Rastegari, M. (2021). MobileViT: Light-weight, general-purpose, and mobile-friendly vision transformer. arXiv."},{"key":"ref_12","unstructured":"Li, Y., Yuan, G., Wen, Y., Hu, J., Evangelidis, G., Tulyakov, S., and Ren, J. (December, January 28). EfficientFormer: Vision transformers at MobileNet speed. Proceedings of the Advances in Neural Information Processing Systems, New Orleans, LA, USA."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Shaker, A., Maaz, M., Rasheed, H., Khan, S., Yang, M.H., and Khan, F.S. (2023, January 2\u20136). SwiftFormer: Efficient Additive Attention for Transformer-based Real-time Mobile Vision Applications. Proceedings of the IEEE\/CVF International Conference on Computer Vision (ICCV), Paris, France.","DOI":"10.1109\/ICCV51070.2023.01598"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Liu, H.I., Galindo, M., Xie, H., Wong, L.K., Shuai, H.H., Li, Y.H., and Cheng, W.H. (2024). Lightweight Deep Learning for Resource-Constrained Environments: A Survey. arXiv.","DOI":"10.1145\/3657282"},{"key":"ref_15","unstructured":"Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. (2016). Understanding deep learning requires rethinking generalization. arXiv."},{"key":"ref_16","first-page":"1876","article-title":"Spectrally-normalized margin bounds for neural networks","volume":"30","author":"Bartlett","year":"2017","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_17","unstructured":"Raghu, M., Poole, B., Kleinberg, J., Ganguli, S., and Sohl-Dickstein, J. (2017, January 6\u201311). On the expressive power of deep neural networks. Proceedings of the International Conference on Machine Learning, Sydney, Australia."},{"key":"ref_18","unstructured":"Frankle, J., and Carbin, M. (2018). The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv."},{"key":"ref_19","unstructured":"Jiang, Y., Neyshabur, B., Mobahi, H., Krishnan, D., and Bengio, S. (2019). Fantastic generalization measures and where to find them. arXiv."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Wang, X., and Jia, W. (2025). Optimizing Edge AI: A Comprehensive Survey on Data, Model, and System Strategies. arXiv.","DOI":"10.32388\/IZOHCH"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"228","DOI":"10.1145\/3724420","article-title":"Empowering Edge Intelligence: A Comprehensive Survey on On-Device AI Models","volume":"57","author":"Wang","year":"2025","journal-title":"ACM Comput. Surv."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Fang, A., Kornblith, S., and Schmidt, L. (2023, January 10\u201316). Does progress on ImageNet transfer to real-world datasets?. Proceedings of the Advances in Neural Information Processing Systems, New Orleans, LA, USA.","DOI":"10.52202\/075280-1089"},{"key":"ref_23","unstructured":"Vishniakov, K., Shen, Z., and Liu, Z. (2023). ConvNet vs. Transformer, Supervised vs. CLIP: Beyond ImageNet Accuracy. arXiv."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Tuggener, L., Schmidhuber, J., and Stadelmann, T. (2022). Is it enough to optimize CNN architectures on ImageNet?. Front. Comput. Sci., 4.","DOI":"10.3389\/fcomp.2022.1041703"},{"key":"ref_25","unstructured":"Vapnik, V.N. (1998). Statistical Learning Theory, Wiley."},{"key":"ref_26","unstructured":"Liang, T., Poggio, T., Rakhlin, A., and Stokes, J. (2019, January 16\u201318). Fisher-rao metric, geometry, and complexity of neural networks. Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics, Naha, Japan."},{"key":"ref_27","unstructured":"Gonon, A., Brisebarre, N., Riccietti, E., and Gribonval, R. (2023). A path-norm toolkit for modern networks: Consequences, promises and challenges. arXiv."},{"key":"ref_28","unstructured":"Neyshabur, B., Bhojanapalli, S., and Srebro, N. (2017). A pac-bayesian approach to spectrally-normalized margin bounds for neural networks. arXiv."},{"key":"ref_29","unstructured":"Kutin, S., and Niyogi, P. (2013). Almost Everywhere Algorithmic Stability and Generalization Error. arXiv."},{"key":"ref_30","unstructured":"Carvalho, B.W., Garcez, A.S., Lamb, L.C., and Brazil, E.V. (2025). Grokking explained: A statistical phenomenon. arXiv."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"627","DOI":"10.1007\/s42979-025-04182-z","article-title":"Grokking in Neural Networks: A Review","volume":"6","author":"Agrawal","year":"2025","journal-title":"SN Comput. Sci."},{"key":"ref_32","first-page":"34651","article-title":"Towards understanding grokking: An effective theory of representation learning","volume":"35","author":"Liu","year":"2022","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_33","unstructured":"Power, A., Burda, Y., Edwards, H., Babuschkin, I., and Misra, V. (2022). Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"124003","DOI":"10.1088\/1742-5468\/ac3a74","article-title":"Deep double descent: Where bigger models and more data hurt","volume":"2021","author":"Nakkiran","year":"2021","journal-title":"J. Stat. Mech. Theory Exp."},{"key":"ref_35","unstructured":"Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M.M.A., Yang, Y., and Zhou, Y. (2017). Deep learning scaling is predictable, empirically. arXiv."},{"key":"ref_36","unstructured":"Morris, J.X., Sitawarin, C., Guo, C., Kokhlikyan, N., Suh, G.E., Rush, A.M., Chaudhuri, K., and Mahloujifar, S. (2025). How much do language models memorize?. arXiv."},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Zhang, W., Ding, P.L.K., Li, B., and Liu, H. (2025). xScore: A Simple Metric for Cross-Domain Robustness in Lightweight Vision Models. Algorithms, 19.","DOI":"10.3390\/a19010014"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Zhang, Y., Xiang, T., Hospedales, T.M., and Lu, H. (2018, January 18\u201322). Deep mutual learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00454"},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"1798","DOI":"10.1109\/TPAMI.2013.50","article-title":"Representation learning: A review and new perspectives","volume":"35","author":"Bengio","year":"2013","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_40","unstructured":"Neyshabur, B. (2017). Implicit regularization in deep learning. arXiv."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"1789","DOI":"10.1007\/s11263-021-01453-z","article-title":"Knowledge Distillation: A Survey","volume":"129","author":"Gou","year":"2021","journal-title":"Int. J. Comput. Vis."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1109\/MCAS.2023.3302182","article-title":"Tiny Machine Learning: Progress and Futures","volume":"23","author":"Lin","year":"2023","journal-title":"IEEE Circuits Syst. Mag."},{"key":"ref_43","doi-asserted-by":"crossref","first-page":"259","DOI":"10.1145\/3578938","article-title":"Efficient deep learning: A survey on making deep learning models smaller, faster, and better","volume":"55","author":"Menghani","year":"2023","journal-title":"ACM Comput. Surv."},{"key":"ref_44","doi-asserted-by":"crossref","unstructured":"Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.C. (2018, January 18\u201322). MobileNetV2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00474"},{"key":"ref_45","unstructured":"Howard, A., Sandler, M., Chu, G., Chen, L.C., Chen, B., Tan, M., Wang, W., Zhu, Y., Pang, R., and Vasudevan, V. (November, January 27). Searching for MobileNetV3. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Seoul, Republic of Korea."},{"key":"ref_46","first-page":"5630616","article-title":"Robust Aerial Person Detection with Lightweight Distillation Network for Edge Deployment","volume":"62","author":"Zhang","year":"2024","journal-title":"IEEE Trans. Geosci. Remote. Sens."},{"key":"ref_47","unstructured":"Zhang, W. (2025, October 31). xScore: Evaluation Framework with Training Wrapper and Dataset Loader. GitHub Repository. Available online: https:\/\/github.com\/javawormer\/beyond_imagenet."},{"key":"ref_48","unstructured":"Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the Knowledge in a Neural Network. arXiv."},{"key":"ref_49","unstructured":"Tan, M., and Le, Q.V. (2019, January 9\u201315). EfficientNet: Rethinking model scaling for convolutional neural networks. Proceedings of the International Conference on Machine Learning, Long Beach, CA, USA."},{"key":"ref_50","doi-asserted-by":"crossref","unstructured":"Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., and Xie, S. (2022). A ConvNet for the 2020s. arXiv.","DOI":"10.1109\/CVPR52688.2022.01167"},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Fei-Fei, L. (2009, January 20\u201325). ImageNet: A large-scale hierarchical image database. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Miami, FL, USA.","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Kornblith, S., Shlens, J., and Le, Q.V. (2019, January 16\u201320). Do Better ImageNet Models Transfer Better?. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.","DOI":"10.1109\/CVPR.2019.00277"},{"key":"ref_53","unstructured":"Krizhevsky, A., and Hinton, G. (2009). Learning Multiple Layers of Features from Tiny Images, University of Toronto. CIFAR-10 and CIFAR-100 Datasets."},{"key":"ref_54","unstructured":"Khosla, A., Jayadevaprakash, B., Yao, B., and Fei-Fei, L. (2011, January 20\u201325). Novel dataset for fine-grained image categorization: Stanford Dogs. Proceedings of the CVPR Workshop on Fine-Grained Visual Categorization, Colorado Springs, CO, USA."},{"key":"ref_55","doi-asserted-by":"crossref","first-page":"180161","DOI":"10.1038\/sdata.2018.161","article-title":"The HAM10000 dataset: A large collection of multi-source dermatoscopic images of common pigmented skin lesions","volume":"5","author":"Tschandl","year":"2018","journal-title":"Sci. Data"},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Quattoni, A., and Torralba, A. (2009). Recognizing indoor scenes. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, IEEE.","DOI":"10.1109\/CVPRW.2009.5206537"}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/19\/4\/262\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,4,1]],"date-time":"2026-04-01T08:13:14Z","timestamp":1775031194000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/19\/4\/262"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,4,1]]},"references-count":56,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2026,4]]}},"alternative-id":["a19040262"],"URL":"https:\/\/doi.org\/10.3390\/a19040262","relation":{},"ISSN":["1999-4893"],"issn-type":[{"value":"1999-4893","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,4,1]]}}}