{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,18]],"date-time":"2026-08-18T04:19:30Z","timestamp":1787026770905,"version":"build-2736575974"},"reference-count":39,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2025,10,14]],"date-time":"2025-10-14T00:00:00Z","timestamp":1760400000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Neurorobot."],"abstract":"<jats:p>Load imbalance is a major performance bottleneck in training mixture-of-experts (MoE) models, as unbalanced expert loads can lead to routing collapse. Most existing approaches address this issue by introducing auxiliary loss functions to balance the load; however, the hyperparameters within these loss functions often need to be tuned for different tasks. Furthermore, increasing the number of activated experts tends to exacerbate load imbalance, while fixing the activation count can reduce the model\u2019s confidence in handling difficult tasks. To address these challenges, this paper proposes a dynamically balanced routing strategy that employs a threshold-based dynamic routing algorithm. After each routing step, the method adjusts expert weights to influence the load distribution in the subsequent routing. Unlike loss-function-based balancing methods, our approach operates directly at the routing level, avoiding gradient perturbations that could degrade model quality, while dynamically routing to make more efficient use of computational resources. Experiments on Natural Language Understanding (NLU) benchmarks demonstrate that the proposed method achieves accuracy comparable to top-2 routing, while significantly reducing the load standard deviation (e.g., from 12.25 to 1.18 on MNLI). In addition, threshold-based dynamic expert activation reduces model parameters and provides a new perspective for mitigating load imbalance among experts.<\/jats:p>","DOI":"10.3389\/fnbot.2025.1590994","type":"journal-article","created":{"date-parts":[[2025,10,14]],"date-time":"2025-10-14T05:27:39Z","timestamp":1760419659000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["Adaptive-expert-weight-based load balance scheme for dynamic routing of MoE"],"prefix":"10.3389","volume":"19","author":[{"given":"Jialin","family":"Wen","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xiaojun","family":"Li","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Junping","family":"Yao","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xinyan","family":"Kong","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Peng","family":"Cheng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2025,10,14]]},"reference":[{"key":"ref9007","first-page":"17300","volume-title":"AdaMV-MoE: Adaptive multi-task vision mixture-of-experts[C\/OL]\/\/2023 IEEE\/CVF International Conference on Computer Vision (ICCV)","author":"Chen","year":"2023"},{"key":"ref9003","first-page":"1","article-title":"PaLM: Scaling language modeling with pathways","volume":"24","author":"Chowdhery","year":"2023","journal-title":"Journal of Machine Learning Research"},{"key":"ref1","author":"Clark","year":"2022"},{"key":"ref2","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2209.13071","article-title":"Diversified dynamic routing for vision tasks","author":"Csaba","year":"2022","journal-title":"arXiv"},{"key":"ref3","author":"Fan","year":"2024"},{"key":"ref4","volume-title":"A review of sparse expert models in deep learning arxiv","author":"Fedus","year":""},{"key":"ref5","doi-asserted-by":"publisher","first-page":"5232","DOI":"10.48550\/arXiv.2101.03961","article-title":"Switch transformers: scaling to trillion parameter models with simple and efficient sparsity","volume":"23","author":"Fedus","year":"","journal-title":"J. Mach. Learn. Res."},{"key":"ref6","author":"Folino","year":"2024"},{"key":"ref7","author":"Gao","year":"2024"},{"key":"ref8","author":"Guo","year":"2024"},{"key":"ref9","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.neucom.2024.128773","article-title":"Preserving text space integrity for robust compositional zero-shot learning via mixture of pretrained experts","volume":"614","author":"Hao","year":"2025","journal-title":"Neurocomputing"},{"key":"ref9005","doi-asserted-by":"publisher","first-page":"12883","DOI":"10.18653\/v1\/2024.acl-long.696","article-title":"Harder task needs more experts: Dynamic routing in moe models. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistic","author":"Huang","year":"2024","journal-title":"Bangkok, Thailand"},{"key":"ref10","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2401.04088","article-title":"Mixtral of experts","author":"Jiang","year":"2024","journal-title":"arXiv"},{"key":"ref11","author":"Jiang","year":"2023"},{"key":"ref12","article-title":"GShard: scaling Giant models with conditional computation and automatic sharding","author":"Lepikhin","year":"2020","journal-title":"Arxiv"},{"key":"ref13","doi-asserted-by":"publisher","first-page":"24520","DOI":"10.1038\/s41598-025-10408-0","article-title":"Graph-based vision transformer with sparsity for training on small datasets from scratch","volume":"15","author":"Li","year":"2025","journal-title":"Sci. Rep."},{"key":"ref14","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2412.14219","article-title":"A survey on inference optimization techniques for mixture of experts models","author":"Liu","year":"2024","journal-title":"arXiv"},{"key":"ref15","article-title":"Routers in vision mixture of experts: an Empirical study","volume-title":"Trans. Mach. Learn. Res","author":"Liu","year":"2024"},{"key":"ref16","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-73397-0_18","article-title":"MM1: methods, Analysis & Insights from multimodal LLM pre-training","author":"Mckinzie","year":"2024","journal-title":"Arxiv"},{"key":"ref17","author":"Mustafa","year":"2022"},{"key":"ref18","author":"Nguyen","year":"2023"},{"key":"ref19","author":"Radford","year":"2021"},{"key":"ref20","author":"Riquelme","year":"2021"},{"key":"ref21","doi-asserted-by":"publisher","first-page":"101429","DOI":"10.1016\/j.csl.2022.101429","article-title":"On the effect of dropping layers of pre-trained transformer models","volume":"77","author":"Sajjad","year":"2023","journal-title":"Comput. Speech Lang."},{"key":"ref9004","article-title":"Outrageously large neural networks: The sparsely-gated mixture-of-experts layer","author":"Shazeer","year":"2017","journal-title":"Arxiv"},{"key":"ref22","article-title":"Llama 2: open foundation and fine-tuned chat models","volume-title":"ArXiv","author":"Touvron","year":"2023"},{"key":"ref9006","first-page":"1","article-title":"Attention is all you need","volume-title":"Advances in Neural Information Processing Systems 30","author":"Vaswani","year":"2017"},{"key":"ref23","author":"Wang","year":"2018"},{"key":"ref9008","article-title":"Auxiliary-loss-free load balancing strategy for mixture-of-experts","author":"Wang","year":"2024","journal-title":"Arxiv"},{"key":"ref24","article-title":"Omni-SMoLA: Boosting generalist multimodal models with soft mixture of low-rank experts","author":"Wu","year":"2024","journal-title":"Arxiv"},{"key":"ref25","doi-asserted-by":"publisher","first-page":"123431","DOI":"10.1016\/j.eswa.2024.123431","article-title":"Unraveling complex data diversity in underwater acoustic target recognition through convolution-based mixture of experts","volume":"249","author":"Xie","year":"2024","journal-title":"Expert Syst. Appl."},{"key":"ref9001","doi-asserted-by":"publisher","first-page":"12858","DOI":"10.18653\/v1\/2023.findings-emnlp.856","article-title":"Towards being parameter-efficient: a stratified sparsely activated transformer with dynamic capacity","author":"Xu","year":"2023","journal-title":"Findings of the Association for Computational Linguistics: EMNLP 2023"},{"key":"ref26","author":"Xu","year":"2024"},{"key":"ref27","doi-asserted-by":"crossref","DOI":"10.1109\/ICCEA62105.2024.10604245","article-title":"Research on the structure and realization of mixture-of-experts","author":"Yan","year":"2024"},{"key":"ref28","author":"Yang","year":"2024"},{"key":"ref29","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2105.03036","article-title":"SpeechMoE: Scaling to large acoustic models with dynamic routing mixture of experts","author":"You","year":"2021","journal-title":"arXiv"},{"key":"ref9002","first-page":"16812","article-title":"Efficient Deweather Mixture-of-Experts with Uncertainty-Aware Feature-Wise Linear Modulation","volume":"38","author":"Zhang","year":"2024","journal-title":"38th AAAI Conference on artificial intelligence, Assoc Advancement Artificial Intelligence"},{"key":"ref30","author":"Zhou","year":"2020"},{"key":"ref31","author":"Zhu","year":"2024"}],"container-title":["Frontiers in Neurorobotics"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2025.1590994\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,14]],"date-time":"2025-10-14T05:27:43Z","timestamp":1760419663000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnbot.2025.1590994\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,14]]},"references-count":39,"alternative-id":["10.3389\/fnbot.2025.1590994"],"URL":"https:\/\/doi.org\/10.3389\/fnbot.2025.1590994","relation":{},"ISSN":["1662-5218"],"issn-type":[{"value":"1662-5218","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,10,14]]},"article-number":"1590994"}}