{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,16]],"date-time":"2026-06-16T20:01:30Z","timestamp":1781640090243,"version":"3.54.5"},"reference-count":43,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2026,3,12]],"date-time":"2026-03-12T00:00:00Z","timestamp":1773273600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,4,2]],"date-time":"2026-04-02T00:00:00Z","timestamp":1775088000000},"content-version":"vor","delay-in-days":21,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"the Natural Science Basis Research Plan in Shaanxi Province of China"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Int J Comput Intell Syst"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>\n                    Mixture of Experts (MoE) models reduce computational costs through a sparse activation strategy, making it widely adopted in large-scale model training. However, MoE structures still face issues such as knowledge redundancy and token overflow caused by static routing. Existing solutions typically involve increasing expert capacity or applying stricter loss constraints to balance expert loads. These approaches can lead to wasted computational resources due to complex loss functions and free expert capacity. This paper proposes an optimization strategy that combines consensus experts with dynamic capacity allocation. Firstly, in the MoE structure of the consensus expert, the\n                    <jats:bold>ARIMA<\/jats:bold>\n                    time series prediction algorithm is used to predict future load distribution, and the experts\u2019 dynamic capacity is allocated to alleviate routing fluctuations during the initial training phase. Secondly, by comparing the predicted load with the actual load, the loss function is reconstructed to avoid pursuing absolute balance. Experiments on the\n                    <jats:bold>Switch Transformer-base-8<\/jats:bold>\n                    model show that our method achieves an average accuracy of 81.55% across 9 GLUE benchmark tasks\u2014an improvement of 1.07% over the baseline. The gains are notable on imbalanced datasets (e.g., MRPC), with improvements of up to 1.8%. Further experiments demonstrate the method\u2019s effectiveness at expert numbers of 16 and 32.\n                  <\/jats:p>","DOI":"10.1007\/s44196-026-01236-9","type":"journal-article","created":{"date-parts":[[2026,3,12]],"date-time":"2026-03-12T11:16:50Z","timestamp":1773314210000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Consensus-Expert DynamicMoE: ARIMA-based Capacity Prediction with Adaptive Load Balancing for Sparse Models"],"prefix":"10.1007","volume":"19","author":[{"given":"Jia-Lin","family":"Wen","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8638-636X","authenticated-orcid":false,"given":"Xiao-Jun","family":"Li","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jun-Ping","family":"Yao","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hai-Feng","family":"Sun","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Xiang-Rui","family":"An","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,3,12]]},"reference":[{"key":"1236_CR1","doi-asserted-by":"publisher","unstructured":"Wang, J. et al.: A comprehensive review of multimodal large language models: performance and challenges across different tasks. Preprint at (2024). https:\/\/doi.org\/10.48550\/arXiv.2408.01319","DOI":"10.48550\/arXiv.2408.01319"},{"key":"1236_CR2","unstructured":"Brown, T. B. et al.: Language models are Few-Shot Learners. In advances in neural information processing systems. NEURIPS. 33, (2020)."},{"key":"1236_CR3","doi-asserted-by":"publisher","unstructured":"Noels, S., De Blaere, J. & De Bie, T.: A Dutch financial large language model. In 5th ACM International Conference on AIin Finance, ICAIF. 283\u2013291 (2024). https:\/\/doi.org\/10.1145\/3677052.3698628","DOI":"10.1145\/3677052.3698628"},{"key":"1236_CR4","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","volume":"620","author":"K Singhal","year":"2023","unstructured":"Singhal, K., et al.: Large language models encode clinical knowledge. Nature 620, 172 (2023)","journal-title":"Nature"},{"key":"1236_CR5","doi-asserted-by":"publisher","unstructured":"Zoph, B. et al.: ST-MoE: designing stable and transferable sparse expert models. Preprint at (2022). https:\/\/doi.org\/10.48550\/arXiv.2202.08906","DOI":"10.48550\/arXiv.2202.08906"},{"key":"1236_CR6","doi-asserted-by":"crossref","unstructured":"Cai, W. et al.: A Survey on mixture of experts. Arxiv (2024).","DOI":"10.36227\/techrxiv.172055626.64129172\/v1"},{"key":"1236_CR7","unstructured":"Vaswani, A. et al.: Attention is all you need. in (eds Guyon, I. et al.,) 30 (2017)."},{"key":"1236_CR8","doi-asserted-by":"publisher","unstructured":"Yan, Q., Li, Y., Ma, N. & Wan, F et al.: Research on the structure and realization of mixture-of-experts. In 2024 5th International Conference on Computer Engineering and AApplication, ICCEA. 354\u2013359 (IEEE, 2024). https:\/\/doi.org\/10.1109\/ICCEA62105.2024.10604245.","DOI":"10.1109\/ICCEA62105.2024.10604245"},{"key":"1236_CR9","doi-asserted-by":"publisher","unstructured":"Liu, J. et al.: A survey on inference optimization techniques for mixture of experts models. Preprint at (2024). https:\/\/doi.org\/10.48550\/arXiv.2412.14219","DOI":"10.48550\/arXiv.2412.14219"},{"key":"1236_CR10","doi-asserted-by":"publisher","unstructured":"Chen, T. et al.: AdaMV-MoE: adaptive multi-task vision mixture-of-experts. In 2023 IEEE\/CVF International Conference on Computer Vision (ICCV) 17300\u201317311 (IEEE, Paris, France) (2023). https:\/\/doi.org\/10.1109\/ICCV51070.2023.01591","DOI":"10.1109\/ICCV51070.2023.01591"},{"key":"1236_CR11","doi-asserted-by":"publisher","unstructured":"Zareapoor, M., Shamsolmoali, P. & Vesaghati, F et al.: Efficient routing in sparse mixture-of-experts. in 2024 International Joint Conference on Neural Networks (IJCNN) 1\u20138 (IEEE, Yokohama, Japan) (2024). https:\/\/doi.org\/10.1109\/IJCNN60899.2024.10650737","DOI":"10.1109\/IJCNN60899.2024.10650737"},{"key":"1236_CR12","doi-asserted-by":"publisher","unstructured":"Li, P., et al.: Merge then compress: demystify efficient SMoE with hints from Its routing policy. Preprint at (2024). https:\/\/doi.org\/10.48550\/arXiv.2310.01334","DOI":"10.48550\/arXiv.2310.01334"},{"key":"1236_CR13","doi-asserted-by":"publisher","unstructured":"Pedicir, E., Miller, L. & Robinson, L et al.: Novel token-level recurrent routing for enhanced mixture-of-experts performance. Preprint at (2024). https:\/\/doi.org\/10.22541\/au.172479156.65470023\/v1","DOI":"10.22541\/au.172479156.65470023\/v1"},{"key":"1236_CR14","doi-asserted-by":"publisher","unstructured":"Qiu, Z. et al.: Layerwise Recurrent router for mixture-of-experts. Preprint at (2024). https:\/\/doi.org\/10.48550\/arXiv.2408.06793","DOI":"10.48550\/arXiv.2408.06793"},{"key":"1236_CR15","doi-asserted-by":"publisher","unstructured":"Dai, D. et al.: DeepSeekMoE: Towards ultimate expert specialization in mixture-of-experts language models. Preprint at (2024). https:\/\/doi.org\/10.48550\/arXiv.2401.06066","DOI":"10.48550\/arXiv.2401.06066"},{"key":"1236_CR16","doi-asserted-by":"publisher","unstructured":"Lo, K. M., Huang, Z., Qiu, Z., Wang, Z. & Fu, J et al.: A closer look into mixture-of-experts in large language models. Preprint at (2024). https:\/\/doi.org\/10.48550\/arXiv.2406.18219","DOI":"10.48550\/arXiv.2406.18219"},{"key":"1236_CR17","doi-asserted-by":"crossref","unstructured":"Di Palo, F., Singhi, P. & Fadlallah, B., et al.: Performance-Guided LLM Knowledge Distillation for Efficient Text Classification at Scale. in 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024 (eds Al-Onaizan, Y., Bansal, M. & Chen, Y.) 3675\u20133687 (ByteDance Inc; Intuit Inc; Adobe Inc; Florida International University; Figma Inc,) (2024)","DOI":"10.18653\/v1\/2024.emnlp-main.215"},{"key":"1236_CR18","unstructured":"Liu, T., Blondel, M., Ruiz, C. R. & Puigcerver, J., et al.: Routers in vision mixture of experts: an empirical study. Trans. Mac. Learning Res. (2024)"},{"key":"1236_CR19","doi-asserted-by":"publisher","unstructured":"Yang, Y. et al.: XMoE: sparse models with fine-grained and adaptive expert selection. In Findings of the Association for Computational Linguistics ACL 2024 11664\u201311674 (Bangkok, Thailand) (2024). https:\/\/doi.org\/10.18653\/v1\/2024.findings-acl.694","DOI":"10.18653\/v1\/2024.findings-acl.694"},{"key":"1236_CR20","doi-asserted-by":"crossref","unstructured":"Esiobu, D. et al., ROBBIE: Robust Bias evaluation of large generative language models. in 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP, 3764\u20133814 (2023).","DOI":"10.18653\/v1\/2023.emnlp-main.230"},{"key":"1236_CR21","doi-asserted-by":"publisher","unstructured":"Arnold, S., Fietta, M. & Yesilbas, D. Routing in sparsely-gated language models responds to context. Preprint at (2024). https:\/\/doi.org\/10.48550\/arXiv.2409.14107","DOI":"10.48550\/arXiv.2409.14107"},{"key":"1236_CR22","doi-asserted-by":"publisher","unstructured":"Lu, X. et al.: Not all experts are equal: efficient expert pruning and skipping for mixture-of-experts large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) 6159\u20136172 (Bangkok, Thailand) (2024). https:\/\/doi.org\/10.18653\/v1\/2024.acl-long.334","DOI":"10.18653\/v1\/2024.acl-long.334"},{"key":"1236_CR23","doi-asserted-by":"publisher","unstructured":"Zhao, H. et al.: HyperMoE: towards better mixture of experts via transferring among experts. Preprint at (2024). https:\/\/doi.org\/10.48550\/arXiv.2402.12656","DOI":"10.48550\/arXiv.2402.12656"},{"key":"1236_CR24","doi-asserted-by":"publisher","unstructured":"Do, G. et al.: HyperRouter: towards efficient training and inference of sparse mixture of experts. Preprint at (2023). https:\/\/doi.org\/10.48550\/arXiv.2312.07035","DOI":"10.48550\/arXiv.2312.07035"},{"key":"1236_CR25","doi-asserted-by":"publisher","unstructured":"Wu, X., Huang, S., Wang, W. & Wei, F. Multi-head mixture-of-experts. Preprint at (2024). https:\/\/doi.org\/10.48550\/arXiv.2404.15045","DOI":"10.48550\/arXiv.2404.15045"},{"key":"1236_CR26","doi-asserted-by":"crossref","unstructured":"Zhou, Y. et al.: Mixture-of-experts with expert choice routing. In Advances in Neural Information Processing Systems. 35, 7103\u20137114 (Curran Associates, Inc., 2022).","DOI":"10.52202\/068431-0515"},{"key":"1236_CR27","unstructured":"Roller, S., Sukhbaatar, S., Szlam, A. & Weston, J. Hash Layers For Large Sparse Models. in Advances in Neural Information Processing Systems, (NEURIPS). 34, (2021)."},{"key":"1236_CR28","unstructured":"Lepikhin, D. et al.: GShard: scaling giant models with conditional computation and automatic sharding. Arxiv (2020)."},{"key":"1236_CR29","doi-asserted-by":"publisher","unstructured":"Fedus, W., Zoph, B. & Shazeer, N.: Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. Preprint at (2022). https:\/\/doi.org\/10.48550\/arXiv.2101.03961","DOI":"10.48550\/arXiv.2101.03961"},{"key":"1236_CR30","doi-asserted-by":"publisher","unstructured":"Wang, W. et al.: Prophet: fine-grained load balancing for parallel training of large-scale MoE models. In 2023 IEEE International Conference on Cluster Computing, 82\u201394 (IEEE, Santa Fe, NM, USA). https:\/\/doi.org\/10.1109\/CLUSTER52292.2023.00015. (2023).","DOI":"10.1109\/CLUSTER52292.2023.00015"},{"key":"1236_CR31","unstructured":"Li, J. et al.: LocMoE: A low-overhead moe for large language model training. In Proceedings of the 33th International Joint Conference on Artificial Intelligence, IJCAI, 6377\u20136387 (Int Joint Conf Artifical Intelligence, 2024)."},{"key":"1236_CR32","doi-asserted-by":"publisher","unstructured":"DeepSeek-AI et al.: DeepSeek-V2: a strong, economical, and efficient mixture-of-experts language model. Preprint at (2024). https:\/\/doi.org\/10.48550\/arXiv.2405.04434","DOI":"10.48550\/arXiv.2405.04434"},{"key":"1236_CR33","doi-asserted-by":"publisher","first-page":"677","DOI":"10.1109\/TPDS.2025.3539297","volume":"36","author":"Y Zeng","year":"2025","unstructured":"Zeng, Y., et al.: Efficientmoe: optimizing mixture-of-experts model training with adaptive load balance. IEEE Trans. Parallel Distrib. Syst. 36, 677\u2013688 (2025)","journal-title":"IEEE Trans. Parallel Distrib. Syst."},{"key":"1236_CR34","doi-asserted-by":"publisher","unstructured":"Cong, P. et al.: Prediction is all moe needs: expert load distribution goes from fluctuating to stabilizing. Preprint at (2024). https:\/\/doi.org\/10.48550\/arXiv.2404.16914","DOI":"10.48550\/arXiv.2404.16914"},{"key":"1236_CR35","doi-asserted-by":"publisher","unstructured":"Makris, I. et al.: Exploring Load Forecasting: Bridging Statistical Methods to Deep Learning Techniques in Real-World Environment. In 2024 20th International Conference on Distributed Computing in Smart Systems and the Internet of Things, DCOSS-IOT, 491\u2013496 (IEEE Comp Soc Tech Comm Parallel Proc; Technol Innovat Inst) (2024). https:\/\/doi.org\/10.1109\/DCOSS-IoT61029.2024.00078","DOI":"10.1109\/DCOSS-IoT61029.2024.00078"},{"key":"1236_CR36","doi-asserted-by":"publisher","first-page":"274","DOI":"10.1016\/j.knosys.2018.10.041","volume":"164","author":"R Salles","year":"2019","unstructured":"Salles, R., et al.: Nonstationary time series transformation methods: an experimental review. Knowl. Based Syst. 164, 274\u2013291 (2019)","journal-title":"Knowl. Based Syst."},{"key":"1236_CR37","doi-asserted-by":"publisher","DOI":"10.3390\/fi15080255","volume":"15","author":"VI Kontopoulou","year":"2023","unstructured":"Kontopoulou, V.I., et al.: A review of ARIMA vs. machine learning approaches for time series forecasting in data driven networks. Future Internet 15, 255 (2023)","journal-title":"Future Internet"},{"key":"1236_CR38","doi-asserted-by":"publisher","unstructured":"Zeng, Z. et al.: AdaMoE: Token-adaptive routing with null experts for mixture-of-experts language models. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2406.13233 (2024)","DOI":"10.48550\/arXiv.2406.13233"},{"key":"1236_CR39","doi-asserted-by":"publisher","unstructured":"Wang, A. et al.: GLUE: a multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 353\u2013355 (Brussels, Belgium) (2018). https:\/\/doi.org\/10.18653\/v1\/W18-5446","DOI":"10.18653\/v1\/W18-5446"},{"key":"1236_CR40","doi-asserted-by":"publisher","first-page":"13807","DOI":"10.1609\/aaai.v37i11.26617","volume":"37","author":"Y Xie","year":"2023","unstructured":"Xie, Y., Huang, S., Chen, T., Wei, F.: MoEC: Mixture of Expert Clusters. AAAI 37, 13807\u201313815 (2023)","journal-title":"AAAI"},{"key":"1236_CR41","doi-asserted-by":"publisher","unstructured":"Pan, X. et al.: FSMoE: a flexible and scalable training system for sparse mixture-of-experts models. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS.1, 524\u2013539 (2025) https:\/\/doi.org\/10.1145\/3669940.3707272","DOI":"10.1145\/3669940.3707272"},{"key":"1236_CR42","doi-asserted-by":"publisher","unstructured":"Huang, Q. et al.: Harder task needs more experts: dynamic routing in MoE models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. 1, 12883\u201312895 (Bangkok, Thailand) (2024). https:\/\/doi.org\/10.18653\/v1\/2024.acl-long.696","DOI":"10.18653\/v1\/2024.acl-long.696"},{"key":"1236_CR43","unstructured":"Zhou, W. et al.: BERT loses patience: fast and robust inference with early exit. In Proceedings of the 34th International Conference on Neural Information Processing Systems. (Red Hook, NY, USA, 2020)."}],"container-title":["International Journal of Computational Intelligence Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s44196-026-01236-9","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s44196-026-01236-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s44196-026-01236-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,16]],"date-time":"2026-06-16T19:48:22Z","timestamp":1781639302000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s44196-026-01236-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,3,12]]},"references-count":43,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,12]]}},"alternative-id":["1236"],"URL":"https:\/\/doi.org\/10.1007\/s44196-026-01236-9","relation":{},"ISSN":["1875-6883"],"issn-type":[{"value":"1875-6883","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,3,12]]},"assertion":[{"value":"17 September 2025","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"19 January 2026","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"13 February 2026","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 March 2026","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no competing interests. All authors certify that they have no affiliations with or involvement in any organization or entity with any financial interest or non-financial interest in the subject matter or materials discussed in this manuscript.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing Interests"}}],"article-number":"157"}}