{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,10]],"date-time":"2026-07-10T11:13:18Z","timestamp":1783681998296,"version":"3.55.0"},"reference-count":26,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2026,7,10]],"date-time":"2026-07-10T00:00:00Z","timestamp":1783641600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Data"],"abstract":"<jats:p>Large Language Models increasingly rely on Mixture-of-Experts architectures to scale model capacity while controlling computational cost. However, most MoE systems employ static expert routing strategies that allocate identical computational resources to all tokens, regardless of their uncertainty or difficulty. This paper proposes Confidence-Based Dynamic Routing (CBDR), an uncertainty-aware expert allocation mechanism for multitask fine-tuning of large language models. CBDR estimates token-level confidence from the entropy of the gating distribution and dynamically adjusts the number of activated experts accordingly. Tokens with high confidence are processed with fewer experts, while uncertain tokens receive additional expert capacity. To mitigate expert imbalance, we introduce Confidence-Based Importance Balancing (CBIB), a training objective that encourages balanced expert utilization. Experiments on multiple biomedical question-answering datasets demonstrate that CBDR improves model performance across multiple biomedical QA datasets, achieving absolute improvements of up to 3% in F1 score on the BioASQ benchmark while maintaining efficient expert utilization. These results suggest that uncertainty-aware routing provides a practical mechanism for adaptive compute allocation in sparse large language models.<\/jats:p>","DOI":"10.3390\/data11070172","type":"journal-article","created":{"date-parts":[[2026,7,10]],"date-time":"2026-07-10T09:46:47Z","timestamp":1783676807000},"page":"172","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Uncertainty-Aware Expert Allocation for Efficient Multitask Fine-Tuning of Large Language Models"],"prefix":"10.3390","volume":"11","author":[{"ORCID":"https:\/\/orcid.org\/0009-0002-7095-2616","authenticated-orcid":false,"given":"Maab","family":"Elhassan","sequence":"first","affiliation":[{"name":"Electrical and Computer Engineering, Korea University, Seoul 02841, Republic of Korea"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8938-9080","authenticated-orcid":false,"given":"Minhee","family":"Jun","sequence":"additional","affiliation":[{"name":"Electrical and Computer Engineering, Catholic University of America, Washington, DC 20064, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8744-4514","authenticated-orcid":false,"given":"Hanseok","family":"Ko","sequence":"additional","affiliation":[{"name":"Electrical and Computer Engineering, Catholic University of America, Washington, DC 20064, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2026,7,10]]},"reference":[{"key":"ref_1","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, \u0141., and Polosukhin, I. (2017, January 4\u20139). Attention Is All You Need. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Long Beach, CA, USA."},{"key":"ref_2","unstructured":"Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., and Askell, A. (2020, January 6\u201312). Language Models Are Few-Shot Learners. Proceedings of the NeurIPS, Vancouver, BC, Canada."},{"key":"ref_3","unstructured":"Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., and Anadkat, S. (2023). GPT-4 Technical Report. arXiv."},{"key":"ref_4","unstructured":"Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. (2019, January 10\u201315). Parameter-Efficient Transfer Learning for NLP. Proceedings of the ICML, Long Beach, CA, USA."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Li, X., and Liang, P. (2021, January 1\u20136). Prefix-Tuning: Optimizing Continuous Prompts for Generation. Proceedings of the ACL, Bangkok, Thailand.","DOI":"10.18653\/v1\/2021.acl-long.353"},{"key":"ref_6","unstructured":"Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. (2022, January 25\u201329). LoRA: Low-Rank Adaptation of Large Language Models. Proceedings of the ICLR, Virtual."},{"key":"ref_7","unstructured":"Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. (2017, January 24\u201326). Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. Proceedings of the ICLR, Toulon, France."},{"key":"ref_8","first-page":"1","article-title":"Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity","volume":"23","author":"Fedus","year":"2022","journal-title":"J. Mach. Learn. Res."},{"key":"ref_9","unstructured":"Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., and Chen, Z. (2021, January 3\u20137). GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. Proceedings of the ICLR, Vienna, Austria."},{"key":"ref_10","unstructured":"Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019, January 2\u20137). BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the NAACL-HLT, Minneapolis, MN, USA."},{"key":"ref_11","unstructured":"Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. (2018). Improving Language Understanding by Generative Pre-Training, OpenAI. OpenAI Technical Report."},{"key":"ref_12","unstructured":"Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language Models Are Unsupervised Multitask Learners, OpenAI. OpenAI Technical Report."},{"key":"ref_13","unstructured":"Kaplan, J., McCandlish, S., Henighan, T., Brown, T., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. (2020). Scaling Laws for Neural Language Models. arXiv."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. arXiv.","DOI":"10.52202\/075280-0441"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"41","DOI":"10.1023\/A:1007379606734","article-title":"Multitask Learning","volume":"28","author":"Caruana","year":"1997","journal-title":"Mach. Learn."},{"key":"ref_16","first-page":"1","article-title":"Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer","volume":"21","author":"Raffel","year":"2020","journal-title":"J. Mach. Learn. Res."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"5586","DOI":"10.1109\/TKDE.2021.3070203","article-title":"A Survey on Multi-Task Learning","volume":"34","author":"Zhang","year":"2022","journal-title":"IEEE Trans. Knowl. Data Eng."},{"key":"ref_18","unstructured":"Graves, A. (2016). Adaptive Computation Time for Recurrent Neural Networks. arXiv."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Huang, Q., An, Z., Zhuang, N., Tao, M., Zhang, C., Jin, Y., Xu, K., Xu, L., Chen, L., and Huang, S. (2024). Harder Tasks Need More Experts: Dynamic Routing in Mixture-of-Experts Models. arXiv.","DOI":"10.18653\/v1\/2024.acl-long.696"},{"key":"ref_20","unstructured":"Yun, L., Zhuang, Y., Fu, Y., Xing, E.P., and Zhang, H. (2024). Toward Inference-Optimal Mixture-of-Experts Large Language Models. arXiv."},{"key":"ref_21","unstructured":"Jiang, A.Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D.S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., and Saulnier, L. (2023). Mistral 7B. arXiv."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Nentidis, A., Katsimpras, G., Krithara, A., Krallinger, M., Rodr\u00edguez-Ortega, M., Loukachevitch, N., Sakhovskiy, A., Tutubalina, E., Dimitriadis, D., and Tsoumakas, G. (2025, January 6\u201310). Overview of BioASQ 2025: The Thirteenth BioASQ Challenge on Large-Scale Biomedical Semantic Indexing and Question Answering. Proceedings of the CLEF Working Notes, Lucca, Italy.","DOI":"10.1007\/978-3-032-04354-2_12"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Tsatsaronis, G., Balikas, G., Malakasiotis, P., Partalas, I., Zschunke, M., Alvers, M.R., Weissenborn, D., Krithara, A., Petridis, S., and Polychronopoulos, D. (2015). An Overview of the BioASQ Large-Scale Biomedical Semantic Indexing and Question Answering Competition. BMC Bioinform., 16.","DOI":"10.1186\/s12859-015-0564-6"},{"key":"ref_24","unstructured":"(2024, September 09). BioASQ Dataset. Available online: http:\/\/bioasq.org."},{"key":"ref_25","unstructured":"Jin, Q., Dhingra, B., Liu, Z., Cohen, W., and Lu, X. (2019, January 3\u20137). PubMedQA: A Dataset for Biomedical Research Question Answering. Proceedings of the EMNLP, Hong Kong, China."},{"key":"ref_26","unstructured":"Kell, G., Roberts, A., Umansky, S., Khare, Y., Ahmed, N., Patel, N., Simela, C., Coumbe, J., Rozario, J., and Griffiths, R.R. (2024). RealMedQA: A Pilot Biomedical Question Answering Dataset Containing Realistic Clinical Questions. arXiv."}],"container-title":["Data"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2306-5729\/11\/7\/172\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,10]],"date-time":"2026-07-10T10:20:43Z","timestamp":1783678843000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2306-5729\/11\/7\/172"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,10]]},"references-count":26,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2026,7]]}},"alternative-id":["data11070172"],"URL":"https:\/\/doi.org\/10.3390\/data11070172","relation":{},"ISSN":["2306-5729"],"issn-type":[{"value":"2306-5729","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,10]]}}}