{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,13]],"date-time":"2026-07-13T08:08:02Z","timestamp":1783930082212,"version":"3.55.0"},"reference-count":56,"publisher":"Institution of Engineering and Technology (IET)","issue":"3","license":[{"start":{"date-parts":[[2026,5,15]],"date-time":"2026-05-15T00:00:00Z","timestamp":1778803200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"},{"start":{"date-parts":[[2026,5,15]],"date-time":"2026-05-15T00:00:00Z","timestamp":1778803200000},"content-version":"tdm","delay-in-days":0,"URL":"http:\/\/doi.wiley.com\/10.1002\/tdm_license_1.1"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["62576159"],"award-info":[{"award-number":["62576159"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["61402220"],"award-info":[{"award-number":["61402220"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["ietresearch.onlinelibrary.wiley.com"],"crossmark-restriction":true},"short-container-title":["CAAI Trans on Intel Tech"],"published-print":{"date-parts":[[2026,6]]},"abstract":"<jats:title>ABSTRACT<\/jats:title>\n                  <jats:p>Parameter\u2010efficient fine\u2010tuning (PEFT) has become a crucial paradigm for domain adaptation, achieving strong performance by updating only a small fraction of model parameters. Among various PEFT methods, low\u2010rank adaptation (LoRA) is widely adopted due to its structural simplicity and computational efficiency. However, in multitask scenarios, LoRA often suffers from performance degradation due to task interference. Recent extensions, such as mixture\u2010of\u2010experts (MoE) and asymmetric LoRA variants, attempt to mitigate these issues; however, their reliance on fixed subspace\u2010mixing strategies limits flexibility and makes the models more sensitive to input noise and data sparsity. Moreover, vanilla LoRA typically initialises low\u2010rank matrices with Gaussian noise or zeros and optimises them in unconstrained subspaces, which may disrupt the structured representations learnt by pretrained models. In this article, we propose M3LoRA, a novel PEFT framework that leverages multiple low\u2010rank matrices with mixture\u2010of\u2010subspaces and minor singular components initialisation. M3LoRA utilises multiple low\u2010rank matrices to minimise interference between task\u2010specific subspaces while maintaining representational capacity. Furthermore, it employs a learnable mixing matrix positioned between the down\u2010projection and up\u2010projection matrices to dynamically combine their subspaces, thereby enabling adaptive task\u2010specific combination mechanisms. Additionally, it initialises these low\u2010rank matrices within a subspace orthogonal to the principal singular components of pretrained weights\u2014termed the minor singular components\u2014thereby leveraging directions unexplored during pretraining to better capture task\u2010specific features from labelled data. Extensive experiments on a wide range of benchmark datasets demonstrate that M3LoRA achieves substantial improvements over existing PEFT baselines, particularly in multi\u2010task scenarios where task interference often degrades conventional LoRA performance.<\/jats:p>","DOI":"10.1049\/cit2.70144","type":"journal-article","created":{"date-parts":[[2026,5,15]],"date-time":"2026-05-15T07:30:13Z","timestamp":1778830213000},"page":"681-694","update-policy":"https:\/\/doi.org\/10.1002\/crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["M3LoRA: Flexible Task Adaptation via Multiple Low\u2010Rank Matrices With Mixture\u2010of\u2010Subspaces and Minor Singular Components Initialization"],"prefix":"10.1049","volume":"11","author":[{"given":"Xu","family":"Luo","sequence":"first","affiliation":[{"name":"School of Computer University of South China  Hengyang China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3369-3101","authenticated-orcid":false,"given":"Yongbin","family":"Liu","sequence":"additional","affiliation":[{"name":"School of Computer University of South China  Hengyang China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Chunping","family":"Ouyang","sequence":"additional","affiliation":[{"name":"School of Computer University of South China  Hengyang China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ying","family":"Yu","sequence":"additional","affiliation":[{"name":"School of Computer University of South China  Hengyang China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yang","family":"Yang","sequence":"additional","affiliation":[{"name":"College of Computer Science and Technology Zhejiang University  Hangzhou China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"265","published-online":{"date-parts":[[2026,5,15]]},"reference":[{"key":"e_1_2_11_2_1","doi-asserted-by":"publisher","DOI":"10.52202\/075280-2020"},{"key":"e_1_2_11_3_1","doi-asserted-by":"crossref","unstructured":"P.Wang L.Li Z.Shao et\u00a0al. \u201cMath\u2010Shepherd: Verify and Reinforce LLMS Step\u2010by\u2010Step Without Human Annotations \u201d preprint arXiv:2312.08935 (2023).","DOI":"10.18653\/v1\/2024.acl-long.510"},{"key":"e_1_2_11_4_1","unstructured":"H.Ivison Y.Wang V.Pyatkin et\u00a0al. \u201cCamels in a Changing Climate: Enhancing LM Adaptation With Tulu 2 \u201d (2023) https:\/\/arxiv.org\/abs\/2311.10702.2023."},{"issue":"2","key":"e_1_2_11_5_1","first-page":"3","article-title":"Lora: Low\u2010Rank Adaptation of Large Language Models","volume":"1","author":"Hu E. J.","year":"2022","journal-title":"International Conference on Learning Representations"},{"key":"e_1_2_11_6_1","doi-asserted-by":"publisher","DOI":"10.1049\/cit2.12062"},{"key":"e_1_2_11_7_1","doi-asserted-by":"crossref","unstructured":"W.Feng C.Hao Y.Zhang Y.Han andH.Wang \u201cMixture\u2010of\u2010Loras: An Efficient Multitask Tuning for Large Language Models \u201d preprint arXiv:2403.03432 (2024).","DOI":"10.63317\/5m6ctqrk686e"},{"key":"e_1_2_11_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/3626772.3657722"},{"key":"e_1_2_11_9_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52733.2024.01533"},{"key":"e_1_2_11_10_1","doi-asserted-by":"publisher","DOI":"10.52202\/079017-0304"},{"key":"e_1_2_11_11_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v39i20.35509"},{"key":"e_1_2_11_12_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.inffus.2025.103081"},{"key":"e_1_2_11_13_1","doi-asserted-by":"crossref","unstructured":"Y.Zhou C.Yao andJ.Chen \u201cCola: Collaborative Low\u2010Rank Adaptation \u201d preprint arXiv:2505.15471 (2025).","DOI":"10.18653\/v1\/2025.findings-acl.726"},{"key":"e_1_2_11_14_1","doi-asserted-by":"publisher","DOI":"10.52202\/079017-3846"},{"key":"e_1_2_11_15_1","doi-asserted-by":"crossref","unstructured":"H.Wang Y.Li S.Wang G.Chen andY.Chen \u201cMilora: Harnessing Minor Singular Components for Parameter\u2010Efficient LLM Finetuning \u201d preprint arXiv:2406.09044 (2024).","DOI":"10.18653\/v1\/2025.naacl-long.248"},{"key":"e_1_2_11_16_1","first-page":"6","article-title":"Compressing Pre\u2010Trained Language Models Using Progressive Low Rank Decomposition","volume":"35","author":"Hajimolahoseini H.","year":"2021","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_11_17_1","unstructured":"P.Sharma J. T.Ash andD.Misra \u201cThe Truth Is in There: Improving Reasoning in Language Models With Layer\u2010Selective Rank Reduction \u201d preprint arXiv:2312.13558 (2023)."},{"key":"e_1_2_11_18_1","doi-asserted-by":"publisher","DOI":"10.52202\/079017-2319"},{"key":"e_1_2_11_19_1","doi-asserted-by":"crossref","unstructured":"X.Han W.Zhao N.Ding Z.Liu andM.Sun \u201cPTR: Prompt Tuning With Rules for Text Classification \u201d preprint arXiv:2105.11259 (2021).","DOI":"10.1016\/j.aiopen.2022.11.003"},{"key":"e_1_2_11_20_1","doi-asserted-by":"crossref","unstructured":"B.Lester R.Al\u2010Rfou andN.Constant \u201cThe Power of Scale for Parameter\u2010Efficient Prompt Tuning \u201d preprint arXiv:2104.08691 (2021).","DOI":"10.18653\/v1\/2021.emnlp-main.243"},{"key":"e_1_2_11_21_1","unstructured":"X. L.LiandP.Liang \u201cPrefix\u2010Tuning: Optimizing Continuous Prompts for Generation \u201d preprint arXiv:2101.00190 (2021)."},{"key":"e_1_2_11_22_1","doi-asserted-by":"publisher","DOI":"10.1049\/cit2.12086"},{"key":"e_1_2_11_23_1","unstructured":"N.Houlsby A.Giurgiu S.Jastrzebski et\u00a0al. \u201cParameter\u2010Efficient Transfer Learning for NLP \u201d preprint arXiv:1902.00751 (2019)."},{"key":"e_1_2_11_24_1","doi-asserted-by":"crossref","unstructured":"A.R\u00fcckl\u00e9 G.Geigle M.Glockner et\u00a0al. \u201cAdapterDrop: On the Efficiency of Adapters in Transformers \u201d preprint arXiv:2010.11918 (2021).","DOI":"10.18653\/v1\/2021.emnlp-main.626"},{"key":"e_1_2_11_25_1","doi-asserted-by":"publisher","DOI":"10.52202\/079017-1311"},{"key":"e_1_2_11_26_1","unstructured":"Y.Qin X.Wang Y.Su et\u00a0al. \u201cExploring Universal Intrinsic Task Subspace via Prompt Tuning \u201d preprint arXiv:2110.07867 (2021)."},{"key":"e_1_2_11_27_1","doi-asserted-by":"publisher","DOI":"10.3724\/2096\u20107004.di.2025.0009"},{"key":"e_1_2_11_28_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.eacl-main.239"},{"key":"e_1_2_11_29_1","unstructured":"Q.Zhang M.Chen A.Bukharin et\u00a0al. \u201cAdaLoRA: Adaptive Budget Allocation for Parameter\u2010Efficient Fine\u2010Tuning \u201d preprint arXiv:2303.10512 (2023)."},{"key":"e_1_2_11_30_1","unstructured":"S. Y.Liu C. Y.Wang H.Yin et\u00a0al. \u201cDoRA: Weight\u2010Decomposed Low\u2010Rank Adaptation \u201d preprint arXiv:2402.09353 (2024)."},{"key":"e_1_2_11_31_1","unstructured":"D. J.Kopiczko T.Blankevoort andY. M.Asano \u201cVeRA: Vector\u2010Based Random Matrix Adaptation \u201d preprint arXiv:2310.11454 (2023)."},{"key":"e_1_2_11_32_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.emnlp-main.57"},{"key":"e_1_2_11_33_1","unstructured":"Z.Gao Q.Wang A.Chen et\u00a0al. \u201cParameter\u2010Efficient Fine\u2010Tuning With Discrete Fourier Transform \u201d preprint arXiv:2405.03003 (2024)."},{"key":"e_1_2_11_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/5.726791"},{"key":"e_1_2_11_35_1","doi-asserted-by":"publisher","DOI":"10.1038\/14819"},{"key":"e_1_2_11_36_1","doi-asserted-by":"publisher","DOI":"10.1162\/neco.2006.18.7.1527"},{"key":"e_1_2_11_37_1","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v39i5.32510"},{"key":"e_1_2_11_38_1","unstructured":"C.Gao K.Chen J.Rao et\u00a0al. \u201cHigher Layers Need More Lora Experts \u201d preprint arXiv:2402.08562 (2024)."},{"key":"e_1_2_11_39_1","doi-asserted-by":"crossref","unstructured":"T.Wu J.Wang Z.Zhao andN.Wong \u201cMixture\u2010of\u2010Subspaces in Low\u2010Rank Adaptation \u201d preprint arXiv:2406.11909 (2024).","DOI":"10.18653\/v1\/2024.emnlp-main.450"},{"key":"e_1_2_11_40_1","unstructured":"H.Yang L.Wang andM. Z.Hossain \u201cSRLoRA: Subspace Recomposition in Low\u2010Rank Adaptation via Importance\u2010Based Fusion and Reinitialization \u201d preprint arXiv:2505.12433 (2025)."},{"key":"e_1_2_11_41_1","unstructured":"M.Conover M.Hayes A.Mathur et\u00a0al. \u201cFree Dolly: Introducing the World\u2019s First Truly Open Instruction\u2010Tuned LLM \u201d (2023) https:\/\/www.databricks.com\/blog\/2023\/04\/12\/dolly\u2010first\u2010open\u2010commercially\u2010viable\u2010instruction\u2010tuned\u2010llm."},{"key":"e_1_2_11_42_1","unstructured":"D.Hendrycks C.Burns S.Basart et\u00a0al. \u201cMeasuring Massive Multitask Language Understanding \u201d preprint arXiv:2009.03300 (2020)."},{"key":"e_1_2_11_43_1","unstructured":"Alignment\u2010Lab\u2010AI \u201cLawyer\u2010Instruct \u201d (2024)."},{"key":"e_1_2_11_44_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.acl-long.865"},{"key":"e_1_2_11_45_1","doi-asserted-by":"publisher","DOI":"10.7759\/cureus.40895"},{"key":"e_1_2_11_46_1","unstructured":"K.Cobbe V.Kosaraju M.Bavarian et\u00a0al. \u201cTraining Verifiers to Solve Math Word Problems \u201d preprint arXiv:2110.14168 (2021)."},{"key":"e_1_2_11_47_1","doi-asserted-by":"crossref","unstructured":"N.Wang H.Yang andC. D.Wang \u201cFingpt: Instruction Tuning Benchmark for Open\u2010Source Large Language Models in Financial Datasets \u201d preprint arXiv:2310.04793 (2023).","DOI":"10.2139\/ssrn.4489826"},{"key":"e_1_2_11_48_1","article-title":"Openorca: An Open Dataset of Gpt Augmented Flan Reasoning Traces","author":"Lian W.","year":"2023","journal-title":"Hugging Face Dataset Repository"},{"key":"e_1_2_11_49_1","doi-asserted-by":"crossref","unstructured":"M.Suzgun N.Scales N.Sch\u00e4rli et\u00a0al. \u201cChallenging Big\u2010Bench Tasks and Whether Chain\u2010of\u2010Thought Can Solve Them \u201d preprint arXiv:2210.09261 (2022).","DOI":"10.18653\/v1\/2023.findings-acl.824"},{"key":"e_1_2_11_50_1","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2026.3668991"},{"key":"e_1_2_11_51_1","doi-asserted-by":"publisher","DOI":"10.1049\/cit2.70004"},{"key":"e_1_2_11_52_1","unstructured":"A.Grattafiori A.Dubey A.Jauhri et\u00a0al. \u201cThe LLaMA 3 Herd of Models \u201d preprint arXiv:2407.21783 (2024)."},{"key":"e_1_2_11_53_1","unstructured":"X.Liu Y.Zheng Z.Du et\u00a0al. \u201cGPT Understands Too \u201d preprint arXiv:2103.10385 (2021)."},{"key":"e_1_2_11_54_1","doi-asserted-by":"publisher","DOI":"10.52202\/068431-0142"},{"key":"e_1_2_11_55_1","unstructured":"K.B\u00fcy\u00fckaky\u00fcz \u201cOLoRA: Orthonormal Low\u2010Rank Adaptation of Large Language Models \u201d preprint arXiv:2406.01775 (2024)."},{"key":"e_1_2_11_56_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.xinn.2025.101084"},{"key":"e_1_2_11_57_1","doi-asserted-by":"publisher","DOI":"10.1088\/1361\u20106501\/add6c8"}],"container-title":["CAAI Transactions on Intelligence Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/ietresearch.onlinelibrary.wiley.com\/doi\/pdf\/10.1049\/cit2.70144","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/ietresearch.onlinelibrary.wiley.com\/doi\/full-xml\/10.1049\/cit2.70144","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/ietresearch.onlinelibrary.wiley.com\/doi\/pdf\/10.1049\/cit2.70144","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,13]],"date-time":"2026-07-13T07:25:19Z","timestamp":1783927519000},"score":1,"resource":{"primary":{"URL":"https:\/\/ietresearch.onlinelibrary.wiley.com\/doi\/10.1049\/cit2.70144"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,5,15]]},"references-count":56,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2026,6]]}},"alternative-id":["10.1049\/cit2.70144"],"URL":"https:\/\/doi.org\/10.1049\/cit2.70144","archive":["Portico"],"relation":{},"ISSN":["2468-6557","2468-2322"],"issn-type":[{"value":"2468-6557","type":"print"},{"value":"2468-2322","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,5,15]]},"assertion":[{"value":"2025-11-12","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-03-23","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-05-15","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}