{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,24]],"date-time":"2026-05-24T10:05:06Z","timestamp":1779617106344,"version":"3.53.1"},"reference-count":36,"publisher":"SAGE Publications","issue":"1-2","license":[{"start":{"date-parts":[[2026,3,11]],"date-time":"2026-03-11T00:00:00Z","timestamp":1773187200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/journals.sagepub.com\/page\/policies\/text-and-data-mining-license"}],"content-domain":{"domain":["journals.sagepub.com"],"crossmark-restriction":true},"short-container-title":["Web Intelligence"],"published-print":{"date-parts":[[2026,5]]},"abstract":"<jats:p>Pretrained transformer models have demonstrated excellent performance on complex tasks. To improve their inference efficiency, recent studies have introduced the multi-exit mechanism, which enables early exiting through multiple intermediate classifiers. However, the deep architectures of pretrained transformers cause severe gradient conflicts during multi-exit fine-tuning, leading to degraded shallow-exit accuracy and reduced early-exit efficiency. To address this issue, we propose Separate Reverse, a multi-exit training strategy specifically designed for pretrained transformer models. The method iteratively integrates reverse iterative optimization and hierarchical knowledge distillation from deeper to shallower exits, maintaining pretrained parameter integrity, enhances the representation capacity of shallow exits, and coordinates gradient updates across exits to achieve a balanced optimization between shallow and deep classifiers. Experiments on multiple GLUE benchmark datasets using BERT demonstrate that our method significantly improves shallow-exit accuracy, maintains main-exit performance, and accelerates inference for simple samples by a large margin.<\/jats:p>","DOI":"10.1177\/24056456261426603","type":"journal-article","created":{"date-parts":[[2026,3,11]],"date-time":"2026-03-11T18:37:24Z","timestamp":1773254244000},"page":"3-19","update-policy":"https:\/\/doi.org\/10.1177\/sage-journals-update-policy","source":"Crossref","is-referenced-by-count":0,"title":["Separate Reverse: A Gradient-Conflict-Free Training Framework for Multi-Exit Transformers"],"prefix":"10.1177","volume":"24","author":[{"given":"Lisha","family":"Gao","sequence":"first","affiliation":[{"name":"Nanjing Power Supply Branch, State Grid Jiangsu Electric Power Co. Ltd, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Shuo","family":"Han","sequence":"additional","affiliation":[{"name":"Nanjing Power Supply Branch, State Grid Jiangsu Electric Power Co. Ltd, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yue","family":"Wu","sequence":"additional","affiliation":[{"name":"Nanjing Power Supply Branch, State Grid Jiangsu Electric Power Co. Ltd, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hanlin","family":"Guan","sequence":"additional","affiliation":[{"name":"Nanjing Power Supply Branch, State Grid Jiangsu Electric Power Co. Ltd, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Aijia","family":"Zhong","sequence":"additional","affiliation":[{"name":"School of Computer Science and Engineering, Southeast University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Weilong","family":"Xin","sequence":"additional","affiliation":[{"name":"School of Computer Science and Engineering, Southeast University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jinghui","family":"Zhang","sequence":"additional","affiliation":[{"name":"School of Computer Science and Engineering, Southeast University, Nanjing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"179","published-online":{"date-parts":[[2026,3,11]]},"reference":[{"key":"e_1_3_2_2_1","unstructured":"Aghajanyan A. Yu L. Conneau A. Hsu W.-N. Hambardzumyan K. Zhang S. Roller S. Goyal N. Levy O. Zettlemoyer L. (2023). Scaling laws for generative mixed-modal language models. In Proceedings of the 40th international conference on machine learning (ICML'23) (Vol. 202 pp. 265\u2013279). JMLR.org."},{"key":"e_1_3_2_3_1","doi-asserted-by":"crossref","unstructured":"Bajpai D. J. Hanawal M. K. (2024). CeeBERT: Cross-domain inference in early exit BERT. arXiv preprint arXiv:2405.15039.","DOI":"10.18653\/v1\/2024.findings-acl.101"},{"key":"e_1_3_2_4_1","doi-asserted-by":"publisher","DOI":"10.1109\/MCI.2014.2307227"},{"key":"e_1_3_2_5_1","doi-asserted-by":"publisher","DOI":"10.2172\/2340760"},{"key":"e_1_3_2_6_1","unstructured":"Chen Y. Pan X. Li Y. Ding B. Zhou J. (2023). EE-LLM: Large-scale training and inference of early-exit large language models with 3D parallelism. arXiv preprint arXiv:2312.04916."},{"key":"e_1_3_2_7_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2020.113193"},{"key":"e_1_3_2_8_1","unstructured":"Devlin J. Chang M.-W. Lee K. Toutanova K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American Chapter of the Association for Computational Linguistics: Human language technologies (NAACL-HLT) (Vol. 1 pp. 4171\u20134186). Association for Computational Linguistics."},{"key":"e_1_3_2_9_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neucom.2023.126690"},{"key":"e_1_3_2_10_1","unstructured":"Geng S. Gao P. Fu Z. Zhang Y. (2021). RomeBERT: Robust training of multi-exit BERT. arXiv preprint arXiv:2101.09755."},{"key":"e_1_3_2_11_1","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-021-01453-z"},{"key":"e_1_3_2_12_1","first-page":"9782","article-title":"DynabERT: Dynamic BERT with adaptive width and depth","volume":"33","author":"Hou L.","year":"2020","unstructured":"Hou L., Huang Z., Shang L., Jiang X., Chen X., Liu Q. (2020). DynabERT: Dynamic BERT with adaptive width and depth. Advances in Neural Information Processing Systems, 33, 9782\u20139793.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_3_2_13_1","unstructured":"Huang G. Chen D. Li T. Wu F. Van Der Maaten L. Weinberger K. Q. (2017). Multi-scale dense networks for resource efficient image classification. arXiv preprint arXiv:1703.09844."},{"key":"e_1_3_2_14_1","doi-asserted-by":"crossref","unstructured":"Ji Y. Wang J. Li J. Chen Q. Chen W. Zhang M. (2023). Early exit with disentangled representation and equiangular tight frame. In Findings of the Association for Computational Linguistics: ACL 2023 (pp. 14128\u201314142). Association for Computational Linguistics.","DOI":"10.18653\/v1\/2023.findings-acl.889"},{"key":"e_1_3_2_15_1","doi-asserted-by":"crossref","unstructured":"Kurtic E. Campos D. F. Nguyen T. Frantar E. Kurtz M. Fineran B. Goin M. Alistarh D. (2022). The optimal BERT surgeon: Scalable and accurate second-order pruning for large language models. arXiv preprint arXiv:2203.07259.","DOI":"10.18653\/v1\/2022.emnlp-main.279"},{"key":"e_1_3_2_16_1","doi-asserted-by":"crossref","unstructured":"Laskaridis S. Kouris A. Lane N. D. (2021). Adaptive inference through early-exit networks: Design challenges and directions. In Proceedings of the 5th International workshop on embedded and mobile deep learning (MobiSys \u201921) (pp. 1\u20136). ACM.","DOI":"10.1145\/3469116.3470012"},{"key":"e_1_3_2_17_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.engappai.2023.106035"},{"key":"e_1_3_2_18_1","unstructured":"Liao Z. Couillet R. Mahoney M. W. (2020). Sparse quantized spectral clustering. arXiv preprint arXiv:2010.01376."},{"key":"e_1_3_2_19_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3587464","article-title":"OdeBERT: One-stage deep-supervised early-exiting BERT for fast inference in user intent classification","volume":"22","author":"Liu Y.","year":"2023","unstructured":"Liu Y., Hao T., Liu H., Mu Y., Weng H., Wang F. L. (2023). OdeBERT: One-stage deep-supervised early-exiting BERT for fast inference in user intent classification. ACM Transactions on Asian and Low-Resource Language Information Processing, 22, 1\u201318.","journal-title":"ACM Transactions on Asian and Low-Resource Language Information Processing"},{"key":"e_1_3_2_20_1","doi-asserted-by":"crossref","unstructured":"Liu C. Tao C. Feng J. Zhao D. (2022). Multi-granularity structural knowledge distillation for language model compression. In Proceedings of the 60th annual meeting of the Association for Computational Linguistics (Volume 1: Long papers) (pp. 1001\u20131011). Association for Computational Linguistics.","DOI":"10.18653\/v1\/2022.acl-long.71"},{"key":"e_1_3_2_21_1","doi-asserted-by":"crossref","unstructured":"Liu W. Zhou P. Zhao Z. Wang Z. Deng H. Ju Q. (2020). FastBERT: A self-distilling BERT with adaptive inference time. arXiv preprint arXiv:2004.02178.","DOI":"10.18653\/v1\/2020.acl-main.537"},{"key":"e_1_3_2_22_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.acha.2021.12.009"},{"key":"e_1_3_2_23_1","article-title":"Are sixteen heads really better than one?","author":"Michel P.","year":"2019","unstructured":"Michel P., Levy O., Neubig G. (2019). Are sixteen heads really better than one? In Proceedings of the 33rd international conference on neural information processing systems (pp. 14037\u201314047). Curran Associates Inc.","journal-title":"Proceedings of the 33rd international conference on neural information processing systems"},{"key":"e_1_3_2_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/3698767"},{"key":"e_1_3_2_25_1","doi-asserted-by":"crossref","unstructured":"Rotem D. Hassid M. Mamou J. Schwartz R. (2023). Finding the SWEET spot: Analysis and improvement of adaptive inference in low resource settings. In Proceedings of the 61st annual meeting of the association for computational linguistics (Vol. 1: Long Papers pp. 14836\u201314851). Association for Computational Linguistics.","DOI":"10.18653\/v1\/2023.acl-long.829"},{"key":"e_1_3_2_26_1","unstructured":"Schuster T. Fisch A. Gupta J. Dehghani M. Bahri D. Tran V. Q. Tay Y. Metzler D. (2022). Confident adaptive language modeling. arXiv preprint arXiv:2207.07061."},{"key":"e_1_3_2_27_1","doi-asserted-by":"crossref","unstructured":"Tang S. Wang Y. Kong Z. Zhang T. Li Y. Ding C. Wang Y. Liang Y. Xu D. (2023). You need multiple exiting: Dynamic early exiting for accelerating unified vision language model. In Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition (pp. 10781\u201310791). IEEE.","DOI":"10.1109\/CVPR52729.2023.01038"},{"key":"e_1_3_2_28_1","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00577"},{"key":"e_1_3_2_29_1","doi-asserted-by":"crossref","unstructured":"Wang A. Singh A. Michael J. Hill F. Levy O. Bowman S. R. (2018). GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP (pp. 353\u2013355). Association for Computational Linguistics.","DOI":"10.18653\/v1\/W18-5446"},{"key":"e_1_3_2_30_1","doi-asserted-by":"crossref","unstructured":"Xin J. Tang R. Lee J. Yu Y. Lin J. (2020). DeeBERT: Dynamic early exiting for accelerating BERT inference. arXiv preprint arXiv:2004.12993.","DOI":"10.18653\/v1\/2020.acl-main.204"},{"key":"e_1_3_2_31_1","doi-asserted-by":"crossref","unstructured":"Xin J. Tang R. Yu Y. Lin J. (2021). BERxiT: Early exiting for BERT with better fine-tuning and extension to regression. In Proceedings of the 16th conference of the European Chapter of the Association for Computational Linguistics (EACL 2021) (pp. 91\u2013104). Association for Computational Linguistics.","DOI":"10.18653\/v1\/2021.eacl-main.8"},{"key":"e_1_3_2_32_1","doi-asserted-by":"crossref","unstructured":"Xin J. Tang R. Yu Y. Lin J. J. (2021). BERxiT: Early exiting for BERT with better fine-tuning and extension to regression. In Proceedings of the 16th conference of the European Chapter of the Association for Computational Linguistics (EACL) (pp. 91\u2013104). Association for Computational Linguistics .","DOI":"10.18653\/v1\/2021.eacl-main.8"},{"key":"e_1_3_2_33_1","doi-asserted-by":"crossref","unstructured":"Xu J. Pan J. Zhou Y. Chen S. Li J. Lian Y. Wu J. Dai G. (2025). Specee: Accelerating large language model inference with speculative early exiting. In Proceedings of the 52nd annual international symposium on computer architecture (ISCA) (pp. 467\u2013481).","DOI":"10.1145\/3695053.3730996"},{"key":"e_1_3_2_34_1","unstructured":"Xu D. Yin W. Jin X. Zhang Y. Wei S. Xu M. Liu X. (2023). LLMCAD: Fast and scalable on-device large language model inference. arXiv preprint arXiv:2309.04255."},{"key":"e_1_3_2_35_1","doi-asserted-by":"crossref","unstructured":"Yi R. Guo L. Wei S. Zhou A. Wang S. Xu M. (2025). EdgeMoE: Empowering sparse large language models on mobile devices. arXiv preprint arXiv:2308.14352.","DOI":"10.1109\/TMC.2025.3546466"},{"key":"e_1_3_2_36_1","unstructured":"Yu H. Li H. Hua G. Huang G. Shi H. (2022). Boosted dynamic neural networks. arXiv preprint arXiv:2211.16726."},{"key":"e_1_3_2_37_1","doi-asserted-by":"crossref","unstructured":"Zhu W. Wang P. Ni Y. Xie G. Wang X. (2023). BADGE: Speeding up BERT inference after deployment via block-wise bypasses and divergence-based early exiting. In Proceedings of the 61st annual meeting of the Association for Computational Linguistics (Volume 5: Industry track) (pp. 500\u2013509). Association for Computational Linguistics.","DOI":"10.18653\/v1\/2023.acl-industry.48"}],"container-title":["Web Intelligence"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/24056456261426603","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/full-xml\/10.1177\/24056456261426603","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/journals.sagepub.com\/doi\/pdf\/10.1177\/24056456261426603","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,5,24]],"date-time":"2026-05-24T09:46:16Z","timestamp":1779615976000},"score":1,"resource":{"primary":{"URL":"https:\/\/journals.sagepub.com\/doi\/10.1177\/24056456261426603"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,3,11]]},"references-count":36,"journal-issue":{"issue":"1-2","published-print":{"date-parts":[[2026,5]]}},"alternative-id":["10.1177\/24056456261426603"],"URL":"https:\/\/doi.org\/10.1177\/24056456261426603","relation":{},"ISSN":["2405-6456","2405-6464"],"issn-type":[{"value":"2405-6456","type":"print"},{"value":"2405-6464","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,3,11]]}}}