{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,27]],"date-time":"2026-05-27T17:03:15Z","timestamp":1779901395100,"version":"3.53.1"},"reference-count":21,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2022,2,14]],"date-time":"2022-02-14T00:00:00Z","timestamp":1644796800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Qinghai Provincial Science and Technology Research Program","award":["No.2021-QY-206"],"award-info":[{"award-number":["No.2021-QY-206"]}]},{"name":"Guangdong Provincial Science and Technology Research Program","award":["No.2020A050515014"],"award-info":[{"award-number":["No.2020A050515014"]}]},{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"publisher","award":["grant No. 62071201"],"award-info":[{"award-number":["grant No. 62071201"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>Massively pre-trained transformer models such as BERT have gained great success in many downstream NLP tasks. However, they are computationally expensive to fine-tune, slow for inference, and have large storage requirements. So, transfer learning with adapter modules has been introduced and has become a remarkable solution for those problems. Nevertheless, recent studies reveal that the parameters in adapters are actually still quite redundant, which could slow down inference speed when fusing multiple adapters for a specific downstream task, and thus, they can be further reduced. To address this issue, we propose three novel ways to prune the adapter modules iteratively based on the prestigious Lottery Ticket Hypothesis. Extensive experiments on the GLUE datasets show that the pruned adapters can achieve state-of-the-art results, with sizes reduced significantly while performance remains unchanged, and some pruned adapters even outperform the ones with the same size that are fine-tuned alone without pruning.<\/jats:p>","DOI":"10.3390\/a15020063","type":"journal-article","created":{"date-parts":[[2022,2,14]],"date-time":"2022-02-14T20:26:42Z","timestamp":1644870402000},"page":"63","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Pruning Adapters with Lottery Ticket"],"prefix":"10.3390","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0999-6559","authenticated-orcid":false,"given":"Jiarun","family":"Wu","sequence":"first","affiliation":[{"name":"Department of Computer Science, Jinan University, Guangzhou 510632, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5849-8268","authenticated-orcid":false,"given":"Qingliang","family":"Chen","sequence":"additional","affiliation":[{"name":"Department of Computer Science, Jinan University, Guangzhou 510632, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,2,14]]},"reference":[{"key":"ref_1","unstructured":"Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (2017). Attention is All you Need. Advances in Neural Information Processing Systems, Curran Associates, Inc."},{"key":"ref_2","unstructured":"Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv."},{"key":"ref_3","unstructured":"Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv."},{"key":"ref_4","unstructured":"Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., and Le, Q.V. (2019). Xlnet: Generalized autoregressive pretraining for language understanding. arXiv."},{"key":"ref_5","unstructured":"Kaplan, J., McCandlish, S., Henighan, T., Brown, T.B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. (2020). Scaling laws for neural language models. arXiv."},{"key":"ref_6","unstructured":"Chaudhuri, K., and Salakhutdinov, R. (2019, January 9\u201315). Parameter-Efficient Transfer Learning for NLP. Proceedings of the 36th International Conference on Machine Learning, Long Beach, CA, USA."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Pfeiffer, J., Kamath, A., R\u00fcckl\u00e9, A., Cho, K., and Gurevych, I. (2020). AdapterFusion: Non-destructive task composition for transfer learning. arXiv.","DOI":"10.18653\/v1\/2021.eacl-main.39"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"R\u00fcckl\u00e9, A., Geigle, G., Glockner, M., Beck, T., Pfeiffer, J., Reimers, N., and Gurevych, I. (2020). AdapterDrop: On the Efficiency of Adapters in Transformers. arXiv.","DOI":"10.18653\/v1\/2021.emnlp-main.626"},{"key":"ref_9","unstructured":"Frankle, J., and Carbin, M. (2019, January 6\u20139). The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks. Proceedings of the 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA. Available online: https:\/\/openreview.net\/."},{"key":"ref_10","unstructured":"Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019, January 2\u20137). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, MN, USA."},{"key":"ref_11","unstructured":"Bengio, Y., Roux, N.L., Vincent, P., Delalleau, O., and Marcotte, P. (2005, January 5\u20138). Convex Neural Networks. Proceedings of the Advances in Neural Information Processing Systems 18 Neural Information Processing Systems\u2014NIPS 2005, Vancouver, BC, Canada."},{"key":"ref_12","unstructured":"Shawe-Taylor, J., Zemel, R.S., Bartlett, P.L., Pereira, F.C.N., and Weinberger, K.Q. (2011, January 12\u201314). Algorithms for Hyper-Parameter Optimization. Proceedings of the Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems 2011, Granada, Spain."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. (2018, January 1). GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Brussels, Belgium.","DOI":"10.18653\/v1\/W18-5446"},{"key":"ref_14","unstructured":"Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., and Weinberger, K.Q. (2014, January 8\u201313). Do Deep Nets Really Need to be Deep?. Proceedings of the Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, Montreal, QC, Canada."},{"key":"ref_15","unstructured":"Hinton, G., Vinyals, O., and Dean, J. (2015, January 7\u201312). Distilling the Knowledge in a Neural Network. Proceedings of the NIPS Deep Learning and Representation Learning Workshop, Montreal, QC, Canada."},{"key":"ref_16","first-page":"598","article-title":"Optimal Brain Damage","volume":"Volume 2","author":"Touretzky","year":"1990","journal-title":"Advances in Neural Information Processing Systems"},{"key":"ref_17","unstructured":"Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., and Garnett, R. (2015). Learning both Weights and Connections for Efficient Neural Network. Advances in Neural Information Processing Systems, Curran Associates, Inc."},{"key":"ref_18","unstructured":"Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017). Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv."},{"key":"ref_19","unstructured":"Bellec, G., Kappel, D., Maass, W., and Legenstein, R.A. (May, January 30). Deep Rewiring: Training very sparse deep networks. Proceedings of the 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada. Available online: https:\/\/openreview.net\/."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Pfeiffer, J., Vuli\u0107, I., Gurevych, I., and Ruder, S. (2020, January 16\u201320). MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online.","DOI":"10.18653\/v1\/2020.emnlp-main.617"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Wang, R., Tang, D., Duan, N., Wei, Z., Huang, X., Cao, C., Jiang, D., and Zhou, M. (2020). K-adapter: Infusing knowledge into pre-trained models with adapters. arXiv.","DOI":"10.18653\/v1\/2021.findings-acl.121"}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/15\/2\/63\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:19:22Z","timestamp":1760134762000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/15\/2\/63"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,2,14]]},"references-count":21,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2022,2]]}},"alternative-id":["a15020063"],"URL":"https:\/\/doi.org\/10.3390\/a15020063","relation":{},"ISSN":["1999-4893"],"issn-type":[{"value":"1999-4893","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,2,14]]}}}