{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,3]],"date-time":"2026-04-03T03:42:53Z","timestamp":1775187773936,"version":"3.50.1"},"reference-count":30,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2023,1,28]],"date-time":"2023-01-28T00:00:00Z","timestamp":1674864000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Ministry of Science and Technology, Taiwan","award":["MOST 111-2221-E-006-001"],"award-info":[{"award-number":["MOST 111-2221-E-006-001"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Recent advances with large-scale pre-trained language models (e.g., BERT) have brought significant potential to natural language processing. However, the large model size hinders their use in IoT and edge devices. Several studies have utilized task-specific knowledge distillation to compress the pre-trained language models. However, to reduce the number of layers in a large model, a sound strategy for distilling knowledge to a student model with fewer layers than the teacher model is lacking. In this work, we present Layer-wise Adaptive Distillation (LAD), a task-specific distillation framework that can be used to reduce the model size of BERT. We design an iterative aggregation mechanism with multiple gate blocks in LAD to adaptively distill layer-wise internal knowledge from the teacher model to the student model. The proposed method enables an effective knowledge transfer process for a student model, without skipping any teacher layers. The experimental results show that both the six-layer and four-layer LAD student models outperform previous task-specific distillation approaches during GLUE tasks.<\/jats:p>","DOI":"10.3390\/s23031483","type":"journal-article","created":{"date-parts":[[2023,1,30]],"date-time":"2023-01-30T02:28:34Z","timestamp":1675045714000},"page":"1483","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":18,"title":["LAD: Layer-Wise Adaptive Distillation for BERT Model Compression"],"prefix":"10.3390","volume":"23","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4347-0232","authenticated-orcid":false,"given":"Ying-Jia","family":"Lin","sequence":"first","affiliation":[{"name":"Department of Computer Science and Information Engineering, National Cheng Kung University, Tainan 70101, Taiwan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Kuan-Yu","family":"Chen","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Information Engineering, National Cheng Kung University, Tainan 70101, Taiwan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8890-8544","authenticated-orcid":false,"given":"Hung-Yu","family":"Kao","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Information Engineering, National Cheng Kung University, Tainan 70101, Taiwan"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,1,28]]},"reference":[{"key":"ref_1","unstructured":"Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019). Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Association for Computational Linguistics."},{"key":"ref_2","unstructured":"Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv."},{"key":"ref_3","first-page":"5753","article-title":"Xlnet: Generalized autoregressive pretraining for language understanding","volume":"Volume 32","author":"Wallach","year":"2019","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"ref_4","unstructured":"Clark, K., Luong, M.T., Le, Q.V., and Manning, C.D. (2022, November 01). ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators. Available online: https:\/\/openreview.net\/forum?id=r1xMH1BtvB."},{"key":"ref_5","unstructured":"Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. (2022, November 01). Improving Language Understanding by Generative Pre-Training. Available online: https:\/\/cdn.openai.com\/research-covers\/language-unsupervised\/language_understanding_paper.pdf."},{"key":"ref_6","first-page":"1","article-title":"Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer","volume":"21","author":"Raffel","year":"2020","journal-title":"J. Mach. Learn. Res."},{"key":"ref_7","first-page":"1877","article-title":"Language Models are Few-Shot Learners","volume":"Volume 33","author":"Larochelle","year":"2020","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"ref_8","unstructured":"Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the knowledge in a neural network. arXiv."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Bucilu\u01ce, C., Caruana, R., and Niculescu-Mizil, A. (2006, January 20\u201323). Model compression. Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Philadelphia, PA, USA.","DOI":"10.1145\/1150402.1150464"},{"key":"ref_10","unstructured":"Sanh, V., Debut, L., Chaumond, J., and Wolf, T. (2019). DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. arXiv."},{"key":"ref_11","unstructured":"Jiao, X., Yin, Y., Shang, L., Jiang, X., Chen, X., Li, L., Wang, F., and Liu, Q. (2020). Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2020, Association for Computational Linguistics."},{"key":"ref_12","unstructured":"Sun, Z., Yu, H., Song, X., Liu, R., Yang, Y., and Zhou, D. (2020). Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics."},{"key":"ref_13","first-page":"5776","article-title":"MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers","volume":"Volume 33","author":"Larochelle","year":"2020","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"ref_14","unstructured":"Sun, S., Cheng, Y., Gan, Z., and Liu, J. (2019). Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Association for Computational Linguistics."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Passban, P., Wu, Y., Rezagholizadeh, M., and Liu, Q. (2021, January 2\u20139). ALP-KD: Attention-Based Layer Projection for Knowledge Distillation. Proceedings of the AAAI Conference on Artificial Intelligence, Virtual.","DOI":"10.1609\/aaai.v35i15.17610"},{"key":"ref_16","unstructured":"Tenney, I., Das, D., and Pavlick, E. (2019). Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics."},{"key":"ref_17","first-page":"2377","article-title":"Training Very Deep Networks","volume":"Volume 28","author":"Cortes","year":"2015","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"ref_18","unstructured":"Li, J., Liu, X., Zhao, H., Xu, R., Yang, M., and Jin, Y. (2020). Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics."},{"key":"ref_19","unstructured":"Ba, J.L., Kiros, J.R., and Hinton, G.E. (2016). Layer normalization. arXiv."},{"key":"ref_20","unstructured":"Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S.R. (2022, November 01). GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. Available online: https:\/\/openreview.net\/forum?id=rJ4km2R5t7."},{"key":"ref_21","unstructured":"Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C.D., Ng, A., and Potts, C. (2013, January 18\u201321). Recursive deep models for semantic compositionality over a sentiment treebank. Proceedings of the EMNLP, Seattle, MA, USA."},{"key":"ref_22","unstructured":"Dolan, W.B., and Brockett, C. Automatically Constructing a Corpus of Sentential Paraphrases. Proceedings of the International Workshop on Paraphrasing, Available online: https:\/\/aclanthology.org\/I05-5002."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Dagan, I., Glickman, O., and Magnini, B. (2005). Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment, Springer.","DOI":"10.1007\/11736790_9"},{"key":"ref_24","unstructured":"Bar Haim, R., Dagan, I., Dolan, B., Ferro, L., Giampiccolo, D., Magnini, B., and Szpektor, I. The Second PASCAL Recognising Textual Entailment Challenge. Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, Available online: https:\/\/citeseerx.ist.psu.edu\/document?repid=rep1&type=pdf&doi=33f25fae10da978fad3f48eb6bded2f733b28e92."},{"key":"ref_25","unstructured":"Giampiccolo, D., Magnini, B., Dagan, I., and Dolan, B. (2007). Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, Association for Computational Linguistics."},{"key":"ref_26","unstructured":"Bentivogli, L., Dagan, I., Dang, H.T., Giampiccolo, D., and Magnini, B. (2022, November 01). The Fifth PASCAL Recognizing Textual Entailment Challenge, Text Analysis Conference (TAC), Available online: https:\/\/tac.nist.gov\/\/publications\/2009\/additional.papers\/RTE5_overview.proceedings.pdf."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Xu, C., Zhou, W., Ge, T., Wei, F., and Zhou, M. (2020, January 16\u201320). BERT-of-Theseus: Compressing BERT by Progressive Module Replacing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online.","DOI":"10.18653\/v1\/2020.emnlp-main.633"},{"key":"ref_28","first-page":"5998","article-title":"Attention is All you Need","volume":"Volume 30","author":"Guyon","year":"2017","journal-title":"Proceedings of the Advances in Neural Information Processing Systems"},{"key":"ref_29","unstructured":"Glorot, X., and Bengio, Y. (2010, January 13\u201315). Understanding the difficulty of training deep feedforward neural networks. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, Sardinia, Italy."},{"key":"ref_30","unstructured":"Loshchilov, I., and Hutter, F. (2022, November 01). Decoupled Weight Decay Regularization. Available online: https:\/\/openreview.net\/forum?id=Bkg6RiCqY7."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/3\/1483\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T18:18:39Z","timestamp":1760120319000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/23\/3\/1483"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,1,28]]},"references-count":30,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2023,2]]}},"alternative-id":["s23031483"],"URL":"https:\/\/doi.org\/10.3390\/s23031483","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,1,28]]}}}