{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,29]],"date-time":"2026-09-29T19:16:12Z","timestamp":1790709372328,"version":"4.1.0"},"reference-count":28,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2022,2,14]],"date-time":"2022-02-14T00:00:00Z","timestamp":1644796800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001602","name":"Science Foundation Ireland","doi-asserted-by":"publisher","award":["18\/CRT\/6224"],"award-info":[{"award-number":["18\/CRT\/6224"]}],"id":[{"id":"10.13039\/501100001602","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001602","name":"Science Foundation Ireland","doi-asserted-by":"publisher","award":["13\/RC\/2077"],"award-info":[{"award-number":["13\/RC\/2077"]}],"id":[{"id":"10.13039\/501100001602","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001602","name":"Science Foundation Ireland","doi-asserted-by":"publisher","award":["13\/RC\/2106"],"award-info":[{"award-number":["13\/RC\/2106"]}],"id":[{"id":"10.13039\/501100001602","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Marie Sk\u0142odowska-Curie","award":["713567"],"award-info":[{"award-number":["713567"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>Neural machine translation (NMT) systems have greatly improved the quality available from machine translation (MT) compared to statistical machine translation (SMT) systems. However, these state-of-the-art NMT models need much more computing power and data than SMT models, a requirement that is unsustainable in the long run and of very limited benefit in low-resource scenarios. To some extent, model compression\u2014more specifically state-of-the-art knowledge distillation techniques\u2014can remedy this. In this work, we investigate knowledge distillation on a simulated low-resource German-to-English translation task. We show that sequence-level knowledge distillation can be used to train small student models on knowledge distilled from large teacher models. Part of this work examines the influence of hyperparameter tuning on model performance when lowering the number of Transformer heads or limiting the vocabulary size. Interestingly, the accuracy of these student models is higher than that of the teachers in some cases even though the student model training times are shorter in some cases. In a novel contribution, we demonstrate for a specific MT service provider that in the post-deployment phase, distilled student models can reduce emissions, as well as cost purely in monetary terms, by almost 50%.<\/jats:p>","DOI":"10.3390\/info13020088","type":"journal-article","created":{"date-parts":[[2022,2,14]],"date-time":"2022-02-14T20:26:42Z","timestamp":1644870402000},"page":"88","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":17,"title":["Knowledge Distillation: A Method for Making Neural Machine Translation More Efficient"],"prefix":"10.3390","volume":"13","author":[{"given":"Wandri","family":"Jooste","sequence":"first","affiliation":[{"name":"ML-Labs, ADAPT Centre, Dublin City University, D09 Y074 Dublin, Ireland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1680-0099","authenticated-orcid":false,"given":"Rejwanul","family":"Haque","sequence":"additional","affiliation":[{"name":"School of Computing, National College of Ireland, D01 Y300 Dublin, Ireland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5736-5930","authenticated-orcid":false,"given":"Andy","family":"Way","sequence":"additional","affiliation":[{"name":"ML-Labs, ADAPT Centre, Dublin City University, D09 Y074 Dublin, Ireland"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,2,14]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"LeCun","year":"2015","journal-title":"Nature"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Strubell, E., Ganesh, A., and McCallum, A. (2019, January 28). Energy and Policy Considerations for Deep Learning in NLP. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy.","DOI":"10.18653\/v1\/P19-1355"},{"key":"ref_3","unstructured":"Yusuf, M., Surana, P., Gupta, G., and Ramesh, K. (2021). Curb Your Carbon Emissions: Benchmarking Carbon Emissions in Machine Translation. arXiv."},{"key":"ref_4","unstructured":"Thompson, N.C., Greenewald, K., Lee, K., and Manso, G.F. (2021, December 02). Deep Learning\u2019s Diminishing Returns. Available online: https:\/\/spectrum.ieee.org\/deep-learning-computational-cost."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Fei-Fei, L. (2009, January 20\u201325). Imagenet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Bucilu\u01ce, C., Caruana, R., and Niculescu-Mizil, A. (2006, January 20\u201323). Model compression. Proceedings of the Twelfth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Philadelphia, PA, USA.","DOI":"10.1145\/1150402.1150464"},{"key":"ref_7","unstructured":"Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the Knowledge in a Neural Network. arXiv."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Kim, Y., and Rush, A.M. (2016, January 1\u20135). Sequence-Level Knowledge Distillation. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, TX, USA.","DOI":"10.18653\/v1\/D16-1139"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Currey, A., Mathur, P., and Dinu, G. (2020, January 3). Distilling Multiple Domains for Neural Machine Translation. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online.","DOI":"10.18653\/v1\/2020.emnlp-main.364"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Gordon, M., and Duh, K. (2020, January 10). Distill, Adapt, Distill: Training Small, In-Domain Models for Neural Machine Translation. Proceedings of the Fourth Workshop on Neural Generation and Translation, Online.","DOI":"10.18653\/v1\/2020.ngt-1.12"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Zhang, H., Qiu, S., and Wu, S. (2021, January 15). Dual knowledge distillation for bidirectional neural machine translation. Proceedings of the 2021 International Joint Conference on Neural Networks (IJCNN), Shenzhen, China.","DOI":"10.1109\/IJCNN52387.2021.9534396"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Wang, F., Yan, J., Meng, F., and Zhou, J. (2021). Selective Knowledge Distillation for Neural Machine Translation. arXiv.","DOI":"10.18653\/v1\/2021.acl-long.504"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Passban, P., Wu, Y., Rezagholizadeh, M., and Liu, Q. (2021, January 2\u20139). ALP-KD: Attention-Based Layer Projection for Knowledge Distillation. Proceedings of the the Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI-21), Online.","DOI":"10.1609\/aaai.v35i15.17610"},{"key":"ref_14","unstructured":"Dakwale, P. (2020). Strategies for Effective Utilization of Training Data for Machine Translation. [Ph.D. Thesis, University of Amsterdam]."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Mikolov, T., Karafi\u00e1t, M., Burget, L., Cernock\u1ef3, J., and Khudanpur, S. (2010, January 26\u201330). Recurrent neural network based language model. Proceedings of the 11th Annual Conference of the International Speech Communication Association, Makuhari, Chiba, Japan.","DOI":"10.21437\/Interspeech.2010-343"},{"key":"ref_16","first-page":"1","article-title":"Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning","volume":"248","author":"Henderson","year":"2020","journal-title":"J. Mach. Learn. Res."},{"key":"ref_17","unstructured":"Tiedemann, J. (2012, January 23\u201325). Parallel Data, Tools and Interfaces in OPUS. Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC\u201912), Istanbul, Turkey."},{"key":"ref_18","unstructured":"Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., and Zens, R. (2007, January 25\u201327). Moses: Open Source Toolkit for Statistical Machine Translation. Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, Prague, Czech Republic."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Sennrich, R., Haddow, B., and Birch, A. (2016, January 7\u201312). Neural Machine Translation of Rare Words with Subword Units. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany.","DOI":"10.18653\/v1\/P16-1162"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. (2002, January 7\u201312). BLEU: A Method for Automatic Evaluation of Machine Translation. Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, Philadelphia, PA, USA.","DOI":"10.3115\/1073083.1073135"},{"key":"ref_21","unstructured":"Snover, M., Dorr, B., Schwartz, R., Micciulla, L., and Makhoul, J. (2006, January 8\u201312). A study of translation edit rate with targeted human annotation. Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers, Cambridge, MA, USA."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Popovi\u0107, M. (2015, January 17\u201318). chrF: Character n-gram F-score for automatic MT evaluation. Proceedings of the Tenth Workshop on Statistical Machine Translation, Lisbon, Portugal.","DOI":"10.18653\/v1\/W15-3049"},{"key":"ref_23","unstructured":"Clark, J.H., Dyer, C., Lavie, A., and Smith, N.A. (, January 19\u201324). Better hypothesis testing for statistical machine translation: Controlling for optimizer instability. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Portland, OG, USA."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Junczys-Dowmunt, M., Grundkiewicz, R., Dwojak, T., Hoang, H., Heafield, K., Neckermann, T., Seide, F., Germann, U., Aji, A.F., and Bogoychev, N. (2018, January 15\u201320). Marian: Fast Neural Machine Translation in C++. Proceedings of the ACL 2018, System Demonstrations, Melbourne, Australia.","DOI":"10.18653\/v1\/P18-4020"},{"key":"ref_25","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017, January 4\u20139). Attention is All you Need. Proceedings of the Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, Long Beach, CA, USA."},{"key":"ref_26","unstructured":"Gordon, M.A., and Duh, K. (2019). Explaining Sequence-Level Knowledge Distillation as Data-Augmentation for Neural Machine Translation. arXiv."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"54","DOI":"10.1145\/3381831","article-title":"Green AI","volume":"63","author":"Schwartz","year":"2019","journal-title":"Commun. ACM"},{"key":"ref_28","unstructured":"Lacoste, A., Luccioni, A., Schmidt, V., and Dandres, T. (2019). Quantifying the Carbon Emissions of Machine Learning. arXiv."}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/13\/2\/88\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:19:10Z","timestamp":1760134750000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/13\/2\/88"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,2,14]]},"references-count":28,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2022,2]]}},"alternative-id":["info13020088"],"URL":"https:\/\/doi.org\/10.3390\/info13020088","relation":{},"ISSN":["2078-2489"],"issn-type":[{"value":"2078-2489","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,2,14]]}}}