{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,3]],"date-time":"2026-09-03T03:48:20Z","timestamp":1788407300630,"version":"build-2803163510"},"reference-count":41,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2022,4,23]],"date-time":"2022-04-23T00:00:00Z","timestamp":1650672000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100003130","name":"Research Foundation - Flanders","doi-asserted-by":"publisher","award":["77410"],"award-info":[{"award-number":["77410"]}],"id":[{"id":"10.13039\/501100003130","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Horizon 2020 Research and Innovation Programme","award":["101017255"],"award-info":[{"award-number":["101017255"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>We consider neural sign language translation: machine translation from signed to written languages using encoder\u2013decoder neural networks. Translating sign language videos to written language text is especially complex because of the difference in modality between source and target language and, consequently, the required video processing. At the same time, sign languages are low-resource languages, their datasets dwarfed by those available for written languages. Recent advances in written language processing and success stories of transfer learning raise the question of how pretrained written language models can be leveraged to improve sign language translation. We apply the Frozen Pretrained Transformer (FPT) technique to initialize the encoder, decoder, or both, of a sign language translation model with parts of a pretrained written language model. We observe that the attention patterns transfer in zero-shot to the different modality and, in some experiments, we obtain higher scores (from 18.85 to 21.39 BLEU-4). Especially when gloss annotations are unavailable, FPTs can increase performance on unseen data. However, current models appear to be limited primarily by data quality and only then by data quantity, limiting potential gains with FPTs. Therefore, in further research, we will focus on improving the representations used as inputs to translation models.<\/jats:p>","DOI":"10.3390\/info13050220","type":"journal-article","created":{"date-parts":[[2022,4,23]],"date-time":"2022-04-23T08:14:06Z","timestamp":1650701646000},"page":"220","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":13,"title":["Leveraging Frozen Pretrained Written Language Models for Neural Sign Language Translation"],"prefix":"10.3390","volume":"13","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1103-2441","authenticated-orcid":false,"given":"Mathieu","family":"De Coster","sequence":"first","affiliation":[{"name":"IDLab-AIRO, Ghent University\u2014IMEC, Technologiepark-Zwijnaarde 126, 9052 Ghent, Belgium"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9373-1210","authenticated-orcid":false,"given":"Joni","family":"Dambre","sequence":"additional","affiliation":[{"name":"IDLab-AIRO, Ghent University\u2014IMEC, Technologiepark-Zwijnaarde 126, 9052 Ghent, Belgium"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,4,23]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Camgoz, N.C., Hadfield, S., Koller, O., Ney, H., and Bowden, R. (2018, January 18\u201322). Neural sign language translation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.","DOI":"10.1109\/CVPR.2018.00812"},{"key":"ref_2","unstructured":"Espl\u00e0-Gomis, M., Forcada, M., Ram\u00edrez-S\u00e1nchez, G., and Hoang, H.T. (2019, January 19\u201323). ParaCrawl: Web-scale parallel corpora for the languages of the EU. Proceedings of the MTSummit, Dublin, Ireland."},{"key":"ref_3","unstructured":"Moryossef, A., Yin, K., Neubig, G., and Goldberg, Y. (2021, January 16\u201320). Data Augmentation for Sign Language Gloss Translation. Proceedings of the 1st International Workshop on Automatic Translation for Signed and Spoken Languages (AT4SSL), Online."},{"key":"ref_4","unstructured":"Zhang, X., and Duh, K. (2021, January 16\u201320). Approaching Sign Language Gloss Translation as a Low-Resource Machine Translation Task. Proceedings of the 1st International Workshop on Automatic Translation for Signed and Spoken Languages (AT4SSL), Online."},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Zhou, H., Zhou, W., Qi, W., Pu, J., and Li, H. (2021, January 19\u201325). Improving Sign Language Translation with Monolingual Data by Sign Back-Translation. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.00137"},{"key":"ref_6","unstructured":"De Coster, M., D\u2019Oosterlinck, K., Pizurica, M., Rabaey, P., Verlinden, S., Van Herreweghe, M., and Dambre, J. (2021, January 20). Frozen Pretrained Transformers for Neural Sign Language Translation. Proceedings of the 1st International Workshop on Automatic Translation for Signed and Spoken Languages (AT4SSL), Virtual."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Zoph, B., Yuret, D., May, J., and Knight, K. (2016, January 1\u20134). Transfer Learning for Low-Resource Neural Machine Translation. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, TX, USA.","DOI":"10.18653\/v1\/D16-1163"},{"key":"ref_8","unstructured":"Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019, January 2\u20137). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers); Association for Computational Linguistics, Minneapolis, MN, USA."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"264","DOI":"10.1162\/tacl_a_00313","article-title":"Leveraging pre-trained checkpoints for sequence generation tasks","volume":"8","author":"Rothe","year":"2020","journal-title":"Trans. Assoc. Comput. Linguist."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Artetxe, M., Ruder, S., and Yogatama, D. (2020, January 5\u201310). On the Cross-lingual Transferability of Monolingual Representations. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.","DOI":"10.18653\/v1\/2020.acl-main.421"},{"key":"ref_11","unstructured":"Gogoulou, E., Ekgren, A., Isbister, T., and Sahlgren, M. (2021). Cross-lingual Transfer of Monolingual Models. arXiv."},{"key":"ref_12","first-page":"200","article-title":"Multimodal few-shot learning with frozen language models","volume":"34","author":"Tsimpoukelli","year":"2021","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Lu, K., Grover, A., Abbeel, P., and Mordatch, I. (2021). Pretrained transformers as universal computation engines. arXiv.","DOI":"10.1609\/aaai.v36i7.20729"},{"key":"ref_14","unstructured":"Camgoz, N.C., Koller, O., Hadfield, S., and Bowden, R. (2020, January 14\u201319). Sign language transformers: Joint end-to-end sign language recognition and translation. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Online."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Yin, K., and Read, J. (2020, January 8\u201313). Better Sign Language Translation with STMC-Transformer. Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain.","DOI":"10.18653\/v1\/2020.coling-main.525"},{"key":"ref_16","unstructured":"Bungeroth, J., and Ney, H. (2004, January 24\u201330). Statistical sign language translation. Proceedings of the Workshop on Representation and Processing of Sign Languages, LREC, Citeseer, Lisbon, Portugal."},{"key":"ref_17","unstructured":"Morrissey, S., Way, A., Stein, D., Bungeroth, J., and Ney, H. (2007, January 10\u201314). Combining data-driven MT systems for improved sign language translation. Proceedings of the Machine Translation Summit XI, Copenhagen, Denmark."},{"key":"ref_18","unstructured":"Stein, D., Schmidt, C., and Ney, H. (2010, January 2\u20133). Sign language machine translation overkill. Proceedings of the International Workshop on Spoken Language Translation (IWSLT), Paris, France."},{"key":"ref_19","unstructured":"Forster, J., Schmidt, C., Koller, O., Bellgardt, M., and Ney, H. (2014, January 26\u201331). Extensions of the Sign Language Recognition and Translation Corpus RWTH-PHOENIX-Weather. Proceedings of the LREC, Reykjavik, Iceland."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Pfau, R., Steinbach, M., and Woll, B. (2012). Transcription. Sign Language, De Gruyter Mouton.","DOI":"10.1515\/9783110261325"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Vermeerbergen, M., Leeson, L., and Crasborn, O.A. (2007). Simultaneity in Signed Languages: Form and Function, John Benjamins Publishing.","DOI":"10.1075\/cilt.281"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"168","DOI":"10.1016\/j.langcom.2005.10.004","article-title":"Past and current trends in sign language research","volume":"26","author":"Vermeerbergen","year":"2006","journal-title":"Lang. Commun."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Orbay, A., and Akarun, L. (2020, January 16\u201320). Neural sign language translation by learning tokenization. Proceedings of the 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), Buenos Aires, Argentina.","DOI":"10.1109\/FG47880.2020.00002"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"768","DOI":"10.1109\/TMM.2021.3059098","article-title":"Spatial-temporal multi-cue network for sign language recognition and translation","volume":"24","author":"Zhou","year":"2021","journal-title":"IEEE Trans. Multimed."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"De Coster, M., Shterionov, D., Van Herreweghe, M., and Dambre, J. (2022). Machine Translation from Signed to Spoken Languages: State of the Art and Challenges. arXiv.","DOI":"10.1007\/s10209-023-00992-1"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Luong, M.T., Pham, H., and Manning, C.D. (2015, January 17\u201321). Effective Approaches to Attention-based Neural Machine Translation. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal.","DOI":"10.18653\/v1\/D15-1166"},{"key":"ref_27","first-page":"9","article-title":"Language models are unsupervised multitask learners","volume":"1","author":"Radford","year":"2019","journal-title":"OpenAI Blog"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Imamura, K., and Sumita, E. (2019, January 4). Recycling a pre-trained BERT encoder for neural machine translation. Proceedings of the 3rd Workshop on Neural Generation and Translation, Hong Kong, China.","DOI":"10.18653\/v1\/D19-5603"},{"key":"ref_29","unstructured":"Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv."},{"key":"ref_30","unstructured":"Miyazaki, T., Morita, Y., and Sano, M. (2020, January 11\u201316). Machine translation from spoken language to Sign language using pre-trained language model as encoder. Proceedings of the LREC2020 9th Workshop on the Representation and Processing of Sign Languages: Sign Language Resources in the Service of the Language Community, Technological Challenges and Application Perspectives, Marseille, France."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Szegedy, C., Ioffe, S., Vanhoucke, V., and Alemi, A.A. (2017, January 4\u20139). Inception-v4, inception-resnet and the impact of residual connections on learning. Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, San Francisco, CA, USA.","DOI":"10.1609\/aaai.v31i1.11231"},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"2306","DOI":"10.1109\/TPAMI.2019.2911077","article-title":"Weakly supervised learning with multi-stream CNN-LSTM-HMMs to discover sequential parallelism in sign language videos","volume":"42","author":"Koller","year":"2019","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Graves, A., Fern\u00e1ndez, S., Gomez, F., and Schmidhuber, J. (2006, January 25\u201329). Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. Proceedings of the 23rd International Conference on Machine Learning, Pittsburgh, PA, USA.","DOI":"10.1145\/1143844.1143891"},{"key":"ref_34","unstructured":"Kingma, D.P., and Ba, J. (2015, January 7\u20139). Adam: A Method for Stochastic Optimization. Proceedings of the 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA."},{"key":"ref_35","unstructured":"Wallach, H., Larochelle, H., Beygelzimer, A., d\u2019Alch\u00e9-Buc, F., Fox, E., and Garnett, R. (2019). PyTorch: An Imperative Style, High-Performance Deep Learning Library. Advances in Neural Information Processing Systems 32, Curran Associates, Inc."},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Kreutzer, J., Bastings, J., and Riezler, S. (2019, January 3\u20137). Joey NMT: A Minimalist NMT Toolkit for Novices. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations, Hong Kong, China.","DOI":"10.18653\/v1\/D19-3019"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. (2002, January 7\u201317). Bleu: A method for automatic evaluation of machine translation. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, PA, USA.","DOI":"10.3115\/1073083.1073135"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Lin, C.Y., and Och, F.J. (2004, January 21\u201326). Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), Barcelona, Spain.","DOI":"10.3115\/1218955.1219032"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Popovi\u0107, M. (2015, January 17\u201318). chrF: Character n-gram F-score for automatic MT evaluation. Proceedings of the Tenth Workshop on Statistical Machine Translation, Lisbon, Portugal.","DOI":"10.18653\/v1\/W15-3049"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1162\/coli_a_00367","article-title":"On the linguistic representational power of neural machine translation models","volume":"46","author":"Belinkov","year":"2020","journal-title":"Comput. Linguist."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"3411","DOI":"10.1109\/TSP.2020.2993164","article-title":"Analyzing upper bounds on mean absolute errors for deep neural network-based vector-to-vector regression","volume":"68","author":"Qi","year":"2020","journal-title":"IEEE Trans. Signal Process."}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/13\/5\/220\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:59:22Z","timestamp":1760137162000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/13\/5\/220"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,4,23]]},"references-count":41,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2022,5]]}},"alternative-id":["info13050220"],"URL":"https:\/\/doi.org\/10.3390\/info13050220","relation":{},"ISSN":["2078-2489"],"issn-type":[{"value":"2078-2489","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,4,23]]}}}