{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,12]],"date-time":"2026-02-12T12:15:58Z","timestamp":1770898558321,"version":"3.50.1"},"reference-count":40,"publisher":"MDPI AG","issue":"5","license":[{"start":{"date-parts":[[2021,5,2]],"date-time":"2021-05-02T00:00:00Z","timestamp":1619913600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100012166","name":"National Key Research and Development Program of China","doi-asserted-by":"publisher","award":["2017YFC0820700"],"award-info":[{"award-number":["2017YFC0820700"]}],"id":[{"id":"10.13039\/501100012166","id-type":"DOI","asserted-by":"publisher"}]},{"name":"National Language Commission Research Project","award":["ZDI135-96"],"award-info":[{"award-number":["ZDI135-96"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Symmetry"],"abstract":"<jats:p>Named entity recognition (NER) is an important task in the processing of natural language, which needs to determine entity boundaries and classify them into pre-defined categories. For low-resource languages, most state-of-the-art systems require tens of thousands of annotated sentences to obtain high performance. However, there is minimal annotated data available about Uyghur and Hungarian (UH languages) NER tasks. There are also specificities in each task\u2014differences in words and word order across languages make it a challenging problem. In this paper, we present an effective solution to providing a meaningful and easy-to-use feature extractor for named entity recognition tasks: fine-tuning the pre-trained language model. Therefore, we propose a fine-tuning method for a low-resource language model, which constructs a fine-tuning dataset through data augmentation; then the dataset of a high-resource language is added; and finally the cross-language pre-trained model is fine-tuned on this dataset. In addition, we propose an attention-based fine-tuning strategy that uses symmetry to better select relevant semantic and syntactic information from pre-trained language models and apply these symmetry features to name entity recognition tasks. We evaluated our approach on Uyghur and Hungarian datasets, which showed wonderful performance compared to some strong baselines. We close with an overview of the available resources for named entity recognition and some of the open research questions.<\/jats:p>","DOI":"10.3390\/sym13050786","type":"journal-article","created":{"date-parts":[[2021,5,2]],"date-time":"2021-05-02T08:05:21Z","timestamp":1619942721000},"page":"786","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":24,"title":["Low-Resource Named Entity Recognition via the Pre-Training Model"],"prefix":"10.3390","volume":"13","author":[{"given":"Siqi","family":"Chen","sequence":"first","affiliation":[{"name":"Xinjiang Multilingual Information Technology Laboratory, Xinjiang Multilingual Information Technology Research Center, School of Software, Xinjiang University, Urumqi 832001, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0974-4378","authenticated-orcid":false,"given":"Yijie","family":"Pei","sequence":"additional","affiliation":[{"name":"Xinjiang Multilingual Information Technology Laboratory, Xinjiang Multilingual Information Technology Research Center, School of Software, Xinjiang University, Urumqi 832001, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2589-8377","authenticated-orcid":false,"given":"Zunwang","family":"Ke","sequence":"additional","affiliation":[{"name":"College of Information Science and Engineering, Xinjiang University, Urumqi 832001, China"}]},{"given":"Wushour","family":"Silamu","sequence":"additional","affiliation":[{"name":"College of Information Science and Engineering, Xinjiang University, Urumqi 832001, China"}]}],"member":"1968","published-online":{"date-parts":[[2021,5,2]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Bharadwaj, A., Mortensen, D., Dyer, C., and Carbonell, J. (2016, January 1\u20135). Phonologically Aware Neural Model for Named Entity Recognition in Low Resource Transfer Settings. Proceedings of the Conference on Empirical Methods in Natural Language Processing, Austin, TX, USA.","DOI":"10.18653\/v1\/D16-1153"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Tsai, C.T., Mayhew, S., and Roth, D. (2016, January 1). Cross-Lingual Named Entity Recognition via Wikification. Proceedings of the CoNLL, Berlin, Germany.","DOI":"10.18653\/v1\/K16-1022"},{"key":"ref_3","first-page":"4071","article-title":"Improving Low Resource Named Entity Recognition using Cross-lingual Knowledge Transfer","volume":"1","author":"Feng","year":"2018","journal-title":"IJCAI"},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Bari, M.S., Joty, S., and Jwalapuram, P. (2020, January 7\u201312). Zero-resource cross-lingual named entity recognition. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.","DOI":"10.1609\/aaai.v34i05.6237"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Xie, J., Yang, Z., Neubig, G., Smith, N.A., and Carbonell, J. (2018). Neural cross-lingual named entity recognition with minimal resources. arXiv.","DOI":"10.18653\/v1\/D18-1034"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Ni, J., Dinu, G., and Florian, R. (2017). Weakly supervised cross-lingual named entity recognition via effective annotation and representation projection. arXiv.","DOI":"10.18653\/v1\/P17-1135"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Rijhwani, S., Zhou, S., Neubig, G., and Carbonell, J. (2020). Soft Gazetteers for Low-Resource Named Entity Recognition. arXiv.","DOI":"10.18653\/v1\/2020.acl-main.722"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzm\u00e1n, F., Grave, E., Ott, M., Zettlemoyer, L., and Stoyanov, V. (2019). Unsupervised cross-lingual representation learning at scale. arXiv.","DOI":"10.18653\/v1\/2020.acl-main.747"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Wei, J., and Zou, K. (2019). Eda: Easy data augmentation techniques for boosting performance on text classification tasks. arXiv.","DOI":"10.18653\/v1\/D19-1670"},{"key":"ref_10","unstructured":"Shleifer, S. (2019). Low resource text classification with ulmfit and backtranslation. arXiv."},{"key":"ref_11","unstructured":"Berthelot, D., Carlini, N., Goodfellow, I., Papernot, N., Oliver, A., and Raffel, C. (2019). Mixmatch: A holistic approach to semi-supervised learning. arXiv."},{"key":"ref_12","unstructured":"Xie, Q., Dai, Z., Hovy, E., Luong, M.T., and Le, Q.V. (2019). Unsupervised data augmentation for consistency training. arXiv."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Dai, X., and Adel, H. (2020). An analysis of simple data augmentation for named entity recognition. arXiv.","DOI":"10.18653\/v1\/2020.coling-main.343"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Peters, M.E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (2018). Deep contextualized word representations. arXiv.","DOI":"10.18653\/v1\/N18-1202"},{"key":"ref_15","unstructured":"Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"597","DOI":"10.1162\/tacl_a_00288","article-title":"Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond","volume":"7","author":"Artetxe","year":"2019","journal-title":"Trans. Assoc. Comput. Linguist."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Eisenschlos, J.M., Ruder, S., Czapla, P., Kardas, M., Gugger, S., and Howard, J. (2019). MultiFiT: Efficient multi-lingual language model fine-tuning. arXiv.","DOI":"10.18653\/v1\/D19-1572"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Howard, J., and Ruder, S. (2018). Universal language model fine-tuning for text classification. arXiv.","DOI":"10.18653\/v1\/P18-1031"},{"key":"ref_19","unstructured":"Bradbury, J., Merity, S., Xiong, C., and Socher, R. (2018). Quasi-recurrent neural networks. arXiv."},{"key":"ref_20","unstructured":"Lample, G., and Conneau, A. (2019). Cross-lingual language model pretraining. arXiv."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Sennrich, R., Haddow, B., and Birch, A. (2015). Neural machine translation of rare words with subword units. arXiv.","DOI":"10.18653\/v1\/P16-1162"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1245","DOI":"10.1109\/TMM.2017.2648498","article-title":"Diversified visual attention networks for fine-grained object classification","volume":"19","author":"Zhao","year":"2017","journal-title":"IEEE Trans. Multimed."},{"key":"ref_23","unstructured":"Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv."},{"key":"ref_24","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. arXiv."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Gao, M., Xiao, Q., Wu, S., and Deng, K. (2019). An Improved Method for Named Entity Recognition and Its Application to CEMR. Future Internet, 11.","DOI":"10.3390\/fi11090185"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Shen, T., Zhou, T., Long, G., Jiang, J., Pan, S., and Zhang, C. (2018, January 2\u20137). Disan: Directional self-attention network for rnn\/cnn-free language understanding. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.","DOI":"10.1609\/aaai.v32i1.11941"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Tan, Z., Wang, M., Xie, J., Chen, Y., and Shi, X. (2018, January 2\u20137). Deep semantic role labeling with self-attention. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.","DOI":"10.1609\/aaai.v32i1.11928"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Han, X., and Eisenstein, J. (2019). Unsupervised domain adaptation of contextualized embeddings for sequence labeling. arXiv.","DOI":"10.18653\/v1\/D19-1433"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Zhang, Z., Han, X., Liu, Z., Jiang, X., Sun, M., and Liu, Q. (2019). ERNIE: Enhanced language representation with informative entities. arXiv.","DOI":"10.18653\/v1\/P19-1139"},{"key":"ref_30","unstructured":"Szarvas, G., Farkas, R., Felf\u00f6ldi, L., Kocsor, A., and Csirik, J. (2006, January 22\u201328). A highly accurate Named Entity corpus for Hungarian. Proceedings of the LREC, Genoa, Italy."},{"key":"ref_31","unstructured":"Sang, E.F., and De Meulder, F. (2003). Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. arXiv."},{"key":"ref_32","unstructured":"Schwenk, H., Chaudhary, V., Sun, S., Gong, H., and Guzm\u00e1n, F. (2019). Wikimatrix: Mining 135m parallel sentences in 1620 language pairs from wikipedia. arXiv."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Yao, L., Huang, H., Wang, K.W., Chen, S.H., and Xiong, Q. (2020). Fine-Grained Mechanical Chinese Named Entity Recognition Based on ALBERT-AttBiLSTM-CRF and Transfer Learning. Symmetry, 12.","DOI":"10.3390\/sym12121986"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Sheng, J., Wumaier, A., and Li, Z. (2020). POISE: Efficient Cross-Domain Chinese Named Entity Recognization via Transfer Learning. Symmetry, 12.","DOI":"10.3390\/sym12101673"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Ma, X., and Hovy, E. (2016). End-to-end sequence labeling via bi-directional lstm-cnns-crf. arXiv.","DOI":"10.18653\/v1\/P16-1101"},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"602","DOI":"10.1016\/j.neunet.2005.06.042","article-title":"Framewise phoneme classification with bidirectional LSTM and other neural network architectures","volume":"18","author":"Graves","year":"2005","journal-title":"Neur. Netw."},{"key":"ref_37","unstructured":"Nguyen, T.H., and Grishman, R. Event detection and domain adaptation with convolutional neural networks. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Cho, K., Van Merri\u00ebnboer, B., Bahdanau, D., and Bengio, Y. (2014). On the properties of neural machine translation: Encoder-decoder approaches. arXiv.","DOI":"10.3115\/v1\/W14-4012"},{"key":"ref_39","unstructured":"Wu, Y., Schuster, M., Chen, Z., Le, Q.V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., and Macherey, K. (2016). Google\u2019s neural machine translation system: Bridging the gap between human and machine translation. arXiv."},{"key":"ref_40","unstructured":"Da, K. (2014). A method for stochastic optimization. arXiv."}],"container-title":["Symmetry"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2073-8994\/13\/5\/786\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T05:56:38Z","timestamp":1760162198000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2073-8994\/13\/5\/786"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,5,2]]},"references-count":40,"journal-issue":{"issue":"5","published-online":{"date-parts":[[2021,5]]}},"alternative-id":["sym13050786"],"URL":"https:\/\/doi.org\/10.3390\/sym13050786","relation":{},"ISSN":["2073-8994"],"issn-type":[{"value":"2073-8994","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,5,2]]}}}