{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,11]],"date-time":"2026-06-11T16:13:53Z","timestamp":1781194433346,"version":"3.54.1"},"reference-count":46,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2018,3,27]],"date-time":"2018-03-27T00:00:00Z","timestamp":1522108800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>Text on the Internet is written in different languages and scripts that can be divided into different language groups. Most of the errors in language identification occur with similar languages. To improve the performance of short-text language identification, we propose four different levels of hierarchical language identification methods and conducted comparative tests in this paper. The efficiency of the algorithms was evaluated on sentences from 97 languages, and its macro-averaged F1-score reached in four-stage language identification was 0.9799. The experimental results verified that, after script identification, language group identification and similar language group identification, the performance of the language identification algorithm improved with each stage. Notably, the language identification accuracy between similar languages improved substantially. We also investigated how foreign content in a language affects language identification.<\/jats:p>","DOI":"10.3390\/a11040039","type":"journal-article","created":{"date-parts":[[2018,3,27]],"date-time":"2018-03-27T12:17:24Z","timestamp":1522153044000},"page":"39","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["On Hierarchical Text Language-Identification Algorithms"],"prefix":"10.3390","volume":"11","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8860-951X","authenticated-orcid":false,"given":"Maimaitiyiming","family":"Hasimu","sequence":"first","affiliation":[{"name":"School of Information Science and Engineering, Xinjiang University, Urumqi 830046, China"},{"name":"Key Multi-lingual Laboratory of Xinjiang, Urumqi 830046, China"},{"name":"Department of Computer, Hotan Teachers College, Hotan 848000, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wushour","family":"Silamu","sequence":"additional","affiliation":[{"name":"School of Information Science and Engineering, Xinjiang University, Urumqi 830046, China"},{"name":"Key Multi-lingual Laboratory of Xinjiang, Urumqi 830046, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2018,3,27]]},"reference":[{"key":"ref_1","first-page":"21","article-title":"Optimizing n-gram order of an n-gram based language identification algorithm for 68 written languages","volume":"2","author":"Choong","year":"2009","journal-title":"Int. J. Adv. ICT Emerg. Reg."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"491","DOI":"10.1016\/j.ipm.2015.12.003","article-title":"Effective language identification of forum texts based on statistical approaches","volume":"52","author":"Abainia","year":"2016","journal-title":"Inf. Process. Manag. Int. J."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"307","DOI":"10.1016\/j.csl.2012.01.004","article-title":"Factors that affect the accuracy of text-based language identification","volume":"26","author":"Botha","year":"2012","journal-title":"Comput. Speech Lang."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"457","DOI":"10.1016\/j.jksuci.2014.12.004","article-title":"Word-length algorithm for language identification of under-resourced languages","volume":"28","author":"Selamat","year":"2015","journal-title":"J. King Saud Univ.-Comput. Inf. Sci."},{"key":"ref_5","unstructured":"Cavnar, W.B., and Trenkle, J.M. (1994, January 11\u201313). N-Gram-Based Text Categorization. Proceedings of the 3rd Annual Symposium on Document Analysis and Information Retrieval (SDAIR-94), Las Vegas, NV, USA."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Singh, A.K. (2006, January 23). Study Some Distance Measures for Language and Encoding Identification. Proceedings of the Workshop on Linguistic Distance, Sydney, Australia.","DOI":"10.3115\/1641976.1641985"},{"key":"ref_7","unstructured":"Baldwin, T., and Lui, M. (2010, January 2\u20134). Language Identfication: The Long and the Short of the Matter. Proceedings of the Human Language Technologies: The 2010 Annual Conference of the North American of the ACL, Los Angeles, CA, USA."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"534","DOI":"10.1016\/j.diin.2012.05.004","article-title":"Finding and identifying text in 900+ languages","volume":"9","author":"Brown","year":"2012","journal-title":"Digit. Investig."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"534","DOI":"10.3923\/itj.2007.534.540","article-title":"Unicode Aided Language Identification across Multiple Scripts and Heterogeneous Data","volume":"6","author":"Hanif","year":"2007","journal-title":"Inf. Technol. J."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Rowe, B.M., and Levine, D.P. A Concise Introduction to Linguistics, Routledge. Retrieved 26 January 2017.","DOI":"10.4324\/9781315227283"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"17758","DOI":"10.1073\/pnas.1212380109","article-title":"The great human expansion","volume":"109","author":"Henn","year":"2017","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"ref_12","first-page":"354","article-title":"Tree-stage Short Text Language Identification Algorithm","volume":"15","author":"Hasimu","year":"2017","journal-title":"J. Digit. Lang. Identif."},{"key":"ref_13","unstructured":"\u017deljko, A. (2012, January 8\u201312). Slovene-Croatian Treebank Transfer Using Bilingual Lexicon Improves Croatian Dependency Parsing. Proceedings of the 15th International Multiconference Information Society, Ljubljana, Slovenija."},{"key":"ref_14","unstructured":"Sibun, P., and Reynar, J.C. (1996, January 15\u201317). Languge identification: Examining the issues. Proceedings of the 5th Symposium on Document Analysis and Information Retrieval, Las Vegas, NV, USA."},{"key":"ref_15","unstructured":"Majlis, M. (2012, January 25\u201327). Yet another language identifier. Proceedings of the Student Research Workshop at the Conference of the European Chapter of the Association for Computational Linguistics, Avignon, France."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"152","DOI":"10.1016\/j.physa.2017.05.011","article-title":"From language identification to language distance","volume":"484","author":"Gamallo","year":"2017","journal-title":"Phys. Stat. Mech. Appl."},{"key":"ref_17","first-page":"183","article-title":"Natural language identification using corpus-based models","volume":"13","author":"Souter","year":"1994","journal-title":"Hermes J. Linguist."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"843","DOI":"10.1126\/science.267.5199.843","article-title":"Gauging similarity with n-grams: Language-independent categorization of text","volume":"267","author":"Damashek","year":"1995","journal-title":"Science"},{"key":"ref_19","unstructured":"Goldhahn, D., and Quasthoff, U. (2014, January 26\u201331). Vocabulary-Based Language Similarity using Web Corpora. Proceedings of the Ninth International Conference on Language Resource and Evaluation, Reykjavik, Iceland."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Zampieri, M., Tan, L., Ljube\u0161i\u0107, N., and Tiedemann, J. (2014, January 23). A Report on the DSL Shared Task 2014. Proceedings of the Workshop on Applying NlP TOOLS to Similar Languages, Dublin, Ireland.","DOI":"10.3115\/v1\/W14-5307"},{"key":"ref_21","unstructured":"Zampieri, M., Tan, L., Ljube\u0161i\u0107, N., Tiedemann, J., and Nakov, P. (2015, January 10). Overview of the DSL Shared Task 2015. Proceedings of the LT4VarDial Workshop, Hissar, Bulgaria."},{"key":"ref_22","unstructured":"Zampieri, M., Malmasi, S., Ljube\u0161i\u0107, N., Nakov, P., Ali, A., and Tiedemann, J. (2016, January 12). Discriminating Between Similar Languages and Arabic Dialect Identification: A Report on the Third DSL Shared Task. Proceedings of the Third Workshop on NLP for Similar Languages, Varieties and Dialects, Osaka, Japan."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Zampieri, M., Malmasi, S., Ljube\u0161i\u0107, N., Nakov, P., Ali, A., Tiedemann, J., Scherrer, Y., and Aepli, N. (2017, January 3). Findings of the VarDial Evaluation Campaign. Proceedings of the VarDial Workshop, Valencia, Spain.","DOI":"10.18653\/v1\/W17-1201"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Criscuolo, M., and Aluisio, S. (2017, January 3). Discriminating between similar languages with word level convolutional neural networks. Proceedings of the VarDial Workshop, Valencia, Spain.","DOI":"10.18653\/v1\/W17-1215"},{"key":"ref_25","unstructured":"Belinkov, Y., and Glass, G. (arXiv, 2016). A Character-level Convolutional Neural Network for Distinguishing Similar Languages and Dialects, arXiv."},{"key":"ref_26","unstructured":"Bjerva, B. (arXiv, 2016). Byte-based Language Identification with deep Convolutional Networks, arXiv."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Goutte, C., L\u00e9ger, S., and Carpuat, M. (2014, January 23). The NRC System for Discriminating Similar Languages. Proceedings of the First Workshop on Applying NLP Tools to Similar languages, Varieties and Dialects, Dublin, Ireland.","DOI":"10.3115\/v1\/W14-5316"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Porta, J., and Sancho, J.L. (2014, January 23). Using Maximum Entropy Models to Discriminate between Similar Languages and Varieties. Proceedings of the First Workshop on Applying NLP Tools to Similar languages, Varieties and Dialects, Dublin, Ireland.","DOI":"10.3115\/v1\/W14-5314"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Lui, M., Letcher, N., Adams, O., Long, D., Cook, P., and Baldwin, T. (2014, January 23). Exploring Methods and Resources for Discriminating similar languages. Proceedings of the First Workshop on Applying NLP Tools to Similar Languages, Varieties and Dialects, Dublin, Ireland.","DOI":"10.3115\/v1\/W14-5315"},{"key":"ref_30","unstructured":"Fabra-Boluda, R., Rangel, F., and Rosso, P. (2015, January 10). NLEL UPV authoritas participation at Discrimination between Similar Language (DSL) 2015 shared task. Proceedings of the LT4VarDial Workshop, Hissar, Bulgaria."},{"key":"ref_31","unstructured":"\u00c1cs, J., Grad-Gyenge, L., and de Rezende Oliveira, T.B.R. (2015, January 10). A two-level classifier for discriminating similar languages. Proceedings of the Joint Workshop on Language Technology for Closely Related Languages, Varieties and Dialects, Hissar, Bulgaria."},{"key":"ref_32","unstructured":"Goutte, C., L\u00e9ger, S., Malmasi, S., and Zampieri, M. (arXiv, 2016). Discriminating Similar Languages: Evaluations and Explorations, arXiv."},{"key":"ref_33","unstructured":"Hofmann, M., and Ralf, K. (2014). Rapid Miner Data Mining Use Cases and Business Analytics Applications, CRC Press."},{"key":"ref_34","unstructured":"(2018, March 07). Polyglot. Available online: https:\/\/github.com\/saffsd\/polyglot."},{"key":"ref_35","unstructured":"(2018, March 07). Langid.py. Available online: https:\/\/github.com\/saffsd\/langid.py."},{"key":"ref_36","unstructured":"Lui, M., and BaldWin, T. (2012, January 8\u201314). Langid.py: An off the shelf Language Identification Tool. Proceedings of the 50th Annual of the Association for Computational Linguistics, Jeju Island, Korea."},{"key":"ref_37","unstructured":"Goldhahn, D., Eckart, T., and Quasthoff, U. (2012, January 23\u201325). Building Large Monolingual Dictionaries at the Leipzig Corpora Collection: From 100 to 200 languages. Proceedings of the 8th International Language Resources and Evaluation (LREC\u201912), Istanbul, Turkey."},{"key":"ref_38","unstructured":"(2018, March 07). About the Leipzig Corpora Collection. Available online: http:\/\/asvdoku.informatik.uni-leipzig.de\/corpora\/."},{"key":"ref_39","unstructured":"(2018, March 07). Codes for the Representation of Names of Languages, Available online: http:\/\/www.loc.gov\/standards\/iso639-2\/php\/code_list.php."},{"key":"ref_40","first-page":"1289","article-title":"An extensive empirical study of feature selection metrics for text classification","volume":"3","author":"Forman","year":"2003","journal-title":"J. Mach. Learn. Res."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"226","DOI":"10.1016\/j.knosys.2012.06.005","article-title":"A Novel Probabilistic Feature Selection Method for Text Classification","volume":"36","author":"Uysal","year":"2012","journal-title":"Knowl. Based Syst."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"473","DOI":"10.1016\/j.ipm.2016.12.004","article-title":"Feature Selection Based on a Normalized Difference Measure for Text Classification","volume":"53","author":"Rehman","year":"2017","journal-title":"Inf. Process. Manag."},{"key":"ref_43","unstructured":"Brank, J., Grobelnik, M., Milic-Frayling, N., and Mladenic, D. (2002, January 8\u201312). Interaction of feature selection methods and linear classification models. Proceedings of the Workshop on Text Learning held at International Conference on Machine Learning (LCML), Sydney, Australia."},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/505282.505283","article-title":"Machine learning, in automated text categorization","volume":"34","author":"Sebastitiani","year":"2002","journal-title":"ACM Comput. Surv."},{"key":"ref_45","first-page":"2825","article-title":"Scikit-learn: Machine Learning in Python","volume":"12","author":"Pedregosa","year":"2011","journal-title":"JMLR"},{"key":"ref_46","doi-asserted-by":"crossref","unstructured":"Joachims, T. (2002). Learning to Classify Using Support Vector Machines, Kluwer. Dissertation.","DOI":"10.1007\/978-1-4615-0907-3"}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/11\/4\/39\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T14:58:40Z","timestamp":1760194720000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/11\/4\/39"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,3,27]]},"references-count":46,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2018,4]]}},"alternative-id":["a11040039"],"URL":"https:\/\/doi.org\/10.3390\/a11040039","relation":{},"ISSN":["1999-4893"],"issn-type":[{"value":"1999-4893","type":"electronic"}],"subject":[],"published":{"date-parts":[[2018,3,27]]}}}