{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,9]],"date-time":"2026-10-09T21:19:31Z","timestamp":1791580771345,"version":"4.3.4"},"reference-count":63,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2022,10,31]],"date-time":"2022-10-31T00:00:00Z","timestamp":1667174400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100006769","name":"Russian Science Foundation","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100006769","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Artif. Intell."],"abstract":"<jats:p>This article proposes a hybrid model for the estimation of the complexity of legal documents in Russian. The model consists of two main modules: linguistic feature extractor and a transformer-based neural encoder. The set of linguistic metrics includes both non-specific metrics traditionally used to predict complexity, as well as style-specific metrics developed in order to deal with the peculiarities of official texts. The model was trained on a dataset constructed from text sequences from Russian textbooks. Training data were collected on either subjects related to the topic of legal documents such as Jurisprudence, Economics, Social Sciences, or subjects characterized by the use of general languages such as Literature, History, and Culturology. The final set of materials used contain 48 thousand selected text blocks having various subjects and level-of-complexity identifiers. We have tested the baseline fine-tuned BERT model, models trained on linguistic features, and models trained on features in combination with BERT predictions. The scores show that a hybrid approach to complexity estimation can provide high-quality results in terms of different metrics. The model has been tested on three sets of legal documents.<\/jats:p>","DOI":"10.3389\/frai.2022.1008530","type":"journal-article","created":{"date-parts":[[2022,10,31]],"date-time":"2022-10-31T12:19:33Z","timestamp":1667218773000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":17,"title":["A hybrid model of complexity estimation: Evidence from Russian legal texts"],"prefix":"10.3389","volume":"5","author":[{"given":"Olga","family":"Blinova","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Nikita","family":"Tarasov","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1965","published-online":{"date-parts":[[2022,10,31]]},"reference":[{"key":"B1","first-page":"80","article-title":"Determination of stylistic and genre characteristics of text collections based on part-of-speech compatibility,","author":"Antonova","year":"2011","journal-title":"Trudy mezhdunarodnoj Konferencii"},{"key":"B2","doi-asserted-by":"publisher","first-page":"376","DOI":"10.1111\/j.1467-6478.2011.00549.x","article-title":"Can the law speak directly to its subjects? The limitation of plain language","volume":"38","author":"Assy","year":"2013","journal-title":"J. Law Soc"},{"key":"B3","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1515\/ijld-2017-1000","article-title":"On drafting, interpreting, and translating legal texts across languages and cultures","volume":"2","author":"Azuelos-Atias","year":"2017","journal-title":"Int. J. Legal Dis"},{"key":"B4","unstructured":"BegtinI. Plain Russian Language2016"},{"key":"B5","doi-asserted-by":"publisher","first-page":"63","DOI":"10.1007\/s10648-011-9181-8","article-title":"Reconstructing readability: recent developments and recommendations in the analysis of text difficulty","volume":"24","author":"Benjamin","year":"2012","journal-title":"Educ. Psychol. Rev"},{"key":"B6","first-page":"1","article-title":"Zipf's law of abbreviation as a language universal,","volume-title":"Proceedings of the Leiden Workshop on Capturing Phylogenetic Algorithms for Linguistics","author":"Bentz","year":"2016"},{"key":"B7","doi-asserted-by":"crossref","first-page":"8","DOI":"10.18653\/v1\/W18-6002","article-title":"Using universal dependencies in cross-linguistic complexity research,","author":"Berdicevskis","year":"2018","journal-title":"Proceedings of the Second Workshop on Universal Dependencies (UDW 2018)"},{"key":"B8","doi-asserted-by":"publisher","first-page":"13","DOI":"10.25080\/MAJORA-8B375195-003","article-title":"Hyperopt: a python library for optimizing the hyperparameters of machine learning algorithms,","author":"Bergstra","year":"2013","journal-title":"Proceedings of the 12th Python in Science Conference, Vol. 13"},{"key":"B9","article-title":"Grammatical complexity in academic English,","author":"Biber","year":"2016","journal-title":"Linguistic Change in Writing"},{"key":"B10","first-page":"61","article-title":"Decisions of russian constitutional court: lexical complexity analysis in shallow diachrony,","volume-title":"CEUR Workshop Proceedings. Vol-2813. Proceedings of the International Conference \u201cInternet and Modern Society\u201d","author":"Blinova","year":""},{"key":"B11","doi-asserted-by":"publisher","first-page":"76","DOI":"10.28995\/2075-7182-2020-19-76-92","article-title":"Modeling lemma frequency bands for lexical complexity assessment of russian texts","volume":"19","author":"Blinova","year":"","journal-title":"Comput. Linguist. Intell. Technol"},{"key":"B12","doi-asserted-by":"publisher","first-page":"135","DOI":"10.1162\/tacl_a_00051","article-title":"Enriching word vectors with subword information","volume":"5","author":"Bojanowski","year":"2017","journal-title":"Trans. Assoc. Comput. Linguist"},{"key":"B13","doi-asserted-by":"publisher","first-page":"464","DOI":"10.1016\/j.procs.2018.11.108","article-title":"A neural network model for the evaluation of text complexity in italian language: a representation point of view","volume":"145","author":"Bosco","year":"2018","journal-title":"Procedia Comput. Sci"},{"key":"B14","doi-asserted-by":"publisher","first-page":"785","DOI":"10.1145\/2939672.2939785","article-title":"Xgboost: a scalable tree boosting system,","author":"Chen","year":"2016","journal-title":"Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining"},{"key":"B15","doi-asserted-by":"publisher","first-page":"97","DOI":"10.1075\/itl.165.2.01col","article-title":"Computational assessment of text readability: a survey of current and future research. Recent advances in automatic readability assessment and text simplification","volume":"165","author":"Collins-Thompson","year":"2014","journal-title":"Int. J. Appl. Linguist"},{"key":"B16","first-page":"193","article-title":"A language modeling approach to predicting reading difficulty,","author":"Collins-Thompson","year":"2004","journal-title":"Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004"},{"key":"B17","unstructured":"Conll 2018 Shared Task2018"},{"key":"B18","unstructured":"Consultant Plus: Legal Reference System2022"},{"key":"B19","author":"Dahl","year":"1993","journal-title":"The Growth and Maintenance of Linguistic Complexity"},{"key":"B20","doi-asserted-by":"crossref","first-page":"1","DOI":"10.18653\/v1\/2020.bea-1.1","article-title":"Linguistic features for readability assessment,","volume-title":"Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications","author":"Deutsch","year":"2020"},{"key":"B21","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1810.04805","article-title":"Bert: Pre-training of deep bidirectional transformers for language understanding","author":"Devlin","year":"2018","journal-title":"arXiv preprint arXiv:1810.04805"},{"key":"B22","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyg.2021.703690","article-title":"A comparative study of educational texts for native, foreign, and bilingual young speakers of russian: are simplified texts equally simple?","author":"Dmitrieva","year":"2021","journal-title":"Front. Psychol"},{"key":"B23","doi-asserted-by":"publisher","first-page":"125","DOI":"10.21128\/1812-7126-2017-3-125-133","article-title":"\u201cthe art of legal writing\u201d: a quantitative analysis of russian constitutional court rulings","volume":"118","author":"Dmitrieva","year":"2017","journal-title":"Sravnitel'noe Konstitutsionnoe Obozrenie"},{"key":"B24","first-page":"991","article-title":"Processing of static and dynamic texts: an eye-tracking study of russian,","author":"Dobrego","year":"2016","journal-title":"3rd International Multidisciplinary Scientific Conference on Social Sciences and Arts SGEM 2016, Vol. 11"},{"key":"B25","author":"Druzhkin","year":"2016","journal-title":"Readability Metrics for Russian"},{"key":"B26","first-page":"276","article-title":"A comparison of features for automatic readability assessment,","author":"Feng","year":"2010","journal-title":"COLING'10: Proceedings of the 23rd International Conference on Computational Linguistics"},{"key":"B27","doi-asserted-by":"crossref","first-page":"129","DOI":"10.1017\/CBO9780511597855.005","article-title":"Syntactic complexity,","volume-title":"Natural Language Parsing: Psychological, Computational, and Theoretical Perspectives","author":"Frazier","year":"1985"},{"key":"B28","unstructured":"Garant: Legal Information Portal2022"},{"key":"B29","first-page":"284","article-title":"Efficiency of text readability features in russian academic texts,","author":"Ivanov","year":"2018","journal-title":"Komp'juternaja Lingvistika i Intellektual'nye Tehnologii 2018, Vol. 17"},{"key":"B30","author":"Klyshinskij","year":"2013","journal-title":"Formation of a Model of Compatibility of Russian Words and The study of its Propertie"},{"key":"B31","author":"Knutov","year":"2020","journal-title":"Complexity of Russian Laws. The Experience of Syntactic Analysis"},{"key":"B32","first-page":"320","article-title":"Morphological analyzer and generator for russian and ukrainian languages,","volume-title":"Analysis of Images, Social Networks and Texts. AIST 2015. Communications in Computer and Information Science","author":"Korobov","year":"2015"},{"key":"B33","author":"Kuchakov","year":"2018","journal-title":"The Complexity of Legal Acts in Russia: Lexical and Syntactic Quality of Texts: Analytic Note"},{"key":"B34","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1905.07213","article-title":"Adaptation of deep bidirectional multilingual transformers for russian language","author":"Kuratov","year":"2019","journal-title":"arXiv preprint arXiv:1905.07213"},{"key":"B35","doi-asserted-by":"publisher","first-page":"333","DOI":"10.1111\/modl.12468","article-title":"Measuring syntactic complexity in l2 writing using fine-grained clausal and phrasal indices","volume":"102","author":"Kyle","year":"2018","journal-title":"Modern Lang. J"},{"key":"B36","first-page":"408","article-title":"On determining the complexity of russian texts,","author":"Ljashevskaja","year":"1996","journal-title":"XVII Aprel'skaia Mezhdunarodnaia Nauchnaia Konferentsiia po Problemam Razvitiia Ekonomiki i Obshchestva: v 4 kn"},{"key":"B37","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1711.05101","article-title":"Decoupled weight decay regularization","author":"Loshchilov","year":"2017","journal-title":"arXiv preprint arXiv:1711.05101"},{"key":"B38","doi-asserted-by":"publisher","first-page":"100529","DOI":"10.1016\/j.asw.2021.100529","article-title":"Automated assessment of learner text complexity","volume":"49","author":"Lyashevskaya","year":"2021","journal-title":"Assess. Writing"},{"key":"B39","doi-asserted-by":"publisher","first-page":"125","DOI":"10.1515\/lity.2001.001","article-title":"The worlds simplest grammars are creole grammar","volume":"5","author":"McWhorter","year":"2001","journal-title":"Linguist. Typol"},{"key":"B40","doi-asserted-by":"publisher","author":"Merchant","year":"2020","DOI":"10.18653\/v1\/2020.blackboxnlp-1.4"},{"key":"B41","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1301.3781","article-title":"Efficient estimation of word representations in vector space","author":"Mikolov","year":"2013","journal-title":"arXiv preprint arXiv:1301.3781"},{"key":"B42","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2207.06265","article-title":"A transfer learning based model for text readability assessment in german","author":"Mohtaj","year":"2022","journal-title":"arXiv preprint arXiv:2207.06265"},{"key":"B43","doi-asserted-by":"publisher","first-page":"426","DOI":"10.22363\/2687-0088-30132","article-title":"Text complexity and linguistic features: Their correlation in english and russian","volume":"26","author":"Morozov","year":"2022","journal-title":"Russian J. Linguist"},{"key":"B44","author":"Nagel,'","year":"2017","journal-title":"Word-Formation Mechanisms in the Processes of Perception, Identification, and Use of Language"},{"key":"B45","doi-asserted-by":"crossref","first-page":"110","DOI":"10.1093\/oso\/9780199545216.003.0008","article-title":"Linguistic complexity : a comprehensive definition and survey,","volume-title":"Language Complexity as an Evolving Variable","author":"Nichols","year":"2009"},{"key":"B46","first-page":"86","article-title":"Automation of text perception quality assessments,","author":"Oborneva","year":"2005","journal-title":"Vestnik Moskovskogo Gorodskogo Pedagogicheskogo Universiteta"},{"key":"B47","doi-asserted-by":"publisher","first-page":"1532","DOI":"10.3115\/v1\/D14-1162","article-title":"Glove: global vectors for word representation,","author":"Pennington","year":"2014","journal-title":"Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing"},{"key":"B48","doi-asserted-by":"crossref","first-page":"289","DOI":"10.18653\/v1\/W16-0534","article-title":"Insights from russian second language readability classification: complexity-dependent training requirements, and feature evaluation of multiple categories,","volume-title":"Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications","author":"Reynolds","year":"2016"},{"key":"B49","doi-asserted-by":"publisher","DOI":"10.3390\/app10207285","article-title":"Automatic classification of text complexity","author":"Santucci","year":"2020","journal-title":"Appl. Sci"},{"key":"B50","author":"Savel'ev","year":"2019","journal-title":"Decisions of Arbitration Courts of Russian Federation: Lexical and Syntactic Quality of Texts, Analytic Note"},{"key":"B51","doi-asserted-by":"publisher","first-page":"371","DOI":"10.22363\/2687-0088-30178","article-title":"What neural networks know about linguistic complexity","volume":"26","author":"Sharoff","year":"2022","journal-title":"Russian J. Linguist"},{"key":"B52","doi-asserted-by":"publisher","first-page":"132","DOI":"10.1007\/978-3-030-04497-8_11","article-title":"Readability formula for russian texts: a modified version","volume":"11289","author":"Solnyshkina","year":"2018","journal-title":"Adv. Comput. Intell"},{"key":"B53","first-page":"18","article-title":"Text complexity as interdisciplinary problem,","author":"Solnyshkina","year":"2022","journal-title":"Voprosy Kognitivnoy Lingvistiki"},{"key":"B54","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1909.09586","article-title":"Understanding lstm-a tutorial into long short-term memory recurrent neural networks","author":"Staudemeyer","year":"2019","journal-title":"arXiv preprint arXiv:1909.09586"},{"key":"B55","unstructured":"StrakaM. Strakov\u00e1J. Universal Dependencies 2.5 Models for UDPipe2019"},{"key":"B56","doi-asserted-by":"crossref","first-page":"6","DOI":"10.1515\/9783110229226.6","article-title":"Introduction: linguistic complexity: second language acquisition, indigenization, contact,","volume-title":"Linguistic Complexity: Second Language Acquisition, Indigenization, Contact","author":"Szmrecsanyi","year":"2012"},{"key":"B57","first-page":"41","article-title":"Automated assessment of language proficiency on german data,","author":"Sz\u00fcgyi","year":"2019","journal-title":"Proceedings of the 15th Conference on Natural Language Processing, KONVENS 2019"},{"key":"B58","author":"Tiersma","year":"1999","journal-title":"Legal Language"},{"key":"B59","author":"Trudgill","year":"2011","journal-title":"Sociolinguistic Typology: Social Determinants of Linguistic Complexity"},{"key":"B60","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-demos.6","article-title":"Huggingface's transformers: State-of-the-art natural language processing","author":"Wolf","year":"2019","journal-title":"arXiv preprint arXiv:1910.03771"},{"key":"B61","author":"Wydick","year":"2019","journal-title":"Plain English for Lawyers"},{"key":"B62","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1906.07580","article-title":"Text readability assessment for second language learners","author":"Xia","year":"2019","journal-title":"arXiv preprint arXiv:1906.07580"},{"key":"B63","first-page":"84","article-title":"Experience of quantitative-typological study of varieties of oral speech,","author":"Zhuravlev","year":"1988","journal-title":"Raznovidnosti Gorodskoi Ustnoi Rechi. Sbornik Nauchnykh Trudov"}],"container-title":["Frontiers in Artificial Intelligence"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2022.1008530\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,11,29]],"date-time":"2023-11-29T20:53:20Z","timestamp":1701291200000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/frai.2022.1008530\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,10,31]]},"references-count":63,"alternative-id":["10.3389\/frai.2022.1008530"],"URL":"https:\/\/doi.org\/10.3389\/frai.2022.1008530","relation":{},"ISSN":["2624-8212"],"issn-type":[{"value":"2624-8212","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,10,31]]},"article-number":"1008530"}}