{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,20]],"date-time":"2026-03-20T08:56:06Z","timestamp":1773996966086,"version":"3.50.1"},"reference-count":71,"publisher":"MIT Press - Journals","issue":"1","license":[{"start":{"date-parts":[[2021,3,6]],"date-time":"2021-03-06T00:00:00Z","timestamp":1614988800000},"content-version":"vor","delay-in-days":5,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,4,21]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>We present a set of novel neural supervised and unsupervised approaches for determining the readability of documents. In the unsupervised setting, we leverage neural language models, whereas in the supervised setting, three different neural classification architectures are tested. We show that the proposed neural unsupervised approach is robust, transferable across languages, and allows adaptation to a specific readability task and data set. By systematic comparison of several neural architectures on a number of benchmark and new labeled readability data sets in two languages, this study also offers a comprehensive analysis of different neural approaches to readability classification. We expose their strengths and weaknesses, compare their performance to current state-of-the-art classification approaches to readability, which in most cases still rely on extensive feature engineering, and propose possibilities for improvements.<\/jats:p>","DOI":"10.1162\/coli_a_00398","type":"journal-article","created":{"date-parts":[[2021,3,5]],"date-time":"2021-03-05T18:59:47Z","timestamp":1614970787000},"page":"141-179","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":72,"title":["Supervised and Unsupervised Neural Approaches to Text Readability"],"prefix":"10.1162","volume":"47","author":[{"given":"Matej","family":"Martinc","sequence":"first","affiliation":[{"name":"Jo\u017eef Stefan Institute, Ljubljana, Slovenia"},{"name":"Jo\u017eef Stefan International Postgraduate School, Ljubljana, Slovenia. matej.martinc@ijs.si"}]},{"given":"Senja","family":"Pollak","sequence":"additional","affiliation":[{"name":"Jo\u017eef Stefan Institute, Ljubljana, Slovenia. senja.pollak@ijs.si"}]},{"given":"Marko","family":"Robnik-\u0160ikonja","sequence":"additional","affiliation":[{"name":"University of Ljubljana, Faculty of Computer and Information Science, Ljubljana, Slovenia. marko.robnik@fri.uni-lj.si"}]}],"member":"281","published-online":{"date-parts":[[2021,4,21]]},"reference":[{"key":"2021042218044910500_bib1","first-page":"1","article-title":"Analysing the readability of English and non-English texts in the classroom with LIX","volume-title":"Seventh Australian Reading Association Conference","author":"Anderson","year":"1981"},{"key":"2021042218044910500_bib2","doi-asserted-by":"crossref","first-page":"421","DOI":"10.1162\/tacl_a_00278","article-title":"Multiattentive recurrent neural network architecture for multilingual readability assessment","volume":"7","author":"Azpiazu","year":"2019","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2021042218044910500_bib3","article-title":"Neural machine translation by jointly learning to align and translate","author":"Bahdanau","year":"2014","journal-title":"arXiv preprint arXiv:1409.0473"},{"key":"2021042218044910500_bib4","article-title":"An empirical evaluation of generic convolutional and recurrent networks for sequence modeling","author":"Bai","year":"2018","journal-title":"arXiv preprint arXiv:1803.01271"},{"key":"2021042218044910500_bib5","article-title":"Longformer: The long-document transformer","author":"Beltagy","year":"2020","journal-title":"arXiv preprint arXiv:2004.05150"},{"issue":"2","key":"2021042218044910500_bib6","doi-asserted-by":"crossref","first-page":"157","DOI":"10.1109\/72.279181","article-title":"Learning long-term dependencies with gradient descent is difficult","volume":"5","author":"Bengio","year":"1994","journal-title":"IEEE Transactions on Neural Networks"},{"key":"2021042218044910500_bib7","doi-asserted-by":"crossref","first-page":"31","DOI":"10.3115\/1219044.1219075","article-title":"NLTK: The natural language toolkit","volume-title":"Proceedings of the ACL 2004 on Interactive Poster and Demonstration Sessions","author":"Bird","year":"2004"},{"key":"2021042218044910500_bib8","volume-title":"Development of Readability Analysis","author":"Bormuth","year":"1969"},{"key":"2021042218044910500_bib9","doi-asserted-by":"crossref","first-page":"1724","DOI":"10.3115\/v1\/D14-1179","article-title":"Learning phrase representations using RNN encoder\u2013decoder for statistical machine translation","volume-title":"Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Cho","year":"2014"},{"issue":"2","key":"2021042218044910500_bib10","doi-asserted-by":"crossref","first-page":"97","DOI":"10.1075\/itl.165.2.01col","article-title":"Computational assessment of text readability: A survey of current and future research","volume":"165","author":"Collins-Thompson","year":"2014","journal-title":"ITL-International Journal of Applied Linguistics"},{"issue":"Aug","key":"2021042218044910500_bib11","first-page":"2493","article-title":"Natural language processing (almost) from scratch","volume":"12","author":"Collobert","year":"2011","journal-title":"Journal of Machine Learning Research"},{"key":"2021042218044910500_bib12","first-page":"670","article-title":"Supervised learning of universal sentence representations from natural language inference data","volume-title":"Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing","author":"Conneau","year":"2017"},{"key":"2021042218044910500_bib13","volume-title":"Common European Framework of Reference for Languages: Learning, Teaching, Assessment","author":"Council of Europe, Council for Cultural Co-operation. Education Committee. Modern Languages Division","year":"2001"},{"issue":"5\u20136","key":"2021042218044910500_bib14","doi-asserted-by":"crossref","first-page":"340","DOI":"10.1080\/0163853X.2017.1296264","article-title":"Predicting text comprehension, processing, and familiarity in adult readers: New approaches to readability formulas","volume":"54","author":"Crossley","year":"2017","journal-title":"Discourse Processes"},{"key":"2021042218044910500_bib15","first-page":"37","article-title":"A formula for predicting readability: Instructions","author":"Dale","year":"1948","journal-title":"Educational Research Bulletin"},{"key":"2021042218044910500_bib16","doi-asserted-by":"crossref","first-page":"187","DOI":"10.2307\/747483","article-title":"On the failure of readability formulas to define readable texts: A case study from adaptations","author":"Davison","year":"1982","journal-title":"Reading Research Quarterly"},{"key":"2021042218044910500_bib17","article-title":"Linguistic features for readability assessment","author":"Deutsch","year":"2020","journal-title":"arXiv preprint arXiv:2006.00377"},{"key":"2021042218044910500_bib18","first-page":"4171","article-title":"BERT: Pre-training of deep bidirectional transformers for language understanding","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Devlin","year":"2019"},{"key":"2021042218044910500_bib19","article-title":"Text simplification: A survey","author":"Feng","year":"2008"},{"key":"2021042218044910500_bib20","first-page":"229","article-title":"Cognitively motivated features for readability assessment","volume-title":"Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009)","author":"Feng","year":"2009"},{"key":"2021042218044910500_bib21","first-page":"276","article-title":"A comparison of features for automatic readability assessment","volume-title":"COLING 2010: Posters","author":"Feng","year":"2010"},{"key":"2021042218044910500_bib22","first-page":"335","article-title":"Automatic text difficulty estimation using embeddings and neural networks","volume-title":"European Conference on Technology Enhanced Learning","author":"Filighera","year":"2019"},{"key":"2021042218044910500_bib23","first-page":"29","article-title":"Lexical tightness and text complexity","volume-title":"Proceedings of the Workshop on Natural Language Processing for Improving Textual Accessibility","author":"Flor","year":"2013"},{"key":"2021042218044910500_bib24","first-page":"241","article-title":"A data set of syntactic-ngrams over time from a very large corpus of English books","volume-title":"Second Joint Conference on Lexical and Computational Semantics","author":"Goldberg","year":"2013"},{"key":"2021042218044910500_bib25","volume-title":"Deep Learning","author":"Goodfellow","year":"2016"},{"key":"2021042218044910500_bib26","volume-title":"The Technique of Clear Writing","author":"Gunning","year":"1952"},{"key":"2021042218044910500_bib27","volume-title":"Cohesion in English","author":"Halliday","year":"1976"},{"key":"2021042218044910500_bib28","doi-asserted-by":"crossref","first-page":"3651","DOI":"10.18653\/v1\/P19-1356","article-title":"What does BERT learn about the structure of language?","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Jawahar","year":"2019"},{"key":"2021042218044910500_bib29","first-page":"155","article-title":"A domain independent approach for extracting terms from research papers","volume-title":"Australasian Database Conference","author":"Jiang","year":"2015"},{"issue":"1958","key":"2021042218044910500_bib30","first-page":"253","article-title":"Application de l\u2019indice de flesch \u00e0 la langue fran\u00e7aise","volume":"19","author":"Kandel","year":"1958","journal-title":"Cahiers Etudes de Radio-T\u00e9l\u00e9vision"},{"key":"2021042218044910500_bib31","first-page":"2741","article-title":"Character-aware neural language models","volume-title":"AAAI","author":"Kim","year":"2016"},{"key":"2021042218044910500_bib32","doi-asserted-by":"crossref","DOI":"10.21236\/ADA006655","volume-title":"Derivation of New Readability Formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy Enlisted Personnel","author":"Kincaid","year":"1975"},{"key":"2021042218044910500_bib33","article-title":"Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing","author":"Kudo","year":"2018","journal-title":"arXiv preprint arXiv:1808.06226"},{"key":"2021042218044910500_bib34","article-title":"Pearson\u2019s text complexity measure","author":"Landauer","year":"2011"},{"key":"2021042218044910500_bib35","volume-title":"Korpusi slovenskega jezika Gigafida, KRES, ccGigafida in ccKRES: gradnja, vsebina, uporaba","author":"Logar","year":"2012"},{"key":"2021042218044910500_bib36","first-page":"4768","article-title":"A unified approach to interpreting model predictions","volume-title":"Advances in Neural Information Processing Systems","author":"Lundberg","year":"2017"},{"key":"2021042218044910500_bib37","first-page":"548","article-title":"Ranking-based readability assessment for early primary children\u2019s literature","volume-title":"Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Ma","year":"2012"},{"key":"2021042218044910500_bib38","first-page":"1099","article-title":"Automated scoring: Beyond natural language processing","volume-title":"Proceedings of the 27th International Conference on Computational Linguistics","author":"Madnani","year":"2018"},{"issue":"6","key":"2021042218044910500_bib39","doi-asserted-by":"crossref","first-page":"644","DOI":"10.1002\/asi.24293","article-title":"Is crosslingual readability assessment possible?","volume":"71","author":"Madrazo Azpiazu","year":"2020","journal-title":"Journal of the Association for Information Science and Technology"},{"issue":"8","key":"2021042218044910500_bib40","first-page":"639","article-title":"SMOG grading\u2014a new readability formula","volume":"12","author":"McLaughlin","year":"1969","journal-title":"Journal of Reading"},{"key":"2021042218044910500_bib41","first-page":"605","article-title":"Empirical evaluation and combination of advanced language modeling techniques","volume-title":"Twelfth Annual Conference of the International Speech Communication Association","author":"Mikolov","year":"2011"},{"key":"2021042218044910500_bib42","first-page":"3111","article-title":"Distributed representations of words and phrases and their compositionality","volume-title":"Advances in Neural Information Processing Systems","author":"Mikolov","year":"2013"},{"key":"2021042218044910500_bib43","article-title":"Text as environment: A deep reinforcement learning text readability assessment model","author":"Mohammadi","year":"2019","journal-title":"arXiv preprint arXiv:1912.05957"},{"key":"2021042218044910500_bib44","doi-asserted-by":"crossref","first-page":"45","DOI":"10.18653\/v1\/W18-0505","article-title":"Estimating linguistic complexity for science texts","volume-title":"Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications","author":"Nadeem","year":"2018"},{"key":"2021042218044910500_bib45","first-page":"96","article-title":"Online readability and text complexity analysis with TextEvaluator","volume-title":"Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations","author":"Napolitano","year":"2015"},{"key":"2021042218044910500_bib46","doi-asserted-by":"crossref","first-page":"1532","DOI":"10.3115\/v1\/D14-1162","article-title":"GloVe: Global vectors for word representation","volume-title":"Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)","author":"Pennington","year":"2014"},{"key":"2021042218044910500_bib47","first-page":"2227","article-title":"Deep contextualized word representations","volume-title":"Proceedings of NAACL-HLT","author":"Peters","year":"2018"},{"issue":"1","key":"2021042218044910500_bib48","doi-asserted-by":"crossref","first-page":"89","DOI":"10.1016\/j.csl.2008.04.003","article-title":"A machine learning approach to reading level assessment","volume":"23","author":"Petersen","year":"2009","journal-title":"Computer Speech & Language"},{"key":"2021042218044910500_bib49","first-page":"186","article-title":"Revisiting readability: A unified framework for predicting text quality","volume-title":"Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing","author":"Pitler","year":"2008"},{"key":"2021042218044910500_bib50","first-page":"523","article-title":"Reading level assessment using support vector machines and statistical language models","volume-title":"Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics","author":"Schwarm","year":"2005"},{"key":"2021042218044910500_bib51","first-page":"49","article-title":"A two-stage approach for generating unbiased estimates of text complexity","volume-title":"Proceedings of the Workshop on Natural Language Processing for Improving Textual Accessibility","author":"Sheehan","year":"2013"},{"issue":"2","key":"2021042218044910500_bib52","doi-asserted-by":"crossref","first-page":"i","DOI":"10.1002\/j.2333-8504.2010.tb02235.x","article-title":"Generating automated text complexity classifications that are aligned with targeted text complexity standards","volume":"2010","author":"Sheehan","year":"2010","journal-title":"ETS Research Report Series"},{"issue":"2","key":"2021042218044910500_bib53","doi-asserted-by":"crossref","first-page":"184","DOI":"10.1086\/678294","article-title":"The TextEvaluator tool: Helping teachers and test developers select texts for use in instruction and assessment","volume":"115","author":"Sheehan","year":"2014","journal-title":"The Elementary School Journal"},{"issue":"1","key":"2021042218044910500_bib54","first-page":"198","article-title":"Predicting Slovene text complexity using readability measures","volume":"59","author":"\u0160kvorc","year":"2019","journal-title":"Contributions to Contemporary History (Spec. Issue on Digital Humanities and Language Technologies"},{"key":"2021042218044910500_bib55","first-page":"1","article-title":"Automated readability index","volume-title":"AMRL-TR. Aerospace Medical Research Laboratories (US)","author":"Smith","year":"1967"},{"key":"2021042218044910500_bib56","first-page":"987","article-title":"Are cohesive features relevant for text readability evaluation?","volume-title":"Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers","author":"Todirascu","year":"2016"},{"key":"2021042218044910500_bib57","doi-asserted-by":"crossref","first-page":"104","DOI":"10.1007\/978-3-030-58323-1_11","article-title":"FinEst BERT and CroSloEngual BERT","volume-title":"International Conference on Text, Speech, and Dialogue","author":"Ul\u010dar","year":"2020"},{"key":"2021042218044910500_bib58","doi-asserted-by":"crossref","first-page":"297","DOI":"10.18653\/v1\/W18-0535","article-title":"OneStopEnglish corpus: A new corpus for automatic readability assessment and text simplification","volume-title":"Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications","author":"Vajjala","year":"2018"},{"key":"2021042218044910500_bib59","first-page":"163","article-title":"On improving the accuracy of readability classification using insights from second language acquisition","volume-title":"Proceedings of the Seventh Workshop on Building Educational Applications Using NLP","author":"Vajjala","year":"2012"},{"key":"2021042218044910500_bib60","volume-title":"Text and Context: Explorations in the Semantics and Pragmatics of Discourse","author":"Van Dijk","year":"1977"},{"key":"2021042218044910500_bib61","first-page":"5998","article-title":"Attention is all you need","volume-title":"Advances in Neural Information Processing Systems","author":"Vaswani","year":"2017"},{"key":"2021042218044910500_bib62","article-title":"Linformer: Self-attention with linear complexity","author":"Wang","year":"2020","journal-title":"arXiv preprint arXiv:2006.04768"},{"key":"2021042218044910500_bib63","first-page":"1995","article-title":"Dueling network architectures for deep reinforcement learning","volume-title":"International Conference on Machine Learning","author":"Wang","year":"2016"},{"issue":"3","key":"2021042218044910500_bib64","doi-asserted-by":"crossref","first-page":"327","DOI":"10.1093\/ijl\/ecl017","article-title":"Michael Hoey. Lexical priming: A new theory of words and language.","volume":"19","author":"Williams","year":"2006","journal-title":"International Journal of Lexicography"},{"key":"2021042218044910500_bib65","doi-asserted-by":"crossref","first-page":"12","DOI":"10.18653\/v1\/W16-0502","article-title":"Text readability assessment for second language learners","volume-title":"Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications","author":"Xia","year":"2016"},{"key":"2021042218044910500_bib66","first-page":"2048","article-title":"Show, attend and tell: Neural image caption generation with visual attention","author":"Xu","year":"2015","journal-title":"International Conference on Machine Learning"},{"issue":"1","key":"2021042218044910500_bib67","doi-asserted-by":"crossref","first-page":"283","DOI":"10.1162\/tacl_a_00139","article-title":"Problems in current text simplification research: New data can help","volume":"3","author":"Xu","year":"2015","journal-title":"Transactions of the Association of Computational Linguistics"},{"key":"2021042218044910500_bib68","first-page":"1480","article-title":"Hierarchical attention networks for document classification","volume-title":"Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Yang","year":"2016"},{"key":"2021042218044910500_bib69","first-page":"649","article-title":"Character-level convolutional networks for text classification","volume-title":"Advances in Neural Information Processing Systems","author":"Zhang","year":"2015"},{"key":"2021042218044910500_bib70","first-page":"19","article-title":"Aligning books and movies: Towards story-like visual explanations by watching movies and reading books","volume-title":"Proceedings of the IEEE International Conference on Computer Vision","author":"Zhu","year":"2015"},{"key":"2021042218044910500_bib71","first-page":"131","article-title":"Ugotavljanje avtorstva besedil: primer \u201ctrenirkarjev.\u201d","volume-title":"Language Technologies: Proceedings of the 17th International Multiconference Information Society - IS 2014","author":"Zwitter Vitez","year":"2014"}],"container-title":["Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/direct.mit.edu\/coli\/article-pdf\/47\/1\/141\/1911429\/coli_a_00398.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/direct.mit.edu\/coli\/article-pdf\/47\/1\/141\/1911429\/coli_a_00398.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,4,22]],"date-time":"2021-04-22T23:08:53Z","timestamp":1619132933000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/coli\/article\/47\/1\/141\/97334\/Supervised-and-Unsupervised-Neural-Approaches-to"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,3]]},"references-count":71,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2021,4,21]]},"published-print":{"date-parts":[[2021,4,21]]}},"URL":"https:\/\/doi.org\/10.1162\/coli_a_00398","relation":{},"ISSN":["0891-2017","1530-9312"],"issn-type":[{"value":"0891-2017","type":"print"},{"value":"1530-9312","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021,3]]},"published":{"date-parts":[[2021,3]]}}}