{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,17]],"date-time":"2025-10-17T14:09:30Z","timestamp":1760710170117},"reference-count":83,"publisher":"MIT Press - Journals","issue":"2","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Computational Linguistics"],"published-print":{"date-parts":[[2020,6]]},"abstract":"<jats:p> Despite an ever-growing number of word representation models introduced for a large number of languages, there is a lack of a standardized technique to provide insights into what is captured by these models. Such insights would help the community to get an estimate of the downstream task performance, as well as to design more informed neural architectures, while avoiding extensive experimentation that requires substantial computational resources not all researchers have access to. A recent development in NLP is to use simple classification tasks, also called probing tasks, that test for a single linguistic feature such as part-of-speech. Existing studies mostly focus on exploring the linguistic information encoded by the continuous representations of English text. However, from a typological perspective the morphologically poor English is rather an outlier: The information encoded by the word order and function words in English is often stored on a subword, morphological level in other languages. To address this, we introduce 15 type-level probing tasks such as case marking, possession, word length, morphological tag count, and pseudoword identification for 24 languages. We present a reusable methodology for creation and evaluation of such tests in a multilingual setting, which is challenging because of a lack of resources, lower quality of tools, and differences among languages. We then present experiments on several diverse multilingual word embedding models, in which we relate the probing task performance for a diverse set of languages to a range of five classic NLP tasks: POS-tagging, dependency parsing, semantic role labeling, named entity recognition, and natural language inference. We find that a number of probing tests have significantly high positive correlation to the downstream tasks, especially for morphologically rich languages. We show that our tests can be used to explore word embeddings or black-box neural models for linguistic cues in a multilingual setting. We release the probing data sets and the evaluation suite LINSPECTOR with https:\/\/github.com\/UKPLab\/linspector . <\/jats:p>","DOI":"10.1162\/coli_a_00376","type":"journal-article","created":{"date-parts":[[2020,3,23]],"date-time":"2020-03-23T20:08:32Z","timestamp":1584994112000},"page":"335-385","source":"Crossref","is-referenced-by-count":6,"title":["LINSPECTOR: Multilingual Probing Tasks for Word Representations"],"prefix":"10.1162","volume":"46","author":[{"given":"G\u00f6zde G\u00fcl","family":"\u015eahin","sequence":"first","affiliation":[{"name":"AIPHES and UKP Lab \/ TU Darmstadt, Technische Universit\u00e4t Darmstadt, Department of Computer Science."}]},{"given":"Clara","family":"Vania","sequence":"additional","affiliation":[{"name":"New York University."}]},{"given":"Ilia","family":"Kuznetsov","sequence":"additional","affiliation":[{"name":"AIPHES and UKP Lab \/ TU Darmstadt."}]},{"given":"Iryna","family":"Gurevych","sequence":"additional","affiliation":[{"name":"AIPHES and UKP Lab \/ TU Darmstadt."}]}],"member":"281","reference":[{"key":"bib1","first-page":"1","volume-title":"International Conference on Learning Representations","author":"Adi Yossi","year":"2017"},{"key":"bib2","doi-asserted-by":"publisher","DOI":"10.4103\/2229-3485.192046"},{"key":"bib3","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-2049"},{"key":"bib4","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-1080"},{"key":"bib5","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00254"},{"key":"bib6","doi-asserted-by":"publisher","DOI":"10.3115\/1613715.1613756"},{"key":"bib7","first-page":"104","volume-title":"Proceedings of the KONVENS GermEval Shared Task on Named Entity Recognition","author":"Benikova Darina","year":"2014"},{"key":"bib8","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1313"},{"key":"bib9","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9781139164894"},{"key":"bib10","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00051"},{"key":"bib11","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D15-1075"},{"key":"bib12","first-page":"136","volume-title":"50th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, July 8-14, 2012","author":"Bruni Elia","year":"2012"},{"key":"bib13","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/S17-2002"},{"key":"bib14","first-page":"55","volume-title":"Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies","author":"Che Wanxiang","year":"2018"},{"key":"bib15","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-1152"},{"key":"bib16","unstructured":"Comrie, Bernard and Maria Polinsky. 1998. The great Daghestanian case hoax. In A. Siewierska and J. Jung Song, eds. Case, Typology and Grammar, pages 95\u2013114."},{"key":"bib17","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-1198"},{"key":"bib18","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1269"},{"key":"bib19","volume-title":"The World Atlas of Language Structures Online","author":"Corbett Greville G.","year":"2013"},{"key":"bib20","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00271"},{"key":"bib21","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N18-2085"},{"key":"bib22","doi-asserted-by":"publisher","DOI":"10.1007\/s10579-017-9390-y"},{"key":"bib23","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-1036"},{"key":"bib24","first-page":"1","volume-title":"5th International Conference on Learning Representations, ICLR 2017, Conference Track Proceedings","author":"Dozat Timothy","year":"2017"},{"key":"bib25","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-3022"},{"key":"bib26","first-page":"2106","volume-title":"Proceedings of the Ninth International Conference on Language Resources and Evaluation, LREC 2014","author":"Erten Begum","year":"2014"},{"key":"bib27","doi-asserted-by":"publisher","DOI":"10.1162\/coli.2008.07-017-R1-06-83"},{"key":"bib28","first-page":"271","volume-title":"Proceedings of the 21st Nordic Conference on Computational Linguistics","author":"Fares Murhaf","year":"2017"},{"key":"bib29","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/N15-1184"},{"key":"bib30","doi-asserted-by":"publisher","DOI":"10.1145\/371920.372094"},{"issue":"2","key":"bib31","first-page":"23","volume":"12","author":"Gage Philip","year":"1994","journal-title":"C Users Journal"},{"key":"bib32","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W18-2501"},{"key":"bib33","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00032"},{"key":"bib34","first-page":"413","volume-title":"Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)","author":"Ghaddar Abbas","year":"2017"},{"key":"bib35","doi-asserted-by":"publisher","DOI":"10.3115\/1596409.1596411"},{"key":"bib36","doi-asserted-by":"publisher","DOI":"10.1007\/s10579-015-9310-y"},{"key":"bib37","first-page":"2989","volume-title":"Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)","author":"Heinzerling Benjamin","year":"2018"},{"key":"bib38","doi-asserted-by":"publisher","DOI":"10.1162\/COLI_a_00237"},{"key":"bib39","first-page":"315","volume-title":"Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Hohensee Matt","year":"2012"},{"key":"bib40","first-page":"873","volume-title":"Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Huang Eric","year":"2012"},{"key":"bib41","author":"Huang Zhiheng","year":"2015","journal-title":"CoRR, arXiv preprint arXiv: 1508.01991"},{"key":"bib42","volume-title":"The World Atlas of Language Structures Online","author":"Iggesen Oliver A.","year":"2013"},{"key":"bib43","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/W14-5807"},{"key":"bib44","doi-asserted-by":"publisher","DOI":"10.3758\/BRM.42.3.627"},{"key":"bib45","first-page":"2741","volume-title":"Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence","author":"Kim Yoon","year":"2016"},{"key":"bib46","first-page":"1868","volume-title":"Proceedings of the Eleventh International Conference on Language Resources and Evaluation, LREC 2018","author":"Kirov Christo","year":"2018"},{"key":"bib47","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D15-1246"},{"key":"bib48","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W16-2512"},{"key":"bib49","first-page":"1","volume-title":"6th International Conference on Learning Representations, ICLR 2018, Conference Track Proceedings","author":"Lample Guillaume","year":"2018"},{"key":"bib50","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P14-2050"},{"key":"bib51","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D15-1176"},{"key":"bib52","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W16-2503"},{"key":"bib53","first-page":"104","volume-title":"Proceedings of the Seventeenth Conference on Computational Natural Language Learning, CoNLL 2013","author":"Luong Thang","year":"2013"},{"key":"bib54","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W18-6011"},{"key":"bib55","doi-asserted-by":"publisher","DOI":"10.3115\/1219840.1219852"},{"key":"bib56","first-page":"1","volume-title":"1st International Conference on Learning Representations, ICLR 2013, Workshop Track Proceedings","author":"Mikolov Tomas","year":"2013"},{"key":"bib57","first-page":"3111","volume-title":"Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2","author":"Mikolov Tomas","year":"2013"},{"key":"bib58","doi-asserted-by":"publisher","DOI":"10.1080\/01690969108406936"},{"key":"bib59","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-5301"},{"key":"bib60","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W16-2504"},{"key":"bib61","unstructured":"Nivre, Joakim, Mitchell Abrams, \u017eeljko Agi\u0107, Lars Ahrenberg, Lene Antonsen, Katya Aplonova, and Maria Jesus Aranzabe. 2018. Universal dependencies 2.3. LINDAT\/CLARIN digital library at the Institute of Formal and Applied Linguistics (\u00daFAL), Faculty of Mathematics and Physics, Charles University."},{"key":"bib62","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/D14-1162"},{"key":"bib63","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N18-1202"},{"key":"bib64","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-1170"},{"key":"bib65","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P16-1140"},{"key":"bib66","first-page":"2690","volume-title":"Proceedings of the 27th International Conference on Computational Linguistics","author":"Rogers Anna","year":"2018"},{"key":"bib67","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/S17-1017"},{"key":"bib68","doi-asserted-by":"publisher","DOI":"10.1145\/365628.365657"},{"key":"bib69","doi-asserted-by":"publisher","DOI":"10.1613\/jair.1.11640"},{"key":"bib71","doi-asserted-by":"publisher","DOI":"10.3115\/1119176.1119195"},{"key":"bib72","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D15-1036"},{"key":"bib73","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P16-1162"},{"key":"bib74","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D16-1159"},{"key":"bib75","unstructured":"Sylak-Glassman, John. 2016. The composition and use of the universal morphological feature schema (UniMorph schema). Technical report, Center for Language and Speech Processing, Johns Hopkins University."},{"key":"bib76","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/P15-2111"},{"key":"bib77","first-page":"1","volume-title":"Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP","author":"Tal Linzen Tal","year":"2018"},{"key":"bib78","first-page":"1","volume-title":"International Conference on Learning Representations","author":"Tenney Ian","year":"2019"},{"key":"bib79","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D15-1243"},{"key":"bib80","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1278"},{"key":"bib81","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P17-1184"},{"key":"bib82","first-page":"69","volume-title":"CoCo@NIPS","author":"Veldhoen Sara","year":"2016"},{"key":"bib83","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N18-1101"},{"key":"bib84","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D17-1312"}],"container-title":["Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/coli_a_00376","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,3,12]],"date-time":"2021-03-12T21:28:33Z","timestamp":1615584513000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/coli\/article\/46\/2\/335-385\/93365"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,6]]},"references-count":83,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2020,6]]}},"alternative-id":["10.1162\/coli_a_00376"],"URL":"https:\/\/doi.org\/10.1162\/coli_a_00376","relation":{},"ISSN":["0891-2017","1530-9312"],"issn-type":[{"value":"0891-2017","type":"print"},{"value":"1530-9312","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,6]]}}}