{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,9]],"date-time":"2025-12-09T06:14:40Z","timestamp":1765260880627,"version":"3.46.0"},"reference-count":81,"publisher":"Springer Science and Business Media LLC","issue":"4","license":[{"start":{"date-parts":[[2025,6,28]],"date-time":"2025-06-28T00:00:00Z","timestamp":1751068800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,6,28]],"date-time":"2025-06-28T00:00:00Z","timestamp":1751068800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"ILC - PISA"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Lang Resources &amp; Evaluation"],"published-print":{"date-parts":[[2025,12]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>The paper introduces Parallel Trees, a novel multilingual treebank collection that includes 20 treebanks for 10 languages. The distinguishing property of this resource is that the sentences of each language are annotated using two syntactic representation paradigms (SRPs), respectively based on the notions of dependency and constituency. By aligning the annotations of existing resources, Parallel Trees represents an example of exploiting pre-existing treebanks to adapt them to novel applications. To illustrate its potential, we present a case study where the resource is employed as a benchmark to investigate whether and how BERT, one of the first prominent neural language models (NLMs), is sensitive to the dependency- and constituency-based approaches for representing the syntactic structure of a sentence. The case study results indicate that the model\u2019s sensitivity fluctuates across languages and experimental settings. The unique nature of the Parallel Trees resource creates the prerequisites for innovative studies comparing dependency and phrase-structure trees, allowing for more focused investigations without the interference of lexical variation.<\/jats:p>","DOI":"10.1007\/s10579-025-09826-3","type":"journal-article","created":{"date-parts":[[2025,6,27]],"date-time":"2025-06-27T22:51:44Z","timestamp":1751064704000},"page":"3445-3485","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Parallel Trees: a novel resource with aligned dependency\nand constituency syntactic representations"],"prefix":"10.1007","volume":"59","author":[{"given":"Chiara","family":"Alzetta","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alessio","family":"Miaschi","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Felice","family":"Dell\u2019Orletta","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Giulia","family":"Venturi","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Simonetta","family":"Montemagni","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2025,6,28]]},"reference":[{"key":"9826_CR1","unstructured":"Abeill\u00e9, A., Cl\u00e9ment, L., & Kinyon, A. (2000) Building a treebank for French. In Proceedings of the second international conference on language resources and evaluation (LREC\u201900). European Language Resources Association (ELRA)"},{"key":"#cr-split#-9826_CR2.1","unstructured":"Afonso, S., Bick, E., Haber, R. (2002) Floresta sint\u00e1(c)tica: a treebank for portuguese. In Manuel Gonz\u00e1lez Rodrigues"},{"key":"#cr-split#-9826_CR2.2","unstructured":"Carmen Paz Suarez Araujo (ed) Proceedings of the third international conference on language resources and evaluation (LREC 2002)(Las Palmas de Gran Canaria Espanha 29-31 de Maio de 2002) ELRA"},{"issue":"11","key":"9826_CR3","doi-asserted-by":"publisher","first-page":"1585","DOI":"10.3844\/jcssp.2020.1585.1597","volume":"16","author":"I Alfina","year":"2020","unstructured":"Alfina, I., Budi, I., & Suhartanto, H. (2020). Tree rotations for dependency trees: Converting the head-directionality of noun phrases. Journal of Computer Science, 16(11), 1585\u20131597.","journal-title":"Journal of Computer Science"},{"key":"9826_CR4","first-page":"91","volume":"57","author":"HM Alonso","year":"2016","unstructured":"Alonso, H. M., & Zeman, D. (2016). Universal dependencies for the ancora treebanks. Procesamiento del Lenguaje Natural, 57, 91\u201398.","journal-title":"Procesamiento del Lenguaje Natural"},{"key":"9826_CR5","unstructured":"Alzetta, C., Dell\u2019Orletta, F., Montemagni, S., et\u00a0al (2017) Dangerous relations in dependency treebanks. In Proceedings of the 16th international workshop on treebanks and linguistic theories, pp 201\u2013210"},{"key":"9826_CR6","unstructured":"Arnard\u00f3ttir, \u00de., Hafsteinsson, H., Sigur\u00f0sson, EF., et\u00a0al. (2020) A universal dependencies conversion pipeline for a penn-format constituency treebank. In Proceedings of the fourth workshop on universal dependencies (UDW 2020) (pp 16\u201325). Association for Computational Linguistics"},{"key":"9826_CR7","doi-asserted-by":"publisher","unstructured":"Arps, D., Samih, Y., Kallmeyer, L., et\u00a0al (2022) Probing for constituency structure in neural language models. In: Goldberg Y, Kozareva Z, Zhang Y (eds) Findings of the association for computational linguistics: EMNLP 2022 (pp 6738\u20136757). Association for Computational Linguistics, https:\/\/doi.org\/10.18653\/v1\/2022.findings-emnlp.502","DOI":"10.18653\/v1\/2022.findings-emnlp.502"},{"key":"9826_CR8","doi-asserted-by":"publisher","unstructured":"Arps, D., Kallmeyer, L., Samih, Y., et\u00a0al. (2024) Multilingual nonce dependency treebanks: Understanding how language models represent and process syntactic structure. In Duh K, Gomez H, Bethard S (eds) Proceedings of the 2024 conference of the North American chapter of the association for computational linguistics: Human language technologies (Volume 1: Long Papers). Association for Computational Linguistics, pp 7822\u20137844, https:\/\/doi.org\/10.18653\/v1\/2024.naacl-long.433,","DOI":"10.18653\/v1\/2024.naacl-long.433"},{"key":"9826_CR9","doi-asserted-by":"crossref","unstructured":"Arwidarasti, JN., Alfina, I., & Krisnadhi, AA. (2019) Converting an indonesian constituency treebank to the Penn Treebank format. In 2019 International conference on Asian language processing (IALP) (pp 331\u2013336), IEEE","DOI":"10.1109\/IALP48816.2019.9037723"},{"key":"9826_CR10","unstructured":"Beeching, E., Fourrier, C., Habib, N., et\u00a0al (2023) Open llm leaderboard. https:\/\/huggingface.co\/spaces\/open-llm-leaderboard\/open_llm_leaderboard"},{"key":"9826_CR11","first-page":"8","volume-title":"Computational linguistics in the Netherlands","author":"L Van der Beek","year":"2002","unstructured":"Van der Beek, L., Bouma, G., Malouf, R., & Van Noord, G. (2002). The Alpino dependency treebank. Computational linguistics in the Netherlands (pp. 8\u201322). Leiden: Brill."},{"issue":"1","key":"9826_CR12","doi-asserted-by":"publisher","first-page":"207","DOI":"10.1162\/coli_a_00422","volume":"48","author":"Y Belinkov","year":"2022","unstructured":"Belinkov, Y. (2022). Probing classifiers: promises, shortcomings, and advances. Comput Linguist, 48(1), 207\u2013219. https:\/\/doi.org\/10.1162\/coli_a_00422","journal-title":"Comput Linguist"},{"key":"9826_CR13","unstructured":"Belinkov, Y., M\u00e0rquez, L., Sajjad, H., Durrani, N., Dalvi, F., & Glass, J. (2017). Evaluating layers of representation in neural machine translation on part-of-speech and semantic tagging tasks. In Proceedings of the eighth international joint conference on natural language processing (Volume 1: Long Papers). Asian Federation of Natural Language Processing, pp 1\u201310"},{"key":"9826_CR14","unstructured":"Bick, E. (2016) Constraint grammar-based conversion of dependency treebanks. In Proceedings of the 13th international conference on natural language processing. NLP Association of India, pp. 109\u2013114"},{"key":"9826_CR15","doi-asserted-by":"publisher","unstructured":"Blevins, T., Levy, O., & Zettlemoyer, L. (2018) Deep RNNs encode soft hierarchical syntax. In Proceedings of the 56th annual meeting of the association for computational linguistics (Volume 2: Short Papers). Association for Computational Linguistics, pp 14\u2013https:\/\/doi.org\/10.18653\/v1\/P18-2003","DOI":"10.18653\/v1\/P18-2003"},{"key":"9826_CR16","doi-asserted-by":"publisher","DOI":"10.1002\/9781119428589","volume-title":"Application of Graph Rewriting to Natural Language Processing","author":"G Bonfante","year":"2018","unstructured":"Bonfante, G., Guillaume, B., & Perrier, G. (2018). Application of Graph Rewriting to Natural Language Processing. John Wiley & Sons."},{"key":"9826_CR17","doi-asserted-by":"crossref","unstructured":"Bosco, C. (2007) Multiple-step treebank conversion: From dependency to Penn format. In Proceedings of the linguistic annotation workshop. Association for Computational Linguistics, pp. 164\u2013167","DOI":"10.3115\/1642059.1642085"},{"key":"9826_CR18","unstructured":"Bosco, C., Lombardo, V., Vassallo, D., & Lesmo, L. (2000). Building a treebank for Italian: A data-driven annotation schema. In Proceedings of the second international conference on language resources and evaluation (LREC\u201900). European Language Resources Association (ELRA)"},{"key":"9826_CR19","unstructured":"Bosco, C., Sanguinetti, M., Lesmo, L., et\u00a0al (2012) The parallel-tut: A multilingual and multiformat treebank. In Proceedings of the eight international conference on language resources and evaluation (LREC\u201912), European Language Resources Association (ELRA) (pp 1932\u20131938)"},{"key":"9826_CR20","unstructured":"Bosco, C., Montemagni, S., & Simi, M. (2013) Converting Italian treebanks: Towards an Italian Stanford dependency treebank. In Proceedings of the 7th linguistic annotation workshop and interoperability with discourse. Association for Computational Linguistics (pp 61\u201369)"},{"key":"9826_CR21","unstructured":"Bouma, G., & van Noord, G. (2017) Increasing return on annotation investment: The automatic construction of a Universal Dependency treebank for Dutch. In Proceedings of the NoDaLiDa 2017 Workshop on Universal Dependencies (UDW 2017). Association for Computational Linguistics, Gothenburg, Sweden, pp 19\u201326"},{"key":"9826_CR22","doi-asserted-by":"publisher","first-page":"141","DOI":"10.1007\/11816508_16","volume-title":"Advances in Natural Language Processing","author":"M Civit","year":"2006","unstructured":"Civit, M., Mart\u00ed, M. A., Buf\u00ed, N., et al. (2006). Cat3lb and cast3lb: From constituents to dependencies. In T. Salakoski, F. Ginter, & S. Pyysalo (Eds.), Advances in Natural Language Processing (pp. 141\u2013152). Springer."},{"key":"9826_CR23","doi-asserted-by":"publisher","unstructured":"Conneau, A., Kruszewski, G., Lample, G., et\u00a0al (2018) What you can cram into a single \\$ &!#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th annual meeting of the association for computational linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Melbourne, Australia, pp 2126\u20132136, https:\/\/doi.org\/10.18653\/v1\/P18-1198","DOI":"10.18653\/v1\/P18-1198"},{"key":"9826_CR24","doi-asserted-by":"publisher","unstructured":"Devlin, J., Chang, MW., Lee, K., et\u00a0al. (2019) BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: Human language technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, pp 4171\u20134186, https:\/\/doi.org\/10.18653\/v1\/N19-1423","DOI":"10.18653\/v1\/N19-1423"},{"key":"9826_CR25","doi-asserted-by":"crossref","unstructured":"Dinakaramani, A., Rashel, F., Luthfi, A., & Manurung, R. (2014). Designing an Indonesian part of speech tagset and manually tagged Indonesian corpus. In 2014 International conference on Asian language processing (IALP), IEEE, pp 66\u201369","DOI":"10.1109\/IALP.2014.6973519"},{"key":"9826_CR26","unstructured":"Dryer, M.S, & Haspelmath, M. (eds) (2013) WALS Online. Max Planck Institute for Evolutionary Anthropology"},{"key":"9826_CR27","doi-asserted-by":"publisher","first-page":"34","DOI":"10.1162\/tacl_a_00298","volume":"8","author":"A Ettinger","year":"2020","unstructured":"Ettinger, A. (2020). What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8, 34\u201348.","journal-title":"Transactions of the Association for Computational Linguistics"},{"issue":"3","key":"9826_CR28","doi-asserted-by":"publisher","first-page":"304","DOI":"10.1016\/S0019-9958(65)90232-9","volume":"8","author":"H Gaifman","year":"1965","unstructured":"Gaifman, H. (1965). Dependency systems and phrase-structure systems. Information and control, 8(3), 304\u2013337.","journal-title":"Information and control"},{"key":"9826_CR29","doi-asserted-by":"publisher","unstructured":"Gauthier, J., Hu, J., Wilcox, E., Qian, P., & Levy, R. (2020) SyntaxGym: An online platform for targeted evaluation of language models. In Celikyilmaz A, Wen TH (eds) Proceedings of the 58th annual meeting of the association for computational linguistics: System demonstrations. Association for Computational Linguistics, Online, pp 70\u201376, https:\/\/doi.org\/10.18653\/v1\/2020.acl-demos.10,","DOI":"10.18653\/v1\/2020.acl-demos.10"},{"key":"9826_CR30","doi-asserted-by":"publisher","unstructured":"Gerdes, K., Guillaume, B., Kahane, S., & Perrier, G. (2018). SUD or surface-syntactic universal dependencies: An annotation scheme near-isomorphic to UD. In: Proceedings of the second workshop on universal dependencies (UDW 2018). Association for Computational Linguistics, pp 66\u201374, https:\/\/doi.org\/10.18653\/v1\/W18-6008","DOI":"10.18653\/v1\/W18-6008"},{"issue":"4","key":"9826_CR31","doi-asserted-by":"publisher","first-page":"511","DOI":"10.2307\/411934","volume":"40","author":"DG Hays","year":"1964","unstructured":"Hays, D. G. (1964). Dependency theory: A formalism and some observations. Language, 40(4), 511\u2013525.","journal-title":"Language"},{"key":"9826_CR32","doi-asserted-by":"publisher","unstructured":"Hewitt, J., & Liang, P. (2019) Designing and interpreting probes with control tasks. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP). Association for Computational Linguistics, pp 2733\u20132743, https:\/\/doi.org\/10.18653\/v1\/D19-1275","DOI":"10.18653\/v1\/D19-1275"},{"key":"9826_CR33","doi-asserted-by":"publisher","unstructured":"Hewitt, J., & Manning, CD. (2019). A structural probe for finding syntax in word representations. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: Human language technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics (pp 4129\u20134138), https:\/\/doi.org\/10.18653\/v1\/N19-1419","DOI":"10.18653\/v1\/N19-1419"},{"key":"9826_CR34","doi-asserted-by":"publisher","DOI":"10.1515\/ling.1980.18.3-4.179","volume-title":"Constituency and dependency","author":"RA Hudson","year":"1980","unstructured":"Hudson, R. A. (1980). Constituency and dependency. Walter de Gruyter."},{"key":"9826_CR35","doi-asserted-by":"publisher","unstructured":"Jawahar, G., Sagot, B., & Seddah, D. (2019) What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, pp 3651\u20133657, https:\/\/doi.org\/10.18653\/v1\/P19-1356","DOI":"10.18653\/v1\/P19-1356"},{"key":"9826_CR36","doi-asserted-by":"crossref","unstructured":"Kara, N., Mar\u015fan, B., \u00d6z\u00e7elik, M., et\u00a0al (2020) Creating a syntactically felicitous constituency treebank for turkish. In: 2020 Innovations in intelligent systems and applications conference (ASYU) (pp 1\u20136), IEEE","DOI":"10.1109\/ASYU50717.2020.9259873"},{"key":"9826_CR37","doi-asserted-by":"crossref","unstructured":"Kogkalidis, K., & Wijnholds, G. (2022). Discontinuous constituency and BERT: A case study of Dutch. Findings of the Association for Computational Linguistics: ACL 2022 (pp. 3776\u20133785). Association for Computational Linguistics.","DOI":"10.18653\/v1\/2022.findings-acl.298"},{"key":"9826_CR38","doi-asserted-by":"publisher","first-page":"11","DOI":"10.1007\/978-3-031-02131-2_2","volume-title":"Dependency parsing","author":"S K\u00fcbler","year":"2009","unstructured":"K\u00fcbler, S., McDonald, R., & Nivre, J. (2009). Dependency parsing. Dependency parsing (pp. 11\u201320). Cham p: Springer."},{"key":"9826_CR39","doi-asserted-by":"publisher","unstructured":"Kulmizev, A., Ravishankar, V., Abdou, M., et\u00a0al (2020) Do neural language models show preferences for syntactic formalisms? In Proceedings of the 58th annual meeting of the association for computational linguistics. association for computational linguistics, Online, pp 4077\u20134091, https:\/\/doi.org\/10.18653\/v1\/2020.acl-main.375","DOI":"10.18653\/v1\/2020.acl-main.375"},{"key":"9826_CR40","doi-asserted-by":"crossref","unstructured":"Kuzgun, A., Y\u0131ld\u0131z OK,, Cesur N., et\u00a0al. (2021) From constituency to UD-style dependency: Building the first conversion tool of Turkish. In Proceedings of the international conference on recent advances in natural language processing (RANLP 2021). INCOMA Ltd., Held Online, pp 761\u2013769","DOI":"10.26615\/978-954-452-072-4_087"},{"key":"9826_CR41","unstructured":"Liang, P., Bommasani, R., Lee, T., et\u00a0al. (2023). Holistic evaluation of language models. Transactions on Machine Learning Research https:\/\/openreview.net\/forum?id=iO4LZibEqW, featured Certification, Expert Certification"},{"key":"9826_CR42","unstructured":"Linh, HM., Huyen, NTM., Luong, VX., et\u00a0al (2020) VLSP 2020 shared task: Universal Dependency parsing for Vietnamese. In Proceedings of the 7th International workshop on Vietnamese language and speech processing. Association for Computational Lingustics (pp. 77\u201383)"},{"issue":"1","key":"9826_CR43","doi-asserted-by":"publisher","first-page":"152","DOI":"10.1162\/nol_a_00029","volume":"2","author":"A Lopopolo","year":"2021","unstructured":"Lopopolo, A., van den Bosch, A., Petersson, K. M., et al. (2021). Distinguishing syntactic operations in the brain: Dependency and phrase-structure parsing. Neurobiology of Language, 2(1), 152\u2013175.","journal-title":"Neurobiology of Language"},{"key":"9826_CR44","doi-asserted-by":"crossref","unstructured":"Marcus, M., Kim, G., Marcinkiewicz, MA., MacIntyre, R., Bies, A., Ferguson, M., & Schasberger, B. (1994). The Penn treebank: Annotating predicate argument structure. In Human language technology: Proceedings of a workshop held at plainsboro, March 8-11, 1994","DOI":"10.3115\/1075812.1075835"},{"issue":"2","key":"9826_CR45","doi-asserted-by":"publisher","first-page":"255","DOI":"10.1162\/coli_a_00402","volume":"47","author":"MC de Marneffe","year":"2021","unstructured":"de Marneffe, M. C., Manning, C. D., Nivre, J., et al. (2021). Universal Dependencies. Computational Linguistics, 47(2), 255\u2013308. https:\/\/doi.org\/10.1162\/coli_a_00402","journal-title":"Computational Linguistics"},{"key":"9826_CR46","volume-title":"Syntax","author":"P Matthews","year":"1981","unstructured":"Matthews, P. (1981). Syntax. Cambridge University Press."},{"key":"9826_CR47","doi-asserted-by":"publisher","unstructured":"Maudslay, R.H., & Cotterell, R. (2021) Do syntactic probes probe syntax? experiments with jabberwocky probing. In Toutanova K, Rumshisky A, Zettlemoyer L, et\u00a0al (eds) Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Online, pp 124\u20131https:\/\/doi.org\/10.18653\/v1\/2021.naacl-main.11","DOI":"10.18653\/v1\/2021.naacl-main.11"},{"key":"9826_CR48","doi-asserted-by":"crossref","unstructured":"Mel\u2019\u010duk, I., & Polgu\u00e8re, A. (2009) Dependency in linguistic description. Dependency in Linguistic Description pp 1\u2013308","DOI":"10.1075\/slcs.111.03mel"},{"key":"9826_CR49","first-page":"303","volume":"8","author":"IA Mel\u2019\u010duk","year":"1988","unstructured":"Mel\u2019\u010duk, I. A., et al. (1988). Dependency syntax: Theory and practice. SUNY Press, 8, 303\u2013332.","journal-title":"SUNY Press"},{"key":"9826_CR50","doi-asserted-by":"publisher","unstructured":"Miaschi, A., Brunato, D., Dell\u2019Orletta, F., & Venturi, G. (2020). Linguistic profiling of a neural language model. In Proceedings of the 28th international conference on computational linguistics. International Committee on Computational Linguistics,(Online), pp 745\u2013756, https:\/\/doi.org\/10.18653\/v1\/2020.coling-main.65","DOI":"10.18653\/v1\/2020.coling-main.65"},{"issue":"1","key":"9826_CR51","doi-asserted-by":"publisher","first-page":"25","DOI":"10.4000\/ijcol.965","volume":"8","author":"A Miaschi","year":"2022","unstructured":"Miaschi, A., Sarti, G., Brunato, D., et al. (2022). Probing linguistic knowledge in Italian neural language models across language varieties. Italian Journal of Computational Linguistics, 8(1), 25\u201344. https:\/\/doi.org\/10.4000\/ijcol.965","journal-title":"Italian Journal of Computational Linguistics"},{"key":"9826_CR52","doi-asserted-by":"publisher","DOI":"10.3390\/info14030144","author":"A Miaschi","year":"2023","unstructured":"Miaschi, A., Alzetta, C., Brunato, D., et al. (2023). Testing the effectiveness of the diagnostic probing paradigm on Italian treebanks. Information. https:\/\/doi.org\/10.3390\/info14030144","journal-title":"Information"},{"key":"9826_CR53","doi-asserted-by":"publisher","unstructured":"M\u00fcller-Eberstein, M., van\u00a0der Goot, R., & Plank, B. (2022) Probing for labeled dependency trees. In Muresan S, Nakov P, Villavicencio A (eds) Proceedings of the 60th annual meeting of the association for computational linguistics (Volume 1: Long Papers). Association for Computational Linguistics, pp 7711\u20137726, https:\/\/doi.org\/10.18653\/v1\/2022.acl-long.532,","DOI":"10.18653\/v1\/2022.acl-long.532"},{"key":"9826_CR54","doi-asserted-by":"crossref","unstructured":"Mu\u00f1oz-Ortiz, A., Vilares, D., & G\u00f3mez-Rodr\u00edguez, C. (2023) Assessment of pre-trained models across languages and grammars. arXiv:2309.11165","DOI":"10.18653\/v1\/2023.ijcnlp-main.23"},{"issue":"7","key":"9826_CR55","doi-asserted-by":"publisher","first-page":"2867","DOI":"10.1007\/s10670-022-00657-0","volume":"89","author":"RM Nefdt","year":"2023","unstructured":"Nefdt, R. M., & Baggio, G. (2023). Notational variants and cognition: The case of dependency grammar. Erkenntnis, 89(7), 2867\u20132897.","journal-title":"Erkenntnis"},{"key":"9826_CR56","first-page":"222","volume-title":"Empirical methods in natural language generation","author":"A Nenkova","year":"2009","unstructured":"Nenkova, A., Chae, J., Louis, A., & Pitler, E. (2009). Structural features for predicting the linguistic quality of text. In V. John (Ed.), Empirical methods in natural language generation (pp. 222\u2013241). Springer."},{"key":"9826_CR57","doi-asserted-by":"crossref","unstructured":"Nguyen, PT., Vu, XL., Nguyen, TMH., et\u00a0al (2009) Building a large syntactically-annotated corpus of Vietnamese. In Proceedings of the third linguistic annotation workshop (LAW III). Association for Computational Linguistics, pp 182\u2013185","DOI":"10.3115\/1698381.1698416"},{"key":"9826_CR58","unstructured":"Nivre, J., de\u00a0Marneffe, MC., Ginter, F., et\u00a0al (2016) Universal Dependencies v1: A multilingual treebank collection. In Calzolari N, Choukri K, Declerck T, et\u00a0al (eds) Proceedings of the tenth international conference on language resources and evaluation (LREC\u201916). European Language Resources Association (ELRA), pp 1659\u20131666, https:\/\/aclanthology.org\/L16-1262"},{"key":"9826_CR59","unstructured":"Nivre, J., de\u00a0Marneffe, MC., Ginter, F., et\u00a0al (2020) Universal Dependencies v2: An evergrowing multilingual treebank collection. In Calzolari N, B\u00e9chet F, Blache P, et\u00a0al (eds) Proceedings of the twelfth language resources and evaluation conference. European Language Resources Association, pp 4034\u20134043, https:\/\/aclanthology.org\/2020.lrec-1.497"},{"key":"9826_CR60","doi-asserted-by":"publisher","unstructured":"Oota, SR., Marreddy, M., Gupta, M., & Bapi, R. (2023). How does the brain process syntactic structure while listening? In Rogers A, Boyd-Graber J, Okazaki N (eds) Findings of the association for computational linguistics: ACL 2023. Association for Computational Linguistics, Toronto, Canada, pp 6624\u20136647, https:\/\/doi.org\/10.18653\/v1\/2023.findings-acl.415","DOI":"10.18653\/v1\/2023.findings-acl.415"},{"key":"9826_CR61","first-page":"604","volume-title":"The Routledge handbook of syntax","author":"T Osborne","year":"2014","unstructured":"Osborne, T. (2014). Dependency grammar. In E. John (Ed.), The Routledge handbook of syntax (pp. 604\u2013626). Routledge."},{"key":"9826_CR62","doi-asserted-by":"publisher","unstructured":"Prange, J., Schneider, N., & Kong, L. (2022). Linguistic frameworks go toe-to-toe at neuro-symbolic language modeling. In Proceedings of the 2022 conference of the north american chapter of the association for computational linguistics: Human language technologies. Association for Computational Linguistics, pp 4375\u20134391, https:\/\/doi.org\/10.18653\/v1\/2022.naacl-main.325","DOI":"10.18653\/v1\/2022.naacl-main.325"},{"key":"9826_CR63","unstructured":"Rademaker, A., Chalub, F., Real, L., et\u00a0al. (2017) Universal dependencies for portuguese. In: Proceedings of the Fourth International Conference on Dependency Linguistics (Depling), pp 197\u2013206"},{"key":"9826_CR64","doi-asserted-by":"publisher","first-page":"167","DOI":"10.1075\/slcs.39.66ram","volume":"39","author":"O Rambow","year":"1997","unstructured":"Rambow, O., & Joshi, A. (1997). A formal look at dependency grammars and phrase-structure grammars, with special consideration of word-order phenomena. Recent trends in meaning-text theory, 39, 167\u2013190.","journal-title":"Recent trends in meaning-text theory"},{"key":"9826_CR65","unstructured":"R\u00f6gnvaldsson, E., Ingason, AK., Sigur\u00f0sson, EF., & Wallenberg, J. (2012) The Icelandic parsed historical corpus (IcePaHC). In: Proceedings of the eighth international conference on language resources and evaluation (LREC\u201912). European Language Resources Association (ELRA), Istanbul, Turkey, pp 1977\u20131984"},{"key":"9826_CR66","doi-asserted-by":"crossref","unstructured":"Seddah, D., Tsarfaty, R., K\u00fcbler, S., Candito, M., Choi, J. D., (2013). Overview of the spmrl 2013 shared task: A cross-framework evaluation of parsing morphologically rich languages. In Proceedings of the fourth workshop on statistical parsing of morphologically-rich languages, Association for Computational Linguistics, pp 146\u2013182","DOI":"10.18653\/v1\/W13-4917"},{"key":"9826_CR67","unstructured":"Seddah, D., de\u00a0la Clergerie, E., Sagot, B., et\u00a0al (2018) Cheating a parser to death: Data-driven cross-treebank annotation transfer. In Proceedings of the eleventh international conference on language resources and evaluation (LREC 2018). European Language Resources Association (ELRA)"},{"key":"9826_CR68","unstructured":"Srivastava, A., Rastogi, A., Rao, A., et\u00a0al. (2023) Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv:2206.04615"},{"key":"9826_CR69","unstructured":"Taul\u00e9, M., Mart\u00ed, MA., & Recasens, M. (2008) AnCora: Multilevel annotated corpora for Catalan and Spanish. In Proceedings of the sixth international conference on language resources and evaluation (LREC\u201908). European Language Resources Association (ELRA)"},{"key":"9826_CR70","unstructured":"Tenney, I., Xia, P., Chen, B., Wang, A., Poliak, A., McCoy, R. T., & Pavlick, E. (2019). What do you learn from context? probing for sentence structure in contextualized word representations. In Proceedings of the 7th international conference on learning representations (ICLR 2019)"},{"key":"9826_CR71","doi-asserted-by":"crossref","unstructured":"Vilares, D., Strzyz, M., S\u00f8gaard, A. (2020). Parsing as pretraining. In Proceedings of the AAAI Conference on Artificial Intelligence, pp 9114\u20139121","DOI":"10.1609\/aaai.v34i05.6446"},{"key":"9826_CR72","unstructured":"Wagner, J., Zarrie\u00df, S. (2023). Probing BERT\u2019s ability to encode sentence modality and modal verb sense across varieties of English. In Amblard M, Breitholtz E (eds) Proceedings of the 15th International Conference on Computational Semantics. Association for Computational Linguistics, pp 28\u201338, https:\/\/aclanthology.org\/2023.iwcs-1.3"},{"key":"9826_CR73","doi-asserted-by":"crossref","unstructured":"Waldis, A., Perlitz, Y., Choshen, L., Hou, Y., & Gurevych, I.(2024). Holmes: Benchmark the linguistic competence of language models. arXiv:2404.18923","DOI":"10.1162\/tacl_a_00718"},{"key":"9826_CR74","doi-asserted-by":"publisher","unstructured":"Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2018). GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. Association for Computational Linguistics, pp 353\u2013355, https:\/\/doi.org\/10.18653\/v1\/W18-5446","DOI":"10.18653\/v1\/W18-5446"},{"key":"9826_CR75","volume-title":"SuperGLUE: a stickier benchmark for general-purpose language understanding systems","author":"A Wang","year":"2019","unstructured":"Wang, A., Pruksachatkun, Y., Nangia, N., et al. (2019). SuperGLUE: a stickier benchmark for general-purpose language understanding systems. Curran Associates Inc."},{"key":"9826_CR76","doi-asserted-by":"publisher","unstructured":"Warstadt, A., Cao, Y., Grosu, I., et\u00a0al. (2019a) Investigating BERT\u2019s knowledge of language: Five analysis methods with NPIs. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP). Association for Computational Linguistics, Hong Kong, China, pp 2877\u20132887, https:\/\/doi.org\/10.18653\/v1\/D19-1286","DOI":"10.18653\/v1\/D19-1286"},{"key":"9826_CR77","doi-asserted-by":"publisher","first-page":"625","DOI":"10.1162\/tacl_a_00290","volume":"7","author":"A Warstadt","year":"2019","unstructured":"Warstadt, A., Singh, A., & Bowman, S. R. (2019). Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7, 625\u2013641.","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"9826_CR78","doi-asserted-by":"publisher","first-page":"377","DOI":"10.1162\/tacl_a_00321","volume":"8","author":"A Warstadt","year":"2020","unstructured":"Warstadt, A., Parrish, A., Liu, H., et al. (2020). BLiMP: The benchmark of linguistic minimal pairs for English. Transactions of the Association for Computational Linguistics, 8, 377\u2013392. https:\/\/doi.org\/10.1162\/tacl_a_00321","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"9826_CR79","doi-asserted-by":"publisher","unstructured":"Wolf, T., Debut, L., Sanh, V., et\u00a0al (2020) Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: System demonstrations. Association for Computational Linguistics, Online, pp 38\u2013https:\/\/doi.org\/10.18653\/v1\/2020.emnlp-demos.6","DOI":"10.18653\/v1\/2020.emnlp-demos.6"},{"key":"9826_CR80","unstructured":"Zeman, D., Nivre, J., & alii, (2021) Universal dependencies 2.9. LINDAT\/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (\u00daFAL), Faculty of Mathematics and Physics, Charles University"}],"container-title":["Language Resources and Evaluation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10579-025-09826-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10579-025-09826-3\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10579-025-09826-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,12,9]],"date-time":"2025-12-09T05:17:22Z","timestamp":1765257442000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10579-025-09826-3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,28]]},"references-count":81,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2025,12]]}},"alternative-id":["9826"],"URL":"https:\/\/doi.org\/10.1007\/s10579-025-09826-3","relation":{},"ISSN":["1574-020X","1574-0218"],"issn-type":[{"type":"print","value":"1574-020X"},{"type":"electronic","value":"1574-0218"}],"subject":[],"published":{"date-parts":[[2025,6,28]]},"assertion":[{"value":"2 April 2025","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"28 June 2025","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they do not have any Conflict of interest.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}},{"value":"Our work has limited ethical implications since we mainly introduced probing tasks to evaluate a pre-trained neural language model. The treebanks we relied upon were used in compliance with the Terms of Use and the resources and materials produced during this study will be distributed in compliance with the license agreement of each source treebank. The research questions were investigated relying on the treebanks for which we were able to recover parallel sentences (as defined in this work), thus the scalability of our approach is limited to treebanks distributed across multiple annotation formats. We need to point out that our conclusions only concern the sample of treebanks that we analyzed in this study.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Compliance with Ethical Standards"}}]}}