{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,24]],"date-time":"2026-04-24T19:00:20Z","timestamp":1777057220160,"version":"3.51.4"},"reference-count":46,"publisher":"Springer Science and Business Media LLC","issue":"4","license":[{"start":{"date-parts":[[2023,7,23]],"date-time":"2023-07-23T00:00:00Z","timestamp":1690070400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,7,23]],"date-time":"2023-07-23T00:00:00Z","timestamp":1690070400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Lang Resources &amp; Evaluation"],"published-print":{"date-parts":[[2023,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Transformer-based pre-trained Language Models (PLMs) have emerged as the foundations for the current state-of-the-art algorithms in most natural language processing tasks, in particular when applied to context rich data such as sentences or paragraphs. However, their impact on the tasks defined in terms of abstract individual word properties, not necessary tied to their specific use in a particular sentence, has been inadequately explored, which is a notable research gap. Addressing this gap is crucial for advancing our understanding of natural language processing. To fill this void, we concentrate on classification of semantic relations: given a pair of concepts (words or word sequences) the aim is to identify the semantic label to describe their relationship. E.g. in the case of the pair <jats:italic>green\/colour<\/jats:italic>, \u201cis a\u201d is a suitable relation while \u201cpart of\u201d, \u201cproperty of\u201d, and \u201copposite of\u201d are not suitable. This classification is independent of a particular sentence in which these concepts might have been used. We are first to incorporate a language model into both existing approaches to this task, namely path-based and distribution-based methods. Our transformer-based approaches exhibit significant improvements over the state-of-the-art and come remarkably close to achieving human-level performance on rigorous benchmarks. We are also first to provide evidence that the standard datasets over-state the performance due to the effect of \u201clexical memorisation.\u201d We reduce this effect by applying lexical separation. On the new benchmark datasets, the algorithmic performance remains significantly below human-level, highlighting that the task of semantic relation classification is still unresolved, particularly for language models of the sizes commonly used at the time of our study. We also identify additional challenges that PLM-based approaches face and conduct extensive ablation studies and other experiments to investigate the sensitivity of our findings to specific modelling and implementation choices. Furthermore, we examine the specific relations that pose greater challenges and discuss the trade-offs between accuracy and processing time.<\/jats:p>","DOI":"10.1007\/s10579-023-09677-w","type":"journal-article","created":{"date-parts":[[2023,7,23]],"date-time":"2023-07-23T15:01:23Z","timestamp":1690124483000},"page":"1463-1486","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Fine-tuning language models to recognize semantic relations"],"prefix":"10.1007","volume":"57","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-9313-2234","authenticated-orcid":false,"given":"Dmitri","family":"Roussinov","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Serge","family":"Sharoff","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nadezhda","family":"Puchnina","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2023,7,23]]},"reference":[{"key":"9677_CR1","doi-asserted-by":"crossref","unstructured":"Aghajanyan, A., Maillard, J., Shrivastava, A., Diedrick, K., Haeger, M., Li, H., Mehdad, Y., Stoyanov, V., Kumar, A., Lewis, M., & Gupta, S. (2020). Conversational semantic parsing. https:\/\/arxiv.org\/abs\/2009.13655","DOI":"10.18653\/v1\/2020.emnlp-main.408"},{"key":"9677_CR2","doi-asserted-by":"crossref","unstructured":"Barkan, O., Caciularu, A., & Dagan, I. (2020). Within-between lexical relation classification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing","DOI":"10.18653\/v1\/2020.emnlp-main.284"},{"key":"9677_CR3","unstructured":"Baroni, M., & Lenci, A. (2011). How we BLESSed distributional semantic evaluation. In Proceedings of 2011 workshop on geometrical models of natural language semantics"},{"key":"9677_CR4","unstructured":"Brunner, G., Liu, Y., Pascual, D., Richter, O., Ciaramita, M., & Wattenhofer, R. (2020). On identifiability in transformers. In Inetrnational Conference on Learning Rerpresentations"},{"key":"9677_CR5","doi-asserted-by":"crossref","unstructured":"Cho, K., van Merrienboer, B., Gulcehre, C., Bougares, F., & Y\u00a0Bengio, HS. (2014). Learning phrase representations using rnn encoder-decoder for statistical machine translation. In 55th Annual Meeting of the Association for Computational Linguistics","DOI":"10.3115\/v1\/D14-1179"},{"key":"9677_CR6","doi-asserted-by":"crossref","unstructured":"Clark, K., Khandelwal, U., Levy, O., & Manning, CD. (2019). What does bert look at? an analysis of bert\u2019s attention. In ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP","DOI":"10.18653\/v1\/W19-4828"},{"key":"9677_CR7","unstructured":"Devlin, J., Chang, MW., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. In arXiv preprint arXiv:1810.04805"},{"key":"9677_CR8","doi-asserted-by":"publisher","first-page":"34","DOI":"10.1162\/tacl_a_00298","volume":"8","author":"A Ettinger","year":"2020","unstructured":"Ettinger, A. (2020). What bert is not: Lessons from a new suite of psycholinguistic diagnostics for language models. In: Transactions of the Association for Computational Linguistics, Volume 8: 34\u201338","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"9677_CR9","doi-asserted-by":"crossref","unstructured":"Gupta, S., Shah, R., Mohit, M., & Kumar, A. (2018). Semantic parsing for task oriented dialog using hierarchical representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","DOI":"10.18653\/v1\/D18-1300"},{"key":"9677_CR10","doi-asserted-by":"crossref","unstructured":"Hearst, M. A. (1992). Automatic acquisition of hyponyms from large text corpora. In COLING 1992 volume 2: The 14th international conference on computational linguistics.","DOI":"10.3115\/992133.992154"},{"key":"9677_CR11","doi-asserted-by":"crossref","unstructured":"Kassner, N., Krojer, B., & Schutze, H. (2020). Are pretrained language models symbolic reasoners over knowledge? In 24th Conference on Computational Natural Language Learning","DOI":"10.18653\/v1\/2020.conll-1.45"},{"key":"9677_CR12","doi-asserted-by":"crossref","unstructured":"Kim, N., Patel, R., Poliak, A., Xia, P., Wang, A., McCoy, T., Tenney, I., Ross, A., Linzen, T., Durme, BV., Bowman, SR., & Pavlick, E. (2019). Probing what different nlp tasks teach machines about function word comprehension. In Eighth Joint Conference on Lexical and Computational Semantics","DOI":"10.18653\/v1\/S19-1026"},{"key":"9677_CR13","unstructured":"Lample, G., & Conneau, A. (2019). Cross-lingual language model pretraining. In arXiv preprint arXiv:1901.07291"},{"key":"9677_CR14","doi-asserted-by":"crossref","unstructured":"Levy, O., Remus, S., Biemann, C., & Dagan, I. (2015). Do supervised distributional methods really learn lexical inference relations? In Proceedings of NAACL-HLT","DOI":"10.3115\/v1\/N15-1098"},{"key":"9677_CR15","doi-asserted-by":"crossref","unstructured":"McCoy, RT., Pavlick, E., & Linzen, T. (2019). Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In: ACL","DOI":"10.18653\/v1\/P19-1334"},{"key":"9677_CR16","unstructured":"Mickus, T., Paperno, D., Constant, M., & van Deemter, K. (2020). What do you mean, bert? assessing bert as a distributional semantics model. In Proceedings of the Society for Computation in Linguistics Vol. 3"},{"key":"9677_CR17","unstructured":"Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., & Dean, J. (2013). Distributed representations of words and phrases and their compositionality. Advances in Neural Information Processing Systems, 26."},{"key":"9677_CR18","unstructured":"Nakashole, N., Weikum, G., & Suchanek, F. (2012). Patty: A taxonomy of relational patterns with semantic types. In 2012 Joint Conference EMNLP and CoNLL"},{"key":"9677_CR19","doi-asserted-by":"crossref","unstructured":"Necsulescu, S., Mendes, S., Jurgens, D., Bel, N., & Navigli, R. (2015). Reading between the lines: Overcoming data sparsity for accurate classification of lexical relationships. In M. Palmer, G. Boleda, P. Rosso (Eds.), Proceedings of the fourth joint conference on lexical and computational semantics (pp. 182\u2013192). Denver, Colorado: Association for Computational Linguistics.","DOI":"10.18653\/v1\/S15-1021"},{"key":"9677_CR20","unstructured":"Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., & Duchesnay, E. (2011). Machine learning in python. The Journal of Machine Learning Research, 12, 2825\u20132830."},{"key":"9677_CR21","doi-asserted-by":"crossref","unstructured":"Pennington, J., Socher, R., & Manning, C. (2014). Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) (pp. 1532\u20131543).","DOI":"10.3115\/v1\/D14-1162"},{"key":"9677_CR22","doi-asserted-by":"crossref","unstructured":"Petroni, F., Rocktaschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A. (2019). Language models as knowledge bases? In Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","DOI":"10.18653\/v1\/D19-1250"},{"key":"9677_CR23","first-page":"5485","volume":"21","author":"C Raffel","year":"2020","unstructured":"Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, PJ. (2020). Exploring the limits of transfer learning with a unified text-to-text transformer. In: Journal of Machine Learning Research (JMLR), 21: 5485\u20135551","journal-title":"Journal of Machine Learning Research (JMLR)"},{"key":"9677_CR24","volume-title":"2013","author":"S Riedel","year":"2013","unstructured":"Riedel, S., Yao, L., McCallum, A., & Marlin, M. B. (2013). NAACL-HLT: Relation extraction with matrix factorization and universal schemas. In Proceedings of the 2013 conference of the North American chapter of the association for computational linguistics: human language technologies (pp. 74\u201384)."},{"key":"9677_CR25","doi-asserted-by":"crossref","unstructured":"Roberts, A., Raffel, C., & Shazeer, N. (2020). How much knowledge can you pack into the parameters of a language model? In Conference on Empirical Methods in Natural Language Processing","DOI":"10.18653\/v1\/2020.emnlp-main.437"},{"key":"9677_CR26","doi-asserted-by":"publisher","first-page":"842","DOI":"10.1162\/tacl_a_00349","volume":"8","author":"A Rogers","year":"2020","unstructured":"Rogers, A., Kovaleva, O., & Rumshisky, A. (2020). A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8, 842\u2013866.","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"9677_CR27","doi-asserted-by":"crossref","unstructured":"Roller, S., Kiela, D., & Nickel, M. (2018). Hearst patterns revisited: Automatic hypernym detection from large text corpora. arXiv preprint arXiv:1806.03191","DOI":"10.18653\/v1\/P18-2057"},{"key":"9677_CR28","doi-asserted-by":"crossref","unstructured":"Roussinov, D., & Puchnina, N. (2019). Combining neural networks and pattern matching for ontology mining-a meta learning inspired approach. In Proceedings of the 13th IEEE International Conference on Semantic Computing (ICSC)","DOI":"10.1109\/ICOSC.2019.8665528"},{"key":"9677_CR29","doi-asserted-by":"crossref","unstructured":"Roussinov, D., & Turetken, O. (2007). Semantic verification in an online fact seeking environment. In ACM Conference on Information and Knowledge Management","DOI":"10.1145\/1321440.1321453"},{"key":"9677_CR30","doi-asserted-by":"crossref","unstructured":"Santus, E., Yung, F., Lenci, A., & Huang, CR. (2015). Evalution 1.0: an evolving semantic dataset for training and evaluation of distributional semantic models. In Proceedings of the 4th Workshop on Linked Data in Linguistics: Resources and Applications, pp 64\u201369","DOI":"10.18653\/v1\/W15-4208"},{"key":"9677_CR31","unstructured":"Santus, E., Lenci, A., Chiu, TS., Lu, Q., & Huang, CR. (2016). Nine features in a random forest to learn taxonomical semantic relations. arXiv preprint arXiv:1603.08702"},{"key":"9677_CR32","doi-asserted-by":"crossref","unstructured":"Schlichtkrull, M., Kipf, T., Bloem, P., van\u00a0den Berg, R., Titov, I., & Welling, M. (2018). Modeling relational data with graph convolutional networks. In The Semantic Web. ESWC 2018. Lecture Notes in Computer Science","DOI":"10.1007\/978-3-319-93417-4_38"},{"key":"9677_CR33","doi-asserted-by":"publisher","first-page":"163","DOI":"10.1017\/S1351324919000354","volume":"26","author":"S Sharoff","year":"2020","unstructured":"Sharoff, S. (2020). Finding next of kin: Cross-lingual embedding spaces for related languages. Journal of Natural Language Engineering 26","journal-title":"Journal of Natural Language Engineering"},{"key":"9677_CR34","unstructured":"Shi, P., & Lin, J. (2020). Simple bert models for relation extraction and semantic role labeling. https:\/\/arxiv.org\/pdf\/1904.05255.pdf"},{"key":"9677_CR35","unstructured":"Shwartz, V., & Dagan, I. (2016). Path-based vs. distributional information in recognizing lexical semantic relations. arXiv preprint arXiv:1608.05014"},{"key":"9677_CR36","doi-asserted-by":"crossref","unstructured":"Shwartz, V., Goldberg, Y., & Dagan, I. (2016). Improving hypernymy detection with an integrated path-based and distributional method. arXiv preprint arXiv:1603.06076","DOI":"10.18653\/v1\/P16-1226"},{"key":"9677_CR37","doi-asserted-by":"crossref","unstructured":"Shwartz, V., Santus, E., & Schlechtweg, D. (2017). Hypernyms under siege: Linguistically-motivated artillery for hypernymy detection. In: EACL 2017","DOI":"10.18653\/v1\/E17-1007"},{"key":"9677_CR38","unstructured":"Snow, R., Jurafsky, D., & Ng, A. Y. (2004). Learning syntactic patterns for automatic hypernym discovery. Advances in Neural Information Processing Systems, 17."},{"key":"9677_CR39","volume-title":"Synthesis Lectures on Human Language Technologies","author":"A S\u00f8gaard","year":"2019","unstructured":"S\u00f8gaard, A., Vuli\u0107, I., Ruder, S., & Faruqui, M. (2019). Cross-Lingual Word Embeddings. Synthesis Lectures on Human Language Technologies, Morgan & Claypool Publishers"},{"key":"9677_CR40","unstructured":"Tenney, I., Xia, P., Chen, B., Wang, A., Poliak, A., McCoy, R. T., Kim, N., Durme, B. V., Bowman, S. R., Das, D., & Pavlick, E. (2019). What do you learn from context? probing for sentence structure in contextualized word representations. arXiv preprint arXiv:1905.06316"},{"key":"9677_CR41","unstructured":"Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30."},{"key":"9677_CR42","doi-asserted-by":"publisher","unstructured":"Vuli\u0107, I., Ponti, EM., Litschko, R., Glava\u0161, G., & Korhonen, A. (2020). Probing pretrained language models for lexical semantics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, Online, (pp 7222\u20137240), https:\/\/doi.org\/10.18653\/v1\/2020.emnlp-main.586,","DOI":"10.18653\/v1\/2020.emnlp-main.586"},{"key":"9677_CR43","doi-asserted-by":"crossref","unstructured":"Wang, C., He, X., & Zhou, A. (2019). Spherere: Distinguishing lexical relations with hyperspherical relation embeddings. In Proceedings of the 57th annual meeting of the association for computational linguistics (pp. 1727\u20131737).","DOI":"10.18653\/v1\/P19-1169"},{"key":"9677_CR44","doi-asserted-by":"crossref","unstructured":"Wang, C., Qiu, M., Huang, J., & He, X. (2021). Keml: A knowledge-enriched meta-learning framework for lexical relation classification. In: AAAI","DOI":"10.1609\/aaai.v35i15.17640"},{"key":"9677_CR45","doi-asserted-by":"crossref","unstructured":"Washio, K., & Kato, T. (2018). Filling missing paths: Modeling co-occurrences of word pairs and dependency paths for recognizing lexical semantic relations. arXiv preprint arXiv:1809.03411","DOI":"10.18653\/v1\/N18-1102"},{"key":"9677_CR46","doi-asserted-by":"crossref","unstructured":"Zhou, X., Zhang, Y., Cui, L., & Huang, D. (2020). Evaluating commonsense in pre-trained language models. In Proceedings of AAAI Conference","DOI":"10.1609\/aaai.v34i05.6523"}],"container-title":["Language Resources and Evaluation"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10579-023-09677-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10579-023-09677-w\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10579-023-09677-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,11,17]],"date-time":"2023-11-17T15:08:33Z","timestamp":1700233713000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10579-023-09677-w"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,7,23]]},"references-count":46,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2023,12]]}},"alternative-id":["9677"],"URL":"https:\/\/doi.org\/10.1007\/s10579-023-09677-w","relation":{},"ISSN":["1574-020X","1574-0218"],"issn-type":[{"value":"1574-020X","type":"print"},{"value":"1574-0218","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,7,23]]},"assertion":[{"value":"15 May 2023","order":1,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 July 2023","order":2,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}