{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,20]],"date-time":"2026-05-20T21:19:57Z","timestamp":1779311997558,"version":"3.51.4"},"reference-count":36,"publisher":"Cambridge University Press (CUP)","issue":"2","license":[{"start":{"date-parts":[[2023,3,16]],"date-time":"2023-03-16T00:00:00Z","timestamp":1678924800000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["cambridge.org"],"crossmark-restriction":true},"short-container-title":["Nat. Lang. Eng."],"published-print":{"date-parts":[[2024,3]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>In this paper, we study natural language paraphrasing from both corpus creation and modeling points of view. We focus in particular on the methodology that allows the extraction of challenging examples of paraphrase pairs in their natural textual context, leading to a dataset potentially more suitable for evaluating the models\u2019 ability to represent meaning, especially in document context, when compared with those gathered using various sentence-level heuristics. To this end, we introduce the Turku Paraphrase Corpus, the first large-scale, fully manually annotated corpus of paraphrases in Finnish. The corpus contains 104,645 manually labeled paraphrase pairs, of which 98% are verified to be true paraphrases, either universally or within their present context. In order to control the diversity of the paraphrase pairs and avoid certain biases easily introduced in automatic candidate extraction, the paraphrases are manually collected from different paraphrase-rich text sources. This allows us to create a challenging dataset including longer and more lexically diverse paraphrases than can be expected from those collected through heuristics. In addition to quality, this also allows us to keep the original document context for each pair, making it possible to study paraphrasing in context. To our knowledge, this is the first paraphrase corpus which provides the original document context for the annotated pairs.<\/jats:p><jats:p>We also study several paraphrase models trained and evaluated on the new data. Our initial paraphrase classification experiments indicate a challenging nature of the dataset when classifying using the detailed labeling scheme used in the corpus annotation, the accuracy substantially lacking behind human performance. However, when evaluating the models on a large scale paraphrase retrieval task on almost 400M candidate sentences, the results are highly encouraging, 29\u201353% of the pairs being ranked in the top 10 depending on the paraphrase type. The Turku Paraphrase Corpus is available at <jats:uri xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" xlink:href=\"https:\/\/github.com\/TurkuNLP\/Turku-paraphrase-corpus\">github.com\/TurkuNLP\/Turku-paraphrase-corpus<\/jats:uri> as well as through the popular HuggingFace datasets under the CC-BY-SA license.<\/jats:p>","DOI":"10.1017\/s1351324923000086","type":"journal-article","created":{"date-parts":[[2023,3,16]],"date-time":"2023-03-16T10:14:08Z","timestamp":1678961648000},"page":"319-353","update-policy":"https:\/\/doi.org\/10.1017\/policypage","source":"Crossref","is-referenced-by-count":2,"title":["Towards diverse and contextually anchored paraphrase modeling: A dataset and baselines for Finnish"],"prefix":"10.1017","volume":"30","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4580-5366","authenticated-orcid":false,"given":"Jenna","family":"Kanerva","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Filip","family":"Ginter","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Li-Hsin","family":"Chang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Iiro","family":"Rastas","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Valtteri","family":"Skantsi","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jemina","family":"Kilpel\u00e4inen","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hanna-Mari","family":"Kupari","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Aurora","family":"Piirto","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jenna","family":"Saarni","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Maija","family":"Sev\u00f3n","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Otto","family":"Tarkka","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"56","published-online":{"date-parts":[[2023,3,16]]},"reference":[{"key":"S1351324923000086_ref35","volume-title":"Advances in Neural Information Processing Systems","author":"Wang","year":"2019"},{"key":"S1351324923000086_ref15","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.ngt-1.6"},{"key":"S1351324923000086_ref31","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W19-5003"},{"key":"S1351324923000086_ref29","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W17-2619"},{"key":"S1351324923000086_ref16","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W18-6317"},{"key":"S1351324923000086_ref34","unstructured":"Virtanen, A. , Kanerva, J. , Ilo, R. , Luoma, J. , Luotolahti, J. , Salakoski, T. , Ginter, F. and Pyysalo, S. (2019). Multilingual is not enough: BERT for Finnish. arXiv preprint arXiv:1912.07076."},{"key":"S1351324923000086_ref2","first-page":"9433","article-title":"Construction of paraphrasing dataset for Punjabi: A deep learning approach","volume":"29","author":"Arwinder Singh","year":"2020","journal-title":"International Journal of Advanced Science and Technology"},{"key":"S1351324923000086_ref5","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.747"},{"key":"S1351324923000086_ref18","doi-asserted-by":"publisher","DOI":"10.1109\/TBDATA.2019.2921572"},{"key":"S1351324923000086_ref3","doi-asserted-by":"publisher","DOI":"10.1162\/COLI_a_00166"},{"key":"S1351324923000086_ref28","unstructured":"Scherrer, Y. (2020). TaPaCo: A corpus of sentential paraphrases for 73 languages. In Proceedings of the Twelfth Language Resources and Evaluation Conference, Marseille, France. European Language Resources Association, pp. 6868\u20136873."},{"key":"S1351324923000086_ref10","doi-asserted-by":"crossref","unstructured":"Dong, Q. , Wan, X. and Cao, Y. (2021). ParaSCI: A large scientific paraphrase dataset for longer paraphrase generation. arXiv preprint arXiv:2101.08382.","DOI":"10.18653\/v1\/2021.eacl-main.33"},{"key":"S1351324923000086_ref1","article-title":"Evaluation of state-of-the-art paraphrase identification and its application to automatic plagiarism detection","volume":"34","author":"Altheneyan","year":"2019","journal-title":"International Journal of Pattern Recognition and Artificial Intelligence"},{"key":"S1351324923000086_ref19","unstructured":"Kanerva, J. , Ginter, F. , Chang, L.-H. , Rastas, I. , Skantsi, V. , Kilpel\u00e4inen, J. , Kupari, H.-M. , Piirto, A. , Saarni, J. , Sev\u00f3n, M. and Tarkka, O. (2021a). Annotation guidelines for the Turku Paraphrase Corpus. Technical report, University of Turku, arXiv:2108.07499."},{"key":"S1351324923000086_ref13","unstructured":"Ganitkevitch, J. and Callison-Burch, C. (2014). The multilingual paraphrase database. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC\u201914), Reykjavik, Iceland. European Language Resources Association (ELRA), pp. 4276\u20134283."},{"key":"S1351324923000086_ref20","unstructured":"Kanerva, J. , Ginter, F. , Chang, L.-H. , Rastas, I. , Skantsi, V. , Kilpel\u00e4inen, J. , Kupari, H.-M. , Saarni, J. , Sev\u00f3n, M. and Tarkka, O. (2021b). Finnish paraphrase corpus. In Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa), Reykjavik, Iceland (Online). Link\u00f6ping University Electronic Press, Sweden, pp. 288\u2013298."},{"key":"S1351324923000086_ref17","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.611"},{"key":"S1351324923000086_ref6","unstructured":"Creutz, M. (2018). Open Subtitles paraphrase corpus for six languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA), pp. 1364\u20131369."},{"key":"S1351324923000086_ref33","unstructured":"Tiedemann, J. (2020). The Tatoeba Translation Challenge \u2013 Realistic data sets for low resource and multilingual MT. In Proceedings of the Fifth Conference on Machine Translation, Online. Association for Computational Linguistics, pp. 1174\u20131182."},{"key":"S1351324923000086_ref11","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-75477-2_42"},{"key":"S1351324923000086_ref7","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00449"},{"key":"S1351324923000086_ref9","unstructured":"Dolan, W.B. and Brockett, C. (2005). Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP 2005), Jeju Island, Korea. Asian Federation of Natural Language Processing, pp. 9\u201316."},{"key":"S1351324923000086_ref14","unstructured":"Ganitkevitch, J. , Van Durme, B. and Callison-Burch, C. (2013). PPDB: The paraphrase database. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Atlanta, Georgia. Association for Computational Linguistics, pp. 758\u2013764."},{"key":"S1351324923000086_ref24","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D15-1163"},{"key":"S1351324923000086_ref32","unstructured":"Tiedemann, J. (2012). Parallel data, tools and interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC\u201912), Istanbul, Turkey. European Language Resources Association (ELRA), pp. 2214\u20132218."},{"key":"S1351324923000086_ref27","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-1410"},{"key":"S1351324923000086_ref25","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00293"},{"key":"S1351324923000086_ref36","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-1042"},{"key":"S1351324923000086_ref23","unstructured":"Luotolahti, J. , Kanerva, J. , Laippala, V. , Pyysalo, S. and Ginter, F. (2015). Towards universal web parsebanks. In Proceedings of the Third International Conference on Dependency Linguistics (Depling 2015), Uppsala, Sweden. Uppsala University, Uppsala, Sweden, pp. 211\u2013220."},{"key":"S1351324923000086_ref12","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-5503"},{"key":"S1351324923000086_ref22","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D17-1126"},{"key":"S1351324923000086_ref30","unstructured":"Shimohata, M. , Sumita, E. and Matsumoto, Y. (2004). Building a paraphrase corpus for speech translation. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC\u201904), Lisbon, Portugal. European Language Resources Association (ELRA), pp. 1407\u20131410."},{"key":"S1351324923000086_ref4","unstructured":"Chang, L.-H. Pyysalo, S. Kanerva, J. and Ginter, F. (2021). Quantitative evaluation of alternative translations in a corpus of highly dissimilar Finnish paraphrases. In Proceedings for the First Workshop on Modelling Translation: Translatology in the Digital Age online. Association for Computational Linguistics, pp. 100\u2013107."},{"key":"S1351324923000086_ref26","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-71746-3_18"},{"key":"S1351324923000086_ref8","unstructured":"Devlin, J. , Chang, M.-W. , Lee, K. and Toutanova, K. (20190. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota. Association for Computational Linguistics, pp. 4171\u20134186."},{"key":"S1351324923000086_ref21","unstructured":"Kanerva, J. , Ginter, F. , Miekka, N. , Leino, A. and Salakoski, T. (2018). Turku neural parser pipeline: An end-to-end system for the CoNLL 2018 shared task. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, Brussels, Belgium. Association for Computational Linguistics, pp. 133\u2013142."}],"container-title":["Natural Language Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.cambridge.org\/core\/services\/aop-cambridge-core\/content\/view\/S1351324923000086","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,4,1]],"date-time":"2024-04-01T13:29:11Z","timestamp":1711978151000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.cambridge.org\/core\/product\/identifier\/S1351324923000086\/type\/journal_article"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,3,16]]},"references-count":36,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2024,3]]}},"alternative-id":["S1351324923000086"],"URL":"https:\/\/doi.org\/10.1017\/s1351324923000086","relation":{},"ISSN":["1351-3249","1469-8110"],"issn-type":[{"value":"1351-3249","type":"print"},{"value":"1469-8110","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,3,16]]},"assertion":[{"value":"\u00a9 The Author(s), 2023. Published by Cambridge University Press","name":"copyright","label":"Copyright","group":{"name":"copyright_and_licensing","label":"Copyright and Licensing"}},{"value":"This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (https:\/\/creativecommons.org\/licenses\/by\/4.0\/), which permits unrestricted re-use, distribution, and reproduction in any medium, provided the original work is properly cited.","name":"license","label":"License","group":{"name":"copyright_and_licensing","label":"Copyright and Licensing"}},{"value":"This content has been made available to all.","name":"free","label":"Free to read"}]}}