{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,5,13]],"date-time":"2025-05-13T22:00:25Z","timestamp":1747173625567,"version":"3.40.5"},"reference-count":82,"publisher":"Cambridge University Press (CUP)","issue":"2","license":[{"start":{"date-parts":[[2023,6,1]],"date-time":"2023-06-01T00:00:00Z","timestamp":1685577600000},"content-version":"unspecified","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["cambridge.org"],"crossmark-restriction":true},"short-container-title":["Nat. Lang. Eng."],"published-print":{"date-parts":[[2024,3]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Despite recent breakthroughs in Machine Learning for Natural Language Processing, the Natural Language Inference (NLI) problems still constitute a challenge. To this purpose, we contribute a new dataset that focuses exclusively on the factivity phenomenon; however, our task remains the same as other NLI tasks, that is prediction of entailment, contradiction, or neutral (ECN). In this paper, we describe the LingFeatured NLI corpus and present the results of analyses designed to characterize the factivity\/non-factivity opposition in natural language. The dataset contains entirely natural language utterances in Polish and gathers 2432 verb-complement pairs and 309 unique verbs. The dataset is based on the National Corpus of Polish (NKJP) and is a representative subcorpus in regard to syntactic construction [V][\u017ce][cc]. We also present an extended version of the set (3035 sentences) consisting more sentences with internal negations. We prepared deep learning benchmarks for both sets. We found that transformer BERT-based models working on sentences obtained relatively good results (<jats:inline-formula><jats:alternatives><jats:inline-graphic xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" mime-subtype=\"png\" xlink:href=\"S1351324923000220_inline1.png\"\/><jats:tex-math>$\\approx 89\\%$<\/jats:tex-math><\/jats:alternatives><\/jats:inline-formula>F1 score on base dataset). Even though better results were achieved using linguistic features (<jats:inline-formula><jats:alternatives><jats:inline-graphic xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" mime-subtype=\"png\" xlink:href=\"S1351324923000220_inline2.png\"\/><jats:tex-math>$\\approx 91\\%$<\/jats:tex-math><\/jats:alternatives><\/jats:inline-formula>F1 score on base dataset), this model requires more human labor (humans in the loop) because features were prepared manually by expert linguists. BERT-based models consuming only the input sentences show that they capture most of the complexity of NLI\/factivity. Complex cases in the phenomenon\u2014for example, cases with entitlement (E) and non-factive verbs\u2014still remain an open issue for further research.<\/jats:p>","DOI":"10.1017\/s1351324923000220","type":"journal-article","created":{"date-parts":[[2023,6,1]],"date-time":"2023-06-01T08:54:26Z","timestamp":1685609666000},"page":"385-416","update-policy":"https:\/\/doi.org\/10.1017\/policypage","source":"Crossref","is-referenced-by-count":0,"title":["Polish natural language inference and factivity: An expert-based dataset and benchmarks"],"prefix":"10.1017","volume":"30","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0506-4751","authenticated-orcid":false,"given":"Daniel","family":"Ziembicki","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0617-7301","authenticated-orcid":false,"given":"Karolina","family":"Seweryn","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3407-7570","authenticated-orcid":false,"given":"Anna","family":"Wr\u00f3blewska","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"56","published-online":{"date-parts":[[2023,6,1]]},"reference":[{"key":"S1351324923000220_ref10","doi-asserted-by":"publisher","DOI":"10.1007\/s12136-015-0269-5"},{"volume-title":"Coreference: Annotation, Resolution and Evaluation in Polish","year":"2014","author":"Ogrodniczuk","key":"S1351324923000220_ref55"},{"key":"S1351324923000220_ref24","doi-asserted-by":"publisher","DOI":"10.2307\/412067"},{"key":"S1351324923000220_ref41","first-page":"55","article-title":"Some observations on factivity","volume":"4","author":"Karttunen","year":"1971","journal-title":"Research on Language and Social Interaction"},{"key":"S1351324923000220_ref63","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i05.6397"},{"volume-title":"Lingwistyczna analiza zjawiska faktywno\u015bci (na materiale wsp\u00f3\u0142czesnej polszczyzny)","year":"2022","author":"Ziembicki","key":"S1351324923000220_ref81"},{"key":"S1351324923000220_ref68","doi-asserted-by":"publisher","DOI":"10.1007\/s10579-009-9089-9"},{"key":"S1351324923000220_ref12","unstructured":"Danielewiczowa, M. (2002). Wiedza i niewiedza. In Studium polskich czasownik\u00f3w epistemicznych."},{"volume-title":"Proceedings of the PolEval 2020 Workshop","year":"2020","author":"K\u0142eczek","key":"S1351324923000220_ref44"},{"key":"S1351324923000220_ref65","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-1228"},{"key":"S1351324923000220_ref9","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-02151-0"},{"key":"S1351324923000220_ref1","doi-asserted-by":"publisher","DOI":"10.1007\/s11050-016-9122-7"},{"key":"S1351324923000220_ref74","first-page":"934","volume-title":"Semantics and Linguistic Theory","author":"Tonhauser","year":"2016"},{"key":"S1351324923000220_ref17","unstructured":"Devlin, J. , Chang, M.-W. , Lee, K. and Toutanova, K. (2019). BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Minneapolis: Association for Computational Linguistics, pp. 4171\u20134186."},{"key":"S1351324923000220_ref75","doi-asserted-by":"publisher","DOI":"10.1007\/s12136-012-0150-8"},{"key":"S1351324923000220_ref35","doi-asserted-by":"publisher","DOI":"10.1515\/9783110214260.397"},{"key":"S1351324923000220_ref43","doi-asserted-by":"publisher","DOI":"10.1016\/j.lingua.2015.06.004"},{"volume-title":"Factivity and Prosody in Turkish Attitude Reports","year":"2017","author":"\u00d6zy\u0131ld\u0131z","key":"S1351324923000220_ref58"},{"key":"S1351324923000220_ref8","first-page":"177","volume-title":"Machine Learning Challenges Workshop","author":"Dagan","year":"2005"},{"key":"S1351324923000220_ref60","first-page":"159","volume-title":"Semantics and Linguistic Theory","author":"Partee","year":"2014"},{"key":"S1351324923000220_ref6","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D17-1070"},{"key":"S1351324923000220_ref25","doi-asserted-by":"publisher","DOI":"10.1075\/ijcl.10.4.03gra"},{"key":"S1351324923000220_ref22","first-page":"231","article-title":"Toward a grammar of exclamations","volume":"11","author":"Elliott","year":"1974","journal-title":"Foundations of Language"},{"key":"S1351324923000220_ref23","doi-asserted-by":"publisher","DOI":"10.1353\/lan.2006.0136"},{"key":"S1351324923000220_ref56","doi-asserted-by":"crossref","unstructured":"Oren, Y. , Sagawa, S. , Hashimoto, T.B. and Liang, P. (2019). Distributionally robust language modeling. arXiv preprint arXiv: 1909.02060 .","DOI":"10.18653\/v1\/D19-1432"},{"volume-title":"FraCaS: Using the Framework","year":"1996","author":"Cooper","key":"S1351324923000220_ref7"},{"key":"S1351324923000220_ref53","unstructured":"Mroczkowski, R. , Rybak, P. , Wr\u00f3blewska, A. and Gawlik, I. (2021). HerBERT: efficiently pretrained transformer-based language model for Polish. In Proceedings of the 8th Workshop on Balto-Slavic Natural Language Processing, Kiyv: Association for Computational Linguistics, pp. 1\u201310."},{"volume-title":"TimeBank 1.2. LDC2006T08","year":"2006","author":"Pustejovsky","key":"S1351324923000220_ref62"},{"key":"S1351324923000220_ref73","unstructured":"Stalnaker, R. , Munitz, M.K. and Unger, P. (1977). Pragmatic presuppositions. In Proceedings of the Texas Conference on Per Formatives, Presuppositions, and Implicatures. Arlington: Center for Applied Linguistics, ERIC, pp. 135\u2013148."},{"key":"S1351324923000220_ref78","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D16-1177"},{"key":"S1351324923000220_ref33","first-page":"91","volume-title":"Syntax and Semantics","author":"Hooper","year":"1975"},{"key":"S1351324923000220_ref48","doi-asserted-by":"publisher","DOI":"10.1016\/B0-08-043076-7\/03008-4"},{"key":"S1351324923000220_ref20","doi-asserted-by":"publisher","DOI":"10.1016\/j.pragma.2020.04.011"},{"volume-title":"Meaning and Grammar: An Introduction to Semantics","year":"2000","author":"Chierchia","key":"S1351324923000220_ref5"},{"key":"S1351324923000220_ref36","doi-asserted-by":"publisher","DOI":"10.1145\/1837885.1837906"},{"key":"S1351324923000220_ref28","doi-asserted-by":"crossref","unstructured":"Gururangan, S. , Swayamdipta, S. , Levy, O. , Schwartz, R. , Bowman, S.R. and Smith, N.A. (2018). Annotation artifacts in natural language inference data. arXiv preprint arXiv: 1803.02324 .","DOI":"10.18653\/v1\/N18-2017"},{"key":"S1351324923000220_ref42","first-page":"705","volume-title":"Semantics and Linguistic Theory","author":"Karttunen","year":"2016"},{"key":"S1351324923000220_ref45","first-page":"345","article-title":"Fact","volume":"1","author":"Kiparsky","year":"1971","journal-title":"Semantics"},{"key":"S1351324923000220_ref51","unstructured":"Minard, A.-L. , Speranza, M. , Urizar, R. , Altuna, B. , van Erp, M. , Schoen, A. and van Son, C. (2016a). MEANTIME, the NewsReader multilingual event and time corpus. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC\u201916). Portoro\u017e: European Language Resources Association (ELRA), pp. 4417\u20134422."},{"key":"S1351324923000220_ref64","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-1228"},{"volume-title":"The Stanford Encyclopedia of Philosophy","year":"2021","author":"Speaks","key":"S1351324923000220_ref72"},{"key":"S1351324923000220_ref46","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D15-1189"},{"key":"S1351324923000220_ref32","doi-asserted-by":"crossref","unstructured":"He, H. , Zha, S. and Wang, H. (2019). Unlearn dataset bias in natural language inference by fitting the residual. arXiv preprint arXiv: 1908.10763 .","DOI":"10.18653\/v1\/D19-6115"},{"key":"S1351324923000220_ref76","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00335"},{"key":"S1351324923000220_ref50","doi-asserted-by":"crossref","unstructured":"McCoy, R.T. , Pavlick, E. and Linzen, T. (2019). Right for the wrong reasons: diagnosing syntactic heuristics in natural language inference. arXiv preprint arXiv: 1902.01007 .","DOI":"10.18653\/v1\/P19-1334"},{"key":"S1351324923000220_ref79","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N18-1101"},{"key":"S1351324923000220_ref80","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i05.6510"},{"key":"S1351324923000220_ref34","doi-asserted-by":"publisher","DOI":"10.3115\/1564131.1564137"},{"key":"S1351324923000220_ref38","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.768"},{"key":"S1351324923000220_ref57","first-page":"88","article-title":"A corpus-based study on \u2018regret\u2019 as a factive verb and its complements","volume":"2","author":"\u00d6zt\u00fcrk","year":"2017","journal-title":"European Journal of Foreign Language Teaching"},{"key":"S1351324923000220_ref3","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1163\/9789004253162_004","volume-title":"Presuppositions and Discourse: Essays Offered to Hans Kamp","author":"Beaver","year":"2010"},{"key":"S1351324923000220_ref37","doi-asserted-by":"publisher","DOI":"10.1016\/j.lingua.2018.12.002"},{"volume-title":"Narodowy Korpus J\u0119zyka Polskiego (National Corpus of Polish Language)","year":"2012","author":"Przepi\u00f3rkowski","key":"S1351324923000220_ref61"},{"key":"S1351324923000220_ref2","first-page":"69","article-title":"Factivity, belief and discourse","volume":"1","author":"Anand","year":"2014","journal-title":"The Art and Craft of Semantics: A Festschrift for Irene Heim"},{"key":"S1351324923000220_ref66","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N18-1067"},{"key":"S1351324923000220_ref4","doi-asserted-by":"publisher","DOI":"10.1023\/A:1010933404324"},{"volume-title":"Factive and Assertive Attitude Reports","year":"2019","author":"Dj\u00e4rv","key":"S1351324923000220_ref19"},{"key":"S1351324923000220_ref21","doi-asserted-by":"publisher","DOI":"10.1163\/18756735-90000845"},{"key":"S1351324923000220_ref82","doi-asserted-by":"publisher","DOI":"10.1093\/llc\/fqq018"},{"key":"S1351324923000220_ref15","first-page":"107","article-title":"The commitmentbank: investigating projection in naturally occurring discourse","volume":"23","author":"de Marneffe","year":"2019","journal-title":"Proceedings of Sinn und Bedeutung"},{"key":"S1351324923000220_ref30","doi-asserted-by":"publisher","DOI":"10.1111\/j.1933-1592.2010.00338.x"},{"key":"S1351324923000220_ref16","doi-asserted-by":"publisher","DOI":"10.1016\/B978-0-12-545850-4.50012-1"},{"key":"S1351324923000220_ref11","doi-asserted-by":"publisher","DOI":"10.1016\/j.langsci.2018.07.004"},{"key":"S1351324923000220_ref47","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511813313"},{"key":"S1351324923000220_ref54","unstructured":"Nairn, R. , Condoravdi, C. and Karttunen, L. (2006). Computing relative polarity for textual inference. In Proceedings of the Fifth International Workshop on Inference in Computational Semantics (icos-5)."},{"key":"S1351324923000220_ref67","doi-asserted-by":"publisher","DOI":"10.1023\/B:LING.0000023378.71748.db"},{"key":"S1351324923000220_ref27","first-page":"19","article-title":"Factive verbs and presuppositions for \u2018regret\u2019 and \u2018know\u2019","volume":"10","author":"Grigore","year":"2016","journal-title":"Revista Rom\u00e2n\u0103 de Filosofie Analitic\u0103"},{"key":"S1351324923000220_ref29","unstructured":"Hanink, E. and Bochnak, M.R. (2017). Factivity and two types of embedded clauses in Washo. In Proceedings of the 47th Annual Meeting of North-East Linguistic Society (NELS 47), pp. 65\u201378."},{"key":"S1351324923000220_ref71","doi-asserted-by":"publisher","DOI":"10.1080\/0163853X.2016.1150660"},{"key":"S1351324923000220_ref77","doi-asserted-by":"publisher","DOI":"10.5840\/logos-episteme20112155"},{"key":"S1351324923000220_ref39","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-1630"},{"key":"S1351324923000220_ref13","unstructured":"Dasgupta, I. , Guo, D. , Stuhlm\u00fcller, A. , Gershman, S.J. and Goodman, N.D. (2018). Evaluating compositionality in sentence embeddings. arXiv preprint arXiv: 1802.04302 ."},{"key":"S1351324923000220_ref69","doi-asserted-by":"publisher","DOI":"10.1162\/COLI_a_00096"},{"key":"S1351324923000220_ref26","doi-asserted-by":"publisher","DOI":"10.1163\/9789004368811_003"},{"key":"S1351324923000220_ref52","unstructured":"Minard, A.-L. , Speranza, M. , Urizar, R. , Altuna, B. , Van Erp, M. , Schoen, A. and Van Son, C. (2016b). Meantime, the newsreader multilingual event and time corpus. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC\u201916), pp. 4417\u20134422."},{"key":"S1351324923000220_ref70","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.emnlp-main.381"},{"key":"S1351324923000220_ref40","doi-asserted-by":"publisher","DOI":"10.2307\/412084"},{"key":"S1351324923000220_ref18","doi-asserted-by":"publisher","DOI":"10.1007\/s11098-017-0929-y"},{"key":"S1351324923000220_ref31","doi-asserted-by":"publisher","DOI":"10.1007\/s12136-012-0163-3"},{"key":"S1351324923000220_ref49","unstructured":"Lotan, A. , Stern, A. and Dagan, I. (2013). Truthteller: annotating predicate truth. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 752\u2013757."},{"volume-title":"Word Frequency List of American English","year":"2010","author":"Davies","key":"S1351324923000220_ref14"},{"key":"S1351324923000220_ref59","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.conll-1.28"}],"container-title":["Natural Language Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.cambridge.org\/core\/services\/aop-cambridge-core\/content\/view\/S1351324923000220","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,21]],"date-time":"2024-10-21T16:54:44Z","timestamp":1729529684000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.cambridge.org\/core\/product\/identifier\/S1351324923000220\/type\/journal_article"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,1]]},"references-count":82,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2024,3]]}},"alternative-id":["S1351324923000220"],"URL":"https:\/\/doi.org\/10.1017\/s1351324923000220","relation":{},"ISSN":["1351-3249","1469-8110"],"issn-type":[{"type":"print","value":"1351-3249"},{"type":"electronic","value":"1469-8110"}],"subject":[],"published":{"date-parts":[[2023,6,1]]},"assertion":[{"value":"\u00a9 The Author(s), 2023. Published by Cambridge University Press","name":"copyright","label":"Copyright","group":{"name":"copyright_and_licensing","label":"Copyright and Licensing"}},{"value":"This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (https:\/\/creativecommons.org\/licenses\/by\/4.0\/), which permits unrestricted re-use, distribution and reproduction, provided the original article is properly cited.","name":"license","label":"License","group":{"name":"copyright_and_licensing","label":"Copyright and Licensing"}},{"value":"This content has been made available to all.","name":"free","label":"Free to read"}]}}