{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,10]],"date-time":"2026-04-10T03:07:46Z","timestamp":1775790466185,"version":"3.50.1"},"reference-count":61,"publisher":"MIT Press - Journals","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Transactions of the Association for Computational Linguistics"],"published-print":{"date-parts":[[2019,11]]},"abstract":"<jats:p> We analyze human\u2019s disagreements about the validity of natural language inferences. We show that, very often, disagreements are not dismissible as annotation \u201cnoise\u201d, but rather persist as we collect more ratings and as we vary the amount of context provided to raters. We further show that the type of uncertainty captured by current state-of-the-art models for natural language inference is not reflective of the type of uncertainty present in human disagreements. We discuss implications of our results in relation to the recognizing textual entailment (RTE)\/natural language inference (NLI) task. We argue for a refined evaluation objective that requires models to explicitly capture the full distribution of plausible human judgments. <\/jats:p>","DOI":"10.1162\/tacl_a_00293","type":"journal-article","created":{"date-parts":[[2019,11,13]],"date-time":"2019-11-13T19:34:37Z","timestamp":1573673677000},"page":"677-694","source":"Crossref","is-referenced-by-count":60,"title":["Inherent Disagreements in Human Textual Inferences"],"prefix":"10.1162","volume":"7","author":[{"given":"Ellie","family":"Pavlick","sequence":"first","affiliation":[{"name":"Brown University."}]},{"given":"Tom","family":"Kwiatkowski","sequence":"additional","affiliation":[{"name":"Google Research."}]}],"member":"281","reference":[{"key":"bib1","volume-title":"Proc. Subjectivity, Ambiguity and Disagreement in Crowdsourcing (SAD)","volume":"1","author":"Aroyo Lora","year":"2018"},{"key":"bib2","doi-asserted-by":"crossref","first-page":"1210","DOI":"10.3115\/v1\/P14-1114","volume-title":"Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Beltagy Islam","year":"2014"},{"key":"bib3","doi-asserted-by":"crossref","first-page":"632","DOI":"10.18653\/v1\/D15-1075","volume-title":"Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing","author":"Bowman Samuel R.","year":"2015"},{"key":"bib4","first-page":"670","volume-title":"Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing","author":"Bunescu Razvan","year":"2008"},{"key":"bib5","first-page":"1","volume-title":"Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon\u2019s Mechanical Turk","author":"Callison-Burch Chris","year":"2010"},{"key":"bib6","volume-title":"Proceedings of the Computing Natural Language Inference Workshop","author":"Chatzikyriakidis Stergios","year":"2017"},{"key":"bib7","first-page":"33","volume-title":"Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing","author":"Chklovski Timothy","year":"2004"},{"key":"bib8","unstructured":"Robin Cooper, Dick Crouch, Jan Van Eijck, Chris Fox, Johan Van Genabith, Jan Jaspars, Hans Kamp, David Milward, Manfred Pinkal, Massimo Poesio, and Steve Pullman. 1996, Using the framework. Technical report, Technical Report LRE 62-051 D-16, The FraCaS Consortium."},{"key":"bib9","doi-asserted-by":"crossref","first-page":"177","DOI":"10.1007\/11736790_9","volume-title":"Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Textual Entailment","author":"Dagan Ido","year":"2006"},{"key":"bib10","author":"Dasgupta Ishita","year":"2018","journal-title":"arXiv preprint arXiv:1802.04302"},{"key":"bib11","first-page":"4171","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Devlin Jacob","year":"2019"},{"key":"bib12","first-page":"2164","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Dumitrache Anca","year":"2019"},{"key":"bib13","volume-title":"CrowdSem Workshop at the International Semantic Web Conference","author":"Dumitrache Anca","year":"2013"},{"key":"bib14","first-page":"440","volume-title":"Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing","author":"Erk Katrin","year":"2009"},{"key":"bib15","first-page":"1790","volume-title":"Proceedings of the 27th International Conference on Computational Linguistics","author":"Ettinger Allyson","year":"2018"},{"key":"bib16","doi-asserted-by":"crossref","unstructured":"Christiane Fellbaum. 1998. WordNet, Wiley Online Library.","DOI":"10.7551\/mitpress\/7287.001.0001"},{"key":"bib17","first-page":"618","volume-title":"Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing","author":"Finkel Jenny Rose","year":"2006"},{"key":"bib18","first-page":"1300","volume":"99","author":"Friedman Nir","year":"1999","journal-title":"IJCAI"},{"key":"bib19","unstructured":"Konstantina Garoufi. 2007. Towards a Better Understanding of Applied Textual Entailment. Ph.D. thesis, Citeseer."},{"issue":"4","key":"bib20","first-page":"561","volume":"2","author":"Geis Michael L.","year":"1971","journal-title":"Linguistic inquiry"},{"key":"bib21","volume-title":"COLING 1992 Volume 2: The 15th International Conference on Computational Linguistics","author":"Hearst Marti A.","year":"1992"},{"key":"bib22","author":"J\u00f3zefowicz Rafal","year":"2016","journal-title":"ArXiv"},{"key":"bib23","first-page":"556","volume-title":"Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Jurgens David","year":"2013"},{"key":"bib24","first-page":"233","volume":"10","author":"Karttunen Lauri","year":"2014","journal-title":"Empirical Issues in Syntax and Semantics"},{"key":"bib25","first-page":"1","volume-title":"Proceedings of the NIPS Workshop on Probabilistic Programming: Foundations and Applications","author":"Kimmig Angelika","year":"2012"},{"key":"bib26","first-page":"3474","volume-title":"Advances in Neural Information Processing Systems","author":"Kuleshov Volodymyr","year":"2015"},{"key":"bib28","author":"de Marneffe Marie-Catherine","year":"2018","journal-title":"Sinn und Bedeutung 23 (Poster)"},{"key":"bib29","doi-asserted-by":"publisher","DOI":"10.1162\/COLI_a_00097"},{"key":"bib30","doi-asserted-by":"crossref","first-page":"43","DOI":"10.18653\/v1\/W16-1706","volume-title":"Proceedings of the 10th Linguistic Annotation Workshop held in conjunction with ACL 2016 (LAW-X 2016)","author":"Mart\u00ednez Alonso H\u00e9ctor","year":"2016"},{"key":"bib31","first-page":"1357","volume-title":"Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Mart\u00ednez Alonso H\u00e9ctor","year":"2015"},{"key":"bib32","doi-asserted-by":"crossref","first-page":"3428","DOI":"10.18653\/v1\/P19-1334","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"McCoy Tom","year":"2019"},{"issue":"3","key":"bib33","doi-asserted-by":"crossref","first-page":"373","DOI":"10.1111\/j.1755-2567.1970.tb00434.x","volume":"36","author":"Montague Richard","year":"1970","journal-title":"Theoria"},{"key":"bib34","volume-title":"Proceedings of Workshop on Subjectivity, Ambiguity, and Disagreement (SAD)","author":"Palomaki Jennimaria","year":"2018"},{"issue":"2","key":"bib35","doi-asserted-by":"crossref","first-page":"219","DOI":"10.1007\/s10579-012-9188-x","volume":"46","author":"Passonneau Rebecca J.","year":"2012","journal-title":"Language Resources and Evaluation"},{"key":"bib36","doi-asserted-by":"crossref","first-page":"2164","DOI":"10.18653\/v1\/P16-1204","volume-title":"Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Pavlick Ellie","year":"2016"},{"key":"bib37","doi-asserted-by":"crossref","first-page":"114","DOI":"10.18653\/v1\/S16-2014","volume-title":"Proceedings of the Fifth Joint Conference on Lexical and Computational Semantics","author":"Pavlick Ellie","year":"2016"},{"key":"bib38","doi-asserted-by":"crossref","first-page":"742","DOI":"10.3115\/v1\/E14-1078","volume-title":"Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics","author":"Plank Barbara","year":"2014"},{"key":"bib39","doi-asserted-by":"crossref","first-page":"76","DOI":"10.3115\/1608829.1608840","volume-title":"Proceedings of the Workshop on Frontiers in Corpus Annotations II: Pie in the Sky","author":"Poesio Massimo","year":"2005"},{"key":"bib40","first-page":"1778","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Poesio Massimo","year":"2019"},{"key":"bib41","first-page":"513","volume-title":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)","author":"Poliak Adam","year":"2018"},{"key":"bib42","doi-asserted-by":"crossref","first-page":"67","DOI":"10.18653\/v1\/D18-1007","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Poliak Adam","year":"2018"},{"issue":"6","key":"bib43","doi-asserted-by":"crossref","first-page":"1138","DOI":"10.1016\/j.lingua.2011.02.004","volume":"121","author":"Recasens Marta","year":"2011","journal-title":"Lingua"},{"key":"bib44","first-page":"8","volume-title":"Coling 2008: Proceedings of the workshop on Human Judgements in Computational Linguistics","author":"Reidsma Dennis","year":"2008"},{"issue":"1","key":"bib45","doi-asserted-by":"crossref","first-page":"107","DOI":"10.1007\/s10994-006-5833-1","volume":"62","author":"Richardson Matthew","year":"2006","journal-title":"Machine Learning"},{"key":"bib46","volume-title":"1st Workshop on Human-Centered Machine Learning at SIGCHI","author":"Schaekermann Mike","year":"2016"},{"key":"bib47","first-page":"309","volume-title":"Semantics and Linguistic Theory","volume":"20","author":"Simons Mandy","year":"2010"},{"key":"bib48","first-page":"3039","volume-title":"Advances in Neural Information Processing Systems","author":"Smith Nathaniel J.","year":"2013"},{"key":"bib49","first-page":"254","volume-title":"Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing","author":"Snow Rion","year":"2008"},{"key":"bib50","volume-title":"Natural Language Parsing","author":"Tanenhaus M.","year":"1985"},{"issue":"3","key":"bib51","doi-asserted-by":"crossref","first-page":"495","DOI":"10.1093\/jos\/ffy007","volume":"35","author":"Tonhauser Judith","year":"2018","journal-title":"Journal of Semantics"},{"key":"bib52","doi-asserted-by":"crossref","first-page":"333","DOI":"10.1007\/s11168-008-9059-1","volume":"6","author":"Versley Y.","year":"2008","journal-title":"Research on Language and Computation"},{"key":"bib53","doi-asserted-by":"crossref","first-page":"120","DOI":"10.18653\/v1\/W19-0410","volume-title":"Proceedings of the 13th International Conference on Computational Semantics - Long Papers","author":"Westera Matthijs","year":"2019"},{"issue":"2","key":"bib54","doi-asserted-by":"crossref","first-page":"416","DOI":"10.1111\/cogs.12512","volume":"42","author":"White Aaron S.","year":"2018","journal-title":"Cognitive Science"},{"key":"bib55","first-page":"996","volume-title":"Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)","author":"White Aaron Steven","year":"2017"},{"key":"bib56","volume-title":"48th Annual Meeting of the North East Linguistic Society","author":"White Aaron Steven","year":"2017"},{"key":"bib57","first-page":"1112","volume-title":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)","author":"Williams Adina","year":"2018"},{"issue":"3","key":"bib58","doi-asserted-by":"crossref","first-page":"77","DOI":"10.1145\/175247.175255","volume":"37","author":"Zadeh Lotfi A.","year":"1994","journal-title":"Communications of the ACM"},{"issue":"2","key":"bib59","doi-asserted-by":"crossref","first-page":"103","DOI":"10.1109\/91.493904","volume":"4","author":"Zadeh Lotfi A.","year":"1996","journal-title":"IEEE Transactions on Fuzzy Systems"},{"key":"bib60","doi-asserted-by":"crossref","first-page":"694","DOI":"10.1145\/775047.775151","volume-title":"Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","author":"Zadrozny Bianca","year":"2002"},{"key":"bib61","doi-asserted-by":"crossref","first-page":"31","DOI":"10.3115\/1631862.1631868","volume-title":"Proceedings of the ACL Workshop on Empirical Modeling of Semantic Equivalence and Entailment","author":"Zaenen Annie","year":"2005"},{"key":"bib62","doi-asserted-by":"crossref","first-page":"379","DOI":"10.1162\/tacl_a_00068","volume":"5","author":"Zhang Sheng","year":"1996","journal-title":"Transactions of the Association for Computational Linguistics"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/tacl_a_00293","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,3,12]],"date-time":"2021-03-12T21:39:30Z","timestamp":1615585170000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/43531"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,11]]},"references-count":61,"alternative-id":["10.1162\/tacl_a_00293"],"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00293","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,11]]}}}