{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,26]],"date-time":"2025-11-26T16:35:32Z","timestamp":1764174932626},"reference-count":56,"publisher":"MIT Press - Journals","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Transactions of the Association for Computational Linguistics"],"published-print":{"date-parts":[[2020,12]]},"abstract":"<jats:p> Recent success of pre-trained language models (LMs) has spurred widespread interest in the language capabilities that they possess. However, efforts to understand whether LM representations are useful for symbolic reasoning tasks have been limited and scattered. In this work, we propose eight reasoning tasks, which conceptually require operations such as comparison, conjunction, and composition. A fundamental challenge is to understand whether the performance of a LM on a task should be attributed to the pre-trained representations or to the process of fine-tuning on the task data. To address this, we propose an evaluation protocol that includes both zero-shot evaluation (no fine-tuning), as well as comparing the learning curve of a fine-tuned LM to the learning curve of multiple controls, which paints a rich picture of the LM capabilities. Our main findings are that: (a) different LMs exhibit qualitatively different reasoning abilities, e.g., RoBERTa succeeds in reasoning tasks where BERT fails completely; (b) LMs do not reason in an abstract manner and are context-dependent, e.g., while RoBERTa can compare ages, it can do so only when the ages are in the typical range of human ages; (c) On half of our reasoning tasks all models fail completely. Our findings and infrastructure can help future work on designing new datasets, models, and objective functions for pre-training. <\/jats:p>","DOI":"10.1162\/tacl_a_00342","type":"journal-article","created":{"date-parts":[[2020,12,4]],"date-time":"2020-12-04T20:06:24Z","timestamp":1607112384000},"page":"743-758","source":"Crossref","is-referenced-by-count":55,"title":["oLMpics-On What Language Model Pre-training Captures"],"prefix":"10.1162","volume":"8","author":[{"given":"Alon","family":"Talmor","sequence":"first","affiliation":[{"name":"The Allen Institute for AI"},{"name":"Tel-Aviv University."}]},{"given":"Yanai","family":"Elazar","sequence":"additional","affiliation":[{"name":"The Allen Institute for AI"},{"name":"Bar-Ilan University."}]},{"given":"Yoav","family":"Goldberg","sequence":"additional","affiliation":[{"name":"The Allen Institute for AI"},{"name":"Bar-Ilan University."}]},{"given":"Jonathan","family":"Berant","sequence":"additional","affiliation":[{"name":"The Allen Institute for AI"},{"name":"Tel-Aviv University."}]}],"member":"281","reference":[{"key":"bib1","author":"Adi Yossi","year":"2016","journal-title":"arXiv preprint arXiv:1608.04207"},{"key":"bib2","volume-title":"Thirtieth AAAI Conference on Artificial Intelligence","author":"Bagherinezhad Hessam","year":"2016"},{"key":"bib3","doi-asserted-by":"crossref","first-page":"241","DOI":"10.1007\/978-94-009-2727-8_10","volume-title":"Philosophy, language, and artificial intelligence","author":"Barwise Jon","year":"1981"},{"key":"bib4","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00254"},{"key":"bib5","first-page":"2216","volume-title":"Advances in Neural Information Processing Systems","author":"Blier L\u00e9onard","year":"2018"},{"key":"bib6","doi-asserted-by":"crossref","first-page":"pages 1657\u2013page","DOI":"10.18653\/v1\/P17-1152","volume-title":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Chen Qian","year":"2017"},{"key":"bib7","author":"Coenen Andy","year":"2019","journal-title":"arXiv preprint arXiv: 1906.02715"},{"key":"bib8","first-page":"3079","volume-title":"Advances in Neural Information Processing Systems 28","author":"Dai Andrew M.","year":"2015"},{"key":"bib9","volume-title":"North American Association for Computational Linguistics (NAACL)","author":"Devlin J.","year":"2019"},{"key":"bib10","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00280"},{"key":"bib11","doi-asserted-by":"crossref","first-page":"pages 3973\u2013page","DOI":"10.18653\/v1\/P19-1388","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Elazar Yanai","year":"2019"},{"key":"bib12","author":"Ettinger Allyson","year":"2019","journal-title":"arXiv preprint arXiv:1907.13528"},{"key":"bib13","doi-asserted-by":"crossref","first-page":"134","DOI":"10.18653\/v1\/W16-2524","volume-title":"Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP","author":"Ettinger Allyson","year":"2016"},{"key":"bib14","doi-asserted-by":"crossref","DOI":"10.7551\/mitpress\/7287.001.0001","volume-title":"WordNet: An Electronic Lexical Database","author":"Fellbaum C.","year":"1998"},{"key":"bib15","doi-asserted-by":"crossref","first-page":"266","DOI":"10.18653\/v1\/P17-1025","volume-title":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Forbes Maxwell","year":"2017"},{"key":"bib16","author":"Goldberg Yoav","year":"2019","journal-title":"arXiv preprint arXiv:1901.05287"},{"key":"bib17","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1145\/2509558.2509563","volume-title":"Proceedings of the 2013 Workshop on Automated Knowledge Base Construction","author":"Gordon Jonathan","year":"2013"},{"key":"bib18","doi-asserted-by":"crossref","first-page":"22","DOI":"10.18653\/v1\/D15-1003","volume-title":"Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing","author":"Herbelot Aur\u00e9lie","year":"2015"},{"key":"bib19","doi-asserted-by":"crossref","first-page":"2733","DOI":"10.18653\/v1\/D19-1275","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Hewitt John","year":"2019"},{"key":"bib20","first-page":"4129","volume-title":"Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT","author":"Hewitt John","year":"2019"},{"key":"bib21","author":"Jiang Zhengbao","year":"2019","journal-title":"arXiv preprint arXiv:1911.12543"},{"key":"bib22","doi-asserted-by":"crossref","first-page":"7811","DOI":"10.18653\/v1\/2020.acl-main.698","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Kassner Nora","year":"2020"},{"issue":"23","key":"bib23","doi-asserted-by":"crossref","first-page":"11213","DOI":"10.1073\/pnas.1900952116","volume":"116","author":"Kim Judy S.","year":"2019","journal-title":"Proceedings of the National Academy of Sciences"},{"key":"bib24","doi-asserted-by":"crossref","DOI":"10.1093\/acprof:oso\/9780199290932.001.0001","volume-title":"Donald Davidson\u2019s truth-theoretic semantics","author":"Lepore Ernest","year":"2007"},{"key":"bib25","first-page":"188","volume":"178","author":"Lewis David","year":"1975","journal-title":"Formal semantics-the essential readings,"},{"key":"bib26","first-page":"241","volume-title":"Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP","author":"Lin Yongjie","year":"2019"},{"key":"bib27","doi-asserted-by":"crossref","first-page":"521","DOI":"10.1162\/tacl_a_00115","volume":"4","author":"Linzen Tal","year":"2016","journal-title":"TACL"},{"key":"bib28","volume":"4","author":"Linzen Tal","year":"2016","journal-title":"Transactions of the Association for Computational Linguistics (TACL)"},{"key":"bib29","author":"Liu Yinhan","year":"2019","journal-title":"arXiv preprint arXiv:1907.11692"},{"key":"bib30","volume-title":"EMNLP","author":"Mihaylov Todor","year":"2018"},{"key":"bib31","first-page":"4885","volume-title":"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","author":"Nie Yixin","year":"2020"},{"key":"bib32","first-page":"1532","volume-title":"Empirical Methods in Natural Language Processing (EMNLP)","author":"Pennington J.","year":"2014"},{"key":"bib33","volume-title":"North American Association for Computational Linguistics (NAACL)","author":"Peters M. E.","year":"2018"},{"key":"bib34","doi-asserted-by":"crossref","first-page":"1499","DOI":"10.18653\/v1\/D18-1179","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Peters Matthew","year":"2018"},{"key":"bib35","doi-asserted-by":"crossref","first-page":"2463","DOI":"10.18653\/v1\/D19-1250","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Petroni Fabio","year":"2019"},{"key":"bib36","first-page":"pages 2858\u2013page","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Pezzelle Sandro","year":"2019"},{"issue":"8","key":"bib37","volume":"1","author":"Radford Alec","year":"2019","journal-title":"OpenAI Blog"},{"key":"bib38","doi-asserted-by":"crossref","first-page":"pages 196\u2013pages","DOI":"10.18653\/v1\/K19-1019","volume-title":"Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)","author":"Rozen Ohad","year":"2019"},{"key":"bib39","doi-asserted-by":"crossref","first-page":"pages 1715\u2013page","DOI":"10.18653\/v1\/P16-1162","volume-title":"Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Sennrich Rico","year":"2016"},{"key":"bib40","volume-title":"Transactions of the Association for Computational Linguistics (TACL)","author":"Shwartz Vered","year":"2019"},{"key":"bib41","volume-title":"Thirty-First AAAI Conference on Artificial Intelligence","author":"Speer Robyn","year":"2017"},{"key":"bib42","volume-title":"North American Association for Computational Linguistics (NAACL)","author":"Talmor A.","year":"2018"},{"key":"bib43","volume-title":"North American Association for Computational Linguistics (NAACL)","author":"Talmor A.","year":"2019"},{"key":"bib44","doi-asserted-by":"crossref","first-page":"4593","DOI":"10.18653\/v1\/P19-1452","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Tenney Ian","year":"2019"},{"key":"bib45","volume-title":"International Conference on Learning Representations","author":"Tenney Ian","year":"2019"},{"key":"bib46","first-page":"5998","volume-title":"Advances in Neural Information Processing Systems","author":"Vaswani Ashish","year":"2017"},{"key":"bib47","doi-asserted-by":"crossref","DOI":"10.1145\/2629489","volume":"57","author":"Vrande\u010di\u0107 D.","year":"2014","journal-title":"Communications of the ACM"},{"key":"bib48","first-page":"5310","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Wallace Eric","year":"2019"},{"key":"bib49","first-page":"2786","volume-title":"Advances in Neural Information Processing Systems 30","author":"Wang Mingzhe","year":"2017"},{"key":"bib50","first-page":"2870","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Warstadt Alex","year":"2019"},{"key":"bib51","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00021"},{"key":"bib52","doi-asserted-by":"crossref","first-page":"pages 644\u2013pages","DOI":"10.18653\/v1\/P18-2102","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)","author":"Yang Yiben","year":"2018"},{"key":"bib53","volume-title":"Empirical Methods in Natural Language Processing (EMNLP)","author":"Yang Z.","year":"2018"},{"key":"bib54","first-page":"5753","volume-title":"Advances in neural information processing systems","author":"Yang Zhilin","year":"2019"},{"key":"bib55","author":"Yogatama D.","year":"2019","journal-title":"arXiv preprint arXiv:1901.11373"},{"key":"bib56","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Zellers Rowan","year":"2018"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/tacl_a_00342","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,3,12]],"date-time":"2021-03-12T21:39:47Z","timestamp":1615585187000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/96476"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,12]]},"references-count":56,"alternative-id":["10.1162\/tacl_a_00342"],"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00342","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,12]]}}}