{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,30]],"date-time":"2026-09-30T05:11:52Z","timestamp":1790745112667,"version":"4.1.0"},"reference-count":67,"publisher":"MIT Press - Journals","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Transactions of the Association for Computational Linguistics"],"published-print":{"date-parts":[[2020,12]]},"abstract":"<jats:p> Open-domain question answering (QA) involves many knowledge and reasoning challenges, but are successful QA models actually learning such knowledge when trained on benchmark QA tasks? We investigate this via several new diagnostic tasks probing whether multiple-choice QA models know definitions and taxonomic reasoning\u2014two skills widespread in existing benchmarks and fundamental to more complex reasoning. We introduce a methodology for automatically building probe datasets from expert knowledge sources, allowing for systematic control and a comprehensive evaluation. We include ways to carefully control for artifacts that may arise during this process. Our evaluation confirms that transformer-based multiple-choice QA models are already predisposed to recognize certain types of structural linguistic knowledge. However, it also reveals a more nuanced picture: their performance notably degrades even with a slight increase in the number of \u201chops\u201d in the underlying taxonomic hierarchy, and with more challenging distractor candidates. Further, existing models are far from perfect when assessed at the level of clusters of semantically connected probes, such as all hypernym questions about a single concept. <\/jats:p>","DOI":"10.1162\/tacl_a_00331","type":"journal-article","created":{"date-parts":[[2020,9,16]],"date-time":"2020-09-16T13:46:25Z","timestamp":1600263985000},"page":"572-588","source":"Crossref","is-referenced-by-count":6,"title":["What Does My QA Model Know? Devising Controlled Probes Using Expert Knowledge"],"prefix":"10.1162","volume":"8","author":[{"given":"Kyle","family":"Richardson","sequence":"first","affiliation":[{"name":"Allen Institute for AI, Seattle, WA, USA."}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ashish","family":"Sabharwal","sequence":"additional","affiliation":[{"name":"Allen Institute for AI, Seattle, WA, USA."}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","reference":[{"key":"bib1","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00254"},{"key":"bib2","volume-title":"Proceedings of ICLR","author":"Bhagavatula Chandra","year":"2020"},{"key":"bib3","volume-title":"Proceedings of Machine Reading for Question Answering (MRQA) Workshop at ACL","author":"Boratko Michael","year":"2018"},{"key":"bib4","volume-title":"Proceedings of ACL","author":"Chen Qian","year":"2017"},{"key":"bib5","volume-title":"Proceedings of ACL","author":"Chrupa\u0142a Grzegorz","year":"2019"},{"key":"bib6","volume-title":"Twenty-Seventh IAAI Conference","author":"Clark Peter","year":"2015"},{"key":"bib7","author":"Clark Peter","year":"2018","journal-title":"arXiv preprint arXiv: 1803.05457"},{"key":"bib8","author":"Clark Peter","year":"2019","journal-title":"arXiv preprint arXiv:1909. 01958"},{"key":"bib9","volume-title":"Proceedings of AKBC","author":"Clark Peter","year":"2013"},{"key":"bib10","volume-title":"Proceedings of IJCAI","author":"Clark Peter","year":"2020"},{"key":"bib11","volume-title":"Proceedings of EMNLP","author":"Conneau Alexis","year":"2017"},{"issue":"1","key":"bib12","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1609\/aimag.v37i1.2637","volume":"37","author":"Davis Ernest","year":"2016","journal-title":"AI Magazine"},{"key":"bib13","volume-title":"Proceedings of NAACL","author":"Devlin Jacob","year":"2019"},{"key":"bib14","author":"Gardner Matt","year":"2018","journal-title":"arXiv preprint arXiv:1803.07640"},{"key":"bib15","volume-title":"Proceedings of EMNLP","author":"Geiger Atticus","year":"2019"},{"key":"bib16","author":"Geiger Atticus","year":"2020","journal-title":"arXiv preprint arXiv: 2004.14623"},{"key":"bib17","volume-title":"Proceedings of ACL","author":"Glockner Max","year":"2018"},{"key":"bib18","volume-title":"Proceedings of NAACL","author":"Gururangan Suchin","year":"2018"},{"key":"bib19","volume-title":"Proceedings of Recent Advances in NLP","author":"Havasi Catherine","year":"2007"},{"key":"bib20","volume-title":"Proceedings of EMNLP","author":"Hewitt John","year":"2019"},{"key":"bib21","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1162\/tacl_a_00080","volume":"4","author":"Hill Felix","year":"2016","journal-title":"TACL"},{"key":"bib22","volume-title":"Proceedings of ACL","author":"Jauhar Sujay Kumar","year":"2016"},{"key":"bib23","volume-title":"Proceedings of EMNLP","author":"Jia Robin","year":"2017"},{"key":"bib24","author":"Jiang Zhengbao","year":"2019","journal-title":"arXiv preprint arXiv: 1911.12543"},{"key":"bib25","volume-title":"Proceedings of ACL","author":"Kang Dongyeop","year":"2018"},{"key":"bib26","volume-title":"Proceedings of CVPR","author":"Kembhavi Aniruddha","year":"2017"},{"key":"bib27","volume-title":"Proceedings of AAAI","author":"Khot Tushar","year":"2020"},{"key":"bib28","volume-title":"Proceedings of EMNLP","author":"Lai Guokun","year":"2017"},{"key":"bib29","volume-title":"Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications","author":"Liang Chen","year":"2018"},{"key":"bib30","volume-title":"Proceedings of NAACL","author":"Liu Nelson F.","year":"2019"},{"key":"bib31","author":"Liu Yinhan","year":"2019","journal-title":"arXiv preprint arXiv:1907.11692"},{"key":"bib32","volume-title":"Proceedings of ACL","author":"Thomas McCoy R.","year":"2019"},{"key":"bib33","volume-title":"Proceedings of EMNLP","author":"Mihaylov Todor","year":"2018"},{"issue":"11","key":"bib34","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1145\/219717.219748","volume":"38","author":"Miller George A.","year":"1995","journal-title":"Communications of the ACM"},{"key":"bib35","author":"Nangia Nikita","year":"2019","journal-title":"arXiv preprint arXiv:1905.10425"},{"key":"bib36","volume-title":"Proceedings of ACL","author":"Navigli Roberto","year":"2010"},{"key":"bib37","volume-title":"Proceedings of the International Conference on Formal Ontology in Information Systems-Volume","author":"Niles Ian","year":"2001"},{"key":"bib38","volume-title":"Proceedings of EMNLP","author":"Pennington Jeffrey","year":"2014"},{"key":"bib39","volume-title":"Proceedings of NAACL","author":"Peters Matthew E.","year":"2018"},{"key":"bib40","volume-title":"Proceedings of EMNLP","author":"Peters Matthew E.","year":"2019"},{"key":"bib41","volume-title":"Proceedings of EMNLP","author":"Petroni Fabio","year":"2019"},{"key":"bib42","volume-title":"Proceedings of NAACL","author":"Pilehvar Mohammad Taher","year":"2019"},{"key":"bib43","volume-title":"Proceedings of EMNLP","author":"Poliak Adam","year":"2018"},{"key":"bib44","volume-title":"Proceedings of *SEM","author":"Poliak Adam","year":"2018"},{"key":"bib45","unstructured":"Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training."},{"key":"bib46","author":"Raffel Colin","year":"2019","journal-title":"arXiv preprint arXiv:1910.10683"},{"key":"bib47","volume-title":"Proceedings of the Conference on Logic and Machine Learning in Natural Language","author":"Ranta Aarne","year":"2017"},{"key":"bib48","volume-title":"Proceedings of EACL","author":"Reddy Sathish","year":"2017"},{"key":"bib49","volume-title":"Proceedings of AAAI","author":"Richardson Kyle","year":"2020"},{"key":"bib50","volume-title":"Proceedings of CoNLL","author":"Rozen Ohad","year":"2019"},{"key":"bib51","author":"Sakaguchi Keisuke","year":"2020","journal-title":"Proceedings of AAAI"},{"key":"bib52","volume-title":"Proceedings of EMNLP","author":"Sap Maarten","year":"2019"},{"key":"bib53","author":"Schick Timo","year":"2020","journal-title":"Proceedings of AAAI"},{"key":"bib54","volume-title":"Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval","author":"Seyler Dominic","year":"2017"},{"key":"bib55","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","author":"Shah Meet","year":"2019"},{"key":"bib56","author":"Si Chenglei","year":"2019","journal-title":"arXiv preprint arXiv:1910.12391"},{"key":"bib57","author":"Talmor Alon","year":"2019","journal-title":"ArXiv: arXiv preprint arXiv:1912.13283"},{"key":"bib58","volume-title":"Proceedings of NAACL","author":"Talmor Alon","year":"2019"},{"key":"bib59","volume-title":"Proceedings of ACL","author":"Tandon Niket","year":"2017"},{"key":"bib60","first-page":"5998","volume-title":"Proceedings of NeurIPS","author":"Vaswani Ashish","year":"2017"},{"key":"bib61","author":"Vig Jesse","year":"2020","journal-title":"arXiv preprint arXiv:2004.12265"},{"key":"bib62","volume-title":"Proceedings of EMNLP","author":"Warstadt Alex","year":"2019"},{"key":"bib63","volume-title":"Webster\u2019s revised unabridged dictionary of the english language","author":"Webster Noah","year":"1913"},{"key":"bib64","volume-title":"Proceedings of the 3rd Workshop on Noisy User-generated Text","author":"Welbl Johannes","year":"2017"},{"key":"bib65","author":"Wolf Thomas","year":"2019","journal-title":"arXiv preprint arXiv:1910.03771"},{"key":"bib66","volume-title":"Proceedings of ACL","author":"Yanaka Hitomi","year":"2020"},{"key":"bib67","volume-title":"Proceedings of EMNLP","author":"Zellers Rowan","year":"2018"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/tacl_a_00331","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,3,12]],"date-time":"2021-03-12T21:39:43Z","timestamp":1615585183000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/96481"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,12]]},"references-count":67,"alternative-id":["10.1162\/tacl_a_00331"],"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00331","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,12]]}}}