{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,12]],"date-time":"2026-05-12T04:22:37Z","timestamp":1778559757255,"version":"3.51.4"},"reference-count":25,"publisher":"MIT Press - Journals","content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Transactions of the Association for Computational Linguistics"],"published-print":{"date-parts":[[2019,11]]},"abstract":"<jats:p> We present the Natural Questions corpus, a question answering data set. Questions consist of real anonymized, aggregated queries issued to the Google search engine. An annotator is presented with a question along with a Wikipedia page from the top 5 search results, and annotates a long answer (typically a paragraph) and a short answer (one or more entities) if present on the page, or marks null if no long\/short answer is present. The public release consists of 307,373 training examples with single annotations; 7,830 examples with 5-way annotations for development data; and a further 7,842 examples with 5-way annotated sequestered as test data. We present experiments validating quality of the data. We also describe analysis of 25-way annotations on 302 examples, giving insights into human variability on the annotation task. We introduce robust metrics for the purposes of evaluating question answering systems; demonstrate high human upper bounds on these metrics; and establish baseline results using competitive methods drawn from related literature. <\/jats:p>","DOI":"10.1162\/tacl_a_00276","type":"journal-article","created":{"date-parts":[[2019,8,2]],"date-time":"2019-08-02T18:32:33Z","timestamp":1564770753000},"page":"453-466","source":"Crossref","is-referenced-by-count":923,"title":["Natural Questions: A Benchmark for Question Answering                     Research"],"prefix":"10.1162","volume":"7","author":[{"given":"Tom","family":"Kwiatkowski","sequence":"first","affiliation":[{"name":"Google Research."}]},{"given":"Jennimaria","family":"Palomaki","sequence":"additional","affiliation":[{"name":"Google Research"}]},{"given":"Olivia","family":"Redfield","sequence":"additional","affiliation":[{"name":"Google Research"}]},{"given":"Michael","family":"Collins","sequence":"additional","affiliation":[{"name":"Google Research"}]},{"given":"Ankur","family":"Parikh","sequence":"additional","affiliation":[{"name":"Google Research"}]},{"given":"Chris","family":"Alberti","sequence":"additional","affiliation":[{"name":"Google Research"}]},{"given":"Danielle","family":"Epstein","sequence":"additional","affiliation":[{"name":"Google Research"}]},{"given":"Illia","family":"Polosukhin","sequence":"additional","affiliation":[{"name":"Google Research"}]},{"given":"Jacob","family":"Devlin","sequence":"additional","affiliation":[{"name":"Google Research"}]},{"given":"Kenton","family":"Lee","sequence":"additional","affiliation":[{"name":"Google Research"}]},{"given":"Kristina","family":"Toutanova","sequence":"additional","affiliation":[{"name":"Google Research"}]},{"given":"Llion","family":"Jones","sequence":"additional","affiliation":[{"name":"Google Research"}]},{"given":"Matthew","family":"Kelcey","sequence":"additional","affiliation":[{"name":"Google Research"}]},{"given":"Ming-Wei","family":"Chang","sequence":"additional","affiliation":[{"name":"Google Research"}]},{"given":"Andrew M.","family":"Dai","sequence":"additional","affiliation":[{"name":"Google Research"}]},{"given":"Jakob","family":"Uszkoreit","sequence":"additional","affiliation":[{"name":"Google Research"}]},{"given":"Quoc","family":"Le","sequence":"additional","affiliation":[{"name":"Google Research"}]},{"given":"Slav","family":"Petrov","sequence":"additional","affiliation":[{"name":"Google Research"}]}],"member":"281","reference":[{"key":"bib2","doi-asserted-by":"crossref","first-page":"632","DOI":"10.18653\/v1\/D15-1075","volume-title":"Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing","author":"Bowman Samuel R.","year":"2015"},{"key":"bib3","doi-asserted-by":"crossref","first-page":"1870","DOI":"10.18653\/v1\/P17-1171","volume-title":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Chen Danqi","year":"2017"},{"key":"bib4","doi-asserted-by":"crossref","first-page":"2174","DOI":"10.18653\/v1\/D18-1241","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Choi Eunsol","year":"2018"},{"key":"bib5","doi-asserted-by":"crossref","first-page":"845","DOI":"10.18653\/v1\/P18-1078","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Clark Christopher","year":"2018"},{"key":"bib6","volume-title":"A Probabilistic Theory of Pattern Recognition","volume":"31","author":"Devroye Luc","year":"1997","edition":"2"},{"key":"bib7","doi-asserted-by":"crossref","first-page":"37","DOI":"10.18653\/v1\/W18-2605","volume-title":"Proceedings of the Workshop on Machine Reading for Question Answering","author":"He Wei","year":"2018"},{"key":"bib8","volume-title":"COLING 1992 Volume 2: The 15th International Conference on Computational Linguistics","author":"Hearst Marti A.","year":"1992"},{"issue":"15","key":"bib9","volume-title":"Proceedings of the 28th International Conference on Neural Information Processing Systems","author":"Hermann Karl Moritz","year":"2015"},{"key":"bib10","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Hill Felix","year":"2015"},{"key":"bib11","first-page":"2021","volume-title":"Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing","author":"Jia Robin","year":"2017"},{"key":"bib12","doi-asserted-by":"crossref","first-page":"1601","DOI":"10.18653\/v1\/P17-1147","volume-title":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Joshi Mandar","year":"2017"},{"key":"bib13","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00023"},{"key":"bib14","first-page":"785","volume-title":"Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing","author":"Lai Guokun","year":"2017"},{"key":"bib15","doi-asserted-by":"crossref","first-page":"2381","DOI":"10.18653\/v1\/D18-1260","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Mihaylov Todor","year":"2018"},{"key":"bib16","volume-title":"Proceedings of the Workshop on Cognitive Computation: Integrating Neural and Symbolic Approaches","author":"Nguyen Tri","year":"2016"},{"key":"bib17","doi-asserted-by":"crossref","first-page":"2230","DOI":"10.18653\/v1\/D16-1241","volume-title":"Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing","author":"Onishi Takeshi","year":"2016"},{"key":"bib18","doi-asserted-by":"crossref","first-page":"1525","DOI":"10.18653\/v1\/P16-1144","volume-title":"Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Paperno Denis","year":"2016"},{"key":"bib19","first-page":"311","volume-title":"Proceedings of 40th Annual Meeting of the Association for Computational Linguistics","author":"Papineni Kishore","year":"2002"},{"key":"bib20","doi-asserted-by":"crossref","first-page":"2249","DOI":"10.18653\/v1\/D16-1244","volume-title":"Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing","author":"Parikh Ankur","year":"2016"},{"key":"bib21","doi-asserted-by":"crossref","first-page":"784","DOI":"10.18653\/v1\/P18-2124","volume-title":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)","author":"Rajpurkar Pranav","year":"2018"},{"key":"bib22","doi-asserted-by":"crossref","first-page":"2383","DOI":"10.18653\/v1\/D16-1264","volume-title":"Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing","author":"Rajpurkar Pranav","year":"2016"},{"key":"bib24","first-page":"193","volume-title":"Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing","author":"Richardson Matthew","year":"2013"},{"key":"bib25","first-page":"1112","volume-title":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)","author":"Williams Adina","year":"2018"},{"key":"bib26","doi-asserted-by":"crossref","first-page":"2013","DOI":"10.18653\/v1\/D15-1237","volume-title":"Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing","author":"Yi Yang","year":"2015"},{"key":"bib27","doi-asserted-by":"crossref","first-page":"2369","DOI":"10.18653\/v1\/D18-1259","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Yang Zhilin","year":"2018"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mitpressjournals.org\/doi\/pdf\/10.1162\/tacl_a_00276","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,3,12]],"date-time":"2021-03-12T21:39:25Z","timestamp":1615585165000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/43518"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,11]]},"references-count":25,"alternative-id":["10.1162\/tacl_a_00276"],"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00276","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,11]]}}}