{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T03:43:17Z","timestamp":1782790997781,"version":"3.54.5"},"reference-count":38,"publisher":"MIT Press - Journals","license":[{"start":{"date-parts":[[2021,9,10]],"date-time":"2021-09-10T00:00:00Z","timestamp":1631232000000},"content-version":"vor","delay-in-days":252,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,9,8]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Systems for Open-Domain Question Answering (OpenQA) generally depend on a retriever for finding candidate passages in a large corpus and a reader for extracting answers from those passages. In much recent work, the retriever is a learned component that uses coarse-grained vector representations of questions and passages. We argue that this modeling choice is insufficiently expressive for dealing with the complexity of natural language questions. To address this, we define ColBERT-QA, which adapts the scalable neural retrieval model ColBERT to OpenQA. ColBERT creates fine-grained interactions between questions and passages. We propose an efficient weak supervision strategy that iteratively uses ColBERT to create its own training data. This greatly improves OpenQA retrieval on Natural Questions, SQuAD, and TriviaQA, and the resulting system attains state-of-the-art extractive OpenQA performance on all three datasets.<\/jats:p>","DOI":"10.1162\/tacl_a_00405","type":"journal-article","created":{"date-parts":[[2021,9,20]],"date-time":"2021-09-20T19:23:56Z","timestamp":1632165836000},"page":"929-944","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":42,"title":["Relevance-guided Supervision for OpenQA with ColBERT"],"prefix":"10.1162","volume":"9","author":[{"given":"Omar","family":"Khattab","sequence":"first","affiliation":[{"name":"Stanford University, United States. okhattab@stanford.edu"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Christopher","family":"Potts","sequence":"additional","affiliation":[{"name":"Stanford University, United States. cgpotts@stanford.edu"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Matei","family":"Zaharia","sequence":"additional","affiliation":[{"name":"Stanford University, United States. matei@cs.stanford.edu"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2021,9,8]]},"reference":[{"key":"2021091013023961500_bib1","doi-asserted-by":"publisher","first-page":"3490","DOI":"10.18653\/v1\/D19-1352","article-title":"Cross-domain modeling of sentence-level evidence for document retrieval","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Yilmaz","year":"2019"},{"key":"2021091013023961500_bib2","article-title":"Learning to retrieve reasoning paths over wikipedia graph for question answering","author":"Asai","year":"2020","journal-title":"ICLR 2020"},{"key":"2021091013023961500_bib3","first-page":"1533","article-title":"Semantic parsing on Freebase from question-answer pairs","volume-title":"Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing","author":"Berant","year":"2013"},{"key":"2021091013023961500_bib4","doi-asserted-by":"publisher","first-page":"1870","DOI":"10.18653\/v1\/P17-1171","article-title":"Reading Wikipedia to answer open-domain questions","volume-title":"55th Annual Meeting of the Association for Computational Linguistics, ACL 2017","author":"Chen","year":"2017"},{"key":"2021091013023961500_bib5","first-page":"9392","article-title":"Slice-based learning: A programming model for residual learning in critical data slices","volume":"32","author":"Chen","year":"2019","journal-title":"Advances in Neural Information Processing Systems"},{"issue":"1","key":"2021091013023961500_bib6","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1609\/aimag.v37i1.2636","article-title":"My computer is an honor student \u2013 but how intelligent is it? Standardized tests as a measure of AI","volume":"37","author":"Clark","year":"2016","journal-title":"AI Magazine"},{"key":"2021091013023961500_bib7","doi-asserted-by":"publisher","first-page":"65","DOI":"10.1145\/3077136.3080832","article-title":"Neural ranking models with weak supervision","volume-title":"Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval","author":"Dehghani","year":"2017"},{"key":"2021091013023961500_bib8","first-page":"4171","article-title":"BERT: Pre-training of deep bidirectional transformers for language understanding","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Devlin","year":"2019"},{"issue":"3","key":"2021091013023961500_bib9","doi-asserted-by":"publisher","first-page":"59","DOI":"10.1609\/aimag.v31i3.2303","article-title":"Building Watson: An overview of the deepQA project","volume":"31","author":"Ferrucci","year":"2010","journal-title":"AI Magazine"},{"key":"2021091013023961500_bib10","article-title":"Retrieval augmented language model pre-training","author":"Guu","year":"2020","journal-title":"ICML 2020"},{"key":"2021091013023961500_bib11","doi-asserted-by":"publisher","first-page":"325","DOI":"10.3115\/1034678.1034731","article-title":"Deep read: A reading comprehension system","volume-title":"Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics","author":"Hirschman","year":"1999"},{"key":"2021091013023961500_bib12","doi-asserted-by":"publisher","first-page":"633","DOI":"10.3115\/v1\/D14-1070","article-title":"A neural network for factoid question answering over paragraphs","volume-title":"Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Iyyer","year":"2014"},{"key":"2021091013023961500_bib13","doi-asserted-by":"publisher","DOI":"10.1109\/TBDATA.2019.2921572","article-title":"Billion-scale similarity search with GPUs","author":"Johnson","year":"2019","journal-title":"IEEE Transactions on Big Data"},{"key":"2021091013023961500_bib14","doi-asserted-by":"publisher","first-page":"1601","DOI":"10.18653\/v1\/P17-1147","article-title":"TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension","volume-title":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Joshi","year":"2017"},{"key":"2021091013023961500_bib15","doi-asserted-by":"publisher","first-page":"6769","DOI":"10.18653\/v1\/2020.emnlp-main.550","article-title":"Dense passage retrieval for open-domain question answering","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Karpukhin","year":"2020"},{"key":"2021091013023961500_bib16","doi-asserted-by":"publisher","first-page":"39","DOI":"10.1145\/3397271.3401075","article-title":"ColBERT: Efficient and effective passage search via contextualized late interaction over BERT","volume-title":"Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval","author":"Khattab","year":"2020"},{"key":"2021091013023961500_bib17","doi-asserted-by":"publisher","first-page":"576","DOI":"10.18653\/v1\/D18-1055","article-title":"Adaptive document retrieval for deep question answering","volume-title":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","author":"Kratzwald","year":"2018"},{"key":"2021091013023961500_bib18","doi-asserted-by":"publisher","first-page":"271","DOI":"10.3115\/v1\/P14-1026","article-title":"Learning to automatically solve algebra word problems","volume-title":"Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)","author":"Kushman","year":"2014"},{"key":"2021091013023961500_bib19","doi-asserted-by":"publisher","first-page":"453","DOI":"10.1162\/tacl_a_00276","article-title":"Natural questions: A benchmark for question answering research","volume":"7","author":"Kwiatkowski","year":"2019","journal-title":"Transactions of the Association for Computational Linguistics"},{"key":"2021091013023961500_bib20","doi-asserted-by":"crossref","first-page":"6086","DOI":"10.18653\/v1\/P19-1612","article-title":"Latent retrieval for weakly supervised open domain question answering","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Lee","year":"2019"},{"key":"2021091013023961500_bib21","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.703","article-title":"BART: Denoising sequenceto-sequence pre-training for natural language generation, translation, and comprehension","author":"Lewis","year":"2019","journal-title":"arXiv preprint arXiv:1910.13461"},{"key":"2021091013023961500_bib22","article-title":"Retrieval-augmented generation for knowledge-intensive NLP tasks","author":"Lewis","year":"2020","journal-title":"NeurIPS 2020"},{"key":"2021091013023961500_bib23","doi-asserted-by":"publisher","first-page":"993","DOI":"10.1145\/3331184.3331316","article-title":"Content-based weak supervision for ad-hoc re-ranking","volume-title":"Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval","author":"MacAvaney","year":"2019"},{"key":"2021091013023961500_bib24","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-1284","article-title":"A discrete hard em approach for weakly supervised question answering","author":"Min","year":"2019","journal-title":"arXiv preprint arXiv:1909.04849"},{"key":"2021091013023961500_bib25","article-title":"Knowledge guided text retrieval and reading for open domain question answering","author":"Min","year":"2019","journal-title":"arXiv preprint arXiv:1911.03868"},{"key":"2021091013023961500_bib26","article-title":"MS MARCO: A human generated machine reading comprehension dataset","volume-title":"CoCo@ NIPS","author":"Nguyen","year":"2016"},{"key":"2021091013023961500_bib27","doi-asserted-by":"publisher","first-page":"2383","DOI":"10.18653\/v1\/D16-1264","article-title":"SQuAD: 100,000+ questions for machine comprehension of text","volume-title":"Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing","author":"Rajpurkar","year":"2016"},{"key":"2021091013023961500_bib28","first-page":"193","article-title":"MCTest: A challenge dataset for the open-domain machine comprehension of text","volume-title":"Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing","author":"Richardson","year":"2013"},{"key":"2021091013023961500_bib29","volume-title":"The Probabilistic Relevance Framework: BM25 and Beyond","author":"Robertson","year":"2009"},{"key":"2021091013023961500_bib30","article-title":"Okapi at TREC-3","author":"Robertson","year":"1995","journal-title":"NIST Special Publication"},{"key":"2021091013023961500_bib31","first-page":"5998","article-title":"Attention is all you need","volume-title":"Advances in Neural Information Processing Systems 30","author":"Vaswani","year":"2017"},{"key":"2021091013023961500_bib32","doi-asserted-by":"publisher","first-page":"200","DOI":"10.1145\/345508.345577","article-title":"Building a question answering test collection","volume-title":"Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval","author":"Voorhees","year":"2000"},{"key":"2021091013023961500_bib33","doi-asserted-by":"publisher","first-page":"5881","DOI":"10.18653\/v1\/D19-1599","article-title":"Multi-passage BERT: A globally normalized bert model for open-domain question answering","volume-title":"Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)","author":"Wang","year":"2019"},{"key":"2021091013023961500_bib34","doi-asserted-by":"publisher","first-page":"38","DOI":"10.18653\/v1\/2020.emnlp-demos.6","article-title":"Transformers: State-of-the-art natural language processing","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations","author":"Wolf","year":"2020"},{"issue":"4","key":"2021091013023961500_bib35","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3239571","article-title":"Anserini: Reproducible ranking baselines using lucene","volume":"10","author":"Yang","year":"2018","journal-title":"Journal of Data and Information Quality (JDIQ)"},{"key":"2021091013023961500_bib36","doi-asserted-by":"publisher","first-page":"72","DOI":"10.18653\/v1\/N19-4013","article-title":"End-to-end open-domain question answering with bertserini","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)","author":"Yang","year":"2019"},{"key":"2021091013023961500_bib37","doi-asserted-by":"publisher","first-page":"2013","DOI":"10.18653\/v1\/D15-1237","article-title":"WikiQA: A challenge dataset for open-domain question answering","volume-title":"Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing","author":"Yi","year":"2015"},{"key":"2021091013023961500_bib38","doi-asserted-by":"publisher","first-page":"474","DOI":"10.1145\/3366423.3380131","article-title":"Selective weak supervision for neural information retrieval","volume-title":"Proceedings of The Web Conference 2020","author":"Zhang","year":"2020"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00405\/1962458\/tacl_a_00405.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00405\/1962458\/tacl_a_00405.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,9,20]],"date-time":"2021-09-20T19:24:19Z","timestamp":1632165859000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00405\/107205\/Relevance-guided-Supervision-for-OpenQA-with"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021]]},"references-count":38,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00405","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021]]},"published":{"date-parts":[[2021]]}}}