{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,16]],"date-time":"2026-07-16T05:03:54Z","timestamp":1784178234792,"version":"3.55.0"},"reference-count":48,"publisher":"MIT Press - Journals","license":[{"start":{"date-parts":[[2021,4,28]],"date-time":"2021-04-28T00:00:00Z","timestamp":1619568000000},"content-version":"vor","delay-in-days":117,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["direct.mit.edu"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,4,26]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Dual encoders perform retrieval by encoding documents and queries into dense low-dimensional vectors, scoring each document by its inner product with the query. We investigate the capacity of this architecture relative to sparse bag-of-words models and attentional neural networks. Using both theoretical and empirical analysis, we establish connections between the encoding dimension, the margin between gold and lower-ranked documents, and the document length, suggesting limitations in the capacity of fixed-length encodings to support precise retrieval of long documents. Building on these insights, we propose a simple neural model that combines the efficiency of dual encoders with some of the expressiveness of more costly attentional architectures, and explore sparse-dense hybrids to capitalize on the precision of sparse retrieval. These models outperform strong alternatives in large-scale retrieval.<\/jats:p>","DOI":"10.1162\/tacl_a_00369","type":"journal-article","created":{"date-parts":[[2021,4,28]],"date-time":"2021-04-28T23:50:23Z","timestamp":1619653823000},"page":"329-345","update-policy":"https:\/\/doi.org\/10.1162\/mitpressjournals.corrections.policy","source":"Crossref","is-referenced-by-count":196,"title":["Sparse, Dense, and Attentional Representations for Text Retrieval"],"prefix":"10.1162","volume":"9","author":[{"given":"Yi","family":"Luan","sequence":"first","affiliation":[{"name":"Google Research, United States. luanyi@google.com"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jacob","family":"Eisenstein","sequence":"additional","affiliation":[{"name":"Google Research, United States. jeisenstein@google.com"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Kristina","family":"Toutanova","sequence":"additional","affiliation":[{"name":"Google Research, United States. kristout@google.com"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael","family":"Collins","sequence":"additional","affiliation":[{"name":"Google Research, United States. mjcollins@google.com"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"281","published-online":{"date-parts":[[2021,4,26]]},"reference":[{"issue":"4","key":"2021060823400738300_bib1","doi-asserted-by":"crossref","first-page":"671","DOI":"10.1016\/S0022-0000(03)00025-4","article-title":"Database-friendly random projections: Johnson-Lindenstrauss with binary coins","volume":"66","author":"Achlioptas","year":"2003","journal-title":"Journal of Computer and System Sciences"},{"key":"2021060823400738300_bib2","doi-asserted-by":"crossref","DOI":"10.1109\/FOCS.2017.65","article-title":"Optimal compression of approximate inner products and dimension reduction","volume-title":"58th Annual Symposium on Foundations of Computer Science (FOCS)","author":"Alon","year":"2017"},{"key":"2021060823400738300_bib3","doi-asserted-by":"crossref","DOI":"10.1142\/9789813272880_0182","article-title":"Approximate nearest neighbor search in high dimensions","author":"Andoni","year":"2019","journal-title":"Proceedings of the International Congress of Mathematicians (ICM 2018)"},{"key":"2021060823400738300_bib4","article-title":"A compressed sensing view of unsupervised text embeddings, bag-of-n-grams, and LSTMs","volume-title":"Proceedings of the International Conference on Learning Representations (ICLR)","author":"Arora","year":"2018"},{"issue":"Nov","key":"2021060823400738300_bib5","first-page":"441","article-title":"Limitations of learning via embeddings in Euclidean half spaces","volume":"3","author":"Ben-David","year":"2002","journal-title":"Journal of Machine Learning Research"},{"issue":"12","key":"2021060823400738300_bib6","doi-asserted-by":"crossref","first-page":"4203","DOI":"10.1109\/TIT.2005.858979","article-title":"Decoding by linear programming","volume":"51","author":"Candes","year":"2005","journal-title":"IEEE Transactions on Information Theory"},{"key":"2021060823400738300_bib7","article-title":"Overview of the TREC 2019 deep learning track","volume-title":"Text REtrieval Conference (TREC)","author":"Craswell","year":"2020"},{"key":"2021060823400738300_bib8","doi-asserted-by":"crossref","DOI":"10.1145\/3366423.3380258","article-title":"Context-aware document term weighting for ad-hoc search","volume-title":"Proceedings of The Web Conference 2020","author":"Dai","year":"2020"},{"key":"2021060823400738300_bib9","article-title":"Context-aware sentence\/passage term importance estimation for first stage retrieval","volume-title":"Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval","author":"Dai","year":"2020"},{"key":"2021060823400738300_bib10","article-title":"BERT: Pre-training of deep bidirectional transformers for language understanding","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","author":"Devlin","year":"2019"},{"key":"2021060823400738300_bib11","article-title":"TREC complex answer retrieval overview","volume-title":"Text REtrieval Conference (TREC)","author":"Dietz","year":"2018"},{"key":"2021060823400738300_bib12","article-title":"Complementing lexical retrieval with semantic residual embedding","author":"Gao","year":"2020","journal-title":"CoRR"},{"key":"2021060823400738300_bib13","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/K19-1049","article-title":"Learning dense representations for entity retrieval","volume-title":"Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)","author":"Gillick","year":"2019"},{"key":"2021060823400738300_bib14","doi-asserted-by":"crossref","DOI":"10.1145\/2983323.2983769","article-title":"A deep relevance matching model for ad-hoc retrieval","volume-title":"Proceedings of the 25th ACM International on Conference on Information and Knowledge Management","author":"Guo","year":"2016"},{"key":"2021060823400738300_bib15","article-title":"Quantization based fast inner product search","volume-title":"Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS)","author":"Guo","year":"2016"},{"key":"2021060823400738300_bib16","article-title":"Accelerating large-scale inference with anisotropic vector quantization","volume-title":"Proceedings of the 37th International Conference on Machine Learning","author":"Guo","year":"2020"},{"key":"2021060823400738300_bib17","doi-asserted-by":"crossref","DOI":"10.1145\/2505515.2505665","article-title":"Learning deep structured semantic models for web search using clickthrough data","volume-title":"Proceedings of the International Conference on Information and Knowledge Management (CIKM)","author":"Huang","year":"2013"},{"key":"2021060823400738300_bib18","article-title":"Poly-encoders: Transformer architectures and pre-training strategies for fast and accurate multi-sentence scoring","volume-title":"Proceedings of the International Conference on Learning Representations (ICLR)","author":"Humeau","year":"2020"},{"issue":"3","key":"2021060823400738300_bib19","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/2483699.2483706","article-title":"Optimal bounds for Johnson-Lindenstrauss transforms and streaming problems with subconstant error","volume":"9","author":"Jayram","year":"2013","journal-title":"ACM Transactions on Algorithms (TALG)"},{"issue":"189\u2013206","key":"2021060823400738300_bib20","first-page":"1","article-title":"Extensions of Lipschitz mappings into a Hilbert space","volume":"26","author":"Johnson","year":"1984","journal-title":"Contemporary Mathematics"},{"key":"2021060823400738300_bib21","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-642-22935-0_53","article-title":"Almost optimal explicit Johnson-Lindenstrauss families","volume-title":"Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques","author":"Kane","year":"2011"},{"key":"2021060823400738300_bib22","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/2020.emnlp-main.550","article-title":"Dense passage retrieval for open-domain question answering","volume-title":"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)","author":"Karpukhin","year":"2020"},{"key":"2021060823400738300_bib23","doi-asserted-by":"crossref","DOI":"10.1145\/3397271.3401075","article-title":"ColBERT","volume-title":"Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval","author":"Khattab","year":"2020"},{"key":"2021060823400738300_bib24","doi-asserted-by":"crossref","first-page":"453","DOI":"10.1162\/tacl_a_00276","article-title":"Natural questions: A benchmark for question answering research","volume":"7","author":"Kwiatkowski","year":"2019","journal-title":"Transactions of the Association for Computational Linguistics"},{"issue":"3","key":"2021060823400738300_bib25","doi-asserted-by":"crossref","first-page":"242","DOI":"10.1145\/502115.502117","article-title":"Scaling question answering to the web","volume":"19","author":"Kwok","year":"2001","journal-title":"ACM Transactions on Information Systems (TOIS)"},{"key":"2021060823400738300_bib26","doi-asserted-by":"crossref","DOI":"10.1109\/FOCS.2017.64","article-title":"Optimality of the Johnson-Lindenstrauss lemma","volume-title":"2017 IEEE 58th Annual Symposium on Foundations of Computer Science (FOCS)","author":"Larsen","year":"2017"},{"key":"2021060823400738300_bib27","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/P19-1612","article-title":"Latent retrieval for weakly supervised open domain question answering","volume-title":"Proceedings of the Association for Computational Linguistics (ACL)","author":"Lee","year":"2019"},{"key":"2021060823400738300_bib28","article-title":"Zero-shot neural retrieval via domain-targeted synthetic query generation","author":"Ji","year":"2020","journal-title":"CoRR"},{"issue":"1","key":"2021060823400738300_bib29","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1561\/1500000061","article-title":"An introduction to neural information retrieval","volume":"13","author":"Mitra","year":"2018","journal-title":"Foundations and Trends\u00ae in Information Retrieval"},{"key":"2021060823400738300_bib30","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/D16-1261","article-title":"Improving information extraction by acquiring external evidence with reinforcement learning","volume-title":"Proceedings of Empirical Methods in Natural Language Processing (EMNLP)","author":"Narasimhan","year":"2016"},{"key":"2021060823400738300_bib31","article-title":"MS MARCO: A human generated machine reading comprehension dataset","author":"Nguyen","year":"2016"},{"key":"2021060823400738300_bib32","article-title":"Passage re-ranking with BERT","author":"Nogueira","year":"2019","journal-title":"CoRR"},{"key":"2021060823400738300_bib33","article-title":"From doc2query to doctttttquery","author":"Nogueira","year":"2019"},{"key":"2021060823400738300_bib34","article-title":"Multi-stage document ranking with BERT","author":"Nogueira","year":"2019","journal-title":"CoRR"},{"key":"2021060823400738300_bib35","article-title":"Document expansion by query prediction","author":"Nogueira","year":"2019","journal-title":"CoRR"},{"key":"2021060823400738300_bib36","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/D19-1410","article-title":"Sentence-BERT: Sentence embeddings using Siamese BERT-networks","volume-title":"Proceedings of Empirical Methods in Natural Language Processing (EMNLP)","author":"Reimers","year":"2019"},{"issue":"4","key":"2021060823400738300_bib37","doi-asserted-by":"crossref","first-page":"333","DOI":"10.1561\/1500000019","article-title":"The probabilistic relevance framework: BM25 and beyond","volume":"3","author":"Robertson","year":"2009","journal-title":"Foundations and Trends in Information Retrieval"},{"key":"2021060823400738300_bib38","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/P19-1436","article-title":"Real-time open-domain question answering with dense-sparse phrase index","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Seo","year":"2019"},{"issue":"1","key":"2021060823400738300_bib39","doi-asserted-by":"crossref","first-page":"11","DOI":"10.1108\/eb026526","article-title":"A statistical interpretation of term specificity and its application in retrieval","volume":"28","author":"Jones","year":"1972","journal-title":"Journal of Documentation"},{"key":"2021060823400738300_bib40","article-title":"Can unconditional language models recover arbitrary sentences?","volume-title":"Advances in Neural Information Processing Systems","author":"Subramani","year":"2019"},{"key":"2021060823400738300_bib41","volume-title":"The Random Projection Method","author":"Vempala","year":"2004"},{"issue":"4","key":"2021060823400738300_bib42","doi-asserted-by":"crossref","first-page":"361","DOI":"10.1017\/S1351324901002789","article-title":"The TREC question answering track","volume":"7","author":"Voorhees","year":"2001","journal-title":"Natural Language Engineering"},{"key":"2021060823400738300_bib43","article-title":"No training required: Exploring random encoders for sentence classification","volume-title":"Proceedings of the International Conference on Learning Representations (ICLR)","author":"Wieting","year":"2019"},{"key":"2021060823400738300_bib44","article-title":"Approximate nearest neighbor negative contrastive learning for dense text retrieval","author":"Xiong","year":"2020","journal-title":"CoRR"},{"key":"2021060823400738300_bib45","article-title":"IDST at TREC 2019 deep learning track: Deep cascade ranking with generation-based document expansion and pre-trained language modeling","volume-title":"Text REtrieval Conference (TREC)","author":"Yan","year":"2020"},{"issue":"4","key":"2021060823400738300_bib46","doi-asserted-by":"crossref","DOI":"10.1145\/3239571","article-title":"Anserini: Reproducible ranking baselines using lucene","volume":"10","author":"Yang","year":"2018","journal-title":"Journal of Data and Information Quality"},{"key":"2021060823400738300_bib47","article-title":"Simple applications of BERT for ad hoc document retrieval","author":"Yang","year":"2019","journal-title":"CoRR"},{"key":"2021060823400738300_bib48","article-title":"Breaking the softmax bottleneck: A high-rank RNN language model","volume-title":"Proceedings of the International Conference on Learning Representations (ICLR)","author":"Yang","year":"2018"}],"container-title":["Transactions of the Association for Computational Linguistics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00369\/1924040\/tacl_a_00369.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"http:\/\/direct.mit.edu\/tacl\/article-pdf\/doi\/10.1162\/tacl_a_00369\/1924040\/tacl_a_00369.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,6,9]],"date-time":"2021-06-09T09:37:10Z","timestamp":1623231430000},"score":1,"resource":{"primary":{"URL":"https:\/\/direct.mit.edu\/tacl\/article\/doi\/10.1162\/tacl_a_00369\/100684\/Sparse-Dense-and-Attentional-Representations-for"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021]]},"references-count":48,"URL":"https:\/\/doi.org\/10.1162\/tacl_a_00369","relation":{},"ISSN":["2307-387X"],"issn-type":[{"value":"2307-387X","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2021]]},"published":{"date-parts":[[2021]]}}}