{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,3]],"date-time":"2026-11-03T13:01:36Z","timestamp":1793710896105,"version":"4.6.1"},"reference-count":60,"publisher":"Association for Computing Machinery (ACM)","issue":"5","license":[{"start":{"date-parts":[[2024,4,29]],"date-time":"2024-04-29T00:00:00Z","timestamp":1714348800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Netherlands Organization for Scientific Research","award":["NWO STW # 17752; NWO CI # CISC.CC.016"],"award-info":[{"award-number":["NWO STW # 17752; NWO CI # CISC.CC.016"]}]},{"name":"Facebook Research"},{"name":"Innovation Exchange Amsterdam"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2024,9,30]]},"abstract":"<jats:p>Modern transformer-based information retrieval models achieve state-of-the-art performance across various benchmarks. The self-attention of the transformer models is a powerful mechanism to contextualize terms over the whole input but quickly becomes prohibitively expensive for long input as required in document retrieval. Instead of focusing on the model itself to improve efficiency, this paper explores different bag of words document representations that encode full documents by only a fraction of their characteristic terms, allowing us to control and reduce the input length. We experiment with various models for document retrieval on MS MARCO data, as well as zero-shot document retrieval on Robust04, and show large gains in efficiency while retaining reasonable effectiveness. Inference time efficiency gains are both lowering the time and memory complexity in a controllable way, allowing for further trading off memory footprint and query latency. More generally, this line of research connects traditional IR models with neural \u201cNLP\u201d models and offers novel ways to explore the space between (efficient, but less effective) traditional rankers and (effective, but less efficient) neural rankers elegantly.<\/jats:p>","DOI":"10.1145\/3640460","type":"journal-article","created":{"date-parts":[[2024,2,9]],"date-time":"2024-02-09T06:53:57Z","timestamp":1707461637000},"page":"1-27","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":8,"title":["Revisiting Bag of Words Document Representations for Efficient Ranking with Transformers"],"prefix":"10.1145","volume":"42","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1964-1356","authenticated-orcid":false,"given":"David","family":"Rau","sequence":"first","affiliation":[{"name":"ILLC, University of Amsterdam, Amsterdam, Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9772-1095","authenticated-orcid":false,"given":"Mostafa","family":"Dehghani","sequence":"additional","affiliation":[{"name":"Deep Mind, San Francisco, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6614-0087","authenticated-orcid":false,"given":"Jaap","family":"Kamps","sequence":"additional","affiliation":[{"name":"University of Amsterdam, Amsterdam, Netherlands"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,4,29]]},"reference":[{"key":"e_1_3_2_2_2","unstructured":"Payal Bajaj Daniel Campos Nick Craswell Li Deng Jianfeng Gao Xiaodong Liu Rangan Majumder Andrew McNamara Bhaskar Mitra Tri Nguyen et\u00a0al. 2016. MS MARCO: A human generated machine reading comprehension dataset. (2016). arxiv:1611.09268"},{"key":"e_1_3_2_3_2","article-title":"Longformer: The long-document transformer","author":"Beltagy Iz","year":"2020","unstructured":"Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv:2004.05150 (2020).","journal-title":"arXiv:2004.05150"},{"key":"e_1_3_2_4_2","article-title":"Generating long sequences with sparse transformers","volume":"1904","author":"Child Rewon","year":"2019","unstructured":"Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating long sequences with sparse transformers. CoRR abs\/1904.10509 (2019). arXiv:1904.10509http:\/\/arxiv.org\/abs\/1904.10509","journal-title":"CoRR"},{"key":"e_1_3_2_5_2","unstructured":"Nick Craswell Bhaskar Mitra Emine Yilmaz and Daniel Campos. 2021. Overview of the TREC 2020 deep learning track. (2021). arxiv:2102.07662"},{"key":"e_1_3_2_6_2","doi-asserted-by":"crossref","unstructured":"Nick Craswell Bhaskar Mitra Emine Yilmaz Daniel Campos and Ellen M. Voorhees. 2020. Overview of the TREC 2019 deep learning track. (2020). arxiv:2003.07820","DOI":"10.6028\/NIST.SP.1266.deep-overview"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/3331184.3331303"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401204"},{"key":"e_1_3_2_9_2","article-title":"Transformer-XL: Attentive language models beyond a fixed-length context","author":"Dai Zihang","year":"2019","unstructured":"Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860 (2019).","journal-title":"arXiv preprint arXiv:1901.02860"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/2983323.2983814"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/2970398.2970408"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/3077136.3080832"},{"key":"e_1_3_2_13_2","unstructured":"Jacob Devlin Ming-Wei Chang Kenton Lee and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. (2018). arxiv:1810.04805"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","unstructured":"Thibault Formal Carlos Lassance Benjamin Piwowarski and St\u00e9phane Clinchant. 2022. From distillation to hard negative sampling: Making sparse neural IR models more effective(SIGIR \u201922). Association for Computing Machinery New York NY USA 2353\u20132359. 10.1145\/3477495.3531857","DOI":"10.1145\/3477495.3531857"},{"key":"e_1_3_2_15_2","article-title":"COIL: Revisit exact lexical match in information retrieval with contextualized inverted list","author":"Gao Luyu","year":"2021","unstructured":"Luyu Gao, Zhuyun Dai, and Jamie Callan. 2021. COIL: Revisit exact lexical match in information retrieval with contextualized inverted list. arXiv preprint arXiv:2104.07186 (2021).","journal-title":"arXiv preprint arXiv:2104.07186"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV.2005.239"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/1008992.1009025"},{"key":"e_1_3_2_18_2","article-title":"Mitigating the position bias of transformer models in passage re-ranking","volume":"2101","author":"Hofst\u00e4tter Sebastian","year":"2021","unstructured":"Sebastian Hofst\u00e4tter, Aldo Lipani, Sophia Althammer, Markus Zlabinger, and Allan Hanbury. 2021. Mitigating the position bias of transformer models in passage re-ranking. CoRR abs\/2101.06980 (2021). arXiv:2101.06980https:\/\/arxiv.org\/abs\/2101.06980","journal-title":"CoRR"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/3404835.3462889"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/3331184.3331344"},{"key":"e_1_3_2_21_2","article-title":"Local self-attention over long text for efficient document retrieval","volume":"2005","author":"Hofst\u00e4tter Sebastian","year":"2020","unstructured":"Sebastian Hofst\u00e4tter, Hamed Zamani, Bhaskar Mitra, Nick Craswell, and Allan Hanbury. 2020. Local self-attention over long text for efficient document retrieval. CoRR abs\/2005.04908 (2020). arXiv:2005.04908https:\/\/arxiv.org\/abs\/2005.04908","journal-title":"CoRR"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/2505515.2505665"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.findings-emnlp.412"},{"key":"e_1_3_2_24_2","article-title":"A statistical interpretation of term specificity and its application in retrieval","author":"Jones Karen Sparck","year":"1972","unstructured":"Karen Sparck Jones. 1972. A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation (1972).","journal-title":"Journal of Documentation"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-12275-0_48"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401075"},{"key":"e_1_3_2_27_2","article-title":"Adam: A method for stochastic optimization","author":"Kingma Diederik P.","year":"2014","unstructured":"Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).","journal-title":"arXiv preprint arXiv:1412.6980"},{"key":"e_1_3_2_28_2","article-title":"Reformer: The efficient transformer","author":"Kitaev Nikita","year":"2020","unstructured":"Nikita Kitaev, \u0141ukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451 (2020).","journal-title":"arXiv preprint arXiv:2001.04451"},{"key":"e_1_3_2_29_2","article-title":"PARADE: Passage representation aggregation for document reranking","volume":"2008","author":"Li Canjia","year":"2020","unstructured":"Canjia Li, Andrew Yates, Sean MacAvaney, Ben He, and Yingfei Sun. 2020. PARADE: Passage representation aggregation for document reranking. CoRR abs\/2008.09093 (2020). arXiv:2008.09093https:\/\/arxiv.org\/abs\/2008.09093","journal-title":"CoRR"},{"key":"e_1_3_2_30_2","article-title":"Distilling dense representations for ranking using tightly-coupled teachers","author":"Lin Sheng-Chieh","year":"2020","unstructured":"Sheng-Chieh Lin, Jheng-Hong Yang, and Jimmy Lin. 2020. Distilling dense representations for ranking using tightly-coupled teachers. arXiv preprint arXiv:2010.11386 (2020).","journal-title":"arXiv preprint arXiv:2010.11386"},{"key":"e_1_3_2_31_2","unstructured":"Nelson F. Liu Kevin Lin John Hewitt Ashwin Paranjape Michele Bevilacqua Fabio Petroni and Percy Liang. 2023. Lost in the Middle: How Language Models Use Long Contexts. arxiv:2307.03172 [cs.CL]"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1147\/rd.22.0159"},{"key":"e_1_3_2_33_2","unstructured":"Sean MacAvaney Sergey Feldman Nazli Goharian Doug Downey and Arman Cohan. 2020. ABNIRML: Analyzing the Behavior of Neural IR Models. (2020). arxiv:2011.00696"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401262"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401263"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.5555\/2999792.2999959"},{"key":"e_1_3_2_37_2","article-title":"Incorporating query term independence assumption for efficient retrieval and ranking using deep neural networks","volume":"1907","author":"Mitra Bhaskar","year":"2019","unstructured":"Bhaskar Mitra, Corby Rosset, David Hawking, Nick Craswell, Fernando D. Diaz, and Emine Yilmaz. 2019. Incorporating query term independence assumption for efficient retrieval and ranking using deep neural networks. ArXiv abs\/1907.03693 (2019).","journal-title":"ArXiv"},{"key":"e_1_3_2_38_2","unstructured":"Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage re-ranking with BERT. (2019). arxiv:1901.04085"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.findings-emnlp.63"},{"key":"e_1_3_2_40_2","article-title":"PyTorch: An imperative style, high-performance deep learning library","volume":"1912","author":"Paszke Adam","year":"2019","unstructured":"Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas K\u00f6pf, Edward Z. Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An imperative style, high-performance deep learning library. CoRR abs\/1912.01703 (2019). arXiv:1912.01703http:\/\/arxiv.org\/abs\/1912.01703","journal-title":"CoRR"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.findings-acl.98"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/3539813.3545144"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1561\/1500000019"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1002\/asi.4630270302"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00353"},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.1108\/eb026526"},{"key":"e_1_3_2_47_2","article-title":"Adaptive attention span in transformers","author":"Sukhbaatar Sainbayar","year":"2019","unstructured":"Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. 2019. Adaptive attention span in transformers. arXiv preprint arXiv:1905.07799 (2019).","journal-title":"arXiv preprint arXiv:1905.07799"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2310.07712"},{"key":"e_1_3_2_49_2","first-page":"9438","volume-title":"International Conference on Machine Learning","author":"Tay Yi","year":"2020","unstructured":"Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan. 2020. Sparse Sinkhorn Attention. In International Conference on Machine Learning. PMLR, 9438\u20139447."},{"key":"e_1_3_2_50_2","article-title":"Long range arena: A benchmark for efficient transformers","volume":"2011","author":"Tay Yi","year":"2020","unstructured":"Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. 2020. Long range arena: A benchmark for efficient transformers. CoRR abs\/2011.04006 (2020). arXiv:2011.04006https:\/\/arxiv.org\/abs\/2011.04006","journal-title":"CoRR"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1145\/3530811"},{"key":"e_1_3_2_52_2","article-title":"Linformer: Self-attention with linear complexity","author":"Wang Sinong","year":"2020","unstructured":"Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. 2020. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768 (2020).","journal-title":"arXiv preprint arXiv:2006.04768"},{"key":"e_1_3_2_53_2","article-title":"HuggingFace\u2019s transformers: State-of-the-art natural language processing","volume":"1910","author":"Wolf Thomas","year":"2019","unstructured":"Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R\u00e9mi Louf, Morgan Funtowicz, and Jamie Brew. 2019. HuggingFace\u2019s transformers: State-of-the-art natural language processing. CoRR abs\/1910.03771 (2019). arXiv:1910.03771http:\/\/arxiv.org\/abs\/1910.03771","journal-title":"CoRR"},{"key":"e_1_3_2_54_2","volume-title":"International Conference on Learning Representations (ICLR)","author":"Wu Zhanghao","year":"2020","unstructured":"Zhanghao Wu, Zhijian Liu, Ji Lin, Yujun Lin, and Song Han. 2020. Lite transformer with long-short range attention. In International Conference on Learning Representations (ICLR)."},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1145\/3077136.3080809"},{"key":"e_1_3_2_56_2","volume-title":"TREC","author":"Yan Ming","year":"2019","unstructured":"Ming Yan, Chenliang Li, Chen Wu, Bin Bi, Wei Wang, Jiangnan Xia, and Luo Si. 2019. IDST at TREC 2019 deep learning track: Deep cascade ranking with generation-based document expansion and pre-trained language modeling. In TREC."},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/3077136.3080721"},{"key":"e_1_3_2_58_2","unstructured":"Manzil Zaheer Guru Prashanth Guruganesh Avinava Dubey Joshua Ainslie Chris Alberti Santiago Ontanon Philip Minh Pham Anirudh Ravula Qifan Wang Li Yang and Amr Mahmoud El Houssieny Ahmed. 2020. Big Bird: Transformers for longer sequences. https:\/\/proceedings.neurips.cc\/paper\/2020\/file\/c8512d142a2d849725f31a9a7a361ab9-Paper.pdf"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/3269206.3271800"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-72240-1_11"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/2766462.2767700"}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3640460","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3640460","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T20:58:18Z","timestamp":1750280298000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3640460"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,4,29]]},"references-count":60,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2024,9,30]]}},"alternative-id":["10.1145\/3640460"],"URL":"https:\/\/doi.org\/10.1145\/3640460","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"value":"1046-8188","type":"print"},{"value":"1558-2868","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,4,29]]},"assertion":[{"value":"2023-03-03","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-11-05","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-04-29","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}