{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,30]],"date-time":"2026-09-30T09:49:38Z","timestamp":1790761778091,"version":"4.1.0"},"reference-count":114,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2023,9,27]],"date-time":"2023-09-27T00:00:00Z","timestamp":1695772800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62272439"],"award-info":[{"award-number":["62272439"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Google Cloud"},{"name":"Google TPU Research Cloud"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2024,3,31]]},"abstract":"<jats:p>Pre-trained transformer models, such as BERT and T5, have shown to be highly effective at ad hoc passage and document ranking. Due to the inherent sequence length limits of these models, they need to process document passages one at a time rather than processing the entire document sequence at once. Although several approaches for aggregating passage-level signals into a document-level relevance score have been proposed, there has yet to be an extensive comparison of these techniques. In this work, we explore strategies for aggregating relevance signals from a document\u2019s passages into a final ranking score. We find that passage representation aggregation techniques can significantly improve over score aggregation techniques proposed in prior work, such as taking the maximum passage score. We call this new approach PARADE. In particular, PARADE can significantly improve results on collections with broad information needs where relevance signals can be spread throughout the document (such as TREC Robust04 and GOV2). Meanwhile, less complex aggregation techniques may work better on collections with an information need that can often be pinpointed to a single passage (such as TREC DL and TREC Genomics). We also conduct efficiency analyses and highlight several strategies for improving transformer-based aggregation.<\/jats:p>","DOI":"10.1145\/3600088","type":"journal-article","created":{"date-parts":[[2023,5,26]],"date-time":"2023-05-26T11:13:12Z","timestamp":1685099592000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":27,"title":["PARADE: Passage Representation Aggregation forDocument Reranking"],"prefix":"10.1145","volume":"42","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-2758-3296","authenticated-orcid":false,"given":"Canjia","family":"Li","sequence":"first","affiliation":[{"name":"University of Chinese Academy of Sciences, China and Max Planck Institute for Informatics"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5970-880X","authenticated-orcid":false,"given":"Andrew","family":"Yates","sequence":"additional","affiliation":[{"name":"University of Amsterdam, Netherlands and Max Planck Institute for Informatics"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8914-2659","authenticated-orcid":false,"given":"Sean","family":"MacAvaney","sequence":"additional","affiliation":[{"name":"University of Glasgow, United Kingdom"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2699-9209","authenticated-orcid":false,"given":"Ben","family":"He","sequence":"additional","affiliation":[{"name":"University of Chinese Academy of Sciences, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0615-2569","authenticated-orcid":false,"given":"Yingfei","family":"Sun","sequence":"additional","affiliation":[{"name":"University of Chinese Academy of Sciences, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2023,9,27]]},"reference":[{"key":"e_1_3_2_2_2","series-title":"Lecture Notes in Computer Science","doi-asserted-by":"crossref","first-page":"537","DOI":"10.1007\/978-3-319-89500-0","volume-title":"ECIR","author":"Ai Qingyao","year":"2018","unstructured":"Qingyao Ai, Brendan O\u2019Connor, and W. Bruce Croft. 2018. A neural passage model for ad-hoc document retrieval. In ECIR(Lecture Notes in Computer Science, Vol. 10772). Springer, 537\u2013543."},{"key":"e_1_3_2_3_2","first-page":"2654","volume-title":"NIPS","author":"Ba Jimmy","year":"2014","unstructured":"Jimmy Ba and Rich Caruana. 2014. Do deep nets really need to be deep? In NIPS. 2654\u20132662."},{"key":"e_1_3_2_4_2","article-title":"Layer normalization","volume":"1607","author":"Ba Lei Jimmy","year":"2016","unstructured":"Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. CoRR abs\/1607.06450 (2016).","journal-title":"CoRR"},{"key":"e_1_3_2_5_2","article-title":"Longformer: The long-document transformer","volume":"2004","author":"Beltagy Iz","year":"2020","unstructured":"Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. CoRR abs\/2004.05150 (2020).","journal-title":"CoRR"},{"key":"e_1_3_2_6_2","series-title":"Lecture Notes in Computer Science","doi-asserted-by":"crossref","first-page":"162","DOI":"10.1007\/978-3-540-78646-7_17","volume-title":"ECIR","author":"Bendersky Michael","year":"2008","unstructured":"Michael Bendersky and Oren Kurland. 2008. Utilizing passage-based language models for document retrieval. In ECIR(Lecture Notes in Computer Science, Vol. 4956). Springer, 162\u2013174."},{"issue":"8","key":"e_1_3_2_7_2","doi-asserted-by":"crossref","first-page":"1798","DOI":"10.1109\/TPAMI.2013.50","article-title":"Representation learning: A review and new perspectives","volume":"35","author":"Bengio Yoshua","year":"2013","unstructured":"Yoshua Bengio, Aaron C. Courville, and Pascal Vincent. 2013. Representation learning: A review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell. 35, 8 (2013), 1798\u20131828.","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"e_1_3_2_8_2","article-title":"Understanding performance of long-document ranking models through comprehensive evaluation and leaderboarding","volume":"2207","author":"Boytsov Leonid","year":"2022","unstructured":"Leonid Boytsov, Tianyi Lin, Fangwei Gao, Yutian Zhao, Jeffrey Huang, and Eric Nyberg. 2022. Understanding performance of long-document ranking models through comprehensive evaluation and leaderboarding. CoRR abs\/2207.01262 (2022).","journal-title":"CoRR"},{"key":"e_1_3_2_9_2","first-page":"302","volume-title":"SIGIR","author":"Callan James P.","year":"1994","unstructured":"James P. Callan. 1994. Passage-level evidence in document retrieval. In SIGIR. ACM\/Springer, 302\u2013310."},{"key":"e_1_3_2_10_2","article-title":"Enhanced news retrieval: Passages lead the way!","author":"Catena M.","year":"2019","unstructured":"M. Catena, O. Frieder, Cristina Ioana Muntean, F. Nardini, R. Perego, and N. Tonellotto. 2019. Enhanced news retrieval: Passages lead the way! In SIGIR.","journal-title":"SIGIR"},{"key":"e_1_3_2_11_2","article-title":"Simplified TinyBERT: Knowledge distillation for document retrieval","volume":"2009","author":"Chen Xuanang","year":"2020","unstructured":"Xuanang Chen, Ben He, Kai Hui, Le Sun, and Yingfei Sun. 2020. Simplified TinyBERT: Knowledge distillation for document retrieval. CoRR abs\/2009.07531 (2020).","journal-title":"CoRR"},{"key":"e_1_3_2_12_2","first-page":"1617","volume-title":"SIGIR","author":"Chen Xuanang","year":"2021","unstructured":"Xuanang Chen, Ben He, Kai Hui, Yiran Wang, Le Sun, and Yingfei Sun. 2021. Contextualized offline relevance weighting for efficient and effective neural retrieval. In SIGIR. ACM, 1617\u20131621."},{"key":"e_1_3_2_13_2","series-title":"NIST Special Publication","volume-title":"TREC","author":"Chen Xuanang","year":"2020","unstructured":"Xuanang Chen, Ben He, Le Sun, and Yingfei Sun. 2020. ICIP at TREC-2020 Deep Learning Track. In TREC(NIST Special Publication, Vol. 1266). National Institute of Standards and Technology (NIST)."},{"key":"e_1_3_2_14_2","volume-title":"TREC","author":"Chen Xuanang","year":"2019","unstructured":"Xuanang Chen, Canjia Li, Ben He, and Yingfei Sun. 2019. UCAS at TREC-2019 Deep Learning Track. In TREC."},{"key":"e_1_3_2_15_2","article-title":"Generating long sequences with sparse transformers","volume":"1904","author":"Child Rewon","year":"2019","unstructured":"Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating long sequences with sparse transformers. CoRR abs\/1904.10509 (2019).","journal-title":"CoRR"},{"key":"e_1_3_2_16_2","volume-title":"ICLR","author":"Clark Kevin","year":"2020","unstructured":"Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: Pre-training text encoders as discriminators rather than generators. In ICLR. OpenReview.net."},{"key":"e_1_3_2_17_2","volume-title":"TREC","author":"Craswell Nick","year":"2020","unstructured":"Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. 2020. Overview of the TREC 2020 Deep Learning Track. In TREC."},{"key":"e_1_3_2_18_2","volume-title":"Text REtrieval Conference (TREC)","author":"Craswell Nick","year":"2022","unstructured":"Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Jimmy Lin. 2022. Overview of the TREC 2021 Deep Learning Track. In Text REtrieval Conference (TREC). TREC. Retrieved from https:\/\/www.microsoft.com\/en-us\/research\/publication\/overview-of-the-trec-2021-deep-learning-track\/."},{"key":"e_1_3_2_19_2","volume-title":"TREC","author":"Craswell Nick","year":"2019","unstructured":"Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M. Voorhees. 2019. Overview of the TREC 2019 Deep Learning Track. In TREC."},{"key":"e_1_3_2_20_2","article-title":"Context-aware sentence\/passage term importance estimation for first stage retrieval","volume":"1910","author":"Dai Zhuyun","year":"2019","unstructured":"Zhuyun Dai and Jamie Callan. 2019. Context-aware sentence\/passage term importance estimation for first stage retrieval. CoRR abs\/1910.10687 (2019).","journal-title":"CoRR"},{"key":"e_1_3_2_21_2","first-page":"985","volume-title":"SIGIR","author":"Dai Zhuyun","year":"2019","unstructured":"Zhuyun Dai and Jamie Callan. 2019. Deeper text understanding for IR with contextual neural language modeling. In SIGIR. ACM, 985\u2013988."},{"key":"e_1_3_2_22_2","first-page":"126","volume-title":"WSDM","author":"Dai Zhuyun","year":"2018","unstructured":"Zhuyun Dai, Chenyan Xiong, Jamie Callan, and Zhiyuan Liu. 2018. Convolutional neural networks for soft-matching n-grams in ad-hoc search. In WSDM. ACM, 126\u2013134."},{"key":"e_1_3_2_23_2","volume-title":"NAACL-HLT","author":"Devlin Jacob","year":"2019","unstructured":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT."},{"key":"e_1_3_2_24_2","first-page":"375","volume-title":"SIGIR","author":"Fan Yixing","year":"2018","unstructured":"Yixing Fan, Jiafeng Guo, Yanyan Lan, Jun Xu, Chengxiang Zhai, and Xueqi Cheng. 2018. Modeling diverse relevance patterns in ad-hoc retrieval. In SIGIR. ACM, 375\u2013384."},{"issue":"2","key":"e_1_3_2_25_2","article-title":"Diagnostic evaluation of information retrieval models","volume":"29","author":"Fang Hui","year":"2011","unstructured":"Hui Fang, Tao Tao, and Chengxiang Zhai. 2011. Diagnostic evaluation of information retrieval models. ACM Trans. Inf. Syst. 29, 2 (2011).","journal-title":"ACM Trans. Inf. Syst."},{"key":"e_1_3_2_26_2","article-title":"SPLADE v2: Sparse lexical and expansion model for information retrieval","author":"Formal Thibault","year":"2021","unstructured":"Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and St\u00e9phane Clinchant. 2021. SPLADE v2: Sparse lexical and expansion model for information retrieval. arXiv preprint arXiv:2109.10086 (2021).","journal-title":"arXiv preprint arXiv:2109.10086"},{"key":"e_1_3_2_27_2","volume-title":"ICTIR","author":"Gao Luyu","year":"2020","unstructured":"Luyu Gao, Zhuyun Dai, and Jamie Callan. 2020. Understanding BERT rankers under distillation. In ICTIR."},{"key":"e_1_3_2_28_2","article-title":"EARL: Speedup transformer-based rankers with pre-computed representation","volume":"2004","author":"Gao Luyu","year":"2020","unstructured":"Luyu Gao, Zhuyun Dai, and James P. Callan. 2020. EARL: Speedup transformer-based rankers with pre-computed representation. ArXiv abs\/2004.13313 (2020).","journal-title":"ArXiv"},{"key":"e_1_3_2_29_2","first-page":"55","volume-title":"CIKM","author":"Guo Jiafeng","year":"2016","unstructured":"Jiafeng Guo, Yixing Fan, Qingyao Ai, and W. Bruce Croft. 2016. A deep relevance matching model for ad-hoc retrieval. In CIKM. ACM, 55\u201364."},{"key":"e_1_3_2_30_2","volume-title":"TREC","author":"Hersh William","year":"2007","unstructured":"William Hersh, Aaron Cohen, Lynn Ruslen, and Phoebe Roberts. 2007. TREC 2007 Genomics Track overview. In TREC."},{"key":"e_1_3_2_31_2","volume-title":"TREC","author":"Hersh William","year":"2006","unstructured":"William Hersh, Aaron M. Cohen, Phoebe Roberts, and Hari Krishna Rekapalli. 2006. TREC 2006 Genomics Track overview. In TREC."},{"key":"e_1_3_2_32_2","article-title":"Distilling the knowledge in a neural network","volume":"1503","author":"Hinton Geoffrey E.","year":"2015","unstructured":"Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. CoRR abs\/1503.02531 (2015).","journal-title":"CoRR"},{"key":"e_1_3_2_33_2","article-title":"Improving efficient neural ranking models with cross-architecture knowledge distillation","volume":"2010","author":"Hofst\u00e4tter Sebastian","year":"2020","unstructured":"Sebastian Hofst\u00e4tter, Sophia Althammer, Michael Schr\u00f6der, Mete Sertkan, and Allan Hanbury. 2020. Improving efficient neural ranking models with cross-architecture knowledge distillation. CoRR abs\/2010.02666 (2020).","journal-title":"CoRR"},{"key":"e_1_3_2_34_2","series-title":"Lecture Notes in Computer Science","doi-asserted-by":"crossref","first-page":"238","DOI":"10.1007\/978-3-030-72113-8_16","volume-title":"ECIR","author":"Hofst\u00e4tter Sebastian","year":"2021","unstructured":"Sebastian Hofst\u00e4tter, Aldo Lipani, Sophia Althammer, Markus Zlabinger, and Allan Hanbury. 2021. Mitigating the position bias of transformer models in passage re-ranking. In ECIR(Lecture Notes in Computer Science, Vol. 12656). Springer, 238\u2013253."},{"key":"e_1_3_2_35_2","first-page":"2021","volume-title":"SIGIR","author":"Hofst\u00e4tter Sebastian","year":"2020","unstructured":"Sebastian Hofst\u00e4tter, Hamed Zamani, Bhaskar Mitra, Nick Craswell, and Allan Hanbury. 2020. Local self-attention over long text for efficient document retrieval. In SIGIR. ACM, 2021\u20132024."},{"key":"e_1_3_2_36_2","volume-title":"TREC","author":"Hofst\u00e4tter Sebastian","year":"2019","unstructured":"Sebastian Hofst\u00e4tter, Markus Zlabinger, and Allan Hanbury. 2019. TU Wien @ TREC deep learning\u201919\u2014Simple contextualization for re-ranking. In TREC."},{"key":"e_1_3_2_37_2","article-title":"Interpretable & time-budget-constrained contextualization for re-ranking","volume":"2002","author":"Hofst\u00e4tter Sebastian","year":"2020","unstructured":"Sebastian Hofst\u00e4tter, Markus Zlabinger, and Allan Hanbury. 2020. Interpretable & time-budget-constrained contextualization for re-ranking. CoRR abs\/2002.01854 (2020).","journal-title":"CoRR"},{"key":"e_1_3_2_38_2","first-page":"3031","volume-title":"CIKM","author":"Hofst\u00e4tter Sebastian","year":"2020","unstructured":"Sebastian Hofst\u00e4tter, Markus Zlabinger, Mete Sertkan, Michael Schr\u00f6der, and Allan Hanbury. 2020. Fine-grained relevance annotations for multi-task document ranking and question answering. In CIKM. ACM, 3031\u20133038."},{"key":"e_1_3_2_39_2","first-page":"2042","volume-title":"NIPS","author":"Hu Baotian","year":"2014","unstructured":"Baotian Hu, Zhengdong Lu, Hang Li, and Qingcai Chen. 2014. Convolutional neural network architectures for matching natural language sentences. In NIPS. 2042\u20132050."},{"key":"e_1_3_2_40_2","first-page":"2333","volume-title":"CIKM","author":"Huang Po-Sen","year":"2013","unstructured":"Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry P. Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In CIKM. ACM, 2333\u20132338."},{"key":"e_1_3_2_41_2","first-page":"1049","volume-title":"EMNLP","author":"Hui Kai","year":"2017","unstructured":"Kai Hui, Andrew Yates, Klaus Berberich, and Gerard de Melo. 2017. PACRR: A position-aware neural IR model for relevance matching. In EMNLP. Association for Computational Linguistics, 1049\u20131058."},{"key":"e_1_3_2_42_2","first-page":"279","volume-title":"WSDM","author":"Hui Kai","year":"2018","unstructured":"Kai Hui, Andrew Yates, Klaus Berberich, and Gerard de Melo. 2018. Co-PACRR: A context-aware neural IR model for ad-hoc retrieval. In WSDM. ACM, 279\u2013287."},{"key":"e_1_3_2_43_2","volume-title":"ICLR","author":"Humeau Samuel","year":"2020","unstructured":"Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2020. Poly-encoders: Architectures and pre-training strategies for fast and accurate multi-sentence scoring. In ICLR. OpenReview.net."},{"key":"e_1_3_2_44_2","first-page":"4594","volume-title":"EMNLP (Findings)","author":"Jiang Jyun-Yu","year":"2020","unstructured":"Jyun-Yu Jiang, Chenyan Xiong, Chia-Jung Lee, and Wei Wang. 2020. Long document ranking with query-directed sparse transformer. In EMNLP (Findings). Association for Computational Linguistics, 4594\u20134605."},{"key":"e_1_3_2_45_2","article-title":"TinyBERT: Distilling BERT for natural language understanding","volume":"1909","author":"Jiao Xiaoqi","year":"2019","unstructured":"Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2019. TinyBERT: Distilling BERT for natural language understanding. CoRR abs\/1909.10351 (2019).","journal-title":"CoRR"},{"key":"e_1_3_2_46_2","first-page":"963","volume-title":"SIGIR","author":"Keikha Mostafa","year":"2014","unstructured":"Mostafa Keikha, Jae Hyun Park, and W. Bruce Croft. 2014. Evaluating answer passages using summarization measures. In SIGIR. 963\u2013966."},{"key":"e_1_3_2_47_2","doi-asserted-by":"crossref","first-page":"81","DOI":"10.1145\/2682862.2682877","volume-title":"ADCS","author":"Keikha Mostafa","year":"2014","unstructured":"Mostafa Keikha, Jae Hyun Park, W. Bruce Croft, and Mark Sanderson. 2014. Retrieving passages and finding answers. In ADCS. 81\u201384."},{"key":"e_1_3_2_48_2","volume-title":"SIGIR","author":"Khattab Omar","year":"2020","unstructured":"Omar Khattab and Matei Zaharia. 2020. ColBERT: Efficient and effective passage search via contextualized late interaction over BERT. In SIGIR."},{"key":"e_1_3_2_49_2","first-page":"3147","volume-title":"CIKM","author":"Kim Youngwoo","year":"2021","unstructured":"Youngwoo Kim, Razieh Rahimi, Hamed R. Bonab, and James Allan. 2021. Query-driven segment selection for ranking long documents. In CIKM. ACM, 3147\u20133151."},{"key":"e_1_3_2_50_2","series-title":"NIST Special Publication","volume-title":"TREC","author":"Knafou Julien","year":"2020","unstructured":"Julien Knafou, Matthew Jeffryes, Sohrab Ferdowsi, and Patrick Ruch. 2020. SIB text mining at TREC 2020 Deep Learning Track. In TREC(NIST Special Publication, Vol. 1266). National Institute of Standards and Technology (NIST)."},{"key":"e_1_3_2_51_2","doi-asserted-by":"crossref","first-page":"120","DOI":"10.1145\/383952.383972","volume-title":"SIGIR","author":"Lavrenko Victor","year":"2001","unstructured":"Victor Lavrenko and W. Bruce Croft. 2001. Relevance-based language models. In SIGIR. ACM, 120\u2013127."},{"issue":"4","key":"e_1_3_2_52_2","doi-asserted-by":"crossref","first-page":"1234","DOI":"10.1093\/bioinformatics\/btz682","article-title":"BioBERT: A pre-trained biomedical language representation model for biomedical text mining","volume":"36","author":"Lee Jinhyuk","year":"2020","unstructured":"Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. BioBERT: A pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 36, 4 (2020), 1234\u20131240.","journal-title":"Bioinformatics"},{"key":"e_1_3_2_53_2","first-page":"4482","volume-title":"EMNLP","author":"Li Canjia","year":"2018","unstructured":"Canjia Li, Yingfei Sun, Ben He, Le Wang, Kai Hui, Andrew Yates, Le Sun, and Jungang Xu. 2018. NPRF: A neural pseudo relevance feedback framework for ad-hoc information retrieval. In EMNLP. Association for Computational Linguistics, 4482\u20134491."},{"key":"e_1_3_2_54_2","unstructured":"Canjia Li and Andrew Yates. 2020. MPII at the TREC 2020 Deep Learning Track. In TREC ."},{"key":"e_1_3_2_55_2","volume-title":"NTCIR","author":"Li Canjia","year":"2020","unstructured":"Canjia Li and Andrew Yates. 2020. MPII at the NTCIR-15 WWW-3 task. In NTCIR."},{"key":"e_1_3_2_56_2","first-page":"2207","volume-title":"SIGIR","author":"Li Minghan","year":"2021","unstructured":"Minghan Li and \u00c9ric Gaussier. 2021. KeyBLD: Selecting key blocks with local pre-ranking for long document information retrieval. In SIGIR. ACM, 2207\u20132211."},{"key":"e_1_3_2_57_2","article-title":"A few brief notes on DeepImpact, COIL, and a conceptual framework for information retrieval techniques","author":"Lin Jimmy","year":"2021","unstructured":"Jimmy Lin and Xueguang Ma. 2021. A few brief notes on DeepImpact, COIL, and a conceptual framework for information retrieval techniques. arXiv preprint arXiv:2106.14807 (2021).","journal-title":"arXiv preprint arXiv:2106.14807"},{"key":"e_1_3_2_58_2","article-title":"Pretrained transformers for text ranking: BERT and beyond","author":"Lin Jimmy","year":"2020","unstructured":"Jimmy Lin, Rodrigo Nogueira, and Andrew Yates. 2020. Pretrained transformers for text ranking: BERT and beyond. arXiv preprint arXiv:2010.06467 (2020).","journal-title":"arXiv preprint arXiv:2010.06467"},{"key":"e_1_3_2_59_2","article-title":"Is searching full text more effective than searching abstracts?","volume":"10","author":"Lin Jimmy J.","year":"2009","unstructured":"Jimmy J. Lin. 2009. Is searching full text more effective than searching abstracts? BMC Bioinform. 10 (2009).","journal-title":"BMC Bioinform."},{"key":"e_1_3_2_60_2","first-page":"375","volume-title":"CIKM","author":"Liu Xiaoyong","year":"2002","unstructured":"Xiaoyong Liu and W. Bruce Croft. 2002. Passage retrieval based on language models. In CIKM. ACM, 375\u2013382."},{"key":"e_1_3_2_61_2","first-page":"5070","volume-title":"ACL","author":"Liu Yang","year":"2019","unstructured":"Yang Liu and Mirella Lapata. 2019. Hierarchical transformers for multi-document summarization. In ACL. 5070\u20135081."},{"key":"e_1_3_2_62_2","volume-title":"Representation Learning for Natural Language Processing","author":"Liu Zhiyuan","year":"2020","unstructured":"Zhiyuan Liu, Yankai Lin, and Maosong Sun. 2020. Representation Learning for Natural Language Processing. Springer."},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00369"},{"key":"e_1_3_2_64_2","first-page":"283","volume-title":"WSDM","author":"Ma Xinyu","year":"2021","unstructured":"Xinyu Ma, Jiafeng Guo, Ruqing Zhang, Yixing Fan, Xiang Ji, and Xueqi Cheng. 2021. PROP: Pre-training with representative words prediction for ad-hoc retrieval. In WSDM. ACM, 283\u2013291."},{"key":"e_1_3_2_65_2","first-page":"1318","volume-title":"SIGIR","author":"Ma Xinyu","year":"2021","unstructured":"Xinyu Ma, Jiafeng Guo, Ruqing Zhang, Yixing Fan, Yingyan Li, and Xueqi Cheng. 2021. B-PROP: Bootstrapped pre-training with representative words prediction for ad-hoc retrieval. In SIGIR. ACM, 1318\u20131327."},{"key":"e_1_3_2_66_2","volume-title":"SIGIR","author":"MacAvaney Sean","year":"2020","unstructured":"Sean MacAvaney, Franco Maria Nardini, Raffaele Perego, Nicola Tonellotto, Nazli Goharian, and Ophir Frieder. 2020. Efficient document re-ranking for transformers by precomputing term representations. In SIGIR."},{"key":"e_1_3_2_67_2","volume-title":"SIGIR","author":"MacAvaney Sean","year":"2020","unstructured":"Sean MacAvaney, Franco Maria Nardini, Raffaele Perego, Nicola Tonellotto, Nazli Goharian, and Ophir Frieder. 2020. Expansion via prediction of importance with contextualization. In SIGIR."},{"key":"e_1_3_2_68_2","first-page":"1101","volume-title":"SIGIR","author":"MacAvaney Sean","year":"2019","unstructured":"Sean MacAvaney, Andrew Yates, Arman Cohan, and Nazli Goharian. 2019. CEDR: Contextualized embeddings for document ranking. In SIGIR. ACM, 1101\u20131104."},{"key":"e_1_3_2_69_2","first-page":"ACM, 1723\u20131727","volume-title":"SIGIR","author":"Mallia Antonio","year":"2021","unstructured":"Antonio Mallia, Omar Khattab, Torsten Suel, and Nicola Tonellotto. 2021. Learning passage impacts for inverted indexes. In SIGIR. ACM, 1723\u20131727."},{"key":"e_1_3_2_70_2","doi-asserted-by":"crossref","DOI":"10.1017\/CBO9780511809071","volume-title":"Introduction to Information Retrieval","author":"Manning Christopher D.","year":"2008","unstructured":"Christopher D. Manning, Prabhakar Raghavan, and Hinrich Sch\u00fctze. 2008. Introduction to Information Retrieval. Cambridge University Press."},{"key":"e_1_3_2_71_2","series-title":"NIST Special Publication","volume-title":"TREC","author":"Mitra Bhaskar","year":"2019","unstructured":"Bhaskar Mitra and Nick Craswell. 2019. Duet at TREC 2019 Deep Learning Track. In TREC(NIST Special Publication, Vol. 1250). National Institute of Standards and Technology (NIST)."},{"key":"e_1_3_2_72_2","first-page":"1291","volume-title":"WWW","author":"Mitra Bhaskar","year":"2017","unstructured":"Bhaskar Mitra, Fernando Diaz, and Nick Craswell. 2017. Learning to match using local and distributed representations of text for web search. In WWW. ACM, 1291\u20131299."},{"key":"e_1_3_2_73_2","article-title":"Conformer-kernel with query term independence for document retrieval","volume":"2007","author":"Mitra Bhaskar","year":"2020","unstructured":"Bhaskar Mitra, Sebastian Hofst\u00e4tter, Hamed Zamani, and Nick Craswell. 2020. Conformer-kernel with query term independence for document retrieval. CoRR abs\/2007.10434 (2020).","journal-title":"CoRR"},{"key":"e_1_3_2_74_2","doi-asserted-by":"crossref","first-page":"101","DOI":"10.1007\/978-3-031-28241-6_7","volume-title":"Advances in Information Retrieval: 45th European Conference on Information Retrieval, ECIR 2023, Dublin, Ireland, April 2\u20136, 2023, Proceedings, Part III","author":"Nguyen Thong","year":"2023","unstructured":"Thong Nguyen, Sean MacAvaney, and Andrew Yates. 2023. A unified framework for learned sparse retrieval. In Advances in Information Retrieval: 45th European Conference on Information Retrieval, ECIR 2023, Dublin, Ireland, April 2\u20136, 2023, Proceedings, Part III. Springer, 101\u2013116."},{"key":"e_1_3_2_75_2","unstructured":"CEUR Workshop Proceedings CoCo@NIPS 1773 Tri Nguyen Mir Rosenberg Xia Song Jianfeng Gao Saurabh Tiwary Rangan Majumder Li Deng MS MARCO: A human generated MAchine Reading COmprehension dataset 2016"},{"key":"e_1_3_2_76_2","article-title":"Passage re-ranking with BERT","volume":"1901","author":"Nogueira Rodrigo","year":"2019","unstructured":"Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage re-ranking with BERT. CoRR abs\/1901.04085 (2019).","journal-title":"CoRR"},{"key":"e_1_3_2_77_2","volume-title":"Findings of EMNLP","author":"Nogueira Rodrigo","year":"2020","unstructured":"Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020. Document ranking with a pretrained sequence-to-sequence model. In Findings of EMNLP."},{"key":"e_1_3_2_78_2","article-title":"Document expansion by query prediction","volume":"1904","author":"Nogueira Rodrigo","year":"2019","unstructured":"Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. 2019. Document expansion by query prediction. CoRR abs\/1904.08375 (2019).","journal-title":"CoRR"},{"key":"e_1_3_2_79_2","first-page":"2793","volume-title":"AAAI","author":"Pang Liang","year":"2016","unstructured":"Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu, Shengxian Wan, and Xueqi Cheng. 2016. Text matching as image recognition. In AAAI. AAAI Press, 2793\u20132799."},{"key":"e_1_3_2_80_2","series-title":"NIST Special Publication","volume-title":"TREC","author":"Pradeep Ronak","year":"2020","unstructured":"Ronak Pradeep, Xueguang Ma, Xinyu Zhang, Hang Cui, Ruizhou Xu, Rodrigo Frassetto Nogueira, and Jimmy Lin. 2020. H2oloo at TREC 2020: When all you got is a hammer... deep learning, health misinformation, and precision medicine. In TREC(NIST Special Publication, Vol. 1266). National Institute of Standards and Technology (NIST)."},{"key":"e_1_3_2_81_2","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","volume":"1910","author":"Raffel Colin","year":"2019","unstructured":"Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR abs\/1910.10683 (2019).","journal-title":"CoRR"},{"key":"e_1_3_2_82_2","first-page":"247","article-title":"The Technion at the WWW-3 task: Cluster-based document retrieval","author":"Raiber Fiana","year":"2020","unstructured":"Fiana Raiber and Oren Kurland. 2020. The Technion at the WWW-3 task: Cluster-based document retrieval. In NTCIR. 247\u2013248.","journal-title":"NTCIR"},{"key":"e_1_3_2_83_2","first-page":"232","volume-title":"SIGIR","author":"Robertson Stephen E.","year":"1994","unstructured":"Stephen E. Robertson and Steve Walker. 1994. Some simple effective approximations to the 2-Poisson model for probabilistic weighted retrieval. In SIGIR. ACM\/Springer, 232\u2013241."},{"key":"e_1_3_2_84_2","volume-title":"TREC","author":"Robertson Stephen E.","year":"1995","unstructured":"Stephen E. Robertson, Steve Walker, Micheline Hancock-Beaulieu, Mike Gatford, and A. Payne. 1995. Okapi at TREC-4. In TREC."},{"key":"e_1_3_2_85_2","article-title":"Relevance feedback in information retrieval","author":"Rocchio J. J.","year":"1971","unstructured":"J. J. Rocchio. 1971. Relevance feedback in information retrieval. In The Smart Retrieval System - Experiments in Automatic Document Processing, G. Salton (Ed.). Englewood Cliffs, NJ: Prentice-Hall, 313\u2013323.","journal-title":"The Smart Retrieval System - Experiments in Automatic Document Processing"},{"key":"e_1_3_2_86_2","series-title":"Lecture Notes in Computer Science","doi-asserted-by":"crossref","first-page":"319","DOI":"10.1007\/978-3-319-28940-3_25","volume-title":"AIRS","author":"Sakai Tetsuya","year":"2015","unstructured":"Tetsuya Sakai, Lifeng Shang, Zhengdong Lu, and Hang Li. 2015. Topic set size design with the evaluation measures for short text conversation. In AIRS(Lecture Notes in Computer Science, Vol. 9460). Springer, 319\u2013331."},{"key":"e_1_3_2_87_2","volume-title":"NTCIR","author":"Sakai Tetsuya","year":"2020","unstructured":"Tetsuya Sakai, Sijie Tao, Zhaohao Zeng, Yukun Zheng, Jiaxin Mao, Zhumin Chu, Yiqun Liu, Maria Maistro, Zhicheng Dou, Nicola Ferro, and Ian Soboroff. 2020. Overview of the NTCIR-15 We want web with CENTRE (WWW-3) task. In NTCIR."},{"key":"e_1_3_2_88_2","series-title":"Lecture Notes in Computer Science","doi-asserted-by":"crossref","first-page":"92","DOI":"10.1007\/978-3-642-15825-4_10","volume-title":"ICANN","author":"Scherer Dominik","year":"2010","unstructured":"Dominik Scherer, Andreas C. M\u00fcller, and Sven Behnke. 2010. Evaluation of pooling operations in convolutional architectures for object recognition. In ICANN(Lecture Notes in Computer Science, Vol. 6354). Springer, 92\u2013101."},{"issue":"2","key":"e_1_3_2_89_2","doi-asserted-by":"crossref","first-page":"159","DOI":"10.1007\/s10791-020-09369-x","article-title":"A passage-based approach to learning to rank documents","volume":"23","author":"Sheetrit Eilon","year":"2020","unstructured":"Eilon Sheetrit, Anna Shtok, and Oren Kurland. 2020. A passage-based approach to learning to rank documents. Inf. Retr. J. 23, 2 (2020), 159\u2013186.","journal-title":"Inf. Retr. J."},{"key":"e_1_3_2_90_2","first-page":"235","article-title":"KASYS at the NTCIR-15 WWW-3 task","author":"Shinden Kohei","year":"2020","unstructured":"Kohei Shinden, Atsuki Maruta, and Makoto P. Kato. 2020. KASYS at the NTCIR-15 WWW-3 task. In NTCIR. 235\u2013238.","journal-title":"NTCIR"},{"key":"e_1_3_2_91_2","volume-title":"EMNLP","author":"Sun Siqi","year":"2019","unstructured":"Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019. Patient knowledge distillation for BERT model compression. In EMNLP."},{"key":"e_1_3_2_92_2","article-title":"Distilling knowledge for fast retrieval-based chat-bots","volume":"2004","author":"Tahami Amir Vakili","year":"2020","unstructured":"Amir Vakili Tahami, Kamyar Ghajar, and Azadeh Shakery. 2020. Distilling knowledge for fast retrieval-based chat-bots. CoRR abs\/2004.11045 (2020).","journal-title":"CoRR"},{"key":"e_1_3_2_93_2","article-title":"Distilling task-specific knowledge from BERT into simple neural networks","volume":"1903","author":"Tang Raphael","year":"2019","unstructured":"Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. 2019. Distilling task-specific knowledge from BERT into simple neural networks. CoRR abs\/1903.12136 (2019).","journal-title":"CoRR"},{"key":"e_1_3_2_94_2","article-title":"Well-read students learn better: The impact of student initialization on knowledge distillation","volume":"1908","author":"Turc Iulia","year":"2019","unstructured":"Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Well-read students learn better: The impact of student initialization on knowledge distillation. CoRR abs\/1908.08962 (2019).","journal-title":"CoRR"},{"key":"e_1_3_2_95_2","first-page":"5998","volume-title":"NIPS","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS. 5998\u20136008."},{"key":"e_1_3_2_96_2","volume-title":"ICLR","author":"Wang Wei","year":"2020","unstructured":"Wei Wang, Bin Bi, Ming Yan, Chen Wu, Jiangnan Xia, Zuyi Bao, Liwei Peng, and Luo Si. 2020. StructBERT: Incorporating language structures into pre-training for deep language understanding. In ICLR. OpenReview.net."},{"key":"e_1_3_2_97_2","first-page":"848","volume-title":"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)","author":"Wu Chuhan","year":"2021","unstructured":"Chuhan Wu, Fangzhao Wu, Tao Qi, and Yongfeng Huang. 2021. Hi-transformer: Hierarchical interactive transformer for efficient and effective long document modeling. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers). Association for Computational Linguistics, 848\u2013853. DOI:10.18653\/v1\/2021.acl-short.107"},{"key":"e_1_3_2_98_2","first-page":"2421","volume-title":"WWW","author":"Wu Zhijing","year":"2020","unstructured":"Zhijing Wu, Jiaxin Mao, Yiqun Liu, Jingtao Zhan, Yukun Zheng, Min Zhang, and Shaoping Ma. 2020. Leveraging passage-level cumulative gain for document ranking. In WWW. ACM\/IW3C2, 2421\u20132431."},{"key":"e_1_3_2_99_2","first-page":"605","volume-title":"SIGIR","author":"Wu Zhijing","year":"2019","unstructured":"Zhijing Wu, Jiaxin Mao, Yiqun Liu, Min Zhang, and Shaoping Ma. 2019. Investigating passage-level relevance and its role in document-level relevance judgment. In SIGIR. ACM, 605\u2013614."},{"key":"e_1_3_2_100_2","first-page":"55","volume-title":"SIGIR","author":"Xiong Chenyan","year":"2017","unstructured":"Chenyan Xiong, Zhuyun Dai, Jamie Callan, Zhiyuan Liu, and Russell Power. 2017. End-to-end neural ad-hoc ranking with kernel pooling. In SIGIR. ACM, 55\u201364."},{"key":"e_1_3_2_101_2","volume-title":"TREC","author":"Yan Ming","year":"2019","unstructured":"Ming Yan, Chenliang Li, Chen Wu, Bin Bi, Wei Wang, Jiangnan Xia, and Luo Si. 2019. IDST at TREC 2019 Deep Learning Track: Deep cascade ranking with generation-based document expansion and pre-trained language modeling. In TREC."},{"key":"e_1_3_2_102_2","first-page":"1725","volume-title":"CIKM","author":"Yang Liu","year":"2020","unstructured":"Liu Yang, Mingyang Zhang, Cheng Li, Michael Bendersky, and Marc Najork. 2020. Beyond 512 tokens: Siamese multi-depth transformer-based hierarchical encoder for long-form document matching. In CIKM. ACM, 1725\u20131734."},{"issue":"4","key":"e_1_3_2_103_2","first-page":"16:1\u201316:20","article-title":"Anserini: Reproducible ranking baselines using Lucene","volume":"10","author":"Yang Peilin","year":"2018","unstructured":"Peilin Yang, Hui Fang, and Jimmy Lin. 2018. Anserini: Reproducible ranking baselines using Lucene. J. Data Inf. Qual. 10, 4 (2018), 16:1\u201316:20.","journal-title":"J. Data Inf. Qual."},{"key":"e_1_3_2_104_2","first-page":"1129","volume-title":"SIGIR","author":"Yang Wei","year":"2019","unstructured":"Wei Yang, Kuang Lu, Peilin Yang, and Jimmy Lin. 2019. Critically examining the \u201cneural hype\u201d: Weak baselines and the additivity of effectiveness gains from neural ranking models. In SIGIR. ACM, 1129\u20131132."},{"key":"e_1_3_2_105_2","first-page":"1480","volume-title":"NAACL-HLT","author":"Yang Zichao","year":"2016","unstructured":"Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In NAACL-HLT. 1480\u20131489."},{"key":"e_1_3_2_106_2","first-page":"3181","volume-title":"CIKM","author":"Yates Andrew","year":"2020","unstructured":"Andrew Yates, Kevin Martin Jose, Xinyu Zhang, and Jimmy Lin. 2020. Flexible IR pipelines with Capreolus. In CIKM. 3181\u20133188."},{"key":"e_1_3_2_107_2","volume-title":"EMNLP","author":"Yilmaz Zeynep Akkalyoncu","year":"2019","unstructured":"Zeynep Akkalyoncu Yilmaz, Shengjin Wang, Wei Yang, Haotian Zhang, and Jimmy Lin. 2019. Applying BERT to document retrieval with Birch. In EMNLP."},{"key":"e_1_3_2_108_2","volume-title":"NeurIPS","author":"Zaheer Manzil","year":"2020","unstructured":"Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Onta\u00f1\u00f3n, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020. BigBird: Transformers for longer sequences. In NeurIPS."},{"key":"e_1_3_2_109_2","first-page":"700","volume-title":"WSDM","author":"Zamani Hamed","year":"2018","unstructured":"Hamed Zamani, Bhaskar Mitra, Xia Song, Nick Craswell, and Saurabh Tiwary. 2018. Neural ranking models with multiple document fields. In WSDM. ACM, 700\u2013708."},{"key":"e_1_3_2_110_2","first-page":"5059","volume-title":"ACL","author":"Zhang Xingxing","year":"2019","unstructured":"Xingxing Zhang, Furu Wei, and Ming Zhou. 2019. HIBERT: Document level pre-training of hierarchical bidirectional transformers for document summarization. In ACL. 5059\u20135069."},{"key":"e_1_3_2_111_2","doi-asserted-by":"crossref","first-page":"150","DOI":"10.1007\/978-3-030-72240-1_11","volume-title":"Advances in Information Retrieval: 43rd European Conference on IR Research, ECIR 2021, Virtual Event, March 28 \u2013 April 1, 2021, Proceedings, Part II","author":"Zhang Xinyu","year":"2021","unstructured":"Xinyu Zhang, Andrew Yates, and Jimmy Lin. 2021. Comparing score aggregation approaches for document retrieval with pretrained transformers. In Advances in Information Retrieval: 43rd European Conference on IR Research, ECIR 2021, Virtual Event, March 28 \u2013 April 1, 2021, Proceedings, Part II. Springer-Verlag, Berlin, 150\u2013163. DOI:10.1007\/978-3-030-72240-1_11"},{"key":"e_1_3_2_112_2","volume-title":"ICLR","author":"Zhao Chen","year":"2020","unstructured":"Chen Zhao, Chenyan Xiong, Corby Rosset, Xia Song, Paul N. Bennett, and Saurabh Tiwary. 2020. Transformer-XH: Multi-evidence reasoning with eXtra Hop attention. In ICLR. OpenReview.net."},{"key":"e_1_3_2_113_2","article-title":"Sparta: Efficient open-domain question answering via sparse transformer matching retrieval","author":"Zhao Tiancheng","year":"2020","unstructured":"Tiancheng Zhao, Xiaopeng Lu, and Kyusong Lee. 2020. Sparta: Efficient open-domain question answering via sparse transformer matching retrieval. arXiv preprint arXiv:2009.13013 (2020).","journal-title":"arXiv preprint arXiv:2009.13013"},{"key":"e_1_3_2_114_2","first-page":"892","volume-title":"ACL","author":"Zhou Jie","year":"2019","unstructured":"Jie Zhou, Xu Han, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. 2019. GEAR: Graph-based evidence aggregating and reasoning for fact verification. In ACL. Association for Computational Linguistics, 892\u2013901."},{"key":"e_1_3_2_115_2","first-page":"1483","volume-title":"SIGIR","author":"Zhuang Shengyao","year":"2021","unstructured":"Shengyao Zhuang and Guido Zuccon. 2021. TILDE: Term Independent Likelihood moDEl for passage re-ranking. In SIGIR. 1483\u20131492."}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3600088","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3600088","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T16:36:50Z","timestamp":1750178210000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3600088"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,27]]},"references-count":114,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2024,3,31]]}},"alternative-id":["10.1145\/3600088"],"URL":"https:\/\/doi.org\/10.1145\/3600088","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"value":"1046-8188","type":"print"},{"value":"1558-2868","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,9,27]]},"assertion":[{"value":"2021-12-15","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-05-10","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-09-27","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}