{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,25]],"date-time":"2026-06-25T15:54:24Z","timestamp":1782402864927,"version":"3.54.5"},"reference-count":104,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2022,12,21]],"date-time":"2022-12-21T00:00:00Z","timestamp":1671580800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Natural Science Foundation of China","doi-asserted-by":"crossref","award":["62006218, 61902381, and 61872338"],"award-info":[{"award-number":["62006218, 61902381, and 61872338"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100004739","name":"Youth Innovation Promotion Association CAS","doi-asserted-by":"crossref","award":["20144310, and 2021100"],"award-info":[{"award-number":["20144310, and 2021100"]}],"id":[{"id":"10.13039\/501100004739","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Lenovo-CAS Joint Lab Youth Scientist Project"},{"name":"Foundation and Frontier Research Key Program of Chongqing Science and Technology Commission","award":["cstc2017jcyjBX0059"],"award-info":[{"award-number":["cstc2017jcyjBX0059"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2023,4,30]]},"abstract":"<jats:p>\n            Recently, we have witnessed the bloom of neural ranking models in the information retrieval (IR) field. So far, much effort has been devoted to developing effective neural ranking models that can generalize well on new data. There has been less attention paid to the robustness perspective. Unlike the effectiveness, which is about the average performance of a system under normal purpose, robustness cares more about the system performance in the worst case or under malicious operations instead. When a new technique enters into the real-world application, it is critical to know not only how it works in average, but also how would it behave in abnormal situations. So, we raise the question in this work: Are neural ranking models robust? To answer this question, first, we need to clarify what we refer to when we talk about the robustness of ranking models in IR. We show that robustness is actually a multi-dimensional concept and there are three ways to define it in IR: (1) the\n            <jats:italic>performance variance<\/jats:italic>\n            under the independent and identically distributed (I.I.D.) setting; (2) the\n            <jats:italic>out-of-distribution (OOD) generalizability<\/jats:italic>\n            ; and (3) the\n            <jats:italic>defensive ability<\/jats:italic>\n            against adversarial operations. The latter two definitions can be further specified into two different perspectives, respectively, leading to five robustness tasks in total. Based on this taxonomy, we build corresponding benchmark datasets, design empirical experiments, and systematically analyze the robustness of several representative neural ranking models against traditional probabilistic ranking models and learning-to-rank (LTR) models. The empirical results show that there is no simple answer to our question. While neural ranking models are less robust against other IR models in most cases, some of them can still win two out of five tasks. This is the first comprehensive study on the robustness of neural ranking models. We believe the way we study the robustness as well as our findings would be beneficial to the IR community. We will also release all the data and codes to facilitate the future research in this direction.\n          <\/jats:p>","DOI":"10.1145\/3534928","type":"journal-article","created":{"date-parts":[[2022,6,21]],"date-time":"2022-06-21T09:19:48Z","timestamp":1655803188000},"page":"1-36","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":8,"title":["Are Neural Ranking Models Robust?"],"prefix":"10.1145","volume":"41","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2492-0640","authenticated-orcid":false,"given":"Chen","family":"Wu","sequence":"first","affiliation":[{"name":"CAS Key Lab of Network Data Science and Technology, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4294-2541","authenticated-orcid":false,"given":"Ruqing","family":"Zhang","sequence":"additional","affiliation":[{"name":"CAS Key Lab of Network Data Science and Technology, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9509-8674","authenticated-orcid":false,"given":"Jiafeng","family":"Guo","sequence":"additional","affiliation":[{"name":"CAS Key Lab of Network Data Science and Technology, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4317-2702","authenticated-orcid":false,"given":"Yixing","family":"Fan","sequence":"additional","affiliation":[{"name":"CAS Key Lab of Network Data Science and Technology, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5201-8195","authenticated-orcid":false,"given":"Xueqi","family":"Cheng","sequence":"additional","affiliation":[{"name":"CAS Key Lab of Network Data Science and Technology, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China; University of Chinese Academy of Sciences, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2022,12,21]]},"reference":[{"key":"e_1_3_2_2_2","article-title":"DocBERT: BERT for document classification","author":"Adhikari Ashutosh","year":"2019","unstructured":"Ashutosh Adhikari, Achyudh Ram, Raphael Tang, and Jimmy Lin. 2019. DocBERT: BERT for document classification. arXiv preprint arXiv:1904.08398 (2019).","journal-title":"arXiv preprint arXiv:1904.08398"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.3115\/1220575.1220695"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACCESS.2018.2807385"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D18-1316"},{"key":"e_1_3_2_6_2","first-page":"3","volume-title":"ACM Sigir Forum","author":"Broder Andrei","year":"2002","unstructured":"Andrei Broder. 2002. A taxonomy of web search. In ACM Sigir Forum, Vol. 36. ACM New York, NY, 3\u201310."},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1145\/1102351.1102363"},{"issue":"23","key":"e_1_3_2_8_2","first-page":"81","article-title":"From RankNet to LambdaRank to LambdaMART: An overview","volume":"11","author":"Burges Christopher J. C.","year":"2010","unstructured":"Christopher J. C. Burges. 2010. From RankNet to LambdaRank to LambdaMART: An overview. Learning 11, 23-581 (2010), 81.","journal-title":"Learning"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1561\/1500000021"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/3128572.3140448"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3210141"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1145\/1645953.1646059"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.3390\/make2040030"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/3331184.3331356"},{"key":"e_1_3_2_15_2","first-page":"641","volume-title":"Proceedings of the 14th International Conference on Neural Information Processing Systems","volume":"1","author":"Crammer Koby","year":"2001","unstructured":"Koby Crammer, Yoram Singer et\u00a0al. 2001. Pranking with ranking. In Proceedings of the 14th International Conference on Neural Information Processing Systems, Vol. 1. 641\u2013647."},{"key":"e_1_3_2_16_2","volume-title":"Search Engines: Information Retrieval in Practice","author":"Croft W. Bruce","year":"2010","unstructured":"W. Bruce Croft, Donald Metzler, and Trevor Strohman. 2010. Search Engines: Information Retrieval in Practice, Vol. 520. Addison-Wesley Reading."},{"key":"e_1_3_2_17_2","first-page":"293","volume-title":"Proceedings of the Conference on Empirical Methods in Natural Language Processing","author":"Cucerzan Silviu","year":"2004","unstructured":"Silviu Cucerzan and Eric Brill. 2004. Spelling correction as an iterative process that exploits the collective knowledge of web users. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 293\u2013300."},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3331184.3331303"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1145\/3159652.3159659"},{"key":"e_1_3_2_20_2","unstructured":"Matt Davis. 2003. Psycholinguistic Evidence on Scrambled Letters in Reading. https:\/\/www.mrc-cbu.cam.ac.uk\/people\/matt.davis\/cmabridge\/."},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P18-2006"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3209980"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.5555\/2946645.2946704"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/SPW.2018.00016"},{"key":"e_1_3_2_25_2","article-title":"Adversarial texts with gradient methods","author":"Gong Zhitao","year":"2018","unstructured":"Zhitao Gong, Wenlu Wang, Bo Li, Dawn Song, and Wei-Shinn Ku. 2018. Adversarial texts with gradient methods. arXiv preprint arXiv:1801.07175 (2018).","journal-title":"arXiv preprint arXiv:1801.07175"},{"key":"e_1_3_2_26_2","article-title":"Explaining and harnessing adversarial examples","author":"Goodfellow Ian J.","year":"2014","unstructured":"Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 (2014).","journal-title":"arXiv preprint arXiv:1412.6572"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1145\/3331184.3331414"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3210012"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/3340531.3412330"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/2983323.2983769"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1145\/3331184.3331403"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1145\/3077136.3080749"},{"key":"e_1_3_2_33_2","article-title":"Scaling out-of-distribution detection for real-world settings","author":"Hendrycks Dan","year":"2019","unstructured":"Dan Hendrycks, Steven Basart, Mantas Mazeika, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. 2019. Scaling out-of-distribution detection for real-world settings. arXiv preprint arXiv:1911.11132 (2019).","journal-title":"arXiv preprint arXiv:1911.11132"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.00823"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.244"},{"key":"e_1_3_2_36_2","first-page":"513","volume-title":"Proceedings of the European Conference on Artificial Intelligence","author":"Hofst\u00e4tter Sebastian","year":"2020","unstructured":"Sebastian Hofst\u00e4tter, Markus Zlabinger, and Allan Hanbury. 2020. Interpretable & time-budget-constrained contextualization for re-ranking. In Proceedings of the European Conference on Artificial Intelligence. IOS Press, 513\u2013520."},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/2505515.2505665"},{"key":"e_1_3_2_38_2","doi-asserted-by":"crossref","unstructured":"Samuel Huston and W. Bruce Croft. 2014. Parameters learned in the comparison of retrieval models using term dependencies. Ir University of Massachusetts. Technical Report.","DOI":"10.1145\/2661829.2661894"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D17-1215"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v34i05.6311"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/775047.775067"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/1150402.1150429"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2020.acl-main.245"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1162\/tacl_a_00300"},{"issue":"3","key":"e_1_3_2_45_2","first-page":"883","article-title":"Multi-head multi-layer attention to deep language representations for grammatical error detection","volume":"23","author":"Kaneko Masahiro","year":"2019","unstructured":"Masahiro Kaneko and Mamoru Komachi. 2019. Multi-head multi-layer attention to deep language representations for grammatical error detection. Computaci\u00f3n y Sistemas 23, 3 (2019), 883\u2013891.","journal-title":"Computaci\u00f3n y Sistemas"},{"key":"e_1_3_2_46_2","first-page":"4171","volume-title":"Proceedings of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)","author":"Kenton Jacob Devlin, Ming-Wei Chang,","year":"2019","unstructured":"Jacob Devlin, Ming-Wei Chang, KentonLee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT). 4171\u20134186."},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401075"},{"key":"e_1_3_2_48_2","article-title":"Adam: A method for stochastic optimization","author":"Kingma Diederik P.","year":"2014","unstructured":"Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).","journal-title":"arXiv preprint arXiv:1412.6980"},{"key":"e_1_3_2_49_2","first-page":"5637","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Koh Pang Wei","year":"2021","unstructured":"Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao et\u00a0al. 2021. Wilds: A benchmark of in-the-wild distribution shifts. In Proceedings of the International Conference on Machine Learning. PMLR, 5637\u20135664."},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0004-3702(99)00101-0"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.14722\/ndss.2019.23138"},{"key":"e_1_3_2_52_2","first-page":"3866","volume-title":"Proceedings of the International Conference on Machine Learning","author":"Li Yandong","year":"2019","unstructured":"Yandong Li, Lijun Li, Liqiang Wang, Tong Zhang, and Boqing Gong. 2019. NATTACK: Learning the distributions of adversarial examples for an improved black-box attack on deep neural networks. In Proceedings of the International Conference on Machine Learning. PMLR, 3866\u20133876."},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.5555\/3304222.3304355"},{"key":"e_1_3_2_54_2","first-page":"40","volume-title":"ACM SIGIR Forum","author":"Lin Jimmy","year":"2019","unstructured":"Jimmy Lin. 2019. The neural hype and comparisons against weak baselines. In ACM SIGIR Forum, Vol. 52. ACM New York, NY, 40\u201351."},{"key":"e_1_3_2_55_2","doi-asserted-by":"crossref","unstructured":"Tie-Yan Liu. 2009. Learning to rank for information retrieval. Foundations and Trends\u00ae in Information Retrieval 3.3 (2009) 225\u2013331.","DOI":"10.1561\/1500000016"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3209999"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1145\/3404835.3462869"},{"key":"e_1_3_2_58_2","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Madry Aleksander","year":"2018","unstructured":"Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2018. Towards deep learning models resistant to adversarial attacks. In Proceedings of the International Conference on Learning Representations."},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3210155"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/K18-1007"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1145\/3038912.3052579"},{"key":"e_1_3_2_62_2","article-title":"Adversarial training methods for semi-supervised text classification","author":"Miyato Takeru","year":"2016","unstructured":"Takeru Miyato, Andrew M. Dai, and Ian Goodfellow. 2016. Adversarial training methods for semi-supervised text classification. arXiv preprint arXiv:1605.07725 (2016).","journal-title":"arXiv preprint arXiv:1605.07725"},{"key":"e_1_3_2_63_2","article-title":"Task-generalizable adversarial attack based on perceptual metric","author":"Naseer Muzammal","year":"2018","unstructured":"Muzammal Naseer, Salman H. Khan, Shafin Rahman, and Fatih Porikli. 2018. Task-generalizable adversarial attack based on perceptual metric. arXiv preprint arXiv:1811.09020 (2018).","journal-title":"arXiv preprint arXiv:1811.09020"},{"key":"e_1_3_2_64_2","article-title":"Passage Re-ranking with BERT","author":"Nogueira Rodrigo","year":"2019","unstructured":"Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage Re-ranking with BERT. arXiv preprint arXiv:1901.04085 (2019).","journal-title":"arXiv preprint arXiv:1901.04085"},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","DOI":"10.5555\/1795943"},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.1145\/3132847.3132914"},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","DOI":"10.1109\/EuroSP.2016.36"},{"key":"e_1_3_2_68_2","article-title":"Evaluating the robustness of retrieval pipelines with query variation generators","author":"Penha Gustavo","year":"2021","unstructured":"Gustavo Penha, Arthur C\u00e2mara, and Claudia Hauff. 2021. Evaluating the robustness of retrieval pipelines with query variation generators. arXiv preprint arXiv:2111.13057 (2021).","journal-title":"arXiv preprint arXiv:2111.13057"},{"key":"e_1_3_2_69_2","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/D14-1162"},{"key":"e_1_3_2_70_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N18-1202"},{"key":"e_1_3_2_71_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/P19-1561"},{"key":"e_1_3_2_72_2","article-title":"Introducing LETOR 4.0 datasets","author":"Qin Tao","year":"2013","unstructured":"Tao Qin and Tie-Yan Liu. 2013. Introducing LETOR 4.0 datasets. arXiv preprint arXiv:1306.2597 (2013).","journal-title":"arXiv preprint arXiv:1306.2597"},{"key":"e_1_3_2_73_2","volume-title":"Proceedings of the International Conference on Learning Representations","author":"Qin Zhen","year":"2020","unstructured":"Zhen Qin, Le Yan, Honglei Zhuang, Yi Tay, Rama Kumar Pasumarthi, Xuanhui Wang, Michael Bendersky, and Marc Najork. 2020. Are neural rankers still outperformed by gradient boosted decision trees? In Proceedings of the International Conference on Learning Representations."},{"key":"e_1_3_2_74_2","volume-title":"Proceedings of the Language Resources and Evaluation Conference (LREC)","author":"Radev Dragomir R.","year":"2002","unstructured":"Dragomir R. Radev, Hong Qi, Harris Wu, and Weiguo Fan. 2002. Evaluating web-based question answering systems. In Proceedings of the Language Resources and Evaluation Conference (LREC). Citeseer."},{"key":"e_1_3_2_75_2","doi-asserted-by":"publisher","DOI":"10.1145\/3077136.3080785"},{"key":"e_1_3_2_76_2","doi-asserted-by":"publisher","DOI":"10.1109\/MAES.2007.327521"},{"key":"e_1_3_2_77_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4471-2099-5_24"},{"key":"e_1_3_2_78_2","article-title":"Towards crafting text adversarial samples","author":"Samanta Suranjana","year":"2017","unstructured":"Suranjana Samanta and Sameep Mehta. 2017. Towards crafting text adversarial samples. arXiv preprint arXiv:1707.02812 (2017).","journal-title":"arXiv preprint arXiv:1707.02812"},{"key":"e_1_3_2_79_2","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2018\/601"},{"key":"e_1_3_2_80_2","volume-title":"Introduction to Information Retrieval","author":"Sch\u00fctze Hinrich","year":"2008","unstructured":"Hinrich Sch\u00fctze, Christopher D. Manning, and Prabhakar Raghavan. 2008. Introduction to Information Retrieval. Vol. 39. Cambridge University Press, Cambridge, UK."},{"key":"e_1_3_2_81_2","doi-asserted-by":"publisher","DOI":"10.1109\/MDAT.2020.2971217"},{"key":"e_1_3_2_82_2","doi-asserted-by":"publisher","DOI":"10.1145\/3437963.3441772"},{"key":"e_1_3_2_83_2","volume-title":"Proceedings of the 2nd International Conference on Learning Representations","author":"Szegedy Christian","year":"2014","unstructured":"Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna Estrach, Dumitru Erhan, Ian Goodfellow, and Robert Fergus. 2014. Intriguing properties of neural networks. In Proceedings of the 2nd International Conference on Learning Representations."},{"key":"e_1_3_2_84_2","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2011.5995347"},{"key":"e_1_3_2_85_2","doi-asserted-by":"publisher","DOI":"10.3389\/frai.2021.576892"},{"key":"e_1_3_2_86_2","volume-title":"Proceedings of the 12th Text REtrieval Conference","author":"Voorhees Ellen M.","year":"2003","unstructured":"Ellen M. Voorhees. 2003. Overview of the TREC 2003 robust retrieval track. In Proceedings of the 12th Text REtrieval Conference."},{"key":"e_1_3_2_87_2","volume-title":"Proceedings of the 13th Text REtrieval Conference","author":"Voorhees Ellen M.","year":"2004","unstructured":"Ellen M. Voorhees. 2004. Overview of the TREC 2004 robust track. In Proceedings of the 13th Text REtrieval Conference."},{"key":"e_1_3_2_88_2","article-title":"Overview of the TREC 2005 robust retrieval track","author":"Voorhees Ellen M.","year":"2006","unstructured":"Ellen M. Voorhees. 2006. Overview of the TREC 2005 robust retrieval track. In Proceedings of the Text REtrieval Conference.","journal-title":"Proceedings of the Text REtrieval Conference"},{"key":"e_1_3_2_89_2","doi-asserted-by":"publisher","DOI":"10.1145\/2009916.2010190"},{"key":"e_1_3_2_90_2","article-title":"Maximal Jacobian-based saliency map attack","author":"Wiyatno Rey","year":"2018","unstructured":"Rey Wiyatno and Anqi Xu. 2018. Maximal Jacobian-based saliency map attack. arXiv preprint arXiv:1808.07945 (2018).","journal-title":"arXiv preprint arXiv:1808.07945"},{"key":"e_1_3_2_91_2","article-title":"Google\u2019s neural machine translation system: Bridging the gap between human and machine translation","author":"Wu Yonghui","year":"2016","unstructured":"Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey et\u00a0al. 2016. Google\u2019s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 (2016).","journal-title":"arXiv preprint arXiv:1609.08144"},{"key":"e_1_3_2_92_2","doi-asserted-by":"publisher","DOI":"10.1145\/3077136.3080809"},{"key":"e_1_3_2_93_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.naacl-main.321"},{"key":"e_1_3_2_94_2","doi-asserted-by":"publisher","DOI":"10.1145\/3239571"},{"key":"e_1_3_2_95_2","article-title":"Data augmentation for BERT fine-tuning in open-domain question answering","author":"Yang Wei","year":"2019","unstructured":"Wei Yang, Yuqing Xie, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. 2019. Data augmentation for BERT fine-tuning in open-domain question answering. arXiv preprint arXiv:1904.06652 (2019).","journal-title":"arXiv preprint arXiv:1904.06652"},{"key":"e_1_3_2_96_2","doi-asserted-by":"publisher","DOI":"10.1145\/3336191.3371868"},{"key":"e_1_3_2_97_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/D19-1352"},{"key":"e_1_3_2_98_2","first-page":"268","volume-title":"ACM SIGIR Forum","author":"Zhai Chengxiang","year":"2017","unstructured":"Chengxiang Zhai and John Lafferty. 2017. A study of smoothing methods for language models applied to ad hoc information retrieval. In ACM SIGIR Forum, Vol. 51. ACM New York, NY, 268\u2013276."},{"key":"e_1_3_2_99_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSE.2019.2962027"},{"key":"e_1_3_2_100_2","doi-asserted-by":"publisher","DOI":"10.1145\/2484028.2484127"},{"key":"e_1_3_2_101_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2013.08.004"},{"key":"e_1_3_2_102_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.01610"},{"key":"e_1_3_2_103_2","doi-asserted-by":"publisher","DOI":"10.1145\/1183614.1183696"},{"key":"e_1_3_2_104_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.emnlp-main.225"},{"key":"e_1_3_2_105_2","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390524"}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3534928","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3534928","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:02:54Z","timestamp":1750186974000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3534928"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,12,21]]},"references-count":104,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2023,4,30]]}},"alternative-id":["10.1145\/3534928"],"URL":"https:\/\/doi.org\/10.1145\/3534928","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"value":"1046-8188","type":"print"},{"value":"1558-2868","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,12,21]]},"assertion":[{"value":"2021-08-10","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-04-30","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-12-21","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}