{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,11]],"date-time":"2026-08-11T15:56:14Z","timestamp":1786463774431,"version":"build-2736575974"},"reference-count":80,"publisher":"Association for Computing Machinery (ACM)","issue":"6","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2026,7,31]]},"abstract":"<jats:p>\n                    Query Performance Prediction (QPP) estimates the effectiveness of retrieval systems for a given query, offering valuable insights for search effectiveness and query processing. Despite extensive research, a critical gap remains in understanding how well QPPs generalize across diverse retrieval paradigms and collections, a question of robustness that has significant implications for their practical utility. This article provides the first comprehensive cross-paradigm evaluation of QPP robustness and generalization capabilities, examining state-of-the-art QPPs including NQC, WIG, LETOR-based features, and newly explored dense-based predictors MQPPF and BERT-QPP. We systematically assess their performance across diverse sparse (BM25, DFree with and without query expansion), hybrid (SPLADE), and dense (ColBERT, TCT-ColBERT) rankers on four benchmark collections: TREC Robust, GOV2, WT10G, and MS-MARCO. The results reveal fundamental robustness challenges: predictors exhibit significant variability in accuracy, with collection being the dominant factor, followed by ranker type. Some sparse predictors perform adequately on specific collections such as TREC Robust and GOV2, but critically fail to generalize to other collections like WT10G and MS-MARCO. Dense-based predictors, while showing promise in specific scenarios with dense rankers, similarly lack generalization to sparse contexts. We demonstrate that these generalization failures severely limit practical applications: QPP-driven selective query processing achieves only marginal gains (\n                    <jats:inline-formula content-type=\"math\/tex\">\n                      <jats:tex-math notation=\"LaTeX\" version=\"MathJax\">\\(\\approx\\)<\/jats:tex-math>\n                    <\/jats:inline-formula>\n                    4% NDCG improvement), with reliability varying dramatically across settings. Our findings underscore that current QPP methods lack the robustness necessary for real-world deployment and highlight the urgent need for predictors that generalize reliably across diverse collections, align with modern dense retrieval architectures, and provide consistent utility for downstream applications. We publicly release our data and code to facilitate future research on robust QPP methods (\n                    <jats:ext-link xmlns:xlink=\"http:\/\/www.w3.org\/1999\/xlink\" ext-link-type=\"uri\" xlink:href=\"https:\/\/github.com\/adrianchifu\/UncoveringTheLimitationsofQPP\/\">https:\/\/github.com\/adrianchifu\/UncoveringTheLimitationsofQPP\/<\/jats:ext-link>\n                    ).\n                  <\/jats:p>","DOI":"10.1145\/3774427","type":"journal-article","created":{"date-parts":[[2025,11,20]],"date-time":"2025-11-20T13:05:26Z","timestamp":1763643926000},"page":"1-28","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":2,"title":["Uncovering the Limitations of Query Performance Prediction: Failures, Insights, and Implications for Selective Query Processing"],"prefix":"10.1145","volume":"44","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4680-5528","authenticated-orcid":false,"given":"Adrian-Gabriel","family":"Chifu","sequence":"first","affiliation":[{"name":"Aix Marseille Universit\u00e9, Universit\u00e9 de Toulon, CNRS, LIS, Marseille, France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9610-5306","authenticated-orcid":false,"given":"S\u00e9bastien","family":"D\u00e9jean","sequence":"additional","affiliation":[{"name":"IMT, UMR 5219 CNRS, UPS, Universit\u00e9 de Toulouse, Toulouse, France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2528-441X","authenticated-orcid":false,"given":"Moncef","family":"Garouani","sequence":"additional","affiliation":[{"name":"IRIT, UMR 5505 CNRS, Universit\u00e9 Toulouse Capitole, Toulouse, France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9273-2193","authenticated-orcid":false,"given":"Josiane","family":"Mothe","sequence":"additional","affiliation":[{"name":"IRIT, UMR 5505 CNRS, Universit\u00e9 Toulouse Capitole, Toulouse, France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-7272-5644","authenticated-orcid":false,"given":"Di\u00e9go","family":"Ortiz","sequence":"additional","affiliation":[{"name":"IRIT, UMR 5505 CNRS, Toulouse, France"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4022-7344","authenticated-orcid":false,"given":"Md Zia","family":"Ullah","sequence":"additional","affiliation":[{"name":"Edinburgh Napier University, Edinburgh, Scotland, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,8,11]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-24752-4_10"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1145\/582415.582416"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1145\/3583780.3615270"},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1145\/3459637.3482063"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-72240-1_15"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-56069-9_51"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1145\/1835449.1835650"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1145\/3726302.3730348"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1145\/1835449.1835683"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-99739-7_8"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-56063-7_38"},{"key":"e_1_3_2_13_2","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3210121"},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1145\/564376.564429"},{"key":"e_1_3_2_15_2","doi-asserted-by":"crossref","unstructured":"Steve Cronen-Townsend Yun Zhou and W. Bruce Croft. 2004. A Language Modeling Framework for Selective Query Expansion. Technical Report IR338. Center for Intelligent Information Retrieval.","DOI":"10.21236\/ADA477592"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-56060-6_13"},{"key":"e_1_3_2_17_2","doi-asserted-by":"publisher","DOI":"10.1145\/3545112"},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/3341105.3373904"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.52202\/075280-0441"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1145\/3231937"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.3390\/app11199075"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/1277741.1277841"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.1145\/3578337.3605136"},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-28244-7_15"},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-72113-8_8"},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-022-09407-w"},{"key":"e_1_3_2_27_2","unstructured":"Thibault Formal Carlos Lassance Benjamin Piwowarski and St\u00e9phane Clinchant. 2021. SPLADE v2: Sparse lexical and expansion model for information retrieval. arXiv:2109.10086. Retrieve from https:\/\/arxiv.org\/abs\/2109.10086"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/3477495.3531857"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/3539618.3592046"},{"key":"e_1_3_2_30_2","volume-title":"ACM SIGIR 2005 Workshop on Predicting Query Difficulty - Methods and Applications","author":"Grivolla Jens","year":"2005","unstructured":"Jens Grivolla, Pierre Jourlin, and Renato de Mori. 2005. Automatic classification of queries by expected retrieval performance. In ACM SIGIR 2005 Workshop on Predicting Query Difficulty - Methods and Applications. ACM, Brazil."},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-00958-7_28"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-12275-0_20"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/1458082.1458311"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-30213-1_5"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/2600428.2609553"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1145\/3397271.3401075"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ins.2023.119015"},{"key":"e_1_3_2_38_2","unstructured":"Sheng-Chieh Lin Jheng-Hong Yang and Jimmy Lin. 2020. Distilling dense representations for ranking using tightly-coupled teachers. arXiv:2010.11386. Retrieved from https:\/\/arxiv.org\/abs\/2010.11386"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2021.repl4nlp-1.17"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/3446426"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1145\/2493175.2493176"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1145\/3539618.3591919"},{"key":"e_1_3_2_43_2","unstructured":"Chuan Meng Negar Arabzadeh Arian Askari Mohammad Aliannejadi and Maarten de Rijke. 2024. Query performance prediction using relevance judgments generated by large language models. arXiv:2404.01012. Retrieved from https:\/\/arxiv.org\/abs\/2404.01012"},{"key":"e_1_3_2_44_2","unstructured":"Bhaskar Mitra and Nick Craswell. 2017. Neural models for information retrieval. arXiv:1705.01509. Retrieved from https:\/\/arxiv.org\/abs\/1705.01509"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3210146"},{"key":"e_1_3_2_46_2","first-page":"29","volume-title":"Proceedings of the Query Performance Prediction and Its Evaluation in New Tasks, co-located with the 45th European Conference on Information Retrieval (ECIR) (QPP++ \u201923)","author":"Mothe Josiane","year":"2023","unstructured":"Josiane Mothe. 2023. On correlation to evaluate QPP. In Proceedings of the Query Performance Prediction and Its Evaluation in New Tasks, co-located with the 45th European Conference on Information Retrieval (ECIR) (QPP++ \u201923), 29\u201336. Retrieved from https:\/\/hal.science\/hal-04230663"},{"key":"e_1_3_2_47_2","first-page":"7","volume-title":"Proceedings of ACM Conference on Research and Development in Information Retrieval, SIGIR, Predicting Query Difficulty \u2013 Methods and Applications Workshop","author":"Mothe Josiane","year":"2005","unstructured":"Josiane Mothe and Ludovic Tanguy. 2005. Linguistic features to predict query difficulty. In Proceedings of ACM Conference on Research and Development in Information Retrieval, SIGIR, Predicting Query Difficulty \u2013 Methods and Applications Workshop. ACM, New York, NY, 7\u201310."},{"key":"e_1_3_2_48_2","unstructured":"Josiane Mothe and Md Zia Ullah. 2023. Selective query processing: A risk-sensitive selection of system configurations. arXiv: 2305.18311. Retrieved from https:\/\/arxiv.org\/abs\/2305.18311"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-31865-1_37"},{"key":"e_1_3_2_50_2","volume-title":"Proceedings of the Query Performance Prediction and its Applications in the Era of Large Language Models, co-located with the 47th European Conference on Information Retrieval (ECIR \u201925)","author":"Parry Andrew","year":"2025","unstructured":"Andrew Parry, Jan Heinrich Merker, Simon Ruth, Maik Fr\u00f6be, and Harrisen Scells. 2025. Corpora performance prediction. In Proceedings of the Query Performance Prediction and its Applications in the Era of Large Language Models, co-located with the 47th European Conference on Information Retrieval (ECIR \u201925)."},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-16321-0_21"},{"key":"e_1_3_2_52_2","unstructured":"Ronak Pradeep Rodrigo Nogueira and Jimmy Lin. 2021. The Expando-Mono-Duo design pattern for text ranking with pretrained sequence-to-sequence models. arXiv:2101.05667. Retrieved from https:\/\/arxiv.org\/abs\/2101.05667"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-009-9123-y"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1145\/3020165.3020183"},{"key":"e_1_3_2_55_2","unstructured":"Hossein A. Rahmani Clemencia Siro Mohammad Aliannejadi Nick Craswell Charles L. A. Clarke Guglielmo Faggioli Bhaskar Mitra Paul Thomas and Emine Yilmaz. 2025. Judging the judges: A collection of LLM-generated relevance judgements. arXiv:2502.13908. Retrieved from https:\/\/arxiv.org\/abs\/2502.13908"},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1145\/2600428.2609581"},{"key":"e_1_3_2_57_2","doi-asserted-by":"publisher","DOI":"10.1561\/1500000019"},{"key":"e_1_3_2_58_2","doi-asserted-by":"crossref","unstructured":"Stephen E. Robertson Steve Walker Susan Jones Micheline M. Hancock-Beaulieu Mike Gatford. 1995. Okapi at TREC-3. British Library Research and Development Department London UK.","DOI":"10.6028\/NIST.SP.500-225.city"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1145\/3077136.3080665"},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2018.10.009"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-88717-8_9"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10994-024-06659-z"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022.naacl-main.272"},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3210078"},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-04417-5_30"},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.1145\/1835449.1835494"},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","DOI":"10.1145\/2180868.2180873"},{"key":"e_1_3_2_68_2","doi-asserted-by":"publisher","DOI":"10.1145\/3539618.3592082"},{"key":"e_1_3_2_69_2","doi-asserted-by":"publisher","DOI":"10.1145\/2661829.2661906"},{"key":"e_1_3_2_70_2","unstructured":"Nandan Thakur Nils Reimers Andreas R\u00fcckl\u00e9 Abhishek Srivastava and Iryna Gurevych. 2021. BEIR: A Heterogeneous Benchmark for Zero-Shot Evaluation of Information Retrieval Models. Retrieved from https:\/\/openreview.net\/forum?id=wCu6T5xFjeJ"},{"key":"e_1_3_2_71_2","doi-asserted-by":"publisher","DOI":"10.1145\/3166072.3166079"},{"key":"e_1_3_2_72_2","doi-asserted-by":"publisher","DOI":"10.1145\/3626772.3657707"},{"key":"e_1_3_2_73_2","first-page":"913","volume-title":"Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval","author":"Ustinovskiy Yury","year":"2013","unstructured":"Yury Ustinovskiy and Pavel Serdyukov. 2013. Session-based query performance prediction. In Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, New York, NY, 913\u2013916."},{"key":"e_1_3_2_74_2","unstructured":"Maria Vlachou and Craig Macdonald. 2023. On coherence-based predictors for dense query performance prediction. arXiv:2310.11405. Retrieved from https:\/\/arxiv.org\/abs\/2310.11405"},{"key":"e_1_3_2_75_2","doi-asserted-by":"publisher","DOI":"10.1109\/TKDE.2017.2740926"},{"key":"e_1_3_2_76_2","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3210041"},{"issue":"1","key":"e_1_3_2_77_2","first-page":"24","article-title":"Query intent detection based on query log mining","volume":"13","author":"Zamora Juan","year":"2014","unstructured":"Juan Zamora, Marcelo Mendoza, and H\u00e9ctor Allende. 2014. Query intent detection based on query log mining. Journal of Web Engineering 13, 1&2 (2014), 24\u201352.","journal-title":"Journal of Web Engineering"},{"key":"e_1_3_2_78_2","doi-asserted-by":"publisher","DOI":"10.1145\/3404835.3463039"},{"key":"e_1_3_2_79_2","doi-asserted-by":"publisher","DOI":"10.1145\/1183614.1183696"},{"key":"e_1_3_2_80_2","doi-asserted-by":"publisher","DOI":"10.1145\/1277741.1277835"},{"issue":"4","key":"e_1_3_2_81_2","first-page":"322","article-title":"How reliable are the results of large-scale information retrieval experiments","volume":"16","author":"Zobel Justin","year":"1998","unstructured":"Justin Zobel. 1998. How reliable are the results of large-scale information retrieval experiments? ACM Transactions on Information Systems 16, 4 (1998), 322\u2013351.","journal-title":"ACM Transactions on Information Systems"}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3774427","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,11]],"date-time":"2026-08-11T13:46:20Z","timestamp":1786455980000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3774427"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,31]]},"references-count":80,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2026,7,31]]}},"alternative-id":["10.1145\/3774427"],"URL":"https:\/\/doi.org\/10.1145\/3774427","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"value":"1046-8188","type":"print"},{"value":"1558-2868","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,31]]},"assertion":[{"value":"2025-06-15","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-10-28","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2026-08-11","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}