{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,16]],"date-time":"2026-01-16T17:55:21Z","timestamp":1768586121497,"version":"3.49.0"},"reference-count":59,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2016,6,9]],"date-time":"2016-06-09T00:00:00Z","timestamp":1465430400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2016,9,14]]},"abstract":"<jats:p>The task of query performance prediction is to estimate the effectiveness of search performed in response to a query when no relevance judgments are available. We present a novel probabilistic analysis of the performance prediction task. The analysis gives rise to a general prediction framework that uses pseudo-effective or ineffective document lists that are retrieved in response to the query. These lists serve as reference to the result list at hand, the effectiveness of which we want to predict. We show that many previously proposed prediction methods can be explained using our framework. More generally, we shed new light on existing prediction methods and establish formal common grounds to seemingly different prediction approaches. In addition, we formally demonstrate the connection between prediction using reference lists and fusion of retrieved lists, and provide empirical support to this connection. Through an extensive empirical exploration, we study various factors that affect the quality of prediction using reference lists.<\/jats:p>","DOI":"10.1145\/2926790","type":"journal-article","created":{"date-parts":[[2016,6,10]],"date-time":"2016-06-10T13:54:00Z","timestamp":1465566840000},"page":"1-34","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":41,"title":["Query Performance Prediction Using Reference Lists"],"prefix":"10.1145","volume":"34","author":[{"given":"Anna","family":"Shtok","sequence":"first","affiliation":[{"name":"Technion\u2014Israel Institute of Technology, Haifa, Israel"}]},{"given":"Oren","family":"Kurland","sequence":"additional","affiliation":[{"name":"Technion\u2014Israel Institute of Technology, Haifa, Israel"}]},{"given":"David","family":"Carmel","sequence":"additional","affiliation":[{"name":"Yahoo Lab, Haifa, Haifa, Israel"}]}],"member":"320","published-online":{"date-parts":[[2016,6,9]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Proceedings of the 9th Text Retrieval Conference (TREC\u201900)","author":"Allan James","year":"2000","unstructured":"James Allan , Margaret E. Connell , W. Bruce Croft , Fang-Fang Feng , David Fisher , and Xiaoyan Li . 2000 . INQUERY and TREC-9 . In Proceedings of the 9th Text Retrieval Conference (TREC\u201900) . 551--562. NIST Special Publication 500-249. James Allan, Margaret E. Connell, W. Bruce Croft, Fang-Fang Feng, David Fisher, and Xiaoyan Li. 2000. INQUERY and TREC-9. In Proceedings of the 9th Text Retrieval Conference (TREC\u201900). 551--562. NIST Special Publication 500-249."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-24752-4_10"},{"key":"e_1_2_1_3_1","volume-title":"Proceedings of the 29th European Conference on Information Retrieval (ECIR\u201907)","author":"Javed","unstructured":"Javed A. Aslam and Virgiliu Pavlu. 2007. Query hardness estimation using Jensen-Shannon divergence among multiple scoring functions . In Proceedings of the 29th European Conference on Information Retrieval (ECIR\u201907) . 198--209. Javed A. Aslam and Virgiliu Pavlu. 2007. Query hardness estimation using Jensen-Shannon divergence among multiple scoring functions. In Proceedings of the 29th European Conference on Information Retrieval (ECIR\u201907). 198--209."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835449.1835650"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/952532.952695"},{"key":"e_1_2_1_6_1","volume-title":"Proceedings of the 14th Text Retrieval Conference (TREC\u201904)","author":"Bernstein Y.","unstructured":"Y. Bernstein , B. Billerbeck , S. Garcia , N. Lester , F. Scholer , and J. Zobel . 2005. RMIT University at TREC 2005: Terabyte and robust track . In Proceedings of the 14th Text Retrieval Conference (TREC\u201904) . Y. Bernstein, B. Billerbeck, S. Garcia, N. Lester, F. Scholer, and J. Zobel. 2005. RMIT University at TREC 2005: Terabyte and robust track. In Proceedings of the 14th Text Retrieval Conference (TREC\u201904)."},{"key":"e_1_2_1_7_1","volume-title":"Estimating the Query Difficulty for Information Retrieval. Morgan & Claypool","author":"Carmel David","unstructured":"David Carmel and Elad Yom-Tov . 2010. Estimating the Query Difficulty for Information Retrieval. Morgan & Claypool , San Francisco, CA . David Carmel and Elad Yom-Tov. 2010. Estimating the Query Difficulty for Information Retrieval. Morgan & Claypool, San Francisco, CA."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/1148170.1148238"},{"key":"e_1_2_1_9_1","volume-title":"Information Retrieval: Recent Research from the Center for Intelligent Information Retrieval","author":"Croft W. Bruce","year":"2000","unstructured":"W. Bruce Croft (Ed.). 2000 . Combining approaches to information retrieval. In Advances in Information Retrieval: Recent Research from the Center for Intelligent Information Retrieval . W. Bruce Croft (Ed.). Number 7 in The Kluwer International Series on Information Retrieval. Kluwer . 1--36. W. Bruce Croft (Ed.). 2000. Combining approaches to information retrieval. In Advances in Information Retrieval: Recent Research from the Center for Intelligent Information Retrieval. W. Bruce Croft (Ed.). Number 7 in The Kluwer International Series on Information Retrieval. Kluwer. 1--36."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/564376.564429"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-25631-8_29"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2559170"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/2009916.2010063"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-28997-2_25"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835449.1835626"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(199009)41:6<391::AID-ASI1>3.0.CO;2-9"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/1099554.1099722"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/1277741.1277841"},{"key":"e_1_2_1_20_1","volume-title":"Proceedings of the 2nd Text REtrieval Conference (TREC\u201994)","author":"Edward","unstructured":"Edward A. Fox and Joseph A. Shaw. 1994. Combination of multiple searches . In Proceedings of the 2nd Text REtrieval Conference (TREC\u201994) . Edward A. Fox and Joseph A. Shaw. 1994. Combination of multiple searches. In Proceedings of the 2nd Text REtrieval Conference (TREC\u201994)."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/1008992.1009104"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/1458082.1458311"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-30213-1_5"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/1571941.1572038"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.5555\/2040317.2040323"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2396761.2396866"},{"key":"e_1_2_1_28_1","volume-title":"Language Modeling for Information Retrieval","author":"Lafferty John","unstructured":"John Lafferty and ChengXiang Zhai . 2003. Probabilistic relevance models based on document and query generation . In Language Modeling for Information Retrieval . W. Bruce Croft and John Lafferty (Eds.). Number 13 in Information Retrieval Book Series. Kluwer . 1--10. John Lafferty and ChengXiang Zhai. 2003. Probabilistic relevance models based on document and query generation. In Language Modeling for Information Retrieval. W. Bruce Croft and John Lafferty (Eds.). Number 13 in Information Retrieval Book Series. Kluwer. 1--10."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/383952.383972"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/258525.258587"},{"key":"e_1_2_1_31_1","volume-title":"ACM SIGIR 2005 Workshop on Predicting Query Difficulty\u2014Methods and Applications.","author":"Mothe Josiane","year":"2005","unstructured":"Josiane Mothe and Ludovic Tanguy . 2005 . Linguistic features to predict query difficulty . In ACM SIGIR 2005 Workshop on Predicting Query Difficulty\u2014Methods and Applications. Josiane Mothe and Ludovic Tanguy. 2005. Linguistic features to predict query difficulty. In ACM SIGIR 2005 Workshop on Predicting Query Difficulty\u2014Methods and Applications."},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.5555\/1928328.1928355"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2600428.2609581"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1108\/eb026647"},{"key":"e_1_2_1_35_1","volume-title":"The SMART Retrieval System: Experiments in Automatic Document Processing","author":"Rocchio Joseph John","unstructured":"Joseph John Rocchio . 1971. Relevance feedback in information retrieval . In The SMART Retrieval System: Experiments in Automatic Document Processing , Gerard Salton (Ed.). Prentice Hall , Upper Saddle River, NJ, 313--323. Joseph John Rocchio. 1971. Relevance feedback in information retrieval. In The SMART Retrieval System: Experiments in Automatic Document Processing, Gerard Salton (Ed.). Prentice Hall, Upper Saddle River, NJ, 313--323."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/1148170.1148192"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/1571941.1572055"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.v55:7"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-04417-5_30"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835449.1835494"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/2180868.2180873"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835449.1835549"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/1321440.1321528"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/383952.383961"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/312624.312698"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0306-4573(00)00015-7"},{"key":"e_1_2_1_47_1","volume-title":"Proceedings of the 13th Text Retrieval Conference (TREC\u201904)","author":"Tomlinson Stephen","year":"2004","unstructured":"Stephen Tomlinson . 2004 . Robust, web and terabyte retrieval with hummingbird search server at TREC 2004 . In Proceedings of the 13th Text Retrieval Conference (TREC\u201904) . Stephen Tomlinson. 2004. Robust, web and terabyte retrieval with hummingbird search server at TREC 2004. In Proceedings of the 13th Text Retrieval Conference (TREC\u201904)."},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/502585.502608"},{"key":"e_1_2_1_49_1","volume-title":"Information Retrieval","author":"van Rijsbergen C. J.","unstructured":"C. J. van Rijsbergen . 1979. Information Retrieval ( 2 nd ed.). Butterworths , Oxford, UK . C. J. van Rijsbergen. 1979. Information Retrieval (2nd ed.). Butterworths, Oxford, UK.","edition":"2"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/1148170.1148239"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/290941.290991"},{"key":"e_1_2_1_52_1","volume-title":"Overview of the TREC 2004 robust retrieval track. In Proceedings of the 13th Text Retrieval Conference (TREC\u201904)","author":"Voorhees Ellen M.","year":"2004","unstructured":"Ellen M. Voorhees . 2004 . Overview of the TREC 2004 robust retrieval track. In Proceedings of the 13th Text Retrieval Conference (TREC\u201904) . Ellen M. Voorhees. 2004. Overview of the TREC 2004 robust retrieval track. In Proceedings of the 13th Text Retrieval Conference (TREC\u201904)."},{"key":"e_1_2_1_53_1","volume-title":"Harman","author":"Voorhees Ellen M.","year":"2005","unstructured":"Ellen M. Voorhees and Donna K . Harman . 2005 . TREC : Experiments and Evaluation in Information Retrieval. MIT Press , Cambridge, MA. Ellen M. Voorhees and Donna K. Harman. 2005. TREC: Experiments and Evaluation in Information Retrieval. MIT Press, Cambridge, MA."},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390374"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1145\/1852102.1852106"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/1076034.1076121"},{"key":"e_1_2_1_57_1","volume-title":"ACM SIGIR 2005 Workshop on Predicting Query Difficulty\u2014Methods and Applications.","author":"Yom-Tov Elad","year":"2005","unstructured":"Elad Yom-Tov , Shai Fine , David Carmel , and Adam Darlow . 2005 b. Metasearch and federation using query difficulty prediction . In ACM SIGIR 2005 Workshop on Predicting Query Difficulty\u2014Methods and Applications. Elad Yom-Tov, Shai Fine, David Carmel, and Adam Darlow. 2005b. Metasearch and federation using query difficulty prediction. In ACM SIGIR 2005 Workshop on Predicting Query Difficulty\u2014Methods and Applications."},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/383952.384019"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.5555\/1793274.1793285"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/1183614.1183696"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/1277741.1277835"}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2926790","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2926790","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:56:21Z","timestamp":1750222581000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2926790"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2016,6,9]]},"references-count":59,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2016,9,14]]}},"alternative-id":["10.1145\/2926790"],"URL":"https:\/\/doi.org\/10.1145\/2926790","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"value":"1046-8188","type":"print"},{"value":"1558-2868","type":"electronic"}],"subject":[],"published":{"date-parts":[[2016,6,9]]},"assertion":[{"value":"2015-02-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2015-11-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2016-06-09","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}