{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,11]],"date-time":"2026-07-11T02:30:51Z","timestamp":1783737051249,"version":"3.55.0"},"reference-count":83,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2021,9,8]],"date-time":"2021-09-08T00:00:00Z","timestamp":1631059200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"\u201cDAta BenchmarK for Keyword-based Access and Retrieval\u201d"},{"name":"University of Padova Strategic Research Infrastructure","award":["2017"],"award-info":[{"award-number":["2017"]}]},{"DOI":"10.13039\/501100000923","name":"Australian Research Council","doi-asserted-by":"crossref","award":["DP190101113"],"award-info":[{"award-number":["DP190101113"]}],"id":[{"id":"10.13039\/501100000923","id-type":"DOI","asserted-by":"crossref"}]},{"DOI":"10.13039\/501100003977","name":"Israel Science Foundation","doi-asserted-by":"crossref","award":["1136\/17"],"award-info":[{"award-number":["1136\/17"]}],"id":[{"id":"10.13039\/501100003977","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2022,1,31]]},"abstract":"<jats:p>Several recent studies have explored the interaction effects between topics, systems, corpora, and components when measuring retrieval effectiveness. However, all of these previous studies assume that a topic or information need is represented by a single query. In reality, users routinely reformulate queries to satisfy an information need. In recent years, there has been renewed interest in the notion of \u201cquery variations\u201d which are essentially multiple user formulations for an information need. Like many retrieval models, some queries are highly effective while others are not. This is often an artifact of the collection being searched which might be more or less sensitive to word choice. Users rarely have perfect knowledge about the underlying collection, and so finding queries that work is often a trial-and-error process. In this work, we explore the fundamental problem of system interaction effects between collections, ranking models, and queries. To answer this important question, we formalize the analysis using ANalysis Of VAriance (ANOVA) models to measure multiple components effects across collections and topics by nesting multiple query variations within each topic. Our findings show that query formulations have a comparable effect size of the topic factor itself, which is known to be the factor with the greatest effect size in prior ANOVA studies. Both topic and formulation have a substantially larger effect size than any other factor, including the ranking algorithms and, surprisingly, even query expansion. This finding reinforces the importance of further research in understanding the role of query rewriting in IR related tasks.<\/jats:p>","DOI":"10.1145\/3470563","type":"journal-article","created":{"date-parts":[[2021,9,8]],"date-time":"2021-09-08T15:31:23Z","timestamp":1631115083000},"page":"1-36","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":25,"title":["Topic Difficulty: Collection and Query Formulation Effects"],"prefix":"10.1145","volume":"40","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1902-9087","authenticated-orcid":false,"given":"J. Shane","family":"Culpepper","sequence":"first","affiliation":[{"name":"RMIT University, Melbourne, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-5070-2049","authenticated-orcid":false,"given":"Guglielmo","family":"Faggioli","sequence":"additional","affiliation":[{"name":"University of Padua, Padova, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9219-6239","authenticated-orcid":false,"given":"Nicola","family":"Ferro","sequence":"additional","affiliation":[{"name":"University of Padua, Padova, Italy"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Oren","family":"Kurland","sequence":"additional","affiliation":[{"name":"Technion, Israel Institute of Technology, Technion City, Haifa, Israel"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2021,9,8]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"TREC 2017 common core track overview. In Proceedings of the 26th Text REtrieval Conference. National Institute of Standards and Technology, 114","author":"Allan J.","unstructured":"J. Allan , D. K. Harman , E. Kanoulas , D. Li , C. Van Gysel , and E. M. Voorhees . 2018 . TREC 2017 common core track overview. In Proceedings of the 26th Text REtrieval Conference. National Institute of Standards and Technology, 114 . (See alsoVoorhees and Ellis [76]). J. Allan, D. K. Harman, E. Kanoulas, D. Li, C. Van Gysel, and E. M. Voorhees. 2018. TREC 2017 common core track overview. In Proceedings of the 26th Text REtrieval Conference. National Institute of Standards and Technology, 114. (See alsoVoorhees and Ellis [76])."},{"key":"e_1_2_1_2_1","volume-title":"TREC 2018 common core track overview. In Proceedings of the 26th Text REtrieval Conference Proceedings (TREC","author":"Allan J.","year":"2018","unstructured":"J. Allan , D. K. Harman , E. Kanoulas , and E. M. Voorhees . 2019 . TREC 2018 common core track overview. In Proceedings of the 26th Text REtrieval Conference Proceedings (TREC 2018 ). E. M. Voorhees and A. Ellis (Eds.), National Institute of Standards and Technology (NIST), Gaithersburg, MD. J. Allan, D. K. Harman, E. Kanoulas, and E. M. Voorhees. 2019. TREC 2018 common core track overview. In Proceedings of the 26th Text REtrieval Conference Proceedings (TREC 2018). E. M. Voorhees and A. Ellis (Eds.), National Institute of Standards and Technology (NIST), Gaithersburg, MD."},{"key":"e_1_2_1_3_1","volume-title":"Advances in Information Retrieval","author":"Amati Giambattista","unstructured":"Giambattista Amati , Claudio Carpineto , and Giovanni Romano . 2004. Query difficulty, robustness, and selective application of query expansion . In Advances in Information Retrieval . Sharon McDonald and John Tait (Eds.), Springer Berlin , 127\u2013137. Giambattista Amati, Claudio Carpineto, and Giovanni Romano. 2004. Query difficulty, robustness, and selective application of query expansion. In Advances in Information Retrieval. Sharon McDonald and John Tait (Eds.), Springer Berlin, 127\u2013137."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3451964.3451967"},{"key":"e_1_2_1_5_1","volume-title":"Proceedings of the 29th European Conference on IR Research. 198\u2013209","author":"Aslam J. A.","unstructured":"J. A. Aslam and V. Pavlu . 2007. Query hardness estimation using Jensen-Shannon divergence among multiple scoring functions . In Proceedings of the 29th European Conference on IR Research. 198\u2013209 . J. A. Aslam and V. Pavlu. 2007. Query hardness estimation using Jensen-Shannon divergence among multiple scoring functions. In Proceedings of the 29th European Conference on IR Research. 198\u2013209."},{"key":"e_1_2_1_6_1","volume-title":"Proceedings of the 38th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. R. Baeza-Yates, M. Lalmas, A. Moffat, and B. Ribeiro-Neto (Eds.), ACM Press","author":"Bailey P.","unstructured":"P. Bailey , A. Moffat , F. Scholer , and P. Thomas . 2015. User variability and IR system evaluation . In Proceedings of the 38th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. R. Baeza-Yates, M. Lalmas, A. Moffat, and B. Ribeiro-Neto (Eds.), ACM Press , New York, NY, 625\u2013634. P. Bailey, A. Moffat, F. Scholer, and P. Thomas. 2015. User variability and IR system evaluation. In Proceedings of the 38th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. R. Baeza-Yates, M. Lalmas, A. Moffat, and B. Ribeiro-Neto (Eds.), ACM Press, New York, NY, 625\u2013634."},{"key":"e_1_2_1_7_1","volume-title":"Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. 395\u2013404","author":"Bailey P.","unstructured":"P. Bailey , A. Moffat , F. Scholer , and P. Thomas . 2017. Retrieval consistency in the presence of query variations . In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. 395\u2013404 . P. Bailey, A. Moffat, F. Scholer, and P. Thomas. 2017. Retrieval consistency in the presence of query variations. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. 395\u2013404."},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1009984519381"},{"key":"e_1_2_1_9_1","volume-title":"Proceedings of the 16th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 339\u2013346","author":"Belkin N. J.","unstructured":"N. J. Belkin , C. Cool , W. B. Croft , and J. P. Callan . 1993. The effect of multiple query variations on information retrieval system performance . In Proceedings of the 16th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 339\u2013346 . N. J. Belkin, C. Cool, W. B. Croft, and J. P. Callan. 1993. The effect of multiple query variations on information retrieval system performance. In Proceedings of the 16th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 339\u2013346."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1016\/0306-4573(94)00057-A"},{"key":"e_1_2_1_11_1","volume-title":"Proceedings of the 22nd Australasian Document Computing Symposium. 1\u20138.","author":"Benham R.","unstructured":"R. Benham and J. S. Culpepper . 2017. Risk-reward trade-offs in rank fusion . In Proceedings of the 22nd Australasian Document Computing Symposium. 1\u20138. R. Benham and J. S. Culpepper. 2017. Risk-reward trade-offs in rank fusion. In Proceedings of the 22nd Australasian Document Computing Symposium. 1\u20138."},{"key":"e_1_2_1_12_1","volume-title":"Proceedings of the 1st Biennial Conference on Design of Experimental Search & Information Retrieval Systems. Omar Alonso and Gianmaria Silvello (Eds.), CEUR, 62\u201367","author":"Benham R.","unstructured":"R. Benham , J. S. Culpepper , L. Gallagher , X. Lu , and J. Mackenzie . 2018. Towards efficient and effective query variant generation . In Proceedings of the 1st Biennial Conference on Design of Experimental Search & Information Retrieval Systems. Omar Alonso and Gianmaria Silvello (Eds.), CEUR, 62\u201367 . R. Benham, J. S. Culpepper, L. Gallagher, X. Lu, and J. Mackenzie. 2018. Towards efficient and effective query variant generation. In Proceedings of the 1st Biennial Conference on Design of Experimental Search & Information Retrieval Systems. Omar Alonso and Gianmaria Silvello (Eds.), CEUR, 62\u201367."},{"key":"e_1_2_1_13_1","volume-title":"Proceedings of the 26th Text REtrieval Conference. NIST.","author":"Benham R.","unstructured":"R. Benham , L. Gallagher , J. Mackenzie , T. T. Damessie , R.-C. Chen , F. Scholer , A. Moffat , and J. S. Culpepper . 2017. RMIT at the 2017 TREC CORE track . In Proceedings of the 26th Text REtrieval Conference. NIST. R. Benham, L. Gallagher, J. Mackenzie, T. T. Damessie, R.-C. Chen, F. Scholer, A. Moffat, and J. S. Culpepper. 2017. RMIT at the 2017 TREC CORE track. In Proceedings of the 26th Text REtrieval Conference. NIST."},{"key":"e_1_2_1_14_1","volume-title":"Proceedings of theText REtrieval Conference.","author":"Benham R.","unstructured":"R. Benham , L. Gallagher , J. Mackenzie , B. Liu , X. Lu , F. Scholer , A. Moffat , and J. S. Culpepper . 2018. RMIT at the 2018 TREC CORE track . In Proceedings of theText REtrieval Conference. R. Benham, L. Gallagher, J. Mackenzie, B. Liu, X. Lu, F. Scholer, A. Moffat, and J. S. Culpepper. 2018. RMIT at the 2018 TREC CORE track. In Proceedings of theText REtrieval Conference."},{"key":"e_1_2_1_15_1","doi-asserted-by":"crossref","unstructured":"D. Bodoff and P. Li. 2007. Test theory for assessing IR test collections. In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. W. Kraaij A. P. de Vries C. L. A. Clarke N. Fuhr and N. Kando (Eds.) ACM Press New York NY 367\u2013374. D. Bodoff and P. Li. 2007. Test theory for assessing IR test collections. In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. W. Kraaij A. P. de Vries C. L. A. Clarke N. Fuhr and N. Kando (Eds.) ACM Press New York NY 367\u2013374.","DOI":"10.1145\/1277741.1277805"},{"key":"e_1_2_1_16_1","volume-title":"Generalizability Theory","author":"Brennan R. L.","unstructured":"R. L. Brennan . 2001. Generalizability Theory . Springer , Berlin . R. L. Brennan. 2001. Generalizability Theory. Springer, Berlin."},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-007-9032-x"},{"key":"e_1_2_1_18_1","volume-title":"Proceedings of the 3rd Text REtrieval Conference.","author":"Buckley C.","unstructured":"C. Buckley , G. Salton , J. Allan , and A. Singhal . 1995. Automatic query expansion using SMART: TREC 3 . In Proceedings of the 3rd Text REtrieval Conference. C. Buckley, G. Salton, J. Allan, and A. Singhal. 1995. Automatic query expansion using SMART: TREC 3. In Proceedings of the 3rd Text REtrieval Conference."},{"key":"e_1_2_1_19_1","volume-title":"Text REtrieval Conference Experiment and Evaluation in Information Retrieval. D. K. Harman and E. M. Voorhees (Eds.), MIT Press, Cambridge MA, 53\u201378","author":"Buckley C.","unstructured":"C. Buckley and E. M. Voorhees . 2005. Retrieval system evaluation . In Text REtrieval Conference Experiment and Evaluation in Information Retrieval. D. K. Harman and E. M. Voorhees (Eds.), MIT Press, Cambridge MA, 53\u201378 . C. Buckley and E. M. Voorhees. 2005. Retrieval system evaluation. In Text REtrieval Conference Experiment and Evaluation in Information Retrieval. D. K. Harman and E. M. Voorhees (Eds.), MIT Press, Cambridge MA, 53\u201378."},{"key":"e_1_2_1_20_1","unstructured":"K. P. Burnham and D. R. Anderson. 2002. Model Selection and Multimodel Inference. A Practical Information-Theoretic Approach (2nd ed.). Springer Berlin. K. P. Burnham and D. R. Anderson. 2002. Model Selection and Multimodel Inference. A Practical Information-Theoretic Approach (2nd ed.). Springer Berlin."},{"key":"e_1_2_1_21_1","doi-asserted-by":"crossref","unstructured":"D. Carmel and E. Yom-Tov. 2010. Estimating the Query Difficulty for Information Retrieval. Morgan & Claypool Publishers. D. Carmel and E. Yom-Tov. 2010. Estimating the Query Difficulty for Information Retrieval. Morgan & Claypool Publishers.","DOI":"10.1007\/978-3-031-02272-2"},{"key":"e_1_2_1_22_1","volume-title":"Proceedings of the 18th Text REtrieval Conference.","author":"Carterette B.","unstructured":"B. Carterette , V. Pavlu , H. Fang , and E. Kanoulas . 2009. Million query track 2009 overview . In Proceedings of the 18th Text REtrieval Conference. B. Carterette, V. Pavlu, H. Fang, and E. Kanoulas. 2009. Million query track 2009 overview. In Proceedings of the 18th Text REtrieval Conference."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/2094072.2094076"},{"key":"e_1_2_1_24_1","volume-title":"Proceedings of the 2nd International Conference on Human Language Technology Research. 104\u2013109","author":"Cronen-Townsend S.","unstructured":"S. Cronen-Townsend and W. B. Croft . 2002. Quantifying query ambiguity . In Proceedings of the 2nd International Conference on Human Language Technology Research. 104\u2013109 . S. Cronen-Townsend and W. B. Croft. 2002. Quantifying query ambiguity. In Proceedings of the 2nd International Conference on Human Language Technology Research. 104\u2013109."},{"key":"e_1_2_1_25_1","volume-title":"ProcProceedings of the 21st Australasian Document Computing Symposium. 41\u201348","author":"Dammesie T.","unstructured":"T. Dammesie , F. Scholer , and J. S. Culpepper . 2016. The influence of topic difficulty, relevance level, and document ordering on relevance judging . In ProcProceedings of the 21st Australasian Document Computing Symposium. 41\u201348 . T. Dammesie, F. Scholer, and J. S. Culpepper. 2016. The influence of topic difficulty, relevance level, and document ordering on relevance judging. In ProcProceedings of the 21st Australasian Document Computing Symposium. 41\u201348."},{"key":"e_1_2_1_26_1","volume-title":"Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval.","author":"Faggioli G.","unstructured":"G. Faggioli , M. Ferrante , N. Ferro , R. Perego , and N. Tonellotto . 2021. Hierarchical dependence-aware evaluation measures for conversational search . In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. G. Faggioli, M. Ferrante, N. Ferro, R. Perego, and N. Tonellotto. 2021. Hierarchical dependence-aware evaluation measures for conversational search. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval."},{"key":"e_1_2_1_27_1","volume-title":"Proceedings of the 43rd European Conference on IR Research.","author":"Faggioli G.","unstructured":"G. Faggioli and N. Ferro . 2021. System effect estimation by sharding: A comparison between ANOVA approaches to detect significant differences . In Proceedings of the 43rd European Conference on IR Research. G. Faggioli and N. Ferro. 2021. System effect estimation by sharding: A comparison between ANOVA approaches to detect significant differences. In Proceedings of the 43rd European Conference on IR Research."},{"key":"e_1_2_1_28_1","volume-title":"Proceedings of the 43rd European Conference on IR Research.","author":"Faggioli G.","unstructured":"G. Faggioli , O. Zendel , J. S. Culpepper , N. Ferro , and Scholer F . 2021. An enhanced evaluation framework for query performance prediction . In Proceedings of the 43rd European Conference on IR Research. G. Faggioli, O. Zendel, J. S. Culpepper, N. Ferro, and Scholer F.2021. An enhanced evaluation framework for query performance prediction. In Proceedings of the 43rd European Conference on IR Research."},{"key":"e_1_2_1_29_1","volume-title":"Proceedings of the 26th Text REtrieval Conference. National Institute of Standards and Technology, 114 (See Voorhees and Ellis [76]).","author":"Ferro N.","year":"2018","unstructured":"N. Ferro . 2018 . IMS @ TREC 2017 Core Track . In Proceedings of the 26th Text REtrieval Conference. National Institute of Standards and Technology, 114 (See Voorhees and Ellis [76]). N. Ferro. 2018. IMS @ TREC 2017 Core Track. In Proceedings of the 26th Text REtrieval Conference. National Institute of Standards and Technology, 114 (See Voorhees and Ellis [76])."},{"key":"e_1_2_1_30_1","volume-title":"CLEF 2009: Grid@CLEF pilot track overview. In Multilingual Information Access Evaluation I. Text Retrieval Experiments. CLEF 2009. Lecture Notes in Computer Science. Springer","author":"Ferro N.","unstructured":"N. Ferro and D. Harman . 2010 . CLEF 2009: Grid@CLEF pilot track overview. In Multilingual Information Access Evaluation I. Text Retrieval Experiments. CLEF 2009. Lecture Notes in Computer Science. Springer , Berlin, 552\u2013565. N. Ferro and D. Harman. 2010. CLEF 2009: Grid@CLEF pilot track overview. In Multilingual Information Access Evaluation I. Text Retrieval Experiments. CLEF 2009. Lecture Notes in Computer Science. Springer, Berlin, 552\u2013565."},{"key":"e_1_2_1_31_1","first-page":"59","article-title":"Using collection shards to study retrieval performance effect sizes","volume":"5","author":"Ferro N.","year":"2019","unstructured":"N. Ferro , Y. Kim , and M. Sanderson . 2019 . Using collection shards to study retrieval performance effect sizes . ACM Transactions on Information Systems 5 , 44 (2019), 59 . N. Ferro, Y. Kim, and M. Sanderson. 2019. Using collection shards to study retrieval performance effect sizes. ACM Transactions on Information Systems 5, 44 (2019), 59.","journal-title":"ACM Transactions on Information Systems"},{"key":"e_1_2_1_32_1","volume-title":"Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. 901\u2013904","author":"Ferro N.","unstructured":"N. Ferro and M. Sanderson . 2017. Sub-corpora impact on system effectiveness . In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. 901\u2013904 . N. Ferro and M. Sanderson. 2017. Sub-corpora impact on system effectiveness. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. 901\u2013904."},{"key":"e_1_2_1_33_1","doi-asserted-by":"crossref","unstructured":"N. Ferro and M. Sanderson. 2019. Improving the accuracy of system performance estimation by using shards. In Proceedings of the 42nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. B. Piwowarski M. Chevalier E. Gaussier Y. Maarek J.-Y. Nie and F. Scholer (Eds.) ACM Press New York NY 805\u2013814. N. Ferro and M. Sanderson. 2019. Improving the accuracy of system performance estimation by using shards. In Proceedings of the 42nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. B. Piwowarski M. Chevalier E. Gaussier Y. Maarek J.-Y. Nie and F. Scholer (Eds.) ACM Press New York NY 805\u2013814.","DOI":"10.1145\/3331184.3338062"},{"key":"e_1_2_1_34_1","volume-title":"Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval. 25\u201334","author":"Ferro N.","unstructured":"N. Ferro and G. Silvello . 2016. A general linear mixed models approach to study system component effects . In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval. 25\u201334 . N. Ferro and G. Silvello. 2016. A general linear mixed models approach to study system component effects. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval. 25\u201334."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.23910"},{"key":"e_1_2_1_36_1","volume-title":"Proceedings of the 19th ACM International Conference on Information and Knowledge Management. 979\u2013988","author":"Hauff C.","unstructured":"C. Hauff , D. Kelly , and L. Azzopardi . 2010. A comparison of user and system query performance predictions . In Proceedings of the 19th ACM International Conference on Information and Knowledge Management. 979\u2013988 . C. Hauff, D. Kelly, and L. Azzopardi. 2010. A comparison of user and system query performance predictions. In Proceedings of the 19th ACM International Conference on Information and Knowledge Management. 979\u2013988."},{"key":"e_1_2_1_37_1","volume-title":"Inferring query performance using pre-retrieval predictors","author":"He Ben","unstructured":"Ben He and Iadh Ounis . 2004. Inferring query performance using pre-retrieval predictors . In String Processing and Information Retrieval. Alberto Apostolico and Massimo Melucci (Eds.), Springer , Berlin , 43\u201354. Ben He and Iadh Ounis. 2004. Inferring query performance using pre-retrieval predictors. In String Processing and Information Retrieval. Alberto Apostolico and Massimo Melucci (Eds.), Springer, Berlin, 43\u201354."},{"key":"e_1_2_1_38_1","volume-title":"Handbook of Statistics \u2013 Analysis of Variance","author":"Ito P. K.","unstructured":"P. K. Ito . 1980. Robustness of ANOVA and MANOVA test procedures . In Handbook of Statistics \u2013 Analysis of Variance , P. R. Krishnaiah (Ed.), Vol. 1 . Elsevier , The Netherlands , 199\u2013236. P. K. Ito. 1980. Robustness of ANOVA and MANOVA test procedures. In Handbook of Statistics \u2013 Analysis of Variance, P. R. Krishnaiah (Ed.), Vol. 1. Elsevier, The Netherlands, 199\u2013236."},{"key":"e_1_2_1_39_1","volume-title":"Proceedings of the 23rd International Conference on Information and Knowledge Management, X. Li, X. S. Wang, M. Garofalakis, I. Soboroff, T. Suel, and M. Wang (Eds.), ACM Press","author":"Jones T.","year":"1843","unstructured":"T. Jones , A. Turpin , S. Mizzaro , F. Scholer , and M. Sanderson . 2014. Size and source matter: Understanding inconsistencies in test collection-based evaluation . In Proceedings of the 23rd International Conference on Information and Knowledge Management, X. Li, X. S. Wang, M. Garofalakis, I. Soboroff, T. Suel, and M. Wang (Eds.), ACM Press , New York, NY , 1843 \u20131846. T. Jones, A. Turpin, S. Mizzaro, F. Scholer, and M. Sanderson. 2014. Size and source matter: Understanding inconsistencies in test collection-based evaluation. In Proceedings of the 23rd International Conference on Information and Knowledge Management, X. Li, X. S. Wang, M. Garofalakis, I. Soboroff, T. Suel, and M. Wang (Eds.), ACM Press, New York, NY, 1843\u20131846."},{"key":"e_1_2_1_40_1","volume-title":"Rank Correlation Methods","author":"Kendall M. G.","unstructured":"M. G. Kendall . 1948. Rank Correlation Methods . Griffin , Oxford, England . M. G. Kendall. 1948. Rank Correlation Methods. Griffin, Oxford, England."},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1214\/aoms\/1177729694"},{"key":"e_1_2_1_42_1","unstructured":"M. H. Kutner C. J. Nachtsheim J. Neter and W. Li. 2005. Applied Linear Statistical Models (5th ed.). McGraw-Hill\/Irwin New York. M. H. Kutner C. J. Nachtsheim J. Neter and W. Li. 2005. Applied Linear Statistical Models (5th ed.). McGraw-Hill\/Irwin New York."},{"key":"e_1_2_1_43_1","volume-title":"TREC 2004 robust track experiments using PIRCS. In Proceedings of the Text REtrieval Conference. National Institute of Standards and Technology.","author":"Kwok Kui-Lam","year":"2004","unstructured":"Kui-Lam Kwok , Laszlo Grunfeld , HL Sun , Peter Deng , and N Dinstl . 2004 . TREC 2004 robust track experiments using PIRCS. In Proceedings of the Text REtrieval Conference. National Institute of Standards and Technology. Kui-Lam Kwok, Laszlo Grunfeld, HL Sun, Peter Deng, and N Dinstl. 2004. TREC 2004 robust track experiments using PIRCS. In Proceedings of the Text REtrieval Conference. National Institute of Standards and Technology."},{"key":"e_1_2_1_44_1","volume-title":"Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 120\u2013127","author":"Lavrenko V.","unstructured":"V. Lavrenko and W. B. Croft . 2001. Relevance-based language models . In Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 120\u2013127 . V. Lavrenko and W. B. Croft. 2001. Relevance-based language models. In Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 120\u2013127."},{"key":"e_1_2_1_45_1","volume-title":"Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval. 47\u201350","author":"Liu B.","unstructured":"B. Liu , N. Craswell , O. Kurland , and J. S. Culpepper . 2019. A comparative analysis of human and automatic query variants . In Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval. 47\u201350 . B. Liu, N. Craswell, O. Kurland, and J. S. Culpepper. 2019. A comparative analysis of human and automatic query variants. In Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval. 47\u201350."},{"key":"e_1_2_1_46_1","volume-title":"Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval,27\u201334","author":"Lu Xiaolu","unstructured":"Xiaolu Lu , Oren Kurland , J. Shane Culpepper , Nick Craswell , and Ofri Rom . [2019]. Relevance modeling with multiple query variations . In Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval,27\u201334 . Xiaolu Lu, Oren Kurland, J. Shane Culpepper, Nick Craswell, and Ofri Rom. [2019]. Relevance modeling with multiple query variations. In Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval,27\u201334."},{"key":"e_1_2_1_47_1","doi-asserted-by":"crossref","unstructured":"S. Maxwell and H. D. Delaney. 2004. Designing Experiments and Analyzing Data. A Model Comparison Perspective (2nd ed.). Lawrence Erlbaum Associates Mahwah (NJ). S. Maxwell and H. D. Delaney. 2004. Designing Experiments and Analyzing Data. A Model Comparison Perspective (2nd ed.). Lawrence Erlbaum Associates Mahwah (NJ).","DOI":"10.4324\/9781410609243"},{"key":"e_1_2_1_48_1","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1080\/00031305.1978.10479236","article-title":"Variations of box plots","volume":"32","author":"McGill R.","year":"1978","unstructured":"R. McGill , J. W. Tukey , and W. A. Larsen . 1978 . Variations of box plots . The American Statistician 32 , 1 (Feb. 1978), 12\u201316. R. McGill, J. W. Tukey, and W. A. Larsen. 1978. Variations of box plots. The American Statistician 32, 1 (Feb. 1978), 12\u201316.","journal-title":"The American Statistician"},{"key":"e_1_2_1_49_1","unstructured":"W. Mendenhall and T. Sincich. 2012. A Second Course in Statistics. Regression Analysis (7th ed.). Prentice Hall. W. Mendenhall and T. Sincich. 2012. A Second Course in Statistics. Regression Analysis (7th ed.). Prentice Hall."},{"key":"e_1_2_1_50_1","volume-title":"The good, the bad, the difficult, and the easy: Something wrong with information retrieval evaluation? In Advances in Information Retrieval","author":"Mizzaro Stefano","unstructured":"Stefano Mizzaro . 2008. The good, the bad, the difficult, and the easy: Something wrong with information retrieval evaluation? In Advances in Information Retrieval . Craig Macdonald, Iadh Ounis, Vassilis Plachouras, Ian Ruthven, and Ryen W. White (Eds.), Springer , Berlin , 642\u2013646. Stefano Mizzaro. 2008. The good, the bad, the difficult, and the easy: Something wrong with information retrieval evaluation? In Advances in Information Retrieval. Craig Macdonald, Iadh Ounis, Vassilis Plachouras, Ian Ruthven, and Ryen W. White (Eds.), Springer, Berlin, 642\u2013646."},{"key":"e_1_2_1_51_1","volume-title":"Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 479\u2013486","author":"Mizzaro S.","unstructured":"S. Mizzaro and S. Robertson . 2007. HITS hits TREC: Exploring IR evaluation results with network analysis . In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 479\u2013486 . S. Mizzaro and S. Robertson. 2007. HITS hits TREC: Exploring IR evaluation results with network analysis. In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 479\u2013486."},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1037\/1082-989X.8.4.434"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-009-9125-9"},{"key":"e_1_2_1_54_1","volume-title":"Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 843\u2013844","author":"Peng J.","unstructured":"J. Peng , C. Macdonald , B. He , V. Plachouras , and I. Ounis . 2007. Incorporating term dependency in the DFR framework . In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 843\u2013844 . J. Peng, C. Macdonald, B. He, V. Plachouras, and I. Ounis. 2007. Incorporating term dependency in the DFR framework. In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 843\u2013844."},{"key":"e_1_2_1_55_1","volume-title":"Proceedings of the KDD 2020 Workshop on Conversational Systems Towards Mainstream Adoption Co-Located with the 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2666","author":"Penha G.","unstructured":"G. Penha and C. Hauff . 2020. Challenges in the evaluation of conversational search systems . Proceedings of the KDD 2020 Workshop on Conversational Systems Towards Mainstream Adoption Co-Located with the 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2666 . G. Penha and C. Hauff. 2020. Challenges in the evaluation of conversational search systems. Proceedings of the KDD 2020 Workshop on Conversational Systems Towards Mainstream Adoption Co-Located with the 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2666."},{"key":"e_1_2_1_56_1","volume-title":"Proceedings of the String Processing and Information Retrieval. E. Chavez and S. Lonardi (Eds.),Springer","author":"P\u00e9rez-Iglesias J.","unstructured":"J. P\u00e9rez-Iglesias and L. Araujo . 2010. Standard deviation as a query hardness estimator . In Proceedings of the String Processing and Information Retrieval. E. Chavez and S. Lonardi (Eds.),Springer , Berlin, 207\u2013212. J. P\u00e9rez-Iglesias and L. Araujo. 2010. Standard deviation as a query hardness estimator. In Proceedings of the String Processing and Information Retrieval. E. Chavez and S. Lonardi (Eds.),Springer, Berlin, 207\u2013212."},{"key":"e_1_2_1_57_1","volume-title":"Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 275\u2013281","author":"Ponte J.","unstructured":"J. Ponte and W. B. Croft . 1998. A language modeling approach to information retrieval . In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 275\u2013281 . J. Ponte and W. B. Croft. 1998. A language modeling approach to information retrieval. In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 275\u2013281."},{"key":"e_1_2_1_58_1","volume-title":"Proceedings of the 37th International ACM SIGIR Conference on Research and Development in Information Retrieval","author":"Raiber Fiana","unstructured":"Fiana Raiber and Oren Kurland . 2014. Query-performance prediction: Setting the expectations straight . In Proceedings of the 37th International ACM SIGIR Conference on Research and Development in Information Retrieval . Shlomo Geva, Andrew Trotman, Peter Bruza, Charles L. A. Clarke, and Kalervo J\u00e4rvelin (Eds.), ACM Press , New York, NY , 13\u201322. Fiana Raiber and Oren Kurland. 2014. Query-performance prediction: Setting the expectations straight. In Proceedings of the 37th International ACM SIGIR Conference on Research and Development in Information Retrieval. Shlomo Geva, Andrew Trotman, Peter Bruza, Charles L. A. Clarke, and Kalervo J\u00e4rvelin (Eds.), ACM Press, NewYork, NY, 13\u201322."},{"key":"e_1_2_1_59_1","volume-title":"Proceedings of the 35th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval.W. Hersh, J. Callan, Y. Maarek, and M. Sanderson (Eds.), ACM Press","author":"Robertson S. E.","unstructured":"S. E. Robertson and E. Kanoulas . 2012. On Per-topic Variance in IR Evaluation . In Proceedings of the 35th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval.W. Hersh, J. Callan, Y. Maarek, and M. Sanderson (Eds.), ACM Press , New York, NY, 891\u2013900. S. E. Robertson and E. Kanoulas. 2012. On Per-topic Variance in IR Evaluation. In Proceedings of the 35th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval.W. Hersh, J. Callan, Y. Maarek, and M. Sanderson (Eds.), ACM Press, New York, NY, 891\u2013900."},{"key":"e_1_2_1_60_1","volume-title":"The SMART Retrieval System: Experiments in Automatic Document Processing","author":"Rocchio J. J.","unstructured":"J. J. Rocchio . 1971. The SMART Retrieval System: Experiments in Automatic Document Processing . Prentice Hall , Englewood Cliffs, NJ . J. J. Rocchio. 1971. The SMART Retrieval System: Experiments in Automatic Document Processing. Prentice Hall, Englewood Cliffs, NJ."},{"key":"e_1_2_1_61_1","volume-title":"Do easy topics predict effectiveness better than difficult topics? In Advances in Information Retrieval","author":"Roitero Kevin","unstructured":"Kevin Roitero , Eddy Maddalena , and Stefano Mizzaro . 2017. Do easy topics predict effectiveness better than difficult topics? In Advances in Information Retrieval . Joemon M. Jose, Claudia Hauff, Ismail Sengor Alt\u0131ngovde, Dawei Song, Dyaa Albakour, Stuart Watt, and John Tait (Eds.), Springer , Cham , 605\u2013611. Kevin Roitero, Eddy Maddalena, and Stefano Mizzaro. 2017. Do easy topics predict effectiveness better than difficult topics? In Advances in Information Retrieval. Joemon M. Jose, Claudia Hauff, Ismail Sengor Alt\u0131ngovde, Dawei Song, Dyaa Albakour, Stuart Watt, and John Tait (Eds.), Springer, Cham, 605\u2013611."},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1145\/3077136.3080665"},{"key":"e_1_2_1_63_1","volume-title":"A GLM Approach","author":"Rutherford A.","unstructured":"A. Rutherford . 2011. ANOVA and ANCOVA. A GLM Approach ( 2 nd ed.). John Wiley & Sons , New York, NY . A. Rutherford. 2011. ANOVA and ANCOVA. A GLM Approach (2nd ed.). John Wiley & Sons, New York, NY.","edition":"2"},{"key":"e_1_2_1_64_1","volume-title":"Bridging Between Information Retrieval and Databases - PROMISE Winter School","author":"Sakai T.","year":"2013","unstructured":"T. Sakai . 2014. Metrics , statistics, tests . In Bridging Between Information Retrieval and Databases - PROMISE Winter School 2013 , Revised Tutorial Lectures. N. Ferro (Ed.), Springer , 116\u2013163. T. Sakai. 2014. Metrics, statistics, tests. In Bridging Between Information Retrieval and Databases - PROMISE Winter School 2013, Revised Tutorial Lectures. N. Ferro (Ed.), Springer, 116\u2013163."},{"key":"e_1_2_1_65_1","volume-title":"Proceedings of the 21st ACM International Conference on Information and Knowledge Management. 1965\u20131969","author":"Sanderson M.","unstructured":"M. Sanderson , A. Turpin , Y. Zhang , and F. Scholer . 2012. Differences in effectiveness across sub-collections . In Proceedings of the 21st ACM International Conference on Information and Knowledge Management. 1965\u20131969 . M. Sanderson, A. Turpin, Y. Zhang, and F. Scholer. 2012. Differences in effectiveness across sub-collections. In Proceedings of the 21st ACM International Conference on Information and Knowledge Management. 1965\u20131969."},{"key":"e_1_2_1_66_1","doi-asserted-by":"crossref","first-page":"123","DOI":"10.1080\/00031305.1987.10475459","article-title":"The effects of violations of independence assumptions in the one-way ANOVA","volume":"41","author":"Scariano S. M.","year":"1987","unstructured":"S. M. Scariano and J. M. Davenport . 1987 . The effects of violations of independence assumptions in the one-way ANOVA . The American Statistician 41 , 2 (1987), 123 \u2013 129 . S. M. Scariano and J. M. Davenport. 1987. The effects of violations of independence assumptions in the one-way ANOVA. The American Statistician 41, 2 (1987), 123\u2013129.","journal-title":"The American Statistician"},{"key":"e_1_2_1_67_1","volume-title":"Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval. 623\u2013632","author":"Scholer F.","unstructured":"F. Scholer , D. Kelly , W.-C. Wu , H. S. Lee , and W. Webber . 2013. The effect of threshold priming and need for cognition on relevance calibration and assessment . In Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval. 623\u2013632 . F. Scholer, D. Kelly, W.-C. Wu, H. S. Lee, and W. Webber. 2013. The effect of threshold priming and need for cognition on relevance calibration and assessment. In Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval. 623\u2013632."},{"key":"e_1_2_1_68_1","doi-asserted-by":"crossref","unstructured":"R. J. Shavelson and N. M. Webb. 1991. Generalizability Theory. A Primer. SAGE Publishing. R. J. Shavelson and N. M. Webb. 1991. Generalizability Theory. A Primer. SAGE Publishing.","DOI":"10.1037\/10109-051"},{"key":"e_1_2_1_69_1","volume-title":"Proceedings of the 4th ACM International Conference on Web Search and Data Mining. 795\u2013804","author":"Sheldon D.","unstructured":"D. Sheldon , M. Shokouhi , M. Szummer , and N. Craswell . 2011. LambdaMerge: Merging the results of query reformulations . In Proceedings of the 4th ACM International Conference on Web Search and Data Mining. 795\u2013804 . D. Sheldon, M. Shokouhi, M. Szummer, and N. Craswell. 2011. LambdaMerge: Merging the results of query reformulations. In Proceedings of the 4th ACM International Conference on Web Search and Data Mining. 795\u2013804."},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/2926790"},{"key":"e_1_2_1_71_1","doi-asserted-by":"publisher","DOI":"10.1145\/2180868.2180873"},{"key":"e_1_2_1_72_1","volume-title":"Proceedings of the Text REtrieval Conference. 385\u2013398","author":"Tague-Sutcliffe J. M.","unstructured":"J. M. Tague-Sutcliffe and J. Blustein . 1994. A Statistical analysis of the TREC-3 data . In Proceedings of the Text REtrieval Conference. 385\u2013398 . J. M. Tague-Sutcliffe and J. Blustein. 1994. A Statistical analysis of the TREC-3 data. In Proceedings of the Text REtrieval Conference. 385\u2013398."},{"key":"e_1_2_1_73_1","volume-title":"Overview of the TREC 2004 robust retrieval track. In Proceedings of the Text REtrieval Conference.","author":"Voorhees E. M.","year":"2004","unstructured":"E. M. Voorhees . 2004 . Overview of the TREC 2004 robust retrieval track. In Proceedings of the Text REtrieval Conference. E. M. Voorhees. 2004. Overview of the TREC 2004 robust retrieval track. In Proceedings of the Text REtrieval Conference."},{"key":"e_1_2_1_74_1","volume-title":"Overview of the TREC 2005 robust retrieval track. In Proceedings of the Text REtrieval Conference.","author":"Voorhees E. M.","year":"2005","unstructured":"E. M. Voorhees . 2005 . Overview of the TREC 2005 robust retrieval track. In Proceedings of the Text REtrieval Conference. E. M. Voorhees. 2005. Overview of the TREC 2005 robust retrieval track. In Proceedings of the Text REtrieval Conference."},{"key":"e_1_2_1_75_1","volume-title":"Proceedings of the 27th International Conference on Information and Knowledge Management","author":"Voorhees E. M.","unstructured":"E. M. Voorhees . 2018. On building fair and reusable test collections using bandit techniques . In Proceedings of the 27th International Conference on Information and Knowledge Management , A. Cuzzocrea, J. Allan, N. W. Paton, D. Srivastava, R. Agrawal, A. Broder, M. J. Zaki, S. Candan, A. Labrinidis, A. Schuster, and H. Wang (Eds.), ACM Press , New York, NY , 407\u2013416. E. M. Voorhees. 2018. On building fair and reusable test collections using bandit techniques. In Proceedings of the 27th International Conference on Information and Knowledge Management, A. Cuzzocrea, J. Allan, N. W. Paton, D. Srivastava, R. Agrawal, A. Broder, M. J. Zaki, S. Candan, A. Labrinidis, A. Schuster, and H. Wang (Eds.), ACM Press, New York, NY, 407\u2013416."},{"key":"e_1_2_1_76_1","volume-title":"The Twenty-Sixth Text REtrieval Conference Proceedings (TREC","author":"Voorhees E. M.","year":"2018","unstructured":"E. M. Voorhees and A. Ellis ( Eds .). 2018 . The Twenty-Sixth Text REtrieval Conference Proceedings (TREC 2017). National Institute of Standards and Technology (NIST) Gaithersburg, MD. E. M. Voorhees and A. Ellis (Eds.). 2018. The Twenty-Sixth Text REtrieval Conference Proceedings (TREC 2017). National Institute of Standards and Technology (NIST) Gaithersburg, MD."},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.1145\/3086701"},{"key":"e_1_2_1_78_1","doi-asserted-by":"crossref","unstructured":"M. P. Wand and M. C. Jones. 1995. Kernel Smoothing. Chapman and Hall\/CRC. M. P. Wand and M. C. Jones. 1995. Kernel Smoothing. Chapman and Hall\/CRC.","DOI":"10.1007\/978-1-4899-4493-1"},{"key":"e_1_2_1_79_1","volume-title":"Proceedings of the 19th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 4\u201311","author":"Xu J.","unstructured":"J. Xu and W. B. Croft . 1996. Query expansion using local and global document analysis . In Proceedings of the 19th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 4\u201311 . J. Xu and W. B. Croft. 1996. Query expansion using local and global document analysis. In Proceedings of the 19th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 4\u201311."},{"key":"e_1_2_1_80_1","volume-title":"Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 587\u2013594","author":"Yilmaz E.","unstructured":"E. Yilmaz , J. A. Aslam , and S. E. Robertson . 2008. A New rank correlation coefficient for information retrieval . In Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 587\u2013594 . E. Yilmaz, J. A. Aslam, and S. E. Robertson. 2008. A New rank correlation coefficient for information retrieval. In Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 587\u2013594."},{"key":"e_1_2_1_81_1","volume-title":"Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 512\u2013519","author":"Yom-Tov E.","unstructured":"E. Yom-Tov , S. Fine , D. Carmel , and A. Darlow . 2005. Learning to estimate query difficulty: Including applications to missing content detection and distributed information retrieval . In Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 512\u2013519 . E. Yom-Tov, S. Fine, D. Carmel, and A. Darlow. 2005. Learning to estimate query difficulty: Including applications to missing content detection and distributed information retrieval. In Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 512\u2013519."},{"key":"e_1_2_1_82_1","volume-title":"Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. 909\u2013912","author":"Zampieri F.","unstructured":"F. Zampieri , K. Roitero , J. S. Culpepper , O. Kurland , and S. Mizzaro . 2019. On topic difficulty in IR evaluation: The effect of systems, corpora, and system components . In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. 909\u2013912 . F. Zampieri, K. Roitero, J. S. Culpepper, O. Kurland, and S. Mizzaro. 2019. On topic difficulty in IR evaluation: The effect of systems, corpora, and system components. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. 909\u2013912."},{"key":"e_1_2_1_83_1","volume-title":"Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. 395\u2013404","author":"Zendel O.","unstructured":"O. Zendel , A. Shtok , F. Raiber , O. Kurland , and J. S. Culpepper . 2019. Information needs, queries, and query performance prediction . In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. 395\u2013404 . O. Zendel, A. Shtok, F. Raiber, O. Kurland, and J. S. Culpepper. 2019. Information needs, queries, and query performance prediction. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. 395\u2013404."}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3470563","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3470563","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:18:54Z","timestamp":1750191534000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3470563"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,9,8]]},"references-count":83,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2022,1,31]]}},"alternative-id":["10.1145\/3470563"],"URL":"https:\/\/doi.org\/10.1145\/3470563","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"value":"1046-8188","type":"print"},{"value":"1558-2868","type":"electronic"}],"subject":[],"published":{"date-parts":[[2021,9,8]]},"assertion":[{"value":"2020-08-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-06-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-09-08","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}