{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,25]],"date-time":"2026-03-25T14:38:52Z","timestamp":1774449532513,"version":"3.50.1"},"reference-count":82,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2017,6,5]],"date-time":"2017-06-05T00:00:00Z","timestamp":1496620800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"Australian Research Council's Discovery Projects Scheme","award":["DP140102655"],"award-info":[{"award-number":["DP140102655"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2017,7,31]]},"abstract":"<jats:p>\n            Information retrieval systems aim to help users satisfy information needs. We argue that the goal of the person using the system, and the pattern of behavior that they exhibit as they proceed to attain that goal, should be incorporated into the methods and techniques used to evaluate the effectiveness of IR systems, so that the resulting effectiveness scores have a useful interpretation that corresponds to the users\u2019 search experience. In particular, we investigate the role of search task complexity, and show that it has a direct bearing on the number of relevant answer documents sought by users in response to an information need, suggesting that useful effectiveness metrics must be\n            <jats:italic>goal sensitive<\/jats:italic>\n            . We further suggest that user behavior while scanning results listings is affected by the rate at which their goal is being realized, and hence that appropriate effectiveness metrics must be\n            <jats:italic>adaptive<\/jats:italic>\n            to the presence (or not) of relevant documents in the ranking. In response to these two observations, we present a new effectiveness metric, INST, that has both of the desired properties: INST employs a parameter\n            <jats:italic>T<\/jats:italic>\n            , a direct measure of the user\u2019s search goal that adjusts the top-weightedness of the evaluation score; moreover, as progress towards the target\n            <jats:italic>T<\/jats:italic>\n            is made, the modeled user behavior is adapted, to reflect the remaining expectations. INST is experimentally compared to previous effectiveness metrics, including Average Precision (AP), Normalized Discounted Cumulative Gain (NDCG), and Rank-Biased Precision (RBP), demonstrating our claims as to INST\u2019s usefulness. Like RBP, INST is a weighted-precision metric, meaning that each score can be accompanied by a\n            <jats:italic>residual<\/jats:italic>\n            that quantifies the extent of the score uncertainty caused by unjudged documents. As part of our experimentation, we use crowd-sourced data and score residuals to demonstrate that a wide range of queries arise for even quite specific information needs, and that these variant queries introduce significant levels of residual uncertainty into typical experimental evaluations. These causes of variability have wide-reaching implications for experiment design, and for the construction of test collections.\n          <\/jats:p>","DOI":"10.1145\/3052768","type":"journal-article","created":{"date-parts":[[2017,6,7]],"date-time":"2017-06-07T12:47:23Z","timestamp":1496839643000},"page":"1-38","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":77,"title":["Incorporating User Expectations and Behavior into the Measurement of Search Effectiveness"],"prefix":"10.1145","volume":"35","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6638-0232","authenticated-orcid":false,"given":"Alistair","family":"Moffat","sequence":"first","affiliation":[{"name":"The University of Melbourne, Victoria, Australia"}]},{"given":"Peter","family":"Bailey","sequence":"additional","affiliation":[{"name":"Microsoft, Barton, ACT, Australia"}]},{"given":"Falk","family":"Scholer","sequence":"additional","affiliation":[{"name":"RMIT University, Victoria, Australia"}]},{"given":"Paul","family":"Thomas","sequence":"additional","affiliation":[{"name":"Microsoft, Barton, ACT, Australia"}]}],"member":"320","published-online":{"date-parts":[[2017,6,5]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.10217"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/1480506.1480508"},{"key":"e_1_2_1_3_1","unstructured":"L. W. Anderson and D. A. Krathwohl. 2001. A Taxonomy for Learning Teaching and Assessing: A Revision of Bloom\u2019s Taxonomy of Educational Objectives. Longman.  L. W. Anderson and D. A. Krathwohl. 2001. A Taxonomy for Learning Teaching and Assessing: A Revision of Bloom\u2019s Taxonomy of Educational Objectives. Longman."},{"key":"e_1_2_1_4_1","volume-title":"Proc. ACM Conf. on Research and Development in Information Retrieval (SIGIR\u201913)","author":"Azzopardi L.","unstructured":"L. Azzopardi , D. Kelly , and K. Brennan . 2013. How query cost affects search behavior . In Proc. ACM Conf. on Research and Development in Information Retrieval (SIGIR\u201913) . 23--32. L. Azzopardi, D. Kelly, and K. Brennan. 2013. How query cost affects search behavior. In Proc. ACM Conf. on Research and Development in Information Retrieval (SIGIR\u201913). 23--32."},{"key":"e_1_2_1_5_1","volume-title":"Proc. ACM Conf. on Research and Development in Information Retrieval (SIGIR\u201908)","author":"Bailey P.","unstructured":"P. Bailey , N. Craswell , I. Soboroff , P. Thomas , A. P. de Vries , and E. Yilmaz . 2008. Relevance assessment: Are judges exchangeable and does it matter? In Proc. ACM Conf. on Research and Development in Information Retrieval (SIGIR\u201908) . 667--674. P. Bailey, N. Craswell, I. Soboroff, P. Thomas, A. P. de Vries, and E. Yilmaz. 2008. Relevance assessment: Are judges exchangeable and does it matter? In Proc. ACM Conf. on Research and Development in Information Retrieval (SIGIR\u201908). 667--674."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835449.1835606"},{"key":"e_1_2_1_7_1","volume-title":"Proc. ACM Conf. on Research and Development in Information Retrieval (SIGIR\u201915)","author":"Bailey P.","unstructured":"P. Bailey , A. Moffat , F. Scholer , and P. Thomas . 2015. User variability and IR system evaluation . In Proc. ACM Conf. on Research and Development in Information Retrieval (SIGIR\u201915) . 625--634. P. Bailey, A. Moffat, F. Scholer, and P. Thomas. 2015. User variability and IR system evaluation. In Proc. ACM Conf. on Research and Development in Information Retrieval (SIGIR\u201915). 625--634."},{"key":"e_1_2_1_8_1","volume-title":"Proc. ACM Conf. on Research and Development in Information Retrieval (SIGIR\u201916)","author":"Bailey P.","unstructured":"P. Bailey , A. Moffat , F. Scholer , and P. Thomas . 2016. UQV100: A test collection with query variability . In Proc. ACM Conf. on Research and Development in Information Retrieval (SIGIR\u201916) . 725--728. Public data: http:\/\/dx.doi.org\/10.4225\/49\/5726E597B8376. 10.4225\/49 P. Bailey, A. Moffat, F. Scholer, and P. Thomas. 2016. UQV100: A test collection with query variability. In Proc. ACM Conf. on Research and Development in Information Retrieval (SIGIR\u201916). 725--728. Public data: http:\/\/dx.doi.org\/10.4225\/49\/5726E597B8376."},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1009984519381"},{"key":"e_1_2_1_10_1","volume-title":"Proc. ACM International Conf. on Information and Knowledge Management (CIKM\u201913)","author":"Baskaya F.","unstructured":"F. Baskaya , H. Keskustalo , and K. J\u00e4rvelin . 2013. Modeling behavioral factors in interactive information retrieval . In Proc. ACM International Conf. on Information and Knowledge Management (CIKM\u201913) . 2297--2302. F. Baskaya, H. Keskustalo, and K. J\u00e4rvelin. 2013. Modeling behavioral factors in interactive information retrieval. In Proc. ACM International Conf. on Information and Knowledge Management (CIKM\u201913). 2297--2302."},{"key":"e_1_2_1_11_1","volume-title":"Proc. ACM Conf. on Research and Development in Information Retrieval (SIGIR\u201993)","author":"Belkin N. J.","unstructured":"N. J. Belkin , C. Cool , W. B. Croft , and J. P. Callan . 1993. Effect of multiple query representations on information retrieval system performance . In Proc. ACM Conf. on Research and Development in Information Retrieval (SIGIR\u201993) . 339--346. N. J. Belkin, C. Cool, W. B. Croft, and J. P. Callan. 1993. Effect of multiple query representations on information retrieval system performance. In Proc. ACM Conf. on Research and Development in Information Retrieval (SIGIR\u201993). 339--346."},{"key":"e_1_2_1_12_1","doi-asserted-by":"crossref","unstructured":"N. J. Belkin P. Kantor E. A. Fox and J. A. Shaw. 1995. Combining the evidence of multiple query representations for information retrieval. Information Processing 8 Management 31 3 (1995) 431--448.  N. J. Belkin P. Kantor E. A. Fox and J. A. Shaw. 1995. Combining the evidence of multiple query representations for information retrieval. Information Processing 8 Management 31 3 (1995) 431--448.","DOI":"10.1016\/0306-4573(94)00057-A"},{"key":"e_1_2_1_13_1","volume-title":"Proc. European Conf. in Information Retrieval (ECIR\u201904)","author":"Bell D. J.","unstructured":"D. J. Bell and I. Ruthven . 2004. Searchers\u2019 assessments of task complexity for web searching . In Proc. European Conf. in Information Retrieval (ECIR\u201904) . 57--71. D. J. Bell and I. Ruthven. 2004. Searchers\u2019 assessments of task complexity for web searching. In Proc. European Conf. in Information Retrieval (ECIR\u201904). 57--71."},{"key":"e_1_2_1_14_1","volume-title":"The IIR evaluation model: A framework for evaluation of interactive information retrieval systems. Information Research 8, 3","author":"Borlund P.","year":"2003","unstructured":"P. Borlund . 2003. The IIR evaluation model: A framework for evaluation of interactive information retrieval systems. Information Research 8, 3 ( 2003 ). P. Borlund. 2003. The IIR evaluation model: A framework for evaluation of interactive information retrieval systems. Information Research 8, 3 (2003)."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-007-9032-x"},{"key":"e_1_2_1_16_1","volume-title":"Proc. Text Retrieval Conf. (TREC\u201999)","author":"Buckley C.","unstructured":"C. Buckley and J. Walz . 1999. The TREC-8 query track . In Proc. Text Retrieval Conf. (TREC\u201999) . NIST Special Publication 500-246. C. Buckley and J. Walz. 1999. The TREC-8 query track. In Proc. Text Retrieval Conf. (TREC\u201999). NIST Special Publication 500-246."},{"key":"e_1_2_1_17_1","volume-title":"Proc. ACM Conf. on Research and Development in Information Retrieval (SIGIR\u201907)","author":"B\u00fcttcher S.","unstructured":"S. B\u00fcttcher , C. L. A. Clarke , P. C. K. Yeung , and I. Soboroff . 2007. Reliable information retrieval evaluation with incomplete and biased judgements . In Proc. ACM Conf. on Research and Development in Information Retrieval (SIGIR\u201907) . 63--70. S. B\u00fcttcher, C. L. A. Clarke, P. C. K. Yeung, and I. Soboroff. 2007. Reliable information retrieval evaluation with incomplete and biased judgements. In Proc. ACM Conf. on Research and Development in Information Retrieval (SIGIR\u201907). 63--70."},{"key":"e_1_2_1_18_1","doi-asserted-by":"crossref","unstructured":"K. Bystr\u00f6m and K. J\u00e4rvelin. 1995. Task complexity affects information seeking and use. Information Processing 8 Management 31 2 (1995) 191--213.  K. Bystr\u00f6m and K. J\u00e4rvelin. 1995. Task complexity affects information seeking and use. Information Processing 8 Management 31 2 (1995) 191--213.","DOI":"10.1016\/0306-4573(95)80035-R"},{"key":"e_1_2_1_19_1","volume-title":"Proc. ACM International Conf. on Information and Knowledge Management (CIKM). 135--144","author":"Carterette B.","unstructured":"B. Carterette , E. Kanoulas , and E. Yilmaz . 2012. Incorporating variability in user behavior into systems based evaluation . In Proc. ACM International Conf. on Information and Knowledge Management (CIKM). 135--144 . B. Carterette, E. Kanoulas, and E. Yilmaz. 2012. Incorporating variability in user behavior into systems based evaluation. In Proc. ACM International Conf. on Information and Knowledge Management (CIKM). 135--144."},{"key":"e_1_2_1_20_1","volume-title":"Proc. ACM International Conf. on Information and Knowledge Management (CIKM\u201909)","author":"Chapelle O.","unstructured":"O. Chapelle , D. Metzler , Y. Zhang , and P. Grinspan . 2009. Expected reciprocal rank for graded relevance . In Proc. ACM International Conf. on Information and Knowledge Management (CIKM\u201909) . 621--630. O. Chapelle, D. Metzler, Y. Zhang, and P. Grinspan. 2009. Expected reciprocal rank for graded relevance. In Proc. ACM International Conf. on Information and Knowledge Management (CIKM\u201909). 621--630."},{"key":"e_1_2_1_21_1","doi-asserted-by":"crossref","unstructured":"A. Chuklin I. Markov and M. de Rijke. 2015. Click Models for Web Search. Morgan 8 Claypool.  A. Chuklin I. Markov and M. de Rijke. 2015. Click Models for Web Search. Morgan 8 Claypool.","DOI":"10.1007\/978-3-031-02294-4"},{"key":"e_1_2_1_22_1","volume-title":"Overview of the TREC 2004 terabyte track. In Proc. Text Retrieval Conf. (TREC\u201904)","author":"Clarke C. L. A.","unstructured":"C. L. A. Clarke , N. Craswell , and I. Soboroff . 2004 . Overview of the TREC 2004 terabyte track. In Proc. Text Retrieval Conf. (TREC\u201904) . C. L. A. Clarke, N. Craswell, and I. Soboroff. 2004. Overview of the TREC 2004 terabyte track. In Proc. Text Retrieval Conf. (TREC\u201904)."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.5090190108"},{"key":"e_1_2_1_24_1","volume-title":"Proc. Recherche d\u2019Information Etses Applications (RIAO\u201904)","author":"de Vries A. P.","unstructured":"A. P. de Vries , G. Kazai , and M. Lalmas . 2004. Tolerance to irrelevance: A user-effort evaluation of retrieval systems without predefined retrieval unit . In Proc. Recherche d\u2019Information Etses Applications (RIAO\u201904) . 463--473. A. P. de Vries, G. Kazai, and M. Lalmas. 2004. Tolerance to irrelevance: A user-effort evaluation of retrieval systems without predefined retrieval unit. In Proc. Recherche d\u2019Information Etses Applications (RIAO\u201904). 463--473."},{"key":"e_1_2_1_25_1","volume-title":"Proceedings of the 3rd Symposium on Information Interaction in Context. ACM, 185--194","author":"Dumais S. T.","unstructured":"S. T. Dumais , G. Buscher , and E. Cutrell . 2010. Individual differences in gaze patterns for web search . In Proceedings of the 3rd Symposium on Information Interaction in Context. ACM, 185--194 . S. T. Dumais, G. Buscher, and E. Cutrell. 2010. Individual differences in gaze patterns for web search. In Proceedings of the 3rd Symposium on Information Interaction in Context. ACM, 185--194."},{"key":"e_1_2_1_26_1","volume-title":"Proc. ACM Conf. on Research and Development in Information Retrieval (SIGIR\u201916)","author":"Ferro N.","unstructured":"N. Ferro and G. Silvello . 2016. A general linear mixed models approach to study system component effects . In Proc. ACM Conf. on Research and Development in Information Retrieval (SIGIR\u201916) . 25--34. N. Ferro and G. Silvello. 2016. A general linear mixed models approach to study system component effects. In Proc. ACM Conf. on Research and Development in Information Retrieval (SIGIR\u201916). 25--34."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1037\/h0031619"},{"key":"e_1_2_1_28_1","volume-title":"Proc. International Conf. Asia-Pacific Digital Libraries (ICADL\u201912)","author":"Fujikawa K.","unstructured":"K. Fujikawa , H. Joho , and S. Nakayama . 2012. Constraint can affect human perception, behaviour, and performance of search . In Proc. International Conf. Asia-Pacific Digital Libraries (ICADL\u201912) . 39--48. K. Fujikawa, H. Joho, and S. Nakayama. 2012. Constraint can affect human perception, behaviour, and performance of search. In Proc. International Conf. Asia-Pacific Digital Libraries (ICADL\u201912). 39--48."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1002\/meet.14504301167"},{"key":"e_1_2_1_30_1","volume-title":"TREC: Experiment and Evaluation in Information Retrieval","author":"Harman D. K.","unstructured":"D. K. Harman . 2005. The TREC test collections . In TREC: Experiment and Evaluation in Information Retrieval , E. M. Voorhees and D. K. Harman (Eds.). MIT Press , Chapter 2, 21--52. D. K. Harman. 2005. The TREC test collections. In TREC: Experiment and Evaluation in Information Retrieval, E. M. Voorhees and D. K. Harman (Eds.). MIT Press, Chapter 2, 21--52."},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/582415.582418"},{"key":"e_1_2_1_32_1","volume-title":"Proc. European Conf. in Information Retrieval (ECIR\u201916)","author":"Jiang J.","unstructured":"J. Jiang and J. Allan . 2016. Adaptive effort for search evaluation metrics . In Proc. European Conf. in Information Retrieval (ECIR\u201916) . 187--199. J. Jiang and J. Allan. 2016. Adaptive effort for search evaluation metrics. In Proc. European Conf. in Information Retrieval (ECIR\u201916). 187--199."},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2396761.2396779"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/2808194.2809465"},{"key":"e_1_2_1_35_1","volume-title":"Proc. ACM International Conf. on Information and Knowledge Management (CIKM\u201908)","author":"Kinney K. A.","unstructured":"K. A. Kinney , S. B Huffman , and J. Zhai . 2008. How evaluator domain expertise affects search result relevance judgments . In Proc. ACM International Conf. on Information and Knowledge Management (CIKM\u201908) . 591--598. K. A. Kinney, S. B Huffman, and J. Zhai. 2008. How evaluator domain expertise affects search result relevance judgments. In Proc. ACM International Conf. on Information and Knowledge Management (CIKM\u201908). 591--598."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1207\/s15430421tip4104_2"},{"key":"e_1_2_1_37_1","doi-asserted-by":"crossref","unstructured":"G. Kumaran and J. Allan. 2008. Adapting information retrieval systems to user queries. Information Processing 8 Management 44 6 (2008) 1838--1862.  G. Kumaran and J. Allan. 2008. Adapting information retrieval systems to user queries. Information Processing 8 Management 44 6 (2008) 1838--1862.","DOI":"10.1016\/j.ipm.2007.12.006"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-016-9282-6"},{"key":"e_1_2_1_39_1","volume-title":"Proc. ACM International Conf. on Information and Knowledge Management (CIKM\u201915)","author":"Maxwell D.","unstructured":"D. Maxwell , L. Azzopardi , K. J\u00e4rvelin , and H. Keskustalo . 2015. Searching and stopping: An analysis of stopping rules and strategies . In Proc. ACM International Conf. on Information and Knowledge Management (CIKM\u201915) . 313--322. D. Maxwell, L. Azzopardi, K. J\u00e4rvelin, and H. Keskustalo. 2015. Searching and stopping: An analysis of stopping rules and strategies. In Proc. ACM International Conf. on Information and Knowledge Management (CIKM\u201915). 313--322."},{"key":"e_1_2_1_40_1","volume-title":"Proc. ACM Conf. on Research and Development in Information Retrieval (SIGIR\u201905)","author":"Metzler D.","unstructured":"D. Metzler and W. B. Croft . 2005. A Markov random field model for term dependencies . In Proc. ACM Conf. on Research and Development in Information Retrieval (SIGIR\u201905) . 472--479. D. Metzler and W. B. Croft. 2005. A Markov random field model for term dependencies. In Proc. ACM Conf. on Research and Development in Information Retrieval (SIGIR\u201905). 472--479."},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.2105\/AJPH.2014.302070"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-45068-6_1"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/3015022.3015025"},{"key":"e_1_2_1_44_1","volume-title":"Proc. Australasian Document Computing Symp. (ADCS\u201915)","author":"Moffat A.","unstructured":"A. Moffat , P. Bailey , F. Scholer , and P. Thomas . 2015. INST: An adaptive metric for information retrieval evaluation . In Proc. Australasian Document Computing Symp. (ADCS\u201915) . 5:1--5:4. A. Moffat, P. Bailey, F. Scholer, and P. Thomas. 2015. INST: An adaptive metric for information retrieval evaluation. In Proc. Australasian Document Computing Symp. (ADCS\u201915). 5:1--5:4."},{"key":"e_1_2_1_45_1","volume-title":"Proc. Australasian Document Computing Symp. (ADCS\u201912)","author":"Moffat A.","unstructured":"A. Moffat , F. Scholer , and P. Thomas . 2012. Models and metrics: IR evaluation as a user process . In Proc. Australasian Document Computing Symp. (ADCS\u201912) . 47--54. A. Moffat, F. Scholer, and P. Thomas. 2012. Models and metrics: IR evaluation as a user process. In Proc. Australasian Document Computing Symp. (ADCS\u201912). 47--54."},{"key":"e_1_2_1_46_1","volume-title":"Proc. ACM International Conf. on Information and Knowledge Management (CIKM\u201915)","author":"Moffat A.","unstructured":"A. Moffat , F. Scholer , P. Thomas , and P. Bailey . 2015. Pooled evaluation over query variations: Users are as diverse as systems . In Proc. ACM International Conf. on Information and Knowledge Management (CIKM\u201915) . 1759--1762. A. Moffat, F. Scholer, P. Thomas, and P. Bailey. 2015. Pooled evaluation over query variations: Users are as diverse as systems. In Proc. ACM International Conf. on Information and Knowledge Management (CIKM\u201915). 1759--1762."},{"key":"e_1_2_1_47_1","volume-title":"Proc. ACM International Conf. on Information and Knowledge Management (CIKM\u201913)","author":"Moffat A.","unstructured":"A. Moffat , P. Thomas , and F. Scholer . 2013. Users versus models: What observation tells us about effectiveness metrics . In Proc. ACM International Conf. on Information and Knowledge Management (CIKM\u201913) . 659--668. A. Moffat, P. Thomas, and F. Scholer. 2013. Users versus models: What observation tells us about effectiveness metrics. In Proc. ACM International Conf. on Information and Knowledge Management (CIKM\u201913). 659--668."},{"key":"e_1_2_1_48_1","volume-title":"Proc. ACM Conf. on Research and Development in Information Retrieval (SIGIR\u201907)","author":"Moffat A.","unstructured":"A. Moffat , W. Webber , and J. Zobel . 2007. Strategic system comparisons via targeted relevance judgments . In Proc. ACM Conf. on Research and Development in Information Retrieval (SIGIR\u201907) . 375--382. A. Moffat, W. Webber, and J. Zobel. 2007. Strategic system comparisons via targeted relevance judgments. In Proc. ACM Conf. on Research and Development in Information Retrieval (SIGIR\u201907). 375--382."},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/1416950.1416952"},{"key":"e_1_2_1_50_1","volume-title":"Working Notes of the Conference and Labs of the Evaluation Forum (CLEF\u201915)","author":"Palotti J.","unstructured":"J. Palotti , G. Zuccon , L. Goeuriot , L. Kelly , A. Hanbury , G. J. F. Jones , M. Lupu , and P. Pecina . 2015. CLEF eHealth evaluation lab 2015, task 2: Retrieving information about medical symptoms . In Working Notes of the Conference and Labs of the Evaluation Forum (CLEF\u201915) . J. Palotti, G. Zuccon, L. Goeuriot, L. Kelly, A. Hanbury, G. J. F. Jones, M. Lupu, and P. Pecina. 2015. CLEF eHealth evaluation lab 2015, task 2: Retrieving information about medical symptoms. In Working Notes of the Conference and Labs of the Evaluation Forum (CLEF\u201915)."},{"key":"e_1_2_1_51_1","volume-title":"Proc. ACM Conf. on Research and Development in Information Retrieval (SIGIR\u201912)","author":"Robertson S. E.","unstructured":"S. E. Robertson and E. Kanoulas . 2012. On per-topic variance in IR evaluation . In Proc. ACM Conf. on Research and Development in Information Retrieval (SIGIR\u201912) . 891--900. S. E. Robertson and E. Kanoulas. 2012. On per-topic variance in IR evaluation. In Proc. ACM Conf. on Research and Development in Information Retrieval (SIGIR\u201912). 891--900."},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/1148170.1148261"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2911492"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-015-9273-z"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-008-9059-7"},{"key":"e_1_2_1_56_1","volume-title":"Proc. Conf. Conceptions of Library and Information Science (COLIS\u201996)","author":"Saracevic T.","year":"1996","unstructured":"T. Saracevic . 1996 . Relevance reconsidered . In Proc. Conf. Conceptions of Library and Information Science (COLIS\u201996) . 201--218. T. Saracevic. 1996. Relevance reconsidered. In Proc. Conf. Conceptions of Library and Information Science (COLIS\u201996). 201--218."},{"key":"e_1_2_1_57_1","volume-title":"Proc. Conf. on Web Search and Data Mining (WSDM\u201911)","author":"Sheldon D.","unstructured":"D. Sheldon , M. Shokouhi , M. Szummer , and N. Craswell . 2011. Lambdamerge: Merging the results of query reformulations . In Proc. Conf. on Web Search and Data Mining (WSDM\u201911) . 795--804. D. Sheldon, M. Shokouhi, M. Szummer, and N. Craswell. 2011. Lambdamerge: Merging the results of query reformulations. In Proc. Conf. on Web Search and Data Mining (WSDM\u201911). 795--804."},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.1145\/2391224.2391227"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1145\/2348283.2348300"},{"key":"e_1_2_1_60_1","volume-title":"Technical Report 5428. Computer Laboratory","author":"Sp\u00e4rck Jones K.","year":"1977","unstructured":"K. Sp\u00e4rck Jones and R. G. Bates . 1977 . Report on the Design Study for the \u201cIdeal\u201d Information Retrieval Test Collection. Technical Report 5428. Computer Laboratory , University of Cambridge. British Library Research and Development Report . K. Sp\u00e4rck Jones and R. G. Bates. 1977. Report on the Design Study for the \u201cIdeal\u201d Information Retrieval Test Collection. Technical Report 5428. Computer Laboratory, University of Cambridge. British Library Research and Development Report."},{"key":"e_1_2_1_62_1","doi-asserted-by":"crossref","unstructured":"K. Sparck Jones S. Walker and S. E. Robertson. 2000. A probabilistic model of information retrieval: Development and comparative experiments. Part 1. Information Processing 8 Management 36 6 (2000) 779--808.  K. Sparck Jones S. Walker and S. E. Robertson. 2000. A probabilistic model of information retrieval: Development and comparative experiments. Part 1. Information Processing 8 Management 36 6 (2000) 779--808.","DOI":"10.1016\/S0306-4573(00)00015-7"},{"key":"e_1_2_1_63_1","volume-title":"Proc. ACM Conf. on Research and Development in Information Retrieval (SIGIR\u201914)","author":"Stanton I.","unstructured":"I. Stanton , S. Ieong , and N. Mishra . 2014. Circumlocution in diagnostic medical queries . In Proc. ACM Conf. on Research and Development in Information Retrieval (SIGIR\u201914) . 133--142. I. Stanton, S. Ieong, and N. Mishra. 2014. Circumlocution in diagnostic medical queries. In Proc. ACM Conf. on Research and Development in Information Retrieval (SIGIR\u201914). 133--142."},{"key":"e_1_2_1_64_1","volume-title":"Proc. Information Interaction in Context Symp (IIiX). 239--242","author":"Thomas P.","unstructured":"P. Thomas , A. Moffat , P. Bailey , and F. Scholer . 2014. Modeling decision points in user search behavior . In Proc. Information Interaction in Context Symp (IIiX). 239--242 . P. Thomas, A. Moffat, P. Bailey, and F. Scholer. 2014. Modeling decision points in user search behavior. In Proc. Information Interaction in Context Symp (IIiX). 239--242."},{"key":"e_1_2_1_65_1","volume-title":"Proc. Asia Information Retrieval Societies Conf. (AIRS\u201913)","author":"Thomas P.","unstructured":"P. Thomas , F. Scholer , and A. Moffat . 2013. What users do: The eyes have it . In Proc. Asia Information Retrieval Societies Conf. (AIRS\u201913) . 416--427. P. Thomas, F. Scholer, and A. Moffat. 2013. What users do: The eyes have it. In Proc. Asia Information Retrieval Societies Conf. (AIRS\u201913). 416--427."},{"key":"e_1_2_1_66_1","doi-asserted-by":"crossref","unstructured":"E. G. Toms H. O\u2019Brien T. Mackenzie C. Jordan L. Freund S. Toze E. Dawe and A. Macnutt. 2008. Task effects on interactive search: The query factor. In Focused Access to XML Documents. Springer 359--372.  E. G. Toms H. O\u2019Brien T. Mackenzie C. Jordan L. Freund S. Toze E. Dawe and A. Macnutt. 2008. Task effects on interactive search: The query factor. In Focused Access to XML Documents. Springer 359--372.","DOI":"10.1007\/978-3-540-85902-4_31"},{"key":"e_1_2_1_67_1","volume-title":"Proc. ACM Conf. on Research and Development in Information Retrieval (SIGIR\u201909)","author":"Turpin A.","unstructured":"A. Turpin , F. Scholer , K. J\u00e4rvelin , M. Wu , and J. S. Culpepper . 2009. Including summaries in system evaluation . In Proc. ACM Conf. on Research and Development in Information Retrieval (SIGIR\u201909) . 508--515. A. Turpin, F. Scholer, K. J\u00e4rvelin, M. Wu, and J. S. Culpepper. 2009. Including summaries in system evaluation. In Proc. ACM Conf. on Research and Development in Information Retrieval (SIGIR\u201909). 508--515."},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0306-4573(99)00028-X"},{"key":"e_1_2_1_69_1","volume-title":"Variations in relevance judgments and the measurement of retrieval effectiveness. Information Processing 8 Management 36, 5","author":"Voorhees E. M","year":"2000","unstructured":"E. M Voorhees . 2000. Variations in relevance judgments and the measurement of retrieval effectiveness. Information Processing 8 Management 36, 5 ( 2000 ), 697--716. E. M Voorhees. 2000. Variations in relevance judgments and the measurement of retrieval effectiveness. Information Processing 8 Management 36, 5 (2000), 697--716."},{"key":"e_1_2_1_70_1","volume-title":"Overview of the TREC 2002 question answering track. In Proc. Text Retrieval Conf. (TREC\u201902)","author":"Voorhees E. M.","year":"2002","unstructured":"E. M. Voorhees . 2002 a. Overview of the TREC 2002 question answering track. In Proc. Text Retrieval Conf. (TREC\u201902) . E. M. Voorhees. 2002a. Overview of the TREC 2002 question answering track. In Proc. Text Retrieval Conf. (TREC\u201902)."},{"key":"e_1_2_1_71_1","volume-title":"Overview of TREC 2002. In Proc. Text Retrieval Conf. (TREC\u201902)","author":"Voorhees E. M.","year":"2002","unstructured":"E. M. Voorhees . 2002 b. Overview of TREC 2002. In Proc. Text Retrieval Conf. (TREC\u201902) . E. M. Voorhees. 2002b. Overview of TREC 2002. In Proc. Text Retrieval Conf. (TREC\u201902)."},{"key":"e_1_2_1_72_1","volume-title":"Overview of the TREC 2003 robust retrieval track. In Proc. Text Retrieval Conf. (TREC\u201903)","author":"Voorhees E. M.","year":"2003","unstructured":"E. M. Voorhees . 2003 . Overview of the TREC 2003 robust retrieval track. In Proc. Text Retrieval Conf. (TREC\u201903) . E. M. Voorhees. 2003. Overview of the TREC 2003 robust retrieval track. In Proc. Text Retrieval Conf. (TREC\u201903)."},{"key":"e_1_2_1_73_1","volume-title":"Proc. ACM International Conf. on Information and Knowledge Management (CIKM\u201908)","author":"Webber W.","unstructured":"W. Webber , A. Moffat , and J. Zobel . 2008. Statistical power in retrieval experimentation . In Proc. ACM International Conf. on Information and Knowledge Management (CIKM\u201908) . 571--580. W. Webber, A. Moffat, and J. Zobel. 2008. Statistical power in retrieval experimentation. In Proc. ACM International Conf. on Information and Knowledge Management (CIKM\u201908). 571--580."},{"key":"e_1_2_1_74_1","doi-asserted-by":"publisher","DOI":"10.1145\/1852102.1852106"},{"key":"e_1_2_1_75_1","volume-title":"Proc. ACM Conf. on Research and Development in Information Retrieval (SIGIR\u201908)","author":"Webber W.","unstructured":"W. Webber , A. Moffat , J. Zobel , and T. Sakai . 2008. Precision-at-ten considered redundant . In Proc. ACM Conf. on Research and Development in Information Retrieval (SIGIR\u201908) . 695--696. W. Webber, A. Moffat, J. Zobel, and T. Sakai. 2008. Precision-at-ten considered redundant. In Proc. ACM Conf. on Research and Development in Information Retrieval (SIGIR\u201908). 695--696."},{"key":"e_1_2_1_76_1","volume-title":"Proc. Conf. on the World Wide Web (WWW\u201907)","author":"White R. W.","unstructured":"R. W. White and S. M. Drucker . 2007. Investigating behavioral variability in web search . In Proc. Conf. on the World Wide Web (WWW\u201907) . ACM, 21--30. R. W. White and S. M. Drucker. 2007. Investigating behavioral variability in web search. In Proc. Conf. on the World Wide Web (WWW\u201907). ACM, 21--30."},{"key":"e_1_2_1_77_1","volume-title":"Proc. ACM International Conf. on Information and Knowledge Management (CIKM\u201906)","author":"White R. W.","unstructured":"R. W. White and D. Kelly . 2006. A study on the effects of personalization and task information on implicit feedback performance . In Proc. ACM International Conf. on Information and Knowledge Management (CIKM\u201906) . 297--306. R. W. White and D. Kelly. 2006. A study on the effects of personalization and task information on implicit feedback performance. In Proc. ACM International Conf. on Information and Knowledge Management (CIKM\u201906). 297--306."},{"key":"e_1_2_1_78_1","volume-title":"Managing Gigabytes: Compressing and Indexing Documents and Images","author":"Witten I. H.","year":"1999","unstructured":"I. H. Witten , A. Moffat , and T. C. Bell . 1999 . Managing Gigabytes: Compressing and Indexing Documents and Images ( 2 nd ed.). Morgan Kaufmann . I. H. Witten, A. Moffat, and T. C. Bell. 1999. Managing Gigabytes: Compressing and Indexing Documents and Images (2nd ed.). Morgan Kaufmann.","edition":"2"},{"key":"e_1_2_1_79_1","volume-title":"Proc. Information Interaction in Context Symp (IIiX). 254--257","author":"Wu W.-C.","unstructured":"W.-C. Wu , D. Kelly , A. Edwards , and J. Arguello . 2012. Grannies, tanning beds, tattoos and NASCAR: Evaluation of search tasks with varying levels of cognitive complexity . In Proc. Information Interaction in Context Symp (IIiX). 254--257 . W.-C. Wu, D. Kelly, A. Edwards, and J. Arguello. 2012. Grannies, tanning beds, tattoos and NASCAR: Evaluation of search tasks with varying levels of cognitive complexity. In Proc. Information Interaction in Context Symp (IIiX). 254--257."},{"key":"e_1_2_1_80_1","volume-title":"Proc. ACM Conf. on Research and Development in Information Retrieval (SIGIR\u201914)","author":"Wu W.-C.","unstructured":"W.-C. Wu , D. Kelly , and A. Sud . 2014. Using information scent and need for cognition to understand online search behavior . In Proc. ACM Conf. on Research and Development in Information Retrieval (SIGIR\u201914) . 557--566. W.-C. Wu, D. Kelly, and A. Sud. 2014. Using information scent and need for cognition to understand online search behavior. In Proc. ACM Conf. on Research and Development in Information Retrieval (SIGIR\u201914). 557--566."},{"key":"e_1_2_1_81_1","volume-title":"Proc. ACM International Conf. on Information and Knowledge Management (CIKM\u201910)","author":"Yilmaz E.","unstructured":"E. Yilmaz , M. Shokouhi , N. Craswell , and S. Robertson . 2010. Expected browsing utility for web search evaluation . In Proc. ACM International Conf. on Information and Knowledge Management (CIKM\u201910) . 1561--1564. E. Yilmaz, M. Shokouhi, N. Craswell, and S. Robertson. 2010. Expected browsing utility for web search evaluation. In Proc. ACM International Conf. on Information and Knowledge Management (CIKM\u201910). 1561--1564."},{"key":"e_1_2_1_82_1","volume-title":"Proc. ACM International Conf. on Information and Knowledge Management (CIKM\u201914)","author":"Yilmaz E.","unstructured":"E. Yilmaz , M. Verma , N. Craswell , F. Radlinski , and P. Bailey . 2014. Relevance and effort: An analysis of document utility . In Proc. ACM International Conf. on Information and Knowledge Management (CIKM\u201914) . 91--100. E. Yilmaz, M. Verma, N. Craswell, F. Radlinski, and P. Bailey. 2014. Relevance and effort: An analysis of document utility. In Proc. ACM International Conf. on Information and Knowledge Management (CIKM\u201914). 91--100."},{"key":"e_1_2_1_83_1","doi-asserted-by":"publisher","DOI":"10.1145\/290941.291014"}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3052768","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3052768","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T03:36:56Z","timestamp":1750217816000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3052768"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,6,5]]},"references-count":82,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2017,7,31]]}},"alternative-id":["10.1145\/3052768"],"URL":"https:\/\/doi.org\/10.1145\/3052768","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"value":"1046-8188","type":"print"},{"value":"1558-2868","type":"electronic"}],"subject":[],"published":{"date-parts":[[2017,6,5]]},"assertion":[{"value":"2016-05-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2016-11-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2017-06-05","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}