{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,25]],"date-time":"2025-10-25T12:08:30Z","timestamp":1761394110589,"version":"3.41.2"},"reference-count":48,"publisher":"Emerald","issue":"6","license":[{"start":{"date-parts":[[2011,11,29]],"date-time":"2011-11-29T00:00:00Z","timestamp":1322524800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.emerald.com\/insight\/site-policies"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2011,11,29]]},"abstract":"<jats:sec><jats:title content-type=\"abstract-heading\">Purpose<\/jats:title><jats:p>This work aims to investigate the sensitivity of ranking performance with respect to the topic distribution of queries selected for ranking evaluation.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Design\/methodology\/approach<\/jats:title><jats:p>The authors reweight queries used in two TREC tasks to make them match three real background topic distributions, and show that the performance rankings of retrieval systems are quite different.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Findings<\/jats:title><jats:p>It is found that search engines tend to perform similarly on queries about the same topic; and search engine performance is sensitive to the topic distribution of queries used in evaluation.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Originality\/value<\/jats:title><jats:p>Using experiments with multiple real\u2010world query logs, the paper demonstrates weaknesses in the current evaluation model of retrieval systems.<\/jats:p><\/jats:sec>","DOI":"10.1108\/14684521111193184","type":"journal-article","created":{"date-parts":[[2011,11,26]],"date-time":"2011-11-26T07:10:21Z","timestamp":1322291421000},"page":"893-908","source":"Crossref","is-referenced-by-count":4,"title":["Topic\u2010sensitive search engine evaluation"],"prefix":"10.1108","volume":"35","author":[{"given":"Na","family":"Dai","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Brian D.","family":"Davison","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"140","reference":[{"key":"key2022020819474415200_b1","doi-asserted-by":"crossref","unstructured":"Adar, E., Weld, D.S., Bershad, B.N. and Gribble, S.S. (2007), \u201cWhy we search: visualizing and predicting user behaviour\u201d, Proceedings of the 16th International Conference on World Wide Web, ACM, New York, NY, pp. 161\u201070.","DOI":"10.1145\/1242572.1242595"},{"key":"key2022020819474415200_b2","doi-asserted-by":"crossref","unstructured":"Agichtein, E., Brill, E. and Dumais, S.T. (2006), \u201cImproving web search ranking by incorporating user behaviour\u201d, Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM Press, New York, NY, pp. 19\u201026.","DOI":"10.1145\/1148170.1148177"},{"key":"key2022020819474415200_b3","doi-asserted-by":"crossref","unstructured":"Allan, J., Carterette, B., Aslam, J.A., Pavlu, V., Dachev, B. and Kanoulas, E. (2007), \u201cMillion query track 2007 overview\u201d, Proceedings of the 16th Text REtrieval Conference (TREC), available at: http:\/\/trec.nist.gov\/pubs\/trec16\/papers\/1MQ.OVERVIEW16.pdf (accessed 19 September 2011).","DOI":"10.21236\/ADA477388"},{"key":"key2022020819474415200_b4","unstructured":"Baeza\u2010Yates, R. and Ribeiro\u2010Neto, B. (2011), Modern Information Retrieval, 2nd ed., Addison Wesley\/ACM Press, New York, NY."},{"key":"key2022020819474415200_b5","doi-asserted-by":"crossref","unstructured":"Beitzel, S.M., Jensen, E.C., Chowdhury, A. and Frieder, O. (2007), \u201cVarying approaches to topical web query classification\u201d, Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM Press, New York, NY, pp. 783\u20104.","DOI":"10.1145\/1277741.1277907"},{"key":"key2022020819474415200_b6","doi-asserted-by":"crossref","unstructured":"Beitzel, S.M., Jensen, E.C., Chowdhury, A., Grossman, D. and Frieder, O. (2004), \u201cHourly analysis of a very large topically categorized web query log\u201d, Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM Press, New York, NY, pp. 321\u20108.","DOI":"10.1145\/1008992.1009048"},{"key":"key2022020819474415200_b7","doi-asserted-by":"crossref","unstructured":"Beitzel, S.M., Jensen, E.C., Lewis, D.D., Chowdhury, A. and Frieder, O. (2007), \u201cAutomatic classification of web queries using very large unlabeled query logs\u201d, ACM Transactions on Information Systems, Vol. 25 No. 2, p. 9.","DOI":"10.1145\/1229179.1229183"},{"key":"key2022020819474415200_b8","unstructured":"Bhandari, S.K. and Davison, B.D. (2007), Leveraging Search Engine Results for Query Classification, Technical Report LU\u2010CSE\u201007\u2010013, Department of Computer Science and Engineering, Lehigh University, Lehigh, PA."},{"key":"key2022020819474415200_b9","doi-asserted-by":"crossref","unstructured":"Broder, A. (2002), \u201cA taxonomy of web search\u201d, SIGIR Forum, Vol. 36 No. 2, pp. 3\u201010.","DOI":"10.1145\/792550.792552"},{"key":"key2022020819474415200_b10","doi-asserted-by":"crossref","unstructured":"Broder, A., Fontoura, M., Gabrilovich, E., Joshi, A., Josifovski, V. and Zhang, T. (2007), \u201cRobust classification of rare queries using web knowledge\u201d, Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM Press, New York, NY, pp. 231\u20108.","DOI":"10.1145\/1277741.1277783"},{"key":"key2022020819474415200_b11","unstructured":"Bruza, P.D. and Dennis, S. (1997), \u201cQuery reformulation on the internet: empirical data and the hyperindex search engine\u201d, Proceedings of RIAO: Computer\u2010Assisted Information Searching on Internet, Centre de Hautes Etudes Internationales d'Informatique Documentaires, Montreal, pp. 488\u201099."},{"key":"key2022020819474415200_b12","doi-asserted-by":"crossref","unstructured":"Carterette, B., Pavlu, V., Kanoulas, E., Aslam, J.A. and Allan, J. (2008), \u201cEvaluation over thousands of queries\u201d, Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, New York, NY, pp. 651\u20108.","DOI":"10.1145\/1390334.1390445"},{"key":"key2022020819474415200_b13","doi-asserted-by":"crossref","unstructured":"Chen, H. and Dumais, S. (2000), \u201cBringing order to the web: automatically categorizing search results\u201d, Proceedings of the SIGCGHI Conference on Human Factors in Computing Systems, ACM Press, New York, NY, pp. 145\u201052.","DOI":"10.1145\/332040.332418"},{"key":"key2022020819474415200_b14","doi-asserted-by":"crossref","unstructured":"Jansen, B.J. and Spink, A. (2006), \u201cHow are we searching the world wide web? A comparison of nine search engine transaction logs\u201d, Information Processing and Management, Vol. 42 No. 1, pp. 248\u201063.","DOI":"10.1016\/j.ipm.2004.10.007"},{"key":"key2022020819474415200_b15","doi-asserted-by":"crossref","unstructured":"Jansen, B.J. and Pooch, U.W. (2001), \u201cWeb user studies: a review and framework for future work\u201d, Journal of the American Society of Information Science, Vol. 52 No. 3, pp. 235\u201046.","DOI":"10.1002\/1097-4571(2000)9999:9999<::AID-ASI1607>3.0.CO;2-F"},{"key":"key2022020819474415200_b16","doi-asserted-by":"crossref","unstructured":"Jansen, B.J., Booth, D.L. and Spink, A. (2007), \u201cDetermining the user intent of web search engine queries\u201d, Proceedings of the 16th World Wide Web Conference, ACM Pres, New York, NY, pp. 1149\u201050.","DOI":"10.1145\/1242572.1242739"},{"key":"key2022020819474415200_b17","doi-asserted-by":"crossref","unstructured":"Jansen, B.J., Spink, A. and Saracevic, T. (2000), \u201cReal life, real users, and real needs: a study and analysis of user queries on the web\u201d, Information Processing and Management, Vol. 36 No. 2, pp. 207\u201027.","DOI":"10.1016\/S0306-4573(99)00056-4"},{"key":"key2022020819474415200_b18","doi-asserted-by":"crossref","unstructured":"Jensen, E.C., Beitzel, S.M., Chowdhury, A. and Frieder, O. (2006), \u201cQuery phrase suggestion from topically tagged session logs\u201d, in Carbonell, J.G. and Siekmann, J. (Eds), Flexible Query Answering Systems, Lecture Notes in Computer Science, 4027, Springer, Berlin, pp. 185\u201096.","DOI":"10.1007\/11766254_16"},{"key":"key2022020819474415200_b19","doi-asserted-by":"crossref","unstructured":"Joachims, T. (2002), \u201cOptimizing search engines using clickthrough data\u201d, Proceedings of the ACM Conference on Knowledge Discovery and Data Mining, ACM Press, New York, NY, pp. 133\u201042.","DOI":"10.1145\/775047.775067"},{"key":"key2022020819474415200_b20","doi-asserted-by":"crossref","unstructured":"Joachims, T., Granka, L., Pan, B., Hembrooke, H. and Gay, G. (2005), \u201cAccurately interpreting clickthrough data as implicit feedback\u201d, Proceedings of the 28th Annual ACM SIGIR Conference on Research and Development in Information Retrieval, ACM Press, New York, NY, pp. 154\u201061.","DOI":"10.1145\/1076034.1076063"},{"key":"key2022020819474415200_b21","doi-asserted-by":"crossref","unstructured":"Joachims, T., Granka, L., Pan, B., Hembrooke, H., Radlinski, F. and Gay, G. (2007), \u201cEvaluating the accuracy of implicit feedback from clicks and query reformulations in web search\u201d, ACM Transactions on Information Systems, Vol. 25 No. 2, p. 7.","DOI":"10.1145\/1229179.1229181"},{"key":"key2022020819474415200_b22","doi-asserted-by":"crossref","unstructured":"Kang, I.\u2010H. and Kim, G. (2003), \u201cQuery type classification for web document retrieval\u201d, Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM Press, New York, NY, pp. 64\u201071.","DOI":"10.1145\/860435.860449"},{"key":"key2022020819474415200_b23","doi-asserted-by":"crossref","unstructured":"Kendall, M.G. (1938), \u201cA new measure of rank correlation\u201d, Biometrika, Vol. 30 Nos 1\/2, pp. 81\u201093.","DOI":"10.1093\/biomet\/30.1-2.81"},{"key":"key2022020819474415200_b24","doi-asserted-by":"crossref","unstructured":"Lee, U., Liu, Z. and Cho, J. (2005), \u201cAutomatic identification of user goals in web search\u201d, Proceedings of the 14th World Wide Web Conference, ACM Press, New York, NY, pp. 391\u2010400.","DOI":"10.1145\/1060745.1060804"},{"key":"key2022020819474415200_b25","doi-asserted-by":"crossref","unstructured":"Li, Y., Zheng, Z. and Dai, H.K. (2005), \u201cKDD CUP\u20102005 Report: facing a great challenge\u201d, SIGKDD Explorations, Vol. 7 No. 2, pp. 91\u20109.","DOI":"10.1145\/1117454.1117466"},{"key":"key2022020819474415200_b26","doi-asserted-by":"crossref","unstructured":"Liu, F., Yu, C. and Meng, W. (2002), \u201cPersonalized web search by mapping user queries to categories\u201d, Proceedings of the 11th International Conference on Information and Knowledge Management, ACM, New York, NY, pp. 558\u201065.","DOI":"10.1145\/584792.584884"},{"key":"key2022020819474415200_b27","unstructured":"McCallum, A.K. (1996), Bow: a toolkit for statistical language modeling, text retrieval, classification and clustering, available at: www.cs.cmu.edu\/\u223cmccallum\/bow (accessed 19 September 2011)."},{"key":"key2022020819474415200_b28","doi-asserted-by":"crossref","unstructured":"Manning, C.D., Raghavan, P. and Sch\u00fctze, H. (2008), Introduction to Information Retrieval, Cambridge University Press, Cambridge.","DOI":"10.1017\/CBO9780511809071"},{"key":"key2022020819474415200_b29","unstructured":"NIST (2011), Text REtrieval Conference (TREC) home page, available at: http:\/\/trec.nist.gov\/ (accessed 19 September 2011)."},{"key":"key2022020819474415200_b30","unstructured":"ODP (2011), The dmoz Open Directory Project (ODP), available at: http:\/\/www.dmoz.org\/ (accessed 19 September 2011)."},{"key":"key2022020819474415200_b31","unstructured":"Ozmutlu, H.C., Ozmutlu, S. and Spink, A. (2006), \u201cTopic estimation of web search transaction log queries using Monte Carlo simulation\u201d, Proceedings of the Australasian World Wide Web Conference, Southern Cross University, Noosa, pp. 118\u201022."},{"key":"key2022020819474415200_b32","doi-asserted-by":"crossref","unstructured":"Pass, G., Chowdhury, A. and Torgeson, C. (2006), \u201cA picture of search\u201d, Proceedings of the 1st International Conference on Scalable Information Systems, ACM Press, New York, NY, p. 1.","DOI":"10.1145\/1146847.1146848"},{"key":"key2022020819474415200_b33","doi-asserted-by":"crossref","unstructured":"Phan, N., Bailey, P. and Wilkinson, R. (2007), \u201cUnderstanding the relationship of information need specificity to search query length\u201d, Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, New York, NY, pp. 709\u201010.","DOI":"10.1145\/1277741.1277870"},{"key":"key2022020819474415200_b34","doi-asserted-by":"crossref","unstructured":"Piwowarski, B. and Zaragoza, H. (2007), \u201cPredictive user click models based on click\u2010through history\u201d, Proceedings of the Sixteenth ACM Conference on Information and Knowledge Management, ACM, New York, NY, pp. 175\u201082.","DOI":"10.1145\/1321440.1321467"},{"key":"key2022020819474415200_b35","doi-asserted-by":"crossref","unstructured":"Pu, H.\u2010T., Chuang, S.\u2010L. and Yang, C. (2002), \u201cSubject categorization of query terms for exploring web users' search interests\u201d, Journal of the American Society for Information Science and Technology, Vol. 53 No. 8, pp. 617\u201030.","DOI":"10.1002\/asi.10071"},{"key":"key2022020819474415200_b36","doi-asserted-by":"crossref","unstructured":"Rose, D.E. and Levinson, D. (2004), \u201cUnderstanding user goals in web search\u201d, Proceedings of the 13th World Wide Web Conference (WWW), ACM Press, New York, NY, pp. 13\u201019.","DOI":"10.1145\/988672.988675"},{"key":"key2022020819474415200_b37","doi-asserted-by":"crossref","unstructured":"Shen, D., Sun, J.\u2010T., Yang, Q. and Chen, Z. (2006a), \u201cBuilding bridges for web query classification\u201d, Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM Press, New York, NY, pp. 131\u20108.","DOI":"10.1145\/1148170.1148196"},{"key":"key2022020819474415200_b38","doi-asserted-by":"crossref","unstructured":"Shen, D., Pan, R., Sun, J.\u2010T., Pan, J.J., Wu, K., Yin, J. and Yang, Q. (2006b), \u201cQuery enrichment for web\u2010query classification\u201d, ACM Transactions on Information Systems, Vol. 24 No. 3, pp. 320\u201052.","DOI":"10.1145\/1165774.1165776"},{"key":"key2022020819474415200_b39","doi-asserted-by":"crossref","unstructured":"Spink, A. and Jansen, B.J. (2004), Web Search: Public Searching of the Web, Springer, Berlin.","DOI":"10.1007\/1-4020-2269-7"},{"key":"key2022020819474415200_b40","unstructured":"Spink, A., Bateman, J. and Jansen, B.J. (1998), \u201cSearching heterogeneous collections on the web: behavior of Excite users\u201d, Information Research, Vol. 4 No. 2, pp. 317\u201028."},{"key":"key2022020819474415200_b41","doi-asserted-by":"crossref","unstructured":"Spink, A., Bateman, J. and Jansen, B.J. (1999), \u201cSearching the web: a survey of Excite users\u201d, Internet Research: Electronic Networking Applications and Policy, Vol. 9 No. 2, pp. 117\u201028.","DOI":"10.1108\/10662249910264882"},{"key":"key2022020819474415200_b42","doi-asserted-by":"crossref","unstructured":"Spink, A., Jansen, B.J. and Ozmutlu, H.C. (2000), \u201cUse of query reformulation and relevance feedback by Excite users\u201d, Internet Research: Electronic Networking Applications and Policy, Vol. 10 No. 4, pp. 317\u201028.","DOI":"10.1108\/10662240010342621"},{"key":"key2022020819474415200_b43","doi-asserted-by":"crossref","unstructured":"Spink, A., Wolfram, D., Jansen, B.J. and Saracevic, T. (2001), \u201cSearching the web: the public and their queries\u201d, Journal of the American Society for Information Science, Vol. 53 No. 2, pp. 226\u201034.","DOI":"10.1002\/1097-4571(2000)9999:9999<::AID-ASI1591>3.0.CO;2-R"},{"key":"key2022020819474415200_b44","doi-asserted-by":"crossref","unstructured":"Taksa, I., Zelikovitz, S. and Spink, A. (2007), \u201cUsing web search logs to identify query classification terms\u201d, Proceedings of the Fourth International Conference on Information Technology, IEEE Computer Society, Washington, DC, pp. 469\u201074.","DOI":"10.1109\/ITNG.2007.202"},{"key":"key2022020819474415200_b45","doi-asserted-by":"crossref","unstructured":"Vogel, D., Bickel, S., Haider, P., Schimpfky, R., Siemen, P., Bridges, S. and Scheffer, T. (2005), \u201cClassifying search engine queries using the web as background knowledge\u201d, SIGKDD Explorations, Vol. 7 No. 2, pp. 117\u201022.","DOI":"10.1145\/1117454.1117469"},{"key":"key2022020819474415200_b46","doi-asserted-by":"crossref","unstructured":"Wang, X. and Zhai, C. (2007), \u201cLearn from web search logs to organize search results\u201d, Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, New York, NY, pp. 87\u201094.","DOI":"10.1145\/1277741.1277759"},{"key":"key2022020819474415200_b47","doi-asserted-by":"crossref","unstructured":"Yilmaz, E., Kanoulas, E. and Aslam, J.A. (2008), \u201cA simple and efficient sampling method for estimating AP and NDCG\u201d, Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, New York, NY, pp. 603\u201010.","DOI":"10.1145\/1390334.1390437"},{"key":"key2022020819474415200_b48","unstructured":"Zhang, Y. and Moffat, A. (2006), \u201cSome observations on user search behaviour\u201d, Australian Journal of Intelligent Information Processing Systems, Vol. 9 No. 2, pp. 1\u20108."}],"container-title":["Online Information Review"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/www.emeraldinsight.com\/doi\/full-xml\/10.1108\/14684521111193184","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/14684521111193184\/full\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/14684521111193184\/full\/html","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,25]],"date-time":"2025-07-25T00:41:46Z","timestamp":1753404106000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.emerald.com\/oir\/article\/35\/6\/893-908\/314118"}},"subtitle":[],"editor":[{"given":"Dirk","family":"Lewandowski","sequence":"first","affiliation":[],"role":[{"role":"editor","vocabulary":"crossref"}]}],"short-title":[],"issued":{"date-parts":[[2011,11,29]]},"references-count":48,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2011,11,29]]}},"alternative-id":["10.1108\/14684521111193184"],"URL":"https:\/\/doi.org\/10.1108\/14684521111193184","relation":{},"ISSN":["1468-4527"],"issn-type":[{"type":"print","value":"1468-4527"}],"subject":[],"published":{"date-parts":[[2011,11,29]]}}}