{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,17]],"date-time":"2026-07-17T04:23:27Z","timestamp":1784262207438,"version":"3.55.0"},"reference-count":28,"publisher":"Emerald","issue":"6","license":[{"start":{"date-parts":[[2011,11,29]],"date-time":"2011-11-29T00:00:00Z","timestamp":1322524800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.emerald.com\/insight\/site-policies"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2011,11,29]]},"abstract":"<jats:sec><jats:title content-type=\"abstract-heading\">Purpose<\/jats:title><jats:p>The purpose of this paper is to introduce two new automatic methods for evaluating the performance of search engines. The reported study uses the methods to experimentally investigate which search engine among three popular search engines (Ask.com, Bing and Google) gives the best performance.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Design\/methodology\/approach<\/jats:title><jats:p>The study assesses the performance of three search engines. For each one the weighted average of similarity degrees between its ranked result list and those of its metasearch engines is measured. Next these measures are compared to establish which search engine gives the best performance. To compute the similarity degree between the lists two measures called the \u201ctendency degree\u201d and \u201ccoverage degree\u201d are introduced; the former assesses a search engine in terms of results presentation and the latter evaluates it in terms of retrieval effectiveness. The performance of the search engines is experimentally assessed based on the 50 topics of the 2002 TREC web track. The effectiveness of the methods is also compared with human\u2010based ones.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Findings<\/jats:title><jats:p>Google outperformed the others, followed by Bing and Ask.com. Moreover significant degrees of consistency \u2013 92.87 percent and 91.93 percent \u2013 were found between automatic and human\u2010based approaches.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Practical implications<\/jats:title><jats:p>The findings of this work could help users to select a truly effective search engine. The results also provide motivation for the vendors of web search engines to improve their technology.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Originality\/value<\/jats:title><jats:p>The paper focuses on two novel automatic methods to evaluate the performance of search engines and provides valuable experimental results on three popular ones.<\/jats:p><\/jats:sec>","DOI":"10.1108\/14684521111193229","type":"journal-article","created":{"date-parts":[[2011,11,26]],"date-time":"2011-11-26T07:03:35Z","timestamp":1322291015000},"page":"957-971","source":"Crossref","is-referenced-by-count":8,"title":["Automatic performance evaluation of web search engines using judgments of metasearch engines"],"prefix":"10.1108","volume":"35","author":[{"given":"Hamid","family":"Sadeghi","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"140","reference":[{"key":"key2022020920033159400_b1","doi-asserted-by":"crossref","unstructured":"Ali, R. and Beg, M.M.S. (2009), \u201cAutomatic performance evaluation of web search systems using rough set based rank aggregation\u201d, Proceedings of the First International Conference on Intelligent Human Computer Interaction, Springer, Berlin, pp. 344\u201058.","DOI":"10.1007\/978-81-8489-203-1_34"},{"key":"key2022020920033159400_b2","doi-asserted-by":"crossref","unstructured":"Bar\u2010Ilan, J. (2005), \u201cComparing rankings of search results on the web\u201d, Information Processing and Management, Vol. 41 No. 6, pp. 1511\u20109.","DOI":"10.1016\/j.ipm.2005.03.008"},{"key":"key2022020920033159400_b3","doi-asserted-by":"crossref","unstructured":"Bar\u2010Ilan, J., Mat\u2010Hassan, M. and Levene, M. (2006), \u201cMethods for comparing rankings of search engine results\u201d, Computer Networks, Vol. 50 No. 10, pp. 1448\u201063.","DOI":"10.1016\/j.comnet.2005.10.020"},{"key":"key2022020920033159400_b4","doi-asserted-by":"crossref","unstructured":"Bar\u2010Ilan, J., Keenoy, K., Yaari, E. and Levene, M. (2007), \u201cUser rankings of search engine results\u201d, Journal of the American Society for Information Science and Technology, Vol. 58 No. 9, pp. 1254\u201066.","DOI":"10.1002\/asi.20608"},{"key":"key2022020920033159400_b5","doi-asserted-by":"crossref","unstructured":"Bar\u2010Yossef, Z. and Gurevich, M. (2008), \u201cRandom sampling from a search engine's index\u201d, Journal of the ACM, Vol. 55 No. 5, pp. 1\u201074.","DOI":"10.1145\/1411509.1411514"},{"key":"key2022020920033159400_b6","doi-asserted-by":"crossref","unstructured":"Beg, M.M.S. (2005), \u201cA subjective measure of web search quality\u201d, Information Sciences, Vol. 169 Nos 3\u20104, pp. 365\u201081.","DOI":"10.1016\/j.ins.2004.07.003"},{"key":"key2022020920033159400_b7","doi-asserted-by":"crossref","unstructured":"Can, F., Nuray, R. and Sevdik, A.B. (2004), \u201cAutomatic performance evaluation of web search engines\u201d, Information Processing and Management, Vol. 40 No. 3, pp. 495\u2010514.","DOI":"10.1016\/S0306-4573(03)00040-2"},{"key":"key2022020920033159400_b8","unstructured":"Carterette, B. and Jones, R. (2007), \u201cEvaluating search engines by modeling the relationship between relevance and clicks\u201d, Proceedings of the 21st Annual Conference on Advances in Neural Information Processing Systems, Curran Associates, Red Hook, NY, pp. 217\u201024."},{"key":"key2022020920033159400_b9","doi-asserted-by":"crossref","unstructured":"Cen, R., Liu, Y., Zhang, M., Ru, L. and Ma, S. (2009), \u201cAutomatic search engine performance evaluation with the wisdom of crowds\u201d, Lecture Notes in Computer Science, Vol. 5839, pp. 351\u201062.","DOI":"10.1007\/978-3-642-04769-5_31"},{"key":"key2022020920033159400_b10","unstructured":"Craswell, N. and Hawking, D. (2002), \u201cOverview of the TREC\u20102002 web track\u201d, Proceedings of the 11th Text REtrieval Conference (TREC), National Institute of Standards and Technology, Gaithersburg, MD, pp. 86\u201095."},{"key":"key2022020920033159400_b11","doi-asserted-by":"crossref","unstructured":"Diaconis, P. and Graham, R.L. (1977), \u201cSpearman's footrule as a measure of disarray\u201d, Journal of the Royal Statistical Society, Series B (Methodological), Vol. 39 No. 2, pp. 262\u20108.","DOI":"10.1111\/j.2517-6161.1977.tb01624.x"},{"key":"key2022020920033159400_b12","doi-asserted-by":"crossref","unstructured":"Dwork, C., Kumar, R., Naor, M. and Sivakumar, D. (2001), \u201cRank aggregation methods for the web\u201d, Proceedings of the 10th International World Wide Web Conference, ACM, New York, NY, pp. 613\u201022.","DOI":"10.1145\/371920.372165"},{"key":"key2022020920033159400_b13","doi-asserted-by":"crossref","unstructured":"Fagin, R., Kumar, R. and Sivakumar, D. (2003), \u201cComparing top k lists\u201d, SIAM Journal on Discrete Mathematics, Vol. 17 No. 1, pp. 134\u201060.","DOI":"10.1137\/S0895480102412856"},{"key":"key2022020920033159400_b14","doi-asserted-by":"crossref","unstructured":"Gulli, A. and Signorini, A. (2005), \u201cThe indexable web is more than 11.5 billion pages\u201d, Proceedings of the 14th International Conference on World Wide Web, ACM, New York, NY, pp. 902\u20103.","DOI":"10.1145\/1062745.1062789"},{"key":"key2022020920033159400_b15","doi-asserted-by":"crossref","unstructured":"H\u00f6chst\u00f6tter, N. and Lewandowski, D. (2009), \u201cWhat users see \u2013 structures in search engine results pages\u201d, Information Sciences, Vol. 179 No. 12, pp. 1796\u2010812.","DOI":"10.1016\/j.ins.2009.01.028"},{"key":"key2022020920033159400_b16","doi-asserted-by":"crossref","unstructured":"Lewandowski, D. (2008a), \u201cThe retrieval effectiveness of web search engines: considering results descriptions\u201d, Journal of Documentation, Vol. 64 No. 6, pp. 915\u201037.","DOI":"10.1108\/00220410810912451"},{"key":"key2022020920033159400_b17","doi-asserted-by":"crossref","unstructured":"Lewandowski, D. (2008b), \u201cA three\u2010year study on the freshness of web search engine databases\u201d, Journal of Information Science, Vol. 34 No. 6, pp. 817\u201031.","DOI":"10.1177\/0165551508089396"},{"key":"key2022020920033159400_b18","doi-asserted-by":"crossref","unstructured":"McCown, F. and Nelson, M.L. (2007), \u201cAgreeing to disagree: search engines and their public interfaces\u201d, Proceedings of the 7th ACM\/IEEE\u2010CS Joint Conference on Digital Libraries, ACM, New York, NY, pp. 309\u201018.","DOI":"10.1145\/1255175.1255237"},{"key":"key2022020920033159400_b19","doi-asserted-by":"crossref","unstructured":"Meng, W., Yu, C. and Liu, K.L. (2002), \u201cBuilding efficient and effective metasearch engines\u201d, ACM Computing Surveys, Vol. 34 No. 1, pp. 48\u201089.","DOI":"10.1145\/505282.505284"},{"key":"key2022020920033159400_b20","unstructured":"Microsoft News Center (2009), Microsoft, Yahoo! change search landscape, available at: www.microsoft.com\/presspass\/press\/2009\/jul09\/07\u201029release.mspx (accessed 14 December 2010)."},{"key":"key2022020920033159400_b21","doi-asserted-by":"crossref","unstructured":"Nuray, R. and Can, F. (2006), \u201cAutomatic ranking of information retrieval systems using data fusion\u201d, Information Processing and Management, Vol. 42 No. 3, pp. 595\u2010614.","DOI":"10.1016\/j.ipm.2005.03.023"},{"key":"key2022020920033159400_b22","doi-asserted-by":"crossref","unstructured":"Shang, Y. and Li, L. (2002), \u201cPrecision evaluation of search engines\u201d, World Wide Web: Internet and Web Information Systems, Vol. 5 No. 2, pp. 159\u201073.","DOI":"10.1023\/A:1019679624079"},{"key":"key2022020920033159400_b23","doi-asserted-by":"crossref","unstructured":"Shi, Z., Wang, B., Li, P. and Shi, Z. (2010), \u201cUsing global statistics to rank retrieval systems without relevance judgments\u201d, IFIP Advances in Information and Communication Technology, Vol. 340, pp. 183\u201092.","DOI":"10.1007\/978-3-642-16327-2_24"},{"key":"key2022020920033159400_b24","doi-asserted-by":"crossref","unstructured":"Spink, A., Jansen, B.J., Blakely, C. and Koshman, S. (2006), \u201cA study of results overlap and uniqueness among major web search engines\u201d, Information Processing and Management, Vol. 42 No. 5, pp. 1379\u201091.","DOI":"10.1016\/j.ipm.2005.11.001"},{"key":"key2022020920033159400_b25","doi-asserted-by":"crossref","unstructured":"Spoerri, A. (2007), \u201cUsing the structure of overlap between search results to rank retrieval systems without relevance judgments\u201d, Information Processing and Management, Vol. 43 No. 4, pp. 1059\u201070.","DOI":"10.1016\/j.ipm.2006.09.009"},{"key":"key2022020920033159400_b26","doi-asserted-by":"crossref","unstructured":"Voorhees, E. (2002), \u201cOverview of TREC 2002\u201d, Proceedings of the 11th Text REtrieval Conference (TREC), National Institute of Standards and Technology, Gaithersburg, MD, pp. 1\u201015.","DOI":"10.6028\/NIST.SP.500-251"},{"key":"key2022020920033159400_b27","doi-asserted-by":"crossref","unstructured":"Zhang, J. and Fei, W. (2010), \u201cSearch engines' responses to several search feature selections\u201d, The International Information & Library Review, Vol. 42 No. 3, pp. 212\u201025.","DOI":"10.1080\/10572317.2010.10762866"},{"key":"key2022020920033159400_b28","doi-asserted-by":"crossref","unstructured":"Zhang, J. and Lin, S. (2007), \u201cMultiple language supports in search engines\u201d, Online Information Review, Vol. 31 No. 4, pp. 516\u201032.","DOI":"10.1108\/14684520710780458"}],"container-title":["Online Information Review"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/www.emeraldinsight.com\/doi\/full-xml\/10.1108\/14684521111193229","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/14684521111193229\/full\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/14684521111193229\/full\/html","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,25]],"date-time":"2025-07-25T00:41:47Z","timestamp":1753404107000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.emerald.com\/oir\/article\/35\/6\/957-971\/314105"}},"subtitle":[],"editor":[{"given":"Dirk","family":"Lewandowski","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2011,11,29]]},"references-count":28,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2011,11,29]]}},"alternative-id":["10.1108\/14684521111193229"],"URL":"https:\/\/doi.org\/10.1108\/14684521111193229","relation":{},"ISSN":["1468-4527"],"issn-type":[{"value":"1468-4527","type":"print"}],"subject":[],"published":{"date-parts":[[2011,11,29]]}}}