{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,1]],"date-time":"2026-03-01T11:01:21Z","timestamp":1772362881924,"version":"3.50.1"},"reference-count":25,"publisher":"Emerald","issue":"6","license":[{"start":{"date-parts":[[2011,11,29]],"date-time":"2011-11-29T00:00:00Z","timestamp":1322524800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.emerald.com\/insight\/site-policies"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2011,11,29]]},"abstract":"<jats:sec><jats:title content-type=\"abstract-heading\">Purpose<\/jats:title><jats:p>The aim of this paper is to develop a methodology for assessing search results retrieved from different sources.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Design\/methodology\/approach<\/jats:title><jats:p>This is a two phase method, where in the first stage users select and rank the ten best search results from a randomly ordered set. In the second stage they are asked to choose the best pre\u2010ranked result from a set of possibilities. This two\u2010stage method allows users to consider each search result separately (in the first stage) and to express their views on the rankings as a whole, as they were retrieved by the search provider. The method was tested in a user study that compared different country\u2010specific search results of Google and Live Search (now Bing). The users were Israelis and the search results came from six sources: Google Israel, Google.com, Google UK, Live Search Israel, Live Search US and Live Search UK. The users evaluated the results of nine pre\u2010selected queries, created their own preferred ranking and picked the best ranking from the six sources.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Findings<\/jats:title><jats:p>The results indicate that the group of users in this study preferred their local Google interface, i.e. Google succeeded in its country\u2010specific customisation of search results. Live Search was much less successful in this aspect.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Research limitations\/implications<\/jats:title><jats:p>Search engines are highly dynamic, thus the findings of the case study have to be viewed cautiously.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Originality\/value<\/jats:title><jats:p>The main contribution of the paper is a two\u2010phase methodology for comparing and evaluating search results from different sources.<\/jats:p><\/jats:sec>","DOI":"10.1108\/14684521111193166","type":"journal-article","created":{"date-parts":[[2011,11,26]],"date-time":"2011-11-26T07:13:41Z","timestamp":1322291621000},"page":"854-868","source":"Crossref","is-referenced-by-count":14,"title":["A method to assess search engine results"],"prefix":"10.1108","volume":"35","author":[{"given":"Judit","family":"Bar\u2010Ilan","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mark","family":"Levene","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"140","reference":[{"key":"key2022032120164047000_b1","doi-asserted-by":"crossref","unstructured":"Bar\u2010Ilan, J. and Gutman, T. (2005), \u201cHow do search engines respond to some non\u2010English queries?\u201d, Journal of Information Science, Vol. 31 No. 1, pp. 13\u201028.","DOI":"10.1177\/0165551505049255"},{"key":"key2022032120164047000_b2","doi-asserted-by":"crossref","unstructured":"Bar\u2010Ilan, J., Keenoy, K., Levene, M. and Yaari, E. (2007), \u201cUser rankings of search engine results\u201d, Journal of the American Society for Information Science and Technology, Vol. 58 No. 9, pp. 1254\u201066.","DOI":"10.1002\/asi.20608"},{"key":"key2022032120164047000_b3","doi-asserted-by":"crossref","unstructured":"Bar\u2010Ilan, J., Keenoy, K., Levene, M. and Yaari, E. (2009), \u201cPresentation bias is significant in determining user preference for search results \u2013 a user study\u201d, Journal of the American Society for Information Science and Technology, Vol. 60 No. 1, pp. 135\u201049.","DOI":"10.1002\/asi.20941"},{"key":"key2022032120164047000_b4","unstructured":"ComScore (2010), \u201cComScore releases June 2010 US search engine rankings\u201d, available at: www.comscore.com\/Press_Events\/Press_Releases\/2010\/7\/comScore_Releases_June_2010_U.S._Search_Engine_Rankings (accessed 16 August 2010)."},{"key":"key2022032120164047000_b5","unstructured":"Dou, Z., Song, R. and Wen, J.R. (2007), \u201cA large scale evaluation and analysis of personalized search strategies\u201d, Proceedings of WWW2007, ACM, ACM, New York, NY, pp. 581\u201090."},{"key":"key2022032120164047000_b6","unstructured":"Enquiro (2005), \u201cDid\u2010It, Enquiro and Eyetools uncover search's golden triangle\u201d, available at: http:\/\/web.archive.org\/web\/20060101194915\/http:\/\/www.enquiro.com\/eye\u2010tracking\u2010pr.asp (accessed 16 August 2010)."},{"key":"key2022032120164047000_b7","unstructured":"Google (2010), \u201cLanguage tools\u201d, available at: www.google.com\/language_tools (accessed 16 August 2010)."},{"key":"key2022032120164047000_b8","doi-asserted-by":"crossref","unstructured":"Gordon, M. and Pathak, P. (1999), \u201cFinding information on the World Wide Web: the retrieval effectiveness of search engines\u201d, Information Processing and Management, Vol. 35 No. 2, pp. 141\u201080.","DOI":"10.1016\/S0306-4573(98)00041-7"},{"key":"key2022032120164047000_b9","unstructured":"Griesbaum, J. (2004), \u201cEvaluation of three German search engines: Altavista.de, Google.de and Lycos.de\u201d, Information Research, Vol. 9 No. 4, available at: http:\/\/informationr.net\/ir\/9\u20104\/paper189.html (accessed 28 December 2010)."},{"key":"key2022032120164047000_b10","doi-asserted-by":"crossref","unstructured":"Hawking, D., Craswell, N., Bailey, P. and Griffiths, K. (2001), \u201cMeasuring search engine quality\u201d, Information Retrieval, Vol. 4 No. 1, pp. 33\u201059.","DOI":"10.1023\/A:1011468107287"},{"key":"key2022032120164047000_b11","doi-asserted-by":"crossref","unstructured":"J\u00e4rvelin, K. and Kek\u00e4l\u00e4inen, J. (2000), \u201cIR evaluation methods for retrieving highly relevant documents\u201d, Proceedings of SIGIR'00, Vol. 00, ACM, New York, NY, pp. 41\u20108.","DOI":"10.1145\/345508.345545"},{"key":"key2022032120164047000_b12","doi-asserted-by":"crossref","unstructured":"Joachims, T., Granka, L., Pan, B., Hembrooke, H., Radlinksi, F. and Gay, G. (2007), \u201cEvaluating the accuracy of implicit feedback from clicks and query reformulations in web search\u201d, ACM Transactions on Information Systems, Vol. 25 No. 2.","DOI":"10.1145\/1229179.1229181"},{"key":"key2022032120164047000_b13","doi-asserted-by":"crossref","unstructured":"Lazarinis, F., Vilares, J., Tait, J. and Efthimiadis, E. (2009), \u201cCurrent research issues and trends in non\u2010English web searching\u201d, Information Retrieval, Vol. 12 No. 3, pp. 230\u201050.","DOI":"10.1007\/s10791-009-9093-0"},{"key":"key2022032120164047000_b14","doi-asserted-by":"crossref","unstructured":"Lewandowski, D. (2008), \u201cThe retrieval effectiveness of web search engines: considering results descriptions\u201d, Journal of Documentation, Vol. 64 No. 6, pp. 915\u201037.","DOI":"10.1108\/00220410810912451"},{"key":"key2022032120164047000_b15","unstructured":"Live Search (2009), \u201cSearch worldwide\u201d, available at: http:\/\/search.live.com\/worldwide.aspx (accessed 26 January 2009)."},{"key":"key2022032120164047000_b16","doi-asserted-by":"crossref","unstructured":"Mizzaro, S. (1998), \u201cHow many relevances in information retrieval?\u201d, Interacting with Computers, Vol. 10 No. 3, pp. 305\u201022.","DOI":"10.1016\/S0953-5438(98)00012-5"},{"key":"key2022032120164047000_b17","unstructured":"PEW (2009), \u201cPew Internet & American Life Project tracking surveys (March 2000\u2010September 2009)\u201d, Pew Internet & American Life Project, available at: www.pewinternet.org\/Static\u2010Pages\/Trend\u2010Data.aspx (accessed 16 August 2010)."},{"key":"key2022032120164047000_b18","doi-asserted-by":"crossref","unstructured":"Pitkow, J., Sch\u00fctze, H., Cass, T., Cooley, R., Turnball, D., Edmonds, A., Adar, E. and Breuel, T. (2002), \u201cPersonalized search\u201d, Communications of the ACM, Vol. 45 No. 9, pp. 50\u20105.","DOI":"10.1145\/567498.567526"},{"key":"key2022032120164047000_b19","doi-asserted-by":"crossref","unstructured":"Saracevic, T. (1975), \u201cRelevance: a review of and a framework for the thinking on the notion in information science\u201d, Journal of the American Society for Information Science, Vol. 26 No. 6, pp. 321\u201043.","DOI":"10.1002\/asi.4630260604"},{"key":"key2022032120164047000_b20","unstructured":"Saracevic, T. (1996), \u201cRelevance reconsidered\u201d, Proceedings of the 2nd Conference on Conceptions of Library and Information Science, available at: http:\/\/comminfo.rutgers.edu\/\u223ctefko\/CoLIS2_1996.doc (accessed 3 August 2011)."},{"key":"key2022032120164047000_b21","doi-asserted-by":"crossref","unstructured":"Shen, X., Tan, B. and Zhai, C.X. (2005), \u201cImplicit user modeling for personalized search\u201d, Proceedings of CIKM05, ACM, New York, NY, pp. 824\u201031.","DOI":"10.1145\/1099554.1099747"},{"key":"key2022032120164047000_b22","doi-asserted-by":"crossref","unstructured":"Spink, A. and Jansen, B.J. (2004), Web Search: Public Searching of the Web, Kluwer Academic Publishers, Dordrecht.","DOI":"10.1007\/1-4020-2269-7"},{"key":"key2022032120164047000_b23","doi-asserted-by":"crossref","unstructured":"Teevan, J., Dumais, S.T. and Horvitz, E. (2005), \u201cPersonalizing search via automated analysis of interests and activities\u201d, Proceedings of SIGIR05, ACM, New York, NY, pp. 449\u201056.","DOI":"10.1145\/1076034.1076111"},{"key":"key2022032120164047000_b24","unstructured":"V\u00e9ronis, J. (2006), \u201cA comparative study of six search engines\u201d, available at: www.up.univ\u2010mrs.fr\/veronis\/pdf\/2006\u2010comparative\u2010study.pdf (accessed 28 December 2010)."},{"key":"key2022032120164047000_b25","unstructured":"Walla (2010), \u201cTIM survey July\u201d, (in Hebrew), available at: http:\/\/b.walla.co.il\/?w=\/3050\/1719860 (accessed 16 August 2010)."}],"container-title":["Online Information Review"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/www.emeraldinsight.com\/doi\/full-xml\/10.1108\/14684521111193166","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/14684521111193166\/full\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/14684521111193166\/full\/html","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,25]],"date-time":"2025-07-25T00:41:46Z","timestamp":1753404106000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.emerald.com\/oir\/article\/35\/6\/854-868\/314103"}},"subtitle":[],"editor":[{"given":"Dirk","family":"Lewandowski","sequence":"first","affiliation":[],"role":[{"role":"editor","vocabulary":"crossref"}]}],"short-title":[],"issued":{"date-parts":[[2011,11,29]]},"references-count":25,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2011,11,29]]}},"alternative-id":["10.1108\/14684521111193166"],"URL":"https:\/\/doi.org\/10.1108\/14684521111193166","relation":{},"ISSN":["1468-4527"],"issn-type":[{"value":"1468-4527","type":"print"}],"subject":[],"published":{"date-parts":[[2011,11,29]]}}}