{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,25]],"date-time":"2025-10-25T12:00:11Z","timestamp":1761393611137,"version":"3.41.2"},"reference-count":28,"publisher":"Emerald","issue":"4\/5","license":[{"start":{"date-parts":[[2007,7,12]],"date-time":"2007-07-12T00:00:00Z","timestamp":1184198400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.emerald.com\/insight\/site-policies"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2007,7,12]]},"abstract":"<jats:sec><jats:title content-type=\"abstract-heading\">Purpose<\/jats:title><jats:p>The aim of the paper is to put forward a structured mechanism for web search evaluation. The paper seeks to point to useful scientific research and show how information practitioners can use these methods in evaluation of search on the web for their users.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Design\/methodology\/approach<\/jats:title><jats:p>The paper puts forward an approach which utilizes traditional laboratory\u2010based evaluation measures such as average precision\/precision at<jats:italic>N<\/jats:italic>documents, augmented with diagnostic measures such as link broken, etc., which are used to show why precision measures are depressed as well as the quality of the search engines crawling mechanism.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Findings<\/jats:title><jats:p>The paper shows how to use diagnostic measures in conjunction with precision in order to evaluate web search.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Practical implications<\/jats:title><jats:p>The methodology presented in this paper will be useful to any information professional who regularly uses web search as part of their information seeking and needs to evaluate web search services.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Originality\/value<\/jats:title><jats:p>The paper argues that the use of diagnostic measures is essential in web search, as precision measures on their own do not allow a searcher to understand why search results differ between search engines.<\/jats:p><\/jats:sec>","DOI":"10.1108\/00012530710817573","type":"journal-article","created":{"date-parts":[[2007,9,8]],"date-time":"2007-09-08T07:02:29Z","timestamp":1189234949000},"page":"352-366","source":"Crossref","is-referenced-by-count":14,"title":["Evaluation of web search for the information practitioner"],"prefix":"10.1108","volume":"59","author":[{"given":"A.","family":"MacFarlane","sequence":"first","affiliation":[]}],"member":"140","reference":[{"key":"key2022020420291189600_b1","unstructured":"Aitchison, J. and Cleverdon, C. (1963), \u201cA report on a test of the index of metallurgical literature of Western Reserve University\u201d, The College of Aeronautics, Cranfield."},{"key":"key2022020420291189600_b2","doi-asserted-by":"crossref","unstructured":"Belkin, N., Oddy, R. and Brooks, H. (1982), \u201cASK for information retrieval: part 1. background and theory\u201d, Journal of Documentation, Vol. 38 No. 2 (reprinted in Spark\u2010Jones, K. and Willett, P. (1997), Readings in Information Retrieval, Morgan Kaufmann, San Francisco, CA, pp. 299\u2010304).","DOI":"10.1108\/eb026722"},{"key":"key2022020420291189600_b3","doi-asserted-by":"crossref","unstructured":"Broder, A. (2002), \u201cA taxonomy of web search\u201d, SIGIR Forum, Vol. 36 No. 2, pp. 3\u201010.","DOI":"10.1145\/792550.792552"},{"key":"key2022020420291189600_b4","doi-asserted-by":"crossref","unstructured":"Chowdhury, A. and Soboroff, I. (2002), \u201cAutomatic evaluation of World Wide Web search services\u201d, in Beaulieu, M., Baeza\u2010Yates, R., Myaeng, S. and Jarvelin, K. (Eds), Proceedings of the Twenty\u2010fifth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2002), Tampere, pp. 421\u20102.","DOI":"10.1145\/564376.564474"},{"key":"key2022020420291189600_b5","doi-asserted-by":"crossref","unstructured":"Clarke, C., Craswell, N. and Soboroff, I. (2005), \u201cOverview of the TREC 2004 Terabyte Track\u201d, in Voorhees, E. and Buckland, L. (Eds), NIST Special Publication 500\u2010261: The Eleventh Text REtrieval Conference, available at: http:\/\/trec.nist.gov\/pubs\/trec13\/t13_proceedings.html.","DOI":"10.6028\/NIST.SP.500-261.terabyte-overview"},{"key":"key2022020420291189600_b6","unstructured":"Craswell, N. and Hawking, D. (2005), \u201cOverview of the TREC 2004 Web Track\u201d, in Voorhees, E. and Buckland, L. (Eds), NIST Special Publication 500\u2010261: The Eleventh Text REtrieval Conference, available at: http:\/\/trec.nist.gov\/pubs\/trec13\/t13_proceedings.html (accessed 8 March 2007)."},{"key":"key2022020420291189600_b7","doi-asserted-by":"crossref","unstructured":"Gordon, M. and Pathak, P. (1999), \u201cFinding information on the World Wide Web: the retrieval effectiveness of search engines\u201d, Information Processing and Management, Vol. 35 No. 2, pp. 141\u201080.","DOI":"10.1016\/S0306-4573(98)00041-7"},{"key":"key2022020420291189600_b8","unstructured":"Hawking, D. (2001), \u201cOverview of the TREC\u20109 Web Track\u201d, in Voorhees, E. and Harman, D. (Eds), NIST Special Publication 500\u2010249: The Ninth Text REtrieval Conference, available at: http:\/\/trec.nist.gov\/pubs\/trec9\/t9_proceedings.html."},{"key":"key2022020420291189600_b9","unstructured":"Hawking, D. and Craswell, N. (2002), \u201cOverview of the TREC 2001 Web Track\u201d, in Voorhees, E. and Harman, D. (Eds), NIST Special Publication 500\u2010250: The Tenth Text REtrieval Conference, available at: http:\/\/trec.nist.gov\/pubs\/trec10\/t10_proceedings.html."},{"key":"key2022020420291189600_b10","unstructured":"Hawking, D. and Craswell, N. (2003), \u201cOverview of the TREC 2002 Web Track\u201d, in Voorhees, E. and Buckland, L. (Eds), NIST Special Publication 500\u2010251: The Eleventh Text REtrieval Conference, pp. 78\u201092, available at: http:\/\/trec.nist.gov\/pubs\/trec11\/t11_proceedings.html."},{"key":"key2022020420291189600_b14","doi-asserted-by":"crossref","unstructured":"Hawking, D. and Thistlewaite, P. (1998), \u201cOverview of the TREC\u20106 Very Large Collection Track\u201d, in Voorhees, E. and Harman, D. (Eds), NIST Special Publication 500\u2010242: The Sixth Text REtrieval Conference, pp. 93\u2010106, available at: http:\/\/trec.nist.gov\/pubs\/trec6\/t6_proceedings.html.","DOI":"10.6028\/NIST.SP.500-240.vlc-overview"},{"key":"key2022020420291189600_b11","unstructured":"Hawking, D., Craswell, N. and Thistlewaite, P. (1999), \u201cOverview of the TREC\u20107 Very Large Collection Track\u201d, in Voorhees, E. and Harman, D. (Eds), NIST Special Publication 500\u2010242: The Seventh Text REtrieval Conference, pp. 91\u2010104, available at: http:\/\/trec.nist.gov\/pubs\/trec7\/t7_proceedings.html."},{"key":"key2022020420291189600_b15","doi-asserted-by":"crossref","unstructured":"Hawking, D., Voorhees, E. and Craswell, N. (2000), \u201cOverview of the TREC\u20108 Web Track\u201d, in Voorhees, E. and Harman, D. (Eds), NIST Special Publication 500\u2010246: The Eighth Text REtrieval Conference, pp. 131\u201050, available at: http:\/\/trec.nist.gov\/pubs\/trec8\/t8_proceedings.html.","DOI":"10.6028\/NIST.SP.500-246"},{"key":"key2022020420291189600_b12","doi-asserted-by":"crossref","unstructured":"Hawking, D., Craswell, N., Bailey, P. and Griffiths, K. (2001), \u201cMeasuring search engine quality\u201d, Information Retrieval, Vol. 4 No. 1, pp. 33\u201059.","DOI":"10.1023\/A:1011468107287"},{"key":"key2022020420291189600_b13","unstructured":"Hawking, D., Craswell, N., Wilkinson, R. and Wu, M. (2004), \u201cOverview of the TREC 2003 Web Track\u201d, in Voorhees, E. and Buckland, L. (Eds), NIST Special Publication 500\u2010255: The Twelfth Text REtrieval Conference, pp. 78\u201094, available at: http:\/\/trec.nist.gov\/pubs\/trec12\/t12_proceedings.html."},{"key":"key2022020420291189600_b16","doi-asserted-by":"crossref","unstructured":"Henzinger, M., Motwani, R. and Silverstein, C. (2002), \u201cChallenges in web search engines\u201d, SIGIR Forum, Vol. 36 No. 2, pp. 11\u201022.","DOI":"10.1145\/792550.792553"},{"key":"key2022020420291189600_b17","doi-asserted-by":"crossref","unstructured":"Hull, D. (1993), \u201cUsing statistical testing in the evaluation of retrieval experiments\u201d, in Korfage, R., Rasmussen, E. and Willett, P. (Eds), Proccedings of the 16th Annual ACM Conference on Research and Development in Information Retrieval, SIGIR\u201993, Pittsburgh, PA, pp. 329\u201038.","DOI":"10.1145\/160688.160758"},{"key":"key2022020420291189600_b18","doi-asserted-by":"crossref","unstructured":"Jansen, B., Spink, A., Bateman, J. and Saracevic, T. (1998), \u201cReal life information retrieval: a study of user queries on the Web\u201d, SIGIR Forum, Vol. 32 No. 1, pp. 5\u201017.","DOI":"10.1145\/281250.281253"},{"key":"key2022020420291189600_b19","doi-asserted-by":"crossref","unstructured":"Leighton, H.V. and Srivastava, J. (1999), \u201cFirst 20 precision among world wide web search services (search engines)\u201d, Journal of the American Society for Information Science, Vol. 50 No. 10, pp. 870\u201081.","DOI":"10.1002\/(SICI)1097-4571(1999)50:10<870::AID-ASI4>3.0.CO;2-G"},{"key":"key2022020420291189600_b20","doi-asserted-by":"crossref","unstructured":"Oppenheim, C., Morris, A., McKnight, C. and Lowley, S. (2000), \u201cThe evaluation of WWW search engines\u201d, Journal of Documentation, Vol. 56 No. 2, pp. 190\u2010211.","DOI":"10.1108\/00220410010803810"},{"key":"key2022020420291189600_b21","unstructured":"Rowntree, D. (1981), Statistics Without Tears: An Introduction for Non\u2010Mathematicians, Penguin, London."},{"key":"key2022020420291189600_b22","doi-asserted-by":"crossref","unstructured":"Sanderson, M. and Zobel, J. (2005), \u201cInformation retrieval systems evaluation: effort, sensitivity and reliability\u201d, Proceedings of the 28th Annual International ACM Conference on Research and Development in Information Retrieval, SIGIR 2005, Salvador, pp. 162\u20109.","DOI":"10.1145\/1076034.1076064"},{"key":"key2022020420291189600_b23","doi-asserted-by":"crossref","unstructured":"Silverstein, S., Henzinger, M. and Marais, H. (1999), \u201cand Moricz Analysis of a very large web search engine query log\u201d, SIGIR Forum, Vol. 33 No. 1, pp. 6\u201012.","DOI":"10.1145\/331403.331405"},{"key":"key2022020420291189600_b24","unstructured":"Van Rijsbergen, C. (1979), \u201cInformation Retrieval\u201d, 2nd ed., Butterworths, London, available at: www.dcs.gla.ac.uk\/Keith\/Preface.html (accessed 7 March 2007)."},{"key":"key2022020420291189600_b25","doi-asserted-by":"crossref","unstructured":"Vaughan, L. (2004), \u201cNew measurements for search engine evaluation proposed and tested\u201d, Information Processing and Management, Vol. 40 No. 4, pp. 677\u201091.","DOI":"10.1016\/S0306-4573(03)00043-8"},{"key":"key2022020420291189600_b26","doi-asserted-by":"crossref","unstructured":"Voorhees, E. and Harman, D. (2000), \u201cOverview of the Eighth Text REtrieval Conference\u201d, in Voorhees, E. and Harman, D. (Eds), NIST Special Publication 500\u2010246: The Eighth Text REtrieval Conference, pp. 1\u201024, available at: http:\/\/trec.nist.gov\/pubs\/trec8\/t8_proceedings.htm.","DOI":"10.6028\/NIST.SP.500-246"},{"key":"key2022020420291189600_b27","unstructured":"Wu, G. and Li, J. (1999), \u201cComparing web search performance in searching consumer health information: evaluation and recommendations\u201d, Bulletin of the Medical Library Association, Vol. 87 No. 4, pp. 456\u201061."},{"key":"key2022020420291189600_b28","unstructured":"Zeldman, J. (2001), Taking Your Talent to the Web: Making the Transition from Graphic Design to Web Design, Waite Group Press, Boston, MA."}],"container-title":["Aslib Proceedings"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/www.emeraldinsight.com\/doi\/full-xml\/10.1108\/00012530710817573","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/00012530710817573\/full\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/00012530710817573\/full\/html","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,24]],"date-time":"2025-07-24T11:36:51Z","timestamp":1753357011000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.emerald.com\/ajim\/article\/59\/4-5\/352-366\/126468"}},"subtitle":[],"editor":[{"given":"David","family":"Bawden","sequence":"first","affiliation":[]}],"short-title":[],"issued":{"date-parts":[[2007,7,12]]},"references-count":28,"journal-issue":{"issue":"4\/5","published-print":{"date-parts":[[2007,7,12]]}},"alternative-id":["10.1108\/00012530710817573"],"URL":"https:\/\/doi.org\/10.1108\/00012530710817573","relation":{},"ISSN":["0001-253X"],"issn-type":[{"type":"print","value":"0001-253X"}],"subject":[],"published":{"date-parts":[[2007,7,12]]}}}