{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T18:48:33Z","timestamp":1760122113537,"version":"3.41.2"},"reference-count":36,"publisher":"Emerald","issue":"6","license":[{"start":{"date-parts":[[2011,11,29]],"date-time":"2011-11-29T00:00:00Z","timestamp":1322524800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.emerald.com\/insight\/site-policies"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2011,11,29]]},"abstract":"<jats:sec><jats:title content-type=\"abstract-heading\">Purpose<\/jats:title><jats:p>The intent of this work is to evaluate several generalist and health\u2010specific search engines for retrieval of health information by consumers: to compare the retrieval effectiveness of these engines for different types of clinical queries, medical specialties and condition severity; and to compare the use of evaluation metrics for binary relevance scales and for graded ones.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Design\/methodology\/approach<\/jats:title><jats:p>The authors conducted a study in which users evaluated the relevance of documents retrieved by four search engines for two different health information needs. Users could choose between generalist (Bing, Google, Sapo and Yahoo!) and health\u2010specific (MedlinePlus, SapoSa\u00fade and WebMD) search engines. The authors then analysed the differences between search engines and groups of information needs with six different measures: graded average precision (gap), average precision (ap), gap@5, gap@10, ap@5 and ap@10.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Findings<\/jats:title><jats:p>The results show that generalist web search engines surpass the precision of health\u2010specific engines. Google has the best performance, mainly in the top ten results. It was found that information needs associated with severe conditions are associated with higher precision, as are overview and psychiatry questions.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Originality\/value<\/jats:title><jats:p>The study is one of the first to use a recently proposed measure to evaluate the effectiveness of retrieval systems with graded relevance scales. It includes tasks from several medical specialties, types of clinical questions and different levels of severity which, to the best of the authors' knowledge, has not been done before. Moreover, users have considerable involvement in the experiment. The results help in understanding how search engines differ in their responses to health information needs, what types of online health information are more common on the web and how to improve this type of search.<\/jats:p><\/jats:sec>","DOI":"10.1108\/14684521111193175","type":"journal-article","created":{"date-parts":[[2011,11,26]],"date-time":"2011-11-26T07:09:01Z","timestamp":1322291341000},"page":"869-892","source":"Crossref","is-referenced-by-count":10,"title":["Comparative evaluation of web search engines in health information retrieval"],"prefix":"10.1108","volume":"35","author":[{"given":"Carla","family":"Teixeira Lopes","sequence":"first","affiliation":[]},{"given":"Cristina","family":"Ribeiro","sequence":"additional","affiliation":[]}],"member":"140","reference":[{"key":"key2022021220120529100_b1","doi-asserted-by":"crossref","unstructured":"Bin, L. and Lun, K. (2001), \u201cThe retrieval effectiveness of medical information on the web\u201d, International Journal of Medical Informatics, Vol. 62 Nos 2\u20103, pp. 155\u201063.","DOI":"10.1016\/S1386-5056(01)00159-9"},{"key":"key2022021220120529100_b2","unstructured":"Borlund, P. (2003), \u201cThe IIR evaluation model: a framework for evaluation of interactive information retrieval systems\u201d, Information Research, Vol. 8 No. 3, available at: http:\/\/informationr.net\/ir\/8\u20103\/paper152.html (accessed 23 August 2011)."},{"key":"key2022021220120529100_b3","unstructured":"Chu, H. and Rosenthal, M. (1996), \u201cSearch engines for the World Wide Web: a comparative study and evaluation methodology\u201d, Proceedings of the 59th American Society for Information Science Annual Meeting, Vol. 33, Information Today, Medford, NJ, pp. 127\u201035."},{"key":"key2022021220120529100_b4","unstructured":"Eysenbach, G. and Thomson, M. (2007), \u201cThe FA4CT algorithm: a new model and tool for consumers to assess and filter health information on the internet\u201d, MEDINFO 2007 \u2013 Proceedings of the 12th World Congress on Health (Medical) Informatics \u2013 Building Sustainable Health Systems, Studies in Health Technology and Informatics, IOS Press, Amsterdam, pp. 142\u20106."},{"key":"key2022021220120529100_b5","unstructured":"Fox, S. (2006), Online health search 2006, Pew Internet & American Life Project, 29 October, available at: www.pewinternet.org\/\u223c\/media\/\/Files\/Reports\/2006\/PIP_Online_Health_2006.pdf.pdf (accessed 22 August 2011)."},{"key":"key2022021220120529100_b6","unstructured":"Fox, S. and Jones, S. (2009), The social life of health information, Pew Internet & American Life Project, 11 June, available at: www.pewinternet.org\/\u223c\/media\/\/Files\/Reports\/2009\/PIP_Health_2009.pdf (accessed 22 August 2011)."},{"key":"key2022021220120529100_b7","doi-asserted-by":"crossref","unstructured":"Gordon, M. and Pathak, P. (1999), \u201cFinding information on the World Wide Web: the retrieval effectiveness of search engines\u201d, Information Processing & Management, Vol. 35 No. 2, pp. 141\u201080.","DOI":"10.1016\/S0306-4573(98)00041-7"},{"key":"key2022021220120529100_b8","doi-asserted-by":"crossref","unstructured":"Graber, M.A., Bergus, G.R. and York, C. (1999), \u201cUsing the World Wide Web to answer clinical questions: how efficient are different methods of information retrieval?\u201d, The Journal of Family Practice, Vol. 48 No. 7, pp. 520\u20104.","DOI":"10.2307\/585676"},{"key":"key2022021220120529100_b9","unstructured":"Gwizdka, J. and Chignell, M.H. (1999), Towards Information Retrieval Measures for Evaluation of Web Search Engines, Interactive Media Lab, University of Toronto, Toronto."},{"key":"key2022021220120529100_b10","doi-asserted-by":"crossref","unstructured":"Hawking, D., Craswell, N., Bailey, P. and Griffiths, K. (2001), \u201cMeasuring search engine quality\u201d, Information Retrieval, Vol. 4 No. 1, pp. 33\u201059.","DOI":"10.1023\/A:1011468107287"},{"key":"key2022021220120529100_b11","doi-asserted-by":"crossref","unstructured":"Hersh, W. (2008), Information Retrieval: A Health and Biomedical Perspective, 3rd ed., Health Informatics Series, Springer, New York, NY.","DOI":"10.1007\/978-0-387-78703-9_2"},{"key":"key2022021220120529100_b12","doi-asserted-by":"crossref","unstructured":"Ilic, D., Bessell, T.L., Silagy, C.A. and Green, S. (2003), \u201cSpecialized medical search\u2010engines are no better than general search\u2010engines in sourcing consumer information about androgen deficiency\u201d, Human Reproduction, Vol. 18 No. 3, pp. 557\u201061.","DOI":"10.1093\/humrep\/deg154"},{"key":"key2022021220120529100_b13","unstructured":"Ingwersen, P. (2009), \u201cThe user in interactive information retrieval evaluation\u201d, presentation at European Summer School in Information Retrieval, Padua."},{"key":"key2022021220120529100_b14","doi-asserted-by":"crossref","unstructured":"Ingwersen, P. and J\u00e4rvelin, K. (2005), The Turn: Integration of Information Seeking and Retrieval in Context, Information Retrieval Series, Springer, Dordrecht.","DOI":"10.1145\/1113343.1113351"},{"key":"key2022021220120529100_b15","doi-asserted-by":"crossref","unstructured":"J\u00e4rvelin, K. and Kek\u00e4l\u00e4inen, J. (2002), \u201cCumulated gain\u2010based evaluation of IR techniques\u201d, ACM Transactions on Information Systems, Vol. 20 No. 4, pp. 422\u201046.","DOI":"10.1145\/582415.582418"},{"key":"key2022021220120529100_b16","doi-asserted-by":"crossref","unstructured":"Johnson, P., Chen, J., Eng, J., Makary, M. and Fishman, E. (2008), \u201cA comparison of world wide web resources for identifying medical information\u201d, Academic Radiology, Vol. 15 No. 9, pp. 1165\u201072.","DOI":"10.1016\/j.acra.2008.02.010"},{"key":"key2022021220120529100_b17","doi-asserted-by":"crossref","unstructured":"Jones, D. and Timm, D. (2008), \u201cConsumer health search engines comparison\u201d, Journal of Hospital Librarianship, Vol. 8 No. 4, pp. 418\u201032.","DOI":"10.1080\/15323260802391936"},{"key":"key2022021220120529100_b18","unstructured":"Knight, D., Holt, A. and Warren, J. (2009), \u201cSearch engines: a study of nine search engines in four categories\u201d, Journal of Health Informatics in Developing Countries, Vol. 3 No. 1, pp. 1\u20108."},{"key":"key2022021220120529100_b19","unstructured":"Kumar, A. (2005), \u201cHealth search tool evaluation\u201d, Masters thesis in biomedical informatics, Oregon Health and Science University, Portland, OR."},{"key":"key2022021220120529100_b20","doi-asserted-by":"crossref","unstructured":"Lewandowski, D. (2008), \u201cThe retrieval effectiveness of web search engines: considering results descriptions\u201d, Journal of Documentation, Vol. 64 No. 6, pp. 915\u201037.","DOI":"10.1108\/00220410810912451"},{"key":"key2022021220120529100_b21","unstructured":"Lopes, C. and Ribeiro, C. (2010a), \u201cContext in health information retrieval: what and where\u201d, Proceedings of the Fourth Workshop on Human\u2010Computer Interaction and Information Retrieval, available at: http:\/\/research.microsoft.com\/en\u2010us\/um\/people\/ryenw\/hcir2010\/docs\/papers\/lopes_pp9.pdf (accessed 22 August 2011)."},{"key":"key2022021220120529100_b22","doi-asserted-by":"crossref","unstructured":"Lopes, C. and Ribeiro, C. (2010b), \u201cContext effect on query formulation and subjective relevance in health searches\u201d, IIiX'10: Proceedings of the Third Symposium on Information Interaction in Context, ACM, New York, NY, pp. 205\u201014.","DOI":"10.1145\/1840784.1840815"},{"key":"key2022021220120529100_b23","doi-asserted-by":"crossref","unstructured":"Manning, C., Raghavan, P. and Sch\u00fctze, H. (2008), Introduction to Information Retrieval, Cambridge University Press, Cambridge.","DOI":"10.1017\/CBO9780511809071"},{"key":"key2022021220120529100_b24","unstructured":"Marketing Charts (2010), Top 10 health & medical information websites \u2013 July 2010, available at: www.marketingcharts.com\/interactive\/top\u201010\u2010health\u2010medical\u2010information\u2010websites\u2010july\u20102010\u201013919\/ (accessed 16 August 2011)."},{"key":"key2022021220120529100_b25","unstructured":"Market Share (2011), Search engine market share, available at: http:\/\/marketshare.hitslink.com\/search\u2010engine\u2010market\u2010share.aspx?qprid=4 (accessed 19 August 2011)."},{"key":"key2022021220120529100_b26","doi-asserted-by":"crossref","unstructured":"Robertson, S., Kanoulas, E. and Yilmaz, E. (2010), \u201cExtending average precision to graded relevance judgments\u201d, Proceeding of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, Geneva, ACM, New York, NY, pp. 603\u201010.","DOI":"10.1145\/1835449.1835550"},{"key":"key2022021220120529100_b27","doi-asserted-by":"crossref","unstructured":"Sanderson, M. (2010), \u201cTest collection based evaluation of information retrieval systems\u201d, Foundations and Trends in Information Retrieval, Vol. 4 No. 4, pp. 247\u2010375.","DOI":"10.1561\/1500000009"},{"key":"key2022021220120529100_b28","doi-asserted-by":"crossref","unstructured":"Shang, Y. and Li, L. (2002), \u201cPrecision evaluation of search engines\u201d, World Wide Web, Vol. 5 No. 2, pp. 159\u201073.","DOI":"10.1023\/A:1019679624079"},{"key":"key2022021220120529100_b29","doi-asserted-by":"crossref","unstructured":"Su, L. (2003a), \u201cA comprehensive and systematic model of user evaluation of web search engines: I. Theory and background\u201d, Journal of the American Society for Information Science and Technology, Vol. 54 No. 13, pp. 1175\u201092.","DOI":"10.1002\/asi.10303"},{"key":"key2022021220120529100_b30","doi-asserted-by":"crossref","unstructured":"Su, L. (2003b), \u201cA comprehensive and systematic model of user evaluation of web search engines: II. An evaluation by undergraduates\u201d, Journal of the American Society for Information Science and Technology, Vol. 54 No. 13, pp. 1193\u2010223.","DOI":"10.1002\/asi.10334"},{"key":"key2022021220120529100_b31","doi-asserted-by":"crossref","unstructured":"Tang, M.C. and Sun, Y. (2003), \u201cEvaluation of web\u2010based search engines using user\u2010effort measures\u201d, Libres: Library and Information Science Research Electronic Journal, Vol. 13 No. 2, available at: http:\/\/libres.curtin.edu.au\/libres13n2\/index.htm (accessed 22 August 2011).","DOI":"10.32655\/LIBRES.2003.2.1"},{"key":"key2022021220120529100_b32","doi-asserted-by":"crossref","unstructured":"Tang, T., Craswell, N., Hawking, D., Griffiths, K. and Christensen, H. (2006), \u201cQuality and relevance of domain\u2010specific search: a case study in mental health\u201d, Information Retrieval, Vol. 9 No. 2, pp. 207\u201025.","DOI":"10.1007\/s10791-006-7150-5"},{"key":"key2022021220120529100_b33","doi-asserted-by":"crossref","unstructured":"Vaughan, L. (2004), \u201cNew measurements for search engine evaluation proposed and tested\u201d, Information Processing and Management, Vol. 40 No. 4, pp. 677\u201091.","DOI":"10.1016\/S0306-4573(03)00043-8"},{"key":"key2022021220120529100_b34","doi-asserted-by":"crossref","unstructured":"White, R. and Horvitz, E. (2009), \u201cCyberchondria: studies of the escalation of medical concerns in web search\u201d, ACM Transactions on Information Systems, Vol. 27 No. 4, pp. 23:1\u201023:37.","DOI":"10.1145\/1629096.1629101"},{"key":"key2022021220120529100_b35","unstructured":"Wu, G. and Li, J. (1999), \u201cComparing web search engine performance in searching consumer health information: evaluation and recommendations\u201d, Bulletin of the Medical Library Association, Vol. 87 No. 4, pp. 456\u201061."},{"key":"key2022021220120529100_b36","doi-asserted-by":"crossref","unstructured":"Yu, H. and Kaufman, D. (2007), A cognitive evaluation of four online search engines for answering definitional questions posed by physicians, Pacific Symposium on Biocomputing, 328\u201039, available at: http:\/\/psb.stanford.edu\/psb\u2010online\/proceedings\/psb07\/yu.pdf (accessed 22 August 2011).","DOI":"10.1142\/9789812772435_0031"}],"container-title":["Online Information Review"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/www.emeraldinsight.com\/doi\/full-xml\/10.1108\/14684521111193175","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/14684521111193175\/full\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/14684521111193175\/full\/html","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,25]],"date-time":"2025-07-25T00:41:46Z","timestamp":1753404106000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.emerald.com\/oir\/article\/35\/6\/869-892\/314109"}},"subtitle":[],"editor":[{"given":"Dirk","family":"Lewandowski","sequence":"first","affiliation":[]}],"short-title":[],"issued":{"date-parts":[[2011,11,29]]},"references-count":36,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2011,11,29]]}},"alternative-id":["10.1108\/14684521111193175"],"URL":"https:\/\/doi.org\/10.1108\/14684521111193175","relation":{},"ISSN":["1468-4527"],"issn-type":[{"type":"print","value":"1468-4527"}],"subject":[],"published":{"date-parts":[[2011,11,29]]}}}