{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,9]],"date-time":"2025-10-09T20:58:19Z","timestamp":1760043499717,"version":"3.41.2"},"reference-count":34,"publisher":"Emerald","issue":"6","license":[{"start":{"date-parts":[[2011,11,29]],"date-time":"2011-11-29T00:00:00Z","timestamp":1322524800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.emerald.com\/insight\/site-policies"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2011,11,29]]},"abstract":"<jats:sec><jats:title content-type=\"abstract-heading\">Purpose<\/jats:title><jats:p>The purpose of this study is to develop an evaluation criteria system for search engines using the Delphi method and the analytic hierarchy process, and to conduct an empirical study on Chinese search engines to verify it. This is because the evaluation criteria in current research rarely considers expert opinions and experience, and seldom calculates the weight of each criterion by quantitative methods.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Design\/methodology\/approach<\/jats:title><jats:p>Expert investigation of the web\u2010based Delphi method was employed to develop the evaluation criteria, and the analytic hierarchy process based on an exponential scale was adopted to calculate the weight of each criterion. An empirical study was designed to test the evaluation criteria system. A total of 12 study participants were invited to evaluate six popular Chinese search engines by means of e\u2010mail questionnaires. All the calculation processes were executed automatically by Java programming.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Findings<\/jats:title><jats:p>The indexing structure and search method were the core criteria for evaluating the search engines. Google (using simplified Chinese characters) achieved the best performance among the six typical Chinese search engines.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Originality\/value<\/jats:title><jats:p>The criteria for search engine evaluation were determined and weighted by the web\u2010based Delphi method and the analytic hierarchy process based on an exponential scale.<\/jats:p><\/jats:sec>","DOI":"10.1108\/14684521111193210","type":"journal-article","created":{"date-parts":[[2011,11,26]],"date-time":"2011-11-26T07:03:58Z","timestamp":1322291038000},"page":"942-956","source":"Crossref","is-referenced-by-count":16,"title":["Using a Delphi method and the analytic hierarchy process to evaluate Chinese search engines"],"prefix":"10.1108","volume":"35","author":[{"given":"Qinghua","family":"Zhu","sequence":"first","affiliation":[]},{"given":"Jia","family":"Tina Du","sequence":"additional","affiliation":[]},{"given":"Fei","family":"Meng","sequence":"additional","affiliation":[]},{"given":"Kewen","family":"Wu","sequence":"additional","affiliation":[]},{"given":"Xiaoling","family":"Sun","sequence":"additional","affiliation":[]}],"member":"140","reference":[{"key":"key2022012220275396900_b1","doi-asserted-by":"crossref","unstructured":"Bar\u2010Ilan, J. (2002), \u201cMethods for measuring search engine performance over time\u201d, Journal of the American Society for Information Science and Technology, Vol. 53 No. 4, pp. 308\u201019.","DOI":"10.1002\/asi.10047"},{"key":"key2022012220275396900_b2","doi-asserted-by":"crossref","unstructured":"Bar\u2010Ilan, J. and Gutman, T. (2005), \u201cHow do search engines respond to non\u2010English queries?\u201d, Journal of Information Science, Vol. 31 No. 1, pp. 13\u201028.","DOI":"10.1177\/0165551505049255"},{"key":"key2022012220275396900_b3","unstructured":"Bell, S. (1998), \u201cGuidelines for the development of methodologies to evaluate internet search engines: please do a couple of searches and send the feedback to me\u201d, MSc thesis, University of Strathclyde, Glasgow."},{"key":"key2022012220275396900_b4","doi-asserted-by":"crossref","unstructured":"Can, F., Nuray, R. and Sevdik, A.B. (2004), \u201cAutomatic performance evaluation of web search engines\u201d, Information Processing and Management, Vol. 40 No. 3, pp. 495\u2010514.","DOI":"10.1016\/S0306-4573(03)00040-2"},{"key":"key2022012220275396900_b5","doi-asserted-by":"crossref","unstructured":"Chang, Y.K., Morales\u2010Arroyo, M.A. and Spink, A. (2010), \u201cMultimedia search capabilities of Chinese language search engines\u201d, Information Processing and Management, Vol. 46 No. 3, pp. 308\u201019.","DOI":"10.1016\/j.ipm.2009.07.003"},{"key":"key2022012220275396900_b6","doi-asserted-by":"crossref","unstructured":"Chau, M., Fang, X. and Yang, C.C. (2007), \u201cWeb searching in Chinese: a study of a search engine in Hong Kong\u201d, Journal of the American Society for Information Science and Technology, Vol. 58 No. 7, pp. 1044\u201054.","DOI":"10.1002\/asi.20592"},{"key":"key2022012220275396900_b7","unstructured":"Chu, H.T. and Rosenthal, M. (1996), \u201cSearch engines for the world wide web: a comparative study and evaluation methodology\u201d, Proceedings of the 59th Annual Meeting of the American Society for Information Science, Information Today, Medford, NJ, pp. 127\u201035."},{"key":"key2022012220275396900_b8","unstructured":"Cleverdon, C.W. (1964), Evaluation of Operational Information Retrieval Systems. Part 1: Identification of Criteria, College of Aeronautics, Cranfield."},{"key":"key2022012220275396900_b9","unstructured":"CNNIC (2010), Statistical report of the development of Chinese internet, available at: www.cnnic.net.cn\/uploadfiles\/pdf\/2010\/7\/15\/100708.pdf (accessed 30 August 2010)."},{"key":"key2022012220275396900_b10","unstructured":"CNNIC (2011), Statistical report of the development of Chinese internet, available at: www.cnnic.net.cn\/dtygg\/dtgg\/201101\/P020110119328960192287.pdf (accessed 19 January 2011)."},{"key":"key2022012220275396900_b11","doi-asserted-by":"crossref","unstructured":"ComScore (2010), Data passport first half 2010, available at: www2.comscore.com\/l\/1552\/\u2010comScoreDataPassport\u20101H10\u2010pdf\/OTF4X (accessed 7 January 2011).","DOI":"10.1016\/S1350-4789(10)70025-8"},{"key":"key2022012220275396900_b12","doi-asserted-by":"crossref","unstructured":"Crudge, S.E. and Johnson, F.C. (2004), \u201cUsing the information seeker to elicit construct models for search engine evaluation\u201d, Journal of the American Society for Information Science and Technology, Vol. 55 No. 9, pp. 794\u2010806.","DOI":"10.1002\/asi.20023"},{"key":"key2022012220275396900_b13","doi-asserted-by":"crossref","unstructured":"Dong, X. and Su, L.T. (1997), \u201cSearch engines on the world wide web and information retrieval from the internet: a review and evaluation\u201d, Online and CDROM Review, Vol. 21 No. 2, pp. 67\u201082.","DOI":"10.1108\/eb024608"},{"key":"key2022012220275396900_b14","doi-asserted-by":"crossref","unstructured":"Hawking, D., Craswell, N., Bailey, P. and Griffiths, K. (2001), \u201cMeasuring search engine quality\u201d, Information Retrieval, Vol. 4 No. 1, pp. 33\u201059.","DOI":"10.1023\/A:1011468107287"},{"key":"key2022012220275396900_b15","doi-asserted-by":"crossref","unstructured":"Kochtanek, T.R. and Hein, K.K. (1999), \u201cDelphi study of digital libraries\u201d, Information Processing & Management, Vol. 35 No. 3, pp. 245\u201054.","DOI":"10.1016\/S0306-4573(98)00060-0"},{"key":"key2022012220275396900_b16","unstructured":"Lawrence, S. (2000), \u201cContext in web search\u201d, IEEE Data Engineering Bulletin, Vol. 23 No. 3, pp. 25\u201032."},{"key":"key2022012220275396900_b17","doi-asserted-by":"crossref","unstructured":"Leighton, H.V. and Srivastava, J. (1999), \u201cFirst twenty precision among world wide web search services (search engines): AltaVista, Excite, Hotbot, Infoseek, Lycos\u201d, Journal of the American Society for Information Science, Vol. 50 No. 10, pp. 870\u201081.","DOI":"10.1002\/(SICI)1097-4571(1999)50:10<870::AID-ASI4>3.0.CO;2-G"},{"key":"key2022012220275396900_b18","doi-asserted-by":"crossref","unstructured":"Lewandowski, D. (2008), \u201cThe retrieval effectiveness of web search engines: considering results descriptions\u201d, Journal of Documentation, Vol. 64 No. 6, pp. 915\u201037.","DOI":"10.1108\/00220410810912451"},{"key":"key2022012220275396900_b19","doi-asserted-by":"crossref","unstructured":"Li, K.F., Wang, Y.L., Nishio, S.J. and Yu, W. (2005), \u201cA formal approach to evaluate and compare internet search engines: a case study on searching the Chinese web\u201d, in Zhang, Y. (Ed.), Web Technologies Research and Development \u2013 APWEB 2005, Vol. 2005, Springer\u2010Verlag, Berlin, pp. 195\u2010206.","DOI":"10.1007\/978-3-540-31849-1_20"},{"key":"key2022012220275396900_b20","unstructured":"Lv, Y.J. (2001), \u201cResearch on scale system evaluation\u201d, AHP, in System Engineering Society of China (Ed.), Theory and Methods for Decision Science, Ocean Press, Beijing, pp. 50\u20108 (in Chinese)."},{"key":"key2022012220275396900_b21","unstructured":"Lv, Y.J., Zhang, W. and Ceng, X.L. (2003), \u201cComparative study on exponential scale and 1\u20109 scale\u201d, Chinese Journal of Engineering Mathematics, Vol. 20 No. 8, pp. 77\u20108 (in Chinese)."},{"key":"key2022012220275396900_b22","doi-asserted-by":"crossref","unstructured":"Mahdi, I.M., Riley, M.J., Fereig, S.M. and Alex, A.P. (1993), \u201cA multi\u2010criteria approach to contractor selection\u201d, Engineering, Construction and Architectural Management, Vol. 9 No. 1, pp. 29\u201037.","DOI":"10.1108\/eb021204"},{"key":"key2022012220275396900_b23","doi-asserted-by":"crossref","unstructured":"Moukdad, H. and Large, A. (2001), \u201cInformation retrieval from full\u2010text Arabic databases: can search engines designed for English do the job?\u201d, Libri, Vol. 51 No. 2, pp. 63\u201074.","DOI":"10.1515\/LIBR.2001.63"},{"key":"key2022012220275396900_b24","doi-asserted-by":"crossref","unstructured":"Ogilvie, P. and Callan, J. (2003), \u201cCombining document representations for known\u2010item search\u201d, Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, New York, NY, pp. 143\u201050.","DOI":"10.1145\/860435.860463"},{"key":"key2022012220275396900_b25","doi-asserted-by":"crossref","unstructured":"Saracevic, T. (1995), \u201cEvaluation of evaluation in information retrieval\u201d, Proceedings of the 18th ACM\u2010SIGIR International Conference on Research and Development in Information Retrieval, ACM, New York, NY, pp. 138\u201046.","DOI":"10.1145\/215206.215351"},{"key":"key2022012220275396900_b26","doi-asserted-by":"crossref","unstructured":"Saaty, T.L. (1980), The Analytic Hierarchy Process, McGraw\u2010Hill, New York, NY.","DOI":"10.21236\/ADA214804"},{"key":"key2022012220275396900_b27","doi-asserted-by":"crossref","unstructured":"Shon, T.H. and Swatman, P.M.C. (1998), \u201cIdentifying effectiveness criteria for internet payment systems\u201d, Internet Research, Vol. 8 No. 3, pp. 202\u201018.","DOI":"10.1108\/10662249810217759"},{"key":"key2022012220275396900_b28","doi-asserted-by":"crossref","unstructured":"Spink, A. (2002), \u201cA user\u2010centered approach to evaluating human interaction with web search engines: an exploratory study\u201d, Information Processing & Management, Vol. 38 No. 3, pp. 401\u201026.","DOI":"10.1016\/S0306-4573(01)00036-X"},{"key":"key2022012220275396900_b29","doi-asserted-by":"crossref","unstructured":"Su, L.T. (2003a), \u201cA comprehensive and systematic model of user evaluation of web search engines: I. Theory and background\u201d, Journal of the American Society for Information Science and Technology, Vol. 54 No. 13, pp. 1175\u201092.","DOI":"10.1002\/asi.10303"},{"key":"key2022012220275396900_b30","doi-asserted-by":"crossref","unstructured":"Su, L.T. (2003b), \u201cA comprehensive and systematic model of user evaluation of web search engines: II. An evaluation by undergraduates\u201d, Journal of the American Society for Information Science and Technology, Vol. 54 No. 13, pp. 1193\u2010223.","DOI":"10.1002\/asi.10334"},{"key":"key2022012220275396900_b31","unstructured":"Tomaiuolo, N.G. and Packer, J.G. (1996), \u201cAn analysis of internet search engines \u2013 assessment of over 200 search queries\u201d, Computers in Libraries, Vol. 16 No. 6, pp. 58\u201062."},{"key":"key2022012220275396900_b32","doi-asserted-by":"crossref","unstructured":"Vaughan, L. (2004), \u201cNew measurements for search engine evaluation proposed and tested\u201d, Information Processing and Management, Vol. 40 No. 4, pp. 677\u201091.","DOI":"10.1016\/S0306-4573(03)00043-8"},{"key":"key2022012220275396900_b33","doi-asserted-by":"crossref","unstructured":"Yadav, S.B. (2010), \u201cA conceptual model for user\u2010centered quality information retrieval on the world wide web\u201d, Journal of Intelligent Information Systems, Vol. 35 No. 1, pp. 91\u2010121.","DOI":"10.1007\/s10844-009-0090-y"},{"key":"key2022012220275396900_b34","unstructured":"Yong, Y.W., Keng, K.A. and Leng, T.L. (1989), \u201cA Delphi forecast for the Singapore tourism industry: future scenario and marketing implications\u201d, European Journal of Marketing, Vol. 23 No. 11, pp. 15\u201026."}],"container-title":["Online Information Review"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/www.emeraldinsight.com\/doi\/full-xml\/10.1108\/14684521111193210","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/14684521111193210\/full\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/14684521111193210\/full\/html","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,25]],"date-time":"2025-07-25T00:41:47Z","timestamp":1753404107000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.emerald.com\/oir\/article\/35\/6\/942-956\/314113"}},"subtitle":["A case study on Chinese search engines"],"editor":[{"given":"Dirk","family":"Lewandowski","sequence":"first","affiliation":[]}],"short-title":[],"issued":{"date-parts":[[2011,11,29]]},"references-count":34,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2011,11,29]]}},"alternative-id":["10.1108\/14684521111193210"],"URL":"https:\/\/doi.org\/10.1108\/14684521111193210","relation":{},"ISSN":["1468-4527"],"issn-type":[{"type":"print","value":"1468-4527"}],"subject":[],"published":{"date-parts":[[2011,11,29]]}}}