{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,8]],"date-time":"2025-10-08T15:26:37Z","timestamp":1759937197080,"version":"3.41.2"},"reference-count":51,"publisher":"Emerald","issue":"1","license":[{"start":{"date-parts":[[2013,1,11]],"date-time":"2013-01-11T00:00:00Z","timestamp":1357862400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.emerald.com\/insight\/site-policies"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2013,1,11]]},"abstract":"<jats:sec><jats:title content-type=\"abstract-heading\">Purpose<\/jats:title><jats:p>The purpose of this study is to compare the evaluation of search result lists and documents, in particular evaluation criteria, elements, association between criteria and elements, pre\/post and evaluation activities, and the time spent on evaluation.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Design\/methodology\/approach<\/jats:title><jats:p>The study analyzed the data collected from 31 general users through prequestionnaires, think aloud protocols and logs, and post questionnaires. Types of evaluation criteria, elements, associations between criteria and elements, evaluation activities and their associated pre\/post activities, and time were analyzed based on open coding.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Findings<\/jats:title><jats:p>The study identifies the similarities and differences of list and document evaluation by analyzing 21 evaluation criteria applied, 13 evaluation elements examined, pre\/post and evaluation activities performed and time spent. In addition, the authors also explored the time spent in evaluating lists and documents for different types of tasks.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Research limitations\/implications<\/jats:title><jats:p>This study helps researchers understand the nature of list and document evaluation. Additionally, this study connects elements that participants examined to criteria they applied, and further reveals problems associated with the lack of integration between list and document evaluation. The findings of this study suggest more elements, especially at list level, be available to support users applying their evaluation criteria. Integration of list and document evaluation and integration of pre, evaluation and post evaluation activities for the interface design is the absolute solution for effective evaluation.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Originality\/value<\/jats:title><jats:p>This study fills a gap in current research in relation to the comparison of list and document evaluation.<\/jats:p><\/jats:sec>","DOI":"10.1108\/00220411311295324","type":"journal-article","created":{"date-parts":[[2013,1,10]],"date-time":"2013-01-10T15:18:03Z","timestamp":1357831083000},"page":"49-80","source":"Crossref","is-referenced-by-count":7,"title":["Search result list evaluation versus document evaluation: similarities and differences"],"prefix":"10.1108","volume":"69","author":[{"given":"Iris","family":"Xie","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"suffix":"III","given":"Edward","family":"Benoit","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"140","reference":[{"key":"key2022012320352936900_b1","doi-asserted-by":"crossref","unstructured":"Aula, A., Majaranta, P. and R\u00e4ih\u00e4, K.\u2010J. (2005), \u201cEye\u2010tracking reveals the personal styles for search result evaluation\u201d, in Costabile, M.F. and Patern\u00f2, F. (Eds), INTERACT 2005 IFIP TC13 International Conference, Lecture Notes in Computer Science, Vol. 3585, pp. 1058\u201061.","DOI":"10.1007\/11555261_104"},{"key":"key2022012320352936900_b2","doi-asserted-by":"crossref","unstructured":"Bade, D. (2007), \u201cRelevance ranking is not relevance ranking or, when the user is not the user, the search results are not search results\u201d, Online Information Review, Vol. 31 No. 6, pp. 831\u201044.","DOI":"10.1108\/14684520710841793"},{"key":"key2022012320352936900_b4","doi-asserted-by":"crossref","unstructured":"Bales, S. and Wang, P. (2005), \u201cConsolidating user relevance criteria: a meta\u2010ethnography of empirical studies\u201d, poster paper in Proceedings of the 68th ASIS&T Annual Meeting, Vol. 42 No. 1, available at: http:\/\/onlinelibrary.wiley.com\/doi\/10.1002\/meet.14504201277\/abstract.","DOI":"10.1002\/meet.14504201277"},{"key":"key2022012320352936900_b5","doi-asserted-by":"crossref","unstructured":"Barry, C.L. (1994), \u201cUser\u2010defined relevance criteria: an exploratory study\u201d, Journal of the American Society for Information Science, Vol. 45 No. 3, pp. 150\u20109.","DOI":"10.1002\/(SICI)1097-4571(199404)45:3<149::AID-ASI5>3.0.CO;2-J"},{"key":"key2022012320352936900_b6","doi-asserted-by":"crossref","unstructured":"Barry, C.L. (1998), \u201cDocument representations and clues to document relevance\u201d, Journal of the American Society for Information Science, Vol. 49 No. 14, pp. 1293\u2010303.","DOI":"10.1002\/(SICI)1097-4571(1998)49:14<1293::AID-ASI7>3.0.CO;2-E"},{"key":"key2022012320352936900_b7","doi-asserted-by":"crossref","unstructured":"Barry, C.L. and Schamber, L. (1998), \u201cUsers' criteria for relevance evaluation: a cross\u2010situational comparison\u201d, Information Processing & Management, Vol. 34 Nos 2\/3, pp. 219\u201036.","DOI":"10.1016\/S0306-4573(97)00078-2"},{"key":"key2022012320352936900_b8","doi-asserted-by":"crossref","unstructured":"Borlund, P. (2003), \u201cThe concept of relevance in IR\u201d, Journal of the American Society for Information Science and Technology, Vol. 54 No. 10, pp. 913\u201025.","DOI":"10.1002\/asi.10286"},{"key":"key2022012320352936900_b9","doi-asserted-by":"crossref","unstructured":"Chi, E.H., Pirolli, P., Chen, K. and Pitkow, J. (2001), \u201cUsing information scent to model user information needs and actions on the web\u201d, SIGCHI '01, Vol. 3 No. 1, pp. 490\u20107.","DOI":"10.1145\/365024.365325"},{"key":"key2022012320352936900_b11","doi-asserted-by":"crossref","unstructured":"Fitzgerald, M.A. and Galloway, C. (2001), \u201cRelevance judging, evaluation, and decision making in virtual libraries: a descriptive study\u201d, Journal of the American Society for Information Science and Technology, Vol. 52 No. 12, pp. 989\u20101010.","DOI":"10.1002\/asi.1152"},{"key":"key2022012320352936900_b12","doi-asserted-by":"crossref","unstructured":"Fogg, B.J., Soohoo, C., Danielson, D.R., Marable, L., Stanford, J. and Trauber, E.R. (2003), \u201cHow do users evaluate the credibility of web sites? A study with over 2,500 participants\u201d, Proceedings of the Conference on Designing for User Experiences, San Francisco, available at: http:\/\/portal.acm.org\/ citation.cfm?doid_997078.997097.","DOI":"10.1145\/997078.997097"},{"key":"key2022012320352936900_b13","doi-asserted-by":"crossref","unstructured":"Granka, L., Joachims, T. and Gay, G. (2004), \u201cEye\u2010tracking analysis of user behavior in www search\u201d, in Sanderson, M., J\u00e4rvelin, K., Allan, J. and Bruza, P. (Eds), SIGIR 2004: Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Sheffield, UK, July 25\u201029, 2004, ACM, New York, NY, pp. 478\u20109.","DOI":"10.1145\/1008992.1009079"},{"key":"key2022012320352936900_b14","doi-asserted-by":"crossref","unstructured":"Greisdorf, H. (2003), \u201cRelevance thresholds: a multi\u2010stage predictive model of how users evaluate information\u201d, Information Processing and Management, Vol. 39 No. 3, pp. 403\u201023.","DOI":"10.1016\/S0306-4573(02)00032-8"},{"key":"key2022012320352936900_b15","doi-asserted-by":"crossref","unstructured":"Hansen, P. and Karlgren, J. (2005), \u201cEffects of foreign language and task scenario on relevance assessment\u201d, Journal of Documentation, Vol. 61 No. 5, pp. 623\u201039.","DOI":"10.1108\/00220410510625831"},{"key":"key2022012320352936900_b16","doi-asserted-by":"crossref","unstructured":"Hoeber, O. and Yang, X.D. (2009), \u201cHotMap: supporting visual explorations of web search results\u201d, Journal of the American Society for Information Science and Technology, Vol. 60 No. 1, pp. 90\u2010110.","DOI":"10.1002\/asi.20957"},{"key":"key2022012320352936900_b17","unstructured":"Holsti, O.R. (1969), Content Analysis for the Social Sciences and Humanities, Addison\u2010Wesley, Reading, MA."},{"key":"key2022012320352936900_b18","doi-asserted-by":"crossref","unstructured":"Janes, J.W. (1994), \u201cOther people's judgments: a comparison of users' and others' judgments of document relevance, topicality, and utility\u201d, Journal of the American Society for Information Science, Vol. 45 No. 3, pp. 160\u201071.","DOI":"10.1002\/(SICI)1097-4571(199404)45:3<160::AID-ASI6>3.0.CO;2-4"},{"key":"key2022012320352936900_b19","doi-asserted-by":"crossref","unstructured":"Jansen, B.J., Spink, A., Bateman, J. and Saracevic, T. (1998), \u201cReal life information retrieval: a study of user queries on the Web\u201d, SIGIR Forum, Vol. 32 No. 1, pp. 5\u201017.","DOI":"10.1145\/281250.281253"},{"key":"key2022012320352936900_b20","unstructured":"Jansen, B. and Spink, A. (2003), \u201cAn analysis of web information seeking and use: documents retrieved versus documents viewed\u201d, Proceedings of the 4th International Conference on Internet Computing, Las Vegas, NV, pp. 65\u20109."},{"key":"key2022012320352936900_b21","doi-asserted-by":"crossref","unstructured":"Jansen, B. and Spink, A. (2006), \u201cHow are we searching the world wide web? A comparison of nine search engine transaction logs\u201d, Information Processing & Management, Vol. 42 No. 1, pp. 248\u201063.","DOI":"10.1016\/j.ipm.2004.10.007"},{"key":"key2022012320352936900_b22","doi-asserted-by":"crossref","unstructured":"Jansen, B., Zhang, M. and Schultz, C. (2009), \u201cBrand and its effect on user perception of search engine performance\u201d, Journal of the American Society for Information Science & Technology, Vol. 60 No. 8, pp. 1572\u201095.","DOI":"10.1002\/asi.21081"},{"key":"key2022012320352936900_b23","doi-asserted-by":"crossref","unstructured":"Joachims, T., Granka, L., Pan, B., Hembrooke, H. and Gay, G. (2005), \u201cAccurately interpreting clickthrough data as implicit feedback\u201d, Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, New York, NY, pp. 15\u201019.","DOI":"10.1145\/1076034.1076063"},{"key":"key2022012320352936900_b24","doi-asserted-by":"crossref","unstructured":"Kelly, D. and Cool, C. (2002), \u201cThe effects of topic familiarity on information search behavior\u201d, Proceedings of the Second ACM\/IEEE Joint Conference on Digital Libraries, ACM, New York, NY, pp. 74\u20105.","DOI":"10.1145\/544220.544232"},{"key":"key2022012320352936900_b25","doi-asserted-by":"crossref","unstructured":"Kelly, D., Murdock, V., Yuan, X.J., Croft, W.B. and Belkin, N.J. (2002), \u201cFeatures of documents relevant to task\u2010 and fact\u2010 oriented questions\u201d, Proceedings of the Eleventh International Conference on Information and Knowledge Management (CIKM \u201802), ACM, New York, NY, pp. 645\u20107.","DOI":"10.1145\/584792.584907"},{"key":"key2022012320352936900_b26","doi-asserted-by":"crossref","unstructured":"Maglaughlin, K.L. and Sonnenwald, D.H. (2002), \u201cUser perspectives on relevance criteria: a comparison among relevant, partially relevant, and not\u2010relevant judgments\u201d, Journal of the American Society for Information Science and Technology, Vol. 53 No. 5, pp. 327\u201042.","DOI":"10.1002\/asi.10049"},{"key":"key2022012320352936900_b27","doi-asserted-by":"crossref","unstructured":"Metzger, M., Flanagin, A. and Zwarun, L. (2003), \u201cCollege student web use, perceptions of information credibility, and verification behavior\u201d, Computers & Education, Vol. 41 No. 3, pp. 271\u201090.","DOI":"10.1016\/S0360-1315(03)00049-6"},{"key":"key2022012320352936900_b29","doi-asserted-by":"crossref","unstructured":"Park, T.K. (1994), \u201cToward a theory of user\u2010based relevance: a call for a new paradigm of inquiry\u201d, Journal of the American Society for Information Science, Vol. 45 No. 3, pp. 135\u201041.","DOI":"10.1002\/(SICI)1097-4571(199404)45:3<135::AID-ASI3>3.0.CO;2-1"},{"key":"key2022012320352936900_b30","doi-asserted-by":"crossref","unstructured":"Purgailis Parker, L.M. and Johnson, R.E. (1990), \u201cDoes order of presentation affect users\u2019 judgment of documents?\u201d, Journal of the American Society for Information Science, Vol. 41 No. 7, pp. 493\u20104.","DOI":"10.1002\/(SICI)1097-4571(199010)41:7<493::AID-ASI2>3.0.CO;2-0"},{"key":"key2022012320352936900_b31","doi-asserted-by":"crossref","unstructured":"Rele, R.S. and Duchowski, A.T. (2005), \u201cUsing eye tracking to evaluate alternative search results interfaces\u201d, Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 49 No. 15, pp. 1459\u201063.","DOI":"10.1177\/154193120504901508"},{"key":"key2022012320352936900_b32","doi-asserted-by":"crossref","unstructured":"Rieh, S.Y. (2002), \u201cJudgment of information quality and cognitive authority in the web\u201d, Journal of the American Society for Information Science and Technology, Vol. 53 No. 2, pp. 145\u201061.","DOI":"10.1002\/asi.10017"},{"key":"key2022012320352936900_b34","doi-asserted-by":"crossref","unstructured":"Saracevic, T. (2007a), \u201cRelevance: a review of the literature and a framework for thinking on the notion in information science. Part II: nature and manifestations of relevance\u201d, Journal of the American Society for Information Science and Technology, Vol. 58 No. 13, pp. 1915\u201033.","DOI":"10.1002\/asi.20682"},{"key":"key2022012320352936900_b35","doi-asserted-by":"crossref","unstructured":"Saracevic, T. (2007b), \u201cA relevance: a review of the literature and a framework for thinking on the notion in information science. Part III: behavior and effects of relevance\u201d, Journal of the American Society for Information Science and Technology, Vol. 58 No. 13, pp. 2126\u201044.","DOI":"10.1002\/asi.20681"},{"key":"key2022012320352936900_b36","doi-asserted-by":"crossref","unstructured":"Savolainen, R. and Kari, J. (2006), \u201cUser\u2010defined relevance criteria in web searching\u201d, Journal of Documentation, Vol. 62 No. 6, pp. 685\u2010707.","DOI":"10.1108\/00220410610714921"},{"key":"key2022012320352936900_b38","doi-asserted-by":"crossref","unstructured":"Schamber, L., Eisenberg, M.B. and Nilan, M.S. (1990), \u201cA re\u2010examination of relevance: toward a dynamic, situational definition\u201d, Information Processing & Management, Vol. 26 No. 6, pp. 321\u201043.","DOI":"10.1016\/0306-4573(90)90050-C"},{"key":"key2022012320352936900_b39","doi-asserted-by":"crossref","unstructured":"Shneiderman, B., Byrd, D. and Croft, W.B. (1997), \u201cClarifying search: a user\u2010interface framework for text searches\u201d, D\u2010Lib Magazine, Vol. 3 No. 1, available at: www.dlib.org\/dlib\/january97\/retrieval\/01shneiderman.html.","DOI":"10.1045\/january97-shneiderman"},{"key":"key2022012320352936900_b40","doi-asserted-by":"crossref","unstructured":"Silverstein, C., Henzinger, M., Marais, H. and Moricz, M. (1999), \u201cAnalysis of a very large web search engine query log\u201d, ACM SIGIR Forum, Vol. 33 No. 1, pp. 6\u201012.","DOI":"10.1145\/331403.331405"},{"key":"key2022012320352936900_b41","doi-asserted-by":"crossref","unstructured":"Spink, A., Greisdorf, R. and Bateman, J. (1998), \u201cFrom highly relevant to non\u2010relevant: examining different regions of relevance\u201d, Information Processing & Management, Vol. 34 No. 5, pp. 599\u2010621.","DOI":"10.1016\/S0306-4573(98)00025-9"},{"key":"key2022012320352936900_b42","doi-asserted-by":"crossref","unstructured":"Spink, A., Wolfram, D., Jansen, M.B.J. and Saracevic, T. (2001), \u201cSearching the web: the public and their queries\u201d, Journal of the American Society for Information Science and Technology, Vol. 52 No. 3, pp. 226\u201034.","DOI":"10.1002\/1097-4571(2000)9999:9999<::AID-ASI1591>3.0.CO;2-R"},{"key":"key2022012320352936900_b43","unstructured":"Strauss, A. and Corbin, J. (1990), Basics of Qualitative Research, Sage, Newbury Park, CA."},{"key":"key2022012320352936900_b44","doi-asserted-by":"crossref","unstructured":"Su, L.T. (1994), \u201cThe relevance of recall and precision in user evaluation\u201d, Journal of the American Society for Information Science, Vol. 45 No. 3, pp. 204\u201017.","DOI":"10.1002\/(SICI)1097-4571(199404)45:3<207::AID-ASI10>3.0.CO;2-1"},{"key":"key2022012320352936900_b45","doi-asserted-by":"crossref","unstructured":"Tombros, A., Ruthven, I. and Jose, J.M. (2005), \u201cHow users assess web pages for information seeking\u201d, Journal of the American Society for Information Science and Technology, Vol. 56 No. 4, pp. 327\u201044.","DOI":"10.1002\/asi.20106"},{"key":"key2022012320352936900_b46","doi-asserted-by":"crossref","unstructured":"Vakkari, P. and Hakala, N. (2000), \u201cChanges in relevance criteria and problem stages in task performance\u201d, Journal of Documentation, Vol. 56 No. 5, pp. 540\u201062.","DOI":"10.1108\/EUM0000000007127"},{"key":"key2022012320352936900_b47","doi-asserted-by":"crossref","unstructured":"Voiskunskii, V.G. (1997), \u201cEvaluation of search results: a new approach\u201d, Journal of the American Society for Information Science, Vol. 48 No. 2, pp. 133\u201042.","DOI":"10.1002\/(SICI)1097-4571(199702)48:2<133::AID-ASI4>3.0.CO;2-Z"},{"key":"key2022012320352936900_b48","doi-asserted-by":"crossref","unstructured":"Wang, P. and Soergel, D. (1998), \u201cA cognitive model of document use during a research project. Study I: document selection\u201d, Journal of the American Society for Information Science, Vol. 49 No. 2, pp. 115\u201033.","DOI":"10.1002\/(SICI)1097-4571(199802)49:2<115::AID-ASI3>3.0.CO;2-T"},{"key":"key2022012320352936900_b49","doi-asserted-by":"crossref","unstructured":"Wang, P. and Soergel, D. (1999), \u201cA cognitive model of document use during a research project. Study II: decision at the reading and citing stages\u201d, Journal of the American Society for Information Science, Vol. 50 No. 2, pp. 98\u2010114.","DOI":"10.1002\/(SICI)1097-4571(1999)50:2<98::AID-ASI2>3.0.CO;2-L"},{"key":"key2022012320352936900_b50","unstructured":"Xie, I., Benoit, E. III and Zhang, H. (2010), \u201cHow do users evaluate individual documents? An analysis of dimensions of evaluation activities\u201d, Information Research, Vol. 15 No. 4, available at: http:\/\/informationr.net\/ir\/15\u20104\/colis723.html."},{"key":"key2022012320352936900_b51","doi-asserted-by":"crossref","unstructured":"Xu, Y.C. and Chen, Z. (2006), \u201cRelevance judgments: what do information users consider beyond topicality\u201d, Journal of the American Society for Information Science and Technology, Vol. 57 No. 7, pp. 961\u201073.","DOI":"10.1002\/asi.20361"},{"key":"key2022012320352936900_frg3","doi-asserted-by":"crossref","unstructured":"Balatsoukas, P., Morris, A. and O'Brien, A. (2009), \u201cAn evaluation framework of user interaction with metadata surrogates\u201d, Journal of Information Science, Vol. 35 No. 3, pp. 321\u201039.","DOI":"10.1177\/0165551508099090"},{"key":"key2022012320352936900_frg10","unstructured":"Cool, C., Belkin, N.J., Frieder, O. and Kantor, P. (1993), \u201cCharacteristics of texts affecting relevance judgments\u201d, in Williams, M.E. (Ed.), Proceedings of the 14th National Online Meeting, Learned Information, Oxford, pp. 77\u201084."},{"key":"key2022012320352936900_frg28","doi-asserted-by":"crossref","unstructured":"Park, T.K. (1993), \u201cThe nature of relevance in information retrieval: an empirical study\u201d, Library Quarterly, Vol. 63 No. 3, pp. 318\u201051.","DOI":"10.1086\/602592"},{"key":"key2022012320352936900_frg33","unstructured":"Saracevic, T. (1969), \u201cComparative effects of titles, abstracts and full text on relevance judgments\u201d, Proceedings of the American Society for Information Science, Vol. 6, pp. 293\u20109."},{"key":"key2022012320352936900_frg37","unstructured":"Schamber, L. (1991), \u201cUsers' criteria for evaluation in a multimedia environment\u201d, Proceedings of the 54th ASIS Annual Meeting, Vol. 28, pp. 126\u201033."}],"container-title":["Journal of Documentation"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/www.emeraldinsight.com\/doi\/full-xml\/10.1108\/00220411311295324","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/00220411311295324\/full\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/00220411311295324\/full\/html","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,24]],"date-time":"2025-07-24T23:38:32Z","timestamp":1753400312000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.emerald.com\/jd\/article\/69\/1\/49-80\/205386"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2013,1,11]]},"references-count":51,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2013,1,11]]}},"alternative-id":["10.1108\/00220411311295324"],"URL":"https:\/\/doi.org\/10.1108\/00220411311295324","relation":{},"ISSN":["0022-0418"],"issn-type":[{"type":"print","value":"0022-0418"}],"subject":[],"published":{"date-parts":[[2013,1,11]]}}}