{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,12,29]],"date-time":"2025-12-29T19:01:32Z","timestamp":1767034892435,"version":"3.41.2"},"reference-count":47,"publisher":"Emerald","issue":"5","license":[{"start":{"date-parts":[[2009,9,4]],"date-time":"2009-09-04T00:00:00Z","timestamp":1252022400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.emerald.com\/insight\/site-policies"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2009,9,4]]},"abstract":"<jats:sec><jats:title content-type=\"abstract-heading\">Purpose<\/jats:title><jats:p>The purpose of this paper is to examine changes in relevance assessments, specifically the selection of relevance criteria by subjects as they move through the information search process.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Design\/methodology\/approach<\/jats:title><jats:p>The paper examines the relevance criteria choices of 39 subjects in relation to search stage. Subjects were assigned a specific search task in a controlled test. Statistics were collected and analyzed using descriptive statistics and the chi\u2010square goodness\u2010of\u2010fit tests.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Findings<\/jats:title><jats:p>The statistically significant findings identified a number of commonly reported relevance criteria, which varied over an information search process for relevant and partially relevant judgments. These results provide statistical confirmations of previous studies, and extend these findings identifying specific criteria for both relevant and partially relevant judgments.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Research limitations\/implications<\/jats:title><jats:p>The study only examines a short duration search process and since the convenience sample of subjects were from similar backgrounds and were assigned similar tasks, the study did not explicitly examine the impact of contextual factors such as user experience, background or task in relation to relevance criteria choices.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Practical implications<\/jats:title><jats:p>The paper has implications for the development of search systems which are adaptive and recognize the cognitive changes which occur during the information search process. Examining and identifying relevance criteria beyond topicality and the importance of those criteria to a user can help in the generation of better search queries.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Originality\/value<\/jats:title><jats:p>The paper adds more rigorous statistical analysis to the study of relevance criteria and the information search process.<\/jats:p><\/jats:sec>","DOI":"10.1108\/00220410910983083","type":"journal-article","created":{"date-parts":[[2009,10,5]],"date-time":"2009-10-05T12:29:00Z","timestamp":1254745740000},"page":"719-744","source":"Crossref","is-referenced-by-count":11,"title":["Examination of relevance criteria choices and the information search process"],"prefix":"10.1108","volume":"65","author":[{"given":"Art","family":"Taylor","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xiangmin","family":"Zhang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"William J.","family":"Amadio","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"140","reference":[{"key":"key2022031820365838900_b1","doi-asserted-by":"crossref","unstructured":"Barry, C.L. (1998), \u201cDocument representations and clues to document relevance\u201d, Journal of the American Society for Information Science, Vol. 49 No. 14, pp. 1293\u2010303.","DOI":"10.1002\/(SICI)1097-4571(1998)49:14<1293::AID-ASI7>3.0.CO;2-E"},{"key":"key2022031820365838900_b2","doi-asserted-by":"crossref","unstructured":"Barry, C.L. and Schamber, L. (1998), \u201cUsers' criteria for relevance evaluation: a cross\u2010situational comparison\u201d, Information Processing and Management, Vol. 34 Nos 2\u20103, pp. 219\u201036.","DOI":"10.1016\/S0306-4573(97)00078-2"},{"key":"key2022031820365838900_b3","doi-asserted-by":"crossref","unstructured":"Barry, C.L. (1994), \u201cUser\u2010defined relevance criteria: an exploratory study\u201d, Journal of the American Society for Information Science, Vol. 45, pp. 149\u201059.","DOI":"10.1002\/(SICI)1097-4571(199404)45:3<149::AID-ASI5>3.0.CO;2-J"},{"key":"key2022031820365838900_b4","unstructured":"Bateman, J. (1998), \u201cChanges in relevance criteria: a longitudinal study\u201d, Proceedings of the ASIS Annual Meeting, October 25\u201029, Pittsburgh, PA, Vol. 35, pp. 23\u201032."},{"key":"key2022031820365838900_b5","doi-asserted-by":"crossref","unstructured":"Belkin, N. (1982), \u201cASK for information retrieval \u2013 part I\u201d, Journal of Documentation, Vol. 38, pp. 61\u201072.","DOI":"10.1108\/eb026722"},{"key":"key2022031820365838900_b7","doi-asserted-by":"crossref","unstructured":"Belkin, N.J. (2000), \u201cHelping people find what they don't know\u201d, Communications of ACM, Vol. 43, pp. 58\u201061.","DOI":"10.1145\/345124.345143"},{"key":"key2022031820365838900_b9","doi-asserted-by":"crossref","unstructured":"Borlund, P. (2003), \u201cThe concept of relevance in IR\u201d, Journal of the American Society for Information Science and Technology, Vol. 54, pp. 913\u201025.","DOI":"10.1002\/asi.10286"},{"key":"key2022031820365838900_b10","doi-asserted-by":"crossref","unstructured":"Borlund, P. and Ingwersen, P. (1998), \u201cMeasures of relative relevance and ranked half\u2010life: performance indicators for interactive IR\u201d, SIGIR '98: Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, August 24\u201028, Melbourne, Australia, ACM, pp. 324\u201031.","DOI":"10.1145\/290941.291019"},{"key":"key2022031820365838900_b11","doi-asserted-by":"crossref","unstructured":"Choi, Y. and Rasmussen, E.M. (2002), \u201cUsers' relevance criteria in image retrieval in American history\u201d, Information Processing & Management, Vol. 38, pp. 695\u2010726.","DOI":"10.1016\/S0306-4573(01)00059-0"},{"key":"key2022031820365838900_b12","doi-asserted-by":"crossref","unstructured":"Cooper, W.S. (1971), \u201cA definition of relevance for information retrieval\u201d, Information Storage and Retrieval, Vol. 7, pp. 19\u201037.","DOI":"10.1016\/0020-0271(71)90024-6"},{"key":"key2022031820365838900_b13","doi-asserted-by":"crossref","unstructured":"Cosijn, E. and Ingwersen, P. (2001), \u201cDimensions of relevance\u201d, Information Processing and Management, Vol. 36 No. 4, pp. 533\u201050.","DOI":"10.1016\/S0306-4573(99)00072-2"},{"key":"key2022031820365838900_b14","doi-asserted-by":"crossref","unstructured":"Crystal, A. and Greenberg, J. (2006), \u201cRelevance criteria identified by health information users during web searches\u201d, Journal of the American Society for Information Science and Technology, Vol. 57 No. 10, pp. 1368\u201082.","DOI":"10.1002\/asi.20436"},{"key":"key2022031820365838900_b15","doi-asserted-by":"crossref","unstructured":"Cuadra, C.A. and Katter, R.V. (1967), \u201cOpening the black box of \u2018relevance\u2019\u201d, Journal of Documentation, Vol. 23, pp. 291\u2010303.","DOI":"10.1108\/eb026436"},{"key":"key2022031820365838900_b16","doi-asserted-by":"crossref","unstructured":"Ellis, D. (1989), \u201cA behavioural approach to information science retrieval design\u201d, Journal of Documentation, Vol. 45, pp. 171\u2010212.","DOI":"10.1108\/eb026843"},{"key":"key2022031820365838900_b17","doi-asserted-by":"crossref","unstructured":"Ellis, D. (1997), \u201cModeling the information seeking patterns of engineers and research scientists in an industrial environment\u201d, Journal of Documentation, Vol. 53 No. 4, pp. 384\u2010403.","DOI":"10.1108\/EUM0000000007204"},{"key":"key2022031820365838900_b18","doi-asserted-by":"crossref","unstructured":"Harter, S.P. (1992), \u201cPsychological relevance and information science\u201d, Journal of the American Society for Information Science, Vol. 43 No. 9, pp. 602\u201015.","DOI":"10.1002\/(SICI)1097-4571(199210)43:9<602::AID-ASI3>3.0.CO;2-Q"},{"key":"key2022031820365838900_b19","doi-asserted-by":"crossref","unstructured":"Hirsh, S.G. (1999), \u201cChildren's relevance criteria and information seeking on electronic resources\u201d, Journal of the American Society for Information Science, Vol. 50 No. 14, pp. 1265\u201083.","DOI":"10.1002\/(SICI)1097-4571(1999)50:14<1265::AID-ASI2>3.0.CO;2-E"},{"key":"key2022031820365838900_b20","doi-asserted-by":"crossref","unstructured":"Hj\u00f8rland, B. (2000), \u201cRelevance research: the missing perspective(s): non\u2010relevance and epistemological relevance\u201d, Journal of the American Society for Information Science, Vol. 51, pp. 209\u201011.","DOI":"10.1002\/(SICI)1097-4571(2000)51:2<209::AID-ASI14>3.0.CO;2-B"},{"key":"key2022031820365838900_b21","unstructured":"Kuhlthau, C. (1993), Seeking Meaning: A Process Approach to Library and Information Services, Ablex Publishing Company, Norwood, NJ."},{"key":"key2022031820365838900_b22","doi-asserted-by":"crossref","unstructured":"Kuhlthau, C.C. (1991), \u201cInside the search process: information seeking from the user's perspective\u201d, Journal of the American Society for Information Science, Vol. 42 No. 5, pp. 361\u201071.","DOI":"10.1002\/(SICI)1097-4571(199106)42:5<361::AID-ASI6>3.0.CO;2-#"},{"key":"key2022031820365838900_b23","doi-asserted-by":"crossref","unstructured":"Maglaughlin, K.L. and Sonnenwald, D.H. (2002), \u201cUser perspectives on relevance criteria: a comparison among relevant, partially relevant, and not\u2010relevant judgments\u201d, Journal of the American Society of Information Science and Technology, Vol. 53 No. 5, pp. 327\u201042.","DOI":"10.1002\/asi.10049"},{"key":"key2022031820365838900_b24","doi-asserted-by":"crossref","unstructured":"Mizzaro, S. (1998), \u201cHow many relevances in information retrieval?\u201d, Interacting with Computers, Vol. 10, pp. 303\u201020.","DOI":"10.1016\/S0953-5438(98)00012-5"},{"key":"key2022031820365838900_b25","doi-asserted-by":"crossref","unstructured":"Park, K. (1993), \u201cThe nature of relevance in information retrieval: an empirical study\u201d, Library Quarterly, Vol. 63 No. 3, pp. 318\u201051.","DOI":"10.1086\/602592"},{"key":"key2022031820365838900_b26","doi-asserted-by":"crossref","unstructured":"Saracevic, T. (1975), \u201cRelevance: a review of and a framework for the thinking on the notion in information science\u201d, Journal of the American Society for Information Science, Vol. 26, pp. 321\u201043.","DOI":"10.1002\/asi.4630260604"},{"key":"key2022031820365838900_b27","unstructured":"Sarcevic, T. (1996), \u201cRelevance reconsidered\u201d, Proceedings of the Second Conference on Conceptions of Library and Information Science (CoLIS 2), October 14\u201017, Copenhagen, pp. 201\u201018."},{"key":"key2022031820365838900_b28","unstructured":"Schamber, L. (1991), \u201cUsers' criteria for evaluation in a multimedia environment\u201d, in Griffiths, J.\u2010M. (Ed.), Proceedings of the American Society for Information Science (ASIS) 54th Annual Meeting, Vol. 28, pp. 126\u201033."},{"key":"key2022031820365838900_b29","unstructured":"Schamber, L. and Bateman, J. (1996), \u201cUser criteria in relevance evaluation: Toward development of a measurement scale\u201d, Proceedings of the 59th Annual Meeting of the American Society for Information Science, Baltimore, MD, October 21\u201026."},{"key":"key2022031820365838900_b31","doi-asserted-by":"crossref","unstructured":"Schamber, L., Eisenberg, M.B. and Nilan, M.S. (1990), \u201cA re\u2010examination of relevance: toward a dynamic, situational definition\u201d, Information Processing and Management, Vol. 26 No. 6, pp. 755\u201076.","DOI":"10.1016\/0306-4573(90)90050-C"},{"key":"key2022031820365838900_b32","doi-asserted-by":"crossref","unstructured":"Spink, A., Greisdorf, H. and Bateman, J. (1998), \u201cFrom highly relevant to not relevant: examining different regions of relevance\u201d, Information Processing and Management, Vol. 34 No. 5, pp. 599\u2010621.","DOI":"10.1016\/S0306-4573(98)00025-9"},{"key":"key2022031820365838900_b33","unstructured":"Su, L.T. (1993), \u201cIs relevance an adequate criterion for retrieval system evaluation: an empirical inquiry into user's evaluation\u201d, Proceedings of the American Society for Information Science, Learned Information, Medford, NJ, pp. 93\u2010103."},{"key":"key2022031820365838900_b36","doi-asserted-by":"crossref","unstructured":"Tang, R. and Solomon, P. (1998), \u201cToward an understanding of the dynamics of relevance judgment: an analysis of one person's search behavior\u201d, Information Processing and Management, Nos 2\u20103, pp. 237\u201056.","DOI":"10.1016\/S0306-4573(97)00081-2"},{"key":"key2022031820365838900_b37","doi-asserted-by":"crossref","unstructured":"Tang, R.S. and Solomon, P. (2001), \u201cUse of relevance criteria across stages of document evaluation: on the complementarity of experimental and naturalistic studies\u201d, Journal of the American Society for Information Science and Technology, Vol. 52 No. 8, pp. 676\u201085.","DOI":"10.1002\/asi.1116"},{"key":"key2022031820365838900_b39","doi-asserted-by":"crossref","unstructured":"Taylor, A.R., Cool, C., Belkin, N.J. and Amadio, W.J. (2007), \u201cRelationships between categories of relevance criteria and stage in task completion\u201d, Information Processing and Management, Vol. 43, pp. 1071\u201084.","DOI":"10.1016\/j.ipm.2006.09.008"},{"key":"key2022031820365838900_b38","doi-asserted-by":"crossref","unstructured":"Taylor, R.S. (1968), \u201cQuestion\u2010negotiation and information seeking in libraries\u201d, College Research Libraries, Vol. 29 No. 3, pp. 178\u201094.","DOI":"10.5860\/crl_29_03_178"},{"key":"key2022031820365838900_b40","doi-asserted-by":"crossref","unstructured":"Vakkari, P. (2000), \u201cRelevance and contributing information types of searched documents in task performance\u201d, Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, July 24\u201028, Athens, Greece, ACM Press, New York, NY, pp. 2\u20109.","DOI":"10.1145\/345508.345512"},{"key":"key2022031820365838900_b41","doi-asserted-by":"crossref","unstructured":"Vakkari, P. and Hakala, N. (2000), \u201cChanges in relevance criteria and problem stages in task performance\u201d, Journal of Documentation, Vol. 56 No. 5, pp. 540\u201062.","DOI":"10.1108\/EUM0000000007127"},{"key":"key2022031820365838900_b43","doi-asserted-by":"crossref","unstructured":"Wang, P. and Soergel, D. (1998), \u201cA cognitive model of document use during a research project. Study I: document selection\u201d, Journal of the American Society of Information Science and Technology, Vol. 49 No. 2, pp. 115\u201033.","DOI":"10.1002\/(SICI)1097-4571(199802)49:2<115::AID-ASI3>3.0.CO;2-T"},{"key":"key2022031820365838900_b44","doi-asserted-by":"crossref","unstructured":"Wang, P. and White, M.D. (1999), \u201cA cognitive model of document use during a research project. Study II: decisions at the reading and writing stages\u201d, Journal of the American Society for Information Science, Vol. 50 No. 2, pp. 98\u2010114.","DOI":"10.1002\/(SICI)1097-4571(1999)50:2<98::AID-ASI2>3.0.CO;2-L"},{"key":"key2022031820365838900_b46","doi-asserted-by":"crossref","unstructured":"Wilson, P. (1973), \u201cSituational relevance\u201d, Information Storage and Retrieval, Vol. 9 No. 8, pp. 457\u201071.","DOI":"10.1016\/0020-0271(73)90096-X"},{"key":"key2022031820365838900_b48","doi-asserted-by":"crossref","unstructured":"Wilson, T.D. (1999), \u201cModels in information behavior research\u201d, Journal of Documentation, Vol. 55, pp. 49\u2010270.","DOI":"10.1108\/EUM0000000007145"},{"key":"key2022031820365838900_b49","doi-asserted-by":"crossref","unstructured":"Xu, Y. and Chen, Z. (2006), \u201cRelevance judgment: what do information consumers consider beyond topicality?\u201d, Journal of the American Society for Information Science and Technology, Vol. 57 No. 7, pp. 961\u201073.","DOI":"10.1002\/asi.20361"},{"key":"key2022031820365838900_frg1","doi-asserted-by":"crossref","unstructured":"Belkin, N.J., Oddy, R.N. and Brooks, H. (1982), \u201cASK for information retrieval \u2013 part II\u201d, Journal of Documentation, Vol. 38 No. 3, pp. 145\u201064.","DOI":"10.1108\/eb026726"},{"key":"key2022031820365838900_frg2","doi-asserted-by":"crossref","unstructured":"Bookstein, A. (1979), \u201cRelevance\u201d, Journal of the American Society for Information Science, Vol. 30 No. 5, pp. 269\u201073.","DOI":"10.1002\/asi.4630300505"},{"key":"key2022031820365838900_frg3","doi-asserted-by":"crossref","unstructured":"Swanson, D.R. (1986), \u201cSubjective versus objective relevance in bibliographic retrieval systems\u201d, The Library Quarterly, Vol. 56, pp. 389\u201098.","DOI":"10.1086\/601800"},{"key":"key2022031820365838900_frg4","doi-asserted-by":"crossref","unstructured":"Swanson, D.R. (1988), \u201cHistorical note: information retrieval and the future of an illusion\u201d, Journal of the American Society for Information Science, Vol. 39, pp. 92\u20108.","DOI":"10.1002\/(SICI)1097-4571(198803)39:2<92::AID-ASI4>3.0.CO;2-P"},{"key":"key2022031820365838900_frg5","doi-asserted-by":"crossref","unstructured":"Vakkari, P., Pennanen, M. and Serola, S. (2003), \u201cChanges in search terms and tactics while writing a research proposal: a longitudinal case study\u201d, Information Processing and Management, Vol. 39, pp. 445\u201063.","DOI":"10.1016\/S0306-4573(02)00031-6"},{"key":"key2022031820365838900_frg6","doi-asserted-by":"crossref","unstructured":"Wilson, T.D. (1997), \u201cInformation behavior: an interdisciplinary perspective\u201d, Information Processing and Management, Vol. 83 No. 4, pp. 551\u201072.","DOI":"10.1016\/S0306-4573(97)00028-9"}],"container-title":["Journal of Documentation"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/www.emeraldinsight.com\/doi\/full-xml\/10.1108\/00220410910983083","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/00220410910983083\/full\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/00220410910983083\/full\/html","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,24]],"date-time":"2025-07-24T23:38:08Z","timestamp":1753400288000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.emerald.com\/jd\/article\/65\/5\/719-744\/194194"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2009,9,4]]},"references-count":47,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2009,9,4]]}},"alternative-id":["10.1108\/00220410910983083"],"URL":"https:\/\/doi.org\/10.1108\/00220410910983083","relation":{},"ISSN":["0022-0418"],"issn-type":[{"type":"print","value":"0022-0418"}],"subject":[],"published":{"date-parts":[[2009,9,4]]}}}