{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,14]],"date-time":"2026-05-14T22:19:31Z","timestamp":1778797171833,"version":"3.51.4"},"reference-count":37,"publisher":"Emerald","issue":"4","license":[{"start":{"date-parts":[[2013,7,19]],"date-time":"2013-07-19T00:00:00Z","timestamp":1374192000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.emerald.com\/insight\/site-policies"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2013,7,19]]},"abstract":"<jats:sec><jats:title content-type=\"abstract-heading\">Purpose<\/jats:title><jats:p>The purpose of this paper is to better understand the dynamic nature of the relevance judgment process and the influence of work task on that process.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Design\/methodology\/approach<\/jats:title><jats:p>The empirical study reported here examined the information seeking behavior of a group of undergraduate college students assigned a set of research assignments (work tasks). Subjects recorded their selection of documents used for an assignment and the criteria used to judge those documents relevant. Statistical analysis was used to associate relevance judgments and the criteria used to make those judgments with work tasks.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Findings<\/jats:title><jats:p>Findings indicate a strong statistical association between work task and criteria used to judge relevance. Findings also include identification of specific criteria used to judge relevance and the relative importance of those criteria based on frequency of selection of criteria for a work task.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Research limitations\/implications<\/jats:title><jats:p>Findings provide additional insights into the dynamic nature of the relevance judgment process. Relevance judgment influences revealed in these findings in the form of criteria used to make relevance judgments further explicate the relevance judgment process and provide suggestions for the improvement of information retrieval systems and information literacy efforts.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Originality\/value<\/jats:title><jats:p>Understanding the relevance judgment process is critical to understanding information behavior in general. Few studies have examined relevance criteria selections as part of the relevance judgment process and fewer still have studied these selections in relation to work tasks. A better understanding of this relationship is an essential part of understanding the dynamic nature of the relevance judgment process and its influences.<\/jats:p><\/jats:sec>","DOI":"10.1108\/jd-12-2011-0054","type":"journal-article","created":{"date-parts":[[2013,7,19]],"date-time":"2013-07-19T08:50:36Z","timestamp":1374223836000},"page":"523-544","source":"Crossref","is-referenced-by-count":8,"title":["Examination of work task and criteria choices for the relevance judgment process"],"prefix":"10.1108","volume":"69","author":[{"given":"Arthur","family":"Taylor","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"140","reference":[{"key":"key2022031219442290300_b1","doi-asserted-by":"crossref","unstructured":"Barry, C.L. (1994), \u201cUser\u2010defined relevance criteria: an exploratory study\u201d, Journal of the American Society for Information Science, Vol. 45 No. 3, pp. 149\u2010159.","DOI":"10.1002\/(SICI)1097-4571(199404)45:3<149::AID-ASI5>3.0.CO;2-J"},{"key":"key2022031219442290300_b2","doi-asserted-by":"crossref","unstructured":"Barry, C.L. and Schamber, L. (1998), \u201cUsers' criteria for relevance evaluation: a cross\u2010situational comparison\u201d, Information Processing & Management, Vol. 34 Nos 2\/3, pp. 219\u2010236.","DOI":"10.1016\/S0306-4573(97)00078-2"},{"key":"key2022031219442290300_b3","unstructured":"Bateman, J. (1998), \u201cChanges in relevance criteria: a longitudinal study\u201d, Proceedings of the ASIS Annual Meeting, Pittsburgh, PA, October 25\u201029, Vol. 35, pp. 23\u201032."},{"key":"key2022031219442290300_b4","unstructured":"Cool, C., Belkin, N.J., Kantor, P. and Frieder, O. (1993), \u201cCharacteristics of texts affecting relevance judgments\u201d, in Williams, M.E. (Ed.), Proceedings of the Fourteenth National Online Meeting, Medford, NJ, pp. 77\u201083."},{"key":"key2022031219442290300_b5","doi-asserted-by":"crossref","unstructured":"Cosijn, E. (2006), \u201cRelevance judgements within the context of work tasks\u201d, Proceedings of the 1st International Conference on Information Interaction in Context, New York, NY, pp. 20\u201029.","DOI":"10.1145\/1164820.1164827"},{"key":"key2022031219442290300_b6","doi-asserted-by":"crossref","unstructured":"Cosijn, E. and Ingwersen, P. (2001), \u201cDimensions of relevance\u201d, Information Processing & Management, Vol. 36 No. 4, pp. 533\u2010550.","DOI":"10.1016\/S0306-4573(99)00072-2"},{"key":"key2022031219442290300_b7","doi-asserted-by":"crossref","unstructured":"Crystal, A. and Greenberg, J. (2006), \u201cRelevance criteria identified by health information users during web searches\u201d, Journal of the American Society for Information Science and Technology, Vol. 57 No. 10, pp. 1368\u20101382.","DOI":"10.1002\/asi.20436"},{"key":"key2022031219442290300_b8","doi-asserted-by":"crossref","unstructured":"De Sabatta, S. and Reichenbacher, T. (2012), \u201cCriteria of geographic relevance: an experimental study\u201d, International Journal of Geographical Information Science, Vol. 26 No. 8, pp. 1495\u20101520.","DOI":"10.1080\/13658816.2011.639303"},{"key":"key2022031219442290300_b9","doi-asserted-by":"crossref","unstructured":"Greisdorf, H. (2003), \u201cRelevance thresholds: a multi\u2010stage predictive model of how users evaluate information\u201d, Information Processing & Management, Vol. 39 No. 3, pp. 403\u2010423.","DOI":"10.1016\/S0306-4573(02)00032-8"},{"key":"key2022031219442290300_b10","doi-asserted-by":"crossref","unstructured":"Gross, M. and Latham, D. (2009), \u201cUndergraduate perceptions of information literacy: defining, attaining and self\u2010assessing skills\u201d, College & Research Libraries, Vol. 70 No. 4, pp. 336\u2010350.","DOI":"10.5860\/crl.70.4.336"},{"key":"key2022031219442290300_b12","unstructured":"Hansen, P. (1999), \u201cUser interface design for IR interaction: a task\u2010oriented approach\u201d, in Aparac, T., Saracevic, T., Ingwersen, P. and Vakkari, P. (Eds), CoLIS 3, pp. 191\u2010205."},{"key":"key2022031219442290300_b11","doi-asserted-by":"crossref","unstructured":"Hirsh, S.G. (1999), \u201cChildren's relevance criteria and information seeking on electronic resources\u201d, Journal of the American Society for Information Science, Vol. 50 No. 14, pp. 1265\u20101283.","DOI":"10.1002\/(SICI)1097-4571(1999)50:14<1265::AID-ASI2>3.0.CO;2-E"},{"key":"key2022031219442290300_b13","doi-asserted-by":"crossref","unstructured":"Ingwersen, P. and Jarvelin, K. (2005), \u201cInformation retrieval in context: IRiX\u201d, ACMSIGIR Forum, Vol. 39, ACM Press, New York, NY, pp. 31\u201039.","DOI":"10.1145\/1113343.1113351"},{"key":"key2022031219442290300_b14","unstructured":"Li, Y. (2005), \u201cRelationships among work tasks, search tasks, and interactive information searching behavior\u201d, Journal of the American Society of Information Science and Technology, Vol. 60 No. 5, pp. 327\u2010342."},{"key":"key2022031219442290300_b15","doi-asserted-by":"crossref","unstructured":"Maglaughlin, K.L. and Sonnenwald, D.H. (2002), \u201cUser perspectives on relevance criteria: a comparison among relevant, partially relevant, and not\u2010relevant judgments\u201d, Journal of the American Society of Information Science and Technology, Vol. 53 No. 5, pp. 327\u2010342.","DOI":"10.1002\/asi.10049"},{"key":"key2022031219442290300_b16","doi-asserted-by":"crossref","unstructured":"Park, T.K. (1993), \u201cThe nature of relevance in information retrieval: an empirical study\u201d, Library Quarterly, Vol. 63 No. 3, pp. 318\u2010351.","DOI":"10.1086\/602592"},{"key":"key2022031219442290300_b17","doi-asserted-by":"crossref","unstructured":"Raper, J. (2007), \u201cGeographic relevance\u201d, Journal of Documentation, Vol. 63 No. 6, pp. 836\u2010852.","DOI":"10.1108\/00220410710836385"},{"key":"key2022031219442290300_b18","doi-asserted-by":"crossref","unstructured":"Reichenbacher, T. and De Sabbata, S. (2011), \u201cGeographic relevance \u2013 different notions on geographies and relevancies\u201d, Geographic Information Retrieval, Vol. 3 No. 2, (SIGSPATIAL Special Issue).","DOI":"10.1145\/2047296.2047310"},{"key":"key2022031219442290300_b19","doi-asserted-by":"crossref","unstructured":"Saracevic, T. (1975), \u201cRelevance: a review of and a framework for the thinking on the notion in information science\u201d, Journal of the American Society for Information Science, Vol. 26, pp. 321\u2010343.","DOI":"10.1002\/asi.4630260604"},{"key":"key2022031219442290300_b20","unstructured":"Saracevic, T. (1996), \u201cRelevance reconsidered\u201d, Proceedings of the Second Conference on Conceptions of Library and Information Science (CoLIS 2), pp. 201\u2010218."},{"key":"key2022031219442290300_b21","doi-asserted-by":"crossref","unstructured":"Saracevic, T. (2007a), \u201cRelevance: a review of the literature and a framework for thinking on the notion in information science. Part II: nature and manifestations of relevance\u201d, Journal of the American Society for Information Science and Technology, Vol. 58 No. 13, pp. 1915\u20101933.","DOI":"10.1002\/asi.20682"},{"key":"key2022031219442290300_b22","doi-asserted-by":"crossref","unstructured":"Saracevic, T. (2007b), \u201cRelevance: a review of the literature and a framework for thinking on the notion in information science. Part III: behavior and effects of relevance\u201d, Journal of the American Society for Information Science and Technology, Vol. 58 No. 13, pp. 2126\u20102144.","DOI":"10.1002\/asi.20681"},{"key":"key2022031219442290300_b23","unstructured":"Schamber, L. (1991), \u201cUser's criteria for evaluation in a multimedia environment\u201d, in Griffiths, J. (Ed.), ASIS '91: Proceedings of the American Society for Information Science (ASIS) 54th Annual Meeting, Washington, DC, October 30\u2010November 2, Vol. 28, pp. 126\u2010133."},{"key":"key2022031219442290300_b24","unstructured":"Schamber, L. and Bateman, J. (1996), \u201cUser criteria in relevance evaluation: toward development of a measurement scale\u201d, Proceedings of the ASIS Annual Meeting, Vol. 33, pp. 218\u2010225."},{"key":"key2022031219442290300_b25","doi-asserted-by":"crossref","unstructured":"Schamber, L., Eisenberg, M.B. and Nilan, M.S. (1990), \u201cA re\u2010examination of relevance: toward a dynamic, situational definition\u201d, Information Processing & Management, Vol. 26 No. 6, pp. 755\u2010776.","DOI":"10.1016\/0306-4573(90)90050-C"},{"key":"key2022031219442290300_b26","doi-asserted-by":"crossref","unstructured":"Spink, A., Greisdorf, H. and Bateman, J. (1998), \u201cFrom highly relevant to not relevant: examining different regions of relevance\u201d, Information Processing & Management, Vol. 34 No. 5, pp. 599\u2010621.","DOI":"10.1016\/S0306-4573(98)00025-9"},{"key":"key2022031219442290300_b27","doi-asserted-by":"crossref","unstructured":"Tang, R. and Solomon, P. (1998), \u201cToward an understanding of the dynamics of relevance judgment: an analysis of one person's search behavior\u201d, Information Processing & Management, Vol. 34 Nos 2\/3, pp. 237\u2010256.","DOI":"10.1016\/S0306-4573(97)00081-2"},{"key":"key2022031219442290300_b28","doi-asserted-by":"crossref","unstructured":"Tang, R. and Solomon, P. (2001), \u201cUse of relevance criteria across stages of document evaluation: on the complementarity of experimental and naturalistic studies\u201d, Journal of the American Society for Information Science and Technology, Vol. 52 No. 8, pp. 676\u2010685.","DOI":"10.1002\/asi.1116"},{"key":"key2022031219442290300_b29","unstructured":"Taylor, A.R. (2012), \u201cA study of the information search behaviour of the millennial generation\u201d, Information Research, Vol. 17 No. 1."},{"key":"key2022031219442290300_b31","doi-asserted-by":"crossref","unstructured":"Taylor, A.R., Xiangmin, Z. and Amadio, W.J. (2009), \u201cExamination of relevance criteria choices and the information\u201d, Journal of Documentation, Vol. 65 No. 5, pp. 719\u2010744.","DOI":"10.1108\/00220410910983083"},{"key":"key2022031219442290300_b30","doi-asserted-by":"crossref","unstructured":"Taylor, A.R., Cool, C., Belkin, N.J. and Amadio, W.J. (2007), \u201cRelationships between categories of relevance criteria and stage in task completion\u201d, Information Processing & Management, Vol. 43 No. 4, pp. 1071\u20101084.","DOI":"10.1016\/j.ipm.2006.09.008"},{"key":"key2022031219442290300_b32","doi-asserted-by":"crossref","unstructured":"Tombros, A., Ruthven, I. and Jose, J.M. (2005), \u201cHow users assess web pages for information seeking\u201d, Journal of the American Society for Information Science and Technology, Vol. 56 No. 4, pp. 327\u2010344.","DOI":"10.1002\/asi.20106"},{"key":"key2022031219442290300_b33","doi-asserted-by":"crossref","unstructured":"Vakkari, P. (1999), \u201cTask complexity, problem structure and information actions: integrating studies on information seeking and retrieval\u201d, Information Processing & Management, Vol. 35, pp. 819\u2010837.","DOI":"10.1016\/S0306-4573(99)00028-X"},{"key":"key2022031219442290300_b34","doi-asserted-by":"crossref","unstructured":"Vakkari, P. and Hakala, N. (2000), \u201cChanges in relevance criteria and problem stages in task performance\u201d, Journal of Documentation, Vol. 56 No. 5, pp. 540\u2010562.","DOI":"10.1108\/EUM0000000007127"},{"key":"key2022031219442290300_b35","doi-asserted-by":"crossref","unstructured":"Wang, P. and White, M.D. (1999), \u201cA cognitive model of document use during a research project. Study II: decisions at the reading and writing stages\u201d, Journal of the American Society for Information Science, Vol. 50 No. 2, pp. 98\u2010114.","DOI":"10.1002\/(SICI)1097-4571(1999)50:2<98::AID-ASI2>3.0.CO;2-L"},{"key":"key2022031219442290300_b36","doi-asserted-by":"crossref","unstructured":"Xu, Y. and Chen, Z. (2006), \u201cRelevance judgment: what do information consumers consider beyond topicality?\u201d, Journal of the American Society for Information Science and Technology, Vol. 57 No. 7, pp. 961\u2010973.","DOI":"10.1002\/asi.20361"},{"key":"key2022031219442290300_b37","doi-asserted-by":"crossref","unstructured":"Yuan, X., Belkin, N.J. and Kim, J. (2002), \u201cThe relationship between ASK and relevance criteria\u201d, SIGIR '02: Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, New York, NY, pp. 359\u2010360.","DOI":"10.1145\/564376.564443"}],"container-title":["Journal of Documentation"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/www.emeraldinsight.com\/doi\/full-xml\/10.1108\/JD-12-2011-0054","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/JD-12-2011-0054\/full\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/JD-12-2011-0054\/full\/html","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,24]],"date-time":"2025-07-24T22:35:32Z","timestamp":1753396532000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.emerald.com\/jd\/article\/69\/4\/523-544\/215277"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2013,7,19]]},"references-count":37,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2013,7,19]]}},"alternative-id":["10.1108\/JD-12-2011-0054"],"URL":"https:\/\/doi.org\/10.1108\/jd-12-2011-0054","relation":{},"ISSN":["0022-0418"],"issn-type":[{"value":"0022-0418","type":"print"}],"subject":[],"published":{"date-parts":[[2013,7,19]]}}}