{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,4]],"date-time":"2026-04-04T06:06:30Z","timestamp":1775282790048,"version":"3.50.1"},"reference-count":24,"publisher":"Emerald","issue":"2","license":[{"start":{"date-parts":[[2011,3,8]],"date-time":"2011-03-08T00:00:00Z","timestamp":1299542400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.emerald.com\/insight\/site-policies"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2011,3,8]]},"abstract":"<jats:sec><jats:title content-type=\"abstract-heading\">Purpose<\/jats:title><jats:p>This study intends to identify factors that affect relevance judgment of retrieved information as part of the 2007 TREC Legal track interactive task.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Design\/methodology\/approach<\/jats:title><jats:p>Data were gathered and analyzed from the participants of the 2007 TREC Legal track interactive task using a questionnaire which includes not only a list of 80 relevance factors identified in prior research, but also a space for expressing their thoughts on relevance judgment in the process.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Findings<\/jats:title><jats:p>This study finds that topicality remains a primary criterion, out of various options, for determining relevance, while specificity of the search request, task, or retrieved results also helps greatly in relevance judgment.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Research limitations\/implications<\/jats:title><jats:p>Relevance research should focus on the topicality and specificity of what is being evaluated as well as conducted in real environments.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Practical implications<\/jats:title><jats:p>If multiple relevance factors are presented to assessors, the total number in a list should be below ten to take account of the limited processing capacity of human beings' short\u2010term memory. Otherwise, the assessors might either completely ignore or inadequately consider some of the relevance factors when making judgment decisions.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Originality\/value<\/jats:title><jats:p>This study presents a method for reducing the artificiality of relevance research design, an apparent limitation in many related studies. Specifically, relevance judgment was made in this research as part of the 2007 TREC Legal track interactive task rather than a study devised for the sake of it. The assessors also served as searchers so that their searching experience would facilitate their subsequent relevance judgments.<\/jats:p><\/jats:sec>","DOI":"10.1108\/00220411111109467","type":"journal-article","created":{"date-parts":[[2011,2,26]],"date-time":"2011-02-26T08:26:28Z","timestamp":1298708788000},"page":"264-278","source":"Crossref","is-referenced-by-count":14,"title":["Factors affecting relevance judgment: a report from TREC Legal track"],"prefix":"10.1108","volume":"67","author":[{"given":"Heting","family":"Chu","sequence":"first","affiliation":[]}],"member":"140","reference":[{"key":"key2021010420210237900_b1","unstructured":"Barry, C.L. (1994), \u201cUser\u2010defined relevance criteria: an exploratory study\u201d, Journal of the American Society for Information Science and Technology, Vol. 45 No. 3, pp. 149\u201059."},{"key":"key2021010420210237900_b2","unstructured":"Barry, C.L. and Schamber, L. (1998), \u201cUsers' criteria for relevance evaluation: a cross\u2010situational comparison\u201d, Information Processing and Management, Vol. 34 Nos 2\/3, pp. 219\u201036."},{"key":"key2021010420210237900_b3","unstructured":"Bateman, J. (1999), \u201cModeling the importance of end\u2010user relevance criteria\u201d, Proceedings of the 62nd Annual Meeting of the American Society for Information Science and Technology, Vol. 36, pp. 396\u2010406."},{"key":"key2021010420210237900_b4","unstructured":"Borlund, P. (2003), \u201cThe concept of relevance\u201d, Journal of the American Society for Information Science and Technology, Vol. 54 No. 10, pp. 913\u201025."},{"key":"key2021010420210237900_b5","unstructured":"Choi, Y. and Rasumussen, E.M. (2002), \u201cUsers' relevance criteria in image retrieval in American history\u201d, Information Processing and Management, Vol. 38 No. 5, pp. 695\u2010726."},{"key":"key2021010420210237900_b6","doi-asserted-by":"crossref","unstructured":"Chu, H. et al. (2007), \u201cTREC 2007 Legal track interactive task: a report from the LIU team\u201d, The 16th Text REtrieval Conference (TREC 2007) Proceedings, NIST Special Publication: SP 500\u2010274, National Institute of Standards and Technology, Gaithersburg, MD, available at: http:\/\/trec.nist.gov\/pubs\/trec16\/papers\/longislandu.legal.final.pdf (accessed April 26, 2010).","DOI":"10.6028\/NIST.SP.500-274.legal-liu.chu"},{"key":"key2021010420210237900_b7","unstructured":"Crystal, A. and Greenberg, J. (2006), \u201cRelevance criteria identified by health information users during web searches\u201d, Journal of the American Society for Information Science and Technology, Vol. 57 No. 10, pp. 1368\u201082."},{"key":"key2021010420210237900_b8","unstructured":"Eisenberg, M.B. and Barry, C.L. (1988), \u201cOrder effects: a study of the possible influence of presentation order on user judgments of document relevance\u201d, Journal of the American Society of Information Science, Vol. 39 No. 5, pp. 293\u2010300."},{"key":"key2021010420210237900_b9","doi-asserted-by":"crossref","unstructured":"Huang, M.H. and Wang, H.Y. (2004), \u201cThe influence of document presentation order and number of documents judged on users' judgments of relevance\u201d, Journal of the American Society for Information Science and Technology, Vol. 55 No. 11, pp. 970\u20109.","DOI":"10.1002\/asi.20047"},{"key":"key2021010420210237900_b10","unstructured":"Maglaughlin, K.L. and Sonnenwald, D.H. (2002), \u201cUser perspectives on relevance criteria: a comparison among relevant, partially relevant, and not\u2010relevant judgments\u201d, Journal of the American Society for Information Science and Technology, Vol. 53 No. 5, pp. 327\u201042."},{"key":"key2021010420210237900_b11","unstructured":"Miller, G.A. (1956), \u201cThe magical number seven plus or minus two: some limits on our capacity for processing information\u201d, Psychological Review, Vol. 63 No. 2, pp. 81\u201097."},{"key":"key2021010420210237900_b12","unstructured":"Ruthven, I., Baillie, M. and Elsweiler, D. (2007), \u201cThe relative effects of knowledge, interest and confidence in assessing relevance\u201d, Journal of Documentation, Vol. 63 No. 4, pp. 482\u2010504."},{"key":"key2021010420210237900_b13","doi-asserted-by":"crossref","unstructured":"Saracevic, T. (2007a), \u201cRelevance: a review of the literature and a framework for thinking on the notion in information science. Part II: nature and manifestations of relevance\u201d, Journal of the American Society for Information Science and Technology, Vol. 58 No. 13, pp. 1915\u201033.","DOI":"10.1002\/asi.20682"},{"key":"key2021010420210237900_b14","doi-asserted-by":"crossref","unstructured":"Saracevic, T. (2007b), \u201cRelevance: a review of the literature and a framework for thinking on the notion in information science. Part III: behavior and effects of relevance\u201d, Journal of the American Society for Information Science and Technology, Vol. 58 No. 13, pp. 2126\u201044.","DOI":"10.1002\/asi.20681"},{"key":"key2021010420210237900_b15","unstructured":"Schamber, L. (1991), \u201cUsers' criteria for evaluation in a multimedia environment\u201d, Proceedings of the 54th Annual Meeting of the American Society for formation Science, Vol. 28, pp. 126\u201033."},{"key":"key2021010420210237900_b16","unstructured":"Schamber, L. (1994), \u201cRelevance and information behavior\u201d, Annual Review of Information Science and Technology, Vol. 29, pp. 3\u201048."},{"key":"key2021010420210237900_b17","unstructured":"Schamber, L., Eisenberg, M.B. and Nilan, M.S. (1990), \u201cA re\u2010examination of relevance: toward a dynamic, situational definition\u201d, Information Processing and Management, Vol. 26 No. 6, pp. 755\u201076."},{"key":"key2021010420210237900_b18","unstructured":"Tang, R. and Solomon, P. (2001), \u201cUse of relevance criteria across stages of document evaluation: on the complementarity of experimental and naturalistic studies\u201d, Journal of the American Society for Information Science and Technology, Vol. 52 No. 8, pp. 676\u201085."},{"key":"key2021010420210237900_b19","unstructured":"Taylor, A.R., Cool, C., Belkin, N.J. and Amadio, W.J. (2007), \u201cRelationships between categories of relevance criteria and stage in task completion\u201d, Information Processing and Management, Vol. 43 No. 4, pp. 1071\u201084."},{"key":"key2021010420210237900_b20","unstructured":"Tomlinson, S., Oard, D.W., Baron, J.R. and Thompson, P. (2007), \u201cOverview of the TREC 2007 Legal track\u201d, The 16th Text Retrieval Conference (TREC 2007) Proceedings, NIST Special Publication: 500\u2010274, National Institute of Standards and Technology, Gaithersburg, MD, pp. 51\u201084."},{"key":"key2021010420210237900_b21","unstructured":"Twait, M. (2005), \u201cUndergraduate students' source selection criteria: a qualitative study\u201d, The Journal of Academic Librarianship, Vol. 31 No. 6, pp. 567\u201073."},{"key":"key2021010420210237900_b22","unstructured":"Vakkari, P. and Hakala, N. (2000), \u201cChanges in relevance criteria and problem stages in task performance\u201d, Journal of Documentation, Vol. 56 No. 5, pp. 540\u201062."},{"key":"key2021010420210237900_b23","unstructured":"Xu, Y. and Chen, Z. (2006), \u201cRelevance judgment: what do information users consider beyond topicality?\u201d, Journal of the American Society for Information Science and Technology, Vol. 57 No. 7, pp. 961\u201073."},{"key":"key2021010420210237900_b24","unstructured":"Xu, Y. and Wang, D. (2008), \u201cOrder effect in relevance judgment\u201d, Journal of the American Society for Information Science and Technology, Vol. 59 No. 8, pp. 1264\u201075."}],"container-title":["Journal of Documentation"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/www.emeraldinsight.com\/doi\/full-xml\/10.1108\/00220411111109467","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/00220411111109467\/full\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/00220411111109467\/full\/html","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,24]],"date-time":"2025-07-24T23:38:19Z","timestamp":1753400299000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.emerald.com\/jd\/article\/67\/2\/264-278\/216524"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2011,3,8]]},"references-count":24,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2011,3,8]]}},"alternative-id":["10.1108\/00220411111109467"],"URL":"https:\/\/doi.org\/10.1108\/00220411111109467","relation":{},"ISSN":["0022-0418"],"issn-type":[{"value":"0022-0418","type":"print"}],"subject":[],"published":{"date-parts":[[2011,3,8]]}}}