{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,4]],"date-time":"2026-04-04T06:10:52Z","timestamp":1775283052441,"version":"3.50.1"},"reference-count":32,"publisher":"Emerald","issue":"4","license":[{"start":{"date-parts":[[2007,7,31]],"date-time":"2007-07-31T00:00:00Z","timestamp":1185840000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.emerald.com\/insight\/site-policies"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2007,7,31]]},"abstract":"<jats:sec><jats:title content-type=\"abstract-heading\">Purpose<\/jats:title><jats:p>The purpose of this paper is to examine how different aspects of an assessor's context, in particular their knowledge of a search topic, their interest in the search topic and their confidence in assessing relevance for a topic, affect the relevance judgements made and the assessor's ability to predict which documents they will assess as being relevant.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Design\/methodology\/approach<\/jats:title><jats:p>The study was conducted as part of the Text REtrieval Conference (TREC) HARD track. Using a specially constructed questionnaire information was sought on TREC assessors' personal context and, using the TREC assessments gathered, the responses were correlated to the questionnaire questions and the final relevance decisions.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Findings<\/jats:title><jats:p>This study found that each of the three factors (interest, knowledge and confidence) had an affect on how many documents were assessed as relevant and the balance between how many documents were marked as marginally or highly relevant. Also these factors are shown to affect an assessors' ability to predict what information they will finally mark as being relevant.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Research limitations\/implications<\/jats:title><jats:p>The major limitation is that the research is conducted within the TREC initiative. This means that we can report on results but cannot report on discussions with the assessors. The research implications are numerous but mainly on the effect of personal context on the outcomes of a user study.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Practical implications<\/jats:title><jats:p>One major consequence is that we should take more account of how we construct search tasks for IIR evaluation to create tasks that are interesting and relevant to experimental subjects.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Originality\/value<\/jats:title><jats:p>Examining different search variables within one study to compare the relative effects on these variables on the search outcomes.<\/jats:p><\/jats:sec>","DOI":"10.1108\/00220410710758986","type":"journal-article","created":{"date-parts":[[2007,7,21]],"date-time":"2007-07-21T07:02:24Z","timestamp":1185001344000},"page":"482-504","source":"Crossref","is-referenced-by-count":22,"title":["The relative effects of knowledge, interest and confidence in assessing relevance"],"prefix":"10.1108","volume":"63","author":[{"given":"Ian","family":"Ruthven","sequence":"first","affiliation":[]},{"given":"Mark","family":"Baillie","sequence":"additional","affiliation":[]},{"given":"David","family":"Elsweiler","sequence":"additional","affiliation":[]}],"member":"140","reference":[{"key":"key2022020520110504400_b1","unstructured":"Allan, J. (2005), \u201cHard track overview in TREC 2004: high accuracy retrieval from documents\u201d, Proceedings of 13th Text Retrieval Conference (TREC\u201013), Gaithersburg, 16\u201019 November, NIST Special Publication: SP 500\u2010261, NIST, Gaithersburg, MD."},{"key":"key2022020520110504400_b3","doi-asserted-by":"crossref","unstructured":"Baillie, M. and Ruthven, I. (2006), \u201cExamining assessor attributes at Hard 2005\u201d, Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in IR, Seattle. 6\u201011 August, ACM Press, New York, NY.","DOI":"10.1145\/1148170.1148279"},{"key":"key2022020520110504400_b2","unstructured":"Baillie, M., Elsweiler, D., Nicol, E., Ruthven, I., Sweeney, S., Yakici, M., Crestani, F. and Landoni, M. (2006), \u201cUniversity of Strathclyde: the i\u2010lab's first big day out at TREC HARD\u201d, in Voorhees, E.M. and Buckland, L.P. (Eds), Proceedings of the 14th Text Retrieval Conference (TREC\u201014), Gaithersburg. 15\u201018 November, NIST Special Publication no. 500\u2010266, NIST, Gaithersburg, MD."},{"key":"key2022020520110504400_b4","doi-asserted-by":"crossref","unstructured":"Barry, C.L. and Schamber, L. (1998), \u201cUsers' criteria for relevance evaluation: a cross\u2010situational comparison\u201d, Information Processing and Management, Vol. 34 Nos 2\/3, pp. 219\u201036.","DOI":"10.1016\/S0306-4573(97)00078-2"},{"key":"key2022020520110504400_b5","doi-asserted-by":"crossref","unstructured":"Borlund, P. (2003), \u201cThe concept of relevance in IR\u201d, Journal of the American Society for Information Science and Technology, Vol. 54 No. 10, pp. 913\u201025.","DOI":"10.1002\/asi.10286"},{"key":"key2022020520110504400_b6","doi-asserted-by":"crossref","unstructured":"Brandow, R., Mitze, K. and Rau, L.F. (1995), \u201cAutomatic condensation of electronic publications by sentence selection\u201d, Information Processing and Management, Vol. 31 No. 5, pp. 675\u201085.","DOI":"10.1016\/0306-4573(95)00052-I"},{"key":"key2022020520110504400_b7","unstructured":"Buckley, C. and Voorhees, E.M. (2005), \u201cRetrieval systems evaluation\u201d, in Voorhees, E.M. and Harman, D.K. (Eds), TREC Experiment and Evaluation in Information Retrieval, MIT Press, Cambridge, MA, pp. 53\u201078)."},{"key":"key2022020520110504400_b8","doi-asserted-by":"crossref","unstructured":"Choi, Y. and Rasumussen, E.M. (2002), \u201cUsers' relevance criteria in image retrieval in American history\u201d, Information Processing & Management, Vol. 38 No. 5, pp. 695\u2010726.","DOI":"10.1016\/S0306-4573(01)00059-0"},{"key":"key2022020520110504400_b9","doi-asserted-by":"crossref","unstructured":"Eisenberg, M. and Barry, C. (1998), \u201cOrder effects: a study of the possible influence of presentation order on user judgements of document relevance\u201d, Journal of the American Society of Information Science., Vol. 39 No. 5, pp. 293\u2010300.","DOI":"10.1002\/(SICI)1097-4571(198809)39:5<293::AID-ASI1>3.0.CO;2-I"},{"key":"key2022020520110504400_b10","unstructured":"Eisenberg, M. and Hu, X. (1987), \u201cDichotomous relevance judgments and the evaluation of information systems\u201d, Proceedings of the American Society for Information Science 50th Annual Meeting, Boston, 4\u20108 October, Learned Information, Medford, NJ, pp. 66\u20109."},{"key":"key2022020520110504400_b11","doi-asserted-by":"crossref","unstructured":"Florance, V. and Marchionini, G. (1995), \u201cInformation processing in the context of medical care\u201d, in Fox, E.A., Ingwerson, P. and Fide, R. (Eds), Proceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in IR. Seattle. 9\u201013 July, ACM Press, New York, NY, pp. 158\u201063.","DOI":"10.1145\/215206.215353"},{"key":"key2022020520110504400_b12","unstructured":"Harper, D.J., Muresan, G., Liu, B., Koychev, D., Wettschereck, D. and Wiratunga, N. (2005), \u201cThe Robert Gordon University's Hard Track experiments at TREC 2004\u201d, Proceedings of 13th Text Retrieval Conference (TREC\u201013), Gaithersburg, 16\u201019 November, NIST Special Publication: SP 500\u2010261, NIST, Gaithersburg, MD."},{"key":"key2022020520110504400_b13","doi-asserted-by":"crossref","unstructured":"Hsieh\u2010Yee, I. (1993), \u201cEffects of search experience and subject knowledge on the search tactics of novice and experienced searchers\u201d, Journal of the American Society for Information Science, Vol. 44 No. 3, pp. 161\u201074.","DOI":"10.1002\/(SICI)1097-4571(199304)44:3<161::AID-ASI5>3.0.CO;2-8"},{"key":"key2022020520110504400_b14","doi-asserted-by":"crossref","unstructured":"Huang, M.\u2010H. and Wang, H.\u2010Y. (2004), \u201cThe influence of document presentation order and number of documents judged on users' judgements of relevance\u201d, Journal of the American Society for Information Science and Technology, Vol. 55 No. 11, pp. 970\u20109.","DOI":"10.1002\/asi.20047"},{"key":"key2022020520110504400_b15","doi-asserted-by":"crossref","unstructured":"J\u00e4rvelin, K. and Kek\u00e4l\u00e4inen, J. (2000), \u201cIR evaluation methods for retrieving highly relevant documents\u201d, Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in IR, Athens, 24\u201028 July, ACM Press, New York, NY, pp. 41\u20108.","DOI":"10.1145\/345508.345545"},{"key":"key2022020520110504400_b16","doi-asserted-by":"crossref","unstructured":"Katter, R.V. (1968), \u201cThe influence of scale form on relevance judgments\u201d, Information Storage and Retrieval., Vol. 4 No. 1, pp. 1\u201011.","DOI":"10.1016\/0020-0271(68)90002-8"},{"key":"key2022020520110504400_b17","doi-asserted-by":"crossref","unstructured":"Lesk, M.E. and Salton, G. (1969), \u201cRelevance assessments and retrieval system evaluation\u201d, Information Storage and Retrieval, Vol. 4, pp. 343\u201059.","DOI":"10.1016\/0020-0271(68)90029-6"},{"key":"key2022020520110504400_b18","doi-asserted-by":"crossref","unstructured":"Michel, D. (1994), \u201cWhat is used during cognitive processing in information retrieval and library searching? Eleven sources of search information\u201d, Journal of the American Society for Information Science and Technology, Vol. 45 No. 7, pp. 498\u2010514.","DOI":"10.1002\/(SICI)1097-4571(199408)45:7<498::AID-ASI5>3.0.CO;2-T"},{"key":"key2022020520110504400_b19","doi-asserted-by":"crossref","unstructured":"Mizzaro, S. (1997), \u201cRelevance: the whole history\u201d, Journal of the American Society for Information Science and Technology, Vol. 48 No. 9, pp. 810\u201032.","DOI":"10.1002\/(SICI)1097-4571(199709)48:9<810::AID-ASI6>3.0.CO;2-U"},{"key":"key2022020520110504400_b20","doi-asserted-by":"crossref","unstructured":"Mizzaro, S. (1999), \u201cMeasuring the agreement among relevance judges\u201d, in Draper, S.W., Dunlop, M.D., Ruthven, I. and Rijsbergen, C.J. (Eds), Proceedings of the Final Mira Workshop (Mira '99), Glasgow, 14\u201016 April, published in Electronic Workshops in Computing, April, pp. 1\u201013.","DOI":"10.14236\/ewic\/MIRA1999.5"},{"key":"key2022020520110504400_b21","unstructured":"Ogilvie, P. and Callan, J.P. (2002), \u201cExperiments using the lemur toolkit\u201d, Proceedings of 10th Text Retrieval Conference (TREC\u201010), Gaithersburg, 13\u201016 November, NIST Special Publication: SP 500\u2010250, NIST, Gaithersburg, MD, pp. 103\u20108."},{"key":"key2022020520110504400_b22","doi-asserted-by":"crossref","unstructured":"Ruthven, I. (2005), \u201cIntegrating approaches to relevance\u201d, in Spink, A. and Cole, C. (Eds), New Directions in Cognitive Information Retrieval, Information Retrieval Series, Vol. 19, Springer, Dordrecht, pp. 61\u201080.","DOI":"10.1007\/1-4020-4014-8_4"},{"key":"key2022020520110504400_b23","doi-asserted-by":"crossref","unstructured":"Sormunen, E. (2002), \u201cLiberal relevance criteria of TREC \u2013 counting on negligible documents?\u201d, Proceedings of the 25th Annual International Conference on Research and Development in IR, Tampere. 11\u201015 August, ACM Press, New York, NY, pp. 324\u201030.","DOI":"10.1145\/564376.564433"},{"key":"key2022020520110504400_b24","doi-asserted-by":"crossref","unstructured":"Spink, A., Greisdorf, H. and Bateman, J. (1998), \u201cFrom highly relevant to not relevant: examining different regions of relevance\u201d, Information Processing and Management, Vol. 34 No. 5, pp. 599\u2010621.","DOI":"10.1016\/S0306-4573(98)00025-9"},{"key":"key2022020520110504400_b25","doi-asserted-by":"crossref","unstructured":"Tiamiyu, M.A. and Ajiferuke, I.Y. (1988), \u201cA total relevance and document interaction effects model for the evaluation of IR processes\u201d, Information Processing and Management, Vol. 24 No. 4, pp. 391\u2010404.","DOI":"10.1016\/0306-4573(88)90043-X"},{"key":"key2022020520110504400_b26","doi-asserted-by":"crossref","unstructured":"Tombros, A. and Crestani, F. (2000), \u201cUsers' perception of relevance of spoken documents\u201d, Journal of the American Society for Information Science, Vol. 51 No. 9, pp. 929\u201039.","DOI":"10.1002\/1097-4571(2000)51:10<929::AID-ASI50>3.0.CO;2-D"},{"key":"key2022020520110504400_b27","doi-asserted-by":"crossref","unstructured":"Tombros, A., Ruthven, I. and Jose, J.M. (2005), \u201cHow users assess web pages for information\u2010seeking\u201d, Journal of the American Society for Information Science and Technology, Vol. 56 No. 5, pp. 327\u201044.","DOI":"10.1002\/asi.20106"},{"key":"key2022020520110504400_b28","doi-asserted-by":"crossref","unstructured":"Vakkari, P. and Hakala, N. (2000), \u201cChanges in relevance criteria and problem stages in task performance\u201d, Journal of Documentation, Vol. 56 No. 5, pp. 540\u201062.","DOI":"10.1108\/EUM0000000007127"},{"key":"key2022020520110504400_b29","doi-asserted-by":"crossref","unstructured":"Vakkari, P. and Sormunen, S. (2004), \u201cThe influence of relevance levels on the effectiveness of interactive information retrieval\u201d, Journal of the American Society for Information Science, Vol. 55 No. 11, pp. 963\u20109.","DOI":"10.1002\/asi.20046"},{"key":"key2022020520110504400_b30","doi-asserted-by":"crossref","unstructured":"Voorhees, E.M. (2001), \u201cEvaluation by highly relevant documents\u201d, Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, New Orleans, 10\u201012 September, ACM Press, New York, NY, pp. 74\u201082.","DOI":"10.1145\/383952.383963"},{"key":"key2022020520110504400_b31","doi-asserted-by":"crossref","unstructured":"Voorhees, E.M. and Buckland, L.P. (Eds) (2006), Proceedings of the Fourteenth Text Retrieval Conference (TREC 2005), NIST, Gaithersburg, MD.","DOI":"10.6028\/NIST.SP.500-266"},{"key":"key2022020520110504400_b32","doi-asserted-by":"crossref","unstructured":"White, R.W., Ruthven, I. and Jose, J.M. (2002), \u201cFinding relevant documents using top\u2010ranking sentences: an evaluation of two alternative schemes\u201d, Proceedings of the 25th Annual International Conference on Research and Development in IR, Tampere, 11\u201015 August, ACM Press, New York, NY, pp. 57\u201064.","DOI":"10.1145\/564376.564389"}],"container-title":["Journal of Documentation"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/www.emeraldinsight.com\/doi\/full-xml\/10.1108\/00220410710758986","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/00220410710758986\/full\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/00220410710758986\/full\/html","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,24]],"date-time":"2025-07-24T23:37:49Z","timestamp":1753400269000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.emerald.com\/jd\/article\/63\/4\/482-504\/213721"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2007,7,31]]},"references-count":32,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2007,7,31]]}},"alternative-id":["10.1108\/00220410710758986"],"URL":"https:\/\/doi.org\/10.1108\/00220410710758986","relation":{},"ISSN":["0022-0418"],"issn-type":[{"value":"0022-0418","type":"print"}],"subject":[],"published":{"date-parts":[[2007,7,31]]}}}