{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,25]],"date-time":"2026-03-25T00:50:53Z","timestamp":1774399853420,"version":"3.50.1"},"reference-count":38,"publisher":"Emerald","issue":"6","license":[{"start":{"date-parts":[[2014,10,7]],"date-time":"2014-10-07T00:00:00Z","timestamp":1412640000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.emerald.com\/insight\/site-policies"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2014,10,7]]},"abstract":"\n Purpose<\/jats:title>\n \u2013 The purpose of this paper is to examine how various types of TREC data can be used to better understand relevance and serve as test-bed for exploring relevance. The author proposes that there are many interesting studies that can be performed on the TREC data collections that are not directly related to evaluating systems but to learning more about human judgements of information and relevance and that these studies can provide useful research questions for other types of investigation. <\/jats:p>\n <\/jats:sec>\n \n Design\/methodology\/approach<\/jats:title>\n \u2013 Through several case studies the author shows how existing data from TREC can be used to learn more about the factors that may affect relevance judgements and interactive search decisions and answer new research questions for exploring relevance. <\/jats:p>\n <\/jats:sec>\n \n Findings<\/jats:title>\n \u2013 The paper uncovers factors, such as familiarity, interest and strictness of relevance criteria, that affect the nature of relevance assessments within TREC, contrasting these against findings from user studies of relevance. <\/jats:p>\n <\/jats:sec>\n \n Research limitations\/implications<\/jats:title>\n \u2013 The research only considers certain uses of TREC data and assessment given by professional relevance assessors but motivates further exploration of the TREC data so that the research community can further exploit the effort involved in the construction of TREC test collections. <\/jats:p>\n <\/jats:sec>\n \n Originality\/value<\/jats:title>\n \u2013 The paper presents an original viewpoint on relevance investigations and TREC itself by motivating TREC as a source of inspiration on understanding relevance rather than purely as a source of evaluation material.<\/jats:p>\n <\/jats:sec>","DOI":"10.1108\/jd-02-2014-0031","type":"journal-article","created":{"date-parts":[[2014,10,9]],"date-time":"2014-10-09T06:52:11Z","timestamp":1412837531000},"page":"1098-1117","source":"Crossref","is-referenced-by-count":9,"title":["Relevance behaviour in TREC"],"prefix":"10.1108","volume":"70","author":[{"given":"Ian","family":"Ruthven","sequence":"first","affiliation":[]}],"member":"140","reference":[{"key":"key2023032015222004200_b1","unstructured":"Allan, J.\n (2003), \u201cHARD track overview in TREC 2003: high accuracy retrieval from documents\u201d, in \n Voorhees, E.M.\n and \n Buckland, L.P.\n (Eds), Proceedings of the Twelfth Text REtrieval Conference, TREC, Special Publication 500-255 National Institute of Standards and Technology (NIST), Gaithersburg, pp. 24-37."},{"key":"key2023032015222004200_b2","doi-asserted-by":"crossref","unstructured":"Allan, J.\n (2004), \u201cHARD track overview in TREC 2004 \u2013 high accuracy retrieval from documents\u201d, in \n Voorhees, E.M.\n and \n Buckland, L.P.\n (Eds), Proceedings of the Thirteenth Text REtrieval Conference, TREC, Special Publication 500-261 National Institute of Standards and Technology (NIST), Gaithersburg, pp. 25-35.","DOI":"10.21236\/ADA455426"},{"key":"key2023032015222004200_b3","doi-asserted-by":"crossref","unstructured":"Allan, J.\n (2005), \u201cHARD track overview in TREC 2005 high accuracy retrieval from documents\u201d, in \n Voorhees, E.M.\n and \n Buckland, L.P.\n (Eds), Proceedings of the Fourteenth Text REtrieval Conference, TREC, Special Publication 500-266 National Institute of Standards and Technology (NIST), Gaithersburg, pp. 51-68.","DOI":"10.21236\/ADA439446"},{"key":"key2023032015222004200_b4","doi-asserted-by":"crossref","unstructured":"Barry, C.L.\n and \n Schamber, L.\n (1998), \u201cUsers\u2019 criteria for relevance evaluation: a cross-situational comparison\u201d, Information Processing & Management, Vol. 34 Nos 2\/3, pp. 219-236.","DOI":"10.1016\/S0306-4573(97)00078-2"},{"key":"key2023032015222004200_b5","unstructured":"Bateman, J.\n (1998), \u201cChanges in relevance criteria: a longitudinal study\u201d, in \n Preston, C.M.\n (Ed.), Proceedings of the ASIS Annual Meeting, American Society for Information Science and Technology, Pittsburgh, PA, Vol. 35, pp. 23-32."},{"key":"key2023032015222004200_b6","doi-asserted-by":"crossref","unstructured":"Choi, Y.\n and \n Rasmussen, E.\n (2002), \u201cUsers\u2019 relevance criteria in image retrieval in American history\u201d, Information Processing & Management, Vol. 38 No. 5, pp. 695-726.","DOI":"10.1016\/S0306-4573(01)00059-0"},{"key":"key2023032015222004200_b7","doi-asserted-by":"crossref","unstructured":"Chu, H.\n (2007), \u201cFactors affecting relevance judgment: a report from the TREC legal track\u201d, Journal of Documentation, Vol. 6 No. 2, pp. 264-278.","DOI":"10.1108\/00220411111109467"},{"key":"key2023032015222004200_b8","doi-asserted-by":"crossref","unstructured":"Compton, R.\n , \n Banich, M.\n , \n Mohanty, A.\n , \n Milham, M.\n , \n Herrington, J.\n , \n Miller, G.\n , \n Scalf, P.\n , \n Webb, A.\n and \n Heller, W.\n (2003), \u201cPaying attention to emotion: an fMRI investigation of cognitive and emotional Stroop tasks\u201d, Cognitive, Affective, & Behavioral Neuroscience, Vol. 3 No. 2, pp. 81-96.","DOI":"10.3758\/CABN.3.2.81"},{"key":"key2023032015222004200_b9","doi-asserted-by":"crossref","unstructured":"Eisenberg, M.\n and \n Barry, C.\n (1988), \u201cOrder effects: a study of the possible influence of presentation order on user judgements of document relevance\u201d, Journal of the American Society of Information Science, Vol. 39 No. 5, pp. 293-300.","DOI":"10.1002\/(SICI)1097-4571(198809)39:5<293::AID-ASI1>3.0.CO;2-I"},{"key":"key2023032015222004200_b10","unstructured":"Fang, X.\n (2001), \u201cWeb searching behavior: selection of search terms\u201d, in \n Cheok, A.D.\n and \n Chittaro, L.\n (Eds), Proceedings of the Ninth International Conference on Human-Computer Interaction, Vol. 1, ACM, New York, NY, pp. 898-902."},{"key":"key2023032015222004200_b11","doi-asserted-by":"crossref","unstructured":"Florance, V.\n and \n Marchionini, G.\n (1995), \u201cInformation processing in the context of medical care\u201d, in \n Fox, E.A.\n , \n Ingwersen, P.\n and \n Fidel, R.\n (Eds), Proceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, New York, NY, pp. 158-163.","DOI":"10.1145\/215206.215353"},{"key":"key2023032015222004200_b12","doi-asserted-by":"crossref","unstructured":"Ford, N.\n (1999), \u201cThe growth of understanding in information science: towards a developmental model\u201d, Journal of the American Society for Information Science, Vol. 50 No. 12, pp. 1141-1152.","DOI":"10.1002\/(SICI)1097-4571(1999)50:12<1141::AID-ASI19>3.0.CO;2-X"},{"key":"key2023032015222004200_b13","doi-asserted-by":"crossref","unstructured":"Ford, N.\n , \n Miller, D.\n and \n Moss, N.\n (2001), \u201cThe role of individual differences in internet searching: an empirical study\u201d, Journal of the American Society for Information Science and Technology, Vol. 52 No. 12, pp. 1049-1066.","DOI":"10.1002\/asi.1165"},{"key":"key2023032015222004200_b14","doi-asserted-by":"crossref","unstructured":"Gwizdka, J.\n and \n Lopatovska, I.\n (2009), \u201cThe role of subjective factors in the information search process\u201d, Journal of the American Society for Information Science and Technology, Vol. 60 No. 12, pp. 2452-2464.","DOI":"10.1002\/asi.21183"},{"key":"key2023032015222004200_b15","doi-asserted-by":"crossref","unstructured":"Harter, S.P.\n (1996), \u201cVariations in relevance assessments and the measurement of retrieval effectiveness\u201d, Journal of the American Society for Information Science, Vol. 47 No. 1, pp. 37-49.","DOI":"10.1002\/(SICI)1097-4571(199601)47:1<37::AID-ASI4>3.0.CO;2-3"},{"key":"key2023032015222004200_b16","doi-asserted-by":"crossref","unstructured":"Huang, M.-H.\n and \n Wang, H.-Y.\n (2004), \u201cThe influence of document presentation order and number of documents judged on users\u2019 judgements of relevance\u201d, Journal of the American Society for Information Science and Technology, Vol. 55 No. 11, pp. 970-979.","DOI":"10.1002\/asi.20047"},{"key":"key2023032015222004200_b17","doi-asserted-by":"crossref","unstructured":"Kelly, D.\n , \n Fu, X.\n and \n Shah, C.\n (2010), \u201cEffects of position and number of relevant documents retrieved on users\u2019 evaluations of system performance\u201d, ACM Transactions on Information Systems, Vol. 28 No. 2, pp. 1-29.","DOI":"10.1145\/1740592.1740597"},{"key":"key2023032015222004200_b18","doi-asserted-by":"crossref","unstructured":"Lau-Gesk, L.\n and \n Meyers-Levy, J.\n (2009), \u201cEmotional persuasion: when the valence versus the resource demands of emotions influence consumers\u2019 attitudes\u201d, Journal of Consumer Research, Vol. 36 No. 4, pp. 585-599.","DOI":"10.1086\/605297"},{"key":"key2023032015222004200_b19","doi-asserted-by":"crossref","unstructured":"Lin, J.\n and \n Zhang, P.\n (2007), \u201cDeconstructing nuggets: the stability and reliability of complex question answering evaluation\u201d, in \n Clarke, C.L.\n A., \n Fuhr, N.\n , \n Kando, N.\n , \n Kraaijj, W.\n and \n de Vries, A.P.\n (Eds), Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, New York, NY, pp. 327-334.","DOI":"10.1145\/1277741.1277799"},{"key":"key2023032015222004200_b20","doi-asserted-by":"crossref","unstructured":"Ruthven, I.\n (2005), \u201cIntegrating approaches to relevance\u201d, in \n Spink, A.\n and \n Cole, C.\n (Eds), New Directions in Cognitive Information Retrieval, Springer, Dordrecht, pp. 61-80.","DOI":"10.1007\/1-4020-4014-8_4"},{"key":"key2023032015222004200_b21","doi-asserted-by":"crossref","unstructured":"Ruthven, I.\n , \n Baillie, M.\n and \n Elsweiler, D.\n (2007), \u201cThe relative effects of knowledge, interest and confidence in assessing relevance\u201d, Journal of Documentation, Vol. 63 No. 4, pp. 482-504.","DOI":"10.1108\/00220410710758986"},{"key":"key2023032015222004200_b22","unstructured":"Saracevic, T.\n (1996), \u201cRelevance reconsidered. Information science: integration in perspectives\u201d, in \n Ingwersen, P.\n and \n Pors, N.O.\n (Eds), Proceedings of the Second Conference on Conceptions of Library and Information Science, The Royal School of Librarianship, Copenhagen, pp. 201-218."},{"key":"key2023032015222004200_b23","doi-asserted-by":"crossref","unstructured":"Saracevic, T.\n (2007a), \u201cRelevance: a review of the literature and a framework for thinking on the notion in information science. Part II: nature and manifestation of relevance\u201d, Journal of the American Society for Information Science and Technology, Vol. 58 No. 13, pp. 1915-1933.","DOI":"10.1002\/asi.20682"},{"key":"key2023032015222004200_b24","doi-asserted-by":"crossref","unstructured":"Saracevic, T.\n (2007b), \u201cRelevance: a review of the literature and a framework for thinking on the notion in information science. Part III: behavior and effects of relevance\u201d, Journal of the American Society for Information Science and Technology, Vol. 58 No. 13, pp. 2126-2144.","DOI":"10.1002\/asi.20681"},{"key":"key2023032015222004200_b25","doi-asserted-by":"crossref","unstructured":"Saracevic, T.\n and \n Kantor, P.\n (1988), \u201cA study of information seeking and retrieving. II. Users, questions, and effectiveness\u201d, Journal of the American Society for Information Science, Vol. 39 No. 3, pp. 177-196.","DOI":"10.1002\/(SICI)1097-4571(198805)39:3<177::AID-ASI3>3.0.CO;2-F"},{"key":"key2023032015222004200_b26","doi-asserted-by":"crossref","unstructured":"Scholer, F.\n , \n Turpin, A.\n and \n Sanderson, M.\n (2011), \u201cQuantifying test collection quality based on the consistency of relevance judgements\u201d, in \n Ma, W.-Y.\n , \n Nie, J.-Y.\n , \n Baeza-Yates, R.\n , \n Chua, T.-S.\n and \n Croft, W.B.\n (Eds), Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, New York, NY, pp. 1063-1072.","DOI":"10.1145\/2009916.2010057"},{"key":"key2023032015222004200_b27","doi-asserted-by":"crossref","unstructured":"Sherry, A.\n and \n Henson, R.K.\n (2005), \u201cConducting and interpreting canonical correlation analysis in personality research: a user-friendly primer\u201d, Journal of Personality Assessment, Vol. 84 No. 1, pp. 37-48.","DOI":"10.1207\/s15327752jpa8401_09"},{"key":"key2023032015222004200_b28","doi-asserted-by":"crossref","unstructured":"Smith, C.L.\n and \n Kantor, P.B.\n (2008), \u201cUser adaptation: good results from poor systems\u201d, in \n Myaeng, S.H.\n , \n Oard, D.W.\n , \n Sebastiani, F.\n , \n Chau, T.-S.\n and \n Leong, M.-K.\n (Eds), Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, New York, NY, pp. 147-154.","DOI":"10.1145\/1390334.1390362"},{"key":"key2023032015222004200_b29","doi-asserted-by":"crossref","unstructured":"Smucker, M.D.\n and \n Jethani, C.P.\n (2010), \u201cHuman performance and retrieval precision revisited\u201d, in \n Chen, H.-H.\n , \n Efthimiadis, E.N.\n , \n Savoy, J.\n , \n Crestani, F.\n and \n Marchand-Mallet, S.\n (Eds), Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, New York, NY, pp. 595-602.","DOI":"10.1145\/1835449.1835549"},{"key":"key2023032015222004200_b30","doi-asserted-by":"crossref","unstructured":"Smucker, M.D.\n and \n Jethani, C.P.\n (2011), \u201cMeasuring assessor accuracy: a comparison of NIST assessors and user study participants\u201d, in \n Ma, W.-Y.\n , \n Nie, J.-Y.\n , \n Baeza-Yates, R.\n , \n Chua, T.-S.\n and \n Croft, W.B.\n (Eds), Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, New York, NY, pp. 1231-1232.","DOI":"10.1145\/2009916.2010134"},{"key":"key2023032015222004200_b32","doi-asserted-by":"crossref","unstructured":"Sormunen, E.\n (2002), \u201cLiberal relevance criteria of TREC: counting on negligible documents?\u201d, in \n J\u00e4rvelin, K.\n , \n Beaulieu, M.\n , \n Greisdorf, H.\n and \n Bateman, J.\n (Eds), Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, New York, NY, pp. 324-330.","DOI":"10.1145\/564376.564433"},{"key":"key2023032015222004200_b31","doi-asserted-by":"crossref","unstructured":"Sp\u00e4rck Jones, K.\n (2006), \u201cWhat's the value of TREC: is there a gap to jump or a chasm to bridge?\u201d, SIGIR Forum, Vol. 40 No. 1, pp. 10-20.","DOI":"10.1145\/1147197.1147198"},{"key":"key2023032015222004200_b33","doi-asserted-by":"crossref","unstructured":"Spink, A.\n , \n Greisdorf, H.\n and \n Bateman, J.\n (1998), \u201cFrom highly relevant to not relevant: examining different regions relevance\u201d, Information Processing & Management, Vol. 34 No. 5, pp. 599-621.","DOI":"10.1016\/S0306-4573(98)00025-9"},{"key":"key2023032015222004200_b34","doi-asserted-by":"crossref","unstructured":"Turpin, A.H.\n and \n Hersh, W.\n (2001), \u201cWhy batch and user evaluations do not give the same results\u201d, in \n Kraft, D.H.\n , \n Croft, W.B.\n , \n Harper, D.J.\n and \n Zobel, J.\n (Eds), Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, New York, NY, pp. 225-231.","DOI":"10.1145\/383952.383992"},{"key":"key2023032015222004200_b35","doi-asserted-by":"crossref","unstructured":"Vakkari, P.\n and \n Hakala, N.\n (2000), \u201cChanges in relevance criteria and problem stages in task performance\u201d, Journal of Documentation, Vol. 56 No. 5, pp. 540-562.","DOI":"10.1108\/EUM0000000007127"},{"key":"key2023032015222004200_b36","doi-asserted-by":"crossref","unstructured":"Voorhees, E.M.\n (2000), \u201cVariations in relevance judgments and the measurement of retrieval effectiveness\u201d, Information Processing & Management, Vol. 36 No. 5, pp. 697-716.","DOI":"10.1016\/S0306-4573(00)00010-8"},{"key":"key2023032015222004200_b37","doi-asserted-by":"crossref","unstructured":"Voorhees, E.M.\n (2001), \u201cEvaluation by highly relevant documents\u201d, in \n Kraft, D.H.\n , \n Croft, W.B.\n , \n Harper, D.J.\n and \n Zobel, J.\n (Eds), Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, New York, NY, pp. 74-82.","DOI":"10.1145\/383952.383963"},{"key":"key2023032015222004200_b38","unstructured":"Voorhees, E.M.\n and \n Harman, D.K.\n (2005), TREC: Experiment and Evaluation in Information Retrieval, The MIT Press, Cambridge, MA."}],"container-title":["Journal of Documentation"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/www.emeraldinsight.com\/doi\/full-xml\/10.1108\/JD-02-2014-0031","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/JD-02-2014-0031\/full\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/JD-02-2014-0031\/full\/html","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,24]],"date-time":"2025-07-24T22:33:16Z","timestamp":1753396396000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.emerald.com\/jd\/article\/70\/6\/1098-1117\/203132"}},"subtitle":[],"editor":[{"given":"Professor","family":"Peter Willett","sequence":"first","affiliation":[]}],"short-title":[],"issued":{"date-parts":[[2014,10,7]]},"references-count":38,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2014,10,7]]}},"alternative-id":["10.1108\/JD-02-2014-0031"],"URL":"https:\/\/doi.org\/10.1108\/jd-02-2014-0031","relation":{},"ISSN":["0022-0418"],"issn-type":[{"value":"0022-0418","type":"print"}],"subject":[],"published":{"date-parts":[[2014,10,7]]}}}