{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,1]],"date-time":"2025-11-01T21:30:30Z","timestamp":1762032630046,"version":"3.41.2"},"reference-count":20,"publisher":"Emerald","issue":"5","license":[{"start":{"date-parts":[[2005,10,1]],"date-time":"2005-10-01T00:00:00Z","timestamp":1128124800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.emerald.com\/insight\/site-policies"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2005,10,1]]},"abstract":"<jats:sec><jats:title content-type=\"abstract-heading\">Purpose<\/jats:title><jats:p>This paper aims to investigate how readers assess relevance of retrieved documents in a foreign language they know well compared with their native language, and whether work\u2010task scenario descriptions have effect on the assessment process.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Design\/methodology\/approach<\/jats:title><jats:p>Queries, test collections, and relevance assessments were used from the 2002 Interactive CLEF. Swedish first\u2010language speakers, fluent in English, were given simulated information\u2010seeking scenarios and presented with retrieval results in both languages. Twenty\u2010eight subjects in four groups were asked to rate the retrieved text documents by relevance. A two\u2010level work\u2010task scenario description framework was developed and applied to facilitate the study of context effects on the assessment process.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Findings<\/jats:title><jats:p>Relevance assessment takes longer in a foreign language than in the user first language. The quality of assessments by comparison with pre\u2010assessed results is inferior to those made in the users' first language. Work\u2010task scenario descriptions had an effect on the assessment process, both by measured access time and by self\u2010report by subjects. However, effects on results by traditional relevance ranking were detectable. This may be an argument for extending the traditional IR experimental topical relevance measures to cater for context effects.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Originality\/value<\/jats:title><jats:p>An extended two\u2010level work\u2010task scenario description framework was developed and applied. Contextual aspects had an effect on the relevance assessment process. English texts took longer to assess than Swedish and were assessed less well, especially for the most difficult queries. The IR research field needs to close this gap and to design information access systems with users' language competence in mind.<\/jats:p><\/jats:sec>","DOI":"10.1108\/00220410510625831","type":"journal-article","created":{"date-parts":[[2007,1,15]],"date-time":"2007-01-15T15:39:04Z","timestamp":1168875544000},"page":"623-639","source":"Crossref","is-referenced-by-count":19,"title":["Effects of foreign language and task scenario on relevance assessment"],"prefix":"10.1108","volume":"61","author":[{"given":"Preben","family":"Hansen","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jussi","family":"Karlgren","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"140","reference":[{"key":"key2022030819454699700_b1","doi-asserted-by":"crossref","unstructured":"Barry, C.L. (1994), \u201cUser\u2010defined relevance criteria: an exploratory study\u201d, Journal of the American Society for Information Science, Vol. 45, pp. 149\u201059.","DOI":"10.1002\/(SICI)1097-4571(199404)45:3<149::AID-ASI5>3.0.CO;2-J"},{"key":"key2022030819454699700_b2","doi-asserted-by":"crossref","unstructured":"Barry, C.L. and Schamber, L. (1998), \u201cUsers' criteria for relevance evaluation: a cross\u2010situational comparison\u201d, Information Processing and Management, Vol. 24 Nos 2\u20103, pp. 219\u201036.","DOI":"10.1016\/S0306-4573(97)00078-2"},{"key":"key2022030819454699700_b3","unstructured":"Borlund, P. (2000), \u201cEvaluation of interactive information retrieval systems\u201d, doctoral dissertation, \u00c5bo Academi, \u00c5bo."},{"key":"key2022030819454699700_b4","doi-asserted-by":"crossref","unstructured":"Brajnik, G., Mizzaro, S. and Tasso, C. (1996), \u201cEvaluating user interfaces to information retrieval systems. A case study on user support\u201d, in Frei, H\u2010P., Harman, D., Sch\u00e4uble, P. and Wilkinson, R. (Eds), Proceedings of the 19th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR'96), Zurich, 18\u201022 August, pp. 128\u201036.","DOI":"10.1145\/243199.243249"},{"key":"key2022030819454699700_b5","unstructured":"Cleverdon, C., Mills, J. and Keen, M. (1966), ASLIB Cranfield Research Project: Factors Determining the Performance of Indexing Systems, Aslib, London."},{"key":"key2022030819454699700_b7","doi-asserted-by":"crossref","unstructured":"Cosijn, E. and Ingwersen, P. (2000), \u201cDimensions of relevance\u201d, Information Processing & Management, Vol. 36, pp. 533\u201050.","DOI":"10.1016\/S0306-4573(99)00072-2"},{"key":"key2022030819454699700_b8","doi-asserted-by":"crossref","unstructured":"Gonzalo, J. and Oard, D. (2003), \u201cInteractive track\u201d, in Peters, C., Braschler, M., Gonzalo, J. and Kluck, M. (Eds), Advances in Cross\u2010Language Information Retrieval, Proceedings of 3rd Workshop of the Cross\u2010Language Evaluation Forum, CLEF 2002, Rome, September, Lecture Notes in Computer Science 2785, Springer, Berlin, pp. 372\u201082.","DOI":"10.1007\/978-3-540-45237-9_33"},{"key":"key2022030819454699700_b9","unstructured":"Hackos, J.T. and Redish, J.C. (1998), User and Task Analysis for Interface Design, Wiley, New York, NY."},{"key":"key2022030819454699700_b10","unstructured":"Hansen, P. and J\u00e4rvelin, K. (2000), \u201cThe information\u2010seeking and retrieval process at the Swedish patent \u2013 and registration office. Moving from lab\u2010based to real life work\u2010task environment\u201d, Proceedings of the ACM\u2010SIGIR 2000 Workshop on Patent Retrieval, Athens, July, pp. 43\u201053."},{"key":"key2022030819454699700_b11","unstructured":"Hearst, M. (1999), \u201cUser interfaces and visualization\u201d, in Baeza\u2010Yates, R. and Ribeiro\u2010Neto, B. (Eds), Modern Information Retrieval, ACM Press, New York, NY, pp. 257\u2010323."},{"key":"key2022030819454699700_b12","doi-asserted-by":"crossref","unstructured":"Kek\u00e4l\u00e4inen, J. and J\u00e4rvelin, K. (2002), \u201cUsing graded relevance assessments in IR evaluation\u201d, Journal of the American Society for Information Science and Technology, Vol. 53 No. 13, pp. 1120\u20109.","DOI":"10.1002\/asi.10137"},{"key":"key2022030819454699700_b13","doi-asserted-by":"crossref","unstructured":"Mizzaro, S. (1997), \u201cRelevance: the whole history\u201d, Journal of the American Society for Information Science, Vol. 48, pp. 810\u201032.","DOI":"10.1002\/(SICI)1097-4571(199709)48:9<810::AID-ASI6>3.0.CO;2-U"},{"key":"key2022030819454699700_b14","doi-asserted-by":"crossref","unstructured":"Oard, D. (2001), \u201cEvaluating interactive cross\u2010language information retrieval: document selection\u201d, in Peters, C. (Ed.), Cross\u2010Language Information Retrieval and Evaluation: Workshop of Cross\u2010Language Evaluation Forum, CLEF, Lisbon, 21\u201022 September, revised papers, pp. 57\u201071.","DOI":"10.1007\/3-540-44645-1_6"},{"key":"key2022030819454699700_b15","unstructured":"Ruthven, I., Lalmas, M. and van Rijsbergen, K. (2002), \u201cRanking expansion terms with partial and ostensive evidence\u201d, in Bruce, H., Fidel, R., Ingwersen, P. and Vakkari, P. (Eds), Emerging Frameworks and Methods \u2013 CoLIS4. Proceedings of the 4th International Conference on Conceptions of Library and Information Science \u2013 CoLIS4, Seattle, WA, July, Libraries Unlimited, Greenwood Village, CO, pp. 199\u2010220."},{"key":"key2022030819454699700_b16","unstructured":"Saracevic, T. (1996), \u201cRelevance reconsidered '96\u201d, in Ingwersen, P. and Pors, N.O. (Eds), CoLIS 2. 2nd International Conference on Conceptions of Library and Information Science: Integration in Perspective, The Royal School of Librarianship, Copenhagen, 13\u201016 October, pp. 201\u201018."},{"key":"key2022030819454699700_b17","unstructured":"Schamber, L. (1994), \u201cRelevance and information behaviour\u201d, Annual Review of Information Science and Technology (ARIST), Vol. 29, pp. 3\u201048."},{"key":"key2022030819454699700_b18","doi-asserted-by":"crossref","unstructured":"Vipond, D. and Hunt, R. (1984), \u201cPoint\u2010driven understanding: pragmatic and cognitive dimensions of literary reading\u201d, Poetics, Vol. 13, pp. 261\u201077.","DOI":"10.1016\/0304-422X(84)90005-6"},{"key":"key2022030819454699700_b19","doi-asserted-by":"crossref","unstructured":"Voorhees, E. (2000), \u201cVariations in relevance judgment and the measurement of retrieval effectiveness\u201d, Information Processing and Management, Vol. 36, pp. 607\u2010716.","DOI":"10.1016\/S0306-4573(00)00010-8"},{"key":"key2022030819454699700_b20","doi-asserted-by":"crossref","unstructured":"Zwaan, R.A. (1993), Aspects of Literary Comprehension. A Cognitive Approach, John Benjamins, Amsterdam (Utrecht publications in general and comparative literature).","DOI":"10.1075\/upal.29"},{"key":"key2022030819454699700_frd1","unstructured":"The Cross\u2010Language Evaluation (CLEF) (2000), available at: www.clef\u2010campaign.org."}],"container-title":["Journal of Documentation"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/www.emeraldinsight.com\/doi\/full-xml\/10.1108\/00220410510625831","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/00220410510625831\/full\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/00220410510625831\/full\/html","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,24]],"date-time":"2025-07-24T23:37:33Z","timestamp":1753400253000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.emerald.com\/jd\/article\/61\/5\/623-639\/203421"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2005,10,1]]},"references-count":20,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2005,10,1]]}},"alternative-id":["10.1108\/00220410510625831"],"URL":"https:\/\/doi.org\/10.1108\/00220410510625831","relation":{},"ISSN":["0022-0418"],"issn-type":[{"type":"print","value":"0022-0418"}],"subject":[],"published":{"date-parts":[[2005,10,1]]}}}