{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,8,22]],"date-time":"2025-08-22T05:15:32Z","timestamp":1755839732398,"version":"3.41.0"},"reference-count":77,"publisher":"Association for Computing Machinery (ACM)","issue":"6","license":[{"start":{"date-parts":[[2024,10,22]],"date-time":"2024-10-22T00:00:00Z","timestamp":1729555200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2024,11,30]]},"abstract":"<jats:p>Traditional search result evaluation metrics in information retrieval, such as MAP and NDCG, naively focus on topical relevance between a document and search topic and assume this relationship as mono-dimensional and often binary. They neglect document content overlap and assume gains piling up as the searcher examines the ranked list at greater length. We propose a novel search result evaluation framework based on multidimensional, graded relevance assessments, explicit modelling of document overlaps and attributes affecting document usability beyond relevance. Document relevance to a search task is seen to consist of several content themes and document usability attributes. Documents may also overlap regarding their content themes. Attributes such as document readability, trustworthiness, or language represent the entire document\u2019s usability in the search task context, for a given searcher and her motivating task. The proposed framework evaluates the quality of a ranked search result, taking into account the contribution of each successive document, with estimated overlap across themes, and usability based on its attributes.<\/jats:p>","DOI":"10.1145\/3675162","type":"journal-article","created":{"date-parts":[[2024,7,1]],"date-time":"2024-07-01T17:09:39Z","timestamp":1719853779000},"page":"1-38","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["A Blueprint of IR Evaluation Integrating Task and User Characteristics"],"prefix":"10.1145","volume":"42","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7655-8930","authenticated-orcid":false,"given":"Kalervo","family":"Jarvelin","sequence":"first","affiliation":[{"name":"Faculty of Information Technology and Communication Sciences, Tampere University, Tampere, Finland"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1885-5045","authenticated-orcid":false,"given":"Eero","family":"Sormunen","sequence":"additional","affiliation":[{"name":"Faculty of Information Technology and Communication Sciences, Tampere University, Tampere, Finland"}]}],"member":"320","published-online":{"date-parts":[[2024,10,22]]},"reference":[{"key":"e_1_3_3_2_1","doi-asserted-by":"crossref","DOI":"10.1007\/978-3-319-29659-3","volume-title":"Recommender Systems: The Textbook","author":"Aggarwal C. C.","year":"2016","unstructured":"C. C. Aggarwal. 2016. Recommender Systems: The Textbook, Vol. 1. Springer International Publishing."},{"key":"e_1_3_3_3_1","first-page":"10","volume-title":"Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR \u201901)","author":"Allan J.","year":"2001","unstructured":"J. Allan, R. Gupta, and V. Khandelwal. 2001. Temporal summaries of news topics. In: Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR \u201901). W. B. Croft, D. J. Harper, D. H. Kraft, and J. Zobel (Eds.), ACM, New York, NY, 10\u201318."},{"key":"e_1_3_3_4_1","doi-asserted-by":"crossref","unstructured":"J. Allan. 2004. HARD Track Overview in TREC 2004 High Accuracy Retrieval from Documents. Computer Science Department Faculty Publication Series. 117. University of Massachusetts. Retrieved from https:\/\/scholarworks.umass.edu\/cs_faculty_pubs\/117","DOI":"10.21236\/ADA455426"},{"key":"e_1_3_3_5_1","first-page":"314","volume-title":"Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR \u201903)","author":"Allan J.","year":"2003","unstructured":"J. Allan, A. Bolivar, and C. Wade. 2003. Retrieval and novelty detection at the sentence level. I Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR \u201903). C. L. A. Clarke, G. V. Cormack, J. Callan, D. Hawking, and A. F. Smeaton (Eds.), ACM, New York, NY, 314\u2013321."},{"issue":"5","key":"e_1_3_3_6_1","doi-asserted-by":"crossref","first-page":"311","DOI":"10.1093\/comjnl\/41.5.311","article-title":"The incremental searcher satisfaction model for information retrieval","volume":"41","author":"van der Weide T. P.","year":"1999","unstructured":"T. P. van der Weide, P. Huibers, and P. van Bommel. 1999. The incremental searcher satisfaction model for information retrieval. Comput. J. 41, 5 (1999), 311\u2013318.","journal-title":"Comput. J."},{"key":"e_1_3_3_7_1","series-title":"Lecture Notes in Computer Science","doi-asserted-by":"crossref","first-page":"112","DOI":"10.1007\/978-3-642-33247-0_13","volume-title":"Proceedings of the 3rd International Conference of the CLEF Initiative (CLEF \u201912). Information Access Evaluation. Multilinguality, Multimodality, and Visual Analytics","volume":"7488","author":"Angelini M.","year":"2012","unstructured":"M. Angelini, N. Ferro, K. J\u00e4rvelin, H. Keskustalo, A. Pirkola, G. Santucci, and G. Silvello. 2012a. Cumulated relative position: A metric for ranking evaluation. In Proceedings of the 3rd International Conference of the CLEF Initiative (CLEF \u201912). Information Access Evaluation. Multilinguality, Multimodality, and Visual Analytics. Catarci, T., Forner, P., Hiemstra, D., Pe\u00f1as, A., Santucci, G. (Eds.), Lecture Notes in Computer Science, Vol. 7488. Springer, Berlin, 112\u2013121. DOI: 10.1007\/978-3-642-33247-0_13"},{"key":"e_1_3_3_8_1","doi-asserted-by":"crossref","first-page":"194","DOI":"10.1145\/2362724.2362757","volume-title":"Proceedings of the 4th Information Interaction in Context Symposium (IIIX \u201912)","author":"Angelini M.","year":"2012","unstructured":"M. Angelini N. Ferro, G. Silvello, and G. Santucci. 2012b. Visual interactive failure analysis: Supporting users in information retrieval evaluation. In Proceedings of the 4th Information Interaction in Context Symposium (IIIX \u201912). ACM, New York, NY, 194\u2013203. DOI: 10.1145\/2362724.2362757"},{"issue":"3","key":"e_1_3_3_9_1","doi-asserted-by":"crossref","first-page":"145","DOI":"10.1002\/(SICI)1097-4571(199404)45:3<149::AID-ASI5>3.0.CO;2-J","article-title":"User-defined relevance criteria: An exploratory study","volume":"45","author":"Barry C. L.","year":"1994","unstructured":"C. L. Barry. 1994. User-defined relevance criteria: An exploratory study. J. Am. Soc. Inform. Sci. 45, 3 (1994), 145\u2013159.","journal-title":"J. Am. Soc. Inform. Sci."},{"key":"e_1_3_3_10_1","first-page":"219","volume-title":"Inform. Proc. Manag.","volume":"34","author":"Barry C. L.","year":"1998","unstructured":"C. L. Barry and L. Schamber. 1998. Users\u2019 criteria for relevance evaluation: A cross-situational comparison. Inform. Proc. Manag. 34, 2\u20133 (1998), 219\u2013236."},{"key":"e_1_3_3_11_1","first-page":"133","article-title":"Anomalous state of knowledge for information retrieval","volume":"5","author":"Belkin N. J.","year":"1980","unstructured":"N. J. Belkin 1980. Anomalous state of knowledge for information retrieval. Can. J. Inf. Sci. 5 (1980), 133\u2013143.","journal-title":"Can. J. Inf. Sci."},{"key":"e_1_3_3_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/2888422.2888424"},{"issue":"10","key":"e_1_3_3_13_1","doi-asserted-by":"crossref","first-page":"913","DOI":"10.1002\/asi.10286","article-title":"The concept of relevance in IR","volume":"54","author":"Borlund P.","year":"2003","unstructured":"P. Borlund. 2003. The concept of relevance in IR. J. Am. Soc. Inform. Sci. Tech. 54, 10 (2003), 913\u2013925.","journal-title":"J. Am. Soc. Inform. Sci. Tech."},{"key":"e_1_3_3_14_1","doi-asserted-by":"publisher","DOI":"10.1108\/JD-06-2015-0068"},{"key":"e_1_3_3_15_1","first-page":"324","volume-title":"Proceedings of the 21st ACM SIGIR conference on research and development of information retrieval.","author":"Borlund P.","year":"1998","unstructured":"P. Borlund and P. Ingwersen. 1998. Measures of relative relevance and ranked half-life: Performance indicators for interactive IR. In Proceedings of the 21st ACM SIGIR conference on research and development of information retrieval. ACM, New York, NY, 324\u2013331."},{"issue":"2","key":"e_1_3_3_16_1","doi-asserted-by":"crossref","first-page":"191","DOI":"10.1016\/0306-4573(95)80035-R","article-title":"Task complexity affects information seeking and use","volume":"31","author":"Bystr\u00f6m K.","year":"1995","unstructured":"K. Bystr\u00f6m and K. J\u00e4rvelin. 1995. Task complexity affects information seeking and use. Inform. Proc. Manag. 31, 2 (1995), 191\u2013213.","journal-title":"Inform. Proc. Manag."},{"issue":"2","key":"e_1_3_3_17_1","doi-asserted-by":"crossref","first-page":"209","DOI":"10.1145\/3130348.3130369","article-title":"The use of MMR, diversity-based reranking for reordering documents and producing summaries","volume":"51","author":"Carbonell J.","year":"2017","unstructured":"J. Carbonell and J. Goldstein. 2017. The use of MMR, diversity-based reranking for reordering documents and producing summaries. SIGIR Forum 51, 2 (July 2017), 209\u2013210. [Original publication in 1998].","journal-title":"SIGIR Forum"},{"key":"e_1_3_3_18_1","first-page":"533","volume-title":"Inform. Proc. Mana.","author":"Cosijn E.","year":"2000","unstructured":"E. Cosijn and P. Ingwersen, 2000. Dimensions of relevance. Inform. Proc. Mana. 36 (2000) 533\u2013550."},{"key":"e_1_3_3_19_1","volume-title":"Search Engines: Information Retrieval in Practice","author":"Croft W. B.","year":"2010","unstructured":"W. B. Croft, D. Metzler, and T. Strohman. 2010. Search Engines: Information Retrieval in Practice. Addison-Wesley, Reading, MA."},{"key":"e_1_3_3_20_1","volume-title":"International Communication Association Annual Meeting","author":"Dervin B.","year":"1983","unstructured":"B. Dervin. 1983. An overview of sense-making research: Concepts, methods and results to date. In International Communication Association Annual Meeting. Dallas, Texas, May 1983."},{"key":"e_1_3_3_21_1","first-page":"123","volume-title":"TREC: Experiment and Evaluation In Information Retrieval","author":"Dumais S. T.","year":"2005","unstructured":"S. T. Dumais, and N. J. Belkin. 2005. The TREC interactive tracks: Putting the user into search. In TREC: Experiment and Evaluation In Information Retrieval. E. M. Voorhees and D. K. Harman (Eds.), MIT Press, Cambridge, MA, 123\u2013152."},{"key":"e_1_3_3_22_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.23416"},{"key":"e_1_3_3_23_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(199404)45:3<124::AID-ASI2>3.0.CO;2-8"},{"issue":"3","key":"e_1_3_3_24_1","doi-asserted-by":"crossref","first-page":"46","DOI":"10.1145\/3190580.3190588","article-title":"An information nutritional label for online documents","volume":"51","author":"Fuhr N.","year":"2017","unstructured":"N. Fuhr, A. Giachanou, G. Grefenstette, I. Gurevych, A. Hanselowski, K. Jarvelin, R. Jones, Y. Liu, J. Mothe, W. Nejdl, I. Peters, and B. Stein. 2017. An information nutritional label for online documents. SIGIR Forum 51, 3 (2017), 46\u201366.","journal-title":"SIGIR Forum"},{"issue":"1","key":"e_1_3_3_25_1","doi-asserted-by":"crossref","first-page":"32","DOI":"10.1145\/239041.239045","article-title":"A probabilistic relational algebra for the integration of information retrieval and database systems","volume":"15","author":"Fuhr N.","year":"1997","unstructured":"N. Fuhr and T. R\u00f6lleke. 1997. A probabilistic relational algebra for the integration of information retrieval and database systems. ACM Trans. Inform. Syst. 15, 1 (1997), 32\u201366.","journal-title":"ACM Trans. Inform. Syst."},{"key":"e_1_3_3_26_1","doi-asserted-by":"crossref","first-page":"40","DOI":"10.3115\/1117575.1117580","volume-title":"NAACL-ANLP 2000 Workshop: Automatic Summarization","author":"Goldstein J.","year":"2000","unstructured":"J. Goldstein, V. O. Mittal, J. G. Carbonell, and M. Kantrowitz, 2000. Multi-document summarization by sentence extraction. In NAACL-ANLP 2000 Workshop: Automatic Summarization, 40\u201348."},{"key":"e_1_3_3_27_1","first-page":"403","volume-title":"Inform. Proc. Manag.","volume":"39","author":"Greisdorf H.","year":"2003","unstructured":"H. Greisdorf. 2003. Relevance thresholds: A multi-stage predictive model of how users evaluate information. Inform. Proc. Manag. 39 (2003), 403\u2013423."},{"key":"e_1_3_3_28_1","doi-asserted-by":"crossref","first-page":"113","DOI":"10.1002\/aris.1440400111","article-title":"TREC: An overview","volume":"40","author":"Harman D. K.","year":"2006","unstructured":"D. K. Harman and E. M. Voorhees. 2006. TREC: An overview. Ann. Rev. Inf. Sci. Technol. 40 (2006), 113.","journal-title":"Ann. Rev. Inf. Sci. Technol."},{"key":"e_1_3_3_29_1","first-page":"3","article-title":"Evaluation of Information Retrieval Systems: Approaches, Issues, and Methods","volume":"32","author":"Harter S. P.","year":"1997","unstructured":"S. P. Harter and C. A. Hert. 1997. Evaluation of Information Retrieval Systems: Approaches, Issues, and Methods. Ann. Rev. Inf. Sci. Technol. 32 (1997), 3\u201394.","journal-title":"Ann. Rev. Inf. Sci. Technol."},{"issue":"3","key":"e_1_3_3_30_1","doi-asserted-by":"crossref","first-page":"7792","DOI":"10.1093\/bjsw\/bct160","article-title":"Selective clients\u2019 trajectories in case files: Filtering out information in the recording process in child protection","volume":"45","author":"Huuskonen S.","year":"2015","unstructured":"S. Huuskonen and P. Vakkari. 2015. Selective clients\u2019 trajectories in case files: Filtering out information in the recording process in child protection. Br. J. Soc. Work 45, 3 (2015), 7792\u2013808.","journal-title":"Br. J. Soc. Work"},{"key":"e_1_3_3_31_1","first-page":"173","volume-title":"Inform. Proc. Manag.","volume":"31","author":"Iivonen M.","year":"1995","unstructured":"M. Iivonen. 1995. Consistency in the selection of search concepts and search terms. Inform. Proc. Manag. 31, 2 (1995), 173\u2013190."},{"key":"e_1_3_3_32_1","unstructured":"INEX INitiative for the Evaluation of XML Retrieval 2014. Retrieved November 26 2022 from https:\/\/inex.mmci.uni-saarland.de\/index.html"},{"key":"e_1_3_3_33_1","doi-asserted-by":"publisher","DOI":"10.1108\/eb026960"},{"key":"e_1_3_3_34_1","first-page":"448 pages","volume-title":"The Turn: Integration of Information Seeking and Retrieval in Context.","author":"Ingwersen P.","year":"2005","unstructured":"P. Ingwersen and K. J\u00e4rvelin 2005. The Turn: Integration of Information Seeking and Retrieval in Context. Springer, 448 pages."},{"key":"e_1_3_3_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/2699660"},{"key":"e_1_3_3_36_1","series-title":"Lecture Notes in Computer Science","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1007\/978-3-540-78646-7_4","volume-title":"Proceedings of the 30th European Conference on Information Retrieval (ECIR \u201908)","volume":"4956","author":"J\u00e4rvelin K.","year":"2008","unstructured":"K. J\u00e4rvelin, S. Price, L. Delcambre, and M. L. Nielsen. 2008. Discounted cumulated gain based evaluation of multiple-query IR sessions. In Proceedings of the 30th European Conference on Information Retrieval (ECIR \u201908). Lecture Notes in Computer Science, Vol. 4956, Springer, 4\u201315."},{"key":"e_1_3_3_37_1","first-page":"41","volume-title":"Proceedings of the 23rd Annual Conference on Research and Development in Information Retrieval (ACM SIGIR \u201900)","author":"J\u00e4rvelin K.","year":"2000","unstructured":"K. J\u00e4rvelin and J. Kek\u00e4l\u00e4inen. 2000. IR evaluation methods for highly relevant documents. In Proceedings of the 23rd Annual Conference on Research and Development in Information Retrieval (ACM SIGIR \u201900). ACM, New York, NY, 41\u201348."},{"issue":"4","key":"e_1_3_3_38_1","doi-asserted-by":"crossref","first-page":"422","DOI":"10.1145\/582415.582418","article-title":"Cumulated gain-based evaluation of IR techniques","volume":"20","author":"J\u00e4rvelin K.","year":"2002","unstructured":"K. J\u00e4rvelin and J. Kek\u00e4l\u00e4inen. 2002. Cumulated gain-based evaluation of IR techniques. ACM Trans. Inform. Syst. 20, 4, 422\u2013446.","journal-title":"ACM Trans. Inform. Syst."},{"issue":"1","key":"e_1_3_3_39_1","first-page":"9","article-title":"On conceptual models for information seeking and retrieval research","volume":"9","author":"J\u00e4rvelin K.","year":"2003","unstructured":"K. J\u00e4rvelin and T. D. Wilson. 2003. On conceptual models for information seeking and retrieval research. Inform. Res. 9, 1 (2003), 9\u20131.","journal-title":"Inform. Res."},{"key":"e_1_3_3_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3130348.3130352"},{"key":"e_1_3_3_41_1","first-page":"253","volume-title":"Proceedings of the 4th CoLIS Conference","author":"Kek\u00e4l\u00e4inen J.","year":"2002","unstructured":"J. Kek\u00e4l\u00e4inen and K. J\u00e4rvelin. 2002a. Evaluating information retrieval systems under the challenges of interaction and multi-dimensional dynamic relevance. In Proceedings of the 4th CoLIS Conference, 253\u2013270."},{"issue":"13","key":"e_1_3_3_42_1","doi-asserted-by":"crossref","first-page":"1120","DOI":"10.1002\/asi.10137","article-title":"Using graded relevance assessments in IR evaluation","volume":"53","author":"Kek\u00e4l\u00e4inen J.","year":"2002","unstructured":"J. Kek\u00e4l\u00e4inen and K. J\u00e4rvelin. 2002b. Using graded relevance assessments in IR evaluation. J. Am. Soc. Inform. Sci. Tech. 53, 13 (2002b), 1120\u20131129.","journal-title":"J. Am. Soc. Inform. Sci. Tech."},{"key":"e_1_3_3_43_1","first-page":"675","volume-title":"Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (ACM SIGIR \u201908)","author":"Keskustalo H.","year":"2008","unstructured":"H. Keskustalo, K. J\u00e4rvelin, A. Pirkola, and J. Kek\u00e4l\u00e4inen. 2008. Intuition-supporting visualization of user\u2019s performance based on explicit negative higher-order relevance. In Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (ACM SIGIR \u201908). ACM, New York, NY, 675\u2013682."},{"key":"e_1_3_3_44_1","volume-title":"Task-Based Information Access in Molecular Medicine: Task Performance, Barriers, and Searching within a Heterogeneous Information Environment","author":"Kumpulainen M.","year":"2013","unstructured":"M. Kumpulainen. 2013. Task-Based Information Access in Molecular Medicine: Task Performance, Barriers, and Searching within a Heterogeneous Information Environment. Acta Univ. Tamperensis 1879, Tampere University, Tampere, Finland."},{"issue":"5","key":"e_1_3_3_45_1","doi-asserted-by":"crossref","first-page":"856","DOI":"10.1108\/JD-06-2013-0082","article-title":"Trails across the heterogeneous information environment: manual integration patterns of search systems in molecular medicine","volume":"70","author":"Kumpulainen S.","year":"2014","unstructured":"S. Kumpulainen. 2014. Trails across the heterogeneous information environment: manual integration patterns of search systems in molecular medicine. J. Doc. 70, 5 (2014), 856\u2013877.","journal-title":"J. Doc."},{"key":"e_1_3_3_46_1","first-page":"95","volume-title":"Proceedings of the Information Interaction in Context conference (IIiX \u201910)","author":"Kumpulainen S.","year":"2010","unstructured":"S. Kumpulainen and K. J\u00e4rvelin. 2010. Information Interaction in molecular medicine: Integrated use of multiple channels. In Proceedings of the Information Interaction in Context conference (IIiX \u201910). ACM, New York, NY, 95\u2013104. DOI: 10.1145\/1840784.1840800"},{"issue":"1","key":"e_1_3_3_47_1","doi-asserted-by":"crossref","first-page":"86","DOI":"10.1002\/asi.21672","article-title":"Barriers to task-based information access in molecular medicine","volume":"63","author":"Kumpulainen S.","year":"2012","unstructured":"S. Kumpulainen and K. J\u00e4rvelin 2012. Barriers to task-based information access in molecular medicine. J. Am. Soc. Inform. Sci. Tech. 63, 1 (2012), 86\u201397.","journal-title":"J. Am. Soc. Inform. Sci. Tech."},{"issue":"4","key":"e_1_3_3_48_1","doi-asserted-by":"crossref","first-page":"535","DOI":"10.1108\/EUM0000000007104","article-title":"Consistency of textual expression in newspaper articles","volume":"57","author":"Lehtokangas R.","year":"2001","unstructured":"R. Lehtokangas and K. J\u00e4rvelin. 2001. Consistency of textual expression in newspaper articles. J. Doc. 57, 4 (2001), 535\u2013548.","journal-title":"J. Doc."},{"key":"e_1_3_3_49_1","doi-asserted-by":"crossref","first-page":"1771","DOI":"10.1002\/asi.21359","article-title":"An exploration of the relationships between work task and interactive information search behavior","volume":"61","author":"Li Y.","year":"2010","unstructured":"Y. Li and N. J. Belkin. 2010. An exploration of the relationships between work task and interactive information search behavior. J. Am. Soc. Inform. Sci. Tech. 61 (2010), 1771\u20131789.","journal-title":"J. Am. Soc. Inform. Sci. Tech."},{"key":"e_1_3_3_50_1","first-page":"103007","volume-title":"Inform. Proc. Manag.","volume":"59","author":"Liu J.","year":"2022","unstructured":"J. Liu. 2022. Toward Cranfield-inspired reusability assessment in interactive information retrieval evaluation. Inform. Proc. Manag. 59, 5 (2022), 103007. DOI: 10.1016\/j.ipm.2022.103007"},{"issue":"5","key":"e_1_3_3_51_1","doi-asserted-by":"crossref","first-page":"327","DOI":"10.1002\/asi.10049","article-title":"User perspectives on relevance criteria: A comparison among relevant, partially relevant and not-relevant judgments","volume":"53","author":"Maghlaughlin K. L.","year":"2002","unstructured":"K. L. Maghlaughlin and D. H. Sonnenwald. 2002. User perspectives on relevance criteria: A comparison among relevant, partially relevant and not-relevant judgments. J. Am. Soc. Inform. Sci. Tech. 53, 5 (2002), 327\u2013342.","journal-title":"J. Am. Soc. Inform. Sci. Tech."},{"key":"e_1_3_3_52_1","doi-asserted-by":"crossref","first-page":"36","DOI":"10.1016\/j.eswa.2018.03.058","article-title":"A recent overview of the state-of-the-art elements of text classification","volume":"106","author":"Miro\u0144czuk M. M.","year":"2018","unstructured":"M. M. Miro\u0144czuk and J. Protasiewicz. 2018. A recent overview of the state-of-the-art elements of text classification. Expert Syst. Appl. 106 (2018), 36\u201354.","journal-title":"Expert Syst. Appl."},{"key":"e_1_3_3_53_1","first-page":"3","volume-title":"The Soar Papers","author":"Newell A.","year":"1993","unstructured":"A. Newell. 1993. Heuristic programming: Ill-structured problems. In The Soar Papers (Vol. 1) Research on Integrated Intelligence. MIT Press, 3\u201354."},{"key":"e_1_3_3_54_1","first-page":"369","volume-title":"Inform. Proc. Manag.","volume":"37","author":"Over P.","year":"2001","unstructured":"P. Over. 2001. The TREC interactive track: an annotated bibliography. Inform. Proc. Manag. 37, 3 (2001), 369\u2013381."},{"issue":"2","key":"e_1_3_3_55_1","doi-asserted-by":"crossref","first-page":"145","DOI":"10.1002\/asi.10017","article-title":"Judgment of information quality and cognitive authority in the Web","volume":"53","author":"Rieh S. Y.","year":"2002","unstructured":"S. Y. Rieh. 2002. Judgment of information quality and cognitive authority in the Web. J. Am. Soc. Inform. Sci. Tech. 53, 2 (2002), 145\u2013161.","journal-title":"J. Am. Soc. Inform. Sci. Tech."},{"key":"e_1_3_3_56_1","first-page":"603","volume-title":"Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval (SIGIR \u201910)","author":"Robertson S. E.","year":"2010","unstructured":"S. E. Robertson, E. Kanoulas, and E. Yilmaz. 2010. Extending average precision to graded relevance judgments. In Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval (SIGIR \u201910). ACM, New York, NY, 603\u2013610. DOI: 10.1145\/1835449.1835550"},{"issue":"2","key":"e_1_3_3_57_1","doi-asserted-by":"crossref","first-page":"143","DOI":"10.1504\/IJKEDM.2015.071284","article-title":"On the influence of training data quality on text document classification using machine learning methods","volume":"3","author":"Saarikoski J.","year":"2015","unstructured":"J. Saarikoski, H. Joutsijoki, K. J\u00e4rvelin, J. Laurikkala, and M. Juhola. 2015. On the influence of training data quality on text document classification using machine learning methods. Int. J. Knowl. Eng. Data Min. 3, 2 (2015), 143\u2013168.","journal-title":"Int. J. Knowl. Eng. Data Min."},{"key":"e_1_3_3_58_1","volume-title":"Information Searching in Authentic Work Tasks: A Field Study on the Effects of Task Type and Complexity","author":"Saastamoinen M.","year":"2016","unstructured":"M. Saastamoinen 2016. Information Searching in Authentic Work Tasks: A Field Study on the Effects of Task Type and Complexity. Acta Univ. Tamperensis 2244, Tampere University, Tampere, Finland."},{"key":"e_1_3_3_59_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.23766"},{"key":"e_1_3_3_60_1","doi-asserted-by":"publisher","DOI":"10.1561\/1500000009Export"},{"issue":"6","key":"e_1_3_3_61_1","doi-asserted-by":"crossref","first-page":"321","DOI":"10.1002\/asi.4630260604","article-title":"Relevance: A review of and framework for the thinking on the notion in information science","volume":"26","author":"Saracevic T.","year":"1975","unstructured":"T. Saracevic. 1975. Relevance: A review of and framework for the thinking on the notion in information science. J. Am. Soc. Inform. Sci. 26, 6 (1975), 321\u2013343.","journal-title":"J. Am. Soc. Inform. Sci."},{"issue":"13","key":"e_1_3_3_62_1","doi-asserted-by":"crossref","first-page":"1915","DOI":"10.1002\/asi.20682","article-title":"Relevance: A review of the literature and a framework for thinking on the notion in information science. Part II: nature and manifestations of relevance","volume":"58","author":"Saracevic T.","year":"2007","unstructured":"T. Saracevic. 2007. Relevance: A review of the literature and a framework for thinking on the notion in information science. Part II: nature and manifestations of relevance. J. Am. Soc. Inform. Sci. Tech. 58, 13 (2007), 1915\u20131933.","journal-title":"J. Am. Soc. Inform. Sci. Tech."},{"key":"e_1_3_3_63_1","series-title":"Learned Information","first-page":"329","volume-title":"Proceedings of the 12th National Online Meeting","author":"Saracevic T.","year":"1991","unstructured":"T. Saracevic, H. Mokros, L. Su, and A. Spink. 1991. Nature of interaction between users and intermediaries in online searching. In Proceedings of the 12th National Online Meeting. Learned Information, Medford, NJ, 329\u2013341."},{"key":"e_1_3_3_64_1","first-page":"3","volume-title":"Annual Review of Information Science and Technology.","author":"Schamber L.","year":"1994","unstructured":"L. Schamber. 1994. Relevance and information behavior. In: Annual Review of Information Science and Technology. M. Williams (Ed.), Vol. 29. Information Today, Medford, NJ, 3\u201348."},{"key":"e_1_3_3_65_1","first-page":"7","volume-title":"Proceedings of the EuroVis Workshop on Visual Analytics (EuroVA \u201922)","author":"Schmid J.","year":"2022","unstructured":"J. Schmid, L. Cibulski, I. Al-Hazwani, and J. Bernard. 2022. RankASco: A visual analytics approach to leverage attribute-based user preferences for item rankings. In Proceedings of the EuroVis Workshop on Visual Analytics (EuroVA \u201922), 7\u201311. DOI: 10.2312\/eurova.20221072"},{"key":"e_1_3_3_66_1","doi-asserted-by":"crossref","unstructured":"I. Soboroff and D. Harman. (2004). Overview of the TREC 2003 Novelty Track. NIST Gaithersburg MD. Retrieved from https:\/\/trec.nist.gov\/pubs\/trec12\/papers\/NOVELTY.OVERVIEW.pdf","DOI":"10.6028\/NIST.SP.500-255.novelty-overview"},{"key":"e_1_3_3_67_1","volume-title":"A Method for Measuring Wide Range Performance of Boolean Queries in Full-Text Databases.","author":"Sormunen E.","year":"2000","unstructured":"E. Sormunen. 2000. A Method for Measuring Wide Range Performance of Boolean Queries in Full-Text Databases. Doctoral Thesis. University of Tampere, Tampere. Acta Electronica Universitatis Tamperensis. Retrieved from http:\/\/acta.uta.fi\/pdf\/951-44-4732-8.pdf"},{"key":"e_1_3_3_68_1","first-page":"324","volume-title":"Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR \u201902).","author":"Sormunen E.","year":"2002","unstructured":"E. Sormunen. 2002. Liberal relevance criteria of TREC\u2014Counting on negligible documents? In Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR \u201902). ACM, New York, NY, 324\u2013330."},{"key":"e_1_3_3_69_1","volume-title":"Multidimensional Relevance Assessments: A Method for the Estimation of Redundancy in Document Contents","author":"Sormunen E.","year":"2016","unstructured":"E. Sormunen. 2016. Multidimensional Relevance Assessments: A Method for the Estimation of Redundancy in Document Contents. University of Tampere, School of Information Sciences. Unpublished Research Note."},{"key":"e_1_3_3_70_1","doi-asserted-by":"crossref","first-page":"358","DOI":"10.1108\/EUM0000000007087","article-title":"Document text characteristics affect the ranking of the most relevant documents by expanded structured queries","volume":"57","author":"Sormunen E.","year":"2001","unstructured":"E. Sormunen, J. Kek\u00e4l\u00e4inen, J. Koivisto, and K. J\u00e4rvelin. 2001. Document text characteristics affect the ranking of the most relevant documents by expanded structured queries. J. Doc. 57 (2001), 358\u2013376.","journal-title":"J. Doc."},{"issue":"2","key":"e_1_3_3_71_1","doi-asserted-by":"crossref","first-page":"161","DOI":"10.1002\/1097-4571(2000)9999:9999<::AID-ASI1564>3.0.CO;2-L","article-title":"Regions and levels: Measuring and mapping users\u2019 relevance judgments","volume":"52","author":"Spink A.","year":"2001","unstructured":"A. Spink and H. Greisdorf. 2001. Regions and levels: Measuring and mapping users\u2019 relevance judgments. J. Am. Soc. Inform. Sci. Tech. 52, 2 (2001), 161\u2013173.","journal-title":"J. Am. Soc. Inform. Sci. Tech."},{"key":"e_1_3_3_72_1","unstructured":"Stanford 2022. Evaluation of XML Retrieval. Retrieved November 26 2022 from https:\/\/nlp.stanford.edu\/IR-book\/html\/htmledition\/evaluation-of-xml-retrieval-1.html"},{"issue":"1","key":"e_1_3_3_73_1","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1177\/0165551515615833","article-title":"Searching as Learning. A Systematization based on Literature","volume":"42","author":"Vakkari P.","year":"2016","unstructured":"P. Vakkari. 2016. Searching as Learning. A Systematization based on Literature. J. Inform. Sci. 42, 1 (2016), 7\u201318.","journal-title":"J. Inform. Sci."},{"key":"e_1_3_3_74_1","volume-title":"Proceedings of the Conference on Human Information Interaction & Retrieval (CHIIR \u201918)","author":"Vakkari P.","year":"2018","unstructured":"P. Vakkari. 2018. Information search processes in complex tasks. In Proceedings of the Conference on Human Information Interaction & Retrieval (CHIIR \u201918). ACM, New York, NY. DOI: 10.1145\/3176349.3176570"},{"key":"e_1_3_3_75_1","doi-asserted-by":"crossref","first-page":"540","DOI":"10.1108\/EUM0000000007127","article-title":"Changes in relevance criteria and problem stages in task performance","volume":"56","author":"Vakkari P.","year":"2000","unstructured":"P. Vakkari and N. Hakala. 2000. Changes in relevance criteria and problem stages in task performance. J. Doc. 56 (2000), 540\u2013562.","journal-title":"J. Doc."},{"key":"e_1_3_3_76_1","doi-asserted-by":"crossref","unstructured":"P. Vakkari and E. Sormunen 2004. The influence of relevance levels on the effectiveness of interactive information retrieval. J. Am. Soc. Inform. Sci. Tech nol. 58 11 (2004) 963\u2013969.","DOI":"10.1002\/asi.20046"},{"key":"e_1_3_3_77_1","doi-asserted-by":"publisher","DOI":"10.1561\/1500000081"},{"key":"e_1_3_3_78_1","unstructured":"W. X. Zhao K. Zhou J. Li T. Tang X. Wang Y. Hou Y. Min B. Zhang J. Zhang Z. Dong Y. Du C. Yang Y. Chen Z. Chen J. Jiang R. Ren Y. Li X. Tang Z. Liu P. Liu J.-Y. Nie and J.-R. Wen. 2023. A survey of large language models. arXiv:2303.18223. Retrieved from https:\/\/arxiv.org\/abs\/2303.18223"}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3675162","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3675162","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:05:36Z","timestamp":1750291536000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3675162"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,10,22]]},"references-count":77,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2024,11,30]]}},"alternative-id":["10.1145\/3675162"],"URL":"https:\/\/doi.org\/10.1145\/3675162","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"type":"print","value":"1046-8188"},{"type":"electronic","value":"1558-2868"}],"subject":[],"published":{"date-parts":[[2024,10,22]]},"assertion":[{"value":"2023-06-28","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-06-17","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2024-10-22","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}