{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,21]],"date-time":"2025-10-21T00:47:21Z","timestamp":1761007641938,"version":"build-2065373602"},"reference-count":22,"publisher":"Wiley","issue":"1","license":[{"start":{"date-parts":[[2011,2,3]],"date-time":"2011-02-03T00:00:00Z","timestamp":1296691200000},"content-version":"vor","delay-in-days":94,"URL":"http:\/\/onlinelibrary.wiley.com\/termsAndConditions#vor"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc of Assoc for Info"],"published-print":{"date-parts":[[2010,11]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>User satisfaction, though difficult to measure, is the main goal of Information Retrieval (IR) systems. In recent years, as Interactive Information Retrieval (IIR) systems have become increasingly popular, user effectiveness also has become critical in evaluating IIR systems. However, existing measures in IR evaluation are not particularly suitable for gauging user satisfaction and user effectiveness. In this paper, we propose two new measures to evaluate IIR systems, the Normalized Task Completion Time (<jats:italic>NT<\/jats:italic>) and the Normalized User Effectiveness (<jats:italic>NUE<\/jats:italic>). The two measures overcome limitations of existing measures and are efficient to calculate in that they do not need a large pool of search tasks. A user study was conducted to investigate the relationships between the two measures and the <jats:italic>user satisfaction<\/jats:italic> and effectiveness of a given IR system. The <jats:italic>learning effects<\/jats:italic> described by <jats:italic>NT, NUE<\/jats:italic>, and the <jats:italic>task completion time<\/jats:italic> were also studied and compared. The results show that <jats:italic>NT<\/jats:italic> is strongly correlated with <jats:italic>user satisfaction, NUE<\/jats:italic> is a better indicator of system effectiveness than <jats:italic>task completion time<\/jats:italic>, and both new measures are superior to <jats:italic>task completion time<\/jats:italic> in describing the <jats:italic>learning effect<\/jats:italic> of the given IR system.<\/jats:p>","DOI":"10.1002\/meet.14504701144","type":"journal-article","created":{"date-parts":[[2011,2,3]],"date-time":"2011-02-03T16:35:46Z","timestamp":1296750946000},"page":"1-9","source":"Crossref","is-referenced-by-count":3,"title":["New measures for the evaluation of interactive information retrieval systems: Normalized task completion time and normalized user effectiveness"],"prefix":"10.1002","volume":"47","author":[{"given":"Jing","family":"Cheng","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xiao","family":"Hu","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"P. Bryan","family":"Heidorn","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"311","published-online":{"date-parts":[[2011,2,3]]},"reference":[{"key":"e_1_2_9_2_1","doi-asserted-by":"crossref","unstructured":"Allen B.(1994).Perceptual speed learning and information retrieval performance. InProceedings of the 17thACM SIGIR Conference on Research and Development in Information Retrieval(pp.71\u201380).","DOI":"10.1007\/978-1-4471-2099-5_8"},{"key":"e_1_2_9_3_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.21300"},{"key":"e_1_2_9_4_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(199601)47:1<85::AID-ASI8>3.0.CO;2-Z"},{"key":"e_1_2_9_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/1394251.1394261"},{"volume-title":"Interaction in information systems: A review of research from document retrieval to knowledge\u2010based systems","year":"1985","author":"Belkin N.J.","key":"e_1_2_9_6_1"},{"issue":"3","key":"e_1_2_9_7_1","article-title":"The IIR evaluation model: a framework for evaluation of interactive information retrieval systems","volume":"8","author":"Borlund P","year":"2003","journal-title":"Information Research"},{"key":"e_1_2_9_8_1","doi-asserted-by":"publisher","DOI":"10.1108\/EUM0000000007198"},{"volume-title":"Factors determining the performance of indexing systems. Vol. 1: Design. Vol 2: Results","year":"1966","author":"Cleverdon C. W.","key":"e_1_2_9_9_1"},{"key":"e_1_2_9_10_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.4630240204"},{"key":"e_1_2_9_11_1","doi-asserted-by":"crossref","unstructured":"Dunlop M.(1997).Time relevance and interaction modeling for information retrieval. In Proceedings of the 20th ACM SIGIR Conference on Research and Development in Information Retrieval (pp.206\u2013213).","DOI":"10.1145\/258525.258569"},{"key":"e_1_2_9_12_1","doi-asserted-by":"crossref","unstructured":"Geva S. Kamps J. Peters C. Sakai T. Trotman A. and Voorhees E. (editors) (2009). In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation. Boston MA USA.","DOI":"10.1145\/1670564.1670567"},{"key":"e_1_2_9_13_1","doi-asserted-by":"crossref","unstructured":"Gwizdka J.(2008).Revisiting search task difficulty: behavioral and individual difference measures. In Proceedings of the 71th Annual Meeting of the American Society for Information Science and Technology.","DOI":"10.1002\/meet.2008.1450450249"},{"key":"e_1_2_9_14_1","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9781139644082"},{"key":"e_1_2_9_15_1","first-page":"147","volume-title":"Readings in Information Retrieval","author":"Jones K. S.","year":"1997"},{"issue":"1","key":"e_1_2_9_16_1","first-page":"1","article-title":"Methods for evaluating interactive information retrieval systems with users","volume":"3","author":"Kelly D.","year":"2009","journal-title":"Foundations and Trends in Information Retrieval"},{"key":"e_1_2_9_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/1740592.1740597"},{"key":"e_1_2_9_18_1","doi-asserted-by":"publisher","DOI":"10.1016\/0306-4573(92)90007-M"},{"key":"e_1_2_9_19_1","doi-asserted-by":"crossref","unstructured":"Turpin A. H.&Hersh W.(2001).Why batch and user evaluations do notgive the same results. In Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (pp.225\u2013231).","DOI":"10.1145\/383952.383992"},{"key":"e_1_2_9_20_1","doi-asserted-by":"crossref","unstructured":"Voorhees E.(1998).Variations in relevance judgments and the measurement of retrieval effectiveness. In Proceedings of the 21th ACM SIGIR Conference on Research and Development in Information Retrieval (pp.315\u2013323).","DOI":"10.1145\/290941.291017"},{"key":"e_1_2_9_21_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0306-4573(99)00043-6"},{"key":"e_1_2_9_22_1","doi-asserted-by":"publisher","DOI":"10.2514\/8.155"},{"key":"e_1_2_9_23_1","doi-asserted-by":"crossref","unstructured":"Xu Y.&Mease D.(2009).Evaluating web search using task completion time. InProceedings of the 32ndannual international ACM SIGIR conference on Research and Development in Information Retrieval(pp.676\u2013677).","DOI":"10.1145\/1571941.1572073"}],"container-title":["Proceedings of the American Society for Information Science and Technology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/api.wiley.com\/onlinelibrary\/tdm\/v1\/articles\/10.1002%2Fmeet.14504701144","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/api.wiley.com\/onlinelibrary\/tdm\/v1\/articles\/10.1002%2Fmeet.14504701144","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/asistdl.onlinelibrary.wiley.com\/doi\/pdf\/10.1002\/meet.14504701144","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,20]],"date-time":"2025-10-20T14:53:00Z","timestamp":1760971980000},"score":1,"resource":{"primary":{"URL":"https:\/\/asistdl.onlinelibrary.wiley.com\/doi\/10.1002\/meet.14504701144"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2010,11]]},"references-count":22,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2010,11]]}},"alternative-id":["10.1002\/meet.14504701144"],"URL":"https:\/\/doi.org\/10.1002\/meet.14504701144","archive":["Portico"],"relation":{},"ISSN":["0044-7870","1550-8390"],"issn-type":[{"type":"print","value":"0044-7870"},{"type":"electronic","value":"1550-8390"}],"subject":[],"published":{"date-parts":[[2010,11]]}}}