{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,26]],"date-time":"2026-03-26T18:35:15Z","timestamp":1774550115205,"version":"3.50.1"},"reference-count":16,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2017,8,2]],"date-time":"2017-08-02T00:00:00Z","timestamp":1501632000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGIR Forum"],"published-print":{"date-parts":[[2017,8,2]]},"abstract":"<jats:p>This paper proposes evaluation methods based on the use of non-dichotomous relevance judgements in IR experiments. It is argued that evaluation methods should credit IR methods for their ability to retrieve highly relevant documents. This is desirable from the user point of view in modem large IR environments. The proposed methods are (1) a novel application of P-R curves and average precision computations based on separate recall bases for documents of different degrees of relevance, and (2) two novel measures computing the cumulative gain the user obtains by examining the retrieval result up to a given ranked position. We then demonstrate the use of these evaluation methods in a case study on the effectiveness of query types, based on combinations of query structures and expansion, in retrieving documents of various degrees of relevance. The test was run with a best match retrieval system (In- Query I) in a text database consisting of newspaper articles. The results indicate that the tested strong query structures are most effective in retrieving highly relevant documents. The differences between the query types are practically essential and statistically significant. More generally, the novel evaluation methods and the case demonstrate that non-dichotomous relevance assessments are applicable in IR experiments, may reveal interesting phenomena, and allow harder testing of IR methods.<\/jats:p>","DOI":"10.1145\/3130348.3130374","type":"journal-article","created":{"date-parts":[[2017,8,2]],"date-time":"2017-08-02T19:36:12Z","timestamp":1501702572000},"page":"243-250","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":155,"title":["IR evaluation methods for retrieving highly relevant documents"],"prefix":"10.1145","volume":"51","author":[{"given":"Kalervo","family":"J\u00e4rvelin","sequence":"first","affiliation":[{"name":"University of Tampere, Finland"}]},{"given":"Jaana","family":"Kek\u00e4l\u00e4inen","sequence":"additional","affiliation":[{"name":"University of Tampere, Finland"}]}],"member":"320","published-online":{"date-parts":[[2017,8,2]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"E.M. Voorhees & D","author":"Allan J.","year":"1997","unstructured":"J. Allan , J. Callan , B. Croft , L. Ballesteros , J. Broglio , J. Xu & H. Shu . INQUERY at TREC 5 . In E.M. Voorhees & D .K. Harrnan (Eds.), Information technology: The Fifth Text Retrieval Conference (TREC- 5). Gaithersburg, MD : National Institute of Standards and Technology , 119--132, 1997 . J. Allan, J. Callan, B. Croft, L. Ballesteros, J. Broglio, J. Xu & H. Shu. INQUERY at TREC 5. In E.M. Voorhees & D.K. Harrnan (Eds.), Information technology: The Fifth Text Retrieval Conference (TREC-5). Gaithersburg, MD: National Institute of Standards and Technology, 119--132, 1997."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3166.3197"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/290941.291019"},{"key":"e_1_2_1_4_1","volume-title":"Practical nonparametric statistics","author":"Conover W.J.","year":"1980","unstructured":"W.J. Conover . Practical nonparametric statistics ( 2 nd ed.). New York : John Wiley & Sons , 1980 . W.J. Conover. Practical nonparametric statistics (2nd ed.). New York: John Wiley & Sons, 1980.","edition":"2"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1108\/eb026953"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(199508)46:7%3C478::AID-ASI2%3E3.0.CO;2-#"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1515\/libr.1995.45.3-4.160"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1108\/eb026869"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1009983401464"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/290941.290978"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4615-5705-0"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(199505)46:4%3C272::AID-ASI4%3E3.0.CO;2-T"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1108\/eb026654"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(198805)39:3%3C161::AID-ASI2%3E3.0.CO;2-0"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1016\/0306-4573(94)90065-5"},{"key":"e_1_2_1_17_1","volume-title":"d~ssertation. Department of Information Studies","author":"Sormunen E.","year":"2000","unstructured":"E. Sormunen . A Method for Measuring Wtde Range Performance of Boolean Queries in Full-Text Databases. Ph. D. d~ssertation. Department of Information Studies , University of Tampere , 2000 . E. Sormunen. A Method for Measuring Wtde Range Performance of Boolean Queries in Full-Text Databases. Ph.D. d~ssertation. Department of Information Studies, University of Tampere, 2000."}],"container-title":["ACM SIGIR Forum"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3130348.3130374","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3130348.3130374","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T02:26:17Z","timestamp":1750213577000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3130348.3130374"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,8,2]]},"references-count":16,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2017,8,2]]}},"alternative-id":["10.1145\/3130348.3130374"],"URL":"https:\/\/doi.org\/10.1145\/3130348.3130374","relation":{},"ISSN":["0163-5840"],"issn-type":[{"value":"0163-5840","type":"print"}],"subject":[],"published":{"date-parts":[[2017,8,2]]},"assertion":[{"value":"2017-08-02","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}