{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,24]],"date-time":"2025-11-24T09:54:20Z","timestamp":1763978060367,"version":"3.41.0"},"reference-count":47,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2018,9,29]],"date-time":"2018-09-29T00:00:00Z","timestamp":1538179200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["J. Data and Information Quality"],"published-print":{"date-parts":[[2018,9,30]]},"abstract":"<jats:p>The evaluation of retrieval effectiveness by means of test collections is a commonly used methodology in the information retrieval field. Some researchers have addressed the quite fascinating research question of whether it is possible to evaluate effectiveness completely automatically, without human relevance assessments. Since human relevance assessment is one of the main costs of building a test collection, both in human time and money resources, this rather ambitious goal would have a practical impact. In this article, we reproduce the main results on evaluating information retrieval systems without relevance judgments; furthermore, we generalize such previous work to analyze the effect of test collections, evaluation metrics, and pool depth. We also expand the idea to semi-automatic evaluation and estimation of topic difficulty. Our results show that (i) previous work is overall reproducible, although some specific results are not; (ii) collection, metric, and pool depth impact the automatic evaluation of systems, which is anyway accurate in several cases; (iii) semi-automatic evaluation is an effective methodology; and (iv) automatic evaluation can (to some extent) be used to predict topic difficulty.<\/jats:p>","DOI":"10.1145\/3241064","type":"journal-article","created":{"date-parts":[[2018,10,1]],"date-time":"2018-10-01T12:15:55Z","timestamp":1538396155000},"page":"1-32","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["Reproduce. Generalize. Extend. On Information Retrieval Evaluation without Relevance Judgments"],"prefix":"10.1145","volume":"10","author":[{"given":"Kevin","family":"Roitero","sequence":"first","affiliation":[{"name":"University of Udine, Udine, ITA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Marco","family":"Passon","sequence":"additional","affiliation":[{"name":"University of Udine, Udine, ITA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Giuseppe","family":"Serra","sequence":"additional","affiliation":[{"name":"University of Udine, Udine, ITA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Stefano","family":"Mizzaro","sequence":"additional","affiliation":[{"name":"University of Udine, Udine, ITA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2018,9,29]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation","volume":"15","author":"Alonso Omar","year":"2009","unstructured":"Omar Alonso and Stefano Mizzaro . 2009 . Can we get rid of TREC assessors? Using Mechanical Turk for relevance assessment . In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation , Vol. 15 . 16. Omar Alonso and Stefano Mizzaro. 2009. Can we get rid of TREC assessors? Using Mechanical Turk for relevance assessment. In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation, Vol. 15. 16."},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/860435.860501"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.5555\/2592907.2592915"},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/2499178.2499184"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/1008992.1009093"},{"key":"e_1_2_1_6_1","volume-title":"The TREC 2006 terabyte track. In TREC","volume":"6","author":"B\u00fcttcher Stefan","year":"2006","unstructured":"Stefan B\u00fcttcher , Charles L. A. Clarke , and Ian Soboroff . 2006 . The TREC 2006 terabyte track. In TREC , Vol. 6 . 39. http:\/\/trec.nist.gov\/pubs\/trec15\/papers\/TERA06.OVERVIEW.pdf. Stefan B\u00fcttcher, Charles L. A. Clarke, and Ian Soboroff. 2006. The TREC 2006 terabyte track. In TREC, Vol. 6. 39. http:\/\/trec.nist.gov\/pubs\/trec15\/papers\/TERA06.OVERVIEW.pdf."},{"key":"e_1_2_1_7_1","volume-title":"Estimating the Query Difficulty for Information Retrieval","author":"Carmel David","unstructured":"David Carmel and Elad Yom-Tov . 2010. Estimating the Query Difficulty for Information Retrieval ( 1 st ed.). Morgan and Claypool Publishers . David Carmel and Elad Yom-Tov. 2010. Estimating the Query Difficulty for Information Retrieval (1st ed.). Morgan and Claypool Publishers.","edition":"1"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835449.1835540"},{"volume-title":"Human-Based Query Difficulty Prediction","author":"Chifu Adrian-Gabriel","key":"e_1_2_1_9_1","unstructured":"Adrian-Gabriel Chifu , S\u00e9bastien D\u00e9jean , Stefano Mizzaro , and Josiane Mothe . 2017. Human-Based Query Difficulty Prediction . Springer International Publishing , Cham , 343--356. Adrian-Gabriel Chifu, S\u00e9bastien D\u00e9jean, Stefano Mizzaro, and Josiane Mothe. 2017. Human-Based Query Difficulty Prediction. Springer International Publishing, Cham, 343--356."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/122860.122861"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1080\/01621459.1961.10482090"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3020206"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2964797.2964808"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3190580.3190586"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/1629096.1629099"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/1008992.1009104"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-009-9101-4"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835449.1835654"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/1458082.1458311"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/258525.258587"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/2970398.2970429"},{"key":"e_1_2_1_22_1","volume-title":"Multi-armed bandits for adjudicating documents in pooling-based evaluation of information retrieval systems. Information Processing 8 Management 53, 5","author":"Losada David E.","year":"2017","unstructured":"David E. Losada , Javier Parapar , and Alvaro Barreiro . 2017. Multi-armed bandits for adjudicating documents in pooling-based evaluation of information retrieval systems. Information Processing 8 Management 53, 5 ( 2017 ), 1005--1025. David E. Losada, Javier Parapar, and Alvaro Barreiro. 2017. Multi-armed bandits for adjudicating documents in pooling-based evaluation of information retrieval systems. Information Processing 8 Management 53, 5 (2017), 1005--1025."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-016-9282-6"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2914696"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/1277741.1277824"},{"key":"e_1_2_1_26_1","volume-title":"ACM Conference on Research and Development in Information Retrieval, SIGIR, Predicting Query Difficulty-Methods and Applications Workshop. 7--10","author":"Mothe Josiane","year":"2005","unstructured":"Josiane Mothe and Ludovic Tanguy . 2005 . Linguistic features to predict query difficulty . In ACM Conference on Research and Development in Information Retrieval, SIGIR, Predicting Query Difficulty-Methods and Applications Workshop. 7--10 . Josiane Mothe and Ludovic Tanguy. 2005. Linguistic features to predict query difficulty. In ACM Conference on Research and Development in Information Retrieval, SIGIR, Predicting Query Difficulty-Methods and Applications Workshop. 7--10."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/860435.860510"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1145\/1183614.1183630"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-20161-5_14"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-56608-5_55"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/2180868.2180873"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/383952.383961"},{"volume-title":"Document Retrieval Systems","author":"Jones Karen Sparck","key":"e_1_2_1_33_1","unstructured":"Karen Sparck Jones . 1988. A statistical interpretation of term specificity and its application in retrieval . In Document Retrieval Systems . Taylor Graham Publishing , London, UK , 132--142. http:\/\/dl.acm.org\/citation.cfm?id&equals;106765.106782 Karen Sparck Jones. 1988. A statistical interpretation of term specificity and its application in retrieval. In Document Retrieval Systems. Taylor Graham Publishing, London, UK, 132--142. http:\/\/dl.acm.org\/citation.cfm?id&equals;106765.106782"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1002\/meet.14504201175"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2006.09.009"},{"key":"e_1_2_1_36_1","volume-title":"Simoni","author":"Tassey Gregory","year":"2010","unstructured":"Gregory Tassey , Brent R. Rowe , Dallas W. Wood , Albert N. Link , and Diglio A . Simoni . 2010 . Economic impact assessment of NIST's text REtrieval conference (TREC) program. Report prepared for National Institute of Technology (NIST) ( 2010). https:\/\/trec.nist.gov\/pubs\/2010.economic.impact.pdf. Gregory Tassey, Brent R. Rowe, Dallas W. Wood, Albert N. Link, and Diglio A. Simoni. 2010. Economic impact assessment of NIST's text REtrieval conference (TREC) program. Report prepared for National Institute of Technology (NIST) (2010). https:\/\/trec.nist.gov\/pubs\/2010.economic.impact.pdf."},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3166072.3166079"},{"key":"e_1_2_1_38_1","volume-title":"Overview of the TREC 2004 robust track. In TREC","volume":"4","author":"Voorhees Ellen M.","year":"2004","unstructured":"Ellen M. Voorhees . 2004 . Overview of the TREC 2004 robust track. In TREC , Vol. 4 . http:\/\/trec.nist.gov\/pubs\/trec13\/papers\/ROBUST.OVERVIEW.pdf. Ellen M. Voorhees. 2004. Overview of the TREC 2004 robust track. In TREC, Vol. 4. http:\/\/trec.nist.gov\/pubs\/trec13\/papers\/ROBUST.OVERVIEW.pdf."},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/564376.564432"},{"volume-title":"Overview of the Eighth Text REtrieval Conference (TREC-8). In Proceedings of the 8th Text REtrieval Conference (TREC-8). 1--24","author":"Ellen","key":"e_1_2_1_40_1","unstructured":"Ellen M. Voorhees and Donna Harman. 2000 . Overview of the Eighth Text REtrieval Conference (TREC-8). In Proceedings of the 8th Text REtrieval Conference (TREC-8). 1--24 . http:\/\/trec.nist.gov\/pubs\/trec23\/papers\/overview-web.pdf. Ellen M. Voorhees and Donna Harman. 2000. Overview of the Eighth Text REtrieval Conference (TREC-8). In Proceedings of the 8th Text REtrieval Conference (TREC-8). 1--24. http:\/\/trec.nist.gov\/pubs\/trec23\/papers\/overview-web.pdf."},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/1852102.1852106"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/584792.584908"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/952532.952693"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/1183614.1183633"},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390435"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.5555\/1793274.1793285"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/290941.291014"}],"container-title":["Journal of Data and Information Quality"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3241064","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3241064","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T00:43:47Z","timestamp":1750207427000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3241064"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2018,9,29]]},"references-count":47,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2018,9,30]]}},"alternative-id":["10.1145\/3241064"],"URL":"https:\/\/doi.org\/10.1145\/3241064","relation":{},"ISSN":["1936-1955","1936-1963"],"issn-type":[{"type":"print","value":"1936-1955"},{"type":"electronic","value":"1936-1963"}],"subject":[],"published":{"date-parts":[[2018,9,29]]},"assertion":[{"value":"2017-10-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-07-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-09-29","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}