{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,5]],"date-time":"2025-10-05T19:51:31Z","timestamp":1759693891397},"reference-count":29,"publisher":"Association for Computing Machinery (ACM)","issue":"1","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2008,12]]},"abstract":"<jats:p>In information retrieval research, comparing retrieval approaches requires test collections consisting of documents, user requests and relevance assessments. Obtaining relevance assessments that are as sound and complete as possible is crucial for the comparison of retrieval approaches. In XML retrieval, the problem of obtaining sound and complete relevance assessments is further complicated by the structural relationships between retrieval results.<\/jats:p>\n          <jats:p>A major difference between XML retrieval and flat document retrieval is that the relevance of elements (the retrievable units) is not independent of that of related elements. This has major consequences for the gathering of relevance assessments. This article describes investigations into the creation of sound and complete relevance assessments for the evaluation of content-oriented XML retrieval as carried out at INEX, the evaluation campaign for XML retrieval. The campaign, now in its seventh year, has had three substantially different approaches to gather assessments and has finally settled on a highlighting method for marking relevant passages within documents\u2014even though the objective is to collect assessments at element level. The different methods of gathering assessments at INEX are discussed and contrasted. The highlighting method is shown to be the most reliable of the methods.<\/jats:p>","DOI":"10.1145\/1416950.1416951","type":"journal-article","created":{"date-parts":[[2008,12,23]],"date-time":"2008-12-23T14:15:53Z","timestamp":1230041753000},"page":"1-37","update-policy":"http:\/\/dx.doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":19,"title":["Sound and complete relevance assessment for XML retrieval"],"prefix":"10.1145","volume":"27","author":[{"given":"Benjamin","family":"Piwowarski","sequence":"first","affiliation":[{"name":"Yahoo! Research Latin America"}]},{"given":"Andrew","family":"Trotman","sequence":"additional","affiliation":[{"name":"University of Otago, Dunedin, New Zealand"}]},{"given":"Mounia","family":"Lalmas","sequence":"additional","affiliation":[{"name":"Queen Mary, University of London, U.K."}]}],"member":"320","published-online":{"date-parts":[[2008,12,23]]},"reference":[{"key":"e_1_2_1_1_1","volume-title":"Proceedings of the ACM SIGIR Workshop on XML.","author":"Baeza-Yates R.","unstructured":"Baeza-Yates , R. , Fuhr , N. , and Maarek , Y. S . (Eds.). 2002 . Proceedings of the ACM SIGIR Workshop on XML. Baeza-Yates, R., Fuhr, N., and Maarek, Y. S. (Eds.). 2002. Proceedings of the ACM SIGIR Workshop on XML."},{"key":"e_1_2_1_2_1","volume-title":"(Eds.)","author":"Blanken H. M.","year":"2003","unstructured":"Blanken , H. M. , Grabs , T. , Schek , H.-J. , Schenkel , R. , and Weikum , G . (Eds.) . 2003 . Intelligent Search on XML Data, Applications, Languages, Models, Implementations, and Benchmarks. Lecture Notes in Computer Science, vol. 2818 . Springer , Berlin, Germany. Blanken, H. M., Grabs, T., Schek, H.-J., Schenkel, R., and Weikum, G. (Eds.). 2003. Intelligent Search on XML Data, Applications, Languages, Models, Implementations, and Benchmarks. Lecture Notes in Computer Science, vol. 2818. Springer, Berlin, Germany."},{"key":"e_1_2_1_3_1","volume-title":"Proceedings of the ACM SIGIR Workshop on XML.","author":"Carmel D.","unstructured":"Carmel , D. , Maarek , Y. , and Soffer , A . (Eds.). 2000 . Proceedings of the ACM SIGIR Workshop on XML. Carmel, D., Maarek, Y., and Soffer, A. (Eds.). 2000. Proceedings of the ACM SIGIR Workshop on XML."},{"key":"e_1_2_1_4_1","volume-title":"Proceedings of the INEX Workshop on Element Retrieval Methodology, 2nd ed. 4--5.","author":"Clarke C.","year":"2005","unstructured":"Clarke , C. 2005 . Range results in XML retrieval . In Proceedings of the INEX Workshop on Element Retrieval Methodology, 2nd ed. 4--5. Clarke, C. 2005. Range results in XML retrieval. In Proceedings of the INEX Workshop on Element Retrieval Methodology, 2nd ed. 4--5."},{"key":"e_1_2_1_5_1","unstructured":"Cleverdon C. W. Mills J. and Keen M. 1966. Factors determining the performance of indexing systems: Cranfield University Cranfield Bedforshine U.K.  Cleverdon C. W. Mills J. and Keen M. 1966. Factors determining the performance of indexing systems: Cranfield University Cranfield Bedforshine U.K."},{"key":"e_1_2_1_6_1","volume-title":"Proceedings of the 6th Text REtrieval Conference (TREC-6), 171--186","author":"Cormack G. V.","unstructured":"Cormack , G. V. , Palmer , C. R. , To , S. S. L. , and Clarke , C. L. A. 1997. Passage-based refinement (multitext experiements for TREC-6) . In Proceedings of the 6th Text REtrieval Conference (TREC-6), 171--186 . Cormack, G. V., Palmer, C. R., To, S. S. L., and Clarke, C. L. A. 1997. Passage-based refinement (multitext experiements for TREC-6). In Proceedings of the 6th Text REtrieval Conference (TREC-6), 171--186."},{"key":"e_1_2_1_7_1","volume-title":"Proceedings of the INEX Workshop.","author":"Denoyer L.","unstructured":"Denoyer , L. and Gallinari , P . 2006. The Wikipedia XML corpus . In Proceedings of the INEX Workshop. Denoyer, L. and Gallinari, P. 2006. The Wikipedia XML corpus. In Proceedings of the INEX Workshop."},{"key":"e_1_2_1_8_1","volume-title":"Overview of the INEX 2007 ad hoc track. In Proceedings of INEX. 1--22","author":"Fuhr N.","unstructured":"Fuhr , N. , Kamps , J. , Lalmas , M. , Malik , S. , and Trotman , A . 2007 . Overview of the INEX 2007 ad hoc track. In Proceedings of INEX. 1--22 . Fuhr, N., Kamps, J., Lalmas, M., Malik, S., and Trotman, A. 2007. Overview of the INEX 2007 ad hoc track. In Proceedings of INEX. 1--22."},{"key":"e_1_2_1_9_1","volume-title":"Proceedings of the 1st Text REtrieval Conference (TREC-1).","author":"Harman D.","year":"1992","unstructured":"Harman , D. 1992 . Overview of the first text retrieval conference (TREC-1) . In Proceedings of the 1st Text REtrieval Conference (TREC-1). Harman, D. 1992. Overview of the first text retrieval conference (TREC-1). In Proceedings of the 1st Text REtrieval Conference (TREC-1)."},{"key":"e_1_2_1_10_1","volume-title":"The TREC conferences. In Proceedings of the HIM International Conference.","author":"Harman D.","year":"1995","unstructured":"Harman , D. 1995 . The TREC conferences. In Proceedings of the HIM International Conference. Harman, D. 1995. The TREC conferences. In Proceedings of the HIM International Conference."},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/1277741.1277877"},{"key":"e_1_2_1_12_1","volume-title":"INEX 2005 evaluation metrics. In Proceedings of INEX.","author":"Kazai G.","unstructured":"Kazai , G. and Lalmas , M . 2005 . INEX 2005 evaluation metrics. In Proceedings of INEX. Kazai, G. and Lalmas, M. 2005. INEX 2005 evaluation metrics. In Proceedings of INEX."},{"key":"e_1_2_1_13_1","volume-title":"Proceedings of the 26th European Colloquium on Information Retrieval Research (ECIR","author":"Kazai G.","year":"2004","unstructured":"Kazai , G. , Masood , S. , and Lalmas , M . 2004. A study of the assessment of relevance for the INEX'02 test collection . In Proceedings of the 26th European Colloquium on Information Retrieval Research (ECIR 2004 ). Kazai, G., Masood, S., and Lalmas, M. 2004. A study of the assessment of relevance for the INEX'02 test collection. In Proceedings of the 26th European Colloquium on Information Retrieval Research (ECIR 2004)."},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.10137"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/1273221.1273225"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.10056"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-34963-1_1"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/1183614.1183631"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-34963-1_4"},{"key":"e_1_2_1_20_1","volume-title":"-M","author":"Pehcevski J.","year":"2005","unstructured":"Pehcevski , J. , Thom , J. A. and Vercoustre , A . -M . 2005 . Users and assessors in the context of INEX: Are relevance dimensions relevant&quest; In Proceedings of the INEX Workshop on Element Retrieval Methodology , 2 nd ed. 47--62. Pehcevski, J., Thom, J. A. and Vercoustre, A.-M. 2005. Users and assessors in the context of INEX: Are relevance dimensions relevant&quest; In Proceedings of the INEX Workshop on Element Retrieval Methodology, 2nd ed. 47--62.","edition":"2"},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/1031171.1031246"},{"key":"e_1_2_1_22_1","first-page":"82","article-title":"Individual differences in organizing, searching and retrieving information","volume":"28","author":"Saracevic T.","year":"1991","unstructured":"Saracevic , T. 1991 . Individual differences in organizing, searching and retrieving information . Proc. Amer. Soc. Inform. Sci. 28 , 82 -- 86 . Saracevic, T. 1991. Individual differences in organizing, searching and retrieving information. Proc. Amer. Soc. Inform. Sci. 28, 82--86.","journal-title":"Proc. Amer. Soc. Inform. Sci."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1007\/11424550_33"},{"key":"e_1_2_1_24_1","volume-title":"Proceedings of the INEX Workshop on Element Retrieval Methodology, 2nd ed. 63--69","author":"Trotman A.","year":"2005","unstructured":"Trotman , A. 2005 . Wanted: Element retrieval users . In Proceedings of the INEX Workshop on Element Retrieval Methodology, 2nd ed. 63--69 . Trotman, A. 2005. Wanted: Element retrieval users. In Proceedings of the INEX Workshop on Element Retrieval Methodology, 2nd ed. 63--69."},{"key":"e_1_2_1_25_1","volume-title":"Proceedings of the SIGIR Workshop on XML Element Retrieval Methodology. 43--50","author":"Trotman A.","unstructured":"Trotman , A. and Geva , S . 2006. Passage retrieval and other XML-retrieval tasks . In Proceedings of the SIGIR Workshop on XML Element Retrieval Methodology. 43--50 . Trotman, A. and Geva, S. 2006. Passage retrieval and other XML-retrieval tasks. In Proceedings of the SIGIR Workshop on XML Element Retrieval Methodology. 43--50."},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0306-4573(00)00010-8"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/1099554.1099641"},{"key":"e_1_2_1_28_1","volume-title":"Proceedings of the INEX Workshop on Element Retrieval Methodology, 2nd ed. 70--79","author":"Woodley A.","unstructured":"Woodley , A. and Geva , S . 2005. Fine tuning INEX . In Proceedings of the INEX Workshop on Element Retrieval Methodology, 2nd ed. 70--79 . Woodley, A. and Geva, S. 2005. Fine tuning INEX. In Proceedings of the INEX Workshop on Element Retrieval Methodology, 2nd ed. 70--79."},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/290941.291014"}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/1416950.1416951","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,29]],"date-time":"2022-12-29T07:36:19Z","timestamp":1672299379000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1416950.1416951"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2008,12]]},"references-count":29,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2008,12]]}},"alternative-id":["10.1145\/1416950.1416951"],"URL":"https:\/\/doi.org\/10.1145\/1416950.1416951","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"value":"1046-8188","type":"print"},{"value":"1558-2868","type":"electronic"}],"subject":[],"published":{"date-parts":[[2008,12]]},"assertion":[{"value":"2007-06-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2008-03-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2008-12-23","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}