{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,15]],"date-time":"2026-10-15T02:22:07Z","timestamp":1792030927184,"version":"4.4.0"},"reference-count":55,"publisher":"Association for Computing Machinery (ACM)","issue":"4","content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2006,10]]},"abstract":"<jats:p>\n            We propose and evaluate a family of measures, the eXtended Cumulated Gain (XCG) measures, for the evaluation of content-oriented XML retrieval approaches. Our aim is to provide an evaluation framework that allows the consideration of dependency among XML document components. In particular, two aspects of dependency are considered: (1) near-misses, which are document components that are structurally related to relevant components, such as a neighboring paragraph or container section, and (2) overlap, which regards the situation wherein the same text fragment is referenced multiple times, for example, when a paragraph and its container section are both retrieved. A further consideration is that the measures should be flexible enough so that different models of user behavior may be instantiated within. Both system- and user-oriented aspects are investigated and both recall and precision-like qualities are measured. We evaluate the reliability of the proposed measures based on the INEX 2004 test collection. For example, the effects of assessment variation and topic set size on evaluation stability are investigated, and the upper and lower bounds of expected error rates are established. The evaluation demonstrates that the XCG measures are stable and reliable, and in particular, that the novel measures of effort-precision and gain-recall (\n            <jats:italic>ep<\/jats:italic>\n            \/\n            <jats:italic>gr<\/jats:italic>\n            ) show comparable behavior to established IR measures like precision and recall.\n          <\/jats:p>","DOI":"10.1145\/1185877.1185883","type":"journal-article","created":{"date-parts":[[2007,1,16]],"date-time":"2007-01-16T19:38:29Z","timestamp":1168976309000},"page":"503-542","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":30,"title":["eXtended cumulated gain measures for the evaluation of content-oriented XML retrieval"],"prefix":"10.1145","volume":"24","author":[{"given":"Gabriella","family":"Kazai","sequence":"first","affiliation":[{"name":"Queen Mary, University of London, London, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mounia","family":"Lalmas","sequence":"additional","affiliation":[{"name":"Queen Mary, University of London, London, UK"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2006,10]]},"reference":[{"key":"e_1_2_1_2_1","volume-title":"Proceedings of the SIGIR Workshop on XML and Information Retrieval.]]","author":"Baeza-Yates R.","unstructured":"Baeza-Yates , R. , Fuhr , N. , and Maarek , Y ., eds. 2002 . Proceedings of the SIGIR Workshop on XML and Information Retrieval.]] Baeza-Yates, R., Fuhr, N., and Maarek, Y., eds. 2002. Proceedings of the SIGIR Workshop on XML and Information Retrieval.]]"},{"key":"e_1_2_1_3_1","volume-title":"Modern Information Retrieval","author":"Baeza-Yates R.","unstructured":"Baeza-Yates , R. and Ribeiro-Neto , B. 1999. Modern Information Retrieval . Addison Wesley Reading , MA .]] Baeza-Yates, R. and Ribeiro-Neto, B. 1999. Modern Information Retrieval. Addison Wesley Reading, MA.]]"},{"key":"e_1_2_1_4_1","volume-title":"eds","author":"Blanken H. M.","year":"2003","unstructured":"Blanken , H. M. , Grabs , T. , Schek , H.-J. , Schenkel , R. , and Weikum , G. , eds . 2003 . Intelligent Search on XML Data, Applications, Languages, Models, Implementations, and Benchmarks. Lecture Notes in Computer Science, vol. 2818 . Springer-Verlag .]] Blanken, H. M., Grabs, T., Schek, H.-J., Schenkel, R., and Weikum, G., eds. 2003. Intelligent Search on XML Data, Applications, Languages, Models, Implementations, and Benchmarks. Lecture Notes in Computer Science, vol. 2818. Springer-Verlag.]]"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.10286"},{"key":"e_1_2_1_6_1","unstructured":"Bray T. Paoli J. and Sperberg-McQueen C. M. 1998. Extensible markup language (XML) 1.0. http:\/\/www.w3.org\/TR\/1998\/REC-xml-19980210 W3C Recommendation. Tech. Rep. W3C (World Wide Web Consortium). Feb.]]  Bray T. Paoli J. and Sperberg-McQueen C. M. 1998. Extensible markup language (XML) 1.0. http:\/\/www.w3.org\/TR\/1998\/REC-xml-19980210 W3C Recommendation. Tech. Rep. W3C (World Wide Web Consortium). Feb.]]"},{"key":"e_1_2_1_7_1","volume-title":"SIGIR: Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM","author":"Buckley C.","unstructured":"Buckley , C. and Voorhees , E. M . 2000. Evaluating evaluation measure stability . In SIGIR: Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM , New York. 33--40.]] 10.1145\/345508.345543 Buckley, C. and Voorhees, E. M. 2000. Evaluating evaluation measure stability. In SIGIR: Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, New York. 33--40.]] 10.1145\/345508.345543"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1016\/0306-4573(92)90031-T"},{"key":"e_1_2_1_9_1","volume-title":"Tech. Rep. Fermi ESPRIT BRA 8134","author":"Chiaramella Y.","year":"1996","unstructured":"Chiaramella , Y. , Mulhem , P. , and Fourel , F . 1996 . A model for multimedia information retrieval. Tech. Rep. Fermi ESPRIT BRA 8134 , University of Glasgow.]] Chiaramella, Y., Mulhem, P., and Fourel, F. 1996. A model for multimedia information retrieval. Tech. Rep. Fermi ESPRIT BRA 8134, University of Glasgow.]]"},{"key":"e_1_2_1_10_1","unstructured":"Clark J. and DeRose S. 1999. XML path language (XPath) version 1.0. W3C Recommendation. http:\/\/www.w3.org\/TR\/xpath. Tech. Rep. REC-xpath-19991116 WWW Consortium. Nov.]]  Clark J. and DeRose S. 1999. XML path language (XPath) version 1.0. W3C Recommendation. http:\/\/www.w3.org\/TR\/xpath. Tech. Rep. REC-xpath-19991116 WWW Consortium. Nov.]]"},{"key":"e_1_2_1_11_1","volume-title":"Practical Non-Parametric Statistics","author":"Conover W.","unstructured":"Conover , W. 1980. Practical Non-Parametric Statistics , 2 nd ed. John Wiley , New York .]] Conover, W. 1980. Practical Non-Parametric Statistics, 2nd ed. John Wiley, New York.]]","edition":"2"},{"key":"e_1_2_1_12_1","doi-asserted-by":"crossref","first-page":"30","DOI":"10.1002\/asi.5090190108","article-title":"Expected search length: A single measure of retrieval effectiveness based on the weak ordering action of retrieval systems","volume":"19","author":"Cooper W.","year":"1968","unstructured":"Cooper , W. 1968 . Expected search length: A single measure of retrieval effectiveness based on the weak ordering action of retrieval systems . American Documentation 19 , 1, 30 -- 41 .]] Cooper, W. 1968. Expected search length: A single measure of retrieval effectiveness based on the weak ordering action of retrieval systems. American Documentation 19, 1, 30--41.]]","journal-title":"American Documentation"},{"key":"e_1_2_1_13_1","volume-title":"Proceedings of the Recherche d'Informations Assistee par Ordinateur (RIAO) Conference","author":"de Vries A.","unstructured":"de Vries , A. , Kazai , G. , and Lalmas , M . 2004. Tolerance to irrelevance: A user-effort oriented evaluation of retrieval systems without predefined retrieval unit . In Proceedings of the Recherche d'Informations Assistee par Ordinateur (RIAO) Conference . Avignon, France.]] de Vries, A., Kazai, G., and Lalmas, M. 2004. Tolerance to irrelevance: A user-effort oriented evaluation of retrieval systems without predefined retrieval unit. In Proceedings of the Recherche d'Informations Assistee par Ordinateur (RIAO) Conference. Avignon, France.]]"},{"key":"e_1_2_1_14_1","volume-title":"Proceedings of the 2nd Workshop of the INitiative for the Evaluation of XML Retrieval (INEX)","author":"Fuhr N.","year":"2003","unstructured":"Fuhr , N. , Lalmas , M. , and Malik , S ., eds. 2004 . Proceedings of the 2nd Workshop of the INitiative for the Evaluation of XML Retrieval (INEX) . Dagstuhl, Germany, Dec. 15--17 , 2003 . http:\/\/inex.is.informatik.uni-duisburg.de:2003\/proceedings.pdf.]] Fuhr, N., Lalmas, M., and Malik, S., eds. 2004. Proceedings of the 2nd Workshop of the INitiative for the Evaluation of XML Retrieval (INEX). Dagstuhl, Germany, Dec. 15--17, 2003. http:\/\/inex.is.informatik.uni-duisburg.de:2003\/proceedings.pdf.]]"},{"key":"e_1_2_1_15_1","volume-title":"Advances in XML Information Retrieval: 3rd Workshop of the INitiative for the Evaluation of XML Retrieval (INEX 2004","author":"Fuhr N.","year":"2004","unstructured":"Fuhr , N. , Lalmas , M. , Malik , S. , and Szlavik , Z ., eds. 2005 . Advances in XML Information Retrieval: 3rd Workshop of the INitiative for the Evaluation of XML Retrieval (INEX 2004 ). Schloss Dagstuhl, Germany, 6- -8 Dec. 2004 . Lecture Notes in Computer Science, vol. 3493. Springer.]] Fuhr, N., Lalmas, M., Malik, S., and Szlavik, Z., eds. 2005. Advances in XML Information Retrieval: 3rd Workshop of the INitiative for the Evaluation of XML Retrieval (INEX 2004). Schloss Dagstuhl, Germany, 6--8 Dec. 2004. Lecture Notes in Computer Science, vol. 3493. Springer.]]"},{"key":"e_1_2_1_16_1","volume-title":"Proceedings of the 2nd workshop of the initiative for the Evaluation of XML Retrieval","author":"Fuhr N.","year":"2003","unstructured":"Fuhr , N. , Malik , S. , and Lalmas , M . 2004. Overview of the INitiative for the Evaluation of XML Retrieval (INEX) 2003 . In Proceedings of the 2nd workshop of the initiative for the Evaluation of XML Retrieval . Dagstuhl, Germany. 1--11. http:\/\/inex.is.informatik.uni-duisburg.de : 2003 \/proceedings.pdf.]] Fuhr, N., Malik, S., and Lalmas, M. 2004. Overview of the INitiative for the Evaluation of XML Retrieval (INEX) 2003. In Proceedings of the 2nd workshop of the initiative for the Evaluation of XML Retrieval. Dagstuhl, Germany. 1--11. http:\/\/inex.is.informatik.uni-duisburg.de:2003\/proceedings.pdf.]]"},{"key":"e_1_2_1_17_1","unstructured":"Goevert N. Fuhr N. Lalmas M. and Kazai G. 2006. Evaluating the effectiveness of content-oriented XML retrieval methods. J. Inf. Retrieval (to appear).]] 10.1007\/s10791-006-9008-2   Goevert N. Fuhr N. Lalmas M. and Kazai G. 2006. Evaluating the effectiveness of content-oriented XML retrieval methods. J. Inf. Retrieval (to appear).]] 10.1007\/s10791-006-9008-2"},{"key":"e_1_2_1_18_1","volume-title":"Proceedings of the 1st Workshop of the INitiative for the Evaluation of XML Retrieval (INEX)","author":"G\u00f6vert N.","year":"2002","unstructured":"G\u00f6vert , N. and Kazai , G . 2003. Overview of the INitiative for the Evaluation of XML retrieval (INEX) 2002 . In Proceedings of the 1st Workshop of the INitiative for the Evaluation of XML Retrieval (INEX) . Dagstuhl, Germany, 8- -11 Dec. 2002 , Sophia Antipolis, France. 1--17.]] G\u00f6vert, N. and Kazai, G. 2003. Overview of the INitiative for the Evaluation of XML retrieval (INEX) 2002. In Proceedings of the 1st Workshop of the INitiative for the Evaluation of XML Retrieval (INEX). Dagstuhl, Germany, 8--11 Dec. 2002, Sophia Antipolis, France. 1--17.]]"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(199601)47:1%3C37::AID-ASI4%3E3.3.CO;2-I"},{"key":"e_1_2_1_20_1","volume-title":"Proceedings of the TREC Conference.]]","author":"Hawking D.","unstructured":"Hawking , D. , Voorhees , E. , Craswell , N. , and Bailey , P . 1999. Overview of the TREC-8 Web Track . In Proceedings of the TREC Conference.]] Hawking, D., Voorhees, E., Craswell, N., and Bailey, P. 1999. Overview of the TREC-8 Web Track. In Proceedings of the TREC Conference.]]"},{"key":"e_1_2_1_21_1","volume-title":"SIGIR: Proceedings of the 16th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM New York. 329--338","author":"Hull D.","year":"1993","unstructured":"Hull , D. 1993 . Using statistical testing in the evaluation of retrieval experiments . In SIGIR: Proceedings of the 16th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM New York. 329--338 .]] 10.1145\/160688.160758 Hull, D. 1993. Using statistical testing in the evaluation of retrieval experiments. In SIGIR: Proceedings of the 16th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM New York. 329--338.]] 10.1145\/160688.160758"},{"key":"e_1_2_1_22_1","volume-title":"SIGIR: Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM New York. 41--48","author":"J\u00e4rvelin K.","unstructured":"J\u00e4rvelin , K. and Kek\u00e4l\u00e4inen , J . 2000. IR evaluation methods for retrieving highly relevant documents . In SIGIR: Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM New York. 41--48 .]] 10.1145\/345508.345545 J\u00e4rvelin, K. and Kek\u00e4l\u00e4inen, J. 2000. IR evaluation methods for retrieving highly relevant documents. In SIGIR: Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM New York. 41--48.]] 10.1145\/345508.345545"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/582415.582418"},{"key":"e_1_2_1_24_1","unstructured":"Kando N. Kuriyama K. and Yoshioka M. 2001. Information retrieval system evaluation using multi-grade relevance judgements - Discussion on averageable single-numbered measures (in japanese). Tech. Rep.]]  Kando N. Kuriyama K. and Yoshioka M. 2001. Information retrieval system evaluation using multi-grade relevance judgements - Discussion on averageable single-numbered measures (in japanese). Tech. Rep.]]"},{"key":"e_1_2_1_25_1","volume-title":"Proceedings of the INEX Workshop on Element Retrieval Methodology. Glasgow","author":"Kazai G.","year":"2005","unstructured":"Kazai , G. and Lalmas , M . 2005. Notes on what to measure in INEX . In Proceedings of the INEX Workshop on Element Retrieval Methodology. Glasgow , July 2005 .]] Kazai, G. and Lalmas, M. 2005. Notes on what to measure in INEX. In Proceedings of the INEX Workshop on Element Retrieval Methodology. Glasgow, July 2005.]]"},{"key":"e_1_2_1_26_1","volume-title":"INEX 2005 evaluation metrics. In Advances in XML Information Retrieval and Evaluation: 4th Workshop of the INitiative for the Evaluation of XML Retrieval (INEX 2005","author":"Kazai G.","year":"2005","unstructured":"Kazai , G. and Lalmas , M . 2006 . INEX 2005 evaluation metrics. In Advances in XML Information Retrieval and Evaluation: 4th Workshop of the INitiative for the Evaluation of XML Retrieval (INEX 2005 ). Schloss Dagstuhl, 28- -30 Nov. 2005 . Lecture Notes in Computer Science vol. 3977. Springer-Verlag. 16--29.]] Kazai, G. and Lalmas, M. 2006. INEX 2005 evaluation metrics. In Advances in XML Information Retrieval and Evaluation: 4th Workshop of the INitiative for the Evaluation of XML Retrieval (INEX 2005). Schloss Dagstuhl, 28--30 Nov. 2005. Lecture Notes in Computer Science vol. 3977. Springer-Verlag. 16--29.]]"},{"key":"e_1_2_1_27_1","volume-title":"SIGIR: Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM","author":"Kazai G.","year":"2004","unstructured":"Kazai , G. , Lalmas , M. , and de Vries , A. P. 2004 . The overlap problem in content-oriented XML retrieval evaluation . In SIGIR: Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM , New York. 72--79.]] 10.1145\/1008992.1009008 Kazai, G., Lalmas, M., and de Vries, A. P. 2004. The overlap problem in content-oriented XML retrieval evaluation. In SIGIR: Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, New York. 72--79.]] 10.1145\/1008992.1009008"},{"key":"e_1_2_1_28_1","volume-title":"Advances in XML Information Retrieval: 3rd Workshop of the INitiative for the Evaluation of XML Retrieval (INEX 2004","author":"Kazai G.","year":"2005","unstructured":"Kazai , G. , Lalmas , M. , and de Vries , A. P. 2005 . Reliability tests for the XCG and inex-2002 metrics . In Advances in XML Information Retrieval: 3rd Workshop of the INitiative for the Evaluation of XML Retrieval (INEX 2004 ). Schloss Dagstuhl, 6--8 Germany , Dec. 2004. Lecture Notes in Computer Science vol. 3493. Springer-Verlag. 60--72.]] Kazai, G., Lalmas, M., and de Vries, A. P. 2005. Reliability tests for the XCG and inex-2002 metrics. In Advances in XML Information Retrieval: 3rd Workshop of the INitiative for the Evaluation of XML Retrieval (INEX 2004). Schloss Dagstuhl, 6--8 Germany, Dec. 2004. Lecture Notes in Computer Science vol. 3493. Springer-Verlag. 60--72.]]"},{"key":"e_1_2_1_29_1","volume-title":"Proceedings of the 2nd workshop of the Initiative for the Evaluation of XML Retrieval","author":"Kazai G.","year":"2003","unstructured":"Kazai , G. , Lalmas , M. , and Piwowarski , B . 2004. INEX relevance assessment guide . In Proceedings of the 2nd workshop of the Initiative for the Evaluation of XML Retrieval . Dagstuhl, Germany. 204--209. http:\/\/inex.is.informatik.uni-duisburg.de : 2003 \/proceedings.pdf.]] Kazai, G., Lalmas, M., and Piwowarski, B. 2004. INEX relevance assessment guide. In Proceedings of the 2nd workshop of the Initiative for the Evaluation of XML Retrieval. Dagstuhl, Germany. 204--209. http:\/\/inex.is.informatik.uni-duisburg.de:2003\/proceedings.pdf.]]"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2005.01.004"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.10137"},{"key":"e_1_2_1_32_1","volume-title":"INEX 2004 retrieval task and result submission specification. In Proceedings of the Advances in XML Information Retrieval: 3rd Workshop of the Initiative for the Evaluation of XML Retrieval","volume":"3493","author":"Lalmas M.","unstructured":"Lalmas , M. and Malik , S . 2004 . INEX 2004 retrieval task and result submission specification. In Proceedings of the Advances in XML Information Retrieval: 3rd Workshop of the Initiative for the Evaluation of XML Retrieval . Schloss Dagstuhl, Germany. Lecture Notes in Computer Science vol. 3493 . Springer-Verlag.]] Lalmas, M. and Malik, S. 2004. INEX 2004 retrieval task and result submission specification. In Proceedings of the Advances in XML Information Retrieval: 3rd Workshop of the Initiative for the Evaluation of XML Retrieval. Schloss Dagstuhl, Germany. Lecture Notes in Computer Science vol. 3493. Springer-Verlag.]]"},{"key":"e_1_2_1_33_1","volume-title":"Advances in XML Information Retrieval and Evaluation: 4th Workshop of the INitiative for the Evaluation of XML Retrieval (INEX","volume":"3977","author":"Larsen B.","year":"2005","unstructured":"Larsen , B. , Malik , S. , and Tombros , A . 2006. The interactive track at INEX 2005 . In Advances in XML Information Retrieval and Evaluation: 4th Workshop of the INitiative for the Evaluation of XML Retrieval (INEX 2005 ). Schloss Dagstuhl, Germany, 28--30 Nov. Lecture Notes in Computer Science vol. 3977 . Springer-Verlag. 404--417.]] 10.1007\/978-3-540-34963-1_30 Larsen, B., Malik, S., and Tombros, A. 2006. The interactive track at INEX 2005. In Advances in XML Information Retrieval and Evaluation: 4th Workshop of the INitiative for the Evaluation of XML Retrieval (INEX 2005). Schloss Dagstuhl, Germany, 28--30 Nov. Lecture Notes in Computer Science vol. 3977. Springer-Verlag. 404--417.]] 10.1007\/978-3-540-34963-1_30"},{"key":"e_1_2_1_34_1","doi-asserted-by":"crossref","first-page":"343","DOI":"10.1016\/0020-0271(68)90029-6","article-title":"Relevance assessments and retrieval system evaluation","volume":"4","author":"Lesk M.","year":"1969","unstructured":"Lesk , M. and Salton , G. 1969 . Relevance assessments and retrieval system evaluation . Inf. Storage and Retrieval 4 , 4, 343 -- 359 .]] Lesk, M. and Salton, G. 1969. Relevance assessments and retrieval system evaluation. Inf. Storage and Retrieval 4, 4, 343--359.]]","journal-title":"Inf. Storage and Retrieval"},{"key":"e_1_2_1_35_1","volume-title":"Overview of INEX 2005. In Advances in XML Information Retrieval and Evaluation: 4th Workshop of the INitiative for the Evaluation of XML Retrieval (INEX","volume":"3977","author":"Malik S.","year":"2005","unstructured":"Malik , S. , Kazai , G. , Lalmas , M. , and Fuhr , N . 2006 . Overview of INEX 2005. In Advances in XML Information Retrieval and Evaluation: 4th Workshop of the INitiative for the Evaluation of XML Retrieval (INEX 2005 ). Schloss Dagstuhl, Germany, 28--30 Nov. Lecture Notes in Computer Science vol. 3977 . Springer-Verlag. 1--15.]] 10.1007\/978-3-540-34963-1_1 Malik, S., Kazai, G., Lalmas, M., and Fuhr, N. 2006. Overview of INEX 2005. In Advances in XML Information Retrieval and Evaluation: 4th Workshop of the INitiative for the Evaluation of XML Retrieval (INEX 2005). Schloss Dagstuhl, Germany, 28--30 Nov. Lecture Notes in Computer Science vol. 3977. Springer-Verlag. 1--15.]] 10.1007\/978-3-540-34963-1_1"},{"key":"e_1_2_1_36_1","volume-title":"Overview of INEX 2004. In Advances in XML Information Retrieval: 3rd Workshop of the INitiative for the Evaluation of XML Retrieval (INEX","volume":"3493","author":"Malik S.","year":"2004","unstructured":"Malik , S. , Lalmas , M. , and Fuhr , N . 2005 . Overview of INEX 2004. In Advances in XML Information Retrieval: 3rd Workshop of the INitiative for the Evaluation of XML Retrieval (INEX 2004 ). Schloss Dagstuhl, Germany, 6--8 Dec. Lecture Notes in Computer Science vol. 3493 . Springer-Verlag. 1--15.]] Malik, S., Lalmas, M., and Fuhr, N. 2005. Overview of INEX 2004. In Advances in XML Information Retrieval: 3rd Workshop of the INitiative for the Evaluation of XML Retrieval (INEX 2004). Schloss Dagstuhl, Germany, 6--8 Dec. Lecture Notes in Computer Science vol. 3493. Springer-Verlag. 1--15.]]"},{"key":"e_1_2_1_37_1","volume-title":"Proceedings of the 2nd Workshop of the INitiative for the Evaluation of XML retrieval (INEX)","author":"Piwowarski B.","year":"2003","unstructured":"Piwowarski , B. and Gallinari , P . 2004. Expected ratio of relevant units: A measure for structured document information retrieval . In Proceedings of the 2nd Workshop of the INitiative for the Evaluation of XML retrieval (INEX) . Dagstuhl, Germany , Dec. 2003 . 158--166.]] Piwowarski, B. and Gallinari, P. 2004. Expected ratio of relevant units: A measure for structured document information retrieval. In Proceedings of the 2nd Workshop of the INitiative for the Evaluation of XML retrieval (INEX). Dagstuhl, Germany, Dec. 2003. 158--166.]]"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/1198296.1198297"},{"key":"e_1_2_1_39_1","volume-title":"SIGIR: Proceedings of the 12th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM","author":"Raghavan V. V.","unstructured":"Raghavan , V. V. , Bollmann , P. , and Jung , G. S . 1989. Retrieval system evaluation using recall and precision: Problems and answers . In SIGIR: Proceedings of the 12th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM , New York. 59--68.]] 10.1145\/75334.75342 Raghavan, V. V., Bollmann, P., and Jung, G. S. 1989. Retrieval system evaluation using recall and precision: Problems and answers. In SIGIR: Proceedings of the 12th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, New York. 59--68.]] 10.1145\/75334.75342"},{"key":"e_1_2_1_40_1","unstructured":"Rijsbergen C. J. V. 1979. Information Retrieval. Butterworth-Heinemann Newton MA http:\/\/www.dcs.glasgow.ac.uk\/Keith\/Preface.html.]]   Rijsbergen C. J. V. 1979. Information Retrieval. Butterworth-Heinemann Newton MA http:\/\/www.dcs.glasgow.ac.uk\/Keith\/Preface.html.]]"},{"key":"e_1_2_1_41_1","volume-title":"Proceedings of the NTCIR Workshop 4 Meeting Working Notes.]]","author":"Sakai T.","year":"2004","unstructured":"Sakai , T. 2004 . New performance metrics based on multigrade relevance: Their application to question answering . In Proceedings of the NTCIR Workshop 4 Meeting Working Notes.]] Sakai, T. 2004. New performance metrics based on multigrade relevance: Their application to question answering. In Proceedings of the NTCIR Workshop 4 Meeting Working Notes.]]"},{"key":"e_1_2_1_42_1","series-title":"Lecture Notes in Computer Science","volume-title":"The reliability of metrics based on graded relevance","author":"Sakai T.","unstructured":"Sakai , T. 2005. The reliability of metrics based on graded relevance . In AIRS, G. G. Lee et al. eds. Lecture Notes in Computer Science vol. 3689 . Springer-Verlag . 1--16.]] 10.1007\/11562382_1 Sakai, T. 2005. The reliability of metrics based on graded relevance. In AIRS, G. G. Lee et al. eds. Lecture Notes in Computer Science vol. 3689. Springer-Verlag. 1--16.]] 10.1007\/11562382_1"},{"key":"e_1_2_1_43_1","volume-title":"SIGIR: Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM","author":"Sanderson M.","unstructured":"Sanderson , M. and Zobel , J . 2005. Information retrieval system evaluation: Effort, sensitivity, and reliability . In SIGIR: Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM , New York. 162--169.]] 10.1145\/1076034.1076064 Sanderson, M. and Zobel, J. 2005. Information retrieval system evaluation: Effort, sensitivity, and reliability. In SIGIR: Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, New York. 162--169.]] 10.1145\/1076034.1076064"},{"key":"e_1_2_1_44_1","article-title":"Relevance and information behavior. Ammal","author":"Schamber L.","year":"1994","unstructured":"Schamber , L. 1994 . Relevance and information behavior. Ammal Rev. Inf. Sci. Technol. 3--48.]] Schamber, L. 1994. Relevance and information behavior. Ammal Rev. Inf. Sci. Technol. 3--48.]]","journal-title":"Rev. Inf. Sci. Technol. 3--48.]]"},{"key":"e_1_2_1_45_1","volume-title":"eds","author":"Sparck Jones K.","year":"1997","unstructured":"Sparck Jones , K. and Willett , P. , eds . 1997 . Readings in Information Retrieval. Morgan Kaufmann , San Francisco, CA.]] Sparck Jones, K. and Willett, P., eds. 1997. Readings in Information Retrieval. Morgan Kaufmann, San Francisco, CA.]]"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1016\/0306-4573(92)90005-K"},{"key":"e_1_2_1_47_1","volume-title":"Proceedings of the 3rd Workshop of the INitiative for the Evaluation of XML Retrieval (INEX)","author":"Tombros T.","year":"2004","unstructured":"Tombros , T. , Larsen , B. , and Malik , S . 2005. The interactive track at INEX 2004 . In Proceedings of the 3rd Workshop of the INitiative for the Evaluation of XML Retrieval (INEX) . Dagstuhl, Germany , Dec. 2004 .]] Tombros, T., Larsen, B., and Malik, S. 2005. The interactive track at INEX 2004. In Proceedings of the 3rd Workshop of the INitiative for the Evaluation of XML Retrieval (INEX). Dagstuhl, Germany, Dec. 2004.]]"},{"key":"e_1_2_1_48_1","volume-title":"Advances in XML Information Retrieval: 3rd Workshop of the INitiative for the Evaluation of XML Retrieval (INEX 2004","author":"Trotman A.","year":"2004","unstructured":"Trotman , A. and Sigurbj\u00f6rnsson , B . 2005. Narrowed Extended XPath I (NEXI) . In Advances in XML Information Retrieval: 3rd Workshop of the INitiative for the Evaluation of XML Retrieval (INEX 2004 ). Schloss Dagstuhl, Germany, 6- -8 Dec. 2004 . Lecture Notes in Computer Science vol. 3493. Springer-Verlag. 41--53.]] Trotman, A. and Sigurbj\u00f6rnsson, B. 2005. Narrowed Extended XPath I (NEXI). In Advances in XML Information Retrieval: 3rd Workshop of the INitiative for the Evaluation of XML Retrieval (INEX 2004). Schloss Dagstuhl, Germany, 6--8 Dec. 2004. Lecture Notes in Computer Science vol. 3493. Springer-Verlag. 41--53.]]"},{"key":"e_1_2_1_49_1","volume-title":"Proceedings of the 24th BCS-IRSG European Colloquium on IR Research. Springer-Verlag. 268--283","author":"Vegas J.","unstructured":"Vegas , J. , de la Fuente , P. , and Crestani , F . 2002. A graphical user interface for structured document retrieval . In Proceedings of the 24th BCS-IRSG European Colloquium on IR Research. Springer-Verlag. 268--283 .]] Vegas, J., de la Fuente, P., and Crestani, F. 2002. A graphical user interface for structured document retrieval. In Proceedings of the 24th BCS-IRSG European Colloquium on IR Research. Springer-Verlag. 268--283.]]"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0306-4573(00)00010-8"},{"key":"e_1_2_1_51_1","volume-title":"SIGIR: Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM New York. 74--82","author":"Voorhees E. M.","year":"2001","unstructured":"Voorhees , E. M. 2001 . Evaluation by highly relevant documents . In SIGIR: Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM New York. 74--82 .]] 10.1145\/383952.383963 Voorhees, E. M. 2001. Evaluation by highly relevant documents. In SIGIR: Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM New York. 74--82.]] 10.1145\/383952.383963"},{"key":"e_1_2_1_52_1","volume-title":"Overview of the TREC 2003 question answering track. In Proceedings of the Text REtrieval Conference","author":"Voorhees E. M.","year":"2003","unstructured":"Voorhees , E. M. 2003 a. Overview of the TREC 2003 question answering track. In Proceedings of the Text REtrieval Conference . Gaithersburg, Germany.]] Voorhees, E. M. 2003a. Overview of the TREC 2003 question answering track. In Proceedings of the Text REtrieval Conference. Gaithersburg, Germany.]]"},{"key":"e_1_2_1_53_1","volume-title":"Overview of the TREC 2003 robust retrieval track. In Proceedings of the TREC Conference. 69--77","author":"Voorhees E. M.","year":"2003","unstructured":"Voorhees , E. M. 2003 b. Overview of the TREC 2003 robust retrieval track. In Proceedings of the TREC Conference. 69--77 .]] Voorhees, E. M. 2003b. Overview of the TREC 2003 robust retrieval track. In Proceedings of the TREC Conference. 69--77.]]"},{"key":"e_1_2_1_54_1","volume-title":"SIGIR: Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM","author":"Voorhees E. M.","unstructured":"Voorhees , E. M. and Buckley , C . 2002. The effect of topic set size on retrieval experiment error . In SIGIR: Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM , New York. 316--323.]] 10.1145\/564376.564432 Voorhees, E. M. and Buckley, C. 2002. The effect of topic set size on retrieval experiment error. In SIGIR: Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, New York. 316--323.]] 10.1145\/564376.564432"},{"key":"e_1_2_1_55_1","volume-title":"TREC: Experiment and Evaluation in Information Retrieval","author":"Voorhees E. M.","year":"2005","unstructured":"Voorhees , E. M. and Harman , D. K . 2005 . TREC: Experiment and Evaluation in Information Retrieval . MIT Press Cambridge , MA .]] Voorhees, E. M. and Harman, D. K. 2005. TREC: Experiment and Evaluation in Information Retrieval. MIT Press Cambridge, MA.]]"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1016\/0306-4573(95)00061-5"}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/1185877.1185883","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,12,28]],"date-time":"2022-12-28T21:40:06Z","timestamp":1672263606000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1185877.1185883"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2006,10]]},"references-count":55,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2006,10]]}},"alternative-id":["10.1145\/1185877.1185883"],"URL":"https:\/\/doi.org\/10.1145\/1185877.1185883","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"value":"1046-8188","type":"print"},{"value":"1558-2868","type":"electronic"}],"subject":[],"published":{"date-parts":[[2006,10]]},"assertion":[{"value":"2006-10-01","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}