{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T04:28:21Z","timestamp":1750307301103,"version":"3.41.0"},"reference-count":11,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2011,1,3]],"date-time":"2011-01-03T00:00:00Z","timestamp":1294012800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGIR Forum"],"published-print":{"date-parts":[[2011,1,3]]},"abstract":"<jats:p>The Third International Workshop on Evaluating Information Access (EVIA 2010) was held at the National Institute of Informatics, Tokyo, on June 15th, 2010. Themes were evaluation depth; the management of collaborative experiments; result diversity; and user studies.<\/jats:p>","DOI":"10.1145\/1924475.1924490","type":"journal-article","created":{"date-parts":[[2011,1,24]],"date-time":"2011-01-24T14:58:13Z","timestamp":1295881093000},"page":"67-72","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["EVIA 2010"],"prefix":"10.1145","volume":"44","author":[{"given":"William","family":"Webber","sequence":"first","affiliation":[{"name":"University of Melbourne"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Tetsuya","family":"Sakai","sequence":"additional","affiliation":[{"name":"Microsoft Research, Asia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mark","family":"Sanderson","sequence":"additional","affiliation":[{"name":"RMIT University"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2011,1,3]]},"reference":[{"key":"e_1_2_1_1_1","first-page":"16","volume-title":"Marco Dussin, and Nicola Ferro. 10 years of CLEF data in DIRECT: Where we are and where we can go. In Sakai et al. {8}","author":"Agosti Maristella","unstructured":"Maristella Agosti , Giorgio Maria Di Nunzio , Marco Dussin, and Nicola Ferro. 10 years of CLEF data in DIRECT: Where we are and where we can go. In Sakai et al. {8} , pages 16 -- 24 . Maristella Agosti, Giorgio Maria Di Nunzio, Marco Dussin, and Nicola Ferro. 10 years of CLEF data in DIRECT: Where we are and where we can go. In Sakai et al. {8}, pages 16--24."},{"key":"e_1_2_1_2_1","first-page":"60","volume-title":"Sakai et al. {8}","author":"Lamm Katrin","unstructured":"Katrin Lamm , Thomas Mandl , Christa Womser-Hacker , and Werner Greve . The influence of expectation and system performance on user satisfaction with retrieval systems . In Sakai et al. {8} , pages 60 -- 68 . Katrin Lamm, Thomas Mandl, Christa Womser-Hacker, and Werner Greve. The influence of expectation and system performance on user satisfaction with retrieval systems. In Sakai et al. {8}, pages 60--68."},{"key":"e_1_2_1_3_1","first-page":"1","volume-title":"Sakai et al. {8}","author":"Nave John","unstructured":"John Nave . Microsoft's Bing and user behaviour evaluation . In Sakai et al. {8} , pages 1 -- 1 . John Nave. Microsoft's Bing and user behaviour evaluation. In Sakai et al. {8}, pages 1--1."},{"key":"e_1_2_1_4_1","first-page":"2","volume-title":"Sakai et al. {8}","author":"Pal Sukomal","unstructured":"Sukomal Pal , Mandar Mitra , and Samaresh Maiti . Estimating pool-depth on per query basis . In Sakai et al. {8} , pages 2 -- 6 . Sukomal Pal, Mandar Mitra, and Samaresh Maiti. Estimating pool-depth on per query basis. In Sakai et al. {8}, pages 2--6."},{"key":"e_1_2_1_5_1","first-page":"69","volume-title":"Sakai et al. {8}","author":"Park Keun Chan","unstructured":"Keun Chan Park , Jihee Ryu , Kyung-min Kim, and Sung Hyon Myaeng . A game-based evaluation method for subjective tasks involving text content analysis . In Sakai et al. {8} , pages 69 -- 76 . Keun Chan Park, Jihee Ryu, Kyung-min Kim, and Sung Hyon Myaeng. A game-based evaluation method for subjective tasks involving text content analysis. In Sakai et al. {8}, pages 69--76."},{"key":"e_1_2_1_6_1","first-page":"25","volume-title":"Sakai et al. {8}","author":"Sakai Tetsuya","unstructured":"Tetsuya Sakai and Chin-Yew Lin . Ranking retrieval systems without relevance assessments: revisited . In Sakai et al. {8} , pages 25 -- 33 . Tetsuya Sakai and Chin-Yew Lin. Ranking retrieval systems without relevance assessments: revisited. In Sakai et al. {8}, pages 25--33."},{"key":"e_1_2_1_7_1","first-page":"42","volume-title":"Sakai et al. {8}","author":"Sakai Tetsuya","unstructured":"Tetsuya Sakai , Nick Craswell , Ruihua Song , Stephen Robertson , Zhicheng Dou , and Chin-Yew Lin . Simple evaluation metrics for diversified search results . In Sakai et al. {8} , pages 42 -- 50 . Tetsuya Sakai, Nick Craswell, Ruihua Song, Stephen Robertson, Zhicheng Dou, and Chin-Yew Lin. Simple evaluation metrics for diversified search results. In Sakai et al. {8}, pages 42--50."},{"key":"e_1_2_1_8_1","volume-title":"Proceedings of the Third International Workshop on Evaluating Information Access","author":"Sakai Tetsuya","year":"2010","unstructured":"Tetsuya Sakai , Mark Sanderson , and William Webber , editors. Proceedings of the Third International Workshop on Evaluating Information Access , Tokyo, Japan , June 2010 . URL http:\/\/research.nii.ac.jp\/ntcir\/workshop\/OnlineProceedings8\/EVIA\/. Tetsuya Sakai, Mark Sanderson, and William Webber, editors. Proceedings of the Third International Workshop on Evaluating Information Access, Tokyo, Japan, June 2010. URL http:\/\/research.nii.ac.jp\/ntcir\/workshop\/OnlineProceedings8\/EVIA\/."},{"key":"e_1_2_1_9_1","first-page":"34","volume-title":"Sakai et al. {8}","author":"Soboroff Ian","unstructured":"Ian Soboroff . Test collection diagnosis and treatment . In Sakai et al. {8} , pages 34 -- 41 . Ian Soboroff. Test collection diagnosis and treatment. In Sakai et al. {8}, pages 34--41."},{"key":"e_1_2_1_10_1","first-page":"51","volume-title":"Sakai et al. {8}","author":"Song Ruihua","unstructured":"Ruihua Song , Dongjie Qi , Hua Liu , Tetsuya Sakai , Jian-Yun Nie , Hsiao-Wuen Hon , and Yong Yu . Constructing a test collection with multi-intent queries . In Sakai et al. {8} , pages 51 -- 59 . Ruihua Song, Dongjie Qi, Hua Liu, Tetsuya Sakai, Jian-Yun Nie, Hsiao-Wuen Hon, and Yong Yu. Constructing a test collection with multi-intent queries. In Sakai et al. {8}, pages 51--59."},{"key":"e_1_2_1_11_1","first-page":"7","volume-title":"Sakai et al. {8}","author":"Webber William","unstructured":"William Webber , Alistair Moffat , and Justin Zobel . The effect of pooling and evaluation depth on metric stability . In Sakai et al. {8} , pages 7 -- 15 . William Webber, Alistair Moffat, and Justin Zobel. The effect of pooling and evaluation depth on metric stability. In Sakai et al. {8}, pages 7--15."}],"container-title":["ACM SIGIR Forum"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1924475.1924490","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/1924475.1924490","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T10:59:53Z","timestamp":1750244393000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/1924475.1924490"}},"subtitle":["the third international workshop on evaluating information access"],"short-title":[],"issued":{"date-parts":[[2011,1,3]]},"references-count":11,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2011,1,3]]}},"alternative-id":["10.1145\/1924475.1924490"],"URL":"https:\/\/doi.org\/10.1145\/1924475.1924490","relation":{},"ISSN":["0163-5840"],"issn-type":[{"type":"print","value":"0163-5840"}],"subject":[],"published":{"date-parts":[[2011,1,3]]},"assertion":[{"value":"2011-01-03","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}