{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T04:18:14Z","timestamp":1750306694037,"version":"3.41.0"},"reference-count":53,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2013,1,21]],"date-time":"2013-01-21T00:00:00Z","timestamp":1358726400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGIR Forum"],"published-print":{"date-parts":[[2013,12,21]]},"abstract":"<jats:p>The SIGIR 2013 Workshop on Modeling User Behavior of Information Retrieval Evaluation brought together researchers interested in improving Cranfield-style evaluation of information retrieval through the modeling of user behavior. The workshop included two invited talks, ten short paper presentations, and breakout groups. Workshop participants brainstormed research questions of interest and formed breakout groups to explore these questions in greater depth. In addition to summarizing the invited talks and presentations, this report details the results of the breakout groups, which provide a set of research directions for the improvement of information retrieval evaluation.<\/jats:p>","DOI":"10.1145\/2568388.2568403","type":"journal-article","created":{"date-parts":[[2014,1,28]],"date-time":"2014-01-28T13:49:22Z","timestamp":1390916962000},"page":"84-95","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":8,"title":["Report on the SIGIR 2013 workshop on modeling user behavior for information retrieval evaluation (MUBE 2013)"],"prefix":"10.1145","volume":"47","author":[{"given":"Charles L.A.","family":"Clarke","sequence":"first","affiliation":[{"name":"University of Waterloo, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Luanne","family":"Freund","sequence":"additional","affiliation":[{"name":"University of British Columbia, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mark D.","family":"Smucker","sequence":"additional","affiliation":[{"name":"University of Waterloo, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Emine","family":"Yilmaz","sequence":"additional","affiliation":[{"name":"University College London, UK"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2013,1,21]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/133160.133169"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2009916.2009965"},{"key":"e_1_2_1_3_1","first-page":"1","volume-title":"Clarke et al. {19}","author":"Ageev M.","unstructured":"M. Ageev , D. Lagun , and E. Agichtein . Towards task-based snippet evaluation: preliminary results and challenges . In Clarke et al. {19} , pages 1 -- 2 . M. Ageev, D. Lagun, and E. Agichtein. Towards task-based snippet evaluation: preliminary results and challenges. In Clarke et al. {19}, pages 1--2."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-010-9133-9"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/1645953.1646034"},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/2009916.2009923"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/1924475.1924484"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/2484028.2484049"},{"key":"e_1_2_1_9_1","first-page":"3","volume-title":"Clarke et al. {19}","author":"Azzopardi L.","unstructured":"L. Azzopardi , C. Wilkie , and T. Russell-Rose . Towards measures and models of findability . In Clarke et al. {19} , pages 3 -- 4 . L. Azzopardi, C. Wilkie, and T. Russell-Rose. Towards measures and models of findability. In Clarke et al. {19}, pages 3--4."},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/LAWEB.2005.23"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2348283.2348301"},{"key":"e_1_2_1_12_1","first-page":"5","volume-title":"Clarke et al. {19}","author":"Bruza P.","unstructured":"P. Bruza , G. Zuccon , and L. Sitbon . Modelling the information seeking user by the decisions they make . In Clarke et al. {19} , pages 5 -- 6 . P. Bruza, G. Zuccon, and L. Sitbon. Modelling the information seeking user by the decisions they make. In Clarke et al. {19}, pages 5--6."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/2009916.2010037"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/2063576.2063668"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2396761.2396782"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/1645953.1646033"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/365024.365325"},{"key":"e_1_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2484028.2484071"},{"key":"e_1_2_1_19_1","volume-title":"Proceedings of the SIGIR 2013 Workshop on Modeling User Behavior for Information Retrieval Evaluation (MUBE 2013)","author":"Clarke C. L. A.","year":"2013","unstructured":"C. L. A. Clarke , L. Freund , M. D. Smucker , and E. Yilmaz , editors . Proceedings of the SIGIR 2013 Workshop on Modeling User Behavior for Information Retrieval Evaluation (MUBE 2013) , 2013 . C. L. A. Clarke, L. Freund, M. D. Smucker, and E. Yilmaz, editors. Proceedings of the SIGIR 2013 Workshop on Modeling User Behavior for Information Retrieval Evaluation (MUBE 2013), 2013."},{"key":"e_1_2_1_20_1","first-page":"463","volume-title":"RIAO","author":"de Vries A. P.","year":"2004","unstructured":"A. P. de Vries , G. Kazai , and M. Lalmas . Tolerance to irrelevance: A user-effort oriented evaluation of retrieval systems without predefined retrieval unit . In RIAO , pages 463 -- 473 , 2004 . A. P. de Vries, G. Kazai, and M. Lalmas. Tolerance to irrelevance: A user-effort oriented evaluation of retrieval systems without predefined retrieval unit. In RIAO, pages 463--473, 2004."},{"key":"e_1_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1145\/258525.258569"},{"key":"e_1_2_1_22_1","first-page":"2","volume-title":"SPIRE","author":"Dupret G.","year":"2011","unstructured":"G. Dupret . Discounted cumulative gain and user decision models . In SPIRE , pages 2 -- 13 , 2011 . G. Dupret. Discounted cumulative gain and user decision models. In SPIRE, pages 2--13, 2011."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835449.1835538"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/1718487.1718515"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/582415.582418"},{"key":"e_1_2_1_26_1","first-page":"7","volume-title":"Clarke et al. {19}","author":"Jiang J.","unstructured":"J. Jiang and D. He . Simulating user selections of query suggestions . In Clarke et al. {19} , pages 7 -- 8 . J. Jiang and D. He. Simulating user selections of query suggestions. In Clarke et al. {19}, pages 7--8."},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2009916.2010056"},{"key":"e_1_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-04769-5_6"},{"key":"e_1_2_1_29_1","first-page":"9","volume-title":"Clarke et al. {19}","author":"Kharitonov E.","unstructured":"E. Kharitonov , C. Macdonald , P. Serdyukov , and I. Ounis . Incorporating efficiency in evaluation . In Clarke et al. {19} , pages 9 -- 10 . E. Kharitonov, C. Macdonald, P. Serdyukov, and I. Ounis. Incorporating efficiency in evaluation. In Clarke et al. {19}, pages 9--10."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2006.06.006"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390340"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-008-9067-7"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/1416950.1416952"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/1135777.1136000"},{"key":"e_1_2_1_35_1","doi-asserted-by":"crossref","DOI":"10.1093\/acprof:oso\/9780195173321.001.0001","volume-title":"Information Foraging Theory","author":"Pirolli P.","year":"2007","unstructured":"P. Pirolli . Information Foraging Theory . Oxford University Press , 2007 . P. Pirolli. Information Foraging Theory. Oxford University Press, 2007."},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/1871437.1871563"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390453"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/860435.860475"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2484028.2484031"},{"key":"e_1_2_1_40_1","volume-title":"The Second International Workshop on Evaluating Information Access (EVIA)","author":"Sakai T.","year":"2008","unstructured":"T. Sakai and S. Robertson . Modelling a user population for designing information retrieval metrics . In The Second International Workshop on Evaluating Information Access (EVIA) , 2008 . T. Sakai and S. Robertson. Modelling a user population for designing information retrieval metrics. In The Second International Workshop on Evaluating Information Access (EVIA), 2008."},{"key":"e_1_2_1_41_1","first-page":"11","volume-title":"Clarke et al. {19}","author":"Scholer F.","unstructured":"F. Scholer , P. Thomas , and A. Moffat . Observing users to validate models . In Clarke et al. {19} , pages 11 -- 12 . F. Scholer, P. Thomas, and A. Moffat. Observing users to validate models. In Clarke et al. {19}, pages 11--12."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/1148170.1148250"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/2348283.2348300"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/2391224.2391227"},{"key":"e_1_2_1_45_1","first-page":"13","volume-title":"Clarke et al. {19}","author":"Tran V. T.","unstructured":"V. T. Tran and N. Fuhr . Markov modeling for user interaction in retrieval . In Clarke et al. {19} , pages 13 -- 14 . V. T. Tran and N. Fuhr. Markov modeling for user interaction in retrieval. In Clarke et al. {19}, pages 13--14."},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1145\/1571941.1572029"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-540-24752-4_23"},{"key":"e_1_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-04417-5_15"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/1871437.1871672"},{"key":"e_1_2_1_50_1","first-page":"15","volume-title":"Clarke et al. {19}","author":"Younus A.","unstructured":"A. Younus , M. A. Qureshi , C. O'Riordan , and G. Pasi . Personalization for difficult queries . In Clarke et al. {19} , pages 15 -- 16 . A. Younus, M. A. Qureshi, C. O'Riordan, and G. Pasi. Personalization for difficult queries. In Clarke et al. {19}, pages 15--16."},{"key":"e_1_2_1_51_1","first-page":"17","volume-title":"Clarke et al. {19}","author":"Zengin M.","unstructured":"M. Zengin and B. Carterette . User judgements of document similarity . In Clarke et al. {19} , pages 17 -- 18 . M. Zengin and B. Carterette. User judgements of document similarity. In Clarke et al. {19}, pages 17--18."},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-009-9099-7"},{"key":"e_1_2_1_53_1","first-page":"19","volume-title":"Clarke et al. {19}","author":"Zhou K.","unstructured":"K. Zhou , T. Sakai , M. Lalmas , Z. Dou , and J. M. Jose . Evaluating heterogeneous information access (position paper) . In Clarke et al. {19} , pages 19 -- 20 . K. Zhou, T. Sakai, M. Lalmas, Z. Dou, and J. M. Jose. Evaluating heterogeneous information access (position paper). In Clarke et al. {19}, pages 19--20."}],"container-title":["ACM SIGIR Forum"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2568388.2568403","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/2568388.2568403","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T07:01:38Z","timestamp":1750230098000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/2568388.2568403"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2013,1,21]]},"references-count":53,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2013,12,21]]}},"alternative-id":["10.1145\/2568388.2568403"],"URL":"https:\/\/doi.org\/10.1145\/2568388.2568403","relation":{},"ISSN":["0163-5840"],"issn-type":[{"type":"print","value":"0163-5840"}],"subject":[],"published":{"date-parts":[[2013,1,21]]},"assertion":[{"value":"2013-01-21","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}