{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,17]],"date-time":"2026-02-17T03:40:18Z","timestamp":1771299618697,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":16,"publisher":"ACM","license":[{"start":{"date-parts":[[2021,10,26]],"date-time":"2021-10-26T00:00:00Z","timestamp":1635206400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000001","name":"National Science Foundation","doi-asserted-by":"publisher","award":["2106152"],"award-info":[{"award-number":["2106152"]}],"id":[{"id":"10.13039\/100000001","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2021,10,26]]},"DOI":"10.1145\/3459637.3482190","type":"proceedings-article","created":{"date-parts":[[2021,11,15]],"date-time":"2021-11-15T15:31:19Z","timestamp":1636990279000},"page":"3258-3262","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":18,"title":["State-Aware Meta-Evaluation of Evaluation Metrics in Interactive Information Retrieval"],"prefix":"10.1145","author":[{"given":"Jiqun","family":"Liu","sequence":"first","affiliation":[{"name":"The University of Oklahoma, Norman, OK, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ran","family":"Yu","sequence":"additional","affiliation":[{"name":"University of Bonn, Bonn, Germany"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2021,10,30]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/2888422.2888424"},{"key":"e_1_3_2_1_2_1","volume-title":"Data requirements for evaluation of personalization of information retrieval-a position paper. arXiv preprint arXiv:1809.02412","author":"Belkin Nicholas J","year":"2018","unstructured":"Nicholas J Belkin , Daniel Hienert , Philipp Mayr , and Chirag Shah . 2018. Data requirements for evaluation of personalization of information retrieval-a position paper. arXiv preprint arXiv:1809.02412 ( 2018 ). Nicholas J Belkin, Daniel Hienert, Philipp Mayr, and Chirag Shah. 2018. Data requirements for evaluation of personalization of information retrieval-a position paper. arXiv preprint arXiv:1809.02412 (2018)."},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/2505515.2505714"},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/3077136.3080804"},{"key":"e_1_3_2_1_5_1","volume-title":"Proc. HCIR","author":"Cole Michael","year":"2009","unstructured":"Michael Cole , Jingjing Liu , Nicholas Belkin , Ralf Bierig , Jacek Gwizdka , C Liu , Jin Zhang , and X Zhang . 2009 . Usefulness as the criterion for evaluation of interactive information retrieval . Proc. HCIR (2009), 1--4. Michael Cole, Jingjing Liu, Nicholas Belkin, Ralf Bierig, Jacek Gwizdka, C Liu, Jin Zhang, and X Zhang. 2009. Usefulness as the criterion for evaluation of interactive information retrieval. Proc. HCIR (2009), 1--4."},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3274784.3274786"},{"key":"e_1_3_2_1_7_1","volume-title":"Variable selection using mean decrease accuracy and mean decrease gini based on random forest. In 2016 7th ieee international conference on software engineering and service science (icsess)","author":"Han Hong","unstructured":"Hong Han , Xiaoling Guo , and Hua Yu. 2016. Variable selection using mean decrease accuracy and mean decrease gini based on random forest. In 2016 7th ieee international conference on software engineering and service science (icsess) . IEEE , 219--224. Hong Han, Xiaoling Guo, and Hua Yu. 2016. Variable selection using mean decrease accuracy and mean decrease gini based on random forest. In 2016 7th ieee international conference on software engineering and service science (icsess). IEEE, 219--224."},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-30671-1_14"},{"key":"e_1_3_2_1_9_1","volume-title":"Report on the CHIIR 2018 Workshop on Evaluation of Personalisation in Information Retrieval (WEPIR","volume":"52","author":"Jones Gareth JF","year":"2018","unstructured":"Gareth JF Jones , Nicholas J Belkin , S\u00e9amus Lawless , and Gabriella Pasi . 2018 . Report on the CHIIR 2018 Workshop on Evaluation of Personalisation in Information Retrieval (WEPIR 2018). In ACM SIGIR Forum , Vol. 52 . ACM New York, NY, USA, 129--134. Gareth JF Jones, Nicholas J Belkin, S\u00e9amus Lawless, and Gabriella Pasi. 2018. Report on the CHIIR 2018 Workshop on Evaluation of Personalisation in Information Retrieval (WEPIR 2018). In ACM SIGIR Forum, Vol. 52. ACM New York, NY, USA, 129--134."},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-76941-7_43"},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2021.102522"},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/3406522.3446015"},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/3295750.3298922"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/3343413.3377976"},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/3292500.3330981"},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/3077136.3080737"}],"event":{"name":"CIKM '21: The 30th ACM International Conference on Information and Knowledge Management","location":"Virtual Event Queensland Australia","acronym":"CIKM '21","sponsor":["SIGWEB ACM Special Interest Group on Hypertext, Hypermedia, and Web","SIGIR ACM Special Interest Group on Information Retrieval"]},"container-title":["Proceedings of the 30th ACM International Conference on Information & Knowledge Management"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3459637.3482190","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3459637.3482190","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3459637.3482190","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:12:12Z","timestamp":1750191132000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3459637.3482190"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,10,26]]},"references-count":16,"alternative-id":["10.1145\/3459637.3482190","10.1145\/3459637"],"URL":"https:\/\/doi.org\/10.1145\/3459637.3482190","relation":{},"subject":[],"published":{"date-parts":[[2021,10,26]]},"assertion":[{"value":"2021-10-30","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}