{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,4]],"date-time":"2026-04-04T06:15:04Z","timestamp":1775283304301,"version":"3.50.1"},"publisher-location":"New York, NY, USA","reference-count":48,"publisher":"ACM","license":[{"start":{"date-parts":[[2020,3,14]],"date-time":"2020-03-14T00:00:00Z","timestamp":1584144000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2020,3,14]]},"DOI":"10.1145\/3343413.3377959","type":"proceedings-article","created":{"date-parts":[[2020,3,12]],"date-time":"2020-03-12T18:39:33Z","timestamp":1584038373000},"page":"213-222","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":14,"title":["Data-Driven Evaluation Metrics for Heterogeneous Search Engine Result Pages"],"prefix":"10.1145","author":[{"given":"Leif","family":"Azzopardi","sequence":"first","affiliation":[{"name":"University of Strathclyde, Glasgow, United Kingdom"}]},{"given":"Ryen W.","family":"White","sequence":"additional","affiliation":[{"name":"Microsoft, Redmond, WA, USA"}]},{"given":"Paul","family":"Thomas","sequence":"additional","affiliation":[{"name":"Microsoft, Canberra, Australia"}]},{"given":"Nick","family":"Craswell","sequence":"additional","affiliation":[{"name":"Microsoft, Bellevue, WA, USA"}]}],"member":"320","published-online":{"date-parts":[[2020,3,14]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.21300"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3291992.3291996"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3210024"},{"key":"e_1_3_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1145\/2505515.2505693"},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2348283.2348343"},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3210027"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/3331184.3331398"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835449.1835606"},{"key":"e_1_3_2_1_9_1","volume-title":"ACM Sigir forum","author":"Broder Andrei","unstructured":"Andrei Broder . 2002. A taxonomy of web search . In ACM Sigir forum , Vol. 36 . ACM , 3--10. Andrei Broder. 2002. A taxonomy of web search. In ACM Sigir forum, Vol. 36. ACM, 3--10."},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835449.1835459"},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/2009916.2010037"},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1145\/1645953.1646033"},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/1772690.1772714"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390446"},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0121945"},{"key":"e_1_3_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/1242572.1242651"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/1840784.1840812"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-008-9045-0"},{"key":"e_1_3_2_1_19_1","first-page":"32","article-title":"Some Common Mistakes In IR Evaluation, And How They Can Be Avoided","volume":"52","author":"Fuhr Norbert","year":"2017","unstructured":"Norbert Fuhr . 2017 . Some Common Mistakes In IR Evaluation, And How They Can Be Avoided . SIGIR Forum , Vol. 52 , 2 (2017), 32 -- 41 . Norbert Fuhr. 2017. Some Common Mistakes In IR Evaluation, And How They Can Be Avoided. SIGIR Forum , Vol. 52, 2 (2017), 32--41.","journal-title":"SIGIR Forum"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/2505515.2505682"},{"key":"e_1_3_2_1_21_1","doi-asserted-by":"publisher","DOI":"10.1561\/1500000051"},{"key":"e_1_3_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/1242572.1242739"},{"key":"e_1_3_2_1_23_1","volume-title":"ACM Trans. Inf. Syst.","volume":"20","author":"Jaana Kalervo","year":"2002","unstructured":"Kalervo J\"arvelin and Jaana Kek\"al\"ainen. 2002 . Cumulated Gain-based Evaluation of IR Techniques . ACM Trans. Inf. Syst. , Vol. 20 , 4 (Oct. 2002), 422--446. Kalervo J\"arvelin and Jaana Kek\"al\"ainen. 2002. Cumulated Gain-based Evaluation of IR Techniques. ACM Trans. Inf. Syst. , Vol. 20, 4 (Oct. 2002), 422--446."},{"key":"e_1_3_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-30671-1_14"},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/2854946.2855005"},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3132847.3133033"},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2009916.2010056"},{"key":"e_1_3_2_1_28_1","doi-asserted-by":"publisher","DOI":"10.1023\/A:1009902609570"},{"key":"e_1_3_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.10137"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2914737"},{"key":"e_1_3_2_1_31_1","volume-title":"JASIST","volume":"59","author":"Lorigo Lori","year":"2008","unstructured":"Lori Lorigo , Maya Haridasan , Hr\u00f6nn Brynjarsd\u00f3ttir , Ling Xia , Thorsten Joachims , Geri Gay , Laura Granka , Fabio Pellacini , and Bing Pan . 2008 . Eye Tracking and Online Search . JASIST , Vol. 59 , 7 (2008), 1041--1052. Lori Lorigo, Maya Haridasan, Hr\u00f6nn Brynjarsd\u00f3ttir, Ling Xia, Thorsten Joachims, Geri Gay, Laura Granka, Fabio Pellacini, and Bing Pan. 2008. Eye Tracking and Online Search. JASIST , Vol. 59, 7 (2008), 1041--1052."},{"key":"e_1_3_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3077136.3080795"},{"key":"e_1_3_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/2838931.2838938"},{"key":"e_1_3_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3052768"},{"key":"e_1_3_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/2505515.2507665"},{"key":"e_1_3_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3210075"},{"key":"e_1_3_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/1416950.1416952"},{"key":"e_1_3_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835449.1835550"},{"key":"e_1_3_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/2484028.2484031"},{"key":"e_1_3_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.1561\/1500000009"},{"key":"e_1_3_2_1_41_1","volume-title":"Proc. of the 35th ACM SIGIR Conference . 95--104","author":"Mark","unstructured":"Mark D. Smucker and Charles L.A. Clarke. 2012. Time-Based Calibration of Effectiveness Measures . In Proc. of the 35th ACM SIGIR Conference . 95--104 . Mark D. Smucker and Charles L.A. Clarke. 2012. Time-Based Calibration of Effectiveness Measures. In Proc. of the 35th ACM SIGIR Conference . 95--104."},{"key":"e_1_3_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390364"},{"key":"e_1_3_2_1_43_1","volume-title":"TREC: Experiment and Evaluation in Information Retrieval","author":"Voorhees Ellen","year":"2005","unstructured":"Ellen Voorhees and Donna Harman . 2005 . TREC: Experiment and Evaluation in Information Retrieval . The MIT press . Ellen Voorhees and Donna Harman. 2005. TREC: Experiment and Evaluation in Information Retrieval .The MIT press."},{"key":"e_1_3_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/3269206.3269242"},{"key":"e_1_3_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1145\/3209978.3209990"},{"key":"e_1_3_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10791-017-9316-8"},{"key":"e_1_3_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/1871437.1871672"},{"key":"e_1_3_2_1_48_1","doi-asserted-by":"publisher","DOI":"10.1145\/3077136.3080841"}],"event":{"name":"CHIIR '20: Conference on Human Information Interaction and Retrieval","location":"Vancouver BC Canada","acronym":"CHIIR '20","sponsor":["SIGIR ACM Special Interest Group on Information Retrieval","SIGCHI ACM Special Interest Group on Computer-Human Interaction"]},"container-title":["Proceedings of the 2020 Conference on Human Information Interaction and Retrieval"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3343413.3377959","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3343413.3377959","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:13:06Z","timestamp":1750201986000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3343413.3377959"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,3,14]]},"references-count":48,"alternative-id":["10.1145\/3343413.3377959","10.1145\/3343413"],"URL":"https:\/\/doi.org\/10.1145\/3343413.3377959","relation":{},"subject":[],"published":{"date-parts":[[2020,3,14]]},"assertion":[{"value":"2020-03-14","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}