{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,24]],"date-time":"2025-06-24T19:10:11Z","timestamp":1750792211005,"version":"3.41.0"},"publisher-location":"New York, NY, USA","reference-count":31,"publisher":"ACM","license":[{"start":{"date-parts":[[2017,8,7]],"date-time":"2017-08-07T00:00:00Z","timestamp":1502064000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100000038","name":"Natural Sciences and Engineering Research Council of Canada","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100000038","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":[],"published-print":{"date-parts":[[2017,8,7]]},"DOI":"10.1145\/3077136.3080808","type":"proceedings-article","created":{"date-parts":[[2017,7,28]],"date-time":"2017-07-28T19:35:01Z","timestamp":1501270501000},"page":"415-424","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":10,"title":["Online In-Situ Interleaved Evaluation of Real-Time Push Notification Systems"],"prefix":"10.1145","author":[{"given":"Adam","family":"Roegiest","sequence":"first","affiliation":[{"name":"University of Waterloo, Waterloo, ON, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Luchen","family":"Tan","sequence":"additional","affiliation":[{"name":"University of Waterloo, Waterloo, ON, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jimmy","family":"Lin","sequence":"additional","affiliation":[{"name":"University of Waterloo, Waterloo, ON, Canada"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2017,8,7]]},"reference":[{"key":"e_1_3_2_1_1_1","doi-asserted-by":"crossref","unstructured":"Azzah Al-Maskari Mark Sanderson Paul Clough and Eija Airio. 2008. The Good and the Bad System: Does the Test Collection Predict Users' Effectiveness? SIGIR. 59--66. Azzah Al-Maskari Mark Sanderson Paul Clough and Eija Airio. 2008. The Good and the Bad System: Does the Test Collection Predict Users' Effectiveness? SIGIR. 59--66.","DOI":"10.1145\/1390334.1390347"},{"key":"e_1_3_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-1-4615-0933-2"},{"key":"e_1_3_2_1_3_1","doi-asserted-by":"crossref","unstructured":"James Allan Ben Carterette and Joshua Lewis. 2005. When Will Information Retrieval Be \"Good Enough\"? User Effectiveness as a Function of Retrieval Accuracy. In SIGIR. 433--440. James Allan Ben Carterette and Joshua Lewis. 2005. When Will Information Retrieval Be \"Good Enough\"? User Effectiveness as a Function of Retrieval Accuracy. In SIGIR. 433--440.","DOI":"10.1145\/1076034.1076109"},{"volume-title":"TREC 2015 Temporal Summarization Track Overview TREC.","year":"2015","author":"Aslam Javed","key":"e_1_3_2_1_4_1"},{"key":"e_1_3_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/1390334.1390447"},{"key":"e_1_3_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/138859.138861"},{"key":"e_1_3_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1145\/2094072.2094078"},{"key":"e_1_3_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-36973-5_41"},{"key":"e_1_3_2_1_9_1","unstructured":"Allan Hanbury Henning M\u00fcller Krisztian Balog Torben Brodt Gordon V. Cormack Ivan Eggel Tim Gollub Frank Hopfgartner Jayashree Kalpathy-Cramer Noriko Kando Anastasia Krithara Jimmy Lin Simon Mercer and Martin Potthast. 2015. Evaluation-as-a-Service: Overview and Outlook. arXiv:1512.07454. Allan Hanbury Henning M\u00fcller Krisztian Balog Torben Brodt Gordon V. Cormack Ivan Eggel Tim Gollub Frank Hopfgartner Jayashree Kalpathy-Cramer Noriko Kando Anastasia Krithara Jimmy Lin Simon Mercer and Martin Potthast. 2015. Evaluation-as-a-Service: Overview and Outlook. arXiv:1512.07454."},{"key":"e_1_3_2_1_10_1","doi-asserted-by":"crossref","unstructured":"William Hersh Andrew Turpin Susan Price Benjamin Chan Dale Kramer Lynetta Sacherek and Daniel Olson. 2000. Do Batch and User Evaluations Give the Same Results? SIGIR. 17--24. William Hersh Andrew Turpin Susan Price Benjamin Chan Dale Kramer Lynetta Sacherek and Daniel Olson. 2000. Do Batch and User Evaluations Give the Same Results? SIGIR. 17--24.","DOI":"10.1145\/345508.345539"},{"key":"e_1_3_2_1_11_1","doi-asserted-by":"crossref","unstructured":"Katja Hofmann Shimon Whiteson and Maarten de Rijke. 2011. A Probabilistic Method for Inferring Preferences from Clicks CIKM. 249--258. Katja Hofmann Shimon Whiteson and Maarten de Rijke. 2011. A Probabilistic Method for Inferring Preferences from Clicks CIKM. 249--258.","DOI":"10.1145\/2063576.2063618"},{"key":"e_1_3_2_1_12_1","doi-asserted-by":"crossref","unstructured":"Ron Kohavi Randal M. Henne and Dan Sommerfield. 2007. Practical Guide to Controlled Experiments on the Web: Listen to Your Customers not to the HiPPO. In KDD. 959--967. Ron Kohavi Randal M. Henne and Dan Sommerfield. 2007. Practical Guide to Controlled Experiments on the Web: Listen to Your Customers not to the HiPPO. In KDD. 959--967.","DOI":"10.1145\/1281192.1281295"},{"key":"e_1_3_2_1_13_1","doi-asserted-by":"crossref","unstructured":"David D. Lewis. 1995. The TREC-4 Filtering Track. In TREC. 165--180. David D. Lewis. 1995. The TREC-4 Filtering Track. In TREC. 165--180.","DOI":"10.6028\/NIST.SP.500-236.filtering-overview"},{"key":"e_1_3_2_1_14_1","doi-asserted-by":"crossref","unstructured":"Jimmy Lin Miles Efron Yulu Wang and Garrick Sherman. 2014. Overview of the TREC-2014 Microblog Track. TREC. Jimmy Lin Miles Efron Yulu Wang and Garrick Sherman. 2014. Overview of the TREC-2014 Microblog Track. TREC.","DOI":"10.6028\/NIST.SP.500-308.microblog-overview"},{"key":"e_1_3_2_1_15_1","doi-asserted-by":"crossref","unstructured":"Jimmy Lin Miles Efron Yulu Wang Garrick Sherman and Ellen Voorhees. 2015. Overview of the TREC-2015 Microblog Track. TREC. Jimmy Lin Miles Efron Yulu Wang Garrick Sherman and Ellen Voorhees. 2015. Overview of the TREC-2015 Microblog Track. TREC.","DOI":"10.6028\/NIST.SP.500-319.microblog-overview"},{"volume-title":"Overview of the TREC 2016 Real-Time Summarization Track TREC.","year":"2016","author":"Lin Jimmy","key":"e_1_3_2_1_16_1"},{"key":"e_1_3_2_1_17_1","doi-asserted-by":"crossref","unstructured":"Abhinav Mehrotra Veljko Pejovic Jo Vermeulen Robert Hendley and Mirco Musolesi. 2016. My Phone and Me: Understanding People's Receptivity to Mobile Notifications CHI. 1021--1032. Abhinav Mehrotra Veljko Pejovic Jo Vermeulen Robert Hendley and Mirco Musolesi. 2016. My Phone and Me: Understanding People's Receptivity to Mobile Notifications CHI. 1021--1032.","DOI":"10.1145\/2858036.2858566"},{"key":"e_1_3_2_1_18_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2911494"},{"key":"e_1_3_2_1_19_1","doi-asserted-by":"crossref","unstructured":"Filip Radlinski and Nick Craswell 2010. Comparing the Sensitivity of Information Retrieval Metrics SIGIR. 667--674. Filip Radlinski and Nick Craswell 2010. Comparing the Sensitivity of Information Retrieval Metrics SIGIR. 667--674.","DOI":"10.1145\/1835449.1835560"},{"key":"e_1_3_2_1_20_1","doi-asserted-by":"crossref","unstructured":"Filip Radlinski and Nick Craswell. 2013. Optimized Interleaving for Online Retrieval Evaluation WSDM. 245--254. Filip Radlinski and Nick Craswell. 2013. Optimized Interleaving for Online Retrieval Evaluation WSDM. 245--254.","DOI":"10.1145\/2433396.2433429"},{"volume-title":"The TREC 2002 Filtering Track Report. In TREC.","year":"2002","author":"Robertson Stephen","key":"e_1_3_2_1_21_1"},{"volume-title":"de Vries","year":"2013","author":"Said Alan","key":"e_1_3_2_1_22_1"},{"key":"e_1_3_2_1_23_1","doi-asserted-by":"crossref","unstructured":"Mark Sanderson Monica Paramita Paul Clough and Evangelos Kanoulas. 2010. Do User Preferences and Evaluation Measures Line Up? SIGIR. 555--562. Mark Sanderson Monica Paramita Paul Clough and Evangelos Kanoulas. 2010. Do User Preferences and Evaluation Measures Line Up? SIGIR. 555--562.","DOI":"10.1145\/1835449.1835542"},{"volume-title":"Overview of the Living Labs for Information Retrieval Evaluation (LL4IR) CLEF Lab","year":"2015","author":"Schuth Anne","key":"e_1_3_2_1_24_1"},{"key":"e_1_3_2_1_25_1","doi-asserted-by":"crossref","unstructured":"Anne Schuth Katja Hofmann and Filip Radlinski. 2015. Predicting Search Satisfaction Metrics with Interleaved Comparisons SIGIR. 463--472. Anne Schuth Katja Hofmann and Filip Radlinski. 2015. Predicting Search Satisfaction Metrics with Interleaved Comparisons SIGIR. 463--472.","DOI":"10.1145\/2766462.2767695"},{"key":"e_1_3_2_1_26_1","doi-asserted-by":"crossref","unstructured":"Mark Smucker and Chandra Jethani. 2010. Human Performance and Retrieval Precision Revisited SIGIR. 595--602. Mark Smucker and Chandra Jethani. 2010. Human Performance and Retrieval Precision Revisited SIGIR. 595--602.","DOI":"10.1145\/1835449.1835549"},{"key":"e_1_3_2_1_27_1","doi-asserted-by":"crossref","unstructured":"Ian Soboroff Iadh Ounis Craig Macdonald and Jimmy Lin. 2012. Overview of the TREC-2012 Microblog Track. In TREC. Ian Soboroff Iadh Ounis Craig Macdonald and Jimmy Lin. 2012. Overview of the TREC-2012 Microblog Track. In TREC.","DOI":"10.6028\/NIST.SP.500-298.microblog-overview"},{"volume-title":"Clarke","year":"2016","author":"Tan Luchen","key":"e_1_3_2_1_28_1"},{"volume-title":"Hersh","year":"2001","author":"Turpin Andrew","key":"e_1_3_2_1_29_1"},{"key":"e_1_3_2_1_30_1","doi-asserted-by":"crossref","unstructured":"Andrew Turpin and Falk Scholer. 2006. User Performance versus Precision Measures for Simple Search Tasks SIGIR. 11--18. Andrew Turpin and Falk Scholer. 2006. User Performance versus Precision Measures for Simple Search Tasks SIGIR. 11--18.","DOI":"10.1145\/1148170.1148176"},{"key":"e_1_3_2_1_31_1","doi-asserted-by":"crossref","unstructured":"Yulu Wang Garrick Sherman Jimmy Lin and Miles Efron. 2015. Assessor Differences and User Preferences in Tweet Timeline Generation SIGIR. 615--624. Yulu Wang Garrick Sherman Jimmy Lin and Miles Efron. 2015. Assessor Differences and User Preferences in Tweet Timeline Generation SIGIR. 615--624.","DOI":"10.1145\/2766462.2767699"}],"event":{"name":"SIGIR '17: The 40th International ACM SIGIR conference on research and development in Information Retrieval","sponsor":["SIGIR ACM Special Interest Group on Information Retrieval"],"location":"Shinjuku Tokyo Japan","acronym":"SIGIR '17"},"container-title":["Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval"],"original-title":[],"link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3077136.3080808","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3077136.3080808","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,24]],"date-time":"2025-06-24T18:42:03Z","timestamp":1750790523000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3077136.3080808"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2017,8,7]]},"references-count":31,"alternative-id":["10.1145\/3077136.3080808","10.1145\/3077136"],"URL":"https:\/\/doi.org\/10.1145\/3077136.3080808","relation":{},"subject":[],"published":{"date-parts":[[2017,8,7]]},"assertion":[{"value":"2017-08-07","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}