{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:20:05Z","timestamp":1750220405417,"version":"3.41.0"},"reference-count":89,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2021,11,22]],"date-time":"2021-11-22T00:00:00Z","timestamp":1637539200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/501100000923","name":"Australian Research Council ARC","doi-asserted-by":"crossref","award":["LP130100563 ARC, DP190101113 ARC, and DE200100064"],"award-info":[{"award-number":["LP130100563 ARC, DP190101113 ARC, and DE200100064"]}],"id":[{"id":"10.13039\/501100000923","id-type":"DOI","asserted-by":"crossref"}]},{"name":"Real Thing Entertainment Pty Ltd."}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Inf. Syst."],"published-print":{"date-parts":[[2022,7,31]]},"abstract":"<jats:p>In many search scenarios, such as exploratory, comparative, or survey-oriented search, users interact with dynamic search systems to satisfy multi-aspect information needs. These systems utilize different dynamic approaches that exploit various user feedback granularity types. Although studies have provided insights about the role of many components of these systems, they used black-box and isolated experimental setups. Therefore, the effects of these components or their interactions are still not well understood. We address this by following a methodology based on Analysis of Variance (ANOVA). We built a Grid Of Points that consists of systems based on different ways to instantiate three components: initial rankers, dynamic rerankers, and user feedback granularity. Using evaluation scores based on the TREC Dynamic Domain collections, we built several ANOVA models to estimate the effects. We found that (i) although all components significantly affect search effectiveness, the initial ranker has the largest effective size, (ii) the effect sizes of these components vary based on the length of the search session and the used effectiveness metric, and (iii) initial rankers and dynamic rerankers have more prominent effects than user feedback granularity. To improve effectiveness, we recommend improving the quality of initial rankers and dynamic rerankers. This does not require eliciting detailed user feedback, which might be expensive or invasive.<\/jats:p>","DOI":"10.1145\/3483237","type":"journal-article","created":{"date-parts":[[2021,11,22]],"date-time":"2021-11-22T17:41:55Z","timestamp":1637602915000},"page":"1-47","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Component-based Analysis of Dynamic Search Performance"],"prefix":"10.1145","volume":"40","author":[{"given":"Ameer","family":"Albahem","sequence":"first","affiliation":[{"name":"RMIT University, Melbourne, Australia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9913-433X","authenticated-orcid":false,"given":"Damiano","family":"Spina","sequence":"additional","affiliation":[{"name":"RMIT University, Melbourne, Australia"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9094-0810","authenticated-orcid":false,"given":"Falk","family":"Scholer","sequence":"additional","affiliation":[{"name":"RMIT University, Melbourne, Australia"}]},{"given":"Lawrence","family":"Cavedon","sequence":"additional","affiliation":[{"name":"RMIT University, Melbourne, Australia"}]}],"member":"320","published-online":{"date-parts":[[2021,11,22]]},"reference":[{"doi-asserted-by":"publisher","key":"e_1_3_5_2_2","DOI":"10.5555\/2931100"},{"doi-asserted-by":"publisher","key":"e_1_3_5_3_2","DOI":"10.1145\/1498759.1498766"},{"doi-asserted-by":"publisher","key":"e_1_3_5_4_2","DOI":"10.1145\/3121050.3121059"},{"key":"e_1_3_5_5_2","first-page":"607","volume-title":"Proc. ECIR","author":"Albahem Ameer","year":"2019","unstructured":"Ameer Albahem , Damiano Spina , Falk Scholer , and Lawrence Cavedon . 2019. Meta-evaluation of dynamic search: How do metrics capture topical relevance, diversity and user effort? In Proc. ECIR . Springer International Publishing, Cham, 607\u2013620."},{"doi-asserted-by":"publisher","key":"e_1_3_5_6_2","DOI":"10.1145\/3291992.3291996"},{"doi-asserted-by":"publisher","key":"e_1_3_5_7_2","DOI":"10.1145\/582415.582416"},{"doi-asserted-by":"publisher","key":"e_1_3_5_8_2","DOI":"10.1145\/1645953.1646031"},{"doi-asserted-by":"publisher","key":"e_1_3_5_9_2","DOI":"10.1145\/3357375"},{"key":"e_1_3_5_10_2","volume-title":"Proc. TREC","author":"Bo Zhou","year":"2008","unstructured":"Zhou Bo , Qi Fang , Rongwei Cen , Min Zhang , Yiqun Liu , and Shaoping Ma . 2008. THUIR at TREC2008: Relevance feedback track1. In Proc. TREC , Vol. 500-277. National Institute of Standards and Technology (NIST), Gaithersburg, MD."},{"doi-asserted-by":"publisher","key":"e_1_3_5_11_2","DOI":"10.1145\/2911451.2914695"},{"key":"e_1_3_5_12_2","volume-title":"Proc. TREC","author":"Buckley Chris","year":"2008","unstructured":"Chris Buckley and Robertson Robertson . 2008. Relevance feedback track overview: TREC 2008. In Proc. TREC . National Institute of Standards and Technology (NIST), Gaithersburg, MD."},{"doi-asserted-by":"publisher","key":"e_1_3_5_13_2","DOI":"10.1145\/290941.291025"},{"key":"e_1_3_5_14_2","volume-title":"Proc. TREC","author":"Carterette Ben","year":"2014","unstructured":"Ben Carterette , Evangelos Kanoulas , Mark Hall , and Paul Clough . 2014. Overview of the TREC 2014 session track. In Proc. TREC . National Institute of Standards and Technology (NIST), Gaithersburg, MD."},{"doi-asserted-by":"publisher","key":"e_1_3_5_15_2","DOI":"10.1145\/2094072.2094076"},{"doi-asserted-by":"publisher","key":"e_1_3_5_16_2","DOI":"10.1145\/1645953.1646033"},{"doi-asserted-by":"publisher","key":"e_1_3_5_17_2","DOI":"10.1145\/3357384.3358158"},{"doi-asserted-by":"publisher","key":"e_1_3_5_18_2","DOI":"10.1145\/3397271.3401200"},{"doi-asserted-by":"publisher","key":"e_1_3_5_19_2","DOI":"10.1145\/1390334.1390446"},{"key":"e_1_3_5_20_2","volume-title":"Proc. TREC","author":"Clarke Charles L. A.","year":"2012","unstructured":"Charles L. A. Clarke , Nick Craswell , and Ellen M. Voorhees . 2012. Overview of the TREC 2012 web track. In Proc. TREC . National Institute of Standards and Technology (NIST), Gaithersburg, MD."},{"doi-asserted-by":"publisher","key":"e_1_3_5_21_2","DOI":"10.1145\/3404835.3462804"},{"doi-asserted-by":"publisher","key":"e_1_3_5_22_2","DOI":"10.1145\/2348283.2348296"},{"doi-asserted-by":"publisher","key":"e_1_3_5_23_2","DOI":"10.1145\/2484028.2484095"},{"key":"e_1_3_5_24_2","first-page":"33","volume-title":"Proc. ECIR","author":"Faggioli Guglielmo","year":"2021","unstructured":"Guglielmo Faggioli and Nicola Ferro . 2021. System effect estimation by sharding: A comparison between ANOVA approaches to detect significant differences. In Proc. ECIR . Springer International Publishing, Cham, 33\u201346."},{"unstructured":"Marco\n\t\t\t\t\t\t\tFerrante Nicola\n\t\t\t\t\t\t\tFerro and \n\t\t\t\t\t\t\tNorbert\n\t\t\t\t\t\t\tFuhr\n\t\t\t\t\t. 2021. Towards Meaningful Statements in IR Evaluation. Mapping Evaluation Measures to Interval Scales. \n\t\t\t\t\t\tarxiv:2101.02668\n\t\t\t\t\t [cs.IR]. Retrieved from https:\/\/arxiv.org\/abs\/2101.02668.","key":"e_1_3_5_25_2"},{"doi-asserted-by":"publisher","key":"e_1_3_5_26_2","DOI":"10.1145\/3331184.3338062"},{"doi-asserted-by":"publisher","key":"e_1_3_5_27_2","DOI":"10.1145\/2911451.2911530"},{"doi-asserted-by":"publisher","key":"e_1_3_5_28_2","DOI":"10.1002\/asi.23910"},{"doi-asserted-by":"publisher","key":"e_1_3_5_29_2","DOI":"10.5555\/2040369.2040406"},{"doi-asserted-by":"publisher","key":"e_1_3_5_30_2","DOI":"10.1145\/312624.312637"},{"doi-asserted-by":"publisher","key":"e_1_3_5_31_2","DOI":"10.1007\/s10791-009-9101-4"},{"doi-asserted-by":"publisher","key":"e_1_3_5_32_2","DOI":"10.1145\/2806416.2806455"},{"key":"e_1_3_5_33_2","series-title":"Handbook of Statistics","doi-asserted-by":"crossref","first-page":"199","DOI":"10.1016\/S0169-7161(80)01009-7","volume-title":"Analysis of Variance","author":"Ito P. K.","year":"1980","unstructured":"P. K. Ito . 1980. 7 robustness of ANOVA and MANOVA test procedures. In Analysis of Variance . Handbook of Statistics, Vol. 1. Elsevier, Amsterdam, 199\u2013236."},{"doi-asserted-by":"publisher","key":"e_1_3_5_34_2","DOI":"10.1145\/3077136.3080840"},{"doi-asserted-by":"publisher","key":"e_1_3_5_35_2","DOI":"10.1145\/3077136.3080805"},{"doi-asserted-by":"publisher","key":"e_1_3_5_36_2","DOI":"10.1145\/2488388.2488446"},{"key":"e_1_3_5_37_2","volume-title":"Proc. TREC","author":"Jiyun Luo","year":"2015","unstructured":"Luo Jiyun and Hui Yang . 2015. Re-ranking via user feedback: Georgetown university at TREC 2015 DD track. In Proc. TREC . National Institute of Standards and Technology (NIST), Gaithersburg, MD."},{"key":"e_1_3_5_38_2","volume-title":"Proc. TREC","author":"Joganah Robin","year":"2015","unstructured":"Robin Joganah , Richard Khoury , and Luc Lamontagne . 2015. Laval university and lakehead university at TREC dynamic domain 2015: Combination of techniques for subtopics coverage. In Proc. TREC . National Institute of Standards and Technology (NIST), Gaithersburg, MD."},{"doi-asserted-by":"publisher","key":"e_1_3_5_39_2","DOI":"10.1145\/2009916.2010056"},{"doi-asserted-by":"publisher","key":"e_1_3_5_40_2","DOI":"10.1145\/2882782"},{"doi-asserted-by":"publisher","key":"e_1_3_5_41_2","DOI":"10.1145\/383952.383972"},{"doi-asserted-by":"publisher","key":"e_1_3_5_42_2","DOI":"10.1145\/502585.502592"},{"doi-asserted-by":"publisher","key":"e_1_3_5_43_2","DOI":"10.1145\/3077136.3080664"},{"doi-asserted-by":"publisher","key":"e_1_3_5_44_2","DOI":"10.1145\/2983323.2983816"},{"doi-asserted-by":"publisher","key":"e_1_3_5_45_2","DOI":"10.1145\/3404835.3463034"},{"doi-asserted-by":"publisher","key":"e_1_3_5_46_2","DOI":"10.1145\/3397271.3401084"},{"doi-asserted-by":"publisher","key":"e_1_3_5_47_2","DOI":"10.1145\/3341981.3344224"},{"doi-asserted-by":"publisher","key":"e_1_3_5_48_2","DOI":"10.1145\/2505515.2523648"},{"doi-asserted-by":"publisher","key":"e_1_3_5_49_2","DOI":"10.1145\/2600428.2609629"},{"doi-asserted-by":"publisher","key":"e_1_3_5_50_2","DOI":"10.1016\/j.ipm.2018.04.006"},{"doi-asserted-by":"publisher","key":"e_1_3_5_51_2","DOI":"10.1007\/s10791-019-09353-0"},{"doi-asserted-by":"publisher","key":"e_1_3_5_52_2","DOI":"10.1561\/1500000061"},{"key":"e_1_3_5_53_2","volume-title":"On Effective Dynamic Search Systems","author":"Moraes Felipe","year":"2017","unstructured":"Felipe Moraes . 2017. On Effective Dynamic Search Systems . Master thesis. University of Minas Gerais."},{"doi-asserted-by":"publisher","key":"e_1_3_5_54_2","DOI":"10.1145\/3121050.3121065"},{"key":"e_1_3_5_55_2","volume-title":"Proc. TREC","author":"Moraes Felipe","year":"2016","unstructured":"Felipe Moraes , Rodrygo L. T. Santos , and Nivio Ziviani . 2016. UFMG at the TREC 2016 dynamic domain track. In Proc. TREC . National Institute of Standards and Technology (NIST), Gaithersburg, MD."},{"key":"e_1_3_5_56_2","volume-title":"Proc. TREC","author":"Over Paul","year":"1999","unstructured":"Paul Over . 1999. TREC-7 interactive track report. In Proc. TREC (Gaithersburg, Maryland). National Institute of Standards and Technology (NIST), Gaithersburg, MD."},{"doi-asserted-by":"publisher","key":"e_1_3_5_57_2","DOI":"10.1145\/290941.291008"},{"doi-asserted-by":"publisher","key":"e_1_3_5_58_2","DOI":"10.1145\/3340531.3411914"},{"key":"e_1_3_5_59_2","volume-title":"Proc. TREC","author":"Rahimi Razieh","year":"2016","unstructured":"Razieh Rahimi and Grace Hui Yang . 2016. An investigation of basic retrieval models for the dynamic domain task. In Proc. TREC . National Institute of Standards and Technology (NIST), Gaithersburg, MD."},{"doi-asserted-by":"publisher","key":"e_1_3_5_60_2","DOI":"10.1145\/3360487"},{"doi-asserted-by":"publisher","key":"e_1_3_5_61_2","DOI":"10.1145\/2484028.2484089"},{"doi-asserted-by":"publisher","key":"e_1_3_5_62_2","DOI":"10.1145\/2063576.2063623"},{"key":"e_1_3_5_63_2","volume-title":"Relevance Feedback in Information Retrieval","author":"Rocchio J. J.","year":"1971","unstructured":"J. J. Rocchio . 1971. Relevance Feedback in Information Retrieval . Prentice Hall, USA."},{"doi-asserted-by":"publisher","key":"e_1_3_5_64_2","DOI":"10.1145\/3366424.3384362"},{"doi-asserted-by":"publisher","key":"e_1_3_5_65_2","DOI":"10.1145\/1772690.1772780"},{"doi-asserted-by":"publisher","key":"e_1_3_5_66_2","DOI":"10.1561\/1500000040"},{"doi-asserted-by":"publisher","key":"e_1_3_5_67_2","DOI":"10.1145\/1076034.1076047"},{"doi-asserted-by":"publisher","key":"e_1_3_5_68_2","DOI":"10.1145\/2808194.2809457"},{"key":"e_1_3_5_69_2","volume-title":"Proc. CHIIR","author":"S\u00f8rig Esben","year":"2019","unstructured":"Esben S\u00f8rig , Nicolas Collignon , Rebecca Fiebrink , and Noriko Kando . 2019. Evaluation of rich and explicit feedback for exploratory search. In Proc. CHIIR . ACM, New York, NY."},{"key":"e_1_3_5_70_2","first-page":"109","volume-title":"Proc. TREC","author":"Stephen Robertson","year":"1996","unstructured":"Robertson Stephen , S. Walker , S. Jones , M. M. Hancock-Beaulieu , and M. Gatford . 1996. Okapi at TREC-3. In Proc. TREC . National Institute of Standards and Technology (NIST), Gaithersburg, MD, 109\u2013126."},{"key":"e_1_3_5_71_2","volume-title":"Proc. TREC","author":"Tague-Sutcliffe J.","year":"1994","unstructured":"J. Tague-Sutcliffe and J. Blustein . 1994. A statistical analysis of the TREC-3 data. In Proc. TREC . National Institute of Standards and Technology (NIST), Gaithersburg, MD."},{"key":"e_1_3_5_72_2","volume-title":"Proc. TREC","author":"Tang Zhiwen","year":"2017","unstructured":"Zhiwen Tang and Grace Hui Yang . 2017. Georgetown university at TREC 2017 dynamic domain track. In Proc. TREC . National Institute of Standards and Technology (NIST), Gaithersburg, MD."},{"doi-asserted-by":"publisher","key":"e_1_3_5_73_2","DOI":"10.1145\/3086701"},{"doi-asserted-by":"publisher","key":"e_1_3_5_74_2","DOI":"10.1145\/3331184.3331233"},{"doi-asserted-by":"publisher","key":"e_1_3_5_75_2","DOI":"10.1145\/2766462.2767710"},{"doi-asserted-by":"publisher","key":"e_1_3_5_76_2","DOI":"10.1145\/2911451.2911498"},{"doi-asserted-by":"publisher","key":"e_1_3_5_77_2","DOI":"10.1145\/2983921"},{"doi-asserted-by":"publisher","key":"e_1_3_5_78_2","DOI":"10.1145\/3442381.3449831"},{"doi-asserted-by":"publisher","key":"e_1_3_5_79_2","DOI":"10.1145\/3121050.3121101"},{"key":"e_1_3_5_80_2","volume-title":"Proc. TREC","author":"Yang Hui","year":"2015","unstructured":"Hui Yang , John Frank , and Ian Soboroff . 2015. TREC 2015 dynamic domain track overview. In Proc. TREC . National Institute of Standards and Technology (NIST), Gaithersburg, MD."},{"key":"e_1_3_5_81_2","volume-title":"Proc. TREC","author":"Yang Hui","year":"2016","unstructured":"Hui Yang and Ian Soboroff . 2016. TREC 2016 dynamic domain track overview. In Proc. TREC . National Institute of Standards and Technology (NIST), Gaithersburg, MD."},{"key":"e_1_3_5_82_2","volume-title":"Proc. TREC","author":"Yang Hui","year":"2017","unstructured":"Hui Yang , Zhiwen Tang , and Ian Soboroff . 2017. TREC 2017 dynamic domain track overview. In Proc. TREC , Vol. 500-324. National Institute of Standards and Technology (NIST), Gaithersburg, MD."},{"doi-asserted-by":"publisher","key":"e_1_3_5_83_2","DOI":"10.1145\/3331184.3331279"},{"doi-asserted-by":"publisher","key":"e_1_3_5_84_2","DOI":"10.1145\/984321.984322"},{"key":"e_1_3_5_85_2","volume-title":"Proc. TREC","author":"Zhang Weimin","year":"2017","unstructured":"Weimin Zhang , Yong-Cheng Hu , Rongqian Jia , Xianfa Wang , Le Zhang , Yue Feng , Simon Chun Ho Yu , Yuanhai Xue , Xiaoming Yu , Yue Liu , and Xueqi Cheng . 2017. ICTNET at TREC 2017 dynamic domain track. In Proc. TREC . National Institute of Standards and Technology (NIST), Gaithersburg, MD."},{"doi-asserted-by":"publisher","key":"e_1_3_5_86_2","DOI":"10.1145\/3397271.3401467"},{"key":"e_1_3_5_87_2","volume-title":"Proc. TREC","author":"Zhao Le","year":"2008","unstructured":"Le Zhao , Chenmin Liang , and Jamie Callan . 2008. Extending relevance model for relevance feedback. In Proc. TREC . National Institute of Standards and Technology (NIST), Gaithersburg, MD, 500\u2013277."},{"key":"e_1_3_5_88_2","first-page":"2527","volume-title":"Proc. AAAI","author":"Zhiwen Tang","year":"2020","unstructured":"Tang Zhiwen and Grace Hui Yang . 2020. Corpus-level end-to-end exploration for interactive systems. In Proc. AAAI . AAAI Press, 2527\u20132534."},{"doi-asserted-by":"publisher","key":"e_1_3_5_89_2","DOI":"10.1145\/3366423.3380047"},{"key":"e_1_3_5_90_2","first-page":"293","volume-title":"Proc. SIGIR","author":"Zhu Yadong","year":"2014","unstructured":"Yadong Zhu , Yanyan Lan , Jiafeng Guo , Xueqi Cheng , and Shuzi Niu . 2014. Learning for search result diversification. In Proc. SIGIR . ACM, New York, NY, 293\u2013302."}],"container-title":["ACM Transactions on Information Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3483237","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3483237","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T20:18:57Z","timestamp":1750191537000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3483237"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,11,22]]},"references-count":89,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2022,7,31]]}},"alternative-id":["10.1145\/3483237"],"URL":"https:\/\/doi.org\/10.1145\/3483237","relation":{},"ISSN":["1046-8188","1558-2868"],"issn-type":[{"type":"print","value":"1046-8188"},{"type":"electronic","value":"1558-2868"}],"subject":[],"published":{"date-parts":[[2021,11,22]]},"assertion":[{"value":"2020-12-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-08-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-11-22","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}