{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,16]],"date-time":"2026-07-16T05:14:36Z","timestamp":1784178876417,"version":"3.55.0"},"reference-count":50,"publisher":"Association for Computing Machinery (ACM)","issue":"1","license":[{"start":{"date-parts":[[2024,6,1]],"date-time":"2024-06-01T00:00:00Z","timestamp":1717200000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["SIGIR Forum"],"published-print":{"date-parts":[[2024,6]]},"abstract":"<jats:p>This article reviews how empirical research of exploratory search is conducted. We investigated aspects of interdisciplinarity, study settings and evaluation methodologies from a systematically selected sample of 231 publications, including a total of 172 articles with empirical studies. Our results show that exploratory search is highly interdisciplinary, with the most frequently occurring publication venues including high impact venues in information science, information systems and human-computer interaction. However, taken in aggregate, the breadth of study settings investigated was limited. We found that a majority of studies (77%) focused on evaluating novel retrieval systems as opposed to investigating users' search processes. Furthermore, a disproportionate number of studies were based on scientific literature search (20.7%), a majority of which only considered searching for Computer Science articles. Study participants were generally from convenience samples, with 75% of studies composed exclusively of students and other academics. The methodologies used for evaluation were mostly quantitative, but lacked consistency between studies and validated questionnaires were rarely used. In discussion, we offer a critical analysis of our findings and suggest potential improvements for future exploratory search studies.<\/jats:p>","DOI":"10.1145\/3687273.3687278","type":"journal-article","created":{"date-parts":[[2024,8,7]],"date-time":"2024-08-07T22:24:53Z","timestamp":1723069493000},"page":"1-19","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Unexplored Frontiers: A Review of Empirical Studies of Exploratory Search"],"prefix":"10.1145","volume":"58","author":[{"given":"Alan","family":"Medlar","sequence":"first","affiliation":[{"name":"University of Helsinki, Finland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Denis","family":"Kotkov","sequence":"additional","affiliation":[{"name":"University of Helsinki, Finland"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dorota","family":"G\u0142owacka","sequence":"additional","affiliation":[{"name":"University of Helsinki, Finland"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,8,7]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1002\/asi.23617"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2856767.2856786"},{"key":"e_1_2_1_3_1","volume-title":"The design of browsing and berrypicking techniques for the online search interface. Online review","author":"Bates Marcia J","year":"1989","unstructured":"Marcia J Bates. The design of browsing and berrypicking techniques for the online search interface. Online review, 1989."},{"key":"e_1_2_1_4_1","doi-asserted-by":"publisher","DOI":"10.1108\/eb026722"},{"issue":"3","key":"e_1_2_1_5_1","first-page":"8","article-title":"The IIR evaluation model: A framework for evaluation of interactive information retrieval systems","volume":"8","author":"Borlund Pia","year":"2003","unstructured":"Pia Borlund. The IIR evaluation model: A framework for evaluation of interactive information retrieval systems. Information Research, 8(3):8--3, 2003.","journal-title":"Information Research"},{"issue":"3","key":"e_1_2_1_6_1","first-page":"189","article-title":"A \"quick and dirty\" usability scale","volume":"189","author":"Brooke John","year":"1996","unstructured":"John Brooke. SUS: A \"quick and dirty\" usability scale. Usability Evaluation in Industry, 189(3): 189--194, 1996.","journal-title":"Usability Evaluation in Industry"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10639-019-09921-3"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.2196\/jmir.3111"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1108\/13673279810249369"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/3172944.3172986"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3159652.3159661"},{"key":"e_1_2_1_12_1","volume-title":"HCIR 2010","author":"Diriye Abdigani","year":"2010","unstructured":"Abdigani Diriye, Max L Wilson, Ann Blandford, and Anastasios Tombros. Revisiting exploratory search from the HCI perspective. HCIR 2010, 99, 2010."},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/1840784.1840810"},{"key":"e_1_2_1_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/2637002.2637012"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2449396.2449413"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2017.04.001"},{"key":"e_1_2_1_17_1","volume-title":"The weirdest people in the world? Behavioral and Brain Sciences, 33(2--3):61--83","author":"Henrich Joseph","year":"2010","unstructured":"Joseph Henrich, Steven J Heine, and Ara Norenzayan. The weirdest people in the world? Behavioral and Brain Sciences, 33(2--3):61--83, 2010."},{"key":"e_1_2_1_18_1","volume-title":"Methods for evaluating interactive information retrieval systems with users. Foundations and Trends\u00ae in Information Retrieval, 3(1--2):1--224","author":"Kelly Diane","year":"2009","unstructured":"Diane Kelly. Methods for evaluating interactive information retrieval systems with users. Foundations and Trends\u00ae in Information Retrieval, 3(1--2):1--224, 2009."},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1145\/3020165.3020180"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1145\/3459637.3482103"},{"key":"e_1_2_1_21_1","first-page":"18","volume-title":"HCIR 2008","author":"Kules Bill","year":"2008","unstructured":"Bill Kules and Robert Capra. Creating exploratory tasks for a faceted search interface. HCIR 2008, pages 18--21, 2008."},{"key":"e_1_2_1_22_1","first-page":"159","volume-title":"Biometrics","author":"Richard Landis J.","year":"1977","unstructured":"J. Richard Landis and Gary G Koch. The measurement of observer agreement for categorical data. Biometrics, pages 159--174, 1977."},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.3233\/SW-140134"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.5555\/1855210.1855217"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1145\/1835449.1835457"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1145\/3477495.3531675"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1080\/01972243.1984.9959994"},{"issue":"1","key":"e_1_2_1_28_1","first-page":"35","article-title":"Information seeking in full-text end-user-oriented search systems: The roles of domain and search expertise","volume":"15","author":"Marchionini Gary","year":"1993","unstructured":"Gary Marchionini. Information seeking in full-text end-user-oriented search systems: The roles of domain and search expertise. Library & Information Science Research, 15(1):35--69, 1993.","journal-title":"Library & Information Science Research"},{"key":"e_1_2_1_29_1","doi-asserted-by":"publisher","DOI":"10.1145\/1121949.1121979"},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1145\/3269206.3269297"},{"key":"e_1_2_1_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/2911451.2911455"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3025171.3025205"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3406522.3446020"},{"key":"e_1_2_1_34_1","first-page":"1141","volume-title":"The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval","author":"Mitsui Matthew","year":"2018","unstructured":"Matthew Mitsui, Jiqun Liu, and Chirag Shah. How much is too much? Whole session vs. first query behaviors in task type prediction. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 1141--1144, 2018."},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1016\/S0140-6736(99)04149-5"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ipm.2012.08.005"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.ijhcs.2003.08.005"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3038462.3038465"},{"key":"e_1_2_1_39_1","doi-asserted-by":"publisher","DOI":"10.1002\/(SICI)1097-4571(2000)51:6<558::AID-ASI7>3.0.CO;2-9"},{"key":"e_1_2_1_40_1","doi-asserted-by":"publisher","DOI":"10.2196\/jmir.5661"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.1145\/2043932.2043962"},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1145\/2484028.2484084"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1177\/0165551517713168"},{"key":"e_1_2_1_44_1","volume-title":"Coagmento-a collaborative information seeking, synthesis and sense-making framework. Integrated demo at CSCW","author":"Shah Chirag","year":"2010","unstructured":"Chirag Shah. Coagmento-a collaborative information seeking, synthesis and sense-making framework. Integrated demo at CSCW, 2010."},{"key":"e_1_2_1_45_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-39200-9_31"},{"key":"e_1_2_1_46_1","first-page":"359","volume-title":"International Workshop of the Initiative for the Evaluation of XML Retrieval","author":"Toms Elaine G.","year":"2007","unstructured":"Elaine G. Toms, Heather O'Brien, Tayze Mackenzie, Chris Jordan, Luanne Freund, Sandra Toze, Emilie Dawe, and Alexandra Macnutt. Task effects on interactive search: The query factor. In International Workshop of the Initiative for the Evaluation of XML Retrieval, pages 359--372. Springer, 2007."},{"key":"e_1_2_1_47_1","volume-title":"Analyzing the past to prepare for the future: Writing a literature review. MIS Quarterly","author":"Webster Jane","year":"2002","unstructured":"Jane Webster and Richard T. Watson. Analyzing the past to prepare for the future: Writing a literature review. MIS Quarterly, pages xiii--xxiii, 2002."},{"key":"e_1_2_1_48_1","volume-title":"Exploratory search: Beyond the query-response paradigm. Synthesis Lectures on Information Concepts, Retrieval, and Services, 1(1):1--98","author":"White Ryen W.","year":"2009","unstructured":"Ryen W. White and Resa A. Roth. Exploratory search: Beyond the query-response paradigm. Synthesis Lectures on Information Concepts, Retrieval, and Services, 1(1):1--98, 2009."},{"key":"e_1_2_1_49_1","first-page":"1","volume-title":"Proceedings of the Symposium on Human-Computer Interaction and Information Retrieval","author":"Barbara","year":"2012","unstructured":"Barbara M. Wildemuth and Luanne Freund. Assigning search tasks designed to elicit exploratory search behaviors. In Proceedings of the Symposium on Human-Computer Interaction and Information Retrieval, pages 1--10, 2012."},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1145\/2362724.2362768"}],"container-title":["ACM SIGIR Forum"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3687273.3687278","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3687273.3687278","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T01:10:01Z","timestamp":1750295401000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3687273.3687278"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,6]]},"references-count":50,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2024,6]]}},"alternative-id":["10.1145\/3687273.3687278"],"URL":"https:\/\/doi.org\/10.1145\/3687273.3687278","relation":{},"ISSN":["0163-5840"],"issn-type":[{"value":"0163-5840","type":"print"}],"subject":[],"published":{"date-parts":[[2024,6]]},"assertion":[{"value":"2024-08-07","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}