{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,2,21]],"date-time":"2025-02-21T22:16:32Z","timestamp":1740176192601,"version":"3.37.3"},"reference-count":28,"publisher":"Springer Science and Business Media LLC","issue":"4","license":[{"start":{"date-parts":[[2021,9,9]],"date-time":"2021-09-09T00:00:00Z","timestamp":1631145600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,9,9]],"date-time":"2021-09-09T00:00:00Z","timestamp":1631145600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Data Sci. Eng."],"published-print":{"date-parts":[[2021,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Evaluation of interactive search systems and study of users\u2019 struggling search behaviors require a significant number of search tasks. However, generation of such tasks is inherently difficult, as each task is supposed to trigger struggling search behavior rather than simple search behavior. To the best of our knowledge, there has not been a commonly used task set for research in struggling search. Moreover, the everchanging landscape of information needs would render old task sets less ideal if not unusable for evaluation. To deal with this problem, we propose a crowd-powered task generation method and develop a platform to efficiently generate struggling search tasks on basis of online wikis such as Wikipedia. Our experiments and analysis show that the generated tasks are qualified to emulate struggling search behaviors consisting of \u201crepeated similar queries\u201d and \u201cquick-back clicks\u201d; tasks of diverse topics, high quality and difficulty can be created using this method. For benefit of the community, we publicly released a task generation platform <jats:italic>TaskGenie<\/jats:italic>, a task set of 80 topically diverse struggling search tasks with \u201cbaselines,\u201d and the corresponding anonymized user behavior logs.<\/jats:p>","DOI":"10.1007\/s41019-021-00171-3","type":"journal-article","created":{"date-parts":[[2021,9,9]],"date-time":"2021-09-09T16:07:25Z","timestamp":1631203645000},"page":"472-484","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["A Crowd-Powered Task Generation Method for Study of Struggling Search"],"prefix":"10.1007","volume":"6","author":[{"ORCID":"https:\/\/orcid.org\/0000-0001-7093-2477","authenticated-orcid":false,"given":"Luyan","family":"Xu","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xuan","family":"Zhou","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2021,9,9]]},"reference":[{"key":"171_CR1","doi-asserted-by":"crossref","unstructured":"Ageev M, Guo Q, Lagun D, Agichtein E (2011) Find it if you can: a game for modeling different types of web search success using interaction data. In: SIGIR, pp. 345\u2013354","DOI":"10.1145\/2009916.2009965"},{"key":"171_CR2","doi-asserted-by":"crossref","unstructured":"Aula A, Khan RM, Guan Z (2010) How does search behavior change as search becomes more difficult? In: SIGCHI, pp. 35\u201344","DOI":"10.1145\/1753326.1753333"},{"key":"171_CR3","doi-asserted-by":"crossref","unstructured":"Baykan E, Henzinger M, Marian L, Weber I (2009) Purely url-based topic classification. In: WWW, pp. 1109\u20131110","DOI":"10.1145\/1526709.1526880"},{"issue":"3","key":"171_CR4","doi-asserted-by":"publisher","first-page":"463","DOI":"10.1162\/COLI_a_00166","volume":"39","author":"R Bhagat","year":"2013","unstructured":"Bhagat R, Hovy E (2013) What is a paraphrase? Comput. Linguistics 39(3):463\u2013472","journal-title":"Comput. Linguistics"},{"key":"171_CR5","doi-asserted-by":"crossref","unstructured":"Braarud P\u00d8, Kirwan B (2010) Task complexity: what challenges the crew and how do they cope. In: Simulator-based Human Factors Studies Across 25 Years, pp 233\u2013251. Springer","DOI":"10.1007\/978-0-85729-003-8_15"},{"key":"171_CR6","doi-asserted-by":"crossref","unstructured":"Capra R, Arguello J, O\u2019Brien H, Li Y, Choi B (2018) The effects of manipulating task determinability on search behaviors and outcomes. In: SIGIR, pp 445\u2013454","DOI":"10.1145\/3209978.3210047"},{"key":"171_CR7","doi-asserted-by":"publisher","DOI":"10.4324\/9781315835839","volume-title":"Introduction to text linguistics","author":"RA De Beaugrande","year":"1981","unstructured":"De Beaugrande RA, Dressler WU (1981) Introduction to text linguistics, vol 1. Longman, London"},{"key":"171_CR8","doi-asserted-by":"crossref","unstructured":"Gadiraju U, M\u00f6ller S, N\u00f6llenburg M, Saupe D, Egger-Lampl S, Archambault D, Fisher B (2017) Crowdsourcing versus the laboratory: towards human-centered experiments using the crowd. In: Evaluation in the crowd. Crowdsourcing and human-centered experiments, pp 6\u201326. Springer","DOI":"10.1007\/978-3-319-66435-4_2"},{"key":"171_CR9","doi-asserted-by":"crossref","unstructured":"Gadiraju U, Yu R, Dietze S, Holtz P (2018) Analyzing knowledge gain of users in informational search sessions on the web. In: CHIIR 2018","DOI":"10.1145\/3176349.3176381"},{"key":"171_CR10","doi-asserted-by":"crossref","unstructured":"Hassan A, Jones R, Klinkner KL (2010) Beyond dcg: user behavior as a predictor of a successful search. In: WSDM, pp 221\u2013230","DOI":"10.1145\/1718487.1718515"},{"key":"171_CR11","doi-asserted-by":"crossref","unstructured":"Hassan A, White RW, Dumais ST, Wang YM (2014) Struggling or exploring?: disambiguating long search sessions. In: WSDM, pp 53\u201362","DOI":"10.1145\/2556195.2556221"},{"key":"171_CR12","doi-asserted-by":"crossref","unstructured":"Hassan\u00a0Awadallah A, White RW, Pantel P, Dumais ST, Wang YM (2014) Supporting complex search tasks. In: CIKM, pp 829\u2013838","DOI":"10.1145\/2661829.2661912"},{"issue":"1","key":"171_CR13","first-page":"1","volume":"43","author":"M Kellar","year":"2006","unstructured":"Kellar M, Watters C, Shepherd M (2006) A goal-based classification of web information tasks. ASIST 43(1):1\u201322","journal-title":"ASIST"},{"key":"171_CR14","doi-asserted-by":"crossref","unstructured":"Kenter T, De\u00a0Rijke M (2015) Short text similarity with word embeddings. In: CIKM, pp 1411\u20131420","DOI":"10.1145\/2806416.2806475"},{"key":"171_CR15","doi-asserted-by":"crossref","unstructured":"Kim Y, Hassan A, White RW, Zitouni I (2014) Modeling dwell time to predict click-level satisfaction. In: WSDM, pp 193\u2013202","DOI":"10.1145\/2556195.2556220"},{"key":"171_CR16","volume-title":"Understanding english grammar","author":"M Kolln","year":"1982","unstructured":"Kolln M, Funk R (1982) Understanding english grammar. Longman, London"},{"issue":"1","key":"171_CR17","first-page":"1","volume":"49","author":"C Liu","year":"2012","unstructured":"Liu C, Liu J, Cole M, Belkin NJ, Zhang X (2012) Task difficulty and domain knowledge effects on information search behaviors. ASIS&T 49(1):1\u201310","journal-title":"ASIS&T"},{"key":"171_CR18","doi-asserted-by":"crossref","unstructured":"Liu J, Kim CS, Creel C (2013) Why do users feel search task difficult? In: The 76th ASIS&T. American Society for Information Science","DOI":"10.1002\/meet.14505001125"},{"key":"171_CR19","volume-title":"Looking for information: a survey of research on information seeking, needs, and behavior","author":"JE Mai","year":"2016","unstructured":"Mai JE (2016) Looking for information: a survey of research on information seeking, needs, and behavior. Emerald Group Publishing, Bingley"},{"key":"171_CR20","doi-asserted-by":"crossref","unstructured":"Odijk D, White RW, Hassan\u00a0Awadallah A, Dumais ST (2015) Struggling and success in web search. In: CIKM, pp 1551\u20131560","DOI":"10.1145\/2806416.2806488"},{"key":"171_CR21","first-page":"2","volume":"5","author":"P Pirolli","year":"2005","unstructured":"Pirolli P, Card S (2005) The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. IA 5:2\u20134","journal-title":"IA"},{"issue":"1","key":"171_CR22","doi-asserted-by":"publisher","first-page":"27","DOI":"10.1093\/applin\/22.1.27","volume":"22","author":"P Robinson","year":"2001","unstructured":"Robinson P (2001) Task complexity, task difficulty, and task production: exploring interactions in a componential framework. Appl Linguistics 22(1):27\u201357","journal-title":"Appl Linguistics"},{"issue":"3","key":"171_CR23","doi-asserted-by":"publisher","first-page":"346","DOI":"10.1177\/0165551512466974","volume":"39","author":"G Singer","year":"2013","unstructured":"Singer G, Norbisrath U, Lewandowski D (2013) Ordinary search engine users carrying out complex search tasks. J Inf Sci 39(3):346\u2013358","journal-title":"J Inf Sci"},{"key":"171_CR24","doi-asserted-by":"crossref","unstructured":"Singer P, Lemmerich F, West R, Zia L, Wulczyn E, Strohmaier M, Leskovec J (2017) Why we read wikipedia. In: WWW, pp 1591\u20131600","DOI":"10.1145\/3038912.3052716"},{"key":"171_CR25","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9781139525305","volume-title":"Interactions with search systems","author":"RW White","year":"2016","unstructured":"White RW (2016) Interactions with search systems. Cambridge University Press, Cambridge"},{"issue":"1","key":"171_CR26","first-page":"1","volume":"1","author":"RW White","year":"2009","unstructured":"White RW, Roth RA (2009) Exploratory search: beyond the query-response paradigm. Synth Lect Inf Concepts Retrieval Serv 1(1):1\u201398","journal-title":"Synth Lect Inf Concepts Retrieval Serv"},{"key":"171_CR27","doi-asserted-by":"crossref","unstructured":"Xu L, Zhou X (2019) Generating tasks for study of struggling search. In: Proceedings of the 2019 conference on human information interaction and retrieval, pp 267\u2013270","DOI":"10.1145\/3295750.3298949"},{"key":"171_CR28","doi-asserted-by":"crossref","unstructured":"Xu L, Zhou X, Gadiraju U (2020) Taskgenie: crowd-powered task generation for struggling search. In: International conference on web information systems engineering, pp 3\u201320. Springer","DOI":"10.1007\/978-3-030-62008-0_1"}],"container-title":["Data Science and Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s41019-021-00171-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s41019-021-00171-3\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s41019-021-00171-3.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2021,10,27]],"date-time":"2021-10-27T17:32:57Z","timestamp":1635355977000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s41019-021-00171-3"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,9,9]]},"references-count":28,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2021,12]]}},"alternative-id":["171"],"URL":"https:\/\/doi.org\/10.1007\/s41019-021-00171-3","relation":{},"ISSN":["2364-1185","2364-1541"],"issn-type":[{"type":"print","value":"2364-1185"},{"type":"electronic","value":"2364-1541"}],"subject":[],"published":{"date-parts":[[2021,9,9]]},"assertion":[{"value":"27 April 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"25 July 2021","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"27 August 2021","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 September 2021","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no known competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}